In Washington DC last week Google CEO Eric Schmidt defended the company’s business practices when he appeared before a Senate antitrust panel. Down the road at Georgetown University the following day, his colleague Darcy Dapra was doing a similar thing to an audience of scholarly publishers.

Mr Schmidt’s appearance was to reject claims that Google, which celebrates its 13th birthday this week, gives its growing portfolio of online business preferred placements in search results.

According to panel member Richard Blumenthal , a Democrat Senator from Conneticut and the state’s attorney general: “Google is a great American success story, but its size, position and power in the marketplace have raised concerns about its business practices, and raised the question of what responsibilities come with that power.”

Scholarly journals have similar concerns about the power of the world’s favourite search engine. In February many were alarmed to learn that one of Google’s regular algorithm tweaks had triggered a downturn in traffic.

A Guardian technology blog listed bmj.com as one of the affected sites, adding: “The BMJ might somehow stagger on via its expensive subscriptions, but… changes to Google’s algorithm aren’t without collateral damage.”

The blog post generated a debate about how much collateral damage had or had not been done by the algorithm change. It also led to us reviewing how we allow Google to index BMJ content.

In a nutshell, if you allow Google’s automated web crawlers onto your site, you need to ensure that they see what a typical visitor to your site sees.

Author written extracts that show above the fold are fine (not every article has an abstract, of course, included research published decades ago).

But Google has a problem with paywalls and registration pages, particularly for news, editorials, letters, reviews, obituaries, and reviews, which are typically free for general publications. If you get it “wrong”, as the New England Journal of Medicine reportedly did in May, Google’s web search quality team de-indexes you.

Darcy outlined Google’s indexing guidelines when she spoke at last week’s publisher meeting. It was organised by HighWire Press,  Stanford University’s online hosting partner for hundreds of scholarly journals, including the BMJ and its stable of more than 30 specialist journals.

One publisher accused Google of “dictating how we run our business.” Darcy argued that Google “really wants people to get to your content” and it was “down to how discoverable you want to make your content.”

In essence, publishers face four options, all of which have business implications. We can:

• Block some or all content from the Google web crawler (not great for promoting discoverability)
• First click free. This means that if someone clicks on Google search result they hit the article rather than a paywall, but if they then browse within the site the paywall launches.
• Make the entire article free.
• Provide a free pdf of the first page, which, if the article is only one page, means you’re making it free.

The BMJ decided to remove registration to its free online archive after meeting Darcy earlier this year.

But we had a separate (and extensive) discussion about the how important Google traffic is to bmj.com. We are not a consumer publication. Our target audience is doctors, most of whom access bmj.com via a personal or institutional subscription, or because their BMA membership includes free access. Does it matter, therefore, if our Google ranking goes down?

We concluded that Google is important to us, as it is to other organisations. The New York Times’ article about Schmidt’s Senate committee appearance notes that the company is “both admired and feared.”

But Google inspires confidence also. Its 13-year history has been one of consistent innovation, including the arrival of Google Scholar in 2004 and the beta launch of a citation tool this year that allows authors to keep track of citations to their articles.

Many web users, whatever their background, now habitually use Google to find articles in favour of a publication’s, search engine. They do this even if they know the publication and have a rough idea when an article appeared and who authored it.

And the BMJ is an open access journal. All of its research is free, and much of this is press released because it is of interest to the wider public. Google helps the public locate links to that research.

Darcy inspired a similar confidence last week by listening to publishers’ concerns and feeling our pain. But regulators across the world, concerned that the company’s dominance could be stifling competition, means Mr Schmidt faces a tougher task. Will Google’s teens be terrible?

David Payne is editor, bmj.com