Publishers have more AI visibility options than they think

If you're publishing online, you may think you only have two options when it comes to AI and LLM visibility: give them everything, or give them nothing. A few years ago this was true, but the reality now is much more nuanced and, according to our research, hardly any publishers are taking advantage of the tools now at their disposal.

Your options are still not quite as plentiful as in traditional SEO, but when it comes to how AI uses your content, and how it's presented to users, you've got more to play with than you might expect.

Training and retrieval are now separate

For many large publishers, this has been the most meaningful change. Although the large AI labs were always cagey about how they scraped the web, and what they did with the content they scraped (resulting in many, many lawsuits), the received wisdom was always that if you let them in, they'll hoover up your articles and use them for training data.

For companies like online stores or SaaS providers, this wasn't necessarily a bad thing. They want LLMs to know about their products, and the writing on their website was primarily to promote those products. But for companies where the writing was the product, this behaviour was much less welcome. The unwritten agreement with Google has always been that you can index our writing if you send us visitors. ChatGPT, Claude and other LLMs are designed to keep users within their apps, so publishers didn't even benefit from more website traffic by allowing AI bots access.

As a result, many large publishers implemented blanket blocks for AI bots, many sued the developers of LLMs for using their writing without permission, and many have since struck licensing agreements where they are paid in return for their editorial being used as training data.

We were recently asked to investigate how a subset of large publishers operating in the UK are treating AI access, and our research found that almost half were completely closed to every single mainstream AI bot, with many choosing to block them at the CDN level.

As a result, most large AI labs have now broken out their crawlers to have distinct functions, giving publishers more control over what happens to their writing. For example OpenAI's bots now come in three distinct flavours:

• GPTBot, which scrapes web content to use for training data
• OAI-SearchBot, which searches on a user's behalf to provide an answer
• ChatGPT-User, which fetches a specific page on behalf of a user

So if a publisher doesn't want their content used to train ChatGPT, but does want ChatGPT to cite them, or access their editorial on behalf of a user, they can block GPTBot, but allow the other two. This strikes a good balance between protecting intellectual property, but still being visible to users.

At a minimum, businesses which make money from their writing should be blocking the main training bots. There's simply no reason to give your content away for free, except for altruism.

Google provides the finest-grained controls

Google's current tooling is the most mature, but also the most complex to understand. When configured correctly it allows publishers to control what their writing is used for, and where. We'll walk through our recommended setup for most publishers, which incidentally is exactly how this article is structured.

If you allow Googlebot to index your content, you're already set up for inclusion in AI Overviews and AI Mode, as those products draw on the Google index directly.

Google allows you to opt out of having your content included in training data by disallowing Google-Extended in your robots.txt file. However what trips a lot of people up is the fact that Google-Extended is not its own crawler. Think of it as a specific feature of normal Googlebot that your robots.txt file is turning off via the disallow line.

If you move your bot blocking to the CDN (as many of the publishers we probed have, usually to entirely block bad actors who ignore robots.txt directives, such is their appetite for training data), and disallow Google-Extended there, your content is still going into Google's training data. Googlebot is allowed through, sees no Google-Extended directive in the robots.txt, and gladly consumes all your writing.

So, you've got your Google-Extended configured correctly. The next tool at your disposal is snippet control, which denotes exactly how Google can use your content in AI Overviews. In the same way you wrap page text elements in <h>, <span> or <p> tags to designate their styling, you can individually wrap text in snippet tags to control what's included in Google's AI answers.

You can also apply snippet limits at a page level for global controls:

• nosnippet, which prevents anything on the page being used
• max-snippet:[n], where n is the maximum number of characters that can be used in an AI answer

This can be a useful tool where you have proprietary data or editorial. A 200-character teaser could be an effective way to squeeze a click out of an AI result.

Our research on this showed a sharp split between fully closed, and fully open. Only around 15% of the publishers we investigated had some kind of mixed access policy which took into account the technical nuances and tooling that now exist. And even more surprisingly, only one of the publishers we probed made effective use of snippet controls.

The temptation for many businesses which rely on SEO and AEO will be to maximise visibility at all costs. Throw the doors open and hope the traffic arrives. However we would heavily caution against sacrificing your IP at the altar of clicks, especially when the increased use of LLMs is having a disastrous effect on click-through rates.

Cloudflare have been loud advocates of bot blocking, and claim their CDN (which blocks bots at the edge, before they even have a chance to look at a site's robots.txt file) has had a meaningful impact on publishers' abilities to strike licensing deals with AI labs. The FT, The Atlantic, Ziff Davis, Condé Nast and the AP are all named as publishers who are booking meaningful revenue from AI labs as a result of cutting off access entirely and heading to the negotiating table.

This is also existential for AI labs. Since the very beginning LLMs have been plagued by quality and reliability issues, and the main route to minimising those issues is training the models on, and allowing them to retrieve, high quality data.

At the top end, more and more of the world's top producers of high-quality written materials have been cutting off LLMs. Meanwhile at the bottom end, the volume of low-quality written material online, readily indexable and retrievable, has skyrocketed. Ironically most of this material is written by those same LLMs. OpenAI, Anthropic, Google and others need quality sources to draw from, or their products become worse.

This is still difficult technically

Marking up a page for yet another marketing channel is not something anyone will look forward to, but we expect in time snippet controls will be baked into most popular content management systems.

We have created a simple WordPress plugin which allows you to add data-nosnippet tags to any layout block, giving you element-by-element control over what content Google can use in AI Overviews.

Some common errors we found in our research:

• Other structured data overrides a data-nosnippet declaration. So if your article body as rendered on the page has data-nosnippet declarations, but you're also outputting the full text in the articleBody field of your JSON-LD, your nosnippet rules are ignored.
• Many popular SEO suites default to a max-snippet value of -1 out of the box, and we found many publishers using this setting most likely unintentionally, allowing Gemini to quote as much of their article as it felt like. Where a user is asking a specific question, which is answered in the article, this setting could cost you a click. In a few cases, we found sites using paywalls or registration walls but allowing these answers to escape to AI Overviews with a badly configured max-snippet value.

This is a fast-moving and interesting aspect of AEO, so if you're reading this article more than a few months after publication, it's very likely things have changed substantially. But we believe there's an interesting middle ground for LLM access to editorial that almost nobody is occupying right now. By taking advantage of the new tooling available to them, publishers can maximise visibility amongst humans, control access for bots, and protect their intellectual property to make licensing deals possible.