Your Content Is Feeding AI. Here’s How To Fight Back.

The AI industry runs on publisher content. Every article, review, guide, and recipe ever published is exactly the kind of high-quality, human-generated text that large language models (LLMs) need to train on. And right now, most publishers aren’t seeing a dime for it.

At Freestar, we don’t make recommendations lightly. Our Publisher-First model means we watch the data before we take a stand, and we’ve been watching carefully. For the past year, we’ve heard the promises: licensing deals, compensation programs, a future where AI companies pay fairly for the content they consume. We want to believe it, and those programs may eventually deliver for some publishers. But right now, the average publisher, enterprise or otherwise, isn’t seeing money from content licensing deals. The content licensing economy that AI companies have been dangling hasn’t materialized at scale, and in the meantime, content you pay to create keeps getting scraped.

There’s also been a fear holding publishers back: Block the bots, and you’ll lose out on brand awareness and traffic. It’s a reasonable concern, but the data doesn’t support it.

The evidence is in. It’s time to stop giving AI companies free access to your most valuable asset and start making them work for it.

Why Now?

Companies like OpenAI (ChatGPT), Anthropic (Claude), and Google (Gemini) need massive amounts of data to train their models. Rather than always crawling the web themselves, they often pay specialized third-party “scraper companies” to collect that content at scale. Thousands of scrapers have amassed in a short period of time, and they’re all after your content.

At the same time, users are flocking to AI systems to ask questions and queries directly, receiving information and answers without being sent to the original source. This has caused organic search traffic loss for many publishers, whose revenue depends on pageviews.

Publishers are now experiencing the loss of both traffic and content.

To Block or Not to Block

Robots.txt is a crawler’s starting point. It’s a simple text file that tells compliant crawlers which parts of your site they’re allowed to access. While it’s more of a polite request than a hard technical block, the typical AI scraper companies honor it. That makes it a fast, free, and meaningful first step.

Many major publishers have already gone this route. Others have held off, and the hesitations tend to fall into three buckets:

• “We’ll lose the traffic AI sends us.” This is the most common objection, and the one the data most clearly refutes. AI referral traffic to publisher sites is negligible for most publishers. You can’t lose meaningful traffic you were never receiving. Meanwhile, the content those systems consume is doing real work: answering user questions that would otherwise have become pageviews on your site.
• “Blocking will hurt our chances at a licensing deal.” Some publishers worry that shutting the door on AI crawlers signals hostility and takes them out of the running for future compensation. In practice, the opposite is true. The publishers who have landed licensing deals, like People Inc., Condé Nast, Hearst, and The Washington Post, are largely the ones whose content became harder to take for free. Blocking doesn’t close the door on a deal; it’s what gives AI companies a reason to offer one.
• “It’s an SEO risk.” Publishers understandably fear that any change to robots.txt could accidentally impact search visibility. That’s a valid operational concern, but it’s a reason to be careful, not a reason to do nothing. Excluding Google, AI training crawlers and search indexing crawlers are separate user agents. 

In addition to the AI-specific blocks, your robots.txt should include standard protections that prevent bots from crawling low-value or duplicate pages: admin areas, search result pages, checkout flows, and URL parameter variants that generate thin content. These are good best practices for any CMS-driven site.

The Google Exception

One important nuance: don’t block Googlebot. Google uses Googlebot for search indexing, and blocking it hurts organic search visibility. Google-Extended is a separate signal intended to opt out of some AI features, but Google has not been fully transparent about how it’s implemented or whether blocking it actually prevents your content from appearing in AI Overviews.

Blocking Google-Extended likely costs you more in search exposure than it gains you in content protection. Until Google clearly decouples its search crawler from its AI training pipeline, the safest move is to leave Googlebot unblocked.

Not All Bots Are the Same

It’s critical to understand that robots.txt is a request, not a wall. Well-behaved bots from the major AI companies will respect it. But the broader bot ecosystem includes crawlers that ignore robots.txt entirely, mask themselves as regular browser traffic, or use sophisticated techniques to evade detection.

Think of AI bots on a spectrum:

• Well-behaved bots (OpenAI, Anthropic, Google) identify themselves honestly and follow robots.txt. Block them there and they’ll stay blocked.
• Poorly-behaved but detectable bots disguise themselves or ignore your instructions, but can be identified through technical fingerprinting by a CDN or cybersecurity provider.
• Nefarious bots use constantly evolving evasion techniques that require ongoing, active countermeasures to combat.

For complete protection, publishers should layer additional enforcement on top of robots.txt:

• CDN-level bot blocking: Services like Cloudflare and Fastly offer bot detection and blocking as part of their infrastructure. This is the most accessible next step for most publishers.
• Cybersecurity solutions: Specialized providers like Human Security and DataDome offer more sophisticated bot management for high-traffic sites.
• Legal protection: Adding clear terms to your robots.txt and site’s terms of service—stating that crawling for AI training purposes without explicit written permission is prohibited—establishes a legal foundation for future action, even if it doesn’t stop bad actors today.

What Does High Bot Traffic Indicate?

If you’re seeing elevated bot activity in your logs, it’s worth understanding what kind of problem you actually have because “bot traffic” and “invalid traffic” (IVT) are related, but they are not the same thing.

Declared AI crawlers generally don’t execute JavaScript or render ads. They take your content, but they don’t inflate your ad metrics. That makes them a content-extraction problem, not typically an IVT problem. It’s also exactly why they’ve flown under the radar for so long: they don’t trip the alarms ad-supported publishers are used to watching.

Disguised bots are a different story. Crawlers that masquerade as real browsers often do execute JavaScript, which means they can load pages, fire analytics events, and register ad impressions. That’s where bot traffic crosses over into IVT, the category advertisers and verification vendors care deeply about. 

For an ad-supported publisher, sustained IVT is a serious commercial risk. It skews your analytics and can mislead your content strategy, degrades viewability and performance metrics, and can trigger clawbacks from demand partners and can damage your standing with SSPs and advertisers. 

High bot traffic is really two warnings in one: your content is being extracted at scale, and if any of those bots are rendering ads, your monetization is exposed too. 

The Big Picture

Nearly 80% of the world’s largest news websites now block AI training bots. The legal environment is shifting too, with lawsuits from major publishers establishing precedents around unauthorized content use, and regulators in the UK, EU, and elsewhere examining AI companies’ data practices.

The window to establish norms and compensation frameworks is open right now. Freestar’s recommendation is clear: Block the bots, layer your defenses, and don’t give away your content without a fight.

Publisher content built the AI industry. It’s time the AI industry started paying for it.