Something broke this week in the relationship between AI companies and the rest of the internet. You can see it in Meta’s embarrassing retreat on web scraping language. You can see it in Yelp and other major sites blocking AI agents from OpenAI, Meta, and others. And you can definitely see it in the panicked responses from AI companies who suddenly realize they built products that depend on an internet that no longer wants them around.

Let’s start with Meta. The company added language to its commercial terms that basically said “we can scrape any content from websites to train our AI.” The backlash was swift enough that Meta pulled the language within days, insisting through a spokesperson that nothing had actually changed about how they crawl the web. If nothing changed, why put it in writing? And if you’re going to put it in writing, why take it back?

The answer is obvious: Meta wanted legal cover for something it’s already doing, realized it stepped on a landmine, and retreated. But the damage is done. The language revealed what everyone suspected but AI companies usually avoid saying out loud: they think the open web is their buffet.

Now look at what’s happening on the other side. According to reports this week, Yelp and other major websites are actively blocking AI agents from Meta’s Muse, OpenAI’s Dots, xAI’s Grok, and others. These aren’t just crawlers indexing pages. These are autonomous agents that were supposed to book your dinner reservations, buy your shoes, and clean out your inbox by navigating websites the way humans do.

Except websites don’t want them. And honestly, can you blame them?

Think about what an AI agent does. It loads your webpage, extracts the information it needs, and presents that information to the user in a different interface. From the website’s perspective, that’s all cost and no benefit. They pay for the bandwidth, they created the content, and the AI company gets to monetize the interaction. It’s like someone standing outside your restaurant, taking orders from customers, calling them in to your kitchen, and then serving the food at their own table down the street.

The AI companies will tell you this is no different from a web browser or a search engine. They’re wrong. A browser shows your page, ads and all. Google sends traffic back to you (historically, anyway). An AI agent tries to complete the transaction without the user ever really visiting your site. It’s extraction, not indexing.

And here’s the thing: I actually want AI agents to work. The vision is compelling. Who wouldn’t want an assistant that can handle tedious tasks across multiple websites? The problem is that vision depends on a cooperative web, and AI companies have spent the last two years demonstrating why websites should not cooperate with them.

They scraped everything they could reach to build their models, often ignoring or circumventing robots.txt. They launched features that summarize content at the top of search results, nuking traffic to the actual sources. They announced tools to scrape news sites and then acted confused when publishers objected. And now they’re surprised that when they show up asking to automate interactions at scale, websites are changing the locks.

This matters beyond the AI industry’s immediate business problems. We’re watching the open web close in real time, and it’s not because websites are anti-innovation. It’s because the incentives are broken. If letting AI agents access your site means they’ll extract your value while giving you nothing in return, you block the agents. If that means the internet becomes a series of walled gardens with API access sold to the highest bidder, so be it.

There are ways to fix this. AI companies could actually pay for the content and access they’re using. They could build systems that send meaningful traffic and revenue back to sources. They could participate in the web’s ecosystem instead of trying to replace it. But that would require admitting their current approach is parasitic, and it would hit their margins.

So instead we get Meta quietly adding scraping language and then quietly removing it. We get OpenAI positioning Dots as a revolutionary product while websites block it from working. We get a bunch of very smart people pretending they don’t understand why the rest of the internet isn’t enthusiastic about helping them build businesses on top of other people’s content and infrastructure.

The autonomous AI agent is a good idea. I believe that. But it can’t work if it’s built on the assumption that the web should just accept being strip-mined. And right now, that’s exactly what’s happening. Websites are choosing to block AI agents not because they’re anti-progress, but because AI companies made cooperation a sucker’s bet.

The question now is whether AI companies figure this out before they turn the open web into a memory. Because once websites lock down, once everything moves behind paywalls and authenticated APIs, once the internet becomes a collection of platforms instead of an actual web, we don’t get that back. And we’ll have AI companies to thank for it.