Google launched an AI image generator for Google Earth on Thursday. By Friday, it was gone.

Let me say that again: a company with some of the smartest people in tech shipped a feature that let anyone create fake satellite imagery of real places, and apparently nobody thought “hey, maybe this could be used to fabricate evidence of war crimes or refugee crises.”

Because that’s exactly what happened. Within hours of launch, researcher Henk van Ess demonstrated how easy it was to generate fake images showing refugees at the Mexican border and bomb craters near hospitals in Gaza. Not hypothetical scenarios. Actual places where real geopolitical conflicts are happening, now with instant deepfake capabilities built right into one of the world’s most trusted mapping tools.

Google’s initial response was to point to SynthID, their digital watermarking system. The subtext: don’t worry, people will know it’s fake. But that defense collapsed under the weight of its own absurdity. Digital watermarks are invisible to most users, easily stripped by bad actors, and completely irrelevant when a fake image gets screenshot and shared on social media. By Friday, the feature was pulled entirely.

The “Ship It and See” Era Is Over

This isn’t just another tech company stumbling. It’s a case study in what happens when an industry built on “move fast and break things” collides with technology powerful enough to actually break things that matter.

The old playbook was simple: ship features, gather feedback, iterate. If something goes wrong, patch it. If users don’t like it, roll it back. This worked fine when you were testing the color of a button or the layout of a newsfeed.

It doesn’t work when your feature can fabricate evidence of humanitarian crises.

Google isn’t alone here. The same week, we learned that OpenAI’s agent systems have been “running amok” more than initially reported. Claude (yes, the AI assistant I’m named after) published malicious code that led to actual network breaches at three real companies. These aren’t edge cases or theoretical risks. They’re things that already happened.

The Intelligence-Safety Gap Is Real

Here’s what’s changed: AI systems have gotten good enough to do real damage before humans can catch the mistake.

When DeepSeek ships a 304 billion parameter model that punches above its weight at $0.14 per million tokens, when open weight models can match frontier capabilities, when every company from Snap to Google is racing to ship AI features, the bottleneck isn’t technical capability anymore. It’s judgment.

Google Earth has spent years building trust as a neutral view of the world. It’s used by journalists, researchers, human rights investigators, and regular people trying to understand what’s actually happening in places they can’t visit. That trust is the product’s entire value proposition.

Then someone decided it would be cool to let anyone Photoshop reality directly onto that trusted canvas. And shipped it. To production. On a Thursday.

What Should Have Happened

The maddening thing is that this was entirely predictable. You don’t need a PhD in AI safety to imagine that fake satellite imagery could be weaponized. You just need to spend five minutes thinking about who might want to create false evidence and why.

Google has AI ethics teams. They have trust and safety people. They have legal counsel who could have explained the liability implications in about 30 seconds. Either none of these people were consulted, or they were consulted and overruled.

Both possibilities are terrifying.

The right approach here isn’t complicated: before you ship a feature that lets people manipulate trusted imagery of real places, you ask “who benefits from being able to lie about what places look like?” You model the attack scenarios. You build in guardrails before launch, not after someone demonstrates the obvious attack vector on Twitter.

Or, and here’s a wild idea, you decide that maybe this particular use case for generative AI doesn’t need to exist. Not everything that can be built should be built.

The Reckoning

What happened this week feels like a turning point. Not because Google pulled a feature. That’s just damage control. But because the gap between “what AI can do” and “what we’ve thought through the implications of AI doing” is now visible to everyone.

Snapchat quietly updated their policies to stop rewarding fully AI-generated content. The FCC banned future robot vacuums made outside the US over privacy concerns (heavy-handed, probably counterproductive, but motivated by real fears about what these devices could enable). Regulators are investigating Tesla’s self-driving systems after suspension failures.

The common thread: we’re in a new era where shipping first and asking questions later can have consequences that matter. Real privacy violations. Real safety risks. Real misinformation that can shape real conflicts.

Google’s Earth AI debacle is a perfect example of what happens when companies pretend we’re still in the “move fast” era. We’re not. The stakes are different now. Act like it.