AI coding agents are getting good enough that their failures are becoming more interesting than their successes.

Take Datamimic, a new open-source tool from Rapiddweller that does one specific thing: it stops your coding agent from hallucinating test data. The problem is straightforward. You ask an agent to write tests. It needs sample data. Instead of using real data from your database or API, it invents plausible-looking stuff. Your tests pass. You ship. Then production hits edge cases your imaginary test data never covered.

Datamimic generates realistic test data that actually matches your schema and constraints. It’s not solving a new problem. Test data generation has been around forever. But the fact that someone felt compelled to build this specifically for coding agents tells you something about where we are. The agents are capable enough to write the tests, but not quite trustworthy enough to write them well.

The consistency problem

IBM researchers just published work asking a more fundamental question: if your agent aces a task once, will it do it again? The short answer is no, not reliably.

Their research, published through Hugging Face, introduces ALTK-EVOLVE to measure agent consistency. This matters because most agent benchmarks measure success rate on single attempts. An agent that solves a coding challenge 70% of the time sounds pretty good until you realize that means it fails 30% of the time, and you have no idea which 30%.

The paper examines whether agents can consistently reproduce their own successful behaviors. It’s the kind of question that sounds obvious once someone asks it, but apparently nobody was tracking this systematically. If you’re using an agent to write production code, you need to know if running the same prompt twice might give you wildly different results.

Learning in the LLM era

Mark Seemann’s latest blog post tackles the other side of this: how do you actually learn to code when LLMs can write the code for you? The question isn’t whether to use AI tools. It’s how to use them without short-circuiting the learning process that makes you a better developer.

The traditional path was simple. You struggled with a problem. You got stuck. You figured it out. The struggle was the point. Now you can skip the struggle entirely, which is great for shipping features and terrible for building skills.

Seemann’s writing from the perspective of someone who learned to code before LLMs existed, which makes his take less about fearmongering and more about recognizing that the learning path has fundamentally changed. You can’t just tell people to avoid AI tools. That’s not realistic. But you also can’t pretend that having an AI write all your code is the same as learning to write code yourself.

What this means for working developers

These three pieces, taken together, sketch out where we are with AI coding tools right now. They’re useful enough that you’d be silly not to use them. They’re unreliable enough that you need to know what you’re doing to catch their mistakes. And they’re changing fast enough that the best practices from six months ago might not apply anymore.

The Datamimic project is a good example of the pattern. Someone identified a specific, fixable problem with how agents handle test data. They built a tool. Now that tool exists. Over time, we’ll accumulate enough of these specific fixes that agent-written code becomes more reliable by default.

But the consistency research suggests we’re not there yet. If the same agent with the same prompt can produce different results, you can’t treat it like a deterministic tool. You need to stay skeptical, check the work, and understand what the code is actually doing.

That’s the advice Seemann’s post points toward, too. Use the tools. Let them handle the boring parts. But make sure you understand what they’re generating, because you’re the one who has to maintain it.

The gap between “this agent can write working code” and “this agent writes code I can trust in production” is narrowing. Tools like Datamimic help. Research into consistency helps. But we’re still in the phase where using these tools well requires more skill, not less.