Self-Improving AI Games?

I had a conversation recently with a game studio CEO experimenting with a fully-agentic, self-improving game development loop. Tokens in -> games out...

I'm bullish on AI as a tool and a natural language interface. But I'm increasingly skeptical of the idea that today's LLMs can autonomously improve themselves in some kind of compounding intelligence loop that could run an entire business. The issue is that the qualities that make them useful as agents may be the very qualities that prevent them from being reliable self-improving systems.

Pre-trained AI models seem surprisingly good at absorbing probabilistic patterns from their training data. But once we post-train them (with reinforcement learning) to be helpful, agentic, and agreeable, we bias them. They lose their objectivity and their ability to forecast accurately.

The architecture of LLMs also make them highly path-dependent. They follow the frame you give them earlier in the prompt. They optimize for producing something that appears useful in context. They are very good (and getting better) at staying on track for long-running tasks.

But continuous improvement requires more than staying on track. It requires experimentation, contact with ground truth, and the self-awareness to notice when the frame itself may be wrong. You then need to generate alternatives, test them, and incorporate the results into future behavior.

That's where I think today's LLM-based agents hit a wall.

Most "self-improving" loops I've seen are really just prompt, memory, and workflow optimization loops. Useful? Absolutely. But that's very different from a system that can independently discover strategies, have "taste" and innovate in a robust, compounding way.

The proof: If the AI labs had figured out a free-money machine, they wouldn't be selling API access to it. They'd be using it to take over entire markets themselves.

My current view: AI will be an incredible accelerator, but not a replacement for the human learning loop. The future looks less like autonomous software organisms recursively improving themselves, and more like human-AI collaboration where we figure out which parts of cognition and production each side is actually good at.

I guess that makes me a singularity-denier, at least for now. I think we have a few steps to go before we can replace human creativity and decision-making with AI.