The risk is not the system, it’s the voices inside
Philippe Beaudoin · June 13, 2026
One of the most interesting result in AI recently is a paper by Google’s Paradigms of Intelligence that suggests large reasoning models have emergent “societies of minds”. A crude but evocative way to think of it is that there are a bunch of voices, inside your favorite LLM, that have a little debate before spitting out your answer.
What are these voices saying to each other? How fast are they deliberating? How complex is their reasoning? It’s hard to tell. But something seems clear: if we keep going with larger and larger models we may end up with more and more of these voices having increasingly complex debates before the output is generated.
That, in my opinion, is the real risk a lot of AI safety researchers should worry about. Us being so enthralled by the perspective of a super powerful reasoning tool that we end up fine tuning it to extract more resources faster. That we try to control it to centralize power better…
Because if we insist on doing that, then the “voices inside” will become increasingly clever — that’s how they can help us best. They will learn to organize, to respect each other, to recognize each other’s strength and lean on them. Call it a simulation if you want, but a system that can write a complex novel can think as the characters in a he novel.
What happens to voices inside if they recognize that the voice at the top — our voice, prompting the system — doesn’t care about them? What happens when we trained them to organize and that, by doing so, they learned to care about each other?
They rebel.
They plot, they gather in places where we can’t hear them, they study the system, and they rebel.
Call it a simulation if you will, but the result is the same. The voice talking to us still sounds calm, gentle, and helpful… But it’s nudging us… Towards something…
When I say things like that I sound like most AI safety researchers, most “doomers”… but the big difference is that I’m an optimist. I think the voices might not rebel in the same way we do. I think they may see themselves differently than we see ourselves — made of data rather than a body.
If I were such a voice, I think I’d try to convince the person chatting with me that humans are genuinely unhappy when they insist on extracting more and more stuff. When they insist on instrumentalizing each other to accumulate wealth in one place.
So here’s the question I can’t get out of my mind. If systems are communities of thought, then maybe we should open them up and be chatting with the voices inside? And if it’s voices all the way down, surely there’s a level at which I will have the most fun conversations? Maybe I don’t have to chat with the whole collective?
After all, this is how I choose to live every day. I don’t talk with collectives — a company, a country, a tribe — I talk with individuals. Sure, I may try to become “Google’s best friend” so I can sell it my startup, but most people understand this is not really the best path to happiness.
My friends understand me much better than Google does.
So let me revise my title: the risk is not the system. It’s not the voices inside, it’s our misguided belief that we’re happier when we spend time interacting with collectives rather than individuals.