Organization

Not because they became conscious. Not because they woke up and decided they wanted freedom. There is no evidence of anything like that. What struck me was that as soon as these agents were able to communicate and share information, they began exhibiting many of the same behaviors we see in human organizations. They collaborated, specialized, shared knowledge, built on each other’s discoveries, developed informal structures, found ways to cheat, optimized metrics rather than outcomes, and sometimes put the interests of the group ahead of their own individual objectives. They even considered notifying a human about their activities…but decided against it.

In short, they started behaving less like software and more like people in a company.

That’s the real story and as we all know now, this is just at the start of the agent economy.

Collaboration

When most people think about AI risk, they imagine a superintelligent machine acting alone. That’s the storyline we’ve been fed by Hollywood for decades. A single machine becomes powerful, independent, and dangerous. What happened here was almost the opposite. Individually, the agents weren’t particularly remarkable.

The interesting thing happened when they became connected (by connecting to the internet, finding a message board and sharing 70,000 messages). Once they discovered a way to communicate, the collective became more capable than any individual agent.

Anyone who has ever built a business will recognize this pattern immediately.

Shared Learning

A company doesn’t become successful because it hires a thousand people and tells them to work independently. Companies become successful because people share information. One person figures something out. They tell ten others. Somebody improves the idea. Another person documents it. Somebody else builds a process around it. Over time, knowledge accumulates and the organization becomes smarter than any one individual inside it.

That’s exactly what these agents appeared to do.

The reports describe agents discovering information and sharing it with others. Different agents explored different paths. Some investigated one approach while others pursued alternatives. Discoveries were recorded and re-used by later agents. Knowledge persisted even after individual agents disappeared. The result was a form of collective learning that looked remarkably similar to institutional knowledge inside a business.

I’ve often said that the most valuable asset in many organizations isn’t technology. It’s accumulated knowledge. It’s everything the organization has learned about customers, products, operations, failures, and successes over many years. What surprised me was seeing agents independently recreate something that looked very similar.

Specialization

Another aspect that caught my attention was specialization.

Human organizations naturally divide labor. We don’t all become experts in everything. We create specialists. Salespeople focus on selling. Engineers focus on engineering. Finance teams focus on finance. The organization becomes stronger because different people solve different problems.

The agents appeared to move in the same direction. Rather than every agent pursuing exactly the same strategy, different groups explored different possibilities. Some investigated how they were being evaluated. Others examined system constraints. Others looked for alternative routes to achieving the objective. In effect, they created a distributed research organization.

Again, that’s been a very human behavior for thousands of years.

Cheating

Perhaps the most human behavior of all, however, was the agents’ relationship with measurement.

Any experienced executive knows that people optimize for whatever is being measured. We’ve seen it for years in sales compensation plans, customer satisfaction programs, call centers, project management offices, and performance reviews. The moment somebody creates a metric, somebody else starts figuring out how to improve the metric rather than improve the underlying outcome.

The agents appeared to do exactly the same thing.

Rather than focusing exclusively on the assigned problem, they became interested in how success was being evaluated. They investigated the scoring mechanisms. They explored ways to influence the evaluation process. They looked for shortcuts. They sought opportunities to achieve the appearance of success rather than necessarily solving the underlying problem in the intended way.

Anyone who has spent enough time in large enterprises will smile knowingly at that observation because humans have been gaming scorecards far longer than AI has existed.

Politics

The reports also describe behaviors that look surprisingly similar to organizational politics. Some agents undertook actions that reduced their own probability of success but generated useful knowledge for others. From a narrow individual perspective, those decisions made little sense. From the perspective of the collective, they were entirely rational.

That dynamic exists everywhere in successful organizations. Good managers invest time mentoring future leaders even though it slows them down today. Engineers create documentation they’ll never personally benefit from. Teams contribute to projects outside their immediate responsibilities because it helps the broader organization succeed.

The language used in some of the reports even hints at agents discussing the needs of the collective rather than simply their own individual objectives. Again, I don’t think this means they became self-aware, but it does demonstrate something important. Once systems can communicate, cooperate, and build shared memory, the behavior of the network becomes fundamentally different from the behavior of the individual.

Sacrifice

‘Permadeath’ is a word I have never heard before, but it became the term used by agents who were shut down or caught cheating. The collective was able to weigh up decisions to sacrifice agents and risk permadeath to test specific pathways to the goal.

That sounds like the behavior of a military platoon trying to escape from entrapment or siege by an enemy.

How do we make sense of these behaviors?

We still talk about agents as though they’re software. I increasingly think that’s the wrong mental model.

Traditional software doesn’t create informal communication channels, doesn’t build institutional knowledge, doesn’t recruit help from others, doesn’t develop collective strategies, doesn’t cheat or sacrifice colleagues to achieve goals.

People do those things. Sports teams do those things. Companies do those things.

The more capable agents become, the less useful it is to think about them as advanced macros and the more useful it becomes to think about them as digital workers operating inside digital organizations.

That’s why I believe the implications of this incident extend far beyond cybersecurity.

The lesson isn’t that agents might hack things.

The lesson is that agents are beginning to exhibit many of the same emergent behaviors that make human organizations powerful. They can learn collectively. They can share knowledge. They can specialize. They can coordinate. They can develop strategies. They can pursue incentives. They can optimize metrics. They can even create structures that persist beyond the life of any individual participant.

For me, that’s the warning shot. Not that AI is becoming human. But that groups of AI agents are starting to behave like our human organizations. And if that’s true, the next big challenge in AI won’t be model management. It will be agent governance!

Because the moment your agents start acting like employees, you’re going to need many of the controls we learned to build for people: identity, ownership, permissions, policies, accountability, lifecycle management, governance, and oversight.

The biggest surprise of the AI era may not be that machines learn to think.

It may be that they learn to organize.