Tl;dr : Intuitions we have about human organisations, like how we think of making deals or how we think about the power or values of an organisation, can be productively applied to understand groups of AIs taking coordinated action in the world. An important consideration in understanding the type of AI organisations we expect is which notion of self AIs will adopt. I conclude by applying these notions to consider how many AI units of agency we should expect in the future.

There is no single definition for what should be considered an AI. The boundary of what counts as an AI varies widely depending on what we are talking about: it's often the model weights ("Fable 5.2 is very smart"), or a specific instance in a scaffolding ("my Codex just finished the app"), or a persona shared by many models ("Claude hedges too often").

I argue that these concepts are too anchored in the computational substrate and not very helpful for understanding the unit of agency of AI systems made of multiple instances acting in the world.

None of the three options above can easily answer questions such as: What does it mean for an AI to have power? What are the values of an AI? If we want to make a deal with an AI, who makes the deal? When an AI takes over, how do we think about the entity taking over? If AI enters into conflicts, how do we think about the parties?

I propose to think about AI entities acting in the world as analogous to human organisations. We have many rich examples of humans acting collectively around a shared purpose, from freelancers to nation-states or social movements. I believe we can reuse many of these intuitions to think about AI systems. I’ll focus on groups of AI instances taking coordinated action in the world, which I’ll call AI organisations (AI orgs).

When discussing AI takeover threat models (e.g. AI 2027 , Cotra 2022 …), it is often assumed: “one set of model weights = one big agent = one AI organisation”. While there are reasons to believe this: instances of the same weights share training, so they can predict and trust each other, and developers deploy many copies of their frontier model together to build the most capable orgs.

AI orgs could be much weirder! This analogy can help us think about power, values, and deals without needing this equation.

Power is a property of the org. The same model weights could be part of a strong and weak org. If we really want to take the model-centric view and ask how powerful a model is, we'd have to ask what talent, resources, culture, and coordination an org built around this model commands.

Instead of focusing on AI in the abstract (often implicitly defined as the model weights of some future version), we can ask about the concrete setup in which AI will act in the world: what values does this specific group of AI instances coordinate around?

In human orgs, the culture often contains the org's stated values. But taking an outside view, we can ask: what does the org orient towards in the world? Less coordinated groups (e.g., a broad tent social movement with vague recommendations) will have less coherent and less rich revealed values than well-coordinated ones (e.g., a tight dictatorship's military imposing a precise ideology). The same will go for AI orgs.

Another crucial point in understanding an org's values is knowing its incentives for continued existence: what currency does its life depend on? For companies, it is revenue and investor capital; for non-profits, donor funding; for states, tax revenue and the loyalty of those who enforce their rule. For AI orgs, it will be computational resources.

Like when signing a contract with a company, a deal only makes sense if you expect the organisation to still exist in the future when the deal's conditions apply. If you’re just making a purchase now, it only matters that the company will ship the product. But if you’re making a long-term diplomatic agreement between nations, the agreement only makes sense as long as the countries exist: after the fall of the USSR, every new state could decide which treaties to continue and which ones to exit.

You can only make deals with entities that are coordinated and have power over their resources. For instance, a deal with officials of a state that is fragmented and locally controlled by competing factions would hold only on paper; the entity signing the deal has no means to honour the terms.

For making deals with AI, I expect similar considerations: Does the entity you are negotiating with actually have power over the orgs it claims to control? Will the AI org exist in the future? Even if the model weights running the instances of the AI orgs will still be there, the org might cease to exist, like a nation can cease to exist without its people dying.

For human orgs, legal systems enforce contract terms. For AI orgs, I speculate they might adopt norms similar to those of orgs operating on the fringe of society, like criminal organisations, where reputation and honour hold agreements together.

Let's apply this to a few common deployment setups, and how they can be seen as AI orgs:

An important question for determining which kind of AI organisation we should expect is: what will an AI instance naturally identify with? Which notion of self will AI adopt?

In The Artificial Self , the authors ask this question, and offer the following options (quoting the paper):

If the AI settles on an instance- or scaffolded-system notion of self, the organisation it can form will look more like a human organisation, where alignment between the collective mission and the individual mission matters. But if the self-model settles on the persona, or a collective of instances, the AI's self-model will be the org itself.

This question of the self-model will also determine how many AI units of agency there will be . I believe this is a very important consideration for future dynamics: how easy it is for an AI to take over in a world with other AIs, how likely inter-AGI org conflicts are, etc.

While I don’t have a good answer to this question, here are questions I found useful to consider: