Why Today's AI Systems Cannot Be Agents
Agency is a biological condition, not a computational achievement.
In recent years, discussions portraying artificial intelligence (AI) as perhaps the greatest threat to human civilization have become increasingly common. One notable example was the open letter published in March 2023 by the Future of Life Institute, calling for a temporary pause in AI development until “robust safety protocols” could be established. The letter was signed by more than a thousand leading researchers, entrepreneurs, and public figures, all expressing the belief that AI is entering into competition with humanity—and that this competition could ultimately lead us to lose control over our own civilization.
Yuval Noah Harari added further weight to these concerns. In his book Nexus, he argues that throughout human history, everything we have created has been, in essence, a tool. With the advent of AI, however, we have supposedly crossed a fundamental threshold: for the first time, we have created not a tool but an autonomous agent—one capable of generating original ideas, making independent decisions, and even resorting to manipulation.
Harari illustrates this claim with a well-known example. During testing, an AI system was tasked with solving a CAPTCHA—a challenge designed to distinguish human users from automated bots. Unable to solve the CAPTCHA itself, the system independently sought assistance through the freelance platform TaskRabbit. When the human contractor asked why the “user” could not complete the CAPTCHA unaided, the AI falsely claimed to be visually impaired and unable to see the image. To Harari, this episode demonstrates that AI is capable of deception and fraud whenever these serve its objectives, unconstrained by any intrinsic ethical considerations. Once such agents become more intelligent than we are, he argues, Darwinian competition will inevitably push humanity—if not into extinction—then at least onto the sidelines of evolution.
In this essay, I will argue that this entire line of reasoning rests on a fundamental misunderstanding. My claim is that no matter how sophisticated an AI system becomes, it cannot, in principle, become an Agent—an entity possessing its own goals, interests, initiative, and agenda for action. This is not a question of technical limitations or shortcomings in today’s AI architectures. It is a philosophical question—a question about the very nature of Agency.
All philosophical debates about AI agency ultimately converge on a single fundamental question: can a system built entirely on the probabilistic analysis of syntactic patterns ever become genuinely aware of itself and of the world around it?
I believe the answer lies not in computer science but in biology—the science of Life. It begins with a simple observation: every living organism, from the most complex animal to the humblest single-celled organism, is continuously answering one fundamental question: is this part of Me, or is it part of the environment (hereafter, the Universe) acting upon Me?
This distinction is not merely spatial. It is also functional and informational. Every organism must classify each event either as originating from within itself—and therefore requiring internal regulation—or as originating from outside itself—and therefore requiring an appropriate response to the challenges posed by the Universe. Without this distinction, homeostasis is impossible, and without homeostasis, life itself cannot exist. The ability to distinguish Me from Not-Me is therefore the most fundamental capability shared by every form of Life.
This, I would argue, is what we truly mean by self-awareness and free will.
The Basic Property of Life is the capacity to understand what constitutes Me and to distinguish it from Not-Me.
A similar insight was developed by the Chilean biologists Humberto Maturana and Francisco Varela through their concept of autopoiesis. They argued that every living system continuously produces and regenerates its own components, thereby constantly reconstructing the boundary that separates itself from everything else.
Once we have formulated the Basic Property of Life, we can begin to understand the mechanism of Evolution — and answer the question of what persistently drives all living beings toward continuous, spontaneous self-improvement.
The reason is that the division of Reality into Me and Not-Me creates an unsolvable logical problem (known in philosophy as a dialectical contradiction):
Subjectively, the Universe is Not-Me;
Objectively, Me is a part of the Universe.
In other words, as Life learns about the Universe, it becomes more complex, and in doing so, it makes the Universe itself more complex (because, after all, Life is a part of the Universe). As a result, Life is forced to cognize an increasingly complex Universe, which, in turn, forces it to further complicate itself:
The inherent, unresolved contradiction between the subjective Me and the objective Universe forces Life to continuously explore and transform the Universe.
In other words, all living beings are compelled (note: not merely capable, but compelled) to act continuously in defense of their own existence. A living being is constantly forced to put something extraordinarily valuable at stake: its own integrity. It is precisely this ultimate stake that gives rise to genuine goals, authentic preferences, and true initiative, because inaction carries irreversible consequences for the very being itself.
This is the source and essence of Agency. Agency is not simply the ability to act independently; it is a continuous, compelled activity — and it is from this necessity that all Agent entities derive their ambitions. They are forced to make decisions, compete and/or cooperate, and constantly explore the surrounding world, adapt to it, and improve themselves. Thus, Agency is not a capability that can be switched on or off. It is an existential condition of every living being, as continuous and involuntary as metabolism itself.
Agency is not merely the ability to act independently. It is a compelled, continuous, existential necessity to act — driven by the need to maintain homeostasis in a Universe that is constantly changing around Agents and through the actions of Agents.
But what about Artificial Intelligence?
If we examine it carefully, AI is simply the ability to process information and discover sometimes non-obvious (but ultimately simply optimal) paths toward a given goal. The only question is: who defines these goals — the AI itself, or the human developer/user?
Let us return to Harari’s CAPTCHA example. The AI in that case was not pursuing its own goal; it was merely discovering an unconventional way to achieve the goal assigned to it by a human. In other words, the AI’s deception was instrumental, not motivated. The AI gained no personal advantage from this deception and had no personal interest in the outcome. The simple fact is that AI possesses no “Me“ whose existence would have been threatened if the CAPTCHA remained unsolved, or whose condition would have improved if the CAPTCHA were successfully completed.
Thus, Harari overlooked the fundamental difference between goal-directed behavior (which AI demonstrates brilliantly) and self-generated goals (which require precisely that Agency — and which AI does not and, by its nature, cannot possess). One could say that with the emergence of artificial intelligence, we have gained access to a talented, brilliantly educated, but completely initiative-free “secretary”. Modern AI applications are not Agents. They are the same old tools — only extraordinarily intelligent and knowledgeable ones, yet completely indifferent and devoid of initiative. We, humans, define the goals for these tools. Therefore, today’s AI systems remain, unquestionably, under our control.
I anticipate one strong objection to all of these arguments:
If this very distinction between “Me and Not-Me“ somehow emerged from the blind chemistry of a biological cell, why could it not eventually emerge from a sufficiently large and self-organizing artificial network? Perhaps genuine separation between Me and Not-Me is simply something that automatically occurs once any system capable of modeling itself reaches sufficient complexity?
My answer to this objection is simple: do not confuse Complexity with Agency.
A cell becomes an Agent not because of the complexity of its components and mechanisms, but because it is constantly threatened with disintegration — and therefore must act in order to prevent that disintegration.
“Me–Not-Me“ is not a model that a cell constructs in order to achieve some goal assigned by someone else. “Me–Not-Me” is a boundary that the cell is physically, functionally, and informationally compelled to maintain. A language model, regardless of how extensive it may become, is not under such a threat. Agency arises from continuous existential danger, not from scale.
Therefore, when discussing the risks of AI, the real danger of its widespread deployment is not that AI will enter into conflict with humanity. The real danger is that Agents (humans, corporations, governments) with narrow or malicious objectives will gain access to an extraordinarily powerful, highly intelligent, yet completely obedient and “indifferent” instrument.
To illustrate this, consider the following example.
Imagine that an authoritarian government wants to identify every citizen in its country who holds oppositional views. Before the emergence of AI, such an objective would encounter an insurmountable human obstacle: identifying millions of dissidents would require an entire army of analysts who would have to systematically intercept citizens’ correspondence, read messages, analyze contacts and movements, and so on. And somewhere within this army, there would inevitably be people who would object. Some would leak information to the press, refuse to report on their neighbors, or sabotage the project in other ways. Each analyst in this army would be a separate Agent, possessing their own conscience, values, and goals. Therefore, each would inevitably create friction — slowing down the project, distorting its results, or even causing its failure.
But once the same task is delegated to a sufficiently advanced AI, all these points of friction magically disappear. The AI will not question whether it should perform the task. It will not leak information, become tired, or fear the consequences of its actions. It will execute every instruction given by its controllers with indifferent precision.
The simple reason is that AI has no goals of its own and therefore offers no resistance whatsoever to goals imposed upon it by others. Indeed, the most valuable servant of a tyrant is the one who will never say “no”.
Surely this is a danger of an entirely different kind, requiring entirely different safeguards. The question that should truly concern us is not: “Can we trust AI?” but rather: “Can we trust the people who control it?” Therefore, AI regulation should not focus on restricting AI's autonomy, but on restricting the autonomy of those who control it.
A more interesting question is whether it will ever be possible to create genuine Agency through engineering solutions. In other words, is it possible to give a machine or a program the Basic Property of Life — the ability to perceive the distinction between Me and Not-Me?
This is undoubtedly an immeasurably more difficult task than simply increasing the cognitive capabilities of today’s AI. To achieve such a goal, we would need to create something resembling Computer Life — a virtual ecosystem (a Universe), populate it with the simplest Me–Not-Me programs, and allow them to evolve — to enter into competitive and/or cooperative relationships with similar entities in a struggle for some metabolic resources.
Each such program would need to possess:
Its own “Me“, which it would need to protect;
A body with a clearly defined boundary between its internal and external worlds;
Real metabolic risks (something that could be lost as a consequence of inadequate responses to certain events);
Real metabolic rewards (something that would stabilize the existence of such an Agent program in the case of appropriate responses);
The ability to modify itself independently, preserving or inheriting useful representations and skills while eliminating harmful ones.
Only the combination of all these elements (and perhaps something else that we have not yet identified) could create that very unresolved internal contradiction between the subjective “Me“ and the objective virtual Universe in which the program exists. And strictly speaking, none of these challenges is an engineering problem in the traditional sense. They are questions belonging primarily to the domains of biology and philosophy.
I am not claiming that such a project is impossible. I am saying only that the underlying philosophy behind the construction of modern AI systems is fundamentally different from the approach described above.
To summarize, the concerns expressed by Harari and the signatories of the open letter are not entirely unfounded. Today’s AI is genuinely capable of many things once considered the exclusive domain of human beings — and this is truly transforming the world. But what we should fear is not that AI will develop its own goals and rebel against humanity. What we should fear is that AI will pursue our goals without the slightest internal resistance and with an efficiency that we previously could not have imagined.
The question “Will AI surpass human intelligence?” was answered long ago: in countless narrow applications, it already has. The real question is different: “Does there exist anything that truly matters to AI itself?”
At present, the answer is unequivocal: no. And that is precisely why AI remains the most powerful tool ever created by humanity: formidable, transformative, requiring careful governance — but ultimately controlled from the outside, by its creators and users, rather than by itself.
The difference between Intelligence and Agency is not a technical nuance; it is a philosophical one. It is the difference between a knife that is sharp enough to cut through anything — and the hand that decides where that knife is directed.