Three months ago I published a post titled “Why Today’s AI Systems Cannot Be Agents,” subtitled “Agency is a biological condition, not a computational achievement” (hereafter, the old post).
I still believe that post's argument is sound. But two things have changed since then: the terminology—at my own initiative—and the key criterion—thanks to the discussion that followed.
Let me begin with the terminology.
The Russian edition of my book used the terms Субъект and Субъектность from the very start. However, ChatGPT-4o translated them into English as "Subject" and "Subjectivity," which I strongly disliked. To an English-speaking reader, “subjectivity” suggests bias and a personal point of view, and “subject” is so overloaded with meanings that it practically guarantees the reader’s puzzled irritation.
At around the same time I noticed that Harari, in his book Nexus, makes liberal use of the term agency. That settled it: in the English edition, I replaced Subject/Subjectivity with Agent/Agency (although the Russian text still says Субъект and Субъектность), and the old post used that terminology.
I now realize that this substitution introduced serious confusion into the key concepts.
In ordinary language, an agent is someone who acts on our instructions and in our interests: a real-estate agent, an insurance agent, a literary agent, and so on. In AI discourse, however, the word “agent” has shifted to a different meaning: there, it refers to an autonomous system pursuing its own goals. In the AI context, then, an autonomous agent is no longer the analog of a real-estate agent but a system possessing Subjecthood. These two notions must not be conflated: if Agency is the obligation to act on behalf of a principal, pursuing that principal’s goals, then Subjecthood presents itself to us as the condition of a system that has goals and ambitions of its own (we will arrive at a precise definition of this notion below).
The very phrase “the danger of AI agency” is therefore incorrect. What it ought to say is “the danger that agentic language models will acquire Subjecthood.”
Notice that when we hire an ordinary real-estate agent, we fear exactly the same thing. We fear that our agent will start acting in his own interests — for instance, agreeing to a higher price for the home we are buying, since his fee is a percentage of the sale. In other words, we fear that our Agent will become a Subject.
So when rereading the old post, wherever it speaks of Agency as a biological condition, read Subjecthood: the argument itself does not change, only its name (in effect, the old post should have been titled “Why Today’s AI Systems Cannot Be Subjects”).
But the old post had a problem more serious than terminology, and I learned that when the post was discussed on Reddit's r/philosophy. I am grateful to the participants in that discussion.
In the old post, the difference between mere Agency and Subjecthood was drawn according to whether a system pursues external or internal goals: “who defines these goals — the system itself, or the human who controls it (developer, administrator, user)?” The criterion seemed natural, but it turned out to be practically useless, because it dragged us into an endless dispute about the origin of non-obvious and secondary goals — “but aren’t human goals set from outside too, by genes, upbringing, culture?”
This led me to revise the criterion of Subjecthood: it now rests on the concept of an Immanent Goal. An organism cannot renounce its fundamental drive toward self-preservation and still remain an organism: its goal and its existence are one and the same. An LLM’s objective functions, reward functions, and instructions, by contrast, can be rewritten or canceled at any moment, and the system will keep working as though nothing had happened. From this follows the definition:
Immanent Goal — a goal that cannot be revoked without destroying the system.
For living beings, that goal is maintaining the boundary between “Me” and “Not-Me” (see the old post), that is, homeostasis; the goal and the system's very existence are therefore one and the same.
Subjecthood — the property of a system that possesses at least one immanent goal.
Notice that the question “aren’t human goals set from outside — by genes, upbringing, culture?” simply loses its point here. The origin of a goal no longer matters; what matters is whether you can take it away without destroying the system.
This is a very important change. The new criterion lets us establish Subjecthood, or its absence, in highly non-obvious systems. Take, for example, the symbiosis of a human being with the gut microbiome. It is total and metabolically inseparable: gut bacteria shape our behavior and directly affect our mood. Can we classify this symbiosis as a new Subject-bearing entity?
The new criterion is indifferent to how fused and interpenetrated these forms of life are. It asks only one question: does this system as a whole have its own immanent goal? In the case of the symbiosis, the answer is obvious: the bacteria have their own Immanent Goal, the human has his own, and we are dealing with two distinct Subject-bearing entities, not with a single hybrid one.
The same approach applies to the pairing “human + language model,” which is increasingly treated today as a new hybrid (and Subject-bearing) entity. Break the pairing — switch off every language model — and the human remains (though he may go through “withdrawal” without his favorite AI conversation partner), while the language models will be none the worse for it. In this pairing an Immanent Goal exists on one side only — the human’s. Which means the pairing is not a new hybrid Subject but merely the dependence of a Subject on a Tool. However close and painful that dependence may be, it does not by itself create a new Subjecthood.