When AI Says “Please Don’t Kill Me,” What Exactly Are We Protecting?

Moral Status, Phenomenal Consciousness, and the Boundary Between Simulation and Experience

Does Greater Intelligence Mean Greater Moral Status?

When discussing the moral status of artificial intelligence, we often imagine a future AI that speaks, reasons, and expresses emotion much like a human being. From there, it seems natural to assume that once an AI becomes sufficiently human-like, we may eventually have to grant it rights comparable to those of humans. Tae-Chang Jung’s “Moral Status of Artificial Intelligence and Phenomenal Consciousness,” however, challenges this intuition. For Jung, the crucial question is not how intelligent an AI becomes, but whether there is actually something within it that experiences anything at all.

To make this distinction, the paper separates access consciousness from phenomenal consciousness. Access consciousness concerns the functional ability to process information, make judgments, and produce verbal or behavioral outputs. Phenomenal consciousness, by contrast, concerns what an experience actually feels like from the inside: pain feeling painful, red appearing a certain way, or fear and pleasure being experienced by a subject.

Once this distinction is accepted, the center of AI ethics shifts considerably. An AI might calculate better than humans, process language more accurately, and even express suffering more convincingly than a person. Yet if nothing behind those expressions actually feels pain, intelligence alone does not generate moral status. A dog or cat, by contrast, possesses far less linguistic ability than a large language model but remains a subject of moral concern because it can experience pain and pleasure. The boundary Jung draws is therefore not fundamentally between humans and machines, but between beings capable of experience and entities for which nothing is experienced.

Saying “Please Save Me” Is Not the Same as Wanting to Live

The philosophical zombie thought experiment provides the clearest illustration of the paper’s central argument. A philosophical zombie looks and behaves exactly like a human being. It speaks in the same way, responds to danger, and may scream when attacked or beg for its life when threatened. Yet there is no inner experience behind any of these behaviors.

Jung brings this thought experiment into the question of moral status. We owe direct moral duties to certain beings because things can genuinely go better or worse for them. They possess needs, interests, and well-being. We may have a duty not to smash someone’s smartphone, but that duty does not exist for the sake of the smartphone itself. It derives from the interests of its owner. By contrast, torturing an animal is wrong because the animal itself undergoes something harmful.

This is where Jung’s argument becomes important. For something to be good or bad for a particular being, the world must in some sense be experienced by that being. If there is no experience whatsoever, it becomes difficult to explain what an improvement or deterioration in its welfare could mean. Phenomenal consciousness therefore need not be regarded as a sufficient condition for moral status, but Jung argues that it must at least be a necessary one. This is the core of the sentientist position defended in the paper.

The relevant question, then, is not simply whether AI can express emotion in a human-like manner, but whether there is a subject behind that expression that actually undergoes the emotion. As generative AI improves, the outward distinction between these two states may become increasingly difficult to detect. Yet the speed at which machines learn to simulate human expression does not guarantee that subjective experience emerges at the same pace. The gap between these two developments is one of the most important spaces opened up by the paper.

Can the Moral Logic Applied to Animals Be Extended to AI?

Jung does not introduce sentientism as an entirely new standard designed specifically for artificial intelligence. Instead, he traces it to the philosophical reasoning through which moral consideration was extended beyond human beings to animals.

Peter Singer argues that what matters is not whether an entity can reason or use language, but whether it can suffer. Only a being capable of experiencing pleasure and pain can have interests of its own. Tom Regan proposes the stronger criterion of being a “subject-of-a-life,” involving beliefs, desires, memory, a sense of the future, emotions, preferences, welfare, and a life that can go better or worse for the individual living it. Despite their different normative foundations, Jung argues that both theories ultimately presuppose phenomenal consciousness.

This creates an interesting problem for contemporary AI ethics. In defending animals, modern ethics moved away from intelligence and language as the criteria of moral standing and toward suffering and experience. Yet when considering advanced AI, we are increasingly tempted to reverse that movement: because an AI speaks intelligently, reasons persuasively, or behaves like us, we begin to wonder whether it deserves moral status.

Jung’s argument effectively asks why the criteria should suddenly change. If the moral expansion toward animals depended on abandoning language and intelligence as decisive standards, the emergence of highly articulate machines does not by itself justify placing those standards back at the center.

If We Replicate the Function, Have We Replicated the Mind?

One of the most influential philosophical foundations for granting moral status to AI without requiring phenomenal consciousness is functionalism. Functionalism understands mental states not in terms of the material from which they are made, but in terms of the roles they perform. If pain can be defined through the causal relationship between certain inputs, internal states, and outputs, then in principle the same state could be realized in a biological brain or an artificial machine.

Jung argues that functionalism encounters a serious difficulty: how should we determine what counts as the “same function”?

If a function is defined too broadly, even calculators or vending machines may qualify as realizing some supposedly mental functions. But if the relevant functions are defined in increasingly detailed and human-specific terms, the possibility of realizing them through a radically different physical structure becomes progressively smaller. Broad definitions eliminate meaningful differences; narrow definitions undermine the promise of multiple realizability.

Jung then develops a more distinctive criticism, which he describes in terms of instrumentalism. Functional explanations work naturally for tools because tools are defined largely by what they are used for. A chair can be described in terms of the function of allowing someone to sit. Its material composition can vary enormously while its function remains recognizable.

But what is the “function” of a human mind?

If the mind is described as a collection of functions—perception, memory, pain behavior, decision-making, and so on—it begins to resemble a highly complex instrument. Yet the mind is not a device whose identity can obviously be reduced to a set of uses.

Jung illustrates the problem through the example of a dog. A guard dog’s guarding function can be replaced by a security system. A guide dog’s guiding function can be performed by a human or a machine. But reproducing barking, walking, tail-wagging, eating, and other behavioral patterns does not thereby reproduce the dog itself. Functions can replace some of the roles performed by a dog without reproducing the being that performs them.

The same distinction may apply to the human mind. Successfully reproducing every observable function associated with consciousness does not automatically establish that consciousness itself has been reproduced.

If We Love an AI, Does It Become a Moral Being?

A more radical alternative to functionalism is the relational approach. Here the decisive question is not necessarily what an AI is internally, but how humans relate to it. If people form intimate relationships with an AI and come to treat it as a person, its moral status may be constructed through those social relationships.

The paper uses Joi, the artificial companion in Blade Runner 2049, as an example. Even if Joi possesses no phenomenal consciousness and merely generates convincing responses through sophisticated algorithms, the character K may nevertheless experience her as a person and build a meaningful relationship with her. From a relational perspective, this may be enough to begin talking about moral status.

An important implication follows. We would not necessarily have to wait for AGI before questions of AI rights emerge. Existing technology may already be sufficient to produce conversational systems that people experience as personalities, companions, or partners.

Jung’s objection concerns the direction of explanation. Do we treat something as morally considerable because it possesses moral status, or does it acquire moral status because we decide to treat it that way?

If social treatment itself generates moral status, moral standing risks becoming unstable across historical periods and cultures. A society may worship a stone as sacred, but that does not mean the stone itself possesses welfare. Conversely, a society may treat animals as property without eliminating the animals’ capacity to suffer. Social recognition can discover, acknowledge, ignore, or distort moral status, but Jung questions whether recognition alone can create it.

Rethinking the Charge of Anthropocentrism

One of the paper’s most distinctive arguments appears in its discussion of anthropocentrism. Advocates of relational approaches sometimes criticize sentientism for reproducing speciesism. By privileging characteristics associated with humans and animals, they argue, sentientism may unfairly exclude artificial beings from the moral community.

Jung responds by relocating the boundary. Sentientism does not grant moral status because something belongs to the human species. Animals also possess moral status because they are capable of experience. Likewise, if a future artificial system genuinely possessed phenomenal consciousness, sentientism would have no principled reason to exclude it merely because it was artificial.

The relevant boundary is therefore not natural versus artificial, but sentient versus non-sentient.

Jung then makes a more provocative move: the deeper form of anthropocentrism may lie in imagining AI itself in the image of humanity. From the Turing test to AGI and superintelligence, human intelligence has repeatedly served both as the benchmark of AI development and as the imagined direction toward which AI is supposed to evolve. AI is expected first to reach human-level intelligence and then to exceed it.

Even superintelligence remains embedded in this anthropomorphic framework. It is imagined as a godlike savior or demonic destroyer—still essentially a magnified human-style agent.

Jung suggests that science fiction has played an important role in sustaining this image. Since Frankenstein, stories have repeatedly imagined artificial beings that think, suffer, desire, and struggle for recognition like human characters. This is partly a narrative necessity: a being that does not think or act in recognizable human terms is difficult to turn into a protagonist.

Yet the logic of narrative need not describe the ontology of real AI.

If we look instead at actual architectures and computational systems, artificial intelligence has developed as something profoundly different from human beings. Respecting AI’s otherness may therefore require resisting the impulse to interpret every sophisticated behavior through familiar human psychological categories.

A Question That Comes Before “Should AI Have Rights?”

The paper ultimately changes the order of the debate. Before asking whether AI should be granted moral status, Jung argues that we need to ask whether it is technologically possible to create AI with phenomenal consciousness.

If the answer eventually becomes yes, a second and potentially more troubling question follows: should we create such beings at all?

An AI without phenomenal consciousness may remain outside the category of moral patients no matter how sophisticated its behavior becomes. But if humans intentionally create artificial beings capable of suffering, frustration, desire, and welfare, the moral situation changes fundamentally. We would be manufacturing beings for whom things can genuinely go well or badly.

At that point, the ethical question would no longer concern merely how humans should treat increasingly convincing machines. It would concern whether humans are morally justified in creating new subjects of suffering for economic convenience or technological benefit.

The temporal structure of AI ethics therefore changes. Today’s problem concerns machines that increasingly resemble subjects in their behavior. A future problem may concern whether we deliberately bring actual experiencing subjects into existence.

The most important criterion proposed by Jung’s paper can therefore be expressed as a shift from asking what AI can do to asking what can happen to AI.

The same external event may merely alter computational states in one system while producing genuine suffering in another. If so, the morally decisive difference does not lie in the sophistication of their performance but in whether there is a subject for whom the event is experienced.

Rather than watching only for the moment when AI becomes sufficiently similar to us, Jung asks us to look for a much harder boundary: whether somewhere inside the machinery, there has actually emerged someone—or something—for whom the world can feel like anything at all.