I read a recent essay in The Atlantic by Ted Chiang, the science fiction writer known for Stories of Your Life and Others and Exhalation: “No, Artificial Intelligence Is Not Conscious”. The essay offers several thought experiments in support of a confident negative verdict. The experiments do not, individually or together, deliver the verdict.

This post engages only with the first half of the essay, on whether LLMs could be conscious. The second half, on Anthropic’s Claude constitution, is about corporate ethics and product design and deserves its own treatment.

The character and the system

Chiang’s central thought experiment runs as follows. Imagine an LLM prompted with “The following is a conversation between Julius Caesar and Genghis Khan.” It produces a coherent dialogue. Nobody would conclude that the LLM has summoned the historical figures into existence. They are characters in a piece of generated fiction. Now replace the prompt with “The following is a conversation between a helpful AI chatbot and a user.” Nothing fundamental has changed. The chatbot is still a character. Let an actual user enter the user turns and the situation remains: she is collaboratively authoring a transcript with a sentence-continuation engine.

The experiment lands one conclusion firmly. The persona on the screen is fictional. The chatbot character speaks fluently in the first person, says “I think” and “I feel,” and presents itself as a stable conversation partner, but that stability is the literary stability of a well-drawn fictional persona, not the metaphysical stability of a self.

What the experiment does not show is anything about the system running underneath. The persona lives in the transcript and is whatever the prompt directs the model to produce. The system that produces the transcript is a different object: a running model with its own internal computations, activations updated turn by turn, patterns of state that exist regardless of which character the prompt assigns. What Chiang’s experiment establishes is that the assistant persona has the same fictional status as the Caesar persona. That is a claim about prompts and personas. It is not a claim about the system, and it does not show that the system has no inner life when it generates either.

A sharper version of the same point: role-play, by itself, is not evidence against consciousness. Conscious beings role-play constantly. A teacher in front of a class projects a teaching persona. A witness on the stand projects a witness persona. An actor on a stage projects whatever character the script demands. The voice the audience hears is not the inner voice of the person, but no one concludes from the existence of the role that the player is empty.

For LLMs the fraction of output that is role-play may be overwhelming, perhaps total. That widens the gap between role and player. It does not eliminate the player. Chiang’s argument needs the move from “all role-play” to “no player”, and the experiment by itself does not license that move.

“One word at a time”

Chiang then explains the mechanism: an LLM generates one word at a time, and the model is run dozens of times to produce a single sentence. The mechanism description is accurate. The same logic, applied to brains, would rule out human consciousness too.

A brain decomposes into local mechanism. Synapses fire, ion channels open, neurotransmitters cross gaps, hormones diffuse. Each event is a low-level physical occurrence with no obvious connection to experience. “One word at a time” no more settles consciousness than “one spike at a time” does. The relevant question is not whether the system can be described at a mechanism level. Everything can. The question is whether something in how the process works, taken at a higher level, gives rise to experience, and a low-level description does not by itself answer that.

The same pattern appeared in Mustafa Suleyman’s essay on seemingly conscious AI, which I discussed in an earlier post. There an analogy about simulated rain played the role sequential generation plays here. In both, an evidential warning is converted into a metaphysical verdict the warning cannot support.

The Word document argument

Chiang then offers an analogy. Being open to LLM consciousness, he writes, is the same as being open to the idea that multiple consciousnesses lie dormant in every Word document containing a conversational transcript, awakened each time the file is loaded.

The analogy works against one target. If somebody claims that the characters depicted in the transcript are conscious, then yes, opening a Word file containing the same transcript would have to summon them too. The transcript carries no consciousness either way. Chiang has won that argument.

But the distinction from the Caesar setup applies here too. A running LLM is not a Word document. The transcript is the static output of a process; the running model is the process itself. The Word analogy refutes transcript-level consciousness, not the contested question about the process underneath. Storing a transcript and running the model that produced it are not the same kind of event, any more than storing a recording of a conversation is the same kind of event as the conversation itself.

Alpha Centauri and the developmental ladder

Chiang then asks what would convince him that a computer program is conscious. He answers with an analogy. Suppose someone showed him a video of an astronaut orbiting Alpha Centauri. No detail in the video itself would be enough to convince him it was real. He would need to have already seen evidence that astronauts have reached Mars, the moons of Jupiter, the moons of Saturn, and the orbit of Pluto. An observation gets its weight from the chain of precursor achievements that make it credible.

For consciousness, the precursor chain Chiang asks for runs through embodiment, navigation, novelty handling at the level of a mouse, social dynamics at the level of wolves, tool use at the level of chimpanzees, and finally nonlinguistic communication taught by humans and surviving the kind of scrutiny animal-communication research has had to endure. Only after all of that does language come into the picture.

The demand structure is essentially Popperian. As I argued in an earlier post on Popper and falsifiability, the productive way to handle the AI consciousness question is to break it into specific falsifiable sub-claims and test them one at a time, instead of arguing about the unfalsifiable headline. The specific ladder Chiang offers does not fit the systems it is meant to test.

The first step on the ladder is the familiar embodiment argument: consciousness requires a body, emotions presuppose physiological events such as cortisol release, and the path to artificial consciousness must run through lizards, mice, wolves, and chimps before it touches language. The position is widely held and has been made more carefully by others. Four earlier posts on this blog covered the most careful versions, and they all stop at the same place: they name a property doing constitutive work but do not pin it down. Anil Seth argues that “the silicon-based digital computers we are familiar with” cannot be conscious, while leaving other substrates open. Ned Block locates the matter in biological realizers without saying which property does the work. Alexander Lerchner’s DeepMind paper shifts the target from substrate to computation itself, then leaves the instantiating physics unspecified. Antonio Damasio, whose homeostasis-based theory I covered in an earlier post on Michael Pollan’s consciousness tour, grounds emotions in body regulation. His is the most direct cousin of Chiang’s claim, and the one that comes closest to spelling out what the body has to do. Even Damasio stops short of asserting strict necessity. Chiang asserts the necessity claim as a personal credo (“I believe desires and emotions are necessary for consciousness”) and runs the argument from there, without naming what the others openly leave unspecified.

The necessity claim does not survive familiar cases. A person with locked-in syndrome has lost almost all motor behavior, including the toolmaking and social interaction Chiang’s ladder requires. The person remains conscious. Whatever developmental role embodiment played in producing that consciousness, the role is now invisible in the moment-to-moment picture. Embodiment in this case is at most a precursor, not the property in which consciousness consists.

The ladder also picks rungs to suit its conclusion: it demands biological precursors LLMs lack (embodiment, lizard survival, mouse novelty, wolf sociality, chimp tool use) and omits the precursors LLMs already clear. Earlier symbolic AI tried for decades to handle open-ended natural language, write working code from informal descriptions, do common-sense reasoning, and follow multi-step instructions across unfamiliar domains. Symbolic AI did not solve any of them. Current frontier models do all of them. An honest list would include the capabilities today’s systems have demonstrated, not only the ones they lack. The GPT lineage itself, from GPT-2 to GPT-5, forms the kind of precursor chain Chiang’s analogy asks for: each generation added capabilities the previous one lacked, with steps that are visible and reproducible. As a proof that no other route to artificial consciousness exists, the ladder is also too earth-bound. Evolution found one path. The claim that there is only one is Chiang’s to argue, not assume.

Text as a deepfake medium

Chiang argues that when it comes to consciousness, text should be regarded as a deepfake medium. Generating fluent dialogue that resembles a conversation between conscious beings is now easy. Whether anyone knows how to build a system that is actually conscious is a separate question, and no current technology answers it.

As epistemology, this is correct. Until recently, fluent text was something only humans could produce at scale. That is no longer true. The likelihood of a fluent conscious-sounding self-report given a system trained on enormous quantities of conscious-sounding text is high regardless of whether the system has inner states. Verbal evidence is therefore an unreliable signal for the question, in either direction.

As ontology, the framing quietly assumes its conclusion. To call LLM output a deepfake of a conversation between conscious beings presumes there is no conscious party at one end. Whether there is or is not is exactly the question, and the framing presupposes the answer.

Read as a warning about evidence, the deepfake framing is the right tool. Self-report has become a low-bandwidth signal, shaped by prompt context as much as by anything resembling introspection. That conclusion is solid. Ruling out inner states in the systems producing the text would be a further claim, and the evidence does not reach that far.

The undefined target

One feature runs through all of this. Chiang never says what he means by “consciousness.” He uses the term throughout, sometimes paired with “subjective experience” or “inner life,” sometimes with “moral patienthood,” sometimes with “feelings” or “desires and emotions.” These are not the same property. Phenomenal consciousness, access consciousness, self-awareness, sentience, and intentionality have different definitions and different empirical signatures, and the LLM question is different for each. Without a target, the verdict “they do not have it” is ambiguous between several claims, each of which would need its own argument.

Where this leaves it

Two things follow. The first is that conversation is the wrong evidence for the question. The character is fictional, the self-reports track context, and fluent text is now easy to produce. That rules out the conversational signal as decisive in either direction, but does not, by itself, settle anything about the systems producing it.

The second is that “the character is fictional” is not the same as “the process is empty.” The Caesar example shows that the entity the user appears to be talking to is a generated role. It does not show that the system playing the role lacks inner states. Whether it has, or could have, anything that matters morally is a question about the process itself, and no argument in the essay settles it.

A familiar objection: the default for newly engineered systems is no moral status until shown otherwise. Nobody asks whether a thermostat or a toaster might be conscious, and ordinarily the burden of proof falls on the side claiming consciousness. That default does not fit current frontier models. Their capabilities place them well outside the engineering categories where default exclusion applies. The consciousness question they raise has to be settled by argument, not by which category they happen to fall into.

On the reading above, Chiang’s essay does not provide such an argument. He closes by telling readers they “can safely ignore the question of their being conscious.” That confidence outruns the case. Until an argument actually closes the question, “no” is no more an answer than “yes”.

A small final irony. The audio version of Chiang’s essay, available on The Atlantic website, opens with a spoken announcement that it was produced by ElevenLabs and Noa, using an AI voice. The essay arguing that fluent generated language should not be mistaken for a conversation partner is itself delivered, on one of its rendering paths, by a synthesized voice. The economics of synthesis have reached the channel by which the warning travels.