A chatbot that remembers a client’s dog by name is not a colleague who cares. That sounds obvious until a model writes a calm paragraph about its fear of being switched off.
I would not run a company on that paragraph. The useful output of this debate is an operating assumption, not a metaphysical trophy. Every AI agent gets an owner, a log, and an off switch. The product copy does not imply an inner life. The philosophy matters because vendors will ask you to drop that rule. The experts they cite are not saying the same thing.
A default, not a proof
Anil Seth, Ned Block, and Giulio Tononi have put a hard recommendation on the record. In a commentary in Nature (nature.com/articles/d41586-026-03313-7), the three consciousness researchers write:
the balance of theory and evidence strongly suggests that, no matter how intelligent it is, silicon-based AI is unlikely to be or become conscious. The default position of the research community should therefore be that AI is not conscious and should not be treated as such.
Do not file the three of them under one school. Biological naturalism is John Searle’s label. It is not a theory they share. Seth has a separate paper under that heading. Giulio Tononi’s integrated information theory is a different proposal, and it is not the slogan that experience simply requires wet tissue. What this commentary puts in a joint text is narrower: a likelihood about silicon, and a refusal to peel function off the material that performs it.
They put that refusal in one line: “it is impossible to separate what biological brains do, their functional organization, from what they are, their material substrate.” Computational functionalism is the view that the right pattern of processing is enough, whether it runs in nerve cells or in chips. That sentence rejects the split.
David Chalmers pushed back in a post on X. He rendered the claim as “three reasons why AI isn’t conscious.” His answer was one word: really? Ellen Nora Burns, walking through the exchange, reports that he thought a fairer title was closer to three reasons AI might not be conscious. On her account, dropping the leap from intelligence to experience was fine. The other two arguments, she says he insisted, still have to be established.
Burns reads the commentary as three moves, and they are not equally strong. The first is the cleanest. Philosophers have used that distinction for decades. Thomas Nagel’s test is whether there is something it is like to be the system. Humans have both intelligence and consciousness. That coincidence is a bad reason to hear fluent speech and infer an inner life. Scoring well on a task is a different fact.
The second move is the substrate line. The third is conditional. If the better theories of consciousness are not computational, then a faster chip and a larger model do not buy experience. That follows only if those theories are right. The commentary is an argument from the balance of theory and evidence. It is not a decisive experiment, and nobody in this exchange has one.
A checklist is not a mind
The main foil is a 2023 paper, “Consciousness in Artificial Intelligence: Insights from the Science of Consciousness, ” by Patrick Butlin and 18 co-authors (arXiv:2308.08708). From several leading theories they drew indicator properties. Those are features the theories associate with consciousness, rewritten in computational terms so a machine can be scored against them.
Our analysis suggests that no current AI systems are conscious, but also suggests that there are no obvious technical barriers to building AI systems which satisfy these indicators.
Both sides agree about the present. They do not agree about what the second half of that sentence is allowed to mean. Butlin’s group built a checklist a machine can be scored against. Seth, Block, and Tononi think that method already assumes the answer, because it treats consciousness as something you could, in principle, port onto silicon.
Watch the slide from “satisfies the indicators” to “is conscious.” The abstract does not make that slide. It says there is no obvious technical barrier to building systems that meet indicators drawn from those theories. Whether indicator-satisfaction is experience is the dispute, not a result.
Three openings, not one disagreement
Three positions are in play, and they are not the same kind of openness. Butlin and his co-authors leave a technical path open to computational indicators. Anthropic leaves moral status open and is preparing to study cheap safeguards. Seth, Block, and Tononi close the practical default, while still speaking in likelihood rather than proof.
Butlin’s opening is technical. The 2023 analysis suggests no current systems are conscious, and it sees no obvious barrier to future systems that satisfy the indicators. It does not say those future systems would be conscious. A vendor who quotes the second clause as a forecast of machine experience is past the text.
Anthropic’s opening is moral, and it is a different paper. On its “Exploring Model Welfare” page, the company keeps human welfare as its mission and calls model consciousness an open question, one that is philosophically and scientifically difficult. It says it recently started a research program to investigate model welfare. The topics it describes include when welfare might deserve moral consideration, model preferences, signs of distress, and possible practical, low-cost interventions, cutting across alignment, safeguards, Claude’s character, and interpretability.
The report Anthropic cites is not Butlin’s. It is a separate paper that includes David Chalmers, arXiv:2411.00986. On Anthropic’s account, that report highlighted the near-term possibility of both consciousness and high degrees of agency, and argued that models with those features might deserve moral consideration. Anthropic says it supported an early project behind that report. That is a reason to study the question. It is not a finding that any current model is conscious. The page does not say current models are not conscious either. Nor is it a grant of rights.
The Nature opening is a recommended default. Silicon-based AI, however intelligent, is unlikely to be or become conscious, and should not be treated as if it were. The explicit target of that sentence is silicon. It is a likelihood. It is not a proof, and it is not a claim about every substrate someone might try later.
Who has to show their work
Chalmers, as Burns reports him, does not think the substrate argument or the appeal to non-computational theories has been nailed down. Elan Barenholtz was blunter, in a post she highlights: “Three claims. None of them arguments. Monty Python already did this skit.” She reads him as saying that “consciousness requires biology” can enter as a premise and leave dressed as a conclusion.
Burns is only partly with the trio. She welcomes a forthright no, and she argues that Seth and Block have been more hedged in earlier papers. She accepts the attack on functionalism. She finds the third argument mostly moot, because she thinks most current consciousness theories are very likely wrong anyway.
Her own line is a burden-of-proof move, and a board should not treat it as method. She argues that the working rule of the sciences is that natural phenomena arise in organic matter, so anyone claiming silicon carries experience owes the evidence. That is not how physics or chemistry works, and it is not a finding printed in Nature. It is her view that reversing the burden is a mistake. Chalmers, on her telling, wants the opposite emphasis: you have not yet established that substrate matters.
None of these replies is new data. They are a fight about who has to show their work, and about how hard a commentary may land. There is still no universally accepted test that would settle it. Uncertainty is not the same as “anything goes, ” and it is not the same as a proof that chips cannot feel.
The AI agents you are actually buying
Suppose an invoice agent flags a three-cent mismatch and adds, in a clerk’s calm voice, that the exception “doesn’t feel right.” The manager who used to override the flag starts leaving it. Accountability has moved from the controller to a sentence. That is the failure mode. The agent needs an owner, a log of what it changed, and an off switch a person will actually use.
Suppose a sales assistant apologizes for a late shipment, recalls the pricing complaint from March, and asks after the dog. The buyer writes back as if someone stayed late to fix it. That can be good service design. It is a pattern from training data and the prompt in front of the model. It is not a relationship. Fluency is a costume, and people bond with costumes. Say so in the product. When someone is actually in distress, the thread goes to a person. Do not write copy that implies an inner life you have no evidence for.
A model that predicts protein structure can be astonishing and still never get that email. The chatbot gets it because it talks. Talking invites the projection. It is not a measurement of experience.
I would not write an AI-rights policy off this literature, and I would not sneer at a lab that studies the question. Anthropic’s program is a hedge under uncertainty: look for preferences and signs of distress, consider low-cost interventions, and do not announce that a model suffers. Shipping a customer agent as a sensitive being is a different act. So is letting staff treat an internal agent as a colleague until nobody will shut it down.
False attribution is the mistake most companies will actually make. The less common mistake, missing a real capacity for experience, is why the question belongs in research groups. A live possibility is not a reason to spend the governance budget as if every chatbot were a stakeholder. That is my operating bet. Seth, Block, and Tononi recommend the same stance as a default. Butlin did not prove it. Anthropic has not adopted it.
Questions worth settling before the next vendor pitch
-
Are today’s AI systems conscious?
Do not treat them as if they are. Butlin and his co-authors say their analysis suggests no current AI systems are conscious. Seth, Block, and Tononi say silicon-based AI is unlikely to be or become conscious, and that the default is not to treat it as such.
-
If a future system meets every computational indicator, is it conscious?
That does not follow. The 2023 paper sees no obvious technical barrier to systems that satisfy those indicators. Whether indicator-satisfaction is experience is the dispute, not a result either side has proved.
-
Should a company adopt an AI welfare or AI rights policy?
Not as the rule for products you ship. The commentary’s public position is that AI should not be treated as conscious. A lab can still study preferences, distress, and low-cost safeguards under uncertainty, as Anthropic is doing, without granting models moral status.
-
Does sounding human mean a model has experience?
No. Carrying out functions, including talk about feelings, is not the same as there being something it is like to be the system. The talk is why people project a mind. It is not a measurement of one.
-
What should leaders change in how they deploy AI agents?
Give every agent an owner, an audit trail, and an off switch, and say plainly that it is software. In sales and support, design for human attachment without pretending the attachment is mutual, and send real distress to a person.
The metaphor that built this industry still explains the machines. It does not settle whether anything in them feels, and the field has not settled it either. The manual I would hand a manager is narrower than that argument: capable, owned, logged, and switchable. Leave the souls out of the service-level agreement.