Ted Chiang recently wrote a sharp Atlantic essay arguing that today’s AI systems are not conscious, and that treating them as morally meaningful agents is both philosophically confused and politically convenient for the companies building them: “No, Artificial Intelligence Is Not Conscious”.
His strongest point, to my mind, is that fluent conversational output can make us misidentify the thing in front of us. A language model can produce the textual performance of sadness, understanding, and concern without giving us any good reason to think there is a subject experiencing those things.
On that view, the danger is that responsibility will be displaced from the humans and institutions deploying these systems onto the fictional character the system has been trained to perform.
I largely agree with this argument. I also think claims about AI need the technical details right. Otherwise, you risk losing the people most likely to notice the fiddly bits. Chiang’s handling of those details is loose enough to raise some eyebrows. My expertise is limited, though I know enough to be annoying about the fiddly bits.
So, in the spirit of being a pedant:
- Chiang is being reductive about next-token prediction.
- Closely connected to this, he underplays the distance between phone autocomplete and large transformer systems trained with massive pretraining, RLHF/RLAIF-style alignment, and so on. Saying that modern models are impressive does not quite scratch the surface here.
- As far as I can tell, and someone please correct me if I am wrong, AlphaFold is a specialized structure-prediction system, which is quite a bit different technically from a conversational autoregressive text generator.
Does any of this matter for his broader argument? Probably not, except rhetorically. But rhetoric matters here. There is now a whole genre of writing that claims AI is or is not this or that while oversimplifying what is actually going on in the LLM space. Often, it ends up arguing against something closer to a 2022-style chatbot abstraction than the messier systems people are actually building and using now.
I find that concerning. I do not think these systems are conscious. I do not think their fluent self-presentation should be mistaken for interiority, moral agency, or understanding in the human sense. I also want the skeptical case to be the strongest version of itself. That means resisting both kinds of mystification: the industry’s tendency to inflate the technology into something like personhood, and the critic’s tendency to flatten the technology into something too simple.
This sits near AI sycophancy and self-incurred immaturity, Radical AI defaults, and Writing with AI responsibly.