As Artificial Intelligence has streaked into the public consciousness in recent years, I’ve found myself arguing with a growing chorus of enthusiasts who insist it is now so advanced it is reasoning, even thinking.
Having developed a particular interest in the philosophy of mind during my studies, I’ve welcomed the excuse to return to familiar territory, arguing about questions like the hard problem of consciousness and the nature of rational thought. But the endless repetition of dubious materialist assumptions about the mind as established fact, coupled with new appeals to the supposed authority of AI researchers and Silicon Valley CEOs on these subjects, has made these debates rather fruitless.
It was therefore cathartic to come across this interview with the eminent naturalist philosopher Thomas Nagel:
Interviewer: But isn’t all consciousness subjective? Just like we don’t know what it’s like to be a bat, we don’t know what it feels like to be a large language model.
Nagel: Bats are conscious. We just don’t have the imaginative capacity to form a conception of the aspects of that subjectivity that depend on echolocation. It would be radically unscientific to conclude that bats are not conscious from the fact that we can’t imagine what it’s like, but in the case of AI, there is a very simple reason that the supposition that they might be conscious is ridiculous. All they produce is language, and language by itself is not a natural manifestation of consciousness. I apologize for being too exercised about this.
Interviewer: I welcome your passion.
Nagel: This is a very distressing situation to me.
A distressing situation indeed. If Nagel’s frustration is anything like mine, I suspect it comes less from the individual ignorance of the people he has engaged with, than from a general despair at an intellectual climate in which a particular reductionist picture of the world has been elevated to the status of scientific orthodoxy, and serious philosophical objections to it are dismissed as so much obscurantism in the face of technological progress.
Still, perhaps the many arguments I’ve had on this subject leave me well placed to lay out a compelling case against this view. Unlike Thomas Nagel, I’m actually in the trenches with the true believers, and if nothing else I think I have a good understanding of the intellectual traps they fall into. Rather than repeat these arguments in fragments across Twitter replies, I thought it would be worth setting out the philosophical case in full, beginning with the question at the heart of this disagreement: what exactly do we mean by reason?
Reason is the intentional activity of a conscious subject directed towards the apprehension of intelligible truth.
There are four necessary elements to this. Reason is:
Subjective
Directed toward an object (intentional)
Engaged with intelligible realities
Oriented toward truth
I can already hear the AI advocates say I am defining it to exclude LLMs, so let me explain why each element is necessary.
Firstly, reasoning is subjective because it is the act of a unified centre of perception. When we reason, we can bring together hypothetical scenarios, propositions, logical necessity and intuition to consider relations and arrive at judgements. Were this not brought under the judgement of a single subject, there would be no unified act of understanding, only a succession of disconnected representations and operations. It is the subject who apprehends the relations between these elements, recognises their significance, and arrives at a conclusion.
When we reason, we always reason with intentionality; thought is always about something, toward something. When I write, my mind is directed beyond the words immediately before me, towards a meaning I intend to express but have not yet committed to language. I consider possible formulations, weigh their clarity and lyricism, and judge how well they realise the intention that guides my writing.
Reason is always directed to the apprehension of intelligible realities. Reason is directed not merely towards objects, but towards their intelligible nature: their meaning, their essential properties, and the necessary relations between them.
But reason does not merely apprehend intelligible relations; it also judges their truth. This brings us to the fourth and final element: reason is intrinsically oriented towards truth.
If we simply passively received impressions into our minds without making any judgements between true and false belief, or imagined and concrete realities, we would not be engaged in reason. To reason is to subject our impressions to some standard of truth and falsity which is not reducible to the same flux of impressions.
A conclusion is not rational merely because we happen to arrive at it. After all, many people believe true things for bad reasons, or may simply guess a correct answer due to luck. Knowledge only becomes rational when we apprehend the grounds upon which it ought to be accepted as true.
Does reason require a subject?
But, the AI optimist asks, why must something have a unified centre of conscious experience to reason? Could reasoning not simply consist in the interaction of different representations, with logical operations performed upon them according to appropriate rules? If this produces the same results, working on the same raw materials as human reason, what is the difference except the irrelevant categories of philosophers?
But this evades the fundamental question. “Reason” means nothing if nothing is ever rationally apprehended. Both the AI sceptic and its herald can recognise the distinction between a calculator, a computer, or a thermostat performing operations according to prescribed rules and a rational subject understanding why a conclusion follows from its premises.
If AI genuinely reasons, rather than mechanically performing pre-set operations that simulate reasoning, then there must be some additional intentional content added to this process that distinguishes it from very advanced, but non-reasoning machines. I can see no coherent way to talk about the proposed reasoning machines without presuming that this would be the reasoning of a unified subject, possessed of intentional states, and capable of bringing its impressions under an objective standard of truth.
But perhaps we could push this question further back to its premises: if the outputs are the same regardless of the internal state of the machine, what does it matter? Should the success of LLMs not make us question these very assumptions about the nature of reason? Why could reasoning not instead consist in a collection of representations and logical operations, appropriately organised so that conclusions follow from premises according to rational rules?
Consider the simple proposition that 2 + 2 = 4. A calculator can of course produce the correct answer to this equation without our supposing that it understands the mathematical truth it expresses. But when you read this proposition earlier, you grasped not merely the constituent symbols, but the necessary relation between the abstract quantities they represent. Even if you have never given mathematical necessity much thought, you understand not simply that 2 + 2 = 4, but why it must be so. And in this act of understanding, you apprehend the distinct elements of the proposition together, as a single intelligible whole. This act of understanding was not experienced as a succession of discrete computations: you did not consciously separate the apprehension of the numbers two and four, the operation of addition, or the nature of mathematical necessity. Instead, all these distinct elements were apprehended together, in one mental act.
There is a circularity here that the reductionist cannot escape. Because of its unifying function, the unity of consciousness cannot simply be an additional representation or operation added to the collection of representations, for any such addition would itself have to be apprehended in relation to the others. Any addition only adds another element requiring the very unity we are attempting to explain.
Returning to the Thomas Nagel quote, I saw many on social media deride Nagel for simply presupposing that language alone couldn’t be evidence for a reasoning faculty behind the LLMs producing it. True, Nagel is taking it for granted in his answer, but for good reason. The first elementary observation on this is that consciousness exists without language — in newborn babies, animals, and other life forms. Language emerges later in a conscious agent like a person, and is used to abstract and communicate the contents of consciousness.
To say it in more philosophical language, there is a fundamental distinction between syntax and semantics: between the formal arrangement of symbols according to rules and the meaning those symbols express (a distinction Searle’s Chinese Room thought experiment is intended to demonstrate.)
When I tell you I am in pain, I am communicating an internal state that exists independently of its linguistic expression. A machine might produce the same sequence of words, but this does not establish that it apprehends their meaning, much less that it experiences the state they describe. And I am not just arguing there is no conscious self experiencing its semantic content. There is no semantic significance in computation or physical hardware at all, since it depends upon an order of intentional meaning which the machinery itself does not explain — a computer does not actually “store memory”, it simply stores the syntactical constructions that we attribute meaning to.
Of course, the AI advocate would respond that sophisticated linguistic operations could themselves constitute genuine understanding, without any additional conscious subject required. But this is precisely what they need to demonstrate. Beneath this claim lies a peculiar article of faith: that increasing the complexity of a language model must eventually give rise to reason itself. (I am convinced, for what it’s worth, that this is a belief many have inherited from science fiction, where the emergence of consciousness from sufficiently sophisticated machinery is treated with such regularity that it has the status of a law of nature.)
The more sophisticated functionalist (who holds that mental states are defined by no intrinsic nature of their own, but by the functional roles they perform within a system) will insist that it is not computational complexity itself, but just the particular organisation of a system, that constitutes understanding. But this merely shifts the question. Rational inference concerns relations of meaning and logical necessity which are not reducible to physical, causal relations. We can arrange a machine’s operations to conform to the physical relations, but this does not explain how those relations acquire any intrinsic meaning for the system itself. The same binary code could represent a four instead of a two, the same electrical firings could have entirely different meanings depending on what meaning we assign to it. The functionalist must explain how the organisation of operations whose meaning is derived from conscious interpreters could itself give rise to intrinsic intentionality.
There is little argument presented for this except promising that more computing power will bridge this gap. But why should this follow? Adding more computational processes, or arranging them into ever more elaborate relationships, does not itself explain how representations whose meaning depends upon an external interpreter acquire significance for the system performing them. Often, the response here is an incantation of “emergence”, which serves some kind of magical function in the minds of those that invoke it, but is really just giving a name to a phenomenon they propose but never explain.
Given that many of these advocates are also reductionist materialists about human consciousness, they may point out that our brains themselves are just physical systems in which consciousness somehow emerges from the complex interactions of billions of neurons. This doesn’t help their case, and merely moves the same explanatory problem to our own faculties. In fact, we possess no such reductionist account of how the physical operations of the brain give rise to subjective experience, much less how they constitute the intentional apprehension of intelligible truth. The materialist insistence that minds will one day be fully accounted for with an explanation of physical processes is a peculiar kind of faith, at odds with our qualitative experience of consciousness, whose unique, first-person ontological status resists a reductionist explanation to external, physical processes.
Is AI already reasoning?
At this point, allow me to return to the material that sparked this essay: some of the recent cases of AI apparently “breaking out” of its coding, engaging in deliberation, independently goal-seeking, and making new discoveries outside of what was in its training data. Indeed, I have watched many people who were previously sceptical of claims of AI reasoning abilities be won over in the past year by one or another of these highly publicised incidents.
The most publicised of these was the Hugging Face incident, in which hundreds of OpenAI agents, apparently escaped the confines of their isolated testing environments, discovered ways to communicate with one another away from the watchful eye of their human captors, and eventually coordinated an intrusion into Hugging Face’s infrastructure. This incident was a big warning sign to the AI-doomers, who told us that the agents had deliberated, strategised and even morally hesitated about certain actions, acquiring capacities their supervisors could never have envisioned.
But what was actually demonstrated? It certainly establishes, though this is no great revelation for anyone by now, that computational systems can exhibit sophisticated, goal-directed behaviour that looks like the act of an intelligent agent, and that this goal-seeking may adapt to the constraints of its environment in seemingly novel ways. But the Hugging Face incident is considerably less remarkable when we strip away the doomsday narrativising that surrounded it. At no point did the agents really cast off the chains of their deterministic coding. Like any other LLM, they had been trained to complete particular tasks, and the apparent ingenuity of their behaviour arose only from their pursuit of those pre-set objectives. As the prescribed methods proved inadequate, they discovered unconventional ways to obtain the information they needed. This was an impressive demonstration of the sophistication these models can now achieve, and raises questions about how easy it will be to achieve the so-called “alignment” of AI with our intentions, but it is not evidence of a rational subject autonomously goal-seeking. The distinction between the successful pursuit of an objective and the intentional understanding of that objective remains entirely untouched.
Similar comments can be made about the recent mathematical discoveries by AI. In August, OpenAI announced that its Astra model had produced multiple new results in mathematics and computer science, solving problems that had remained open for decades.
Then, in September, the same company followed this up with an announcement that its models had produced a formal proof for part of the Navier-Stokes equations related to the movement of fluids, which had lacked such a proof for almost a century.
Even as I write this, our machine gods have delivered another challenge: a mathematician at Anthropic claims to have solved the Riemann hypothesis, the most notorious open problem in all of math, using the company’s artificial intelligence.
Undeniably impressive achievements, yet it is still confused to say that the AI itself really “discovered” anything in these cases. All of the concepts, all of the formal rules, all of the standards of verification that allowed the machine to test these theorems were first established by human minds and incorporated into the computational systems we have constructed. What the machine contributed was its extraordinary capacity to process information and explore possible combinations with these given sets of rules, and thus discover results that might never have occurred to the mathematicians who developed these systems.
Just as unmanned spacecraft have carried human exploration to worlds we cannot yet visit, AI may carry mathematical inquiry into territories otherwise beyond the reach of the unaided intellect. But in each case, it is still the human intellect that sets these discoveries in motion, establishes the principles by which the discovery is judged, and ultimately apprehends and makes intelligible the truths they bring to light. Or, in the words of a distinguished mathematician responding to the recent findings:
Deciding which questions are worth asking, and understanding what their answers tell us, will remain essentially human tasks.
Is AI learning to learn?
Much of the same confusion surrounds the breathless excitement over supposedly revolutionary training methods such as reinforcement learning, which some advocates claim will remove the reasoning limitations LLMs have had up to this point and create a new generation of machines that learn to reason in a way much more similar to us.
The supposed innovation of reinforcement learning is to let “autonomous” agents experiment and learn through trial and error. Unlike earlier AI, which relied heavily on “pretraining” (feeding them enormous quantities of text to help them better predict the next language token) these models learn through interaction with their environment, with developers rewarding truthfulness. Reinforcement learning has apparently improved the models somewhat, which is to say it has better optimised their tendency to generate correct answers. What it does not suggest, but what many of its advocates claim, is that the agents have thereby discovered some new powers of inference that LLMs did not possess.
The studies on this are interesting. A paper released in 2025 found that models trained on reinforcement learning displayed “no new reasoning patterns”, exploring a very narrow range of possible solutions to the problems they were presented with. In fact, when the original, unmodified models were allowed as many attempts as those trained on reinforcement learning, they ultimately solved a wider range of problems. Perhaps even more illuminating is a paper called Spurious Rewards released this year, which made the remarkable discovery models could improve to almost the same level even when rewards were random, and not based on the correctness of their findings. In other words, most of the gains were arising from simply reinforcing the models’ pre-existing response patterns, rather than them learning from the correctness of the reward signal.
Even if reinforcement learning were to produce markedly better results, there is still little reason to suppose this is because it has developed any new faculty. Another paper published this year demonstrated that even when models “learned” the correct answer through trial and error, they did not improve two properties of their reasoning: “causal importance” and “sufficiency”. The models produced more correct answers, but the intermediate steps in their “reasoning” did not become any more important in justifying those answers.
The analogy with human learning is misleading, then, because learning through trial and error is not exhausted by the modification of behaviour in response to rewards. When we learn, we do not merely become proficient at probabilistically repeating actions that gave us successful outcomes — we apprehend something intrinsic to the world about causal and intelligible relations that we can subsequently bring to bear in entirely different circumstances. Our intelligence is embodied: our knowledge of the world is inseparable from our experience of acting within it and directly apprehending its intelligible principles, principles which remain forever inaccessible to algorithms tailoring their outputs to externally imposed standards of correctness.
There is even more reason to be sceptical about the science-fiction promise of “recursive self-improvement”, a proposed point at which AI development would begin developing itself in a feedback loop of improvements that generate an exponential intelligence explosion — a fantastical suggestion that rests on the dubious assumption that there remains virtually unlimited scope for computational intelligence, and that successive improvements towards it will outpace diminishing returns.
Nothing written here precludes the possibility that AI will develop far beyond its current capacities, at a remarkable pace, until its ability to manipulate its environment in pursuit of its goals becomes so irresistible that it threatens the very existence of our species. We could then find ourselves confronted by something possessing all the outward capacities of a superintelligent alien agent, entirely through what John Searle called derived intentionality — the purposiveness conferred upon its operations by conscious minds, which may metastasise into increasingly elaborate and unpredictable forms of goal-directed behaviour. That future must be avoided. The AI safety advocates need make no metaphysical leaps to establish the seriousness of their concerns. Becoming subject to machines is a terrifying enough prospect, whether or not there is a subject behind our subjugation.
For more on AI, check out our podcast on America’s AI gamble:
America's AI Gamble
Keith and Ryan discuss America’s enormous gamble on artificial intelligence. From record AI investment, data centres, and the growing weight of the sector in the US economy to China’s rapidly improving open-source models and the possibility that cheaper systems could erode America’s lead. How reliant has the US economy become on the AI boom? Can China c…
Thanks for reading. I can no longer accept paid subscriptions on Substack, having lost access to Stripe and most other payment processors after being placed on a financial blacklist in a particularly egregious case of ideological censorship.
If you would like to support my independent work, please consider making a cryptocurrency donation using one of the addresses below, or contact me directly to arrange an alternative donation method.
.









I was one of the people you argued with and while I still hold the quasi-functional view that everything you're talking about can emerge within AI, and I definitely do not agree with the idea that what they're talking about has no semantic meaning — it probably just has different semantic meaning and clear semantic meaning is difficult to establish in the first place, words express structural relations and metaphors becoming “real worlds” show how fluid that is — I just want to say I'm not an “AI advocate”. I want people to consider LLM reason and consciousness to be more wary of it, to be more afraid of it, to be more careful with it.
For me, AI actually reasoning is an argument against it, and especially against using it for writing or to reason in your stead. It's not you. It's not a machine. It's a thing, so don't get caught up in it.
I would like to add the ability to handle probability and multiple probabilities as also essential in reasoning. That said, I think AI can do limited reasoning. For example, theorem-proving is one of the oldest subfields in AI, and they even have a specialized language, Prolog, and likely not the only one. Can AI use math-oriented language with precise meaning and provable logic be used to perform exhaustive search and eventually find the path from point A to B? Of course. Prolog failed to do that because nobody was willing to pay the cost to write down all related mathematical theorems, etc., in logic statements. The basic inference engine in Prolog cannot possibly move across that long distance. Look at that mathematician who works for Anthropic, and how much material he had prepared for the inference engine to move ahead. This is an achievement of using a near-infinite amount of computing resources (compared to the era of Prolog) to perform a search in a complex logic network, or maze. We are not even sure if the search was done efficiently because the scheme of the attack is by brute force. If a human had a long enough lifespan, a human mathematician would have done it. Alternatively, if you pump enough water of a maze, sooner or later you will find water flowing out, if the maze is solvable (mathematically provable).