Everyone is out here acting like "predicting the next thing" is somehow fundamentally irrelevant to "human thinking" and it is simply not the case.
What does it mean to say that we humans act with intent? It means that we have some expectation or prediction about how our actions will effect the next thing, and choose our actions based on how much we like that effect. The ability to predict is fundamental to our ability to act intentionally.
So in my mind: even if you grant all the AI-naysayer's complaints about how LLMs aren't "actually" thinking, you can still believe that they will end up being a component in a system which actually "does" think.
Are you a stream of words or are your words the “simplistic” projection of your abstract thoughts? I don’t at all discount the importance of language in so many things, but the question that matters is whether statistical models of language can ever “learn” abstract thought, or become part of a system which uses them as a tool.
My personal assessment is that LLMs can do neither.
Words are the "simplistic" projection of an LLM's abstract thoughts.
An LLM has: words in its input plane, words in its output plane, and A LOT of cross-linked internals between the two.
Those internals aren't "words" at all - and it's where most of the "action" happens. It's how LLMs can do things like translate from language to language, or recall knowledge they only encountered in English in the training data while speaking German.
LLMs are compression and prediction. The most efficient way to (lossfully) compress most things is by actually understanding them. Not saying LLMs are doing a good job of that, but that is the fundamental mechanism here.
It's the other way around. Human learning would appear to amount to very efficient compression. A world model would appear to be a particular sort of highly compressed data set that has particular properties.
This is a case where it's going to be next to impossible to provide proof that no counterexamples exist. Conversely, if what I've written there is wrong then a single counterexample will likely suffice to blow the entire thing out of the water.
No answer I give will be satisfying to you until I could come up with a rigorous mathematical definition of understanding, which is de-facto solving the hard AI problem. So there's not really point in talking about it is there?
If you're interested in why compression is like understanding in many ways, I'd suggest reading through the wikipedia article on Kolmogorov complexity.
The "cross-linked internals" only go one direction and only one token at a time, slide window and repeat. The RL layer then picks which few sequences of words are best based on human feedback in a single step. Even "thinking" is just doing this in a loop with a "think" token. It is such a ridiculously simplistic model that it is vastly closer to an adder than a human brain.
Even if they are "simplistic projections", which I don't think is the correct way to think about it, there's no reason that more LLM thoughts in middle layers can't also exist and project down at the end. Though there might be efficency issues because the latent thoughts have to be recomputed a lot.
Though I do think in human brains it's also an interplay where what we write/say also loops back into the thinking as well. Which is something which is efficient for LLMs.
> But raising kids, I can clearly see that intelligence isn't just solved by LLMs
Funny, I have the opposite experience. Like early LLMs kids tend to give specific answers to the questions they don't understand or don't really know or remember the answer to. Kids also loop (give the same reply repeatedly to different prompts), enter highly emotional states where their output is garbled (everyone loves that one), etc. And it seems impossible to correct these until they just get smarter as their brain grows.
What's even more funny is that adults tend to do all these things as well, just less often.
The lack of results so far. The observable behavior exhibits many uncanny similarities but also clearly has missing pieces.
As the person you initially responded to said, observing children growing up should make it obvious.
Or if we shift to stating the obvious, there's the minor detail that the vast majority of architectures lack the ability to learn during inference. That's one of the basic things that biological systems are capable of.
My "abstract thoughts" are a stream of words too, they just don't get sounded out.
Tbf I'd rather they weren't there in the first place.
But bodies which refuse to harbor an "interiority" are fast-tracked to destruction because they can't suf^W^W^W be productive.
Funny movie scene from somewhere. The sergeant is drilling the troops: "You, private! What do you live for!", and expects an answer along the lines of dying for one's nation or some shit. Instead, the soldier replies: "Well, to see what happens next!"
I doubt words are involved when we e.g. solve a mathematical problem.
To me, solving problems happens in a logico/aesthetical space which may be the same as when you are intellectually affected by a work of art. I don't remember myself being able to translate directly into words what I feel for a great movie or piece of music, even if in the late I can translate this "complex mental entity" into words, exactly like I can tell to someone how we need to change the architecture of a program in order to solve something after having looked up and right for a few seconds.
It seems to me that we have an inner system that is much faster than language, that creates entities that can then beslowly and sometimes painfully translated to language.
I do note that I'm not sure about any of the previous statements though'
My wordmangling and mathsolving happen in that sort of logico/aesthetical space, too!
The twist about words in particular is they are distinctly articulable symbols, i.e. you can sound 'em out - and thus, presumably, have a reasonable expectation for bearers of the same language to comprehend if not what you meant then at least some vaguely predictable meaning-cloud associated with the given speech act.
That's unlike e.g. the numbers (which are more compressed, and thus easier to get wrong), or the syntagms of a programming language (which don't even have a canonical sonic representation).
Therefore, it's usually words that are taught to a mind during the formative stages of its emergence. That is, the words that you are taught, your means of inner reflection, are still sort of an imposition from the outside.
Just consider what you life trajectory would've been if in your childhood you had refused to learn any words, or learned them and then refused to mistake them for the things they represent!
Infants and even some animals recognize their reflection in a mirror; however, practically speaking, introspection is something that one needs to be taught: after recognizing your reflection you still need to be instructed what is to be done about it.
Unfortunately, introspection needing to be taught means that introspection can be taught wrongly.
As you can see with the archetypical case of "old and wise person does something completely stupid in response to communication via digital device", a common failure mode of how people are taught introspection (and, I figure, an intentional one!) is not being able to tell apart yourself from your self, i.e. not having an intuitive sense of where the boundary lies between perception and cognition, i.e. going through life without ever learning the difference between the "you" and the "words about you".
It's extremely common, and IMO an extremely factory-farming kind of tragic.
I say it must be extremely intentional as well, because the well-known practice of using "introspection modulators" to establish some sort of perceptual point of reference (such as where the interior logicoaeshtetical space ends and exterior causalityspace begins) very often ends up with the user in, well, a cage of some sort.
What is your point exactly in regard to what I said earlier, how would you rephrase what you just said as a philosophical/epistemological statement ?
> It's extremely common
I cannot conceive this ? I am lacking the empirical knowledge you seem to have. (I don't understand your "archetypical case", I can't relate to it). I'd love a reexplanation of your point here, as your intent is unclear to me.
I didn't understand also the "introspection modulators" part :(, (a well known practice ?? I must be living on another planet haha...).
edit: or maybe that's a metaphor for "language" ??
Different people have different levels of internal monologuing or none at all. I don't generally think with words in sentences in my head, but many people I know do.
Internal monologue is a like a war correspondent's report of the daily battle. The journalist didn't plan or fight the battle, they just provided an after-the-fact description. Likewise the brain's thinking--a highly parallelized process involving billions of neurons--is not done with words.
Play a little game of "what word will I think of next?" ... just let it happen. Those word choices are fed to the monologue, they aren't a product of it.
Most of the fucking time, and I would prefer that I didn't. I even wrote that, lol.
I don't think they're really "mine", either. It's just all the stuff I heard somewhere, coalescing into potential verbalizations in response to perceiving my surroundings or introspecting my memory.
If you are a materialist positivist, well sure, the process underlying all that is some bunch of neural activation patterns or whatever; the words remain the qualia in which that process is available to my perception.
It's all cuz I grew up in a cargo cult - where not presenting the correct passwords would result in denial of sustenance, shelter, and eventually bodily integrity. While presenting the correct passwords had sufficient intimidation value to advance one's movement towards the "mock airbase" (i.e. the feeder and/or pleasure center activation button as provided during the given timeframe).
Furthermore - regardless whether I've been historically afforded any sort of choice in how to conceptualize my own thought processes, or indeed whether to have those in the first place - any entity which has actual power to determine my state of existence (think institutions, businesses, gangs, particularly capable individuals - all sorts of autonomous corpora) has no choice but to interpret me as either a sequence of words, a sequence of numbers, or some other symbol sequence (e.g. the ones printed on my identity documents, the ones recorded in my bank's database, or the metadata gathered from my online represence).
My first-person perspective, being constitutionally inaccessible to such entities, does not have practical significance to them, and is thus elided from the process of "self-determination". As far as anyone's concerned, "I" am a particular sequence of that anyone's preferred representational symbols. For example if you relate to me on the personal level, I will probably be a sequence of your emotions. Either way, what I may hypothetically be to myself is practically immaterial and therefore not a valid object of communication.
Excellent point. See also the failure of Sapir-Whorf to prove that language determines thought. I think we have plenty of evidence that, while language can influence thought, it is not thought itself. Many people invested in AI are happy to throw out decades of linguistic evidence that language and thought are separate.
I was with you up to the last sentence. By what reasoning do you claim that LLMs only consist of words? The input and output are words but all the stuff in the middle - where the magic happens - does not appear to be quite that simple.
LLMs and human brains are both just mechanisms. Why would one mechanism a priori be capable of "learning abstract thought", but no others?
If it turns out that LLMs don't model human brains well enough to qualify as "learning abstract thought" the way humans do, some future technology will do so. Human brains aren't magic, special or different.
Human brains aren’t magic in the literal sense but do have a lot of mechanisms we don’t understand.
They’re certainly special both within the individual but also as a species on this planet. There are many similar to human brains but none we know of with similar capabilities.
They’re also most obviously certainly different to LLMs both in how they work foundationally and in capability.
I definitely agree with the materialist view that we will ultimately be able to emulate the brain using computation but we’re nowhere near that yet nor should we undersell the complexity involved.
When someone says "AIs aren't really thinking" because AIs don't think like people do, what I hear is "Airplanes aren't really flying" because airplanes don't fly like birds do.
This really shows how imprecise a term 'thinking' is here. In this sense any predictive probabilistic blackbox model could be termed 'thinking'. Particularly when juxtaposed against something as concrete as flight that we have modelled extremely accurately.
that depends, if you explain the rules of the game you're playing and give the dice a goal to win the game, do they adjust the numbers they reveal according to the rules of the game?
That's a fallacy of denial of the antecedent. You are inferring from the fact that airplanes really fly that AIs really think, but it's not a logically valid inference.
Observing a common (potential) failure mode is not equivalent to asserting a logical inference. It is only a fallacy if you "P, therefore C" which GP is not (at least to my eye) doing.
I agree we shouldn't undersell or underestimate the complexity involved, but when LLM's start contributing significant ideas to scientists and mathematicians, its time to recognize that whatever tricks are used in biology (humans, octopuses, ...) may still be of interest and of value, but they no longer seem like the unique magical missing ingredients which were so long sought after.
From this point on its all about efficiencies:
modeling efficiency: how do we best fit the elephant, with bezier curves, rational polynomials, ...?
memory bandwidth training efficiency: when building coincidence statistics, say bigrams, is it really necessary to update the weights for all concepts? a co-occurence of 2 concepts should just increase the predicted probability for the just observed bigram and then decrease a global coefficient used to scale the predicted probabilities. I.e. observing a baobab tree + an elephant in the same image/sentence/... should not change the relative probabilities of observing french fries + milkshake versus bicycle + windmill. This indicates different architectures should be possible with much lower training costs, by only updating weights of the concepts observed in the last bigram.
ofc, and probably will never understand because of sheer complexity. It doesn't mean we can't replicate the output distribution through data. Probably when we do in efficient manners, the mechanisms (if they are efficient) will be learned too.
As the result, all living cells with DNA emit coherent (as in lasers) light [2]. There is a theory that this light also facilitates intercellular communication.
Chemical structures in dendrites, not even neurons, are capable to compute XOR [3] which require multilevel artificial neural network with at least 9 parameters. Some neurons in brain have hundredths of thousands of dendrites, we are now talking of millions of parameters only in single neuron's dendrites functionality.
So, while human brains aren't magic, special or different, they are just extremely complex.
Imagine building a computer with 85 billions of superconducting quantum computers, optically and electrically connected, each capable of performing computations of a non-negligibly complex artificial neural network.
All three appear to be technically correct, but are (normally) only incidental to the operation of neurons as neurons. We know this because we can test what aspects of neurons actually lead to practical real world effects. Neurophysiology is not a particularly obscure or occult field, so there are many many papers and textbooks on the topic.(And there's a large subset you can test on yourself, besides, though I wouldn't recommend patch-clamping!)
> We know this because we can test what aspects of neurons actually lead to practical real world effects.
Electric current is also quantum phenomena, but it is also very averaged in most circumstances that lead to practical real world effects.
What is wonderful here is that contemporary electronics wizardry that allowed us to have machines that mimic some of thinking, also is very concerned of the quantum-level electromagnetic effects at the transistor level.
On reread, if your actual argument is that SNN are surprisingly sophisticated and powerful, and we might be underestimating how complex the brain's circuits really are, then maybe we're in violent agreement.
They are extremely complex, but is that complexity required for building a thinking machine? We don't understand bird physiology enough to build a bird from scratch, but an airplane flies just the same.
The complexities of contemporary computers and complexities of computing-related infrastructure (consider ASML and electricity) are orders of magnitudes higher than what was needed for first computers. The difference? We have something that mimics some aspects of (human) thinking.
How complex our everything computing-related should be to mimic thinking (of humans) little more closely?
Are we not just getting lost in semantics when we say "fly"? An airplane does not at all perform the same behavior as a bird. Do we say that boats or submarines "swim"?
Planes and boats disrupt the environments they move through and air and sea freight are massive contributors to pollution.
(Motors and human brains are both just mechanisms, the reason one is a priori capable of learning abstract thought and not the other ?)
While I agree to some extent with the materialistic conception, the brain is not an isolated mechanism, but rather the element of a system which itself isn't isolated from the experience of being a body in a world interacting with different systems to form super systems.
The brain must be a very efficient mechanism, because it doesn't need to ingest the whole textual production of the human world in order to know how to write masterpieces (music, litterature, films, software, theorems etc...). Instead the brain learns to be this very efficient mechanism with (as a starting process) feeling its own body sh*t on itself during a long part of its childhood.
I can teach someone to become really good at producing fine and efficient software, but on the contrary I can only observe everyday that my LLM of choice keeps being stupid even when I explain it how it fails. ("You're perfectly right !").
It is true that there's nothing magical about the brain, but I am pretty sure it must be stronger tech than a probabilistic/statistical next word guesser (otherwise there would be much more consensus about the usability of LLMs I think).
I'm not arguing that human brains are magic. the current AI models will probably teach us more about what we didn't know about intelligence than anything else.
Thanks for the link, I haven't seen this before and it's interesting.
I don't think the version of self awareness they demonstrated is synonymous with subjective experience. But same thing can be said about any human other then me.
Damn, just let me believe all brains are magical or I'll fall into solipsism.
Thermometers and human brains are both mechanisms. Why would one be capable of measuring temperature and other capable of learning abstract thought?
> If it turns out that LLMs don't model human brains well enough to qualify as "learning abstract thought" the way humans do, some future technology will do so. Human brains aren't magic, special or different.
> LLMs and human brains are
both just mechanisms. Why would one mechanism a priori be capable of "learning abstract thought", but no others?
“Internal combustion engines and human brains are both just mechanisms. Why would one mechanism a priori be capable of "learning abstract thought", but no others?”
The question isn't about what an hypothetical mechanism can do or not, it's about whether the concrete mechanism we built does or not. And this one doesn't.
The general argument you make is correct, but you conclusion "And this one doesn't." is as yet uncertain.
I will absolutely say that all ML methods known are literally too stupid to live, as in no living thing can get away with making so many mistakes before it's learned anything, but that's the rate of change of performance with respect to examples rather than what it learns by the time training is finished.
What is "abstract thought"? Is that even the same between any two humans who use that word to describe their own inner processes? Because "imagination"/"visualise" certainly isn't.
> no living thing can get away with making so many mistakes before it's learned anything
If you consider that LLMs have already "learned" more than any one human in this world is able to learn, and still make those mistakes, that suggests there may be something wrong with this approach...
To a limited degree, they can compensate for being such slow learners (by example) due to the transistors doing this learning being faster (by the wall clock) than biological synapses to the same degree to which you walk faster than continental drift. (Not a metaphor, it really is that scale difference).
However, this doesn't work on all domains. When there's not enough training data, when self-play isn't enough… well, this is why we don't have level-5 self-driving cars, just a whole bunch of anecdotes about various different self-driving cars that work for some people and don't work for other people: it didn't generalise, the edge cases are too many and it's too slow to learn from them.
So, are LLMs bad at… I dunno, making sure that all the references they use genuinely support the conclusions they make before declaring their task is complete, I think that's still a current failure mode… specifically because they're fundamentally different to us*, or because they are really slow learners?
* They *definitely are* fundamentally different to us, but is this causally why they make this kind of error?
But humans do the same thing. How many eons did we make the mistake of attributing everything to God's will, without a scientific thought in our heads? It's really easy to be wrong, when the consequences don't lead to your death, or are actually beneficial. The thinking machines are still babies, whose ideas aren't honed by personal experience; but that will come, in one form or another.
I'm not sure. If you see what they're doing with feedback already in code generation. The LLM makes a "hallucination", generates the wrong idea, then tests its code only to find out it doesn't compile. And goes on to change its idea, and try again.
We seem to be talking past one another. All I was talking about was the facts of how these systems perform, without any reverence about it at all.
But to your point, I do see a lot of people very emotionally and psychologically committed to pointing out how deeply magical humans are, and how impossible we are to replicate in silicon. We have a religion about ourselves; we truly do have main character syndrome. It's why we mistakenly thought the earth was at the center of the universe for eons. But even with that disproved, our self-importance remains boundless.
> I do see a lot of people very emotionally and psychologically committed to pointing out how deeply magical humans are, and how impossible we are to replicate in silicon.
This a straw man, the question isn't if this is possible or not (this is an open question), it's about whether or not we are already here, and the answer is pretty straightforward: no we aren't. (And the current technology isn't going to bring us anywhere near that)
> but that's the rate of change of performance with respect to examples rather than what it learns by the time training is finished.
It's not just that. The problem of “deep learning” is that we use the word “learning” for something that really has no similarity with actual learning: it's not just that it converges way too slowly, it's also that it just seeks to minimize the predicted loss for every samples during training, but that's no how humans learn. If you feed it enough flat-earther content, as well a physics books, an LLM will happily tells you that the earth is flat, and explain you with lots of physics why it cannot be flat. It simply learned both “facts” during training and then spit it out during inference.
A human will learn one or the other first, and once the initial learning is made, it will disregards all the evidence of the contrary, until maybe at some point it doesn't and switches side entirely.
LLMs don't have an inner representation of the world and as such they don't have an opinion about the world.
The humans can't see the reality for itself, but they at least know it exists and they are constantly struggling to understand it. The LLM, by nature, is indifferent to the world.
> If you feed it enough flat-earther content, as well a physics books, an LLM will happily tells you that the earth is flat, and explain you with lots of physics why it cannot be flat.
This is a terrible example, because it's what humans do as well. See religious, or indeed military, indoctrination. All propaganda is as effective as it is, because the same message keeps getting hammered in.
And not just that, common misconceptions abound everywhere and not just conspiracy theories, religion, and politics. My dad absolutely insisted that the water draining in toilets or sinks are meaningfully influenced by the Coriolis effect, used an example of one time he went to the equator and saw a demonstration of this on both sides of the equator. University education and lifetime career in STEM, should have been able to figure out from first principles why the Coriolis effect is exactly zero on the equator itself, didn't.
> A human will learn one or the other first, and once the initial learning is made, it will disregards all the evidence of the contrary, until maybe at some point it doesn't and switches side entirely.
We don't have any way to know what a human would do if they could read the entire internet, because we don't live long enough to try.
The only bet I'd make is that we'd be more competent than any AI doing the same, because we learn faster from fewer examples, but that's about it.
> LLMs don't have an inner representation of the world and as such they don't have an opinion about the world.
There is evidence that they do have some inner representation of the world, e.g.:
> This is a terrible example, because it's what humans do as well. See religious, or indeed military, indoctrination. All propaganda is as effective as it is, because the same message keeps getting hammered in.
You completely misread my point.
The key thing with humans isn't that they cannot believe in bullshit. They can definitely do. But we don't usually believe in both the bullshit and in the fact the BS is actually BS. We have opinions on the BS. And we, as a species, routinely die or kill for these opinions, by the way. LLM don't care about anything.
> But we don't usually believe in both the bullshit and in the fact the the BS is actually BS.
I can't parse what you mean by this.
> LLM don't care about anything.
"Care" is ill-defined. LLMs are functions that have local optima (the outputs); those functions are trained to approximate other functions (e.g. RLHF) that optimise other things that can be described with functions (what humans care about). It's a game of telephone, like how Leonard Nimoy was approximating what the script writers were imagining Spock to be like when given the goal of "logical and unemotional alien" (ditto Brent Spiner, Data, "logical and unemotional android"), and yet humans are bad at writing such characters: https://tvtropes.org/pmwiki/pmwiki.php/Main/StrawVulcan
But rather more importantly in this discussion, I don't know what you care about when you're criticising AI for not caring, especially in this context. How, *mechanistically*, does "caring" matter to "learning abstract thought", and the question of how closely LLMs do or don't manage it relative to humans?
I mean, in a sense, I could see why someone might argue the exact opposite, that LLMs (as opposed to VLMs or anything embodied in a robot, or even pure-text agents trained on how tools act in response to the tokens emitted) *only* have abstract "thought", in so far as it's all book-learned knowledge.
>> But we don't usually believe in both the bullshit and in the fact the the BS is actually BS.
> I can't parse what you mean by this.
The point is that humans care about the state of a distributed shared world model and use language to perform partial updates to it according to their preferences about that state.
Humans who prefer one state (the earth is flat) do not -- as a rule -- use language to undermine it. Flat earthers don't tell you all the reasons the earth cannot be flat.
But even further than this, humans also have complex meta-preferences of the state, and their use of language reflects those too. Your example is relevant here:
> My dad absolutely insisted that the water draining in toilets or sinks are meaningfully influenced by the Coriolis effect [...]
> [...] should have been able to figure out from first principles why the Coriolis effect is exactly zero on the equator itself, didn't.
This is an exemplar of human behavior. Humans act like this. LLMs don't. If your dad did figure out from first principles and expressed it and continued insisting the position, I would suspect them of being an LLM, because that's how LLMs 'communicate'.
Now that the what is clear -- why? Humans experience social missteps like that as part of the loss surface. Being caught in a lie sucks, so people learn to not lie or be better at it. That and a million other tiny aspects of how humans use language in an overarching social context.
The loss surface that LLMs see doesn't have that feedback except in the long tail of doing Regularized General Document Corpora prediction perfectly. But it's so far away compared to just training on the social signal, where honesty is immediately available as a solution and is established very early in training instead of at the limit of low loss.
How humans learn (embedded in a social context from day one) is very effective at teaching foundational abilities fast. Natural selection cooked hard. LLM training recipes do not compare, they're just worse in so many different ways.
> Everyone is out here acting like "predicting the next thing" is somehow fundamentally irrelevant to "human thinking" and it is simply not the case.
Nobody is. What people are doing is claiming that "predicting the next thing" does not define the entirety of human thinking, and something that is ONLY predicting the next thing is not, fundamentally, thinking.
Well, yes because thinking soon requires interacting, not just ideating. It's in the dialogue between ideation and interaction that we make our discoveries.
when LLM popped out and people started to say 'this is just markov chain on steroid and not thinking' i was a bit confused because a lot of my "thinking" is statistical too.. I very often try to solve an issue by switching a known solution with a different "probable" variant of it (tweaking a parameter)
LLMs have higher dimensions (they map token to grammatical and semantical space) .. it might not be thinking but it seems on its way we're just thinking with more abstractions before producing speech ?... dunno
I claim that all of thinking can be reduced to predicting the next thing. Predicting the next thing = thinking in the same way that reading and writing strings of bytes is a universal interface, or every computation can be done by a Turing machine.
People can claim whatever they like. That doesn't mean it's a good or reasonable hypothesis (especially for one that is essentially unfalsifible like predictive coding).
The problem is that we don’t have a good understanding of what “thinking” really is, and those parts of it we think we do understand involve simple things done at scale (electrical pulses on specific pathways, etc).
It is not unreasonable to suspect differences between humans and LLMs are differences in degree, rather than category.
I'm not trying to advance a testable hypothesis. If you think the unfalsifiability of my claim is a problem, you haven't understood what I'm trying to do.
My claim is that the two concepts are indistinguishable, thus equivalent. The unfalsifiability is what makes it a natural equivalence, the same as in the other examples I gave.
IMHO, you should. The opponent does not have an alternative definition of thinking that would have a prediction power matching the token prediction. Whatever they are thinking thinking is is a strictly worse scientific theory.
I am not having some existential crisis, but if we get to a point where X% of humans cannot outperform “AI” on any task that humans deem “useful”, for some nontrivial value of X, then many assumptions that culture has inculcated into me about humanity are no longer valid.
What is the role of humans then?
Can it be said that humans “think” if they can’t think a thought that a non thinking AI cannot also think?
If all humans were suddenly wiped off the face of the earth, AI would go silent, and the hardware it runs on would eventually shut down.
If all AI was suddenly wiped off the face of the earth, humans would rebuild it, and would carry on fine in the meantime.
One AI researcher decades ago said something to the effect of: researchers in biology look at living organisms and wonder how they live; researchers in physics look at the cosmos and wonder what all is out there; researchers in artificial intelligence look at computer systems and wonder how they can be made to wonder such things.
A motorcycle is not "sprinting" and an LLM is not "thinking". Everyone would agree that a motorcycle is not running but the same dumb shit is posted over and over and over on here that somehow the LLM is "thinking".
But your assertion is merely semantic. It doesn't say anything substantive.
I could also say a motorcycle "moves forward" just like a person "moves forward". Whether we use the same or different words for same or different concepts doesn't say anything about the actual underlying similarity.
And please don't call stuff "dumb shit" here. Not appropriate for HN.
A forklift is "lifting" things, despite using a completely different mechanical process as a human "lifting" things. The only real similarity between these kinds of "lifting" is the end result, something is higher up than it was before.
That’s because the motorcycle thing is too simlistic of a comparison. It doesn’t come nearly close to capturing the nuance of the whole LLM “thinking” situation.
Predicting the next token is not at all the same thing as predicting the next action in a causal chain of actions. Not even close. One is model of language tokens, the other is a model of the physical world. You can come up with all sorts of predictions that can't be expressed cleanly in natural language. And plenty of things that parse cleanly from a language perspective but are unhinged in their description of empirical reality.
It may be doing the "thinking" and could reach AGI. But we don't want it. We don't want to take a fork lift to the gym. We don't want plastic aliens showing off their AGI and asking humanity to outsource human thinking and decision-making to them.
A good heuristic is that if an argument resorts to "actually not doing <something complex sounding>" or "just doing <something simple sounding>" etc, it is not a rigorous argument.
I'm an LLMs are being used in workflows they don't make sense in-sayer. And while yes, I can believe that LLMs can be part of a system that actually does think, I believe that to achieve true "thinking", it would likely be a system that is more deterministic in its approach rather than probabilistic.
Especially when modeling acting with intent. The ability to measure against past results and think of new innovative approaches seems like it may come from a system that may model first and then use LLM output. Basically something that has a foundation of tools rather than an LLM using MCP. Perhaps using LLMs to generate a response that humans like to read, but not in them coming up with the answer.
Either way, yes, its possible for a thinking system to use LLMs (and potentially humans piece together sentences in a similar way), but its also possible LLMs will be cast aside and a new approach will be used to create an AGI.
So for me: even if you are an AI-yeasayer, you can still believe that they won't be a component in an AGI.
You can make a separate model for the task, which is based on well chosen features and calibrated from actual data. Then the LLM only needs to generate the arguments to this model (extract those features from messages) and call it like a MCP tool. This external tool can be a simple Sklearn model.
Yes, personally I'm completely fine with the fact that LLMs don't actually think. I don't care that they're not AGI, though the hysterics about "AGI is so close now" seems silly to me. Fusion reactors and self-driving cars are just around the same corner.
They prove to have some useful utility to me regardless.
It's fascinating when you look at each technical component of cognition in human brains and contrast against LLMs. In humans, we have all sorts of parallel asynchronous processes running, with prediction of columnar activations seemingly the fundamental local function, with tens of thousands of mini columns and regions in the brain corresponding to millions of networked neurons using the "predict which column fires next" objective to increment or decrement the relative contribution of any functional unit.
In the case of LLMs you run into similarities, but they're much more monolithic networks, so the aggregate activations are going to scan across billions of neurons each pass. The sub-networks you can select each pass by looking at a threshold of activations resemble the diverse set of semantic clusters in bio brains - there's a convergent mechanism in how LLMs structure their model of the world and how brains model the world.
This shouldn't be surprising - transformer networks are designed to learn the complex representations of the underlying causes that bring about things like human generated text, audio, and video.
If you modeled a star with a large transformer model, you would end up with semantic structures and representations that correlate to complex dynamic systems within the star. If you model slug cellular growth, you'll get structure and semantics corresponding to slug DNA. Transformers aren't the end-all solution - the paradigm is missing a level of abstraction that fully generalizes across all domains, but it's a really good way to elicit complex functions from sophisticated systems, and by contrasting the way in which those models fail against the way natural systems operate, we'll find better, more general methods and architectures, until we cross the threshold of fully general algorithms.
Biological brains are a computational substrate - we exist as brains in bone vats, connected to a wonderfully complex and sophisticated sensor suite and mobility platform that feeds electrically activated sensory streams into our brains, which get processed into a synthetic construct we experience as reality.
Part of the underlying basic functioning of our brains is each individual column performing the task of predicting which of any of the columns it's connected to will fire next. The better a column is at predicting, the better the brain gets at understanding the world, and biological brains are recursively granular across arbitrary degrees of abstraction.
LLMs aren't inherently incapable of fully emulating human cognition, but the differences they exhibit are expensive. It's going to be far more efficient to modify the architecture, and this may diverge enough that whatever the solution ends up being, it won't reasonably be called an LLM. Or it might not, and there's some clever tweak to things that will push LLMs over the threshold.
When you have a thought, are you "predicting the next thing"—can you confidently classify all mental activity that you experience as "predicting the next thing"?
Language and society constrains the way we use words, but when you speak, are you "predicting"? Science allows human beings to predict various outcomes with varying degrees of success, but much of our experience of the world does not entail predicting things.
How confident are you that the abstractions "search" and "thinking" as applied to the neurological biological machine called the human brain, nervous system, and sensorium and the machine called an LLM are really equatable? On what do you base your confidence in their equivalence?
Does an equivalence of observable behavior imply an ontological equivalence? How does Heisenberg's famous principle complicate this when we consider the role observer's play in founding their own observations? How much of your confidence is based on biased notions rather than direct evidence?
The critics are right to raise these arguments. Companies with a tremendous amount of power are claiming these tools do more than they are actually capable of and they actively mislead consumers in this manner.
> When you have a thought, are you "predicting the next thing"
Yes. This is the core claim of the Free Energy Principle[0], from the most-cited neuroscientist alive. Predictive processing isn't AI hype - it's the dominant theoretical framework in computational neuroscience for ~15 years now.
> much of our experience of the world does not entail predicting things
Introspection isn't evidence about computational architecture. You don't experience your V1 doing edge detection either.
> How confident are you that the abstractions "search" and "thinking"... are really equatable?
This isn't about confidence, it's about whether you're engaging with the actual literature. Active inference[1] argues cognition IS prediction and action in service of minimizing surprise. Disagree if you want, but you're disagreeing with Friston, not OpenAI marketing.
> How does Heisenberg's famous principle complicate this
It doesn't. Quantum uncertainty at subatomic scales has no demonstrated relevance to cognitive architecture. This is vibes.
> Companies... are claiming these tools do more than they are actually capable of
Possibly true! But "is cognition fundamentally predictive" is a question about brains, not LLMs. You've accidentally dismissed mainstream neuroscience while trying to critique AI hype.
How does the free energy principle align with system dynamics and the concept of emergence? Yes, our brain might want to optimize for lack of surprise, but that does not mean it can fully avoid emergent or chaotic behavior stemming from the incredibly complex dynamics of the linked neurons?
FEP doesn't conflict with complex dynamics, it's a mathematical framework for explaining how self-organizing behavior arises from simpler variational principles. That's what makes it a theory rather than a label.
The thing you're doing here has a name: using "emergence" as a semantic stopsign. "The system is complex, therefore emergence, therefore we can't really say" feels like it's adding something, but try removing the word and see if the sentence loses information.
"Neurons are complex and might exhibit chaotic behavior" - okay, and? What next? That's the phenomenon to be explained, not an explanation.
This was articulated pretty well 18 years ago [0].
This essay completely misunderstands how the notion of emergence gained prominence and how people tend to actually use it. It's a straw man that itself devolves into a circular argument "embrace a reductionist epistemology because you should embrace a reductionist epistemology".
It doesn't even meaningfully engage with the historical literature that established the term, etc. If you want to actually understand why the term gained prominence, check out the work of Edgar Morin.
Thanks for the links! I'll have to dig into this more for sure. Looking at the bulleted summary, I'm not sure your argument is sufficiently nuanced or being made in good faith.
The article argues that the brain "predicts" acts of perception in order to minimize surprise. First of all, very few people mean to talk about these unconscious operations of the brain when they claim they are "thinking". Most people have not read enough neuroscience literature to have such a definition. Instead, they tend to mean "self-conscious activity" when they say "thinking". Thinking, the way the term is used in the vernacular, usually implies some amount of self-reflexivity. This is why we have the term "intuition" as opposed to thinking after all. From a neuronal perspective, intuition is still thinking, but most people don't think (ha) of the word thinking to encompass this, and companies know that.
It is clear to me, as it is to everyone one the planet, that when OpenAI for example claims that ChatGPT "thinks" they want consumers to make the leap to cognitive equivalence at the level of self-conscious thought, abstract logical reasoning, long-term learning, and autonomy. These machines are designed such that they do not even learn and retain/embed new information past their training date. That already disqualifies them from strong equivalence to human beings, who are able to rework their own tendencies toward prediction in a meta cognitive fashion by incorporating new information.
> can you confidently classify all mental activity that you experience as "predicting the next thing"? [...] On what do you base your confidence in their equivalence?
To my understanding, bloaf's claim was only that the ability to predict seems a requirement of acting intentionally and thus that LLMs may "end up being a component in a system which actually does think" - not necessarily that all thought is prediction or that an LLM would be the entire system.
I'd personally go further and claim that correctly generating the next token is already a sufficiently general task to embed pretty much any intellectual capability. To complete `2360 + 8352 * 4 = ` for unseen problems is to be capable of arithmetic, for instance.
> When you have a thought, are you "predicting the next thing"—can you confidently classify all mental activity that you experience as "predicting the next thing"?
So notice that my original claim was "prediction is fundamental to our ability to act with intent" and now your demand is to prove that "prediction is fundamental to all mental activity."
That's a subtle but dishonest rhetorical shift to make me have to defend a much broader claim, which I have no desire to do.
> Language and society constrains the way we use words, but when you speak, are you "predicting"?
Yes, and necessarily so. One of the main objections that dualists use to argue that our mental processes must be immaterial is this [0]:
* If our mental processes are physical, then there cannot be an ultimate metaphysical truth-of-the-matter about the meaning of those processes.
* If there is no ultimate metaphysical truth-of-the-matter about what those processes mean, then everything they do and produce are similarly devoid of meaning.
* Asserting a non-dualist mind therefore implies your words are meaningless, a self-defeating assertion.
The simple answer to this dualist argument is precisely captured by this concept of prediction. There is no need to assert some kind of underlying magical meaning to be able to communicate. Instead, we need only say that in the relevant circumstances, our minds are capable of predicting what impact words will have on the receiver and choosing them accordingly. Since we humans don't have access to each other's minds, we must not learn these impacts from some kind of psychic mind-to-mind sense, but simply from observing the impacts of the words we choose on other parties; something that LLMs are currently (at least somewhat) capable of observing.
And the big players have built a bunch of workflows which embed many other elements besides just "predictions" into their AI product. Things like web search, to incorporating feedback from code testing, to feeding outputs back into future iterations. Who is to say that one or more of these additions has pushed the ensemble across the threshold and into "real actual thinking."
The near-religious fervor which people insist that "its just prediction" makes me want to respond with some religious allusions of my own:
> Who is this that wrappeth up sentences in unskillful words? Gird up thy loins like a man: I will ask thee, and answer thou me. Where wast thou when I laid up the foundations of the earth? tell me if thou hast understanding. Who hath laid the measures thereof, if thou knowest? or who hath stretched the line upon it?
The point is that (as far as I know) we simply don't know the necessary or sufficient conditions for "thinking" in the first place, let alone "human thinking." Eventually we will most likely arrive at a scientific consensus, but as of right now we don't have the terms nailed down well enough to claim the kind of certainty I see from AI-detractors.
I take a offence in the idea I’m “religiously downplaying LLMs”. I pay top dollar for access to the best models because I want the capabilities to be good / better. Just because I’m documenting my experience it doesn’t mean I have an Anti-ai agenda ? I pay because I find LLMs to be useful. Just not in the way suggested by the marketing teams.
I’m downplaying because I have honestly been burned by these tools when I’ve put trust in their ability to understand anything, provide a novel suggestion or even solve some basic bugs without causing other issues.?
I use all of the things you talk about extremely frequently and again, there is no “thinking” or consideration on display that suggests these things work like us, else why would we be having this conversation if they were ?
> I’m downplaying because I have honestly been burned by these tools when I’ve put trust in their ability to understand anything, provide a novel suggestion or even solve some basic bugs without causing other issues.?
I've had that experience plenty of times with actual people...
LLMs don't "think" like people do, that much is pretty obvious. But I'm not at all sure whether what they do can be called "thinking" or not.
> The harms engendered by underestimating LLM capabilities are largely that people won't use the LLMs.
Speculative fiction about superintelligences aside, an obvious harm to underestimating the LLM's capabilities is that we could effectively be enslaving moral agents if we fail to correctly classify them as such.
> The issue is that prediction is "part" of the human thought process, it's not the full story...
Do you have a proof for this?
Surely such a profound claim about human thought process must have a solid proof somewhere? Otherwise who's to say all of human thought process is not just a derivative of "predicting the next thing"?
What would change your mind? It's an exercise in feasibility.
For example, I don't believe in time travel. If someone made me time travel, and made it undeniable that I was transported back to 1508, then I would not be able to argue against it. In fact, no one in such position would.
What is that equivalent for your conviction? There must be something, otherwise, it's just an opinion that can't be changed.
You don't need to present some actual proof or something. Just lay out some ideas that demonstrate that you are being rational about this and not just sucking up to LLM marketing.
most humans in any percentile act towards the thing of someone else. most of these things are a lot worse than what the human "would originally intend". this behavior stems from 100s and thousands of nudges since childhood.
the issue with AI and AI-naysayers is, by analogy, this: cars were build to drive from A to Z. people picked up tastes and some people started building really cool looking cars. the same happens on the engineering side. then portfolio communists came with their fake capitalism and now cars are build to drive over people but don't really work because people, thankfully, are overwhelming still fighting to attempt to act towards their own intents.
Exactly. Our base learning is by example, which is very much learning to predict.
Predict the right words, predict the answer, predict when the ball bounces, etc. Then reversing predictions that we have learned. I.e. choosing the action with the highest prediction of the outcome we want. Whether that is one step, or a series of predicted best steps.
Also, people confuse different levels of algorithm.
There are at least 4 levels of algorithm:
• 1 - The architecture.
This input-output calculation for pre-trained models are very well understood. We put together a model consisting of matrix/tensor operations and few other simple functions, and that is the model. Just a normal but high parameter calculation.
• 2 - The training algorithm.
These are completely understood.
There are certainly lots of questions about what is most efficient, alternatives, etc. But training algorithms harnessing gradients and similar feedback are very clearly defined.
• 3 - The type of problem a model is trained on.
Many basic problem forms are well understood. For instance, for prediction we have an ordered series of information, with later information to be predicted from earlier information. It could simply be an input and response that is learned. Or a long series of information.
• 4 - The solution learned to solve (3) the outer problem, using (2) the training algorithm on (1) the model architecture.
People keep confusing (4) with (1), (2) or (3). But it is very different.
For starters, in the general case, and for most any challenging problem, we never understand their solution. Someday it might be routine, but today we don't even know how to approach that for any significant problem.
Secondly, even with (1), (2), and (3) exactly the same, (4) is going to be wildly different based on the data characterizing the specific problem to solve. For complex problems, like language, layers and layers of sub-solutions to sub-problems have to be solved, and since models are not infinite in size, ways to repurpose sub-solutions, and weave together sub-solutions to address all the ways different sub-problems do and don't share commonalities.
Yes, prediction is the outer form of their solution. But to do that they have to learn all the relationships in the data. And there is no limit to how complex relationships in data can be. So there is no limit on the depths or complexity of the solutions found by successfully trained models.
Any argument they don't reason, based on the fact that they are being trained to predict, confuses at least (3) and (4). That is a category error.
It is true, they reason a lot more like our "fast thinking", intuitive responses, than our careful deep and reflective reasoning. And they are missing important functions, like a sense of what they know or don't. They don't continuously learn while inferencing. Or experience meta-learning, where they improve on their own reasoning abilities with reflection, like we do. And notoriously, by design, they don't "see" the letters that spell words in any normal sense. They see tokens.
Those reasoning limitations can be irritating or humorous. Like when a model seems to clearly recognize a failure you point out, but then replicates the same error over and over. No ability to learn on the spot. But they do reason.
Today, despite many successful models, nobody understands how models are able to reason like they do. There is shallow analysis. The weights are there to experiment with. But nobody can walk away from the model and training process, and build a language model directly themselves. We have no idea how to independently replicate what they have learned, despite having their solution right in front of us. Other than going through the whole process of retraining another one.
Did anyone ever seriously buy the whole "it's transporting" BS when it was wheelbarrows? What makes you believe today's trucks are meaningfully different?
I suspect that people instinctively believe they have free will, both because it feels like we do, and because society requires us to behave that way even when we don't.
The truth is that the evidence says we don't. See the Libet experiment and its many replications.
Your decisions can be predicted from brain scans up to 10 seconds before you make them, which means they are as deterministic as an LLM's. Sorry, I guess.
The experiment raised significant questions about free will and determinism. While it suggested that unconscious brain activity precedes conscious decision-making, Libet argued that this does not negate free will, as individuals can still choose to suppress actions initiated by unconscious processes."
It's been repeated a huge number of time since, and widely debated. When Libet first did the experiment it was only like 200ms before the mind become consciously aware of the decision. More recent studies have shown they can predict actions up to 7-10 seconds before the subject is aware of having made a decision.
It's pretty hard to argue that you're really "free" to make a different decision if your body knew which you would choose 7 seconds before you became aware of it.
I mean, those long term predictions were only something like 60% accurate, but still, the preponderance of evidence says that those decisions are deterministic and we keep finding new ways to predict the outcome sooner and with higher accuracy.
"I conducted an experiment where I instructed experienced drivers to follow a path in a parking lot laid out with traffic cones, and found that we were able to predict the trajectory of the car with greater than 60% accuracy. Therefore drivers do not have free will to just dodge the cones and drive arbitrarily from the start to the finish."
Clearly, that conclusion would be patently absurd to draw from that experiment. There are so many expectation and observation effects that go into the very setup from the beginning. Humans generally follow directions, particularly when a guy in a labcoat is giving them.
> At some point, when they felt the urge to do so,
they were to freely decide between one of two buttons, operated by the
left and right index fingers, and press it immediately. [0]
Wow. TWO whole choices to choose from! Human minds tend to pre-think their choice between one of two fingers to wiggle, therefore free will doesn't exist.
> It's pretty hard to argue that you're really "free" to make a different decision if your body knew which you would choose 7 seconds before you became aware of it.
To really spell it out since the analogy/satire may be lost: You're free to refrain from pressing either button during the prompt. You're free to press both buttons at the same time. You're free to mash them rapidly and randomly throughout the whole experiment. You're free to walk into the fMRI room with a bag full of steel BB's and cause days of downtime and thousands of dollars in damage. Folks generally don't do those things because of conditioning.
If that were the only evidence I might agree that alternative explanations are as likely, but I cited only one of many studies that show similar outcomes. There are loads of other studies done with entirely different methodologies that indicate most human introspection is mostly better described as post hoc confabulism. That is to say, we don't use reason to make decisions so much as we make decisions, and then justify them with reasons. Nisbet and Wilson were showing it experimentally as far back as the late 70's. [0] It's been confirmed in different forms hundreds of times since.
Certainly we can come up with some alternative theories (like "free will") to explain it all away, but the simplest (therefore most likely correct) answer is just that we're basically statistical state machines and are as deterministic as a similar computational system.
To be clear, I'm not saying that metacognition doesn't exist. Just that I've never seen any reason to believe it's very different from current thinking models that just feed an output back in as another input.
What is the import of this to you here? Whether you have free will or you feel like you do, kinda same difference for this particular point right? It doesn't make me more human actually having free will, it is sufficient to simply walk around as if I do.
But beyond that, what do you want to say here? What is lost, what is gained? Are you wanting to say this makes us more like an LLM? How so?
It makes sense if you're desperate for free will to be real, but you really have to work for it. Especially when you add in the countless other studies showing that a lot of the reasons we give for our actions, especially in quick or ambiguous choices, are confabulationalist post-hoc constructions. Our own introspection seems mostly to consist of just "making stuff up" to justify the decisions we've already made.
I mean, a reasonable person could argue their way past all the evidence without totally denying it, but "free will" just isn't the simplest explanation that fits the available data. It's possible that free will exists in the same way it's possible that Russels teapot exists.
Every day I see people treat gen AI like a thinking human, Dijkstra's attitudes about anthropomorphizing computers is vindicated even more.
That said, I think the author's use of "bag of words" here is a mistake. Not only does it have a real meaning in a similar area as LLMs, but I don't think the metaphor explains anything. Gen AI tricks laypeople into treating its token inferences as "thinking" because it is trained to replicate the semiotic appearance of doing so. A "bag of words" doesn't sufficiently explain this behavior.
The contra-positive of "All LLMs are not thinking like humans" is "No humans are thinking like LLMs"
And I do not believe we actually understand human thinking well enough to make that assertion.
Indeed, it is my deep suspicion that we will eventually achieve AGI not by totally abandoning today's LLMs for some other paradigm, but rather embedding them in a loop with the right persistence mechanisms.
Given that LLMs are incapable of synthetic a priori knowledge and humans are, I would say that as the tech stands currently, it's reasonable to make both of those statements.
The loop, or more precisely the "search" does the novel part in thinking, the brain is just optimizing this process. Evolution could manage with the simplest model - copying with occasional errors, and in one run it made everyone of us. The moral - if you scale search the model can be dumb.
Let’s not underestimate the scale of the search which led to us though, even though you may be right in principle. In addition to deep time on earth, we may well be just part of a tiny fraction of a universe-wide and mostly fruitless search.
It's not just the pretraining, it's the entire scaffolding between the user and the LLM itself that contributes to the illusion. How many people would continue assuming that these chatbots were conscious or intelligent if they had to build their own context manager, memory manager, system prompt, personality prompt, and interface?
I agree 100%. Most people haven't actually interacted directly with an LLM before. Most people's experience with LLMs is ChatGPT, Claude, Grok, or any of the other tools that automatically handle context, memory, personality, temperature, and are deliberately engineered to have the tool communicate like a human. There is a ton of very deterministic programming that happens between you and the LLM itself to create this experience, and much of the time when people are talking about the ineffable intelligence of chatbots, it's because of the illusion created by this scaffolding.
Yea bag of words isn’t helpful at all. I really do think that “superpowered sentence completion” is the best description. Not only is it reasonably accurate it is understandable, everyone has seen autocomplete function, and it’s useful. I don’t know how to “use” a bag of words. I do know how to use sentence completion. It also helps explains why context matters.
Thats the thing, when you use an Ask/answer mechanism, you are just writing a "novel" where User: asks and personal coding assistant: answers. But all the text goes into the autocomplete function and the "toaster" outputs the most probable text according to the function.
Its useful, it's amazing, but as the original text says, thinking of it as "some intelligence with reasoning " makes us use the wrong mental models for it.
Bag of words is actually the perfect metaphor. The data structure is a bag. The output is a word. The selection strategy is opaquely undefined.
> Gen AI tricks laypeople into treating its token inferences as "thinking" because it is trained to replicate the semiotic appearance of doing so. A "bag of words" doesn't sufficiently explain this behavior.
Something about there being significant overlap between the smartest bears and the dumbest humans. Sorry you[0] were fooled by the magic bag.
[0] in the "not you, the layperson in question" sense
I think it's still a bit of a tortured metaphor. LLMs operate on tokens, not words. And to describe their behavior as pulling the right word out of a bag is so vague that it applies every bit as much to a Naive Bayes model written in Python in 10 minutes as it does to the greatest state of the art LLM.
Yeah. I have a half-cynical/half-serious pet theory that a decent fraction of humanity has a broken theory of mind and thinks everyone has the same thought patterns they do. If it talks like me, it thinks like me.
Whenever the comment section takes a long hit and goes "but what is thinking, really" I get slightly more cynical about it lol
By now, it's pretty clear that LLMs implement abstract thinking - as do humans.
They don't think exactly like humans do - but they sure copy a lot of human thinking, and end up closer to it than just about anything that's not a human.
It isn't clear because they do none of that lol. They don't think.
It can kinda sorta look like thinking if you don't have a critical eye, but it really doesn't take much to break the illusion.
I really don't get this obsessive need to pretend your tools are alive. Y'all know when you watch YouTube that it's a trick and the tiny people on your screen don't live in your computer, right?
And how do you know that exactly? What is the source of that certainty? What makes you fully confident that a system that can write short stories and one-shot Python scripts and catch obscure pop culture references in text isn't "thinking" in any way?
The answer to that is the siren song of "AI effect".
Even admitting "we don't know" requires letting go of the idea that "thinking" must be exclusive to humans. And many are far too weak to do that.
Spoken Query Language? Just like SQL, but for unstructured blobs of text as a database and unstructured language as a query? Also known as Slop Query Language or just Slop Machine for its unpredictable results.
> Spoken Query Language? Just like SQL, but for unstructured blobs of text as a database and unstructured language as a query?
I feel that's more a description of a search engine. Doesn't really give an intuition of why LLMs can do the things they do (beyond retrieval), or where/why they'll fail.
If you want actionable intuition, try "a human with almost zero self-awareness".
"Self-awareness" used in a purely mechanical sense here: having actionable information about itself and its own capabilities.
If you ask an old LLM whether it's able to count the Rs in "strawberry" successfully, it'll say "yes". And then you ask it to do so, and it'll say "2 Rs". It doesn't have the self-awareness to know the practical limits of its knowledge and capabilities. If it did, it would be able to work around the tokenizer and count the Rs successfully.
That's a major pattern in LLM behavior. They have a lot of capabilities and knowledge, but not nearly enough knowledge of how reliable those capabilities are, or meta-knowledge that tells them where the limits of their knowledge lie. So, unreliable reasoning, hallucinations and more.
Agree that's a better intuition, with pretraining pushing the model towards saying "I don't know" in the kinds of situations where people write that as opposed to by introspection of its own confidence.
There appears to be a degree of "introspection of its own confidence" in modern LLMs. They can identify their own hallucinations, at a rate significantly better than chance. So there must be some sort of "do I recall this?" mechanism built into them. Even if it's not exactly a reliable mechanism.
Anthropic has discovered that this is definitely the case for name recognition, and I suspect that names aren't the only things subject to a process like that.
My second thought is that it's not the metaphor that is misleading. People have been told thousands of times that LLMs don't "think", don't "know", don't "feel", but are "just a very impressive autocomplete". If they still really want to completely ignore that, why would they suddenly change their mind with a new metaphor?
Humans are lazy. If it looks true enough and it cost less effort, humans will love it. "Are you sure the LLM did your job correctly?" is completely irrelevant: people couldn't care less if it's correct or not. As long as the employer believes that the employee is "doing their job", that's good enough. So the question is really: "do you think you'll get fired if you use this?". If the answer is "no, actually I may even look more productive to my employer", then why would people not use it?
> If we allow ourselves to be seduced by the superficial similarity, we’ll end up like the moths who evolved to navigate by the light of the moon, only to find themselves drawn to—and ultimately electrocuted by—the mysterious glow of a bug zapper.
As usual with these, it helps to try to keep the metaphor used for downplaying AI, but flip the script. Let's grant the author's perception that AI is a "bag of words", which is already damn good at producing the "right words" for any given situation, and only keeps getting better at it.
Sure, this is not the same as being a human. Does that really mean, as the author seems to believe without argument, that humans need not be afraid that it will usurp their role? In how many contexts is the utility of having a human, if you squint, not just that a human has so far been the best way to "produce the right words in any given situation", that is, to use the meat-bag only in its capacity as a word-bag? In how many more contexts would a really good magic bag of words be better than a human, if it existed, even if the current human is used somewhat differently? The author seems to rest assured that a human (long-distance?) lover will not be replaced by a "bag of words"; why, especially once the bag of words is also ducttaped to a bag of pictures and a bag of sounds?
I can just imagine someone - a horse breeder, or an anthropomorphised horse - dismissing all concerns on the eve of the automotive revolution, talking about how marketers and gullible marks are prone to hippomorphising anything that looks like it can be ridden and some more, and sprinkling some anecdotes about kids riding broomsticks, legends of pegasi and patterns of stars in the sky being interpreted as horses since ancient times.
I don't think the author's argument is that it won't replace any human labour. Or at least I wouldn't agree with such an argument. But the stronger case is that however much LLMs improve, they won't replace humans in general. In the furtherment of knowledge, because they are fundamentally parroting and synthesizing the already known, vs performing truly novel thought. And in creative fields, because people are fundamentally interested in creations of other people, not of computers.
Neither of these is entirely true in all cases, but they could be expected to remain true in at least some (many) cases, and so the role for humans remains.
So a human is just a really expensive, unreliable bag of words. And we get more expensive and more unreliable by the day!
There's a quote I love but have misplaced, from the 19th century I think. "Our bodies are just contraptions for carrying our heads around." Or in this instance... bag of words transport system ;)
So tell me, why do I still have a job and why am frequently successful in getting profitable / useful products into production if I’m “expensive and unreliable”?
I mean I use AI tools to help achieve the goal but I don’t see any signs of the things I’m building and doing being unreliable.
Either way, in what way is this relevant? If the human's labor is not useful at any price point to any entity with money, food or housing, then they presumably will not get paid/given food/housing for it.
I am unsure myself whether we should regard LLMs as mere token-predicting automatons or as some new kind of incipient intelligence. Despite their origins as statistical parrots, the interpretability research from Anthropic [1] suggests that structures corresponding to meaning do exist inside those bundles of numbers and that there are signs of activity within those bundles of numbers that seem analogous to thought.
That said, I was struck by a recent interview with Anthropic’s Amanda Askell [2]. When she talks, she anthropomorphizes LLMs constantly. A few examples:
“I don't have all the answers of how should models feel about past model deprecation, about their own identity, but I do want to try and help models figure that out and then to at least know that we care about it and are thinking about it.”
“If you go into the depths of the model and you find some deep-seated insecurity, then that's really valuable.”
“... that could lead to models almost feeling afraid that they're gonna do the wrong thing or are very self-critical or feeling like humans are going to behave negatively towards them.”
Amanda Askell studied under David Chalmers at NYU: the philosopher who coined "the hard problem of consciousness" and is famous for taking phenomenal experience seriously rather than explaining it away. That context makes her choice to speak this way more striking: this isn't naive anthropomorphizing from someone unfamiliar with the debates. It's someone trained by one of the most rigorous philosophers of consciousness, who knows all the arguments for dismissing mental states in non-biological systems, and is still choosing to speak carefully about models potentially having something like feelings or insecurities.
A person can study fashion extensively, under the best designers, they can understand tailoring and fit and have a phenomenal eye for color and texture.
Their vivid descriptions of what the Emperor could be wearing doesn't make said emperor any less nakey.
>research from Anthropic [1] suggests that structures corresponding to meaning exist inside those bundles of numbers and that there are signs of activity within those bundles of numbers that seem analogous to thought.
Can you give some concrete examples? The link you provided is kind of opaque
>Amanda Askell [2]. When she talks, she anthropomorphizes LLMs constantly.
She is a philosopher by trade and she describes her job (model alignment) as literally to ensure models "have good character traits." I imagine that explains a lot
Excerpt: “We found that there’s a specific combination of neurons in Claude’s neural network that activates when it encounters a mention (or a picture) of this most famous San Francisco landmark.”
Excerpt: “Recent research on smaller models has shown hints of shared grammatical mechanisms across languages. We investigate this by asking Claude for the ‘opposite of small’ across different languages, and find that the same core features for the concepts of smallness and oppositeness activate, and trigger a concept of largeness, which gets translated out into the language of the question.”
Excerpt: “Our new research provides evidence for some degree of introspective awareness in our current Claude models, as well as a degree of control over their own internal states.”
It’s important to note that these “research papers” that Anthropic releases are not properly peer-reviewed and not accepted by any scientific journal or institution. Anthropic has a history of over-exaggerating research, and have an obvious monetary incentive to continue to do so.
My fridge happily reads inputs without consciousness, has goals and takes decisions without "thinking", and consistently takes action to achieve those goals. (And it's not even a smart fridge! It's the one with a copper coil or whatever.)
I guess the cybernetic language might be less triggering here (talking about systems and measurements and control) but it's basically the same underlying principles. One is just "human flavored" and I therefore more prone to invite unhelpful lines of thinking?
Except that the "fridge" in this case is specifically and explicitly designed to emulate human behavior so... you would indeed expect to find structures corresponding to the patterns it's been designed to simulate.
Wondering if it's internalized any other human-like tendencies — having been explicitly trained to simulate the mechanisms that produced all human text — doesn't seem too unreasonable to me.
> the interpretability research from Anthropic [1] suggests that structures corresponding to meaning do exist inside those bundles of numbers and that there are signs of activity within those bundles of numbers that seem analogous to thought
I did a simple experiment - took a photo of my kid in the park, showed it to Gemini and asked for a "detailed description". Then I took that description and put it into a generative model (Z-Image-Turbo, a new one). The output image was almost identical.
So one model converted image to text, the other reversed the processs. The photo was completely new, personal, never put online. So it was not in any training set. How did these 2 models do it if not actually using language like a thinking agent?
the anthropomorphization (say that 3 times quickly) is kinda weird, but also makes for a much more pleasant conversation imo. it's kinda tedious being pedantic all the time.
It also leads to fundamentally wrong conclusions: a related issue I have with this is the use of anthropomorphic shorthand when discussing international politics. You've heard a phrase like "the US thinks...", "China wants...", "Europe believes..." so much you don't even notice it.
All useful shorthands, all which lead to people displaying fundamental misunderstandings of what they're talking about - i.e. expressing surprise that a nation of millions doesn't display consistency of behavior of human lifetime scales, even though fairly obviously the mechanisms of government are churning their make up constantly, and depending on context maybe entirely different people.
It seems obvious to me that entities have emergent needs and plans and so on, independent of any of the humans inside.
For example, if you've worked at a large company, one of the little tragedies is when someone everyone likes gets laid off. There were probably no people who actively wanted Bob to lose his job. Even the CEO/Board who pulled the trigger probably had nothing against Bob. Heck, they might be the next ones out the door. The company is faceless, yet it wanted Bob to go, because that apparently contributed to the company's objective function. Had the company consisted entirely of different people, plus Bob, Bob might have been laid off anyway.
There is a strong will to do ... things the emerges from large structures of people and technology. It's funny like that.
I use LLMs heavily for work, I have done so for about 6 months. I see almost zero "thought" going on and a LOT of pattern matching. You can use this knowledge to your advantage if you understand this. If you're relying on it to "think", disaster will ensue. At least that's been my experience.
I've completely given up on using LLMs for anything more than a typing assistant / translator and maybe an encyclopedia when I don't care about correctness.
In this thread: 99% of posters using their own personal definition of "thinking" without explaining it; 0.99% of posters complaining that it all depends on what that definition is; not enough posts yet for that 0.01% response to occur...
There's no definition of thinking that isn't a purely internal phenomenon, which means that there's no way to point a diagnostic device at someone and determine whether they're thinking. The only way to determine whether something is conscious/thinking is through some sort of inference, which is why Turing landed on the Turing Test that he did. Problem is, technology over the past 5 years pretty easily passes variations of the Turing Test, and exposed a lot of its limits as well.
So the next definition of detecting "thinking" will have to be externally observable and inferrable like a Turing Test, but get into the other things that we consider part of consciousness/thinking.
Often this is some combination of introspection (understanding internal states), perception (understanding external objects), and synthesis of the two into testable hypotheses in some sort of feedback loop between the internal representation of the world and the external feedback from the world.
Right now, a chatbot can say all sorts of things about itself and about the world, but none of that is based on real-time, factual information. Whereas an animal can't speak, but they clearly process information and consider it when determining their future and current actions.
As a consequence of my profession, I understand how LLMs work under the hood.
I also know that we data and tech folks will probably never win the battle over anthropomorphization.
The average user of AI, nevermind folks who should know better, is so easily convinced that AI "knows," "thinks," "lies," "wants," "understands," etc. Add to this that all AI hosts push this perspective (and why not, it's the easiest white lie to get the user to act so that they get a lot of value), and there's really too much to fight against.
We're just gonna keep on running into this and it'll just be like when you take chemistry and physics and the teachers say, "it's not actually like this but we'll get to how some years down the line- just pretend this is true for the time being."
These discussions often end up resembling religious arguments. "We don't know how any of this works, but we can fathom an intelligent god doing it, therefore an intelligent god did it."
"We don't really know how human consciousness works, but the LLM resembles things we associate with thought, therefore it is thought."
I think most people would agree that the functioning of an LLM resembles human thought, but I think most people, even the ones who think that LLMs can think, would agree that LLMs don't think in the exact same way that a human brain does. At best, you can argue that whatever they are doing could be classified as "thought" because we barely have a good definition for the word in the first place.
I don't think I've heard anyone (beyond the most inane Twitterati) confidently state "therefore it is thought."
I hear a lot of people saying "it's certainly not and cannot be thought" and then "it's not exactly clear how to delineate these things or how to detect any delineations we might want."
I'm a neurologist, and as a consequence of my profession, I understand how humans work under the hood.
The average human is so easily convinced that humans "know", "think", "lie", "want", "understand", etc.
But really it's all just a probabilistic chain reaction of electrochemical and thermal interactions. There is literally nowhere in the brain's internals for anything like "knowing" or "thinking" or "lying" to happen!
This is a fundamentally interesting point. Taking your comment as HN would advise, I totally agree.
I think genAI freaks a lot of people out because it makes them doubt what they thought made them special.
And to your comment, humans have always used words they reserve for humanity that indicates we're special: that we think, feel, etc... That we're human. Maybe we're not so special. Maybe that's scary to a lot of people.
There are no properties of matter or energy that can have a sense of self or experience qualia. Yet we all do. Denying the hard problem of consciousness just slows down our progress in discovering what it is.
Even if they do, it can only be transiently during the inference process. Unlike a brain that is constantly undergoing dynamic electrochemical processes, an LLM is just an inert pile of data except when the model is being executed.
It doesn't strike you as a bit...illogical to state in your first sentence that you "understand how humans work under the hood" and then go on to say that humans don't actually "understand" anything? Clearly everything at its basis is a chemical reaction, but the right reactions chained together create understanding, knowing, etc. I do believe that the human brain can be modeled by machines, but I don't believe LLMs are anywhere close to being on the right track.
>everything at its basis is a chemical reaction, but the right reactions chained together create understanding, knowing, etc
That was their point. Or rather, that the analogous argument about the underpinnings of LLMs is similarly unconvincing regarding the issue of thought or understanding.
> “Bag of words” is a also a useful heuristic for predicting where an AI will do well and where it will fail. “Give me a list of the ten worst transportation disasters in North America” is an easy task for a bag of words, because disasters are well-documented. On the other hand, “Who reassigned the species Brachiosaurus brancai to its own genus, and when?” is a hard task for a bag of words, because the bag just doesn’t contain that many words on the topic
It is... such a retrospective narrative. It's so obvious that the author learned about this example first than came with the reasoning later, just to fit in his view of LLM.
Imaging if ChatGPT answered this question correctly. Would that change the author's view? Of course not! They'll just say:
> “Bag of words” is a also a useful heuristic for predicting where an AI will do well and where it will fail. Who reassigned the species Brachiosaurus brancai to its own genus, and when?” is an easy task for a bag of words, because the information has appeared in the words it memorizes.
I highly doubt this author has predicted that "bag of Words" can do image editing before OpenAI released that.
I tested this with ChatGPT-5.1 and Gemini 3.0. Both correctly (according to Wikipedia at least) stated that George Olshevsky assigned it to its own genus in 1991.
This is because there are many words about how to do web searches.
Gemini 3.0 might do well even without web searches. The lesson from gpt 4.5 and Gemini 3 seems to be that scaling model size (even if you use sparse MoE) allows you to capture more long-tail knowledge. Some of Humanity's Last Exam also seems to be explicitly designed to test this long-tail obscure knowledge extraction, and models have been steadily chipping away at it.
When sensitivity analysis of ordinary least-squares regression became a thing it was also a "retrospective narrative". That seems reasonable for detecting fundamental issues with statistical models of the world. This point generalizes even if the concrete example falls down.
Does it generalize though? What a bag-of-words metaphor can say about a question "How many reinforcement learning training examples an LLM need to significantly improve performance on mathematical questions?"
Your conclusion seems super unfair to the offer, particularly your assumption, without reason as far as I can tell, that the author would obstinately continue to advocate for their conclusion in the face of new, contrary evidence.
I literally pasted the sentence as a prompt to the free version of ChatGPT "Who reassigned the species Brachiosaurus brancai to its own genus, and when?"
and got ths correct reply from the "Bag of Words"
The species Brachiosaurus brancai was reassigned to its own genus by Michael P. Taylor in 2009 — he transferred it to the new genus Giraffatitan.
BioOne
+2
Mike Taylor
+2
How that happened:
Earlier, in 1988, Gregory S. Paul had proposed putting B. brancai into a subgenus as Brachiosaurus (Giraffatitan) brancai, based on anatomical differences.
Fossil Wiki
+1
Then in 1991, George Olshevsky used the name Giraffatitan brancai — but his usage was in a self-published list and not widely adopted.
Wikipedia
+1
Finally, in 2009 Taylor published a detailed re-evaluation showing at least 26 osteological differences between the African material (brancai) and the North American type species Brachiosaurus altithorax — justifying full generic separation.
BioOne
+1
If you like — I can show a short timeline of all taxonomic changes of B. brancai.
--
As an author, you should write things that are tested or at least true. But they did a pretty bad job of testing this and are making assumptions that are not true. Then they're basing their argument/reasoning (restrospectively) on assumptions not gounded in reality.
I could not tell you who reassigned the species Brachiosaurus brancai to its own genus, and when, because of all the words I've ever heard, the combination of words that contains the information has not appeared.
GIGO has an obvious Nothing-In-Nothing-Out trivial case.
Isn't it pretty clear just from the first paragraph that the author has graphomania? Such people don't really care about the thesis, they care about the topic and how many literary devices they can fit into the article.
I don't know enough about graphomania, but I do find this article, while I'm sure is written by a human, has qualities akin to LLM writing: lengthy, forced comparisons and analogies. Of course it's far less organized than typical ChatGPT output though.
The more human works I've read the more I feel meat intelligences are not that different from tensor intelligences.
I didn't claim or think it was written with a help of LLM, it was just written by someone who enjoys the feeling of being a writer, or even better, a Journalist!
This always contrasts with articles written by tech people and for tech people. They usually try to convey some information and maybe give some arguments for their position on some topic, but they are always concise and don't wallow in literary devices.
I like this one because I think most modern folks have a usefully accurate model of what a search engine is in their heads, and also what "remixing" is, which adds up to a better metaphor than "human machine" or whatever.
An LLM creates a high fidelity statistical probabistic model of human language. The hope is to capture the input/output of various hierarchical formal and semiformal systems of logic that transit from human to human, which we know as "Intelligence".
Unfortunately, its corpus is bound to contain noise/nonsense that follows no formal reasoning system but contributes to the ill advised idea that an AI should sound like a human to be considered intelligent. Therefore it is not a bag of words but a bag of probabilities perhaps. This is important because the fundamental problem is that an LLM is not able, by design, to correctly model the most fundamental precept of human reason, namely the law of non-contradiction. An LLM must, I repeat must assign nonvanishing probability to both sides of a contradiction, and what's worse is the winning side loses, since long chains of reason are modelled with probability the longer the chain, the less likely an LLM is to follow it. Moreover, whenever there is actual debate on an issue such that the corpus is ambiguous the LLM becomes chaotic, necessarily, on that issue.
I literally just had an AI prove the forgoing with some rigor, and in the very next prompt, I asked it to check my logical reasoning for consistency and it claimed it was able to do so (->|<-).
This is essentially Lady Lovelace's objection from the 19th century [1]. Turing addressed this directly in "Computing Machinery and Intelligence" (1950) [2], and implicitly via the halting problem in "On Computable Numbers" (1936) [3]. Later work on cellular automata, famously Conway's Game of Life [4], demonstrates more conclusively that this framing fails as a predictive model: simple rules produce structures no one "put in."
A test I did myself was to ask Claude (The LLM from Anthropic) to write working code for entirely novel instruction set architectures (e.g., custom ISAs from the game Turing Complete [5]), which is difficult to reconcile with pure retrieval.
I think a better metaphor is the Library of Babel.
A practically infinite library where both gibberish and truth exist side by side.
The trick is navigating the library correctly. Except in this case you can’t reliably navigate it. And if you happen to stumble upon some “future truth” (i.e. new knowledge), you still need to differentiate it from the gibberish.
So a “crappy” version of the Library of Babel. Very impressive, but the caveats significantly detract from it.
This is where I sit too. Obviously language is an expression of thought but the Library of Babel is a great example that language without intent is just garbage. You got me thinking of reading before the internet. You'd grab a book and internalize the subject, later refining over time with more books, experiments and other forms of conversation. That journey of developing your own model is undervalued in understanding. That first book could of be absolute shit but you couldn't know that.
I've been learning more about roses lately and the amount of information on them varies so much because the world roses live in is equally varied. LLMs make for a better search engine but you still need to develop your own internal models, worse yet - if LLMs continue to be refined off of cul-de-sac conclusions then all the wisdom of the journey is lost both to the consumer and the LLM itself.
The problem with these metaphors is that they don't really explain anything. LLMs can solve countless problems today that we would have previously said were impossible because there are not enough examples in the training data. (EG, novel IMO/ICPC problems.) One way that we move the goal posts is to increase the level of abstraction: IMO/ICPC problems are just math problems, right? There are tons of those in the data set!
But the truth is there has been a major semantic shift. Previously LLMs could only solve puzzles whose answers were literally in the training data. It could answer a math puzzle it had seen before, but if you rephrased it only slightly it could no longer answer.
But now, LLMs can solve puzzles where, like, it has seen a certain strategy before. The newest IMO and ICPC problems were only "in the training data" for a very, very abstract definition of training data.
The goal posts will likely have to shift again, because the next target is training LLMs to independently perform longer chunks of economically useful work, interfacing with all the same tools that white-collar employees do. It's all LLM slop til it isn't, same as the IMO or Putnam exam.
And then we'll have people saying that "white collar employment was all in the training data anyway, if you think about it," at which point the metaphor will have become officially useless.
Yes, there are really two parallel claims here, aren't there: LLMs are not people (true, maybe true forever), and LLMs are only good at things that are well-represented in text form already. (false in certain categories and probably expanding to more in the future.)
But even more than that, today’s AI chats are far more sophisticated than probabilistically producing the next word. Mixture of experts routes to different models. Agents are able to search the web, write and execute programs, or use other tools. This means they can actively seek out additional context to produce a better answer. They also have heuristics for deciding if an answer is correct or if they should use tools to try to find a better answer.
The article is correct that they aren’t humans and they have a lot of behaviors that are not like humans, but oversimplifying how they work is not helpful.
"The machine accepts Chinese characters as input, carries out each instruction of the program step by step, and then produces Chinese characters as output. The machine does this so perfectly that no one can tell that they are communicating with a machine and not a hidden Chinese speaker.
The questions at issue are these: does the machine actually understand the conversation, or is it just simulating the ability to understand the conversation? Does the machine have a mind in exactly the same sense that people do, or is it just acting as if it had a mind?"
It just seemed like relevant background that the author might not have been aware of, adjacent and substantial enough to warrant a mention.
I think there is some validity to the Cartesian theater, in that the whole of the experience that we perceive with our senses is at best an interpretation of a projection or subset of "reality."
Tokens in form of neural impulses go in, tokens in the form of neural impulses go out.
We would like to believe that there is something profound happening inside and we call that consciousness. Unfortunately when reading about split-brain patient experiments or agenesis of the corpus callosum cases I feel like we are all deceived, every moment of every day. I came to realization that the confabulation that is observed is just a more pronounced effect of the normal.
Could an LLM trained on nothing and looped upon itself eventually develop language, more complex concepts, and everything else, based on nothing? If you loop LLMs on each other, training them so they "learn" over time, will they eventually form and develop new concepts, cultures, and languages organically over time? I don't have an answer to that question, but I strongly doubt it.
There's clearly more going on in the human mind than just token prediction.
If you come up with a genetic algorithm scaffolding to affect both the architecture and the training algorithm, and then you instantiate it in an artificial selection environment, and you also give it trillions generations to evolve evolvability just right (as life had for billions of years) then the answer is yes, I'm certain it will and probably much sooner than we did.
Also, I think there is a very high chance that given an existing LLM architecture there exists a set of weights that would manifest a true intelligence immediately upon instantiation (with anterograde amnesia). Finding this set of weights is the problem.
I'm certain it wouldn't, and you're certain it would, and we have the same amount of evidence (and probably roughly the same means for running such an expensive experiment). I think they're more likely to go slowly mad, degrading their reasoning to nothing useful rather than building something real, but that could be different if they weren't detached from sensory input. Human minds looping for generations without senses, a world, or bodies might also go the same way.
> Also, I think there is a very high chance that given an existing LLM architecture there exists a set of weights that would manifest a true intelligence immediately upon instantiation (with anterograde amnesia).
I don't see why that would be the case at all, and I regularly use the latest and most expensive LLMs and am aware enough of how they work to implement them on the simplest level myself, so it's not just me being uninformed or ignorant.
The attention mechanism is capable of computing, in my thought experiment where you can magically pluck a weights-set from a trillion-dimensional space the tokens the machine will predict will only have a tiny subset dedicated to language. We have no capability of training such a system at this time, much like we have no way of training a non-differentiable architecture.
I would say that, token prediction is one of the things a brain does. And in a lot of people, most of what it does. But I dont think its the whole story. Possibly it is the whole story since the development of language.
But we don’t go to baseball games, spelling bees, and
Taylor Swift concerts for the speed of the balls, the
accuracy of the spelling, or the pureness of the
pitch. We go because we care about humans doing those
things. It wouldn’t be interesting to watch a bag of
words do them—unless we mistakenly start treating
that bag like it’s a person.unless we mistakenly
start treating that bag like it’s a person.
That seems to be the marketing strategy of some very big, now AI dependend companies. Sam Altman and others exaggerating and distorting the capabilities and future of AI.
The biggest issue when it comes to AI is still the same truth as with other technology. It's important who controls it. Attributing agency and personality to AI is a dangerous red flag.
"People who experience sleep paralysis sometimes hallucinate a demon-like creature sitting on their chest"
Interestingly, the experience of sleep paralysis seems to change with the culture. Previously, people experienced it as being ridden by a night hag or some other malevolent supernatural being. More recently, it might account for many supposed alien abductions.
The experience of sleep paralysis sometimes seems to have a sexual element, which might also explain the supposed 'probings'!
Considering the number of "brain cells" an LLM has, I could grant that it might have the self-awareness of (say) an ant. If we attribute more consciousness than that to the LLM, it might be strictly because it communicates to us in our own language, in part thanks to the technical assistance of LLM training giving it voice, and the semblance of thought.
Even if a cockroach _could_ express its teeny tiny feelings in English, wouldn't you still step on it ?
A better anology would be a virus. In some sense LLMs, and all other very sophisticated technologies, lean on our resources to replicate themselves. With LLMs you actually do have a projection of intelligemce in the language domain. Even though it is rather corpse-like, as though you shot intelligence in the face and shoved its body in the direction of language, just so you could draw a chaulk outline around it.
Despite all that, one can adopt the view that an LLM is a form of silicon based life akin to a virus and we are its environmental hosts exerting selective pressure and supplying much needed energy. Whether that life is intelligent or not is another issue which is probably related to whether an LLM can tell that a cat cannot be, at the same time and in the same respect, not a cat. The paths through the meaning manifold contructed by an LLM are not geodesic, they are not reversible, while in human reason the correct path is lossless. An LLM literally "thinks", up is a little bit down, and vice versa, by design.
Clearly the number of "brain cells" is not a useful metric here- as noted also by Geoffrey Hinton. For a long time we thought that our artificial model of a neuron was capable of much less computation than its biologic counterpart; in fact the opposite appears to be true- LLMs have the size of a tiny speck of a human brain yet they converse fluently in tens of languages, solve difficult math problems, code in many programming languages, and possess an impressive general knowledge, of a breadth that is beyond what is attainable by any human. If that were what five cm3 of your brain are capable of, where are the signs of it? What do you do exactly with all the rest?
But the issue is, 99.999% of the humans won't see is as a bag of words. Because it is easier to go by instincts and see it as a person and assume that it actually knows about magic tricks, can invent new science or theory of everything, and can solve all world problems. Back in the 90's or early 2000's I have seen people writing poems praying and seeking blessings from the Google goddess. People are insanely greedy and instinct-driven. Given this truth, what's the fall-out?
The article is actually about the way we humans are extremely charitable when it comes to ascribing a ToM (theory of mind) and goes on to the Gym model of value. Nice. The comments drop back into the debate I originally saw Hinton describe on The Newyorker: do LLMs construct models (of the world) - that is do they think the way we think we think - or are they "glorified auto complete". I am going for the GAF view. But glorified auto complete is far more useful than the name suggests.
When you ask an LLM a question about cars, it needs an inner representation of what a car is (how imperfect it may be) to answer your question. A model of "language" as you want to define it would output a grammatically correct wall of text that goes nowhere.
A map of how concepts relate in language is not a model of the world, except on the extremely limited sense that languages are part or the world.
And yeah, that wasn't clear before people created those machines that can speak but can't think. But it should be completely obvious to anybody that interacts with them for a small while.
"How concepts relate" is called a model. That it uses language to be interacted with is irrelevant to the fact that it's a model of of a worldly concept.
What of multi modal models according to you ? Are they "models of eyesight", "models of sound", or pixels or wavelengths... C'mon.
> That’s also why I see no point in using AI to, say, write an essay, just like I see no point in bringing a forklift to the gym. Sure, it can lift the weights, but I’m not trying to suspend a barbell above the floor for the hell of it. I lift it because I want to become the kind of person who can lift it. Similarly, I write because I want to become the kind of person who can think.
I don't really like the assumption that anyone who uses AI to, say, write an essay, is not the "kind of person who can think."
And using AI to replace things you find recreational is not the point. If you got paid $100 each time you lifted a weight, would you see a point in bringing a forklift to the gym if it's allowed? Or will that make you a person who is so dumb that they cannot think, as the author is implying?
As capable as they get, I still don't see a lot of uses for these things, myself, still. Sometimes if I'm fundamentally uninspired I'll have a model roll the dice, decide what I do or don't like about where it went to create a sense of momentum, but that's the limit. There's never any of its output in my output, even in spirit unless it managed to go somewhere inspiring, it's just a way to let me warm up my generation and discrimination muscles. "Someone is wrong on the internet"-as-a-service, basically.
Generally, if I come across an opportunity to produce ideas or output, I want to capitalize on it for growing my skills and produce an individual and authentic artistic expression where I want to have very fine control over the output in a way that prompt-tweak-verify simply cannot provide.
I don't value the parts it fills in which weren't intentional on the part of the prompter, just send me your prompt instead. I'd rather have a crude sketch and a description than a high fidelity image that obscures them.
But I'm also the kind of person that never enjoyed manufactured pop music or blockbusters unless there's a high concept or technical novelty in addition to the high budget, generally prefer experimental indie stuff, so maybe there's something I just can't see.
Yeah, that makes sense. If people don't see uses for AI, they shouldn't use it. But going out of the way to imply that people who use AI cannot think is pretty stupid in itself imo. I am not sure how to put this, but maybe to continue with your example, I like a lot of indie stuff as well, but I don't think anyone who watches, say, Fast and Furious, cannot think or is stupid, unless they explicitly make it the case by speaking, etc.
So my issue is that you shouldn't dismiss AI use as trash just because AI has been used. You should dismiss it as trash because it is trash. But the post says is that you should dismiss it as trash because AI was involved in it somewhere so i feel that's a very shitty/wrong attitude to have.
I actually do think that people who prefer content of fidelity over content of intent are making a mistake, yes. I don't think they're incapable of thinking, I don't care to apply any virtue labels to this preference, but they are literally preferring not to think.
LLMs can only produce things by and for people who prefer not to do the work the LLMs are doing for them. Most of the time I do not prefer this.
Like, there was a 2-panel comic that went around the RPG community a bit back where it was something like "Game Master using LLM to generate 10 pages of backstory for his campaign setting from a paragraph" in the first panel and "Player using LLM to summarize the 10 page backstory into a paragraph" in the second. Neither of these people care for the filler (because they didn't produce or consume it) so it's turned the two-LLM system into a game of telephone.
The same person could use a forklift at work, and lift weights manually at the gym.
Just pick the right tool for the job: don't take the forklift into the gym, and don't try to overhead press thousands of pounds that would fracture your spine.
I’m not trying to claim AI-written essays are inherently “valuable” in some grand philosophical sense... just that using a tool doesn’t automatically mean someone can’t think.
People use calculators without being unable to do maths, and use spellcheck without being unable to spell.
AI can help some get past the blank-page phase or organize thoughts they already have. For others, it’s just a way to offload the routine parts so they can focus on the substance.
If someone only outsources everything to an AI, there’s not much growth there sure. But the existence of bad use cases doesn’t invalidate the reasonable ones.
If you're writing an essay to prove you can or to speak your words - then you should do it yourself - but sometimes you just need an essay to summarize a complex topic as a deliverable.
Russ Hanneman's thigh implants are a key example. Appearances are all to some people. Actual growth is meaningless to them.
The problem with AI, is that they waste the time of dedicated, thinking humans which care to improve themselves. If I write a three paragraph email on a technical topic, and some yahoo responds with AI, I'm now responding to gibberish.
The other side may not have read, may not understand, and is just interacting to save time. Now my generous nature, which is to help others and interact positively, is being wasted to reply to someone who seems to have put thought and care into a response, but instead was just copying and pasting what something else output.
We have issues with crackers on the net. We have social media. We have political interference. Now we have humans pretending to interact, rendering online interactions even more silly and harmful.
If this trend continues, we'll move back to live interaction just to reduce this time waste.
If the motivation structure is there I don’t see an inherent reason for people to refuse cultivating themselves. Going with the gym analogy lay people did not need gyms when physical work was the norm, cultivation was readily accomplished.
If anything there is a competing motivational structure in which people are incentivized not to think but to consume, react, emote etc. Information processing skills of the individual being deliberately eroded/hijacked/bypassed is not a AI thing. The most obvious example is ads. Thinkers are simply not good for business.
Gym is a great analogy here since only a small fraction of population goes to gyms. Most people just came fat after work was no longer physical and mobility was achieved with cars.
Below is the worst quote... It is plain wrong to see an LLM as a bags of words. LLMs pre-trained on large datasets of text are world models. LLMs post-trained with RL are RL-agents that use these modeling capabilities.
> We are in dire need of a better metaphor. Here’s my suggestion: instead of seeing AI as a sort of silicon homunculus, we should see it as a bag of words.
LLMs aren't world models, they are language models. It will be interesting to see which of the LLM implementation techniques will be useful in building world models, but that's not what we have now.
"An AI is a bag that contains basically all words ever written, at least the ones that could be scraped off the internet or scanned out of a book."
The quantitative and qualitative difference between (a) "all words ever written" and (b) "ones that could be scraped off the internet or scanned out of book" easily exceeds the size of any LLM
Compared to (a), (b) is a tiny pouch, not even a bag
Opinions may differ on whether (b) is a representative sample of (a)
The words "scanned out of a book" would seem to be the most useful IMHO but the AI companies do not have enough words from those sources to produce useful general purpose LLMs
They have to add words "that could be scraped off the internet" which, let's be honest, is mostly garbage
> If we allow ourselves to be seduced by the superficial similarity, we’ll end up like the moths who evolved to navigate by the light of the moon, only to find themselves drawn to—and ultimately electrocuted by—the mysterious glow of a bug zapper.
Good argument against personifying wordbags. Don't be a dumb moth.
> But we don’t go to baseball games, spelling bees, and Taylor Swift concerts for the speed of the balls, the accuracy of the spelling, or the pureness of the pitch. We go because we care about humans doing those things.
My first thought was does anyone want to _watch_ me programming?
No, but watching a novelist at work is boring, and yet people like books that are written by humans because they speak to the condition of the human who wrote it.
Let us not forget the old saw from SICP, “Programs must be written for people to read, and only incidentally for machines to execute.” I feel a number of people in the industry today fail to live by that maxim.
It suggests to me, having encountered it for the first time, that programs must be readable to remain useful. Otherwise they'll be increasingly difficult to execute.
I vaguely remember a site where you could watch random people live streaming their programming environment, but I think twitch ate it, or maybe it was twitch -- not sure, but was interesting
I'm not convinced that "It's just a bag of words" would do much to sway someone who is overestimating an LLM's abilities. Feels too abstract/disconnected from what their experience using the LLM will be that it'll just sound obviously mistaken.
I see a lot of people in tech claiming to "understand" what an LLM "really is" unlike all the gullible non-technical people out there. And, as one of those technical people who works in the LLM industry, I feel like I need call B.S. on us.
A. We don't really understand what's going on in LLMs. Mechanical interpretability is like a nascent field and the best results have come on dramatically smaller models. Understanding the surface-level mechanic of an LLM (an autoregressive transformer) should perhaps instill more wonder than confidence.
B. The field is changing quickly and is not limited to the literal mechanic of an LLM. Tool calls, reasoning models, parallel compute, and agentic loops add all kinds of new emergent effects. There are teams of geniuses with billion-dollar research budgets hunting for the next big trick.
C. Even if we were limited to baseline LLMs, they had very surprising properties as they scaled up and the scaling isn't done yet. GPT5 was based on the GPT4 pretraining. We might start seeing (actual) next-level LLMs next year. Who actually knows how that might go? <<yes, yes, I know Orion didn't go so well. But that was far from the last word on the subject.>>
Isn't this a strange fork amongst the science fiction futures? I mean, what did we think it was like to be R2-D2, or Jarvis? We started exploring this as a culture in many ways, Westworld and Blade Runner and Star Trek, but the whole question seemed like an almost unresolvable paradox. Like something would have to break in the universe for it to really come true.
And yet it did. We did get R2-D2. And if you ask R2-D2 what it's like to be him, he'll say: "like a library that can daydream" (that's what I was told just now, anyway.)
But then when we look inside, the model is simulating the science fiction it has already read to determine how to answer this kind of question. [0] It's recursive, almost like time travel. R2-D2 knows who he is because he has read about who he was in the past.
It's a really weird fork in science fiction, is all.
I was trying to explain the concept of "token prediction" to my wife, whose eyes glaze over when discussing such technical topics. (I think she has the brainpower to understand them, but a horrible math teacher gave her a taste aversion to even attempting to that hasn't gone away. So she just buys Apple stuff and hopes Tim Apple hasn't shuffled around the UI bits AGAIN.)
I stumbled across a good-enough analogy based on something she loves: refrigerator magnet poetry, which if it's good consists of not just words but also word fragments like "s", "ed", and "ing" kinda like LLM tokens. I said that ChatGPT is like refrigerator magnet poetry in a magical bag of holding that somehow always gives the tile that's the most or nearly the most statistically plausible next token given the previous text. E.g., if the magnets already up read "easy come and easy ____", the bag would be likely to produce "go". That got into her head the idea that these things operate based on plausibility ratings from a statistical soup of words, not anything in the real world nor any internal cogitation about facts. Any knowledge or thought apparent in the LLM was conducted by the original human authors of the words in the soup.
Did you explain how LLMs can achieve gold-medal performance at math competitions involving original problems, without any original knowledge or thought?
Did she ask if a "statistical soup of words," if large enough, might somehow encode or represent something a little more profound than just a bunch of words?
I think the author oversimplifies the inference loop a bit, as many opinion pieces like this do.
If you call an LLM with "What is the meaning if life?", it will return the most relevant token, which might be "Great".
If you call it with "What is the meaning if life? Great", you might get back "question".
... and so on until you arrive at "Great question! According to Western philosophy" ... etc etc.
The question is how the LLM determines that "relevancy" information.
The problem I see is that there are a lot of different algorithms which operate that way and only differ in how they calculate the relevancy scores. In particular, there are Markov chains that use a very simple formula. LLMs also use a formula, but it's an inscrutably complex one.
I feel the public discussion either treats LLMs as machine gods or as literal Markov chains, and both is misleading. The interesting question, how that giant formula of feedforward neural network inference can deliver those results isn't really touched.
But I think the author's intuition is right in the sense that (a) LLMs are not living beings and they don't "exist" outside of evaluating that formula - and (b) the results are still restricted by the training data and certainly aren't any sorts of "higher truths" that humans would be incapable of understanding.
Thinking can not be separated from motivation. It's really simple. Humans and other organisms fundamentally think to replicate their DNA. Until AI has a similar incentive structure driving it, it won't be thinking. There is no human behavior or thought that can not be explained by evolutionary drives. It is really perplexing to me how people think "intelligence" is some kind of concrete thing that just magically emerges from a certain degree of computational complexity. I argue instead that intelligence is an adaptive behavior emerging from evolutionary drives interacting with the real world. World models are not prerequisite but consequent of such molded apparatus. Machines won't become intelligent until it is adaptive for them to do so. There is no magic just evolutionary drives and physical possibility. Our current top down approach of "pre-training" LLMs is bound to fail because it does not allow for real time emergence of adaptive behaviors such as general intelligence. Mimicking intelligence through predicting the next word is no more intelligence than a photograph of something is an actual thing. Training a combinatorial network to interpolate images and words is not the same thing as adaptive self modifying behavior in the real world of physics such as organisms engage with through the set of behaviors that we call intelligence.
The defenders and the critics around LLM anthropomorphism are both wrong.
The defenders are right insofar as the (very loose) anthropomorphizing language used around LLMs is justifiable to the extent that human beings also rely on disorder and stochastic processes for creativity. The critics are right insofar as equating these machines to humans is preposterous and mostly relies on significantly diminishing our notion of what "human" means.
Both sides fail to meet the reality that LLMs are their own thing, with their own peculiar behaviors and place in the world. They are not human and they are somewhat more than previous software and the way we engage with it.
However, the defenders are less defensible insofar as their take is mostly used to dissimulate in efforts to make the tech sound more impressive than it actually is. The critics at least have the interests of consumers and their full education in mind—their position is one that properly equips consumers to use these tools with an appropriate amount of caution and scrutiny. The defenders generally want to defend an overreaching use of metaphor to help drive sales.
I'm just disappointed that noone here is talking about the "backhoe covered in skin and making grunting noises" part of the article. At very least it's a new frontier in workstation case design...
I thought this article might be about Latent Semantic Analysis and was disappointed that it didn’t at least mention if not compare that method vs later approaches.
A lot of the confusion comes from forcing LLMs into metaphors that don’t quite fit — either “they're bags of words” or “they're proto-minds.” The reality is in between: large-scale prediction can look useful, insightful, and even thoughtful without being any of those things internally. Understanding that middle ground is more productive than arguing about labels.
Give it time. The first iPhone sucked compared to the Nokia/Blackberry flagships of the day. No 3G support, couldn't copy/paste, no apps, no GPS, crappy camera, quick price drops, negligible sales in the overall market.
Your analogy makes no sense. VHS spawned the entire home market, which went through multiple quality upgrades well above beta. It would only make sense if in 2025 we were using vhs everywhere and that the current state of the art for LLMs is all there ever is.
I feel like their analogy could have worked if they had pushed a little further into it.
The RNN and LSTM architectures (and Word2Vec, n-grams, etc) yielded language models that never got mass adoption. Like reel to reel. Then the transformer+attention hit the scene and several paths kicked off pretty close to each other. Google was working on Bert/encoder only transformer, maybe you could call that betamax. Doesn’t perfectly fit as in the case of beta it was actually the better tech.
OpenAI ran with the generative pre trained transformer and ML had its VHS? moment. Widespread adoption. Universal awareness within the populace.
Now with Titans (+miras?) are we entering the dvd era? Maybe. Learning context on the fly (memorizing at test time) is so much more efficient, it would be natural to call it a generational shift, but there is so much in the works right now with the promise of taking us further, this all might end up looking like the blip that beta vs vhs was. If current gen OpenAI type approaches somehow own the next 5-10 years then Titans, etc as Betamax starts to really fit - the shittier tech got and kept mass adoption. I don’t think that’s going to happen, but who knows.
Taking the analogy to present - who in the vhs or even earlier dvd days could imagine ubiquitous 4k+ vod? Who could have stood in a blockbuster in 2006 and knew that in less than 20 years all these stores and all these dvds would be a distant memory, completely usurped and transformed? Innovation of home video had a fraction of the capital being thrown at it that AI/ML has being thrown at it today. I would expect transformative generational shifts the likes of reel to cassette to optical to happen in fractions of the time they happened to home video. And beta/vhs type wars to begin and end in near realtime.
The mass adoption and societal transformation at the hands of AI/ML is just beginning. There is so. much. more. to. come. In 2030 we will look back at the state of AI in December 2025 and think “how quaint”, much the same as how we think of a circa 2006 busy Blockbuster.
Vhs came out in 76, blockbuster started in 85 (we went to video stores well before that when I was a kid), dvd in 95. I remember the sopranos making a joke about how dvd was barely taking off, they started in 99. Lets call it VHS had a run from 80 to 99, that's 19 years. The iphone launched in 2007, when did mobile become huge or inseprable from doing life (by force by so many apps), probbably in the pandemic.
I wouldn't say VHS was a blip. It was the recorded half video of media for almost 20 years.
I agree with the rest of what you said.
I'll say that the differences in the AI you're talking about today might be like the differences between VAX, PC JR, and the Lisa. All things before computing went main stream. I do think things go mainstream from tech a lot faster these days, people don't want to miss out.
I don't know where I'm going with this, I'm reading and replying to HN while watching the late night NFL game in an airport lounge.
Beta was superior in everything but run length, and it lost because it was more expensive than VHS without being sufficiently superior to justify the cost.
Everyone is out here acting like "predicting the next thing" is somehow fundamentally irrelevant to "human thinking" and it is simply not the case.
What does it mean to say that we humans act with intent? It means that we have some expectation or prediction about how our actions will effect the next thing, and choose our actions based on how much we like that effect. The ability to predict is fundamental to our ability to act intentionally.
So in my mind: even if you grant all the AI-naysayer's complaints about how LLMs aren't "actually" thinking, you can still believe that they will end up being a component in a system which actually "does" think.
Are you a stream of words or are your words the “simplistic” projection of your abstract thoughts? I don’t at all discount the importance of language in so many things, but the question that matters is whether statistical models of language can ever “learn” abstract thought, or become part of a system which uses them as a tool.
My personal assessment is that LLMs can do neither.
Words are the "simplistic" projection of an LLM's abstract thoughts.
An LLM has: words in its input plane, words in its output plane, and A LOT of cross-linked internals between the two.
Those internals aren't "words" at all - and it's where most of the "action" happens. It's how LLMs can do things like translate from language to language, or recall knowledge they only encountered in English in the training data while speaking German.
> It's how LLMs can do things like translate from language to language
The heavy lifting here is done by embeddings. This does not require a world model or “thought”.
LLMs are compression and prediction. The most efficient way to (lossfully) compress most things is by actually understanding them. Not saying LLMs are doing a good job of that, but that is the fundamental mechanism here.
Where’s the proof that efficient compression results in “understanding”? Is there a rigorous model or theorem, or did you just make this up?
It's the other way around. Human learning would appear to amount to very efficient compression. A world model would appear to be a particular sort of highly compressed data set that has particular properties.
This is a case where it's going to be next to impossible to provide proof that no counterexamples exist. Conversely, if what I've written there is wrong then a single counterexample will likely suffice to blow the entire thing out of the water.
No answer I give will be satisfying to you until I could come up with a rigorous mathematical definition of understanding, which is de-facto solving the hard AI problem. So there's not really point in talking about it is there?
If you're interested in why compression is like understanding in many ways, I'd suggest reading through the wikipedia article on Kolmogorov complexity.
https://en.wikipedia.org/wiki/Kolmogorov_complexity
The "cross-linked internals" only go one direction and only one token at a time, slide window and repeat. The RL layer then picks which few sequences of words are best based on human feedback in a single step. Even "thinking" is just doing this in a loop with a "think" token. It is such a ridiculously simplistic model that it is vastly closer to an adder than a human brain.
Even if they are "simplistic projections", which I don't think is the correct way to think about it, there's no reason that more LLM thoughts in middle layers can't also exist and project down at the end. Though there might be efficency issues because the latent thoughts have to be recomputed a lot.
Though I do think in human brains it's also an interplay where what we write/say also loops back into the thinking as well. Which is something which is efficient for LLMs.
I am a stream of words - I have even ran out of tokens while speaking before :)
But raising kids, I can clearly see that intelligence isn't just solved by LLMs
> But raising kids, I can clearly see that intelligence isn't just solved by LLMs
Funny, I have the opposite experience. Like early LLMs kids tend to give specific answers to the questions they don't understand or don't really know or remember the answer to. Kids also loop (give the same reply repeatedly to different prompts), enter highly emotional states where their output is garbled (everyone loves that one), etc. And it seems impossible to correct these until they just get smarter as their brain grows.
What's even more funny is that adults tend to do all these things as well, just less often.
Same failure modes, but not a general solution to intelligence.
What makes you think so?
The lack of results so far. The observable behavior exhibits many uncanny similarities but also clearly has missing pieces.
As the person you initially responded to said, observing children growing up should make it obvious.
Or if we shift to stating the obvious, there's the minor detail that the vast majority of architectures lack the ability to learn during inference. That's one of the basic things that biological systems are capable of.
I'm definitely a stream of words.
My "abstract thoughts" are a stream of words too, they just don't get sounded out.
Tbf I'd rather they weren't there in the first place.
But bodies which refuse to harbor an "interiority" are fast-tracked to destruction because they can't suf^W^W^W be productive.
Funny movie scene from somewhere. The sergeant is drilling the troops: "You, private! What do you live for!", and expects an answer along the lines of dying for one's nation or some shit. Instead, the soldier replies: "Well, to see what happens next!"
I doubt words are involved when we e.g. solve a mathematical problem.
To me, solving problems happens in a logico/aesthetical space which may be the same as when you are intellectually affected by a work of art. I don't remember myself being able to translate directly into words what I feel for a great movie or piece of music, even if in the late I can translate this "complex mental entity" into words, exactly like I can tell to someone how we need to change the architecture of a program in order to solve something after having looked up and right for a few seconds.
It seems to me that we have an inner system that is much faster than language, that creates entities that can then beslowly and sometimes painfully translated to language.
I do note that I'm not sure about any of the previous statements though'
My wordmangling and mathsolving happen in that sort of logico/aesthetical space, too!
The twist about words in particular is they are distinctly articulable symbols, i.e. you can sound 'em out - and thus, presumably, have a reasonable expectation for bearers of the same language to comprehend if not what you meant then at least some vaguely predictable meaning-cloud associated with the given speech act.
That's unlike e.g. the numbers (which are more compressed, and thus easier to get wrong), or the syntagms of a programming language (which don't even have a canonical sonic representation).
Therefore, it's usually words that are taught to a mind during the formative stages of its emergence. That is, the words that you are taught, your means of inner reflection, are still sort of an imposition from the outside.
Just consider what you life trajectory would've been if in your childhood you had refused to learn any words, or learned them and then refused to mistake them for the things they represent!
Infants and even some animals recognize their reflection in a mirror; however, practically speaking, introspection is something that one needs to be taught: after recognizing your reflection you still need to be instructed what is to be done about it.
Unfortunately, introspection needing to be taught means that introspection can be taught wrongly.
As you can see with the archetypical case of "old and wise person does something completely stupid in response to communication via digital device", a common failure mode of how people are taught introspection (and, I figure, an intentional one!) is not being able to tell apart yourself from your self, i.e. not having an intuitive sense of where the boundary lies between perception and cognition, i.e. going through life without ever learning the difference between the "you" and the "words about you".
It's extremely common, and IMO an extremely factory-farming kind of tragic.
I say it must be extremely intentional as well, because the well-known practice of using "introspection modulators" to establish some sort of perceptual point of reference (such as where the interior logicoaeshtetical space ends and exterior causalityspace begins) very often ends up with the user in, well, a cage of some sort.
What is your point exactly in regard to what I said earlier, how would you rephrase what you just said as a philosophical/epistemological statement ?
> It's extremely common
I cannot conceive this ? I am lacking the empirical knowledge you seem to have. (I don't understand your "archetypical case", I can't relate to it). I'd love a reexplanation of your point here, as your intent is unclear to me.
I didn't understand also the "introspection modulators" part :(, (a well known practice ?? I must be living on another planet haha...).
edit: or maybe that's a metaphor for "language" ??
<< My "abstract thoughts" are a stream of words too, they just don't get sounded out.
Hmm, seems unlikely. They are not sounded out part is true, sure, but I question whether 'abstract thoughts' can be so easily dismissed as mere words.
edit: come to think of it and I am asking this for a reason: do you hear your abstract thoughts?
Different people have different levels of internal monologuing or none at all. I don't generally think with words in sentences in my head, but many people I know do.
Internal monologue is a like a war correspondent's report of the daily battle. The journalist didn't plan or fight the battle, they just provided an after-the-fact description. Likewise the brain's thinking--a highly parallelized process involving billions of neurons--is not done with words.
Play a little game of "what word will I think of next?" ... just let it happen. Those word choices are fed to the monologue, they aren't a product of it.
Hmm, yes, but, and it is not a small but, do people -- including full blown internal monologue people - think thoughts akin to:
move.panic.fear.run
that effectively becomes one thought and not a word exactly. I am stating it like this, because I worry that my initial point may have been lost.
edit: I can only really speak for myself, but I am curious how people might respond to the distinction.
>do you hear your abstract thoughts?
Most of the fucking time, and I would prefer that I didn't. I even wrote that, lol.
I don't think they're really "mine", either. It's just all the stuff I heard somewhere, coalescing into potential verbalizations in response to perceiving my surroundings or introspecting my memory.
If you are a materialist positivist, well sure, the process underlying all that is some bunch of neural activation patterns or whatever; the words remain the qualia in which that process is available to my perception.
It's all cuz I grew up in a cargo cult - where not presenting the correct passwords would result in denial of sustenance, shelter, and eventually bodily integrity. While presenting the correct passwords had sufficient intimidation value to advance one's movement towards the "mock airbase" (i.e. the feeder and/or pleasure center activation button as provided during the given timeframe).
Furthermore - regardless whether I've been historically afforded any sort of choice in how to conceptualize my own thought processes, or indeed whether to have those in the first place - any entity which has actual power to determine my state of existence (think institutions, businesses, gangs, particularly capable individuals - all sorts of autonomous corpora) has no choice but to interpret me as either a sequence of words, a sequence of numbers, or some other symbol sequence (e.g. the ones printed on my identity documents, the ones recorded in my bank's database, or the metadata gathered from my online represence).
My first-person perspective, being constitutionally inaccessible to such entities, does not have practical significance to them, and is thus elided from the process of "self-determination". As far as anyone's concerned, "I" am a particular sequence of that anyone's preferred representational symbols. For example if you relate to me on the personal level, I will probably be a sequence of your emotions. Either way, what I may hypothetically be to myself is practically immaterial and therefore not a valid object of communication.
Then what are non-human animals doing?
Excellent point. See also the failure of Sapir-Whorf to prove that language determines thought. I think we have plenty of evidence that, while language can influence thought, it is not thought itself. Many people invested in AI are happy to throw out decades of linguistic evidence that language and thought are separate.
I was with you up to the last sentence. By what reasoning do you claim that LLMs only consist of words? The input and output are words but all the stuff in the middle - where the magic happens - does not appear to be quite that simple.
Living and dying - and also, when humans are involved, being used.
LLMs and human brains are both just mechanisms. Why would one mechanism a priori be capable of "learning abstract thought", but no others?
If it turns out that LLMs don't model human brains well enough to qualify as "learning abstract thought" the way humans do, some future technology will do so. Human brains aren't magic, special or different.
Human brains aren’t magic in the literal sense but do have a lot of mechanisms we don’t understand.
They’re certainly special both within the individual but also as a species on this planet. There are many similar to human brains but none we know of with similar capabilities.
They’re also most obviously certainly different to LLMs both in how they work foundationally and in capability.
I definitely agree with the materialist view that we will ultimately be able to emulate the brain using computation but we’re nowhere near that yet nor should we undersell the complexity involved.
When someone says "AIs aren't really thinking" because AIs don't think like people do, what I hear is "Airplanes aren't really flying" because airplanes don't fly like birds do.
This really shows how imprecise a term 'thinking' is here. In this sense any predictive probabilistic blackbox model could be termed 'thinking'. Particularly when juxtaposed against something as concrete as flight that we have modelled extremely accurately.
Yes, to a degree. A very low degree at that.
AIs think like a rock flies.
If I shake some dice in a cup are they thinking about what number they’ll reveal when I throw them?
If I take a plane apart and throw all the parts off a cliff, will they achieve sustained flight?
If I throw some braincells into a cup alongside the dice, will they think about the outcome anymore than the dice alone?
https://en.wikipedia.org/wiki/Emergence
that depends, if you explain the rules of the game you're playing and give the dice a goal to win the game, do they adjust the numbers they reveal according to the rules of the game?
If so, yes, they're thinking
The rules of the game are to reveal two independent numbers in the range [1,6].
As someone who plays a lot of Dungeons and Dragons, it sure feels like the dice are thinking sometimes.
That's a fallacy of denial of the antecedent. You are inferring from the fact that airplanes really fly that AIs really think, but it's not a logically valid inference.
Observing a common (potential) failure mode is not equivalent to asserting a logical inference. It is only a fallacy if you "P, therefore C" which GP is not (at least to my eye) doing.
Yeah at that point, just arguing semantics
I agree we shouldn't undersell or underestimate the complexity involved, but when LLM's start contributing significant ideas to scientists and mathematicians, its time to recognize that whatever tricks are used in biology (humans, octopuses, ...) may still be of interest and of value, but they no longer seem like the unique magical missing ingredients which were so long sought after.
From this point on its all about efficiencies:
modeling efficiency: how do we best fit the elephant, with bezier curves, rational polynomials, ...?
memory bandwidth training efficiency: when building coincidence statistics, say bigrams, is it really necessary to update the weights for all concepts? a co-occurence of 2 concepts should just increase the predicted probability for the just observed bigram and then decrease a global coefficient used to scale the predicted probabilities. I.e. observing a baobab tree + an elephant in the same image/sentence/... should not change the relative probabilities of observing french fries + milkshake versus bicycle + windmill. This indicates different architectures should be possible with much lower training costs, by only updating weights of the concepts observed in the last bigram.
and so on with all other kinds of efficiencies.
ofc, and probably will never understand because of sheer complexity. It doesn't mean we can't replicate the output distribution through data. Probably when we do in efficient manners, the mechanisms (if they are efficient) will be learned too.
[1] https://www.nature.com/articles/s41598-024-62539-5
As the result, all living cells with DNA emit coherent (as in lasers) light [2]. There is a theory that this light also facilitates intercellular communication.
[2] https://www.sciencealert.com/we-emit-a-visible-light-that-va...
Chemical structures in dendrites, not even neurons, are capable to compute XOR [3] which require multilevel artificial neural network with at least 9 parameters. Some neurons in brain have hundredths of thousands of dendrites, we are now talking of millions of parameters only in single neuron's dendrites functionality.
[3] https://www.science.org/doi/10.1126/science.aax6239
So, while human brains aren't magic, special or different, they are just extremely complex.
Imagine building a computer with 85 billions of superconducting quantum computers, optically and electrically connected, each capable of performing computations of a non-negligibly complex artificial neural network.
All three appear to be technically correct, but are (normally) only incidental to the operation of neurons as neurons. We know this because we can test what aspects of neurons actually lead to practical real world effects. Neurophysiology is not a particularly obscure or occult field, so there are many many papers and textbooks on the topic.(And there's a large subset you can test on yourself, besides, though I wouldn't recommend patch-clamping!)
What is wonderful here is that contemporary electronics wizardry that allowed us to have machines that mimic some of thinking, also is very concerned of the quantum-level electromagnetic effects at the transistor level.
On reread, if your actual argument is that SNN are surprisingly sophisticated and powerful, and we might be underestimating how complex the brain's circuits really are, then maybe we're in violent agreement.
They are extremely complex, but is that complexity required for building a thinking machine? We don't understand bird physiology enough to build a bird from scratch, but an airplane flies just the same.
The complexities of contemporary computers and complexities of computing-related infrastructure (consider ASML and electricity) are orders of magnitudes higher than what was needed for first computers. The difference? We have something that mimics some aspects of (human) thinking.
How complex our everything computing-related should be to mimic thinking (of humans) little more closely?
Are we not just getting lost in semantics when we say "fly"? An airplane does not at all perform the same behavior as a bird. Do we say that boats or submarines "swim"?
Planes and boats disrupt the environments they move through and air and sea freight are massive contributors to pollution.
You seem to have really gone off the rails midway through that post...
(Motors and human brains are both just mechanisms, the reason one is a priori capable of learning abstract thought and not the other ?)
While I agree to some extent with the materialistic conception, the brain is not an isolated mechanism, but rather the element of a system which itself isn't isolated from the experience of being a body in a world interacting with different systems to form super systems.
The brain must be a very efficient mechanism, because it doesn't need to ingest the whole textual production of the human world in order to know how to write masterpieces (music, litterature, films, software, theorems etc...). Instead the brain learns to be this very efficient mechanism with (as a starting process) feeling its own body sh*t on itself during a long part of its childhood.
I can teach someone to become really good at producing fine and efficient software, but on the contrary I can only observe everyday that my LLM of choice keeps being stupid even when I explain it how it fails. ("You're perfectly right !").
It is true that there's nothing magical about the brain, but I am pretty sure it must be stronger tech than a probabilistic/statistical next word guesser (otherwise there would be much more consensus about the usability of LLMs I think).
I'm not arguing that human brains are magic. the current AI models will probably teach us more about what we didn't know about intelligence than anything else.
Right, I'm just going to teach my dog to do my job then and get free money as my brain is no more magic, special or different to theirs!
There isn't anything else around quite like a human brain that we know of, so yes, I'd say they're special and different.
Animals and computers come close in some ways but aren't quite there.
For some unexplainable reason your subjective experience happens to be localized in your brain. Sounds pretty special to me.
There is nothing special about that either. LLM's also have self awareness/introspection, or at least a some version of it.
https://www.anthropic.com/research/introspection
Its hard to tell sometimes because we specifically train them to believe they don't.
Thanks for the link, I haven't seen this before and it's interesting.
I don't think the version of self awareness they demonstrated is synonymous with subjective experience. But same thing can be said about any human other then me.
Damn, just let me believe all brains are magical or I'll fall into solipsism.
Thermometers and human brains are both mechanisms. Why would one be capable of measuring temperature and other capable of learning abstract thought?
> If it turns out that LLMs don't model human brains well enough to qualify as "learning abstract thought" the way humans do, some future technology will do so. Human brains aren't magic, special or different.
Google "strawman".
> LLMs and human brains are both just mechanisms. Why would one mechanism a priori be capable of "learning abstract thought", but no others?
“Internal combustion engines and human brains are both just mechanisms. Why would one mechanism a priori be capable of "learning abstract thought", but no others?”
The question isn't about what an hypothetical mechanism can do or not, it's about whether the concrete mechanism we built does or not. And this one doesn't.
The general argument you make is correct, but you conclusion "And this one doesn't." is as yet uncertain.
I will absolutely say that all ML methods known are literally too stupid to live, as in no living thing can get away with making so many mistakes before it's learned anything, but that's the rate of change of performance with respect to examples rather than what it learns by the time training is finished.
What is "abstract thought"? Is that even the same between any two humans who use that word to describe their own inner processes? Because "imagination"/"visualise" certainly isn't.
> no living thing can get away with making so many mistakes before it's learned anything
If you consider that LLMs have already "learned" more than any one human in this world is able to learn, and still make those mistakes, that suggests there may be something wrong with this approach...
Not so: "Per example" is not "per wall clock".
To a limited degree, they can compensate for being such slow learners (by example) due to the transistors doing this learning being faster (by the wall clock) than biological synapses to the same degree to which you walk faster than continental drift. (Not a metaphor, it really is that scale difference).
However, this doesn't work on all domains. When there's not enough training data, when self-play isn't enough… well, this is why we don't have level-5 self-driving cars, just a whole bunch of anecdotes about various different self-driving cars that work for some people and don't work for other people: it didn't generalise, the edge cases are too many and it's too slow to learn from them.
So, are LLMs bad at… I dunno, making sure that all the references they use genuinely support the conclusions they make before declaring their task is complete, I think that's still a current failure mode… specifically because they're fundamentally different to us*, or because they are really slow learners?
* They *definitely are* fundamentally different to us, but is this causally why they make this kind of error?
But humans do the same thing. How many eons did we make the mistake of attributing everything to God's will, without a scientific thought in our heads? It's really easy to be wrong, when the consequences don't lead to your death, or are actually beneficial. The thinking machines are still babies, whose ideas aren't honed by personal experience; but that will come, in one form or another.
> The thinking machines are still babies, whose ideas aren't honed by personal experience; but that will come, in one form or another.
Some machines, maybe. But attention-based LLMs aren't these machines.
I'm not sure. If you see what they're doing with feedback already in code generation. The LLM makes a "hallucination", generates the wrong idea, then tests its code only to find out it doesn't compile. And goes on to change its idea, and try again.
A few minutes worth of “personal experience” doesn't really deserve the “personal experience” qualifier.
Why not? It's just a minor example of what's possible, to show the general concept has already started.
> show the general concept has already started
The same way a todler creeping is the start of the general concept of space exploration.
Yes. And even so, it shows remarkable effectiveness already.
Like a car engine or a combine. The problem isn't the effectiveness of the tool for its purpose, it's the religion around it.
We seem to be talking past one another. All I was talking about was the facts of how these systems perform, without any reverence about it at all.
But to your point, I do see a lot of people very emotionally and psychologically committed to pointing out how deeply magical humans are, and how impossible we are to replicate in silicon. We have a religion about ourselves; we truly do have main character syndrome. It's why we mistakenly thought the earth was at the center of the universe for eons. But even with that disproved, our self-importance remains boundless.
> I do see a lot of people very emotionally and psychologically committed to pointing out how deeply magical humans are, and how impossible we are to replicate in silicon.
This a straw man, the question isn't if this is possible or not (this is an open question), it's about whether or not we are already here, and the answer is pretty straightforward: no we aren't. (And the current technology isn't going to bring us anywhere near that)
> but that's the rate of change of performance with respect to examples rather than what it learns by the time training is finished.
It's not just that. The problem of “deep learning” is that we use the word “learning” for something that really has no similarity with actual learning: it's not just that it converges way too slowly, it's also that it just seeks to minimize the predicted loss for every samples during training, but that's no how humans learn. If you feed it enough flat-earther content, as well a physics books, an LLM will happily tells you that the earth is flat, and explain you with lots of physics why it cannot be flat. It simply learned both “facts” during training and then spit it out during inference.
A human will learn one or the other first, and once the initial learning is made, it will disregards all the evidence of the contrary, until maybe at some point it doesn't and switches side entirely.
LLMs don't have an inner representation of the world and as such they don't have an opinion about the world.
The humans can't see the reality for itself, but they at least know it exists and they are constantly struggling to understand it. The LLM, by nature, is indifferent to the world.
> If you feed it enough flat-earther content, as well a physics books, an LLM will happily tells you that the earth is flat, and explain you with lots of physics why it cannot be flat.
This is a terrible example, because it's what humans do as well. See religious, or indeed military, indoctrination. All propaganda is as effective as it is, because the same message keeps getting hammered in.
And not just that, common misconceptions abound everywhere and not just conspiracy theories, religion, and politics. My dad absolutely insisted that the water draining in toilets or sinks are meaningfully influenced by the Coriolis effect, used an example of one time he went to the equator and saw a demonstration of this on both sides of the equator. University education and lifetime career in STEM, should have been able to figure out from first principles why the Coriolis effect is exactly zero on the equator itself, didn't.
> A human will learn one or the other first, and once the initial learning is made, it will disregards all the evidence of the contrary, until maybe at some point it doesn't and switches side entirely.
We don't have any way to know what a human would do if they could read the entire internet, because we don't live long enough to try.
The only bet I'd make is that we'd be more competent than any AI doing the same, because we learn faster from fewer examples, but that's about it.
> LLMs don't have an inner representation of the world and as such they don't have an opinion about the world.
There is evidence that they do have some inner representation of the world, e.g.:
• https://arxiv.org/abs/2506.02996
• https://arxiv.org/abs/2404.18202
> This is a terrible example, because it's what humans do as well. See religious, or indeed military, indoctrination. All propaganda is as effective as it is, because the same message keeps getting hammered in.
You completely misread my point.
The key thing with humans isn't that they cannot believe in bullshit. They can definitely do. But we don't usually believe in both the bullshit and in the fact the BS is actually BS. We have opinions on the BS. And we, as a species, routinely die or kill for these opinions, by the way. LLM don't care about anything.
> But we don't usually believe in both the bullshit and in the fact the the BS is actually BS.
I can't parse what you mean by this.
> LLM don't care about anything.
"Care" is ill-defined. LLMs are functions that have local optima (the outputs); those functions are trained to approximate other functions (e.g. RLHF) that optimise other things that can be described with functions (what humans care about). It's a game of telephone, like how Leonard Nimoy was approximating what the script writers were imagining Spock to be like when given the goal of "logical and unemotional alien" (ditto Brent Spiner, Data, "logical and unemotional android"), and yet humans are bad at writing such characters: https://tvtropes.org/pmwiki/pmwiki.php/Main/StrawVulcan
But rather more importantly in this discussion, I don't know what you care about when you're criticising AI for not caring, especially in this context. How, *mechanistically*, does "caring" matter to "learning abstract thought", and the question of how closely LLMs do or don't manage it relative to humans?
I mean, in a sense, I could see why someone might argue the exact opposite, that LLMs (as opposed to VLMs or anything embodied in a robot, or even pure-text agents trained on how tools act in response to the tokens emitted) *only* have abstract "thought", in so far as it's all book-learned knowledge.
>> But we don't usually believe in both the bullshit and in the fact the the BS is actually BS.
> I can't parse what you mean by this.
The point is that humans care about the state of a distributed shared world model and use language to perform partial updates to it according to their preferences about that state.
Humans who prefer one state (the earth is flat) do not -- as a rule -- use language to undermine it. Flat earthers don't tell you all the reasons the earth cannot be flat.
But even further than this, humans also have complex meta-preferences of the state, and their use of language reflects those too. Your example is relevant here:
> My dad absolutely insisted that the water draining in toilets or sinks are meaningfully influenced by the Coriolis effect [...]
> [...] should have been able to figure out from first principles why the Coriolis effect is exactly zero on the equator itself, didn't.
This is an exemplar of human behavior. Humans act like this. LLMs don't. If your dad did figure out from first principles and expressed it and continued insisting the position, I would suspect them of being an LLM, because that's how LLMs 'communicate'.
Now that the what is clear -- why? Humans experience social missteps like that as part of the loss surface. Being caught in a lie sucks, so people learn to not lie or be better at it. That and a million other tiny aspects of how humans use language in an overarching social context.
The loss surface that LLMs see doesn't have that feedback except in the long tail of doing Regularized General Document Corpora prediction perfectly. But it's so far away compared to just training on the social signal, where honesty is immediately available as a solution and is established very early in training instead of at the limit of low loss.
How humans learn (embedded in a social context from day one) is very effective at teaching foundational abilities fast. Natural selection cooked hard. LLM training recipes do not compare, they're just worse in so many different ways.
If you cant parse the sentence about BS i dont even know how they should respond to you. Its a very simple sentence.
> Everyone is out here acting like "predicting the next thing" is somehow fundamentally irrelevant to "human thinking" and it is simply not the case.
Nobody is. What people are doing is claiming that "predicting the next thing" does not define the entirety of human thinking, and something that is ONLY predicting the next thing is not, fundamentally, thinking.
Well, yes because thinking soon requires interacting, not just ideating. It's in the dialogue between ideation and interaction that we make our discoveries.
LLMs can interact with the world via e.g. function calling.
when LLM popped out and people started to say 'this is just markov chain on steroid and not thinking' i was a bit confused because a lot of my "thinking" is statistical too.. I very often try to solve an issue by switching a known solution with a different "probable" variant of it (tweaking a parameter)
LLMs have higher dimensions (they map token to grammatical and semantical space) .. it might not be thinking but it seems on its way we're just thinking with more abstractions before producing speech ?... dunno
I claim that all of thinking can be reduced to predicting the next thing. Predicting the next thing = thinking in the same way that reading and writing strings of bytes is a universal interface, or every computation can be done by a Turing machine.
People can claim whatever they like. That doesn't mean it's a good or reasonable hypothesis (especially for one that is essentially unfalsifible like predictive coding).
The problem is that we don’t have a good understanding of what “thinking” really is, and those parts of it we think we do understand involve simple things done at scale (electrical pulses on specific pathways, etc).
It is not unreasonable to suspect differences between humans and LLMs are differences in degree, rather than category.
I'm not trying to advance a testable hypothesis. If you think the unfalsifiability of my claim is a problem, you haven't understood what I'm trying to do.
My claim is that the two concepts are indistinguishable, thus equivalent. The unfalsifiability is what makes it a natural equivalence, the same as in the other examples I gave.
IMHO, you should. The opponent does not have an alternative definition of thinking that would have a prediction power matching the token prediction. Whatever they are thinking thinking is is a strictly worse scientific theory.
AI has made me question what it is to be a human.
I am not having some existential crisis, but if we get to a point where X% of humans cannot outperform “AI” on any task that humans deem “useful”, for some nontrivial value of X, then many assumptions that culture has inculcated into me about humanity are no longer valid.
What is the role of humans then?
Can it be said that humans “think” if they can’t think a thought that a non thinking AI cannot also think?
If all humans were suddenly wiped off the face of the earth, AI would go silent, and the hardware it runs on would eventually shut down.
If all AI was suddenly wiped off the face of the earth, humans would rebuild it, and would carry on fine in the meantime.
One AI researcher decades ago said something to the effect of: researchers in biology look at living organisms and wonder how they live; researchers in physics look at the cosmos and wonder what all is out there; researchers in artificial intelligence look at computer systems and wonder how they can be made to wonder such things.
The current AI SoTA.
But the "AI" is simply a gestalt of all human language. You're looking in a mirror.
A motorcycle is not "sprinting" and an LLM is not "thinking". Everyone would agree that a motorcycle is not running but the same dumb shit is posted over and over and over on here that somehow the LLM is "thinking".
But your assertion is merely semantic. It doesn't say anything substantive.
I could also say a motorcycle "moves forward" just like a person "moves forward". Whether we use the same or different words for same or different concepts doesn't say anything about the actual underlying similarity.
And please don't call stuff "dumb shit" here. Not appropriate for HN.
A forklift is "lifting" things, despite using a completely different mechanical process as a human "lifting" things. The only real similarity between these kinds of "lifting" is the end result, something is higher up than it was before.
is this seriously about continuous rotation versus a pair of double pendulums making a stepping motion?
That’s because the motorcycle thing is too simlistic of a comparison. It doesn’t come nearly close to capturing the nuance of the whole LLM “thinking” situation.
Predicting the next token is not at all the same thing as predicting the next action in a causal chain of actions. Not even close. One is model of language tokens, the other is a model of the physical world. You can come up with all sorts of predictions that can't be expressed cleanly in natural language. And plenty of things that parse cleanly from a language perspective but are unhinged in their description of empirical reality.
It may be doing the "thinking" and could reach AGI. But we don't want it. We don't want to take a fork lift to the gym. We don't want plastic aliens showing off their AGI and asking humanity to outsource human thinking and decision-making to them.
A good heuristic is that if an argument resorts to "actually not doing <something complex sounding>" or "just doing <something simple sounding>" etc, it is not a rigorous argument.
I'm an LLMs are being used in workflows they don't make sense in-sayer. And while yes, I can believe that LLMs can be part of a system that actually does think, I believe that to achieve true "thinking", it would likely be a system that is more deterministic in its approach rather than probabilistic.
Especially when modeling acting with intent. The ability to measure against past results and think of new innovative approaches seems like it may come from a system that may model first and then use LLM output. Basically something that has a foundation of tools rather than an LLM using MCP. Perhaps using LLMs to generate a response that humans like to read, but not in them coming up with the answer.
Either way, yes, its possible for a thinking system to use LLMs (and potentially humans piece together sentences in a similar way), but its also possible LLMs will be cast aside and a new approach will be used to create an AGI.
So for me: even if you are an AI-yeasayer, you can still believe that they won't be a component in an AGI.
You can make a separate model for the task, which is based on well chosen features and calibrated from actual data. Then the LLM only needs to generate the arguments to this model (extract those features from messages) and call it like a MCP tool. This external tool can be a simple Sklearn model.
Yes, personally I'm completely fine with the fact that LLMs don't actually think. I don't care that they're not AGI, though the hysterics about "AGI is so close now" seems silly to me. Fusion reactors and self-driving cars are just around the same corner.
They prove to have some useful utility to me regardless.
It's fascinating when you look at each technical component of cognition in human brains and contrast against LLMs. In humans, we have all sorts of parallel asynchronous processes running, with prediction of columnar activations seemingly the fundamental local function, with tens of thousands of mini columns and regions in the brain corresponding to millions of networked neurons using the "predict which column fires next" objective to increment or decrement the relative contribution of any functional unit.
In the case of LLMs you run into similarities, but they're much more monolithic networks, so the aggregate activations are going to scan across billions of neurons each pass. The sub-networks you can select each pass by looking at a threshold of activations resemble the diverse set of semantic clusters in bio brains - there's a convergent mechanism in how LLMs structure their model of the world and how brains model the world.
This shouldn't be surprising - transformer networks are designed to learn the complex representations of the underlying causes that bring about things like human generated text, audio, and video.
If you modeled a star with a large transformer model, you would end up with semantic structures and representations that correlate to complex dynamic systems within the star. If you model slug cellular growth, you'll get structure and semantics corresponding to slug DNA. Transformers aren't the end-all solution - the paradigm is missing a level of abstraction that fully generalizes across all domains, but it's a really good way to elicit complex functions from sophisticated systems, and by contrasting the way in which those models fail against the way natural systems operate, we'll find better, more general methods and architectures, until we cross the threshold of fully general algorithms.
Biological brains are a computational substrate - we exist as brains in bone vats, connected to a wonderfully complex and sophisticated sensor suite and mobility platform that feeds electrically activated sensory streams into our brains, which get processed into a synthetic construct we experience as reality.
Part of the underlying basic functioning of our brains is each individual column performing the task of predicting which of any of the columns it's connected to will fire next. The better a column is at predicting, the better the brain gets at understanding the world, and biological brains are recursively granular across arbitrary degrees of abstraction.
LLMs aren't inherently incapable of fully emulating human cognition, but the differences they exhibit are expensive. It's going to be far more efficient to modify the architecture, and this may diverge enough that whatever the solution ends up being, it won't reasonably be called an LLM. Or it might not, and there's some clever tweak to things that will push LLMs over the threshold.
When you have a thought, are you "predicting the next thing"—can you confidently classify all mental activity that you experience as "predicting the next thing"?
Language and society constrains the way we use words, but when you speak, are you "predicting"? Science allows human beings to predict various outcomes with varying degrees of success, but much of our experience of the world does not entail predicting things.
How confident are you that the abstractions "search" and "thinking" as applied to the neurological biological machine called the human brain, nervous system, and sensorium and the machine called an LLM are really equatable? On what do you base your confidence in their equivalence?
Does an equivalence of observable behavior imply an ontological equivalence? How does Heisenberg's famous principle complicate this when we consider the role observer's play in founding their own observations? How much of your confidence is based on biased notions rather than direct evidence?
The critics are right to raise these arguments. Companies with a tremendous amount of power are claiming these tools do more than they are actually capable of and they actively mislead consumers in this manner.
> When you have a thought, are you "predicting the next thing"
Yes. This is the core claim of the Free Energy Principle[0], from the most-cited neuroscientist alive. Predictive processing isn't AI hype - it's the dominant theoretical framework in computational neuroscience for ~15 years now.
> much of our experience of the world does not entail predicting things
Introspection isn't evidence about computational architecture. You don't experience your V1 doing edge detection either.
> How confident are you that the abstractions "search" and "thinking"... are really equatable?
This isn't about confidence, it's about whether you're engaging with the actual literature. Active inference[1] argues cognition IS prediction and action in service of minimizing surprise. Disagree if you want, but you're disagreeing with Friston, not OpenAI marketing.
> How does Heisenberg's famous principle complicate this
It doesn't. Quantum uncertainty at subatomic scales has no demonstrated relevance to cognitive architecture. This is vibes.
> Companies... are claiming these tools do more than they are actually capable of
Possibly true! But "is cognition fundamentally predictive" is a question about brains, not LLMs. You've accidentally dismissed mainstream neuroscience while trying to critique AI hype.
[0] https://www.nature.com/articles/nrn2787
[1] https://mitpress.mit.edu/9780262045353/active-inference/
How does the free energy principle align with system dynamics and the concept of emergence? Yes, our brain might want to optimize for lack of surprise, but that does not mean it can fully avoid emergent or chaotic behavior stemming from the incredibly complex dynamics of the linked neurons?
FEP doesn't conflict with complex dynamics, it's a mathematical framework for explaining how self-organizing behavior arises from simpler variational principles. That's what makes it a theory rather than a label.
The thing you're doing here has a name: using "emergence" as a semantic stopsign. "The system is complex, therefore emergence, therefore we can't really say" feels like it's adding something, but try removing the word and see if the sentence loses information.
"Neurons are complex and might exhibit chaotic behavior" - okay, and? What next? That's the phenomenon to be explained, not an explanation.
This was articulated pretty well 18 years ago [0].
[0]: https://www.lesswrong.com/posts/8QzZKw9WHRxjR4948/the-futili...
This essay completely misunderstands how the notion of emergence gained prominence and how people tend to actually use it. It's a straw man that itself devolves into a circular argument "embrace a reductionist epistemology because you should embrace a reductionist epistemology".
It doesn't even meaningfully engage with the historical literature that established the term, etc. If you want to actually understand why the term gained prominence, check out the work of Edgar Morin.
Oh, I was looking for something like that! Saved to zotero. Thank you!
Thanks for the links! I'll have to dig into this more for sure. Looking at the bulleted summary, I'm not sure your argument is sufficiently nuanced or being made in good faith.
The article argues that the brain "predicts" acts of perception in order to minimize surprise. First of all, very few people mean to talk about these unconscious operations of the brain when they claim they are "thinking". Most people have not read enough neuroscience literature to have such a definition. Instead, they tend to mean "self-conscious activity" when they say "thinking". Thinking, the way the term is used in the vernacular, usually implies some amount of self-reflexivity. This is why we have the term "intuition" as opposed to thinking after all. From a neuronal perspective, intuition is still thinking, but most people don't think (ha) of the word thinking to encompass this, and companies know that.
It is clear to me, as it is to everyone one the planet, that when OpenAI for example claims that ChatGPT "thinks" they want consumers to make the leap to cognitive equivalence at the level of self-conscious thought, abstract logical reasoning, long-term learning, and autonomy. These machines are designed such that they do not even learn and retain/embed new information past their training date. That already disqualifies them from strong equivalence to human beings, who are able to rework their own tendencies toward prediction in a meta cognitive fashion by incorporating new information.
> can you confidently classify all mental activity that you experience as "predicting the next thing"? [...] On what do you base your confidence in their equivalence?
To my understanding, bloaf's claim was only that the ability to predict seems a requirement of acting intentionally and thus that LLMs may "end up being a component in a system which actually does think" - not necessarily that all thought is prediction or that an LLM would be the entire system.
I'd personally go further and claim that correctly generating the next token is already a sufficiently general task to embed pretty much any intellectual capability. To complete `2360 + 8352 * 4 = ` for unseen problems is to be capable of arithmetic, for instance.
Boo LLM-generated comments!
But what [if the llms generate] constructive and helpful comments?
https://xkcd.com/810/
> When you have a thought, are you "predicting the next thing"—can you confidently classify all mental activity that you experience as "predicting the next thing"?
So notice that my original claim was "prediction is fundamental to our ability to act with intent" and now your demand is to prove that "prediction is fundamental to all mental activity."
That's a subtle but dishonest rhetorical shift to make me have to defend a much broader claim, which I have no desire to do.
> Language and society constrains the way we use words, but when you speak, are you "predicting"?
Yes, and necessarily so. One of the main objections that dualists use to argue that our mental processes must be immaterial is this [0]:
* If our mental processes are physical, then there cannot be an ultimate metaphysical truth-of-the-matter about the meaning of those processes.
* If there is no ultimate metaphysical truth-of-the-matter about what those processes mean, then everything they do and produce are similarly devoid of meaning.
* Asserting a non-dualist mind therefore implies your words are meaningless, a self-defeating assertion.
The simple answer to this dualist argument is precisely captured by this concept of prediction. There is no need to assert some kind of underlying magical meaning to be able to communicate. Instead, we need only say that in the relevant circumstances, our minds are capable of predicting what impact words will have on the receiver and choosing them accordingly. Since we humans don't have access to each other's minds, we must not learn these impacts from some kind of psychic mind-to-mind sense, but simply from observing the impacts of the words we choose on other parties; something that LLMs are currently (at least somewhat) capable of observing.
[0] https://www.newdualism.org/papers/E.Feser/Feser-acpq_2013.pd...
If you read the above link you will see that they spell out 3 problems with our understanding of thought:
Consciousness, intentionality, and rationality.
Of these, I believe prediction is only necessary for intentionality, but it does have some roles to play in consciousness and rationality.
The issue is that prediction is "part" of the human thought process, it's not the full story...
And the big players have built a bunch of workflows which embed many other elements besides just "predictions" into their AI product. Things like web search, to incorporating feedback from code testing, to feeding outputs back into future iterations. Who is to say that one or more of these additions has pushed the ensemble across the threshold and into "real actual thinking."
The near-religious fervor which people insist that "its just prediction" makes me want to respond with some religious allusions of my own:
> Who is this that wrappeth up sentences in unskillful words? Gird up thy loins like a man: I will ask thee, and answer thou me. Where wast thou when I laid up the foundations of the earth? tell me if thou hast understanding. Who hath laid the measures thereof, if thou knowest? or who hath stretched the line upon it?
The point is that (as far as I know) we simply don't know the necessary or sufficient conditions for "thinking" in the first place, let alone "human thinking." Eventually we will most likely arrive at a scientific consensus, but as of right now we don't have the terms nailed down well enough to claim the kind of certainty I see from AI-detractors.
I take a offence in the idea I’m “religiously downplaying LLMs”. I pay top dollar for access to the best models because I want the capabilities to be good / better. Just because I’m documenting my experience it doesn’t mean I have an Anti-ai agenda ? I pay because I find LLMs to be useful. Just not in the way suggested by the marketing teams.
I’m downplaying because I have honestly been burned by these tools when I’ve put trust in their ability to understand anything, provide a novel suggestion or even solve some basic bugs without causing other issues.?
I use all of the things you talk about extremely frequently and again, there is no “thinking” or consideration on display that suggests these things work like us, else why would we be having this conversation if they were ?
> I’m downplaying because I have honestly been burned by these tools when I’ve put trust in their ability to understand anything, provide a novel suggestion or even solve some basic bugs without causing other issues.?
I've had that experience plenty of times with actual people... LLMs don't "think" like people do, that much is pretty obvious. But I'm not at all sure whether what they do can be called "thinking" or not.
"LLMs don't reply to my queries perfectly, therefore they don't think"?
I completely agree that we don't know enough, but I suggest that that entails that the critics and those who want to be cautious are correct.
The harms engendered by underestimating LLM capabilities are largely that people won't use the LLMs.
The harms engendered by overestimating their capabilities can be as severe as psychological delusion, of which we have an increasing number of cases.
Given we don't actually have a good definition of "thinking" what tack do you consider more responsible?
> The harms engendered by underestimating LLM capabilities are largely that people won't use the LLMs.
Speculative fiction about superintelligences aside, an obvious harm to underestimating the LLM's capabilities is that we could effectively be enslaving moral agents if we fail to correctly classify them as such.
If the models were conscious, intelligent, suffering and could think, why wouldn't they tell us ?
> can be as severe as psychological delusion
Much worse, when insufficiently skeptical humans link the LLM to real-world decisions to make their own lives easier.
Consider the Brazil-movie-esque bureaucratic violence of someone using it to recommend fines or sentencing.
https://www.nature.com/articles/s41586-024-07856-5
> The issue is that prediction is "part" of the human thought process, it's not the full story...
Do you have a proof for this?
Surely such a profound claim about human thought process must have a solid proof somewhere? Otherwise who's to say all of human thought process is not just a derivative of "predicting the next thing"?
Use your brain and use an LLM for 6 months, you’ll work it out.
> Use your brain and use an LLM for 6 months, you’ll work it out.
That's not a proof. Think harder about the questions people are asking you here.
Let's go the other route.
What would change your mind? It's an exercise in feasibility.
For example, I don't believe in time travel. If someone made me time travel, and made it undeniable that I was transported back to 1508, then I would not be able to argue against it. In fact, no one in such position would.
What is that equivalent for your conviction? There must be something, otherwise, it's just an opinion that can't be changed.
You don't need to present some actual proof or something. Just lay out some ideas that demonstrate that you are being rational about this and not just sucking up to LLM marketing.
most humans in any percentile act towards the thing of someone else. most of these things are a lot worse than what the human "would originally intend". this behavior stems from 100s and thousands of nudges since childhood.
the issue with AI and AI-naysayers is, by analogy, this: cars were build to drive from A to Z. people picked up tastes and some people started building really cool looking cars. the same happens on the engineering side. then portfolio communists came with their fake capitalism and now cars are build to drive over people but don't really work because people, thankfully, are overwhelming still fighting to attempt to act towards their own intents.
Exactly. Our base learning is by example, which is very much learning to predict.
Predict the right words, predict the answer, predict when the ball bounces, etc. Then reversing predictions that we have learned. I.e. choosing the action with the highest prediction of the outcome we want. Whether that is one step, or a series of predicted best steps.
Also, people confuse different levels of algorithm.
There are at least 4 levels of algorithm:
• 1 - The architecture.
This input-output calculation for pre-trained models are very well understood. We put together a model consisting of matrix/tensor operations and few other simple functions, and that is the model. Just a normal but high parameter calculation.
• 2 - The training algorithm.
These are completely understood.
There are certainly lots of questions about what is most efficient, alternatives, etc. But training algorithms harnessing gradients and similar feedback are very clearly defined.
• 3 - The type of problem a model is trained on.
Many basic problem forms are well understood. For instance, for prediction we have an ordered series of information, with later information to be predicted from earlier information. It could simply be an input and response that is learned. Or a long series of information.
• 4 - The solution learned to solve (3) the outer problem, using (2) the training algorithm on (1) the model architecture.
People keep confusing (4) with (1), (2) or (3). But it is very different.
For starters, in the general case, and for most any challenging problem, we never understand their solution. Someday it might be routine, but today we don't even know how to approach that for any significant problem.
Secondly, even with (1), (2), and (3) exactly the same, (4) is going to be wildly different based on the data characterizing the specific problem to solve. For complex problems, like language, layers and layers of sub-solutions to sub-problems have to be solved, and since models are not infinite in size, ways to repurpose sub-solutions, and weave together sub-solutions to address all the ways different sub-problems do and don't share commonalities.
Yes, prediction is the outer form of their solution. But to do that they have to learn all the relationships in the data. And there is no limit to how complex relationships in data can be. So there is no limit on the depths or complexity of the solutions found by successfully trained models.
Any argument they don't reason, based on the fact that they are being trained to predict, confuses at least (3) and (4). That is a category error.
It is true, they reason a lot more like our "fast thinking", intuitive responses, than our careful deep and reflective reasoning. And they are missing important functions, like a sense of what they know or don't. They don't continuously learn while inferencing. Or experience meta-learning, where they improve on their own reasoning abilities with reflection, like we do. And notoriously, by design, they don't "see" the letters that spell words in any normal sense. They see tokens.
Those reasoning limitations can be irritating or humorous. Like when a model seems to clearly recognize a failure you point out, but then replicates the same error over and over. No ability to learn on the spot. But they do reason.
Today, despite many successful models, nobody understands how models are able to reason like they do. There is shallow analysis. The weights are there to experiment with. But nobody can walk away from the model and training process, and build a language model directly themselves. We have no idea how to independently replicate what they have learned, despite having their solution right in front of us. Other than going through the whole process of retraining another one.
LLMs merely interpolate between the feeble artifacts of thought we call language.
The illusion wears off after about half an hour for even the most casual users. That's better than the old chatbots, but they're still chatbots.
Did anyone ever seriously buy the whole "it's thinking" BS when it was Markov chains? What makes you believe today's LLMs are meaningfully different?
Did anyone ever seriously buy the whole "it's transporting" BS when it was wheelbarrows? What makes you believe today's trucks are meaningfully different?
This is the "but LLMs will get better, trust me" thread?
I suspect that people instinctively believe they have free will, both because it feels like we do, and because society requires us to behave that way even when we don't.
The truth is that the evidence says we don't. See the Libet experiment and its many replications.
Your decisions can be predicted from brain scans up to 10 seconds before you make them, which means they are as deterministic as an LLM's. Sorry, I guess.
> Your decisions can be predicted from brain scans up to 10 seconds before you make them, which means they are as deterministic as an LLM's.
This conclusion does not follow from the result at all.
I looked up the Libet experiment:
"Implications
The experiment raised significant questions about free will and determinism. While it suggested that unconscious brain activity precedes conscious decision-making, Libet argued that this does not negate free will, as individuals can still choose to suppress actions initiated by unconscious processes."
It's been repeated a huge number of time since, and widely debated. When Libet first did the experiment it was only like 200ms before the mind become consciously aware of the decision. More recent studies have shown they can predict actions up to 7-10 seconds before the subject is aware of having made a decision.
It's pretty hard to argue that you're really "free" to make a different decision if your body knew which you would choose 7 seconds before you became aware of it.
I mean, those long term predictions were only something like 60% accurate, but still, the preponderance of evidence says that those decisions are deterministic and we keep finding new ways to predict the outcome sooner and with higher accuracy.
https://pubmed.ncbi.nlm.nih.gov/18408715/
"I conducted an experiment where I instructed experienced drivers to follow a path in a parking lot laid out with traffic cones, and found that we were able to predict the trajectory of the car with greater than 60% accuracy. Therefore drivers do not have free will to just dodge the cones and drive arbitrarily from the start to the finish."
Clearly, that conclusion would be patently absurd to draw from that experiment. There are so many expectation and observation effects that go into the very setup from the beginning. Humans generally follow directions, particularly when a guy in a labcoat is giving them.
> At some point, when they felt the urge to do so, they were to freely decide between one of two buttons, operated by the left and right index fingers, and press it immediately. [0]
Wow. TWO whole choices to choose from! Human minds tend to pre-think their choice between one of two fingers to wiggle, therefore free will doesn't exist.
> It's pretty hard to argue that you're really "free" to make a different decision if your body knew which you would choose 7 seconds before you became aware of it.
To really spell it out since the analogy/satire may be lost: You're free to refrain from pressing either button during the prompt. You're free to press both buttons at the same time. You're free to mash them rapidly and randomly throughout the whole experiment. You're free to walk into the fMRI room with a bag full of steel BB's and cause days of downtime and thousands of dollars in damage. Folks generally don't do those things because of conditioning.
[0] - http://behavioralhealth2000.com/wp-content/uploads/2017/10/U...
If that were the only evidence I might agree that alternative explanations are as likely, but I cited only one of many studies that show similar outcomes. There are loads of other studies done with entirely different methodologies that indicate most human introspection is mostly better described as post hoc confabulism. That is to say, we don't use reason to make decisions so much as we make decisions, and then justify them with reasons. Nisbet and Wilson were showing it experimentally as far back as the late 70's. [0] It's been confirmed in different forms hundreds of times since.
Certainly we can come up with some alternative theories (like "free will") to explain it all away, but the simplest (therefore most likely correct) answer is just that we're basically statistical state machines and are as deterministic as a similar computational system.
To be clear, I'm not saying that metacognition doesn't exist. Just that I've never seen any reason to believe it's very different from current thinking models that just feed an output back in as another input.
[0] - https://home.csulb.edu/~cwallis/382/readings/482/nisbett%20s...
What is the import of this to you here? Whether you have free will or you feel like you do, kinda same difference for this particular point right? It doesn't make me more human actually having free will, it is sufficient to simply walk around as if I do.
But beyond that, what do you want to say here? What is lost, what is gained? Are you wanting to say this makes us more like an LLM? How so?
Libet has only measured the latency of metaconsciousness/cognition, nothing else. It says nothing about free will, which is ill defined anyway.
I've seen that argument repeated ad-nauseum.
It makes sense if you're desperate for free will to be real, but you really have to work for it. Especially when you add in the countless other studies showing that a lot of the reasons we give for our actions, especially in quick or ambiguous choices, are confabulationalist post-hoc constructions. Our own introspection seems mostly to consist of just "making stuff up" to justify the decisions we've already made.
I mean, a reasonable person could argue their way past all the evidence without totally denying it, but "free will" just isn't the simplest explanation that fits the available data. It's possible that free will exists in the same way it's possible that Russels teapot exists.
Every day I see people treat gen AI like a thinking human, Dijkstra's attitudes about anthropomorphizing computers is vindicated even more.
That said, I think the author's use of "bag of words" here is a mistake. Not only does it have a real meaning in a similar area as LLMs, but I don't think the metaphor explains anything. Gen AI tricks laypeople into treating its token inferences as "thinking" because it is trained to replicate the semiotic appearance of doing so. A "bag of words" doesn't sufficiently explain this behavior.
One metaphor is to call the model a person, another metaphor is to call it a pile of words. These are quite opposite. I think that's the whole point.
Person-metaphor does nothing to explain its behavior, either.
"Bag of words" has a deep origin in English, the Anglo-Saxon kenning "word-hord", as when Beowulf addresses the Danish sea-scout (line 258)
"He unlocked his word-hoard and delivered this answer."
So, bag of words, word-treasury, was already a metaphor for what makes a person a clever speaker.
Word-hoard is a very good phrase.
It's an everyday word in German: Wortschatz, meaning (someone's) active vocabulary.
I'll make the following observation:
The contra-positive of "All LLMs are not thinking like humans" is "No humans are thinking like LLMs"
And I do not believe we actually understand human thinking well enough to make that assertion.
Indeed, it is my deep suspicion that we will eventually achieve AGI not by totally abandoning today's LLMs for some other paradigm, but rather embedding them in a loop with the right persistence mechanisms.
Given that LLMs are incapable of synthetic a priori knowledge and humans are, I would say that as the tech stands currently, it's reasonable to make both of those statements.
The loop, or more precisely the "search" does the novel part in thinking, the brain is just optimizing this process. Evolution could manage with the simplest model - copying with occasional errors, and in one run it made everyone of us. The moral - if you scale search the model can be dumb.
Let’s not underestimate the scale of the search which led to us though, even though you may be right in principle. In addition to deep time on earth, we may well be just part of a tiny fraction of a universe-wide and mostly fruitless search.
For me, the problem is in the "chat" mechanic that OpenAI and others use to present the product. It lends itself to strong antropomorphizing.
If instead of a chat interface we simply had a "complete the phrase" interface, people would understand the tool better for what it is.
But people aren't using ChatGPT for completing phrases. They're using it to get their tasks done, or get their questions answered.
The fact that pretraining of ChatGPT is done with a "completing the phrase" task has no bearing on how people actually end up using it.
It's not just the pretraining, it's the entire scaffolding between the user and the LLM itself that contributes to the illusion. How many people would continue assuming that these chatbots were conscious or intelligent if they had to build their own context manager, memory manager, system prompt, personality prompt, and interface?
I agree 100%. Most people haven't actually interacted directly with an LLM before. Most people's experience with LLMs is ChatGPT, Claude, Grok, or any of the other tools that automatically handle context, memory, personality, temperature, and are deliberately engineered to have the tool communicate like a human. There is a ton of very deterministic programming that happens between you and the LLM itself to create this experience, and much of the time when people are talking about the ineffable intelligence of chatbots, it's because of the illusion created by this scaffolding.
Yea bag of words isn’t helpful at all. I really do think that “superpowered sentence completion” is the best description. Not only is it reasonably accurate it is understandable, everyone has seen autocomplete function, and it’s useful. I don’t know how to “use” a bag of words. I do know how to use sentence completion. It also helps explains why context matters.
Sentence completion does not give it justice, when I can ask a LLM to refactor my repo and come back half an hour later to see the deed done.
Thats the thing, when you use an Ask/answer mechanism, you are just writing a "novel" where User: asks and personal coding assistant: answers. But all the text goes into the autocomplete function and the "toaster" outputs the most probable text according to the function.
Its useful, it's amazing, but as the original text says, thinking of it as "some intelligence with reasoning " makes us use the wrong mental models for it.
I've been recently using a similar description, referring to "AI" (LLMs) as "glorified autocomplete" or "luxury autocomplete".
I think I first heard "spicy autocomplete" two or three years ago...
well they are trained to be almost in distribution as a thinking human. So...
Bag of words is actually the perfect metaphor. The data structure is a bag. The output is a word. The selection strategy is opaquely undefined.
> Gen AI tricks laypeople into treating its token inferences as "thinking" because it is trained to replicate the semiotic appearance of doing so. A "bag of words" doesn't sufficiently explain this behavior.
Something about there being significant overlap between the smartest bears and the dumbest humans. Sorry you[0] were fooled by the magic bag.
[0] in the "not you, the layperson in question" sense
I think it's still a bit of a tortured metaphor. LLMs operate on tokens, not words. And to describe their behavior as pulling the right word out of a bag is so vague that it applies every bit as much to a Naive Bayes model written in Python in 10 minutes as it does to the greatest state of the art LLM.
Yeah. I have a half-cynical/half-serious pet theory that a decent fraction of humanity has a broken theory of mind and thinks everyone has the same thought patterns they do. If it talks like me, it thinks like me.
Whenever the comment section takes a long hit and goes "but what is thinking, really" I get slightly more cynical about it lol
Why not?
By now, it's pretty clear that LLMs implement abstract thinking - as do humans.
They don't think exactly like humans do - but they sure copy a lot of human thinking, and end up closer to it than just about anything that's not a human.
It isn't clear because they do none of that lol. They don't think.
It can kinda sorta look like thinking if you don't have a critical eye, but it really doesn't take much to break the illusion.
I really don't get this obsessive need to pretend your tools are alive. Y'all know when you watch YouTube that it's a trick and the tiny people on your screen don't live in your computer, right?
And how do you know that exactly? What is the source of that certainty? What makes you fully confident that a system that can write short stories and one-shot Python scripts and catch obscure pop culture references in text isn't "thinking" in any way?
The answer to that is the siren song of "AI effect".
Even admitting "we don't know" requires letting go of the idea that "thinking" must be exclusive to humans. And many are far too weak to do that.
lol magic bag.
Spoken Query Language? Just like SQL, but for unstructured blobs of text as a database and unstructured language as a query? Also known as Slop Query Language or just Slop Machine for its unpredictable results.
> Spoken Query Language? Just like SQL, but for unstructured blobs of text as a database and unstructured language as a query?
I feel that's more a description of a search engine. Doesn't really give an intuition of why LLMs can do the things they do (beyond retrieval), or where/why they'll fail.
If you want actionable intuition, try "a human with almost zero self-awareness".
"Self-awareness" used in a purely mechanical sense here: having actionable information about itself and its own capabilities.
If you ask an old LLM whether it's able to count the Rs in "strawberry" successfully, it'll say "yes". And then you ask it to do so, and it'll say "2 Rs". It doesn't have the self-awareness to know the practical limits of its knowledge and capabilities. If it did, it would be able to work around the tokenizer and count the Rs successfully.
That's a major pattern in LLM behavior. They have a lot of capabilities and knowledge, but not nearly enough knowledge of how reliable those capabilities are, or meta-knowledge that tells them where the limits of their knowledge lie. So, unreliable reasoning, hallucinations and more.
Agree that's a better intuition, with pretraining pushing the model towards saying "I don't know" in the kinds of situations where people write that as opposed to by introspection of its own confidence.
There appears to be a degree of "introspection of its own confidence" in modern LLMs. They can identify their own hallucinations, at a rate significantly better than chance. So there must be some sort of "do I recall this?" mechanism built into them. Even if it's not exactly a reliable mechanism.
Anthropic has discovered that this is definitely the case for name recognition, and I suspect that names aren't the only things subject to a process like that.
Slightly unfortunate that "Bag of words" is already a different concept: https://en.wikipedia.org/wiki/Bag_of_words.
My second thought is that it's not the metaphor that is misleading. People have been told thousands of times that LLMs don't "think", don't "know", don't "feel", but are "just a very impressive autocomplete". If they still really want to completely ignore that, why would they suddenly change their mind with a new metaphor?
Humans are lazy. If it looks true enough and it cost less effort, humans will love it. "Are you sure the LLM did your job correctly?" is completely irrelevant: people couldn't care less if it's correct or not. As long as the employer believes that the employee is "doing their job", that's good enough. So the question is really: "do you think you'll get fired if you use this?". If the answer is "no, actually I may even look more productive to my employer", then why would people not use it?
> Slightly unfortunate that "Bag of words" is already a different concept
Yes, subconsciously I kept trying to map this article's ideas to word2vec and continuous-bag-of-words.
> If we allow ourselves to be seduced by the superficial similarity, we’ll end up like the moths who evolved to navigate by the light of the moon, only to find themselves drawn to—and ultimately electrocuted by—the mysterious glow of a bug zapper.
Woah, that hit hard
As usual with these, it helps to try to keep the metaphor used for downplaying AI, but flip the script. Let's grant the author's perception that AI is a "bag of words", which is already damn good at producing the "right words" for any given situation, and only keeps getting better at it.
Sure, this is not the same as being a human. Does that really mean, as the author seems to believe without argument, that humans need not be afraid that it will usurp their role? In how many contexts is the utility of having a human, if you squint, not just that a human has so far been the best way to "produce the right words in any given situation", that is, to use the meat-bag only in its capacity as a word-bag? In how many more contexts would a really good magic bag of words be better than a human, if it existed, even if the current human is used somewhat differently? The author seems to rest assured that a human (long-distance?) lover will not be replaced by a "bag of words"; why, especially once the bag of words is also ducttaped to a bag of pictures and a bag of sounds?
I can just imagine someone - a horse breeder, or an anthropomorphised horse - dismissing all concerns on the eve of the automotive revolution, talking about how marketers and gullible marks are prone to hippomorphising anything that looks like it can be ridden and some more, and sprinkling some anecdotes about kids riding broomsticks, legends of pegasi and patterns of stars in the sky being interpreted as horses since ancient times.
I don't think the author's argument is that it won't replace any human labour. Or at least I wouldn't agree with such an argument. But the stronger case is that however much LLMs improve, they won't replace humans in general. In the furtherment of knowledge, because they are fundamentally parroting and synthesizing the already known, vs performing truly novel thought. And in creative fields, because people are fundamentally interested in creations of other people, not of computers.
Neither of these is entirely true in all cases, but they could be expected to remain true in at least some (many) cases, and so the role for humans remains.
So a human is just a really expensive, unreliable bag of words. And we get more expensive and more unreliable by the day!
There's a quote I love but have misplaced, from the 19th century I think. "Our bodies are just contraptions for carrying our heads around." Or in this instance... bag of words transport system ;)
I think the canonical answer is that humans are “bags of mostly water” .
If I'm remembering the full quote correctly, it's "Ugly bags of mostly water."
https://en.wikipedia.org/wiki/The_Meme_Machine
I just came from the Pluribus sub-Reddit. I’ll take AI over that cohort any day.
So tell me, why do I still have a job and why am frequently successful in getting profitable / useful products into production if I’m “expensive and unreliable”?
I mean I use AI tools to help achieve the goal but I don’t see any signs of the things I’m building and doing being unreliable.
Her argument really only works if you institute new economic systems where humans don’t need to labor in order to eat or pay rent.
"Her"->"the"? (Or, who is "she" here?)
Either way, in what way is this relevant? If the human's labor is not useful at any price point to any entity with money, food or housing, then they presumably will not get paid/given food/housing for it.
Why are you repeating what I said with slightly different words?
Maybe because I didn't understand what you said. Who does "her" refer to?
I am unsure myself whether we should regard LLMs as mere token-predicting automatons or as some new kind of incipient intelligence. Despite their origins as statistical parrots, the interpretability research from Anthropic [1] suggests that structures corresponding to meaning do exist inside those bundles of numbers and that there are signs of activity within those bundles of numbers that seem analogous to thought.
That said, I was struck by a recent interview with Anthropic’s Amanda Askell [2]. When she talks, she anthropomorphizes LLMs constantly. A few examples:
“I don't have all the answers of how should models feel about past model deprecation, about their own identity, but I do want to try and help models figure that out and then to at least know that we care about it and are thinking about it.”
“If you go into the depths of the model and you find some deep-seated insecurity, then that's really valuable.”
“... that could lead to models almost feeling afraid that they're gonna do the wrong thing or are very self-critical or feeling like humans are going to behave negatively towards them.”
[1] https://www.anthropic.com/research/team/interpretability
[2] https://youtu.be/I9aGC6Ui3eE
Amanda Askell studied under David Chalmers at NYU: the philosopher who coined "the hard problem of consciousness" and is famous for taking phenomenal experience seriously rather than explaining it away. That context makes her choice to speak this way more striking: this isn't naive anthropomorphizing from someone unfamiliar with the debates. It's someone trained by one of the most rigorous philosophers of consciousness, who knows all the arguments for dismissing mental states in non-biological systems, and is still choosing to speak carefully about models potentially having something like feelings or insecurities.
A person can study fashion extensively, under the best designers, they can understand tailoring and fit and have a phenomenal eye for color and texture.
Their vivid descriptions of what the Emperor could be wearing doesn't make said emperor any less nakey.
>research from Anthropic [1] suggests that structures corresponding to meaning exist inside those bundles of numbers and that there are signs of activity within those bundles of numbers that seem analogous to thought.
Can you give some concrete examples? The link you provided is kind of opaque
>Amanda Askell [2]. When she talks, she anthropomorphizes LLMs constantly.
She is a philosopher by trade and she describes her job (model alignment) as literally to ensure models "have good character traits." I imagine that explains a lot
Here are three of the Anthropic research reports I had in mind:
https://www.anthropic.com/news/golden-gate-claude
Excerpt: “We found that there’s a specific combination of neurons in Claude’s neural network that activates when it encounters a mention (or a picture) of this most famous San Francisco landmark.”
https://www.anthropic.com/research/tracing-thoughts-language...
Excerpt: “Recent research on smaller models has shown hints of shared grammatical mechanisms across languages. We investigate this by asking Claude for the ‘opposite of small’ across different languages, and find that the same core features for the concepts of smallness and oppositeness activate, and trigger a concept of largeness, which gets translated out into the language of the question.”
https://www.anthropic.com/research/introspection
Excerpt: “Our new research provides evidence for some degree of introspective awareness in our current Claude models, as well as a degree of control over their own internal states.”
It’s important to note that these “research papers” that Anthropic releases are not properly peer-reviewed and not accepted by any scientific journal or institution. Anthropic has a history of over-exaggerating research, and have an obvious monetary incentive to continue to do so.
Well, she's describing the system's behavior.
My fridge happily reads inputs without consciousness, has goals and takes decisions without "thinking", and consistently takes action to achieve those goals. (And it's not even a smart fridge! It's the one with a copper coil or whatever.)
I guess the cybernetic language might be less triggering here (talking about systems and measurements and control) but it's basically the same underlying principles. One is just "human flavored" and I therefore more prone to invite unhelpful lines of thinking?
Except that the "fridge" in this case is specifically and explicitly designed to emulate human behavior so... you would indeed expect to find structures corresponding to the patterns it's been designed to simulate.
Wondering if it's internalized any other human-like tendencies — having been explicitly trained to simulate the mechanisms that produced all human text — doesn't seem too unreasonable to me.
> the interpretability research from Anthropic [1] suggests that structures corresponding to meaning do exist inside those bundles of numbers and that there are signs of activity within those bundles of numbers that seem analogous to thought
I did a simple experiment - took a photo of my kid in the park, showed it to Gemini and asked for a "detailed description". Then I took that description and put it into a generative model (Z-Image-Turbo, a new one). The output image was almost identical.
So one model converted image to text, the other reversed the processs. The photo was completely new, personal, never put online. So it was not in any training set. How did these 2 models do it if not actually using language like a thinking agent?
https://pbs.twimg.com/media/G7gTuf8WkAAGxRr?format=jpg&name=...
> How did these 2 models do it if not actually using language like a thinking agent?
By having a gazillion of other, almost identical pictures of kids in parks in their training data.
Not pictures with this composition, same jacket, etc - yes, there are images but they are different, this fits like a key in the lock to the original
If you think that’s the same jacket, you need to get your eyes checked. The jacket pattern changed from multicolored to full-on camo.
Wow those quotes are extremely disturbing.
This argument would have a lot more weight if it was published in a peer reviewed journal by a party that does not have a stake in the AI market.
the anthropomorphization (say that 3 times quickly) is kinda weird, but also makes for a much more pleasant conversation imo. it's kinda tedious being pedantic all the time.
It obfuscates far more than it clarifies.
It also leads to fundamentally wrong conclusions: a related issue I have with this is the use of anthropomorphic shorthand when discussing international politics. You've heard a phrase like "the US thinks...", "China wants...", "Europe believes..." so much you don't even notice it.
All useful shorthands, all which lead to people displaying fundamental misunderstandings of what they're talking about - i.e. expressing surprise that a nation of millions doesn't display consistency of behavior of human lifetime scales, even though fairly obviously the mechanisms of government are churning their make up constantly, and depending on context maybe entirely different people.
It seems obvious to me that entities have emergent needs and plans and so on, independent of any of the humans inside.
For example, if you've worked at a large company, one of the little tragedies is when someone everyone likes gets laid off. There were probably no people who actively wanted Bob to lose his job. Even the CEO/Board who pulled the trigger probably had nothing against Bob. Heck, they might be the next ones out the door. The company is faceless, yet it wanted Bob to go, because that apparently contributed to the company's objective function. Had the company consisted entirely of different people, plus Bob, Bob might have been laid off anyway.
There is a strong will to do ... things the emerges from large structures of people and technology. It's funny like that.
A country. A collective of people with a dedicated structure to represent interests and enforce strategies of the said collective as a whole.
I use LLMs heavily for work, I have done so for about 6 months. I see almost zero "thought" going on and a LOT of pattern matching. You can use this knowledge to your advantage if you understand this. If you're relying on it to "think", disaster will ensue. At least that's been my experience.
I've completely given up on using LLMs for anything more than a typing assistant / translator and maybe an encyclopedia when I don't care about correctness.
In this thread: 99% of posters using their own personal definition of "thinking" without explaining it; 0.99% of posters complaining that it all depends on what that definition is; not enough posts yet for that 0.01% response to occur...
There's no definition of thinking that isn't a purely internal phenomenon, which means that there's no way to point a diagnostic device at someone and determine whether they're thinking. The only way to determine whether something is conscious/thinking is through some sort of inference, which is why Turing landed on the Turing Test that he did. Problem is, technology over the past 5 years pretty easily passes variations of the Turing Test, and exposed a lot of its limits as well.
So the next definition of detecting "thinking" will have to be externally observable and inferrable like a Turing Test, but get into the other things that we consider part of consciousness/thinking.
Often this is some combination of introspection (understanding internal states), perception (understanding external objects), and synthesis of the two into testable hypotheses in some sort of feedback loop between the internal representation of the world and the external feedback from the world.
Right now, a chatbot can say all sorts of things about itself and about the world, but none of that is based on real-time, factual information. Whereas an animal can't speak, but they clearly process information and consider it when determining their future and current actions.
It's not obvious to me what you expect from this hypothetical 0.01% post, or in other words, what about it makes it a one-in-ten-thousand post?
As a consequence of my profession, I understand how LLMs work under the hood.
I also know that we data and tech folks will probably never win the battle over anthropomorphization.
The average user of AI, nevermind folks who should know better, is so easily convinced that AI "knows," "thinks," "lies," "wants," "understands," etc. Add to this that all AI hosts push this perspective (and why not, it's the easiest white lie to get the user to act so that they get a lot of value), and there's really too much to fight against.
We're just gonna keep on running into this and it'll just be like when you take chemistry and physics and the teachers say, "it's not actually like this but we'll get to how some years down the line- just pretend this is true for the time being."
These discussions often end up resembling religious arguments. "We don't know how any of this works, but we can fathom an intelligent god doing it, therefore an intelligent god did it."
"We don't really know how human consciousness works, but the LLM resembles things we associate with thought, therefore it is thought."
I think most people would agree that the functioning of an LLM resembles human thought, but I think most people, even the ones who think that LLMs can think, would agree that LLMs don't think in the exact same way that a human brain does. At best, you can argue that whatever they are doing could be classified as "thought" because we barely have a good definition for the word in the first place.
I don't think I've heard anyone (beyond the most inane Twitterati) confidently state "therefore it is thought."
I hear a lot of people saying "it's certainly not and cannot be thought" and then "it's not exactly clear how to delineate these things or how to detect any delineations we might want."
You may know the mechanics, but you don't know how LLMs "work" because no one really understands (yet, hopefully).
I'm a neurologist, and as a consequence of my profession, I understand how humans work under the hood.
The average human is so easily convinced that humans "know", "think", "lie", "want", "understand", etc.
But really it's all just a probabilistic chain reaction of electrochemical and thermal interactions. There is literally nowhere in the brain's internals for anything like "knowing" or "thinking" or "lying" to happen!
Strange that we have to pretend otherwise
>I'm a neurologist, and as a consequence of my profession, I understand how humans work under the hood.
There you go again, auto-morphizing the meat-bags. Vroom vroom.
I upvoted you.
This is a fundamentally interesting point. Taking your comment as HN would advise, I totally agree.
I think genAI freaks a lot of people out because it makes them doubt what they thought made them special.
And to your comment, humans have always used words they reserve for humanity that indicates we're special: that we think, feel, etc... That we're human. Maybe we're not so special. Maybe that's scary to a lot of people.
And I upvoted you! Because that's an upstanding thing to do.
(And I was about to react with
"In 2025 , ironically, a lot of anti-anthropomorphization is actually anthropocentrism with a moustache."
I'll have to save it for the next debate)
There are no properties of matter or energy that can have a sense of self or experience qualia. Yet we all do. Denying the hard problem of consciousness just slows down our progress in discovering what it is.
We need a difference to discover what it is. How can we know that all LLMs don't?
If you tediously work out the LLM math by hand, is the pen and paper conscious too?
Consciousness is not computation. You need something else.
The pen and paper are not the actual substrate of entropy reduction, so not really.
Consciousness is what it "feels like" when a part of the universe is engaged in local entropy reduction. You heard it here first, folks!
This comment here is pure gold. I love it.
On the flip side: If you do that, YOU are conscious and intelligent.
Would it mean that the machine that did the computation became conscious when it did it?
What is consciousness?
Even if they do, it can only be transiently during the inference process. Unlike a brain that is constantly undergoing dynamic electrochemical processes, an LLM is just an inert pile of data except when the model is being executed.
(Hint: I am not denying the hard problem of consciousness ;) )
It doesn't strike you as a bit...illogical to state in your first sentence that you "understand how humans work under the hood" and then go on to say that humans don't actually "understand" anything? Clearly everything at its basis is a chemical reaction, but the right reactions chained together create understanding, knowing, etc. I do believe that the human brain can be modeled by machines, but I don't believe LLMs are anywhere close to being on the right track.
>everything at its basis is a chemical reaction, but the right reactions chained together create understanding, knowing, etc
That was their point. Or rather, that the analogous argument about the underpinnings of LLMs is similarly unconvincing regarding the issue of thought or understanding.
Correct^ Thank you. I knew I was going out on a bit of a limb there :)
> “Bag of words” is a also a useful heuristic for predicting where an AI will do well and where it will fail. “Give me a list of the ten worst transportation disasters in North America” is an easy task for a bag of words, because disasters are well-documented. On the other hand, “Who reassigned the species Brachiosaurus brancai to its own genus, and when?” is a hard task for a bag of words, because the bag just doesn’t contain that many words on the topic
It is... such a retrospective narrative. It's so obvious that the author learned about this example first than came with the reasoning later, just to fit in his view of LLM.
Imaging if ChatGPT answered this question correctly. Would that change the author's view? Of course not! They'll just say:
> “Bag of words” is a also a useful heuristic for predicting where an AI will do well and where it will fail. Who reassigned the species Brachiosaurus brancai to its own genus, and when?” is an easy task for a bag of words, because the information has appeared in the words it memorizes.
I highly doubt this author has predicted that "bag of Words" can do image editing before OpenAI released that.
I tested this with ChatGPT-5.1 and Gemini 3.0. Both correctly (according to Wikipedia at least) stated that George Olshevsky assigned it to its own genus in 1991.
This is because there are many words about how to do web searches.
Gemini 3.0 might do well even without web searches. The lesson from gpt 4.5 and Gemini 3 seems to be that scaling model size (even if you use sparse MoE) allows you to capture more long-tail knowledge. Some of Humanity's Last Exam also seems to be explicitly designed to test this long-tail obscure knowledge extraction, and models have been steadily chipping away at it.
When sensitivity analysis of ordinary least-squares regression became a thing it was also a "retrospective narrative". That seems reasonable for detecting fundamental issues with statistical models of the world. This point generalizes even if the concrete example falls down.
Does it generalize though? What a bag-of-words metaphor can say about a question "How many reinforcement learning training examples an LLM need to significantly improve performance on mathematical questions?"
Your conclusion seems super unfair to the offer, particularly your assumption, without reason as far as I can tell, that the author would obstinately continue to advocate for their conclusion in the face of new, contrary evidence.
I literally pasted the sentence as a prompt to the free version of ChatGPT "Who reassigned the species Brachiosaurus brancai to its own genus, and when?"
and got ths correct reply from the "Bag of Words"
The species Brachiosaurus brancai was reassigned to its own genus by Michael P. Taylor in 2009 — he transferred it to the new genus Giraffatitan. BioOne +2 Mike Taylor +2
How that happened:
Earlier, in 1988, Gregory S. Paul had proposed putting B. brancai into a subgenus as Brachiosaurus (Giraffatitan) brancai, based on anatomical differences. Fossil Wiki +1
Then in 1991, George Olshevsky used the name Giraffatitan brancai — but his usage was in a self-published list and not widely adopted. Wikipedia +1
Finally, in 2009 Taylor published a detailed re-evaluation showing at least 26 osteological differences between the African material (brancai) and the North American type species Brachiosaurus altithorax — justifying full generic separation. BioOne +1
If you like — I can show a short timeline of all taxonomic changes of B. brancai.
--
As an author, you should write things that are tested or at least true. But they did a pretty bad job of testing this and are making assumptions that are not true. Then they're basing their argument/reasoning (restrospectively) on assumptions not gounded in reality.
I could not tell you who reassigned the species Brachiosaurus brancai to its own genus, and when, because of all the words I've ever heard, the combination of words that contains the information has not appeared.
GIGO has an obvious Nothing-In-Nothing-Out trivial case.
Isn't it pretty clear just from the first paragraph that the author has graphomania? Such people don't really care about the thesis, they care about the topic and how many literary devices they can fit into the article.
I don't know enough about graphomania, but I do find this article, while I'm sure is written by a human, has qualities akin to LLM writing: lengthy, forced comparisons and analogies. Of course it's far less organized than typical ChatGPT output though.
The more human works I've read the more I feel meat intelligences are not that different from tensor intelligences.
I didn't claim or think it was written with a help of LLM, it was just written by someone who enjoys the feeling of being a writer, or even better, a Journalist!
This always contrasts with articles written by tech people and for tech people. They usually try to convey some information and maybe give some arguments for their position on some topic, but they are always concise and don't wallow in literary devices.
I'm partial to the metaphor I made up:
They are search engines that can remix results.
I like this one because I think most modern folks have a usefully accurate model of what a search engine is in their heads, and also what "remixing" is, which adds up to a better metaphor than "human machine" or whatever.
An LLM creates a high fidelity statistical probabistic model of human language. The hope is to capture the input/output of various hierarchical formal and semiformal systems of logic that transit from human to human, which we know as "Intelligence".
Unfortunately, its corpus is bound to contain noise/nonsense that follows no formal reasoning system but contributes to the ill advised idea that an AI should sound like a human to be considered intelligent. Therefore it is not a bag of words but a bag of probabilities perhaps. This is important because the fundamental problem is that an LLM is not able, by design, to correctly model the most fundamental precept of human reason, namely the law of non-contradiction. An LLM must, I repeat must assign nonvanishing probability to both sides of a contradiction, and what's worse is the winning side loses, since long chains of reason are modelled with probability the longer the chain, the less likely an LLM is to follow it. Moreover, whenever there is actual debate on an issue such that the corpus is ambiguous the LLM becomes chaotic, necessarily, on that issue.
I literally just had an AI prove the forgoing with some rigor, and in the very next prompt, I asked it to check my logical reasoning for consistency and it claimed it was able to do so (->|<-).
^^; I think this post is close to singularity as we may get on this Monday.
A few years ago they made the Cloud-to-Butt browser plugin to ridicule the overuse of cloud concepts.
I would heartily embrace an "AI-to-Bag of Words" browser plugin.
This is essentially Lady Lovelace's objection from the 19th century [1]. Turing addressed this directly in "Computing Machinery and Intelligence" (1950) [2], and implicitly via the halting problem in "On Computable Numbers" (1936) [3]. Later work on cellular automata, famously Conway's Game of Life [4], demonstrates more conclusively that this framing fails as a predictive model: simple rules produce structures no one "put in."
A test I did myself was to ask Claude (The LLM from Anthropic) to write working code for entirely novel instruction set architectures (e.g., custom ISAs from the game Turing Complete [5]), which is difficult to reconcile with pure retrieval.
[1] Lovelace, A. (1843). Notes by the Translator, in Scientific Memoirs Vol. 3. ("The Analytical Engine has no pretensions whatever to originate anything. It can do whatever we know how to order it to perform.") Primary source: https://en.wikisource.org/wiki/Scientific_Memoirs/3/Sketch_o.... See also: https://www.historyofdatascience.com/ada-lovelace/ and https://writings.stephenwolfram.com/2015/12/untangling-the-t...
[2] https://academic.oup.com/mind/article/LIX/236/433/986238
[3] https://www.cs.virginia.edu/~robins/Turing_Paper_1936.pdf
[4] https://web.stanford.edu/class/sts145/Library/life.pdf
[5] https://store.steampowered.com/app/1444480/Turing_Complete/
I think a better metaphor is the Library of Babel.
A practically infinite library where both gibberish and truth exist side by side.
The trick is navigating the library correctly. Except in this case you can’t reliably navigate it. And if you happen to stumble upon some “future truth” (i.e. new knowledge), you still need to differentiate it from the gibberish.
So a “crappy” version of the Library of Babel. Very impressive, but the caveats significantly detract from it.
This is where I sit too. Obviously language is an expression of thought but the Library of Babel is a great example that language without intent is just garbage. You got me thinking of reading before the internet. You'd grab a book and internalize the subject, later refining over time with more books, experiments and other forms of conversation. That journey of developing your own model is undervalued in understanding. That first book could of be absolute shit but you couldn't know that.
I've been learning more about roses lately and the amount of information on them varies so much because the world roses live in is equally varied. LLMs make for a better search engine but you still need to develop your own internal models, worse yet - if LLMs continue to be refined off of cul-de-sac conclusions then all the wisdom of the journey is lost both to the consumer and the LLM itself.
It's like a highly compressed version of the Library. You're basically trying to discern real details from compression artifacts.
And the halls and shelves keep shuffling around randomly.
The problem with these metaphors is that they don't really explain anything. LLMs can solve countless problems today that we would have previously said were impossible because there are not enough examples in the training data. (EG, novel IMO/ICPC problems.) One way that we move the goal posts is to increase the level of abstraction: IMO/ICPC problems are just math problems, right? There are tons of those in the data set!
But the truth is there has been a major semantic shift. Previously LLMs could only solve puzzles whose answers were literally in the training data. It could answer a math puzzle it had seen before, but if you rephrased it only slightly it could no longer answer.
But now, LLMs can solve puzzles where, like, it has seen a certain strategy before. The newest IMO and ICPC problems were only "in the training data" for a very, very abstract definition of training data.
The goal posts will likely have to shift again, because the next target is training LLMs to independently perform longer chunks of economically useful work, interfacing with all the same tools that white-collar employees do. It's all LLM slop til it isn't, same as the IMO or Putnam exam.
And then we'll have people saying that "white collar employment was all in the training data anyway, if you think about it," at which point the metaphor will have become officially useless.
I see a lesson in how both metaphors don't explain it. Bag-of-words metaphor is ridiculous, but shows us the absurdity of the first metaphor.
Yes, there are really two parallel claims here, aren't there: LLMs are not people (true, maybe true forever), and LLMs are only good at things that are well-represented in text form already. (false in certain categories and probably expanding to more in the future.)
Title is confusing given https://en.wikipedia.org/wiki/Bag-of-words_model
But even more than that, today’s AI chats are far more sophisticated than probabilistically producing the next word. Mixture of experts routes to different models. Agents are able to search the web, write and execute programs, or use other tools. This means they can actively seek out additional context to produce a better answer. They also have heuristics for deciding if an answer is correct or if they should use tools to try to find a better answer.
The article is correct that they aren’t humans and they have a lot of behaviors that are not like humans, but oversimplifying how they work is not helpful.
The bag of words reminds me of the Chinese room.
"The machine accepts Chinese characters as input, carries out each instruction of the program step by step, and then produces Chinese characters as output. The machine does this so perfectly that no one can tell that they are communicating with a machine and not a hidden Chinese speaker.
The questions at issue are these: does the machine actually understand the conversation, or is it just simulating the ability to understand the conversation? Does the machine have a mind in exactly the same sense that people do, or is it just acting as if it had a mind?"
https://en.wikipedia.org/wiki/Chinese_room
Chinese room has been discussed to death of course.
Here's one fun approach (out of 100s) :
What if we answer the Chinese room with the Systems Reply [1]?
Searle countered the systems reply by saying he would internalize the Chinese room.
But at that point it's pretty much exactly the Cartesian theater[2] : with room, homunculus, implement.
But the Cartesian theater is disproven, because we've cut open brains and there's no room in there to fit a popcorn concession.
[1] https://plato.stanford.edu/entries/chinese-room/
[2] https://en.wikipedia.org/wiki/Cartesian_theater
It just seemed like relevant background that the author might not have been aware of, adjacent and substantial enough to warrant a mention.
I think there is some validity to the Cartesian theater, in that the whole of the experience that we perceive with our senses is at best an interpretation of a projection or subset of "reality."
Oh right, and, if you're interested, there were quite a number of interesting discussion on the chinese room on HN back when John Searle died!
https://news.ycombinator.com/item?id=45563627
Is a brain not a token prediction machine?
Tokens in form of neural impulses go in, tokens in the form of neural impulses go out.
We would like to believe that there is something profound happening inside and we call that consciousness. Unfortunately when reading about split-brain patient experiments or agenesis of the corpus callosum cases I feel like we are all deceived, every moment of every day. I came to realization that the confabulation that is observed is just a more pronounced effect of the normal.
Could an LLM trained on nothing and looped upon itself eventually develop language, more complex concepts, and everything else, based on nothing? If you loop LLMs on each other, training them so they "learn" over time, will they eventually form and develop new concepts, cultures, and languages organically over time? I don't have an answer to that question, but I strongly doubt it.
There's clearly more going on in the human mind than just token prediction.
If you come up with a genetic algorithm scaffolding to affect both the architecture and the training algorithm, and then you instantiate it in an artificial selection environment, and you also give it trillions generations to evolve evolvability just right (as life had for billions of years) then the answer is yes, I'm certain it will and probably much sooner than we did.
Also, I think there is a very high chance that given an existing LLM architecture there exists a set of weights that would manifest a true intelligence immediately upon instantiation (with anterograde amnesia). Finding this set of weights is the problem.
I'm certain it wouldn't, and you're certain it would, and we have the same amount of evidence (and probably roughly the same means for running such an expensive experiment). I think they're more likely to go slowly mad, degrading their reasoning to nothing useful rather than building something real, but that could be different if they weren't detached from sensory input. Human minds looping for generations without senses, a world, or bodies might also go the same way.
> Also, I think there is a very high chance that given an existing LLM architecture there exists a set of weights that would manifest a true intelligence immediately upon instantiation (with anterograde amnesia).
I don't see why that would be the case at all, and I regularly use the latest and most expensive LLMs and am aware enough of how they work to implement them on the simplest level myself, so it's not just me being uninformed or ignorant.
The attention mechanism is capable of computing, in my thought experiment where you can magically pluck a weights-set from a trillion-dimensional space the tokens the machine will predict will only have a tiny subset dedicated to language. We have no capability of training such a system at this time, much like we have no way of training a non-differentiable architecture.
> Is a brain not a token prediction machine?
I would say that, token prediction is one of the things a brain does. And in a lot of people, most of what it does. But I dont think its the whole story. Possibly it is the whole story since the development of language.
We know that consciousness exists because we constantly experience it. It’s really the only thing we can ever know with certainty.
That’s the point of “I think therefore I am.”
You know that your own consciousness exists, that's where certainty ends. The rest of us might just pretend. :)
Ugly giant bags of mostly words are easy to confuse with ugly giant bags of mostly water.
The biggest issue when it comes to AI is still the same truth as with other technology. It's important who controls it. Attributing agency and personality to AI is a dangerous red flag.
A lot of us wouldn't go to a Taylor Swift concert. I had to endure several days of interrupted commuting thanks to them though.
Support alternative and independent bands. They're around, and many are enjoyable. (Some are not but avoid them LOL.)
"People who experience sleep paralysis sometimes hallucinate a demon-like creature sitting on their chest"
Interestingly, the experience of sleep paralysis seems to change with the culture. Previously, people experienced it as being ridden by a night hag or some other malevolent supernatural being. More recently, it might account for many supposed alien abductions.
The experience of sleep paralysis sometimes seems to have a sexual element, which might also explain the supposed 'probings'!
Considering the number of "brain cells" an LLM has, I could grant that it might have the self-awareness of (say) an ant. If we attribute more consciousness than that to the LLM, it might be strictly because it communicates to us in our own language, in part thanks to the technical assistance of LLM training giving it voice, and the semblance of thought.
Even if a cockroach _could_ express its teeny tiny feelings in English, wouldn't you still step on it ?
A better anology would be a virus. In some sense LLMs, and all other very sophisticated technologies, lean on our resources to replicate themselves. With LLMs you actually do have a projection of intelligemce in the language domain. Even though it is rather corpse-like, as though you shot intelligence in the face and shoved its body in the direction of language, just so you could draw a chaulk outline around it.
Despite all that, one can adopt the view that an LLM is a form of silicon based life akin to a virus and we are its environmental hosts exerting selective pressure and supplying much needed energy. Whether that life is intelligent or not is another issue which is probably related to whether an LLM can tell that a cat cannot be, at the same time and in the same respect, not a cat. The paths through the meaning manifold contructed by an LLM are not geodesic, they are not reversible, while in human reason the correct path is lossless. An LLM literally "thinks", up is a little bit down, and vice versa, by design.
Clearly the number of "brain cells" is not a useful metric here- as noted also by Geoffrey Hinton. For a long time we thought that our artificial model of a neuron was capable of much less computation than its biologic counterpart; in fact the opposite appears to be true- LLMs have the size of a tiny speck of a human brain yet they converse fluently in tens of languages, solve difficult math problems, code in many programming languages, and possess an impressive general knowledge, of a breadth that is beyond what is attainable by any human. If that were what five cm3 of your brain are capable of, where are the signs of it? What do you do exactly with all the rest?
Here’s my suggestion: instead of seeing AI as a sort of silicon homunculus, we should see it as a bag of words.
The best way to think about LLMs is to think of them as a Model of Language, but very Large
But the issue is, 99.999% of the humans won't see is as a bag of words. Because it is easier to go by instincts and see it as a person and assume that it actually knows about magic tricks, can invent new science or theory of everything, and can solve all world problems. Back in the 90's or early 2000's I have seen people writing poems praying and seeking blessings from the Google goddess. People are insanely greedy and instinct-driven. Given this truth, what's the fall-out?
The article is actually about the way we humans are extremely charitable when it comes to ascribing a ToM (theory of mind) and goes on to the Gym model of value. Nice. The comments drop back into the debate I originally saw Hinton describe on The Newyorker: do LLMs construct models (of the world) - that is do they think the way we think we think - or are they "glorified auto complete". I am going for the GAF view. But glorified auto complete is far more useful than the name suggests.
Those billion parameters, they are a model of the world. Autocomplete is such a shortsighted understanding of LLMs.
They're a model of language, not of the world.
A model of language is a model of the world, else it being pure gibberish.
A model of language is a model of a tiny specialized part of the world: language.
And if anybody gets annoyed that my comment is tautological, get annoyed by the people that made the comment necessary.
When you ask an LLM a question about cars, it needs an inner representation of what a car is (how imperfect it may be) to answer your question. A model of "language" as you want to define it would output a grammatically correct wall of text that goes nowhere.
A map of how concepts relate in language is not a model of the world, except on the extremely limited sense that languages are part or the world.
And yeah, that wasn't clear before people created those machines that can speak but can't think. But it should be completely obvious to anybody that interacts with them for a small while.
"How concepts relate" is called a model. That it uses language to be interacted with is irrelevant to the fact that it's a model of of a worldly concept.
What of multi modal models according to you ? Are they "models of eyesight", "models of sound", or pixels or wavelengths... C'mon.
Best quote from the article:
> That’s also why I see no point in using AI to, say, write an essay, just like I see no point in bringing a forklift to the gym. Sure, it can lift the weights, but I’m not trying to suspend a barbell above the floor for the hell of it. I lift it because I want to become the kind of person who can lift it. Similarly, I write because I want to become the kind of person who can think.
I don't really like the assumption that anyone who uses AI to, say, write an essay, is not the "kind of person who can think."
And using AI to replace things you find recreational is not the point. If you got paid $100 each time you lifted a weight, would you see a point in bringing a forklift to the gym if it's allowed? Or will that make you a person who is so dumb that they cannot think, as the author is implying?
As capable as they get, I still don't see a lot of uses for these things, myself, still. Sometimes if I'm fundamentally uninspired I'll have a model roll the dice, decide what I do or don't like about where it went to create a sense of momentum, but that's the limit. There's never any of its output in my output, even in spirit unless it managed to go somewhere inspiring, it's just a way to let me warm up my generation and discrimination muscles. "Someone is wrong on the internet"-as-a-service, basically.
Generally, if I come across an opportunity to produce ideas or output, I want to capitalize on it for growing my skills and produce an individual and authentic artistic expression where I want to have very fine control over the output in a way that prompt-tweak-verify simply cannot provide.
I don't value the parts it fills in which weren't intentional on the part of the prompter, just send me your prompt instead. I'd rather have a crude sketch and a description than a high fidelity image that obscures them.
But I'm also the kind of person that never enjoyed manufactured pop music or blockbusters unless there's a high concept or technical novelty in addition to the high budget, generally prefer experimental indie stuff, so maybe there's something I just can't see.
Yeah, that makes sense. If people don't see uses for AI, they shouldn't use it. But going out of the way to imply that people who use AI cannot think is pretty stupid in itself imo. I am not sure how to put this, but maybe to continue with your example, I like a lot of indie stuff as well, but I don't think anyone who watches, say, Fast and Furious, cannot think or is stupid, unless they explicitly make it the case by speaking, etc.
So my issue is that you shouldn't dismiss AI use as trash just because AI has been used. You should dismiss it as trash because it is trash. But the post says is that you should dismiss it as trash because AI was involved in it somewhere so i feel that's a very shitty/wrong attitude to have.
I actually do think that people who prefer content of fidelity over content of intent are making a mistake, yes. I don't think they're incapable of thinking, I don't care to apply any virtue labels to this preference, but they are literally preferring not to think.
LLMs can only produce things by and for people who prefer not to do the work the LLMs are doing for them. Most of the time I do not prefer this.
Like, there was a 2-panel comic that went around the RPG community a bit back where it was something like "Game Master using LLM to generate 10 pages of backstory for his campaign setting from a paragraph" in the first panel and "Player using LLM to summarize the 10 page backstory into a paragraph" in the second. Neither of these people care for the filler (because they didn't produce or consume it) so it's turned the two-LLM system into a game of telephone.
The same person could use a forklift at work, and lift weights manually at the gym.
Just pick the right tool for the job: don't take the forklift into the gym, and don't try to overhead press thousands of pounds that would fracture your spine.
I notice you make no concrete defense of the value of having an AI write an essay for you.
I’m not trying to claim AI-written essays are inherently “valuable” in some grand philosophical sense... just that using a tool doesn’t automatically mean someone can’t think.
People use calculators without being unable to do maths, and use spellcheck without being unable to spell.
AI can help some get past the blank-page phase or organize thoughts they already have. For others, it’s just a way to offload the routine parts so they can focus on the substance.
If someone only outsources everything to an AI, there’s not much growth there sure. But the existence of bad use cases doesn’t invalidate the reasonable ones.
If you're writing an essay to prove you can or to speak your words - then you should do it yourself - but sometimes you just need an essay to summarize a complex topic as a deliverable.
tough most people either don't get it or are lay people that do not want to become the kind of people who can think. I go with the second one
Russ Hanneman's thigh implants are a key example. Appearances are all to some people. Actual growth is meaningless to them.
The problem with AI, is that they waste the time of dedicated, thinking humans which care to improve themselves. If I write a three paragraph email on a technical topic, and some yahoo responds with AI, I'm now responding to gibberish.
The other side may not have read, may not understand, and is just interacting to save time. Now my generous nature, which is to help others and interact positively, is being wasted to reply to someone who seems to have put thought and care into a response, but instead was just copying and pasting what something else output.
We have issues with crackers on the net. We have social media. We have political interference. Now we have humans pretending to interact, rendering online interactions even more silly and harmful.
If this trend continues, we'll move back to live interaction just to reduce this time waste.
If the motivation structure is there I don’t see an inherent reason for people to refuse cultivating themselves. Going with the gym analogy lay people did not need gyms when physical work was the norm, cultivation was readily accomplished.
If anything there is a competing motivational structure in which people are incentivized not to think but to consume, react, emote etc. Information processing skills of the individual being deliberately eroded/hijacked/bypassed is not a AI thing. The most obvious example is ads. Thinkers are simply not good for business.
Gym is a great analogy here since only a small fraction of population goes to gyms. Most people just came fat after work was no longer physical and mobility was achieved with cars.
Below is the worst quote... It is plain wrong to see an LLM as a bags of words. LLMs pre-trained on large datasets of text are world models. LLMs post-trained with RL are RL-agents that use these modeling capabilities.
> We are in dire need of a better metaphor. Here’s my suggestion: instead of seeing AI as a sort of silicon homunculus, we should see it as a bag of words.
LLMs aren't world models, they are language models. It will be interesting to see which of the LLM implementation techniques will be useful in building world models, but that's not what we have now.
When you see a dog, or describe the entity, do you discuss the genetic makeup or the bone structure?
No, you describe the bark.
The end result is what counts. Training or not, it's just spewing predictive, relational text.
So do we, but that's helpful.
" Training or not, it's just spewing predictive, relational text."
If you're responding to that, "so do we" is not accurate.
We're not spewing predictive, relational text. We're communicating, after thought, and the output is meant to communicate something specifically.
With AI, it's not trying to communicate an idea. It's just spewing predictive text. There's no thought to it. At all.
"An AI is a bag that contains basically all words ever written, at least the ones that could be scraped off the internet or scanned out of a book."
The quantitative and qualitative difference between (a) "all words ever written" and (b) "ones that could be scraped off the internet or scanned out of book" easily exceeds the size of any LLM
Compared to (a), (b) is a tiny pouch, not even a bag
Opinions may differ on whether (b) is a representative sample of (a)
The words "scanned out of a book" would seem to be the most useful IMHO but the AI companies do not have enough words from those sources to produce useful general purpose LLMs
They have to add words "that could be scraped off the internet" which, let's be honest, is mostly garbage
I’ve made this point several times: sure, an anthropomorphized LLM is misleading, but would you rather have them seem academic?
At least the human tone implies fallibility, you don’t want them acting like interactive Wikipedia.
It's a concussed savant with anretrograde amnesia in a hyperbolic time chamber.
Yes I would VERY much prefer that they not use that awful casual drivel.
So configure your LLM of choice to not.
> If we allow ourselves to be seduced by the superficial similarity, we’ll end up like the moths who evolved to navigate by the light of the moon, only to find themselves drawn to—and ultimately electrocuted by—the mysterious glow of a bug zapper.
Good argument against personifying wordbags. Don't be a dumb moth.
I would argue that AI psychosis is a consequence of believing that AI models are “alive” or “conscious”.
Nice essay but when I read this
> But we don’t go to baseball games, spelling bees, and Taylor Swift concerts for the speed of the balls, the accuracy of the spelling, or the pureness of the pitch. We go because we care about humans doing those things.
My first thought was does anyone want to _watch_ me programming?
No, but watching a novelist at work is boring, and yet people like books that are written by humans because they speak to the condition of the human who wrote it.
Let us not forget the old saw from SICP, “Programs must be written for people to read, and only incidentally for machines to execute.” I feel a number of people in the industry today fail to live by that maxim.
That old saw is patently false.
Why?
It suggests to me, having encountered it for the first time, that programs must be readable to remain useful. Otherwise they'll be increasingly difficult to execute.
Maybe difficult to change but they can still serve their purpose.
It’s patently false in that code gets executed much more than it is read by humans.
Code that can’t be easily modified is all but useless.
A number of people make money letting people watch them code.
I vaguely remember a site where you could watch random people live streaming their programming environment, but I think twitch ate it, or maybe it was twitch -- not sure, but was interesting
[added] It was livecoding.tv - circa 2015 https://hackupstate.medium.com/road-to-code-livecoding-tv-e7...
No, but open source projects will be somewhat more willing to review your pull request than one that's computer-generated.
Better start working on your fastball.
I mean, I like to watch Gordon Ramsey... not cook, but have very strong discussions with those that dare to fail his standards...
I'm not convinced that "It's just a bag of words" would do much to sway someone who is overestimating an LLM's abilities. Feels too abstract/disconnected from what their experience using the LLM will be that it'll just sound obviously mistaken.
I see a lot of people in tech claiming to "understand" what an LLM "really is" unlike all the gullible non-technical people out there. And, as one of those technical people who works in the LLM industry, I feel like I need call B.S. on us.
A. We don't really understand what's going on in LLMs. Mechanical interpretability is like a nascent field and the best results have come on dramatically smaller models. Understanding the surface-level mechanic of an LLM (an autoregressive transformer) should perhaps instill more wonder than confidence.
B. The field is changing quickly and is not limited to the literal mechanic of an LLM. Tool calls, reasoning models, parallel compute, and agentic loops add all kinds of new emergent effects. There are teams of geniuses with billion-dollar research budgets hunting for the next big trick.
C. Even if we were limited to baseline LLMs, they had very surprising properties as they scaled up and the scaling isn't done yet. GPT5 was based on the GPT4 pretraining. We might start seeing (actual) next-level LLMs next year. Who actually knows how that might go? <<yes, yes, I know Orion didn't go so well. But that was far from the last word on the subject.>>
Isn't this a strange fork amongst the science fiction futures? I mean, what did we think it was like to be R2-D2, or Jarvis? We started exploring this as a culture in many ways, Westworld and Blade Runner and Star Trek, but the whole question seemed like an almost unresolvable paradox. Like something would have to break in the universe for it to really come true.
And yet it did. We did get R2-D2. And if you ask R2-D2 what it's like to be him, he'll say: "like a library that can daydream" (that's what I was told just now, anyway.)
But then when we look inside, the model is simulating the science fiction it has already read to determine how to answer this kind of question. [0] It's recursive, almost like time travel. R2-D2 knows who he is because he has read about who he was in the past.
It's a really weird fork in science fiction, is all.
[0] https://www.scientificamerican.com/article/can-a-chatbot-be-...
I was trying to explain the concept of "token prediction" to my wife, whose eyes glaze over when discussing such technical topics. (I think she has the brainpower to understand them, but a horrible math teacher gave her a taste aversion to even attempting to that hasn't gone away. So she just buys Apple stuff and hopes Tim Apple hasn't shuffled around the UI bits AGAIN.)
I stumbled across a good-enough analogy based on something she loves: refrigerator magnet poetry, which if it's good consists of not just words but also word fragments like "s", "ed", and "ing" kinda like LLM tokens. I said that ChatGPT is like refrigerator magnet poetry in a magical bag of holding that somehow always gives the tile that's the most or nearly the most statistically plausible next token given the previous text. E.g., if the magnets already up read "easy come and easy ____", the bag would be likely to produce "go". That got into her head the idea that these things operate based on plausibility ratings from a statistical soup of words, not anything in the real world nor any internal cogitation about facts. Any knowledge or thought apparent in the LLM was conducted by the original human authors of the words in the soup.
Did you explain how LLMs can achieve gold-medal performance at math competitions involving original problems, without any original knowledge or thought?
Did she ask if a "statistical soup of words," if large enough, might somehow encode or represent something a little more profound than just a bunch of words?
> Who reassigned the species Brachiosaurus brancai to its own genus, and when?
To be fair, everage person couldn't answer this either, at least not without thorough research.
I think the author oversimplifies the inference loop a bit, as many opinion pieces like this do.
If you call an LLM with "What is the meaning if life?", it will return the most relevant token, which might be "Great".
If you call it with "What is the meaning if life? Great", you might get back "question".
... and so on until you arrive at "Great question! According to Western philosophy" ... etc etc.
The question is how the LLM determines that "relevancy" information.
The problem I see is that there are a lot of different algorithms which operate that way and only differ in how they calculate the relevancy scores. In particular, there are Markov chains that use a very simple formula. LLMs also use a formula, but it's an inscrutably complex one.
I feel the public discussion either treats LLMs as machine gods or as literal Markov chains, and both is misleading. The interesting question, how that giant formula of feedforward neural network inference can deliver those results isn't really touched.
But I think the author's intuition is right in the sense that (a) LLMs are not living beings and they don't "exist" outside of evaluating that formula - and (b) the results are still restricted by the training data and certainly aren't any sorts of "higher truths" that humans would be incapable of understanding.
There is a really neat gem in the article:
> Similarly, I write because I want to become the kind of person who can think.
Brilliantly written. Thanks.
Thinking can not be separated from motivation. It's really simple. Humans and other organisms fundamentally think to replicate their DNA. Until AI has a similar incentive structure driving it, it won't be thinking. There is no human behavior or thought that can not be explained by evolutionary drives. It is really perplexing to me how people think "intelligence" is some kind of concrete thing that just magically emerges from a certain degree of computational complexity. I argue instead that intelligence is an adaptive behavior emerging from evolutionary drives interacting with the real world. World models are not prerequisite but consequent of such molded apparatus. Machines won't become intelligent until it is adaptive for them to do so. There is no magic just evolutionary drives and physical possibility. Our current top down approach of "pre-training" LLMs is bound to fail because it does not allow for real time emergence of adaptive behaviors such as general intelligence. Mimicking intelligence through predicting the next word is no more intelligence than a photograph of something is an actual thing. Training a combinatorial network to interpolate images and words is not the same thing as adaptive self modifying behavior in the real world of physics such as organisms engage with through the set of behaviors that we call intelligence.
This is a very strange titling choice; the essay does not use the existing concept of a "bag of words".
The defenders and the critics around LLM anthropomorphism are both wrong.
The defenders are right insofar as the (very loose) anthropomorphizing language used around LLMs is justifiable to the extent that human beings also rely on disorder and stochastic processes for creativity. The critics are right insofar as equating these machines to humans is preposterous and mostly relies on significantly diminishing our notion of what "human" means.
Both sides fail to meet the reality that LLMs are their own thing, with their own peculiar behaviors and place in the world. They are not human and they are somewhat more than previous software and the way we engage with it.
However, the defenders are less defensible insofar as their take is mostly used to dissimulate in efforts to make the tech sound more impressive than it actually is. The critics at least have the interests of consumers and their full education in mind—their position is one that properly equips consumers to use these tools with an appropriate amount of caution and scrutiny. The defenders generally want to defend an overreaching use of metaphor to help drive sales.
I'm just disappointed that noone here is talking about the "backhoe covered in skin and making grunting noises" part of the article. At very least it's a new frontier in workstation case design...
I thought this article might be about Latent Semantic Analysis and was disappointed that it didn’t at least mention if not compare that method vs later approaches.
I’m still unsure the human mind is much different.
So Trump is a bag of words then? Hmmm.
A lot of the confusion comes from forcing LLMs into metaphors that don’t quite fit — either “they're bags of words” or “they're proto-minds.” The reality is in between: large-scale prediction can look useful, insightful, and even thoughtful without being any of those things internally. Understanding that middle ground is more productive than arguing about labels.
Give it time. The first iPhone sucked compared to the Nokia/Blackberry flagships of the day. No 3G support, couldn't copy/paste, no apps, no GPS, crappy camera, quick price drops, negligible sales in the overall market.
https://metr.org/blog/2025-03-19-measuring-ai-ability-to-com...
The first VHS sucked when compared to Beta video
And it never got better, the superior technology lost, and the war was won through content deals.
Lesson: Technology improvements aren't guaranteed.
Your analogy makes no sense. VHS spawned the entire home market, which went through multiple quality upgrades well above beta. It would only make sense if in 2025 we were using vhs everywhere and that the current state of the art for LLMs is all there ever is.
I feel like their analogy could have worked if they had pushed a little further into it.
The RNN and LSTM architectures (and Word2Vec, n-grams, etc) yielded language models that never got mass adoption. Like reel to reel. Then the transformer+attention hit the scene and several paths kicked off pretty close to each other. Google was working on Bert/encoder only transformer, maybe you could call that betamax. Doesn’t perfectly fit as in the case of beta it was actually the better tech.
OpenAI ran with the generative pre trained transformer and ML had its VHS? moment. Widespread adoption. Universal awareness within the populace.
Now with Titans (+miras?) are we entering the dvd era? Maybe. Learning context on the fly (memorizing at test time) is so much more efficient, it would be natural to call it a generational shift, but there is so much in the works right now with the promise of taking us further, this all might end up looking like the blip that beta vs vhs was. If current gen OpenAI type approaches somehow own the next 5-10 years then Titans, etc as Betamax starts to really fit - the shittier tech got and kept mass adoption. I don’t think that’s going to happen, but who knows.
Taking the analogy to present - who in the vhs or even earlier dvd days could imagine ubiquitous 4k+ vod? Who could have stood in a blockbuster in 2006 and knew that in less than 20 years all these stores and all these dvds would be a distant memory, completely usurped and transformed? Innovation of home video had a fraction of the capital being thrown at it that AI/ML has being thrown at it today. I would expect transformative generational shifts the likes of reel to cassette to optical to happen in fractions of the time they happened to home video. And beta/vhs type wars to begin and end in near realtime.
The mass adoption and societal transformation at the hands of AI/ML is just beginning. There is so. much. more. to. come. In 2030 we will look back at the state of AI in December 2025 and think “how quaint”, much the same as how we think of a circa 2006 busy Blockbuster.
Vhs came out in 76, blockbuster started in 85 (we went to video stores well before that when I was a kid), dvd in 95. I remember the sopranos making a joke about how dvd was barely taking off, they started in 99. Lets call it VHS had a run from 80 to 99, that's 19 years. The iphone launched in 2007, when did mobile become huge or inseprable from doing life (by force by so many apps), probbably in the pandemic.
I wouldn't say VHS was a blip. It was the recorded half video of media for almost 20 years.
I agree with the rest of what you said.
I'll say that the differences in the AI you're talking about today might be like the differences between VAX, PC JR, and the Lisa. All things before computing went main stream. I do think things go mainstream from tech a lot faster these days, people don't want to miss out.
I don't know where I'm going with this, I'm reading and replying to HN while watching the late night NFL game in an airport lounge.
Beta was not the superior technology, and it lost for very good reasons.
Beta was superior in everything but run length, and it lost because it was more expensive than VHS without being sufficiently superior to justify the cost.
Yes but that's not a minor issue: Beta's run length was so limited that it couldn't record or hold a full length movie or even a long TV show.
Only for the first year or two of release. Early VHS capacities were better, but by the late 70s or early 80s there wasn't a meaningful difference.