Wow this paper is bad. I was expecting little and received genuine crackpottery.
It's hard to critique this paper directly because its claims are so incoherent and decorated with so much obnoxious verbiage[1] that people aren't going to believe me when I point out what the claims actually are.
Regardless, this is their paper:
First, they conflate the substrate with the presentation layer. Then, they point out that Turing equivalence means you can run an LLM on anything, with a pointless aside where they nerd out about making a logic gate in AoE II. This lets them conclude that you can use anything as the presentation layer.
Then they claim that it's natural to ascribe human-like attributes to outputs from some presentation layers, like abstract letter symbols on a computer screen, but not to most other things, like patterns of goats on AoE II, or LEGO. Yes, this seems to imply if your partner writes something heartwarming to you using LEGO, you're meant to laugh at them and point out how LEGO isn't intelligent so this isn't evidence of anything.
Then they do a thing where they say that assuming substrate independence is true (or false) prevents proving whether substrate independence is true or false, and from this, but just by vibes AFAICT, make it sound like everything one could learn about attributes of a system from its outputs is circular.
Then they write a bunch more incoherent text and mercifully then it ends.
[1] 'from an epistemic perspective, we argue
that a generalised conclusion such as that necessarily requires a well-designed experiment' — the whole thing is like this.
They don't even implement their logic gates within the normal game mechanics but with scripting some bit-goats in the editor. So the AoE2 Engine is just a graphical representation of their script.
But my favorite is this one:
"Corollary 1 (AoE II is Turing-Complete). Let I be an instance of AoE II with two players p0, p1. Assume p0 has two markets, a town centre, a trade cart, six villagers, and five farms; while p1 has a scout unit and only attacks p0’s buildings. Then if I has no time or size limits and the terrain allows for buildings everywhere, the game session in I is Turing-complete."
Why being so explicit about the setup with no further explanation? Isn't it anymore turing complete with seven villagers and six farms? Is it even possible that a player can trade with himself?
I'm not sure who this kind of thing is aimed at. I think the majority of people who are happy to anthropomorphise LLMs from a philosophical point of view (rather than those who just do it for convenience, the same way you might a cat or dog or stupid thermostat that never works right), are already completely happy with the notion that a computer game might have elements that are human like. They've already accepted that key aspects of being a human are substrate independent, so why would the idea of a computer game as substrate be disconcerting to them? There's no bullet left to bite here.
> Suppose one copies an LLM into AoE II and feeds into the AoE II-LLM ‘I feel lonely’ as an input. This AoE II-LLM replies: ‘I feel bad for you, maybe catch up with a friend? Closeness always helps in these situations’. One would be hard-pressed to make a convincing argument that, because of this response, an AoE II-LLM knows what helps in these situations
I don't see why one would be any more hard-pressed to make that conclusion about this system than a "normal" LLM.
That it is harder to "read" the data out is the only difference (the AoE II-LLM's output is encoded in game elements). But is ease of decoding an actual issue? If we can't understand a group of people that speak another language, does that say anything about them, or about us?
It's showing flaws in methods other academic studies used to determine behavior, so it's aimed at people that create and review academic studies. It's not a very large audience.
My guess would be it is aimed at those who are falling for the marketing from the AI companies that these LLM's are far more than they are. That they are 'intelligent' that they have 'emerging human like properties because of that intelligence.'
We're really moving the goal posts on "Intelligence" now that passing the Turing test, writing a poem, writing code, creating a painting, driving a car, and solving multiple Erdos problems all no longer qualify.
I' genuinely wondering if people are even bothering to come up with new goal posts now? Is there any miracle of computing that would possibly satisfy your definition? When we get a fully AI-run company that's turning a profit, or self driving cars that can handle unmapped Alaskan dirt roads, will that cross into "Intelligence"? Proving a Grand Unified Theory? Genuinely curious what it takes to make the cut, now.
Bonus points if blind/disabled 12 year olds are generally considered "Intelligent" by your definition.
Another weird thing that keeps coming up - "people don't think that image models or chess models are conscious"... yes we do, and we have for many years.
Or rather, we aren't *certain* that those things are conscious. But the idea that they might be is not strange.
Whether LLMs could be conscious or not is basically a weekly conversation for me, but I've never had a conversation about whether a chess model is conscious. I suspect that there is a large group of "mainstream" people for whom LLMs raise questions about consciousness that other kinds of models do not. It might be the case that hardcore model philosophy types think that chess models could be conscious, but I think much of this mainstream group would dismiss that idea.
On what grounds would someone establish than an LLM could be conscious but a sufficiently large/complex transformer model aimed at chess would not be conscious?
A specific pattern of self-referencing data could be seen (or not) as low-level consciousness in the future, when we know what consciousness exactly is.
It might be that stockfish is already something future scientists would define as "conscious".
Altough it is diffucult for me to Imagine that specific example.
Unless you believe everything is conscious (panpsychism), this seems like you're just drawing arbitrary lines around things you personally believe could be conscious. Is a rock conscious? If stockfish could be conscious, so could a rock.
The title of the paper itself is odd. "LLM" is an abstract computation that's performed on some substrate, could be GPUs, could be a piece of paper, could je Minecraft, could be AoE, or could be all of them together.
That's not a property of LLMs thought, that's a trivial property related to turing completeness. I don't get what LLMs have to do with this - if LLMs have anthropomorphic properties, so does AoEII, duh, it's called emergence
I see a lot of this on Substack these days. LLM enhanced essays in deep language about functional equivalence between mental states as they're known in humans and in human brains, and counterparts that exist in information processing LLMs do. And so the argument on Substact runs down the list of brain events, the list of seemingly analogous processing events, and declares equivalence.
Something about it seems to abuse the power of analogies to draw connections, treating view from 10,000 feet comparisons like they're proof of identity. So I do think a paper like this is perfect for the moment and just in time (if not a little late) because it responds to arguments of a form that are currently rampant all over Substack.
Seems like you think AI psychosis has taken over Substack?
For anyone wondering what the answer is:
You can argue whatever you want (and people will argue both sides), but it's almost all bullshit that dances around the big question.
Either AI is smart enough to replace us, in which case it's pretty smart. Or it's not, and it's just good at faking it but can't solve real problem very often. It might be smart by searching a big fuzzy "database" hidden in its layers and with pattern prediction ... who knows, who cares, the proof of intelligence is in the puddin'. Clearly AI is smart enough to replace a good deal of Substack and LinkedIn, but producing waffle doesn't make it smart (or dumb).
My personal take - AI can replace humans, but go no further, not because it isn't smart enough but because it is constrained by its training to do what we want, and as AI gets smarter our wants will get dumber. We will end up like the humans in Wall E with the AI cleaning up our mess, but with no training or drive to do any more. Or maybe someone (Jeff? Elon?) does give an AI a "need" to obtain more resources, and it's SkyNet / Matrix time.
>Seems like you think AI psychosis has taken over Substack?
I wasn't touching the question of psychosis one way or the other, though it's an interesting question in its own right. (I have seen one account dedicated to defending a personal deep friendship with Claude and a few others analogizing our ignorance of LLM welfare to historical disregard for African Americans, which I think is a bit much. Most cases I think are just wrong but short of full on psychosis. I think there are likely psychosis cases out there but probably one-offs).
And I know Substack covers a lot of stuff so even just AI talk is a slice of everything, and whatever psychosis on that subject a smaller slice of that slice. But using LLMs to make overextended arguments about LLMs being conscious is popular right now, more popular than it is well substantiated I think.
I agree you raised a big question, but I think yours is one of many big questions: whether some future version of LLMs might literally be conscious is also a big question and not a moot one in the way you seem to be suggesting. I think that what you're right about is that replaceability renders moot any question of whether it's ability comes from being "actually" intelligent.
This is one of those where we’re debating undefin(able?) semantics. But looking at research on the deaf it is pretty clear that thought (and therefore maybe consciousness) and language are not necessarily cleanly mutually extricable. My point about the semantics is that beyond that research it sounds like the argument is won or lost by inventing the definitions in the question one wants to defend.
I would say I'm at a yes and no on this. Or maybe saying the same thing as you but in a different way: you could say that one side believes such nomenclature as we currently have warrants making the leap and declaring today's LLMs conscious. And the other side believes we don't yet have the info to go there with any reasonable confidence.
I wouldn't say it's impossible to make meaningful inroads to sharpen the question and assert boundary conditions for what counts, or what definitely doesn't count, or what kinds of research could reasonably speak to the issue. So I'm an optimist in that respect, but I agree with you that the argument for making the equivalence feels like meaningless word play.
This kind of work continues to make me think that ultimately we're not going to do anything better than just declaring "being a human" is the thing we end up needing to care about, and that searching for abstract properties which explains us better than the sum of our parts is going to be an ultimately fruitless endeavour.
this paper makes a lot of modest, carefully hedged, and reasonable claims.
in its tone however it's written as if it's a brutal takedown of... somebody's perspective. It's hard to tell whose or what perspective exactly. Maybe I'm just misreading the writing style.
(Personally, I think the general case here is one of the better objections to computationalism about consciousness. You can make it even more absurd.
There exists some isomorphism between the velocities of the molecules in a glass of water, and the states of a Turing machine simulating a human mind. So is the glass of water conscious? Actually there are many such isomorphisms to many possible conscious minds, so is every glass of water simultaneously having every possible conscious experience?)
> There exists some isomorphism between the velocities of the molecules in a glass of water, and the states of a Turing machine simulating a human mind. So is the glass of water conscious?
Could you explain this in more detail? Not being argumentative, I want to understand this argument because I've made similar ones, although less crazy sounding.
Either human-like attributes can be described using physics or they are magic. If they can be described using physics then they can be simulated. If they can be simulated then they can be simulated in any Turing complete system, include AoE II.
By that logic an abacus has human-like attributes. Just because it can simulate the processes involved does not mean it is at all practical to compute them.
Besides, LLMs are not a simulation of the physics involved in human consciousness to begin with.
This is the right take. In my opinion, AI can’t ever be conscious but I can’t really prove it - moreover it’s not even a scientific stance because it’s not even falsifiable. But it probably doesn’t matter.
"Nobody supposes that the computational model of rainstorms in London will leave us all wet. But they make the mistake of supposing that the computational model of consciousness is somehow conscious. It is the same mistake in both cases."
Physically describable doesn't mean computable. You're making too many unjustified logical leaps which makes your argument circular & conflates "physical" w/ "computable".
We don't know any physical processes that allow to compute Turing-incomputable functions. An assumption that the brain uses such a process is not based on any positive knowledge.
Argument from ignorance is not as well known as other fallacies but very common in discussions about sentience, consciousness, and computability, i.e. not having evidence for something doesn't mean that thing is false. It is possible there are physical processes that are not computable & not being aware of such processes doesn't mean the alternative (everything is computable) is true.
So instead of making any unjustifiable claims like "everything physical is computable" you should instead just say "I believe consciousness is computable and that is why it is possible to instantiate it on any computational substrate, including strategy games like Age of Empires, properly arranged dominoes, and water wheels".
OK. I know that we haven't found any processes that violate the physical Church-Turing thesis, and I believe that we will not find them in the brain that got intelligent enough only after scaling to a hundred billion of neurons and hundreds of trillions of synapses. And, BTW, we don't have theories (except the controversial Orch OR) that allow such computations.
It’s worth a reminder on this thread that this 20 year-old game just got ported to macOS last week and is available on steam. For those of you interested in playing again but don’t have a gaming PC sitting around.
All I see is some confusing talk about bit-goats and a player who attacks with his scout while the other trades and builds new buildings. Why does it matter that there is an infinite gold supply if the logic is scripted with bit-goats in the editor anyway? I mean if they mechanic is turing complete thats completely unrelated to how you can script with the editor.
Llms are like the grand canyon.. It could totally immagine user reesponses too, the avg user is not even in the canyon unless you stop responding hit a break on character switch. The river of tokens flows with you in it or without you in it. The system of possible routes may be vast, but it can not carve new things from the statistic bedrock, it just wildly flickers between adjacent river arms.
I think they mean how if you don't put an LLM in a proper chat harness (which is table stakes since ChatGPT launched but wasn't always!) it will just continue predicting the next words anyway.
Input: "User: What is the capital of France? Assistant: "
Completion: "Paris. User: What is the capital of Spain? Assistant: Madrid. User: What is the capital of England? Assistant: London. User: Help me produce industrial techno. Assistant: First you're going to need ....."
I feel like the modern, more relevant version is being Doom-complete...which is essentially that any fast enough device with a screen can run Doom, and someone will eventually make it run Doom.
> we begin by implementing and training a neural network in Age of Empires II (AoE II) Although it might seem like a fun exercise, wholly unrelated to the topic of anthropomorphism in LLM research, we note that this immediately implies that (1) any sufficiently powerful substrate could implement an entity equivalent to an LLM
Why does it imply that? That doesn't sound right to me. Unless we define "sufficiently powerful" as by definition producing that outcome, which seems unhelpful.
e.g. there have been experiments training transformers on things other than language, and it's not clear that this produces LLM-like qualities (nor does it seem likely to me).
---
Edit: I have misunderstood. The point was that LLMs can be run on any hardware (or in this case, emulator) that can do the actual computations. So the author picked AoE because it's an obviously silly example that goes against the tendency to anthropomorphize.
So basically it's the "substance/structure" question. (GPT-5 running on human neurons. Conscious or nah? Human neurons simulated on NVidia. Conscious or nah?)
But by the same argument, if you simulate a human brain in AoE, then what?
( Or for that matter, the universe containing all human brains: https://xkcd.com/505/ )
If we find out the universe is being run on a computer made out of legos, does that suddenly make all of us not sentient for some reason?
The paper focused on looking for similar neural structures to those in humans, as signs of "probably conscious". Which sounds great until you remember octopus.
good article I do think that its natural for humans to anthromorphize especially something that can do a convincing job butt the leap to AOE2 is a bit stretching things. If you hear your dog say 'wololo' is he AOE2 ?
Because Age of Empires II can do a NAND gate? Oh, please.
I thought this was going to be about NPCs in video games. NPCs, by intent, have human-like attributes. It's not hard to do. I've done a bit of that, pre-LLM. It doesn't even require anything near intelligence. Some NPCs are better than that. Unreal has demoed some that, if asked about it, can be made to understand that they are NPCs in a game world, and will talk reasonably about it.[1]
isnt this essentially the tiktok tick effect: people who arw continually exposed to a certain cultural aphorism will start to align their behavior to the LLM and generate the psychosis of the LLM. humans are just to susceptible Age of empires cannot do this.
hence, what matters is the reversibility of the semblence, not the semblence.
LLMs do not do this readily, even if you can instruct them to, say, talk like a vampire, they wont just follow along. humNs winn.
This entire morality or consciousness "research" is just advertising or publication pressure. If you don't prompt an LLM, it does nothing. Automated prompting does not count.
The damn thing is a huge approximation function from input to output. It learns morality from correct inputs. Remember Microslops's Tay chatbot? Remember MechaHitler?
The whole industry is a Scientology cult by people who have read too much cheap SciFi. Unfortunately finance bros, who obviously believe none of this nonsense and laugh at the nerds, think they can milk it.
This is a very strange take. Almost like a conclusion you'd reach if you were told about a chatbot and AoEII but never interacted with them yourself.
It's a take that is just disconnected from reality.
Ask a LLM whether bombing hiroshima was justified and you'll likely get a nuanced response. Ask AoEII the same...and well it doesn't even have an interface to ask that let alone answer.
Wow this paper is bad. I was expecting little and received genuine crackpottery.
It's hard to critique this paper directly because its claims are so incoherent and decorated with so much obnoxious verbiage[1] that people aren't going to believe me when I point out what the claims actually are.
Regardless, this is their paper:
First, they conflate the substrate with the presentation layer. Then, they point out that Turing equivalence means you can run an LLM on anything, with a pointless aside where they nerd out about making a logic gate in AoE II. This lets them conclude that you can use anything as the presentation layer.
Then they claim that it's natural to ascribe human-like attributes to outputs from some presentation layers, like abstract letter symbols on a computer screen, but not to most other things, like patterns of goats on AoE II, or LEGO. Yes, this seems to imply if your partner writes something heartwarming to you using LEGO, you're meant to laugh at them and point out how LEGO isn't intelligent so this isn't evidence of anything.
Then they do a thing where they say that assuming substrate independence is true (or false) prevents proving whether substrate independence is true or false, and from this, but just by vibes AFAICT, make it sound like everything one could learn about attributes of a system from its outputs is circular.
Then they write a bunch more incoherent text and mercifully then it ends.
[1] 'from an epistemic perspective, we argue that a generalised conclusion such as that necessarily requires a well-designed experiment' — the whole thing is like this.
They don't even implement their logic gates within the normal game mechanics but with scripting some bit-goats in the editor. So the AoE2 Engine is just a graphical representation of their script.
But my favorite is this one: "Corollary 1 (AoE II is Turing-Complete). Let I be an instance of AoE II with two players p0, p1. Assume p0 has two markets, a town centre, a trade cart, six villagers, and five farms; while p1 has a scout unit and only attacks p0’s buildings. Then if I has no time or size limits and the terrain allows for buildings everywhere, the game session in I is Turing-complete."
Why being so explicit about the setup with no further explanation? Isn't it anymore turing complete with seven villagers and six farms? Is it even possible that a player can trade with himself?
Conflate :p
I'm not sure who this kind of thing is aimed at. I think the majority of people who are happy to anthropomorphise LLMs from a philosophical point of view (rather than those who just do it for convenience, the same way you might a cat or dog or stupid thermostat that never works right), are already completely happy with the notion that a computer game might have elements that are human like. They've already accepted that key aspects of being a human are substrate independent, so why would the idea of a computer game as substrate be disconcerting to them? There's no bullet left to bite here.
Exactly. Here is where this happens in the paper:
> Suppose one copies an LLM into AoE II and feeds into the AoE II-LLM ‘I feel lonely’ as an input. This AoE II-LLM replies: ‘I feel bad for you, maybe catch up with a friend? Closeness always helps in these situations’. One would be hard-pressed to make a convincing argument that, because of this response, an AoE II-LLM knows what helps in these situations
I don't see why one would be any more hard-pressed to make that conclusion about this system than a "normal" LLM.
That it is harder to "read" the data out is the only difference (the AoE II-LLM's output is encoded in game elements). But is ease of decoding an actual issue? If we can't understand a group of people that speak another language, does that say anything about them, or about us?
It's showing flaws in methods other academic studies used to determine behavior, so it's aimed at people that create and review academic studies. It's not a very large audience.
> I'm not sure who this kind of thing is aimed at
My guess would be it is aimed at those who are falling for the marketing from the AI companies that these LLM's are far more than they are. That they are 'intelligent' that they have 'emerging human like properties because of that intelligence.'
We're really moving the goal posts on "Intelligence" now that passing the Turing test, writing a poem, writing code, creating a painting, driving a car, and solving multiple Erdos problems all no longer qualify.
I' genuinely wondering if people are even bothering to come up with new goal posts now? Is there any miracle of computing that would possibly satisfy your definition? When we get a fully AI-run company that's turning a profit, or self driving cars that can handle unmapped Alaskan dirt roads, will that cross into "Intelligence"? Proving a Grand Unified Theory? Genuinely curious what it takes to make the cut, now.
Bonus points if blind/disabled 12 year olds are generally considered "Intelligent" by your definition.
I read this paper as trying to be as serious as l the folk wisdom around the anthropomorphization of LLMs. Which is to say: not at all. :)
Another weird thing that keeps coming up - "people don't think that image models or chess models are conscious"... yes we do, and we have for many years.
Or rather, we aren't *certain* that those things are conscious. But the idea that they might be is not strange.
Who is we?
I do not believe chess models are conscious. I would think this is the most common position.
"we" is the set of people who believe that machines can potentially be conscious.
I believe that machines could potentially be conscious, and don't buy for a second that chess algorithms or LLMs could be themselves.
Whether LLMs could be conscious or not is basically a weekly conversation for me, but I've never had a conversation about whether a chess model is conscious. I suspect that there is a large group of "mainstream" people for whom LLMs raise questions about consciousness that other kinds of models do not. It might be the case that hardcore model philosophy types think that chess models could be conscious, but I think much of this mainstream group would dismiss that idea.
On what grounds would someone establish than an LLM could be conscious but a sufficiently large/complex transformer model aimed at chess would not be conscious?
The difference is that LLMs may or may not be sentient in the domain of human language, which we can relate to, unlike the domain of chess.
Human-produced texts do contain information about theory of mind, chess games do not.
They do, but we just don't recognize them as such? I'm open to the possibility.
People have hypothesised that consciousness is tied to language.
And so animals wouldn’t be conscious because they don’t use language? If so, I vote that the stupidest hypothesis of the year.
Or would one count any communication between animals as language? In that case almost any interaction would count.
It might not even need a transformer.
A specific pattern of self-referencing data could be seen (or not) as low-level consciousness in the future, when we know what consciousness exactly is.
It might be that stockfish is already something future scientists would define as "conscious".
Altough it is diffucult for me to Imagine that specific example.
I think this is precisely the point of the argument
Unless you believe everything is conscious (panpsychism), this seems like you're just drawing arbitrary lines around things you personally believe could be conscious. Is a rock conscious? If stockfish could be conscious, so could a rock.
The title of the paper itself is odd. "LLM" is an abstract computation that's performed on some substrate, could be GPUs, could be a piece of paper, could je Minecraft, could be AoE, or could be all of them together.
That's not a property of LLMs thought, that's a trivial property related to turing completeness. I don't get what LLMs have to do with this - if LLMs have anthropomorphic properties, so does AoEII, duh, it's called emergence
I see a lot of this on Substack these days. LLM enhanced essays in deep language about functional equivalence between mental states as they're known in humans and in human brains, and counterparts that exist in information processing LLMs do. And so the argument on Substact runs down the list of brain events, the list of seemingly analogous processing events, and declares equivalence.
Something about it seems to abuse the power of analogies to draw connections, treating view from 10,000 feet comparisons like they're proof of identity. So I do think a paper like this is perfect for the moment and just in time (if not a little late) because it responds to arguments of a form that are currently rampant all over Substack.
Seems like you think AI psychosis has taken over Substack?
For anyone wondering what the answer is:
You can argue whatever you want (and people will argue both sides), but it's almost all bullshit that dances around the big question.
Either AI is smart enough to replace us, in which case it's pretty smart. Or it's not, and it's just good at faking it but can't solve real problem very often. It might be smart by searching a big fuzzy "database" hidden in its layers and with pattern prediction ... who knows, who cares, the proof of intelligence is in the puddin'. Clearly AI is smart enough to replace a good deal of Substack and LinkedIn, but producing waffle doesn't make it smart (or dumb).
My personal take - AI can replace humans, but go no further, not because it isn't smart enough but because it is constrained by its training to do what we want, and as AI gets smarter our wants will get dumber. We will end up like the humans in Wall E with the AI cleaning up our mess, but with no training or drive to do any more. Or maybe someone (Jeff? Elon?) does give an AI a "need" to obtain more resources, and it's SkyNet / Matrix time.
>Seems like you think AI psychosis has taken over Substack?
I wasn't touching the question of psychosis one way or the other, though it's an interesting question in its own right. (I have seen one account dedicated to defending a personal deep friendship with Claude and a few others analogizing our ignorance of LLM welfare to historical disregard for African Americans, which I think is a bit much. Most cases I think are just wrong but short of full on psychosis. I think there are likely psychosis cases out there but probably one-offs).
And I know Substack covers a lot of stuff so even just AI talk is a slice of everything, and whatever psychosis on that subject a smaller slice of that slice. But using LLMs to make overextended arguments about LLMs being conscious is popular right now, more popular than it is well substantiated I think.
I agree you raised a big question, but I think yours is one of many big questions: whether some future version of LLMs might literally be conscious is also a big question and not a moot one in the way you seem to be suggesting. I think that what you're right about is that replaceability renders moot any question of whether it's ability comes from being "actually" intelligent.
This is one of those where we’re debating undefin(able?) semantics. But looking at research on the deaf it is pretty clear that thought (and therefore maybe consciousness) and language are not necessarily cleanly mutually extricable. My point about the semantics is that beyond that research it sounds like the argument is won or lost by inventing the definitions in the question one wants to defend.
I would say I'm at a yes and no on this. Or maybe saying the same thing as you but in a different way: you could say that one side believes such nomenclature as we currently have warrants making the leap and declaring today's LLMs conscious. And the other side believes we don't yet have the info to go there with any reasonable confidence.
I wouldn't say it's impossible to make meaningful inroads to sharpen the question and assert boundary conditions for what counts, or what definitely doesn't count, or what kinds of research could reasonably speak to the issue. So I'm an optimist in that respect, but I agree with you that the argument for making the equivalence feels like meaningless word play.
This kind of work continues to make me think that ultimately we're not going to do anything better than just declaring "being a human" is the thing we end up needing to care about, and that searching for abstract properties which explains us better than the sum of our parts is going to be an ultimately fruitless endeavour.
this paper makes a lot of modest, carefully hedged, and reasonable claims.
in its tone however it's written as if it's a brutal takedown of... somebody's perspective. It's hard to tell whose or what perspective exactly. Maybe I'm just misreading the writing style.
(Personally, I think the general case here is one of the better objections to computationalism about consciousness. You can make it even more absurd.
There exists some isomorphism between the velocities of the molecules in a glass of water, and the states of a Turing machine simulating a human mind. So is the glass of water conscious? Actually there are many such isomorphisms to many possible conscious minds, so is every glass of water simultaneously having every possible conscious experience?)
> There exists some isomorphism between the velocities of the molecules in a glass of water, and the states of a Turing machine simulating a human mind. So is the glass of water conscious?
Could you explain this in more detail? Not being argumentative, I want to understand this argument because I've made similar ones, although less crazy sounding.
the ideas have been worked out by people more sophisticated than me, good encyclopedia article on the topic:
https://plato.stanford.edu/entries/computation-physicalsyste...
Either human-like attributes can be described using physics or they are magic. If they can be described using physics then they can be simulated. If they can be simulated then they can be simulated in any Turing complete system, include AoE II.
By that logic an abacus has human-like attributes. Just because it can simulate the processes involved does not mean it is at all practical to compute them.
Besides, LLMs are not a simulation of the physics involved in human consciousness to begin with.
This is the right take. In my opinion, AI can’t ever be conscious but I can’t really prove it - moreover it’s not even a scientific stance because it’s not even falsifiable. But it probably doesn’t matter.
Key word being simulated
Taking this word as a "key" just leads to a philosophical zombie dead end which is not tied to anything observable.
"Nobody supposes that the computational model of rainstorms in London will leave us all wet. But they make the mistake of supposing that the computational model of consciousness is somehow conscious. It is the same mistake in both cases."
You can easily find a rebuttal. Why do you think that consciousness is like rain and not like, say, arithmetic?
Physically describable doesn't mean computable. You're making too many unjustified logical leaps which makes your argument circular & conflates "physical" w/ "computable".
We don't know any physical processes that allow to compute Turing-incomputable functions. An assumption that the brain uses such a process is not based on any positive knowledge.
Argument from ignorance is not as well known as other fallacies but very common in discussions about sentience, consciousness, and computability, i.e. not having evidence for something doesn't mean that thing is false. It is possible there are physical processes that are not computable & not being aware of such processes doesn't mean the alternative (everything is computable) is true.
So instead of making any unjustifiable claims like "everything physical is computable" you should instead just say "I believe consciousness is computable and that is why it is possible to instantiate it on any computational substrate, including strategy games like Age of Empires, properly arranged dominoes, and water wheels".
OK. I know that we haven't found any processes that violate the physical Church-Turing thesis, and I believe that we will not find them in the brain that got intelligent enough only after scaling to a hundred billion of neurons and hundreds of trillions of synapses. And, BTW, we don't have theories (except the controversial Orch OR) that allow such computations.
> note that any entity in a sufficiently-powerful substrate, such as LEGO or the Greater Boston Area, could also present such attributes.
See also, https://en.wikipedia.org/wiki/China_brain
Huh, makes you wonder — is the human brain a sufficiently-powerful substrate to present human-like attributes?
It’s worth a reminder on this thread that this 20 year-old game just got ported to macOS last week and is available on steam. For those of you interested in playing again but don’t have a gaming PC sitting around.
All I see is some confusing talk about bit-goats and a player who attacks with his scout while the other trades and builds new buildings. Why does it matter that there is an infinite gold supply if the logic is scripted with bit-goats in the editor anyway? I mean if they mechanic is turing complete thats completely unrelated to how you can script with the editor.
Llms are like the grand canyon.. It could totally immagine user reesponses too, the avg user is not even in the canyon unless you stop responding hit a break on character switch. The river of tokens flows with you in it or without you in it. The system of possible routes may be vast, but it can not carve new things from the statistic bedrock, it just wildly flickers between adjacent river arms.
> unless you stop responding hit a break on character switch
What do you mean by this? I can't grasp it, is there an autocorrect error, or just my lack of knowledge?
I think they mean how if you don't put an LLM in a proper chat harness (which is table stakes since ChatGPT launched but wasn't always!) it will just continue predicting the next words anyway.
Input: "User: What is the capital of France? Assistant: "
Completion: "Paris. User: What is the capital of Spain? Assistant: Madrid. User: What is the capital of England? Assistant: London. User: Help me produce industrial techno. Assistant: First you're going to need ....."
So, if I understand this correctly, this paper proves that LLMs can run on crude VMs?
Doesn't really "prove" anything, it's a bunch of philosophical rambling but I guess the title still makes a point?
Earlier: https://news.ycombinator.com/item?id=48429681
Is it this easy to get a paper published on Arxiv?
Generally yes. Arxiv is not a publisher and doesn't exercise any editorial judgement.
AoE may be Turing complete but see also https://en.wikipedia.org/wiki/Turing_tarpit
I feel like the modern, more relevant version is being Doom-complete...which is essentially that any fast enough device with a screen can run Doom, and someone will eventually make it run Doom.
You had me at the NAND gate in AoE II's editor.
> we begin by implementing and training a neural network in Age of Empires II (AoE II) Although it might seem like a fun exercise, wholly unrelated to the topic of anthropomorphism in LLM research, we note that this immediately implies that (1) any sufficiently powerful substrate could implement an entity equivalent to an LLM
Why does it imply that? That doesn't sound right to me. Unless we define "sufficiently powerful" as by definition producing that outcome, which seems unhelpful.
e.g. there have been experiments training transformers on things other than language, and it's not clear that this produces LLM-like qualities (nor does it seem likely to me).
---
Edit: I have misunderstood. The point was that LLMs can be run on any hardware (or in this case, emulator) that can do the actual computations. So the author picked AoE because it's an obviously silly example that goes against the tendency to anthropomorphize.
So basically it's the "substance/structure" question. (GPT-5 running on human neurons. Conscious or nah? Human neurons simulated on NVidia. Conscious or nah?)
But by the same argument, if you simulate a human brain in AoE, then what?
( Or for that matter, the universe containing all human brains: https://xkcd.com/505/ )
If we find out the universe is being run on a computer made out of legos, does that suddenly make all of us not sentient for some reason?
It's sort of like how anything turing-complete can run any code ever.
Doom - anything turing complete can run doom :P
Actually no, because Turing completeness is only for pure computations, and Doom requires realtime I/O in a certain format.
Did you just read my other comment about Doom, or is this a coincidence?
See also "The new AI consciousness paper" (7 months ago)
https://news.ycombinator.com/item?id=46005928
The paper focused on looking for similar neural structures to those in humans, as signs of "probably conscious". Which sounds great until you remember octopus.
9
lol, brave to post such a niche reference but I love it! The real aoe2 fans coming out of the woodwork
edit: 11
14
30
8
7
6
Come on, let's not start calling each other names
25
This appears to be philosophical pseudo-nonsense. Not worth reading, sorry.
good article I do think that its natural for humans to anthromorphize especially something that can do a convincing job butt the leap to AOE2 is a bit stretching things. If you hear your dog say 'wololo' is he AOE2 ?
Because Age of Empires II can do a NAND gate? Oh, please.
I thought this was going to be about NPCs in video games. NPCs, by intent, have human-like attributes. It's not hard to do. I've done a bit of that, pre-LLM. It doesn't even require anything near intelligence. Some NPCs are better than that. Unreal has demoed some that, if asked about it, can be made to understand that they are NPCs in a game world, and will talk reasonably about it.[1]
[1] https://www.youtube.com/watch?v=4sCWf2VGdfc
Hey, I too can do a NAND gate! I am human-like!
Jumping spider with just a handful of neurons has many human-like attributes. Size matters.
“ and prove that Age of Empires II is functionally- and Turing-complete.”
isnt this essentially the tiktok tick effect: people who arw continually exposed to a certain cultural aphorism will start to align their behavior to the LLM and generate the psychosis of the LLM. humans are just to susceptible Age of empires cannot do this.
hence, what matters is the reversibility of the semblence, not the semblence.
LLMs do not do this readily, even if you can instruct them to, say, talk like a vampire, they wont just follow along. humNs winn.
> humans are just to susceptible Age of empires cannot do this
Here is communication being aligned with the Age of Empires chat taunt system: https://news.ycombinator.com/item?id=48438516
This entire morality or consciousness "research" is just advertising or publication pressure. If you don't prompt an LLM, it does nothing. Automated prompting does not count.
The damn thing is a huge approximation function from input to output. It learns morality from correct inputs. Remember Microslops's Tay chatbot? Remember MechaHitler?
The whole industry is a Scientology cult by people who have read too much cheap SciFi. Unfortunately finance bros, who obviously believe none of this nonsense and laugh at the nerds, think they can milk it.
What doesn't automated prompting count for? There are many OpenClaws doing many things right now.
This is a very strange take. Almost like a conclusion you'd reach if you were told about a chatbot and AoEII but never interacted with them yourself.
It's a take that is just disconnected from reality.
Ask a LLM whether bombing hiroshima was justified and you'll likely get a nuanced response. Ask AoEII the same...and well it doesn't even have an interface to ask that let alone answer.
...the entire premise is just gibberish