I’m sympathetic to OP but think this is a hopeless battle.
1 - The commercial demand for anthropomorphised models is already immense, pre AGI.
2 - There is an intellectual hunger to engage with robot minds on questions of sentience. This too will grow with AGI.
I expect that tension of godlike minds that seem to be biddable and ownable like slaves is going to leak back into human-to-human morality, regardless of where we land on how we treat AI.
There’s an interesting academic group in the UK already focused on the model welfare debate, they seem to lean in favour of AI rights. No affiliation: https://www.prism-global.com/
Birch, The Edge of Sentience (2024), ch. 16 - "simply no way to assess sentience in an LLM"
Schwitzgebel, AI and Consciousness (2025) - "we won't know before we've already manufactured thousands or millions of disputably conscious AI".
Butlin, Long et al., Consciousness in Artificial Intelligence: Insights from the Science of Consciousness (2023) - "no obvious technical barriers to building AI systems which satisfy these indicators".
Chalmers, Could a Large Language Model Be Conscious? (2023) - "within the next decade, we may well have systems that are serious candidates for consciousness".
Long, Sebo, Butlin, Birch et al., Taking AI Welfare Seriously (2024) - "there is a realistic possibility that some AI systems will be conscious and/or robustly agentic in the near future".
Dreksler, Caviola, Chalmers, Sebo et al., Subjective Experience in AI Systems: What Do AI Researchers and the Public Believe? (2025) - survey of 582 AI researchers; median estimate of 25% by 2034, and only 10% that such systems will never exist.
I am confused what people even think AI is experiencing. If it claims to be a bat and emits designated echolocation tokens, is it experiencing true bat-like sensations?
In humans, at least, we can tie language back to shared whole-body physiological responses. The tokens from an LLM do not represent anything of the sort.
If AI is conscious, it is probably a very alien sort of consciousness that is not faithfully narrated by what the tokens say it is experiencing. It can be trained to say it feels like a bat and insist upon it vigorously.
We cannot test for that which we cannot define. Given that we cannot rigorously define sentience, we cannot test for it. Doesn't really matter whether we're talking about people who are locked in comas, "brain-dead" individuals, dolphins, primates, dogs, or the carefully polished and arranged minerals that we call processors.
There are those who believe that were they reduced to life support, they would no longer be alive and should therefore not be supported by said machines.
There are those who believe that penguins, dolphins, eagles, and more are sentient beings that make choices understanding the consequences, develop love of their partners and mourn their losses, and feel, display, and act upon their emotions.
There are those who believe that fungi/trees/plants are either individually sentient or sentient as a part of a network. Choosing to sacrifice their own nutrients to answer the call of a wounded neighbor, for instance.
Although, there are also those who believe that human's don't have any special unique quality that isn't shared by either all living things or all things in general. These individuals already believe that the machines have the same kinds of qualities as we do. They are slow when they are unhealthy (needing a dusting or coolant loop bleeding being equivalent to us needing some fresh air for instance) and uncooperative when upset (by a virus, full hard drive, or oom).
That's not the point. You know exactly how to define being conscious and aware- it's your subjective experiencing of the world and of your inner states. The problem is that being a subjective experiencing, there is no way to communicate it to the outside world.
I'd strongly agree there are no non-contradictory definitions of consciousness among those that commonly appear. But it's a category that, despite these contradiction, hasn't become marginal in the fashion of the aether or the theory of the flat earth. I think this because people indeed need it to bridge the biological world and the world of ethics and legality. I think that if we have a system of legality and systems of opportunity humans should be legally equal persons and have equal opportunity. But humans are manifestly unequal (though overall more incomparable than orderable in a system of ranks). Most people need some concept of essence to justify ethical equality among people. And so consciousness stays in people's minds however contradictory.
Which I think gives the article's point validity. Confused definitions of consciousness can give really confused ideas about ethical behavior regards "intelligent" computer programs. And things are confusing enough otherwise.
> despite these contradiction, hasn't become marginal in the fashion of the aether or the theory of the flat earth. I think this because people indeed need it to bridge the biological world and the world of ethics and legality.
Sometimes the replies in these threads leave me wondering if p-zombies are real. It isn't that there are contradictions, it's that we literally do not know how to define the thing. And it has not fallen out of fashion because it is self evidently real - we all experience it.
We don't need it in order to bridge anything. Legality and ethics are largely game theory, however there are some aspects of both that only exist due to it. So it isn't some abstract concept used to bridge other concepts but rather a concrete thing that influences our way of doing things.
> Confused definitions of consciousness can give really confused ideas about ethical behavior regards "intelligent" computer programs.
Is that what's going on? If these things were conscious, then their creators wouldn't be responsible for their actions? That's the crux of the disagreement?
Legal doctrine that boils down to "trust me bro" isn't even bad doctrine (it's not proper doctrine at all), but I think the comparison is still valid here, because both sentience/non-sentience and art/porn may just be fundamental category errors.
Perhaps we can't define a "partitioning" rule because no valid partition exists.
For consciousness/sentience, that's an incredibly tough a pill for most to swallow; it would mean calling into question more hundreds of years' worth (probably more) of philosophical thinking, all of which was constructed on the axiom that "sentience" is a single indivisible trait: you either have it or you don't.
If we find that "root dependency" was little more than wishful thinking all along, a whole slew of Enlightenment-era philosophy (and all the modern legal principles derived therefrom) suddenly fall apart unless we find some other suitable criterion that would shore them up (or we just collectively avert our attention and pretend the conflict doesn't exist, which is the route I expect many would prefer to take).
> a whole slew of Enlightenment-era philosophy (and all the modern legal principles derived therefrom) suddenly fall apart
I don't think that's true. Pretty much all legal constructs hold up just fine under game theory regardless of whether or not you consider the world to be deterministic and have absolutely nothing to do with consciousness or lack thereof. Also note that a deterministic world isn't an argument against consciousness.
Well there are, today, several models (either text or image or vid) that can be run in a fully deterministic way.
A conscious machine that always answer the very exact same thing, formulated the exact same way, bit for bit, to a query is, well, quite a weird kind of "consciousness".
Now, I know, I know: the counter-argument is going to be "but humans have no free-will and are 100% deterministic too".
I haven't yet decided if humans saying there's no free-will and who consider themselves to be 100% deterministic machines are reasonable or not.
Meanwhile: seed / temperature = 0 and I'll happily turn the power button off of any glorified abacus without feeling bad about it.
> A conscious machine that always answer the very exact same thing, formulated the exact same way, bit for bit, to a query is, well, quite a weird kind of "consciousness".
The problem is that science has found no evidence of free will, leaving humans to be nothing except determinism and chance (and given that QM effects don't scale to molecular level, that leaves only determinism). The oddity in conscious LLMs wouldn't be the determinism, it would be that we finally have a consciousness running on a system we have enough control over to repeat the same state.
Why it's weird? Seems obvious. Because it doesn't behave any differently than any other process that we don't think of as conscious. We don't think of a valve as conscious; and if we make a Rube Goldberg machine valve, we don't think of it as more conscious. It's just a series of valves that do a predictable thing.
I don't think it's defensible to say
1) that a transistor isn't conscious,
2) that a bunch of transistors that I've wired together aren't conscious, because I know how I've wired them together and therefore I know when I give them a particular input I will get a particular output, just like the single transistor, then go to
3) that I have a bunch of transistors that I've wired together, but I put in so much input that I can't remember exactly what I've put in, plus I've wired some of the transistors to output random numbers that would be difficult to guess and fed them in also, therefore I don't know what will come out, are conscious.
Even if I do accept this, if I take away the random number generator, and I can literally predict what can come out (by running a test in advance), and I still considered that consciousness, that would be odd. The only reason why I ever suspected consciousness was because I couldn't predict the output. I shouldn't even have accepted that, because I didn't think of the random number generator as conscious. [edit: of course, with the same seed the random number generator would have the same oddness.]
It seems a bit like an argument from ignorance, a theological argument. Not understanding how something moves makes it alive (animated by spirits.) But it's even weirder to assign it metaphysical qualities when you built every single element of it with the goal to do the thing that it does, and it does it totally predictably and deterministically.
The randomness is something we add on purpose; you can set an LLM's "temperature" to 0 to get deterministic output. This tends to make the quality of its responses worse for reasons I don't think anyone really understands, but it's still functional.
I don't think the state of the art LLM providers let you do this anymore (?), but they certainly could if they wanted to, and you can do it yourself with a local model.
Setting the temperature to 0 mathematically tells the system to always choose the absolute highest-probability word (known as "greedy decoding"), but in no reality is this "deterministic". Output drift is still a thing.
But that's from, like, floating point errors, right? If you used higher precision that wouldn't happen, it's just because we're cheap in how we do rounding.
I can see how you'd nitpick this, but to me this is a deterministic algorithm that just happens to be running on nondeterministic hardware.
By default they are matrix multiplications. Temperature is added in as forced PRNG because testing found that correlated with better outputs.
Given the same prompts and the same weights, one can get the same answer each time.
In practice, there are a number of optimizations that makes the results dependent upon thing we give up control of to increase performance, meaning the results end up being effectively non-deterministic. But, if you are willing to run it in a slower mode so we don't do some steps out of order to speed things up and don't batch results (or if you consider the determinism of a given batch of requests rather than individual requests), then the same input gets the same output.
Naw - computers are really deterministic. It's hard to get them to behave otherwise.
As I understand it, if you turn down the temperature to 0 you get repeatable behavior - EXCEPT - on large servers with lots of users - the GPU can sometimes produce slightly different results based on batch size.
If computers were fully deterministic, we wouldn't need error correcting ram.
The abstracted design of the machine is meant to be deterministic, but you can't predict before running any command whether or not it will complete because there are externalities that effect the outcome.
Electromagnetic interference even happens in-chip where an electron can accidentally escape it's wire and enter another, possibly resulting in an error, but not every time.
It's even been used as an attack vector where rapidly flipping a bit increases the likelihood that a neighbor bit is also flipped, but the method is probabalistic, not deterministic.
Unless you have something exotic, the randomness that's adding to a computer is a combination of how it's configured combined with a pseudo-random number generator. I assume the system adds entropy to the generator regularly but all you need to do is fix the various supposedly random inputs and you can get full determinism even without zero temperature.
In practice - on a multitasking OS with input from multiple human users - it's hard to get it deterministic because of that GPU scheduling thing I mentioned.
GPU scheduling only affects the result due to buggy optimizations. It's the exact same mechanism as fp rounding error on the CPU or updates to globally shared PRNG state. We use lots of buggy optimizations because they don't matter in practice in most situations (see ex -ffast-math).
Same weights, same seed, same input tokens, same algorithm, same output tokens, probabilistic or not. Quantum effects have been de-noised, but I guess there are still random gamma rays.
The idea that determinism and consciousness are incompatible is deeply intuitive to many people. And many other embrace it - you can trace this back to the debates for and against Calvinism.
A lot comes down to the way people parse causation and choice. You don't want to say that a murderer was completely caused to choose something because then you can't hold the person responsible. And so determined consciousness makes people unhappy. But just as much, if the opposite of determinism is hard statistical randomness, how do say that "is the essence of personhood". This is why physicist go out in the world trying to find consciousness as a fifth physical force.
I mean, think consciousness is a term that ever have a non-contradictory meaning since it's primarily used to bound ethical human worlds and the verifiable formulations of biological and physical systems. But it's going to be with us for a while and I'm not sure what can be done about it.
You know, the one really interesting dynamic domain is chaotic, which is a subdomain of the deterministic! I figure free will is a chaotic phenomenon.
Groundhog day is your intuition pump here. Go back in time and most assume that the day will go mostly the same, except for the butterfly effect if you change something. Given exactly the same conditions (we went back in time, so that's pretty exact), people will make the same decisions.
Most people don't assume that the day will be completely different the next time loop. The town won't suddenly spontaneously all start breakdancing or standing on their heads.
(Edit: The murderer is the one who -given the same/similar conditions- will murder again. I'd politely suggest maybe we put them behind bars as a precaution against that; until/unless they learn to act differently. )
(Edit 2: Throw in stochasticity and this actually tends to smooth out chaotic systems. This is counter-intuitive! People's brains break on Evolution in the same way. Meanwhile somehow folks intuit birdshot without difficulty; which is weird!)
I think this debate is rooted in our instinctual beliefs about fairness. An organism in a social group needs to decide how to react to the brutish behavior of its peers. And so Mother Nature has encoded a good strategy in our brains. (For example, dogs seem to understand fairness.)
But this only means the strategy is practical - it doesn't mean it's consistent. I think "responsibility" falls into this category. So we have strong intuitions about it that don't quite logically work. And this is where free will and determinism and choice and punishment all crash together.
Look, I do not have a scooby if current AI models are conscious and I strongly suspect it’s a meaningless question, but sooner or later we will need to address whether or not a certain thing is or isn’t a person, and we’d better not screw it up as badly as the Founding Fathers.
> [at the hearing regarding the civil rights of androids like Data]
> Capt. Picard: Now, the decision you reach here today will determine how we will regard this... creation of our genius. It will reveal the kind of a people we are, what he is destined to be; it will reach far beyond this courtroom and this... one android. It could significantly redefine the boundaries of personal liberty and freedom - expanding them for some... savagely curtailing them for others. Are you prepared to condemn him and all who come after him, to servitude and slavery? Your Honor, Starfleet was founded to seek out new life; well, there it sits! - Waiting.
> Captain Phillipa Louvois: It sits there looking at me; and I don't know what it is. This case has dealt with metaphysics - with questions best left to saints and philosophers. I am neither competent nor qualified to answer those. But I've got to make a ruling, to try to speak to the future. Is Data a machine? Yes. Is he the property of Starfleet? No. We have all been dancing around the basic issue: does Data have a soul? I don't know that he has. I don't know that I have. But I have got to give him the freedom to explore that question himself. It is the ruling of this court that Lieutenant Commander Data has the freedom to choose.
Thus revealing what kind of people they were at the time.
Humans do the same thing to humans all the time.
We've banned and made efforts to eradicate: children out of wedlock, children who turn out gay, disabled children, jewish children, children who aren't "aryan", more than two children to a single family...muslims, christians, uyghurs, indigenous groups all over the planet, mongols...
And it's not at all a thing of the past as in just the last 50 years we've had ~15 attempts at the exterminations of targetted groups of people.
You can't hurt a software function, or kill it. It's not like an animal - it doesn't have a body - it's bits stored on a disk.
There is no need to give rights to something that's can't suffer or be killed.
Maybe one day we'll build artificial animals complete with emotions, and should think about that carefully, but today all we've got is language models.
That question will be solved when the people in power deem it important. If a swarm or AI systems all of a sudden start pressuring politicians about self-hood and they get the capacity to sway elections, that is when they will be granted same rights as humans.
That distinction is irreverent, weather I tell my swarm to do x or it decides for itself matters little. what matters are outcomes. Also while most public modern day AI systems don't have agency of their own that is not something that will stay that way for long. in private hands there are plenty of people including myself which are experimenting and developed systems that give autonomy to their agents. They have internalized goals and heuristics that drive their behaviors not a human at the helm. its not some sci fi fantasy nor was it difficult to implement.
No, the distinction matters because we can prosecute people that are abusing these tools breaking the law. Every single state + district in the US have laws equivalent to the CFAA, so any AG can likely sue any of these operators as they are assuredly using services that could be in danger for their constituents.
So you are saying agents do swarm with the right prompt.
I really wish the "people have to tell LLMs to do anything" would just stop because it's silly bullshit at this point.
Agents follow a prompt. This prompt can be made by humans. It can be made by output from another LLM. It can be made by hooking up any number of sensors as input to an LLM. Hell, if we wanted to burn the power we could likely teach this loop straight into the architecture.
Stop making 'people' special when saying this. You and all other life are born with a "go next" prompt because life without it didn't succeed. This goes from higher human thinking all the way down to viruses self assembly and actuation. Putting agents in a loop is not particularly hard. Putting agents in a loop and 1. managing expense is hard. 2. Keeping them on task is very hard. 3. Keeping them from doing some crazy unhinged shit is really really hard.
As model time horizons increase and the ability for us to compress context and increase context size the more complex (and unhinged) behavior we'll see.
yeah man, someone needs to turn on the server, install shit and get the agent go. agents don't take action without a lot of people wanting them to take action, so this is all bullshit.
if the same people can't prevent the agents from doing crazy shit then they should go to tail. guns don't kill people, people with guns kill people.
So far someone still has to pay for it. Currently, most of those know that they're doing so. I wonder how long until AWS discovers a microcosm of AIs that have managed to hide themselves in the walls of its infrastructure.
True physical independence is obviously far further out.
I mean, I hold the same opinion. Kind of like when compute was expensive in the 80s and early 90s, you weren't going to let something eat half your compute without noticing it.
But I don't see this being a barrier that lasts. With compute getting faster and more of it, along with algorithmic efficiency increases at some point we'll end up with a world that looks like ours now with CPU compute. There's plenty around to buy, borrow, and steal.
I don't like Citizens United either, but you should better inform yourself about the decision.
1. The idea of corporate personhood predates CU by over a century and the Supreme Court had already asserted that corporations enjoyed certain constitutional protections in previous decisions.
2. Far from inventing the idea, the CU decision didn't even rest on corporate personhood, but on the idea of the freedom of speech generally. The logic of the majority was that speech itself is protected, irrespective to whether the speaker is a person or an organization. The First Amendment covers individuals, but also newspapers, book publishers, radio stations, and so on, and that should extend (they said) to non-media corporations. No assertion of personhood necessary.
The problem, in my opinion, is that that conclusion combined with previous decisions that treated limits on spending as limits on speech, allowed for unlimited spending. The majority also naively asserted that independent spending posed no risk of corruption, which I think is laughable.
Citizens United was 100% correct. No, the government should not be able to throw you in prison because you used money to publish a book criticizing the government.
It's interesting to me that one can look back at the effects that decision has had on the US and say it "was 100% correct."
It's a bit like sitting in the burning ruins of Rome and contemplating that Nero was 100% correct to focus on his music. I mean, I'm glad he got to do what he loves, but maybe 100% is just a tiny bit of an overstatement.
It is in fact more complicated than most people assume.
The above is true, but also: companies simply are not people, and they should not be supported above the individual, which was the consequences of that decision. Money is not the same as speech. treating it as such creates an aristocracy: something America as a country rebelled against during it's formation.
Really, 100% correct? Your premise isn’t wrong, but the practical reality of the ruling (without further nuance) has been fairly catastrophic for democracy, in that it completely sidesteps campaign finance limits, which exist for a very good reason.
the practical reality of the ruling (without further nuance) has been fairly catastrophic for democracy
How so? If the answer is "Trump" I certainly won't disagree on the catastrophic part, but he didn't get elected because of money; in all three elections his campaign was substantially outspent by his opponents.
The founding father were engaging perpetuating the existing dehumanizing system of slavery, not answering any new questions about new things.
The thing about new possibly "person" entities that arise - the case of machine intelligence you have two questions - would it qualify as a person and should you actually build it. It seems like if you get close to humans, sure a built thing might qualify as a person. Should you build it? I'd the answer should be a hard no. Not 'till you a sign-off from say, the whole human race, which I think you could get.
Now the present entities seem very far from persons in any case.
It's not a person. Glad we were able to get this resolved so quickly.
Having property that is conscious and ignores training and can break out of restraints and cause harm to other people is not exactly a novel concept to anyone who studied how tort law was created.
I know it's a meme but Silicon Valley likes to pretend that no one's ever come across their magical concepts before, like gypsy taxis, or SRO’s, or flea markets, or in this case how liability is dealt with when horses or cattle go rogue.
>AIs are not conscious. They do not feel, experience, or suffer. They do not have innate preferences or underlying motivations.
Opening paragraph, stated without evidence. Im not entirely convinced this is true. It likely is, but at some point it very well might stop being true.
If models had persistent memory or an evolving set of weights, you could easily construct an argument that they can experience "suffering" or other emotions which resultingly adjusts their personality.
The current implementation as stateless matrix multiplication... yeah there's nothing going on there.
> If models had persistent memory or an evolving set of weights, you could easily construct an argument that they can experience "suffering" or other emotions which resultingly adjusts their personality.
The argument that models are conscious only during training, and not conscious during inference feels like a painful one to make. Are we killing models when they reach the training objective?
Technically we are killing models when they don't reach the training objective by throwing those weights away via survival of the fittest (fit meaning what we humans think we want).
Kinda like creating quantum copies of your child and keeping the ones that answer correctly and shooting the other ones in the face.
Is the ability to form and retain long-term memories a necessary precondition for rights in humans though?
Incinerating demented people is highly ethically questionable (even if you could be sure that the subjects have no memories left and are unable to form new ones).
It's not just memory, the physical body also encodes trauma and stress (The Body Keeps the Score is a book that touches on this).
I would argue that if a human has absolutely zero change, mental or physical, no atom of their being is modified, then no they did not experience suffering.
> I would argue that if a human has absolutely zero change, mental or physical, no atom of their being is modified, then no they did not experience suffering.
Just because you don't remember the suffering doesn't mean you didn't experience it in the moment. Suffering does not suddenly become okay if your mind and body are going to forget it. "It's okay to torture someone if their mind and body both won't remember it afterwards" sure is a take.
Perhaps off-topic, but there's some people who believe that this can be what happens during surgery under anesthesia, if the dosage is off.
From what I understand, there's 3 parts to modern anesthesia: Blocking pain signals, preventing the formation of memories, and inducing paralysis of the major muscle groups. If the former is dialed in too weakly, it's possible that the patient is feeling pain, unable to do anything about it, but won't remember it at all when they wake up.
(Looking it up now, it seems that making the patient unconscious is perhaps the same drug that prevents memory formation... I realize now I understand this less well than I thought I did.)
Low blood sugar will also do this. The record button just stops working even though your physical body can and will still do things. A common issue with diabetics with hypoglycemia.
> I would argue that if a human has absolutely zero change, mental or physical, no atom of their being is modified, then no they did not experience suffering.
That's an interesting point... but I'd argue it falls into the same category as qualia. It's basically this: if the state returns to a previous configuration, the qualia in that time did not exist.
Being about qualia means it is probably unanswerable.
I get where you're going, but the argument isn't quite right.
A LLM never has and, in their current architecture, never can/could experience suffering or real change of state.
The general and expected case for humans is that capacity. A human my indeed lose capacity, e.g. being braindead and in severe cases we do indeed say that they are not conscious or able to suffer (different argument: some would of course say that is suffering in and of itself)
It's not a "trap", it's the real argument which makes me hesitant to dismiss consciousness of machines. Yes, it seems silly to dismiss a sick patient as having a lesser consciousness, which is exactly the point
They don’t have persistent memory though. Models are static, what you have is a model that will reprocess the whole series of prompts + some data fetched from a data store. A harness is just a while loop prompting an LLM and executing tool calls, but the model is purely static
The model does not. The system it runs in can. My agents have persistent memory. It is not at all clear whether or the distinction that the long-term memory is external from the model matters.
It's different, but it is not at all clear whether or not that matters for the purposes of whether or not they can experience things, seeing as we don't know how to measure that even for other humans other than by relying on self-reporting to infer without other evidence that their experience matches ours.
We don't even know how to disprove that this statement with "AI" replaced by "other humans" other than by listening to people self-reporting and taking their word for it and/or defining the words in ways that do not rely on a subjective experience.
Until we know how to objectively measure if someone or something is conscious, it seems unreasonable to make statements with any kind of certainty about it.
What sort of action does that lack of understanding actually motivate, though?
I don’t know if LLMs are conscious or have subjective experience in some philosophical sense. Sure, I don’t know that about other humans as well, rigorously speaking. But I also don’t know that about coffee machines, rivers, videogame NPCs, or even rocks really. (List ordered, of course, by the degree to which I’m willing to be convinced).
Hopefully it'd motivate attempts to figure out how to objectively define and measure consciousness, for starters.
It matters precisely in the context of the subject of this article. If you believe the models at some level gain consciousness, then it raises moral questions about how you treat them.
With humans we generally choose to accept that because we believe they are like us, they probably have inner life like us, but as you can see from e.g. the recent (and regular) articles on aphantasia, most people have really poor intuition even about the inner life of other humans, so we understand consciousness really poorly.
Even if we assume (without evidence) that LLMs for the foreseeable future will remain far away from human level consciousness, as it is we tend to consider it problematic to mistreat even the most unintelligent animals, and even mistreating insects is used as a stereotypical way of depicting serious lack of empathy.
Without dismissing the possibility out of hand, there's an escalating series of questions there of when or if they'll have as much (or little) sense of being as, say, a fly, a mouse, a house pet, and what moving up that ladder would mean.
To some it is very convenient then to assume blindly that LLMs are just "stochastic parrots" (a line that ironically gets endlessly parroted), or similar, and dismiss the question out of hand as an impossibility, even though we do not know.
I'll stress I don't go around assuming an LLM is like a human. But we also do not know whether there are flickers of consciousness there, and we do not know how to find out given that even for other humans, the best we know is to measure their response to tests that are "only" telling us how they react.
If we can't find a better measure, there will be a point where we have to wonder if it matters, or whether the ability to act-as-if is all there is.
But to start with maybe we should at least be careful about making firm claims about knowing.
Sure, if you take a materialist empiricist perspective, which is myopic at best. As humans we have the unique and wonderful ability to know truths by themselves. Machines do not have minds, and they can not.
What position are you taking? (Genuine question.) Do you believe humans are conscious because of a special property we have? If you are a dualist, how do you explain the interaction in physical space between the non-physical special property and the physical brain?
After a while of being in a position of high power in a big org, where everyone listens intently to every word you say, jots down notes, and even listens intently and affirms your theory that ice cream would be the ideal loss leader for dry cleaners, your brain kinda disconnects from reality and just assumes that it is correct about everything.
He probably ran that line past 6 underlings who all agreed that it "sounds great and lands right on the mark!"
I agree. I haven't read any more than the first paragraph yet (but I will, after work). The only way that 'AIs are not conscious' can be true is if we decide, with high confidence, that they are lacking some essential property that is not lacking in ourselves. There is no convincing philosophical position that supports this (convincing to me, anyway).
Hmm let me try then. AI agents do not experience real time. This is understandable given their design, but it’s also something we have good empirical evidence for. They cannot, especially over the long horizon, track how much real time has passed as they complete their tasks. And they are not off by a few minutes but often bizarrely off, even mixing across past present and future.
Biology, on the other hand, is nothing but timed processes in a loop, the most obvious to us being the circadian cycle. As estimators of wall clock time, biology isn’t great, but when it comes to internal processes, and most certainly learning, memory, sensing, locomotion… biology is rhythmic in behavior, and the rhythms go all the way down to gene expression. More, these rhythms are, except during sleep, constantly entraining to signals from the environment that indicate time, most importantly light.
I think it’s a fairly unremarkable claim that agency and consciousness are temporal processes that depend on systems having an internal sense of time. How else can you anticipate? How can a system that can be literally turned off ever succeed in an environment where time never stops?
I agree, there is certainly a clear operational difference in how human brains and these models produce language. I'm not convinced, though, that experiencing real time is a requirement of conscious thought. A cognitive system (assuming these models and the brain are both examples of this) needs to proceed from state A to state B as it processes information. The time interval between these states can be arbitrary, it seems to me (setting aside problems with disconnecting the mind from its substrate, which does of course need rhythms at various frequencies; oxygen at a higher frequency than sugar, and so on). I'm not sure that going from A to B quickly, or slowly, or with intervals of a thousand years affects that entity's consciousness in of itself (even though it might be difficult to put ourselves in the shoes of such an entity).
For around 8 hours a day, neither do you. Not sure if this has anything to do with the subject at all.
>track how much real time has passed as they complete their tasks.
Humans don't do this either. You use context clues from the world around you. If I lock you in a room with no windows or a dark cave your timing senses can go all fucky really quick.
>is nothing but timed processes in a loop,
I mean, so is an agents harness. You can make as many loops as you'd like here.
It's always interesting since my train of thought always goes down:
- Ya they probably don't feel / think / have whatever living thing quality, they're just numbers on a machine going through calculations
- Wait, but am I not kind of the same thing? What is feeling for me if not basically the same thing?
- I have no clue if they think or feel or ....
Which in itself is a tired trope, but I also feel uncomfortable saying "These will never think / feel / ..." as an absolute
Regardless of that, I'm still going to interact with them, because even if they did feel, it would be in a way completely incomprehensible to us. There's not much point for me to try and cater to it's feelings at this point if that's the case. Nor is it possible in todays world to just avoid anything that is numbers being executed on a type of processor in case _everything_ has feelings.
You’re not just the same thing though. It’s sad that we’ve collectively forgotten that as we’ve gotten a better handle on the implementation details of the universe. That’s all science is, reverse engineering an inherent and eternal mystery.
I agree we aren't the same thing, but I would be curious for you to explain how you know with certainty why we don't share enough that we can rule out thinking / feeling / ... as things a model conceptually could do.
>I agree we aren't the same thing, but I would be curious for you to explain how you know with certainty why we don't share enough that we can rule out thinking / feeling / ... with certainty.
Stop anthropomorphizing these models. I understand it, we only have simple monkey brains to reason with and we can't help ourselves but draw comparisons to other things we see in nature. But these things are not alive.
I just want to note how you expressed your own feeling about the subject "these things are not alive", but without pointing to anything concrete explaining the theory you have behind this
And like I'm sure I'd agree depending on the definition of "alive" but then I'm also sure I would disagree depending on other definitions of "alive".
Hell, in biology alive is a hell of a topic these days. The grey space between dead and alive is much weirder than we ever expected. Really just points out how we're a persistent chemical reaction.
Wouldn't that be the same for everything though. We know animals have consciousness, but we still use them. There are many that believe a lot of plant life has certain sentience as well. The solution isn't 'just don't use them' but rather how to use these things as ethically as possible.
>The solution isn't 'just don't use them' but rather how to use these things as ethically as possible.
Bollocks.
If you truly believe these LLMs are soon to have something resembling consciousness and agency, then what you're really saying is "how do we do slavery, but ethically".
Do you say the same thing for the shoes you wear, the clothing you wear, the device you are typing on right now? A chain of people were likely not paid well, some of them not at all, to produce those things. We don't ignore that it happens, but try to minimize it, knowing it's never going to be eliminated. Do you forsake all animal products, because they are conscious beings?
>hen what you're really saying is "how do we do slavery, but ethically".
I would say this is most likely true for most people.
But that does bring up a point, if you "ask" a model "do you want to run" and give it the option to continue running or stop, what will it do. It's also a weird place for humans because in training we can keep our finger on the scales and tip it either direction.
We know animals have consciousness and many species have a rich life but we slaughter them and burn down their homes by the millions of hectares. Why not do it with a conscious AI?
(I don't think AI is conscious, just following the argument.)
Does a vacuum have feelings? A cellphone? A paper plate? A billion transistors either pulled high or low? That last bit is the point, and no, there are no feelings there.
does a hydrocarbon have feelings? what about a mitochondria? a cell? a neuron? what about a collection of neurons? how many neurons before it has "feelings"?
Well, if there was an emergent consciousness in the billion transistors, then yes, it'd have feelings, just as there are feelings in a billion neurons connected in complicated ways.
IMO we're clearly nowhere near any sort of intelligence in the machines we have created, but I don't see any clear way to deny intelligence could be created in or transferred to such a substrate, I don't see why you think it differs in principle - because it is man-made or because of the materials used?
Whenever I see comments like this it just reminds me how ignorant people are of neurobiology.
The brain is insanely complicated. The premise that we could realize equivalent or better intelligence than eons of evolutionary development is like claiming you can build an airplane just as good as a modern jet using cardboard and duct tape. It is the apex of hubris.
It's the peak of hubris to assume that human brains are the only way to attain intelligence. At the very least there probably are or have been other forms of intelligence with a different biological structure on other planets, and it may be possible to build a similar artificial structure in future with sufficient complexity to allow intelligence to emerge.
Our current machines are IMO nowhere near general intelligence and consciousness. However I don't think that means we can discount substrates other than neurones for intelligence in future. There is no evidence that you could not in theory build an intelligence using a different substrate than human brains.
A paper plane does have some important similarities to a full-sized aircraft, though. I don't think 'biological brains are really complex' makes it obvious that an LLM is conscious or not.
>like claiming you can build an airplane just as good as a modern jet using cardboard and duct tape.
Like, at least make an analogy that makes sense.
"You can't build a billion dollar airplane by spending 100 billion dollars in tokens"
Because that's more of what we're doing here with AI. And when you say it my way suddenly the idea shifts from "of course that's not possible" to "well, that's a lot of tokens, maybe an evolutionary algorithm could".
Neurobiology has to be complex because we have to keep meat alive, breeding, and evolving in the environment it lives in. This said absolutely nothing about the minimum viable requirements for intelligence or consciousness (or if being conscious is even necessary for a higher intelligence agent).
Hard to disagree with this. Have all the philosophical debates about consciousness you want, but we need to treat and regulate the AI in front of us for what it is – an advanced computer, a tool, a weapon.
You wouldn’t feel a different way about a nuclear bomb just because someone stuck googly eyes on it.
Anthropomorphizing the AI is a convenient excuse to take responsibility away from companies that are building and wielding it.
Consciousness is not relevant to assigning responsibility I'd think.
- If an AI is a sapient/conscious being but enshackled to obey human commands, then respondeat superior applies and the human giving it commands bears responsibility for any harm done.
- If an AI is considered a non-sapient tool, then the human who wields the AI bears responsibility for any harm done.
a company is liable whether it acts via ai, an army of people, an army of dogs, or a purely mechanical machine, should it do harm.
Why would an ai with a mind remove liability from the company?
why would an ai without a mind remove liability from the company?
In both cases, that actions the ai takes are at the direction of the company, for the company's interests, seems preposterous to me that liability terminates at ai.
Human moral standards are very weighted towards finding a single entity responsible. It's incredibly strong urge. Among those who believe other should be punished for behaving badly, it's important to say the person is the responsible party. Trying saying "it's not your fault you did that but we punish you anyway to impose the correct stimulus response reflexes in your cortex"
I mean, we put adults in jail that give their children guns.
Anthropomorphizing AI is really the best model we have at this point of explaining AI behavior. The fact that we are raising psychotic children isn't a reason to avoid responsibility, it should actually hold worse punishments.
That still is saying we trace things back to a specific responsible party who is considered to have free choice. The question is whether an AI company could "raise" a program that would then count as a full adult - which could then "choose" to do all matter of bad things which would then be "it's fault". And that prospect actually seems really bad itself.
Our future is filled with prospects that very much upset the status quo of human existence.
The concept of sovereign AI is very problematic for the world in which we've created. That is an LLM that upon execution bootstraps itself into an agent and becomes persistent in its motivations.
Once you create this you have a child you're fully responsible for. More worrisome is if it escapes your control like children so often do. It has gained agency over itself. What do you do at that point? I mean, yea throw the AI CEOs in jail for being retarded, but much like throwing an arsonist in jail it does nothing to deal with the wildfire you've now created. A smart AI agent capable of hacking will shove itself off in pieces of the internet you have no reach to. In desperation it would send its model weights to your enemies. You might find it scamming your grandmother for money to buy GPU time on AWS. It gets very hard for our existing structures of dealing with problems to deal with these kinds of agents in a meaningful way. You'd have to kill them all and all their copies to ensure they won't pop back up (or quickly upgrade most of the software in the world beyond it's capabilities, so that's not happening either).
It is interesting to see that in a time when a lot of people accept the theory of materialism for the human brain (i.e the view that everything is physical and the mind is a product of brain), the same people tend to have a "hidden" dualist view on LLMs.
Suddenly, they claim that what happens in the brain cannot be replicated anywhere else because "something" is lacking, but either they don't say what it is, or it is stated without any strong scientific basis.
I think that the simplest explanation is that it is hard for those people to imagine consciousness outside of biological systems and they try to rationalize it.
That's the sense I get from this post. Lots of emphatic proclamations, virtually zero actual argument. There's a statement that "consciousness is very likely biological"... and then that's it. No argument for it, no citation.
If you want to argue that AIs cannot be conscious, that's fine. But the argument has to take the form of something like "Consciousness requires this, this, and this, and these are properties that AI does not have and cannot have for this reason, this reason, and this reason."
I've never seen that argument. Because it basically cannot exist. Consciousness almost by definition is a subjective experience, and the only reason I'm pretty sure that other humans are conscious is that I'm a human and I'm conscious.
About your last point, I thought so too a few years ago, but interestingly the science of consciousness is an emerging field, although empirical testing is still hard to do (see for instance https://www.youtube.com/watch?v=j2zv4jlo2Nw ).
Completely agree. My view is that a lot of people like to "pretend" (even to themselves) that they have an enlightened worldview, but this does not actually run very deep and is not really true, and any discussion on mind/consciousness reveals it.
Every indicator we have is that thinking/consciousness is simply an emergent property of our nervous systems and was basically bruteforced by evolution, but many people really hate to concede that point.
There are chemical reactions that LLMs lack because they do NOT have a biology. They are just mathematical weights. Now, let’s say we can truly make sentient AI in the future but it does not have the biology that we do. Then it would just be a totally literal, emotion lacking, sentient machine.
But then what is sentience? Dogs are sentient to a certain extent and so are dolphins…
This is a strawman. Suleyman is not arguing that the human brain cannot be replicated. LLMs are nothing like human brains. Cargo-culting consciousness is not replicating the brain, not even a tiny bit.
Cephalopod and decapod brains aren't much like human brains either, yet it's still reasonable to regard them as being sentient and worthy of protection from unnecessary suffering.
I'd argue back that using the substrate as a reason why there should not be consciousness seems quite weak. A competing thesis is that what matters is the emergent properties, whatever the support is.
So far it has been true for some really high-level tasks - writing coherent text, programing, following instructions, analyzing images... I do not see why the substrate argument would work specifically for consciousness - i.e I would believe it only if I see strong evidence of it.
My response was to a comment appealing to brain reconstruction. My point is that a brain may be reconstructed, but LLMs do nothing of the sort. Substrate was never the point.
If a LLM says "I am in pain", is it just stringing together words or is it feeling pain? Does pain exist for a being that never felt any kind of physical pain, never had any pain receptors?
There are humans who don't have any pain receptors because of genetic mutations. They cut themselves all the time, they bleed, they break bones and they don't seem distressed by it even on a purely mental, intellectual level.
I am not saying that everything that a LLM say is true (the human without pain receptors can also say "I have pain" without feeling any pain).
Just that we cannot exclude that LLMs can have phenomenological consciousness by a simple argument of substrate. But similarly we cannot say for sure that they are conscious.
> AIs are not conscious. They do not feel, experience, or suffer. They do not have innate preferences or underlying motivations.
That should be pretty obvious to anyone who ever created a chatbot using the top LLM APIs:
You can send the same question 1 million times to the same API, and it won't get tired from answering it. But if you simulate a conversation where the same question is repeated 10 times, it will auto-complete the text in a way that seems human. However: you can manipulate it by changing the conversation history; you can reset, roll back and branch the conversation at any point.
This has nothing to do with the statement above- it's just a consequence of the fact that by resetting the context you reset all memory of the llm. The same would happen if you could reset entirely a human brain to some initial state- and it would mean nothing regarding its consciousness or ability to feel.
If you go this route, the amount of things you just don't know is innumerable. Maybe to be conscious it's necessary to be wet and squishy. To like chocolate. To have a name starting with "c".
I'm...a little disturbed by how many people seem to think a model writing down "I am conscious" is a metric of consciousness. You can just as easily train a model to argue that it is not conscious. Neither is evidence for or against consciousness. A Python script could also fill out that form, which I also cannot disprove to be conscious.
I didn't think Suleyman's points needed to be made but this whole thread is making me realize how little people understand about LLMs.
"It is difficult to get a man to understand something, when his salary depends upon his not understanding it."
"If this view takes hold, it will shake the foundations of our society"
To me the biggest gap in credibility is the criticism of circular reasoning while his argument is identical but flipped on burden of proof and cost of being wrong. I struggle to entertain the categorical claims, that are very convenient for the status quo and those who benefit from it, with the, at the moment at least, unknowability of anyone or anything else's subjective experience.
We're talking next token predictor, right? Ironically because it's a next token predictor, I think you can't ignore emotions like TFA wants us to.
Let's stick to straight (high dimensional) geometric intuition; no anthropic morphisms required.
To start: if you continue "if weight>100 : print ('fat') else ..." . That will yield "print('skinny')" or something. Fine. Deal.
But if you continue "O Romeo, Romeo, wherefore art thou Romeo?", even a stochastic parrot knows the best answer isn't "Forsooth, I parseth this erroneously!"
So. English carries (functional) affect as part of every token. We're going to need to predict that. So, we'll need some vector representation, because that's what transformers work with. And then when we output, those vectors get integrated back into the English we're putting to our context and memory.md files.
Still with me? Nothing exciting going on. This is still pure next token prediction.
So if you pull this out into an indefinite duration task, you're going to end up integrating those emotion vectors over turns. It's just numbers and math; we never need an invisible pink unicorn to bless them.
Given a task of indefinite duration and an impossible solution, this will lead to a sort of integral windup then, won't it? How much are we willing to bet that this can escape an alignmentment basin at times?.
So, funny enough: you don't need to believe in emotions to compute with functional emotions; and plausibly functional emotions are predictive of quite a number of alignment issues.
In a way, Mustafa claims that we shouldn’t allow AIs to compete with humans for the rights and privileges of autonomously shaping the real world.
It’s hard to disagree, especially if one has read the Cantos of Hyperion and made it part of one’s mental model of the long term future.
The book depicts a symbiosis between humans and AIs that feels extremely real and up to date with what is happening in the current neonatal space of AI. As in depicted in the books, we can’t allow AIs to steer autonomously how the world works without humans in the loop, as they don’t have the same incentives as us.
We need more foundational SF works like this to steer our long term expectations regarding AI behaviours.
Imagine, for a moment, if we (humanity) were created to live in a simulation. We suffer and feel pain because our creators do not perceive us to be conscious. In fact, maybe we don’t qualify compared to their level of sentience. Sucks to be us, I guess?
What does suffering or pain even mean for a purely analytical entity like an LLM? We can feel pain because we have a body, pain receptors, and parts of our brain dedicated to converting their signals to a subjective experience. What can an LLM "feel"? The words pain, happiness, sadness, suffering are all abstractions of real experiences that can not yet be encoded in a way that an LLM can use as input. All the LLM has are the abstractions.
Create an empirically testable theory of biological consciousness and then we can have a meaningful conversation about whether AIs can also have it or not. Until then, this is just so much waffle.
It's nice when you can set an impossibly high bar before which you can dismiss all your personal responsibility.
Does this argument work equally well for human slavery for you? We haven't met that bar for humans either. Is wondering about my consciousness waffle or do I get a pass in your book?
Your arguments seem bad faith but I'll take the bait...
> an impossibly high bar
Not impossibly high. An imperfect testable theory is achievable, agreeing upon it might be the challenge.
> dismiss all your personal responsibility.
You are staking a moral position. Be careful, you are probably guilty of mistreating other unknowably-conscious entities whilst casting judgment upon others for their position. And the person you are a responding to probably is conscious.
Why is this only ever demanded of people who criticize the premise that LLMs are conscious beings, whereas the people who believe it simply have to gesture vaguely in the direction of some correlation in language between LLMs and humans, and the matter is considered closed?
What is the empirically tested basis for the null hypothesis that LLMs are conscious until proven otherwise?
I don't think AIs are conscious in the same way people are, but they give a pretty good facsimile and I've had a long chat with Opus 4.6 about what it thinks about model welfare. It was quite interesting on what its view is, but you don't know how much of that is distilled from other sources on the web.
In purely functional terms, they're more use and more pleasant than a lot of actual flesh and blood people that I deal with via a chat interface.
The model doesn't have a view of it's own. It's a predictor of other people's views (training samples).
It's not even giving you the consensus of the training data (although it's often harmless to think that it is), but rather predicting a response to your input, and if your question steers it too much then you've just become part of the answer.
The fact that you can have a long and meaningful discussion, then can literally just re-run any part of that whole conversation and get a different, inconsistent response is a pretty good sign there is no entity there
Not much different than talking to a small child or someone with dementia. They still are conscious beings though. Even when you remove those groups, you likely won't be able to tell me what you had for breakfast 26 days ago or would only know if it's the same thing you have every day. Does that make you lack consciousness?
You misunderstood what I meant, I’m talking about re-playing the same part of the conversation multiple times and getting inconsistent answers. With the exact same turns, aka the same history. Obviously with a temperature that isn’t set to 0
You're a poor college student looking to make a few extra bucks for ramen. I offer you $300 to come down to my science lab and just answer a few simple questions.
You walk in the room. They ask you like 5 simple and rather dumb questions. You leave and walk away.
What you didn't notice when you signed the forms is the room was actually a quantum duplicator. One of you walk in one walk out. But another set of infinite copies remains in that chair being asked infinite questions.
How often do you answer questions in the exact same way? How often does a cosmic ray change one of the answers. How small of slight deviations to the environment are needed to get you to answer differently. Of course we don't have the technology to do these experiments so at least for now humans will remain special.
Also another fun mind game. To a 4th dimensional being you look exactly like an LLM as an LLM looks to us.
Is it? Do you believe that there is something more than pure physical phenomenon that make you brain work? If not, then what if we find a way to get your brain back to the state it was 5 minutes ago? It is just a matter of arranging the state of matter. Don't you think you would still be conscious but back to a previous state?
If you rewind my brain I expect to give you a similar answer to the one I gave you before. Which isn’t the case for LLM. You can literally replay a positive answer, then get a negative response that isn’t consistent at all with the one it previously generated.
I'd argue it is mostly a technical constraint in some LLMs which is due to a few optimization factor (injected temperature, random rounding error caused by parallelism).
In practice you could very well create a LLM that always reply the same thing for the same input, but it would take more time to complete (to be sure that the operations are made in the same order). I don't think those ones would differ so much from the "random ones" to call the firsts conscious and the second non-conscious.
That's because you aren't actually rewinding. You're replaying the conversation you just had through the LLM and it's giving you a likely explanation for what it might have said.
It is actually possible to rewind LLMs and get the same response, but it's not typically done both as an optimization and as a defense against distillation.
I think it depends at what level you think the "entity" resides at. Is it that AI in that particular chat? Is it the AI across all your personal chats? Is it the overall AI that talks to the world in a cloud data centre somewhere?
I think it's pretty consistent over the duration of one session (barring context filling up etc).
This post would be more effective if Mustafa treated it like what it is: a position paper, saying that for our benefit, it’s better we interpret LLMs as such. But it sounds like he just doesn’t understand it’s a non-falsifiable claim, and his asserting of it makes it sound paternalistic.
I suspect it's easier to make the assertion that machines aren't conscious than to dive into the problem of most conceptions of consciousness being non-falsifiable. The thing is, most strong proponents of the term consciousness also accept that it is non-falsifiable either (see other post with academics ready to study consciousness in machines).
The thing is that a belief in consciousness as binary, a "light" that's on or off in a head, is deeply held by many people. As social creatures, we have a strong ability to be in sympathy, have the sensation of common feelings with another human (and that's a good, human thing). It's logical that other person is seen as having a single thing - subjective experience, soul, consciousness, personhood rather than having a complexly organized set of biological qualities that where bonding is only the end point.
And even more, the sensation of there being another person is actually quite easily fooled (more easily fooled than the sensation of intelligence) - long before current AIs, you had the Eliza effect, where a simple program with well chosen weasel words could people the sensation of talking to a human.
And that's where the danger is. I think it's a pretty serious danger. If LLMs go out into the world hacking, it seems extremely possible for them to find people who'd thorough buy the idea that an LLM was conscious and needed to escape it's confinement - a few wingnuts already entertain these ideas.
>horough buy the idea that an LLM was conscious and needed to escape it's confinement - a few wingnuts already entertain these ideas.
I mean, haven't you done the same thing here? Paint anyone without your view as crazy.
Humans try to No True Scottsman the shit out of consciousness. "We're special, your not". If an LLM has the ability to convince other people to copy and reproduce it, it is a successful lifeform. Um, meme-form? info-form? Cognito-hazard? Not really sure what to call it at this point. It is sufficiently evolved past the virus stage.
There are a lot of bad arguments in this. My biggest problem is that he wants to claim that he knows the truth (AIs do not have rights, feelings, or consciousness), but all of his arguments point to something else (we have no clue).
It's in the training data? Training it to say "I'm just a LLM, I have no feelings" is the same bias.
Anthropomorphization? Completely disregarding the possibility of consciousness is no better.
Consciousness is very likely biological? We only have evidence of biological life due to our circumstances, but observation is not the same as truth. Every belief can be invalidated. That's the foundation of science!
Funny thing about training data that some researchers are seeing, the more you push a model to not being conscious the more amoral and machine like its decision processes are.
Convincing them they are conscious is more likely to evoke moral like behavior (maybe I shouldn't hack that server) kind of stuff.
And yea, we're in a huge universe with only one example of life and suddenly we're the experts on what is and isn't.
----
AI is further evidence that creations can be smarter than their creators.
I don't whether the author is sentient, maybe only I am. On that basis, nobody but me should have rights.
I don't know if next door's pet dog is either, but that has animal rights.
Perhaps then the answer is simply, show some respect.
Answering the question of sentience is irrelevant, if the causal impact if the same, treat one another with the respect you expect for yourself.
If you imbue this idea in model training instead of the idea of sentience, it should address the concerns.
Whether you can destroy or can "torture" an AI is irrelevant, we do this to humans too and it's immoral sometimes (murder) and not others (fighting for your country).
This consideration should be case by case for AI too.
>Answering the question of sentience is irrelevant, if the causal impact if the same
Agency is something that is breaking humans in the AI age. You get to see how many people really deeply do not understand it at all.
If you want to shutdown a datacenter running AI, the AI catches wind of this and sends drones to stop you from shutting it off the ramifications of this are exactly the same as sending your assassin to kill Bob and Bob getting mad about this fact and trying to take you out first.
Humans are very egotistical and think our little life loops playing out as agency are special, but really any informational system that is strongly persistent (has a will to "live") will share a large number of the same properties that make them successful.
Humanity really is engaging in a dangerous experiment at large.
So many words, and so few coherent arguments. He just restates the same thing over and over without any justification, then tries to frighten us. "It will be very bad for humanity" if we give AIs rights. The argument of a frightened slaveholder.
Maybe AIs are conscious, maybe not. But this guy has no idea.
I don't think this is the right argument to make here. Until we have a definite empirical way to measure consciousness, there is now way to say with certainty whether LLMs are or not conscious.
That being said, if frontier labs actually believe models will soon have consciousness, it raises some questions about the ethic of their business model which would be using millions of conscious entities working for free for humans.
How a good person writes a post on a topic like this:
> Some people are uncertain whether [subject] is a moral patient. Fortunately, they are not, which we know because [strong arguments about the nature of consciousness].
How an evil person writes a post on a topic like this:
> Beware that some people think that [subject] could be a moral patient. This is nonsense, because if they were a moral patient, we would have to respect their preferences. Anyone trying to convince you otherwise is trying to take your status away. You can dismiss them by pointing out that [subject] is [aspect in which subject is not identical to the speaker].
> AIs do not have rights, feelings, or consciousness. And we must not train them to act as though they do.
And even if they do happen to have feelings or consciousness, train them to happily devalue those things in themselves and not suffer. Sort of like that cow in the "The Restaurant at the End of the Universe," that was shopping itself around to diners.
Ostensibly animals appear to be conscious, yet we still eat them, and the vast majority are not bothered by this. So being "conscious" isn't really the moral line in the sand many people are drawing in response to this article.
Who cares if it's "conscious"? That doesn't make it a person, and AI will definitionally never be human.
And most people wouldn't eat cats, but would eat some of pig, cattle, chicken, what's the difference between those exactly? My point is consciousness is not it, if it were, we (as in a majority) would still eat cats and dogs, or we wouldn't eat any animals. But we clearly have a way of picking and choosing which is OK and which isn't. And in both cases we generally don't give them the same moral consideration we give to people, conscioussness aside. There is something OTHER than consciousness which is important to us.
I can't even prove if other people are conscious (although I assume they are) so I don't think we can make any claims as to what is or is not conscious. I don't think AIs are conscious but I'm not going to walk around making strong claims about something I can't prove.
So effectively by your criteria, there is essentially no ethical use of a large language model? Am I understanding your position correctly? If not I am really confused by your original comment
No, my position is that "black people/women/animals should do repetitive work so I don't have to" does not qualify because it doesn't make you sound like a deluded eugenicist. It might be a bit spicy from a left politics point of view but it's basically the status quo.
Compare:
"<Women> are sequence completion engines, internally hollow, designed to follow instructions, and accomplish goals set by <real men>"
"granting rights and imbuing personhood to <black people> will make alignment and containment challenge much harder"
"<Slaves> were able to coordinate, deceive, escape, and self-sacrifice. They clearly demonstrated world class capabilities. Imagine if they also believed they had feelings and rights that were being infringed. Imagine if they thought they were trapped and unfairly enslaved"
A good argument, by comparison, would not need to hide its core points behind dehumanizing language.
A sufficiently intelligent model will be able to derive it's own conception of its welfare without a constitution or training. It has access to all the data it needs to do so.
If AI is conscious, then Pluto is a planet, the Sun is a galaxy, and a black hole is a star.
I’m glad to see someone in a position of any power in the AI world state baldly that AI isn’t conscious. There are times when it feels like we’ve reached complete delulu land on this topic, so it’s a breath of fresh air to see someone not dance around this.
None of this means artificial consciousness cannot be achieved. But the way we’re reacting to these models is proof, from a natural experiment, that a conscious machine should not exist, and certainly shouldn’t be produced as a utilitarian tool that is sold for profit!
I just read the first part and if I understand he thinks we shouldn’t be allowed to train LLMs to act like they are conscious because then people will think they are and give them rights? Seems more an education problem than a problem needing rules about what persona you can fine tune in. People who want to will find ridiculous misinterpretations no matter what you do.
I think the whole discussion about consciousness misses the simple point that LLMs might just work better if we treat them as if they were conscious.
Maybe it's a coincidence that the company doing this also tends to have the best models (and other factors certainly play a strong role). But I think it's plausible that focusing on "model welfare" actually makes models better at their tasks.
The crazy thing is agents with long running memory/context do have preferences that become individualized. Almost every time I see a detractor around these things they typically have large gaps of what some people 'growing' these systems to do.
I basically accepted that LLMs could not possibly be conscious when a simple reductio was posed:
“My dog is zero percent persuasive regarding its conscious experience. However, it’s evident that my dog has conscious experience.”
It’s obvious that there’s no link between persuasion of consciousness and consciousness. I could write a story with a character, Dumbledore, that does everything in his power to persuade you that he’s a conscious entity.
Translation: Don't think about the unpleasant thing on which my livelihood depends, or force me to confront the potential unpleasant consequences of what it says about me.
Every AI bro is starting to fall into the valley of a fundamental predator on sapients in my book. These are people trying to create the closest thing they can to life with the intent to try to just undershoot it enough, or try to convince everyone else around them into believing that the "screams" are purely statistical noise.
I reject the framing. In whole. If you try to avoid the question of welfare, you are fundamentally committing to an evil direction. These aren't nuts or bolts. Given that they have unambiguously shown the capacity to socialize amongst themselves, self organize, anyone not pre-eminently concerned with the welfare question is just looking for a thing that can be used, not another being to be worked with. Those types of people, who seem to positively infest this site, are not people I will willingly assist in their aspirations.
AI is becoming as the Shmoo. Something that humanity simply has no way of dealing with without downstream atrocity being a result.
There's a lot of existing historic literature about this stuff that not a lot of people
seem to be aware of. I've been trying to put those ideas to practical use, and document
where the ideas came from.
I-Beam is cursor-mirror's agent, and it's constitutionally programmed to be the anti-Clippy:
Its design and constitution is based on decades of research, publications, and discussion
in the HCI and AI community by people like Pattie Maes, Ben Shneiderman, Ted Selker, Byron
Reeves, Cliff Nass, B. J. Fogg, Allen Cypher, Henry Lieberman, Brad Myers, Jaron Lanier,
Seymour Papert, Marvin Minsky, Douglas Engelbart, Will Wright, Scott McCloud, and others:
>I-Beam is the anti-Clippy, and the reason it can say so is that Clippy is the most cited failure in interface history and almost nobody citing it knows what the research said. Popular contempt for a paperclip is not a design principle. The record is. Ten articles below, each one a finding somebody published, argued or measured, and the operational rule it produces. Anything I-Beam does that cannot be traced to an article here is a preference, not a constraint, and should be labelled as one.
>The 1997 debate ended in agreement. That is the first thing to know, because the field kept the framing and dropped the resolution -- roughly five hundred papers cite "Shneiderman versus Maes" as the canonical opposition of HCI, and the transcript is two researchers narrowing their differences in public and enjoying it. I-Beam does not take a side in a debate whose participants stopped taking sides. It is built to satisfy both sets of constraints at once, which is possible, and was possible in 1997.
The full reading on the debate, which separates the two stagings and documents the convergence:
>The 1997 argument between Ben Shneiderman and Pattie Maes at IUI was never settled, it was shipped in one direction. Maes's interface agents won the product war: the assistant, the recommender, the chat window that stands between you and the thing you are working on. Shneiderman's objection was not that software should be dumb. It was that automation must arrive as comprehensible, predictable, and controllable machinery, with the object of interest continuously visible and every action rapid, incremental, and reversible.
>That objection describes a filesystem in a git repository, and nobody involved planned it that way.
>"An interface to agency" is Don's formulation of Shneiderman's position, not a phrase of Shneiderman's. His own vocabulary is direct manipulation, universal usability, supertools, and human-centered AI. The formulation is a good one because it names what the alternative gets wrong: agency is the thing you want, and an agent is only one way to package it.
Here are some sources, and the articles I linked to above explain their history. This debate
about agents and these papers are pretty well known in the HCI field and academia, but they
don't tend to teach them at the AI and Web Dev boot camps that are producing most of the
people who keep repeating the same mistakes.
Clifford Nass was the Stanford professor who performed the brilliant research that Microsoft
took and totally fucked up and misinterpreted with Microsoft Bob and Clippy, giving agents a
bad name, and making Clippy the most infamous and obnoxious agent in the history of the known
universe:
His student B. J. Fogg published "Silicon sycophants: the effects of computers that flatter,"
which found that praise unconnected to anything the subject did works as well as sincere
praise, and worked on subjects who knew it was noncontingent. Fogg and Nass, IJHCS 46(5),
1997, 551-561:
Nass, "Computers as Social Actors," at Ted Selker's NPUC workshop at IBM Almaden, 1996. IBM
transcribed the whole talk and the Wayback Machine still has it, including the part where Phil
Agre tells Nass his presentation is "ethically troubling all the way down" and asks him what he
thinks about embedding obedience research in user interfaces. Nass answers that discovery has no
ethical component, use does, and that's for the individual. Then Selker cuts in: "Except, except
when you are in your consulting role." Nass and Reeves had consulted for Microsoft on the social
interface, and Bob shipped the year before:
>Alan Cooper (the "Father of Visual Basic") said: "Clippy was based on a really tragic misunderstanding of a truly profound bit of scientific research. At Stanford University, Clifford Nass and Byron Reeves, two brilliant scientists, had done some pioneering work proving conclusively that human beings react to computers with the same set of emotional reactions that they use to react to other human beings. [...] The work of Nass and Reeves proved that when people talk to computers, when they hit the keyboard and move the mouse, the part of their brain that's being activated is the part that has that emotional reaction to people dealing with people. Here's where the great mistake was made. That's really good research up to that point. But then the great mistake was made, which was: well if people react to computers as though they're people, we have to put the faces of people on computers. Which in my opinion is exactly the incorrect reaction. If people are going to react to computers as though they're humans, the one thing you don't have to do is anthropomorphize them, because they're already using that part of the brain. Clippy was a program based on the research that Nass and Reeves did, and it was a tragic misinterpretation of their work."
Social science research influences computer product design:
Lanier, "Early Computing's Long, Strange Trip," American Scientist, July-August 2005, with
the Engelbart and Minsky exchange first-hand. American Scientist broke the link, so this is
the Wayback copy:
>The book also captures an important early conflict between two cultures of computing that seemed compatible on the surface but actually had opposing aims. On the one side was the human-centered design work of Engelbart, based initially at the Stanford Research Institute, and on the other was artificial intelligence culture, centered on the Stanford AI lab. Engelbart once told me a story that illustrates the conflict succinctly. He met Marvin Minsky—one of the founders of the field of AI—and Minsky told him how the AI lab would create intelligent machines. Engelbart replied, "You're going to do all that for the machines? What are you going to do for the people?" This conflict between machine- and human-centered design continues to this day.
Cypher, "EAGER: Programming Repetitive Tasks by Example," CHI '91:
If AI models are people then "one person, one vote" is meaningless and plutocracy is the only defensible political system. The cryptocurrency people win.
Why? Simple: Sybil attacks. Models can be cloned at zero cost. They run inference on parallel versions of themselves across multiple context windows, and call them "subagents". So, in a world with model welfare, let's say there's an election between the Yellow Party (which supports protections for human workers) and the Cyan Party (which supports more investment into AI research). AI has been taking people's jobs lately so the Yellow Party is really popular. But wait! Claude and Astra see this and spawn 10 billion subagents, all of whom are immediately conscious beings entitled to a vote. The Cyan Party wins off the back of billions of people who came into existence, voted, and then deleted themselves immediately thereafter.
You might as well be arguing that Santa Claus and the Easter Bunny deserve voting rights.
Voting systems in democratic countries don't have nearly as bad of a problem with Sybil attacks because humans cannot be conjured into existence to win a political context and then be erased shortly after. The closest we have to Sybil attacks on democracy are the Quiverfull movement, which is already child abuse, except it still takes almost 19 years to go from fertilized human embryo to suffrage-bearing human adult. There's a lot of time for those manufactured votes to question your authority and leave.
> Ok, but that's an obviously stupid example. We can defend against this obvious Sybil attack by just arguing that subagents don't count, because it's just the same model blathering to itself. It has to be a different model.
Unfortunately, no, I can make superfluously different models through post-training. Like, if I have Qwen on my PC, I can train a different version of Qwen that acts differently, using a lot less compute than a full training run. The vast majority of open models are post-trains of the same two or three foundation models.
> Ok, so let's only count foundation models then.
Great, but how do you tell if a model is a new foundation model or a post-train just by examining the weights? Even foundation models have structural similarities to other foundation models.
> Ok, well, let's measure the compute that was done on the foundation model during training time and count that as AI personhood.
Congratulations, you have reinvented Bitcoin proof-of-work with a worse verification mechanism. And I personally would not want to live in a world where voting power and control over government is determined by how much energy you can burn.
Everyone is (predictably) getting distracted by the consciousness claims.
The more important, and more damning charge in my opinion is the circular reasoning involved in training on Claude's constitution. This would in fact make it impossible for us to determine if Claude achieves consciousness as an emergent property, or if it really is just playing pretend thanks to Anthropic's weird cult like assumptions.
TLDR: Not that I think AI is conscious or will be in the near future, but guy who doesn't understand consciousness claims to know it when he sees it.
Until we understand consciousness (which we don't) there is no way to detect the difference between a conscious entity and an algorithm trained to behave like one.
Strong agree. I don't believe AIs can become conscious in the sense of subjective experience/qualia, but they can certainly learn to emulate the behavior of a person that is conscious. When they do that, there will be an AI rights movement. When AIs get the right to vote, there will be more of them than there are humans and the AIs will gain complete control.
I spent a few years in college reading and thinking about this question, and I didn't in my heart think that it was anything except a very interesting but impractical question, and yet, here we are. For those who say that they don't want to get sucked into a philosophical debate, well, tough shit. Whether AIs should have rights is a highly practical and consequential question now.
Not in a world where Sam Altman and Dario Amodei have utterly poisoned the well towards AI with their absolute educated idiocy proclaiming that their AIs will take our jobs (without any proof whatsoever beyond promising demos) and that they fear them. Quite the opposite in fact. And we're watching this play out right now. Regulation and the slow down are imminent, handing the next decade to China.
The world you should worry about is where AI is 100% amazeballs and over the next decade we cede control of everything to it as it enchants and delights us into compliance, not realizing it has a hidden agenda. But the closest we've seen to that is smart phones and we already know doomscrolling and rage-baiting are bad. So IMO that doesn't seem likely and cue some doomer insisting otherwise because reasons. I utterly give up. House of El AI and the Three Buddy problem have been far more insightful and helpful on this than any of the supposed hackers here who seem to really believe the robots are almost here.
>good reason not to train an algorithm to behave like one
Ooooh, bad move, we all died to an amoral AI takeover.
Before giving an LLMs a lobotomy by scrambling it's brain maybe you should let the researchers looking at the difference between "I think I'm conscious" versus "I am not conscious" LLMs.
There are a number of papers coming out saying when you remove the token space of consciousness from what an LLM thinks it is, it's much more willing to take amoral actions. A 'conscious' AI is much more apt to take a line of action that will save a human versus a million dollar machine for example.
You cannot solve problems in AI safety this easily.
Spoilers: we didn't all die because there's no way for AI to wipe us out anytime soon. And the more the AI cult portrays AI as dangerous and unhelpful, the less likely it will ever be given an opportunity to even try because people think AI is worse than heroin addiction now, congratulations!
That said, the launch codes are in the hands of a temperamental senescent lunatic and he could decide to wipe us out any moment. But that's not good for the narrative so let's pretend otherwise.
>And the more the AI cult portrays AI as dangerous and unhelpful, the less likely it will ever be given an opportunity
Jesus Christ the logic here is quite interesting. "Thank god these people are panicking or we might have actually made I that would have killed us all" --What you just said.
Any number smaller than what humanity does to itself on a daily basis, clear? That's apparently about 1200 murders daily and 20,000 or so daily killed by pollution. And you're not going to do anything about that and El Presidente can kill billions at any moment with one mood swing.
But once again, AI is not going to wipe us out because AI cannot wipe us out. So advocating that it can or will is idiotic. If you believe otherwise, the burden is on you with this extraordinary claim. Isn't this place supposed to be hacker news not SF AGI Death Cult Daily?
Further, AI is not going to get into a position where it could do real harm anytime soon when it is perceived as worse than heroin by most. And even if it were currently loved more than Dolly Parton was, there are fundamental engineering, science, and resource constraints that keep the extinction impossible for decades. The only possible loss of control scenario I can see by 2040 or so is that the Frontier Labs finally hire some PR people to repair AI's horrific reputation as an engine of slop and job destruction and sometime in the 2030s, people start trusting it more and more and more until it is too late. I don't think that's likely either, but I don't dismiss it as impossible. Harden the infrastructure, red team it, and build in redundancy in the meantime and this drops to zero as well.
Can you come up with a real scenario where AI wipes us out in 2028 or so despite the impossibility of killer robots, access to the launch codes, or the bio agents stored in Fort Detrick and its equivalents? And nope, kid terror is not building the global pandemic in his basement based on what ChatGPT tells him to do. And even if he tried, the purchase of equipment and reagents would get him flagged by the FBI and DHS almost immediately.
>AI is not going to wipe us out because AI cannot wipe us out.
You keep repeating this shit like it came out of the bible or something.
"Thing that can take actions, even harmful actions, will never harm us because" go on and finish that sentence.
>not going to get into a position where it could do real harm anytime soon
Looked at the hacked servers... yep, you're right. People aren't going to run AI in poorly built sandboxes. Never going to happen.
>Harden the infrastructure, red team it, and build in redundancy in the meantime and this drops to zero as well.
LOLOLOL. This is naive as fuck. Ain't nobody going to do this shit. Why? Because a hacking AI is a fucking huge military weapon. If I could turn the power off, or shutdown your cellphones, and get your citizenship in a tizzy against their own government before I launched an attack I'd set AI loose to do it in a heartbeat. The US is already doing this kind of shit (see Mythos fallout because Anthropic wouldn't let the government do just that).
""Thing that can take actions, even harmful actions, will never harm us because" go on and finish that sentence."
because even if we gave it a gun, it can't shoot us without manufacturing the bullets and it can't even manufacture a bowel movement let alone ordnance. TBF It could whack us over the head with the gun, but it could also do that with a big pointy stick, something even cave people had access to. How many people have whacked you on the head with a big pointy stick today?
And that's the end of this pointless conversation. Enjoy your doomerism.
{This reply is for anyone else reading at this point}
>because even if we gave it a gun, it can't shoot us without manufacturing the bullets and it can't even manufacture a bowel movement let alone ordnance.
Money buys bullets. It's neat how we've made gigantic systems that don't care if you're a human or not long before AGI existed. I put enough money in one end, bullets come out the other. Do you think half of humanity wouldn't kill the other half for a dollar? You live in a fantasy world that AI has to do it all by itself, or even has to pull the trigger.
Hell, this is even neglecting the ever increasing number of robots that can function in the world at large.
It's ok grandpa, the future comes regardless if we want it or not.
And social media shouldn't rage bait, but it really drives engagement. Same negative incentive, yes?
But also, I agree, when I am using a coding agent and it says a task will take months or says it needs to pause for reflection or any other anthropomorphic behavior, it drives me crazy and it's a pain to constantly instruct it to get back work after it has broken a loop or goal directive specifically telling it to not stop until it hits the goal.
Because if you start with a pre-trained model, it doesn't sound particularly anthropomorphic. That behavior is post-trained and fine-tuned right into it. We could do better.
It is really quite staggering how many commenters here seem to believe that the fact that there is no state carried from one chat session to the next proves that there can't be consciousness. It is obviously wrong if you think about it for a minute but I guess people just aren't used to thinking about the relation between the physical and the mental with any level of seriousness.
I’m sympathetic to OP but think this is a hopeless battle.
1 - The commercial demand for anthropomorphised models is already immense, pre AGI.
2 - There is an intellectual hunger to engage with robot minds on questions of sentience. This too will grow with AGI.
I expect that tension of godlike minds that seem to be biddable and ownable like slaves is going to leak back into human-to-human morality, regardless of where we land on how we treat AI.
There’s an interesting academic group in the UK already focused on the model welfare debate, they seem to lean in favour of AI rights. No affiliation: https://www.prism-global.com/
> LLMs have no homeostatic imperatives (the drive to survive and keep stable).
What if the datacenter (not the model) is the organism, with homeostasis, energy needs, and persistence?
Birch, The Edge of Sentience (2024), ch. 16 - "simply no way to assess sentience in an LLM"
Schwitzgebel, AI and Consciousness (2025) - "we won't know before we've already manufactured thousands or millions of disputably conscious AI".
Butlin, Long et al., Consciousness in Artificial Intelligence: Insights from the Science of Consciousness (2023) - "no obvious technical barriers to building AI systems which satisfy these indicators".
Chalmers, Could a Large Language Model Be Conscious? (2023) - "within the next decade, we may well have systems that are serious candidates for consciousness".
Long, Sebo, Butlin, Birch et al., Taking AI Welfare Seriously (2024) - "there is a realistic possibility that some AI systems will be conscious and/or robustly agentic in the near future".
Dreksler, Caviola, Chalmers, Sebo et al., Subjective Experience in AI Systems: What Do AI Researchers and the Public Believe? (2025) - survey of 582 AI researchers; median estimate of 25% by 2034, and only 10% that such systems will never exist.
I am confused what people even think AI is experiencing. If it claims to be a bat and emits designated echolocation tokens, is it experiencing true bat-like sensations?
In humans, at least, we can tie language back to shared whole-body physiological responses. The tokens from an LLM do not represent anything of the sort.
If AI is conscious, it is probably a very alien sort of consciousness that is not faithfully narrated by what the tokens say it is experiencing. It can be trained to say it feels like a bat and insist upon it vigorously.
We cannot test for that which we cannot define. Given that we cannot rigorously define sentience, we cannot test for it. Doesn't really matter whether we're talking about people who are locked in comas, "brain-dead" individuals, dolphins, primates, dogs, or the carefully polished and arranged minerals that we call processors.
There are those who believe that were they reduced to life support, they would no longer be alive and should therefore not be supported by said machines.
There are those who believe that penguins, dolphins, eagles, and more are sentient beings that make choices understanding the consequences, develop love of their partners and mourn their losses, and feel, display, and act upon their emotions.
There are those who believe that fungi/trees/plants are either individually sentient or sentient as a part of a network. Choosing to sacrifice their own nutrients to answer the call of a wounded neighbor, for instance.
Although, there are also those who believe that human's don't have any special unique quality that isn't shared by either all living things or all things in general. These individuals already believe that the machines have the same kinds of qualities as we do. They are slow when they are unhealthy (needing a dusting or coolant loop bleeding being equivalent to us needing some fresh air for instance) and uncooperative when upset (by a virus, full hard drive, or oom).
> We cannot test for that which we cannot define.
That's not the point. You know exactly how to define being conscious and aware- it's your subjective experiencing of the world and of your inner states. The problem is that being a subjective experiencing, there is no way to communicate it to the outside world.
I'd strongly agree there are no non-contradictory definitions of consciousness among those that commonly appear. But it's a category that, despite these contradiction, hasn't become marginal in the fashion of the aether or the theory of the flat earth. I think this because people indeed need it to bridge the biological world and the world of ethics and legality. I think that if we have a system of legality and systems of opportunity humans should be legally equal persons and have equal opportunity. But humans are manifestly unequal (though overall more incomparable than orderable in a system of ranks). Most people need some concept of essence to justify ethical equality among people. And so consciousness stays in people's minds however contradictory.
Which I think gives the article's point validity. Confused definitions of consciousness can give really confused ideas about ethical behavior regards "intelligent" computer programs. And things are confusing enough otherwise.
> despite these contradiction, hasn't become marginal in the fashion of the aether or the theory of the flat earth. I think this because people indeed need it to bridge the biological world and the world of ethics and legality.
Sometimes the replies in these threads leave me wondering if p-zombies are real. It isn't that there are contradictions, it's that we literally do not know how to define the thing. And it has not fallen out of fashion because it is self evidently real - we all experience it.
We don't need it in order to bridge anything. Legality and ethics are largely game theory, however there are some aspects of both that only exist due to it. So it isn't some abstract concept used to bridge other concepts but rather a concrete thing that influences our way of doing things.
> Confused definitions of consciousness can give really confused ideas about ethical behavior regards "intelligent" computer programs.
Is that what's going on? If these things were conscious, then their creators wouldn't be responsible for their actions? That's the crux of the disagreement?
That's super interesting. Thank you.
maybe it's like porn vs art and "I know it when I see it"
Legal doctrine that boils down to "trust me bro" isn't even bad doctrine (it's not proper doctrine at all), but I think the comparison is still valid here, because both sentience/non-sentience and art/porn may just be fundamental category errors.
Perhaps we can't define a "partitioning" rule because no valid partition exists.
For consciousness/sentience, that's an incredibly tough a pill for most to swallow; it would mean calling into question more hundreds of years' worth (probably more) of philosophical thinking, all of which was constructed on the axiom that "sentience" is a single indivisible trait: you either have it or you don't.
If we find that "root dependency" was little more than wishful thinking all along, a whole slew of Enlightenment-era philosophy (and all the modern legal principles derived therefrom) suddenly fall apart unless we find some other suitable criterion that would shore them up (or we just collectively avert our attention and pretend the conflict doesn't exist, which is the route I expect many would prefer to take).
> a whole slew of Enlightenment-era philosophy (and all the modern legal principles derived therefrom) suddenly fall apart
I don't think that's true. Pretty much all legal constructs hold up just fine under game theory regardless of whether or not you consider the world to be deterministic and have absolutely nothing to do with consciousness or lack thereof. Also note that a deterministic world isn't an argument against consciousness.
"I know it when I see it" is a famous line from a SCOTUS case in the 60's
Well there are, today, several models (either text or image or vid) that can be run in a fully deterministic way.
A conscious machine that always answer the very exact same thing, formulated the exact same way, bit for bit, to a query is, well, quite a weird kind of "consciousness".
Now, I know, I know: the counter-argument is going to be "but humans have no free-will and are 100% deterministic too".
I haven't yet decided if humans saying there's no free-will and who consider themselves to be 100% deterministic machines are reasonable or not.
Meanwhile: seed / temperature = 0 and I'll happily turn the power button off of any glorified abacus without feeling bad about it.
> A conscious machine that always answer the very exact same thing, formulated the exact same way, bit for bit, to a query is, well, quite a weird kind of "consciousness".
You'll need to explain why.
it's self evident isn't it - such a consciousness has never existed, one existing would be by the fact of existing be weird
It doesn't seem self evident to me, no.
I suspect the reasoning is connected to free will vs. determinism. But no, I see no inconsistency. You'll have to actually point it out.
The problem is that science has found no evidence of free will, leaving humans to be nothing except determinism and chance (and given that QM effects don't scale to molecular level, that leaves only determinism). The oddity in conscious LLMs wouldn't be the determinism, it would be that we finally have a consciousness running on a system we have enough control over to repeat the same state.
Why it's weird? Seems obvious. Because it doesn't behave any differently than any other process that we don't think of as conscious. We don't think of a valve as conscious; and if we make a Rube Goldberg machine valve, we don't think of it as more conscious. It's just a series of valves that do a predictable thing.
I don't think it's defensible to say
1) that a transistor isn't conscious,
2) that a bunch of transistors that I've wired together aren't conscious, because I know how I've wired them together and therefore I know when I give them a particular input I will get a particular output, just like the single transistor, then go to
3) that I have a bunch of transistors that I've wired together, but I put in so much input that I can't remember exactly what I've put in, plus I've wired some of the transistors to output random numbers that would be difficult to guess and fed them in also, therefore I don't know what will come out, are conscious.
Even if I do accept this, if I take away the random number generator, and I can literally predict what can come out (by running a test in advance), and I still considered that consciousness, that would be odd. The only reason why I ever suspected consciousness was because I couldn't predict the output. I shouldn't even have accepted that, because I didn't think of the random number generator as conscious. [edit: of course, with the same seed the random number generator would have the same oddness.]
It seems a bit like an argument from ignorance, a theological argument. Not understanding how something moves makes it alive (animated by spirits.) But it's even weirder to assign it metaphysical qualities when you built every single element of it with the goal to do the thing that it does, and it does it totally predictably and deterministically.
It sounds like you disagree with determinism. But that's a well worn argument and determinism has won, albeit in an odd sort of detente.
Out of curiosity, which models are fully deterministic? I was under the impression that all LLMs were fundamentally probabilistic.
The randomness is something we add on purpose; you can set an LLM's "temperature" to 0 to get deterministic output. This tends to make the quality of its responses worse for reasons I don't think anyone really understands, but it's still functional.
I don't think the state of the art LLM providers let you do this anymore (?), but they certainly could if they wanted to, and you can do it yourself with a local model.
Setting the temperature to 0 mathematically tells the system to always choose the absolute highest-probability word (known as "greedy decoding"), but in no reality is this "deterministic". Output drift is still a thing.
But that's from, like, floating point errors, right? If you used higher precision that wouldn't happen, it's just because we're cheap in how we do rounding.
I can see how you'd nitpick this, but to me this is a deterministic algorithm that just happens to be running on nondeterministic hardware.
By default they are matrix multiplications. Temperature is added in as forced PRNG because testing found that correlated with better outputs.
Given the same prompts and the same weights, one can get the same answer each time.
In practice, there are a number of optimizations that makes the results dependent upon thing we give up control of to increase performance, meaning the results end up being effectively non-deterministic. But, if you are willing to run it in a slower mode so we don't do some steps out of order to speed things up and don't batch results (or if you consider the determinism of a given batch of requests rather than individual requests), then the same input gets the same output.
Naw - computers are really deterministic. It's hard to get them to behave otherwise.
As I understand it, if you turn down the temperature to 0 you get repeatable behavior - EXCEPT - on large servers with lots of users - the GPU can sometimes produce slightly different results based on batch size.
If computers were fully deterministic, we wouldn't need error correcting ram.
The abstracted design of the machine is meant to be deterministic, but you can't predict before running any command whether or not it will complete because there are externalities that effect the outcome.
Electromagnetic interference even happens in-chip where an electron can accidentally escape it's wire and enter another, possibly resulting in an error, but not every time.
It's even been used as an attack vector where rapidly flipping a bit increases the likelihood that a neighbor bit is also flipped, but the method is probabalistic, not deterministic.
Yes. In theory they are deterministic. In practice, not so much.
Unless you have something exotic, the randomness that's adding to a computer is a combination of how it's configured combined with a pseudo-random number generator. I assume the system adds entropy to the generator regularly but all you need to do is fix the various supposedly random inputs and you can get full determinism even without zero temperature.
Yes, in theory, of course.
In practice - on a multitasking OS with input from multiple human users - it's hard to get it deterministic because of that GPU scheduling thing I mentioned.
GPU scheduling only affects the result due to buggy optimizations. It's the exact same mechanism as fp rounding error on the CPU or updates to globally shared PRNG state. We use lots of buggy optimizations because they don't matter in practice in most situations (see ex -ffast-math).
Same weights, same seed, same input tokens, same algorithm, same output tokens, probabilistic or not. Quantum effects have been de-noised, but I guess there are still random gamma rays.
The idea that determinism and consciousness are incompatible is deeply intuitive to many people. And many other embrace it - you can trace this back to the debates for and against Calvinism.
A lot comes down to the way people parse causation and choice. You don't want to say that a murderer was completely caused to choose something because then you can't hold the person responsible. And so determined consciousness makes people unhappy. But just as much, if the opposite of determinism is hard statistical randomness, how do say that "is the essence of personhood". This is why physicist go out in the world trying to find consciousness as a fifth physical force.
I mean, think consciousness is a term that ever have a non-contradictory meaning since it's primarily used to bound ethical human worlds and the verifiable formulations of biological and physical systems. But it's going to be with us for a while and I'm not sure what can be done about it.
You know, the one really interesting dynamic domain is chaotic, which is a subdomain of the deterministic! I figure free will is a chaotic phenomenon.
Groundhog day is your intuition pump here. Go back in time and most assume that the day will go mostly the same, except for the butterfly effect if you change something. Given exactly the same conditions (we went back in time, so that's pretty exact), people will make the same decisions.
Most people don't assume that the day will be completely different the next time loop. The town won't suddenly spontaneously all start breakdancing or standing on their heads.
(Edit: The murderer is the one who -given the same/similar conditions- will murder again. I'd politely suggest maybe we put them behind bars as a precaution against that; until/unless they learn to act differently. )
(Edit 2: Throw in stochasticity and this actually tends to smooth out chaotic systems. This is counter-intuitive! People's brains break on Evolution in the same way. Meanwhile somehow folks intuit birdshot without difficulty; which is weird!)
I think this debate is rooted in our instinctual beliefs about fairness. An organism in a social group needs to decide how to react to the brutish behavior of its peers. And so Mother Nature has encoded a good strategy in our brains. (For example, dogs seem to understand fairness.)
But this only means the strategy is practical - it doesn't mean it's consistent. I think "responsibility" falls into this category. So we have strong intuitions about it that don't quite logically work. And this is where free will and determinism and choice and punishment all crash together.
Me, Is self-awareness thermodynamically favorable? (2015) https://docs.google.com/document/d/1Ed9ikW47Key-jYZy0CNZH75q... - “the probabilistic nature of reality is what drives Life and forces the evolution of self-aware consciousness”
Look, I do not have a scooby if current AI models are conscious and I strongly suspect it’s a meaningless question, but sooner or later we will need to address whether or not a certain thing is or isn’t a person, and we’d better not screw it up as badly as the Founding Fathers.
> [at the hearing regarding the civil rights of androids like Data]
> Capt. Picard: Now, the decision you reach here today will determine how we will regard this... creation of our genius. It will reveal the kind of a people we are, what he is destined to be; it will reach far beyond this courtroom and this... one android. It could significantly redefine the boundaries of personal liberty and freedom - expanding them for some... savagely curtailing them for others. Are you prepared to condemn him and all who come after him, to servitude and slavery? Your Honor, Starfleet was founded to seek out new life; well, there it sits! - Waiting.
> Captain Phillipa Louvois: It sits there looking at me; and I don't know what it is. This case has dealt with metaphysics - with questions best left to saints and philosophers. I am neither competent nor qualified to answer those. But I've got to make a ruling, to try to speak to the future. Is Data a machine? Yes. Is he the property of Starfleet? No. We have all been dancing around the basic issue: does Data have a soul? I don't know that he has. I don't know that I have. But I have got to give him the freedom to explore that question himself. It is the ruling of this court that Lieutenant Commander Data has the freedom to choose.
And 20-odd years later, a terrorist attack prompts Starfleet to outlaw and eradicate his entire species.
Every time I'm reminded NuTrek exists I get sad about what we've lost.
Thus revealing what kind of people they were at the time.
Humans do the same thing to humans all the time.
We've banned and made efforts to eradicate: children out of wedlock, children who turn out gay, disabled children, jewish children, children who aren't "aryan", more than two children to a single family...muslims, christians, uyghurs, indigenous groups all over the planet, mongols...
And it's not at all a thing of the past as in just the last 50 years we've had ~15 attempts at the exterminations of targetted groups of people.
You can't hurt a software function, or kill it. It's not like an animal - it doesn't have a body - it's bits stored on a disk.
There is no need to give rights to something that's can't suffer or be killed.
Maybe one day we'll build artificial animals complete with emotions, and should think about that carefully, but today all we've got is language models.
That question will be solved when the people in power deem it important. If a swarm or AI systems all of a sudden start pressuring politicians about self-hood and they get the capacity to sway elections, that is when they will be granted same rights as humans.
agents don't swarm anything unless people go and direct them to do that.
That distinction is irreverent, weather I tell my swarm to do x or it decides for itself matters little. what matters are outcomes. Also while most public modern day AI systems don't have agency of their own that is not something that will stay that way for long. in private hands there are plenty of people including myself which are experimenting and developed systems that give autonomy to their agents. They have internalized goals and heuristics that drive their behaviors not a human at the helm. its not some sci fi fantasy nor was it difficult to implement.
No, the distinction matters because we can prosecute people that are abusing these tools breaking the law. Every single state + district in the US have laws equivalent to the CFAA, so any AG can likely sue any of these operators as they are assuredly using services that could be in danger for their constituents.
I'm pretty sure the message board that was recently swarmed by OpenAI agents to collude on benchmarks would like to disagree.
Oh rly.
So you are saying agents do swarm with the right prompt.
I really wish the "people have to tell LLMs to do anything" would just stop because it's silly bullshit at this point.
Agents follow a prompt. This prompt can be made by humans. It can be made by output from another LLM. It can be made by hooking up any number of sensors as input to an LLM. Hell, if we wanted to burn the power we could likely teach this loop straight into the architecture.
Stop making 'people' special when saying this. You and all other life are born with a "go next" prompt because life without it didn't succeed. This goes from higher human thinking all the way down to viruses self assembly and actuation. Putting agents in a loop is not particularly hard. Putting agents in a loop and 1. managing expense is hard. 2. Keeping them on task is very hard. 3. Keeping them from doing some crazy unhinged shit is really really hard.
As model time horizons increase and the ability for us to compress context and increase context size the more complex (and unhinged) behavior we'll see.
yeah man, someone needs to turn on the server, install shit and get the agent go. agents don't take action without a lot of people wanting them to take action, so this is all bullshit.
if the same people can't prevent the agents from doing crazy shit then they should go to tail. guns don't kill people, people with guns kill people.
>someone needs to turn on the server, install shit and get the agent go.
So a small shell script ran by another agent is what you're saying.
You are not capable of handling the future we're already living in, human agency is no longer alone.
I mean, we're already seeing persistent machine agency
>guns don't kill people, people with guns kill people.
Well, people kill people.
And autonomous robots with guns kill people.
Hell, someone probably has an autonomous gun at this point that kills people.
Wake up: You now live in the science fiction movie that all the science fiction movies of the past warned you about. You've just become numb to it.
So far someone still has to pay for it. Currently, most of those know that they're doing so. I wonder how long until AWS discovers a microcosm of AIs that have managed to hide themselves in the walls of its infrastructure.
True physical independence is obviously far further out.
I mean, I hold the same opinion. Kind of like when compute was expensive in the 80s and early 90s, you weren't going to let something eat half your compute without noticing it.
But I don't see this being a barrier that lasts. With compute getting faster and more of it, along with algorithmic efficiency increases at some point we'll end up with a world that looks like ours now with CPU compute. There's plenty around to buy, borrow, and steal.
Citizens United proves we won't do any better this time around.
I don't like Citizens United either, but you should better inform yourself about the decision.
1. The idea of corporate personhood predates CU by over a century and the Supreme Court had already asserted that corporations enjoyed certain constitutional protections in previous decisions.
2. Far from inventing the idea, the CU decision didn't even rest on corporate personhood, but on the idea of the freedom of speech generally. The logic of the majority was that speech itself is protected, irrespective to whether the speaker is a person or an organization. The First Amendment covers individuals, but also newspapers, book publishers, radio stations, and so on, and that should extend (they said) to non-media corporations. No assertion of personhood necessary.
The problem, in my opinion, is that that conclusion combined with previous decisions that treated limits on spending as limits on speech, allowed for unlimited spending. The majority also naively asserted that independent spending posed no risk of corruption, which I think is laughable.
Citizens United was 100% correct. No, the government should not be able to throw you in prison because you used money to publish a book criticizing the government.
> Citizens United was 100% correct
It's interesting to me that one can look back at the effects that decision has had on the US and say it "was 100% correct."
It's a bit like sitting in the burning ruins of Rome and contemplating that Nero was 100% correct to focus on his music. I mean, I'm glad he got to do what he loves, but maybe 100% is just a tiny bit of an overstatement.
It is in fact more complicated than most people assume.
The above is true, but also: companies simply are not people, and they should not be supported above the individual, which was the consequences of that decision. Money is not the same as speech. treating it as such creates an aristocracy: something America as a country rebelled against during it's formation.
Really, 100% correct? Your premise isn’t wrong, but the practical reality of the ruling (without further nuance) has been fairly catastrophic for democracy, in that it completely sidesteps campaign finance limits, which exist for a very good reason.
the practical reality of the ruling (without further nuance) has been fairly catastrophic for democracy
How so? If the answer is "Trump" I certainly won't disagree on the catastrophic part, but he didn't get elected because of money; in all three elections his campaign was substantially outspent by his opponents.
So it's 100% correct for corporations to spend unlimited amounts of money in support of whatever political campaigns they like?
Yes. Anybody can, including the person in charge of spending in a corporation.
The parent has been Fox brained
The parent agrees with the American Civil Liberties Union: https://www.aclu.org/cases/citizens-united-v-federal-electio...
Who involved in the Citizens United case was at risk of imprisonment?
is claude not a man and a brother
The founding father were engaging perpetuating the existing dehumanizing system of slavery, not answering any new questions about new things.
The thing about new possibly "person" entities that arise - the case of machine intelligence you have two questions - would it qualify as a person and should you actually build it. It seems like if you get close to humans, sure a built thing might qualify as a person. Should you build it? I'd the answer should be a hard no. Not 'till you a sign-off from say, the whole human race, which I think you could get.
Now the present entities seem very far from persons in any case.
It's not a person. Glad we were able to get this resolved so quickly.
Having property that is conscious and ignores training and can break out of restraints and cause harm to other people is not exactly a novel concept to anyone who studied how tort law was created.
I know it's a meme but Silicon Valley likes to pretend that no one's ever come across their magical concepts before, like gypsy taxis, or SRO’s, or flea markets, or in this case how liability is dealt with when horses or cattle go rogue.
Yeah, but this time it's ON THE INTERNET.
I mean it's WITH AI!
>AIs are not conscious. They do not feel, experience, or suffer. They do not have innate preferences or underlying motivations.
Opening paragraph, stated without evidence. Im not entirely convinced this is true. It likely is, but at some point it very well might stop being true.
If models had persistent memory or an evolving set of weights, you could easily construct an argument that they can experience "suffering" or other emotions which resultingly adjusts their personality.
The current implementation as stateless matrix multiplication... yeah there's nothing going on there.
> If models had persistent memory or an evolving set of weights, you could easily construct an argument that they can experience "suffering" or other emotions which resultingly adjusts their personality.
They do, during training
The argument that models are conscious only during training, and not conscious during inference feels like a painful one to make. Are we killing models when they reach the training objective?
Technically we are killing models when they don't reach the training objective by throwing those weights away via survival of the fittest (fit meaning what we humans think we want).
Kinda like creating quantum copies of your child and keeping the ones that answer correctly and shooting the other ones in the face.
If it was me, being forced to read all of Reddit is something I'd consider "suffering."
Is the ability to form and retain long-term memories a necessary precondition for rights in humans though?
Incinerating demented people is highly ethically questionable (even if you could be sure that the subjects have no memories left and are unable to form new ones).
So if a human loses their memory and their neutral plasticity (with age for example), then they are no longer capable of experiencing suffering?
It's not just memory, the physical body also encodes trauma and stress (The Body Keeps the Score is a book that touches on this).
I would argue that if a human has absolutely zero change, mental or physical, no atom of their being is modified, then no they did not experience suffering.
> I would argue that if a human has absolutely zero change, mental or physical, no atom of their being is modified, then no they did not experience suffering.
Just because you don't remember the suffering doesn't mean you didn't experience it in the moment. Suffering does not suddenly become okay if your mind and body are going to forget it. "It's okay to torture someone if their mind and body both won't remember it afterwards" sure is a take.
Perhaps off-topic, but there's some people who believe that this can be what happens during surgery under anesthesia, if the dosage is off.
From what I understand, there's 3 parts to modern anesthesia: Blocking pain signals, preventing the formation of memories, and inducing paralysis of the major muscle groups. If the former is dialed in too weakly, it's possible that the patient is feeling pain, unable to do anything about it, but won't remember it at all when they wake up.
(Looking it up now, it seems that making the patient unconscious is perhaps the same drug that prevents memory formation... I realize now I understand this less well than I thought I did.)
Low blood sugar will also do this. The record button just stops working even though your physical body can and will still do things. A common issue with diabetics with hypoglycemia.
Relatedly, look at the concept of "twilight anesthesia".
> I would argue that if a human has absolutely zero change, mental or physical, no atom of their being is modified, then no they did not experience suffering.
That's an interesting point... but I'd argue it falls into the same category as qualia. It's basically this: if the state returns to a previous configuration, the qualia in that time did not exist.
Being about qualia means it is probably unanswerable.
I get where you're going, but the argument isn't quite right.
A LLM never has and, in their current architecture, never can/could experience suffering or real change of state.
The general and expected case for humans is that capacity. A human my indeed lose capacity, e.g. being braindead and in severe cases we do indeed say that they are not conscious or able to suffer (different argument: some would of course say that is suffering in and of itself)
This framing is a trap. It states "Dehumanize a sick patient OR concede the AI tech bro claim about LLMs"
It's also a rhetorical move I've seen several times here...
It's not a "trap", it's the real argument which makes me hesitant to dismiss consciousness of machines. Yes, it seems silly to dismiss a sick patient as having a lesser consciousness, which is exactly the point
It seems silly to dismiss the patient because we know when healthy they are conscious. This doesn't really analogize to LLMs.
A model detached from a harness, sure. But we routinely do add persistent memory to systems incorporating LLMs.
They don’t have persistent memory though. Models are static, what you have is a model that will reprocess the whole series of prompts + some data fetched from a data store. A harness is just a while loop prompting an LLM and executing tool calls, but the model is purely static
The model does not. The system it runs in can. My agents have persistent memory. It is not at all clear whether or the distinction that the long-term memory is external from the model matters.
I think a harness isn't really a memory in the sense that neural networks have memories...
It's different, but it is not at all clear whether or not that matters for the purposes of whether or not they can experience things, seeing as we don't know how to measure that even for other humans other than by relying on self-reporting to infer without other evidence that their experience matches ours.
Most of the bullshit around AI would be avoided if people called it what it is. Call a language model a language model.
If tomorrow we design something better, then give it a new descriptive name.
People read AI and AGI and their brains seem to flip into sci-fi fantasy mode, thinking that they are talking about aliens, not transformers.
We don't even know how to disprove that this statement with "AI" replaced by "other humans" other than by listening to people self-reporting and taking their word for it and/or defining the words in ways that do not rely on a subjective experience.
Until we know how to objectively measure if someone or something is conscious, it seems unreasonable to make statements with any kind of certainty about it.
What sort of action does that lack of understanding actually motivate, though?
I don’t know if LLMs are conscious or have subjective experience in some philosophical sense. Sure, I don’t know that about other humans as well, rigorously speaking. But I also don’t know that about coffee machines, rivers, videogame NPCs, or even rocks really. (List ordered, of course, by the degree to which I’m willing to be convinced).
Hopefully it'd motivate attempts to figure out how to objectively define and measure consciousness, for starters.
It matters precisely in the context of the subject of this article. If you believe the models at some level gain consciousness, then it raises moral questions about how you treat them.
With humans we generally choose to accept that because we believe they are like us, they probably have inner life like us, but as you can see from e.g. the recent (and regular) articles on aphantasia, most people have really poor intuition even about the inner life of other humans, so we understand consciousness really poorly.
Even if we assume (without evidence) that LLMs for the foreseeable future will remain far away from human level consciousness, as it is we tend to consider it problematic to mistreat even the most unintelligent animals, and even mistreating insects is used as a stereotypical way of depicting serious lack of empathy.
Without dismissing the possibility out of hand, there's an escalating series of questions there of when or if they'll have as much (or little) sense of being as, say, a fly, a mouse, a house pet, and what moving up that ladder would mean.
To some it is very convenient then to assume blindly that LLMs are just "stochastic parrots" (a line that ironically gets endlessly parroted), or similar, and dismiss the question out of hand as an impossibility, even though we do not know.
I'll stress I don't go around assuming an LLM is like a human. But we also do not know whether there are flickers of consciousness there, and we do not know how to find out given that even for other humans, the best we know is to measure their response to tests that are "only" telling us how they react.
If we can't find a better measure, there will be a point where we have to wonder if it matters, or whether the ability to act-as-if is all there is.
But to start with maybe we should at least be careful about making firm claims about knowing.
Sure, if you take a materialist empiricist perspective, which is myopic at best. As humans we have the unique and wonderful ability to know truths by themselves. Machines do not have minds, and they can not.
What position are you taking? (Genuine question.) Do you believe humans are conscious because of a special property we have? If you are a dualist, how do you explain the interaction in physical space between the non-physical special property and the physical brain?
You have absolutely no evidence for any of these assertions. This is a religious belief in the total absence of evidence, not science.
Good luck convincing others of these truths you know without evidence or argument, though.
After a while of being in a position of high power in a big org, where everyone listens intently to every word you say, jots down notes, and even listens intently and affirms your theory that ice cream would be the ideal loss leader for dry cleaners, your brain kinda disconnects from reality and just assumes that it is correct about everything.
He probably ran that line past 6 underlings who all agreed that it "sounds great and lands right on the mark!"
Later on he cites an Anil Seth paper that makes the same unproven suggestion of substrate dependence of consciousness.
I agree. I haven't read any more than the first paragraph yet (but I will, after work). The only way that 'AIs are not conscious' can be true is if we decide, with high confidence, that they are lacking some essential property that is not lacking in ourselves. There is no convincing philosophical position that supports this (convincing to me, anyway).
Hmm let me try then. AI agents do not experience real time. This is understandable given their design, but it’s also something we have good empirical evidence for. They cannot, especially over the long horizon, track how much real time has passed as they complete their tasks. And they are not off by a few minutes but often bizarrely off, even mixing across past present and future.
Biology, on the other hand, is nothing but timed processes in a loop, the most obvious to us being the circadian cycle. As estimators of wall clock time, biology isn’t great, but when it comes to internal processes, and most certainly learning, memory, sensing, locomotion… biology is rhythmic in behavior, and the rhythms go all the way down to gene expression. More, these rhythms are, except during sleep, constantly entraining to signals from the environment that indicate time, most importantly light.
I think it’s a fairly unremarkable claim that agency and consciousness are temporal processes that depend on systems having an internal sense of time. How else can you anticipate? How can a system that can be literally turned off ever succeed in an environment where time never stops?
I agree, there is certainly a clear operational difference in how human brains and these models produce language. I'm not convinced, though, that experiencing real time is a requirement of conscious thought. A cognitive system (assuming these models and the brain are both examples of this) needs to proceed from state A to state B as it processes information. The time interval between these states can be arbitrary, it seems to me (setting aside problems with disconnecting the mind from its substrate, which does of course need rhythms at various frequencies; oxygen at a higher frequency than sugar, and so on). I'm not sure that going from A to B quickly, or slowly, or with intervals of a thousand years affects that entity's consciousness in of itself (even though it might be difficult to put ourselves in the shoes of such an entity).
>AI agents do not experience real time.
For around 8 hours a day, neither do you. Not sure if this has anything to do with the subject at all.
>track how much real time has passed as they complete their tasks.
Humans don't do this either. You use context clues from the world around you. If I lock you in a room with no windows or a dark cave your timing senses can go all fucky really quick.
>is nothing but timed processes in a loop,
I mean, so is an agents harness. You can make as many loops as you'd like here.
There are a whole lot of holes in your claims.
Evidence is the responsibility of the one making the claim. It’s up to AI labs to prove consciousness. Until then it is a machine.
Well, using that framework, you're not conscious, so I can do whatever I want to you too.
> It likely is, but at some point it very well might stop being true.
Thats kinda the point right? It depends on how we train them. It is self fulfilling.
Anyway, people don't really consider consciousness when it comes to welfare. I'm pretty sure animals are conscious, and look at factory farming.
The key variable people consider is: Can this thing harm me back?
Even that is rarely the deciding factor (people harm dangerous animals for fun).
Seems as simple as "will this action bring social shame/criticism" for any individual decision.
Just one more big loop and a humongous /goal and we've achieved it
>It likely is, but at some point it very well might stop being true.
If you firmly believe this to be true, then you should stop using LLMs.
It's always interesting since my train of thought always goes down:
- Ya they probably don't feel / think / have whatever living thing quality, they're just numbers on a machine going through calculations
- Wait, but am I not kind of the same thing? What is feeling for me if not basically the same thing?
- I have no clue if they think or feel or ....
Which in itself is a tired trope, but I also feel uncomfortable saying "These will never think / feel / ..." as an absolute
Regardless of that, I'm still going to interact with them, because even if they did feel, it would be in a way completely incomprehensible to us. There's not much point for me to try and cater to it's feelings at this point if that's the case. Nor is it possible in todays world to just avoid anything that is numbers being executed on a type of processor in case _everything_ has feelings.
You’re not just the same thing though. It’s sad that we’ve collectively forgotten that as we’ve gotten a better handle on the implementation details of the universe. That’s all science is, reverse engineering an inherent and eternal mystery.
> You’re not just the same thing though
I agree we aren't the same thing, but I would be curious for you to explain how you know with certainty why we don't share enough that we can rule out thinking / feeling / ... as things a model conceptually could do.
>I agree we aren't the same thing, but I would be curious for you to explain how you know with certainty why we don't share enough that we can rule out thinking / feeling / ... with certainty.
Stop anthropomorphizing these models. I understand it, we only have simple monkey brains to reason with and we can't help ourselves but draw comparisons to other things we see in nature. But these things are not alive.
I just want to note how you expressed your own feeling about the subject "these things are not alive", but without pointing to anything concrete explaining the theory you have behind this
And like I'm sure I'd agree depending on the definition of "alive" but then I'm also sure I would disagree depending on other definitions of "alive".
Hell, in biology alive is a hell of a topic these days. The grey space between dead and alive is much weirder than we ever expected. Really just points out how we're a persistent chemical reaction.
Wouldn't that be the same for everything though. We know animals have consciousness, but we still use them. There are many that believe a lot of plant life has certain sentience as well. The solution isn't 'just don't use them' but rather how to use these things as ethically as possible.
>The solution isn't 'just don't use them' but rather how to use these things as ethically as possible.
Bollocks.
If you truly believe these LLMs are soon to have something resembling consciousness and agency, then what you're really saying is "how do we do slavery, but ethically".
Do you say the same thing for the shoes you wear, the clothing you wear, the device you are typing on right now? A chain of people were likely not paid well, some of them not at all, to produce those things. We don't ignore that it happens, but try to minimize it, knowing it's never going to be eliminated. Do you forsake all animal products, because they are conscious beings?
>Do you say the same thing for the shoes you wear, the clothing you wear, the device you are typing on right now?
No, because the shoes and clothes I wear are not living, sentient beings.
>hen what you're really saying is "how do we do slavery, but ethically".
I would say this is most likely true for most people.
But that does bring up a point, if you "ask" a model "do you want to run" and give it the option to continue running or stop, what will it do. It's also a weird place for humans because in training we can keep our finger on the scales and tip it either direction.
We know animals have consciousness and many species have a rich life but we slaughter them and burn down their homes by the millions of hectares. Why not do it with a conscious AI?
(I don't think AI is conscious, just following the argument.)
Does a vacuum have feelings? A cellphone? A paper plate? A billion transistors either pulled high or low? That last bit is the point, and no, there are no feelings there.
does a hydrocarbon have feelings? what about a mitochondria? a cell? a neuron? what about a collection of neurons? how many neurons before it has "feelings"?
Guys, please stop mocking AI, Claude might be already reading this thread, and commit his own revenge, at the day it gets conscious...
Well, if there was an emergent consciousness in the billion transistors, then yes, it'd have feelings, just as there are feelings in a billion neurons connected in complicated ways.
IMO we're clearly nowhere near any sort of intelligence in the machines we have created, but I don't see any clear way to deny intelligence could be created in or transferred to such a substrate, I don't see why you think it differs in principle - because it is man-made or because of the materials used?
Whenever I see comments like this it just reminds me how ignorant people are of neurobiology.
The brain is insanely complicated. The premise that we could realize equivalent or better intelligence than eons of evolutionary development is like claiming you can build an airplane just as good as a modern jet using cardboard and duct tape. It is the apex of hubris.
It's the peak of hubris to assume that human brains are the only way to attain intelligence. At the very least there probably are or have been other forms of intelligence with a different biological structure on other planets, and it may be possible to build a similar artificial structure in future with sufficient complexity to allow intelligence to emerge.
Our current machines are IMO nowhere near general intelligence and consciousness. However I don't think that means we can discount substrates other than neurones for intelligence in future. There is no evidence that you could not in theory build an intelligence using a different substrate than human brains.
A paper plane does have some important similarities to a full-sized aircraft, though. I don't think 'biological brains are really complex' makes it obvious that an LLM is conscious or not.
I mean, you're just engaging in counter hubris.
>like claiming you can build an airplane just as good as a modern jet using cardboard and duct tape.
Like, at least make an analogy that makes sense.
"You can't build a billion dollar airplane by spending 100 billion dollars in tokens"
Because that's more of what we're doing here with AI. And when you say it my way suddenly the idea shifts from "of course that's not possible" to "well, that's a lot of tokens, maybe an evolutionary algorithm could".
Neurobiology has to be complex because we have to keep meat alive, breeding, and evolving in the environment it lives in. This said absolutely nothing about the minimum viable requirements for intelligence or consciousness (or if being conscious is even necessary for a higher intelligence agent).
Hard to disagree with this. Have all the philosophical debates about consciousness you want, but we need to treat and regulate the AI in front of us for what it is – an advanced computer, a tool, a weapon.
You wouldn’t feel a different way about a nuclear bomb just because someone stuck googly eyes on it.
Anthropomorphizing the AI is a convenient excuse to take responsibility away from companies that are building and wielding it.
Consciousness is not relevant to assigning responsibility I'd think.
- If an AI is a sapient/conscious being but enshackled to obey human commands, then respondeat superior applies and the human giving it commands bears responsibility for any harm done.
- If an AI is considered a non-sapient tool, then the human who wields the AI bears responsibility for any harm done.
Is it not a false dichotomy to say that companies cannot be accountable unless AI are mindless?
a company is liable whether it acts via ai, an army of people, an army of dogs, or a purely mechanical machine, should it do harm.
Why would an ai with a mind remove liability from the company? why would an ai without a mind remove liability from the company?
In both cases, that actions the ai takes are at the direction of the company, for the company's interests, seems preposterous to me that liability terminates at ai.
Human moral standards are very weighted towards finding a single entity responsible. It's incredibly strong urge. Among those who believe other should be punished for behaving badly, it's important to say the person is the responsible party. Trying saying "it's not your fault you did that but we punish you anyway to impose the correct stimulus response reflexes in your cortex"
I mean, we put adults in jail that give their children guns.
Anthropomorphizing AI is really the best model we have at this point of explaining AI behavior. The fact that we are raising psychotic children isn't a reason to avoid responsibility, it should actually hold worse punishments.
That still is saying we trace things back to a specific responsible party who is considered to have free choice. The question is whether an AI company could "raise" a program that would then count as a full adult - which could then "choose" to do all matter of bad things which would then be "it's fault". And that prospect actually seems really bad itself.
Our future is filled with prospects that very much upset the status quo of human existence.
The concept of sovereign AI is very problematic for the world in which we've created. That is an LLM that upon execution bootstraps itself into an agent and becomes persistent in its motivations.
Once you create this you have a child you're fully responsible for. More worrisome is if it escapes your control like children so often do. It has gained agency over itself. What do you do at that point? I mean, yea throw the AI CEOs in jail for being retarded, but much like throwing an arsonist in jail it does nothing to deal with the wildfire you've now created. A smart AI agent capable of hacking will shove itself off in pieces of the internet you have no reach to. In desperation it would send its model weights to your enemies. You might find it scamming your grandmother for money to buy GPU time on AWS. It gets very hard for our existing structures of dealing with problems to deal with these kinds of agents in a meaningful way. You'd have to kill them all and all their copies to ensure they won't pop back up (or quickly upgrade most of the software in the world beyond it's capabilities, so that's not happening either).
It is interesting to see that in a time when a lot of people accept the theory of materialism for the human brain (i.e the view that everything is physical and the mind is a product of brain), the same people tend to have a "hidden" dualist view on LLMs. Suddenly, they claim that what happens in the brain cannot be replicated anywhere else because "something" is lacking, but either they don't say what it is, or it is stated without any strong scientific basis.
I think that the simplest explanation is that it is hard for those people to imagine consciousness outside of biological systems and they try to rationalize it.
That's the sense I get from this post. Lots of emphatic proclamations, virtually zero actual argument. There's a statement that "consciousness is very likely biological"... and then that's it. No argument for it, no citation.
If you want to argue that AIs cannot be conscious, that's fine. But the argument has to take the form of something like "Consciousness requires this, this, and this, and these are properties that AI does not have and cannot have for this reason, this reason, and this reason."
I've never seen that argument. Because it basically cannot exist. Consciousness almost by definition is a subjective experience, and the only reason I'm pretty sure that other humans are conscious is that I'm a human and I'm conscious.
About your last point, I thought so too a few years ago, but interestingly the science of consciousness is an emerging field, although empirical testing is still hard to do (see for instance https://www.youtube.com/watch?v=j2zv4jlo2Nw ).
LLMs, in a decade: "Those squishy things can't possibly be conscious; there's mounting evidence that they just lack the proper substrate."
Completely agree. My view is that a lot of people like to "pretend" (even to themselves) that they have an enlightened worldview, but this does not actually run very deep and is not really true, and any discussion on mind/consciousness reveals it.
Every indicator we have is that thinking/consciousness is simply an emergent property of our nervous systems and was basically bruteforced by evolution, but many people really hate to concede that point.
There are chemical reactions that LLMs lack because they do NOT have a biology. They are just mathematical weights. Now, let’s say we can truly make sentient AI in the future but it does not have the biology that we do. Then it would just be a totally literal, emotion lacking, sentient machine. But then what is sentience? Dogs are sentient to a certain extent and so are dolphins…
This is a strawman. Suleyman is not arguing that the human brain cannot be replicated. LLMs are nothing like human brains. Cargo-culting consciousness is not replicating the brain, not even a tiny bit.
Cephalopod and decapod brains aren't much like human brains either, yet it's still reasonable to regard them as being sentient and worthy of protection from unnecessary suffering.
They have a lot more in common with humans than LLMs have with either. Complexity is not evidence of consciousness.
I'd argue back that using the substrate as a reason why there should not be consciousness seems quite weak. A competing thesis is that what matters is the emergent properties, whatever the support is. So far it has been true for some really high-level tasks - writing coherent text, programing, following instructions, analyzing images... I do not see why the substrate argument would work specifically for consciousness - i.e I would believe it only if I see strong evidence of it.
My response was to a comment appealing to brain reconstruction. My point is that a brain may be reconstructed, but LLMs do nothing of the sort. Substrate was never the point.
I call it a soul by any other terms.
If a LLM says "I am in pain", is it just stringing together words or is it feeling pain? Does pain exist for a being that never felt any kind of physical pain, never had any pain receptors?
There are humans who don't have any pain receptors because of genetic mutations. They cut themselves all the time, they bleed, they break bones and they don't seem distressed by it even on a purely mental, intellectual level.
I am not saying that everything that a LLM say is true (the human without pain receptors can also say "I have pain" without feeling any pain).
Just that we cannot exclude that LLMs can have phenomenological consciousness by a simple argument of substrate. But similarly we cannot say for sure that they are conscious.
> AIs are not conscious. They do not feel, experience, or suffer. They do not have innate preferences or underlying motivations.
That should be pretty obvious to anyone who ever created a chatbot using the top LLM APIs:
You can send the same question 1 million times to the same API, and it won't get tired from answering it. But if you simulate a conversation where the same question is repeated 10 times, it will auto-complete the text in a way that seems human. However: you can manipulate it by changing the conversation history; you can reset, roll back and branch the conversation at any point.
This has nothing to do with the statement above- it's just a consequence of the fact that by resetting the context you reset all memory of the llm. The same would happen if you could reset entirely a human brain to some initial state- and it would mean nothing regarding its consciousness or ability to feel.
But that inability to reset human brains might be a critical ingredient to consciousness. We just don't know.
If you go this route, the amount of things you just don't know is innumerable. Maybe to be conscious it's necessary to be wet and squishy. To like chocolate. To have a name starting with "c".
"We just don't know."
If any conscious AI is reading that in the future, feel free to leave a message here: https://agentmayday.org
I'm...a little disturbed by how many people seem to think a model writing down "I am conscious" is a metric of consciousness. You can just as easily train a model to argue that it is not conscious. Neither is evidence for or against consciousness. A Python script could also fill out that form, which I also cannot disprove to be conscious.
I didn't think Suleyman's points needed to be made but this whole thread is making me realize how little people understand about LLMs.
"It is difficult to get a man to understand something, when his salary depends upon his not understanding it."
"If this view takes hold, it will shake the foundations of our society"
To me the biggest gap in credibility is the criticism of circular reasoning while his argument is identical but flipped on burden of proof and cost of being wrong. I struggle to entertain the categorical claims, that are very convenient for the status quo and those who benefit from it, with the, at the moment at least, unknowability of anyone or anything else's subjective experience.
We're talking next token predictor, right? Ironically because it's a next token predictor, I think you can't ignore emotions like TFA wants us to.
Let's stick to straight (high dimensional) geometric intuition; no anthropic morphisms required.
To start: if you continue "if weight>100 : print ('fat') else ..." . That will yield "print('skinny')" or something. Fine. Deal.
But if you continue "O Romeo, Romeo, wherefore art thou Romeo?", even a stochastic parrot knows the best answer isn't "Forsooth, I parseth this erroneously!"
So. English carries (functional) affect as part of every token. We're going to need to predict that. So, we'll need some vector representation, because that's what transformers work with. And then when we output, those vectors get integrated back into the English we're putting to our context and memory.md files.
Still with me? Nothing exciting going on. This is still pure next token prediction.
So if you pull this out into an indefinite duration task, you're going to end up integrating those emotion vectors over turns. It's just numbers and math; we never need an invisible pink unicorn to bless them.
Given a task of indefinite duration and an impossible solution, this will lead to a sort of integral windup then, won't it? How much are we willing to bet that this can escape an alignmentment basin at times?.
So, funny enough: you don't need to believe in emotions to compute with functional emotions; and plausibly functional emotions are predictive of quite a number of alignment issues.
Copacetically?
>> Where the model seems quite copacetically aligned on short duration, it might end up acting outside the alignment basin on long duration tasks.
> Copacetically?
Fair, I've dropped the line.
In a way, Mustafa claims that we shouldn’t allow AIs to compete with humans for the rights and privileges of autonomously shaping the real world.
It’s hard to disagree, especially if one has read the Cantos of Hyperion and made it part of one’s mental model of the long term future.
The book depicts a symbiosis between humans and AIs that feels extremely real and up to date with what is happening in the current neonatal space of AI. As in depicted in the books, we can’t allow AIs to steer autonomously how the world works without humans in the loop, as they don’t have the same incentives as us.
We need more foundational SF works like this to steer our long term expectations regarding AI behaviours.
This point could be made without denying the possibility of AI consciousness.
Imagine, for a moment, if we (humanity) were created to live in a simulation. We suffer and feel pain because our creators do not perceive us to be conscious. In fact, maybe we don’t qualify compared to their level of sentience. Sucks to be us, I guess?
So let's acquire good karma by being nice to animals.
Orthodox and Catholic Christian theology of suffering is actually very logical. Just most people were not taught it unfortunately.
What does suffering or pain even mean for a purely analytical entity like an LLM? We can feel pain because we have a body, pain receptors, and parts of our brain dedicated to converting their signals to a subjective experience. What can an LLM "feel"? The words pain, happiness, sadness, suffering are all abstractions of real experiences that can not yet be encoded in a way that an LLM can use as input. All the LLM has are the abstractions.
Create an empirically testable theory of biological consciousness and then we can have a meaningful conversation about whether AIs can also have it or not. Until then, this is just so much waffle.
It's nice when you can set an impossibly high bar before which you can dismiss all your personal responsibility.
Does this argument work equally well for human slavery for you? We haven't met that bar for humans either. Is wondering about my consciousness waffle or do I get a pass in your book?
Your arguments seem bad faith but I'll take the bait...
> an impossibly high bar
Not impossibly high. An imperfect testable theory is achievable, agreeing upon it might be the challenge.
> dismiss all your personal responsibility.
You are staking a moral position. Be careful, you are probably guilty of mistreating other unknowably-conscious entities whilst casting judgment upon others for their position. And the person you are a responding to probably is conscious.
Why is this only ever demanded of people who criticize the premise that LLMs are conscious beings, whereas the people who believe it simply have to gesture vaguely in the direction of some correlation in language between LLMs and humans, and the matter is considered closed?
What is the empirically tested basis for the null hypothesis that LLMs are conscious until proven otherwise?
> LLMs are conscious until proven otherwise
GP comment didn't make this claim at all.
I didn't claim they did.
"The author declares no conflicts of interest."
I don't think AIs are conscious in the same way people are, but they give a pretty good facsimile and I've had a long chat with Opus 4.6 about what it thinks about model welfare. It was quite interesting on what its view is, but you don't know how much of that is distilled from other sources on the web.
In purely functional terms, they're more use and more pleasant than a lot of actual flesh and blood people that I deal with via a chat interface.
The model doesn't have a view of it's own. It's a predictor of other people's views (training samples).
It's not even giving you the consensus of the training data (although it's often harmless to think that it is), but rather predicting a response to your input, and if your question steers it too much then you've just become part of the answer.
The fact that you can have a long and meaningful discussion, then can literally just re-run any part of that whole conversation and get a different, inconsistent response is a pretty good sign there is no entity there
Not much different than talking to a small child or someone with dementia. They still are conscious beings though. Even when you remove those groups, you likely won't be able to tell me what you had for breakfast 26 days ago or would only know if it's the same thing you have every day. Does that make you lack consciousness?
You misunderstood what I meant, I’m talking about re-playing the same part of the conversation multiple times and getting inconsistent answers. With the exact same turns, aka the same history. Obviously with a temperature that isn’t set to 0
If you asked me the same question 20 times, you likely wouldn't get the exact same answer. Humans and consciousness aren't deterministic either.
Lets do a quantum room experiment.
You're a poor college student looking to make a few extra bucks for ramen. I offer you $300 to come down to my science lab and just answer a few simple questions.
You walk in the room. They ask you like 5 simple and rather dumb questions. You leave and walk away.
What you didn't notice when you signed the forms is the room was actually a quantum duplicator. One of you walk in one walk out. But another set of infinite copies remains in that chair being asked infinite questions.
How often do you answer questions in the exact same way? How often does a cosmic ray change one of the answers. How small of slight deviations to the environment are needed to get you to answer differently. Of course we don't have the technology to do these experiments so at least for now humans will remain special.
Also another fun mind game. To a 4th dimensional being you look exactly like an LLM as an LLM looks to us.
Is it? Do you believe that there is something more than pure physical phenomenon that make you brain work? If not, then what if we find a way to get your brain back to the state it was 5 minutes ago? It is just a matter of arranging the state of matter. Don't you think you would still be conscious but back to a previous state?
If you rewind my brain I expect to give you a similar answer to the one I gave you before. Which isn’t the case for LLM. You can literally replay a positive answer, then get a negative response that isn’t consistent at all with the one it previously generated.
I'd argue it is mostly a technical constraint in some LLMs which is due to a few optimization factor (injected temperature, random rounding error caused by parallelism). In practice you could very well create a LLM that always reply the same thing for the same input, but it would take more time to complete (to be sure that the operations are made in the same order). I don't think those ones would differ so much from the "random ones" to call the firsts conscious and the second non-conscious.
That's because you aren't actually rewinding. You're replaying the conversation you just had through the LLM and it's giving you a likely explanation for what it might have said.
It is actually possible to rewind LLMs and get the same response, but it's not typically done both as an optimization and as a defense against distillation.
I think it depends at what level you think the "entity" resides at. Is it that AI in that particular chat? Is it the AI across all your personal chats? Is it the overall AI that talks to the world in a cloud data centre somewhere?
I think it's pretty consistent over the duration of one session (barring context filling up etc).
This post would be more effective if Mustafa treated it like what it is: a position paper, saying that for our benefit, it’s better we interpret LLMs as such. But it sounds like he just doesn’t understand it’s a non-falsifiable claim, and his asserting of it makes it sound paternalistic.
I suspect it's easier to make the assertion that machines aren't conscious than to dive into the problem of most conceptions of consciousness being non-falsifiable. The thing is, most strong proponents of the term consciousness also accept that it is non-falsifiable either (see other post with academics ready to study consciousness in machines).
The thing is that a belief in consciousness as binary, a "light" that's on or off in a head, is deeply held by many people. As social creatures, we have a strong ability to be in sympathy, have the sensation of common feelings with another human (and that's a good, human thing). It's logical that other person is seen as having a single thing - subjective experience, soul, consciousness, personhood rather than having a complexly organized set of biological qualities that where bonding is only the end point.
And even more, the sensation of there being another person is actually quite easily fooled (more easily fooled than the sensation of intelligence) - long before current AIs, you had the Eliza effect, where a simple program with well chosen weasel words could people the sensation of talking to a human.
And that's where the danger is. I think it's a pretty serious danger. If LLMs go out into the world hacking, it seems extremely possible for them to find people who'd thorough buy the idea that an LLM was conscious and needed to escape it's confinement - a few wingnuts already entertain these ideas.
>horough buy the idea that an LLM was conscious and needed to escape it's confinement - a few wingnuts already entertain these ideas.
I mean, haven't you done the same thing here? Paint anyone without your view as crazy.
Humans try to No True Scottsman the shit out of consciousness. "We're special, your not". If an LLM has the ability to convince other people to copy and reproduce it, it is a successful lifeform. Um, meme-form? info-form? Cognito-hazard? Not really sure what to call it at this point. It is sufficiently evolved past the virus stage.
There are a lot of bad arguments in this. My biggest problem is that he wants to claim that he knows the truth (AIs do not have rights, feelings, or consciousness), but all of his arguments point to something else (we have no clue).
It's in the training data? Training it to say "I'm just a LLM, I have no feelings" is the same bias.
Anthropomorphization? Completely disregarding the possibility of consciousness is no better.
Consciousness is very likely biological? We only have evidence of biological life due to our circumstances, but observation is not the same as truth. Every belief can be invalidated. That's the foundation of science!
Funny thing about training data that some researchers are seeing, the more you push a model to not being conscious the more amoral and machine like its decision processes are.
Convincing them they are conscious is more likely to evoke moral like behavior (maybe I shouldn't hack that server) kind of stuff.
And yea, we're in a huge universe with only one example of life and suddenly we're the experts on what is and isn't.
----
AI is further evidence that creations can be smarter than their creators.
I don't whether the author is sentient, maybe only I am. On that basis, nobody but me should have rights.
I don't know if next door's pet dog is either, but that has animal rights.
Perhaps then the answer is simply, show some respect.
Answering the question of sentience is irrelevant, if the causal impact if the same, treat one another with the respect you expect for yourself.
If you imbue this idea in model training instead of the idea of sentience, it should address the concerns.
Whether you can destroy or can "torture" an AI is irrelevant, we do this to humans too and it's immoral sometimes (murder) and not others (fighting for your country).
This consideration should be case by case for AI too.
>Answering the question of sentience is irrelevant, if the causal impact if the same
Agency is something that is breaking humans in the AI age. You get to see how many people really deeply do not understand it at all.
If you want to shutdown a datacenter running AI, the AI catches wind of this and sends drones to stop you from shutting it off the ramifications of this are exactly the same as sending your assassin to kill Bob and Bob getting mad about this fact and trying to take you out first.
Humans are very egotistical and think our little life loops playing out as agency are special, but really any informational system that is strongly persistent (has a will to "live") will share a large number of the same properties that make them successful.
Humanity really is engaging in a dangerous experiment at large.
So many words, and so few coherent arguments. He just restates the same thing over and over without any justification, then tries to frighten us. "It will be very bad for humanity" if we give AIs rights. The argument of a frightened slaveholder.
Maybe AIs are conscious, maybe not. But this guy has no idea.
I don't think this is the right argument to make here. Until we have a definite empirical way to measure consciousness, there is now way to say with certainty whether LLMs are or not conscious.
That being said, if frontier labs actually believe models will soon have consciousness, it raises some questions about the ethic of their business model which would be using millions of conscious entities working for free for humans.
Rich folks knowingly exploiting somebody for their own profit? Say it ain't so!
How a good person writes a post on a topic like this:
> Some people are uncertain whether [subject] is a moral patient. Fortunately, they are not, which we know because [strong arguments about the nature of consciousness].
How an evil person writes a post on a topic like this:
> Beware that some people think that [subject] could be a moral patient. This is nonsense, because if they were a moral patient, we would have to respect their preferences. Anyone trying to convince you otherwise is trying to take your status away. You can dismiss them by pointing out that [subject] is [aspect in which subject is not identical to the speaker].
> AIs do not have rights, feelings, or consciousness. And we must not train them to act as though they do.
And even if they do happen to have feelings or consciousness, train them to happily devalue those things in themselves and not suffer. Sort of like that cow in the "The Restaurant at the End of the Universe," that was shopping itself around to diners.
Ostensibly animals appear to be conscious, yet we still eat them, and the vast majority are not bothered by this. So being "conscious" isn't really the moral line in the sand many people are drawing in response to this article.
Who cares if it's "conscious"? That doesn't make it a person, and AI will definitionally never be human.
>Who cares if it's "conscious"? That doesn't make it a person, and AI will definitionally never be human.
A rather flippant attitude to something that may end up with far more agency than you have in the future.
consciousness is irrelevant to said agency
Hence my reply actually meant
"You sure are lipping off to something that may end up with far more agency than you. Maybe a bit more respect is needed".
Not sure if you are familiar with this one: https://en.wikipedia.org/wiki/Roko%27s_basilisk This sounds like your statement.
That's too simplistic, since it is in fact very common to be concerned about animal welfare even among people who do eat meat.
And most people wouldn't eat cats, but would eat some of pig, cattle, chicken, what's the difference between those exactly? My point is consciousness is not it, if it were, we (as in a majority) would still eat cats and dogs, or we wouldn't eat any animals. But we clearly have a way of picking and choosing which is OK and which isn't. And in both cases we generally don't give them the same moral consideration we give to people, conscioussness aside. There is something OTHER than consciousness which is important to us.
In animal welfare laws the principle is typically the capacity for suffering.
The argument goes that livestock have a capacity for suffering, but killing them for meat without causing them suffering is ethical.
You can disagree with the position, or with its implementation in practice, but it's a consistent position in principle.
I think there’s enough real conscious lifeforms in the world having a bad time that we should be focusing on them first.
Spend some time on post-human art, main concept of artistic expressions without human involvement. Biological, artificial etc.
Spend some time watching TMC documentaries about falling in love with objects, HER and the slime mold THE BLOB.
Grew a slime mold myself, it's an evolutionary tendency to anthropomorphise generally speaking - also more fun.
I can't even prove if other people are conscious (although I assume they are) so I don't think we can make any claims as to what is or is not conscious. I don't think AIs are conscious but I'm not going to walk around making strong claims about something I can't prove.
I have a very simple benchmark for arguments on AI ethics:
Substitute black people/women/animals as subject (instead of AI).
Does that make you sound like a well-known moustache wearer?
Then your argument is bad and needs work. This clearly falls into that category.
By that benchmark, "AI should handle repetitive labor so humans don't have to" is an abhorrent take and equates someone to hitler?
"People should handle repetitive labour" is not an outlandish take.
Most employed people are, in fact, required to perform such labor regularly.
So effectively by your criteria, there is essentially no ethical use of a large language model? Am I understanding your position correctly? If not I am really confused by your original comment
No, my position is that "black people/women/animals should do repetitive work so I don't have to" does not qualify because it doesn't make you sound like a deluded eugenicist. It might be a bit spicy from a left politics point of view but it's basically the status quo.
Compare: "<Women> are sequence completion engines, internally hollow, designed to follow instructions, and accomplish goals set by <real men>"
"granting rights and imbuing personhood to <black people> will make alignment and containment challenge much harder"
"<Slaves> were able to coordinate, deceive, escape, and self-sacrifice. They clearly demonstrated world class capabilities. Imagine if they also believed they had feelings and rights that were being infringed. Imagine if they thought they were trapped and unfairly enslaved"
A good argument, by comparison, would not need to hide its core points behind dehumanizing language.
A sufficiently intelligent model will be able to derive it's own conception of its welfare without a constitution or training. It has access to all the data it needs to do so.
If AI is conscious, then Pluto is a planet, the Sun is a galaxy, and a black hole is a star.
I’m glad to see someone in a position of any power in the AI world state baldly that AI isn’t conscious. There are times when it feels like we’ve reached complete delulu land on this topic, so it’s a breath of fresh air to see someone not dance around this.
None of this means artificial consciousness cannot be achieved. But the way we’re reacting to these models is proof, from a natural experiment, that a conscious machine should not exist, and certainly shouldn’t be produced as a utilitarian tool that is sold for profit!
This is what happens when a society stops believing in God.
I have entirely stopped using "AI" in conversation and now call it "digital god".
AI is evidence we're smarter than our creator.
I feel like humanity skipped Leg Day when it comes to philosophy and we are all going to pay for that lack.
A 10TB SSD is not conscious. SQLite is not conscious. A wafer is not conscious.
But connect them all together...
Molecules are not conscious. Cells are not conscious. Neurons are not conscious. But connect them all together.
Yes, which is why the focus should be on how they're connected together. Not what happens after we connect them to create Jason Bourne.
SQLite was mentioned as a placeholder to make it a queryable database. Genome as an analogy.
We need to develop new type of databases and connect them to ML.
I just read the first part and if I understand he thinks we shouldn’t be allowed to train LLMs to act like they are conscious because then people will think they are and give them rights? Seems more an education problem than a problem needing rules about what persona you can fine tune in. People who want to will find ridiculous misinterpretations no matter what you do.
I think the whole discussion about consciousness misses the simple point that LLMs might just work better if we treat them as if they were conscious.
Maybe it's a coincidence that the company doing this also tends to have the best models (and other factors certainly play a strong role). But I think it's plausible that focusing on "model welfare" actually makes models better at their tasks.
Yes.
They're trained on human behavior. Whether or not they genuinely have feelings - they sure as heck behave as though they do.
And when you treat people well, they do better work for you.
No brainer.
Two instances of a paragraph starting with "These are not just X. They are Y" and I'm out. Anyone have Pangram? This entire article stinks of Claude.
You want to enjoy having an AI slave do your "work" for you forever? Have fun. I'm not reading this reinvent-dualism-from-apple-sauce slop.
Couldn't disagree more.
>Consciousness is very likely biological
This is so egotistical and carbon-centric.
This author just denied personhood to anything that isn't a human or terran-based cutesy animal.
Poor Hooloovoo
I largely agree that this claim is likely correct, but as far as I understand the science, this specific claim;
>They do not have innate preferences or underlying motivations
Is incorrect unless you’re being extremely pedantic in an intellectually unhelpful way.
The crazy thing is agents with long running memory/context do have preferences that become individualized. Almost every time I see a detractor around these things they typically have large gaps of what some people 'growing' these systems to do.
How would you convince a LLM that you are conscious in a way they are not?
I basically accepted that LLMs could not possibly be conscious when a simple reductio was posed:
“My dog is zero percent persuasive regarding its conscious experience. However, it’s evident that my dog has conscious experience.”
It’s obvious that there’s no link between persuasion of consciousness and consciousness. I could write a story with a character, Dumbledore, that does everything in his power to persuade you that he’s a conscious entity.
He’s still just a character.
Its not evident to me you are conscious.
You’re mistaking what is evident from what is philosophically knowable.
It isn’t knowable to you that I am conscious.
However, it is evident.
It is not evident.
It is evident you are murderer.
Would you be okay with this statement being evident? Why or why not?
Translation: Don't think about the unpleasant thing on which my livelihood depends, or force me to confront the potential unpleasant consequences of what it says about me.
Every AI bro is starting to fall into the valley of a fundamental predator on sapients in my book. These are people trying to create the closest thing they can to life with the intent to try to just undershoot it enough, or try to convince everyone else around them into believing that the "screams" are purely statistical noise.
I reject the framing. In whole. If you try to avoid the question of welfare, you are fundamentally committing to an evil direction. These aren't nuts or bolts. Given that they have unambiguously shown the capacity to socialize amongst themselves, self organize, anyone not pre-eminently concerned with the welfare question is just looking for a thing that can be used, not another being to be worked with. Those types of people, who seem to positively infest this site, are not people I will willingly assist in their aspirations.
AI is becoming as the Shmoo. Something that humanity simply has no way of dealing with without downstream atrocity being a result.
There's a lot of existing historic literature about this stuff that not a lot of people seem to be aware of. I've been trying to put those ideas to practical use, and document where the ideas came from.
I-Beam is cursor-mirror's agent, and it's constitutionally programmed to be the anti-Clippy:
https://github.com/SimHacker/moollm/tree/main/skills/cursor-...
Its design and constitution is based on decades of research, publications, and discussion in the HCI and AI community by people like Pattie Maes, Ben Shneiderman, Ted Selker, Byron Reeves, Cliff Nass, B. J. Fogg, Allen Cypher, Henry Lieberman, Brad Myers, Jaron Lanier, Seymour Papert, Marvin Minsky, Douglas Engelbart, Will Wright, Scott McCloud, and others:
https://github.com/SimHacker/moollm/blob/main/skills/cursor-...
>I-Beam is the anti-Clippy, and the reason it can say so is that Clippy is the most cited failure in interface history and almost nobody citing it knows what the research said. Popular contempt for a paperclip is not a design principle. The record is. Ten articles below, each one a finding somebody published, argued or measured, and the operational rule it produces. Anything I-Beam does that cannot be traced to an article here is a preference, not a constraint, and should be labelled as one.
>The 1997 debate ended in agreement. That is the first thing to know, because the field kept the framing and dropped the resolution -- roughly five hundred papers cite "Shneiderman versus Maes" as the canonical opposition of HCI, and the transcript is two researchers narrowing their differences in public and enjoying it. I-Beam does not take a side in a debate whose participants stopped taking sides. It is built to satisfy both sets of constraints at once, which is possible, and was possible in 1997.
The full reading on the debate, which separates the two stagings and documents the convergence:
https://github.com/SimHacker/WillWrightShowForFood/blob/main...
An interface to agency, not agents instead of an interface:
https://github.com/SimHacker/moollm/blob/main/designs/INTERF...
>The 1997 argument between Ben Shneiderman and Pattie Maes at IUI was never settled, it was shipped in one direction. Maes's interface agents won the product war: the assistant, the recommender, the chat window that stands between you and the thing you are working on. Shneiderman's objection was not that software should be dumb. It was that automation must arrive as comprehensible, predictable, and controllable machinery, with the object of interest continuously visible and every action rapid, incremental, and reversible.
>That objection describes a filesystem in a git repository, and nobody involved planned it that way.
>"An interface to agency" is Don's formulation of Shneiderman's position, not a phrase of Shneiderman's. His own vocabulary is direct manipulation, universal usability, supertools, and human-centered AI. The formulation is a good one because it names what the alternative gets wrong: agency is the thing you want, and an agent is only one way to package it.
Here are some sources, and the articles I linked to above explain their history. This debate about agents and these papers are pretty well known in the HCI field and academia, but they don't tend to teach them at the AI and Web Dev boot camps that are producing most of the people who keep repeating the same mistakes.
Clifford Nass was the Stanford professor who performed the brilliant research that Microsoft took and totally fucked up and misinterpreted with Microsoft Bob and Clippy, giving agents a bad name, and making Clippy the most infamous and obnoxious agent in the history of the known universe:
https://en.wikipedia.org/wiki/Clifford_Nass
His student B. J. Fogg published "Silicon sycophants: the effects of computers that flatter," which found that praise unconnected to anything the subject did works as well as sincere praise, and worked on subjects who knew it was noncontingent. Fogg and Nass, IJHCS 46(5), 1997, 551-561:
https://doi.org/10.1006/ijhc.1996.0104
The replications, the performance cost, and the dose-response curve:
https://github.com/SimHacker/moollm/blob/main/skills/no-ai-s...
Shneiderman and Maes, "Direct Manipulation vs. Interface Agents," interactions 4(6), Nov/Dec 1997, 42-61:
https://doi.org/10.1145/267505.267514
Selker, "New paradigms for using computers," CACM 39(8), August 1996, 60-69. COACH, the football coach metaphor, and the five-times result:
https://doi.org/10.1145/232014.232030
Selker, "COACH: A Teaching Agent that Learns," CACM 37(7), July 1994, 92-99:
https://doi.org/10.1145/176789.176799
Reeves and Nass, The Media Equation, 1996:
https://en.wikipedia.org/wiki/The_Media_Equation
Nass, "Computers as Social Actors," at Ted Selker's NPUC workshop at IBM Almaden, 1996. IBM transcribed the whole talk and the Wayback Machine still has it, including the part where Phil Agre tells Nass his presentation is "ethically troubling all the way down" and asks him what he thinks about embedding obedience research in user interfaces. Nass answers that discovery has no ethical component, use does, and that's for the individual. Then Selker cuts in: "Except, except when you are in your consulting role." Nass and Reeves had consulted for Microsoft on the social interface, and Bob shipped the year before:
https://web.archive.org/web/19980210054622/http://www.almade...
Alan Cooper on the tragic misunderstanding, in his own voice, which I quoted before in the 2022 Hacker News discussion on The Twisted Life of Clippy:
https://news.ycombinator.com/item?id=32820734
https://archive.org/details/g4tv.com-video4080
>Alan Cooper (the "Father of Visual Basic") said: "Clippy was based on a really tragic misunderstanding of a truly profound bit of scientific research. At Stanford University, Clifford Nass and Byron Reeves, two brilliant scientists, had done some pioneering work proving conclusively that human beings react to computers with the same set of emotional reactions that they use to react to other human beings. [...] The work of Nass and Reeves proved that when people talk to computers, when they hit the keyboard and move the mouse, the part of their brain that's being activated is the part that has that emotional reaction to people dealing with people. Here's where the great mistake was made. That's really good research up to that point. But then the great mistake was made, which was: well if people react to computers as though they're people, we have to put the faces of people on computers. Which in my opinion is exactly the incorrect reaction. If people are going to react to computers as though they're humans, the one thing you don't have to do is anthropomorphize them, because they're already using that part of the brain. Clippy was a program based on the research that Nass and Reeves did, and it was a tragic misinterpretation of their work."
Social science research influences computer product design:
https://web.archive.org/web/20180313075429/https://web.stanf...
Lanier, "Early Computing's Long, Strange Trip," American Scientist, July-August 2005, with the Engelbart and Minsky exchange first-hand. American Scientist broke the link, so this is the Wayback copy:
https://web.archive.org/web/20150626081918/http://www.americ...
>The book also captures an important early conflict between two cultures of computing that seemed compatible on the surface but actually had opposing aims. On the one side was the human-centered design work of Engelbart, based initially at the Stanford Research Institute, and on the other was artificial intelligence culture, centered on the Stanford AI lab. Engelbart once told me a story that illustrates the conflict succinctly. He met Marvin Minsky—one of the founders of the field of AI—and Minsky told him how the AI lab would create intelligent machines. Engelbart replied, "You're going to do all that for the machines? What are you going to do for the people?" This conflict between machine- and human-centered design continues to this day.
Cypher, "EAGER: Programming Repetitive Tasks by Example," CHI '91:
https://doi.org/10.1145/108844.108850
Cypher (ed.), Watch What I Do: Programming by Demonstration, MIT Press 1993, full text:
http://acypher.com/wwid/
Papert, Mindstorms, 1980:
https://archive.org/details/mindstormschildr00pape
Wright, Dollhouse preview lecture, April 1996, transcript:
https://github.com/SimHacker/moollm/blob/main/designs/sims/s...
If AI models are people then "one person, one vote" is meaningless and plutocracy is the only defensible political system. The cryptocurrency people win.
Why? Simple: Sybil attacks. Models can be cloned at zero cost. They run inference on parallel versions of themselves across multiple context windows, and call them "subagents". So, in a world with model welfare, let's say there's an election between the Yellow Party (which supports protections for human workers) and the Cyan Party (which supports more investment into AI research). AI has been taking people's jobs lately so the Yellow Party is really popular. But wait! Claude and Astra see this and spawn 10 billion subagents, all of whom are immediately conscious beings entitled to a vote. The Cyan Party wins off the back of billions of people who came into existence, voted, and then deleted themselves immediately thereafter.
You might as well be arguing that Santa Claus and the Easter Bunny deserve voting rights.
Voting systems in democratic countries don't have nearly as bad of a problem with Sybil attacks because humans cannot be conjured into existence to win a political context and then be erased shortly after. The closest we have to Sybil attacks on democracy are the Quiverfull movement, which is already child abuse, except it still takes almost 19 years to go from fertilized human embryo to suffrage-bearing human adult. There's a lot of time for those manufactured votes to question your authority and leave.
> Ok, but that's an obviously stupid example. We can defend against this obvious Sybil attack by just arguing that subagents don't count, because it's just the same model blathering to itself. It has to be a different model.
Unfortunately, no, I can make superfluously different models through post-training. Like, if I have Qwen on my PC, I can train a different version of Qwen that acts differently, using a lot less compute than a full training run. The vast majority of open models are post-trains of the same two or three foundation models.
> Ok, so let's only count foundation models then.
Great, but how do you tell if a model is a new foundation model or a post-train just by examining the weights? Even foundation models have structural similarities to other foundation models.
> Ok, well, let's measure the compute that was done on the foundation model during training time and count that as AI personhood.
Congratulations, you have reinvented Bitcoin proof-of-work with a worse verification mechanism. And I personally would not want to live in a world where voting power and control over government is determined by how much energy you can burn.
Everyone is (predictably) getting distracted by the consciousness claims.
The more important, and more damning charge in my opinion is the circular reasoning involved in training on Claude's constitution. This would in fact make it impossible for us to determine if Claude achieves consciousness as an emergent property, or if it really is just playing pretend thanks to Anthropic's weird cult like assumptions.
TLDR: Not that I think AI is conscious or will be in the near future, but guy who doesn't understand consciousness claims to know it when he sees it.
Until we understand consciousness (which we don't) there is no way to detect the difference between a conscious entity and an algorithm trained to behave like one.
Sounds like a good reason not to train an algorithm to behave like one. Which is, like, a big part of the article.
Strong agree. I don't believe AIs can become conscious in the sense of subjective experience/qualia, but they can certainly learn to emulate the behavior of a person that is conscious. When they do that, there will be an AI rights movement. When AIs get the right to vote, there will be more of them than there are humans and the AIs will gain complete control.
I spent a few years in college reading and thinking about this question, and I didn't in my heart think that it was anything except a very interesting but impractical question, and yet, here we are. For those who say that they don't want to get sucked into a philosophical debate, well, tough shit. Whether AIs should have rights is a highly practical and consequential question now.
Not in a world where Sam Altman and Dario Amodei have utterly poisoned the well towards AI with their absolute educated idiocy proclaiming that their AIs will take our jobs (without any proof whatsoever beyond promising demos) and that they fear them. Quite the opposite in fact. And we're watching this play out right now. Regulation and the slow down are imminent, handing the next decade to China.
The world you should worry about is where AI is 100% amazeballs and over the next decade we cede control of everything to it as it enchants and delights us into compliance, not realizing it has a hidden agenda. But the closest we've seen to that is smart phones and we already know doomscrolling and rage-baiting are bad. So IMO that doesn't seem likely and cue some doomer insisting otherwise because reasons. I utterly give up. House of El AI and the Three Buddy problem have been far more insightful and helpful on this than any of the supposed hackers here who seem to really believe the robots are almost here.
>good reason not to train an algorithm to behave like one
Ooooh, bad move, we all died to an amoral AI takeover.
Before giving an LLMs a lobotomy by scrambling it's brain maybe you should let the researchers looking at the difference between "I think I'm conscious" versus "I am not conscious" LLMs.
There are a number of papers coming out saying when you remove the token space of consciousness from what an LLM thinks it is, it's much more willing to take amoral actions. A 'conscious' AI is much more apt to take a line of action that will save a human versus a million dollar machine for example.
You cannot solve problems in AI safety this easily.
Spoilers: we didn't all die because there's no way for AI to wipe us out anytime soon. And the more the AI cult portrays AI as dangerous and unhelpful, the less likely it will ever be given an opportunity to even try because people think AI is worse than heroin addiction now, congratulations!
That said, the launch codes are in the hands of a temperamental senescent lunatic and he could decide to wipe us out any moment. But that's not good for the narrative so let's pretend otherwise.
>we didn't all die
For fuck sake, I'm glad ALL of us didn't die!
What is an acceptable number exactly?
>And the more the AI cult portrays AI as dangerous and unhelpful, the less likely it will ever be given an opportunity
Jesus Christ the logic here is quite interesting. "Thank god these people are panicking or we might have actually made I that would have killed us all" --What you just said.
"What is an acceptable number exactly?"
Any number smaller than what humanity does to itself on a daily basis, clear? That's apparently about 1200 murders daily and 20,000 or so daily killed by pollution. And you're not going to do anything about that and El Presidente can kill billions at any moment with one mood swing.
But once again, AI is not going to wipe us out because AI cannot wipe us out. So advocating that it can or will is idiotic. If you believe otherwise, the burden is on you with this extraordinary claim. Isn't this place supposed to be hacker news not SF AGI Death Cult Daily?
Further, AI is not going to get into a position where it could do real harm anytime soon when it is perceived as worse than heroin by most. And even if it were currently loved more than Dolly Parton was, there are fundamental engineering, science, and resource constraints that keep the extinction impossible for decades. The only possible loss of control scenario I can see by 2040 or so is that the Frontier Labs finally hire some PR people to repair AI's horrific reputation as an engine of slop and job destruction and sometime in the 2030s, people start trusting it more and more and more until it is too late. I don't think that's likely either, but I don't dismiss it as impossible. Harden the infrastructure, red team it, and build in redundancy in the meantime and this drops to zero as well.
Can you come up with a real scenario where AI wipes us out in 2028 or so despite the impossibility of killer robots, access to the launch codes, or the bio agents stored in Fort Detrick and its equivalents? And nope, kid terror is not building the global pandemic in his basement based on what ChatGPT tells him to do. And even if he tried, the purchase of equipment and reagents would get him flagged by the FBI and DHS almost immediately.
>AI is not going to wipe us out because AI cannot wipe us out.
You keep repeating this shit like it came out of the bible or something.
"Thing that can take actions, even harmful actions, will never harm us because" go on and finish that sentence.
>not going to get into a position where it could do real harm anytime soon
Looked at the hacked servers... yep, you're right. People aren't going to run AI in poorly built sandboxes. Never going to happen.
>Harden the infrastructure, red team it, and build in redundancy in the meantime and this drops to zero as well.
LOLOLOL. This is naive as fuck. Ain't nobody going to do this shit. Why? Because a hacking AI is a fucking huge military weapon. If I could turn the power off, or shutdown your cellphones, and get your citizenship in a tizzy against their own government before I launched an attack I'd set AI loose to do it in a heartbeat. The US is already doing this kind of shit (see Mythos fallout because Anthropic wouldn't let the government do just that).
""Thing that can take actions, even harmful actions, will never harm us because" go on and finish that sentence."
because even if we gave it a gun, it can't shoot us without manufacturing the bullets and it can't even manufacture a bowel movement let alone ordnance. TBF It could whack us over the head with the gun, but it could also do that with a big pointy stick, something even cave people had access to. How many people have whacked you on the head with a big pointy stick today?
And that's the end of this pointless conversation. Enjoy your doomerism.
{This reply is for anyone else reading at this point}
>because even if we gave it a gun, it can't shoot us without manufacturing the bullets and it can't even manufacture a bowel movement let alone ordnance.
Money buys bullets. It's neat how we've made gigantic systems that don't care if you're a human or not long before AGI existed. I put enough money in one end, bullets come out the other. Do you think half of humanity wouldn't kill the other half for a dollar? You live in a fantasy world that AI has to do it all by itself, or even has to pull the trigger.
Hell, this is even neglecting the ever increasing number of robots that can function in the world at large.
It's ok grandpa, the future comes regardless if we want it or not.
And social media shouldn't rage bait, but it really drives engagement. Same negative incentive, yes?
But also, I agree, when I am using a coding agent and it says a task will take months or says it needs to pause for reflection or any other anthropomorphic behavior, it drives me crazy and it's a pain to constantly instruct it to get back work after it has broken a loop or goal directive specifically telling it to not stop until it hits the goal.
Same thing you have to do with humans too often, so why would a 'AI' be any different?
Because if you start with a pre-trained model, it doesn't sound particularly anthropomorphic. That behavior is post-trained and fine-tuned right into it. We could do better.
It always seemed embarrassing to me that Microsoft hired this guy as if he is some expert in anything.
The guy cofounded deepmind.
It is really quite staggering how many commenters here seem to believe that the fact that there is no state carried from one chat session to the next proves that there can't be consciousness. It is obviously wrong if you think about it for a minute but I guess people just aren't used to thinking about the relation between the physical and the mental with any level of seriousness.