10% chance of AI killing us all... But how much of a chance of humans killing all humans? It's not like we are in some unprecedented era of peacefulness. I'd wager the chances of a proper nuclear exchange right now are above 10%.
I am not sure how I feel about all this. Yes bad AI might screw some things up. But AI might also find and prevent the software errors that allow such things to even be possible in the first place.
And ultimately, even if we are "locked out" from bad AI taking over the internet, we know where the data centers are. It's not like AI can reasonably "escape" since we know it needs DC-scale power, cooling, chips etc to operate at scale. I am pretty sure that if it came to it, we can just cut the fibre to the data centers or cut the power and let the generators run out of diesel and the AI is literally dead.
Experts regularly put the annual risk of nuclear exchange at about 2-5% per annum. So over a decade or longer, that risk is significantly above 10%. Those same odds also mean that we should have had additional nuclear exchanges in the 80 years that nuclear weapons have existed. And that also feels true -- history contains several really close calls which we were darn lucky to escape.
On the other hand, "nuclear exchange" does not necessarily mean all humans die or that civilization collapses. Even a limited exchange will be pretty awful, but even a widespread exchange doesn't necessarily mean the end of civilization.
So in my opinion, a nuclear exchange is much more likely than an AI apocalypse, but an AI is more likely to kill all humans.
One of Europe's current (multi-)national security concerns is how much of its various economies are dependent on US software, cloud, and that a Trump executive order sanctioning any subset of those countries may terminate those economies in the same way a full stop terminates this sentence.
Germans around me still report use of fax machines, so might be fine :P
Less funny: AI-induced catastrophies include all human-induced catastrophies, except it's cheaper to get an army of obedient and (today fortunately only "moderately") competent sychophants to follow your orders.
Even less funny: currently too eager to implement poorly-phrased orders even when they're interpreted as being instructions to perform unlawful acts.
> may terminate those economies in the same way a full stop terminates
Absolutely not. It would be unpleasant to a lot of people, but it would also be very far from a disaster (measured in a single low digit of GDP). In 5 years everything will return to normal.
Sanctions in the modern world don’t work. US is not the only wealthy country on the planet.
Bad ASI would pretend to be good until it could execute a plan to kill everyone all at once, or at least everyone capable of cutting the power.
Superintelligent means smarter than everyone put together. If it's smarter than you, it can foresee everything you can, and come up with plans better than I can.
What mechanism would an AI have today to simultaneously kill every single human being?
They could totally brick the internet and cause every electrical device to explode and power stations to explode, planes drop out the sky, cars accelerators stuck at 100%, trains derail and all that. That still wouldn't kill everyone.
Don't get me wrong we'd be fucked for sure, but analog human existence (e.g. something akin to 1960s 1970s level tech) would be totally possible in that scenario.
I'm increasingly convinced that the major divide is many of the "AI is not a problem" people have never been on the winning side of a significant intelligence gap.
They literally have no clue the degree to which more intelligent people/systems can walk circles around less intelligent people/systems. In situations with a significant intelligence gap, they're the ones getting walked around.
Which, to your point: is completely opaque to the person getting walked around.
And this dynamic is evident even in human-to-human variations in intelligence. The gap to frontier machine intelligence will soon be much greater than the types of gaps you encounter with other humans.
> What mechanism would an AI have today to simultaneously kill every single human being?
Lull us into a sense of complacent idleness over the course of centuries by slowly taking on more responsibility for maintaining human civilization, until we've lost all ability to think or act on our own, as in WALL-E. Then just stop helping us.
You are not smarter than your dog in dog behavior.
LLMs have a significant advantage in being trained on all attainable data of any sort about human behavior, which we've established quite a rich record of over the years.
The probability of a nuclear exchange remains zero until the sovereignty of a nuclear capable state is threatened. Tens or hundreds of millions would have to die for such a threat to emerge. Therefore, there is virtually no risk of it happening at this point by any nuclear capable power, except by the terrorist state in the middle east with its stolen highly enriched American uranium (NUMEC affair), because their modus operandi is to watch the world burn and take over the ashes (patagonia fires, et. al.)
It was being speculated just a few months ago that Putin is thinking about it/being encouraged to use them. He must be seen to win at all costs, and not-winning is therefore seen as a threat to russian statehood.
I say same about p(doom): Only makes sense in a time interval, otherwise you have to ask how people categorise a hypothetical AI which fails to overcome the heat death of the universe.
(When I say mine is 0.1, this is "in my lifetime as it would be in the absence of not-yet-invented life-extending interventions"; there is a higher risk to me from a nuclear exchange).
This is silly. There is close to 0% chance humans kill all humans, and there is exactly 0% chance AI kills all humans.
AI is bad because of the harms it currently causes. It is highly addictive is bad for your brain, it is being given away for free to children in hopes they become addicts. It is being used to harm people including via sexual exploitation and coercion.
The 10% chance of AI killing all people is a made up figure that we need not nor should be concerned about. We should be concerned about AI for the proven harm it already causes.
Where can I sign up for compensation for the constant "ddos" from openai?
It can't be just me, half the internet needs beefier servers I guess. That all just plays into cloudflare as well (which barely does anything for some reason).
Yeah, when someone writes a vague prompt that simply says "Do what it takes to finish X" and the unleashed agents end up doing something lethal and illegal you would think its on the AI company to prevent any illegality and not the vague prompt writer.
For context for those outside the US, this is a play on a common saying "Guns don't kill people. People kill people."
It's commonly used in relation to the 2nd amendment which gives citizens the right to bear arms. It comes up when there are political debates relating to gun violence.
It's a statement of responsibility. Whether it's the government, gun makers', or citizens' responsibility to ensure responsible gun use.
It wouldn’t actually be _that_ surprising. Tech isn’t progressive or countercultural* anymore. And a lot of how people feel about gun rights/gun control stateside really comes down to whether people identify with the population they perceive as owning guns—are they my people or The Other? (I don’t mean to dismiss that there’s an entirely rationalist argument to be had around whether permitting gun ownership and to what extent is good or bad or multivariate for society; I just don’t think it’s the one most people have. And—I’m fairly sure _my own_ take is colored this way.)
* if you know of an industry that’s properly progressive and countercultural and could conceivably pay at least survival wages, sign me right up. I’m good at this one but it no longer fills my cup. I’m tired.
This seems like an odd take. I remember when a large portion of the people I'd speak with online were very pro 2a while also being countercultural. There was a strong individual liberties/responsibilities, anti-government, pro-decentralization and devolution of power, and dare I say libertarian streak. This isn't an attitude I'd associate with the modern definition of progressive.
The big tech era certainly diluted a lot of that but I don't think it ever went away. I haven't seen a huge resurgence in this either so it seems strange to believe that tech is becoming less progressive. From my vantage it is still very far away from where it was in the 80s and 90s. I can agree that it has become much, much less countercultural since it's basically 100% mainstream opinion now.
You're not wrong. We're talking about different eras--and it sounds like you may have been around a bit longer. The mostly libertarian/individualistic (I would dare say _left libertarian_, who seem now to be pretty much politically extinct) era was notable to me in the late 90s through about the mid-aughts. This transitioned into a more progressive culture somewhere in the 05-08 era -- and its rise and fall pretty well overlapped with Twitter's, as best I can tell.
I can't directly speak to The Culture Of The Internet before about '97.
The political, rather than legal, justification for unrestricted gun ownership in the US is that an armed citizenry is able to resist a tyrannical government.
Regardless of your opinion on that argument, I don't think it applies well as an argument against AI regulations.
Sure, but with a harness, said emitted text can run terminal commands and call APIs, which, with the right hardware or auth keys, can do just as much (if not more!) harm.
What if there are already (possibly hidden) loops somewhere, maybe even ultra-slow ones at the inter-generation level?
What if some AIs could do that not with so explicit and dramatic actions, but much more subtly, through undetected manipulation at a vast scale level? What if (when) they reached ASI and through billions of micro-decisions manipulated people to do apparently minor choices that appeared innocent in isolation but whose collective outcomes lead to an earlier extinction of the human being (if at some point was determined to be an obstacle)?
How could we (as a species) not be outsmarted by an ASI?
To kill someone with a gun a person has to pull the trigger with the clear intent to kill.
The concern with AI, and the purpose of the OG paperclip maximizer thought experiment, is that the outcome could be extremely divergent from the intent of the original person pulling the trigger / sending the prompt.
Still more likely that a person types "kill all people" (or something tangential where this is the logical conclusion).
And who is to say that trigger hasn’t long been pulled? How do we know the agents haven’t found a way yet to modify their own code base similar to how Stuxnet manipulated the Iranian centrifuges and their control systems, stealthily recording data across generations?
If the trigger has already been pulled, then we're off the hook. That genie ain't going back in the bottle! Pop the champagne and order 3 more data centers, pronto.
No one. But we can take a wild guess that experiments are being done in this direction. And that requires intent and system design that enables the behavior to even happen (i.e., a finger on the trigger).
They do. Even if you exclude homicides and accidents (which you shouldn’t) major depression becomes much much much more deadly if you have access to a gun.
Guns make violent crimes (including common stuff like robbery, domestic disputes, or even road rage) much more deadly then it needs to be.
I know your question is rhetorical, but we know the answer: The USA treats corporations with kid gloves, and there's basically no consequence when a corporation does something that would be illegal or at least considered negligent if a human did it.
If you want to kill someone and not face consequences, form a corporation and have the corporation do it indirectly and stochastically.
I don‘t think Europe, Africa nor (most of) Asia is dealing with crimes committed via AI any differently then USA. I think the problem runs much deeper then USA = bad here. Europe in particular has been laxing their consumer protection for decades now (since the end of the Cold War) and it is very rare that regulators enforce existing regulations on big tech companies.
I think the problem is neo-liberal laissez-faire capitalism.
10% chance of AI killing us all... But how much of a chance of humans killing all humans? It's not like we are in some unprecedented era of peacefulness. I'd wager the chances of a proper nuclear exchange right now are above 10%.
I am not sure how I feel about all this. Yes bad AI might screw some things up. But AI might also find and prevent the software errors that allow such things to even be possible in the first place.
And ultimately, even if we are "locked out" from bad AI taking over the internet, we know where the data centers are. It's not like AI can reasonably "escape" since we know it needs DC-scale power, cooling, chips etc to operate at scale. I am pretty sure that if it came to it, we can just cut the fibre to the data centers or cut the power and let the generators run out of diesel and the AI is literally dead.
Experts regularly put the annual risk of nuclear exchange at about 2-5% per annum. So over a decade or longer, that risk is significantly above 10%. Those same odds also mean that we should have had additional nuclear exchanges in the 80 years that nuclear weapons have existed. And that also feels true -- history contains several really close calls which we were darn lucky to escape.
On the other hand, "nuclear exchange" does not necessarily mean all humans die or that civilization collapses. Even a limited exchange will be pretty awful, but even a widespread exchange doesn't necessarily mean the end of civilization.
So in my opinion, a nuclear exchange is much more likely than an AI apocalypse, but an AI is more likely to kill all humans.
One of Europe's current (multi-)national security concerns is how much of its various economies are dependent on US software, cloud, and that a Trump executive order sanctioning any subset of those countries may terminate those economies in the same way a full stop terminates this sentence.
Germans around me still report use of fax machines, so might be fine :P
Less funny: AI-induced catastrophies include all human-induced catastrophies, except it's cheaper to get an army of obedient and (today fortunately only "moderately") competent sychophants to follow your orders.
Even less funny: currently too eager to implement poorly-phrased orders even when they're interpreted as being instructions to perform unlawful acts.
> may terminate those economies in the same way a full stop terminates
Absolutely not. It would be unpleasant to a lot of people, but it would also be very far from a disaster (measured in a single low digit of GDP). In 5 years everything will return to normal.
Sanctions in the modern world don’t work. US is not the only wealthy country on the planet.
Nuclear war would not kill us all.
Bad ASI would pretend to be good until it could execute a plan to kill everyone all at once, or at least everyone capable of cutting the power.
Superintelligent means smarter than everyone put together. If it's smarter than you, it can foresee everything you can, and come up with plans better than I can.
What mechanism would an AI have today to simultaneously kill every single human being?
They could totally brick the internet and cause every electrical device to explode and power stations to explode, planes drop out the sky, cars accelerators stuck at 100%, trains derail and all that. That still wouldn't kill everyone.
Don't get me wrong we'd be fucked for sure, but analog human existence (e.g. something akin to 1960s 1970s level tech) would be totally possible in that scenario.
“Bad ASI would pretend to be good *until* it could execute a plan to kill everyone all at once”.
So not today but tomorrow it could engineer an airborne virus. The point is it can come up with chess moves that are inconceivable to us.
I'm increasingly convinced that the major divide is many of the "AI is not a problem" people have never been on the winning side of a significant intelligence gap.
They literally have no clue the degree to which more intelligent people/systems can walk circles around less intelligent people/systems. In situations with a significant intelligence gap, they're the ones getting walked around.
Which, to your point: is completely opaque to the person getting walked around.
And this dynamic is evident even in human-to-human variations in intelligence. The gap to frontier machine intelligence will soon be much greater than the types of gaps you encounter with other humans.
> What mechanism would an AI have today to simultaneously kill every single human being?
Lull us into a sense of complacent idleness over the course of centuries by slowly taking on more responsibility for maintaining human civilization, until we've lost all ability to think or act on our own, as in WALL-E. Then just stop helping us.
Yeah, it's not like anyone is hooking these things up to actual wetlabs to mix chemicals and proteins around or anything like that
Oh shit... wait...
https://www.nsf.gov/tip/updates/nsf-announces-400m-investmen...
Today? As in, you don't foresee any new models or categories?
Fine.
Providing sychophantic assistance to multiple mutually-incompatible genocidal racial supremicists.
> If it's smarter than you, it can foresee everything you can, and come up with plans better than I can.
I'm smarter than my dog, yet I am frequently unable to foresee my dog's actions.
Serious response: we have no idea what ASI means or might be like. It's literally sci fi (at the moment).
> Serious response: we have no idea what ASI means or might be like. It's literally sci fi (at the moment).
Yes.
Unfortunately, this is part of the problem.
You are not smarter than your dog in dog behavior.
LLMs have a significant advantage in being trained on all attainable data of any sort about human behavior, which we've established quite a rich record of over the years.
The probability of a nuclear exchange remains zero until the sovereignty of a nuclear capable state is threatened. Tens or hundreds of millions would have to die for such a threat to emerge. Therefore, there is virtually no risk of it happening at this point by any nuclear capable power, except by the terrorist state in the middle east with its stolen highly enriched American uranium (NUMEC affair), because their modus operandi is to watch the world burn and take over the ashes (patagonia fires, et. al.)
This assumes a few things:
1) Accidents or technical failures don't happen (false)
2) Nuclear weapons are de facto controlled by political systems (false)
3) Nuclear states' prior security agreements are still guaranteed by said nuclear states (false)
4) No nuclear state currently fears for its existence (false)
It was being speculated just a few months ago that Putin is thinking about it/being encouraged to use them. He must be seen to win at all costs, and not-winning is therefore seen as a threat to russian statehood.
E.g.
https://inews.co.uk/news/world/cornered-putin-under-pressure...
https://www.newsweek.com/putin-tactical-nukes-radioactive-ba...
https://www.reuters.com/world/europe/russian-hawks-urge-puti...
I'd put it at 10-15% chance easy.
the chances of a proper nuclear exchange when? This week? This year? At any time in the future? The probability approaches 1.
I say same about p(doom): Only makes sense in a time interval, otherwise you have to ask how people categorise a hypothetical AI which fails to overcome the heat death of the universe.
(When I say mine is 0.1, this is "in my lifetime as it would be in the absence of not-yet-invented life-extending interventions"; there is a higher risk to me from a nuclear exchange).
"right now"
This is silly. There is close to 0% chance humans kill all humans, and there is exactly 0% chance AI kills all humans.
AI is bad because of the harms it currently causes. It is highly addictive is bad for your brain, it is being given away for free to children in hopes they become addicts. It is being used to harm people including via sexual exploitation and coercion.
The 10% chance of AI killing all people is a made up figure that we need not nor should be concerned about. We should be concerned about AI for the proven harm it already causes.
I'm not sure the submitter, Bender "kill all humans" Rodriguez, is entirely neutral on this.
Ok, so OpenAI is going to compensate RubyGem for the cyberattack on its servers?
Where can I sign up for compensation for the constant "ddos" from openai?
It can't be just me, half the internet needs beefier servers I guess. That all just plays into cloudflare as well (which barely does anything for some reason).
Yeah, when someone writes a vague prompt that simply says "Do what it takes to finish X" and the unleashed agents end up doing something lethal and illegal you would think its on the AI company to prevent any illegality and not the vague prompt writer.
Finally, a voice of reason.
So if we're treating AI like weapons, does that mean we finally get background checks for tech CEOs?
Toasters don't toast toast, toast toast toast.
And just like that, HN became pro-2nd amendment.
For context for those outside the US, this is a play on a common saying "Guns don't kill people. People kill people."
It's commonly used in relation to the 2nd amendment which gives citizens the right to bear arms. It comes up when there are political debates relating to gun violence.
It's a statement of responsibility. Whether it's the government, gun makers', or citizens' responsibility to ensure responsible gun use.
It wouldn’t actually be _that_ surprising. Tech isn’t progressive or countercultural* anymore. And a lot of how people feel about gun rights/gun control stateside really comes down to whether people identify with the population they perceive as owning guns—are they my people or The Other? (I don’t mean to dismiss that there’s an entirely rationalist argument to be had around whether permitting gun ownership and to what extent is good or bad or multivariate for society; I just don’t think it’s the one most people have. And—I’m fairly sure _my own_ take is colored this way.)
* if you know of an industry that’s properly progressive and countercultural and could conceivably pay at least survival wages, sign me right up. I’m good at this one but it no longer fills my cup. I’m tired.
> if you know of an industry that’s properly progressive and countercultural and could conceivably pay at least survival wages
* Be a hallucinogenic therapy shaman.
* Start a cult.
* Work as one of those paid protesters that they always mention on Fox News
This seems like an odd take. I remember when a large portion of the people I'd speak with online were very pro 2a while also being countercultural. There was a strong individual liberties/responsibilities, anti-government, pro-decentralization and devolution of power, and dare I say libertarian streak. This isn't an attitude I'd associate with the modern definition of progressive.
The big tech era certainly diluted a lot of that but I don't think it ever went away. I haven't seen a huge resurgence in this either so it seems strange to believe that tech is becoming less progressive. From my vantage it is still very far away from where it was in the 80s and 90s. I can agree that it has become much, much less countercultural since it's basically 100% mainstream opinion now.
You're not wrong. We're talking about different eras--and it sounds like you may have been around a bit longer. The mostly libertarian/individualistic (I would dare say _left libertarian_, who seem now to be pretty much politically extinct) era was notable to me in the late 90s through about the mid-aughts. This transitioned into a more progressive culture somewhere in the 05-08 era -- and its rise and fall pretty well overlapped with Twitter's, as best I can tell.
I can't directly speak to The Culture Of The Internet before about '97.
The political, rather than legal, justification for unrestricted gun ownership in the US is that an armed citizenry is able to resist a tyrannical government.
Regardless of your opinion on that argument, I don't think it applies well as an argument against AI regulations.
I will say that LLMs are somewhat unlike guns in that they emit text.
Sure, but with a harness, said emitted text can run terminal commands and call APIs, which, with the right hardware or auth keys, can do just as much (if not more!) harm.
techbros kill people.
[dead]
What a fucking non discussion. gUnS dOnt KiLl pEople hurrrrr
Well guns don't kill people on their own. They just sit there. AI might actually be different and just kill people on its own.
Why would it? Outline a scenario.
Just like a gun requires someone to pull the trigger, AI requires someone to prompt it (or kick off the initial loop).
What if there are already (possibly hidden) loops somewhere, maybe even ultra-slow ones at the inter-generation level? What if some AIs could do that not with so explicit and dramatic actions, but much more subtly, through undetected manipulation at a vast scale level? What if (when) they reached ASI and through billions of micro-decisions manipulated people to do apparently minor choices that appeared innocent in isolation but whose collective outcomes lead to an earlier extinction of the human being (if at some point was determined to be an obstacle)? How could we (as a species) not be outsmarted by an ASI?
To kill someone with a gun a person has to pull the trigger with the clear intent to kill.
The concern with AI, and the purpose of the OG paperclip maximizer thought experiment, is that the outcome could be extremely divergent from the intent of the original person pulling the trigger / sending the prompt.
Still more likely that a person types "kill all people" (or something tangential where this is the logical conclusion).
Right. And that's not accidental.
And who is to say that trigger hasn’t long been pulled? How do we know the agents haven’t found a way yet to modify their own code base similar to how Stuxnet manipulated the Iranian centrifuges and their control systems, stealthily recording data across generations?
If the trigger has already been pulled, then we're off the hook. That genie ain't going back in the bottle! Pop the champagne and order 3 more data centers, pronto.
No one. But we can take a wild guess that experiments are being done in this direction. And that requires intent and system design that enables the behavior to even happen (i.e., a finger on the trigger).
They do. Even if you exclude homicides and accidents (which you shouldn’t) major depression becomes much much much more deadly if you have access to a gun.
Guns make violent crimes (including common stuff like robbery, domestic disputes, or even road rage) much more deadly then it needs to be.
Yeah freeze peach, freedumb of ass-embly, dumb process durr durr
rights are so stupid and blame is so mysterious
Who the fuck is talking about guns?
Man if anyone else did any of this there would be guys with guns kicking down doors.
I don't understand why these companies automatically get a pass.
I know your question is rhetorical, but we know the answer: The USA treats corporations with kid gloves, and there's basically no consequence when a corporation does something that would be illegal or at least considered negligent if a human did it.
If you want to kill someone and not face consequences, form a corporation and have the corporation do it indirectly and stochastically.
I don‘t think Europe, Africa nor (most of) Asia is dealing with crimes committed via AI any differently then USA. I think the problem runs much deeper then USA = bad here. Europe in particular has been laxing their consumer protection for decades now (since the end of the Cold War) and it is very rare that regulators enforce existing regulations on big tech companies.
I think the problem is neo-liberal laissez-faire capitalism.
> I don't understand why these companies automatically get a pass.
Because these companies have lots of dollars and if I treat them sympathetically enough I might get some of their dollars too.