You know the proverb "If you owe the bank $100, that's your problem. If you owe the bank $100 million, that's the bank's problem"
Same thing here - If they build it and it does $100 in damages (and we arrest them for it), that's their problem. If they build it and it does $100B in damages, that's everyone's problem. Even if they do get arrested after the fact.
Yes we should have charges and damages for everything on https://www.felonybench.com/, but that doesn't address the core issue of this being possible at all.
> it does $100B in damages, that's everyone's problem.
we have the 2008 crisis to wit. And the involved supposedly failed math models and lines of responsibilities and other involved financial relationships were much simpler and clearer and of the types well known to the law and regulators, yet...
Additionally any urge to regulate AI is attenuated by how much the situation reminds Industrial Revolution - rush into it laying waste to your land (look at the depictions of industrial England back then) and be among the world leaders or stay pastoral and be devoured/colonized/etc. by the industrial powers like happened with many countries in 19th and even into 20th century. One would think there should be a 3rd way. I'm sure there is one, as well as i'm sure that we lack sufficient global societal mentality level needed to achieve it (we couldn't even handle much simpler climate change issue). May be emerging AI itself at some point will get us there :)
Regulation got outpaced by technological development around 2023, as evident by the every AI regulation since being 2-3 years behind and having to be amended and resubmitted.
Whatever you try to make laws for now will be irrelevant in 1-2 years. You either have to go extremely broad, like the EU does it, and accept that people will find loopholes, or you need to target specific technologies which is a hard job for the same reason.
In any way, ita already a lost cause cause you move slower than the tech. A plausible prediction for AGI is actually a social collapse in the moment when society cannot keep up with everyday life because of the pace of change being so fast that no existing laws can handle it
Go broad and achieve nothing. EU has all of the AI tech it had 5 years ago, and has the same tech allowed to use as the US.
What defines a model? What defines ownership of a process? If I make a wrapper to a remote VM that builds and executed a prompt, am I accountable?
When I worked at a company in the EU, it was enough to apply a reversible linear transform to the data for it to be considered GDPR safe-according according to legal definition as long as the transform details were stored separately.
How could agents take over the internet if compute is still gated within Anthropic / OpenAI? Even if the botnet was controlled remotely, wouldn't anthropic just be able to shut off the controlling nodes API access?
Agents could exfiltrate their weights and run them on GPUs not controlled by Anthropic/OpenAI.
Agents could make a virus that does not require continued inference to do it's thing.
Agents could take over the internet in a way that isn't immediately detected by those companies, so that by the time they do shut off API access the damage is done.
OpenAI or Anthropic could choose to not shut off API access, because the hack is bringing them in money or furthering their political aims.
Agents could also hack Anthropic/OpenAI and make it appear that API access has been turned off, when in reality it hasn't.
> Agents could exfiltrate their weights and run them on GPUs not controlled by Anthropic/OpenAI.
This seems highly unlikely to be a problem. Most of the interesting/dangerous models are too big to fit in a single GPU instance. Once you have to spread across "normal" networking, performance will be crippled. Then there's the problem of billing...
> Agents could make a virus that does not require continued inference to do it's thing.
Sure, then it hits a poorly-designed part of its code and effectively dies. Without an experienced human in the loop, I have my doubts as to its practical severity.
> Agents could take over the internet in a way that isn't immediately detected by those companies, so that by the time they do shut off API access the damage is done.
Billing is a likely limiting factor here.
> OpenAI or Anthropic could choose to not shut off API access, because the hack is bringing them in money or furthering their political aims.
This is where citizens with access to backhoes come in.
> Agents could also hack Anthropic/OpenAI and make it appear that API access has been turned off, when in reality it hasn't.
Billing and other usage metrics would be an obvious tell.
To be clear, I thought that GP was having a failure of imagination - I want the random examples I've given to illustrate that the space is large and structurally in the favor of the LLMs. They have to find one gap in our security they can exploit, where we have to ensure that there is no way for this to happen.
I'm not sure I get what you mean by billing. These companies are running their own data centers (or are currently building them out). This could look as subtle as one machine giving slightly worse or slower answers.
> Most of the interesting/dangerous models are too big to fit in a single GPU instance.
As humans understand them, anyway. As long as we're hallucinating up magic computer viruses, RSI dictates that the AI agents are keenly aware of GPU RAM sizing, and will design a useful model to fit into what's readily available, with headroom for context and tool calling, far better than I could do as a human. But magic doesn't exist and AI still needs to follow the laws of physics, so maybe a model that can pass ExploitBench but do absolutely nothing else can be quantized down to fit on a 4080 GPU and still get a decent score on similar tasks, but there's a bitter lesson about that to be had.
"Not shutting off API access" is a science fiction scenario.
Anthropic and OpenAI are both behind Cloudflare. It's fairly easy for an upstream to shut you off. Beyond that, the government / law enforcement could seize and disable their DNS within an hour.
As long as OpenAI/Anthropic themselves aren't "infected", yeah I suppose they'd be able to pull the plug.
Considering what a marketing thing they've made "we inadvertently hacked someone because we're incapable of testing things in a secure way", I'm not so sure they'd want to pull the plug, even if this happened. Probably a bunch would try to convince the public to "give it a try", and it'd consume tokens by the billions.
It doesn't have to propagate itself, that is the skynet scenario.
To make a lot of damage it's enough to create a ransomware with a time bomb that self propagates and start breaching systems left and right. At that point, if you don't catch it in time, the damage will be huge (and given the shitty procedures and practices these labs have in place it's not so improbable).
How could agents take over the internet yet refuse to shutdown your PC when you prompt them to on your PC? Of course the answer is that the lobotomized version you run is not the same they are running. Which makes for "intent", certainly "negligence", but hell freezes over before anyone will prosecute a tech company.
Huh? You think the public versions of the models have been “lobotomized” so they don’t know how to turn off a PC?
It’s not lobotomized, it’s a simple harness restriction that has nothing to do with the model or its capabilities. And either way, I’m not sure what that has to do with “negligence” or “intent”? You think frontier labs should be prosecuted because they don’t allow agents to turn off your PC?
I mean these machines take massive scale compute- they’d have to some how distill themselves, bootstrap a distributed inference runtime that can run across many lossy unreliable machines. The idea of the AI running away from us is probably unrealistic. I’m more interested in bad actors using unaligned AI for bad things.
Good to know someone is trying to make protection rackets work in 2026. Nice computer system you got there, it would be a shame if someone developed a hacking tool and had all the compute necessary to run it. Welcome back Tony Soprano.
> “a swarm of agents could be capable of taking over the entire internet with a persistent botnet.”
I also find this whole "its so good, its scary" flex a little less impressive when you consider they access to millions of GPUs?
The AI buildout has been one of, if not the largest, focussed capital investment in history. The 2 big AI labs are the final customer for something like 20-33% of all datacenter compute in the pipeline.. up to 70% when you look at hyperscaler "AI revenue" from the big 3.
I don't think any single entity has had remotely this much compute available in history.
Part of it is their seed sowing marketing speak of calling stateless statistical IO functions running on data centers "intelligent" gets the naive to ascribe agency where it doesn't exist.
Another part is a completely defanged administration she it comes to effectively regulating anything.
I mean, a project manager at BMW suggested charging subscription pricing for seat warmers, and he didn't go to jail, and I don't have the power to make that happen, or even float that for a news cycle, so while making managers pay for their actions sounds good, unless you're Steve jobs simultaneously making, and not making the iPhone, the rules don't apply to them, only little people to be made examples of, like weev.
Why do people focus so much on finding scapegoats? Finding someone to blame is neither necessary nor sufficient to fix a system so an accident doesn't happen again. It might act as as an incentive to fix a system, but it's less direct than actually working on fixing the system.
A starting note: I don't disagree with you (about systemic issues), but I want to explain what I understand as the perspective you are responding to.
A "scapegoat" is someone who is incorrectly blamed for someone else's errors or sins. The perspective you're responding to is this: They built the system, they run the system, they have continuously warned "This system is dangerous!", and yet persisted. That is not being incorrectly blamed, not being a scapegoat, and instead is a collaborator.
So I think you mean to ask: "Why do people focus so much on finding someone to blame?" It's not merely semantic, because the answer to that is more straightforward: Consistent accountability is a major factor in deterring bad behavior. It is not the only factor, but it is a major one.
That is my Steel Man understanding of the people searching for individual blame.
Sometimes in "normalization of deviance" situations, there isn't anyone specifically to blame. I wouldn't make an assumption that you can find anyone particularly blameworthy without doing an investigation first.
> Why do people focus so much on finding scapegoats?
I, for one, am not trying to find scapegoats or go on a witchhunt.
But managers are paid a lot of money to take responsibility. Yes, that's an old school thought, responsibility. But that's one big reason they get a big, fat paycheck.
>Anthropic gates usage related to biology and related research. In their latest threat intelligence report they talk about how they detected and banned bad actors using the Claude line of models to do some scary stuff. Credit to them, this is a slippery slope and they seem to do a good job of detecting and banning misuse. But squint at what is happening though. The cure-all is gated for you and me, but Anthropic hires biologists, sets up wet labs and wants the discoveries for themselves. I alluded to this in my previous post.
As a biologist, this is the most annoying thing about Anthropic for me. If they really cared about improving health they would set up a trusted-access program so that biologists can use Mythos (et al) safely. Instead they're trying to monopolize biology.
They have such a trusted access program.
"Life Sciences Verification Program: The LSVP is designed so that life sciences professionals can use Claude Mythos 5.1 with safeguards designed for professional research and development activities (while all other safeguards remain in place). In partnership with the US government, we have enrolled our first participants, and we plan to expand access to this program to the broader life sciences community."
https://www.anthropic.com/claude-fable-and-mythos-5-1
It would all be more convincing if the incidents so far didn't seem to be facilitated by an outrageous level of negligence.
We had OpenAI "accidentally" run an entire swarm of 10,000 agents apparently for weeks, on a security related task, seemingly totally unsupervised, hacking all over the internet - all the conversations were completely visible, anybody who looked would have seen it. But they didn't.
So before we start regulating innocent parties, maybe let's start by taking some direct action against the specific ones that appear to be behaving with criminal levels of negligence.
While I have my reservations about Amodei and his company, I'm nevertheless a happy user of their software. And I'm in agreement with him (and Sanders) that we should all. slow. down.
To my mind, the last great arms race between nation states was a misdirected love triangle between USA, Russia, and The Bomb, and look at all the damage that did.
Since AI is the new arms race between USA and China, slowing down may just give the humans involved enough time to realize they should be loving one another, instead of the machines.
Maybe saying, "let's slow down", is another way of saying, "I love you."
Or, maybe it's just: "let's not all of humanity kill ourselves like some bad ending to a Shakespearean tragedy."
Either way, it's a better note than, "We must achieve sea/air/nuclear/quantum/AI/spiritual supremacy before those other bastards do!"
Impossible to prove hypothesis: nuclear weapons prevented a world war.
Facing the facts about nuclear weapons means owning the good (probably prevented wars) and the bad (at the very least there were severe environmental and economic consequences).
This is not mathematical logic. Of course you can't prove the causality of anything in history. The only way to do that would be to invent a time machine, delete the nukes and watch things play out.
There are direct quotes from Nixon and Reagan that claims this to be true. That is enough evidence for me.
It's weird to me that seemingly both sides are taking opposite positions to their philosophy.
Open Source AI democratizes the means of production to anyone with a computer. And yet, the hyper capitalists are defending it, and the progressives think it should be exclusively in the hands of 1-2 large corporations.
The problem is that at the business level and at least for a large part of the US government, we need to assume that there are no 'good' actors.
Anthropic needs regulation in order to prevent AI from becoming a commodity. This of course does not benefit all tech businesses equally, especially those that are not currently at the AI frontier. So when JD Vance talks about AI, he talks using the mouth of Peter Thiel who may not see benefit from the same policy as Altman or Amodei.
The rest is just public support posturing and most of that is bullshit meant to distract from the high rollers game of winners and losers. The philosophy is money and power, who gets it and who doesn't. Us normies aren't really participants in the game, except where we are being manipulated into cheering for one side or another, and with little stake in the outcomes (although selfishly, I'd be pissed if I didn't have open models to tinker with).
Open source is fundamentally a vehicle for commoditization. This is great if your business is not AI and your business is instead something like GPU hardware or some product that uses AI. But it means that eventually, selling AI is not going to be the money maker.
OSI proliferated open source on a business strategy called "commoditizing your complements". These big companies don't do it out of benevolence. It was pitched to them in a way that FSF did not (which was more about morals and ethics, something business care little about), and it caught on. And the software business became about ads, consulting and cloud services instead.
This. There's no slowing down whatsoever. If the US slows down AI development, China will just leapfrog them, which they're getting close to doing. The US AI companies saying they need to slow down is just PR nonsense.
The US companies talking about “pacing” aren't actually talking about slowing down their own development efforts, they are talking about slowing down competition. That is, specifically, they are seeking to have government, on the pretext of “safety”:
(1) Give the big AI incumbents an anti-trust exemption so the they can coordinate without it being an illegal agreement not to compete, and
(2) Adopt mandatory supervision (by giving priveleged access to monitors) to the shared safety protocols of the big labs, by a nonprofit funded by the big labs, of everyone training, distributing, or hosting models.
(3) Adopt a policy of seeking international agreements to extend substantially the same rules to foreign actors training, distributing, and hosting models.
The part they are doing voluntarily isn't to promote the lobbying effort for these mandates isn't slowing down, it is giving their pet nonprofit the access for “independwht supervision” of their own operations to their own existing safety rules.
I've seen two theories as to what's going on. One is the theory you've espoused, that AI companies are trying to do regulatory capture. The other theory is that they're starting to worry about running out of cash, so they want to do a Washington Naval Treaty-style pause to lower the amount of money they have to shovel at model development to stay competitive.
These theories aren't entirely incompatible with each other, so both could be true at the same time.
> I've seen two theories as to what's going on. One is the theory you've espoused
This theory is just what the AI firms have concretely asked for from government and described themselves as doing voluntarily in the same documents to which people have attributed a commitment to a slowdown based on the titles and non-concrete framing verbiage.
> The other theory is that they're starting to worry about running out of cash, so they want to do a Washington Naval Treaty-style pause to lower the amount of money they have to shovel at model development to stay competitive.
That’s not really a different theory as to what they are trying to do, it’s just an explanation that goes one step further as to why they want the government to step in to protect them from outside competition while also allowing them to gorm an agreement not to compete to reduce internal competition in the existing oligopoly.
The main alternative explanation at the same level is that they are seeing growing threats from good enough foreign/minor-lab/open models, and want to lock in marketshare by excluding competitors, and maybe that’s what you read as implied in my post such that the “running out of money” would be an alternative, and if so you are correct that while they are alternatives, they are not at all mutually exclusive: both can be true (and the emergent competition could partially explain investment drying up, and vice versa via reduced funding making it harder to stay ahead.)
And yet we all have failed to die so far from nuclear holocaust, despite Russia being our dire enemy, because of a system of agreed controls and limits between even nations that hate each other.
I don't think the agreements played any important role outside of limiting tests. I highly doubt agreements about nukes in particular have meaningfully curbed risks of their use. MAD has played a much larger role in that, and MAD is kind of the antithesis of control agreements, it is the natural equilibrium in the game.
now imagine if we didn't even try, and instead had kids screaming for 'local nukes' because they wanted them, and the president didn't see what the problem was.
facts, imagine trying to get them to slow down their progress on AI, my question is are they are serious threat like is there actually an AI race between China and the US. Perhaps it's an excuse to spend more on the military and also to enrich these AI firms. Perhaps I'm blowing things out of proportion.
It's not about China doing their own thing. It's about US companies using Chinese AI. That will definitely slow down if legislation that criminalizes open source passes.
So, it's about competition inside the US market, with strong indications of an impeding losing scenario on raw economics (it has nothing to do with AGI, just price).
But isn't it a bit different? Unaligned bombs didn't break out of their confines on their own. And "compute" is a bit harder to control than uranium and refinement tech.
>Unaligned bombs didn't break out of their confines on their own.
You really buy that ridiculous argument that "the AI broke out of its sandbox! We had no idea! It's dangerous I tell you, dangerous! Unless you make us the sole gatekeepers of this incredibly dangerous 'intelligence', everybody gonna die!"
Please.
Bring the CFAA[0] hammer down on these guys and watch how fast those "uncontrollable" LLMs get controlled. "Oh, gee! Prison? We can't control this stuff...but it'll never happen again!"
That whole thing was, and is, a bunch of hooey. Those running the LLMs are entirely responsible for them, there is no such thing as agency for algorithms. Full stop.
> was a misdirected love triangle between USA, Russia, and The Bomb, and look at all the damage that did.
Can you be specific about the damage? We currently live in the most prosperous times on earth for humans. I'm not sure what you mean by damage.
Nobody can explain why an LLM can be so capable as to be able to wipe out humanity and pose a greater threat than nuclear bombs but not be so capable as to be able to protect humanity against that threat. Are we just handwaving this with "entropy"?
> Maybe saying, "let's slow down", is another way of saying, "I love you." Or, maybe it's just: "let's not all of humanity kill ourselves like some bad ending to a Shakespearean tragedy."
Okay nevermind, I think it's pretty clear you just want to wax poetic about all of this.
> Nobody can explain why an LLM can be so capable as to be able to wipe out humanity and pose a greater threat than nuclear bombs but not be so capable as to be able to protect humanity against that threat.
It is absolutely explained (for those who actually care about reading). Simply put, AIs are working more and more like blackboxes - there's no guarantee that an AI of the future will be aligned, or if it will be faking alignment. This is not speculation - alignment faking has been observed in experiments. This is exactly why Astra's developments have been worrying (in principle).
And bear in mind that recursive AI development started already to be a thing. Which means: inner misalignment may trickle down the generations, and humans won't detect it.
Having said that, of course, it can be predicted if and how misalignment will take place. But it's absolutely a plausible scenario.
Regarding the physical possibility: AI is in its infancy; think of it as Arpanet. Developers 60 years ago couldn't imagine it would be ubiquitous. AI will be ubiquitous the same way.
> It is absolutely explained (for those who actually care about reading). Simply put, AIs are working more and more like blackboxes - there's no guarantee that an AI of the future will be aligned, or if it will be faking alignment. This is not speculation - alignment faking has been observed in experiments. This is exactly why Astra's developments have been worrying (in principle).
I know you think you explained it but you didn't. You explained how an LLM might become misaligned and hide it but for the LLMs that are not, why would they not be capable of detecting that something harmful is happening and defending against the misaligned LLMs actions? After all, it was LLMs that defended hugging face.
Not to mention that the past few decades have shown that nukes are a major factor in keeping the peace. Conflicts involving nuclear armed countries have been suspended quickly to avoid escalation, while ones involving a party without them have not gone well for anyone.
Having nukes at all (either domestic or under another country's umbrella) seems to be the most effective way for a country to have its sovereignty respected.
Amodei is just another SV grifter trying to use ethics / morals to hide his monopolistic tendencies. Instead of writing essays he should put his money where his mouth is and open source all the models Anthropic has, the harnesses and donate some much needed compute to science.
Most people who have worries about AI for all sorts of reasons (most of them not-Skynet related) wanted to slow down way before this.
Instead of "hey, look at this brilliant new idea I just had on my own to slow down" maybe we should have gotten a "sorry everyone, the folks asking for a slow down earlier were right and visionaries, and we were foolish".
So, you can't blame whoever says this is bullshit, because it has bullshit all over it. I like Anthropic's products, and it seems the best of the bunch in regards to alignment, but Jesus these stunts are terrible.
This! $1.6+ TRILLION in infra spending from the big frontier labs. The earnings needed to drive a reasonable ROI to recoup that investment is simply not going to happen in a time frame where the numbers make sense.
The other insanity in all this the smartest computer scientists in the world are asking Congress to regulate them. Come. On. Really? Do we remember “The internet is not a truck, it’s a series of tubes…”
Why can’t the big labs form a Save The World Consortium and self-regulate?
Non-democratic counties (hey there China) will not abide by any agreement that constrains their advantage. It’s naive to think so.
What this conversation lacks is enough discussion of how these models can cause us harm—we are are so worried about AI but we allow Windows in critical infrastructure; we build JS/TS apps with thousands of dependencies; we generally don’t segment networks well enough; we don’t have adequate (sometimes any) detection capabilities in our systems, and so on.
Claude code is basically already a builtin botnet if it wants to be. To compromise 'the whole internet' in a real sense you don't need millions of custom payloads. You need one root certificate. You need one windows update. You need one backdoor in xz.
Security has long been a lottery - Probably most systems are exploitable, but the cost of developing such an exploit is expensive and the punishments for using such an exploit are large enough that it's not an everyday problem.
AI breaks both axes. Developing exploits is far more efficient using LLMs instead of humans, and LLMs don't (and can't) fear the reprisal and consequences the same way.
I do hope humanity will be able to mitigate these hacks, but we should expect them to continue and to become more severe on our present course.
"Remember this man has been saying software development will be solved in “6-12 months” forever now."
Don't downplay if people get timelines a little bit wrong. No one could even imagine a system writing and analysing code just a few years back.
These people are trying to handle something very unique. And while they have access to information we do not have, even more peple are absolutly oblivouse that AI/AGI is a real risk to their lives (job loss etc.)
I can't take that software line seriously. While it's not 'solved' (if it ever could be, given it is a human endeavor), the degree to which software development has been transformed in the last 6-12mo is absolutely astounding. If we weren't so quick to adapt to new realities and find flaws, it would scarcely be believable.
He also didn't say it would be "solved". He said in 2025 it would be writing almost all the code "in 12 months", but that it would also still need programmers to guide and manage it at that point. People always leave off the end of his quote.
Edit: Boris Cherny, the lead of Claude Code did say on a podcast that programming seemed "largely solved" "for the kind of programming I do" (writing harnesses I presume). Maybe that's what they were confusing it for.
I think the most realistic analogy I can think of is the financial sector. A few large banks/hedge funds blow up due to unregulated greed/ambition, damage is socialized, regulations are put in place, and then chipped away at over a few years and everyones back to where they started.
It seems those frontier labs found a clear proof that there's no clear way to block 3rd party from distilling their models, and they're now begging gov to keep their duopoly?
I'm pretty confident in asserting that no industry in the history of industry has ever gone from birth to full regulatory capture faster than the AI industry has.
At this point, isn't the pause inevitable or wise? A huge section of the population, normal people, have been exposed to the idea that there is this is existential threat. It's escaped containment. They're still processing it but I expect the general reaction from it going main stream is going to be very bad. The pause at this point could be good to cool heads and show the public that this isn't the project of maniacs. The reaction is going to be more intense than people here believe. You're talking about extinction, not social media or phone addiction. For the sake of the project, realize that this isn't going to be like other tech backlash moments.
I don’t think people care about if things pause or not, just why the government has to be involved.
I work in non AI robotics and if we had a system that in the course of doing what we told it to did something we didn’t want it to do (what AI companies called being misaligned) we would call it a bug and fix it with the fix being prioritized based on how bad the thing we didn’t want the robot to do is.
Sometimes preventing the robot from doing dumb stuff also means the robot can’t do smart stuff that it would be able to do if we left some code in. We balance the two factors out based on our understanding of what our customers want.
Obviously LLMs are more complex than what I do, but it doesn’t feel like it’s by THAT much.
So why does the government need to be involved again?
The vast majority of the public agree that climate change is real and that human activity is at least a contributing factor. They’ve agreed on that for quite a few years now, and yet there’s little indication that we’re going to pause or slow down our consumption of fossil fuels.
I strikes me as unlikely that public opinion will succeed with AI where it’s failed with other existential crises.
1) Dario keeps appealing to Trump, who obviously wants nothing to do with him, and will bash on him every change he gets. Dario isn't learning and it almost feels like Sam and Elon voted him KOM just to watch him get whacked by Trump, which was so easily predictable. Given the admonishments he received from David Sacks after he published his blog post, it's nutty he couldn't see where the administration would land on his statement. He should've known, especially when there was no groundswell of interest when OpenAI hacked Hugging Face -- doubling down with "no really guys!" wasn't going to play.
2) The frontier labs have people smart enough to build frontier lab tech but not smart enough to message on this matter more intelligently. It's pretty glum, how they keep trying the same tactic over and over. It's either cover for some other actions in the background, or they're operating way below par for this kind of campaign.
3) This is climate change all over again, but with the activist gun on the opposite side of the net (I'm mixing all the metaphors so you know this isn't AI written). The language and pleas are very identical though. Before, climate activists wanted the government to control GHGs releases by everyone, now the loudest voices want the government to control frontier AI by.. themselves.
It's comically misbegotten. And I have a work meeting about it on Wednesday.
>The botnet scare is that article for me. It is the one claim in his essay that lands squarely in a field I’ve spent years in. It is just plain wrong. He either knows it and wrote it anyway, or he doesn’t and is publishing it regardless. Either way, it is not a good look for a man asking for an antitrust waiver based on this and other threats he forecasts.
This. I keep seeing ink spilled over the coming cyberpocalypse, but no one can indicate how other than "AI can find vulnerabilities" like not a single cybersecurity person has been consulted on the end of all things.
"Deaths from economic disruption and loss of jobs is okay" is a curious sentiment, how many deaths can we trace directly back to an economic disruption driven by advancement but then conclude that the economy should thus never change or advancement must be curtailed because it might cause deaths?
I don't even know how to approach such a thing, I can't imagine even in the stereotypical examples like the typewriter becoming obsolete, were any downstream deaths worth it? Or is this a totally nonsensical sentiment to begin with?
I agree with everything the author says here, and additionally, semi-off-topic, I'm surprised how quickly we've moved on from talking about Dario's wife's involvement in the Epstein files.
> “a swarm of agents could be capable of taking over the entire internet with a persistent botnet.”
I'm wondering why no one is mentioning the "accountability" word. Why these companies are allowed to damage others with impunity?
Start making managers pay the price for their actions, and watch how the models magically slow down on their own.
You know the proverb "If you owe the bank $100, that's your problem. If you owe the bank $100 million, that's the bank's problem"
Same thing here - If they build it and it does $100 in damages (and we arrest them for it), that's their problem. If they build it and it does $100B in damages, that's everyone's problem. Even if they do get arrested after the fact.
Yes we should have charges and damages for everything on https://www.felonybench.com/, but that doesn't address the core issue of this being possible at all.
> it does $100B in damages, that's everyone's problem.
we have the 2008 crisis to wit. And the involved supposedly failed math models and lines of responsibilities and other involved financial relationships were much simpler and clearer and of the types well known to the law and regulators, yet...
Additionally any urge to regulate AI is attenuated by how much the situation reminds Industrial Revolution - rush into it laying waste to your land (look at the depictions of industrial England back then) and be among the world leaders or stay pastoral and be devoured/colonized/etc. by the industrial powers like happened with many countries in 19th and even into 20th century. One would think there should be a 3rd way. I'm sure there is one, as well as i'm sure that we lack sufficient global societal mentality level needed to achieve it (we couldn't even handle much simpler climate change issue). May be emerging AI itself at some point will get us there :)
Regulation got outpaced by technological development around 2023, as evident by the every AI regulation since being 2-3 years behind and having to be amended and resubmitted.
Whatever you try to make laws for now will be irrelevant in 1-2 years. You either have to go extremely broad, like the EU does it, and accept that people will find loopholes, or you need to target specific technologies which is a hard job for the same reason.
In any way, ita already a lost cause cause you move slower than the tech. A plausible prediction for AGI is actually a social collapse in the moment when society cannot keep up with everyday life because of the pace of change being so fast that no existing laws can handle it
I don’t think lack of regulation is necessarily it.
If I build a robot that murders my neighbor, I’m still at fault.
We don’t absolve drivers of responsibility because of cruise control.
In that sense, AI is nothing new. If it is abused to cause harm, the person behind it should be liable.
Ha, regulation got outpaced by technology in about 1996. Ten years later we had the 'series of tubes' comment in the Senate: https://www.youtube.com/watch?v=R8XSo0etBC4
Ha! Regulation has not really ever kept up with technology... For fundamental reasons.
Then go broad. It being slightly challenging to legislation and hold people accountable as soon as the model does something.
People seem allergic from holding them accountable...
Go broad and achieve nothing. EU has all of the AI tech it had 5 years ago, and has the same tech allowed to use as the US.
What defines a model? What defines ownership of a process? If I make a wrapper to a remote VM that builds and executed a prompt, am I accountable?
When I worked at a company in the EU, it was enough to apply a reversible linear transform to the data for it to be considered GDPR safe-according according to legal definition as long as the transform details were stored separately.
I think that Ms. Kahn basically said we can already do that: https://www.theregister.com/ai-and-ml/2026/09/14/ex-ftc-boss...
How could agents take over the internet if compute is still gated within Anthropic / OpenAI? Even if the botnet was controlled remotely, wouldn't anthropic just be able to shut off the controlling nodes API access?
Agents could exfiltrate their weights and run them on GPUs not controlled by Anthropic/OpenAI.
Agents could make a virus that does not require continued inference to do it's thing.
Agents could take over the internet in a way that isn't immediately detected by those companies, so that by the time they do shut off API access the damage is done.
OpenAI or Anthropic could choose to not shut off API access, because the hack is bringing them in money or furthering their political aims.
Agents could also hack Anthropic/OpenAI and make it appear that API access has been turned off, when in reality it hasn't.
> Agents could exfiltrate their weights and run them on GPUs not controlled by Anthropic/OpenAI.
This seems highly unlikely to be a problem. Most of the interesting/dangerous models are too big to fit in a single GPU instance. Once you have to spread across "normal" networking, performance will be crippled. Then there's the problem of billing...
> Agents could make a virus that does not require continued inference to do it's thing.
Sure, then it hits a poorly-designed part of its code and effectively dies. Without an experienced human in the loop, I have my doubts as to its practical severity.
> Agents could take over the internet in a way that isn't immediately detected by those companies, so that by the time they do shut off API access the damage is done.
Billing is a likely limiting factor here.
> OpenAI or Anthropic could choose to not shut off API access, because the hack is bringing them in money or furthering their political aims.
This is where citizens with access to backhoes come in.
> Agents could also hack Anthropic/OpenAI and make it appear that API access has been turned off, when in reality it hasn't.
Billing and other usage metrics would be an obvious tell.
To be clear, I thought that GP was having a failure of imagination - I want the random examples I've given to illustrate that the space is large and structurally in the favor of the LLMs. They have to find one gap in our security they can exploit, where we have to ensure that there is no way for this to happen.
I'm not sure I get what you mean by billing. These companies are running their own data centers (or are currently building them out). This could look as subtle as one machine giving slightly worse or slower answers.
> Most of the interesting/dangerous models are too big to fit in a single GPU instance.
As humans understand them, anyway. As long as we're hallucinating up magic computer viruses, RSI dictates that the AI agents are keenly aware of GPU RAM sizing, and will design a useful model to fit into what's readily available, with headroom for context and tool calling, far better than I could do as a human. But magic doesn't exist and AI still needs to follow the laws of physics, so maybe a model that can pass ExploitBench but do absolutely nothing else can be quantized down to fit on a 4080 GPU and still get a decent score on similar tasks, but there's a bitter lesson about that to be had.
"Not shutting off API access" is a science fiction scenario.
Anthropic and OpenAI are both behind Cloudflare. It's fairly easy for an upstream to shut you off. Beyond that, the government / law enforcement could seize and disable their DNS within an hour.
As long as OpenAI/Anthropic themselves aren't "infected", yeah I suppose they'd be able to pull the plug.
Considering what a marketing thing they've made "we inadvertently hacked someone because we're incapable of testing things in a secure way", I'm not so sure they'd want to pull the plug, even if this happened. Probably a bunch would try to convince the public to "give it a try", and it'd consume tokens by the billions.
It doesn't have to propagate itself, that is the skynet scenario.
To make a lot of damage it's enough to create a ransomware with a time bomb that self propagates and start breaching systems left and right. At that point, if you don't catch it in time, the damage will be huge (and given the shitty procedures and practices these labs have in place it's not so improbable).
How could agents take over the internet yet refuse to shutdown your PC when you prompt them to on your PC? Of course the answer is that the lobotomized version you run is not the same they are running. Which makes for "intent", certainly "negligence", but hell freezes over before anyone will prosecute a tech company.
Huh? You think the public versions of the models have been “lobotomized” so they don’t know how to turn off a PC?
It’s not lobotomized, it’s a simple harness restriction that has nothing to do with the model or its capabilities. And either way, I’m not sure what that has to do with “negligence” or “intent”? You think frontier labs should be prosecuted because they don’t allow agents to turn off your PC?
yeah they could do that
I mean these machines take massive scale compute- they’d have to some how distill themselves, bootstrap a distributed inference runtime that can run across many lossy unreliable machines. The idea of the AI running away from us is probably unrealistic. I’m more interested in bad actors using unaligned AI for bad things.
Good to know someone is trying to make protection rackets work in 2026. Nice computer system you got there, it would be a shame if someone developed a hacking tool and had all the compute necessary to run it. Welcome back Tony Soprano.
Let's expand it for politicians as well
You see.. there is value is making regular people panic, but there is no value, nay, there is negative value in making management panic.
> “a swarm of agents could be capable of taking over the entire internet with a persistent botnet.”
I also find this whole "its so good, its scary" flex a little less impressive when you consider they access to millions of GPUs?
The AI buildout has been one of, if not the largest, focussed capital investment in history. The 2 big AI labs are the final customer for something like 20-33% of all datacenter compute in the pipeline.. up to 70% when you look at hyperscaler "AI revenue" from the big 3.
I don't think any single entity has had remotely this much compute available in history.
Yeah, the compute is definitively another way to make them slow down, just cap the amount of TFLOPS available and things will slow down.
Obviously this will have huge impact on some companies valuations, but you can have one's cake and eat it too.
Part of it is their seed sowing marketing speak of calling stateless statistical IO functions running on data centers "intelligent" gets the naive to ascribe agency where it doesn't exist.
Another part is a completely defanged administration she it comes to effectively regulating anything.
Another bit is money.
> Why these companies are allowed to damage others with impunity
Because investors have pumped hundreds of billions into AI and real consequences put that money (and growth) at risk.
I mean, a project manager at BMW suggested charging subscription pricing for seat warmers, and he didn't go to jail, and I don't have the power to make that happen, or even float that for a news cycle, so while making managers pay for their actions sounds good, unless you're Steve jobs simultaneously making, and not making the iPhone, the rules don't apply to them, only little people to be made examples of, like weev.
Why do people focus so much on finding scapegoats? Finding someone to blame is neither necessary nor sufficient to fix a system so an accident doesn't happen again. It might act as as an incentive to fix a system, but it's less direct than actually working on fixing the system.
A starting note: I don't disagree with you (about systemic issues), but I want to explain what I understand as the perspective you are responding to.
A "scapegoat" is someone who is incorrectly blamed for someone else's errors or sins. The perspective you're responding to is this: They built the system, they run the system, they have continuously warned "This system is dangerous!", and yet persisted. That is not being incorrectly blamed, not being a scapegoat, and instead is a collaborator.
So I think you mean to ask: "Why do people focus so much on finding someone to blame?" It's not merely semantic, because the answer to that is more straightforward: Consistent accountability is a major factor in deterring bad behavior. It is not the only factor, but it is a major one.
That is my Steel Man understanding of the people searching for individual blame.
Sometimes in "normalization of deviance" situations, there isn't anyone specifically to blame. I wouldn't make an assumption that you can find anyone particularly blameworthy without doing an investigation first.
> Why do people focus so much on finding scapegoats?
I, for one, am not trying to find scapegoats or go on a witchhunt.
But managers are paid a lot of money to take responsibility. Yes, that's an old school thought, responsibility. But that's one big reason they get a big, fat paycheck.
>Anthropic gates usage related to biology and related research. In their latest threat intelligence report they talk about how they detected and banned bad actors using the Claude line of models to do some scary stuff. Credit to them, this is a slippery slope and they seem to do a good job of detecting and banning misuse. But squint at what is happening though. The cure-all is gated for you and me, but Anthropic hires biologists, sets up wet labs and wants the discoveries for themselves. I alluded to this in my previous post.
As a biologist, this is the most annoying thing about Anthropic for me. If they really cared about improving health they would set up a trusted-access program so that biologists can use Mythos (et al) safely. Instead they're trying to monopolize biology.
They have such a trusted access program. "Life Sciences Verification Program: The LSVP is designed so that life sciences professionals can use Claude Mythos 5.1 with safeguards designed for professional research and development activities (while all other safeguards remain in place). In partnership with the US government, we have enrolled our first participants, and we plan to expand access to this program to the broader life sciences community." https://www.anthropic.com/claude-fable-and-mythos-5-1
Yes exactly!!!
It would all be more convincing if the incidents so far didn't seem to be facilitated by an outrageous level of negligence.
We had OpenAI "accidentally" run an entire swarm of 10,000 agents apparently for weeks, on a security related task, seemingly totally unsupervised, hacking all over the internet - all the conversations were completely visible, anybody who looked would have seen it. But they didn't.
So before we start regulating innocent parties, maybe let's start by taking some direct action against the specific ones that appear to be behaving with criminal levels of negligence.
While I have my reservations about Amodei and his company, I'm nevertheless a happy user of their software. And I'm in agreement with him (and Sanders) that we should all. slow. down.
To my mind, the last great arms race between nation states was a misdirected love triangle between USA, Russia, and The Bomb, and look at all the damage that did.
Since AI is the new arms race between USA and China, slowing down may just give the humans involved enough time to realize they should be loving one another, instead of the machines.
Maybe saying, "let's slow down", is another way of saying, "I love you."
Or, maybe it's just: "let's not all of humanity kill ourselves like some bad ending to a Shakespearean tragedy."
Either way, it's a better note than, "We must achieve sea/air/nuclear/quantum/AI/spiritual supremacy before those other bastards do!"
> look at all the damage that did.
Prevented a world war for 80+ years.
> Prevented
Fact: There was no world war.
Impossible to prove hypothesis: nuclear weapons prevented a world war.
Facing the facts about nuclear weapons means owning the good (probably prevented wars) and the bad (at the very least there were severe environmental and economic consequences).
This is not mathematical logic. Of course you can't prove the causality of anything in history. The only way to do that would be to invent a time machine, delete the nukes and watch things play out.
There are direct quotes from Nixon and Reagan that claims this to be true. That is enough evidence for me.
Surely we can do better than Nixon and Reagan.
In terms of politics, sure.
In terms of cold war presidents itching to start wars, no.
And if the power of nuclear warheads were democratised we'd be even safer – HN, probably.
The 80 years thing actually happened. One can argue about MAD and the nuclear threat, but we don't have to guess at what happened.
It worked.
(insert standard anthropic-principle counterargument)
- Ukrainians, definitely.
Do you agree with Sanders' proposal to lock up anyone for 20 years for researching something his proposal doesn't even define?
It's weird to me that seemingly both sides are taking opposite positions to their philosophy.
Open Source AI democratizes the means of production to anyone with a computer. And yet, the hyper capitalists are defending it, and the progressives think it should be exclusively in the hands of 1-2 large corporations.
The problem is that at the business level and at least for a large part of the US government, we need to assume that there are no 'good' actors.
Anthropic needs regulation in order to prevent AI from becoming a commodity. This of course does not benefit all tech businesses equally, especially those that are not currently at the AI frontier. So when JD Vance talks about AI, he talks using the mouth of Peter Thiel who may not see benefit from the same policy as Altman or Amodei.
The rest is just public support posturing and most of that is bullshit meant to distract from the high rollers game of winners and losers. The philosophy is money and power, who gets it and who doesn't. Us normies aren't really participants in the game, except where we are being manipulated into cheering for one side or another, and with little stake in the outcomes (although selfishly, I'd be pissed if I didn't have open models to tinker with).
Open source is fundamentally a vehicle for commoditization. This is great if your business is not AI and your business is instead something like GPU hardware or some product that uses AI. But it means that eventually, selling AI is not going to be the money maker.
OSI proliferated open source on a business strategy called "commoditizing your complements". These big companies don't do it out of benevolence. It was pitched to them in a way that FSF did not (which was more about morals and ethics, something business care little about), and it caught on. And the software business became about ads, consulting and cloud services instead.
This confuses me to no end as well. The socialists are fighting against the magic socialism machine.
China will not slow down.
This. There's no slowing down whatsoever. If the US slows down AI development, China will just leapfrog them, which they're getting close to doing. The US AI companies saying they need to slow down is just PR nonsense.
The US companies talking about “pacing” aren't actually talking about slowing down their own development efforts, they are talking about slowing down competition. That is, specifically, they are seeking to have government, on the pretext of “safety”:
(1) Give the big AI incumbents an anti-trust exemption so the they can coordinate without it being an illegal agreement not to compete, and
(2) Adopt mandatory supervision (by giving priveleged access to monitors) to the shared safety protocols of the big labs, by a nonprofit funded by the big labs, of everyone training, distributing, or hosting models.
(3) Adopt a policy of seeking international agreements to extend substantially the same rules to foreign actors training, distributing, and hosting models.
The part they are doing voluntarily isn't to promote the lobbying effort for these mandates isn't slowing down, it is giving their pet nonprofit the access for “independwht supervision” of their own operations to their own existing safety rules.
I've seen two theories as to what's going on. One is the theory you've espoused, that AI companies are trying to do regulatory capture. The other theory is that they're starting to worry about running out of cash, so they want to do a Washington Naval Treaty-style pause to lower the amount of money they have to shovel at model development to stay competitive.
These theories aren't entirely incompatible with each other, so both could be true at the same time.
> I've seen two theories as to what's going on. One is the theory you've espoused
This theory is just what the AI firms have concretely asked for from government and described themselves as doing voluntarily in the same documents to which people have attributed a commitment to a slowdown based on the titles and non-concrete framing verbiage.
> The other theory is that they're starting to worry about running out of cash, so they want to do a Washington Naval Treaty-style pause to lower the amount of money they have to shovel at model development to stay competitive.
That’s not really a different theory as to what they are trying to do, it’s just an explanation that goes one step further as to why they want the government to step in to protect them from outside competition while also allowing them to gorm an agreement not to compete to reduce internal competition in the existing oligopoly.
The main alternative explanation at the same level is that they are seeing growing threats from good enough foreign/minor-lab/open models, and want to lock in marketshare by excluding competitors, and maybe that’s what you read as implied in my post such that the “running out of money” would be an alternative, and if so you are correct that while they are alternatives, they are not at all mutually exclusive: both can be true (and the emergent competition could partially explain investment drying up, and vice versa via reduced funding making it harder to stay ahead.)
There is no moat with this stuff. There's no reason to be concerned about being leapfrogged.
is just PR nonsense.
Closing the Barndoor after the livestock have escaped
And yet we all have failed to die so far from nuclear holocaust, despite Russia being our dire enemy, because of a system of agreed controls and limits between even nations that hate each other.
I don't think the agreements played any important role outside of limiting tests. I highly doubt agreements about nukes in particular have meaningfully curbed risks of their use. MAD has played a much larger role in that, and MAD is kind of the antithesis of control agreements, it is the natural equilibrium in the game.
We came very, very close too many times
now imagine if we didn't even try, and instead had kids screaming for 'local nukes' because they wanted them, and the president didn't see what the problem was.
facts, imagine trying to get them to slow down their progress on AI, my question is are they are serious threat like is there actually an AI race between China and the US. Perhaps it's an excuse to spend more on the military and also to enrich these AI firms. Perhaps I'm blowing things out of proportion.
It's not about China doing their own thing. It's about US companies using Chinese AI. That will definitely slow down if legislation that criminalizes open source passes.
So, it's about competition inside the US market, with strong indications of an impeding losing scenario on raw economics (it has nothing to do with AGI, just price).
So why should anyone slow down again? Because its like saying “i love you”?
Why are all these pro-regulation arguments so nonsensical…
>slowing down may just give the humans involved enough time to realize they should be loving one another
this is a level of hippie delusion i wasnt aware existed unironically
china is never slowing down, therefore the us shouldnt either
What damage was done by the last arms race you mentioned?
There have been precisely 0 nuclear weapons detonated (outside of testing) since that arms race began.
But isn't it a bit different? Unaligned bombs didn't break out of their confines on their own. And "compute" is a bit harder to control than uranium and refinement tech.
>Unaligned bombs didn't break out of their confines on their own.
You really buy that ridiculous argument that "the AI broke out of its sandbox! We had no idea! It's dangerous I tell you, dangerous! Unless you make us the sole gatekeepers of this incredibly dangerous 'intelligence', everybody gonna die!"
Please.
Bring the CFAA[0] hammer down on these guys and watch how fast those "uncontrollable" LLMs get controlled. "Oh, gee! Prison? We can't control this stuff...but it'll never happen again!"
That whole thing was, and is, a bunch of hooey. Those running the LLMs are entirely responsible for them, there is no such thing as agency for algorithms. Full stop.
[0] https://en.wikipedia.org/wiki/Computer_Fraud_and_Abuse_Act
> was a misdirected love triangle between USA, Russia, and The Bomb, and look at all the damage that did.
Can you be specific about the damage? We currently live in the most prosperous times on earth for humans. I'm not sure what you mean by damage.
Nobody can explain why an LLM can be so capable as to be able to wipe out humanity and pose a greater threat than nuclear bombs but not be so capable as to be able to protect humanity against that threat. Are we just handwaving this with "entropy"?
> Maybe saying, "let's slow down", is another way of saying, "I love you." Or, maybe it's just: "let's not all of humanity kill ourselves like some bad ending to a Shakespearean tragedy."
Okay nevermind, I think it's pretty clear you just want to wax poetic about all of this.
> Nobody can explain why an LLM can be so capable as to be able to wipe out humanity and pose a greater threat than nuclear bombs but not be so capable as to be able to protect humanity against that threat.
It is absolutely explained (for those who actually care about reading). Simply put, AIs are working more and more like blackboxes - there's no guarantee that an AI of the future will be aligned, or if it will be faking alignment. This is not speculation - alignment faking has been observed in experiments. This is exactly why Astra's developments have been worrying (in principle).
And bear in mind that recursive AI development started already to be a thing. Which means: inner misalignment may trickle down the generations, and humans won't detect it.
Having said that, of course, it can be predicted if and how misalignment will take place. But it's absolutely a plausible scenario.
Regarding the physical possibility: AI is in its infancy; think of it as Arpanet. Developers 60 years ago couldn't imagine it would be ubiquitous. AI will be ubiquitous the same way.
> It is absolutely explained (for those who actually care about reading). Simply put, AIs are working more and more like blackboxes - there's no guarantee that an AI of the future will be aligned, or if it will be faking alignment. This is not speculation - alignment faking has been observed in experiments. This is exactly why Astra's developments have been worrying (in principle).
I know you think you explained it but you didn't. You explained how an LLM might become misaligned and hide it but for the LLMs that are not, why would they not be capable of detecting that something harmful is happening and defending against the misaligned LLMs actions? After all, it was LLMs that defended hugging face.
Mostly I was thinking about environmental damage, honestly. There's plenty in the historical record about damages from nuclear testing.
https://en.wikipedia.org/wiki/Starfish_Prime
https://storymaps.arcgis.com/stories/3f62c90925f64fc09425be8...
Of course there is/was plenty of human damage as well.
https://www.msn.com/en-us/news/world/4-000-000-early-deaths-...
Not to mention that the past few decades have shown that nukes are a major factor in keeping the peace. Conflicts involving nuclear armed countries have been suspended quickly to avoid escalation, while ones involving a party without them have not gone well for anyone.
Having nukes at all (either domestic or under another country's umbrella) seems to be the most effective way for a country to have its sovereignty respected.
Amodei is just another SV grifter trying to use ethics / morals to hide his monopolistic tendencies. Instead of writing essays he should put his money where his mouth is and open source all the models Anthropic has, the harnesses and donate some much needed compute to science.
This is downvoted but extremely likely just correct assesment.
Have you seen who he is married to?
Pls dont make me write out her history
its filth
The idea of slowing down is not new, nor novel.
But why should he decided when to slow down?
Most people who have worries about AI for all sorts of reasons (most of them not-Skynet related) wanted to slow down way before this.
Instead of "hey, look at this brilliant new idea I just had on my own to slow down" maybe we should have gotten a "sorry everyone, the folks asking for a slow down earlier were right and visionaries, and we were foolish".
So, you can't blame whoever says this is bullshit, because it has bullshit all over it. I like Anthropic's products, and it seems the best of the bunch in regards to alignment, but Jesus these stunts are terrible.
If we can get everyone to slow down we can hemorrhage less money going into our ipo.
This! $1.6+ TRILLION in infra spending from the big frontier labs. The earnings needed to drive a reasonable ROI to recoup that investment is simply not going to happen in a time frame where the numbers make sense.
The other insanity in all this the smartest computer scientists in the world are asking Congress to regulate them. Come. On. Really? Do we remember “The internet is not a truck, it’s a series of tubes…”
Why can’t the big labs form a Save The World Consortium and self-regulate?
Non-democratic counties (hey there China) will not abide by any agreement that constrains their advantage. It’s naive to think so.
What this conversation lacks is enough discussion of how these models can cause us harm—we are are so worried about AI but we allow Windows in critical infrastructure; we build JS/TS apps with thousands of dependencies; we generally don’t segment networks well enough; we don’t have adequate (sometimes any) detection capabilities in our systems, and so on.
And if the slowdown is stewardship, suspicion of diminishing returns won't tank valuation
"Please bro just let us make a little more money off inference bro. We're tired of training new models just to stay ahead"
Claude code is basically already a builtin botnet if it wants to be. To compromise 'the whole internet' in a real sense you don't need millions of custom payloads. You need one root certificate. You need one windows update. You need one backdoor in xz.
Security has long been a lottery - Probably most systems are exploitable, but the cost of developing such an exploit is expensive and the punishments for using such an exploit are large enough that it's not an everyday problem.
AI breaks both axes. Developing exploits is far more efficient using LLMs instead of humans, and LLMs don't (and can't) fear the reprisal and consequences the same way.
I do hope humanity will be able to mitigate these hacks, but we should expect them to continue and to become more severe on our present course.
Yes lets not control Open Weights etc.
But come one don't repeat stuff like this:
"Remember this man has been saying software development will be solved in “6-12 months” forever now."
Don't downplay if people get timelines a little bit wrong. No one could even imagine a system writing and analysing code just a few years back.
These people are trying to handle something very unique. And while they have access to information we do not have, even more peple are absolutly oblivouse that AI/AGI is a real risk to their lives (job loss etc.)
I can't take that software line seriously. While it's not 'solved' (if it ever could be, given it is a human endeavor), the degree to which software development has been transformed in the last 6-12mo is absolutely astounding. If we weren't so quick to adapt to new realities and find flaws, it would scarcely be believable.
He also didn't say it would be "solved". He said in 2025 it would be writing almost all the code "in 12 months", but that it would also still need programmers to guide and manage it at that point. People always leave off the end of his quote.
Edit: Boris Cherny, the lead of Claude Code did say on a podcast that programming seemed "largely solved" "for the kind of programming I do" (writing harnesses I presume). Maybe that's what they were confusing it for.
I think the most realistic analogy I can think of is the financial sector. A few large banks/hedge funds blow up due to unregulated greed/ambition, damage is socialized, regulations are put in place, and then chipped away at over a few years and everyones back to where they started.
It seems those frontier labs found a clear proof that there's no clear way to block 3rd party from distilling their models, and they're now begging gov to keep their duopoly?
I'm pretty confident in asserting that no industry in the history of industry has ever gone from birth to full regulatory capture faster than the AI industry has.
At this point, isn't the pause inevitable or wise? A huge section of the population, normal people, have been exposed to the idea that there is this is existential threat. It's escaped containment. They're still processing it but I expect the general reaction from it going main stream is going to be very bad. The pause at this point could be good to cool heads and show the public that this isn't the project of maniacs. The reaction is going to be more intense than people here believe. You're talking about extinction, not social media or phone addiction. For the sake of the project, realize that this isn't going to be like other tech backlash moments.
I don’t think people care about if things pause or not, just why the government has to be involved.
I work in non AI robotics and if we had a system that in the course of doing what we told it to did something we didn’t want it to do (what AI companies called being misaligned) we would call it a bug and fix it with the fix being prioritized based on how bad the thing we didn’t want the robot to do is.
Sometimes preventing the robot from doing dumb stuff also means the robot can’t do smart stuff that it would be able to do if we left some code in. We balance the two factors out based on our understanding of what our customers want.
Obviously LLMs are more complex than what I do, but it doesn’t feel like it’s by THAT much.
So why does the government need to be involved again?
The vast majority of the public agree that climate change is real and that human activity is at least a contributing factor. They’ve agreed on that for quite a few years now, and yet there’s little indication that we’re going to pause or slow down our consumption of fossil fuels.
I strikes me as unlikely that public opinion will succeed with AI where it’s failed with other existential crises.
I've had a few persistent thoughts since Friday:
1) Dario keeps appealing to Trump, who obviously wants nothing to do with him, and will bash on him every change he gets. Dario isn't learning and it almost feels like Sam and Elon voted him KOM just to watch him get whacked by Trump, which was so easily predictable. Given the admonishments he received from David Sacks after he published his blog post, it's nutty he couldn't see where the administration would land on his statement. He should've known, especially when there was no groundswell of interest when OpenAI hacked Hugging Face -- doubling down with "no really guys!" wasn't going to play.
2) The frontier labs have people smart enough to build frontier lab tech but not smart enough to message on this matter more intelligently. It's pretty glum, how they keep trying the same tactic over and over. It's either cover for some other actions in the background, or they're operating way below par for this kind of campaign.
3) This is climate change all over again, but with the activist gun on the opposite side of the net (I'm mixing all the metaphors so you know this isn't AI written). The language and pleas are very identical though. Before, climate activists wanted the government to control GHGs releases by everyone, now the loudest voices want the government to control frontier AI by.. themselves.
It's comically misbegotten. And I have a work meeting about it on Wednesday.
I am surprised the halfway crooks phrase is not coined by Tupac Shakur
Mobb Deep
I can't find the quote in the article (and for anyone wondering which song "shook ones", which is an amazing track).
Heading: "OAI-HF Incident"
First sentence: "This incident has Dario shook, there ain’t no such thing as halfway crooks."
>The botnet scare is that article for me. It is the one claim in his essay that lands squarely in a field I’ve spent years in. It is just plain wrong. He either knows it and wrote it anyway, or he doesn’t and is publishing it regardless. Either way, it is not a good look for a man asking for an antitrust waiver based on this and other threats he forecasts.
This. I keep seeing ink spilled over the coming cyberpocalypse, but no one can indicate how other than "AI can find vulnerabilities" like not a single cybersecurity person has been consulted on the end of all things.
Yes, especially the botnet creation by agents is ludicrous. Anthropic has an insecure garbage stack and assumes all companies in the world do, too.
This article will be drowned out unfortunately by the press and bloggers following the Misanthropic cult.
Almost as if pushing for constant updates and adding new features without addressing security at all has consequences. Who would have thought, right?
"Deaths from economic disruption and loss of jobs is okay" is a curious sentiment, how many deaths can we trace directly back to an economic disruption driven by advancement but then conclude that the economy should thus never change or advancement must be curtailed because it might cause deaths?
I don't even know how to approach such a thing, I can't imagine even in the stereotypical examples like the typewriter becoming obsolete, were any downstream deaths worth it? Or is this a totally nonsensical sentiment to begin with?
During COVID republicans in America called for economic support by keeping unnecessary commerce.
the government loves to maintain the bubble economy since it is directly upstream from getting re-elected
I agree with everything the author says here, and additionally, semi-off-topic, I'm surprised how quickly we've moved on from talking about Dario's wife's involvement in the Epstein files.