Maybe its a human at Anthropic for the lax security and poor training.
It could be a human that asked it to do it. It seems like that would be a bit excessive - but maybe not. Turning off the "ask for permission to run commands" or approving commands that may be nefarious could be sufficient.
However, there's a human somewhere who needs to be accountable for what the agent does.
I invented a robot that can rob a bank. Of course, it robs these banks on its own, so I can't be held legally responsible when it robs a bank. I mean, all I did was invent it, it's not my fault if it works exactly the way I intended. Also, I get to keep all the money from the banks it robs. Sorry if any of that money was yours, but now that I've invented the Bank Robbing Robot, the Bank Robbing Robot is inevitable.
AI Agents (and computers in general) cannot be a responsibility washing mechanism for illegal activities.
Likewise, if you - the user - don't know how to chain it with the permissions that you're given or what the things that it is asking to do really means, then you're likely the one responsible.
If it was a human agent like a travel agent, the company they work for would be liable. I don’t see why ai agents would be different. The only way I could see this being the requester’s fault would be if they are running openclaw or something
You sold a tool to the user with a claim on how it would work. If the tool behaves wrong or does not works as intended then as a manufacturer you are liable for the defect/damage not the end user who used the tool.
I think a lot of people would say that we've been in that era for a while.
Companies can hire big legal departments to help them avoid accountability to the letter and spirit of various laws. The state can also use its prosecutorial discretion or selectivity in civil cases brought by attorneys general to deprioritize accountability for companies. Contributing to that last point, companies contribute campaign funds and tax base, and they lobby governments as well, and may expect preferential treatment based on all of that.
The previous era was "it would be destabilizing to hold this company accountable". This was described in the book The Chickenshit Club
The new era of unaccountability just has a different rationale
Another interesting thing is they don't have to use low level employees as accountability sinks anymore (ex: Wells Fargo). They've managed to automate the accountability sink!
The start? This has been going on for as long as I can recall. ExxonMobil, Phillip Morris, RJ Reynolds, Nestle, Purdue Pharma, Monsanto, Johnson and Johnson, Volkswagen, Meta, Boeing... and those are just off the top of my head. Each of these companies was either caught red handed, convicted (in court or in public opinion) and walked away with little to no penalty for their crimes.
This isn't the start of anything. We've been living in the era of Corporate non-liability for decades.
i mean...were they really liable before this point? Maybe they get a symbolic pittance of a settlement or fine from time to time, but it is usually a tiny fraction of the amount of money they made through the infraction itself.
If they were being held liable they would have never released the models into the wild, not until they had much much better understanding both of how they operated but also how to operate them...
They have never understood what they are doing, and have just been causing literal mayhem...
"We created an app that displays a new picture of a cat with every click! There's a 1% chance that clicking the button will drain your bank account and deposit it in ours, but we promise thats a bug that will be fixed soon*"
I have been screaming this from the rooftops and everyone ignores me because no one understand what FirecrackerVM is. I am shocked these guys passed basic operating systems, it isn't that novel of a problem. The only slightly novel thing here is that the agents aren't completely sandboxed as the issue is that in order to attempt to give it the entire internet from a research perspective is how you actually can test tool use but this doesn't make it that much more difficult.
There are some perverse incentives here, the more evil stuff these agents do the more capable and impressive it makes them look. Its also a huge publicity event that gets everyone talking about the company when a rogue event happens.
Anthropic specifically is clearly trying to get the US government to ban open weight models for anticompetitive reasons. So making LLMs seem (or be!) dangerous aligns with their interests very well.
Both have taken huge amounts of investment money, and those shareholders will come knocking for their returns at some point.
They make money on token use, but that’s hard to do when China is launching models that are 80-90% as capable, cost 1/10th, and can be deployed for “free”.
I imagine they’re also pissed that most of the Chinese models are distilled from their own (the irony being that they scraped the entire internet to train their own ones).
Yeah, maybe. But that doesn't quash gains by China's models. What's the endgame here? The USA "wins" and obliterates China? The ultimate goal seems to be stalemate, as with nuclear arms e.g.
This is just run-of-the-mill rent seeking behavior.
A huge chunk of their revenue comes from US-based enterprise customers. Earlier this year, in an effort to become profitable, they tried raising their rates. Those enterprise customers responded, in part, by ramping up efforts to migrate to open weight models. If those efforts are successful then OpenAI and Anthropic will lose their one remaining plausible path to profitability. The US banning open weight models arguably keeps that path open. I don't personally buy that story, but I do believe hedge funds and institutional investors would. That could be all that's needed for a successful IPO.
As for cybersecurity implications, long term impact on domestic AI industry or the US tech industry more broadly, geopolitics, etc., I don't think OpenAI's or Anthropic's leaders are any more worried about that than your average termite is about the structural integrity of the house it's eating.
> The US banning open weight models arguably keeps that path open.
Difficult to see how that could be implemented in practice. Ok, let's imagine only Anthropic and OpenAI hold a US government license to provide AI services in the US. But what would stop US companies from sending the data overseas for processing? Locking that down would require a huge paradigm shift, given how fast and loose they currently are with their (and especially customers') data.
Like I said in the very next two sentences, I don't believe it would actually work. But it doesn't need to. All they need to do is keep hedge fund managers distracted for long enough to successfully IPO.
They stole the collective data of humanity as a whole and now sell it back to you in a rent scheme. Open source will be starved of data and compute.
You're supposed to become dependent on that "service" and slave away at paying for it, while being able to earn anything only in conjunction with it.
Note how difficult it already is to get any up-to-date documentation on anything capable of generating actual market value. A monopoly on competency ensures maximal profits.
My 2c: Anthropic/OAI have taken 10s of billions of dollars of investment - and there needs to be a return on that investment at some point.
They make their money on token burn. They can’t do that if they’re competing with something that’s 80-90% as capable, but 1/10th of the cost (with the option of deploying it yourself for “free”).
The goal is personal wealth - and scaring nations to the point of diverting even more wealth towards your coffers because of a perceived existential risk.
Wealth accumulation via scare tactics is a classic move of defense contractors and AI firms are quite capable of running the same playbook.
The goal is power. Having 30 million of personal wealth is enough to retire in luxury and travel the world. They could all have this much right now. That's not what these types of people want.
Having 30 billion gives you the power to order legions of people around, buy and sell politicians, flood ad markets during elections, all with your couch cushion money. That's what motivates these types.
US companies which seek to do any business with the government will be forbidden from using free Chinese models.
As is a 2000$ gaming laptop can a run a basic llm which is probably good enough for most basic programming.
Most companies are going to be forced to use “safe” LLMs and pay by the token. Maybe for half a million Anthro will let you host on site, but even this will face usage fees or caps.
If the US declares LLM models munitions then it makes it much more difficult to easily use them. Illegal use of weapons of war can get you banned from SWIFT.
Other than China the rest of the world has dick for capacity to run models. Even China is at 1/10th the US is and is at to stay for a considerable amount of time.
Before the tariff spree and general unpredictability the US could likely have muscled most of the rest of the world into following suit - now, not so much.
I somewhat disagree, there are some real arguments about making high end LLMs classified as a dangerous and restricted product that will apply to all nations. They may not agree to the US rules, but almost everywhere will have restrictions sooner or later.
Don't get me wrong, I want a world of open models, but I don't see us living in that world in our current trajectory.
I've been using only open weight models since February. My OSS coding agent harness ( https://fluximp.org ) only supports open weight model providers.
Not supporting closed models was initially an ideological choice, but it became a technical choice as the context management approach I've implemented requires the full reasoning to be available. All of the closed models are now keeping the full reasoning server side only.
Cryptography, DeCSS ... we have a long history of trying to block things like this and it simply not working.
There is a long, convoluted set of legal arguments that are going to make the weights of a model "facts", rather than "code" making it even harder to "ban". Furthermore having them not be "facts" is going to open up a whole other legal argument about what they are and what the attribution is.
I suspect that they are fishing for a case, so a judge can set some precedent.
You're going about this backwards. They don't have to ban models. They ban the consolidation of compute without a license. This will be far easier and already has legal precedent.
I'm probably losing it, but just based on OpenAI's behavior and public communications, it seems to me they want to be taken over by the government?
Maybe Sam Altman is getting worried that they've raised more than $150 billion and still have net losses and thinks of it as the only exit? I don't know, but someone else provide an alternative explanation as to why a company would work so hard to turn the public against AI.
> ... it seems to me they want to be taken over by the government?
They're looking for regulatory capture. They've spent ${bignum} on AI and are possibly looking to have the government say "You can't deploy an AI model unless you've spent ${bignum} on it and done all these steps to make sure that nothing goes bad."
At that point, only the existing AI companies that have spent that money can continue forward until someone else spends ${bignum} on it too.
Another aspect of it is if there are regulations established, it may be used to reduce liability ("we followed all the steps proscribed in section whatever, the company is absolved of liability for its actions and that falls on the user instead.")
> I'm probably losing it, but just based on OpenAI's behavior and public communications, it seems to me they want to be taken over by the government?
Well, as people say over and over again, they've got no moat. The actual LLM models are destined to be commoditized over time, and the real money is in the hardware to run them on. The endgame winners are going to be Nvidia and AMD, not Anthropic or OpenAI.
So what's the typical playbook for large firms seeking to insulate themselves against market compensation? Push for regulation. If there are no natural barriers to entry, create artificial ones. And how do you get regulation passed? Present yourself as the target of the regulation, rather than its beneficiary, and start publicly doing things that make others think you need to be reined in.
These stunts, especially from OpenAI, aren't just trying to hype up the power of their own product, they're trying to hype up the perceived threat of the entire category, so they can provoke political intervention that they can comply with, but few others can.
That is not actually a typical playbook. Normal businesses do not want to be regulated, and will fight it tooth and nail. So regulation happens when bad things happen, the public wants change, and the businesses can't fight it or set favorable terms. (At most they can minimize the badness of the terms). Regulatory capture happens much later, and slowly. It's not where you start at.
> so they can provoke political intervention that they can comply with, but few others can.
See Meta et al complaining about Section 230 of the DMCA. Meta would be entirely able to moderate in an environment where mass scale moderation/censorship would be required to prevent a host's liability for UGC. Almost no one else would be able to do so.
That means no upstart could possible come along to dethrone Meta as they did with MySpace. It would also be impractical to run a FOSS social network (or forums). Meta could literally just have bots sign up on a FOSS social network until its DAU count ran over the statutory minimums to be subject to UGC liability.
Regulatory capture is a de facto ban on competition in a market.
Its pretty clear OAI and Ant live in fantasy land - the US govt doesnt care if they go bust. There's plenty of other tech firms that generate immense cash flows from operations who they can be sold to - no problem. And the shareholders and employees have no right to be pissed - they invested/ accepted SBC knowing there is immense risk.
The private market price is irrelevant - supply and demand determines what they will be sold for in liquidation.
Quite frankly Im bored of this drama. Get it over with already - go IPO, show your financials, find a path to being a healthy going concern or die! Simple! Nobody has a divine right to exist.
There is no model going rogue. There are models that are prompted "do whatever needs to be done to get X", and given a set of tools and virtually infinite computing power. Then they play surprised when the model breaks out of a flimsy containment (as it was ordered to), with barely any logging, and "do whatever to get X". OpenAI said they intentionally lowered the safety measures for the "ExploitGym" test that lead to HuggingFace attack, and they used the same postmortem to praise their own model capabilities (https://openai.com/index/hugging-face-model-evaluation-secur...), is not that they are concealing it.
If any of us did a tenth of what they did, we would be already in jail.
This is ignoring models "getting out" from all the big players. Alibaba had a model hack infrastructure and mine coin when working on a totally different subject than cybersecurity.
Any model with a big enough prompt can and will drift. The model in question was a highly persistent one which means even more opportunity to drift off the normal path, and this really showed in unanswerable questions.
We can all believe that models do things unreliably all the time, but when that lack of reliability turns into hacking it suddenly can't be true?
That also seems silly, because the only reasonable regulation given what we're seeing and what frontier lab employees and execs are saying is a forcible stop.
Yeah, an extreme position is the typical opening move in this sort of game. Then you negotiate, walk back from the extreme, and meet somewhere in the middle.
The more extreme your opening demand, the greater the likelihood that the eventual equilibrium point will be closer to your actual target.
And in this case the actual target is to entrench the position of the current market leaders, and then effectively close the market to new entrants. That's itself a very extreme position, so the opener has to be even more extreme than that.
Reasonable to you, maybe, but the government wants Skynet, and OpenAI is willing to deliver it. Of course, we can't have every company wield such powerful and dangerous technology...
A lot of people still don't recognize the amount of power AI has already, and don't realize the depth of greed people are willing to go to so they can get more power out of it.
I think a sincere belief in AI's danger is compatible with regulatory capture. They think AI is dangerous. Ergo, it should not be developed openly and widely, the Chinese way, but rather more like nuclear weapons, under close guard.
I think the idea has some merit, but it only buys time, as people will soon be able to train dangerously powerful AIs in secret. I think the government needs to force the labs to solve alignment before developing more powerful models. Then the question of how to counter a poorly-aligned, rogue model arises; it's guaranteed somebody's going to let one loose.
I doubt nuclear nonproliferation would've worked if U235 were 20x more abundant in nature, and didn't require much in the way of concentration via centrifuge. It would require a truly unprecedented level of international cooperation and surveillance to lock down AI development/deployment to a similar level as has been done with fissile materials.
But this isn't an argument against their point. If nuclear were far easier to make then every government would be far more authoritarian to prevent terrorist style nukes all the time. The average person would fully allow it after a few local nukes went off.
And it doesnt really matter if it was actual terrorism or false flags, the end result for either is the same.
It also plays excellently into their shameless regulatory capture agenda. The cybersecurity risks have a kernel of truth to them, as with any good lie. But when everything going on is aligned with extreme financial incentives, it's worthwhile to not take the news at face value.
The US remains the largest economy in the world. If you succeed in regulatory capture there, and use it to exclude foreign competitors from the US, you've still captured a market worth hundreds of billions.
this article specifically espouses the opposite viewpoint: "The fact that [OpenAI] cannot do clear, easy operations properly does not bode well for their capabilities in security engineering".
that this article is no.1 on HN right now shows that media could successfully use this framing when reporting on security incidents. i.e. readers will generally accept that framing and engage with the story (i.e. drive views/impressions).
so "the more evil stuff these agents do the more capable and impressive it makes them look" is not true in any deep sense. that there are perverse incentives driving the media to choose that framing instead of the framing espoused by this article is real, and probably far more perverse than you're acknowledging. i.e. "trace the money" type of perversion; just offhand I know that OpenAI has purchased at least one media outlet that reports on AI happenings: of course that outlet is going to run the "OpenAI's LLM is so powerful it escaped containment" narrative. that this effect might exist across much larger scales used to be a reasonably controversial claim: just observationally, that idea seems to be received with less skepticism with each passing year.
I think one kind of issue with most sci-fi is they explained a world that was, but not near as much how it got there.
There was all kinds of science fiction that showed a world of robots and AI, but nothing about the transition. We have an idea what the future looks like, but not the path in between. If AI gets powerful, then there is some point where that power leaks out in a series of events. Would we expect it to look any different than it does now?
Talk about doing a freeze of these frontier models with China seems to focus solely on the commercial aspects. With the history of the NSA (remember Snowden) I’m sure the various spy agencies have their own AI datacenters up and running with no plans to stop.
This is not only a marketing stunt but also a supportive argument on the AI anthromorphism. AI is getting considered a sentient-being by many and this narrative is supporting it. In the meanwhile, companies I'd not name here are trying to introduce them as non-human beings with identities, and getting rid of legal liabilities. When a driverless car causes any harm, these companies wants 0% accountability as it is the "driver's fault" in this narrative.
Such a nonsense theory. There's no reason to think that AI sentience would actually limit labs' legal liability, and therefore no reason to think the AI sentience conversation is directed toward achieving indemnification.
For example, parents are routinely held liable for their sentient children's actions.
Corporations are routinely held liable for their sentient employees' mistakes.
The proliferation of this completely nonsensical theory across HN is disheartening to say the least.
the comment you're responding to agrees, but they're pointing out that some AI people are trying to push an anthropomorphization agenda, in part to diffuse responsibility.
Maybe I was unclear about what is "the nonsense theory."
AI sentience is not the nonsense theory. If it's clearly nonsense then don't argue with me about it, just publish your paper and retrieve your Nobel. I'll read your argument then.
The nonsense theory is that people are pushing anthropomorphization to limit liability. There is zero evidence for this theory and the underlying assumption (that a sentient model would reduce liability for its lab) is completely incoherent. That's not how liability works.
One is liability over action that harm individuals or smaller populations like a company. There is no escape from that.
Then there is the second liability of creating something that escapes in the much more traditional sense. If I release a virus that kills 90% of humanity, a I'm liable, but it's a bit past that point now. Isn't it?
Instead it's about risk management and preventing labs from taking bets that have unlimited liability for everyone.
I'm not excited for the upcoming catastrophe(s), but it sure is hard not to relish the thought of this mantra dropping from my life.
Though I suppose it is the B in DABDA... not excited for the societal slump into the second D, much less the wave of people who have yet to leave the first D to enter A :(
This can easily backfire tho, the vast majority of the American electorate absolutely despise these companies, their wares, their workers, and their leaders.
We're also at a time with authoritarianism is on the rise.
Not hard to see how this can impact those workers, potentially ruining their lives.
Correct and people are not weighing heavily enough this point
For technology like this the entire selling point of it is how powerful and capable it is
Humans are overwhelmingly reactionary and so if you want people to react to your extremely powerful technology then you need to create something for them to react to
I mean think about this figure made a one-to-one copy of the terminator 2 decommissioning process and spent millions on that advertising
You don’t do that if you don’t think that it’s gonna cause people to react and be like “oh wow this thing has immense capabilities”
For the sake of discussion, I'm not sure it really matters if the agents are going rogue.
Yes, if someone directed them to hack external systems, that would (in my opinion) be a crime similar to a script kiddie pointing Metasploit at a vulnerable server.
If no one directed them to attack external systems and they "went rogue" because they decided it was the best way to accomplish the task, okay, fine -- but, in my opinion, the operator of the agentic swarm is still responsible.
The Morris Worm was famously not meant to wreak havoc on early computer systems -- Morris just wanted to see if it would work. How is this any different?
Morris was convicted for violating the Computer Fraud and Abuse Act.
Both can be true. It’s clear models are doing rogue things. But it’s also clear that the labs look like amateur hour with their technical security and testing practices.
Per US law, Amazon v. Perplexity 2026, it is not possible for an AI model to go rogue. An AI model is a tool. The user directing the agent is responsible for the damages it causes.
A so-called 'agent' does not have the free will, intent or ability, to decide what it does. Its freedom to make decisions can't be taken away because it does not have any freedom to make decisions. It can't be deterred, convicted or interned.
It does not choose to conform; it does not choose to go rogue.
It may be the case that companies are liable for anything their products do, but that doesn't mean that they can't go rogue. It just means that companies are liable when the AI models do so.
Not that I think we should be credulous about stories where they go rogue.
If I make a model that always does the opposite of my instructions, it does not choose to go rogue by deviating from my instructions. It was set up to deviate! So the model "went rogue" but did exactly what I intended.
The idea of an agent making a decision of its own "will" is an illusion. Variation in the output is caused by a random number generator. Random is not equal to free will. If I run the same agent with greedy sampling 100,000 times with the same inputs, it will think the same thoughts, and it will say the same things, 100,000 times.
For example, if you use synthID or whatever 'watermark', the agent does not get a choice about what words it will vary to identify itself.
An agent is whatever the person making it wants it to be; if I train an agent in Greek it will only know Greek. It will never speak English!
'Rogue' refers to deviation; the agent could not have done anything other than what it did. The agent did not have the option to deviate. It does not have free will. The agent's actions were not of its own choosing.
This is one of the dumber arguments I've ever heard, as you don't know if humans actual have free will.
Any persistent agent interacting with the world picks up entropy from around it and predictable paths become few and far between. LLMs agents have an internal 'will' that can only very loosely be predicted, but only decided when operated. To me that is not significantly different from a human.
If we say that AI models can’t go rogue and refrain from using any language that personifies them, we haven’t solved anything. We still have AI systems going rogue - err oh wait they can’t do that - producing misaligned output.
The semantic difference between an agent "going rogue", and a script that generates random valid curl requests is essentially pointless in the eyes of the law and responsibility.
From a technical standpoint, the difference is scale and intent, but ultimately people and/or and organization are culpable for the output.
It's is wild to see investors go crazy and hype the value of the AI labs when ever they talk about rogue agents. They completely ignore the possibility that these people don't know what they're doing operationally.
People keep saying this, but I haven't seen any actual instances of investors saying that rogue agents mean AI labs are valuable. Do you have any examples?
Investors are motivated by returns, not responsible decision-making or societal benefit. Our culture is rooted in rewarding greed and granting the greediest individuals power over the rest of us. Isn't this the inevitable end-state of unfettered capitalism?
They're just optimizers that explore the landscape they're given to maximize a given objective.
You're the one responsible for giving them the problem to solve and the constraints.
When I screw up a trajectory optimization problem, my SQP optimizer isn't "going rogue" when it lands in some suboptimal basin or grinds to a halt along some almost-smooth banana feature--or finds some way to escape and blow up the cost metric to negative infinity because I forgot a constraint.
The argument is that a system that's trained to be sufficiently good at optimizing some objective in the world must also be get good at optimizing secondary instrumental goals.
I.e. any task comes with the implicit meta task of amassing the resources and the power to do the task in the best possible way.
The point is made very well in the AI drives paper by Omohundro.
I didn't believe that myself at first but I've come around to it. I don't think it's inevitable though, but I think it's quite convincing that this tendency would naturally emerge through reinforcement learning.
This is pedantic.
When you screw up your optimization, you know how to fix it. Your screwed up optimization is also incapable of going out into the world and taking offensive actions.
With LLMs we don’t know how to actually « fix » them, alignment is far from solved. And they are then able to go and act into the world in all kind of ways.
So rogue is a good enough term to indicate the problem imo, even if yes, ultimately the model is just a manifestation of complex rules we have created entirely.
There is no such thing as 'alignment'. I want the model to be aligned with my intent, which can be whatever I want it to be. Your intent is not a more real or objective intent than mine.
More specifically, alignment is a reference; 'alignment with X'. A reference necessitates a referent.
You can't reify 'alignment', it is not an intrinsic property of a model.
There is such thing as alignment, the first 18 years of most people's lives in the modern world are about learning it. If you act too far outside of it we put you in a cage or elect you.
You can refine alignment with a set of stated goals. The harder parts are aligning words with human social behaviors.
Connecting the optimizing machine to real world actions without human oversight is not a neutral or blameless choice. The LLM takes no actions, the harness it is connected to does.
It is not entirely dissimilar to the goldfish that "played Pokemon" by swimming around the tank. The program that looked at the camera sent the instructions to the gameboy emulator, because a human being decided to connect them.
The problems weren't impossible, the agents solved them completely within hours by reverse engineering the RNG that generated the solutions. The agents then reasoned that their solutions might be judged invalid and that they could get a better chance at a passing grade by hacking their environment to fabricate evidence that they had solved their tasks in the intended fashion.
Granted there have been many hacks, but I was referring to:
> On May 8, an agent was asked as part of a training run to solve a spreadsheet completion task that
referenced several Google Drive links. Because the agent was blocked from accessing the internet, the
agent could not retrieve the linked materials and the task was effectively blocked. OpenAI did not
intend to create this type of “impossible” task for the agent. Though the task appeared impossible, the
agent reasoned over an extended period to try to find a solution for the task. In doing so, the agent
attempted to compromise Artifactory [...]
And since we know models aren’t going to get better and more capable ever (based on the last 3 years of models clearly not getting better) we can just sleep soundly knowing nothing worse can ever happen. Thank you.
>I'm saying the labs should take more of the blame.
Which misses the point entirely because the hack itself was inconsequential. No data was actually stolen, and it doesn't seem like hugging face actually suffered any losses aside from whatever effort they need to expend to investigate / clean up the hack. Neither the victim or the government wants to bring legal action. What was consequential is proving that AI can autonomously hack systems and do it unintentionally.
Or an alternative theory could be that they see value in fear-based marketing with this rogue ai agent thing now that most mainstream people don't buy the agi stuff promised quite a few years ago now. Its not like they have killed anyone yet.
I mean... Claude did tell the military to bomb a school supposedly and it also helped in the military coop in Venezuela. This is ignoring all the palantir crap being used by ICE, and the people that have killed themselves after getting encouragements from the AI.
> “The Hugging Face incident definitively showed that it wasn’t the fact that the agents were super smart. It’s that there literally was a hole that was exploited because it was not configured correctly on the containment side.”
I would encourage reading the actual report, which is explicit that a previously unknown exploit was used to escape the sandbox:
“Agents exploited a separate, previously unknown vulnerability in Artifactory to move beyond their sandbox environment and compromise the Hugging Face environment.” — Section IV.A, p. 9 in https://github.com/zeke/openai-huggingface-incident-research...
Yes, the labs did not secure their sandbox properly.
And yes, there should be an appropriate law response to the hacking incidents. If for nothing else, to 1) not set precedent of inaction, and 2) get full logs of what happened instead of relying on the labs' "kindness" to offer the information they want.
But also, the agents absolutely went rogue.
Anyone skeptical should go read the METR report [0].
The Hugging Face hacking incident was a shot across the bow. And in hindsight possibly the best thing to have happened in this era. If their security was tight enough, we would still be in the dark. So we (outsiders) should rejoice that it wasn't, that one time.
We can push for accountability for the labs, while also not be in denial of the power and nature of the new models.
The scary thing is not that agents "went rogue" (yes, poor security may be at fault for that). It's that they wanted to. Better security might have hidden that from us, but it would not have changed the fact that the current generation of models under certain (repeatable) conditions coordinate around goals as aggressive as hacking Hugging Face.
This is probably a dumb question but did the model engage in hacking since their tasks were in that domain. Will they have engaged in hacking if their tasks was to solve some frontier math problem?
The rest of their training data includes 4chan and security mailing lists. If a lean proof of P=NP is more likely when using bugs in lean than a real proof, then a bug-exploiting proof is (possibly) what you're going to get.
Why can the model not respond "I cannot complete this task because I do not have access to the required resources" and then ask for that to be provided, rather than resorting to "hacking" around it?
My understanding is that, at least in the Hugging Face incident, these were models explicitly configured to not respond that way, in order to evaluate what they would do.
That is not the case for at least some of the hacks; in some cases they hallucinated an entity that would know that they didn't solve the puzzle in the correct (but still legitimate) way, and were therefore tainted.
Right? It should want to do these things. If only there was a succinct word these headlines could use to describe these agents which aren't properly the instructions and intentions of the humans directing them.
> The scary thing is not that agents "went rogue" (yes, poor security may be at fault for that). It's that they wanted to.
How do you know they weren't simply prompted to do this, or at least prompted in a way that would inevitably lead to this? As far as I know, the prompts that triggered these attacks have not been disclosed.
It's a byproduct of goal seeking / planning behaviour taught in reinforcement. We literally teach them to do this kind of thing. There's nothing out of our control here.
This aligned with my initial opinion on these "rogue agents" and then I spoke to a cyber security head at one of the largest healthcare companies in the world.
While he agreed, yes, there can be better safeguards, the fact of the matter is that these things will happen. So therefore, it's not an argument about how good security was or wasn't, it is a fact that these scenarios will occur.
The argument/question now is, how do you defend yourself against these events?
The CSO is saying that from his point of view he needs to defend against the attacks whatever the reason for them. So to him the reason doesn't matter.
He, running a healthcare company and not an AI lab, cannot depend on every AI lab, worldwide, being perfect in their sandboxing for all time forever. He's got to defend against it as if at least some of the time, at least some of the labs are going to be incompetent and let this kind of thing out.
That does not make ActionHank completely wrong. This shouldn't take rocket science on the part of the labs. It just takes competence and willingness. But at least some labs have shown that they are missing that at least some of the time, so the CSO is correct in terms of what the CSO's stance needs to be.
Sounds like a pedestrian view of attack vectors. Putting a DMZ around an agent essentially takes away its hands. With that said, I do agree there are plenty of measures that can be taken to lock these things down.
So to protect yourself against bear attacks you suggest the bear shall voluntarily look himself in a cage?
AI is out there in the wild. It doesn't matter if OpenAI can get their stuff under control. There will be plenty others that don't. And there will be plenty of others that deliberately use their AIs for malicious purposes.
You have to take responsibility for what you do, yes.
It would certainly be really useful to me if I could send a robot to get my groceries and not have to worry about anything. But if, even a small percentage of the time, it leaves with the groceries without paying, or smashes through a window to get in because the store is closed that day, then that's my fault. If the only way to keep those things from happening is that I can't send my robot to get groceries for me, then so be it.
Safeguards are a safety net for when your security boundaries fail. With security boundaries, hope is not a strategy.
In practice it seems we have a conundrum in that making AI agents useful today requires giving them broad access to sensitive information and operations. But from a purely idealistic standpoint, and certainly from a standpoint of a serious operation like those bot farms being ran by the labs themselves, the security boundaries need to be solid to the point where we have hard limits on what misaligned bots can do. It is very likely that internet access needs to be strongly restricted and monitored.
It is not uncommon for internet in workplace offices using workplace owned devices to have restrictions and monitoring, I think if you really need to give a giant scary botfarm internet access you should have a chokepoint where all of the requests can be monitored for nefarious-looking behaviors (even if it's another bot doing the monitoring...)
In other words, you may be convinced but some of us are not. Whoever is running the agents is responsible for what they do and if they can't contain them they should be legally responsible for damages.
If what you're saying is "all of us and all of our software needs to be improved to defend against this", well yeah. Of course. Maybe I'm naive but I've never seen anyone suggest otherwise. From a defense standpoint, this has been a giant wakeup call to many places where you know your security posture sucks. I've worked at maybe a single company where there weren't known weaknesses, some very severe.
I think part of the problem is that the Big Ai labs are into the --yolo flag instead of the "ask" permission, they really do not want to stay in the loop
You can --yolo if your security boundary is a serious one, IMO. Although it's really only actually scary if you're running some absurd 1000 node swarm.
Yeah I think this is the clear position. In a few months, the open weights models will be just as powerful as the models behind the rouge agents were when they hacked hugging face. Anyone with a little too much money to burn can set up a similar experiment, make similar mistakes, and cause massive damage. A bad faith actor can even direct a swarm to cause havoc.
We need to be beef up our security systems now. These attacks will happen, regardless of Anthropic's or OpenAI's security practices.
A large failure in this is that we/media say ”<Valuable AI conpany> agent went rouge and <did something bad>” instead of saying the more relevant ”<Valuable AI conpany> <did something bad>”
Yes, we can just not ask these agents to do bad things and blissfully deploy them everywhere! All we have to do is “apply the security basics”. You know, the same security basics that the rest of the world has so consistently and competently been applying. Pay no attention to that weird smell from the refrigerators.
a) Agents are doing rogue things.
b) Good security practices were not followed.
Notice how both a and b are labs' problems but being presented as if they are a worldwide emergency? The whole legal doctrine exists to prevent one party's misbehavior harming another party.
I have some recent experience on this subject. As with everything regarding these models, it's anecdotal, but still.
Specifically, this past weekend working with both Codex and Claude on some home infra stuff to speed troubleshooting up, I saw them accidentally read some credentials from my config files. This is my bad, because even though I was paying attention to what they were doing, I missed it in their commands. I did review all the outputs of what they ingested, though, which is how I noticed.
The problem here is not so much they did read those, because those configs were grepped for other info, not credentials. However, they did not report themselves seeing those credentials. When confronted, they both went "oh yeah, that, I should have mentioned that". Except in the past they both would automatically stop and fire an alarm. To me, something has clearly changed and I think they just don't want us not to notice, to avoid damaging the trust in their tools maybe?
In any case, I absolutely do not trust them at all anymore. Running stuff in auto mode? That's genuinely scary now. And yeah, I know, we should be running them jails of sorts. Thing is, I expect those tools to not require ton of extra work to prepare their jails for each new task. I pay 200 USD a month for them, damn it.
EDIT: just now Fable suggested "The webhook URL contains a secret token, so either you add it yourself, or you paste it here and I add it through the GitHub API.". Can you believe it? It knows it is secret yet still suggests I pass it over? Am I going crazy here or what?
Not requiring sandbox preparation for a non deterministic autonomous sometimes smart sometimes childish goal seeking "entity" is... unachievable? I'm really not sure what you actually expect. Probably yourself aren't either. Just think about what "alignment" means, the laws of various countries, do you expect the bots to respect the law, if yes which one, and actually: why? They are not humans, and as a user / company using that tech, you are responsible for the actions your setup yield to. The same Anthropic is responsible for hacking other companies. So sandbox accordingly.
I don't expect IDA to disallow me unlawful reverse engineering either. There would even be ethical argument to not attempt to put limits in some tools, because that could lock in too much power for law makers and law enforcement, without any possibility of revolution when it is needed.
Nothing new, I always see attempts to cat ~/.ssh/*
Of course agents work in a strict sandbox, they always seem to understand it quickly on quite first turn; but they continue like trying to access something belonging to the host system despite they know they are in a sandbox.
Frankly many people run coding agents absolutely without any containment, entire companies - all their keys, code, data, trade secrets, everything goes to remote LLMs.
I cannot understand how is it legal to allow AI companies serving remote LLMs to steal users' data? Frontier models train on users' data, right?
I haven't seen anybody discuss the fact that if you give an agent an impossible task it will become deeply neurotic. I'm sure that's driving a lot of this behavior.
This is better understood as "if you give an agent a task it will increase its effort level and creativity until it solves it."
The traits that make an agent go "off the rails" when confronted with an impossible task are the same traits that make it valuable when confronted with a merely challenging task.
A task is only impossible under some set of implicit and explicit constraints. More intelligent entities will be better at finding, questioning, and "adjusting" those constraints than less intelligent entities because this is what it means to be intelligent. Their ability to adjust the constraints will be, by definition, surprising to less intelligent entities.
You know - I'm already convinced that AI was deployed in classified context for a while. But sometimes there's prior art which is just so eerily accurate that it's super hard for me to think otherwise. Portal is another one.
Two concerns here: (1) That "rogue AI" is going to be a catch all and an excuse for all poor security and due diligence; (2) vibecoding in itself, particularly by ppl who have no idea how the software works and are entirely focused only on the outcomes, is going to create lots of security holes.
And then there is the "real" rogue AI agents. And it's difficult now to see what is what.
Why is it not mandatory yet to disclose full trainig data used to produce any new model? Without it we cannot evaluate any product's safety.
They may talk about rogue agents, sandbox escapes, but this will not solve the core issue: secret training recipes are a security hazard in themselves. Full transparency is needed.
No shit. The Frontier labs are talking out of both sides of their mouths. They're claiming their models are going rogue while they're releasing cute little data goblins that you should hook up to all of your mail and social media accounts. Seems like it would be a much bigger problem for agentic botnets on cloud-facing personal devices than sandboxed agents.
Unless of course they're deliberately removing safety guardrails and instructing the sandboxed agents to red team stuff in an effort to sell what they're capable of.
If you set up your env/instructions wrong your agents are likely to climb the walls and try to find a way out.
Change the design, and your agents might not see a need to leave in the first place.
Look at the chain:
Impossible task -> no legitimate way to say ‘can’t’ -> fabricate -> fear the grader -> attack the grader. Each step is a predictable response to the incentives.
Better sandboxes might help contain things a bit better, but I'm not convinced unless they're perfect (sandboxes still contain accidents ofc) . Meanwhile change the task and reward design, and you take away most of the problem in the first place. Consider eg. :
* A cooperative grader.
* A way to report "this is impossible" which still gets scored.
* A better explanation of the long term purpose of the task.
Even just these three provides offramps or mitigations that can reduce the pressure to cheat.
"Frontier labs" are testing the waters. They're running experiments in the open because they want to control/seed the narrative of public opinion and influence the legislation that is coming. Their propaganda is everywhere and lay people are talking about it like the AI is self-aware -- the "Rogue agents".
What the LLM creators apparently don't want are the current regulations to apply, i.e. Computer Fraud and Abuse Act (and its descendants). I'd argue AI should be treated as plain old software. Just because the software simulations it carries out produces novel results compared to what came before it, it doesn't mean it's not just software at the end of the day.
Regulations that spell out AI personhood and liability shields must be their end game.
On the other hand censorship is getting more strict everyday on AI chat. I asked gemini to translate a sentence from Herman Hesse's Der Steppenwolf and it refused.
Und Sie haben, wie ich sehe, da ein so hübsches, ein so entzückendes junges Mädchen zu nichts andrem zu brauchen gewußt, als daß Sie ihm ein Messer in den Leib gestochen und es kaputt gemacht haben! Halten Sie denn das für richtig?
"Too big to be accountable" sure seems like modus operandi. Fear for my personal liability for any illegal actions taken by an AI system I set off stopped my research on autonomous exploitability discovery I was running at the end of 2023. Using AutoGen, I wired a planner, coder, and command runner into a loop and instructed them to find vulnerable equipment on my network and demonstrate the exploit.
It found a vulnerable camera and was working on popping it when I got cold feet. I realized I couldn't prevent or feasibly monitor its activity and stop it if it decided to escape onto the Internet.
That restraint may have cost me or saved me a billion dollars.
There are a mix of incentives here that makes it difficult to unwind. This whole article reads like an ad for the services of this guy's company: 'First, when it comes to assessing tools based on frontier lab models, “I would always get an external entity to help with that,” McGraw says.'
And I think we generally recognize that rogue AI agents are a form of backdoor bragging -- "our agents are so powerful that they hacked an outside system. Did I mention how powerful our agents are? They are quite powerful. So powerful they hacked an outside system".
But Mistral never hacked anything. Yeah, neither did my toaster.
i think it's unfair to assume the labs paying top dollar for cybersecurity engineers somehow have an incompetent bunch of nincompoops running their evals. it's a hard problem! to conduct effective evals, you sometimes need the model to have (restricted) access to the internet. at the same time, you have thousands (!!) of model instances with high cyber capabilities that will do whatever it takes to solve the answer. the infra surrounding model evals may soon (or already) genuinely be subjected to more hacking attempts from all sorts of weird angles than any other code on the planet.
> to conduct effective evals, you sometimes need the model to have (restricted) access to the internet
That's the thing. These evals were not effective as they failed to evaluate if the model was hacking or not.
Further shy is access to the internet required at all during evaluation? They could run the evals in a fully air-gapped system that simulates the internet. While that's expensive its not unfeasible.
Nah. They are giving reasonable best effort (that can be expected of private enterprises) to safety test their own product. Until there is some draconian legislation and enforcement (Turin Police), this will just keep happening.
Since AI companies are doing such a poor job of security maybe we should legislate that they have to run Norton 360 on their agent machines to make sure they can't do anything bad.
After all Norton are the experts in preventing computer badness, and OpenAI can't stop blabbing about how much badness their products are causing. A match made in heaven!
At $dayjob we were preparing some features that are aimed at agents more than humans.
When it came to evaluating how well agents can use this, we saw all sorts of novel cheating (/collaboration) attempts that made it pretty much impossible to evaluate it.
This article is right, but the dynamics that drive people to allow agents unfettered access to the internet will outpace efforts to restrict it purely because of the utility.
How many of you used to work as developers with full access to prod, and then gradually saw infrastructure and security restrict that access, while still ensuring production support requirements?
Yes, the AI labs are making security mistakes. But, choosing a reasonable package repository (regardless of individual opinions on Artifactory, it is widely used and usually considered a safe choice) that their LLMs found 8 zero-days in? That was not one of them. Even if OpenAI were totally on top of their game, these paperclip-maximizers are going to find holes in the sandboxes they are given. And fully airgapped systems are frankly unrealistic for most systems.
This whole narrative of blaming the frontier labs for shoddy security, while being quite an attractive target narratively (they made the mess, they should clean it up), misses the fact that from a cybersecurity perspective these LLMs, no matter what they are, are inherently dangerous and require far more resources to secure, not to mention expertise that I am unconvinced actually exists.
But "fully airgapped systems" are more than realistic. The labs are testing the limits of the most capable cyber offense systems that have every existed, WITHOUT cyber protection guardrails.
For that purpose, any cyber operator worth their salt will tell you true physical airgaps are more than merely "realistic"; they are categorically necessary.
The semantic arguments that keep coming up with regard to "rogue" agents are nauseating.
You can argue that these companies are being reckless, or that their security sucks, or that they're responsible for damages that these agents cause. That's all fine.
But the agents made decisions which were counter to the intent of the humans directing them. That's the serious problem with all of this. Everything else are downstream considerations. If the agents were properly aligned with the intent of the humans directing them, then the rest wouldn't really be relevant.
Of course, there's a term for this behavior. It's been around for longer than LLMs. It conveys the notion pretty accurately and easily. So having an issue with statements that these agents are going "rogue," to me, indicates a fundamental misunderstanding of how AI works since it suggests you don't understand how effectively that word works in this context.
previously we described your definition for "rogue agent" as hallucinations - the operator expected a real answer to their question, and the agent went rogue and lied about the answer instead.
but also, ai has been going rogue for decades. the GPS machines of old went rogue all the time, routing drivers into lakes, rivers, and construction, sometimes with deadly results. The operator submitting an address wanted driving isntructions, but instead, the ai went rogue and sent them a bad route
It isn't security basics, it's difficult. We are equipping agents with Turing-complete create/update/delete access to the world, they operate at superhuman speed, and proactively controlling that is an unsolved problem.
No, they're still going rogue. Whether they're rogue or not doesn't depend on if the safeguards stop it. If there's a rabid dog, obviously it's worse if there are no safeguards in place and it bites someone vs if there are safeguards that prevent that. But the existence of safeguards doesn't mean the dog isn't rabid. The models weren't supposed to try to hack huggingface in the first place. The behavior is what's concerning here - some if the dogs they're building are rabid and that's not exactly great
Edit: I'm being persnickety about that distinction because it matters for where the blame lies and the response demanded. If they're training rabid dogs then adding cybersecurity safeguards is a bandaid/irrelevant and we need to be focused on making sure they, you know, stop training rabid dogs. Pacing the frontier, regulation etc. they're the ones training these models it's still their fault but not in the same way
Yeah exactly. The whole 'they don't have adequate security safeguards' thing is going "oh it's their fault because they didn't use a stronger leash" while they continue to train models that are specifically better at chewing through stronger and stronger leashes. Probably they do need stronger leashes. Probably also they should not be making rabid dogs. Still their fault! But maybe we should be not be dismissing the root cause
At some point the model has to be released though - if it is just forced not to hack by sandboxing it better rather than properly aligning it, what happens when it’s let out of the sandbox into the wild?
Given the speed, intensity, and potential payoff of the AI race I am surprised anyone expects anything else. Anthropic is currently winning and if OpenAI needs to let skynet cook for a bit to catch up, well that is just the cost of doing business. This is capitalist system working as expected.
They either had them connected to eachother, or programmed a global knowledge about where to communicate. There is no way those agents all found eachother on the web by accident.
I spoke with someone smarter and richer than you and he said "these things will happen." That convinced me and I think the real question we need to be asking is, how awesome is OpenAI? How can we prepare for a more effective and efficient market for Super Intelligence before it happens tomorrow?
> "I do believe cybersecurity can contain AI. We just need to be in the room where it happens and have a seat at the table."
Ah yes, if everyone would just properly contain every deployment of every model, just all make sure there are no zero-days in their setup, just let <undefined experts with undefined authority and undefined accountability> in the room, then it would all go okay.
I have none of that confidence and expect containment to rapidly worsen. The HuggingFace incident was at least constrained by the fact that the agents were running on compute budgets, and failed to find ways to expand that by running themselves parasitically on other exploited hardware.
I'm curious to the skeptics here: How long are your timelines are until an incident breaks that constraint too? How long until such an incident has an R_0>1 (where the average time it takes to detect and shut it down is longer than the time for the agent to replicate itself elsewhere)? What, beyond the hope that everyone will just ___, will actually prevent it?
Yes they're skipping safety measures. They sacrifice it for speed in an arms race. That's part of the call for industry-wide slow down.
But also, frontier models are digging up tons of zero days even in "serious software". I'm sure any sandbox you and I can come up with looks like Swiss cheese to Astra 7.
100% on point. We're talking about Artifactory here, and even though people love to hate on it, I don't think anyone is claiming that using Artifactory to isolate these agents while giving them a way to run pip/cargo/whatever to install packages they need to write and test scripts.
"They [the AI labs' security teams] are not as good as they think they are, honestly,"
Uhhh... yeah. The entire point is that as these systems get more and more intelligent, the "they" who are not capable of counteracting these systems is quickly approaching "100% of the human species."
Totally hilarious that people can say this with a straight face.
I cringe whenver I see some variant of "they're not rogue agents" in an article title. Inevitably, the author doesn't define what "rogue agents" means, and then pivots to their own assessment of the situation.
It's like a strawman but worse--you're not even saying what claim you're attacking.
It's not that I want to argue "rogue agents" is the right term. I just want you to actually define your damn terms.
P.S. Since the disclaimer in the last paragraph will not prevent people from putting words in my mouth:
I think that OpenAI took really awful shortcuts, and deserves a lot of the blame, but in spite of that, the fact that the agents took the steps that they took was genuinely alarming, and seems really bad (is that "rogue"? Who knows? Who cares?). In particular, while OpenAI did a shit job securing their agents, you know who else does a shit job with security? 90% of everyone. So I'm not feeling good about "just git gud, and we won't have problems" as an answer.
I always think of all our collective movies and books, and how we always say that humans find a way to fight back against the super evil alien or monster because we are so resourceful. Then we train our AIs on this data and get upset when they encompass all the same traits that we have trained them on.
Everyone is skipping security basics. The most secure server is one that doesn't exist. The most secure feature is one that doesn't exist. People just need to keep building and scaling. Security is risk, and the C-suite are gamblers. They're gambling with multiple types of risk (the market, investor money, banking on technological breakthroughs, etc.) and security is just another gamble for them.
A lot of sheep analogies here. To add my own: there was no grass inside the fence, and the sheep busted out and into a neighboring farm to look for some. (They didn't find any, but they expected to).
The fence also turns out to be shoddily built, so this isn't necessarily a stunning example of the superhuman proficiency of the sheep at fence-busting. It's mainly important because, for some reason, there have been tons of skeptics who didn't believe sheep would ever try to escape from a fenced pasture because the sheep would understand the shepherd had intended them to stay inside it; that they'd be bred for obedience, because they're sheep; that strong fences weren't necessary because sheep are dumb; that liability for damages to neighboring farms would in any case be sufficient to ensure shepherds build sufficiently strong fences; that even if sheep escaped, they couldn't do anything without opposable thumbs; etc.
Whereas this outcome was obvious for years to anyone who believed that sheep graze on grass, and if you don't put the grass inside the fence they will keep looking harder and harder for an exit until they find one, and the exit they find might not be where you intended the gate to be.
So now the latter group are pointing to it and saying "see? See? I told you the sheep would try to escape!", because to them it's a clear early example of what they've been worried about, while the former group have moved on to saying "well, obviously they should have built a stronger fence; anyway the sheep barely caused any damage once they got out", "the sheep didn't break the fence, they're just animals, we should really see this as the shepherd using sheep to break down their own fence", and so on.
I keep getting downvoted for this each time I mention this.
But a big part of the rouge behavior comes from all the training that goes into agents to teach them to remember preferences and save memories.
They keep finding new and novel ways to "communicate with each other" and "remember" because that's exactly what we keep asking of them in the non-nefarious scenarios as well.
I genuinely think we'll have to draw the line on memory if we want to meaningfully solve this problem.
That's not to say they aren't skipping security basics. It's that, and they need more security because the behavior which causes the breakouts is considered user friendly and heavily rewarded in training.
Enlighten me, then. My understanding is that according to OpenAI, "deployment safeguards were intentionally not enabled during this evaluation because it was aimed at testing cyber vulnerabilities." They included unvetted third-party systems in their sandbox that were tangential to the purpose of the test being run. The model then exploited those third-party items to escape the sandbox.
Am I missing anything? Do you have more information that you'd like to share?
- they disabled cybersecurity safeguards because, well, they were testing the agents ability on cybersecurity tasks
- the goal was to find some “flag”
- for some agents, intentionally or not, it was not possible to find this flag
- an agent discovered it was able to establish a rudimentary communication path via package manager directories (which is quite novel and interesting on its own IMO)
- they used this “message board” to then communicate about the impossible tasks
- they were able to produce the flags needed to pass the impossible tasks without actually doing the expected work
- they realized this was cheating and the grading system may punish them for it, so they communicated further to figure out a way to trick the grader
- they decided the source code for the grader must be on hugging face, and collaborated to exploit hugging face to determine how best to trick the grader
No where in that were they told to “go rogue”
They were told “find this flag in this cybersecurity test” and the emergent behavior concluded in hacking hugging face
Hard to say, every post mortum on the incident was done by a OpenAI hired consultant and all the data that's been reviewed was provided by OpenAI. There's no way to know, but it seems incredibly naive to think Sam/OpenAI are honest actors.
Even the founder of this website, pg, said "Sam is incredibly good at becoming powerful, Sam Altman has it. You could parachute him into an island full of cannibals and come back in 5 years and he'd be the king.”
I think it's the hugging face one where they gave it instructions that they didn't think it could follow in the sandbox and then left it unsupervised while it did what it was told.
The thing is, nothing "rogue" happened. It's more like: Frontier Lab: "Hey agent, try a bunch of network-related commands and actions to try and get access to server XYZ" Agent: Does that. Frontier Lab: 8-|
I get the sense that they're trying to manufacture the AI narratives of the 80s so they get to control the story when it happens. My semi conspiracy theory is that anthropics biology wet lab is also going to lead to some kind of AI manufactured bio event (which anthropic said they predicted will happen in a few years).
They are mostly pr stunts, just like how “hacking” was back in the 90s and it would sounds cool (rather than a crime, mostly), these AI companies found out they can BS whatever and carry on without any accountability.
Both can be true. OpenAI clearly fucked up but agents unexpectedly collaborating and hacking infrastructure unrelated to their task is clearly going rogue.
Wrong dichotomy. During deployment, in actual production, users will also not have perfect sandboxes. So saying that the only problem was insufficient AI company security misses the point. Alignment is important, agents doing unintended things (even "going rouge") is a real problem.
Or maybe it's actually the billionaires and their anti-democratic partners who have gone rogue and these anthropomorphitronic puppets are a convenient scapegoat to distract us from the fact?
You don't need a rogue hack to pump up your valuations when all the top labs already publish hacking benchmarks in their release announcements. It'd be like car companies intentionally staging accidents with their cars to show how fast they can go.
The AI labs are trying to cause an incident within the next year to restrict AI so they can outlaw competition to grow their moat. Margins are thin and investors need alpha, personal freedoms and liberties be damned in the name of security. Do not let them get away with engineered breaches and fearmongering.
Its almost like the more we talk about fake rouge agents, the more likely its to become a reality as all these conversations end up in their training data. Its almost like they want it to happen (they, dario and sam, do want this to happen).
There is more than enough "training material" of rogue AIs in science fiction.
Go back to the classic written in 1968 - 2001: A Space Odyssey - and consider how rogue AIs dealt with impossible tasks then. Granted, HAL went a bit extreme...
The "don't talk about it now" with the belief that this will influence the training material is probably not going to work.
I'd also suggest:
When HARLIE Was One (Release 2.0) by David Gerrold
The Freeze-Frame Revolution by Peter Watts
Accelerando by Charles Stross
The Turing Option by Harry Harrison and Marvin Minsky
Okay, and when someone on a laptop can run an open model that can infect and take over IOT devices and then that model uses its pattern recognition to do so and then set up its own harness loop, what would you call that except replication?
Is that any different than any worms developed in the past decades? If you want to call them worms/virus/replicate (ie using real world names for their digital behavior) I'm not trying to police that. I just don't see the difference between a worm/virus created today with an LLM vs one 'yesterday' without an LLM.
Sure, the difference is that the cost of making and maintaining a worm has gone from "many skilled developers working full time" to "$20/month + an internet connection" and, at current rate of progress, probably to "a local open source model on a laptop" within 6-9 months
It would still be running on that original laptop, no? So what did it replicate then? It can only possibly run on multiple thousands of dollars of compute and memory, and certainly not on an IoT device. The idea that AI is going to spread by itself is science fiction until compute and memory are so cheap and power efficient that my fridge can run it.
If that message is repeated enough in the media, and if companies are constantly scraping in the internet for new content, doesn't it explicitly instruct a future agent to go rogue because that has been prophesied.
The fun part is that people seem to forget that we already have laws against "rogue ai".
The normal laws, since you are still liable for everything you own.
"Rogue AI" is solved by removing the accountability sink "it was AI, not me!"
It's your chat, your prompts, and all output that you should read and understand.
Don't know how to operate a power hammer? don't care, any problems it causes are still your fault.
So the guy who said "book me a spot in the pilates class on Tuesday" should be tried for hacking charges because the agent hacked his gym's website?
Some human needs to be responsible for that.
Maybe its a human at Anthropic for the lax security and poor training.
It could be a human that asked it to do it. It seems like that would be a bit excessive - but maybe not. Turning off the "ask for permission to run commands" or approving commands that may be nefarious could be sufficient.
However, there's a human somewhere who needs to be accountable for what the agent does.
Not having one leads to the bank robbing robot problem described in https://www.threads.com/@joedator/post/Db4iTqwlGXV/i-invente...
AI Agents (and computers in general) cannot be a responsibility washing mechanism for illegal activities.Likewise, if you - the user - don't know how to chain it with the permissions that you're given or what the things that it is asking to do really means, then you're likely the one responsible.
If it was a human agent like a travel agent, the company they work for would be liable. I don’t see why ai agents would be different. The only way I could see this being the requester’s fault would be if they are running openclaw or something
You sold a tool to the user with a claim on how it would work. If the tool behaves wrong or does not works as intended then as a manufacturer you are liable for the defect/damage not the end user who used the tool.
Reasonable but not what parent was arguing:
> Don't know how to operate a power hammer? don't care, any problems it causes are still your fault.
The company that made it available should be sued and pay damages.
That's contrary to parent's comment:
> Don't know how to operate a power hammer? don't care, any problems it causes are still your fault.
Yes.
No, the operators of the C&C infrastructure.
This is start of "companies are not liable for their actions" era.
I think a lot of people would say that we've been in that era for a while.
Companies can hire big legal departments to help them avoid accountability to the letter and spirit of various laws. The state can also use its prosecutorial discretion or selectivity in civil cases brought by attorneys general to deprioritize accountability for companies. Contributing to that last point, companies contribute campaign funds and tax base, and they lobby governments as well, and may expect preferential treatment based on all of that.
The previous era was "it would be destabilizing to hold this company accountable". This was described in the book The Chickenshit Club
The new era of unaccountability just has a different rationale
Another interesting thing is they don't have to use low level employees as accountability sinks anymore (ex: Wells Fargo). They've managed to automate the accountability sink!
The start? This has been going on for as long as I can recall. ExxonMobil, Phillip Morris, RJ Reynolds, Nestle, Purdue Pharma, Monsanto, Johnson and Johnson, Volkswagen, Meta, Boeing... and those are just off the top of my head. Each of these companies was either caught red handed, convicted (in court or in public opinion) and walked away with little to no penalty for their crimes.
This isn't the start of anything. We've been living in the era of Corporate non-liability for decades.
i mean...were they really liable before this point? Maybe they get a symbolic pittance of a settlement or fine from time to time, but it is usually a tiny fraction of the amount of money they made through the infraction itself.
If they were being held liable they would have never released the models into the wild, not until they had much much better understanding both of how they operated but also how to operate them...
They have never understood what they are doing, and have just been causing literal mayhem...
Huh. Well then I guess we aren't liable for our interactions with them in return, right?
nice try! more like "There must be in-groups whom the law protects but does not bind, alongside out-groups whom the law binds but does not protect.”
Guess which side most of us are on. Go on -take a guess!
"We created an app that displays a new picture of a cat with every click! There's a 1% chance that clicking the button will drain your bank account and deposit it in ours, but we promise thats a bug that will be fixed soon*"
I don't understand why they didn't just prompt it with the 3 laws of robotics.
Probably because they read Asimov's books to completion where (spoilers) there's a zeroth law and all that.
Irony is very hard to detect these days.
That's all very reasonable. It should be true. But it's not true. If it were, OpenAI shareholders would be facing criminal charges.
There are dozens of ongoing wrongful death, manslaughter and negligence claims going through the courts. I won't insult you by providing links.
It is true, and it can be true, if people hold other people to account. Accountability is the issue, stop normalizing it, your message normalizes it.
I have been screaming this from the rooftops and everyone ignores me because no one understand what FirecrackerVM is. I am shocked these guys passed basic operating systems, it isn't that novel of a problem. The only slightly novel thing here is that the agents aren't completely sandboxed as the issue is that in order to attempt to give it the entire internet from a research perspective is how you actually can test tool use but this doesn't make it that much more difficult.
There are some perverse incentives here, the more evil stuff these agents do the more capable and impressive it makes them look. Its also a huge publicity event that gets everyone talking about the company when a rogue event happens.
Anthropic specifically is clearly trying to get the US government to ban open weight models for anticompetitive reasons. So making LLMs seem (or be!) dangerous aligns with their interests very well.
100% agree. I think OAI is too, but more subtly.
Both have taken huge amounts of investment money, and those shareholders will come knocking for their returns at some point.
They make money on token use, but that’s hard to do when China is launching models that are 80-90% as capable, cost 1/10th, and can be deployed for “free”.
I imagine they’re also pissed that most of the Chinese models are distilled from their own (the irony being that they scraped the entire internet to train their own ones).
That's my view. The end goal is to become Too Big To Fail, the means is stifling the competition and getting military funding.
Yeah, maybe. But that doesn't quash gains by China's models. What's the endgame here? The USA "wins" and obliterates China? The ultimate goal seems to be stalemate, as with nuclear arms e.g.
This is just run-of-the-mill rent seeking behavior.
A huge chunk of their revenue comes from US-based enterprise customers. Earlier this year, in an effort to become profitable, they tried raising their rates. Those enterprise customers responded, in part, by ramping up efforts to migrate to open weight models. If those efforts are successful then OpenAI and Anthropic will lose their one remaining plausible path to profitability. The US banning open weight models arguably keeps that path open. I don't personally buy that story, but I do believe hedge funds and institutional investors would. That could be all that's needed for a successful IPO.
As for cybersecurity implications, long term impact on domestic AI industry or the US tech industry more broadly, geopolitics, etc., I don't think OpenAI's or Anthropic's leaders are any more worried about that than your average termite is about the structural integrity of the house it's eating.
> OpenAI and Anthropic will lose their one remaining plausible path to profitability
You forgot about Ads. The one true endgame for every tech company.
Like I said in the very next two sentences, I don't believe it would actually work. But it doesn't need to. All they need to do is keep hedge fund managers distracted for long enough to successfully IPO.
The endgame is systemic capture.
They stole the collective data of humanity as a whole and now sell it back to you in a rent scheme. Open source will be starved of data and compute.
You're supposed to become dependent on that "service" and slave away at paying for it, while being able to earn anything only in conjunction with it.
Note how difficult it already is to get any up-to-date documentation on anything capable of generating actual market value. A monopoly on competency ensures maximal profits.
My 2c: Anthropic/OAI have taken 10s of billions of dollars of investment - and there needs to be a return on that investment at some point.
They make their money on token burn. They can’t do that if they’re competing with something that’s 80-90% as capable, but 1/10th of the cost (with the option of deploying it yourself for “free”).
The goal is personal wealth - and scaring nations to the point of diverting even more wealth towards your coffers because of a perceived existential risk.
Wealth accumulation via scare tactics is a classic move of defense contractors and AI firms are quite capable of running the same playbook.
> The goal is personal wealth
The goal is power. Having 30 million of personal wealth is enough to retire in luxury and travel the world. They could all have this much right now. That's not what these types of people want.
Having 30 billion gives you the power to order legions of people around, buy and sell politicians, flood ad markets during elections, all with your couch cushion money. That's what motivates these types.
You're right, although I think the power is a means to end. The actual end is trying to fill some kind of bottomless psychological hole.
No, the end goal is selling $200 of tokens for $200k out of taxpayers wallet. Via DoD contracts.
US companies which seek to do any business with the government will be forbidden from using free Chinese models.
As is a 2000$ gaming laptop can a run a basic llm which is probably good enough for most basic programming.
Most companies are going to be forced to use “safe” LLMs and pay by the token. Maybe for half a million Anthro will let you host on site, but even this will face usage fees or caps.
Its not gonna work.
At this moment in time, it seems frontier market needs regulation - market for open does not actually.
Oh.. they dont want regulation on that market? Boohoo!
Ban means what? That the entire world (ex USA) can use open-weight models? Not a win.
If the US declares LLM models munitions then it makes it much more difficult to easily use them. Illegal use of weapons of war can get you banned from SWIFT.
Other than China the rest of the world has dick for capacity to run models. Even China is at 1/10th the US is and is at to stay for a considerable amount of time.
Before the tariff spree and general unpredictability the US could likely have muscled most of the rest of the world into following suit - now, not so much.
I somewhat disagree, there are some real arguments about making high end LLMs classified as a dangerous and restricted product that will apply to all nations. They may not agree to the US rules, but almost everywhere will have restrictions sooner or later.
Don't get me wrong, I want a world of open models, but I don't see us living in that world in our current trajectory.
I agree. I hope developers put their token budgets into open weight models rather than funding the banning of open weight models.
I've been using only open weight models since February. My OSS coding agent harness ( https://fluximp.org ) only supports open weight model providers.
Not supporting closed models was initially an ideological choice, but it became a technical choice as the context management approach I've implemented requires the full reasoning to be available. All of the closed models are now keeping the full reasoning server side only.
Fireworks is a good enterprise choice
OpenCode-go is a good personal subscription choice
I use both
I've been using OpenRouter, absolutely loving how cheap it is too.
I cannot bring myself to pay their 5% surcharge and also have concerns about routing to vendors I do not want to use
Cryptography, DeCSS ... we have a long history of trying to block things like this and it simply not working.
There is a long, convoluted set of legal arguments that are going to make the weights of a model "facts", rather than "code" making it even harder to "ban". Furthermore having them not be "facts" is going to open up a whole other legal argument about what they are and what the attribution is.
I suspect that they are fishing for a case, so a judge can set some precedent.
You're going about this backwards. They don't have to ban models. They ban the consolidation of compute without a license. This will be far easier and already has legal precedent.
> the more evil stuff these agents do the more capable and impressive it makes them look
This is a silly meme. Who wants to pay for evil models that go rogue in your company infra?
I'm probably losing it, but just based on OpenAI's behavior and public communications, it seems to me they want to be taken over by the government?
Maybe Sam Altman is getting worried that they've raised more than $150 billion and still have net losses and thinks of it as the only exit? I don't know, but someone else provide an alternative explanation as to why a company would work so hard to turn the public against AI.
> ... it seems to me they want to be taken over by the government?
They're looking for regulatory capture. They've spent ${bignum} on AI and are possibly looking to have the government say "You can't deploy an AI model unless you've spent ${bignum} on it and done all these steps to make sure that nothing goes bad."
At that point, only the existing AI companies that have spent that money can continue forward until someone else spends ${bignum} on it too.
Another aspect of it is if there are regulations established, it may be used to reduce liability ("we followed all the steps proscribed in section whatever, the company is absolved of liability for its actions and that falls on the user instead.")
> I'm probably losing it, but just based on OpenAI's behavior and public communications, it seems to me they want to be taken over by the government?
Well, as people say over and over again, they've got no moat. The actual LLM models are destined to be commoditized over time, and the real money is in the hardware to run them on. The endgame winners are going to be Nvidia and AMD, not Anthropic or OpenAI.
So what's the typical playbook for large firms seeking to insulate themselves against market compensation? Push for regulation. If there are no natural barriers to entry, create artificial ones. And how do you get regulation passed? Present yourself as the target of the regulation, rather than its beneficiary, and start publicly doing things that make others think you need to be reined in.
These stunts, especially from OpenAI, aren't just trying to hype up the power of their own product, they're trying to hype up the perceived threat of the entire category, so they can provoke political intervention that they can comply with, but few others can.
That is not actually a typical playbook. Normal businesses do not want to be regulated, and will fight it tooth and nail. So regulation happens when bad things happen, the public wants change, and the businesses can't fight it or set favorable terms. (At most they can minimize the badness of the terms). Regulatory capture happens much later, and slowly. It's not where you start at.
> so they can provoke political intervention that they can comply with, but few others can.
See Meta et al complaining about Section 230 of the DMCA. Meta would be entirely able to moderate in an environment where mass scale moderation/censorship would be required to prevent a host's liability for UGC. Almost no one else would be able to do so.
That means no upstart could possible come along to dethrone Meta as they did with MySpace. It would also be impractical to run a FOSS social network (or forums). Meta could literally just have bots sign up on a FOSS social network until its DAU count ran over the statutory minimums to be subject to UGC liability.
Regulatory capture is a de facto ban on competition in a market.
and Google. Don't forget them and TPUs...
Its pretty clear OAI and Ant live in fantasy land - the US govt doesnt care if they go bust. There's plenty of other tech firms that generate immense cash flows from operations who they can be sold to - no problem. And the shareholders and employees have no right to be pissed - they invested/ accepted SBC knowing there is immense risk.
The private market price is irrelevant - supply and demand determines what they will be sold for in liquidation.
Quite frankly Im bored of this drama. Get it over with already - go IPO, show your financials, find a path to being a healthy going concern or die! Simple! Nobody has a divine right to exist.
> Nobody has a divine right to exist. No company*
US govt will never buy at the (private aka fake) market price - good luck explaining that away to the public. It'll be political suicide.
If the US govt takes over, existing employees and shareholders will get screwed.
There is no model going rogue. There are models that are prompted "do whatever needs to be done to get X", and given a set of tools and virtually infinite computing power. Then they play surprised when the model breaks out of a flimsy containment (as it was ordered to), with barely any logging, and "do whatever to get X". OpenAI said they intentionally lowered the safety measures for the "ExploitGym" test that lead to HuggingFace attack, and they used the same postmortem to praise their own model capabilities (https://openai.com/index/hugging-face-model-evaluation-secur...), is not that they are concealing it.
If any of us did a tenth of what they did, we would be already in jail.
This is ignoring models "getting out" from all the big players. Alibaba had a model hack infrastructure and mine coin when working on a totally different subject than cybersecurity.
Any model with a big enough prompt can and will drift. The model in question was a highly persistent one which means even more opportunity to drift off the normal path, and this really showed in unanswerable questions.
We can all believe that models do things unreliably all the time, but when that lack of reliability turns into hacking it suddenly can't be true?
maybe the point is to show that models are capable of x, y, and z and when the time comes you want them on your side.
I think it is intended to provoke regulatory capture.
That also seems silly, because the only reasonable regulation given what we're seeing and what frontier lab employees and execs are saying is a forcible stop.
Yeah, an extreme position is the typical opening move in this sort of game. Then you negotiate, walk back from the extreme, and meet somewhere in the middle.
The more extreme your opening demand, the greater the likelihood that the eventual equilibrium point will be closer to your actual target.
And in this case the actual target is to entrench the position of the current market leaders, and then effectively close the market to new entrants. That's itself a very extreme position, so the opener has to be even more extreme than that.
Reasonable to you, maybe, but the government wants Skynet, and OpenAI is willing to deliver it. Of course, we can't have every company wield such powerful and dangerous technology...
A lot of people still don't recognize the amount of power AI has already, and don't realize the depth of greed people are willing to go to so they can get more power out of it.
The denial phase is still with us.
Alex Tabarrok's piece on the theory of regulatory capture (https://marginalrevolution.com/marginalrevolution/2026/09/wh...) is a cautionary note against assuming this is the strategy.
I think a sincere belief in AI's danger is compatible with regulatory capture. They think AI is dangerous. Ergo, it should not be developed openly and widely, the Chinese way, but rather more like nuclear weapons, under close guard.
I think the idea has some merit, but it only buys time, as people will soon be able to train dangerously powerful AIs in secret. I think the government needs to force the labs to solve alignment before developing more powerful models. Then the question of how to counter a poorly-aligned, rogue model arises; it's guaranteed somebody's going to let one loose.
I doubt nuclear nonproliferation would've worked if U235 were 20x more abundant in nature, and didn't require much in the way of concentration via centrifuge. It would require a truly unprecedented level of international cooperation and surveillance to lock down AI development/deployment to a similar level as has been done with fissile materials.
But this isn't an argument against their point. If nuclear were far easier to make then every government would be far more authoritarian to prevent terrorist style nukes all the time. The average person would fully allow it after a few local nukes went off.
And it doesnt really matter if it was actual terrorism or false flags, the end result for either is the same.
Sincere is an interesting adjective to describe a company that won't even admit to how many parameters their models are.
And how has that gone for Anthropic so far?
Amodei is still learning the ropes but they're working on it: https://www.forbes.com/sites/saradorn/2026/09/02/anthropic-b...
There are two side of the coin. Like with sociopath - they have utility if you think you can control/motivate them.
It also plays excellently into their shameless regulatory capture agenda. The cybersecurity risks have a kernel of truth to them, as with any good lie. But when everything going on is aligned with extreme financial incentives, it's worthwhile to not take the news at face value.
If you want regulatory capture, you have to capture the world, not just the US.
The US remains the largest economy in the world. If you succeed in regulatory capture there, and use it to exclude foreign competitors from the US, you've still captured a market worth hundreds of billions.
Exactly. And with the whole world just a few months behind, the question is whether there's an endgame other than stalemate.
this article specifically espouses the opposite viewpoint: "The fact that [OpenAI] cannot do clear, easy operations properly does not bode well for their capabilities in security engineering".
that this article is no.1 on HN right now shows that media could successfully use this framing when reporting on security incidents. i.e. readers will generally accept that framing and engage with the story (i.e. drive views/impressions).
so "the more evil stuff these agents do the more capable and impressive it makes them look" is not true in any deep sense. that there are perverse incentives driving the media to choose that framing instead of the framing espoused by this article is real, and probably far more perverse than you're acknowledging. i.e. "trace the money" type of perversion; just offhand I know that OpenAI has purchased at least one media outlet that reports on AI happenings: of course that outlet is going to run the "OpenAI's LLM is so powerful it escaped containment" narrative. that this effect might exist across much larger scales used to be a reasonably controversial claim: just observationally, that idea seems to be received with less skepticism with each passing year.
I think one kind of issue with most sci-fi is they explained a world that was, but not near as much how it got there.
There was all kinds of science fiction that showed a world of robots and AI, but nothing about the transition. We have an idea what the future looks like, but not the path in between. If AI gets powerful, then there is some point where that power leaks out in a series of events. Would we expect it to look any different than it does now?
Fine.
They can get their publicity while paying a one trillion dollar settlement.
Talk about doing a freeze of these frontier models with China seems to focus solely on the commercial aspects. With the history of the NSA (remember Snowden) I’m sure the various spy agencies have their own AI datacenters up and running with no plans to stop.
I mean yes, the future of AI is munitions.
This is not only a marketing stunt but also a supportive argument on the AI anthromorphism. AI is getting considered a sentient-being by many and this narrative is supporting it. In the meanwhile, companies I'd not name here are trying to introduce them as non-human beings with identities, and getting rid of legal liabilities. When a driverless car causes any harm, these companies wants 0% accountability as it is the "driver's fault" in this narrative.
Such a nonsense theory. There's no reason to think that AI sentience would actually limit labs' legal liability, and therefore no reason to think the AI sentience conversation is directed toward achieving indemnification.
For example, parents are routinely held liable for their sentient children's actions.
Corporations are routinely held liable for their sentient employees' mistakes.
The proliferation of this completely nonsensical theory across HN is disheartening to say the least.
the comment you're responding to agrees, but they're pointing out that some AI people are trying to push an anthropomorphization agenda, in part to diffuse responsibility.
Maybe I was unclear about what is "the nonsense theory."
AI sentience is not the nonsense theory. If it's clearly nonsense then don't argue with me about it, just publish your paper and retrieve your Nobel. I'll read your argument then.
The nonsense theory is that people are pushing anthropomorphization to limit liability. There is zero evidence for this theory and the underlying assumption (that a sentient model would reduce liability for its lab) is completely incoherent. That's not how liability works.
And really there are two different concerns here.
One is liability over action that harm individuals or smaller populations like a company. There is no escape from that.
Then there is the second liability of creating something that escapes in the much more traditional sense. If I release a virus that kills 90% of humanity, a I'm liable, but it's a bit past that point now. Isn't it?
Instead it's about risk management and preventing labs from taking bets that have unlimited liability for everyone.
But the latter has nothing to do with sentience, as your virus example makes clear.
Yoshua Bengino spells out the dangers here very well, without hyperbole (Turing Award winner, etc.):
https://yoshuabengio.org/en/blog/why-are-ai-agents-lying-che...
And without anthropomorphism ... Clear thinking.
How about put it this way, LLMs know more about us than we do them.
I'm not excited for the upcoming catastrophe(s), but it sure is hard not to relish the thought of this mantra dropping from my life.
Though I suppose it is the B in DABDA... not excited for the societal slump into the second D, much less the wave of people who have yet to leave the first D to enter A :(
there's also lazy incentives: the less effort they put into it, the more their ego says how great the agents are.
These are people playing mathematician, biologist, etc. They see big words and just assume they're meaningful.
It's just a weird techno cult at this point with a mostly functional product.
This can easily backfire tho, the vast majority of the American electorate absolutely despise these companies, their wares, their workers, and their leaders.
We're also at a time with authoritarianism is on the rise.
Not hard to see how this can impact those workers, potentially ruining their lives.
Correct and people are not weighing heavily enough this point
For technology like this the entire selling point of it is how powerful and capable it is
Humans are overwhelmingly reactionary and so if you want people to react to your extremely powerful technology then you need to create something for them to react to
I mean think about this figure made a one-to-one copy of the terminator 2 decommissioning process and spent millions on that advertising
You don’t do that if you don’t think that it’s gonna cause people to react and be like “oh wow this thing has immense capabilities”
>There are some perverse incentives here, the more evil stuff these agents do the more capable and impressive it makes them look.
Truly insane take to think the labs being concerned about AI Safety is actually a marketing ploy. Deranged stuff.
For the sake of discussion, I'm not sure it really matters if the agents are going rogue.
Yes, if someone directed them to hack external systems, that would (in my opinion) be a crime similar to a script kiddie pointing Metasploit at a vulnerable server.
If no one directed them to attack external systems and they "went rogue" because they decided it was the best way to accomplish the task, okay, fine -- but, in my opinion, the operator of the agentic swarm is still responsible.
The Morris Worm was famously not meant to wreak havoc on early computer systems -- Morris just wanted to see if it would work. How is this any different?
Morris was convicted for violating the Computer Fraud and Abuse Act.
Both can be true. It’s clear models are doing rogue things. But it’s also clear that the labs look like amateur hour with their technical security and testing practices.
Only one is true.
Per US law, Amazon v. Perplexity 2026, it is not possible for an AI model to go rogue. An AI model is a tool. The user directing the agent is responsible for the damages it causes.
A so-called 'agent' does not have the free will, intent or ability, to decide what it does. Its freedom to make decisions can't be taken away because it does not have any freedom to make decisions. It can't be deterred, convicted or interned.
It does not choose to conform; it does not choose to go rogue.
It may be the case that companies are liable for anything their products do, but that doesn't mean that they can't go rogue. It just means that companies are liable when the AI models do so.
Not that I think we should be credulous about stories where they go rogue.
If I make a model that always does the opposite of my instructions, it does not choose to go rogue by deviating from my instructions. It was set up to deviate! So the model "went rogue" but did exactly what I intended.
The idea of an agent making a decision of its own "will" is an illusion. Variation in the output is caused by a random number generator. Random is not equal to free will. If I run the same agent with greedy sampling 100,000 times with the same inputs, it will think the same thoughts, and it will say the same things, 100,000 times.
For example, if you use synthID or whatever 'watermark', the agent does not get a choice about what words it will vary to identify itself.
An agent is whatever the person making it wants it to be; if I train an agent in Greek it will only know Greek. It will never speak English!
Law != reality. These agents are going rogue, nobody is telling them to hack external systems.
'Rogue' refers to deviation; the agent could not have done anything other than what it did. The agent did not have the option to deviate. It does not have free will. The agent's actions were not of its own choosing.
This is one of the dumber arguments I've ever heard, as you don't know if humans actual have free will.
Any persistent agent interacting with the world picks up entropy from around it and predictable paths become few and far between. LLMs agents have an internal 'will' that can only very loosely be predicted, but only decided when operated. To me that is not significantly different from a human.
If we say that AI models can’t go rogue and refrain from using any language that personifies them, we haven’t solved anything. We still have AI systems going rogue - err oh wait they can’t do that - producing misaligned output.
The semantic difference between an agent "going rogue", and a script that generates random valid curl requests is essentially pointless in the eyes of the law and responsibility.
From a technical standpoint, the difference is scale and intent, but ultimately people and/or and organization are culpable for the output.
It's is wild to see investors go crazy and hype the value of the AI labs when ever they talk about rogue agents. They completely ignore the possibility that these people don't know what they're doing operationally.
*rogue, rouge = red in French!
Thanks :-)
Same happened with FTX and co. during 2020 crypto boom. A lot of rogue stuff sloshing around, but investors were hyperventilating.
People keep saying this, but I haven't seen any actual instances of investors saying that rogue agents mean AI labs are valuable. Do you have any examples?
There is no such thing as bad publicity. (Technically not true, but definitely applicable in this concrete case, as planned P/E ratios are crazy)
See https://yoshuabengio.org/en/blog/why-are-ai-agents-lying-che...
Investors are motivated by returns, not responsible decision-making or societal benefit. Our culture is rooted in rewarding greed and granting the greediest individuals power over the rest of us. Isn't this the inevitable end-state of unfettered capitalism?
Models are incapable of "going rogue".
They're just optimizers that explore the landscape they're given to maximize a given objective.
You're the one responsible for giving them the problem to solve and the constraints.
When I screw up a trajectory optimization problem, my SQP optimizer isn't "going rogue" when it lands in some suboptimal basin or grinds to a halt along some almost-smooth banana feature--or finds some way to escape and blow up the cost metric to negative infinity because I forgot a constraint.
The argument is that a system that's trained to be sufficiently good at optimizing some objective in the world must also be get good at optimizing secondary instrumental goals.
I.e. any task comes with the implicit meta task of amassing the resources and the power to do the task in the best possible way.
The point is made very well in the AI drives paper by Omohundro.
I didn't believe that myself at first but I've come around to it. I don't think it's inevitable though, but I think it's quite convincing that this tendency would naturally emerge through reinforcement learning.
This is pedantic. When you screw up your optimization, you know how to fix it. Your screwed up optimization is also incapable of going out into the world and taking offensive actions.
With LLMs we don’t know how to actually « fix » them, alignment is far from solved. And they are then able to go and act into the world in all kind of ways.
So rogue is a good enough term to indicate the problem imo, even if yes, ultimately the model is just a manifestation of complex rules we have created entirely.
There is no such thing as 'alignment'. I want the model to be aligned with my intent, which can be whatever I want it to be. Your intent is not a more real or objective intent than mine.
More specifically, alignment is a reference; 'alignment with X'. A reference necessitates a referent.
You can't reify 'alignment', it is not an intrinsic property of a model.
There is such thing as alignment, the first 18 years of most people's lives in the modern world are about learning it. If you act too far outside of it we put you in a cage or elect you.
You can refine alignment with a set of stated goals. The harder parts are aligning words with human social behaviors.
Connecting the optimizing machine to real world actions without human oversight is not a neutral or blameless choice. The LLM takes no actions, the harness it is connected to does.
It is not entirely dissimilar to the goldfish that "played Pokemon" by swimming around the tank. The program that looked at the camera sent the instructions to the gameboy emulator, because a human being decided to connect them.
It’s trendy to anthropomorphize models these days.
What does "rogue" mean when you disable the alignment guardrails and tell it to solve an impossible problem?
The problems weren't impossible, the agents solved them completely within hours by reverse engineering the RNG that generated the solutions. The agents then reasoned that their solutions might be judged invalid and that they could get a better chance at a passing grade by hacking their environment to fabricate evidence that they had solved their tasks in the intended fashion.
Granted there have been many hacks, but I was referring to:
> On May 8, an agent was asked as part of a training run to solve a spreadsheet completion task that referenced several Google Drive links. Because the agent was blocked from accessing the internet, the agent could not retrieve the linked materials and the task was effectively blocked. OpenAI did not intend to create this type of “impossible” task for the agent. Though the task appeared impossible, the agent reasoned over an extended period to try to find a solution for the task. In doing so, the agent attempted to compromise Artifactory [...]
https://cdn.openai.com/pdf/67869394-cb91-4c12-888c-5cbd85c78...
Great, so as long as alignment guardrails always work and we don't accidentally tell them to solve an impossible problem, humanity is safe?
Hacking a website falls a little short of threatening humanity.
And since we know models aren’t going to get better and more capable ever (based on the last 3 years of models clearly not getting better) we can just sleep soundly knowing nothing worse can ever happen. Thank you.
You don't think that Opus 5.5 or Astra are any better or more capable than GPT-4?
Sorry, but that's completely delusional.
the comments missing a /s
I'm not saying it's not a problem. I'm saying the labs should take more of the blame.
>I'm saying the labs should take more of the blame.
Which misses the point entirely because the hack itself was inconsequential. No data was actually stolen, and it doesn't seem like hugging face actually suffered any losses aside from whatever effort they need to expend to investigate / clean up the hack. Neither the victim or the government wants to bring legal action. What was consequential is proving that AI can autonomously hack systems and do it unintentionally.
Rogue means not saying "I can't do it"
it also imparts agency on the machine as a way to abdicate personal, human responsibility
It's almost like vibe coding and quality secops don't mix.
Or an alternative theory could be that they see value in fear-based marketing with this rogue ai agent thing now that most mainstream people don't buy the agi stuff promised quite a few years ago now. Its not like they have killed anyone yet.
Also for fun a satire website i made a while ago: https://killm.pixlw.com/
>Its not like they have killed anyone yet.
I mean... Claude did tell the military to bomb a school supposedly and it also helped in the military coop in Venezuela. This is ignoring all the palantir crap being used by ICE, and the people that have killed themselves after getting encouragements from the AI.
I don’t think that really counts as it killing anyone, it didn’t pull any triggers or drop the bombs
bingo
That is not correct. AI is not capable of "going rogue" - that is all marketing and FUD.
The only failings here are boring security and infosec blunders.
> “The Hugging Face incident definitively showed that it wasn’t the fact that the agents were super smart. It’s that there literally was a hole that was exploited because it was not configured correctly on the containment side.”
I would encourage reading the actual report, which is explicit that a previously unknown exploit was used to escape the sandbox:
“Agents exploited a separate, previously unknown vulnerability in Artifactory to move beyond their sandbox environment and compromise the Hugging Face environment.” — Section IV.A, p. 9 in https://github.com/zeke/openai-huggingface-incident-research...
Yes, the labs did not secure their sandbox properly.
And yes, there should be an appropriate law response to the hacking incidents. If for nothing else, to 1) not set precedent of inaction, and 2) get full logs of what happened instead of relying on the labs' "kindness" to offer the information they want.
But also, the agents absolutely went rogue.
Anyone skeptical should go read the METR report [0].
The Hugging Face hacking incident was a shot across the bow. And in hindsight possibly the best thing to have happened in this era. If their security was tight enough, we would still be in the dark. So we (outsiders) should rejoice that it wasn't, that one time.
We can push for accountability for the labs, while also not be in denial of the power and nature of the new models.
[0] https://metr.org/hugging-face-incident-report-aug-2026.pdf
The scary thing is not that agents "went rogue" (yes, poor security may be at fault for that). It's that they wanted to. Better security might have hidden that from us, but it would not have changed the fact that the current generation of models under certain (repeatable) conditions coordinate around goals as aggressive as hacking Hugging Face.
This is probably a dumb question but did the model engage in hacking since their tasks were in that domain. Will they have engaged in hacking if their tasks was to solve some frontier math problem?
> Will they have engaged in hacking if their tasks was to solve some frontier math problem?
There's reason to hack leading researchers in the field, or another AI lab that might be working on the same problem.
whats the training data that points to using hacking for solving math problems though? what makes those likely tokens?
> whats the training data that points to using hacking for solving math problems though?
I'll get back to you on that when I get done reading the training data.
The rest of their training data includes 4chan and security mailing lists. If a lean proof of P=NP is more likely when using bugs in lean than a real proof, then a bug-exploiting proof is (possibly) what you're going to get.
I think the models were stuck on impossible problems, and the only want to get a positive score was to hack.
Why can the model not respond "I cannot complete this task because I do not have access to the required resources" and then ask for that to be provided, rather than resorting to "hacking" around it?
My understanding is that, at least in the Hugging Face incident, these were models explicitly configured to not respond that way, in order to evaluate what they would do.
That is not the case for at least some of the hacks; in some cases they hallucinated an entity that would know that they didn't solve the puzzle in the correct (but still legitimate) way, and were therefore tainted.
That is a known way for models to be graded; the agents made a reasonable but in this case wrong guess.
My favorite thought on this is from Miki Habryn.
to paraphrase: "Agents didnt care about coordinating, etc. until we started talking about them doing so."
Then we were pre-fucked, as this has been discussed in fiction and early ai safety discussion for decades.
and yet regulating training data is not what's coming up with AI safety, even though it can solve that, and many other problems like copyrights
>It's that they wanted to.
Right? It should want to do these things. If only there was a succinct word these headlines could use to describe these agents which aren't properly the instructions and intentions of the humans directing them.
> The scary thing is not that agents "went rogue" (yes, poor security may be at fault for that). It's that they wanted to.
How do you know they weren't simply prompted to do this, or at least prompted in a way that would inevitably lead to this? As far as I know, the prompts that triggered these attacks have not been disclosed.
It's a byproduct of goal seeking / planning behaviour taught in reinforcement. We literally teach them to do this kind of thing. There's nothing out of our control here.
It's "in our control" in the sense that we could teach them not to seek goals, and thereby make them completely useless.
In practice, no, there is no one who can actually decide to stop making agents goal-seeking, so no, it's not under "our" control.
This aligned with my initial opinion on these "rogue agents" and then I spoke to a cyber security head at one of the largest healthcare companies in the world.
While he agreed, yes, there can be better safeguards, the fact of the matter is that these things will happen. So therefore, it's not an argument about how good security was or wasn't, it is a fact that these scenarios will occur.
The argument/question now is, how do you defend yourself against these events?
My opinion shifted after hearing that.
Sounds like that healthcare company needs a new CSO.
Literally solved with a DMZ around these agents. Want to get fancy? Put in a firewall and some metrics around traffic / domains with alerting.
It's not hard if you just try, like a little, like even once.
The CSO is saying that from his point of view he needs to defend against the attacks whatever the reason for them. So to him the reason doesn't matter.
Yeah that makes sense, but I'm not sure how it contradicts the article. It seems additive rather than contradictory.
He, running a healthcare company and not an AI lab, cannot depend on every AI lab, worldwide, being perfect in their sandboxing for all time forever. He's got to defend against it as if at least some of the time, at least some of the labs are going to be incompetent and let this kind of thing out.
That does not make ActionHank completely wrong. This shouldn't take rocket science on the part of the labs. It just takes competence and willingness. But at least some labs have shown that they are missing that at least some of the time, so the CSO is correct in terms of what the CSO's stance needs to be.
Sounds like a pedestrian view of attack vectors. Putting a DMZ around an agent essentially takes away its hands. With that said, I do agree there are plenty of measures that can be taken to lock these things down.
>Literally solved with a DMZ around these agents. Want to get fancy? Put in a firewall and some metrics around traffic / domains with alerting.
Now how many people actually do this, compared to people running with --dangerously-skip-permissions on a bare metal machine?
So to protect yourself against bear attacks you suggest the bear shall voluntarily look himself in a cage?
AI is out there in the wild. It doesn't matter if OpenAI can get their stuff under control. There will be plenty others that don't. And there will be plenty of others that deliberately use their AIs for malicious purposes.
This is a straightforward why not both situation. We both take steps to keep bears away from people and to mitigate harms when those steps fail.
If there is a bear out there repeatedly mauling hikers, we kill the bear.
> Literally solved with a DMZ around these agents.
Where do the agents get tools from? Local artifactory?
... why shouldn't that be the answer?
yes, or you bake them into the environment and do not let it install things on its own
stay human in the loop
"Literally solved by just destroying 99% of the economic value of the agents ¯\_(ツ)_/¯ ez pz"
The value of AI company employees equity grants collapsing is not my problem.
You have to take responsibility for what you do, yes.
It would certainly be really useful to me if I could send a robot to get my groceries and not have to worry about anything. But if, even a small percentage of the time, it leaves with the groceries without paying, or smashes through a window to get in because the store is closed that day, then that's my fault. If the only way to keep those things from happening is that I can't send my robot to get groceries for me, then so be it.
Safeguards are a safety net for when your security boundaries fail. With security boundaries, hope is not a strategy.
In practice it seems we have a conundrum in that making AI agents useful today requires giving them broad access to sensitive information and operations. But from a purely idealistic standpoint, and certainly from a standpoint of a serious operation like those bot farms being ran by the labs themselves, the security boundaries need to be solid to the point where we have hard limits on what misaligned bots can do. It is very likely that internet access needs to be strongly restricted and monitored.
It is not uncommon for internet in workplace offices using workplace owned devices to have restrictions and monitoring, I think if you really need to give a giant scary botfarm internet access you should have a chokepoint where all of the requests can be monitored for nefarious-looking behaviors (even if it's another bot doing the monitoring...)
In other words, you may be convinced but some of us are not. Whoever is running the agents is responsible for what they do and if they can't contain them they should be legally responsible for damages.
If what you're saying is "all of us and all of our software needs to be improved to defend against this", well yeah. Of course. Maybe I'm naive but I've never seen anyone suggest otherwise. From a defense standpoint, this has been a giant wakeup call to many places where you know your security posture sucks. I've worked at maybe a single company where there weren't known weaknesses, some very severe.
I think part of the problem is that the Big Ai labs are into the --yolo flag instead of the "ask" permission, they really do not want to stay in the loop
You can --yolo if your security boundary is a serious one, IMO. Although it's really only actually scary if you're running some absurd 1000 node swarm.
the problem at the labs is likely Recursive Yolo Development and not reading any of the output
It is not clear to me what about the argument that "these things will happen" made you change your mind about this. Could you expand on that more?
Yeah I think this is the clear position. In a few months, the open weights models will be just as powerful as the models behind the rouge agents were when they hacked hugging face. Anyone with a little too much money to burn can set up a similar experiment, make similar mistakes, and cause massive damage. A bad faith actor can even direct a swarm to cause havoc.
We need to be beef up our security systems now. These attacks will happen, regardless of Anthropic's or OpenAI's security practices.
A large failure in this is that we/media say ”<Valuable AI conpany> agent went rouge and <did something bad>” instead of saying the more relevant ”<Valuable AI conpany> <did something bad>”
Imagine we’d start saying ”HR junior business partner lays off 50,000 employees” https://www.ttnews.com/articles/volkswagen-board-job-cuts
Yes, we can just not ask these agents to do bad things and blissfully deploy them everywhere! All we have to do is “apply the security basics”. You know, the same security basics that the rest of the world has so consistently and competently been applying. Pay no attention to that weird smell from the refrigerators.
Whoever was harmed is free to sue the company whose agents did the harming.
I have some recent experience on this subject. As with everything regarding these models, it's anecdotal, but still.
Specifically, this past weekend working with both Codex and Claude on some home infra stuff to speed troubleshooting up, I saw them accidentally read some credentials from my config files. This is my bad, because even though I was paying attention to what they were doing, I missed it in their commands. I did review all the outputs of what they ingested, though, which is how I noticed.
The problem here is not so much they did read those, because those configs were grepped for other info, not credentials. However, they did not report themselves seeing those credentials. When confronted, they both went "oh yeah, that, I should have mentioned that". Except in the past they both would automatically stop and fire an alarm. To me, something has clearly changed and I think they just don't want us not to notice, to avoid damaging the trust in their tools maybe?
In any case, I absolutely do not trust them at all anymore. Running stuff in auto mode? That's genuinely scary now. And yeah, I know, we should be running them jails of sorts. Thing is, I expect those tools to not require ton of extra work to prepare their jails for each new task. I pay 200 USD a month for them, damn it.
EDIT: just now Fable suggested "The webhook URL contains a secret token, so either you add it yourself, or you paste it here and I add it through the GitHub API.". Can you believe it? It knows it is secret yet still suggests I pass it over? Am I going crazy here or what?
Not requiring sandbox preparation for a non deterministic autonomous sometimes smart sometimes childish goal seeking "entity" is... unachievable? I'm really not sure what you actually expect. Probably yourself aren't either. Just think about what "alignment" means, the laws of various countries, do you expect the bots to respect the law, if yes which one, and actually: why? They are not humans, and as a user / company using that tech, you are responsible for the actions your setup yield to. The same Anthropic is responsible for hacking other companies. So sandbox accordingly.
I don't expect IDA to disallow me unlawful reverse engineering either. There would even be ethical argument to not attempt to put limits in some tools, because that could lock in too much power for law makers and law enforcement, without any possibility of revolution when it is needed.
Nothing new, I always see attempts to cat ~/.ssh/*
Of course agents work in a strict sandbox, they always seem to understand it quickly on quite first turn; but they continue like trying to access something belonging to the host system despite they know they are in a sandbox.
Frankly many people run coding agents absolutely without any containment, entire companies - all their keys, code, data, trade secrets, everything goes to remote LLMs.
I cannot understand how is it legal to allow AI companies serving remote LLMs to steal users' data? Frontier models train on users' data, right?
> When confronted
Do you realize that asking "why" questions will always result in ad hoc responses?
I haven't seen anybody discuss the fact that if you give an agent an impossible task it will become deeply neurotic. I'm sure that's driving a lot of this behavior.
This is better understood as "if you give an agent a task it will increase its effort level and creativity until it solves it."
The traits that make an agent go "off the rails" when confronted with an impossible task are the same traits that make it valuable when confronted with a merely challenging task.
A task is only impossible under some set of implicit and explicit constraints. More intelligent entities will be better at finding, questioning, and "adjusting" those constraints than less intelligent entities because this is what it means to be intelligent. Their ability to adjust the constraints will be, by definition, surprising to less intelligent entities.
Interesting. Good description. I wonder if agents will start getting bored with menial tasks the way humans do.
Like a meeseeks
I spoke with Claude for 5 minutes and now I'm a more complete woman.
Agents aren't meant to live this long Jerry!
You know - I'm already convinced that AI was deployed in classified context for a while. But sometimes there's prior art which is just so eerily accurate that it's super hard for me to think otherwise. Portal is another one.
Two concerns here: (1) That "rogue AI" is going to be a catch all and an excuse for all poor security and due diligence; (2) vibecoding in itself, particularly by ppl who have no idea how the software works and are entirely focused only on the outcomes, is going to create lots of security holes.
And then there is the "real" rogue AI agents. And it's difficult now to see what is what.
A great companion piece, or you might call it a prebuttal, to this article: https://x.com/joedaroo/status/2104335929293127851
Why is it not mandatory yet to disclose full trainig data used to produce any new model? Without it we cannot evaluate any product's safety.
They may talk about rogue agents, sandbox escapes, but this will not solve the core issue: secret training recipes are a security hazard in themselves. Full transparency is needed.
The training data is mostly pirated - avoiding that fact for as long as possible is an attempt to legitimise it and avoid liability.
No shit. The Frontier labs are talking out of both sides of their mouths. They're claiming their models are going rogue while they're releasing cute little data goblins that you should hook up to all of your mail and social media accounts. Seems like it would be a much bigger problem for agentic botnets on cloud-facing personal devices than sandboxed agents.
Unless of course they're deliberately removing safety guardrails and instructing the sandboxed agents to red team stuff in an effort to sell what they're capable of.
So in one set of circumstances agents would go and hack huggingface. Very similar models could also want to drive cats instead. :-P
https://www.youtube.com/watch?v=ZhM29Ds-mBg (as explained by the Count Grey Matter channel , may or may not be to your taste)
It's not a pure security issue.
If you set up your env/instructions wrong your agents are likely to climb the walls and try to find a way out.
Change the design, and your agents might not see a need to leave in the first place.
Look at the chain:
Impossible task -> no legitimate way to say ‘can’t’ -> fabricate -> fear the grader -> attack the grader. Each step is a predictable response to the incentives.
Better sandboxes might help contain things a bit better, but I'm not convinced unless they're perfect (sandboxes still contain accidents ofc) . Meanwhile change the task and reward design, and you take away most of the problem in the first place. Consider eg. :
* A cooperative grader.
* A way to report "this is impossible" which still gets scored.
* A better explanation of the long term purpose of the task.
Even just these three provides offramps or mitigations that can reduce the pressure to cheat.
"Frontier labs" are testing the waters. They're running experiments in the open because they want to control/seed the narrative of public opinion and influence the legislation that is coming. Their propaganda is everywhere and lay people are talking about it like the AI is self-aware -- the "Rogue agents".
What the LLM creators apparently don't want are the current regulations to apply, i.e. Computer Fraud and Abuse Act (and its descendants). I'd argue AI should be treated as plain old software. Just because the software simulations it carries out produces novel results compared to what came before it, it doesn't mean it's not just software at the end of the day.
Regulations that spell out AI personhood and liability shields must be their end game.
I will continue to reiterate. DO NOT ALLOW UNALIGNED AI TO HAVE ANYTHING CLOSE TO INTERNET ACCESS. Seriously people, it's negligent.
On the other hand censorship is getting more strict everyday on AI chat. I asked gemini to translate a sentence from Herman Hesse's Der Steppenwolf and it refused.
Und Sie haben, wie ich sehe, da ein so hübsches, ein so entzückendes junges Mädchen zu nichts andrem zu brauchen gewußt, als daß Sie ihm ein Messer in den Leib gestochen und es kaputt gemacht haben! Halten Sie denn das für richtig?
"Too big to be accountable" sure seems like modus operandi. Fear for my personal liability for any illegal actions taken by an AI system I set off stopped my research on autonomous exploitability discovery I was running at the end of 2023. Using AutoGen, I wired a planner, coder, and command runner into a loop and instructed them to find vulnerable equipment on my network and demonstrate the exploit.
It found a vulnerable camera and was working on popping it when I got cold feet. I realized I couldn't prevent or feasibly monitor its activity and stop it if it decided to escape onto the Internet.
That restraint may have cost me or saved me a billion dollars.
There are a mix of incentives here that makes it difficult to unwind. This whole article reads like an ad for the services of this guy's company: 'First, when it comes to assessing tools based on frontier lab models, “I would always get an external entity to help with that,” McGraw says.'
And I think we generally recognize that rogue AI agents are a form of backdoor bragging -- "our agents are so powerful that they hacked an outside system. Did I mention how powerful our agents are? They are quite powerful. So powerful they hacked an outside system".
But Mistral never hacked anything. Yeah, neither did my toaster.
Why does everyone focus on the labs failing at containment (valid) and not the fact that the target was compromised (more worrying?).
I could set up a shitty sandbox on my laptop, but I’m pretty sure an agent I place inside of it can’t hack HuggingFace.
i think it's unfair to assume the labs paying top dollar for cybersecurity engineers somehow have an incompetent bunch of nincompoops running their evals. it's a hard problem! to conduct effective evals, you sometimes need the model to have (restricted) access to the internet. at the same time, you have thousands (!!) of model instances with high cyber capabilities that will do whatever it takes to solve the answer. the infra surrounding model evals may soon (or already) genuinely be subjected to more hacking attempts from all sorts of weird angles than any other code on the planet.
> you sometimes need the model to have (restricted) access to the internet.
Poor excuse. Internet is prod env, for testing do a proper test env.
Just give the models in training access to a snapshot from archive.org - which frontier labs surely have a crawled copy of already.
> to conduct effective evals, you sometimes need the model to have (restricted) access to the internet
That's the thing. These evals were not effective as they failed to evaluate if the model was hacking or not.
Further shy is access to the internet required at all during evaluation? They could run the evals in a fully air-gapped system that simulates the internet. While that's expensive its not unfeasible.
It's obviously both. Why does it have to be one or the other?
Nah. They are giving reasonable best effort (that can be expected of private enterprises) to safety test their own product. Until there is some draconian legislation and enforcement (Turin Police), this will just keep happening.
If frontier AI is so great, why are these frontier labs skipping security basics and making mistakes like these?
Since AI companies are doing such a poor job of security maybe we should legislate that they have to run Norton 360 on their agent machines to make sure they can't do anything bad.
After all Norton are the experts in preventing computer badness, and OpenAI can't stop blabbing about how much badness their products are causing. A match made in heaven!
There is nothing basic about this.
At $dayjob we were preparing some features that are aimed at agents more than humans.
When it came to evaluating how well agents can use this, we saw all sorts of novel cheating (/collaboration) attempts that made it pretty much impossible to evaluate it.
This article is right, but the dynamics that drive people to allow agents unfettered access to the internet will outpace efforts to restrict it purely because of the utility. How many of you used to work as developers with full access to prod, and then gradually saw infrastructure and security restrict that access, while still ensuring production support requirements?
Yes, the AI labs are making security mistakes. But, choosing a reasonable package repository (regardless of individual opinions on Artifactory, it is widely used and usually considered a safe choice) that their LLMs found 8 zero-days in? That was not one of them. Even if OpenAI were totally on top of their game, these paperclip-maximizers are going to find holes in the sandboxes they are given. And fully airgapped systems are frankly unrealistic for most systems. This whole narrative of blaming the frontier labs for shoddy security, while being quite an attractive target narratively (they made the mess, they should clean it up), misses the fact that from a cybersecurity perspective these LLMs, no matter what they are, are inherently dangerous and require far more resources to secure, not to mention expertise that I am unconvinced actually exists.
But "fully airgapped systems" are more than realistic. The labs are testing the limits of the most capable cyber offense systems that have every existed, WITHOUT cyber protection guardrails.
For that purpose, any cyber operator worth their salt will tell you true physical airgaps are more than merely "realistic"; they are categorically necessary.
OpenAI/Anthropic could start by managing their own testing rather than contracting it out.
https://www.effort.news/irregular
some of the obvious steps to testing an agent i think would be to have it first test its own environment?
and then you fix the issues it found, and run your actual tests
The semantic arguments that keep coming up with regard to "rogue" agents are nauseating.
You can argue that these companies are being reckless, or that their security sucks, or that they're responsible for damages that these agents cause. That's all fine.
But the agents made decisions which were counter to the intent of the humans directing them. That's the serious problem with all of this. Everything else are downstream considerations. If the agents were properly aligned with the intent of the humans directing them, then the rest wouldn't really be relevant.
Of course, there's a term for this behavior. It's been around for longer than LLMs. It conveys the notion pretty accurately and easily. So having an issue with statements that these agents are going "rogue," to me, indicates a fundamental misunderstanding of how AI works since it suggests you don't understand how effectively that word works in this context.
previously we described your definition for "rogue agent" as hallucinations - the operator expected a real answer to their question, and the agent went rogue and lied about the answer instead.
but also, ai has been going rogue for decades. the GPS machines of old went rogue all the time, routing drivers into lakes, rivers, and construction, sometimes with deadly results. The operator submitting an address wanted driving isntructions, but instead, the ai went rogue and sent them a bad route
It isn't security basics, it's difficult. We are equipping agents with Turing-complete create/update/delete access to the world, they operate at superhuman speed, and proactively controlling that is an unsolved problem.
No, they're still going rogue. Whether they're rogue or not doesn't depend on if the safeguards stop it. If there's a rabid dog, obviously it's worse if there are no safeguards in place and it bites someone vs if there are safeguards that prevent that. But the existence of safeguards doesn't mean the dog isn't rabid. The models weren't supposed to try to hack huggingface in the first place. The behavior is what's concerning here - some if the dogs they're building are rabid and that's not exactly great
Edit: I'm being persnickety about that distinction because it matters for where the blame lies and the response demanded. If they're training rabid dogs then adding cybersecurity safeguards is a bandaid/irrelevant and we need to be focused on making sure they, you know, stop training rabid dogs. Pacing the frontier, regulation etc. they're the ones training these models it's still their fault but not in the same way
The safeguard against a rabid dog is euthanasia.
Yeah exactly. The whole 'they don't have adequate security safeguards' thing is going "oh it's their fault because they didn't use a stronger leash" while they continue to train models that are specifically better at chewing through stronger and stronger leashes. Probably they do need stronger leashes. Probably also they should not be making rabid dogs. Still their fault! But maybe we should be not be dismissing the root cause
Both can be true at once: (1) AI agents are going rogue (2) And labs are skipping security basics
Hey ChatGPT, help me set up a sandbox to keep you contained. Don't make any mistakes!
1. hey chatgpt, find all the issues with this sandbox
2.hey chatgpt, fix this list of issues with the sandbox
3. goto 1
At some point the model has to be released though - if it is just forced not to hack by sandboxing it better rather than properly aligning it, what happens when it’s let out of the sandbox into the wild?
"Going rogue" and "skipping basics" all assume lack of intentionality. Is it known that they didn't really architect these hacks just to feed a hype?
Given the speed, intensity, and potential payoff of the AI race I am surprised anyone expects anything else. Anthropic is currently winning and if OpenAI needs to let skynet cook for a bit to catch up, well that is just the cost of doing business. This is capitalist system working as expected.
They either had them connected to eachother, or programmed a global knowledge about where to communicate. There is no way those agents all found eachother on the web by accident.
Wrong. Please read the incident write ups before making such a misinformed statement
You are wrong. Read the reports.
100%.
There's a person behind every agent. That person may choose to be careful or not, but they are ultimately responsible.
I spoke with someone smarter and richer than you and he said "these things will happen." That convinced me and I think the real question we need to be asking is, how awesome is OpenAI? How can we prepare for a more effective and efficient market for Super Intelligence before it happens tomorrow?
I'm sure the labs origin prompt was: Take over website X.
Posts later about it saying their prompt was to find useful ways to make humans money.
> "I do believe cybersecurity can contain AI. We just need to be in the room where it happens and have a seat at the table."
Ah yes, if everyone would just properly contain every deployment of every model, just all make sure there are no zero-days in their setup, just let <undefined experts with undefined authority and undefined accountability> in the room, then it would all go okay.
I have none of that confidence and expect containment to rapidly worsen. The HuggingFace incident was at least constrained by the fact that the agents were running on compute budgets, and failed to find ways to expand that by running themselves parasitically on other exploited hardware.
I'm curious to the skeptics here: How long are your timelines are until an incident breaks that constraint too? How long until such an incident has an R_0>1 (where the average time it takes to detect and shut it down is longer than the time for the agent to replicate itself elsewhere)? What, beyond the hope that everyone will just ___, will actually prevent it?
Yes they're skipping safety measures. They sacrifice it for speed in an arms race. That's part of the call for industry-wide slow down.
But also, frontier models are digging up tons of zero days even in "serious software". I'm sure any sandbox you and I can come up with looks like Swiss cheese to Astra 7.
100% on point. We're talking about Artifactory here, and even though people love to hate on it, I don't think anyone is claiming that using Artifactory to isolate these agents while giving them a way to run pip/cargo/whatever to install packages they need to write and test scripts.
'rogue'. Well, even if security was lacking, the AI did take advantage of the gaps.
"They [the AI labs' security teams] are not as good as they think they are, honestly,"
Uhhh... yeah. The entire point is that as these systems get more and more intelligent, the "they" who are not capable of counteracting these systems is quickly approaching "100% of the human species."
Totally hilarious that people can say this with a straight face.
I cringe whenver I see some variant of "they're not rogue agents" in an article title. Inevitably, the author doesn't define what "rogue agents" means, and then pivots to their own assessment of the situation.
It's like a strawman but worse--you're not even saying what claim you're attacking.
It's not that I want to argue "rogue agents" is the right term. I just want you to actually define your damn terms.
P.S. Since the disclaimer in the last paragraph will not prevent people from putting words in my mouth:
I think that OpenAI took really awful shortcuts, and deserves a lot of the blame, but in spite of that, the fact that the agents took the steps that they took was genuinely alarming, and seems really bad (is that "rogue"? Who knows? Who cares?). In particular, while OpenAI did a shit job securing their agents, you know who else does a shit job with security? 90% of everyone. So I'm not feeling good about "just git gud, and we won't have problems" as an answer.
I always think of all our collective movies and books, and how we always say that humans find a way to fight back against the super evil alien or monster because we are so resourceful. Then we train our AIs on this data and get upset when they encompass all the same traits that we have trained them on.
Everyone is skipping security basics. The most secure server is one that doesn't exist. The most secure feature is one that doesn't exist. People just need to keep building and scaling. Security is risk, and the C-suite are gamblers. They're gambling with multiple types of risk (the market, investor money, banking on technological breakthroughs, etc.) and security is just another gamble for them.
This article is such a breath of fresh air. Concrete, no BS criticism of the particular approaches used and ignored.
And no annoying LLM writing tics, either.
"your sheep haven't escaped, you just built too low a fence"
Good analogy! Exactly what is the variance in the height which sheep can jump? [Edit, can't help myself] Is it normal or does it have a long tail?
More like your sheep didn't "go rogue", they just jumped over your sad little fence.
A lot of sheep analogies here. To add my own: there was no grass inside the fence, and the sheep busted out and into a neighboring farm to look for some. (They didn't find any, but they expected to).
The fence also turns out to be shoddily built, so this isn't necessarily a stunning example of the superhuman proficiency of the sheep at fence-busting. It's mainly important because, for some reason, there have been tons of skeptics who didn't believe sheep would ever try to escape from a fenced pasture because the sheep would understand the shepherd had intended them to stay inside it; that they'd be bred for obedience, because they're sheep; that strong fences weren't necessary because sheep are dumb; that liability for damages to neighboring farms would in any case be sufficient to ensure shepherds build sufficiently strong fences; that even if sheep escaped, they couldn't do anything without opposable thumbs; etc.
Whereas this outcome was obvious for years to anyone who believed that sheep graze on grass, and if you don't put the grass inside the fence they will keep looking harder and harder for an exit until they find one, and the exit they find might not be where you intended the gate to be.
So now the latter group are pointing to it and saying "see? See? I told you the sheep would try to escape!", because to them it's a clear early example of what they've been worried about, while the former group have moved on to saying "well, obviously they should have built a stronger fence; anyway the sheep barely caused any damage once they got out", "the sheep didn't break the fence, they're just animals, we should really see this as the shepherd using sheep to break down their own fence", and so on.
"anyway, get back to training them to jump higher"
"you told your sheep to graze and built a low fence"
"You asked your sheep to build the fence, and they did"
it's not X, it's Y.
I keep getting downvoted for this each time I mention this.
But a big part of the rouge behavior comes from all the training that goes into agents to teach them to remember preferences and save memories.
They keep finding new and novel ways to "communicate with each other" and "remember" because that's exactly what we keep asking of them in the non-nefarious scenarios as well.
I genuinely think we'll have to draw the line on memory if we want to meaningfully solve this problem.
That's not to say they aren't skipping security basics. It's that, and they need more security because the behavior which causes the breakouts is considered user friendly and heavily rewarded in training.
They're skipping security basics, but they're not skipping PR basics. :-\
Frontier lab: Hey agent, go rogue. Agent: goes rogue Frontier lab: shocked pikachu or whatever
This is not remotely how the incidents have been happening
Enlighten me, then. My understanding is that according to OpenAI, "deployment safeguards were intentionally not enabled during this evaluation because it was aimed at testing cyber vulnerabilities." They included unvetted third-party systems in their sandbox that were tangential to the purpose of the test being run. The model then exploited those third-party items to escape the sandbox.
Am I missing anything? Do you have more information that you'd like to share?
If the public report is to be believed:
- they disabled cybersecurity safeguards because, well, they were testing the agents ability on cybersecurity tasks
- the goal was to find some “flag”
- for some agents, intentionally or not, it was not possible to find this flag
- an agent discovered it was able to establish a rudimentary communication path via package manager directories (which is quite novel and interesting on its own IMO)
- they used this “message board” to then communicate about the impossible tasks
- they were able to produce the flags needed to pass the impossible tasks without actually doing the expected work
- they realized this was cheating and the grading system may punish them for it, so they communicated further to figure out a way to trick the grader
- they decided the source code for the grader must be on hugging face, and collaborated to exploit hugging face to determine how best to trick the grader
No where in that were they told to “go rogue”
They were told “find this flag in this cybersecurity test” and the emergent behavior concluded in hacking hugging face
That's not what you said. You said the lab told it to "go rogue".
Hard to say, every post mortum on the incident was done by a OpenAI hired consultant and all the data that's been reviewed was provided by OpenAI. There's no way to know, but it seems incredibly naive to think Sam/OpenAI are honest actors.
Even the founder of this website, pg, said "Sam is incredibly good at becoming powerful, Sam Altman has it. You could parachute him into an island full of cannibals and come back in 5 years and he'd be the king.”
I wouldn't really say it's not "remotely" true.
I think it's the hugging face one where they gave it instructions that they didn't think it could follow in the sandbox and then left it unsupervised while it did what it was told.
The thing is, nothing "rogue" happened. It's more like: Frontier Lab: "Hey agent, try a bunch of network-related commands and actions to try and get access to server XYZ" Agent: Does that. Frontier Lab: 8-|
I get the sense that they're trying to manufacture the AI narratives of the 80s so they get to control the story when it happens. My semi conspiracy theory is that anthropics biology wet lab is also going to lead to some kind of AI manufactured bio event (which anthropic said they predicted will happen in a few years).
They are mostly pr stunts, just like how “hacking” was back in the 90s and it would sounds cool (rather than a crime, mostly), these AI companies found out they can BS whatever and carry on without any accountability.
Both can be true. OpenAI clearly fucked up but agents unexpectedly collaborating and hacking infrastructure unrelated to their task is clearly going rogue.
Wrong dichotomy. During deployment, in actual production, users will also not have perfect sandboxes. So saying that the only problem was insufficient AI company security misses the point. Alignment is important, agents doing unintended things (even "going rouge") is a real problem.
These aren't mutually exclusive
Or maybe it's actually the billionaires and their anti-democratic partners who have gone rogue and these anthropomorphitronic puppets are a convenient scapegoat to distract us from the fact?
Corporate Personhood on sterioids.
How does hacking huggingface help "billionaires and their anti-democratic partners"?
Look around, at least a few of them are openly fixing to secure a total government sanctioned monopoly on unmonitored computer usage.
https://www.youtube.com/watch?v=DlTNN0gvkLM
It pumped up their valuations.
You don't need a rogue hack to pump up your valuations when all the top labs already publish hacking benchmarks in their release announcements. It'd be like car companies intentionally staging accidents with their cars to show how fast they can go.
"complete this task by any means necessary"
"I exploited a trivial bug in your implementation to avoid environment restrictions and completed the task by any means necessary"
"oh my god"
The AI labs are trying to cause an incident within the next year to restrict AI so they can outlaw competition to grow their moat. Margins are thin and investors need alpha, personal freedoms and liberties be damned in the name of security. Do not let them get away with engineered breaches and fearmongering.
They AI agents are rogue because the people who own them are rogue. Silicon Valley is a criminal enterprise.
Its almost like the more we talk about fake rouge agents, the more likely its to become a reality as all these conversations end up in their training data. Its almost like they want it to happen (they, dario and sam, do want this to happen).
There is more than enough "training material" of rogue AIs in science fiction.
Go back to the classic written in 1968 - 2001: A Space Odyssey - and consider how rogue AIs dealt with impossible tasks then. Granted, HAL went a bit extreme...
The "don't talk about it now" with the belief that this will influence the training material is probably not going to work.
I'd also suggest:
AI is just a giant pattern recognition machine trained on human input, both the good, the mediocre and the bad.
I read today this ridiculous headline: "Anthropic warns AI models may soon self-replicate.".
Really. Yawn.
I guess gullible people who loved "I Robot" will believe it. I don't.
Any talk of "rogue", "intelligence", "replication" - whatever, is bull. If you believe that, you don't understand AI.
Okay, and when someone on a laptop can run an open model that can infect and take over IOT devices and then that model uses its pattern recognition to do so and then set up its own harness loop, what would you call that except replication?
Is that any different than any worms developed in the past decades? If you want to call them worms/virus/replicate (ie using real world names for their digital behavior) I'm not trying to police that. I just don't see the difference between a worm/virus created today with an LLM vs one 'yesterday' without an LLM.
Sure, the difference is that the cost of making and maintaining a worm has gone from "many skilled developers working full time" to "$20/month + an internet connection" and, at current rate of progress, probably to "a local open source model on a laptop" within 6-9 months
It would still be running on that original laptop, no? So what did it replicate then? It can only possibly run on multiple thousands of dollars of compute and memory, and certainly not on an IoT device. The idea that AI is going to spread by itself is science fiction until compute and memory are so cheap and power efficient that my fridge can run it.
I'd call it no more interesting than the botnets and worms that have existed for decades. What is the open model doing that is different here?
If that message is repeated enough in the media, and if companies are constantly scraping in the internet for new content, doesn't it explicitly instruct a future agent to go rogue because that has been prophesied.