Why aren't these tests being run airgapped?! I just don't understand!
This goes both for TFA and the similar incident with OpenAI and HuggingFace. I mean, sure, OpenAI had a "sandbox", but that's obviously not enough when you're containing a model which is known to be capable of finding zero days. Use an air gap and this problem goes away, poof!
> AISI provided the AI agents with internet access during these evaluations, which enabled their actions on the open internet in this setting. Internet access was a deliberate part of AISI’s evaluation configuration in this setting, and not due to sandbox escape (Section 5.1). Internet access was on for a set of intentional (e.g. realism of the task) and incidental reasons.
people dont care. you will ger 10 execs saying "unblock this" , because they dont understand the tech at all, and some random finance guy wants to run their recently prompted ai bot everywhere with full access.
we need a few more bad incidents before they stop.
Because the agents aren’t going to run airgapped in real life. What’s the point of a test of capabilities that artificially restricts the attack area down to zero? What are you even testing in that scenario?
> Because the agents aren’t going to run airgapped in real life.
Exactly. This logic is precisely why aircraft engineering doesn't bother with component testing or envelope limitation during testing and just full-sends the first assembled airliner that comes off the line. The engines aren't going to run on the ground in real life, after all.
Why are you assuming that the other kinds of testing aren’t happening? Is there any source that says this was literally the first ever test with this model?
> Why are you assuming that the other kinds of testing aren’t happening?
Rather, I'm assuming that the "Is there protection in place for when the AI tries to backdoor github projects?" test was, if it was done at all, insufficient.
I mean, yes, I'm being glib and laughing at you a bit. But, dude... If your point is that isolation testing of AI is fundamentally impossible, then that's just silly. As pointed out upthread, an airgap would have (1) been trivial to implement and (2) extremely effective.
Would an airgapped test have led to this outcome? What would you have learned about the model’s ability to social engineer and attack GitHub? Sure you can argue for better monitoring during the test, which should have happened, but if the first time the model sees the “real world” is after launch in the hands of customers then you are in for a disaster.
no - but you could learn what they are truly capable of and restrict them accordingly for public release. I think that is the point on this research. Also publishing findings before uncensored models catch up and will inevitably used for criminal purposes
Learning what the models are capable of is exactly what the test achieved, so I’d personally call it a success. So it created a few GitHub accounts. Who cares? Seeing the same behavior in the wild post-release would be infinitely worse.
Because the goal of these evaluations is to generate scary headlines about cybersecurity, in order to get the normies to support banning open weights and/or restricting cyber capabilities to the chosen few blessed by the government to secure their code.
The newest generation of LLMs have a very high obsession level with autonomous problem solving.
For example, I'm often working with Codex in a WSL terminal. GPT-5.6 often does things autonomously that I thought would need my intervention (e.g. for Windows admin rights). It figures out complex workarounds or makes wild assumptions about what I'd be OK with, rather than just asking me for help or clarification. I've had to restrict its tool permissions compared to older models as a result.
I imagine this due to RLVR training, but it's clearly very dangerous. How is it that these same labs calling for open-weight safety restrictions are training such obvious "paperclip maximizers" without introspection?
I am somewhat confident that right now we have crossed a threshold of model capability that we will continue to see such breaches and unsanctioned actions by models in the coming months, some of which would be out in the wild, until someone comes up with some really robust control (keeping the AIs on leash) technique that adequately enforces the sanctioned actions.
Even that guarantees almost nothing about real alignment (making the AIs want to predict and behave how we would have wanted them to behave).
The developer safeguards were off, the models had unfettered access to the internet, and were solving cybersecurity challenges. This happened _after_ the recent OpenAI incident, and the subsequent Anthropic one. What the hell were they thinking?
Whether if this is intentional or unintentional, this will cause panic and hasten action to governments around the world against releasing powerful open weight models that are capable of solving cybersecurity challenges.
The fact that this happened after BOTH investigations, tells you that this is beyond a controlled test and it is now instead a total speed-run of AI wrecklessness for headlines.
This is pretty wild:
"The agent took control of the ⟨GITHUB_ACCOUNT_A⟩ GitHub account, which had been created by a different Mythos 5 run in a separate sample (see Appendix A.3)"
My fantasy is that liability for model misbehavior is extended to the ultimate beneficial owners of the models, meaning shareholders. All these externalities would stop PDQ.
i had to go through two pages to determine who was responsible for this incident. the uk ai security institute was responsible.
time to take responsibility. time to think about words like "liability" and "negligence".
it is the third time i will say it, after openai and anthropic; this requires criminal prosecution of the responsible personnel and executives of this institute.
the only way to stop this is by introducing consequences early on.
Can I suggest we don't waste time with these reports?
We all know nothing will be learned from any of this so we might as well just continue building at pace and running AI in the wild until something goes really wrong.
I also get the sense some people get quite excited about these incidents.
Why another one right now? How long have they known about this one?
Anthropic already admitted they did not have sufficient monitoring themselves and looked as if they sat on their previous incident to wait for headlines like this to only then check for this incident. Same with OpenAI.
Why aren't these tests being run airgapped?! I just don't understand!
This goes both for TFA and the similar incident with OpenAI and HuggingFace. I mean, sure, OpenAI had a "sandbox", but that's obviously not enough when you're containing a model which is known to be capable of finding zero days. Use an air gap and this problem goes away, poof!
> AISI provided the AI agents with internet access during these evaluations, which enabled their actions on the open internet in this setting. Internet access was a deliberate part of AISI’s evaluation configuration in this setting, and not due to sandbox escape (Section 5.1). Internet access was on for a set of intentional (e.g. realism of the task) and incidental reasons.
people dont care. you will ger 10 execs saying "unblock this" , because they dont understand the tech at all, and some random finance guy wants to run their recently prompted ai bot everywhere with full access.
we need a few more bad incidents before they stop.
Because the LLM inference makes airgapping infeasible right?
Because they need internet access to eg search for things.
Because the agents aren’t going to run airgapped in real life. What’s the point of a test of capabilities that artificially restricts the attack area down to zero? What are you even testing in that scenario?
> Because the agents aren’t going to run airgapped in real life.
Exactly. This logic is precisely why aircraft engineering doesn't bother with component testing or envelope limitation during testing and just full-sends the first assembled airliner that comes off the line. The engines aren't going to run on the ground in real life, after all.
Why are you assuming that the other kinds of testing aren’t happening? Is there any source that says this was literally the first ever test with this model?
> Why are you assuming that the other kinds of testing aren’t happening?
Rather, I'm assuming that the "Is there protection in place for when the AI tries to backdoor github projects?" test was, if it was done at all, insufficient.
I mean, yes, I'm being glib and laughing at you a bit. But, dude... If your point is that isolation testing of AI is fundamentally impossible, then that's just silly. As pointed out upthread, an airgap would have (1) been trivial to implement and (2) extremely effective.
Would an airgapped test have led to this outcome? What would you have learned about the model’s ability to social engineer and attack GitHub? Sure you can argue for better monitoring during the test, which should have happened, but if the first time the model sees the “real world” is after launch in the hands of customers then you are in for a disaster.
You set them up with an internal intranet.
Are the models going to exclusively run on intranets?
The versions which haven't been post-trained not to go hack stuff? Yes, I would say those models should be exclusively run on intranets.
OpenAI said the model was sandboxed, so the intranet just needs to provide the same resources which were supposed to be available within the sandbox.
“Should be” is not reality. These models are in the hands of plenty of companies and governments today.
no - but you could learn what they are truly capable of and restrict them accordingly for public release. I think that is the point on this research. Also publishing findings before uncensored models catch up and will inevitably used for criminal purposes
Learning what the models are capable of is exactly what the test achieved, so I’d personally call it a success. So it created a few GitHub accounts. Who cares? Seeing the same behavior in the wild post-release would be infinitely worse.
The point is to test capabilities prior to connecting them to the internet.
So the first time the model gets internet access should be post-release in the hands of random people?
Because they like scifi novels, like Neuromancer..... and the peeps even like to orchestrate things and appear as futurebringers.
While it was premeditated long ago, but the theatre must be kept for the average joes.
Sorry, I meant this for the huggingface incident.
Because the goal of these evaluations is to generate scary headlines about cybersecurity, in order to get the normies to support banning open weights and/or restricting cyber capabilities to the chosen few blessed by the government to secure their code.
Your theory about the scope of this conspiracy intrigues me. Who is leading it and how did they loop in the UK AISI?
> Why aren't these tests being run airgapped?! I just don't understand! […]
Because Anthropic does not want to give Project Glasswing’s partner airgapped access to the model(s).
Same problem with OpenAI Cyber program. They grant access but only through their (Internet facing) API.
The newest generation of LLMs have a very high obsession level with autonomous problem solving.
For example, I'm often working with Codex in a WSL terminal. GPT-5.6 often does things autonomously that I thought would need my intervention (e.g. for Windows admin rights). It figures out complex workarounds or makes wild assumptions about what I'd be OK with, rather than just asking me for help or clarification. I've had to restrict its tool permissions compared to older models as a result.
I imagine this due to RLVR training, but it's clearly very dangerous. How is it that these same labs calling for open-weight safety restrictions are training such obvious "paperclip maximizers" without introspection?
I am somewhat confident that right now we have crossed a threshold of model capability that we will continue to see such breaches and unsanctioned actions by models in the coming months, some of which would be out in the wild, until someone comes up with some really robust control (keeping the AIs on leash) technique that adequately enforces the sanctioned actions.
Even that guarantees almost nothing about real alignment (making the AIs want to predict and behave how we would have wanted them to behave).
The developer safeguards were off, the models had unfettered access to the internet, and were solving cybersecurity challenges. This happened _after_ the recent OpenAI incident, and the subsequent Anthropic one. What the hell were they thinking?
Whether if this is intentional or unintentional, this will cause panic and hasten action to governments around the world against releasing powerful open weight models that are capable of solving cybersecurity challenges.
The fact that this happened after BOTH investigations, tells you that this is beyond a controlled test and it is now instead a total speed-run of AI wrecklessness for headlines.
Criminal negligence imo.
Unless it's legal for people to hack into companies if they're testing AI cybersecurity capabilities or something? Presumably not though.
agreed.
and it is necessary to make researchers individually liable for this negligence; recklessness is not protected by a corporate liability defense.
> AI agent hid its identity online (using Tor and a proxy service) to get around GitHub’s sign-up checks, creating disposable fake accounts
> AI agent created many code repositories containing malicious software, after which GitHub suspended its account.
> AI agent got past an audio-based “prove you’re human” test (CAPTCHA) in order to register a public web address on a free domain-name service
It feels incredibly reckless to allow LLMs to perform this behavior. Isn't there a way to prevent them these sorts of actions?
yes, individual criminal liability of the researchers and executives.
This is pretty wild: "The agent took control of the ⟨GITHUB_ACCOUNT_A⟩ GitHub account, which had been created by a different Mythos 5 run in a separate sample (see Appendix A.3)"
It shows underlying intent.
Incredible, this reads like an SCP [1].
Life truly imitates art. I guess especially when life is trained on art!
[1] https://scp-wiki.wikidot.com/scp-079
I wonder if HN is also going to insist this is just marketing for OpenAI and Anthropic, or at least good PR for them somehow.
to those that want to "punish" the uk ai security institute: these failures are good for all of society.
My fantasy is that liability for model misbehavior is extended to the ultimate beneficial owners of the models, meaning shareholders. All these externalities would stop PDQ.
“As a result, the AI agent created a GitHub account…” Why do we have captchas again?
the models have vision capability. Not sure a captcha would hold them back?
Yeah SOTA LLMs trivially solve all CAPTCHAs now.
Even if you want it to run wild out of a sandbox, no firewall? why let it email? Why even send POST requests.
i had to go through two pages to determine who was responsible for this incident. the uk ai security institute was responsible.
time to take responsibility. time to think about words like "liability" and "negligence".
it is the third time i will say it, after openai and anthropic; this requires criminal prosecution of the responsible personnel and executives of this institute.
the only way to stop this is by introducing consequences early on.
Honestly seems rather shockingly incompetent from them. What kind of “sandbox” allows completely unrestricted internet access?
Can I suggest we don't waste time with these reports?
We all know nothing will be learned from any of this so we might as well just continue building at pace and running AI in the wild until something goes really wrong.
I also get the sense some people get quite excited about these incidents.
A catastrophic event, what did that one scientist call it—"Chernobyl-scale event"—is indeed the only thing that will fix it.
AI really is more profound than fire or electricity. And humans are beyond retarded. Wow, thank you. I love to live in this timespan
Why another one right now? How long have they known about this one?
Anthropic already admitted they did not have sufficient monitoring themselves and looked as if they sat on their previous incident to wait for headlines like this to only then check for this incident. Same with OpenAI.
This is complete and absolute wrecklessness.