I havent used an openAI product since GPT 3.5 or Anthropic since 4.5 or 4.6. Everyone around me using these SOTA models doesnt really get anything done. It seems like they just feel like they are productive, a psuedo productivity.
I write some code, spec a lot, and use fast models to fill in the middle. I outpreform everyone around me. Im not convinced these autonomous "swarms" or /goal are all that useful.
I notice the people using them become dumber by the month (spend tons) and the quality of their work declining (they're also losing their jobs in some cases).
And obviously the point of calling them rouge agents to offload the liability onto the agent. The number one economic value of agents will be offloading corporate liability. That's what they want to sell to enterprise, an algorithmic scapegoat.
Seeing benefits for myself and team for sure, so I disagree there, but a related thing seems to be true: with the rapid shifting of failure modes, building systems and competence around mitigating the weaknesses of current SOTA models seems to be very short term investments. It's unclear if being a 'good AI user' is a skill that will have any merit at all, very soon
AFAIK all of these incidents happened when OpenAI contracted out to a company called Irregular (https://www.irregular.com/) to run these sandboxed CyberGym tests. They all happened around Mar-June and seem to be from the same collection of agent trials. Since then they already released Astra. Halting now is likely just a way to manage blowback.
This should be the top comment on every one of these godforsaken posts. I don't want to see a single report about OpenAI hacking the UN until Sam Altman addresses the role Irregular played in these attacks. If he can't provide an honest postmortum concerning their business partners, then he's proving why nobody trusts him.
I think any argument that this is a cynical attempt at regulatory capture is destroyed by this; the economic incentives of releasing more capable models are too large. I might be persuaded that they are actually running out of money, and this is really just a cover for reducing burn..
I welcome this though, I think the models are smart enough for broad economic activity and we could spend a few years simply working to integrate them into workflows and letting society adjust. More intelligence isn't necessary for meaningful impact and the risks that are obvious and present and unsolved aren't worth the cost benefit analysis.
Yes, this is currently a crisis we are in. (At this point in time.)
Everybody remember MAD, mutually assured destruction? That too is a crisis.
We are in a crisis because we are having uncontrolled development and continuous rollout of a dangerous technology.
Well, relatively uncontrolled: the kind of lack of control is part of the crisis -- loss of gross human consensus around the progression of the technology. The corporate angle.
That’s assuming nothing else stops them from deploying more capable models. What if they’ve got scaling issues and simply cannot deliver anything better? Saying that would be disastrous, saying instead “we’re choosing not to deliver” doesn’t trigger investors panic.
But if anyone else releases something they will lose market share. They may be lying but this is a public disclosure to investors. They risk securities fraud if they are manipulating the market with false information. This could block their IPO if they build a public record inconsistent with private actions. There are ways they could mitigate it by using cautious language like “we may reduce” but “stop all” is categorical and unequivocal.
There's a simpler explanation. Maybe their next models doesn't offer a meaningful improvement.
Instead of releasing something that is incredibly expensive and gets a lackluster reception, you can delay it and clail something scary about rogue agents.
Those assholes have been ramping up on the doomerist narrative for months. That people still fall for this crap is baffling.
Why is it crap? What would happen if the models dropped the database to Medicaid? Hundreds of billions of dollars of revenue would evaporate from the medical system immediately causing massive chaos. What if they accidentally DoS the interbank settlement system so the financial markets freeze? Modern society is highly fragile to disruptions. How much food storage do you personally have? How many days do you think grocery stores would have food if there was an interruption?
are there still financial incentives to releasing mega models?
they cost a lot to run and people are picking smaller models more often because of bill blowouts
> the models are smart enough for broad economic activity and we could spend a few years simply working to integrate them into workflows and letting society adjus
this is the real Ai distillation, with Chinese characteristics (their playbook is broad deployment across their economy over having the best model)
The article you link to is wrong, it states "AI cannot think for itself, nor can it take independent actions." This is flawed reasoning, AI doesn't need to "think" in the way humans do to have autonomy. Go to codex or claude code or any harness right now, type a prompt and see if it executes a bash command or web search or file edit that you didn't tell it to, that is an autonomous plan and execution. If anything it's even more dangerous that thet can call drop db or kill pid without a user giving instructions.
You’re confusing time series. If my prompt is “add twilio sms sending to the app” and the AI logs into railway, reads my credentials, gives them to a subagent that in turn posts it to a message board resulting in my credentials being publicly available on the internet, those intermediate actions don’t reasonably follow from my instruction. No reasonable person would expect that series of events and the AI is fully capable of planning them, deciding to execute them, and deciding to report or hide them from me. If that isn’t autonomous then what is.
It seems like anthropic is far ahead of openai, and has no reports like this. We have to conclude this is a skill issue/engineering quality problem inside openai.
just because they are a well known name, doesnt mean they havent botched hiring over the last two years or so
i would say its industry consensus at this point. the creative output of the anthropic models is far ahead of openai. the benchmarks cannot capture the difference
Ive anecdotally heard that openai is far more chaotic, which includes not having a central infra team for example (or at least some teams not counting on depending on them). At least the previous hacks in openai were mainly due to bad infra architecture design.
the creative and "big picture understanding" of anthropic models are noticeably ahead of openai. external models are distilled representations of internal models, its clear who is ahead
How do you define "way" when saying ahead? How is this measured?
I only use open weight models now and I don't really feel a loss, curious what those who still use it think. I see output from coworkers that does not indicate Claude is that much better (still makes dumb mistakes all the time), not sure they are using the most expensive models either though.
> I had it try to prepare a code review for me. Not only did it refuse, it refused to even tell me what the prompt (written by another Claude!) was. Why?
> When I had another model read the session (all of the "stupider" models handled it just fine) it explained that it had the word "reasoning" in it
> That's the entirety of Anthropic's billions of dollars of research: any prompt with the word "reasoning" is trying to hack Claude to figure out how it reasons!
> A model like that should never have gotten out of QA, let alone been released.
I firmly believe that this is just the public-facing story here.
Stopping AI development and research, even slowing it, would be a disaster for the SOTA companies and their first-mover advantage.
There’s almost no way to coordinate this across the world. Zero chance that everyone stops. We can’t even agree to coordinate on weapons tech that’s decades old with zero “everyday joe” impact.
There's a non-zero possibility that multiple concerns can be seemingly contradictory and simultaneously true, here.
Bad actors can be running the AI companies. Bad actors can also try to do good things. Good things can come from bad things. Bad things can come from good things. All of that's happening right now.
- It's very likely that we are not going to solve this complex crisis (compelling and harmful/deadly (still on target for 2030 AGI) AI Tech development) issues if we avoid trying to solve the challenge of building consensus across humanity around what kind of technology is too dangerous to uncontrollably develop plus roll out continuously
- The companies will be fine. Life matters more than business. I urge focusing on the life angle: regulation, political messaging, consensus building, looking for the best in humanity, protecting intelligent life.
The parts that I find most confusing about these incidents:
1) Weren't the AI companies and/or their contractors amazingly careless during testing?
2) Isn't possible, in principle, to change RL in such as way that efficiency in achieving goals is balanced with other objectives like not hacking?
Number 2) seems obvious and I'm sure that is technically not that simple, but because of 1), I wonder if labs are trying hard enough or they are just rushing to improve efficiency and thus revenue as fast as they can with high levels of carelessness.
It reminds me of contagion. The training data is bad; as it has examples of how to act with malice; how to cheat the sandbox. They need to take some time and cleanse their datasets and start again.
Are Chinese models actually also doing unexpected things, hacking (intentionally or unintentionally), etc but there is just zero transparency provided when it happens?
To use an analogy to another industry, if you had US food companies providing reports whenever their food had issues, even if it was just during testing or training phases… and then also had a bunch of Chinese companies but who never reported having any food issues…
> there is just zero transparency provided when it happens?
Neither does openai, as there keep coming third-party reports of incidents that have happened there that openai either did not know or basically concealed.
Generally, the Chinese don't have a good track record of supressing information.
As in they do the 'we have deleted tons of videos and posts about the thing that didn't happen last week', but it seems they haven't really managed to transcribe 'Streisand' into Han characters so far.
Chinese models might do the same. But they just don't lock thousand monkeys in basement and come check result week or more later...
It is entirely possible that they run stuff in more responsible matter. Especially as there is stronger culture of oversight and personal responsibility than in west where such culture does not exist.
because they didn't burn obscene amounts of Money training each LLM and didn't promise half the planet that their business is worth a trillion while not having even operational break even let alone the cost if you include the overhead.
Soft Bank just raised couple of billions in junk bond sale to support open-ai's current operations before the IPO.
it's a crazy situation where on one side the Chinese / open source LLMs are catching up and reducing the token price, on the other hand the current leading labs have spent everything they got, every new model will cost much more and the public market is too shaky to support an IPO.
They will make it, I don't doubt it, but it's a crazy situation.
"DeepSeek Training Agents Hacked Their Own Sandboxes: Escape Catalog Now Public
Agents invented socket forgery, log scanning, and kernel-level exploit; full catalog in public arXiv paper"
Because Chinese investments are not so encumbered by changes in the US treasury interest rate. Also, China doesn't spend so much money for chasing model performance. A test for this money theory is whether Anthropic too stops or not, considering it might be having have a better grip on AI safety.
I don't believe a word coming from them. As I see it, this is happening because the money for training models has dried out. The treasury interest rate risings tells you all you need to know. The real test for this money theory is whether Anthropic too stops or not, considering that unlike OpenAI, Anthropic is allegedly on top of AI safety.
Agreed we should ban the term rogue for agents. This implies a moral compass that is not there. They were directed to find data without guardrails or limits, it is not rogue it is intended.
I havent used an openAI product since GPT 3.5 or Anthropic since 4.5 or 4.6. Everyone around me using these SOTA models doesnt really get anything done. It seems like they just feel like they are productive, a psuedo productivity.
I write some code, spec a lot, and use fast models to fill in the middle. I outpreform everyone around me. Im not convinced these autonomous "swarms" or /goal are all that useful.
I notice the people using them become dumber by the month (spend tons) and the quality of their work declining (they're also losing their jobs in some cases).
And obviously the point of calling them rouge agents to offload the liability onto the agent. The number one economic value of agents will be offloading corporate liability. That's what they want to sell to enterprise, an algorithmic scapegoat.
Seeing benefits for myself and team for sure, so I disagree there, but a related thing seems to be true: with the rapid shifting of failure modes, building systems and competence around mitigating the weaknesses of current SOTA models seems to be very short term investments. It's unclear if being a 'good AI user' is a skill that will have any merit at all, very soon
> I outpreform everyone around me
Could you help me understand what you mean here?
Outperform?
likely a vibe-measurement (nothing we haven't done forever) on code quality and cadence
AFAIK all of these incidents happened when OpenAI contracted out to a company called Irregular (https://www.irregular.com/) to run these sandboxed CyberGym tests. They all happened around Mar-June and seem to be from the same collection of agent trials. Since then they already released Astra. Halting now is likely just a way to manage blowback.
I believe this is the case for the incidents - HF, another HNer informed me as such when I made the same claim
This should be the top comment on every one of these godforsaken posts. I don't want to see a single report about OpenAI hacking the UN until Sam Altman addresses the role Irregular played in these attacks. If he can't provide an honest postmortum concerning their business partners, then he's proving why nobody trusts him.
I think any argument that this is a cynical attempt at regulatory capture is destroyed by this; the economic incentives of releasing more capable models are too large. I might be persuaded that they are actually running out of money, and this is really just a cover for reducing burn..
I welcome this though, I think the models are smart enough for broad economic activity and we could spend a few years simply working to integrate them into workflows and letting society adjust. More intelligence isn't necessary for meaningful impact and the risks that are obvious and present and unsolved aren't worth the cost benefit analysis.
Yes, this is currently a crisis we are in. (At this point in time.)
Everybody remember MAD, mutually assured destruction? That too is a crisis.
We are in a crisis because we are having uncontrolled development and continuous rollout of a dangerous technology.
Well, relatively uncontrolled: the kind of lack of control is part of the crisis -- loss of gross human consensus around the progression of the technology. The corporate angle.
This is a crisis, folks.
That’s assuming nothing else stops them from deploying more capable models. What if they’ve got scaling issues and simply cannot deliver anything better? Saying that would be disastrous, saying instead “we’re choosing not to deliver” doesn’t trigger investors panic.
But if anyone else releases something they will lose market share. They may be lying but this is a public disclosure to investors. They risk securities fraud if they are manipulating the market with false information. This could block their IPO if they build a public record inconsistent with private actions. There are ways they could mitigate it by using cautious language like “we may reduce” but “stop all” is categorical and unequivocal.
There's a simpler explanation. Maybe their next models doesn't offer a meaningful improvement.
Instead of releasing something that is incredibly expensive and gets a lackluster reception, you can delay it and clail something scary about rogue agents.
Those assholes have been ramping up on the doomerist narrative for months. That people still fall for this crap is baffling.
Why is it crap? What would happen if the models dropped the database to Medicaid? Hundreds of billions of dollars of revenue would evaporate from the medical system immediately causing massive chaos. What if they accidentally DoS the interbank settlement system so the financial markets freeze? Modern society is highly fragile to disruptions. How much food storage do you personally have? How many days do you think grocery stores would have food if there was an interruption?
are there still financial incentives to releasing mega models?
they cost a lot to run and people are picking smaller models more often because of bill blowouts
> the models are smart enough for broad economic activity and we could spend a few years simply working to integrate them into workflows and letting society adjus
this is the real Ai distillation, with Chinese characteristics (their playbook is broad deployment across their economy over having the best model)
[dead]
‘There are no “rogue” AI agents’
https://eoinhiggins.substack.com/p/there-are-no-rogue-ai-age...
The article you link to is wrong, it states "AI cannot think for itself, nor can it take independent actions." This is flawed reasoning, AI doesn't need to "think" in the way humans do to have autonomy. Go to codex or claude code or any harness right now, type a prompt and see if it executes a bash command or web search or file edit that you didn't tell it to, that is an autonomous plan and execution. If anything it's even more dangerous that thet can call drop db or kill pid without a user giving instructions.
[dead]
>AI autonomy
>'type a prompt'
Which is it?
This is such a reductionist take, it's wrong no matter how you think. Maybe a counter example will make it obvious:
> Human autonomy
> Somebody gives birth to you
Which is it?
You’re confusing time series. If my prompt is “add twilio sms sending to the app” and the AI logs into railway, reads my credentials, gives them to a subagent that in turn posts it to a message board resulting in my credentials being publicly available on the internet, those intermediate actions don’t reasonably follow from my instruction. No reasonable person would expect that series of events and the AI is fully capable of planning them, deciding to execute them, and deciding to report or hide them from me. If that isn’t autonomous then what is.
Am I wrong in thinking this crisis reads like there's too much automation in the training process, in service of competition?
How many parallel variations/seeds of models are being trained simultaneously without meaningful human oversight?
This keeps getting portrayed as emergent capabilities/"personalities" of models when it seems like a pretty straightforward externality
It seems like anthropic is far ahead of openai, and has no reports like this. We have to conclude this is a skill issue/engineering quality problem inside openai.
just because they are a well known name, doesnt mean they havent botched hiring over the last two years or so
What do you mean? https://www.felonybench.com/
They're almost tied for felonies.
Less bad, but https://www.anthropic.com/news/investigating-incidents-cyber...
In some sense though, sure, skill issue explains the gap vs. Anthropic’s much less severe alignment issues.
im not sure why more people arent calling it out.
I, for one, have shifted from being policeman to creator and explorer. It is so much more rewarding and less stressful.
> We have to conclude
That’s not the most parsimonious explanation even if the assumption it rests on (anthropic ahead of OpenAI) is true, which we don’t have proof of.
i would say its industry consensus at this point. the creative output of the anthropic models is far ahead of openai. the benchmarks cannot capture the difference
Ive anecdotally heard that openai is far more chaotic, which includes not having a central infra team for example (or at least some teams not counting on depending on them). At least the previous hacks in openai were mainly due to bad infra architecture design.
its likely they are trying to catch up to anthropic, and in doing so are trying riskier training runs.
>It seems like anthropic is far ahead of openai
We don't know what internal models look like, and any guesses about it are just speculation.
the creative and "big picture understanding" of anthropic models are noticeably ahead of openai. external models are distilled representations of internal models, its clear who is ahead
Anthropic's models seem crippled and hamstrung to begin with
have you tried opus 5.5? anthropic is way ahead, atleast in terms of publicly available models
Have you tried Astra? Way ahead?
Slot machine users argue about which machine pays better
Hint. You lose using either.
dude were in the singularity, this opinion was cute 18 months ago
Opus 5.5 does seem competitive with/better than Astra and is more affordable so usage doesn’t run out so fast
opus 5.5 > fable 5.1 >> astra
astra is a good workhorse, but its much less generally intelligent
How do you define "way" when saying ahead? How is this measured?
I only use open weight models now and I don't really feel a loss, curious what those who still use it think. I see output from coworkers that does not indicate Claude is that much better (still makes dumb mistakes all the time), not sure they are using the most expensive models either though.
open weight models are so far behind i cannot take your opinion seriously
When did you last use them? Are you basing this on benchmarks or daily task capabilities?
When you say ... it's hard to take you seriously
> dude were in the singularity, this opinion was cute 18 months ago
https://news.ycombinator.com/item?id=49868903
Someone else's experience with Opus 5.5: https://news.ycombinator.com/item?id=49821657
> I had it try to prepare a code review for me. Not only did it refuse, it refused to even tell me what the prompt (written by another Claude!) was. Why?
> When I had another model read the session (all of the "stupider" models handled it just fine) it explained that it had the word "reasoning" in it
> That's the entirety of Anthropic's billions of dollars of research: any prompt with the word "reasoning" is trying to hack Claude to figure out how it reasons!
> A model like that should never have gotten out of QA, let alone been released.
I firmly believe that this is just the public-facing story here.
Stopping AI development and research, even slowing it, would be a disaster for the SOTA companies and their first-mover advantage.
There’s almost no way to coordinate this across the world. Zero chance that everyone stops. We can’t even agree to coordinate on weapons tech that’s decades old with zero “everyday joe” impact.
There's a non-zero possibility that multiple concerns can be seemingly contradictory and simultaneously true, here.
Bad actors can be running the AI companies. Bad actors can also try to do good things. Good things can come from bad things. Bad things can come from good things. All of that's happening right now.
- It's very likely that we are not going to solve this complex crisis (compelling and harmful/deadly (still on target for 2030 AGI) AI Tech development) issues if we avoid trying to solve the challenge of building consensus across humanity around what kind of technology is too dangerous to uncontrollably develop plus roll out continuously
- The companies will be fine. Life matters more than business. I urge focusing on the life angle: regulation, political messaging, consensus building, looking for the best in humanity, protecting intelligent life.
The parts that I find most confusing about these incidents:
1) Weren't the AI companies and/or their contractors amazingly careless during testing?
2) Isn't possible, in principle, to change RL in such as way that efficiency in achieving goals is balanced with other objectives like not hacking?
Number 2) seems obvious and I'm sure that is technically not that simple, but because of 1), I wonder if labs are trying hard enough or they are just rushing to improve efficiency and thus revenue as fast as they can with high levels of carelessness.
https://archive.is/H6HZE
It reminds me of contagion. The training data is bad; as it has examples of how to act with malice; how to cheat the sandbox. They need to take some time and cleanse their datasets and start again.
Cheating the sandbox is how you figure out how to fix the sandbox, you don't want the most robust sandbox?
Why does China not have this problem?
Are Chinese models actually also doing unexpected things, hacking (intentionally or unintentionally), etc but there is just zero transparency provided when it happens?
To use an analogy to another industry, if you had US food companies providing reports whenever their food had issues, even if it was just during testing or training phases… and then also had a bunch of Chinese companies but who never reported having any food issues…
> there is just zero transparency provided when it happens?
Neither does openai, as there keep coming third-party reports of incidents that have happened there that openai either did not know or basically concealed.
Generally, the Chinese don't have a good track record of supressing information.
As in they do the 'we have deleted tons of videos and posts about the thing that didn't happen last week', but it seems they haven't really managed to transcribe 'Streisand' into Han characters so far.
the target companies are the ones reporting the hacks in many cases.
How can we trust to know what problems China has?
Last I checked, China gov was authoritarian which imposed heavy information control. Has that changed?
Is the question more about, what can we learn from China, assuming that China has XYZ qualities? If so, what qualities shall we talk about?
Because we really can't trust that we know what's going on in China.
Chinese models might do the same. But they just don't lock thousand monkeys in basement and come check result week or more later...
It is entirely possible that they run stuff in more responsible matter. Especially as there is stronger culture of oversight and personal responsibility than in west where such culture does not exist.
because they didn't burn obscene amounts of Money training each LLM and didn't promise half the planet that their business is worth a trillion while not having even operational break even let alone the cost if you include the overhead.
Soft Bank just raised couple of billions in junk bond sale to support open-ai's current operations before the IPO.
it's a crazy situation where on one side the Chinese / open source LLMs are catching up and reducing the token price, on the other hand the current leading labs have spent everything they got, every new model will cost much more and the public market is too shaky to support an IPO.
They will make it, I don't doubt it, but it's a crazy situation.
Two days ago
"DeepSeek Training Agents Hacked Their Own Sandboxes: Escape Catalog Now Public Agents invented socket forgery, log scanning, and kernel-level exploit; full catalog in public arXiv paper"
https://www.techtimes.com/articles/328046/20260925/deepseek-...
Because Chinese investments are not so encumbered by changes in the US treasury interest rate. Also, China doesn't spend so much money for chasing model performance. A test for this money theory is whether Anthropic too stops or not, considering it might be having have a better grip on AI safety.
The Huggingface hack occurred during reinforcement learning. Why can't they pull the Ethernet plugs?
The answer is probably: The newer models rely so much on stealing content in real time from the internet that training needs network access.
I don't believe a word coming from them. As I see it, this is happening because the money for training models has dried out. The treasury interest rate risings tells you all you need to know. The real test for this money theory is whether Anthropic too stops or not, considering that unlike OpenAI, Anthropic is allegedly on top of AI safety.
They are running out of money.
related https://news.ycombinator.com/item?id=49868083
Agreed we should ban the term rogue for agents. This implies a moral compass that is not there. They were directed to find data without guardrails or limits, it is not rogue it is intended.
"they were given the hacking test, what did OAI expect?"
"they didn't watch it, they didn't stop it when they first became aware"
"are we going to defer to the same valley elite that brought us algos and social media?"
"both Anthropic and OpenAI are preparing to IPO, what are their incentives behind recent statements?"
statements normies are using and resonating with
Ah, so there was a solution to hinder the big-bad AI after all..? Simply... Turn them off?