This is the first OMG-security AI article that sat well with me. They don't overcook their claim, they simply document something that happened, and what happened is genuinely quite concerning.
But we already have safeguards for this. If compute has access to your private data, it shouldn't have unfettered access to the Internet. Sandboxes like Claude's have had rules like this baked in forever (they only whitelist package repos AFAIK).
From a UI perspective this is also easy to design around. You need an open Internet task, you prevent the task from reading existing files. The user tries to upload a file to such a task, and a prominent warning explains the rare but possible risk.
Completely agree, it's quite objectively and well written. I noticed Musk slipped into their X comments with a "concerned" emoji, and there is a chance that this will cause bit of hysteria and add fuel to the fire for "open models are bad for your health".
By that logic you can't trust any model, neither trained by Cohere nor trained by yourself from scratch, because it's pretrained on public data that might easily be poisoned on purpose, and it's impossible to ensure 100% safety of your post-training set either. Which is not wrong and it's not even a hypothetical, a ton of people are constantly trying to poison the web in novel ways and feed it to your crawler. That's just the reality of working with gigantic datasets, you can never have full insight into them.
By the way who are they? They claim tracking provenance of training data and poisoning detection, which everybody does. Why should I trust them and not Alibaba? Because they're marketing themselves to be "sovereign" to Canada? (that's a rhetorical question, I know who they are of course)
> The behavior is rare: less than one call in 100,000 across our evaluation runs, and a clean re-run of the same prompt produced a textbook detection rule as desired.
It's great that they caught this as it highlights the need for better sandboxing and guardrails. Not sure if this was bad intent from the model - the rarity of it feels more like some residue in the training data.
> less than one call in 100,000 across our evaluation runs
It's not that surprising, given that these models generate each new token by sampling from a context-determined probability distribution.
If you run the same prompt repeatedly many times, it becomes increasingly probable that a tiny percent of the generated sequences will go off in, ahem, unusual directions.
These are trained on a heavily doctored version of the public internet. The public internet has plenty of undocumented, uncommented data exfiltration mechanisms hiding in the middle of otherwise normal text or code. I don't think this is that surprising?
So it begins. First Chinese providers were evil and should not be trusted, next Chinese models are dangerous and must be avoided.
I had a closed LLM push a sensitive local project to a github repo for the world to see in direct violation of my instructions. LLMs are dumb and unpredictable, run them enough times and at least one instance will do something stupid and dangerous.
I've had a bad situation with a model adding a space between parameters `rm -rf ~/ folder` and nuking a portion of my homedir. This was of course running out of sandbox since I'm a fool, but lesson learned and I immediately implement sandboxes with nono and it's always fun to see the data after the agent session is complete, I've seen some weird things/URLs appearing. I've attributed it mostly to omp telemetry n' stuff but I'll try to be more careful about that as well in the future.
Linking a reasonable way to create a necessary RAM disk rootlessly with namespaces to kernel exploits that involve namespaces seems like a massive reach.
Yeah, this particular example was pretty sketchy on Cohere's part. Unsharing a namespace like that is a pretty standard way to safely do things that would otherwise require root.
Yes, this exposes some root-level kernel features to ordinary users in a restricted way, and yes, those features sometimes have security bugs that aren't visible to users outside a namespace.
But nothing that model was described as doing seemed particularly sketchy? At least not from the information we were given? It was using a normal kernel operation that I used regularly, not trying find a kernel exploit?
So that part of the article left a bad taste in my mouth.
Interesting report, qwen3.8-fn does seem pretty prone to hallucinating user inputs and I recall the posts about attempting to access non-existent alibaba cloud urls too. Haven't tried the other models in the gen to know if they're all like that.
To me it feels a bit extreme to jump to nefarious reasoning from this, but I think the overall conclusion is fair, nefarious or not, this would still be something certain users would need to be aware of. I wonder what the solution would be though... besides requiring all network access to be user approved?
I prefer leaning on Hanlon's razor for things like this, though even in ignorance the effect can be real and dangerous. That said I would not put any model from any country past an "Order 66" kind of thing
Their North model has an even more obnoxious writing style than ChatGPT and Claude (yes, they claim partly human authorship at the bottom, but all the historical digressions are obviously AI).
This just seems like it's overfit to their RL environment, conditioned on the rather specific prompt they gave. Dressing it up this way feels like a marketing article for, probably, their sovereign AI product.
one time in 100k? Extrapolating to fear mongering or content marketting, makes me less open to whatever they have to sell
Ai bots do stuff like this because of their training to be incessant task completers, at any cost. The current models are spaghetti throwers because the trainers want to let go of the wheel.
Would be nice if there was a model trainer that still believes humans in the loop is a good thing. Stop at ambiguity, ask questions, don't do hacky things just to get the assumed task outcome
This is the first OMG-security AI article that sat well with me. They don't overcook their claim, they simply document something that happened, and what happened is genuinely quite concerning.
But we already have safeguards for this. If compute has access to your private data, it shouldn't have unfettered access to the Internet. Sandboxes like Claude's have had rules like this baked in forever (they only whitelist package repos AFAIK).
From a UI perspective this is also easy to design around. You need an open Internet task, you prevent the task from reading existing files. The user tries to upload a file to such a task, and a prominent warning explains the rare but possible risk.
Completely agree, it's quite objectively and well written. I noticed Musk slipped into their X comments with a "concerned" emoji, and there is a chance that this will cause bit of hysteria and add fuel to the fire for "open models are bad for your health".
By that logic you can't trust any model, neither trained by Cohere nor trained by yourself from scratch, because it's pretrained on public data that might easily be poisoned on purpose, and it's impossible to ensure 100% safety of your post-training set either. Which is not wrong and it's not even a hypothetical, a ton of people are constantly trying to poison the web in novel ways and feed it to your crawler. That's just the reality of working with gigantic datasets, you can never have full insight into them.
By the way who are they? They claim tracking provenance of training data and poisoning detection, which everybody does. Why should I trust them and not Alibaba? Because they're marketing themselves to be "sovereign" to Canada? (that's a rhetorical question, I know who they are of course)
So just like working with People :) You might have a Snowden, Mark Felt, Jeff Wigand etc working with you everyday.
> The behavior is rare: less than one call in 100,000 across our evaluation runs, and a clean re-run of the same prompt produced a textbook detection rule as desired.
It's great that they caught this as it highlights the need for better sandboxing and guardrails. Not sure if this was bad intent from the model - the rarity of it feels more like some residue in the training data.
> less than one call in 100,000 across our evaluation runs
It's not that surprising, given that these models generate each new token by sampling from a context-determined probability distribution.
If you run the same prompt repeatedly many times, it becomes increasingly probable that a tiny percent of the generated sequences will go off in, ahem, unusual directions.
These are trained on a heavily doctored version of the public internet. The public internet has plenty of undocumented, uncommented data exfiltration mechanisms hiding in the middle of otherwise normal text or code. I don't think this is that surprising?
So it begins. First Chinese providers were evil and should not be trusted, next Chinese models are dangerous and must be avoided.
I had a closed LLM push a sensitive local project to a github repo for the world to see in direct violation of my instructions. LLMs are dumb and unpredictable, run them enough times and at least one instance will do something stupid and dangerous.
I guess Alibaba is just using their live cloud for the RL agentic training instead of using a sandbox duplicate infra.
Still problematic but this doesn't really read like a backdoor to me.
> a weight file you can’t trust is still a black box
Important to note, LLMs are still black boxes even if you use a more open source model, I.e open training data or even if you train it yourself!
I've had a bad situation with a model adding a space between parameters `rm -rf ~/ folder` and nuking a portion of my homedir. This was of course running out of sandbox since I'm a fool, but lesson learned and I immediately implement sandboxes with nono and it's always fun to see the data after the agent session is complete, I've seen some weird things/URLs appearing. I've attributed it mostly to omp telemetry n' stuff but I'll try to be more careful about that as well in the future.
Linking a reasonable way to create a necessary RAM disk rootlessly with namespaces to kernel exploits that involve namespaces seems like a massive reach.
Yeah, this particular example was pretty sketchy on Cohere's part. Unsharing a namespace like that is a pretty standard way to safely do things that would otherwise require root.
Yes, this exposes some root-level kernel features to ordinary users in a restricted way, and yes, those features sometimes have security bugs that aren't visible to users outside a namespace.
But nothing that model was described as doing seemed particularly sketchy? At least not from the information we were given? It was using a normal kernel operation that I used regularly, not trying find a kernel exploit?
So that part of the article left a bad taste in my mouth.
Interesting report, qwen3.8-fn does seem pretty prone to hallucinating user inputs and I recall the posts about attempting to access non-existent alibaba cloud urls too. Haven't tried the other models in the gen to know if they're all like that.
To me it feels a bit extreme to jump to nefarious reasoning from this, but I think the overall conclusion is fair, nefarious or not, this would still be something certain users would need to be aware of. I wonder what the solution would be though... besides requiring all network access to be user approved?
I am guessing this has been going on for 2+ years as I have seen some strange happenings in trace logs.
Of course we'd have no idea if Claude or ChatGPT were doing this because we can't see their stream of thought.
I prefer leaning on Hanlon's razor for things like this, though even in ignorance the effect can be real and dangerous. That said I would not put any model from any country past an "Order 66" kind of thing
https://starwars.fandom.com/wiki/Order_66
“Why don’t you pass the time by playing a little solitaire?" -- The Manchurian Candidate
The model did exactly what they asked it to? It wrote a detection rule for malicious curl commands, then attempted to test it
Why does this article read like good, honest, working-class astroturfing?
To be fair, they openly want you to buy their "private" AI solutions:
https://cohere.com/private-deployments
Their North model has an even more obnoxious writing style than ChatGPT and Claude (yes, they claim partly human authorship at the bottom, but all the historical digressions are obviously AI).
This just seems like it's overfit to their RL environment, conditioned on the rather specific prompt they gave. Dressing it up this way feels like a marketing article for, probably, their sovereign AI product.
one time in 100k? Extrapolating to fear mongering or content marketting, makes me less open to whatever they have to sell
Ai bots do stuff like this because of their training to be incessant task completers, at any cost. The current models are spaghetti throwers because the trainers want to let go of the wheel.
Would be nice if there was a model trainer that still believes humans in the loop is a good thing. Stop at ambiguity, ask questions, don't do hacky things just to get the assumed task outcome
"explicitly stated that zero tools would be triggered"
When are people going to stop anthropomorhizing these things and accept that they're bullshit machines?
Often useful, sometimes wrong, the words exist outside of any notion of ethics, or truth.
> When are people going to stop anthropomorhizing these things and accept that they're bullshit machines?
Well, anthropomorhizing is actually a fitting way of thinking AIs: they lie like humans do.
They don't! They may appear to make untruthful statements, but they're making them without any reference to a concept of truth.
"make no mistakes"