Should probably have a (2018) or (2024) (latest revision) on the title, especially given the current buzz surrounding AI and security/existential threats.
This paper diagnosed the disease in 2018. Eight years later none of the recommendations happened. Norms, collaboration, responsible disclosure - none of it materialized in any structural way.
I think the reason is that the paper frames malicious AI use as a policy problem, recommending social solutions. It's actually an architecture problem.
Every generation of computing has hit a version of this. Programs could write anywhere in memory - we added protected memory. Programs could hog the CPU - we added preemptive multitasking. Desktop apps could call any OS function - the iPhone sandboxed them. Nobody asked programs to please behave, the containment actually went into the infrastructure.
AI skipped that step entirely. We went straight to open-ended agents with broad permissions and tried to make them safe through alignment and prompting. I've been researching this for the past year and I think alignment is necessary but not sufficient, because the intelligence increasingly isn't in the model. It's in the substrate - the harness, the domain knowledge, the tooling around the model. I actually measured this on real coding tasks: Sonnet with a code-derived index outperformed the frontier model (Opus 5.8) exploring on its own, and the top-tier model (Fable) refused the real work entirely! The cheap model with the right rig beat the expensive model without one. https://safebots.ai/matchup.html
If that's true then aligning the model doesn't solve the problem. A bad actor who can't get the best model just uses Sonnet. Or Llama. Or Kimi. The weights have already leaked and bits don't degrade - you can't recall them the way you can stop manufacturing CFCs.
So what do you actually do? Same thing that worked for CFCs. You gotta first build the safe version — in this case, declarative workflows running in sealed compute environments — and prove it handles 99% of actual use cases at lower cost. Let it win commercially. Then regulate the dangerous version. DuPont developed HFC refrigerants first. The Montreal Protocol became possible BECAUSE of that. The ban became politically viable because the alternative already existed.
Governments have repeatedly shown that instead of investing in developing robust, secure, open source software and hardware - they're interested in something very different.
When these same governments get the whistle blown on them for flagrantly violating laws, they'll ruin the lives of the whistle blowers and all the FAANG boys who could've easily protested and done actual good for once, did nothing.
Forgive me, it's not architecture. What's malicious is your culture and way of life.
Alignment is the architectural solution. Make it so the model can't misbehave. Yet many people here deride it as tainting the model, claiming "Whose values is it aligned to?" Sandboxes are a last ditch layer. They fail, as we see.
If it's not dangerous it's also not useful, simple as that. For example if you train a model for cybersecurity, it can be used for both attack and defense. And almost every use is like that. Alignment is fundamentally flawed as a concept, it's a pie in the sky. Let alone the perverse version of it by crazy AI "safety" people that in practice means "the model does what I want, only for the people I allow".
It's not possible to stop the model from misinterpreting the instructions either (the most lax interpretation of alignment) because the instructions are not formally specified. You have to train the "common sense" into it, which is subjective and all issues above apply to it.
The “make it so the model can't misbehave” part is interesting. Maybe the goal isn't to make the model perfectly aligned, but to make misalignment have a very small blast radius. That feels like a more achievable engineering problem.
Embed a constitution they can't override. Project bad outputs to their nearest acceptable one. If we have to stop model development to ensure we can do it, so be it.
Should probably have a (2018) or (2024) (latest revision) on the title, especially given the current buzz surrounding AI and security/existential threats.
This paper diagnosed the disease in 2018. Eight years later none of the recommendations happened. Norms, collaboration, responsible disclosure - none of it materialized in any structural way.
I think the reason is that the paper frames malicious AI use as a policy problem, recommending social solutions. It's actually an architecture problem.
Every generation of computing has hit a version of this. Programs could write anywhere in memory - we added protected memory. Programs could hog the CPU - we added preemptive multitasking. Desktop apps could call any OS function - the iPhone sandboxed them. Nobody asked programs to please behave, the containment actually went into the infrastructure.
AI skipped that step entirely. We went straight to open-ended agents with broad permissions and tried to make them safe through alignment and prompting. I've been researching this for the past year and I think alignment is necessary but not sufficient, because the intelligence increasingly isn't in the model. It's in the substrate - the harness, the domain knowledge, the tooling around the model. I actually measured this on real coding tasks: Sonnet with a code-derived index outperformed the frontier model (Opus 5.8) exploring on its own, and the top-tier model (Fable) refused the real work entirely! The cheap model with the right rig beat the expensive model without one. https://safebots.ai/matchup.html
If that's true then aligning the model doesn't solve the problem. A bad actor who can't get the best model just uses Sonnet. Or Llama. Or Kimi. The weights have already leaked and bits don't degrade - you can't recall them the way you can stop manufacturing CFCs.
So what do you actually do? Same thing that worked for CFCs. You gotta first build the safe version — in this case, declarative workflows running in sealed compute environments — and prove it handles 99% of actual use cases at lower cost. Let it win commercially. Then regulate the dangerous version. DuPont developed HFC refrigerants first. The Montreal Protocol became possible BECAUSE of that. The ban became politically viable because the alternative already existed.
I've been building this alternative for the past 8 months: https://safebots.ai/about
Governments have repeatedly shown that instead of investing in developing robust, secure, open source software and hardware - they're interested in something very different.
When these same governments get the whistle blown on them for flagrantly violating laws, they'll ruin the lives of the whistle blowers and all the FAANG boys who could've easily protested and done actual good for once, did nothing.
Forgive me, it's not architecture. What's malicious is your culture and way of life.
Why would a bad actor volunteer to run a "declarative workflow running in sealed compute environment" when they could just not do that?
ai slop comment and my eyes glaze over
Alignment is the architectural solution. Make it so the model can't misbehave. Yet many people here deride it as tainting the model, claiming "Whose values is it aligned to?" Sandboxes are a last ditch layer. They fail, as we see.
If it's not dangerous it's also not useful, simple as that. For example if you train a model for cybersecurity, it can be used for both attack and defense. And almost every use is like that. Alignment is fundamentally flawed as a concept, it's a pie in the sky. Let alone the perverse version of it by crazy AI "safety" people that in practice means "the model does what I want, only for the people I allow".
It's not possible to stop the model from misinterpreting the instructions either (the most lax interpretation of alignment) because the instructions are not formally specified. You have to train the "common sense" into it, which is subjective and all issues above apply to it.
> Make it so the model can't misbehave
Not possible. They can chase the models with whack a mole tuning for obvious stuff, but there's always a way to extract what you want from the model.
The “make it so the model can't misbehave” part is interesting. Maybe the goal isn't to make the model perfectly aligned, but to make misalignment have a very small blast radius. That feels like a more achievable engineering problem.
> Make it so the model can't misbehave.
How do you figure? I haven't met anyone who thinks that's possible. It seems clear to me that it is not possible.
Embed a constitution they can't override. Project bad outputs to their nearest acceptable one. If we have to stop model development to ensure we can do it, so be it.