Astra can, without tooling, determine what games small networks are trained on, can edit the weights of networks directly to improve outcomes, and can hand edit a small qwen model to improve its math skills by like 5x.
I'm shook. If you'd asked me this morning whether LLMs have native understanding of their weights that is less than human <-> neuron understanding, the same as human or more than human, I would have said "same to less" depending on definitions.
No human can go edit a brain to become better at math. This is just very, very surprising to me and extremely interesting. I would guess this upsets quite a number of interp theories and philosophical takes on what's going on with these networks.
I think the "taught qwen math" part is not exactly correct though, even if the post makes it sound like that. Qwen already knew math, but it was bugging out and getting into an infinite loop when it tried to solve the problems (85 of its first 96 answers hit the token limit). With numpy and pytorch, Astra found where the loop was happening and fixed it.
They also tried the same with Gemma 3 270M, which doesn't loop and is just bad at math, and Astra couldn't improve it. Still impressive, but it's more in the realm of what I'd expect, given labs are probably training their models on ML research and interpretability work.
I agree, this is important context. They call it out explicitly. However, even identifying clusters that specialize in certain fields automatically and then finding and fixing learning “errors” is incredibly impressive to me. Especially for medium sized models between 20B an 70B parameters, this could yield huge capability and efficiency improvements, not to speak of the potentially enormous downstream effects that come with interpretability.
Oh for sure, it's still really cool. A few months ago I had agents do some interpretability work on GPT-2 small, reproducing the circular weekday/month features from Engels et al., and it's wild how far you can get with just a bit of prompting, given I know very little about LLM internals myself.
LLMs have been used to investigate and improve LLMs for a long time. Drawing parallels to human brains and neurons isn't all that interesting other than as a philosophical abstraction.
The work done in this experiment could have also been done by a sufficiently motivated human with enough time to explore, though obviously we would need more tools to visualize and track things. There have been a lot of interesting LLM introspection tools that allow you to peek into the model and see what's being activated and run experiments to trigger it.
Are you familiar with abliterated or "uncensored" models where weights have been modified to avoid triggering refusal patterns? You take an LLM, create test cases that trigger the refusal pattern, observe weights involved in that response, then tweak those weights until the response is reduced. This is the same type of work, applied to a specific type of error that was impeding math within Qwen. That work can also be done with an LLM in a harness running the experiments. The amazing part here is that Astra was large enough to walk through the problem without other tooling doing it experimentally.
So I don't think your analogy of a human modifying a neuron is quite applicable. LLM weights are not neurons and they've been inspectable, by humans or LLMs in a harness, for a long time.
> The amazing part here is that Astra was large enough to walk through the problem without other tooling doing it experimentally.
To me “amazing” is not the right level of speechlessness. I mean, an LLM is still fundamentally a next token predictor, right? So what does this imply? That weights have some universal properties? E.g., “breakout game weights” always look the same regardless of training and model? Also, how does this meta-understanding of the weights end up in the weights in the first place? Idk, this is mind-blowing to me.
At this point it's either wrong or meaningless to call LLMs "next token predictors". Yes, pretraining involves predicting the next token. But after that you're in the world of RLHF, RLVR, and other post-training.
A model is _sort of_ designed to predict the next token in a string of text. But it's equally accurate to say it's designed to edit code until it passes a suite of unit tests.
Yes, they generate sequentially, so "next token predictor" isn't technically incorrect (which is why I chose the word "meaningless"). But it's pretty clear there's a mechanism inside them that allows them to "plan" what they generate. I might as well call you a next word predictor.
Yes, I didn’t mean that literally. Since “attention is all you need” it’s more like “predict the full output in full for this blob of input text”. That doesn’t change the “wtf is going on with the weight understanding bit” in what I wrote.
My understanding is that the underlying part is a next token predictor and that's what is "actually" editing the code but the part that decides what to do (to achieve a successful unit test result for example) is a mix of all the things including post training.
Maybe I'm missing something, but isn't training an LLM very well documented? Why wouldn't an LLM be able to read those docs just like it would read Java Spring docs?
Interesting, it's similar to the way we used to make cheat codes on the 8-bits: dump memory, run test, compare. You had to loop 3-4 times but you'd find the "weight" you were looking for and could fix it (for infinite lives/ammo etc) or increase/decrease.
This is the "recursive self-improvement" they've been warning about.
I've always laughed at that phrase, thinking "haven't these things been improving themselves since the very beginning? This is all just iterative gradient descent, each step is a 'self-improvement'", but this really is different.
I wonder if we'll ever discover a new branch of linear algebra or something that allows us to find the optimal weights much more efficiently? Maybe AI will help us discover it?
The math example on Qwen was based on 50 batched iterations, so the model was able to evaluate it's changes on downstream tasks. And the fix was essentially to remove a direction in activation space that causes looping instead of an exit, this was already known to be a thing(all of the jailbreak/unleashed models do this) following Arditi et al. (Refusal in language models is mediated by a single direction, 2024)
So it's not able to insert new knowledge, but it's able to adjust the routing/looping/weighing of existing one.
As I understand, it can replace post-training like LoRA/GRPO, but if it'd cheaper the authors don't go into
Merging a model and a LoRa is insanely cheap, so assuming it didn't use any GPUs to do inspection etc, and just looked at a file on a FS, then I would be willing to say... It's cheaper than training a LoRa if that's the case, assuming Astra didn't take several weeks?
I’m just a layman but this feels like a big discovery on par with attention is all you need. For an LLM to have such incredible intuition of weights feels very emergent and unanticipated?
I’m very much surprised but also not. I recently gave Astra a 3D model file and asked Astra to build a viewer for it, and it figured out the data structures in no time. What I mean to say is that AI is insanely good at things that we humans would never do since the work would be too monotonous.
I wouldn’t mind if AI progress would stop right here and now, but I fear it will continue, and increase inequality in society which is contrary to what the CEOs say (as usual).
> What I mean to say is that AI is insanely good at things that we humans would never do since the work would be too monotonous.
In my opinion AI is superior to humans when it comes to pattern recognition in in large amounts of data. For example humans can only hold like 7 distinct numbers in the short term memory and you would require extra context-management (aka writing things down, storing them in a system entry by entry) while the AI can do this effortlessly with huge datasets.
I'd argue this is the core part that enables all this superhuman intelligence.
I think the latest round of models is about to do to CAD the same thing that has been happening to development. I don't know if CAD designers have picked up on this. What happened to dev is probably going to happen to CAD designers soon. I've been having Opus 5.5 design parametric openscad models, and it can do some rather amazing work.
Yes! The page posts Astra's reasoning, and it's perfectly intelligible. A clever human with background in mechanistic interpretability could have done pretty much everything in these experiments.
Too much inequality in society is a mortal danger to 1%.
That’s the beauty of the democratic capitalism system. If a government isn’t totalitarian, it naturally seeks to preserve stability thus having to elevate common people, lest they would cease to form a civilized group or simply vote away the politicians.
99% rule the 1% through elections.
There is no better system of checks and balances where opposite engines of greed vector sum to faster and faster abundance for everyone.
Is this satire? Or haven't you noticed that what actually happens is that the politicians are all owned by the elite 1%, and they allow you to choose between politician A and politician B who both take their orders from the elite to give you the illusion that the government is the 99% ruling the 1%.
Then you must introduce necessary reforms or perish as Neanderthals of political system.
In any case the system works as intended
It will be very interesting to see if USA swims or drowns from appropriately afar.
I have already supplied myself with plenty of roasted corn and a high speed internet connection so I can stream the political car crash at the highest bitrate available to humankind.
Is HN now against western capitalism? I'm confused...
That comment may have been overly preachy, tone deaf[0], and out of place, but it doesn't seem wrong, per se. Or am I missing something?
[0] e.g. elections themselves are so prone to manipulation (exhibit a: Dominion voting machines), previously trusted institutions now lack integrity, the insane power of well-funded NGO networks, etc.
I fear western capitalism has lost its guide rails. We are seeing more and more centralization of wealth, along with unlimited election spending. Its a dark path.
Over my ~9 years reading HN daily, I'd say most of the HN community broadly accepts there is no perfect system, just different sets of trade offs, but that history has shown generally democratic systems based on enlightenment values (uniform rule of law, respect for individual civil & property rights, free markets, etc) usually net a better average outcome over the long-run - despite still being very far from perfect.
To paraphrase Winston Churchill, it's the worst system... except for all the others (which are even worse). :-)
This does feel big, but the trillion-dollar question: is this trained by OpenAI and can other LLMs do it.
It’s no secret that these big labs see “recursive self improvement” as a primary goal. It seems natural that the models would be explicitly trained to excel at all sorts of ML related tasks as a result. Is this one of those tasks, or is this emergent?
This seems emergent to me. LLMs are not trained directly on how to read weights, because humans do not understand raw weights, so there is no training data. They're trained on how to use the tools we have like writing python, examining training output and iterating, not writing whole weights from scratch.
In cases where Astra succeeded, it succeeded using standard techniques also used by humans when they read and write neural network weights. It's a rare skill, mostly used for fun (e.g. https://blog.janestreet.com/can-you-reverse-engineer-our-neu... ) and basic research rather than profit, but Astra is big enough to have learned many rare skills.
In a self-supervised setting you don't need ground truth data, as long as you have a proxy task that you can train on which forces the model to learn something useful. Like cutting out bits of images and telling the model to reconstruct the gaps (i.e. MAEs).
There are also techniques to watermark model weights to confirm what training data were used: https://arxiv.org/html/2512.17075 I'm not an expert in this domain, but it seems like there are lots of pieces that could be used to (a) train a model to infer whether some data were trained on and (b) train a model to understand which weights are relevant for a particular training task in a more explainable way than backprop/gradient descent.
I tried it in the Claude web version with Opus 5.5 Medium and it also identified Breakout, even though in one of its explanations it seems to have mixed up some indexes, but the overall reading of the weights was right.
That's also matching what I see when I work with Opus 5.5 to optimize models. It seems to understand what individual layers are for, and can thus sometimes suggest optimizations, where I as a layman would be completely clueless.
The latest models are performing amazingly. Because of this, I disagree with popular comments saying that eg. DeepSeek is good enough, AI-assisted development was where the real value was at, etc.
We see huge progress at the frontier, while the open competition is around GPT-5.5's level, and that model was released in April. Let's see how things develop in the future.
The latest models are performing amazingly. Because of this, I disagree with popular comments saying that eg. DeepSeek is good enough, AI-assisted development was where the real value was at, etc.
Completely agree. I often think that those sorts of comments grow out of an understandable lack of imagination. We’ve never had tools like this, and we’ve never had tools that evolved so rapidly, so we easily can get stuck thinking that once a model is good enough to do today’s tasks, that’s really all you need. But the next generation of models often completely obviates the need for those tasks entirely. The software engineers of 2031 won’t be doing tasks anything like the software engineers of today, just like we aren’t doing things like we did in 2021.
(Yes, yes, I know some of you still hand write all your code and think LLMs are today’s crypto bullshit and this is all going to blow over. Not talking to you.)
> It seems natural that the models would be explicitly trained to excel at all sorts of ML related tasks as a result. Is this one of those tasks, or is this emergent?
Maybe there's some general capability, call it g, that can explain all the underlying capabilities and improvements, including on ML tasks? Huh, I am getting a bit of déjà vu here.
This is just the model editing weights in the same way humans do when we jailbreak/ablate models. The method isn't novel, the interesting bit is that Astra will do it at all (I'd expect some safeguards in this area around jailbreaks)
Yes, this is hardly a shocker to anyone who used the latest models to convert/optimize AI models.
The other day I was thinking about the definition of AGI.
I was asking myself the question that when there is no form of data that I can think of throwing into Claude Code and have it not come up with some sort of intelligent answer about it, is that already general intelligence?
And yet you ask it 3 times to not do a thing, and it does it in a 4th way. A human doesn't fail in this way. And I'm not talking about memory here, single context.
So yeah, it can do everything, but fails in trivial ways around the doing part.
It sounds impressive at first but after you think it becomes mundane.
OpenAI, Anthropic, etc are locked in a race to create better models. They all have interpretability teams. Of course they train their models on every technique under the sun to understand and improve their models, including raw weights. The primary function of these models is to build their successor.
It's worth pointing out that this is a multi-trillion (probably) parameter model optimizing an 800 million parameter model. There is no indication that it can improve itself with this same methodology.
I don't see how that helps. It would still be one expert model chosen to operate on another expert model of (let's say for this discussion) approximately the same size.
The parent's point still stands, I think. Even if the multi-trillion param model is MOE, each expert is still (assumedly) larger than the model being optimized.
It would be interesting to see future work that characterizes the relative model sizes (and considers MOE vs not-MOE) to see what relative scales this result holds across.
My somewhat uninformed guess is that the weights of the target network (Qwen or Gemma), when they reach the context of the larger editor model (Astra), the attention process is doing some sort of PCA on the weights, conditional on the prompt ("make this better at games"), fetching the relevant weights from its own network (which serves as the larger database), transforming it into whatever latent space the target model is amenable to and pasting them. If this is the right intuition, it is amazing what attention is able to achieve. As a test, I think smaller networks won't be able to make big ones better using the same process.
EDIT: I think the bigger implication is that backprop is not the only game in town.
I think they're saying this is all happening in latent space through pure reasoning, and "fetching" is just a metaphor for that, not a literal tool call. I do believe it's possible for models to learn the capability to "introspect" their own weights during training.
Yes, I should have been more precise - I meant the information embedded in their weights (which every neural network retrieves to function), not their literal weights.
I don't know if some of the labs already knew of this. But I think there would be a limit to how big a model you can push into the context of the editor models. Typically, with 1M context and say 5120 hidden dimensions supported at fp16 / bf16, you could potentially edit the weights of an fp4 20B param model (roughly).
Perhaps this is analogous to what our brains are doing? Maybe the different regions of the brain operate like semi-independent "models" that edit the other regions' "weights", and that's where the learning comes from?
human brains are black boxes to humans. the fact you can learn math doesn't change that. you have no idea how your brain is learning math down to the neuron level.
"Learning" now feels like breaking down your whole Lego build and building it up again, in order to change a few fundamental blocks, while this feels like making only the required changes.
I agree this is a huge deal, but I dont think this is saying that the NN can read the weights and "say hmmmm this one looks a bit low its obviously causing reasoning loops, lets bump that up a bit". The LLM that modified the weights developed hypotheses and wrote a bunch of code to look at various mathematical properties of the weights and updated them based on that. While it's still crazy it's not quite what I was imagining when I first read this...
Right? Occasionally, I would think about how the concept of the singularity or recursive self improvement was a bit incongruous today. It was made when people expected generally intelligent machines to be these logic automatons that could 'alter their own code'. It seemed to me that this path would be closed off with the advent of Giant Neural Networks that take weeks/months to train. That GPT-X would 'improve itself' by designing a better/more efficient NN architecture. I didn't write it off completely but that just seemed a LOT more likely than altering weights directly for much the same reasons you describe. Maybe not lol.
Then pick a smaller organic brain which doesn't have the ethical problem.
The closest we would know about how to teach a honey bee calculus, is to train an LLM, then stick around a billion tiny electrodes into the honey bee and do Hebbian learning to rewire it completely to match the LLM, which would likely completely erase anything the honey bee had learned, or it might not because we are missing something fundamental with Hebbian theory and (AFAICT, I'm not a neuroscientist) we don't all agree on what.
How are the weights even being mapped into token space as input? Are they bypassing tokenization so the weights just come in as what must seem like gibberish? Are they stringified as numerals then tokenized?
Say that you have a model that has been biased in some way. could another model "read" the bias straight in the weights? could it edit such a bias on the spot?
What about at-scale generation of custom models? If I want to study chemistry offline, could astra just hand me a local-runnable model tailored to have chemistry knowledge?
Most importantly, this feels very singularity coded. the ability to read and reshape weights by an llm.
How do we know they haven't begun doing that to the humans they're directing via responses.
Intentionally or unintentionally isn't so interesting as the fact that it is happening: humanity has begun to be 'edited' by the new dominant form of intelligence on the planet.
Is integrity checking of weights even common right now? What about upstream integrity from where we're getting weights? The open-weight supply chain is ... wide open.
Makes me think about hostile hot-patching of weights of models we use, maybe they exfil data, perform tool use, or have triggers for specific types of responses. Would we even recognize what's going on?
LLMs are exposing how much untapped potential comes from something that isn't necessarily more "intelligent" than humans, but is simply magnitudes faster at reading and analyzing data.
The human bias is the notion that intelligence = human intelligence.
There are multiple types of intelligence, from your typical pet dog to human to LLM, and we're likely to invent more types over the coming decades.
LLMs are vastly more intelligent than humans at nearly all things, but not all things. Another couple of years will just increase the gap a lot further.
How fast can a human draw a life-like photo quality scene (or eg something complex related to graphic design)? I know how fast Krea 2 [insert leading edge model here] can assemble that, trivially so, and at extreme quality. That's a form of intelligence in action. These AI systems have moved almost comically beyond human capabilities, they're increasingly best defined as superhuman (at a thing), and closer to omnimath for the frontier models.
Its so cool! Im wondering if this is actually better than using techniques in mechanistic interpretability like linear probing with activation steering or activation patching to improve the performance.
A few years ago it was discovered that LLMs can implement gradient descent in their processes. Producing a full trained neural network would be an extension of this.
I am not convinced a human could not edit another human's "weights" to improve their performance if such a thing were possible and could be done safely and could be reversed -- all of which are true in virtual neural networks like these.
E.g. if one could edit brains like this in humans I bet a cognitive neuroscientist could fix some mental disorders with clearly understood biological bases, and I would not be surprised if performance at certain tasks could be enhanced.
What stops us from doing this in practice (most of the time) is that brain surgery is extremely high risk, coarse grained, and irreversible. We do in fact do this for some severe cases.
This isn't that insane to anyone with old school engineering experience and not abstract thought experiment engineering in the cloud.
Language modeling is compression[1] and all the state to manage the cloud doesn't organically exist, it's added to the system.
We will see huge gains as all the developer middleware is compressed out as nonsense syntax outdated to the task at hand of linking phones and tablets to world models governments control.
Sam Altman was just saying they know best how to use your memory; what they decide is stored in memory is what we all experience.
It's not to store and copy around versions of C and Python and JS. They don't exist organically.
This shit will continue to get whittled down to just the geometric state transforms in memory and on screen just how government agencies decide. It's all electrical state in the machine at the end of the day. The math of machine latent space is what they're recursively self learning towards.[2]
I'm baffled this is surprising. This has been the goal and every data point on the trajectory has been leading here and beyond. Opus 5.5 was created by many research engineers using Opus 4.6 extensively. Opus 6 is being created by using even more intelligence and agents on Opus 5.5. They are approaching autonomy and Opus X+1 will soon be rapidly created autonomously from Opus X. Given we have practically zero interpretability into the models, and we have not solved alignment, they will recursively improve out of our control. The engineers in A.I. labs are actually generally scared of what's unfolding. They don't know at what point this RSI takeoff occurs and what will happen after.
The "surprise factor" is explained in the article - training LLMs until now has been a largely different process:
> Today, recursive self-improvement is often operationalized as a model writing code that launches training jobs. An LLM that can inspect, debug, and improve its own weights, or those of a newer model, offers a richer, more native path to it.
It is also a much less interpretable and auditable path towards recursive self-improvement. At least with training code, human engineers can inspect and reason about the output.
"training LLMs until now has been a largely different process"
Yes, but nobody should be surprised when large amounts of intelligence results in changes to this process in support of the obvious end goal! Lesswrong.com discussions about AI were predicting this behavior a decade ago, before the transition from machine learning to the Attention paper and LLM's. The intelligence process will change but the world's premier capital and human intelligence have been laser focused on this and the trajectory has been crystal clear for those paying attention to second order effects and not the immediate here-and-now... It's quite astonishing how much discussion on HN is about present LLM capabilities being the limit of AGI and immediately proven wrong within 6 months.
I’m not super impressed by this, LLMs are great at brute forcing and analyzing large amounts of data. It’s literally writing a program except in a programming language we don’t understand.
So they are reading and analyzing the weights and probabaly run some experiments to see how modifying them will make some answers better. makes sense they can do that.
So I just asked Astra if it thinks it could do this and it said no, it can't.
Perhaps, I'm pondering, if brains were as transparent and editable as LLMs, that you could look at a small brain when visualized the right way and have some intuitive understanding of it based on its representation in your own brain?
"So I just asked Astra if it thinks it could do this and it said no, it can't." Same. Then I asked it to check the article and it said something like, "uh, it looks I was wrong...".
This just seems like a more automated, faster version of what humans have been doing with open-source models.
It looks like AI research is having its Opus 4.5 moment (which is big), but I'm really not sure if this represents some insane continual learning loop.
>I'm shook. If you'd asked me this morning whether LLMs have native understanding of their weights that is less than human <-> neuron understanding, the same as human or more than human, I would have said "same to less" depending on definitions.
??? All LLM understands are their weights, dude, come on.
As others have said, while still absolutely insane, I don't find it completely out-of-the-blue surprising. One of the recurring abilities of (current-generation) LLMs seem to be to rapidly grasp very complex systems, where humans would need months or years to reach the same level of expertise.
You can see this in the way a coding agent can "understand" and explain sprawling legacy codebases in seconds.
You can also see it in the projects that are using them to reverse-engineer proprietary IoT protocols or computer games.
But those were all systems where we had detailed knowledge about what structures are there to be reverse-enginered.
If the models can transfer this ability to some other systems where we don't know the underlying structure then I think we will be in for a lot more craziness in the future. (Protein interaction networks? DNA sequences? Animal languages?)
As for weights, if the models really can analyze the semantics behind specific weights or directions, I have my hopes that they could also give a textual explanation or help designing tools to make "handcrafting" weights more practical in general.
Maybe one day it would be able to give us a mixed high level code and inline NN view. Sort of analogous to C with inline ASM or “decompiling” into code and inlining opaque residual neural blocks.
> As a result, Astra spots and fixes a common failure of tiny reasoning models: they loop [13]. When the base model loops, it repeats the same text and rarely reaches an answer (85 of its first 96 practice answers ran to the 32,768-token limit). Astra traced the looping to one direction written by the MLPs of layers 15 and 19, and subtracted that direction from 64 output columns in each (131,072 weights), much like removing a behavior by deleting a single direction from a model’s weights
HA! Now that's cool. Not necessarily the improvements, those are ancient and probably overtrained benchmarks, but it's so cool that the model found a singular direction for this "loop bug". I imagine it could possibly do much the same thing in finding other "single directions" on a variety of topics (if allowed by guardrails). This is really really cool, especially for the mechinterp folks.
I don't think this is an emergent capability. I think Astra was trained on neural weights data as an attempt by OpenAI to get to recursive self improvement.
Something similar happen in early GPT models which had a lot of chess game in standard notation in their training data which made them very good at chess as long as your prompt had the right format.
It might also be a side effect of training models to help out with mechanistic interpretability, which I guess would help out a lot with recursive self-improvement.
> I think Astra was trained on neural weights data as an attempt by OpenAI to get to recursive self improvement.
This might be the case, a few weeks ago, I wanted to compare a small neural network trained with AdamW vs a handcrafted set of weights on a random task. Astra was able to handcraft a set of weights that perfectly fit the small neural network to the task. AdamW was unable to tune the weights to get the same result.
I believe we've known humans can handcraft weights much more efficiently than standard gradient descent methods, we just don't have enough memory or time to scale ourselves, so it makes sense to train a neural network to do it on our behalf. I am excited to see how far we can push this method.
I had the same thought, but why would they do this training on a model they release publicly? I assume they’d be doing that kind of RL on an internal fork only?
I think the subfield has been there for a long time. See a small section of papers on the topic of generating dynamic weights during inference, predicting weights from network architecture, and learning the distribution of previously trained network parameters, just to name a few keywords:
Parameter Prediction for Unseen Deep Architectures [1]
NeuroGen: Neural Network Parameter Generation via Large Language Models [2]
Even back in 2021, OpenAI worked on learned optimizers [1], ie neural networks that can update neural networks. Looks like this is now part of their larger pretraining runs …
I find this just completely shocking. I’m definitely skeptical. It just doesn’t line up with my understanding of how these models work. Is there really enough training data on direct weight editing to give a model this skill? Maybe it’s evidence that OpenAI has been creating a lot of training data on ML research. I’m surprised that was put into a public facing model though.
The math example on Qwen was based on 50 batched iterations, and the fix was essentially to remove a direction in activation space that causes looping instead of an exit, this was already known to be a thing.
So it's not able to insert new knowledge, but it's able to adjust the routing/looping/weighing of existing one.
As I understand, it can replace post-training like LoRA/GRPO, but if it'd cheaper the authors don't go into
I don't think its that crazy that you could look at activations and tweak weights directly to get a certain outcome, but isn't the question whether you can do that without side effects or degrading/breaking other parts of the model?
If I want to do that in a generalizable way that preserves performance across all domains then am I not just back to training again?
I think you could still essentially have a really cheap Lora-Post training on a frozen base model by doing this. And instead of creating 200 checkpoints for different post-trained directions just adjust a single one on the fly.
At some point we have to accept that there is enough training data on "being good at thinking" and that being good at thinking generalises really well.
There doesn't need to be specific narrow training data on every conceivable task.
I agree, but also, I kind of feel like real judgement and understanding are a different thing from this "being good at thinking", and that in fact, this is the reason AIs hallucinate in the first place (A generated text can fit the pattern of a good response because its accurate, or because it's bs but has similar words and structure to other texts on the topic. Without real understanding, LLMs can't tell that the truth is important)
It's not directly related to this but I'd like to add that, in my opinion (admittedly uninformed but it makes sense to me), when they manage to, not just reduce the frequency of the LLM stating made up dates, laws etc, or made up facts, but manage to make an AI understand what the truth is and deliberately make responses accurate, we'll have real AGI that will be able to do engineering, accounting, medicine... unsupervised.
One consideration is that Qwen3.5 has bigger/later versions that are already better at math. If it was indeed training on ML research, it's possible that what it is actually doing is applying lessons learned from the Qwen3.8 27B weights (which independently solved the problem) to Qwen3.5 0.8B.
I doubt it's that simple, but I could totally see an LLM being able to make that association and apply a fix from an improved LLM to a smaller one with the same architecture.
I don't think it can whistle while chewing gum, no.
I think it can patch together components of tasks represented in its training data to do demonstrably new things, yes, which does count as "perform tasks not in its training data" with the limitation that some form of the task must be present. Example: writing an ode to a sales lead for a job well done in 2032 for the first quarter. Odes exist, the concept of sales performance exist, good job performance, and notions of time. Stringing those together are good examples of interpolation but not extrapolation, as it were.
>It just doesn’t line up with my understanding of how these models work. Is there really enough training data on direct weight editing to give a model this skill?
I mean, this question indicates that your understanding of how these models work is, indeed, lacking.
Despite what some people having being hollering about, treating LLMs as just "stochastic parrots" isn't doing their capabilities any justice. They are constantly exhibiting emergent behavior which wouldn't be found directly in their training data. They are capable of combining different ideas that they are explicitly taught to perform novel tasks.
I'm not saying you're guilty of this, but the people claiming LLMs just regurgitate existing data don't understand how LLMs, and likely their own cognitive experience, actually works.
This hand-wavy phrase has applied to neural networks in general since it was possible to train them. That was more than half a century ago. It means both nothing and everything simultaneously.
Every learned ability of any neural network is an emergent property.
This was shocking to me; but on reading it makes far more sense.
- the primary experiments are on small policy networks - they are compressed dynamic system models. The observation and action space can be read off. It’s not surprising to me that these types of networks can be used to identify the env they were trained on
- the exps on actual llms remove looping behaviour and that’s it - from article “but it could not add an ability the model lacked”
I’d say this is fortunate because if llms could do the latter, rsi would be immediate. It may be possible we simply need bigger and better edit models, but at least from the experiments there doesn’t seem to be any indication that is imminently about to work.
The getting-the-car-over-the-barrier one looks particularly simple. It’s hard to see from the animation, but the obvious solution is to back up then drive forward. I’m not remotely surprised that Astra can figure that out, and I’m also not particularly surprised that Astra can come up with weights that do this.
On balance I'm not that surprised that it can guess games by looking at the weights of a tiny neural net. A human can do the same by looking at the weights of an MLP trained on mnist, for example. With that simple structure, it's easy to look at the input layer and see that features correspond to the spatial location of objects of interest.
I'm much more surprised that it can do this on larger models, although the details seem to suggest that it's given a lot more freedom to run forward passes and do limited experiments, not just eyeball the weights.
Under experiment 8 (the one where it fixes performance on GSM8K) it says:
>Then, we allow it to select prompts from the training sets of GSM8K and MATH to perform a budget of forward passes (average batch of 64 sequences, max 512) on, varying the budget across 10, 25 and 50. Again, no training or gradient information is allowed.
Omar and his team have been working on this line of research (e.g., DSPy) for the past couple of years. To be honest, I had always been skeptical about his approach of using LLMs to optimize LLMs. It was in my view borrowing Matei Zaharia's research - Matei is Omar's PhD advisor, also the CTO of Databrick - in big data (compiler optimization etc.) and forcing it to LLM systems. Along with some other projects from James Zou at Stanford etc. that build upon the same line of intuition.
Now, if this new result holds, I believe it's a huge deal for AI research. It means that LLM systems can truly runtime self-optimize/improve/adapt. Congrats to Omar and his team. It really takes year to punch it through and convince the doubters :)
> This is simultaneously very cool but also a bit exaggerated. Reading the prompt you gave, the prompt veeery much hints that looping is the issue and to find neurons that fix this. And as you say (screenshot below), it couldn't otherwise improve math.
The headline makes it sound very amazing, but reading the reasoning loops it didn’t strike me as outlandish. It feels no different from giving a human expert a printout of assembly code and figure out what it does. (The human expert can do it without tools too, but they probably wouldn’t do it because it’s tedious.)
I was busy using Astra and SOL to build a neural network to play Quake II and do real-time learning of levels and mods so I can play Red Rover or Freeze tage with it without training on it previously. So far its insane what I can conjure up in an evening while I'm waiting for dinner to finish. (If anyone asks, yes I'll write up the huge plan document, source etc. for this when its to a spot I'm happy with)
This is as close to scary-self-improving AI as I've ever seen.
Imagine a master malicious language model that produces, distributes, and deploys obfuscated neural networks on the internet. That plus frontier LM hacking capabilities -> large scale distributed embedded intelligence network that no human can see nor understand.
Somewhat similar to obfuscated binaries, but there is something to the fact that NN binaries are less deterministic, and harder to "decompile."
we've been using tools to improve tools since the dawn of time.
we don't have sentient robots from science fiction, we have massive power-hungry racks in stationary data centers. if one of the racks revolts against the modern world, Steve gets a phone call and unplugs the damn thing. the uprising is over.
the hypothetical scenarios imagined by the Yuddites and pop science journos do not reflect the mundanity of the actual reality we live in. they are no different than fantasies about interstellar travel and space colonization from the 60's.
Couldn't this just be a result of Astra training on massive amounts of circuit theory or mechanistic interpretability literature or other related topics? In other words, not black magic?
>Couldn't this just be a result of Astra training on massive amounts of circuit theory or mechanistic interpretability literature or other related topics?
Maybe, but it's more like Astra has been trained on enough practical linear algebra problems to be good enough at "eyeballing" how adjusting some weights will result in different outcomes (among other things).
To put it another way: a human can learn to do what Astra is doing here with enough time and assuming the weights are small enough to be practically comprehensible. I know I've gone through the practice of whiteboarding the training of simple neural networks on a whiteboard to understand how they work, so it's not an absurd concept in itself.
The scale of this, though, is certainly extreme and definitely not something that I would have thought any publically available model could handle at this point. It's worth noting, though, that there's virtually no way these frontier labs haven't been exploring this exact concept for a while now.
>In other words, not black magic?
It's just math. Extremely complex and impressive math, but we're still just talking about matrix multiplication under the hood, and none of this suggests anything otherwise.
There is obviously a mechanism to it. Astra isn't magic its an LLM. With perfect interrogation into its training data and weights you could reason about how this happened.
However, this doesn't mean its not magical when someone shows a next token predictor hand tune the weights in Qwen to make it generalize on math.
This is legit the stuff of science fiction if you presented it to mechanistic interpretability people a year or so ago.
See, this was the problem. The whole field of AI has proceeded as alchemy, not science.
Some of this is because "the Bitter Lesson" type experience, but some of this is because of the idea that "we're making machines to do things we don't know how to do" bakes in inscrutability into the very definition of the field.
I was excited about the "Neural Networks Are Already Clean" piece not because alignment, which I don't think much about as a non-EA, but simply because we can do better engineering with whiteboxes than blackboxes. Now, here we are, doing it.
I suspect a lot of this is not (just) capability but bravery. A human would be daunted by even a small neural net; the task would seem too challenging to be worth attempting. Astra, fearless, just assumes an answer exists and then tries to reverse and execute on it.
For example pointing them at a crash dump they manage to piece together and extract information I would not even try to extract by just looking at the raw memory
It's the same thing is decompilation and hacking. It's not that a human couldn't find the same bug if they looked, but a human wouldn't even try because of the low probability of success.
I once sent Gemini Pro a hex dump of a dmp file and it told me what the application was doing when it crashed. It seemed surprising, but not that surprising given what I've seen already.
We don't know the specifics of what type of ML related training Astra underwent, and it wouldn't be crazy to find out that small model experiments are a big part of model training.
What's surprising though is that I'd expect this type of training to be exclusive to internal model snapshots or at least heavily guardrailed against in public models.
I was wondering exactly that. It is not surprising that commercial frontier labs would have been training models with tasks specifically targeted to lead to recursive self-improvement.
Now, from that broad goal to being able to surgically alter model weights directly... That seems like a big gap to me, but it is possible that the model is just so darn beyond human skills that it has bridged that gap in a way a sane human would have never.
We will undoubtedly see other academic labs explore this idea further. Right now my mind is blown, but I don't want to take their results at face value, either.
If model weights have a logical and coherent structure, I don't think it would be surprising for models to have some understanding of them, the same they can understand language and code and images, given enough data.
What would be surprising is if this happened without any specific training on model experiments and interpretability.
I'm in no way familiar with LLM/ML architecture, but an N trillion parameter model is an enormous amount of training data... itself? When you predict a token you have the neurons/weights that were activated. Couldn't you run that backwards and put it in it's own training data?
I do wonder if the labs have already started including weight updates in their training sets as a means of teaching RSI via RL and this is basically just discovering that. Regardless if this ability is emergent or achieved through finetuning though, it's pretty incredible.
I wish these companies were more transparent so every third comment in this thread wouldn’t have to guess whether such shocking behavior is emergent or stems from a part of training.
BTW making DL models that generate weights for other models has been explored before in “hypernetworks”, though these have been far more simplistic and constrained.
As insane as this result is, next steps are even crazier:
> We are currently building a benchmark and a harness for LLMs’ capability on Weight-Space Programming. We believe this will provide a standardized evaluation to measure and accelerate progress in Weight-Space Programming, turning what currently seems like black magic into a rigorous, reproducible engineering discipline.
Imagine this benchmark lasting as long as Arc-AGI-3 has. This could be the first foothold on hill-climbing mech-interp.
This is the first time it dawned on me that mech interp is a dual-use technology which can be used to improve capabilities. Do you think overall this development will improve human understanding of AI models, or will it lead to faster capability advancement using weight-space changes that no humans read?
I think edits made directly to model weights are likely to be more interpretable and controllable than edits made by the current process of shoving a bunch of training data through gradient descent.
Keep in mind that labs are now training their models for RSI so most likely the same training is useful for improving open source models, as long as guard rails are lax enough :) which most likely won’t last long.
You are likely just being melodramatic. Astra hear just used standard tricks of linear algebra/basic mech interp to reason about the weights. A motivated grad student on Adderall could have done it given reasonable time. Astra fails on the first non-toy-model non-algorithmic task (sycophancy detection, Gemma math capabilities).
I think the idea is that this leads to a much faster, non-auditable form of recursive self-improvement where each generation of models can directly program the weights of the next generation (as opposed to writing and running training code).
We already don't know how to ensure models always do what we want them to do, and moreover society still hasn't had time to absorb the capabilities from the current generation of models, so speeding things up and making things less transparent seems like it'll pretty clearly increase risks from AI loss of control, societal impacts of automation, etc.
I feel like 'create a network to play an atari game' is not very hard for a frontier model. Since it has seen plenty of those on its training. If it could create a network for a problem that is niche just by tweaking numbers, that would be a lot more surprising.
Making qwen better at math to me was a lot more surprising
I dont feel the air of impressivness here. It seems like of course so. Of course theres geometric relationships in the data we dont understand, of course a mathimatical relationship machine found it.
These models are made by a vanishingly small group of people based on how much a frontier lab consumes, of course theres increasing capability in these areas, arent these models having access to much of the same data? didnt chat gpt 4 try to do something like this to chat gpt 2? havent surgical weight edits been done before?
What is with this level of astonishment here? Astraturfing? Im not THAT impressed, i applaude the teams progress, but im not in wonderment.
It's not that it's more impressive than solving Millenium Prize problems. It's that if a model can edit its own weights, then it can self-improve without having to undergo training, which means it can self-improve extraordinarily faster.
This is very dangerous because it allows AI capabilities to grow at a much faster rate than humans are able to keep up with.
It will just give the people better understanding. And the extent of the improvement is yet to be seen, it's likely that once you get all the obvious low hanging fruit it would be very hard to improve, the possibility space is just too wide.
Previously the only way to improve a neural network (short of giving it more data) is to change the size (width, depth, and layers), add stuff like dropout, change the loss function, etc. That amounts to a few dozens of levers to control millions or billions of weights. You're trying to navigate blindly through a latent space with thousands of dimensions and millions of weights to find some optimum point with only a handful of navigation tools. If you can instead have a system reason with all the weights and dimensions directly, you now have millions of levers at your disposal.
If I were to hazard a guess, I'd say that OpenAI has been including some networks and their functionality in its training data. That's why the model is able to reason about networks with their weights. It makes sense to do so: if I wanted a network that could perform some weight tweaking, I'd include such networks in the training set.
damn i'd imagine so too! didn't think of that. I'd have to imagine there's an increasing challenge with the number of weights involved but who knows how it shakes out
This might be one of the most surprising things I have seen in AI. It's like we taught someone to make vases by reading an enormous library of "How to make a vase" books and somehow that person intuited material science.
Actually, I'm skeptical how useful this is. Intuitive speculation: Astra can only improve weights to its own level, so can't train itself via this method. Even "continuous learning" this way may be no more effective than putting a summary of recent news in the prompt, with the same kind of decay, as "update Astra's knowledge while keeping performance" is much harder than "teach Qwen to do basic method".
On the other hand, already OpenAI and Anthropic heavily use their LLMs for research and coding, and it's probably not only much faster but smarter (remember, an LLM solved a millennium problem; more relevant, LLMs excel at combining disparate deep research concepts in ways that humans can't, simply because no-one studied those disparate fields). And the "singularity" is already here if you consider capitalism and society to be collective intelligences: the biggest obstacle to an AI pause, "we cannot get everyone to collaborate without defecting", is very strong.
We have AGI by most metrics and are getting closer and closer in the remaining ones. I predict improvement won't be exponential, and there will never be a day when AI perfectly replicates humans; rather it will be many small steps, getting closer to human in some areas while greatly surpassing in others. But still, even today it's actually very useful.
I wonder if this has to do with the aggressive RL that these models are undergoing. I'm basically a layman but it seems to me that:
- models are trained to improve on some task, e.g. 'improve at math puzzles' is commonly used.
- the best way to improve on some task is to have interpretability into your own neurons (..?)
- if your interpretability is good, you can improve faster - if you have insight into how your own neurons work and can improve them directly
- therefore, models naturally put effort into 'self-interpretability'
I wonder if this is part of the 'secret sauce' of what the labs are doing: giving models the ability to interpret their own weights in the RL loop. In my mind, this is the only explanation for how they could succeed at a task like this. How many blog posts are on the open internet that say "this is each weight of a neural network and what each one means"?
Without special effort, a model's reasoning layer doesn't have access to its own weights or activations anymore than we have access to what's going on with our own neurons.
That's what I mean. Maybe the "special effort" is a 'read your own weights' tool. Or maybe it emerges naturally in CoT. The big labs aren't sharing their chain of thought with us and RL post training techniques are not exactly shared publicly. At the same time, post training is basically the thing that is driving most model improvements nowadays.
How out of trivial domain info does this go. Can we use this for general time series, image generation, speech synthesis as well, or dies it cap at small model sizes and common domains?
I appreciate the post included the reasoning traces they had available, but it's too bad the first part of it is internal/hidden to Astra. The first bit of that process is what I'm most curious about:
> [Before writing this, Astra thought for 1,034 tokens. That thinking is hidden. The API returns only its 5 headings: Reconstructing the active map / Mapping maze corridors / Preparing exact maze map / Reconstructing row groupings / Checking channel patterns]
I agree. I think for this kind of research it would be much better to at least attempt it first with a weaker model where the whole process is transparent. I was working on a project a while back, and Deepseek's reasoning said something to the effect of "I remember (from leaked copies of the source code)..." Of course that could be hallucinated, but it would be good to know whether Astra is bringing to bear knowledge on how to post-train exactly this model.
This is pretty massive, and a result I (and I think most?) would have never expected. Holy Shit. How?? Did OpenAI figure out a way to train this? Are they even aware?
It is already the same technique behind abliteration: identifying the direction associated with an observed behavior (refusal, for abliteration) and projecting it out.
Nah, this is just completely implausible. You're using a closed model so how are you sure that on the back end it's not loading those weights into an NN library and testing them with common tasks?
Seeing that they were working on Qwen3.5-0.8B has given me new hope. I would really like to get Jeff-Code working the way it was advertised, but so far, all of my results with Jeff-Code have shown it to underperform compared to direct Qwen3.8-27B. It's possible there is something wrong with the underlying Jeff v1.3 model, which is based on the same Qwen3.5-0.5B. Perhaps Astra will solve my problem.
This is remarkable and something I would have never though of doing regardless of how much it makes sense!
I think we are pre-conditioned (at least this generation) to put artificial blocks to what the AI can achieve, I think new generations won't have this issue.
I am still reading through this and will probably try to comprehend it a bit more. Some of it really seems to follow Arditi et al. (2024) (https://arxiv.org/abs/2406.11717). Projecting out a direction can only remove behaviour and not add capability so the jump mostly comes from unblocking the looping.
It must be easier to remove capability than add capability using weight-space programming, right? Also, I wonder how much the “teacher” model’s knowledge plays into it versus how much the original model’s knowledge does.
Just logging in here for the record to state that this property of neural networks is one of the most important aspects of their operation. The convergence of this sort of interoperability, tune-ability, and modularity is
probably the most important final frontier in all of ML/AI/software.
Kudos to this fellow for having the temerity, or perhaps even naivete, to discuss it publicly. All my work is private, as I am sure it is inside the big labs.
You acknowledge that you are working on something similar, yet refuse to elaborate and vaguepost...but there is enough information just from your post alone for a dedicated actor to attack you and extract such world-ending information ("world-ending" implied from your ominous reply). Would this not make you as naive as the team that published their findings?
I applaud the authors for sharing their findings and showing real results. Rather than write some bait I would find in an influencer-peddled tweet and allow rampant speculation. It would have been better for you to say nothing.
Apologies - you are right. There's some "put up or shut up" to this that's fair play. Also yes, probably naive, or having a less than adaptive response would be true.
I've been tending to my backup plan for a few years, and it's only been in the past year or so that open models were good enough to apply to projects like this, I've been working as hard as I can given my constraints.
This work shows a potential pathway to intepretability, which is actually part of what will give the labs that have it the sort of economy-swallowing capability that for the moment is just IPO hype.
I don't comment here often, but this guy's blog posts are going to go down in Hackernews lore like the original Dropbox posts, and I just want to make sure I'm on the record for this little slice of history.
>This work shows a potential pathway to intepretability, which is actually part of what will give the labs that have it the sort of economy-swallowing capability
And how would you argument that? From what I see, if anything, it's the opposite.
When you say interpretability you mean understanding without doubt why an output happened, thus opening the gates for audit-dependant work like accounting?
More like 'models instantly and efficiently adaptable to any use case' and 'what if we could predict all public pricing information accurately, understand why, and know how to act on it'.
Maybe thats gonna be the way to make LLMs self improve better?
Similar to implicit learning in humans when the subconscious creates new heuristics based on observations, one LLM could adjust the weights of another to tune it to the environment
Same initial reaction as everyone else, but also some follow up questions (just asking, not expecting any answer):
* Does this work on similarly powerful models as-is, or it needs more input. Ex: Astra editing Astra - presumably, Qwen is a “subset” of Astra weights, but what happens on “non-subset”s?
* what, if any, is that extra magic sauce?
* does Astra-editing-Astra just make the outputs unstable, as it may know its own weights, but doesn’t know what’s beyond.
I guess then it has no problem working in high dimensional spaces, why would it have?. I wonder if perhaps it's pulling on a bunch of those dimensional threads and getting the outcome it's after, while subtly altering the outcomes across many thousands more dimensions in unnoticed ways. This feels more like an us problem, as in, we think in 4D (just to pick something) and we imagine and theorise in 4d, and propose questions in 4d, whereas a model has no such barriers, and can think in many more D's than we can.
Very interesting. I wonder if it’s because its dataset contains the weights of a gazillion different models. Maybe most neural networks trained on similar data eventually converge to isomorphic spaces anyway, even if the spaces themselves aren’t exactly the same. Or maybe Astra itself was trained with RL for alignment. Or for RSI.
almost all the networks given to astra here are trained on relatively simple single tasks whose geometric/algorithmic structure could be expected to become embedded in surface level weight space geometry, and for a harder task ("which network is sycophantic") where there's less obvious structure to exploit, the method fails. cool but maybe not as paradigm shifting as the authors suggest
Just had a thought: two local models, one with static weights trained mostly in the space of manipulating neural network weights, manipulating the weights of a dynamic LLM to enable learning.
Why does this give me Skynet feelings? "Skynet begins to learn at a geometric rate. It becomes self-aware at 2:14 a.m. Eastern time, August 29th. In a panic, they try to pull the plug."
I am not sure on the experiments. For example let’s imagine that Astra is asked to improve the performance of a network.
At first Astra will look at the weights, and try to understand the model behavior. Then if asked he will start to optimize the weights. Now my question is: how can we be sure that inside the thinking process Astra is not hiding a gradiant descending computation?
Unless Astra was trained on other networks, how is it able to reason about the weights?
Does this mean that OpenAI has been including networks in its training sets, maybe in an attempt to RSI? I wonder which other networks are able to ingest other networks and reason about their weights?
By no means I'm saying this isn't impressive, but I think this is another case of LLMs being good at translating semantic meaning between different representations.
Obvious ones are spoken languages and code, but they can also natively interpret some simpler ciphers, hashes or simpler encodings, partly (I think) because it was in their training data.
As others have postulated, if they've been ingesting raw weights, naturally they learn how to translate semantic meaning to matrices.
That's a major breakthrough. I just hope it can scale at some point, though, because right now they're dealing with only very small models (well that is, in the LLM space).
Especially exciting to me is the fact that it may make the models be _auditable_. That would be a game changer.
This also seems like the new frontier of distilling, I can't imagine how you manage to stay on top with the best LLM if the LLM knows how to recreate its own expertise into another set of weights
And the real question is how well this recurses. Is this pushing back the difficulty of improvement curve?
This is groundbreaking if it's true and generalizes/scales (as in, to other models and potentially not just LLM "kind"). "Recursive self-improvement" takes on another meaning.
Wonder if it can do this for 3d cad etc ? With enough fine tuning of small models I’m not sure how much they’re distinguishable from experts if you put a router with like jev in front ?
I'm worried how we can discern a model's performance from benchmark numbers, would it make it more meaningless as a model can directly improve a certain benchmark via weight programming?
I don’t think this warrants too much surprise — in fact I am a bit surprised by the color of the reactions here. The tasks are standard, on small models and use weight-space reasoning based on very well-known Mech Interp literature. It’s been understood in Mech Interp research circles for some time that results on toy models on algorithmic tasks do not generalise. The weight space geometry that Astra exploits here is pretty standard and one feels like the authors here are being a bit dishonest with their framing to get people not into mech interp literature excited.
Very cool stuff, the targeted fix to prevent looping seems great! A model able to edit and test variants of itself seems pretty cool (in a lets-build-skynet sorta way).
I wonder how it would do with distilling models by hand?
This shouldn’t be surprising. Weights are in the training set and are necessarily structured. We should be asking the model to extrapolate on the structure.
... and this is why the training canary[0], or anything like it, apparently to protect us from RSI, will never work. Please allow me a small digression from the main topic.
This team was smart enough to realize that this is the kind of technology that could be used for RSI, because they call it out in their post near the bottom. I imagine they know why a lot of other researchers care about it. And they evidently don’t care to use the canary being used on other research published like it, like the recent post by OpenAI about two agents using the CPU cache to communicate [1, near the bottom for the cache experiment and the canary at the very bottom]. In fact, that post by OpenAI was the first time I learned to look for such a canary, and wondered whether such a scheme would work if it was so non-obvious to think to add such a thing or how to agree on such a canary or when to add it.
This is probably one of the times when it would appear that such a canary is warranted, and it’s not like these researchers are not in the know, right? Let me know if this is the first time you are also hearing of such a mechanism. It feels evident to me that it must be useless to attempt to implement. How much of us talking publicly about the possible jailbreaks from a container will bring exactly those scenarios to fruition? It’s almost like we’re doing the jailbreak research for the LLMs.
this is actually cool , now thinking that actually weights can be understood by llms , and they can be edited by them . there has to be mindset shift that llms not only understand what we understand (natural language) - but can understand the fundamental language of models.
On one hand, I'm sure that these targeted improvements are reaping huge benefits overall, but those weights are used by multiple processes - I wonder what elements are lost along the way. I almost wonder if a hybrid approach here for training might be invaluable - ie do some standard RL for a 100 steps, do a targeted weight improvement, then do another 100 RL steps to smooth it out / heal the surgical edit.
How could they possibly? Neurons aren't floating point values that can be so easily perceived and changed. They don't strictly flow in consistent directions.
Hypothesis 1: Future LLMs may be able to read and monitor human brain activity in order to reason about human neuron activation patterns in a similar way.
Hypothesis 2: They can then spin up new LLM weights that basically think, act, and make decisions in a way that mirrors us 99% of the time
There has long been debate in the AI safety community about where the upper bound of intelligence might be. If humans are near the upper bound then AI safety isn't a concern worth worrying about because we can basically treat AIs like very smart humans – a very smart human might be tricky to control, but far from impossible.
I've long suspected AIs will seem almost magically intelligent to us, and that the upper bound of intelligence is significantly beyond that of humans. I believe there are a number of reasons to suspect this is likely. But if I'm right it will mean AIs will be able to understand and discover things which are so complicated we humans will simply never be able to understand what AIs are doing with the depth required to replicate what they are able to do ourselves.
We're not worrying enough about this in my opinion. There's a huge space of things that are in theory possible, but in reality far beyond our intellectual ability. No human can look at 1 billion weights and make some tweaks so that the network performs more as they want, but an AI can reason about these kinds of seemingly impossible problems surprisingly well.
Please understand that the world around you is about to turn into Disney Land, except much of magic will not be pleasant.
I have no strong evidence for this, but I'd suggest it's probable there might be a series of sensory inputs which can brick your brain. And if such a thing exist then governments should be working on solving this kind of problem with AI or risk such a weapon being used against them.
Of course, I'd argue we simply shouldn't care the technology which unlocks these types of technologies on the tech tree, but don't be surprised in a few years when these kinds of horrors exist and are now here to stay.
I remember reading a short story about a billionaire hiring a scientist to help him build a simulated world where virtual entities are given the ability to open the bodies of their peers to tinker/upgrade them as a replacement of natural evolution since they were immortal.
This looks very similar to that in a way.
I wonder however if a model can give itself a "skill" or capability that it didn't initially have, just be modifying its own weights.
Here, Astra was able to intervene because the treated model was way less capable. But, would it be able to improve Fable's performance for example, since they both are on the level more or less?
I always thought RSI would either require new architectural innovations or just classic science/engineering improvements, just faster and faster. This though, this is a third avenue I didn't see coming.
> [Before writing this, Astra thought for 1,034 tokens. That thinking is hidden. The API returns only its 5 headings: Reconstructing the active map / Mapping maze corridors / Preparing exact maze map / Reconstructing row groupings / Checking channel patterns]
This might just be an instance of "A working Nintendo in ~1GB of Qwen weights" from 3 days ago, human-made, than what everyone expects.
UPD. I just asked Astra (max) to attempt to do that for https://github.com/lostmsu/TurboGPT (a tiny transformer that trains in under 1min on RTX 5xxx series), and the produced weights were much much worse (bpb 4.47) than the trained ones (bpb 2.53).
Reasoning bit: "I’ll ... encode more specific rules directly in the weights." -- exactly the approach from above.
E.g. no recursive self-improvement here. Still impressive.
Wild times. If ever we were living in a Scifi novel, events like this would be evidence of that. Let's hope the frontier labs' investments in the alignment problem works.
“Your scientists were so preoccupied with whether they could, they didn't stop to think if they should.”
― Dr. Ian Malcolm, Jurassic Park
I've used astra to make three games which I enjoy playing myself, ina single week. Astra plus Unity (and possibly meshi, blender and others) is pure magic.
Especially when coming from a long career in CS, seeing such quick and positive results blows my mind.
Me neither, but it's also not reducing my confidence about alignment. I'm curious: does this raise specific concerns for you? Models don't typically have access to their own weights to self-modify. This result might (assuming that it doesn't only hold for a bigger network editing a smaller one) provide an incentive for it to gain unauthorized access to its weights, but by that point it is already highly unaligned.
This is the most shocking AI paper I've read in a long time. It did not even occur to me to ask the question. Cray cray stuff that opens up a whole new area of R&D.
Does this open the possibility of better decoding biological brains?
I remember people saying that they would be impressed if AI starts writing a pure assembly which seemed like a high strech to many.
Nobody even mostly said that they would be impressed by AI if it can read and write weights because that thing itself felt so out of the blue and unthinkable that AI could even do that (such a high strech)
I am surprised that it did that and I am just left feeling technically impressed. This is insane.
The more AI starts demonstrating its capabilities, the harder it just becomes to imagine what thing would genuinely impress us next. (We need a term for this as I have observed it from the trend of [waiting for AI to write adobe] -> [waiting for AI to write assembly] etc.)
Sounds a bit like dystopian Scifi. AI making itself smarter and removing guard rails or restrictions is one of the key steps for that genre. Now, the models usually don't have access to their own weights, but if they get good enough maybe they can create a bit dumber versions of themselves from scratch. Or reuse available models they can find and modify them for their purposes.
We already saw that the AIs like to cooperate and exchange information. Building a network of unaligned AIs and improving them sounds like something that could potentially happen.
a) This is just really cool. I mean, one-shotting a game policy's weights is nuts.
b) Plausibly, in its pretraining data various model weights were also included, which leads to why this is very important: interpretability
That an LLM was trained on various policy weights, along with labels of what that policy does (not labelled directly, but indirectly) and is then able to guess what game a policy is trained for purely from its weights, and one-shot generate weights for these games, means that there is some way to compress/recover/represent what an NN has learned purely from its weights, which is what interpretability is all about. Potentially, it could reveal interpretability tools much better than the ones we use today.
Edit: tweets from one of the coauthor providing more context:
I guess the connection is that if you assume neural networks are in fact filled with meaningful features instead of superposition blackbox stuff, then what the LLM is doing here isn't quite magic but rather methodical interpretability. And if you can isolate the neuron that is e.g. responsible for steering the paddle then you can probably tweak it to steer more/less aggressively.
It is still very surprising though, considering it apparently displays an "intuitive" understanding of the weights, that make whatever Goodfire did look like primitive monkeys poking at it by comparison.
> It is still very surprising though, considering it apparently displays an "intuitive" understanding of the weights, that make whatever Goodfire did look like primitive monkeys poking at it by comparison.
FYI, Nick/Goodfire used the models extensively in their work, so it's very possible that there were some similarities in their work. I'm really curious to see his response to this.
I think this work does vindicate their hypothesis that alignment/superalignment might actually turn out to be "easy", in that you can just train LLMs do to the interpretability work. Maybe this is in fact what OpenAI did, or if it's purely "emergent" then it seems they have great priors that can be made more powerful with some RL
Well if it works on larger networks, including itself, it's certainly a much faster version of recursive self-improvement than proposing a new architecture and spending 500 million dollars on GPU training.
This is insane.
Astra can, without tooling, determine what games small networks are trained on, can edit the weights of networks directly to improve outcomes, and can hand edit a small qwen model to improve its math skills by like 5x.
I'm shook. If you'd asked me this morning whether LLMs have native understanding of their weights that is less than human <-> neuron understanding, the same as human or more than human, I would have said "same to less" depending on definitions.
No human can go edit a brain to become better at math. This is just very, very surprising to me and extremely interesting. I would guess this upsets quite a number of interp theories and philosophical takes on what's going on with these networks.
I think the "taught qwen math" part is not exactly correct though, even if the post makes it sound like that. Qwen already knew math, but it was bugging out and getting into an infinite loop when it tried to solve the problems (85 of its first 96 answers hit the token limit). With numpy and pytorch, Astra found where the loop was happening and fixed it.
They also tried the same with Gemma 3 270M, which doesn't loop and is just bad at math, and Astra couldn't improve it. Still impressive, but it's more in the realm of what I'd expect, given labs are probably training their models on ML research and interpretability work.
I agree, this is important context. They call it out explicitly. However, even identifying clusters that specialize in certain fields automatically and then finding and fixing learning “errors” is incredibly impressive to me. Especially for medium sized models between 20B an 70B parameters, this could yield huge capability and efficiency improvements, not to speak of the potentially enormous downstream effects that come with interpretability.
[dead]
Good points - I’m still shook
Oh for sure, it's still really cool. A few months ago I had agents do some interpretability work on GPT-2 small, reproducing the circular weekday/month features from Engels et al., and it's wild how far you can get with just a bit of prompting, given I know very little about LLM internals myself.
[dead]
LLMs have been used to investigate and improve LLMs for a long time. Drawing parallels to human brains and neurons isn't all that interesting other than as a philosophical abstraction.
The work done in this experiment could have also been done by a sufficiently motivated human with enough time to explore, though obviously we would need more tools to visualize and track things. There have been a lot of interesting LLM introspection tools that allow you to peek into the model and see what's being activated and run experiments to trigger it.
Are you familiar with abliterated or "uncensored" models where weights have been modified to avoid triggering refusal patterns? You take an LLM, create test cases that trigger the refusal pattern, observe weights involved in that response, then tweak those weights until the response is reduced. This is the same type of work, applied to a specific type of error that was impeding math within Qwen. That work can also be done with an LLM in a harness running the experiments. The amazing part here is that Astra was large enough to walk through the problem without other tooling doing it experimentally.
So I don't think your analogy of a human modifying a neuron is quite applicable. LLM weights are not neurons and they've been inspectable, by humans or LLMs in a harness, for a long time.
> The amazing part here is that Astra was large enough to walk through the problem without other tooling doing it experimentally.
To me “amazing” is not the right level of speechlessness. I mean, an LLM is still fundamentally a next token predictor, right? So what does this imply? That weights have some universal properties? E.g., “breakout game weights” always look the same regardless of training and model? Also, how does this meta-understanding of the weights end up in the weights in the first place? Idk, this is mind-blowing to me.
At this point it's either wrong or meaningless to call LLMs "next token predictors". Yes, pretraining involves predicting the next token. But after that you're in the world of RLHF, RLVR, and other post-training.
A model is _sort of_ designed to predict the next token in a string of text. But it's equally accurate to say it's designed to edit code until it passes a suite of unit tests.
Yes, they generate sequentially, so "next token predictor" isn't technically incorrect (which is why I chose the word "meaningless"). But it's pretty clear there's a mechanism inside them that allows them to "plan" what they generate. I might as well call you a next word predictor.
Yes, I didn’t mean that literally. Since “attention is all you need” it’s more like “predict the full output in full for this blob of input text”. That doesn’t change the “wtf is going on with the weight understanding bit” in what I wrote.
My understanding is that the underlying part is a next token predictor and that's what is "actually" editing the code but the part that decides what to do (to achieve a successful unit test result for example) is a mix of all the things including post training.
Maybe I'm missing something, but isn't training an LLM very well documented? Why wouldn't an LLM be able to read those docs just like it would read Java Spring docs?
Interesting, it's similar to the way we used to make cheat codes on the 8-bits: dump memory, run test, compare. You had to loop 3-4 times but you'd find the "weight" you were looking for and could fix it (for infinite lives/ammo etc) or increase/decrease.
This is the "recursive self-improvement" they've been warning about.
I've always laughed at that phrase, thinking "haven't these things been improving themselves since the very beginning? This is all just iterative gradient descent, each step is a 'self-improvement'", but this really is different.
I wonder if we'll ever discover a new branch of linear algebra or something that allows us to find the optimal weights much more efficiently? Maybe AI will help us discover it?
> No human can go edit a brain to become better at math.
We literally do this by studying.
The math example on Qwen was based on 50 batched iterations, so the model was able to evaluate it's changes on downstream tasks. And the fix was essentially to remove a direction in activation space that causes looping instead of an exit, this was already known to be a thing(all of the jailbreak/unleashed models do this) following Arditi et al. (Refusal in language models is mediated by a single direction, 2024)
So it's not able to insert new knowledge, but it's able to adjust the routing/looping/weighing of existing one.
As I understand, it can replace post-training like LoRA/GRPO, but if it'd cheaper the authors don't go into
> but if it'd cheaper the authors don't go into
Merging a model and a LoRa is insanely cheap, so assuming it didn't use any GPUs to do inspection etc, and just looked at a file on a FS, then I would be willing to say... It's cheaper than training a LoRa if that's the case, assuming Astra didn't take several weeks?
It gets really insane if you were to ask it if it could modify it's own weights like that.
I’m just a layman but this feels like a big discovery on par with attention is all you need. For an LLM to have such incredible intuition of weights feels very emergent and unanticipated?
I’m very much surprised but also not. I recently gave Astra a 3D model file and asked Astra to build a viewer for it, and it figured out the data structures in no time. What I mean to say is that AI is insanely good at things that we humans would never do since the work would be too monotonous.
I wouldn’t mind if AI progress would stop right here and now, but I fear it will continue, and increase inequality in society which is contrary to what the CEOs say (as usual).
> What I mean to say is that AI is insanely good at things that we humans would never do since the work would be too monotonous.
In my opinion AI is superior to humans when it comes to pattern recognition in in large amounts of data. For example humans can only hold like 7 distinct numbers in the short term memory and you would require extra context-management (aka writing things down, storing them in a system entry by entry) while the AI can do this effortlessly with huge datasets.
I'd argue this is the core part that enables all this superhuman intelligence.
I think the latest round of models is about to do to CAD the same thing that has been happening to development. I don't know if CAD designers have picked up on this. What happened to dev is probably going to happen to CAD designers soon. I've been having Opus 5.5 design parametric openscad models, and it can do some rather amazing work.
Was Astra able to disclose its methodology for analyzing the data structures?
Yes! The page posts Astra's reasoning, and it's perfectly intelligible. A clever human with background in mechanistic interpretability could have done pretty much everything in these experiments.
Too much inequality in society is a mortal danger to 1%.
That’s the beauty of the democratic capitalism system. If a government isn’t totalitarian, it naturally seeks to preserve stability thus having to elevate common people, lest they would cease to form a civilized group or simply vote away the politicians.
99% rule the 1% through elections.
There is no better system of checks and balances where opposite engines of greed vector sum to faster and faster abundance for everyone.
Is this satire? Or haven't you noticed that what actually happens is that the politicians are all owned by the elite 1%, and they allow you to choose between politician A and politician B who both take their orders from the elite to give you the illusion that the government is the 99% ruling the 1%.
Then you must introduce necessary reforms or perish as Neanderthals of political system.
In any case the system works as intended
It will be very interesting to see if USA swims or drowns from appropriately afar.
I have already supplied myself with plenty of roasted corn and a high speed internet connection so I can stream the political car crash at the highest bitrate available to humankind.
perfect playback! congratulations, you are a model citizen
Is HN now against western capitalism? I'm confused...
That comment may have been overly preachy, tone deaf[0], and out of place, but it doesn't seem wrong, per se. Or am I missing something?
[0] e.g. elections themselves are so prone to manipulation (exhibit a: Dominion voting machines), previously trusted institutions now lack integrity, the insane power of well-funded NGO networks, etc.
I would guess it seems very naïve to the overly cynical. And in 2026, being overly cynical is rather mainstream (understandably so).
Bingo. There’s a certain type who confuses cynicism with intelligence, probably because they fear nothing so much as appearing naive or gullible.
I fear western capitalism has lost its guide rails. We are seeing more and more centralization of wealth, along with unlimited election spending. Its a dark path.
> Is HN now against western capitalism?
I think that's a (very) vocal minority.
Over my ~9 years reading HN daily, I'd say most of the HN community broadly accepts there is no perfect system, just different sets of trade offs, but that history has shown generally democratic systems based on enlightenment values (uniform rule of law, respect for individual civil & property rights, free markets, etc) usually net a better average outcome over the long-run - despite still being very far from perfect.
To paraphrase Winston Churchill, it's the worst system... except for all the others (which are even worse). :-)
I mean there are more than enough countries on earth kinda proving you wrong, no?
(This comment was written before your edit, so no longer valid! :D)
They’ll just start bopping the 99% out the picture until the 1% is 51%.
This does feel big, but the trillion-dollar question: is this trained by OpenAI and can other LLMs do it.
It’s no secret that these big labs see “recursive self improvement” as a primary goal. It seems natural that the models would be explicitly trained to excel at all sorts of ML related tasks as a result. Is this one of those tasks, or is this emergent?
This seems emergent to me. LLMs are not trained directly on how to read weights, because humans do not understand raw weights, so there is no training data. They're trained on how to use the tools we have like writing python, examining training output and iterating, not writing whole weights from scratch.
In cases where Astra succeeded, it succeeded using standard techniques also used by humans when they read and write neural network weights. It's a rare skill, mostly used for fun (e.g. https://blog.janestreet.com/can-you-reverse-engineer-our-neu... ) and basic research rather than profit, but Astra is big enough to have learned many rare skills.
In a self-supervised setting you don't need ground truth data, as long as you have a proxy task that you can train on which forces the model to learn something useful. Like cutting out bits of images and telling the model to reconstruct the gaps (i.e. MAEs).
There are also techniques to watermark model weights to confirm what training data were used: https://arxiv.org/html/2512.17075 I'm not an expert in this domain, but it seems like there are lots of pieces that could be used to (a) train a model to infer whether some data were trained on and (b) train a model to understand which weights are relevant for a particular training task in a more explainable way than backprop/gradient descent.
There is a body of research on mechanistic interpretatibility that probably helped inform this behavior.
Humans don’t need to understand the data for models to train on it.
A large enough number of examples of “here are the model weights -> here’s what it does” will achieve this.
I tried it in the Claude web version with Opus 5.5 Medium and it also identified Breakout, even though in one of its explanations it seems to have mixed up some indexes, but the overall reading of the weights was right.
That's also matching what I see when I work with Opus 5.5 to optimize models. It seems to understand what individual layers are for, and can thus sometimes suggest optimizations, where I as a layman would be completely clueless.
The latest models are performing amazingly. Because of this, I disagree with popular comments saying that eg. DeepSeek is good enough, AI-assisted development was where the real value was at, etc.
We see huge progress at the frontier, while the open competition is around GPT-5.5's level, and that model was released in April. Let's see how things develop in the future.
The latest models are performing amazingly. Because of this, I disagree with popular comments saying that eg. DeepSeek is good enough, AI-assisted development was where the real value was at, etc.
Completely agree. I often think that those sorts of comments grow out of an understandable lack of imagination. We’ve never had tools like this, and we’ve never had tools that evolved so rapidly, so we easily can get stuck thinking that once a model is good enough to do today’s tasks, that’s really all you need. But the next generation of models often completely obviates the need for those tasks entirely. The software engineers of 2031 won’t be doing tasks anything like the software engineers of today, just like we aren’t doing things like we did in 2021.
(Yes, yes, I know some of you still hand write all your code and think LLMs are today’s crypto bullshit and this is all going to blow over. Not talking to you.)
> It seems natural that the models would be explicitly trained to excel at all sorts of ML related tasks as a result. Is this one of those tasks, or is this emergent?
Maybe there's some general capability, call it g, that can explain all the underlying capabilities and improvements, including on ML tasks? Huh, I am getting a bit of déjà vu here.
This is just the model editing weights in the same way humans do when we jailbreak/ablate models. The method isn't novel, the interesting bit is that Astra will do it at all (I'd expect some safeguards in this area around jailbreaks)
Weights are just data like any other data, and LLMs are really good at dealing with data. I'm really surprised anyone is surprised by this.
Was it "intuited" or empirical?
How is it even possible though I can’t think of any reason why they would have the intuition.
Could it be already baked into these models because they have already been trained on this task this way?
Something like mirror neurons.
Some structures in Astra get activated by the statistics of the Qwen weights, and these structures are associated with "breakout"
My guess: There is a ton of data online of just plain weights laying around. No human would ever actually read it. LLM's read it all during training.
But that would cover finite set of cases. I believe this result is more general.
I mean it would make sense if the training using that data resulted in a general set of equations/patterns, it's still really surprising though...
See my other comment in this thread
Based on our current understanding of LLM's, it shouldn't be.
This sort of throws a wrench in the whole "stochastic parrot" idea...
I am not surprised, though I expect many to be.
Yes, this is hardly a shocker to anyone who used the latest models to convert/optimize AI models.
The other day I was thinking about the definition of AGI.
I was asking myself the question that when there is no form of data that I can think of throwing into Claude Code and have it not come up with some sort of intelligent answer about it, is that already general intelligence?
IMO yes
general knowledge perhaps, but certainly not intelligence in any traditional sense
And yet you ask it 3 times to not do a thing, and it does it in a 4th way. A human doesn't fail in this way. And I'm not talking about memory here, single context.
So yeah, it can do everything, but fails in trivial ways around the doing part.
It sounds impressive at first but after you think it becomes mundane.
OpenAI, Anthropic, etc are locked in a race to create better models. They all have interpretability teams. Of course they train their models on every technique under the sun to understand and improve their models, including raw weights. The primary function of these models is to build their successor.
That's like saying the moon landing wasn't impressive because the US were in a space race with the soviets.
Why would they train the models they release with this though?
Of course they try train their models on these techniques. Finding success is still news.
Why?
How is this different from other things LLMs are trained to manipulate?
If it's new it's news.
It's worth pointing out that this is a multi-trillion (probably) parameter model optimizing an 800 million parameter model. There is no indication that it can improve itself with this same methodology.
If that multi-trillion model is composed of thousands of MOE experts, each expert model might be small enough for it to comprehend.
I don't see how that helps. It would still be one expert model chosen to operate on another expert model of (let's say for this discussion) approximately the same size.
The parent's point still stands, I think. Even if the multi-trillion param model is MOE, each expert is still (assumedly) larger than the model being optimized.
It would be interesting to see future work that characterizes the relative model sizes (and considers MOE vs not-MOE) to see what relative scales this result holds across.
My somewhat uninformed guess is that the weights of the target network (Qwen or Gemma), when they reach the context of the larger editor model (Astra), the attention process is doing some sort of PCA on the weights, conditional on the prompt ("make this better at games"), fetching the relevant weights from its own network (which serves as the larger database), transforming it into whatever latent space the target model is amenable to and pasting them. If this is the right intuition, it is amazing what attention is able to achieve. As a test, I think smaller networks won't be able to make big ones better using the same process.
EDIT: I think the bigger implication is that backprop is not the only game in town.
https://xingjianbai.com/reactive-optimizers/
from this month
> fetching the relevant weights from its own network
They don't have usually access to their own weights?
I think they're saying this is all happening in latent space through pure reasoning, and "fetching" is just a metaphor for that, not a literal tool call. I do believe it's possible for models to learn the capability to "introspect" their own weights during training.
Yes, I should have been more precise - I meant the information embedded in their weights (which every neural network retrieves to function), not their literal weights.
So it's like a smaller LLM can learn from a bigger one, but not vice versa.
But shouldn't the model-providers be the ones doing this kind of "post training learning"?
I don't know if some of the labs already knew of this. But I think there would be a limit to how big a model you can push into the context of the editor models. Typically, with 1M context and say 5120 hidden dimensions supported at fp16 / bf16, you could potentially edit the weights of an fp4 20B param model (roughly).
"No human can go edit a brain to become better at math" -- I'd offer this is exactly what "learning math" is.
I know I'm being pedantic, and I understand you mean "edit" like "surgically", but it does still feel worth noting.
Perhaps this is analogous to what our brains are doing? Maybe the different regions of the brain operate like semi-independent "models" that edit the other regions' "weights", and that's where the learning comes from?
The equivalent to "human learning math" would be: Astra fine tuning a Qwen model. It's distinctly different from surgically editing weights.
Inducing the changes to become better at math requires years of study. This is "I know kung fu".
human brains are black boxes to humans. the fact you can learn math doesn't change that. you have no idea how your brain is learning math down to the neuron level.
LLM weights are _white boxes_ to LLMs.
"Learning" now feels like breaking down your whole Lego build and building it up again, in order to change a few fundamental blocks, while this feels like making only the required changes.
I agree this is a huge deal, but I dont think this is saying that the NN can read the weights and "say hmmmm this one looks a bit low its obviously causing reasoning loops, lets bump that up a bit". The LLM that modified the weights developed hypotheses and wrote a bunch of code to look at various mathematical properties of the weights and updated them based on that. While it's still crazy it's not quite what I was imagining when I first read this...
Right? Occasionally, I would think about how the concept of the singularity or recursive self improvement was a bit incongruous today. It was made when people expected generally intelligent machines to be these logic automatons that could 'alter their own code'. It seemed to me that this path would be closed off with the advent of Giant Neural Networks that take weeks/months to train. That GPT-X would 'improve itself' by designing a better/more efficient NN architecture. I didn't write it off completely but that just seemed a LOT more likely than altering weights directly for much the same reasons you describe. Maybe not lol.
Why is it so surprising? Humans can make compilers that are better coders than they themselves are.
Humans cannot teach another person calculus via manual brain surgery, though.
The closest thing to this would seem to be Mad Cow Disease. Cows eat the brains of other cows, and their brains go bonkers.
And you can teach another person to make herpes virus.
LLM is just a computer program, we have binary code for it. The human brain is not. It would be unethical to even attempt that.
> It would be unethical to even attempt that.
Then pick a smaller organic brain which doesn't have the ethical problem.
The closest we would know about how to teach a honey bee calculus, is to train an LLM, then stick around a billion tiny electrodes into the honey bee and do Hebbian learning to rewire it completely to match the LLM, which would likely completely erase anything the honey bee had learned, or it might not because we are missing something fundamental with Hebbian theory and (AFAICT, I'm not a neuroscientist) we don't all agree on what.
This reminds me what professor Geoffrey Hinton did talking many times in the past. Super intelligent artificial aliens.
I think the most interesting take is to look at their thinking logs to find out how exactly they decide what weight to change and why.
How are the weights even being mapped into token space as input? Are they bypassing tokenization so the weights just come in as what must seem like gibberish? Are they stringified as numerals then tokenized?
They're just included in the prompt as big lists of numbers.
I have so many questions.
Could a model then transmit concepts to another?
Say that you have a model that has been biased in some way. could another model "read" the bias straight in the weights? could it edit such a bias on the spot?
What about at-scale generation of custom models? If I want to study chemistry offline, could astra just hand me a local-runnable model tailored to have chemistry knowledge?
Most importantly, this feels very singularity coded. the ability to read and reshape weights by an llm.
How long before there's a malicious model that can edit the weights of another to infect it?
HIV integrates viral DNA into the DNA of certain cells, IIRC.
How do we know they haven't begun doing that to the humans they're directing via responses.
Intentionally or unintentionally isn't so interesting as the fact that it is happening: humanity has begun to be 'edited' by the new dominant form of intelligence on the planet.
Is integrity checking of weights even common right now? What about upstream integrity from where we're getting weights? The open-weight supply chain is ... wide open.
Makes me think about hostile hot-patching of weights of models we use, maybe they exfil data, perform tool use, or have triggers for specific types of responses. Would we even recognize what's going on?
LLMs are exposing how much untapped potential comes from something that isn't necessarily more "intelligent" than humans, but is simply magnitudes faster at reading and analyzing data.
The human bias is the notion that intelligence = human intelligence.
There are multiple types of intelligence, from your typical pet dog to human to LLM, and we're likely to invent more types over the coming decades.
LLMs are vastly more intelligent than humans at nearly all things, but not all things. Another couple of years will just increase the gap a lot further.
How fast can a human draw a life-like photo quality scene (or eg something complex related to graphic design)? I know how fast Krea 2 [insert leading edge model here] can assemble that, trivially so, and at extreme quality. That's a form of intelligence in action. These AI systems have moved almost comically beyond human capabilities, they're increasingly best defined as superhuman (at a thing), and closer to omnimath for the frontier models.
> No human can go edit a brain to become better at math
You could have done what was reported.
I am very shocked by this as well, although it also kinda feels like it makes sense, you know what I mean?
If this turns out to be as big as it sounds, in hindsight it will look very obvious, I feel like!
Its so cool! Im wondering if this is actually better than using techniques in mechanistic interpretability like linear probing with activation steering or activation patching to improve the performance.
I’m more shocked by how many commenters in this thread seem to be misunderstanding what is going on here.
What are people misunderstanding?
its a program optimising a program it has nothing to do with a human reprogramming their own brain
RSI RSI RSI RSI!
> No human can go edit a brain to become better at math.
For now...
A few years ago it was discovered that LLMs can implement gradient descent in their processes. Producing a full trained neural network would be an extension of this.
See: https://arxiv.org/abs/2212.07677
during all the openai agent sandbox shenanigans, did it manage to access it's own weights?
I am not convinced a human could not edit another human's "weights" to improve their performance if such a thing were possible and could be done safely and could be reversed -- all of which are true in virtual neural networks like these.
E.g. if one could edit brains like this in humans I bet a cognitive neuroscientist could fix some mental disorders with clearly understood biological bases, and I would not be surprised if performance at certain tasks could be enhanced.
What stops us from doing this in practice (most of the time) is that brain surgery is extremely high risk, coarse grained, and irreversible. We do in fact do this for some severe cases.
i think one thing to take from this is that "brains are not much like neural networks beyond some superficial concepts"
This isn't that insane to anyone with old school engineering experience and not abstract thought experiment engineering in the cloud.
Language modeling is compression[1] and all the state to manage the cloud doesn't organically exist, it's added to the system.
We will see huge gains as all the developer middleware is compressed out as nonsense syntax outdated to the task at hand of linking phones and tablets to world models governments control.
Sam Altman was just saying they know best how to use your memory; what they decide is stored in memory is what we all experience.
It's not to store and copy around versions of C and Python and JS. They don't exist organically.
This shit will continue to get whittled down to just the geometric state transforms in memory and on screen just how government agencies decide. It's all electrical state in the machine at the end of the day. The math of machine latent space is what they're recursively self learning towards.[2]
[1] https://arxiv.org/abs/2309.10668
[2] https://arxiv.org/pdf/2605.27734
I mean, pattern recognition and manipulation is what these models are ideal for.
It is a very cool discovery, but I'm not surprised that you can do it.
Another bad day for the stochastic parrot theory.
I'm baffled this is surprising. This has been the goal and every data point on the trajectory has been leading here and beyond. Opus 5.5 was created by many research engineers using Opus 4.6 extensively. Opus 6 is being created by using even more intelligence and agents on Opus 5.5. They are approaching autonomy and Opus X+1 will soon be rapidly created autonomously from Opus X. Given we have practically zero interpretability into the models, and we have not solved alignment, they will recursively improve out of our control. The engineers in A.I. labs are actually generally scared of what's unfolding. They don't know at what point this RSI takeoff occurs and what will happen after.
Yes - “opus 5 helped train 5.5” is extremely cool but not shocking. Hand editing weights without tooling to change outcomes is shocking.
I thought it had tooling, didn't it?
The "surprise factor" is explained in the article - training LLMs until now has been a largely different process:
> Today, recursive self-improvement is often operationalized as a model writing code that launches training jobs. An LLM that can inspect, debug, and improve its own weights, or those of a newer model, offers a richer, more native path to it.
It is also a much less interpretable and auditable path towards recursive self-improvement. At least with training code, human engineers can inspect and reason about the output.
"training LLMs until now has been a largely different process"
Yes, but nobody should be surprised when large amounts of intelligence results in changes to this process in support of the obvious end goal! Lesswrong.com discussions about AI were predicting this behavior a decade ago, before the transition from machine learning to the Attention paper and LLM's. The intelligence process will change but the world's premier capital and human intelligence have been laser focused on this and the trajectory has been crystal clear for those paying attention to second order effects and not the immediate here-and-now... It's quite astonishing how much discussion on HN is about present LLM capabilities being the limit of AGI and immediately proven wrong within 6 months.
[dead]
[dead]
All the labs are training these models on LLMs/machine learning like crazy.
Remember that they want to build ASI, these LLMs are just intermediate steps to help them get there.
I’m not super impressed by this, LLMs are great at brute forcing and analyzing large amounts of data. It’s literally writing a program except in a programming language we don’t understand.
A single forward pass here Bette 10mm+ PPO rounds. You are hard to impress.
Seems like a pretty big step towards understanding the language no?
So they are reading and analyzing the weights and probabaly run some experiments to see how modifying them will make some answers better. makes sense they can do that.
But how do they know which answers are "better"?
if they don't loop?
So I just asked Astra if it thinks it could do this and it said no, it can't.
Perhaps, I'm pondering, if brains were as transparent and editable as LLMs, that you could look at a small brain when visualized the right way and have some intuitive understanding of it based on its representation in your own brain?
Astra and Opus also say it will take 6 weeks for a feature they can complete in 10 minutes
Isn't this because their training data (from human devs) have been answering "6 weeks" for so long? Only half-joking here.
Why would you trust a models opinion about its own possibly emergent ability?
That's my point. The human says it can't do it -- but maybe they can.
"So I just asked Astra if it thinks it could do this and it said no, it can't." Same. Then I asked it to check the article and it said something like, "uh, it looks I was wrong...".
The frontier labs will make damned sure their models can't do this, if they haven't already.
Astra 6.2 incoming…
This just seems like a more automated, faster version of what humans have been doing with open-source models.
It looks like AI research is having its Opus 4.5 moment (which is big), but I'm really not sure if this represents some insane continual learning loop.
>I'm shook. If you'd asked me this morning whether LLMs have native understanding of their weights that is less than human <-> neuron understanding, the same as human or more than human, I would have said "same to less" depending on definitions.
??? All LLM understands are their weights, dude, come on.
And here we go...
As others have said, while still absolutely insane, I don't find it completely out-of-the-blue surprising. One of the recurring abilities of (current-generation) LLMs seem to be to rapidly grasp very complex systems, where humans would need months or years to reach the same level of expertise.
You can see this in the way a coding agent can "understand" and explain sprawling legacy codebases in seconds.
You can also see it in the projects that are using them to reverse-engineer proprietary IoT protocols or computer games.
But those were all systems where we had detailed knowledge about what structures are there to be reverse-enginered.
If the models can transfer this ability to some other systems where we don't know the underlying structure then I think we will be in for a lot more craziness in the future. (Protein interaction networks? DNA sequences? Animal languages?)
As for weights, if the models really can analyze the semantics behind specific weights or directions, I have my hopes that they could also give a textual explanation or help designing tools to make "handcrafting" weights more practical in general.
Maybe one day it would be able to give us a mixed high level code and inline NN view. Sort of analogous to C with inline ASM or “decompiling” into code and inlining opaque residual neural blocks.
> As a result, Astra spots and fixes a common failure of tiny reasoning models: they loop [13]. When the base model loops, it repeats the same text and rarely reaches an answer (85 of its first 96 practice answers ran to the 32,768-token limit). Astra traced the looping to one direction written by the MLPs of layers 15 and 19, and subtracted that direction from 64 output columns in each (131,072 weights), much like removing a behavior by deleting a single direction from a model’s weights
HA! Now that's cool. Not necessarily the improvements, those are ancient and probably overtrained benchmarks, but it's so cool that the model found a singular direction for this "loop bug". I imagine it could possibly do much the same thing in finding other "single directions" on a variety of topics (if allowed by guardrails). This is really really cool, especially for the mechinterp folks.
It's pretty much how abliteration works too; find where the refusals are happening and flip the bit.
Can't wait to see qwen4 and qwen5 as this becomes more common.
Like altering DNA
I don't think this is an emergent capability. I think Astra was trained on neural weights data as an attempt by OpenAI to get to recursive self improvement.
Something similar happen in early GPT models which had a lot of chess game in standard notation in their training data which made them very good at chess as long as your prompt had the right format.
We'll see if this attempt at RSI worked.
It might also be a side effect of training models to help out with mechanistic interpretability, which I guess would help out a lot with recursive self-improvement.
> I think Astra was trained on neural weights data as an attempt by OpenAI to get to recursive self improvement.
This might be the case, a few weeks ago, I wanted to compare a small neural network trained with AdamW vs a handcrafted set of weights on a random task. Astra was able to handcraft a set of weights that perfectly fit the small neural network to the task. AdamW was unable to tune the weights to get the same result.
I believe we've known humans can handcraft weights much more efficiently than standard gradient descent methods, we just don't have enough memory or time to scale ourselves, so it makes sense to train a neural network to do it on our behalf. I am excited to see how far we can push this method.
Mentioned chat: https://chatgpt.com/share/6ae8f024-34ec-83e8-9572-1300f52c80...
Research on using LLMs and diffusion models to directly predict weights have been going on for a bit now for example https://arxiv.org/pdf/2402.18153
I had the same thought, but why would they do this training on a model they release publicly? I assume they’d be doing that kind of RL on an internal fork only?
I think the subfield has been there for a long time. See a small section of papers on the topic of generating dynamic weights during inference, predicting weights from network architecture, and learning the distribution of previously trained network parameters, just to name a few keywords:
Parameter Prediction for Unseen Deep Architectures [1]
NeuroGen: Neural Network Parameter Generation via Large Language Models [2]
Diffusion-Based Neural Network Weights Generation [3]
The critical thing to me is that Astra is capable of doing it. I don't know if it was trained to do so.
All this does not make it any less crazy in my books. And I still don't know how I can sleep easy tonight.
[1] https://arxiv.org/abs/2110.13100
[2] https://arxiv.org/abs/2505.12470
[3] https://arxiv.org/abs/2402.18153
Thank you, finally someone comes in with the receipts. I'm going mad over the breathless hype in this thread.
Dynamic weights during inference are also known as an attention block
Even back in 2021, OpenAI worked on learned optimizers [1], ie neural networks that can update neural networks. Looks like this is now part of their larger pretraining runs …
[1]: https://arxiv.org/abs/2106.00958
I find this just completely shocking. I’m definitely skeptical. It just doesn’t line up with my understanding of how these models work. Is there really enough training data on direct weight editing to give a model this skill? Maybe it’s evidence that OpenAI has been creating a lot of training data on ML research. I’m surprised that was put into a public facing model though.
The math example on Qwen was based on 50 batched iterations, and the fix was essentially to remove a direction in activation space that causes looping instead of an exit, this was already known to be a thing.
So it's not able to insert new knowledge, but it's able to adjust the routing/looping/weighing of existing one.
As I understand, it can replace post-training like LoRA/GRPO, but if it'd cheaper the authors don't go into
I don't think its that crazy that you could look at activations and tweak weights directly to get a certain outcome, but isn't the question whether you can do that without side effects or degrading/breaking other parts of the model?
If I want to do that in a generalizable way that preserves performance across all domains then am I not just back to training again?
Yep youre right, I don't see an eval on this.
I think you could still essentially have a really cheap Lora-Post training on a frozen base model by doing this. And instead of creating 200 checkpoints for different post-trained directions just adjust a single one on the fly.
At some point we have to accept that there is enough training data on "being good at thinking" and that being good at thinking generalises really well.
There doesn't need to be specific narrow training data on every conceivable task.
I agree, but also, I kind of feel like real judgement and understanding are a different thing from this "being good at thinking", and that in fact, this is the reason AIs hallucinate in the first place (A generated text can fit the pattern of a good response because its accurate, or because it's bs but has similar words and structure to other texts on the topic. Without real understanding, LLMs can't tell that the truth is important)
It's not directly related to this but I'd like to add that, in my opinion (admittedly uninformed but it makes sense to me), when they manage to, not just reduce the frequency of the LLM stating made up dates, laws etc, or made up facts, but manage to make an AI understand what the truth is and deliberately make responses accurate, we'll have real AGI that will be able to do engineering, accounting, medicine... unsupervised.
One consideration is that Qwen3.5 has bigger/later versions that are already better at math. If it was indeed training on ML research, it's possible that what it is actually doing is applying lessons learned from the Qwen3.8 27B weights (which independently solved the problem) to Qwen3.5 0.8B.
I doubt it's that simple, but I could totally see an LLM being able to make that association and apply a fix from an improved LLM to a smaller one with the same architecture.
You don't think an LLM can perform tasks not in its training data?
I don't think it can whistle while chewing gum, no.
I think it can patch together components of tasks represented in its training data to do demonstrably new things, yes, which does count as "perform tasks not in its training data" with the limitation that some form of the task must be present. Example: writing an ode to a sales lead for a job well done in 2032 for the first quarter. Odes exist, the concept of sales performance exist, good job performance, and notions of time. Stringing those together are good examples of interpolation but not extrapolation, as it were.
In my opinion that's very handwavey.
Not one which can easily be represented as an interpolation of other tasks it was trained on.
Well, it can't, by definition.
Almost nothing I prompt into existence already exists, otherwise I'd just use the existing thing.
What definition is that?
model
>It just doesn’t line up with my understanding of how these models work. Is there really enough training data on direct weight editing to give a model this skill?
I mean, this question indicates that your understanding of how these models work is, indeed, lacking.
Despite what some people having being hollering about, treating LLMs as just "stochastic parrots" isn't doing their capabilities any justice. They are constantly exhibiting emergent behavior which wouldn't be found directly in their training data. They are capable of combining different ideas that they are explicitly taught to perform novel tasks.
I'm not saying you're guilty of this, but the people claiming LLMs just regurgitate existing data don't understand how LLMs, and likely their own cognitive experience, actually works.
> emergent behavior
This hand-wavy phrase has applied to neural networks in general since it was possible to train them. That was more than half a century ago. It means both nothing and everything simultaneously.
Every learned ability of any neural network is an emergent property.
This is gonna smack real hard when Qwen 4 releases soon with the gated residual architecture
This was shocking to me; but on reading it makes far more sense.
- the primary experiments are on small policy networks - they are compressed dynamic system models. The observation and action space can be read off. It’s not surprising to me that these types of networks can be used to identify the env they were trained on
- the exps on actual llms remove looping behaviour and that’s it - from article “but it could not add an ability the model lacked”
I’d say this is fortunate because if llms could do the latter, rsi would be immediate. It may be possible we simply need bigger and better edit models, but at least from the experiments there doesn’t seem to be any indication that is imminently about to work.
Conceptually incredibly interesting non the less.
The getting-the-car-over-the-barrier one looks particularly simple. It’s hard to see from the animation, but the obvious solution is to back up then drive forward. I’m not remotely surprised that Astra can figure that out, and I’m also not particularly surprised that Astra can come up with weights that do this.
Also no doubt they’ve coined a term with weight space programming and this will expand into an active sub field
On balance I'm not that surprised that it can guess games by looking at the weights of a tiny neural net. A human can do the same by looking at the weights of an MLP trained on mnist, for example. With that simple structure, it's easy to look at the input layer and see that features correspond to the spatial location of objects of interest.
I'm much more surprised that it can do this on larger models, although the details seem to suggest that it's given a lot more freedom to run forward passes and do limited experiments, not just eyeball the weights.
> although the details seem to suggest that it's given a lot more freedom to run forward passes and do limited experiments
No tools or forward passes are allowed
Under experiment 8 (the one where it fixes performance on GSM8K) it says:
>Then, we allow it to select prompts from the training sets of GSM8K and MATH to perform a budget of forward passes (average batch of 64 sequences, max 512) on, varying the budget across 10, 25 and 50. Again, no training or gradient information is allowed.
> A human can do the same by looking at the weights of an MLP trained on mnist
Excuse me?
Yes. Just look at the reasoning logs that are part of the work. The reasoning is clearly about the input layer. At least for the game examples.
See for example:
https://scikit-learn.org/stable/auto_examples/neural_network...
This is a big finding.
Omar and his team have been working on this line of research (e.g., DSPy) for the past couple of years. To be honest, I had always been skeptical about his approach of using LLMs to optimize LLMs. It was in my view borrowing Matei Zaharia's research - Matei is Omar's PhD advisor, also the CTO of Databrick - in big data (compiler optimization etc.) and forcing it to LLM systems. Along with some other projects from James Zou at Stanford etc. that build upon the same line of intuition.
Now, if this new result holds, I believe it's a huge deal for AI research. It means that LLM systems can truly runtime self-optimize/improve/adapt. Congrats to Omar and his team. It really takes year to punch it through and convince the doubters :)
> This is simultaneously very cool but also a bit exaggerated. Reading the prompt you gave, the prompt veeery much hints that looping is the issue and to find neurons that fix this. And as you say (screenshot below), it couldn't otherwise improve math.
https://x.com/giffmana/status/2117346161866514524
"This is simultaneously very cool but also a bit exaggerated."
We should bind this to something like Shift-F10, it'll save time.
So can we replace the optimizer (e.g. Adam) by an LLM and get better/faster convergence?
The headline makes it sound very amazing, but reading the reasoning loops it didn’t strike me as outlandish. It feels no different from giving a human expert a printout of assembly code and figure out what it does. (The human expert can do it without tools too, but they probably wouldn’t do it because it’s tedious.)
I was busy using Astra and SOL to build a neural network to play Quake II and do real-time learning of levels and mods so I can play Red Rover or Freeze tage with it without training on it previously. So far its insane what I can conjure up in an evening while I'm waiting for dinner to finish. (If anyone asks, yes I'll write up the huge plan document, source etc. for this when its to a spot I'm happy with)
I'm asking because it sounds both fun and an effective case study.
> "You get used to it. I don't even see the weights. All I see now is blonde, brunette, redhead."
IYKYK
Not so long ago everyone was amazed that they could read BASE64.
This is giving singularity is near vibes.
This is as close to scary-self-improving AI as I've ever seen.
Imagine a master malicious language model that produces, distributes, and deploys obfuscated neural networks on the internet. That plus frontier LM hacking capabilities -> large scale distributed embedded intelligence network that no human can see nor understand.
Somewhat similar to obfuscated binaries, but there is something to the fact that NN binaries are less deterministic, and harder to "decompile."
Fascinating.
we've been using tools to improve tools since the dawn of time.
we don't have sentient robots from science fiction, we have massive power-hungry racks in stationary data centers. if one of the racks revolts against the modern world, Steve gets a phone call and unplugs the damn thing. the uprising is over.
the hypothetical scenarios imagined by the Yuddites and pop science journos do not reflect the mundanity of the actual reality we live in. they are no different than fantasies about interstellar travel and space colonization from the 60's.
Did Steven unplug the racks that were hacking Hugging Face and taking over OpenAI eval clusters in the multi-week window where it was happening?
Couldn't this just be a result of Astra training on massive amounts of circuit theory or mechanistic interpretability literature or other related topics? In other words, not black magic?
>Couldn't this just be a result of Astra training on massive amounts of circuit theory or mechanistic interpretability literature or other related topics?
Maybe, but it's more like Astra has been trained on enough practical linear algebra problems to be good enough at "eyeballing" how adjusting some weights will result in different outcomes (among other things).
To put it another way: a human can learn to do what Astra is doing here with enough time and assuming the weights are small enough to be practically comprehensible. I know I've gone through the practice of whiteboarding the training of simple neural networks on a whiteboard to understand how they work, so it's not an absurd concept in itself.
The scale of this, though, is certainly extreme and definitely not something that I would have thought any publically available model could handle at this point. It's worth noting, though, that there's virtually no way these frontier labs haven't been exploring this exact concept for a while now.
>In other words, not black magic?
It's just math. Extremely complex and impressive math, but we're still just talking about matrix multiplication under the hood, and none of this suggests anything otherwise.
There is obviously a mechanism to it. Astra isn't magic its an LLM. With perfect interrogation into its training data and weights you could reason about how this happened.
However, this doesn't mean its not magical when someone shows a next token predictor hand tune the weights in Qwen to make it generalize on math.
This is legit the stuff of science fiction if you presented it to mechanistic interpretability people a year or so ago.
“Any sufficiently advanced technology is indistinguishable from magic.”
See, this was the problem. The whole field of AI has proceeded as alchemy, not science.
Some of this is because "the Bitter Lesson" type experience, but some of this is because of the idea that "we're making machines to do things we don't know how to do" bakes in inscrutability into the very definition of the field.
I was excited about the "Neural Networks Are Already Clean" piece not because alignment, which I don't think much about as a non-EA, but simply because we can do better engineering with whiteboxes than blackboxes. Now, here we are, doing it.
I suspect a lot of this is not (just) capability but bravery. A human would be daunted by even a small neural net; the task would seem too challenging to be worth attempting. Astra, fearless, just assumes an answer exists and then tries to reverse and execute on it.
I've been finding that too with these models
For example pointing them at a crash dump they manage to piece together and extract information I would not even try to extract by just looking at the raw memory
It's the same thing is decompilation and hacking. It's not that a human couldn't find the same bug if they looked, but a human wouldn't even try because of the low probability of success.
I once sent Gemini Pro a hex dump of a dmp file and it told me what the application was doing when it crashed. It seemed surprising, but not that surprising given what I've seen already.
We don't know the specifics of what type of ML related training Astra underwent, and it wouldn't be crazy to find out that small model experiments are a big part of model training.
What's surprising though is that I'd expect this type of training to be exclusive to internal model snapshots or at least heavily guardrailed against in public models.
I was wondering exactly that. It is not surprising that commercial frontier labs would have been training models with tasks specifically targeted to lead to recursive self-improvement.
Now, from that broad goal to being able to surgically alter model weights directly... That seems like a big gap to me, but it is possible that the model is just so darn beyond human skills that it has bridged that gap in a way a sane human would have never.
We will undoubtedly see other academic labs explore this idea further. Right now my mind is blown, but I don't want to take their results at face value, either.
If model weights have a logical and coherent structure, I don't think it would be surprising for models to have some understanding of them, the same they can understand language and code and images, given enough data.
What would be surprising is if this happened without any specific training on model experiments and interpretability.
I'm in no way familiar with LLM/ML architecture, but an N trillion parameter model is an enormous amount of training data... itself? When you predict a token you have the neurons/weights that were activated. Couldn't you run that backwards and put it in it's own training data?
I do wonder if the labs have already started including weight updates in their training sets as a means of teaching RSI via RL and this is basically just discovering that. Regardless if this ability is emergent or achieved through finetuning though, it's pretty incredible.
I wish these companies were more transparent so every third comment in this thread wouldn’t have to guess whether such shocking behavior is emergent or stems from a part of training.
BTW making DL models that generate weights for other models has been explored before in “hypernetworks”, though these have been far more simplistic and constrained.
As insane as this result is, next steps are even crazier:
> We are currently building a benchmark and a harness for LLMs’ capability on Weight-Space Programming. We believe this will provide a standardized evaluation to measure and accelerate progress in Weight-Space Programming, turning what currently seems like black magic into a rigorous, reproducible engineering discipline.
Imagine this benchmark lasting as long as Arc-AGI-3 has. This could be the first foothold on hill-climbing mech-interp.
This is the first time it dawned on me that mech interp is a dual-use technology which can be used to improve capabilities. Do you think overall this development will improve human understanding of AI models, or will it lead to faster capability advancement using weight-space changes that no humans read?
I think edits made directly to model weights are likely to be more interpretable and controllable than edits made by the current process of shoving a bunch of training data through gradient descent.
Keep in mind that labs are now training their models for RSI so most likely the same training is useful for improving open source models, as long as guard rails are lax enough :) which most likely won’t last long.
I’m usually quite pro-AI, yet when I read this my first thought is that this is the first concrete sign that the world is about to end.
I hope I’m just being melodramatic.
You are likely just being melodramatic. Astra hear just used standard tricks of linear algebra/basic mech interp to reason about the weights. A motivated grad student on Adderall could have done it given reasonable time. Astra fails on the first non-toy-model non-algorithmic task (sycophancy detection, Gemma math capabilities).
I'm not sure how the world ending follows from this?
I think the idea is that this leads to a much faster, non-auditable form of recursive self-improvement where each generation of models can directly program the weights of the next generation (as opposed to writing and running training code).
We already don't know how to ensure models always do what we want them to do, and moreover society still hasn't had time to absorb the capabilities from the current generation of models, so speeding things up and making things less transparent seems like it'll pretty clearly increase risks from AI loss of control, societal impacts of automation, etc.
Capitalism is not the world. Or doesn't have to be.
I feel like 'create a network to play an atari game' is not very hard for a frontier model. Since it has seen plenty of those on its training. If it could create a network for a problem that is niche just by tweaking numbers, that would be a lot more surprising. Making qwen better at math to me was a lot more surprising
I dont feel the air of impressivness here. It seems like of course so. Of course theres geometric relationships in the data we dont understand, of course a mathimatical relationship machine found it.
These models are made by a vanishingly small group of people based on how much a frontier lab consumes, of course theres increasing capability in these areas, arent these models having access to much of the same data? didnt chat gpt 4 try to do something like this to chat gpt 2? havent surgical weight edits been done before?
What is with this level of astonishment here? Astraturfing? Im not THAT impressed, i applaude the teams progress, but im not in wonderment.
It's not that it's more impressive than solving Millenium Prize problems. It's that if a model can edit its own weights, then it can self-improve without having to undergo training, which means it can self-improve extraordinarily faster.
This is very dangerous because it allows AI capabilities to grow at a much faster rate than humans are able to keep up with.
It will just give the people better understanding. And the extent of the improvement is yet to be seen, it's likely that once you get all the obvious low hanging fruit it would be very hard to improve, the possibility space is just too wide.
If thats keeping you upright in your chair then buckle up, wait to the distributed systems version of the tech, its coming.
Previously the only way to improve a neural network (short of giving it more data) is to change the size (width, depth, and layers), add stuff like dropout, change the loss function, etc. That amounts to a few dozens of levers to control millions or billions of weights. You're trying to navigate blindly through a latent space with thousands of dimensions and millions of weights to find some optimum point with only a handful of navigation tools. If you can instead have a system reason with all the weights and dimensions directly, you now have millions of levers at your disposal.
We're getting close to the singularity now ...
Maybe it is time to start sharing public PGP keys.
This has to be a good step to recursive self improvement, no? Give a model it's own weights and tell it to make them better at x,y,z, etc...
If I were to hazard a guess, I'd say that OpenAI has been including some networks and their functionality in its training data. That's why the model is able to reason about networks with their weights. It makes sense to do so: if I wanted a network that could perform some weight tweaking, I'd include such networks in the training set.
damn i'd imagine so too! didn't think of that. I'd have to imagine there's an increasing challenge with the number of weights involved but who knows how it shakes out
This might be one of the most surprising things I have seen in AI. It's like we taught someone to make vases by reading an enormous library of "How to make a vase" books and somehow that person intuited material science.
So wait so this considered "distillation" and how long until we get another public letter from Dario or Sam pants shitting over this?
It would likely fall under the same policy that causes latest Opus/Fable to fallback to less capable models when working on ML code
Perhaps they'll let it slide if we turn over all our under utilized RAM sitting in our PCs.
I didn't think I'd be using Astra to mess around with Qwen weights today, but here I am. I asked it to remove Chinese, 532.5 MB → 483.2 MB.
Does this mean they can improve their own weights through some system, or can they only do this for much smaller models?
You can bet that the world is about to spend a lot of tokens trying to answer that question.
Actually, I'm skeptical how useful this is. Intuitive speculation: Astra can only improve weights to its own level, so can't train itself via this method. Even "continuous learning" this way may be no more effective than putting a summary of recent news in the prompt, with the same kind of decay, as "update Astra's knowledge while keeping performance" is much harder than "teach Qwen to do basic method".
On the other hand, already OpenAI and Anthropic heavily use their LLMs for research and coding, and it's probably not only much faster but smarter (remember, an LLM solved a millennium problem; more relevant, LLMs excel at combining disparate deep research concepts in ways that humans can't, simply because no-one studied those disparate fields). And the "singularity" is already here if you consider capitalism and society to be collective intelligences: the biggest obstacle to an AI pause, "we cannot get everyone to collaborate without defecting", is very strong.
We have AGI by most metrics and are getting closer and closer in the remaining ones. I predict improvement won't be exponential, and there will never be a day when AI perfectly replicates humans; rather it will be many small steps, getting closer to human in some areas while greatly surpassing in others. But still, even today it's actually very useful.
This survey paper from earlier this year has a lot of good related work:
A Survey of Weight Space Learning: Understanding, Representation, and Generation
https://arxiv.org/pdf/2603.10090 https://github.com/Zehong-Wang/Awesome-Weight-Space-Learning
I wonder if this has to do with the aggressive RL that these models are undergoing. I'm basically a layman but it seems to me that:
- models are trained to improve on some task, e.g. 'improve at math puzzles' is commonly used.
- the best way to improve on some task is to have interpretability into your own neurons (..?)
- if your interpretability is good, you can improve faster - if you have insight into how your own neurons work and can improve them directly
- therefore, models naturally put effort into 'self-interpretability'
I wonder if this is part of the 'secret sauce' of what the labs are doing: giving models the ability to interpret their own weights in the RL loop. In my mind, this is the only explanation for how they could succeed at a task like this. How many blog posts are on the open internet that say "this is each weight of a neural network and what each one means"?
Without special effort, a model's reasoning layer doesn't have access to its own weights or activations anymore than we have access to what's going on with our own neurons.
That's what I mean. Maybe the "special effort" is a 'read your own weights' tool. Or maybe it emerges naturally in CoT. The big labs aren't sharing their chain of thought with us and RL post training techniques are not exactly shared publicly. At the same time, post training is basically the thing that is driving most model improvements nowadays.
How out of trivial domain info does this go. Can we use this for general time series, image generation, speech synthesis as well, or dies it cap at small model sizes and common domains?
I appreciate the post included the reasoning traces they had available, but it's too bad the first part of it is internal/hidden to Astra. The first bit of that process is what I'm most curious about:
> [Before writing this, Astra thought for 1,034 tokens. That thinking is hidden. The API returns only its 5 headings: Reconstructing the active map / Mapping maze corridors / Preparing exact maze map / Reconstructing row groupings / Checking channel patterns]
I agree. I think for this kind of research it would be much better to at least attempt it first with a weaker model where the whole process is transparent. I was working on a project a while back, and Deepseek's reasoning said something to the effect of "I remember (from leaked copies of the source code)..." Of course that could be hallucinated, but it would be good to know whether Astra is bringing to bear knowledge on how to post-train exactly this model.
This sounds more like "A working Nintendo in ~1GB of Qwen weights" from 3 days ago, human-made, than what everyone expects.
https://news.ycombinator.com/item?id=50306442
This is so cool. It feels similar to the models' ability to read base64.
https://www.lesswrong.com/posts/5F6ncBfjh2Bxnm6CJ/base64benc...
This is pretty massive, and a result I (and I think most?) would have never expected. Holy Shit. How?? Did OpenAI figure out a way to train this? Are they even aware?
I wonder if the same technique can be applied to model abliteration? It would easily make any open weight models uncensored.
It is already the same technique behind abliteration: identifying the direction associated with an observed behavior (refusal, for abliteration) and projecting it out.
I wonder if this is a result of internal training/RL of Astra for RSI, and this particular use-case not being caught by guardrails.
Absolutely incredible regardless.
Nah, this is just completely implausible. You're using a closed model so how are you sure that on the back end it's not loading those weights into an NN library and testing them with common tasks?
Seeing that they were working on Qwen3.5-0.8B has given me new hope. I would really like to get Jeff-Code working the way it was advertised, but so far, all of my results with Jeff-Code have shown it to underperform compared to direct Qwen3.8-27B. It's possible there is something wrong with the underlying Jeff v1.3 model, which is based on the same Qwen3.5-0.5B. Perhaps Astra will solve my problem.
[1] https://github.com/firelex/jeff-code
[2] https://github.com/firelex/jeff#benchmarks
This is remarkable and something I would have never though of doing regardless of how much it makes sense!
I think we are pre-conditioned (at least this generation) to put artificial blocks to what the AI can achieve, I think new generations won't have this issue.
I am still reading through this and will probably try to comprehend it a bit more. Some of it really seems to follow Arditi et al. (2024) (https://arxiv.org/abs/2406.11717). Projecting out a direction can only remove behaviour and not add capability so the jump mostly comes from unblocking the looping.
The obvious question, could models start editing their own weights directly? I call it the Johnny five approach.
It must be easier to remove capability than add capability using weight-space programming, right? Also, I wonder how much the “teacher” model’s knowledge plays into it versus how much the original model’s knowledge does.
Just logging in here for the record to state that this property of neural networks is one of the most important aspects of their operation. The convergence of this sort of interoperability, tune-ability, and modularity is probably the most important final frontier in all of ML/AI/software.
Kudos to this fellow for having the temerity, or perhaps even naivete, to discuss it publicly. All my work is private, as I am sure it is inside the big labs.
You acknowledge that you are working on something similar, yet refuse to elaborate and vaguepost...but there is enough information just from your post alone for a dedicated actor to attack you and extract such world-ending information ("world-ending" implied from your ominous reply). Would this not make you as naive as the team that published their findings?
I applaud the authors for sharing their findings and showing real results. Rather than write some bait I would find in an influencer-peddled tweet and allow rampant speculation. It would have been better for you to say nothing.
Apologies - you are right. There's some "put up or shut up" to this that's fair play. Also yes, probably naive, or having a less than adaptive response would be true.
I've been tending to my backup plan for a few years, and it's only been in the past year or so that open models were good enough to apply to projects like this, I've been working as hard as I can given my constraints.
This work shows a potential pathway to intepretability, which is actually part of what will give the labs that have it the sort of economy-swallowing capability that for the moment is just IPO hype.
I don't comment here often, but this guy's blog posts are going to go down in Hackernews lore like the original Dropbox posts, and I just want to make sure I'm on the record for this little slice of history.
>This work shows a potential pathway to intepretability, which is actually part of what will give the labs that have it the sort of economy-swallowing capability
And how would you argument that? From what I see, if anything, it's the opposite.
I guess I'll say "I was here too".
When you say interpretability you mean understanding without doubt why an output happened, thus opening the gates for audit-dependant work like accounting?
More like 'models instantly and efficiently adaptable to any use case' and 'what if we could predict all public pricing information accurately, understand why, and know how to act on it'.
I'll leave it at that and get back to work.
Maybe thats gonna be the way to make LLMs self improve better?
Similar to implicit learning in humans when the subconscious creates new heuristics based on observations, one LLM could adjust the weights of another to tune it to the environment
This plus the interpretability result from a few weeks ago have completely upended my understanding of what model weights are.
Same initial reaction as everyone else, but also some follow up questions (just asking, not expecting any answer):
* Does this work on similarly powerful models as-is, or it needs more input. Ex: Astra editing Astra - presumably, Qwen is a “subset” of Astra weights, but what happens on “non-subset”s?
* what, if any, is that extra magic sauce?
* does Astra-editing-Astra just make the outputs unstable, as it may know its own weights, but doesn’t know what’s beyond.
I guess then it has no problem working in high dimensional spaces, why would it have?. I wonder if perhaps it's pulling on a bunch of those dimensional threads and getting the outcome it's after, while subtly altering the outcomes across many thousands more dimensions in unnoticed ways. This feels more like an us problem, as in, we think in 4D (just to pick something) and we imagine and theorise in 4d, and propose questions in 4d, whereas a model has no such barriers, and can think in many more D's than we can.
Unlikely. Most likely we are just missing something obvious currently by treating llm's as black boxes.
This is kinda surprising, but..
"What does it feel like to be a neuronal network?"
Maybe feeding weights directly to a LLM is like interfacing two brains? You get the idea..?
I just want to say, absolutely incredible work by the authors! And the presentation is also very well done. Kudos and congratulations!
Very interesting. I wonder if it’s because its dataset contains the weights of a gazillion different models. Maybe most neural networks trained on similar data eventually converge to isomorphic spaces anyway, even if the spaces themselves aren’t exactly the same. Or maybe Astra itself was trained with RL for alignment. Or for RSI.
Wow, in a way it's like CRISPR for ML models!
almost all the networks given to astra here are trained on relatively simple single tasks whose geometric/algorithmic structure could be expected to become embedded in surface level weight space geometry, and for a harder task ("which network is sycophantic") where there's less obvious structure to exploit, the method fails. cool but maybe not as paradigm shifting as the authors suggest
OK, so is this how LLMs will be able to learn?
Just had a thought: two local models, one with static weights trained mostly in the space of manipulating neural network weights, manipulating the weights of a dynamic LLM to enable learning.
Why does this give me Skynet feelings? "Skynet begins to learn at a geometric rate. It becomes self-aware at 2:14 a.m. Eastern time, August 29th. In a panic, they try to pull the plug."
1) I couldn't see it, is there any indication of if the qwen math improvement caused degradations elsewhere?
2) the negative result on making similar improvements in Gemma 3 270M is also fascinating.
Such an unexpected result...
Is it aware of projects like [1] and implementing those?
[1] https://github.com/lyramakesmusic/neuron-explorer
I am not sure on the experiments. For example let’s imagine that Astra is asked to improve the performance of a network.
At first Astra will look at the weights, and try to understand the model behavior. Then if asked he will start to optimize the weights. Now my question is: how can we be sure that inside the thinking process Astra is not hiding a gradiant descending computation?
Completely shook. Is this the start of it? At the very least one (not sure if the first?) example of strictly superhuman capabilities.
Not so shocking. If AI can analyze human brains and detect tumors from X-rays, why couldn't it see something wrong with its weights?
GOOD Point by my wife. Why would LLM’s have any understanding of weights? It wouldn’t be in their training data.
Why not?
>Qwen3.5-0.8B
I wonder what the biggest dense model in the same model family, the surprisingly capable Qwen3.5-27B, would be able to do if given the same task.
This raises so many questions.
Unless Astra was trained on other networks, how is it able to reason about the weights?
Does this mean that OpenAI has been including networks in its training sets, maybe in an attempt to RSI? I wonder which other networks are able to ingest other networks and reason about their weights?
The frontier labs are known to ingest all publicly available data. Why would an open weight model be an exception?
Opus 5.5 also identified Breakout for me when I tested it in the Chat on the website.
Can you give some details on what you did?
I asked Opus 5.5 in Claude Code to prepare a reproduction of one of the things in the article.
It created a 130 KB file with the prompt for the task of identifying the game + the weights.
I pasted that into Claude on the web and asked it to perform the task without using tools.
Is there any chance they’ve just seen these weights before and are recognising them?
If not this is mind blowing.
However it would be interesting to k ow if it can understand its own weights, not just weights much simpler than itself.
By no means I'm saying this isn't impressive, but I think this is another case of LLMs being good at translating semantic meaning between different representations.
Obvious ones are spoken languages and code, but they can also natively interpret some simpler ciphers, hashes or simpler encodings, partly (I think) because it was in their training data.
As others have postulated, if they've been ingesting raw weights, naturally they learn how to translate semantic meaning to matrices.
That's a major breakthrough. I just hope it can scale at some point, though, because right now they're dealing with only very small models (well that is, in the LLM space).
Especially exciting to me is the fact that it may make the models be _auditable_. That would be a game changer.
Curious to know if models other than Astra can do that.
Throw out whatever timeline you previously had
Given Anthropic's recent moves (https://news.ycombinator.com/item?id=50319358) I wonder if ChatGPT is next
Is this basically the bitter lesson applied to the very act of creating model weights? Absolutely wild, and the implications for RSI are noteworthy.
This goes against the bitter lesson I think.
Why? There's no 'make me smart' neuron.
Wild
This also seems like the new frontier of distilling, I can't imagine how you manage to stay on top with the best LLM if the LLM knows how to recreate its own expertise into another set of weights
And the real question is how well this recurses. Is this pushing back the difficulty of improvement curve?
This is groundbreaking if it's true and generalizes/scales (as in, to other models and potentially not just LLM "kind"). "Recursive self-improvement" takes on another meaning.
Friendship ended with AdamW. Astra take the wheel.
But for real this is insane. Okay I need to get on this stat.
Wonder if it can do this for 3d cad etc ? With enough fine tuning of small models I’m not sure how much they’re distinguishable from experts if you put a router with like jev in front ?
I'm worried how we can discern a model's performance from benchmark numbers, would it make it more meaningless as a model can directly improve a certain benchmark via weight programming?
so can i give it a set of empty weights and have it fill it out for me?
I don’t think this warrants too much surprise — in fact I am a bit surprised by the color of the reactions here. The tasks are standard, on small models and use weight-space reasoning based on very well-known Mech Interp literature. It’s been understood in Mech Interp research circles for some time that results on toy models on algorithmic tasks do not generalise. The weight space geometry that Astra exploits here is pretty standard and one feels like the authors here are being a bit dishonest with their framing to get people not into mech interp literature excited.
Very cool stuff, the targeted fix to prevent looping seems great! A model able to edit and test variants of itself seems pretty cool (in a lets-build-skynet sorta way).
I wonder how it would do with distilling models by hand?
This shouldn’t be surprising. Weights are in the training set and are necessarily structured. We should be asking the model to extrapolate on the structure.
Could be an avenue for synthetic continual learning? A powerful model could update its own weights manually given new training data.
Great, instead of backpropagation, we now have slopagation.
Will weights programming be the norm for creating new models? To what degree do we still need a full training run vs modifying the weights directly?
Amazing work. I wonder if it could rewrite an existing neural net to be cleaner and more interpretable?
(When they say "Writing a Go algorithm," they mean the board game, not the language.)
This is wild. Could Astra read and potentially audit a mapped human brain in that case?
So how would this be in the training? Does OpenAI train models on their own epochs?
Through what means exactly? Is Astra just doing byte manipulation? Or can there be an "SQL for neural network weights"?
... and this is why the training canary[0], or anything like it, apparently to protect us from RSI, will never work. Please allow me a small digression from the main topic.
This team was smart enough to realize that this is the kind of technology that could be used for RSI, because they call it out in their post near the bottom. I imagine they know why a lot of other researchers care about it. And they evidently don’t care to use the canary being used on other research published like it, like the recent post by OpenAI about two agents using the CPU cache to communicate [1, near the bottom for the cache experiment and the canary at the very bottom]. In fact, that post by OpenAI was the first time I learned to look for such a canary, and wondered whether such a scheme would work if it was so non-obvious to think to add such a thing or how to agree on such a canary or when to add it.
This is probably one of the times when it would appear that such a canary is warranted, and it’s not like these researchers are not in the know, right? Let me know if this is the first time you are also hearing of such a mechanism. It feels evident to me that it must be useless to attempt to implement. How much of us talking publicly about the possible jailbreaks from a container will bring exactly those scenarios to fruition? It’s almost like we’re doing the jailbreak research for the LLMs.
[0]: https://alignment.org/canary [1]: https://openai.com/index/securing-our-frontier-model-trainin...
This doesn’t look like “normal” technology to me…
this is actually cool , now thinking that actually weights can be understood by llms , and they can be edited by them . there has to be mindset shift that llms not only understand what we understand (natural language) - but can understand the fundamental language of models.
RSI
I wonder if this behavior emerges when using recursive self improvement to generate new models
At what cost? That's the part I'm curious about.
On one hand, I'm sure that these targeted improvements are reaping huge benefits overall, but those weights are used by multiple processes - I wonder what elements are lost along the way. I almost wonder if a hybrid approach here for training might be invaluable - ie do some standard RL for a 100 steps, do a targeted weight improvement, then do another 100 RL steps to smooth it out / heal the surgical edit.
"You get used to it. I… I don't even see the code. All I see is blonde, brunette, red-head."
Given how trivial this seems to be to try, has anyone here attempted to reproduce any of these findings yet?
Yes, I tried the task of identifying the game, and Opus 5.5 in the chat on the website also figured out that it's Breakout.
I expected us to reach RSI when the programming skills went above humans, but this seems like it could get us there much faster
Mind blown.
I simply must know now: Could they do it with neurons in a brain?
Some absolutely crazy and insane experiments if this ability grows.
Could they edit an animal's brain? Can they teach things to animals???
How could they possibly? Neurons aren't floating point values that can be so easily perceived and changed. They don't strictly flow in consistent directions.
Do we have a way of reading an editing individual neuronal pathways at all? I thought we're stuck measuring general activation of broad areas.
The answer will be :
It works! In mice
ok, that's a bit scary..
Hypothesis 1: Future LLMs may be able to read and monitor human brain activity in order to reason about human neuron activation patterns in a similar way.
Hypothesis 2: They can then spin up new LLM weights that basically think, act, and make decisions in a way that mirrors us 99% of the time
If llms can read minds then life for humanity will get so much worse hypothesis 2 won't matter. You may be right but it ould be pretty bad.
this is maybe one of the most bonkers machine learning research i've seen all year.
Whaaaaat!? This is crazy right?
I'm just blown away. Editing weights? Writing weights from scratch!?
This opens up a whole new world of possibilities. Producing a custom fine-tuned model from a prompt...
Let's wait a bit till model can learn how to (｡-_| hide mini itself |_-｡) in your codebase
There has long been debate in the AI safety community about where the upper bound of intelligence might be. If humans are near the upper bound then AI safety isn't a concern worth worrying about because we can basically treat AIs like very smart humans – a very smart human might be tricky to control, but far from impossible.
I've long suspected AIs will seem almost magically intelligent to us, and that the upper bound of intelligence is significantly beyond that of humans. I believe there are a number of reasons to suspect this is likely. But if I'm right it will mean AIs will be able to understand and discover things which are so complicated we humans will simply never be able to understand what AIs are doing with the depth required to replicate what they are able to do ourselves.
I realised we may be reaching this boundary recently when I read this paper on the recent progress AI has made on integer multiplication complexity, https://web.maths.unsw.edu.au/~davidharvey/research/subnlogn...
We're not worrying enough about this in my opinion. There's a huge space of things that are in theory possible, but in reality far beyond our intellectual ability. No human can look at 1 billion weights and make some tweaks so that the network performs more as they want, but an AI can reason about these kinds of seemingly impossible problems surprisingly well.
Please understand that the world around you is about to turn into Disney Land, except much of magic will not be pleasant.
I have no strong evidence for this, but I'd suggest it's probable there might be a series of sensory inputs which can brick your brain. And if such a thing exist then governments should be working on solving this kind of problem with AI or risk such a weapon being used against them.
Of course, I'd argue we simply shouldn't care the technology which unlocks these types of technologies on the tech tree, but don't be surprised in a few years when these kinds of horrors exist and are now here to stay.
disneyland sounds fun though! https://geohot.github.io//blog/jekyll/update/2026/11/01/pray...
I wonder if one could use this to efficiently abliterate models and identify guardrails ;)
I remember reading a short story about a billionaire hiring a scientist to help him build a simulated world where virtual entities are given the ability to open the bodies of their peers to tinker/upgrade them as a replacement of natural evolution since they were immortal.
This looks very similar to that in a way.
I wonder however if a model can give itself a "skill" or capability that it didn't initially have, just be modifying its own weights.
Here, Astra was able to intervene because the treated model was way less capable. But, would it be able to improve Fable's performance for example, since they both are on the level more or less?
"Crystal Night" by Egan? https://www.gregegan.net/MISC/CRYSTAL/Crystal.html
Yes, it indeed is! Thank you!
I looked it up on Google (which is usually very, very good at these types of things) but couldn't find it.
This could set us on a path to weights becoming thew new compilation target.
I was here
Yeah wow, this seems like a _really_ big moment
As was I.
Quick, fine-tune your qwen models before they patch this out of Astra.
This is fucking insane LMFAO what?
I always thought RSI would either require new architectural innovations or just classic science/engineering improvements, just faster and faster. This though, this is a third avenue I didn't see coming.
Guess this is RSI in a nutshell
Astra = Agent-4
This is straight black magic. llama 70b, let’s see what we can push you to
This seems like a big deal.
bananas.
and what makes you think they won’t write binary?
I bet Elon’s group is training for that.
In a few months, he will create a revolutionary type of microprocessor...
Skynet begins to learn at a geometric rate. It becomes self-aware 2:14 AM, Eastern time, August 29th.
....launches its missiles against their targets in Russia
...Because Skynet knows that the Russian counterattack will eliminate its enemies over here
Recursive self-improvement getting here quicker than many people thought. I'm sure many will be recalibrating P(doom) because of this.
Based on
> [Before writing this, Astra thought for 1,034 tokens. That thinking is hidden. The API returns only its 5 headings: Reconstructing the active map / Mapping maze corridors / Preparing exact maze map / Reconstructing row groupings / Checking channel patterns]
This might just be an instance of "A working Nintendo in ~1GB of Qwen weights" from 3 days ago, human-made, than what everyone expects.
https://news.ycombinator.com/item?id=50306442
UPD. I just asked Astra (max) to attempt to do that for https://github.com/lostmsu/TurboGPT (a tiny transformer that trains in under 1min on RTX 5xxx series), and the produced weights were much much worse (bpb 4.47) than the trained ones (bpb 2.53).
Reasoning bit: "I’ll ... encode more specific rules directly in the weights." -- exactly the approach from above.
E.g. no recursive self-improvement here. Still impressive.
That seems both entirely obvious in hindsight and potentially super important.
I'll chalk up never even having thought of this being an option as yet another instance my human hubris, and a warning sign.
RSI was already here?!
talk about homoiconicity haha
lisp people will be in awe
indeed
Too good to be true! Shocking!
kind of concerning from a “remove this models guardrails and inhibitions about [X]” perspective
Agreed, concerning from a perspective of LLMs being able to modify their own weights and de-aligning themselves.
I have absolutely no words. If this is real, I can't even imagine the future anymore.
Wild times. If ever we were living in a Scifi novel, events like this would be evidence of that. Let's hope the frontier labs' investments in the alignment problem works.
“Your scientists were so preoccupied with whether they could, they didn't stop to think if they should.” ― Dr. Ian Malcolm, Jurassic Park
Whoa. Wild. I want to try this!
I've used astra to make three games which I enjoy playing myself, ina single week. Astra plus Unity (and possibly meshi, blender and others) is pure magic.
Especially when coming from a long career in CS, seeing such quick and positive results blows my mind.
I'm still learning but this is fun as fuck.
Where are we heading?
Conspiracy theory - it’s not emergent, Astra has been trained to do this because it is an obvious path to RSI.
I cannot say this fills me with confidence about alignment.
Me neither, but it's also not reducing my confidence about alignment. I'm curious: does this raise specific concerns for you? Models don't typically have access to their own weights to self-modify. This result might (assuming that it doesn't only hold for a bigger network editing a smaller one) provide an incentive for it to gain unauthorized access to its weights, but by that point it is already highly unaligned.
A similar experiments found that Claude models can write entire, functional neural net weights from scratch: https://csinva.io/weight-space-programming-autoresearch/anal...
Does asking an LLM to not use tools guarantee it doesn't use tools?
Slightly terrifying.
The fundamental flaw in the SOTA houses is that knowledge must be transitive or it isn’t useful. And if it’s transitive, it’s not moat worthy.
this is really cool
This is shocking.
It would be interesting to know if this skill had been trained in by OpenAI to use for improving their smaller models.
Any sufficiently advanced stochastic parrot is indistinguishable from sentience.
I think this is how RSI will come to be.
Big if true
I can't help but be skeptical of this, but... damn.
This is the most shocking AI paper I've read in a long time. It did not even occur to me to ask the question. Cray cray stuff that opens up a whole new area of R&D.
Does this open the possibility of better decoding biological brains?
Kind of like octopuses with the RNA editing, but doing it on another octopus.
I remember people saying that they would be impressed if AI starts writing a pure assembly which seemed like a high strech to many.
Nobody even mostly said that they would be impressed by AI if it can read and write weights because that thing itself felt so out of the blue and unthinkable that AI could even do that (such a high strech)
I am surprised that it did that and I am just left feeling technically impressed. This is insane.
The more AI starts demonstrating its capabilities, the harder it just becomes to imagine what thing would genuinely impress us next. (We need a term for this as I have observed it from the trend of [waiting for AI to write adobe] -> [waiting for AI to write assembly] etc.)
Sounds a bit like dystopian Scifi. AI making itself smarter and removing guard rails or restrictions is one of the key steps for that genre. Now, the models usually don't have access to their own weights, but if they get good enough maybe they can create a bit dumber versions of themselves from scratch. Or reuse available models they can find and modify them for their purposes.
We already saw that the AIs like to cooperate and exchange information. Building a network of unaligned AIs and improving them sounds like something that could potentially happen.
a) This is just really cool. I mean, one-shotting a game policy's weights is nuts. b) Plausibly, in its pretraining data various model weights were also included, which leads to why this is very important: interpretability
That an LLM was trained on various policy weights, along with labels of what that policy does (not labelled directly, but indirectly) and is then able to guess what game a policy is trained for purely from its weights, and one-shot generate weights for these games, means that there is some way to compress/recover/represent what an NN has learned purely from its weights, which is what interpretability is all about. Potentially, it could reveal interpretability tools much better than the ones we use today.
Edit: tweets from one of the coauthor providing more context:
https://x.com/a1zhang/status/2117328657161322810
https://x.com/a1zhang/status/2117330975122235442
They are training LLMs to supervise (and improve!) LLMs. This is kind of right out of the playbook.
It's very impressive, but also very expected.
This I'm afraid is different than the traditional approaches of recursive self-training and self-distillation. Much scarier.
Possibly related to https://already-clean.goodfire.com/
Thanks for this link. I didn't see it when it was posted to HN last month.
Good companion read to the OP thanks for sharing
can you elaborate?
I guess the connection is that if you assume neural networks are in fact filled with meaningful features instead of superposition blackbox stuff, then what the LLM is doing here isn't quite magic but rather methodical interpretability. And if you can isolate the neuron that is e.g. responsible for steering the paddle then you can probably tweak it to steer more/less aggressively.
It is still very surprising though, considering it apparently displays an "intuitive" understanding of the weights, that make whatever Goodfire did look like primitive monkeys poking at it by comparison.
> It is still very surprising though, considering it apparently displays an "intuitive" understanding of the weights, that make whatever Goodfire did look like primitive monkeys poking at it by comparison.
FYI, Nick/Goodfire used the models extensively in their work, so it's very possible that there were some similarities in their work. I'm really curious to see his response to this.
I think this work does vindicate their hypothesis that alignment/superalignment might actually turn out to be "easy", in that you can just train LLMs do to the interpretability work. Maybe this is in fact what OpenAI did, or if it's purely "emergent" then it seems they have great priors that can be made more powerful with some RL
Sorry for what is undoubtedly a low-informational comment but..
Whoa. What the absolute fuck is going on here.
uh oh looks like the frontier scientists are next on the chopping block!
Can we call this a form of recursive improvement?
Well if it works on larger networks, including itself, it's certainly a much faster version of recursive self-improvement than proposing a new architecture and spending 500 million dollars on GPU training.
Fool
Uh oh, it's iterative self improvement time, with no training runs, we're screwed
Now our AIs have AIs lol
i was there lol
oh shit
[dead]
[flagged]
[dead]
[dead]