Not making any claims about a specific model's capabilities, but all these "can interpret model weights" posts brings to mind the line in Shakespeare's Henry IV
GLENDOWE: I can call spirits from the vasty deep.
HOTSPUR: Why, so can I, or so can any man, but will they come when you do call for them?
I can interpret weights too, but will that interpretation be correct?
> I can interpret weights too, but will that interpretation be correct?
Chances are much higher than a model can figure out what the patterns are, decipher them, and retain that knowledge in a meaningful way it can reason about. Me and you might take years or months to do all of that on our own, with a "map" of the weight we might be able to discuss it, but not as effortlessly as an LLM.
Honestly, I had told a few different colleagues that reading a model is probably not that exclusive to frontier models. It's probably yet another reason OpenAI / Anthropic don't release any big open weight models, but thankfully Google and Microsoft release reasonably sized models. Imagine a new GPT open weight release, it would get dissected drastically.
Further than that, aren't "weights" just another sequence of tokens? We see them as vectors, pass to the API or as strings into a chatbox interface. Is it so surprising that the high-dimensionality linear algebra machine crunches them and gives us back useful vectors that can be mapped to text?
Isn't that what this whole technology paradigm is about? How is it surprising?
Granted, it is cool. But I see so much of "look at this cool thing I did with LLMs" and honestly... we need more structured text, more explicit documentation, more ergonomic and intuitive CRUD applications.
The amount of "look at cool things" is growing way faster than before and they are all sort of low effort now because the "agentic" LLM-VM looping apparatus can do a lot of heavy lifting and run with our prompts.
And I mean, if one has jargon, knowledge, and intention, they can also leverage that into great structured text, ergonomic and intuitive CRUD apps etc, all of which add great value to our mundane pursuits of applied research, software products and services and so on.
Honestly, I begin to think the problem is people dazzling themselves instead of doing the housekeeping that is the bulk of life. Technology choice is entirely secondary.
> Further than that, aren't "weights" just another sequence of tokens?
No, not at all. I recommend you learn some of the basic linear algebra about how these things work before engaging in wild and completely nonsensical speculation.
Simon, could you try reproducing Experiment 5 from the blog post, if you have a chance? Apparently this person tried [1] with API models and only the latest models can do the impressive 10-20x vs. PPO baseline.
> I’m not ready to declare that the age of source code is over and our future involves incomprehensible machines editing their own model weights in incomprehensible ways.
That sounds a bit like biology; maybe AI will become like that.
Why would you want to change weights directly versus running through a new training cycle? For example, I have collected a bunch of data I would like to train weights that I use local inference for. I figured a LoRa would be the way to go, perhaps be more cost effective and generalized too.
In my latest experiments with editing the weights directly, it has edited the weights of my decision head and LoRa checkpoint.
Essentially, my training introduced knowledge and structure which wasn't translating into results. I tried training in a few different ways, different ranks, different things, and I wasn't making any progress.
With weight editing, a larger model was able to eliminate some undesirable pathways which the model was often using. Without going into the nitty-gritty, it basically took my best LoRa checkpoint and gave it the skills that I was unable to get with further training.
Because text-to-weights is a learnable function. Everything else seems to be. So, why not? Why train weights if you could just prompt for them. If we managed to train image edit models, then why can't we train model-edit models?
If you could figure out exactly what certain weights do it would be way quicker to directly edit them, whereas training is non-deterministic and takes ages.
And the modifications of those weights could be tracked in source control and overlayed on top of the base model? I see, there is a lot of potential in that.
Speed, instant finetunes? Highly optimized iteration cycles? The applications are the same for any reason why you might want local llms. This is crazy exciting.
Isn't the network too shallow to be challenging to decode? One can easily produce few layer network to solve specific goal with construction techniques, so not a surprising result in my view.
I wouldn't be surprised at all if 3.8-27B could tweak 3.5-0.8B the same way Astra did, but it wouldn't be a big deal either because that part of the paper has nothing to do with directly tweaking the weights like the first part of the paper, but is instead a fairly straightforward abliteration method on a model that is known for a specific pathological behavior (ending up stuck in a thinking loop).
isn't consciousness just that, a thinking loop with interruptions for sleep until we break beyond repair? and, just maybe, if we managed to figure out the meaning of life, our thinking would satisfyingly stop?
What I want to see someone try and succeed at would be to get the really highly quantized models to loop less. Many people with only 16GB VRAM are running the IQ3_XXS or IQ3_S quants of Qwen3.8-27B. If it could figure out how to improve those quants, that'd actually be pretty substantial. That's all they did for those improved math results is removed the looping. Why not? Please someone. Please.
It's fairly straightforward to write a function that detects looping output -- I wrote one for promptfoo in [1].
For training you would need to hook it up to a LoRA training harness, identify the looping weight direction, and adjust the weights accordingly. I've not (yet) looked into how to do this.
i feel like this could turn out to be a over unity demo where they say the model can improve a model better than it's own. (more energy output than input)
> It’s a cool paper, but some of the wider commentary I’ve seen around it does not feel justified. I’m not ready to declare that the age of source code is over and our future involves incomprehensible machines editing their own model weights in incomprehensible ways.
The age of source code is definitely not going to end anytime soon if ever. But the future will involve incomprehensible machines editing other models in a "fairly certain" way, but we are going to need a lot of tooling to keep up with the process and not just take the model updates as "trust me bro".
I did exactly the same thing with Qwen 3.8 27B but i also asked it something else: a couple years ago i wrote a simple upscaler[0] when i was learning about neural networks. The network is dead simple: it takes a 25 inputs for a 5x5 grid around a pixel and outputs the 2x2 pixel grid that the input pixel should be replaced for a (width2)x(height2) upscale. I trained it on a bunch of random images (in hindsight i should have trained it on game screenshots but w/e) and it looked quite nice.
So i asked Qwen 3.8 27B two things:
1. What it is this network all about? I only gave it the ann.c code that implemented a simple artificial neural network library in C and the setup code that specified the network's shape but not its purpose.
Qwen 3.8 27B tried a bunch of stuff, taking the layers one by one, proding numbers through it, etc and finally figured out it is some sort of 5x5 to 2x2 filter, most likely working with pixels (i never mentioned pixels or graphics), though it wasn't certain about the exact purpose. It did rule out a bunch of uses like being a downscaler or classifier, from its behavior.
2. After i revealed what it was, i asked it if it can improve it but keep it 100% compatible with the existing program that used the weights file. And sure enough, it did improve it. I gave it the original training C code and dataset. It made a few experiments (and wrote its own training C code because it found mine too slow :-P) and ended up improving the weights. The difference[1] is small, but it basically eliminated some small noise that was always there and made some transitions less "halo-y". Of course for the purpose i wanted it (use it to upscale games in realtime) it isn't that great (too heavy) but it was learning material anyway. Though one bonus thing i noticed is that it upscales my random doodles much better than any Krita filter[2] :-P.
The above isn't Qwen 3.8 27B observing the weights directly and making sense out of them (ok actually it did seem to make some sense of the weights in the "#1" but i didn't dump the entire network to it - though thinking about it, it is only around 16384 floats so it might work). It just used a bunch of scripts it wrote in Python (and later C) but i still found it interesting to watch it go at it.
It's a blog post which received feedback from collaborators.
It hasn't gone through the academic paper lifecycle ... the authors themselves refer to it as a post and as preliminary.
I feel like I'm taking crazy pills, but can we be precise when talking about work of this form? It started with referring to arXiv pre-prints as papers, and now we're at blog posts with an academic template as papers.
A paper is a fucking paper: something which has been vetted by the scientific community, with at least some form (often, a few rounds) of anonymous review from people outside your circle of collaborators
This is not there yet, and clearly so due to the various feedback from people (including the two pushback posts on HN) critiquing the lack of discussion about the existence of these techniques in mech interp, etc?
Like you said, in the AI space a pre-print on arXiv is massively more likely than anything that's been through peer-review.
I called it a paper here because it's by people with an academic affiliation and is of comparable quality (if not better than) many of the other "papers" we see.
I also like it when a paper isn't a PDF. PDFs are a bad way of communicating information.
(I will note that the authors themselves didn't call it a paper, their own BibTeX reference calls it a "blog post":)
@misc{takehi2026weightspace,
title = {{Sparks of Weight-Space Programming: LLMs can natively read \& write neural network weights}},
author = {Takehi, Rikiya and Teoh, Jayden and Zhang, Alex and Khattab, Omar},
year = {2026},
month = nov,
howpublished = {\url{https://rikiyat.github.io/blog/weight-space-programming/}},
note = {Blog post}
}
But for (1, 2), just to point out, neither of your criteria would pass muster in the scientific communities who are ultimately responsible for vetting this work, right?
The ideal is: we don't accept papers at NeurIPS (or wherever) because of affiliation, and the badge of "quality" is bestowed by peer review, from other scientists.
> is of comparable quality (if not better than)
But the posts that you refer to in your own post point out several ways in which the quality is not good ...
But, in any case, I find it a bizarre choice to call it a paper when the authors themselves do not refer to it as such. We can agree to disagree, but just offering feedback.
Of course, I agree with you in many respects - but I’ve found the academic paper lifecycle to be tremendously helpful in terms of polishing my contributions (admittedly, not in ML - but in PL / systems) and avoiding content which could be construed as “misleading”
You, being of sound mind, tried the technique and drew reasonable conclusions … but just look at the original post for this blog post: there are a large number of responses by people who clearly don’t understand the technique, missed the experimental weaknesses, couldn’t place the post into the lineage of mech interp results that preceded it … and will leave the discussion with the wrong impression.
In your mind, whose fault is that? Is that that the unavoidable cost of a field, moving too rapidly to engage in peer review?
In its ideal form, the paper lifecycle is not a gate, but a sharpening stone.
I think we have a messy version of peer review playing out right now.
I'm not convinced the alternative - delaying publication by 6 months, keeping the new information available only to a much smaller group until then - would have been better.
All of that said, I did just update my own post to include "The paper (OK, the authors classify is as a “blog post”, but it’s paper-shaped) found that frontier models..." inspired by this conversation.
Because nobody actually reviews papers anyway so it is irrelevant. That might have happened when you only had 12 papers a year and they were things like “gravity exists”, but when you get a million papers a day on topics like “the ethnocentric illusion of west-canadian yarn spinning”, nobody has time for that.
Yeah, that's not gonna happen in ML, where a positive result (even in blog form) can be (hopefully) trivially tested and confirmed.
From memory, yolo and roberta and ppo were initially rejected. After a quick google search, pagerank was initially rejected, word2vec got one weak and one strong reject in reviews, LSTM got rejected (90s but still), dropout, knowledge distillation, and many more such cases. All of these are extremely important, foundational and field defining papers that ended up on arxiv or other preprint places.
If that happened to those papers, there's 0 chance today that it'd be a good idea. The field is simply moving too fast, there's too much being published, and the "top venues" are simply swamped. Blog / preprint / carrier pigeon is ok atm, as long as someone can look at it, try it, and hopefully reproduce it.
Not making any claims about a specific model's capabilities, but all these "can interpret model weights" posts brings to mind the line in Shakespeare's Henry IV
GLENDOWE: I can call spirits from the vasty deep.
HOTSPUR: Why, so can I, or so can any man, but will they come when you do call for them?
I can interpret weights too, but will that interpretation be correct?
> I can interpret weights too, but will that interpretation be correct?
Chances are much higher than a model can figure out what the patterns are, decipher them, and retain that knowledge in a meaningful way it can reason about. Me and you might take years or months to do all of that on our own, with a "map" of the weight we might be able to discuss it, but not as effortlessly as an LLM.
Honestly, I had told a few different colleagues that reading a model is probably not that exclusive to frontier models. It's probably yet another reason OpenAI / Anthropic don't release any big open weight models, but thankfully Google and Microsoft release reasonably sized models. Imagine a new GPT open weight release, it would get dissected drastically.
Further than that, aren't "weights" just another sequence of tokens? We see them as vectors, pass to the API or as strings into a chatbox interface. Is it so surprising that the high-dimensionality linear algebra machine crunches them and gives us back useful vectors that can be mapped to text?
Isn't that what this whole technology paradigm is about? How is it surprising?
Granted, it is cool. But I see so much of "look at this cool thing I did with LLMs" and honestly... we need more structured text, more explicit documentation, more ergonomic and intuitive CRUD applications.
The amount of "look at cool things" is growing way faster than before and they are all sort of low effort now because the "agentic" LLM-VM looping apparatus can do a lot of heavy lifting and run with our prompts.
And I mean, if one has jargon, knowledge, and intention, they can also leverage that into great structured text, ergonomic and intuitive CRUD apps etc, all of which add great value to our mundane pursuits of applied research, software products and services and so on.
Honestly, I begin to think the problem is people dazzling themselves instead of doing the housekeeping that is the bulk of life. Technology choice is entirely secondary.
> Further than that, aren't "weights" just another sequence of tokens?
No, not at all. I recommend you learn some of the basic linear algebra about how these things work before engaging in wild and completely nonsensical speculation.
They did test for correct interpretation, so what point are you making?
And this reminds me of the king in the little prince, who had the power to order the sun to rise or go down. But only once a day at a certain time.
Simon, could you try reproducing Experiment 5 from the blog post, if you have a chance? Apparently this person tried [1] with API models and only the latest models can do the impressive 10-20x vs. PPO baseline.
[1] - https://nitter.netbub.com/AlexKrentsel/status/21173766837246...
Hopefully we get a new benchmark: How well can a model tweak a set of weights to produce pelican on a bike?
I actually build something to help with the whole pelicans on bike problem! https://modelsalt.com
have you tried it on wspbench?
> I’m not ready to declare that the age of source code is over and our future involves incomprehensible machines editing their own model weights in incomprehensible ways.
That sounds a bit like biology; maybe AI will become like that.
Superior as mammals are to plants, so may AI become to us mammals.
Not so sure mammals are "superior" to plants, nor fungi.
Nope squishy is where it's at.
That's paraphrased from Erewhon, which was written over 150 years ago and was the inspiration for Dune.
Why would you want to change weights directly versus running through a new training cycle? For example, I have collected a bunch of data I would like to train weights that I use local inference for. I figured a LoRa would be the way to go, perhaps be more cost effective and generalized too.
In my latest experiments with editing the weights directly, it has edited the weights of my decision head and LoRa checkpoint.
Essentially, my training introduced knowledge and structure which wasn't translating into results. I tried training in a few different ways, different ranks, different things, and I wasn't making any progress.
With weight editing, a larger model was able to eliminate some undesirable pathways which the model was often using. Without going into the nitty-gritty, it basically took my best LoRa checkpoint and gave it the skills that I was unable to get with further training.
Because text-to-weights is a learnable function. Everything else seems to be. So, why not? Why train weights if you could just prompt for them. If we managed to train image edit models, then why can't we train model-edit models?
If you could figure out exactly what certain weights do it would be way quicker to directly edit them, whereas training is non-deterministic and takes ages.
And the modifications of those weights could be tracked in source control and overlayed on top of the base model? I see, there is a lot of potential in that.
Something someone else brought up in one of the previous threads was to identify the refusal routes and remove them.
Because training is more expensive I guess
It would take a lot less FLOPS presumably
Speed, instant finetunes? Highly optimized iteration cycles? The applications are the same for any reason why you might want local llms. This is crazy exciting.
Isn't the network too shallow to be challenging to decode? One can easily produce few layer network to solve specific goal with construction techniques, so not a surprising result in my view.
What do you mean? Is there prior art? Would you call this a LORA?
My first reaction to your question (rhetorical?) was uncharitable... so, I am curious what you meant that it was "easy."
I wonder how far this can be taken. Could Qwen 3.8 can interpret and make improvements to Qwen 3.5 0.8B weights, or even its own weights?
It can, as long as you can give it benchmarks. What it can't do is optimize general intelligence without any external validation.
You can optimize a model by randomly changing its weights and validating results on your task. A monkey can type a book given enough attempts.
I wouldn't be surprised at all if 3.8-27B could tweak 3.5-0.8B the same way Astra did, but it wouldn't be a big deal either because that part of the paper has nothing to do with directly tweaking the weights like the first part of the paper, but is instead a fairly straightforward abliteration method on a model that is known for a specific pathological behavior (ending up stuck in a thinking loop).
isn't consciousness just that, a thinking loop with interruptions for sleep until we break beyond repair? and, just maybe, if we managed to figure out the meaning of life, our thinking would satisfyingly stop?
https://upload.wikimedia.org/wikipedia/en/b/ba/DrawingHands....
What I want to see someone try and succeed at would be to get the really highly quantized models to loop less. Many people with only 16GB VRAM are running the IQ3_XXS or IQ3_S quants of Qwen3.8-27B. If it could figure out how to improve those quants, that'd actually be pretty substantial. That's all they did for those improved math results is removed the looping. Why not? Please someone. Please.
It's fairly straightforward to write a function that detects looping output -- I wrote one for promptfoo in [1].
For training you would need to hook it up to a LoRA training harness, identify the looping weight direction, and adjust the weights accordingly. I've not (yet) looked into how to do this.
[1] https://news.ycombinator.com/item?id=48555634
i feel like this could turn out to be a over unity demo where they say the model can improve a model better than it's own. (more energy output than input)
Is RSI here? No.
The question is can a 4b or smaller improve a 0.5b llm, or would it just get caught in a loop
Still amazing to me that a model that good, Qwen3.8:27b, runs on a laptop.
Wasn’t this one of the main marketing points for the Inkling model as well?
> It’s a cool paper, but some of the wider commentary I’ve seen around it does not feel justified. I’m not ready to declare that the age of source code is over and our future involves incomprehensible machines editing their own model weights in incomprehensible ways.
The age of source code is definitely not going to end anytime soon if ever. But the future will involve incomprehensible machines editing other models in a "fairly certain" way, but we are going to need a lot of tooling to keep up with the process and not just take the model updates as "trust me bro".
i don't get his example, he had a simple model that only plays pong and Qwen can tell by just looking at the weights?
You can see the full prompt which describes the challenge (and includes the model weights) for the Pong example here: https://github.com/RikiyaT/lm_understand-logs/blob/main/exp1...
I did exactly the same thing with Qwen 3.8 27B but i also asked it something else: a couple years ago i wrote a simple upscaler[0] when i was learning about neural networks. The network is dead simple: it takes a 25 inputs for a 5x5 grid around a pixel and outputs the 2x2 pixel grid that the input pixel should be replaced for a (width2)x(height2) upscale. I trained it on a bunch of random images (in hindsight i should have trained it on game screenshots but w/e) and it looked quite nice.
So i asked Qwen 3.8 27B two things:
1. What it is this network all about? I only gave it the ann.c code that implemented a simple artificial neural network library in C and the setup code that specified the network's shape but not its purpose.
Qwen 3.8 27B tried a bunch of stuff, taking the layers one by one, proding numbers through it, etc and finally figured out it is some sort of 5x5 to 2x2 filter, most likely working with pixels (i never mentioned pixels or graphics), though it wasn't certain about the exact purpose. It did rule out a bunch of uses like being a downscaler or classifier, from its behavior.
2. After i revealed what it was, i asked it if it can improve it but keep it 100% compatible with the existing program that used the weights file. And sure enough, it did improve it. I gave it the original training C code and dataset. It made a few experiments (and wrote its own training C code because it found mine too slow :-P) and ended up improving the weights. The difference[1] is small, but it basically eliminated some small noise that was always there and made some transitions less "halo-y". Of course for the purpose i wanted it (use it to upscale games in realtime) it isn't that great (too heavy) but it was learning material anyway. Though one bonus thing i noticed is that it upscales my random doodles much better than any Krita filter[2] :-P.
The above isn't Qwen 3.8 27B observing the weights directly and making sense out of them (ok actually it did seem to make some sense of the weights in the "#1" but i didn't dump the entire network to it - though thinking about it, it is only around 16384 floats so it might work). It just used a bunch of scripts it wrote in Python (and later C) but i still found it interesting to watch it go at it.
[0] https://mastodon.gamedev.place/@badsector/113765818432603610
[1] http://runtimeterror.com/pages/iv/images/a3d94ea369ff95ecf05...
[2] http://runtimeterror.com/pages/iv/images/4e29d027a8aa6e89db5...
Which coding agent harness did you use with Qwen for that?
I've been using Pi though i'm currently doing the NIH thing and making a custom one in C++ with a web-based frontend :-P.
Why do people keep calling this a paper?
It's a blog post which received feedback from collaborators.
It hasn't gone through the academic paper lifecycle ... the authors themselves refer to it as a post and as preliminary.
I feel like I'm taking crazy pills, but can we be precise when talking about work of this form? It started with referring to arXiv pre-prints as papers, and now we're at blog posts with an academic template as papers.
A paper is a fucking paper: something which has been vetted by the scientific community, with at least some form (often, a few rounds) of anonymous review from people outside your circle of collaborators
This is not there yet, and clearly so due to the various feedback from people (including the two pushback posts on HN) critiquing the lack of discussion about the existence of these techniques in mech interp, etc?
Like you said, in the AI space a pre-print on arXiv is massively more likely than anything that's been through peer-review.
I called it a paper here because it's by people with an academic affiliation and is of comparable quality (if not better than) many of the other "papers" we see.
I also like it when a paper isn't a PDF. PDFs are a bad way of communicating information.
(I will note that the authors themselves didn't call it a paper, their own BibTeX reference calls it a "blog post":)
Sure, I agree with point (3).
But for (1, 2), just to point out, neither of your criteria would pass muster in the scientific communities who are ultimately responsible for vetting this work, right?
The ideal is: we don't accept papers at NeurIPS (or wherever) because of affiliation, and the badge of "quality" is bestowed by peer review, from other scientists.
> is of comparable quality (if not better than)
But the posts that you refer to in your own post point out several ways in which the quality is not good ...
But, in any case, I find it a bizarre choice to call it a paper when the authors themselves do not refer to it as such. We can agree to disagree, but just offering feedback.
You're right about this, but I don't regret my choice of words.
I admit that I don't think formal "papers" deserve their elevated status in today's information economy.
I think they add a whole lot of ceremony that usually results in a format that's harder to understand.
So me calling this a "paper" was a little bit of a dig!
Of course, I agree with you in many respects - but I’ve found the academic paper lifecycle to be tremendously helpful in terms of polishing my contributions (admittedly, not in ML - but in PL / systems) and avoiding content which could be construed as “misleading”
You, being of sound mind, tried the technique and drew reasonable conclusions … but just look at the original post for this blog post: there are a large number of responses by people who clearly don’t understand the technique, missed the experimental weaknesses, couldn’t place the post into the lineage of mech interp results that preceded it … and will leave the discussion with the wrong impression.
In your mind, whose fault is that? Is that that the unavoidable cost of a field, moving too rapidly to engage in peer review?
In its ideal form, the paper lifecycle is not a gate, but a sharpening stone.
I think we have a messy version of peer review playing out right now.
I'm not convinced the alternative - delaying publication by 6 months, keeping the new information available only to a much smaller group until then - would have been better.
All of that said, I did just update my own post to include "The paper (OK, the authors classify is as a “blog post”, but it’s paper-shaped) found that frontier models..." inspired by this conversation.
Because nobody actually reviews papers anyway so it is irrelevant. That might have happened when you only had 12 papers a year and they were things like “gravity exists”, but when you get a million papers a day on topics like “the ethnocentric illusion of west-canadian yarn spinning”, nobody has time for that.
It's not considered a paper until it's vetted by others?
Yes: before that stage, it is often called a "pre-print" ... if it is not at the state of a "pre-print", then "blog post" ... :)
In most fields of science, it's not a paper until it's accepted for publication in a journal. That normally includes peer review of some sort.
Yeah, that's not gonna happen in ML, where a positive result (even in blog form) can be (hopefully) trivially tested and confirmed.
From memory, yolo and roberta and ppo were initially rejected. After a quick google search, pagerank was initially rejected, word2vec got one weak and one strong reject in reviews, LSTM got rejected (90s but still), dropout, knowledge distillation, and many more such cases. All of these are extremely important, foundational and field defining papers that ended up on arxiv or other preprint places.
If that happened to those papers, there's 0 chance today that it'd be a good idea. The field is simply moving too fast, there's too much being published, and the "top venues" are simply swamped. Blog / preprint / carrier pigeon is ok atm, as long as someone can look at it, try it, and hopefully reproduce it.