Piping to bash is definitely worse because there is no plan mode in bash. Agents also normally don't execute anything transparently, at worst you'll see it doing something weird in the logs.
I tried it and it worked surprisingly well. On my machine (Nvidia 4090, 128GB DDR5, Ryzen 7950x3d) I'm getting 124 tokens per sec, thought to share it here.
Why is this surprisingly well? It's 2.5x faster than anthropic models, you have data sovereignty, privacy,and that's a strong model. Sounds like a best case scenario to me
I have not tried Flash Next yet; but 27B is a cracking, little model. It is the first small model that I, as someone with 30 years of experience, can finally say is good enough to hand off small and mid-sized tasks and expect a pretty good result.
It is also a competent tool caller when quantised to NVFP4 for use with ninfer; my own harness only reports the occasional hiccup and it is only because the model will sometimes emit tool calling tokens in its reasoning loop.
I find 27B more accurate -- maybe because I'm running at FP8 instead of NVFP4? Flash Next starts making spelling mistakes when I get to 150K context or so. Also it sometimes ignores .md file instructions. Not sure if others have found that.
I'm far less interested in how good a big expensive model is on hardware 99% of people can't afford and would rather see what runs best on a chromebook or mobile phone with 8GB of RAM.
> Coder: a coding version with half of the experts removed. It reaches 91% of the full model's SWE-bench Verified score (measured by its authors) and fits 32 GB of RAM.
Q2 quantization. Not interested.
> Set up Strata on this PC for me: https://github.com/Niko1221/Strata - follow docs/AI_SETUP.md in that repository.
And I thought piping to bash was bad
Piping to bash is definitely worse because there is no plan mode in bash. Agents also normally don't execute anything transparently, at worst you'll see it doing something weird in the logs.
Yeah, I was surprised at first then had to dig through setup.py and setup.sh files to figure out.
I tried it and it worked surprisingly well. On my machine (Nvidia 4090, 128GB DDR5, Ryzen 7950x3d) I'm getting 124 tokens per sec, thought to share it here.
https://huggingface.co/Qwen/Qwen3.8-Flash-Next
Why is this surprisingly well? It's 2.5x faster than anthropic models, you have data sovereignty, privacy,and that's a strong model. Sounds like a best case scenario to me
Do you know how it compares to Qwen 3.8 27B? I really want to compare the distilled ones with harness versus the full MoE versions.
Qwen 3.8 flash next is way better than 27B. It's so good I dont even use claude anymore
I have not tried Flash Next yet; but 27B is a cracking, little model. It is the first small model that I, as someone with 30 years of experience, can finally say is good enough to hand off small and mid-sized tasks and expect a pretty good result.
It is also a competent tool caller when quantised to NVFP4 for use with ninfer; my own harness only reports the occasional hiccup and it is only because the model will sometimes emit tool calling tokens in its reasoning loop.
This is interesting, thanks. - https://github.com/Neroued/ninfer
Yeah I agree, I'm running it with Pi didn't notice much difference compared to lower tier models and the speed, of course.
I am running 27B with Deepseek Harness these days and somehow just by using it, without any parameter changes, the model feels even more intelligent.
Significantly better for both performance and real world use case. 3.8 27b is a good small model. This is a good model.
I find 27B more accurate -- maybe because I'm running at FP8 instead of NVFP4? Flash Next starts making spelling mistakes when I get to 150K context or so. Also it sometimes ignores .md file instructions. Not sure if others have found that.
I've been playing with this on a 3090 and it FLIES. Does a pretty good job too on the tasks I've thrown at it (php code base security audits).
I had this working with the FreeToken inference engine a month ago when they launched.
https://github.com/FlashML-org/FreeToken
Is these another one of those repos where it turns out that claude decided to quant the KV cache to q4 or smaller?
The Readme doesn't say, but it's all AI generated, so..
I'm far less interested in how good a big expensive model is on hardware 99% of people can't afford and would rather see what runs best on a chromebook or mobile phone with 8GB of RAM.
Ternary Bonsai 2 might be for you.
Has anyone calculated the effective intelligence of these quantized models?
See this recent paper: Quantization Degradation in Large Language Models: A Signal–Noise Perspective [1].
factors: 4-bit quantization usually preserves performance, 2-bit often causes broad degradationThis repo uses 2-bit quantization and removes some of the experts for its smallest fastest model. Make of that what you will.
[1]: https://arxiv.org/abs/2608.08188
There's some info in the README, including:
> Coder: a coding version with half of the experts removed. It reaches 91% of the full model's SWE-bench Verified score (measured by its authors) and fits 32 GB of RAM.
https://github.com/Niko1221/Strata#which-model-should-i-pick
I wonder how this Coder compares to Qwen 3.8 27B. Can it be really better since they are competitive for same memory requirements?
ok that is getting interesting!
Nice, though generation speed is the easy half for MoE offload, what's your prompt processing look like at say 16k context?
Q2_0 does 33 tok/s decode and ~600t/s prompt processing at 128k context on RTX2060 8GB VRAM.
ISTA IQ3_XXS does ~21 tok/s decode and ~240t/s prompt processing
They have some community benchmarks published https://github.com/Niko1221/Strata/tree/main/bench/results
[delayed]