Hey Unsloth, your gguf are the first ones I look for when I want to download a gguf model.
Today I was trying in fact to see, what's the smallest Qwen3.8-27B that I could run and get good results, say restricting it to 16GB of ram.. so I went, pick up the Qwen3.8-27B-UD-IQ2_XXS.gguf and them BAM, error on MTP... now I understand why after reading your announcement.
Beyond the space saving, why removing the MTP? improves speed exactly for the group that could benefit from it.
> We also removed the MTP module from smaller quants under UD-Q2_K_XL (8.37GB and lower) to converse around 500MB of disk space - you can use the Q4_0 MTP separate module if needed
Of course. Models don't actually require VRAM. Nor do they require regular RAM. You could have 1 GB of RAM and swap the model to disk as you need different parts of it. And if you didn't have enough disks you could access weights via a network connection.
Might be off-topic but: is it possible to perform such a quantization on Apple devices? Something like Mac Studio Ultra M1 (even if it would take weeks/months)?
Unsloth use a property dataset they don't release, however you can indeed create quantisation locally on your machine and it's pretty easy, llama.cpp comes with everything you need.
Not 1-bit, but I’m getting pretty good results with some light coding using unsloth’s previous 2-bit quant of qwen3.8-27b. With these new quants i may be able to bump up to 3bit, tho it’s already running so slow (15tok/s average for the first 32k of context) that the speed hit might make it not worth the extra smarts
It seems the NVFP4 quants have a preview version of this Unsloth Dynamic 3.0. Is this close to the finished version, or would it be better to switch to one of the newer quants?
What size context are you able to squeeze in with less than 2gb of headroom? I have had some luck using a quantized kv cache but i fear that also decreases overall quality.
Hey Unsloth, your gguf are the first ones I look for when I want to download a gguf model. Today I was trying in fact to see, what's the smallest Qwen3.8-27B that I could run and get good results, say restricting it to 16GB of ram.. so I went, pick up the Qwen3.8-27B-UD-IQ2_XXS.gguf and them BAM, error on MTP... now I understand why after reading your announcement. Beyond the space saving, why removing the MTP? improves speed exactly for the group that could benefit from it.
you can still have it, no?
> We also removed the MTP module from smaller quants under UD-Q2_K_XL (8.37GB and lower) to converse around 500MB of disk space - you can use the Q4_0 MTP separate module if needed
my bad, you are totally right, thanks!
Is it possible to use a model that needs around 64 GB VRAM if you have four GPUs with 16 GB VRAM each?
I do use 2 amd gpus and I get high 40 for generation, 500 for pp and low 20/100 by the end of the context of 256k.
llama-server --host 0.0.0.0 --port 8089 -m Qwen3.8-27B-UD-Q8_u.gguf --spec-type draft-mtp,ngram-mod --spec-draft-n-max 3 --spec-draft-n-min 1
if you have an igpu and want to exclude or just use some gpus you can use
--device Vulkan3,Vulkan2,Vulkan1
in my case vulkan because of amd, you can see your devices with
llama-server2 --list-devices
Available devices: Vulkan0: AMD Radeon Graphics (RADV RAPHAEL_MENDOCINO) (33515 MiB, 29349 MiB free) Vulkan1: AMD Radeon RX 7900 XTX (RADV NAVI31) (24560 MiB, 4911 MiB free) Vulkan2: AMD Radeon RX 7900 XTX (RADV NAVI31) (24560 MiB, 7681 MiB free)
I am getting 14 t/s on my 16 GB card at full context with the UD-Q3_K_XL quant. Model link: https://huggingface.co/unsloth/Qwen3.8-27B-GGUF.
Yes, but unless they support NVLink (they don't), it's quite slow.
Of course. Models don't actually require VRAM. Nor do they require regular RAM. You could have 1 GB of RAM and swap the model to disk as you need different parts of it. And if you didn't have enough disks you could access weights via a network connection.
Might be off-topic but: is it possible to perform such a quantization on Apple devices? Something like Mac Studio Ultra M1 (even if it would take weeks/months)?
Unsloth use a property dataset they don't release, however you can indeed create quantisation locally on your machine and it's pretty easy, llama.cpp comes with everything you need.
Not 1-bit, but I’m getting pretty good results with some light coding using unsloth’s previous 2-bit quant of qwen3.8-27b. With these new quants i may be able to bump up to 3bit, tho it’s already running so slow (15tok/s average for the first 32k of context) that the speed hit might make it not worth the extra smarts
It seems the NVFP4 quants have a preview version of this Unsloth Dynamic 3.0. Is this close to the finished version, or would it be better to switch to one of the newer quants?
The new IQ4XS has been working pretty well so far on 4090 16gb.
What size context are you able to squeeze in with less than 2gb of headroom? I have had some luck using a quantized kv cache but i fear that also decreases overall quality.
No MLX versions for 3.8 though.
Cool. Now run TerminalHard and compare to unquantized 27B.
KLD of 1%, or similar error metric that multiplies, on 10000 tokens would give accumulated error of 2,000,000%