Hey Unsloth, your gguf are the first ones I look for when I want to download a gguf model.
Today I was trying in fact to see, what's the smallest Qwen3.8-27B that I could run and get good results, say restricting it to 16GB of ram.. so I went, pick up the Qwen3.8-27B-UD-IQ2_XXS.gguf and them BAM, error on MTP... now I understand why after reading your announcement.
Beyond the space saving, why removing the MTP? improves speed exactly for the group that could benefit from it.
> We also removed the MTP module from smaller quants under UD-Q2_K_XL (8.37GB and lower) to converse around 500MB of disk space - you can use the Q4_0 MTP separate module if needed
Of course. Models don't actually require VRAM. Nor do they require regular RAM. You could have 1 GB of RAM and swap the model to disk as you need different parts of it. And if you didn't have enough disks you could access weights via a network connection.
Might be off-topic but: is it possible to perform such a quantization on Apple devices? Something like Mac Studio Ultra M1 (even if it would take weeks/months)?
Unsloth use a property dataset they don't release, however you can indeed create quantisation locally on your machine and it's pretty easy, llama.cpp comes with everything you need.
Not 1-bit, but I’m getting pretty good results with some light coding using unsloth’s previous 2-bit quant of qwen3.8-27b. With these new quants i may be able to bump up to 3bit, tho it’s already running so slow (15tok/s average for the first 32k of context) that the speed hit might make it not worth the extra smarts
It seems the NVFP4 quants have a preview version of this Unsloth Dynamic 3.0. Is this close to the finished version, or would it be better to switch to one of the newer quants?
What size context are you able to squeeze in with less than 2gb of headroom? I have had some luck using a quantized kv cache but i fear that also decreases overall quality.
> We also removed the MTP module from smaller quants under UD-Q2_K_XL (8.37GB and lower) to converse around 500MB of disk space - you can use the Q4_0 MTP separate module if needed
llama-server --host 0.0.0.0 --port 8089 -m Qwen3.8-27B-UD-Q8_u.gguf --spec-type draft-mtp,ngram-mod --spec-draft-n-max 3 --spec-draft-n-min 1
if you have an igpu and want to exclude or just use some gpus you can use
--device Vulkan3,Vulkan2,Vulkan1
in my case vulkan because of amd, you can see your devices with
llama-server2 --list-devices
Available devices: Vulkan0: AMD Radeon Graphics (RADV RAPHAEL_MENDOCINO) (33515 MiB, 29349 MiB free) Vulkan1: AMD Radeon RX 7900 XTX (RADV NAVI31) (24560 MiB, 4911 MiB free) Vulkan2: AMD Radeon RX 7900 XTX (RADV NAVI31) (24560 MiB, 7681 MiB free)
KLD of 1%, or similar error metric that multiplies, on 10000 tokens would give accumulated error of 2,000,000%