82 points | by matt_d 2 hours ago
5 comments
Ternary quantization does not make any sense. Vector quantization and trellis based methods are better in this region for PTQ.
If you want sub-2 bit llm, get one that’s already trained in higher precision, and compress it with something like YAQA/QTIP with finetuning or PV-tuning + AQLM/HIGGS
Very interesting, I was just exploring this to hopefully fit one of the latest quantized models in 16GB of VRAM.
This is the only time "1.58 bit" phrase makes more sense than "1 trit"
Who knew that if you actually look at information entropy you can pack stuff better!
Woah. Good science.
Ternary quantization does not make any sense. Vector quantization and trellis based methods are better in this region for PTQ.
If you want sub-2 bit llm, get one that’s already trained in higher precision, and compress it with something like YAQA/QTIP with finetuning or PV-tuning + AQLM/HIGGS
Very interesting, I was just exploring this to hopefully fit one of the latest quantized models in 16GB of VRAM.
This is the only time "1.58 bit" phrase makes more sense than "1 trit"
Who knew that if you actually look at information entropy you can pack stuff better!
Woah. Good science.