🚀 DASLab GGUF Quantization Toolkit

[mkleinegger]

🚀 Announcing the DASLab GGUF Quantization Toolkit

We're excited to release the first open-source toolkit that brings GPTQ + EvoPress to the GGUF format, enabling heterogeneous quantization based on importance. Delivering Higher-quality models, same file size.

---

What's inside

• GPTQ quantization with GGUF export $\rightarrow$ packag delivers error-correcting calibration for improved performance
• EvoPress (ICML '25) $\rightarrow$ packag runs evolutionary search to automatically discover optimal per-layer quantization configs
• Model assembly tools $\rightarrow$ package models to be fully functional with llama.cpp

---

Why it matters

Unlike standard uniform quantization, our toolkit optimizes precision where it matters most.
Critical layers (e.g. attention) can use higher precision, while others (e.g. FFN) compress more aggressively.
With EvoPress search + GPTQ quantization, these trade-offs are discovered automatically.

---

Results

Below are zero-shot evaluations. Full benchmark results are available in the repo.

---

Resources

DASLab GGUF Quantization Toolkit (GitHub Repo Link)

We'd love your feedback, contributions, and experiments! 💡