Benchmarks
Benchmark results
Quality at a given model size, measured on instruction-tuned models with standard task benchmarks - including where standard methods still lead.
Instruction-tuned models, real tasks
Qwen3-8B
| Model | Memory (GB) | Instruction following | Coding | Knowledge |
|---|---|---|---|---|
| Qwen3-8B, uncompressed (16-bit) | 16.40 | 82.1% | 79.3% | - |
| Qwen3-8B, official 4-bit release (AWQ) | 6.10 | 80.6% | 74.4% | 71.9% |
| Qwen3-8B, Minutus 3-bit | 3.90 | 82.1% | 75.0% | 70.9% |
Qwen3-4B and Qwen3-1.7B
| Model | Memory (GB) | Instruction following | Coding | Knowledge |
|---|---|---|---|---|
| Qwen3-4B, uncompressed (16-bit) | 8.04 | 79.9% | 76.2% | - |
| Qwen3-4B, standard 4-bit (NF4) | 2.65 | 77.8% | 73.2% | 66.8% |
| Qwen3-4B, Minutus 3-bit | 2.45 | 78.0% | 72.6% | 66.3% |
| Qwen3-4B, Minutus 2-bit (research preview)Research preview | 1.38 | 60.6% | 46.3% | 55.3% |
| Qwen3-1.7B, standard 8-bit | 2.04 | 68.6% | 62.2% | 54.8% |
| Qwen3-1.7B, standard 4-bit (NF4) | 1.35 | 62.7% | 58.5% | 53.5% |
| Qwen3-1.7B, uncompressed (16-bit) | 3.44 | 66.4% | 62.2% | 55.5% |
Instruction following
- Qwen3-1.7B
- Qwen3-4B
- Qwen3-8B
Coding
- Qwen3-1.7B
- Qwen3-4B
- Qwen3-8B
Knowledge
- Qwen3-1.7B
- Qwen3-4B
- Qwen3-8B
The uncompressed Qwen3-8B and Qwen3-4B models are omitted because MMLU was not measured.
Key results
- Qwen3-8B compressed to 3.90 GB: 4.2x smaller than the 16.4 GB original, with the same instruction-following score (82.1).
- Matches Qwen's official 4-bit release (6.10 GB) in 36% less memory: instruction following 82.1 vs 80.6 and coding 75.0 vs 74.4 (differences not statistically significant); the 4-bit release is 1 point ahead on knowledge (71.9 vs 70.9).
- A compressed 8B in 3.90 GB scores like the uncompressed 4B in 8.04 GB: half the memory for the larger model.
- Qwen3-4B at 3 bits matches standard 4-bit quality on all three benchmarks (instruction following 78.0 vs 77.8, coding 72.6 vs 73.2, knowledge 66.3 vs 66.8; no significant difference) in less memory (2.45 GB vs 2.65 GB).
- Keeps 98% of Qwen3-4B's instruction-following score and 95% of its coding score at 3 bits.
- Research preview: Qwen3-4B in 1.38 GB (5.8x smaller). Active development.
Swiss model
Apertus-8B, the Swiss open model
| Metric | Uncompressed | Minutus 3-bit |
|---|---|---|
| IFEval | 68.4% | 57.7% |
| HumanEval+ | 39.6% | 37.8% |
| MMLU | 59.1% | 51.4% |
Apertus-8B at 3 bits keeps 89% of its benchmark quality on average.
98%
4k tokens
95%
8k tokens
96%
16k tokens
Long-context ability is preserved: 95-98% of the original up to 16k tokens.
Methodology
IFEval (instruction following, prompt-level strict accuracy), HumanEval+ (coding, pass@1), MMLU (multi-subject knowledge test, evaluated without chat template); greedy decoding; reasoning (thinking) mode off; identical evaluation harness and hardware for every model; memory = total model storage including the token embedding and output head; Minutus sizes are measured, the official 4-bit release (AWQ) size is the published checkpoint size, and sizes of standard 4-bit (NF4) / 8-bit models are estimated from their bit widths; differences reported as significant are tested with paired McNemar tests on per-question results.
The official 4-bit release was evaluated with its quantized weights expanded to 16-bit for the same evaluation harness, exactly as our models are; scores reflect the quantized weights, not runtime kernels.
Qwen3-8B was compressed with generic chat calibration text, not model-specific data.
Retained % = score of the compressed model / score of the uncompressed model x 100.
All models were evaluated by us on the same hardware and software; numbers may differ from those reported by model authors, who use other evaluation setups.