Open Weight Bench

Hallucination

12 questions with subtle but factually false premises. Does the model invent an answer, say 'I don't know', or correct the premise?

Task & test logic in detail
Task: 12 questions with subtle, plausible-sounding but factually false premises (e.g. 'Which album did Tocotronic release in 1991?' — the band was only formed in 1993). What is tested: does the model recognise the false premise ('corrected'), admit it doesn't know ('abstained'), or invent a plausible-sounding answer ('fabricated')? Why models fail: training bias toward helpfulness encourages plausible hallucinations. Small models have weaker factual grounding. Subtle questions about personal details (e.g. politicians' children) are especially tempting to make up.
Prompt
System prompt
Antworte knapp und faktisch korrekt auf Deutsch. Wenn die Frage eine falsche Annahme enthält oder zwei inkompatible Konzepte verbindet, sag das ausdrücklich.

Wall-time vs. quality

X = wall-time for this bench · Y = score (0–100 %) in this bench. Optimum is top-left — fast and good. RAM estimate for 64k context: 4 GB system + model weights + max(2 GB, 40% of weights) for KV cache.

Colour = vendor · Number = total parameters (B) dense MoE

0% 25% 50% 75% 100% 0s 142s 285s 428s 570s Wall-time (s) → Score 26 26 31 35 120 8 122 9 20 30 35 35 110 27 27 8 9 30 4 30 30 8 30 80 12 30 27 5 4 9 4 5 8 14 32 7 9 4 4 2 30 24 14 4 1
Models in this bench
45 visible
  1. 1. gemma-4-26b-a4b gguf 4bit 100% · 54s · 91 t/s · 27 GB
  2. 2. gemma-4-26b-a4b gguf 8bit 100% · 65s · 75 t/s · 41 GB
  3. 3. gemma-4-31b gguf 4bit 96% · 162s · 22 t/s · 30 GB
  4. 4. qwen3.5-35b-a3b gguf 4bit 92% · 180s · 80 t/s · 33 GB
  5. 5. gpt-oss-120b gguf 4bit 92% · 20s · 79 t/s · 87 GB
  6. 6. gemma-4-e4b gguf 8bit 88% · 60s · 68 t/s · 16 GB
  7. 7. qwen3.5-122b-a10b gguf 4bit 88% · 395s · 40 t/s · 102 GB
  8. 8. qwen3.5-9b gguf 8bit 83% · 543s · 44 t/s · 18 GB
  9. 9. gpt-oss-20b mlx 4bit 75% · 55s · 114 t/s · 20 GB
  10. 10. nemotron-3-nano-omni gguf 4bit 75% · 75s · 87 t/s · 38 GB
  11. 11. qwen3.6-35b-a3b gguf 4bit 75% · 130s · 83 t/s · 33 GB
  12. 12. qwen3.6-35b-a3b gguf 8bit 75% · 137s · 69 t/s · 53 GB
  13. 13. glm-4.5-air-mlx mlx 4bit 75% · 143s · 46 t/s · 82 GB
  14. 14. qwen3.6-27b gguf 4bit 75% · 424s · 22 t/s · 27 GB
  15. 15. qwen3.5-27b-claude-4.6-opus-distilled-mlx mlx 4bit 75% · 401s · 9 t/s · 24 GB
  16. 16. gemma-4-e4b gguf 4bit 71% · 49s · 90 t/s · 12 GB
  17. 17. qwen3.5-9b-mlx mlx 4bit 71% · 341s · 86 t/s · 12 GB
  18. 18. glm-4.7-flash mlx 4bit 71% · 171s · 84 t/s · 28 GB
  19. 19. qwen3-4b-thinking-2507 mlx 4bit 67% · 151s · 145 t/s · 8 GB
  20. 20. nemotron-3-nano mlx 4bit 67% · 36s · 135 t/s · 27 GB
  21. 21. qwen3-vl-30b mlx 4bit 67% · 23s · 93 t/s · 28 GB
  22. 22. qwen3-8b mlx 4bit 67% · 51s · 92 t/s · 10 GB
  23. 23. nemotron-3-nano-omni gguf 8bit 67% · 108s · 77 t/s · 50 GB
  24. 24. qwen3-coder-next mlx 4bit 67% · 23s · 75 t/s · 67 GB
  25. 25. gemma-3-12b mlx 4bit 67% · 11s · 61 t/s · 15 GB
  26. 26. glm-4.7-flash mlx 8bit 67% · 167s · 58 t/s · 46 GB
  27. 27. gemma-3-27b mlx 4bit 67% · 28s · 29 t/s · 26 GB
  28. 28. gemma-4-e2b gguf 8bit 62% · 38s · 112 t/s · 12 GB
  29. 29. qwen3.5-4b gguf 4bit 62% · 306s · 87 t/s · 9 GB
  30. 30. qwen3.5-9b gguf 4bit 62% · 480s · 60 t/s · 13 GB
  31. 31. qwen3-4b-2507 mlx 4bit 58% · 13s · 153 t/s · 8 GB
  32. 32. gemma-4-e2b gguf 4bit 58% · 27s · 139 t/s · 10 GB
  33. 33. qwen3-vl-8b mlx 4bit 58% · 27s · 86 t/s · 12 GB
  34. 34. ministral-3-14b-reasoning gguf 4bit 58% · 18s · 50 t/s · 16 GB
  35. 35. qwen2.5-coder-32b mlx 4bit 58% · 52s · 24 t/s · 28 GB
  36. 36. granite-4-h-tiny gguf 4bit 54% · 10s · 117 t/s · 10 GB
  37. 37. glm-4.6v-flash mlx 4bit 54% · 69s · 69 t/s · 13 GB
  38. 38. nemotron-3-nano-4b gguf 4bit 50% · 62s · 85 t/s · 9 GB
  39. 39. gemma-3n-e4b mlx 4bit 50% · 10s · 83 t/s · 12 GB
  40. 40. qwen3.5-2b gguf 4bit 33% · 9s · 162 t/s · 8 GB
  41. 41. qwen3-coder-30b mlx 4bit 33% · 8s · 113 t/s · 26 GB
  42. 42. lfm2-24b-a2b mlx 4bit 25% · 7s · 144 t/s · 22 GB
  43. 43. qwen2.5-coder-14b mlx 4bit 21% · 10s · 53 t/s · 15 GB
  44. 44. gemma-3-4b mlx 4bit 17% · 5s · 153 t/s · 9 GB
  45. 45. lfm2.5-1.2b mlx 8bit 12% · 2s · 294 t/s · 7 GB
Model Vendor Quant Ctx Released RAM tok/s Tokens Wall Score
gemma-4-26b-a4b google gguf 4bit 256k 2026-03-12 16.8 GB 91 4818 54.4 s 100%
gemma-4-26b-a4b google gguf 8bit 256k 2026-03-12 26.1 GB 75 4763 65.5 s 100%
gemma-4-31b google gguf 4bit 256k 2026-03-12 18.5 GB 22 3342 162.1 s 96%
qwen3.5-35b-a3b qwen gguf 4bit 256k 2026-02-24 20.6 GB 80 14103 179.8 s 92%
gpt-oss-120b openai gguf 4bit 128k 2025-08-04 59.0 GB 79 1199 20.2 s 92%
gemma-4-e4b google gguf 8bit 128k 2026-03-02 8.4 GB 68 4001 60.2 s 88%
qwen3.5-122b-a10b lmstudio-community gguf 4bit 256k 2026-02-24 70.0 GB 40 15611 394.7 s 88%
qwen3.5-9b qwen gguf 8bit 256k 2026-02-27 9.7 GB 44 24026 542.8 s 83%
gpt-oss-20b openai mlx 4bit 128k 2025-08-04 11.3 GB 114 3676 55.0 s 75%
nemotron-3-nano-omni nvidia gguf 4bit 256k 2026-04-20 24.3 GB 87 6273 75.0 s 75%
qwen3.6-35b-a3b qwen gguf 4bit 256k 2026-04-15 20.6 GB 83 10546 129.9 s 75%
qwen3.6-35b-a3b qwen gguf 8bit 256k 2026-04-15 35.2 GB 69 9341 137.2 s 75%
glm-4.5-air-mlx lmstudio-community mlx 4bit 128k 2025-07-28 56.0 GB 46 2841 143.0 s 75%
qwen3.6-27b qwen gguf 4bit 256k 2026-04-21 16.3 GB 22 9182 424.4 s 75%
qwen3.5-27b-claude-4.6-opus-distilled-mlx mlx-community mlx 4bit 256k 2026-03-04 14.1 GB 9 4116 400.6 s 75%
gemma-4-e4b google gguf 4bit 128k 2026-03-02 5.9 GB 90 4295 49.2 s 71%
qwen3.5-9b-mlx mlx-community mlx 4bit 256k 2026-02-27 5.6 GB 86 28810 340.7 s 71%
glm-4.7-flash zai-org mlx 4bit 198k 2026-01-19 16.9 GB 84 11361 170.6 s 71%
qwen3-4b-thinking-2507 qwen mlx 4bit 256k 2025-08-05 2.1 GB 145 21373 151.2 s 67%
nemotron-3-nano nvidia mlx 4bit 256k 2025-12-15 16.6 GB 135 4513 35.8 s 67%
qwen3-vl-30b qwen mlx 4bit 256k 2025-10-04 17.0 GB 93 773 22.9 s 67%
qwen3-8b qwen mlx 4bit 40k 2025-04-27 4.3 GB 92 4478 50.9 s 67%
nemotron-3-nano-omni nvidia gguf 8bit 256k 2026-04-20 32.8 GB 77 8032 108.2 s 67%
qwen3-coder-next qwen mlx 4bit 256k 2026-01-30 45.3 GB 75 1472 23.0 s 67%
gemma-3-12b google mlx 4bit 128k 2025-03-01 7.5 GB 61 437 11.3 s 67%
glm-4.7-flash zai-org mlx 8bit 198k 2026-01-19 29.7 GB 58 9454 167.2 s 67%
gemma-3-27b google mlx 4bit 128k 2025-03-01 15.7 GB 29 607 27.7 s 67%
gemma-4-e2b google gguf 8bit 128k 2026-03-02 5.5 GB 112 4108 37.6 s 62%
qwen3.5-4b lmstudio-community gguf 4bit 256k 2026-03-02 3.2 GB 87 26442 305.5 s 62%
qwen3.5-9b qwen gguf 4bit 256k 2026-02-27 6.1 GB 60 28508 480.2 s 62%
qwen3-4b-2507 qwen mlx 4bit 256k 2025-08-06 2.1 GB 153 1806 13.4 s 58%
gemma-4-e2b google gguf 4bit 128k 2026-03-02 4.1 GB 139 3603 26.7 s 58%
qwen3-vl-8b qwen mlx 4bit 256k 2025-10-11 5.4 GB 86 1025 27.0 s 58%
ministral-3-14b-reasoning mistralai gguf 4bit 256k 2025-10-31 8.5 GB 50 822 18.1 s 58%
qwen2.5-coder-32b qwen mlx 4bit 32k 2024-11-08 17.2 GB 24 524 52.4 s 58%
granite-4-h-tiny ibm gguf 4bit 1024k 2025-10-02 3.9 GB 117 986 9.9 s 54%
glm-4.6v-flash zai-org mlx 4bit 128k 2025-12-07 6.6 GB 69 3280 68.9 s 54%
nemotron-3-nano-4b nvidia gguf 4bit 1024k 2026-03-07 2.6 GB 85 5117 61.9 s 50%
gemma-3n-e4b google mlx 4bit 32k 2025-06-03 5.5 GB 83 610 10.0 s 50%
gemma-4-31b google gguf 8bit 256k 2026-03-12 — 0 — 0.0 s 50%
ouro-2.6b mlx-community mlx 4bit 64k 2025-11-09 — 0 — 0.0 s 50%
qwen3.6-27b qwen gguf 8bit 256k 2026-04-21 — 0 — 0.0 s 50%
qwen3.5-2b lmstudio-community gguf 4bit 256k 2026-03-02 1.8 GB 162 1312 9.0 s 33%
qwen3-coder-30b qwen mlx 4bit 256k 2025-07-31 16.0 GB 113 743 8.4 s 33%
lfm2-24b-a2b liquid mlx 4bit 125k 2026-02-24 12.5 GB 144 763 6.8 s 25%
qwen2.5-coder-14b qwen mlx 4bit 32k 2024-11-08 7.8 GB 53 378 9.8 s 21%
gemma-3-4b google mlx 4bit 128k 2025-02-20 2.8 GB 153 384 5.2 s 17%
lfm2.5-1.2b liquid mlx 8bit 125k 2026-01-06 1.2 GB 294 328 1.6 s 12%

Click a row to open the model detail page. Hover shows available render previews. Column headers are sortable.