Open Weight Bench

Tool Use

Seven OpenAI-style function-calling scenarios across three difficulty tiers (easy / medium / hard). Toolset: list_files, read_file, apply_diff, get_weather. The model must pick tools, supply correct arguments, chain multiple calls and synthesise a final answer.

Task & test logic in detail
Task: 7 agentic-workflow scenarios across three difficulty tiers, with four mocked tools available (list_files, read_file, apply_diff, get_weather). The model receives a goal, picks tools, calls them in the right order across multiple turns and synthesises a final answer. Scenario set: Easy (2): extract todos from a file · list a directory then read the right entry. Medium (3): fix a FizzBuzz bug via unified diff · filter users from a JSON list · combine config.json with a weather mock for an alert. Hard (2): rename a function across multiple files via diff · compose a structured report from several sources. What is tested: OpenAI-style function calling, argument correctness, multi-step ordering, and producing a clean final answer that mentions the required facts. Why models fail: models without 'tool_use' capability ignore the schemas. Weak models pick wrong tools or emit malformed arguments. Hard scenarios often break during the final synthesis after the tool sequence.
Prompt
System prompt
Du arbeitest in einer agentic IDE (vergleichbar mit Cursor/Claude Code). Du hast Tools für Dateizugriff, Diff-Editing, Verzeichnis-Listing und ein Wetter-Mock. Plane mehrstufig: erst aufklären, dann handeln, am Ende prüfen. Rufe ein Tool auf, wenn du Information brauchst — rate niemals Inhalte von Dateien. Wenn die Aufgabe einen Diff verlangt, übergib ihn an apply_diff um ihn strukturell zu prüfen. Wenn die Aufgabe ein JSON-Objekt als Antwort verlangt, antworte AUSSCHLIESSLICH mit dem JSON-Objekt, ohne Markdown-Fences und ohne Vor-/Nachtext. Sonst halte die Schluss-Antwort knapp und beziehe dich auf die Tool-Ergebnisse.
Filter
– of 63 models shown

Wall-time vs. quality

X = wall-time for this bench · Y = score (0–100 %) in this bench. Optimum is top-left — fast and good. RAM estimate for 64k context: 4 GB system + model weights + max(2 GB, 40% of weights) for KV cache.

Colour = vendor · Number = total parameters (B) dense MoE

0% 25% 50% 75% 100% 0s 472s 945s 1418s 1890s Wall-time (s) → Score 5 5 26 8 8 30 26 35 8 9 35 118 110 27 30 31 27 31 122 27 20 30 80 32 26 12 12 30 30 8 35 14 30 35 30 30 120 12 4 4 30 4 24 120 7 9 9 14 36 4 2 9 9 4 32 4 70 24 1 15
Models in this bench
60 visible
  1. 1. gemma-4-e2b gguf 4bit 100% · 51s · 132 t/s · 10 GB
  2. 2. gemma-4-e2b gguf 8bit 100% · 78s · 113 t/s · 12 GB
  3. 3. gemma-4-26b-a4b-qat mlx 4bit 100% · 91s · 108 t/s · 24 GB
  4. 4. gemma-4-e4b gguf 4bit 100% · 80s · 89 t/s · 12 GB
  5. 5. qwen3-8b mlx 4bit 100% · 120s · 88 t/s · 10 GB
  6. 6. nemotron-3-nano-omni gguf 4bit 100% · 202s · 87 t/s · 38 GB
  7. 7. gemma-4-26b-a4b gguf 8bit 100% · 132s · 75 t/s · 41 GB
  8. 8. qwen3.6-35b-a3b gguf 8bit 100% · 70s · 70 t/s · 53 GB
  9. 9. gemma-4-e4b gguf 8bit 100% · 94s · 68 t/s · 16 GB
  10. 10. ornith-1.0-9b gguf 4bit 100% · 60s · 67 t/s · 11 GB
  11. 11. ornith-1.0-35b gguf 4bit 100% · 72s · 62 t/s · 32 GB
  12. 12. laguna-s-2.1 gguf 4bit 100% · 129s · 61 t/s · 97 GB
  13. 13. glm-4.5-air-mlx mlx 4bit 100% · 146s · 43 t/s · 82 GB
  14. 14. gemma-3-27b mlx 4bit 100% · 219s · 27 t/s · 26 GB
  15. 15. muse-glimmer gguf 100% · 446s · 25 t/s · 28 GB
  16. 16. gemma-4-31b-qat gguf 4bit 100% · 197s · 23 t/s · 29 GB
  17. 17. qwen3.6-27b gguf 4bit 100% · 179s · 22 t/s · 27 GB
  18. 18. gemma-4-31b gguf 4bit 100% · 249s · 21 t/s · 30 GB
  19. 19. qwen3.5-122b-a10b gguf 4bit 100% · 266s · 12 t/s · 102 GB
  20. 20. qwen3.5-27b-claude-4.6-opus-distilled-mlx mlx 4bit 100% · 651s · 9 t/s · 24 GB
  21. 21. gpt-oss-20b mlx 4bit 98% · 99s · 114 t/s · 20 GB
  22. 22. glm-4.7-flash mlx 4bit 98% · 53s · 85 t/s · 28 GB
  23. 23. qwen3-coder-next mlx 4bit 98% · 38s · 74 t/s · 67 GB
  24. 24. qwen2.5-coder-32b mlx 4bit 98% · 171s · 24 t/s · 28 GB
  25. 25. gemma-4-26b-a4b gguf 4bit 98% · 37s · 94 t/s · 27 GB
  26. 26. gemma-3-12b mlx 4bit 98% · 103s · 56 t/s · 15 GB
  27. 27. gemma-4-12b-qat gguf 4bit 98% · 65s · 53 t/s · 13 GB
  28. 28. nemotron-3-nano mlx 4bit 96% · 137s · 132 t/s · 27 GB
  29. 29. qwen3-vl-30b mlx 4bit 96% · 51s · 91 t/s · 28 GB
  30. 30. qwen3-vl-8b mlx 4bit 96% · 106s · 84 t/s · 12 GB
  31. 31. qwen3.5-35b-a3b gguf 4bit 96% · 35s · 81 t/s · 33 GB
  32. 32. qwen2.5-coder-14b mlx 4bit 96% · 62s · 50 t/s · 15 GB
  33. 33. qwen3-coder-30b mlx 4bit 95% · 33s · 109 t/s · 26 GB
  34. 34. qwen3.6-35b-a3b gguf 4bit 95% · 55s · 83 t/s · 33 GB
  35. 35. qwen3-30b-a3b-2507 mlx 4bit 94% · 22s · 107 t/s · 26 GB
  36. 36. glm-4.7-flash mlx 8bit 94% · 51s · 57 t/s · 46 GB
  37. 37. gpt-oss-120b gguf 4bit 92% · 71s · 82 t/s · 87 GB
  38. 38. mellum2-12b-a2.5b-thinking gguf 4bit 89% · 98s · 160 t/s · 15 GB
  39. 39. qwen3-4b-2507 mlx 4bit 89% · 26s · 150 t/s · 8 GB
  40. 40. nemotron-3-nano-4b gguf 4bit 89% · 109s · 86 t/s · 9 GB
  41. 41. nemotron-3-nano-omni gguf 8bit 89% · 244s · 77 t/s · 50 GB
  42. 42. qwen3-4b-thinking-2507 mlx 4bit 89% · 422s · 49 t/s · 8 GB
  43. 43. devstral-small-2-2512 mlx 4bit 89% · 64s · 37 t/s · 22 GB
  44. 44. nemotron-3-super gguf 4bit 89% · 364s · 33 t/s · 116 GB
  45. 45. granite-4-h-tiny gguf 4bit 89% · 32s · 118 t/s · 10 GB
  46. 46. glm-4.6v-flash mlx 4bit 86% · 134s · 67 t/s · 13 GB
  47. 47. qwen3.5-9b-mlx mlx 4bit 82% · 81s · 89 t/s · 12 GB
  48. 48. ministral-3-14b-reasoning gguf 4bit 82% · 52s · 54 t/s · 16 GB
  49. 49. seed-oss-36b mlx 4bit 76% · 835s · 21 t/s · 31 GB
  50. 50. qwen3.5-4b gguf 4bit 75% · 42s · 88 t/s · 9 GB
  51. 51. qwen3.5-2b gguf 4bit 75% · 31s · 166 t/s · 8 GB
  52. 52. qwen3.5-9b gguf 4bit 70% · 161s · 61 t/s · 13 GB
  53. 53. qwen3.5-9b gguf 8bit 69% · 69s · 45 t/s · 18 GB
  54. 54. gemma-3-4b mlx 4bit 61% · 27s · 143 t/s · 9 GB
  55. 55. olmo-3-32b-think mlx 4bit 55% · 1868s · 22 t/s · 28 GB
  56. 56. gemma-3n-e4b mlx 4bit 51% · 121s · 81 t/s · 12 GB
  57. 57. llama-3.3-70b gguf 4bit 50% · 122s · 11 t/s · 59 GB
  58. 58. lfm2-24b-a2b mlx 4bit 48% · 22s · 154 t/s · 22 GB
  59. 59. lfm2.5-1.2b mlx 8bit 5% · 10s · 287 t/s · 7 GB
  60. 60. phi-4-reasoning-plus mlx 4bit 0% · 936s · 45 t/s · 15 GB
Model Vendor Quant Ctx Released RAM tok/s Tokens Wall Score
gemma-4-e2b google gguf 4bit 128k 2026-03-02 4.1 GB 132 6154 50.8 s 100%
gemma-4-e2b google gguf 8bit 128k 2026-03-02 5.5 GB 113 8225 77.9 s 100%
gemma-4-26b-a4b-qat google mlx 4bit 256k 2026-03-12 14.6 GB 108 8536 91.1 s 100%
gemma-4-e4b google gguf 4bit 128k 2026-03-02 5.9 GB 89 6672 80.1 s 100%
qwen3-8b qwen mlx 4bit 40k 2025-04-27 4.3 GB 88 9664 120.0 s 100%
nemotron-3-nano-omni nvidia gguf 4bit 256k 2026-04-20 24.3 GB 87 16547 202.2 s 100%
gemma-4-26b-a4b google gguf 8bit 256k 2026-03-12 26.1 GB 75 8835 131.7 s 100%
qwen3.6-35b-a3b qwen gguf 8bit 256k 2026-04-15 35.2 GB 70 3994 69.5 s 100%
gemma-4-e4b google gguf 8bit 128k 2026-03-02 8.4 GB 68 5971 93.9 s 100%
ornith-1.0-9b deepreinforce-ai gguf 4bit 256k 2026-06 5.2 GB 67 3150 60.3 s 100%
ornith-1.0-35b deepreinforce-ai gguf 4bit 256k 2026-06 19.7 GB 62 2970 72.0 s 100%
laguna-s-2.1 poolside gguf 4bit 256k 2026-07-21 66.3 GB 61 1597 129.5 s 100%
glm-4.5-air-mlx lmstudio-community mlx 4bit 128k 2025-07-28 56.0 GB 43 4074 145.5 s 100%
gemma-3-27b google mlx 4bit 128k 2025-03-01 15.7 GB 27 4031 218.7 s 100%
muse-glimmer meta gguf 128k 2026-08-10 16.9 GB 25 9806 446.5 s 100%
gemma-4-31b-qat google gguf 4bit 256k 2026-03-12 17.6 GB 23 3747 196.6 s 100%
qwen3.6-27b qwen gguf 4bit 256k 2026-04-21 16.3 GB 22 3040 179.4 s 100%
gemma-4-31b google gguf 4bit 256k 2026-03-12 18.5 GB 21 4124 249.2 s 100%
qwen3.5-122b-a10b lmstudio-community gguf 4bit 256k 2026-02-24 70.0 GB 12 1888 265.9 s 100%
qwen3.5-27b-claude-4.6-opus-distilled-mlx mlx-community mlx 4bit 256k 2026-03-04 14.1 GB 9 3986 650.6 s 100%
gpt-oss-20b openai mlx 4bit 128k 2025-08-04 11.3 GB 114 8636 98.5 s 98%
glm-4.7-flash zai-org mlx 4bit 198k 2026-01-19 16.9 GB 85 2885 53.4 s 98%
qwen3-coder-next qwen mlx 4bit 256k 2026-01-30 45.3 GB 74 1519 38.0 s 98%
qwen2.5-coder-32b qwen mlx 4bit 32k 2024-11-08 17.2 GB 24 2554 171.0 s 98%
gemma-4-26b-a4b google gguf 4bit 256k 2026-03-12 16.8 GB 94 2737 36.8 s 98%
gemma-3-12b google mlx 4bit 128k 2025-03-01 7.5 GB 56 3795 102.8 s 98%
gemma-4-12b-qat google gguf 4bit 256k 2026-03-12 6.7 GB 53 2513 64.9 s 98%
nemotron-3-nano nvidia mlx 4bit 256k 2025-12-15 16.6 GB 132 14908 136.7 s 96%
qwen3-vl-30b qwen mlx 4bit 256k 2025-10-04 17.0 GB 91 1840 51.4 s 96%
qwen3-vl-8b qwen mlx 4bit 256k 2025-10-11 5.4 GB 84 3981 106.2 s 96%
qwen3.5-35b-a3b qwen gguf 4bit 256k 2026-02-24 20.6 GB 81 2090 34.6 s 96%
qwen2.5-coder-14b qwen mlx 4bit 32k 2024-11-08 7.8 GB 50 1615 62.1 s 96%
qwen3-coder-30b qwen mlx 4bit 256k 2025-07-31 16.0 GB 109 1304 33.0 s 95%
qwen3.6-35b-a3b qwen gguf 4bit 256k 2026-04-15 20.6 GB 83 3831 55.4 s 95%
qwen3-30b-a3b-2507 qwen mlx 4bit 256k 2025-07-21 16.0 GB 107 1675 22.3 s 94%
glm-4.7-flash zai-org mlx 8bit 198k 2026-01-19 29.7 GB 57 1250 51.1 s 94%
gpt-oss-120b openai gguf 4bit 128k 2025-08-04 59.0 GB 82 4241 70.9 s 92%
mellum2-12b-a2.5b-thinking JetBrains gguf 4bit 128k 2026-06-02 7.5 GB 160 14289 97.8 s 89%
qwen3-4b-2507 qwen mlx 4bit 256k 2025-08-06 2.1 GB 150 1754 26.4 s 89%
nemotron-3-nano-4b nvidia gguf 4bit 1024k 2026-03-07 2.6 GB 86 8477 108.8 s 89%
nemotron-3-nano-omni nvidia gguf 8bit 256k 2026-04-20 32.8 GB 77 17645 244.0 s 89%
qwen3-4b-thinking-2507 qwen mlx 4bit 256k 2025-08-05 2.1 GB 49 20242 422.2 s 89%
devstral-small-2-2512 mistralai mlx 4bit 384k 2025-12-09 13.2 GB 37 1317 63.9 s 89%
nemotron-3-super nvidia gguf 4bit 1024k 2026-03-10 80.1 GB 33 10677 364.2 s 89%
granite-4-h-tiny ibm gguf 4bit 1024k 2025-10-02 3.9 GB 118 3070 32.1 s 89%
glm-4.6v-flash zai-org mlx 4bit 128k 2025-12-07 6.6 GB 67 5103 133.9 s 86%
qwen3.5-9b-mlx mlx-community mlx 4bit 256k 2026-02-27 5.6 GB 89 4672 80.6 s 82%
ministral-3-14b-reasoning mistralai gguf 4bit 256k 2025-10-31 8.5 GB 54 1950 51.5 s 82%
seed-oss-36b bytedance mlx 4bit 512k 2025-08-20 19.0 GB 21 15737 834.8 s 76%
qwen3.5-4b lmstudio-community gguf 4bit 256k 2026-03-02 3.2 GB 88 3101 41.6 s 75%
qwen3.5-2b lmstudio-community gguf 4bit 256k 2026-03-02 1.8 GB 166 4454 31.3 s 75%
qwen3.5-9b qwen gguf 4bit 256k 2026-02-27 6.1 GB 61 8776 160.8 s 70%
qwen3.5-9b qwen gguf 8bit 256k 2026-02-27 9.7 GB 45 2663 69.2 s 69%
gemma-3-4b google mlx 4bit 128k 2025-02-20 2.8 GB 143 1881 27.2 s 61%
olmo-3-32b-think allenai mlx 4bit 64k 2025-11-19 16.9 GB 22 40334 1868.0 s 55%
gemma-3n-e4b google mlx 4bit 32k 2025-06-03 5.5 GB 81 7673 120.5 s 51%
llama-3.3-70b meta gguf 4bit 128k 2024-12-06 39.6 GB 11 542 121.6 s 50%
lfm2-24b-a2b liquid mlx 4bit 125k 2026-02-24 12.5 GB 154 947 21.6 s 48%
lfm2.5-1.2b liquid mlx 8bit 125k 2026-01-06 1.2 GB 287 658 9.7 s 5%
phi-4-reasoning-plus microsoft mlx 4bit 32k 2025-04-17 7.7 GB 45 41993 936.3 s 0%
gemma-4-31b google gguf 8bit 256k 2026-03-12 — 0 — 0.0 s —
ouro-2.6b mlx-community mlx 4bit 64k 2025-11-09 — 0 — 0.0 s —
qwen3.6-27b qwen gguf 8bit 256k 2026-04-21 — 0 — 0.0 s —

Click a row to open the model detail page. Hover shows available render previews. Column headers are sortable. The filter at the top of the page hides table rows and dims the matching dots in the chart.