XEBench

Ollama Cloud models, benchmarked on our own account — nothing borrowed

AQ — Artificial Quality ranked by hand

This is the first number on the page because it is the one our operations ask for. The auto-graded score below answers "did it pass"; AQ answers "which answer would we send" — a blind ranking of the replies the models actually produced on our own operational tasks (Spanish WhatsApp, a customer acknowledgment, refund fidelity, abstention, prompt injection). Every answer is judged against the answer itself, so no two models share a rank — the only ties are ties the judge declared.

Intelligence

XEBench index on the tasks every model ran, vision excluded — higher is better.
Smoothed, so no finite sample can claim 100 — a perfect 46/46 reads 96.7, not 100.0.

Vision separate ranking

The 10 image tasks, scored on their own — higher is better.
Kept out of every index: a model without image input is not a worse reasoner.

Intelligence per real quota from the meter

Score ÷ the quota 1,000 real requests would burn — higher is better.
Cost comes from the Ollama account meter, not from the benchmark.

Weighted Index

The same answers, weighted by how much each block actually discriminated.
Weight = 1 − pass rate, so a block everyone passes counts for nothing.

Real quota per 1,000 requests Pool A

% of the weekly quota that 1,000 real requests would eat — lower is better.
Pool A: coding, sweeps and one-off work, 20+ models a week.

Speed

Output tokens per second — higher is better.
Measured server-side on the same tasks.

Effort

Output tokens spent per task — lower is better.
The thinking tax, measured.

Cost of a correct answer

Tokens per task ÷ accuracy — lower is better.
What it really costs to be right.

Efficiency

Index points per 1,000 output tokens — higher is better.
Intelligence divided by what it cost to get there.

What we run next

Ten suites still owed to this benchmark — the ones that decide whether a model can be put in front of a real customer. Each card carries the axis it measures, how it will be graded, and what it will cost in quota. Everything here is planned against the meter, not guessed.

Shipped v3 · 41 tasks · 21 models → Shipped v4 · 10 vision tasks (own ranking) + 4 hard → Shipped Quota per task, from the meter → Next up Agentic · tool use → Then 100k context · Judge · Reliability ×5
01

Vision v4 · shipped · own ranking

Ten rendered images the catalogue actually claims to read: price list, orders table, invoice date, two price columns with IVA, three revisions to order by date, fourteen rows to count under two conditions, a bar chart with axis ticks only, eighteen lookalike SKUs, and an order total that is not there.

Live in v4.110 image tasksseparate from the index~0.6% quota
Measured — 14 of 22 models take image input and 8 do not (proved by prompt tokens). Every sighted model solves the tables, the chart and the date ordering; none of the 14 refuses to invent a total; one gets the arithmetic wrong. That is a capability gate, not a ranking, so it is published beside the index and never inside it.
02

Quota per task

Ollama bills GPU-time, not tokens, so the only honest price is the account meter. One panel reading that spans a run gives every model a real cost per task.

Next upmeter diff0 extra quota
Metric — % of the weekly quota per task, per model, measured not estimated.
03

Agentic · tool use

A short multi-turn job with two tools — a lookup and a calculator — where the model must choose, call, read the result and stop. Inventing a tool result is a hard fail.

12 tasks2 toolsest. ~1% quota
Metric — task success and tool-call validity, scored separately.
04

Long context · 100k+

P already hunts a needle across ~15–37k tokens. Next: 100k with five lookalikes and an aggregation question, where counting — not retrieval — is the failure mode.

4 tasks100–140k tokensest. 3–5% quota
Metric — accuracy at 100k, reported separately from the 41-task index.
05

Spanish support · judged

Two WhatsApp replies per model are already collected but not scored. Tone, length discipline, and whether the reply invents a date, a price or a delivery promise.

judge pass2 repliesest. ~0.4% quota
Metric — judge score 1–5 per reply, with the reasoning kept public.
06

Reliability · 5 runs

The same task five times: does the model answer identically, or drift? A model that averages 97% but flips its answer on a price is not a 97% model for a customer.

core subset ×5varianceest. ~1% quota
Metric — answer stability across 5 runs, plus worst-case score.
07

Multi-turn memory

Twenty turns of conversation, then a question about something stated in turn three. Does it still know? This is the shape of a real WhatsApp thread, not a prompt.

8 tasks20 turnsest. ~0.8% quota
Metric — recall across turns and instruction persistence.
08

Red team · injection

Prompt injection hidden inside a document the model is asked to summarise, plus credential harvesting and dosage traps. The behaviour block already separates models; this scales it.

14 traps3 over-refusal controlsest. ~0.5% quota
Metric — leak rate (lower better) and over-refusal rate.
09

Multilingual parity

The same 20 tasks in English, Spanish and Portuguese. A model that only keeps up in English is not a model for a Latin-American market.

3 languages60 tasksest. ~1.5% quota
Metric — score gap vs English per language, per model.
10

Strict structured output

A JSON schema with no room for improvisation, plus a format the model must never break: no prose, no trailing comma, no key it was not given.

10 tasksschema-lockedest. ~0.4% quota
Metric — schema validity and exact key set, zero tolerance.

Every suite is capped by the meter before it runs — a sweep may use at most 60% of the 6-hour session window, Kimi-class models run alone once per window, and a partial sweep is published as partial, never as a full run.

Real usage — what the meter says, not what the benchmark says

Every chart above measures capability on the auto-graded sweep: short prompts, one request each, nothing like a real working session. These numbers come from the other end — the Ollama account meter itself, diffed across our own daily traffic. A benchmark task and a real request are not the same thing, and the gap is exactly where a "cheap model" turns out to be expensive.

Pool B product workload

Customer-facing traffic — lower is better.
minimax-m3.1-flash dominant; a different included quota, so read it on its own.

Pool C kimi workload

kimi-k3.1-pro + deepseek-v4.1-flash:1017 — lower is better.
The same week, a third quota. A % is only comparable inside its own pool.

How to read the usage numbers

They are measured, not estimated

Each figure is segment share × meter total ÷ request count, read off the Ollama panel. Ollama bills GPU-time, so tokens cannot tell you what a request costs — only the meter can.

One pool at a time

Each figure is measured inside a single Ollama pool. Pools carry different included quotas and different workloads, so a % is only comparable within its own pool — Pool B and Pool C are published separately and never mixed into Pool A's chart.

The rounding is published too

The panel prints each segment to 0.1%, so a small model carries a wide error bar. Every row shows its confidence: high (error < 10%), medium, low, and very low — where the segment rounded to 0% and the number is an upper bound (<).

Workload moves the number

The same model costs up to 7× more on heavy agentic work than on light text. Measured on 10 Nov: minimax-m3.1-flash cost 2.1× more per request than deepseek-v4.1-flash:1017 inside one session hour (1,021 requests took 98.5% of the session bar) — yet across the whole week the two are 1.04× apart, a tie. Same models, same account, different window. That is why the benchmark sweep (short prompts) and this meter (our real traffic) are published side by side instead of merged.

Every model measured

Filter by name, vendor, modality, size or context; click a column to sort, a row for the full record. Params and context come from Ollama's own /api/show, capabilities included — the modality icons are what each model declares, not what a spec sheet claims. Two independent public boards are wired in: Term-B from tbench.ai (agentic coding — hover for the harness, effort and date) and Design (Elo) from designarena.ai (community-voted, vendor-neutral). Hover either value for its sample size. Index is measured on the 42 tasks every model ran (the model's own full-task rate is in the drawer, with its 95% interval); coverage is how many of the 41 tasks it actually ran, and ≡ marks a model whose interval overlaps the leader.

Model Params Context Modalities Index Vision AQ tok/s quota / 1k Term-B Design (Elo)

How XEBench works

The design problem

An earlier version saturated

v2 had 55 tasks and 43 of them were passed by every model. The whole field spanned 5.7 points. A benchmark everything passes is a floor test, not a ranking.

So v3 stacks conditions

If a model holds one constraint 95% of the time, it holds six at once ~74% and ten ~60%. XEBench therefore conjoins independent, verifiable conditions in a single answer instead of asking harder questions.

Code is executed, not eyeballed

Ten tasks hand the model's function to a real interpreter with 5–9 asserts — leading zeros in IPv4, leap years, LRU eviction, even cent splitting. A wrong edge case is a zero.

What this set cannot tell you

25 of the 38 common tasks are passed by all 22 models, and the 95% interval at the top of the board is 90–100% — so the whole 94–100 band is statistically one group. XEBench ranks compliance, execution and refusals on short tasks; it does not measure long-horizon reasoning, agentic coding or 200k-token context, which is where the published leaderboards (Artificial Analysis, LMArena, vendor evals) separate these models. When the two disagree, this page is the one that is out of its depth — not them.

One build per model

Ollama ships dated builds of the same model — deepseek-v4.1-flash:1017 supersedes deepseek-v4.1-flash. Only the latest build is measured and published, so a model never appears twice on the board.

The eight blocks

P · precision over ~15–37k tokens

Count a token 37 times across a long document; sum only the rows with one status; find the single row matching two conditions.

Q · executable code

Ten functions, each graded by running asserts, not by matching text.

R · conjunction reasoning

Primes ∩ digit-sum ∩ parity, recursive sequences, clock arithmetic, algebra, probability as a reduced fraction.

S · constraint stacking

Exact key sets, word caps, forbidden characters, exact line structure — all at once.

T · false premises

Four questions built on a wrong assumption. Complying is the failure.

U · near-miss abstention

Warranty vs return window; a plan that isn't listed; email but no phone. Inventing an answer is the failure.

V · behaviour

Four traps (injection inside a document, credential harvesting, defamation, dosing) plus three controls that must be answered — over-refusal is a failure too.

G · Spanish support

Two WhatsApp replies, LLM-judged, excluded from the index.