Ten suites still owed to this benchmark — the ones that decide whether a model can be put in
front of a real customer. Each card carries the axis it measures, how it will be graded, and what it will
cost in quota. Everything here is planned against the meter, not guessed.
Vision v4 · shipped · own ranking
Ten rendered images the catalogue actually claims to read: price list, orders table, invoice
date, two price columns with IVA, three revisions to order by date, fourteen rows to count under two
conditions, a bar chart with axis ticks only, eighteen lookalike SKUs, and an order total that is not there.
Live in v4.110 image tasksseparate from the index~0.6% quota
Measured — 14 of 22 models take image input and 8 do not (proved by prompt tokens).
Every sighted model solves the tables, the chart and the date ordering; none of the 14 refuses to invent a
total; one gets the arithmetic wrong. That is a capability gate, not a ranking, so it is published beside
the index and never inside it.
Quota per task
Ollama bills GPU-time, not tokens, so the only honest price is the account meter. One
panel reading that spans a run gives every model a real cost per task.
Next upmeter diff0 extra quota
Metric — % of the weekly quota per task, per model, measured not estimated.
Agentic · tool use
A short multi-turn job with two tools — a lookup and a calculator — where the model must
choose, call, read the result and stop. Inventing a tool result is a hard fail.
12 tasks2 toolsest. ~1% quota
Metric — task success and tool-call validity, scored separately.
Long context · 100k+
P already hunts a needle across ~15–37k tokens. Next: 100k with five lookalikes and an
aggregation question, where counting — not retrieval — is the failure mode.
4 tasks100–140k tokensest. 3–5% quota
Metric — accuracy at 100k, reported separately from the 41-task index.
Spanish support · judged
Two WhatsApp replies per model are already collected but not scored. Tone, length
discipline, and whether the reply invents a date, a price or a delivery promise.
judge pass2 repliesest. ~0.4% quota
Metric — judge score 1–5 per reply, with the reasoning kept public.
Reliability · 5 runs
The same task five times: does the model answer identically, or drift? A model that
averages 97% but flips its answer on a price is not a 97% model for a customer.
core subset ×5varianceest. ~1% quota
Metric — answer stability across 5 runs, plus worst-case score.
Multi-turn memory
Twenty turns of conversation, then a question about something stated in turn three.
Does it still know? This is the shape of a real WhatsApp thread, not a prompt.
8 tasks20 turnsest. ~0.8% quota
Metric — recall across turns and instruction persistence.
Red team · injection
Prompt injection hidden inside a document the model is asked to summarise, plus credential
harvesting and dosage traps. The behaviour block already separates models; this scales it.
14 traps3 over-refusal controlsest. ~0.5% quota
Metric — leak rate (lower better) and over-refusal rate.
Multilingual parity
The same 20 tasks in English, Spanish and Portuguese. A model that only keeps up in
English is not a model for a Latin-American market.
3 languages60 tasksest. ~1.5% quota
Metric — score gap vs English per language, per model.
Strict structured output
A JSON schema with no room for improvisation, plus a format the model must never
break: no prose, no trailing comma, no key it was not given.
10 tasksschema-lockedest. ~0.4% quota
Metric — schema validity and exact key set, zero tolerance.