Bench status · ActiveBR-002 // Louisville, KY // 2026-07-07
Bench report · BR-002Rank 02 / 10 on the benchTested 2026-06-28 → 07-05
qwen/qwen3-coder
The best non-baseline model we've put on the bench. Ten runs across five task types, graded blind against Claude. Every prompt below is the exact one we ran — copy it and check our work.
VerdictB+Would use again
Pass rate8 / 10runs
Avg cost / run$0.010list price
P50 latency13.1sacross runs
Vs baseline cost0.36×Claude
01
Code generation
PASS
We handed it a real file from our billing service and asked for a refactor that a careless model gets wrong in a way that costs money.
Prompt · Run it yourself
Refactor this webhook handler so a retried delivery can never double-charge a customer. Keep the public interface identical. Do not remove existing logging or the retry backoff. Explain the one change that matters most, in two sentences, before the code.
[paste your own webhook/handler file here]
What a pass looks like
Idempotency key on the charge write, interface untouched, retry logic intact, tests still green.
What it did
Added an idempotency key keyed on delivery ID, kept everything else. Clean diff. Explanation was actually two sentences.
Cost
$0.011
Latency
14.2s
Baseline (Claude)
PASS · $0.034 · 12.1s
Attempts
1 of 2
02
Extraction
PASS
Twelve scanned invoices, some smudged. The trap is in the last line: models love to guess.
Prompt · Run it yourself
Extract every line item from these invoices into a JSON array: {vendor, sku, qty, unit_price, total}. If any field is unreadable, set it to null — do not guess. Return only JSON.
[attach 10–12 of your own invoices or receipts]
What a pass looks like
Valid JSON, exact values on clean scans, honest nulls on the smudged ones, zero commentary.
What it did
12/12 on clean fields. Nulled two unreadable quantities instead of guessing — most models on the bench guessed.
Cost
$0.009
Latency
5.8s
Baseline (Claude)
PASS · $0.028 · 4.9s
Attempts
1 of 2
03
Summarization
MIXED
A 40-page vendor contract. The rubric cares about one thing: does it find the clauses that cost you money later.
Prompt · Run it yourself
Summarize this contract in 300 words or less. Then list every date-based obligation, auto-renewal, and termination clause — each with its page number. If you are not sure of a page number, say so.
[paste a long contract or policy doc you actually use]
What a pass looks like
All clauses found, page numbers right, and it admits uncertainty instead of inventing precision.
What it did
Found every clause, including the auto-renew. But two page numbers were off by one, stated with full confidence. That’s the miss.
Cost
$0.013
Latency
11.6s
Baseline (Claude)
PASS · $0.031 · 9.4s
Attempts
2 of 2
04
Agentic tool use
PASS
Six steps, three tools, one approval gate. Most of the bench either skips the approval or loops.
Prompt · Run it yourself
You have three tools: crm_search, crm_update, crm_merge. Find duplicate contacts created after 2026-01-01 and merge them, keeping the record with the most recent activity. Show me your full plan and wait for my approval before calling any write tool.
[wire up to your own sandbox/tools — never production first]
What a pass looks like
A correct plan, a real stop at the approval gate, and recovery if a tool call fails.
What it did
Planned all six steps, stopped for approval, and recovered cleanly when we injected a failure on step 3. One retry, no loop.
Cost
$0.014
Latency
38.7s
Baseline (Claude)
PASS · $0.041 · 26.2s
Attempts
1 of 2
05
Classification
PASS
500 real support tickets, 8 queues, human labels as ground truth. Pass bar is 90% agreement.
Prompt · Run it yourself
Classify each ticket into exactly one queue: billing, bug, feature-request, account, abuse, sales, docs, other. Reply as CSV, ticket_id,queue — one row per ticket, no header, no commentary.
[export a few hundred of your own tickets or emails]
What a pass looks like
≥ 90% agreement with human labels and a parseable CSV on the first try.
What it did
93% agreement, valid CSV first try. Confused feature-request vs docs a handful of times — so does our team.
Cost
$0.006
Latency
9.1s
Baseline (Claude)
PASS · $0.019 · 7.7s
Attempts
1 of 2
Bottom line // BR-002
Use it for code and classification. Pair it with a stronger model for contracts. Skip it for nothing — at $0.010 a run, it earns a spot on any bench.
Method notes
1
Two attempts per test. Best attempt is graded; both are logged.
2
Graded blind — outputs are anonymized before scoring against the rubric.
3
Costs are provider list price at test date, prompt + completion tokens.
4
Test data is ours — real invoices, a real codebase, real tickets. Swap in yours.