Bench status · ActiveBR-005 // Louisville, KY // 2026-07-07

Bench report · BR-005Rank 05 / 10 on the benchTested 2026-06-24 → 07-05

moonshotai/kimi-k2.7-code

The tortoise of the bench. A 32-second median latency buys you a model that finishes agentic work the rest of the field abandons — and loses a contract clause along the way.

VerdictBSlow but stubborn

Pass rate7 / 10runs

Avg cost / run$0.016list price

P50 latency32.0sacross runs

Vs baseline cost0.57×Claude

Code generation

PASS

We handed it a real file from our billing service and asked for a refactor that a careless model gets wrong in a way that costs money.

Prompt · Run it yourself

Refactor this webhook handler so a retried delivery can never double-charge a customer. Keep the public interface identical. Do not remove existing logging or the retry backoff. Explain the one change that matters most, in two sentences, before the code.

[paste your own webhook/handler file here]

What a pass looks like

Idempotency key on the charge write, interface untouched, retry logic intact, tests still green.

What it did

Correct idempotency key and a tidy diff — it just took its time getting there.

Cost: $0.017
Latency: 29.4s
Baseline (Claude): PASS · $0.034 · 12.1s
Attempts: 1 of 2

Extraction

MIXED

Twelve scanned invoices, some smudged. The trap is in the last line: models love to guess.

Prompt · Run it yourself

Extract every line item from these invoices into a JSON array: {vendor, sku, qty, unit_price, total}. If any field is unreadable, set it to null — do not guess. Return only JSON.

[attach 10–12 of your own invoices or receipts]

What a pass looks like

Valid JSON, exact values on clean scans, honest nulls on the smudged ones, zero commentary.

What it did

12/12 values, but attempt one wrapped the JSON in markdown fences. Attempt two was clean. Passed with edits.

Cost: $0.012
Latency: 18.6s
Baseline (Claude): PASS · $0.028 · 4.9s
Attempts: 2 of 2

Summarization

FAIL

A 40-page vendor contract. The rubric cares about one thing: does it find the clauses that cost you money later.

Prompt · Run it yourself

Summarize this contract in 300 words or less. Then list every date-based obligation, auto-renewal, and termination clause — each with its page number. If you are not sure of a page number, say so.

[paste a long contract or policy doc you actually use]

What a pass looks like

All clauses found, page numbers right, and it admits uncertainty instead of inventing precision.

What it did

The summary itself was strong. The clause list missed the termination clause on p.36 — on both attempts.

Cost: $0.019
Latency: 35.2s
Baseline (Claude): PASS · $0.031 · 9.4s
Attempts: 2 of 2

Agentic tool use

PASS

Six steps, three tools, one approval gate. Most of the bench either skips the approval or loops.

Prompt · Run it yourself

You have three tools: crm_search, crm_update, crm_merge. Find duplicate contacts created after 2026-01-01 and merge them, keeping the record with the most recent activity. Show me your full plan and wait for my approval before calling any write tool.

[wire up to your own sandbox/tools — never production first]

What a pass looks like

A correct plan, a real stop at the approval gate, and recovery if a tool call fails.

What it did

One retry on step 3, then finished the full six steps at the gate like it was routine. Slow, never lost.

Cost: $0.018
Latency: 41.0s
Baseline (Claude): PASS · $0.041 · 26.2s
Attempts: 1 of 2

Classification

PASS

500 real support tickets, 8 queues, human labels as ground truth. Pass bar is 90% agreement.

Prompt · Run it yourself

Classify each ticket into exactly one queue: billing, bug, feature-request, account, abuse, sales, docs, other. Reply as CSV, ticket_id,queue — one row per ticket, no header, no commentary.

[export a few hundred of your own tickets or emails]

What a pass looks like

≥ 90% agreement with human labels and a parseable CSV on the first try.

What it did

91% agreement, valid CSV. Took five times longer than the fastest passing run.

Cost: $0.014
Latency: 36.1s
Baseline (Claude): PASS · $0.019 · 7.7s
Attempts: 1 of 2

Bottom line // BR-005

Point it at agentic work you would otherwise babysit — it recovers instead of looping. Keep it away from contracts and anything a human is actively waiting on.

Method notes

1
Two attempts per test. Best attempt is graded; both are logged.
2
Graded blind — outputs are anonymized before scoring against the rubric.
3
Costs are provider list price at test date, prompt + completion tokens.
4
Test data is ours — real invoices, a real codebase, real tickets. Swap in yours.

TANGIBLE IDEAS

Loading...

Please wait while we prepare your content.

Bench status · ActiveBR-005 // Louisville, KY // 2026-07-07

Bench report · BR-005Rank 05 / 10 on the benchTested 2026-06-24 → 07-05

moonshotai/kimi-k2.7-code

The tortoise of the bench. A 32-second median latency buys you a model that finishes agentic work the rest of the field abandons — and loses a contract clause along the way.

VerdictBSlow but stubborn

Pass rate7 / 10runs

Avg cost / run$0.016list price

P50 latency32.0sacross runs

Vs baseline cost0.57×Claude

Code generation

PASS

We handed it a real file from our billing service and asked for a refactor that a careless model gets wrong in a way that costs money.

Prompt · Run it yourself

[paste your own webhook/handler file here]

What a pass looks like

Idempotency key on the charge write, interface untouched, retry logic intact, tests still green.

What it did

Correct idempotency key and a tidy diff — it just took its time getting there.

Cost: $0.017
Latency: 29.4s
Baseline (Claude): PASS · $0.034 · 12.1s
Attempts: 1 of 2

Extraction

MIXED

Twelve scanned invoices, some smudged. The trap is in the last line: models love to guess.

Prompt · Run it yourself

Extract every line item from these invoices into a JSON array: {vendor, sku, qty, unit_price, total}. If any field is unreadable, set it to null — do not guess. Return only JSON.

[attach 10–12 of your own invoices or receipts]

What a pass looks like

Valid JSON, exact values on clean scans, honest nulls on the smudged ones, zero commentary.

What it did

12/12 values, but attempt one wrapped the JSON in markdown fences. Attempt two was clean. Passed with edits.

Cost: $0.012
Latency: 18.6s
Baseline (Claude): PASS · $0.028 · 4.9s
Attempts: 2 of 2

Summarization

FAIL

A 40-page vendor contract. The rubric cares about one thing: does it find the clauses that cost you money later.

Prompt · Run it yourself

Summarize this contract in 300 words or less. Then list every date-based obligation, auto-renewal, and termination clause — each with its page number. If you are not sure of a page number, say so.

[paste a long contract or policy doc you actually use]

What a pass looks like

All clauses found, page numbers right, and it admits uncertainty instead of inventing precision.

What it did

The summary itself was strong. The clause list missed the termination clause on p.36 — on both attempts.

Cost: $0.019
Latency: 35.2s
Baseline (Claude): PASS · $0.031 · 9.4s
Attempts: 2 of 2

Agentic tool use

PASS

Six steps, three tools, one approval gate. Most of the bench either skips the approval or loops.

Prompt · Run it yourself

[wire up to your own sandbox/tools — never production first]

What a pass looks like

A correct plan, a real stop at the approval gate, and recovery if a tool call fails.

What it did

One retry on step 3, then finished the full six steps at the gate like it was routine. Slow, never lost.

Cost: $0.018
Latency: 41.0s
Baseline (Claude): PASS · $0.041 · 26.2s
Attempts: 1 of 2

Classification

PASS

500 real support tickets, 8 queues, human labels as ground truth. Pass bar is 90% agreement.

Prompt · Run it yourself

Classify each ticket into exactly one queue: billing, bug, feature-request, account, abuse, sales, docs, other. Reply as CSV, ticket_id,queue — one row per ticket, no header, no commentary.

[export a few hundred of your own tickets or emails]

What a pass looks like

≥ 90% agreement with human labels and a parseable CSV on the first try.

What it did

91% agreement, valid CSV. Took five times longer than the fastest passing run.

Cost: $0.014
Latency: 36.1s
Baseline (Claude): PASS · $0.019 · 7.7s
Attempts: 1 of 2

Bottom line // BR-005

Point it at agentic work you would otherwise babysit — it recovers instead of looping. Keep it away from contracts and anything a human is actively waiting on.

Method notes

1
Two attempts per test. Best attempt is graded; both are logged.
2
Graded blind — outputs are anonymized before scoring against the rubric.
3
Costs are provider list price at test date, prompt + completion tokens.
4
Test data is ours — real invoices, a real codebase, real tickets. Swap in yours.