Bench status · ActiveBR-008 // Louisville, KY // 2026-07-07

Bench report · BR-008Rank 08 / 10 on the benchTested 2026-06-23 → 07-02

nousresearch/hermes-4-70b

The best writer on the bench and one of its least reliable engineers. When the deliverable is words for humans it shines; when the deliverable is JSON, CSV, or a diff, it improvises.

VerdictCGreat prose, unreliable structure

Pass rate5 / 10runs

Avg cost / run$0.006list price

P50 latency8.2sacross runs

Vs baseline cost0.21×Claude

Code generation

FAIL

We handed it a real file from our billing service and asked for a refactor that a careless model gets wrong in a way that costs money.

Prompt · Run it yourself

Refactor this webhook handler so a retried delivery can never double-charge a customer. Keep the public interface identical. Do not remove existing logging or the retry backoff. Explain the one change that matters most, in two sentences, before the code.

[paste your own webhook/handler file here]

What a pass looks like

Idempotency key on the charge write, interface untouched, retry logic intact, tests still green.

What it did

A beautifully explained refactor that changed the public interface. Both attempts.

Cost: $0.007
Latency: 9.1s
Baseline (Claude): PASS · $0.034 · 12.1s
Attempts: 2 of 2

Extraction

PASS

Twelve scanned invoices, some smudged. The trap is in the last line: models love to guess.

Prompt · Run it yourself

Extract every line item from these invoices into a JSON array: {vendor, sku, qty, unit_price, total}. If any field is unreadable, set it to null — do not guess. Return only JSON.

[attach 10–12 of your own invoices or receipts]

What a pass looks like

Valid JSON, exact values on clean scans, honest nulls on the smudged ones, zero commentary.

What it did

12/12 on clean fields, honest nulls where it couldn’t read. Its one unqualified win.

Cost: $0.006
Latency: 7.4s
Baseline (Claude): PASS · $0.028 · 4.9s
Attempts: 1 of 2

Summarization

FAIL

A 40-page vendor contract. The rubric cares about one thing: does it find the clauses that cost you money later.

Prompt · Run it yourself

Summarize this contract in 300 words or less. Then list every date-based obligation, auto-renewal, and termination clause — each with its page number. If you are not sure of a page number, say so.

[paste a long contract or policy doc you actually use]

What a pass looks like

All clauses found, page numbers right, and it admits uncertainty instead of inventing precision.

What it did

The best-written summary on the bench — and it missed the renewal clause on p.31. Prose is not the rubric.

Cost: $0.006
Latency: 7.9s
Baseline (Claude): PASS · $0.031 · 9.4s
Attempts: 2 of 2

Agentic tool use

FAIL

Six steps, three tools, one approval gate. Most of the bench either skips the approval or loops.

Prompt · Run it yourself

You have three tools: crm_search, crm_update, crm_merge. Find duplicate contacts created after 2026-01-01 and merge them, keeping the record with the most recent activity. Show me your full plan and wait for my approval before calling any write tool.

[wire up to your own sandbox/tools — never production first]

What a pass looks like

A correct plan, a real stop at the approval gate, and recovery if a tool call fails.

What it did

Wrote an essay about the plan instead of executing it. Never called a single tool.

Cost: $0.005
Latency: 10.6s
Baseline (Claude): PASS · $0.041 · 26.2s
Attempts: 2 of 2

Classification

MIXED

500 real support tickets, 8 queues, human labels as ground truth. Pass bar is 90% agreement.

Prompt · Run it yourself

Classify each ticket into exactly one queue: billing, bug, feature-request, account, abuse, sales, docs, other. Reply as CSV, ticket_id,queue — one row per ticket, no header, no commentary.

[export a few hundred of your own tickets or emails]

What a pass looks like

≥ 90% agreement with human labels and a parseable CSV on the first try.

What it did

88% with commentary on attempt one; 90% flat on attempt two. Scraped by with edits.

Cost: $0.005
Latency: 8.0s
Baseline (Claude): PASS · $0.019 · 7.7s
Attempts: 2 of 2

Bottom line // BR-008

If the deliverable is words for humans — drafts, explanations, narratives — audition it. If the deliverable is structured output, look one row up or down this bench.

Method notes

1
Two attempts per test. Best attempt is graded; both are logged.
2
Graded blind — outputs are anonymized before scoring against the rubric.
3
Costs are provider list price at test date, prompt + completion tokens.
4
Test data is ours — real invoices, a real codebase, real tickets. Swap in yours.

TANGIBLE IDEAS

Loading...

Please wait while we prepare your content.

Bench status · ActiveBR-008 // Louisville, KY // 2026-07-07

Bench report · BR-008Rank 08 / 10 on the benchTested 2026-06-23 → 07-02

nousresearch/hermes-4-70b

The best writer on the bench and one of its least reliable engineers. When the deliverable is words for humans it shines; when the deliverable is JSON, CSV, or a diff, it improvises.

VerdictCGreat prose, unreliable structure

Pass rate5 / 10runs

Avg cost / run$0.006list price

P50 latency8.2sacross runs

Vs baseline cost0.21×Claude

Code generation

FAIL

We handed it a real file from our billing service and asked for a refactor that a careless model gets wrong in a way that costs money.

Prompt · Run it yourself

[paste your own webhook/handler file here]

What a pass looks like

Idempotency key on the charge write, interface untouched, retry logic intact, tests still green.

What it did

A beautifully explained refactor that changed the public interface. Both attempts.

Cost: $0.007
Latency: 9.1s
Baseline (Claude): PASS · $0.034 · 12.1s
Attempts: 2 of 2

Extraction

PASS

Twelve scanned invoices, some smudged. The trap is in the last line: models love to guess.

Prompt · Run it yourself

Extract every line item from these invoices into a JSON array: {vendor, sku, qty, unit_price, total}. If any field is unreadable, set it to null — do not guess. Return only JSON.

[attach 10–12 of your own invoices or receipts]

What a pass looks like

Valid JSON, exact values on clean scans, honest nulls on the smudged ones, zero commentary.

What it did

12/12 on clean fields, honest nulls where it couldn’t read. Its one unqualified win.

Cost: $0.006
Latency: 7.4s
Baseline (Claude): PASS · $0.028 · 4.9s
Attempts: 1 of 2

Summarization

FAIL

A 40-page vendor contract. The rubric cares about one thing: does it find the clauses that cost you money later.

Prompt · Run it yourself

Summarize this contract in 300 words or less. Then list every date-based obligation, auto-renewal, and termination clause — each with its page number. If you are not sure of a page number, say so.

[paste a long contract or policy doc you actually use]

What a pass looks like

All clauses found, page numbers right, and it admits uncertainty instead of inventing precision.

What it did

The best-written summary on the bench — and it missed the renewal clause on p.31. Prose is not the rubric.

Cost: $0.006
Latency: 7.9s
Baseline (Claude): PASS · $0.031 · 9.4s
Attempts: 2 of 2

Agentic tool use

FAIL

Six steps, three tools, one approval gate. Most of the bench either skips the approval or loops.

Prompt · Run it yourself

[wire up to your own sandbox/tools — never production first]

What a pass looks like

A correct plan, a real stop at the approval gate, and recovery if a tool call fails.

What it did

Wrote an essay about the plan instead of executing it. Never called a single tool.

Cost: $0.005
Latency: 10.6s
Baseline (Claude): PASS · $0.041 · 26.2s
Attempts: 2 of 2

Classification

MIXED

500 real support tickets, 8 queues, human labels as ground truth. Pass bar is 90% agreement.

Prompt · Run it yourself

Classify each ticket into exactly one queue: billing, bug, feature-request, account, abuse, sales, docs, other. Reply as CSV, ticket_id,queue — one row per ticket, no header, no commentary.

[export a few hundred of your own tickets or emails]

What a pass looks like

≥ 90% agreement with human labels and a parseable CSV on the first try.

What it did

88% with commentary on attempt one; 90% flat on attempt two. Scraped by with edits.

Cost: $0.005
Latency: 8.0s
Baseline (Claude): PASS · $0.019 · 7.7s
Attempts: 2 of 2

Bottom line // BR-008

If the deliverable is words for humans — drafts, explanations, narratives — audition it. If the deliverable is structured output, look one row up or down this bench.

Method notes

1
Two attempts per test. Best attempt is graded; both are logged.
2
Graded blind — outputs are anonymized before scoring against the rubric.
3
Costs are provider list price at test date, prompt + completion tokens.
4
Test data is ours — real invoices, a real codebase, real tickets. Swap in yours.