Please wait while we prepare your content.
Every model, run against our actual work — not synthetic benchmarks. Same prompts, same rubric, real invoices and real codebases. Claude and GPT are baselines, not defaults.
| Run | Prompt | Task | Model | Cost | Latency | Result | Notes |
|---|---|---|---|---|---|---|---|
| BX-042 | Refactor billing webhook handler to idempotent writes | CODE GEN | qwen/qwen3-coder | $0.011 | 14.2s | PASS | Clean diff. Tests stayed green. |
| BX-041 | Refactor billing webhook handler to idempotent writes | CODE GEN | deepseek/deepseek-v4-flash | $0.002 | 6.8s | FAIL | Dropped the retry logic. Silently. |
| BX-040 | Extract line items from 12 scanned invoices to JSON | EXTRACTION | openai/gpt-5-mini | $0.004 | 3.1s | PASS | 11/12 exact. One quantity swap. |
| BX-039 | Extract line items from 12 scanned invoices to JSON | EXTRACTION | qwen/qwen3.7-plus | $0.003 | 4.6s | PASS | Matched baseline at a fraction of the cost. |
| BX-038 | Summarize 40-page vendor contract, flag renewal terms | SUMMARIZ. | anthropic/claude-sonnet-latest | $0.031 | 9.4s | PASS | Baseline. Caught the auto-renew clause. |
| BX-037 | Summarize 40-page vendor contract, flag renewal terms | SUMMARIZ. | nousresearch/hermes-4-70b | $0.006 | 7.9s | FAIL | Missed the renewal clause on p.31. |
| BX-036 | Plan and run a 6-step CRM cleanup via tool calls | AGENTIC | moonshotai/kimi-k2.7-code | $0.018 | 41.0s | PASS | One retry on step 3. Recovered. |
| BX-035 | Plan and run a 6-step CRM cleanup via tool calls | AGENTIC | openai/gpt-oss-120b | $0.005 | 28.4s | FAIL | Looped on step 4. Killed at 3 min. |
| BX-034 | Classify 500 support tickets into 8 queues | CLASSIF. | deepseek/deepseek-v4-pro | $0.009 | 12.0s | PASS | 96% agreement with human labels. |
| BX-033 | Classify 500 support tickets into 8 queues | CLASSIF. | qwen/qwen3-coder-flash | $0.001 | 5.2s | FAIL | 71%. Wrong tool for this job. |