Evidence-driven local routing.

Roles & incumbents

Every routing slot has one incumbent. Challengers must beat the incumbent on acceptance rate, review pain, and hard-failure rate without collapsing the role into a single training family.

Latest run

No archived runs yet. Check back after the next benchmark.

Promotion policy

A challenger may replace an incumbent only if it beats on: acceptance_rate (≥ incumbent), review_pain (≤ incumbent), hard_failure_rate (≤ incumbent), cost_tier (free unless incumbent is also paid), AND preserves family_diversity (promotion must NOT leave the role single-family).

Family diversity guard. If the proposed promotion would leave a role served by only one `family` value across incumbent + challengers, reject regardless of other metrics (memory #895).

After every benchmark run; summary in reports/local-model-benchmark-YYYY-MM-DD.md.

Methodology

• Cases come from real commits — not synthetic prompts. Each case in benchmarks/cases/*.json references a source commit so the task shape stays grounded.
• Automated checkers only. Every case has a programmatic verifier; nothing depends on a human re-scoring run-to-run. Pipelines that need human scoring decay.
• Severity weights. Silent-wrong (model returns confidently incorrect output) counts 3–5× a parse-fail in aggregate scoring — silent errors corrupt downstream work.
• Family diversity. A promotion that leaves a role single-family is rejected even if it wins on every other metric. Same-family agreement is correlated, not independent.
• Cost tier on every model. Free local models and paid cloud models are not interchangeable; the registry tracks this so promotion logic can prefer free tiers when quality is comparable.

Turbo escalation. When the review-role incumbent rejects on a 26b verify-reject on code-review or security-review task, the runner retries against gpt-oss:120b-cloud before falling back to a frontier-API call. Telemetry-tagged turbo-escalation.