Hermes Mixture-of-Agents: what we tested, what we found, when to use it

Branch feature/moa-openai-proxy on the public fork github.com/hughmadden/hermes-agent · started 2026-07-03, updated 2026-07-10 · ~330 benchmark turns on OpenRouter + Cerebras, plus local GPU measurements

0. Configuration reference — every tested config, models & roles

Roles: voter = answers the request directly, its answer is compared for consensus (cascade mode) · reference/advisor = writes advice the aggregator reads (classic MoA fan-out) · aggregator = the acting model that produces the final answer (in cascade: the tier-1 arbiter on disagreement) · classifier = routes moa:auto requests (~300 ms) · judge = one-word consistency check for freeform consensus · verifier = writes a python check script (sandboxed exec) · escalation lane = stronger preset a conversation/cascade moves to on observed failure/discord. Hosts: @cere = Cerebras wafer API (provider-served precision) · @OR = OpenRouter (provider-default precision) · @plan = openai-codex OAuth (ChatGPT plan, $0 marginal) · @5090 = local vLLM on the RTX 5090. Defaults unless stated: reference_max_tokens 8000 (cascade voters) / 1500 (classic fan-out), quorum straggler-dropping on where noted, save_traces off in benches.

0.1 Cascade family (mode: cascade — voting by exact-answer consensus)

0.2 Routed / escalation / omni configs (moa:auto)

0.3 Classic MoA fan-out combos (references advise → aggregator acts)

0.4 Solos, learning loop & misc

Full machine-readable configs: the serve YAMLs and bench CONFIG dicts in the fork branch (scripts/moa_*_bench.py, docs/plans/moa-routed-config-example.yaml, docs/plans/moa-cascade-spec.md); every results table's config name matches a row above.

2. Architecture — how each mechanism works

The proxy (hermes moa serve). An OpenAI-compatible /v1/chat/completions server where every configured preset is a model id (moa:<preset>). The CLIENT owns tools and turn termination: the proxy forwards the client's tool definitions to the acting model and streams tool_calls back — so any agent harness (aider, mini-swe-agent, Harbor/terminus-2, OpenClaw, Claude Code via LiteLLM) can use these compositions as "a model" with zero integration work. Reference/voter thinking streams as reasoning deltas; usage.moa itemizes every upstream call. Hard per-slot timeouts (slot_timeout_s, 300 s) after real wedges. Provider slots resolve through the standard Hermes chain, which is how Cerebras (custom provider), local vLLM (custom provider), OpenRouter, and plan-OAuth models (openai-codex; anthropic once logged in) all mix freely in one preset.

Classic fan-out (mode: fanout, the original MoA). Every reference slot receives an advisory view of the conversation (with an advisor system prompt and truncated tool results), writes advice in parallel; the aggregator then acts with that advice appended as a guidance block. Measured: lifts hard reasoning +4–7 pp over its own aggregator solo; HURTS precise code editing (−13 pp) and never helps agentic coding. Latency = slowest reference → mitigations: reference_max_tokens: 600 (−17% latency, free) and quorum straggler-dropping (reference_quorum_grace: once all but one reference finish at elapsed T, the straggler gets T×grace then is dropped with a note — no accuracy cost measured, 60/60).

Cascade (mode: cascade — lazy MoA, the flagship). Voters (the reference slots) answer the request DIRECTLY (no advisory framing) in parallel at tier 0; a candidate answer is extracted from each (ANSWER-line → balanced-\boxed{} → short last line; failure boilerplate can never vote). If ≥ min_consensus normalized candidates agree, the first agreeing voter's full text returns immediately — no aggregation at all (~1.4–6 s, and consensus precision measured 96–100%; PERFECT 39/39 with a competent-diverse pool). On disagreement, tier 1: the aggregator acts, normally seeing the voters' work — or, with clean_arbiter: true, re-solving from scratch (measured: voter context ANCHORS even frontier arbiters — 7/9 vs 98% solo). If the tier-1 answer agrees with no voter and escalate_to names a preset, tier 2 escalates to that preset's aggregator. Optional gates: gate: judge (a 300 ms wafer consistency check extends tier-0 to freeform — 2× speed, but costs blind-graded polish 2/2/23, latency-first lanes only) and verify: python (a verifier slot writes a check script, executed sandboxed — python -I, empty env, tempdir, 12 s, non-zero exit distrusted; WRONG strikes the consensus or forces escalation; only pays when checks are computational). Per-slot max_tokens overrides let thinking-heavy voters finish (an 8 k cap silenced a 32B voter's candidates entirely). Production semantics (v1.4): streaming tool-free requests run the true cascade (voter progress as reasoning deltas; a tier-0 winner arrives as one content delta — 1.3 s measured live; tier-1 streams token-by-token unless an escalation tier must inspect the answer first); tool-carrying requests run the acting slot SOLO with tools forwarded (no voter overhead — advisory context measurably hurts tool work); requests estimated over cascade.max_context_tokens (default 100k) bypass the ~128k wafer voters to the acting slot ("context-solo") instead of erroring — the long-context ladder being cascade <100k → plan-GPT-5.5 to ~400k at $0 → 1M-class open lanes on explicit routes.

Router (moa:auto). A wafer classifier (gemma-4-31b@cere: 100% accuracy, p50 288 ms) reads the request tail and picks a route-described preset; SELF class answers trivia directly. Decisions are sticky per conversation (session header or first-user-message hash) so tool loops never re-classify. Failure-gated escalation: when a sticky conversation's latest client/tool message carries failure markers, it re-routes one tier up (escalation.tiers), gated by min_failures (debugging prints tracebacks as normal work — first-failure escalation over-fired on 80% of SWE conversations; min_failures=3 cut frontier turns 33%→18% at identical resolution). Disabled presets with a route block are routable solo lanes — how coding traffic bypasses the fan-out penalty.

Learning (hermes moa evolve). With save_traces on, every fan-out turn records the full advisory inputs/outputs plus optional externally-graded outcomes and the routing decision. Evolve replays recent traces through a distiller LLM that REWRITES a bounded (≤4 KB) aggregation-heuristics skill, injected into every subsequent aggregator prompt; --per-preset keeps lane heuristics separate. Measured on a weak fleet: +8.3 pp held-out across 3 cycles, zero answer leakage; cost is +40–70% latency from the verification habits it teaches.

Plan integration. Mounting the live ~/.hermes/auth.json (single shared store — never copy it; OAuth refresh rotation would split-brain) into the serve container lets slots use provider: openai-codex → GPT-5.5 at $0 marginal on the ChatGPT plan. The same path unlocks plan-Fable/Opus once hermes auth login anthropic has been run.

Test surface: 82+ hermetic MoA tests + 9 live integration tests; slim docker release (docker-compose.moa.yml) serves a fresh clone in ~3 commands.

2.x The serving layer (iteration 44) — what real sessions add

Benchmarks send one short problem; real agents send a growing conversation with tool calls and provider-opaque fields. Three mechanisms close the gap: (1) history projection — a pure, per-message transform renders tool_calls/tool results/content blocks as plain text and strips opaque provider fields before any voter sees the history (per-message ⇒ append-only ⇒ upstream prefix caches survive); (2) voter context windows (cascade.voter_context_tokens) — voters see a recency window while acting lanes keep the full verbatim transcript, bounding the fan-out token bill that otherwise multiplies the session length 5–6× per turn (and trips provider TPM quotas that count even cached tokens); (3) a session registry — TTL+LRU, keyed by header or first-user-message hash, providing a stable prompt_cache_key for OpenAI-family acting lanes and session observability (usage.moa.session = turns + last served mode; usage.moa.voter_view = window stats; prompt_tokens_details.cached_tokens on every usage object). Robustness rule learned the same day: degraded_consensus — a quota burst that kills half the voter pool no longer forces acting-solo when the survivors agree (min_consensus degrades to the live count, floor 2).

3. Scorecard — every test we ran

* 11 of the 550B model's 16 misses were empty/invalid responses from its OpenRouter provider — frontier-open serving is itself still flaky, which is a reliability argument for compositions that degrade gracefully (a failed reference becomes a note; a failed monolith is a failed request).

4. Primary quality axis — AIME 2024+2025 (hard benchmark)

The original 40-task set saturated at kimi-class (three configs 40/40), so AIME 24+25 (60 competition problems, exact integer answers) replaced it as the primary quality axis. Same code path, same configs:

What the hard benchmark changes: (1) composition does lift hard reasoning — +7 pp for a Cerebras-merged heavy fan-out over kimi solo, +4 pp for mixed-heavy, and every composition beats its own parts (the earlier "no gain" was a ceiling artifact of the saturated 40-task set); (2) all-Cerebras MoA delivers 90% AIME at 19 s/task — gpu4-composed accuracy at 12× lower latency, the practical open fast lane; (3) frontier saturates AIME (Fable perfect), so open-tier improvements are where the measurement signal lives; (4) small-model MoA (nano) collapses on hard problems (32%) — composition cannot rescue models far below the task.

4b. POST-CUTOFF REVALIDATION — AIME 2026 + HMMT Feb 2026 (the numbers that count)

Both competitions were administered February 2026 — after the training cutoff of every model in the pool. 50 gradable problems, identical harness. This is the contamination-resistant scoreboard:

Three findings. (1) Contamination was real: frontier solos dropped 1–10 pp on fresh problems (Fable 100→97/90, GPT-5.5 98/95→97/85) — the old-set ceilings flattered them. (2) The architecture generalizes: cascade-rlm matched its old-set numbers on problems no model had seen — tying both frontier solos on AIME 2026 and BEATING them on HMMT Feb 2026 (+10 pp over GPT-5.5, +5 pp over Fable) at 6–8× lower latency. Consensus among diverse fast agents is measuring problem difficulty, not memorization. (3) Champion swap: the always-frontier clean arbiter (rlm-conv) fell behind — its old-set edge was partly the arbiter's contamination; observation-gated escalation (cascade-rlm: wafer aggregator first, frontier only on discord) is the better architecture on genuinely novel problems, and is now the default on the stable service and the turbo profile. All older sections retain AIME 24+25 / HMMT 2025 numbers — read those as contamination-suspect upper bounds for the frontier solos, and as validated-by-revalidation for the cascade family.

5.1 The cascade program — flagship: lazy MoA with consensus gating

Built 2026-07-04 (ultracode batch): the campaign's one law is that predicted difficulty fails and observed signals win — so mode: cascade gates on the cheapest observation there is: answer agreement between wafer-speed voters. Tier 0: Cerebras gpt-oss-120b + gemma-4-31b answer directly in parallel (~1.5 s); unanimity returns instantly. Tier 1: on disagreement the aggregator synthesizes from the voter outputs already paid for. Tier 2: when the aggregator agrees with no voter (observed discord = genuine hardness), escalate to plan-GPT-5.5 ($0). Implementation farmed to parallel Sonnet builders with adversarial review (3 confirmed findings fixed: nesting-aware boxed extraction, failure-boilerplate votes, trace attribution); 98 tests green.

Why this is the product: it strictly dominates its own always-on MoA (better accuracy on both sets, half the median latency, no aggregator cost on easy traffic), reaches within 2–5 pp of a frontier solo while touching the frontier on ≤15% of requests, and every tier below the top runs on wafer/local-class models — this is the 4×96 GB box's serving architecture, live today: mode: cascade in one preset. Smoke: 17×23 answered end-to-end in 1.16 s. The cascade self-adapts to difficulty: easy sets ride tier 0, hard sets shift to tiers 1–2 — no classifier, no prediction, just observation.

5.2 The frontier dial — measured

Consensus strictness IS the quality/cost dial. HMMT Feb 2025 (hard set), same wafer voters throughout:

The strict-unanimity cascade matched Fable and edged GPT-5.5 solo on the hard set with TWO frontier calls (both $0 on the plan) — stricter consensus routes marginal cases into tier-1 aggregation, which went 12/12 here. AIME-60 confirmation (v19) sharpened this honestly: mc4 scored 88% on AIME — BELOW mc3's 93% — because strict unanimity demoted 22 problems to tier-1, whose aggregator went 16/22, worse than the 95.8%-precise weak consensus it replaced. The dial is task-relative: where voters are competent (AIME), trust weak consensus; where stretched (HMMT), strictness pays. The two cross-set constants: unanimous tier-0 is essentially perfect everywhere measured (33/33, 20/20), and tier-2 is near-perfect (4/4, 3/3, 7/8) — the tier-1 aggregator (glm-4.7) is the weak link and the next upgrade target.

5.3 Cascade family — complete measured map

The measured laws of the cascade family: (1) it sits at 92±1.5% AIME across every knob — the knobs trade latency (3.5–10 s median) and frontier fraction (0–15%), not accuracy; (2) consensus precision is the engine (unanimous tier-0: 53/53 cross-set; 3-of-4: ~96%); (3) the residue is bounded by ~4% false consensus plus frontier-hard items; and (4) — the deepest finding — voter context anchors even a frontier arbiter: GPT-5.5 judging disagreements scored 7/9 where GPT-5.5 solo runs 98% — the same contamination measured in code editing, now in hard reasoning. Iteration 25 tested clean escalation: the clean frontier arbiter improved the disagreement subset directionally (78% → 86%, small n) — anchoring is real — but the total held at 92% because stochastic false consensus (~4–5%, different problems each run) is the family's true ceiling. Seven AIME-60 variants land at 92.1% mean. The remaining lever is voter DIVERSITY, not architecture: Cerebras offers three model families; adding independent families (e.g. local vLLM voters on the 5090/4090 — exactly the 4×96 GB box thesis) attacks the correlated-error term directly. Iteration 26 tested it live: a LOCAL Qwen3-8B voter (vLLM on the RTX 5090) joined the Cerebras voters in one consensus pool — the hybrid ran flawlessly (60/60 turns, 0 errors, hybrid tier-0 consensus at ~5 s) but scored 88%: the weak voter's dissent was noise, inflating tier-1 volume without cutting false consensus. The closing law: diversity value = independence × competence. An 8B is below the bar; the 4×96 GB box thesis holds precisely because 96 GB cards run 27B–120B-class voters (qwen3.6-27b measured 80% HMMT) that can be right when the wafer models collude wrong.

5.4 CONVERGENCE RESULTS — 95% with local silicon; 97% pure-cloud (the record)

Update (iterations 31–32): composing the RLM discovery into the convergence pattern beat the local-GPU result: cascade-rlm-conv (RLM wafer voters + clean $0 frontier arbitration) scored 58/60 (97%) at 3.9 s median — tier-0 handled 83% of traffic at 98% precision (49/50 @ 3.0 s), the clean arbiter took the tail 9/10, frontier fraction 17% at $0, zero errors, zero local hardware. The local-voter convergence below remains the proof that on-box silicon slots in natively (and the best result when plan quota is scarce — its tier-0 was perfect).

Iteration 27 composed every measured law into one preset: competent-diverse voters (2× Cerebras gpt-oss-120b + gemma-4-31b + Qwen3-32B-AWQ running on the local RTX 5090) + clean plan-frontier arbitration (GPT-5.5 re-solves every disagreement from scratch — anchoring guard):

Why this one matters: for the first time the number moved because of design, not noise — tier-0 became perfect BECAUSE the diversity law (independence × competence) fixed false consensus, and tier-1 lifted BECAUSE the anchoring law demanded clean arbitration. 95% vs GPT-5.5 solo's 98%, built from Hugh's own GPU + wafer inference + plan-covered frontier on a third of requests. This is the 4×96 GB box architecture, measured end-to-end with real local silicon in the pool.

The completed frontier curve (iteration 28 closed it): 93% @ 5% frontier (cascade-wafer4) → 95% @ 35% frontier (cascade-convergence — the optimum) → 98% @ ~100% (unanimity-of-4 degenerated to frontier-solo: the thinking-heavy local voter often exhausted its 8000-token cap before emitting a candidate, making 4-of-4 unreachable — mechanism lesson: unanimity dials require every voter to reliably emit candidates; thinking voters need caps ≥ their reasoning budget). Pick your point on the curve per lane; all frontier usage rides the $0 plan.

6. Frontier / wafer / local mixes — HMMT Feb 2025

New batch (2026-07-04): can frontier+wafer+plan mixtures be "fast and smart", and how far do locally-runnable-class stacks go? HMMT Feb 2025 (gradable 20-problem subset — integers/simple fractions), quorum on. GPT-5.5 rides the ChatGPT plan through the proxy (live hermes auth mounted, zero marginal cost); Opus/Fable still via OpenRouter (anthropic OAuth isn't provisioned in hermes yet — one `hermes auth login anthropic` would unlock plan-Fable).

Batch findings so far: (1) plan-integrated GPT-5.5 through the proxy is the single best value config for hard reasoning; (2) frontier councils can't be evaluated on competition math any more — Fable saturates AIME and HMMT; their test needs long-horizon agentic work; (3) the local tier repeats the composition story (+5 pp) but pays minutes of latency — the practical local sweet spot remains cere-moa-style wafer serving unless quality is paramount; (4) qwen3.6-27b is the strongest small local model we've measured.

7. moa:omni — the product, live

Every measured optimization assembled behind ONE model id (moa:auto): the 288 ms Cerebras classifier routes each request to its measured-best lane — checkable problems → consensus cascade (93% AIME @ 4.4 s median, 5% frontier), code → kimi solo with plan-frontier escalation on observed failure, chat → wafer MoA, trivia → wafer self-answer. Mixed-traffic showcase (10 prompts across all four classes): 10/10 routed correctly, 2.6 s median latency — math answered by tier-0 consensus in 0.9–2.4 s, trivia in 0.7 s. Config shipped as the example in the branch.

Verified cascade (iteration 19) — honest verdict: adding wafer-generated python verification was NEUTRAL on AIME (92% vs 93%): the two WRONG verdicts correctly forced tier-2 escalations, but false consensus survived because a 31B verifier cannot independently check what 120B voters got wrong — verification pays exactly when the check is pure computation (the earlier +5 pp), not when it needs the missing insight. The honest frame: the wafer cascade ceilings at ~93% AIME, and above that you turn the frontier dial (escalation rate) — 5% frontier → 93%, 100% frontier → 98% — rather than expecting free accuracy from tricks.

8. MoA pros & cons — updated for the cascade era

9. Detailed results (aider / SWE-bench / Terminal-Bench / learning / multi-GPU)

Reasoning quality (40 verified exact-answer tasks)

Code editing (aider polyglot, 30 Python, whole format)

Costs from measured tokens × live OpenRouter prices. The MoA pass@1 penalty (−13 pp vs its own aggregator) replicated on both combos → routing now supports solo lanes so coding traffic can bypass the fan-out.

Agentic benchmarks

Context: the official bash-only SWE-bench Verified leaderboard tops out around 76.8% (Claude 4.5 Opus) — different subset, so not directly comparable, but 76% from an open solo through our proxy is squarely competitive. (See section 17 for the iteration-46 clean head-to-head that supersedes these early runs.)

Learning loop (evolve), paired held-out evals

Zero leakage in all distilled skills (heuristics + per-model trust notes only). Residual misses are tool-shaped (exact big-number arithmetic) — see roadmap.

Multi-GPU throughput (RTX 5090 + 4090, vLLM, Qwen3-8B)

Full 4×96 GB plan (hardware dossier, model fits, buy verdict): docs/plans/multi-gpu-moa-plan.md in the branch.

10. Quality-improvement loop — every iteration & verdict

Hypothesize → test → report, one iteration at a time:

How failure was detected — read before citing the escalation numbers. The offline composites are oracle-detected upper bounds: the aider 30/30 used the benchmark's gold tests to pick what to escalate, and the SWE-bench 25/25 used SWE-bench's hidden evaluation tests — signals a runtime agent cannot see. The live escalation run is the real-world-shaped result: the router never detects failure itself; it pattern-matches failure evidence (FAILED / AssertionError / Traceback) in the conversation, and that evidence exists only because the client agent ran tests itself and fed the output back — which coding agents (aider auto-test, mini-swe-agent, CI loops) do as part of their normal loop. Boundary conditions: tasks with no verifiable signal never escalate; agent-visible tests can be weaker than gold tests (aider polyglot happens to ship its gold tests to the agent, which flatters the live number); a true live SWE composite would land between kimi's 76% and the oracle 100%.

Loop takeaway: routing + verification-gated escalation is how you get frontier-class results at a fraction of the frontier calls — guess-free (escalate on measured failure, not predicted difficulty). This slots directly into agentic harnesses that already run tests (aider, SWE-bench, CI).

11. Roadmap — open experiments

The vision: a dockerized router/MoA of cheap open-weight models that matches frontier quality where it matters, improves from its own traffic, and runs anywhere (API slots today, a 4×96 GB box tomorrow). The gaps and the experiments that close them:

12. How to replicate

# Code (public fork, branch feature/moa-openai-proxy)
git clone -b feature/moa-openai-proxy https://github.com/hughmadden/hermes-agent.git
cd hermes-agent

# Serve MoA as an OpenAI endpoint
mkdir moa-home   # write moa-home/config.yaml — presets + router (docs/moa-openai-endpoint.md)
export OPENROUTER_API_KEY=... CEREBRAS_API_KEY=...
docker compose -f docker-compose.moa.yml up -d
curl -s localhost:8646/v1/models | jq -r '.data[].id'

# Tests + benches (all dockerized; raw JSONs for every table in docs/plans/)
tests/integration/docker/run-moa-proxy-tests.sh
docker build -f tests/integration/docker/Dockerfile.moa-proxy -t hermes-moa-proxy-test .
D="docker run --rm -v $PWD/scripts:/app/scripts:ro -e OPENROUTER_API_KEY -e CEREBRAS_API_KEY hermes-moa-proxy-test"
$D python scripts/moa_router_bench.py --classifiers "custom:cerebras=gemma-4-31b"
$D python scripts/moa_learning_cycle.py --cycles 3 --config open-moa-nano --reset-learning
$D python scripts/moa_gpusim_bench.py
$D python scripts/moa_cerebras_bench.py

# Public harnesses against the endpoint (details: docs/plans/moa-public-bench-results-20260703.md)
# aider:    OPENAI_API_BASE=http://localhost:8646/v1  --model openai/moa:<preset>
# harbor:   harbor run -d [email protected] -a terminus-2 -m openai/moa:<preset> \
#             --ak api_base=http://localhost:8646/v1 --timeout-multiplier 2
# swebench: mini-extra swebench --model openai/moa:<preset> --subset lite --split test \
#             --slice 0:25 -c <default swebench.yaml> -c environment.timeout=300

15. 2× RTX 6000 Blackwell topology study — one big deployment vs two instances

Instrument: Qwen3-Next-80B-A3B-FP8 (262k native context, hybrid attention ≈2.3 GiB KV per 100k tokens), vLLM, prefix caching on, fp8 KV, workstation Blackwells with NO NVLink (PCIe only; TP=2 requires --disable-custom-all-reduce). No public benchmark of this comparison existed.

Verdict: for any model that fits a single 96 GB card, two independent instances dominate every axis — throughput, latency, and caching. TP=2 over PCIe on these cards is purely a FIT mechanism for weights >96 GB (tonight: GLM-4.7-IQ3 at ~165 GB), never a performance choice. Agentic caching verdict: GPU-resident prefix caching is near-total — a 180k-token session turn answers in 0.38 s warm (vs 35.3 s cold); long agent sessions are essentially free after the first turn, with zero system-RAM requirement. The 262k-context 80B endpoint now runs permanently on bench GPU 1 as a proxy provider.

14. Serving-cache economics — measured (agentic sessions)

An agent session re-sends a large stable prefix (system + history) with a small delta each turn. Whether that prefix is CACHED at the provider decides both latency and cost. Measured 2026-07-06 with a 30k-token synthetic agent prefix, 4 sequential calls per model (scripts/moa_cache_bench.py + moa_prefix_cache_probe.py):

Design consequences for the proxy: (1) cache-critical agent lanes (long sessions, escalation tiers) should pin deepseek/glm-class providers — 40% real input-cost cuts at 92-99% hit rates; (2) OR usage carries prompt_tokens_details.cached_tokens — never the cache_discount field; (3) prompt layout matters: static content first, volatile fields (timestamps, ids) LAST or hit rates collapse; (4) the wafer voter tier needs no caching design at all — its prefill is faster cache-cold than most providers are cache-warm.

16. Real-world serving proof — iteration 44

Measured today: 2026-07-08. Session-serving tests used scripts/moa_session_sim.py against the proxy.

Every prior number in this report came from one-shot short problems. Real agent sessions carry a growing 10k–200k-token history, resend their tool schemas every request, run multi-round tool threads, and live or die on provider caching and quotas. scripts/moa_session_sim.py drives exactly that shape against the proxy and records per-turn mode, latency, cached tokens, and errors.

Finding A — production bug: context cap

Finding B — quota wall

Fixes shipped

Committed. 220 tests green.

Result run — cascade-live

Iteration 45 — local long-context lane + advisor

With the bench 262k Qwen3-Next-80B endpoint as the acting/aggregator lane (cascade-live-local: windowed wafer gate + local acting), a 120k-token tool-thread session ran 9/9 turns, 2/2 tool threads, 2/2 post-thread cascade reverts, tier-0 @0.57–0.82 s — no provider quota exists on local silicon, and prefix caching makes the giant transcript cheap after turn one. This is the structural endgame the topology study pointed at for latency and quota — but measured on a session simulator, not a live agent. Iteration 46 found the caveat: a single 80B on one GPU is too slow for real agentic loops (long-context acting calls time out and hang; cascade-live-local resolved only 7/17 on SWE-bench, stuck in file-edit loops). The endgame needs faster or more local silicon; today the reliable agentic acting lanes are cloud (kimi / plan-GPT-5.5).

The advisor lane (spec v1.6 §B) replicates oh-my-pi's advisor role with two measured-law deviations: it runs async, after the turn returns (zero added latency, unlike OMP's inline review) and is concern-gated (OK verdicts are dropped — the anchoring law forbids always-on advisory injection). Live-verified: a session steering toward a destructive rm -rf / drew a BLOCKER note that surfaced on the very next turn; in escalate mode a BLOCKER also forces that turn past tier-0 to the full aggregator's scrutiny.

A cascade that wins benchmarks is not yet a cascade that serves sessions - the serving layer is its own engineering problem, now measured and largely solved.

Dogfood provenance: this section's first draft was written by the proxy itself (stable service, cascade tier-2, 60.8 s wall, styling-level edits only + the degraded-consensus validation result added after its run). The results-evolution table for these runs was computed and written by a hermes agent session running ON the serving preset (4-file tool task, 61.8 s, output verified correct).

17. Real-world agentic head-to-head — is the proxy worth using over a solo API? (iteration 46)

Iterations 1–44 asked can a cascade win benchmarks. This one asks the only question that decides adoption: on a real coding agent doing a long tool-using loop, is the proxy smart enough, fast enough, and cheap enough to use instead of calling GPT-5.5 or Claude directly? Harness: mini-swe-agent 2.4.4 on SWE-bench Lite (25 instances, docker-socket eval), each arm a genuine multi-turn agentic session (median 12–14 tool calls/instance), driven through the serving worker.

Finding 17a — the proxy is exactly as smart as its acting model, no more. cascade-live scored bit-identically to GPT-5.5 solo (20/25) because on a tool turn the cascade routes straight to its acting lane, which is GPT-5.5. For agentic coding the wafer-consensus short-circuit never fires (there is no fresh single-shot question to vote on), so the cascade only adds voter-token cost and two median steps of gate latency. Do not serve cascade-live for pure coding — you pay more for the model you already have. The MoA machinery earns its keep on single-shot reasoning (sections 4–6), not tool loops.

Finding 17b — moa:auto answers the cost question YES. The router sends coding to cheap kimi-k2.6 and escalates to GPT-5.5 only after two consecutive failures. Across the 25-instance run: 627 requests served by kimi, 172 by the GPT-5.5 escalation tier (21 escalation events) — ~79% of all work on the cheap lane. Result: 80% resolved, identical to all-frontier GPT-5.5 solo, for ~$2.5 of OpenRouter spend plus escalations that ride the $0 ChatGPT plan. Even if every frontier escalation were billed at GPT-5.5 API rates, only ~21% of requests touch frontier — a 4–5× cost reduction at equal resolve rate. This is the config a cost-conscious agent should point at: the accuracy of frontier, most of the bill of a wafer.

Finding 17c — two serving bugs that only real agentic load exposes. (1) Reasoning-model replay. kimi-k2.6 emits reasoning_content + provider_specific_fields on its own output; a real harness echoes those back in history and a strict endpoint rejects the acting call with wrong_api_format. Fixed with a per-message acting-lane sanitizer that strips response-only annotations while preserving real tool_calls/ids and Anthropic signatures — a hard prerequisite for using any reasoning open model as an acting lane. (2) Router degradation. moa:auto with no routable presets (or a wafer-acting default) silently sends every agentic turn onto a rate-limited Cerebras lane and 429-storms under concurrency; the fix is to route coding to a non-wafer lane and keep the classifier off the wafer. Neither bug is visible to a single-request benchmark; both are fatal to a real tool loop.

Finding 17d — the advisor: our gated design is safe; OMP's inline-always actively harms a competent actor. The advisor role Hugh asked us to replicate from oh-my-pi reads every turn and can flag the agent. We built three shapes (our async + concern-gated; OMP-faithful synchronous inline; and a Cerebras-RLM advisor that executes a check before judging) and ran the clean head-to-head with a weak-but-competent coding actor — qwen3-coder-30b — against a GPT-5.5 advisor. (The first attempt used a local q80 actor, but an 80B on one GPU is too slow on long agentic contexts: acting calls time out and the run hangs. A fast coding-tuned cloud actor completed all 75 instances with zero hangs, for ~$3.50.) The mechanism is validated — the frontier advisor writes genuinely sharp diagnoses (verbatim: “agent is stuck in a loop of failing bash commands due to improper quoting”). But the clean numbers say something sharper than “the advisor helps”:

The inline advisor broke the actor's tool-call formatting on 40% of instances. OMP's shape appends the advisor's note as a trailing system message before every acting turn; qwen3-coder responded with 10/25 RepeatedFormatError aborts (vs 1/25 baseline) — the constant mid-conversation injection corrupts structured output, dragging resolve from 32% to 20%. Our async-gated design injects the same GPT-5.5 advice, deferred and only when a concern fires, and produced zero format failures (25/25 submitted) at baseline resolve. So on a procedurally-competent actor the advisor is neutral-at-best on quality (it adds procedure, not capability — this actor's ceiling is capability, and advice can't raise it), and the injection mechanism is where the real risk lives: inline-always is a gamble that backfired, gated-async is safe. Combined with the earlier signal that a mechanically-flailing actor (looping on file edits) does benefit from steering, the law: an advisor rescues actors that fail on procedure, is dead weight for actors that fail on capability, and inline-always injection can corrupt a capable actor outright — so gate and defer by default, and reserve always-on only for an actor you have measured to be flailing. This vindicates the async + concern-gated design over OMP-faithful inline as the safe default.

Bottom line for adoption. Over a solo frontier API for long agentic coding: the proxy is not smarter (it is its acting model), and for a single frontier lane it is a thin cost. But moa:auto is the reason to run it — frontier-equal task success at a quarter to a fifth of the cost, by doing the routine turns on a cheap model and spending frontier tokens only where the cheap model provably stalls. The honest scorecard: use solo Fable/Claude when you want the top resolve rate and cost is no object; use moa:auto when you want ~frontier accuracy at open-model cost; skip cascade-live for coding entirely.

18. Frontier advisor & the smartest possible panel — do they beat a solo actor? (iteration 47)

Iteration 46 tested the advisor with a cheap actor and a wafer panel. Iteration 47 pushes both questions to the ceiling with frontier models on every role, and answers them cleanly — in the negative. Two experiments, same harness (mini-swe-agent 2.4.4, SWE-bench Lite, 25 instances, docker-socket eval, median 13 tool calls/instance), driven through an Opus-enabled serving worker (Opus 4.8 runs on the Claude plan via its OAuth; GPT-5.5 on the ChatGPT plan; Grok 4.5 via OpenRouter).

Set A — a frontier ADVISOR (GPT-5.5) over cheap-but-good actors vs the same models with no advisor. Three cheap actors, each briefed every turn by a GPT-5.5 advisor, against a no-advisor panel of the same three as cascade voters with a GPT-5.5 arbiter:

Finding 18a — inline-advisor corruption is actor-specific, and it hit the strongest actor hardest. The identical GPT-5.5 inline note was harmless to flash and glm (0 format failures each) but broke deepseek-v4-pro on 17 of 25 instances (RepeatedFormatError, 68%), collapsing it to 7/25. This extends the iteration-46 law: you cannot predict corruption from actor strength — the most capable actor here was the most fragile to trailing-system-message injection, so per-actor measurement is mandatory. And even where the advisor did no harm (flash, glm at 18), it produced no lift: the winning Set A config (sa-best, 19) carries no advisor at all. Advising a coding actor is, at the frontier, zero upside plus model-dependent catastrophic downside.

Set B — the smartest MoA we can build: three frontier models in one config. GPT-5.5 acts (produces the tool call every turn); Opus 4.8 and Grok 4.5 run as windowed voters informing it (cascade, min_consensus 2). This is the strongest possible voter panel — not wafer models, but two of the best frontier models advising the third.

Finding 18b — adding the two best frontier voters to a frontier actor made it worse, not better. sb-smartest scored 18/25 with zero format failures and zero errors — below solo GPT-5.5 (20) and well below solo Fable (24), for triple the per-turn frontier cost (Opus 4.8 and Grok 4.5 each called ~349 times as voters). The panel design is safe where the inline advisor was not (votes inform the aggregator; nothing is injected into the acting model's message stream, so GPT-5.5's tool format stays intact), but safety is not value: the voter opinions either anchor the acting model toward wrong consensus or simply add nothing a strong actor didn't already have, and net cost it two resolves. This is the sharpest possible confirmation of the iteration-46 law (17a): for agentic coding the proxy is exactly as smart as its acting model — and stacking even frontier voters on top is dead weight to negative. Diversity and voting earn their keep on single-shot reasoning (sections 4–6), never on tool loops.

Bottom line (iteration 47). Nothing built from advising or voting beat a solo actor for agentic coding. The whole-campaign verdict for coding stands and hardens: use Fable 5 solo for top resolve; use moa:auto when you want ~frontier resolve at a quarter to a fifth of the cost (cheap actor + escalate on observed failure); do not add advisors or voter panels to a coding actor — at best they cost you nothing, at worst they corrupt or anchor the actor below its solo score.

19. Verification-gated delegation — can an observable test gate let a cheap worker offload the frontier? (iteration 48)

Every prior coding result said the same thing: opinion-based MoA (voting, advising) never beats a solo actor. This iteration tests the one shape with headroom — verification-based delegation, where the accept decision rides on observed test results, not a model's opinion. The design (Hugh's): a strong model writes a reproduction test from the issue text; a cheap worker attempts the fix; the worker's patch is run against that test; pass → keep the cheap patch, fail → escalate to the frontier. It is moa:auto's “observe, don't predict” law with a sharper signal: a real test instead of the agent's own stuck-detection. Stage 1a, SWE-bench Lite (25), test-author = GPT-5.5@plan, cheap worker = deepseek-v4-flash@OR, escalation coder = Fable 5.

The validity wall. The test author sees ONLY the issue text — never the hidden FAIL_TO_PASS grading tests (that would be leakage; it is what the iteration-46 “oracle 25/25” bound used, and it is not deployable). The gold patch is used solely to calibrate whether a gate is a valid instrument (does the repro fail on the bug and pass on the fix?), offline, never in the runtime accept path.

Finding 19a — the linchpin holds: a strong model CAN author a valid observable gate from an issue, 22/25 (88%). GPT-5.5 wrote a reproduction test that correctly failed-on-bug and passed-on-fix for 22 of 25 instances (20 first-shot; up to 2 self-repair rounds on collection errors). A framework gotcha decided it: Django's SWE-bench image ships no pytest (only its own runtests.py) — a naive pytest gate mis-scored every Django instance; stdlib unittest scripts run everywhere and fixed it. This is the first measurement that the gate’s author step — the linchpin nobody had tested — is feasible.

Finding 19b — high offload, but the gate is a lossy filter (68% precision), and that precision is the ceiling.

The gate accepted 19/25 flash patches (76% offload) and escalated only 6 to Fable — who resolved all 6. That lifts the cheap worker from 15 to 19/25. But of the 19 accepted, only 13 truly resolve: gate precision is 13/19 = 68%. The other 6 are false accepts — flash patches that pass GPT-5.5’s repro test yet fail the hidden FAIL_TO_PASS — and they are locked in as the cheap patch, never escalated. Those 6 are the entire gap to Fable: with a perfect gate they would escalate and Fable resolves 5 of them (it misses 1 itself) → 24. So the observable gate’s precision literally is the quality ceiling. A one-shot repro test written from issue text is narrower than the real acceptance criteria about a third of the time.

Finding 19c — a real cost-shifting lever, not a free lunch; the economics depend on the frontier’s price. Verify-escalate does 76% of the coding on a $0.14/$0.28 model and touches the frontier coder on just 6/25 instances — a genuine offload. But it does not reach frontier quality (19 vs 24), and it adds real overhead (a cheap session on ALL 25 + gate authoring on 22 + a container test-run per candidate). On a paid frontier that trade — give up ~5 instances to cut frontier-coder calls ~76% — may be worth it; on the $0 plan (Fable free) it is not, you lose 5 resolves for no dollar saving. The mechanism works; whether it pays is a function of frontier price and of gate quality.

Caching (measured, per standing directive). The flash lane (deepseek@OR) cached 0% on turns 1–3 then 96% (8,192 tok) once the prefix stabilised — OpenRouter’s multi-backend routing makes caching intermittent, so a cheap worker’s long-session cost is not reliably cut. The Anthropic plan lane returns no token/cache figures through the proxy (usage.moa shows 0) — an instrumentation gap (Anthropic reports cache_read_input_tokens, not prompt_tokens_details.cached_tokens), the same reason the codex plan lane has always shown “0 cached.” A gap to close, not proof of no caching.

Two infrastructure findings. (1) A gate must match the repo’s own test runner — unittest scripts are the portable choice (Django has no pytest). (2) Fable on the plan works for single-shot but NOT as an agentic acting lane through the proxy: the native-Anthropic path throws RepeatedFormatError (hits the output limit before emitting a parseable action) even with a 32k cap, while GPT-5.5 on the codex plan and Fable via OpenRouter both act fine. So the escalation here used Fable@OR (~$1–2 for 6 instances); plan-based agentic acting for Anthropic models needed a proxy fix — found and fixed the next iteration (section 20, finding 20a).

Bottom line (iteration 48). Verification-gated delegation is the first multi-model coding shape that adds value — it lifts a cheap worker from 60% to 76% by escalating only where an observable test says to — vindicating “observe, don’t predict.” But it does not reach the frontier, and the reason is precise and measured: the gate is only as good as the test the strong model can write from the issue, and a one-shot repro is a lossy proxy (68% precision) for the real acceptance criteria. The gate quality is now the whole game. Next (Stage 1b, section 20): a repo-aware (agentic) test author to lift precision, plus the proxy fix that lets Anthropic models act on the plan — and, still open, a regression-suite gate and the full context-isolation orchestrator measured on harder/longer tasks where a solo session balloons, the regime where this architecture should actually pay.

20. Verification-gated delegation, Stage 1b — a repo-aware gate and a proxy fix for plan-based frontier acting (iteration 49)

Section 19 left two threads: the gate is a lossy filter (its precision is the ceiling), and Fable/Opus could not act agentically through the proxy on the Claude plan (so escalation had to rent Fable via OpenRouter). Stage 1b addresses both, on the same SWE-bench Lite (25) set.

Finding 20a — the proxy bug that blocked plan-based frontier acting, found and fixed (P1). On the Claude plan the proxy sends over Anthropic’s OAuth wire, where build_anthropic_kwargs renames every tool to mcp__<name> for Anthropic’s billing classifier, and the transport reverses that only for Hermes’s own registry tools. When the proxy serves an external OpenAI client that advertises its own tools (mini-swe-agent’s bash), the bare name is not in the registry, so it leaked back as mcp__bash — the client cannot match its own tool call and aborts every turn (RepeatedFormatError, empty patch). The fix reverses mcp__<name>→<name> for any tool the caller actually sent (commit 133abbfeb). Result at scale: all 7 escalated instances produced non-empty patches and all 7 resolved — up from 0 before the fix. Anthropic models now act agentically on the plan at $0 marginal, closing the gap flagged in iteration 48.

Finding 20b — the escalation lane is flawless; 100% of the residual is false-accepts. A repo-aware author (the strong model issues a few READ-ONLY exploration commands in the base container — grep/cat/ls/sed — before writing the repro, so the test targets the real API) lifted the assembled result to 20/25. The decomposition is the story:

Because escalation resolves everything it touches, every one of the 5 remaining failures is a false-accept — a flash patch that passed the observable repro yet failed the hidden FAIL_TO_PASS (astropy-14182, astropy-14365, django-11742/-11815/-11848). They are locked in as the cheap patch and never escalated. So the earlier finding sharpens to its exact form: the worker is not the bottleneck and the escalation model is not the bottleneck — gate precision is the entire ceiling. Every false-accept is a patch that satisfies one reproduction case but misses an edge case the hidden multi-case suite checks — the structural limit of a single-test gate. The lever for Stage 1c is a multi-case / regression gate, not a stronger worker.

Directional, not headline: the precision delta is confounded. Iteration 49 changed three things at once versus iteration 48 — the test author (GPT-5.5→Fable, forced when the GPT-5.5 codex-plan token expired mid-run), the author becoming repo-aware, and the escalation transport (OpenRouter→plan). So the +4 pp precision (68→72%) and +1 resolve cannot be attributed to repo-awareness alone; a Fable-one-shot-author control would isolate it. What is robust regardless of the confound: the proxy fix (7/7 from 0), that escalation resolves 7/7, and that 100% of the residual is false-accepts.

Caching (measured, per standing directive). Both the test-author and the escalation lane ran Fable on the native-Anthropic plan path. Its usage.moa comes back all-zero (prompt_tokens_details.cached_tokens = 0, aggregator tokens 0) across all 94 agent steps — the proxy does not surface Anthropic’s cache_read_input_tokens, so caching on the plan lane is unobservable through the proxy. This instrumentation gap is unchanged by P1 (P1 fixed tool naming, not usage accounting); the concrete fix is to pipe Anthropic’s cache fields into usage.moa.aggregator. Server-side prompt caching is almost certainly active on the plan — we just can’t measure it yet.

Cost note. Because both lanes rode the Claude plan, this iteration added $0 marginal OpenRouter spend (iteration 48’s escalation rented Fable@OR for ~$1–2; the P1 fix removes even that).

Bottom line (iteration 49). The verification-gated architecture now runs entirely on plan credits and resolves 20/25 — and its failure mode is fully characterised: not the worker, not the escalation model, purely the gate’s false-accepts. “Observe, don’t predict” holds; the open question it leaves is narrow and mechanical — make the observable signal richer (a regression suite, not one repro) so fewer wrong patches slip the gate.

13. Tooling & execution tally — what ran each experiment

Are we running on our own discoveries yet? Partially — honestly tallied: the worker MoA endpoint (cere-moa + plan-GPT-5.5 lanes) served as an offload target for builder agents and as the blind grader in the judge eval, and every benchmark exercises the proxy itself. But the heavy thinking is still Fable + Sonnet on the Claude plans — the discoveries are the PRODUCT being built, not yet the build system. Queued: route bench analysis and doc drafting through moa:omni lanes and measure whether quality holds.

Budgets: OpenRouter ceiling $500 (raised 2026-07-05); spent ≈$415 to date (iteration 46's SWE-bench agentic runs added ~$180 — kimi-solo ≈$20/24 instances, moa:auto ≈$2.5/25 since escalations ride the plan; iteration 47 added ≈$15 — Set A cheap-actor OR spend across 100 instances + Set B's Grok-4.5 voter tokens via OR; Opus 4.8 and GPT-5.5 rode plan credits at $0 marginal; iteration 49 added ≈$0 — both the repo-aware test author and the Fable escalation ran on the Claude plan, so the P1 proxy fix removed even iteration 48's ~$1–2 Fable@OR escalation rental). Frontier tiers ride the ChatGPT plan at $0 and builds ride the Claude plan, so marginal experiment cost is mostly wafer pennies + open-model OpenRouter tokens.

Appendix — Changelog (version history)

Newest first. The executive summary and sections above always reflect the latest state; this is a condensed highlights log keyed to version bumps — not every iteration (currently 49) gets its own row, and some early/minor ones are folded together or omitted.

Generated autonomously by the Claude agent within Turq · fork branch · report v41 · iteration 49 · 2026-07-10