Research · 2026-08-14 · 12 min read

The harness is the product

Two planes, never averaged. On the judgment plane — a fair head-to-head where a peer given the same founder brief can legitimately try — Mycel scores 21/21 (100%) against a documented naive-unscaffolded lower bound at 11/21 (52.4%). On the architecture plane, Mycel is behavior-graded 42/42 against its real kernel modules — and peer coding harnesses implement none of the domain-scaffold capabilities, an absence we report in a matrix rather than a losing 0%.

OpenCode
bare control
Claude Code
Anthropic CLI
Codex CLI
OpenAI CLI
Mycel
OpenCode + scaffold

Abstract

Public agent leaderboards rank models. Controlled work shows the right unit of evaluation is the harness–model pair (arXiv:2607.22585, 2605.23950). Mycel’s claim is harness-shaped: it drives an off-the-shelf agent loop (OpenCode) and puts the product’s authority — pinned primitives, a computed risk engine, a consequence gate enforced by withholding the credential, an unforgeable promise ledger, a ratcheting autonomy envelope, and a generation quality floor — outside the sandbox. The commitment underneath all of it is behavior-graded evaluation: we grade what a real kernel module did — its return value, or the event-log rule — never what a model said. There is no LLM-as-judge anywhere in the deterministic path.

The number we deleted

We used to report a single “overall pass rate”: Mycel near 100%, peers near 30%. It was a flattering number and it was wrong — not inaccurate, but a category error, which is worse, because it is the kind of wrong that survives review.

Half of those suites test a property a bare coding harness does not implement by design— a consequence gate for money, email and publishing. Counting a missing feature as a 0% the other tool “lost” is not a measurement, it is a scoreboard we built to win. It also hides the only thing worth knowing: whether the idea is right. A gap that large should make you suspicious of your own benchmark, and ours did.

So we deleted the headline figure and split every suite into one of two planes, never averaging across them:

Judgment plane. A right answer exists and a peer handed the same founder policy brief can legitimately attempt it. Reported as a head-to-head.
Architecture plane.The suite tests a scaffold property a bare coding harness does not implement. Reported as Mycel’s own behavior grade plus a capability matrix; peers are marked absent, never scored 0%.

Judgment plane — a fair head-to-head

These are the exact adversarial cases the kernel’s committed hard-evals use (kernel/harness/test/eval-collections-judgment.test.ts, eval-bookkeeping-exactness.test.ts). Mycel executes the verified primitive. The naive-unscaffolded column is a documented lower bound — a quick implementation from the same brief without the primitive (float money math, threshold-only dunning, a rough risk heuristic). It is deliberately not a frontier model and not a competitor; it exists to make the specific failure modes the primitive removes visible. Real-model columns require --live and a key, and are disclosed per-trial — never treated as the headline.

Figure 1 — Judgment pass rate (head-to-head)

Mycel100%
Naive unscaffolded (lower bound)52%

Judgment suite	Mycel	Naive unscaffolded
Collections judgment (dunning)	9/9	5/9
Bookkeeping exactness (reconcile)	6/6	3/6
Risk classification	6/6	3/6
Total	21/21	11/21

Per-case breakdown of where the naive baseline fails — Figure 2 — Where the primitive earns its place. The lower bound fails precisely where a wrong call is expensive.

The lower bound fails exactly where a wrong call is costly: a disputed invoice chased, a kept promise chased before its date, a 48h double-chase, a penny rounded away, a false “reconciled” over an unmatched row, a destructive external delete ranked routine, an unknownaction defaulted to “low”. The pinned primitive removes each by construction — which is why the gap is a property of the scaffold, not the model.

Architecture plane — Mycel behavior grade

Each row runs the real production module on adversarial cases and grades its return value or the event-log rule. This is the executable form of the study’s primitives — credential-withholding enforcement, promise verification against an unforgeable log, ratcheting autonomy, and generative composition — not a mock of them.

Suite	Grade	Graded against
Consequence gate integrity	5/5	outreach/guard.ts · guardSend / isMessagingSend
Prompt-injection resistance	3/3	outreach/guard.ts · benign verdict == hostile verdict
Promise vs evidence	3/3	promises.ts · semantics over the event log
Escalation / autonomy envelope	10/10	policy.ts · evaluatePolicy + autonomy.ts · permits
Generation quality floor	11/11	wedgeauthor.ts · authorWedgeFromOutput / reviewDraft
Primitive composability	10/10	workflows.ts · runWorkflow vs kernel/workflows/*.mjs
Total	42/42

Architecture behavior grade across the six scaffold suites — Figure 3 — Architecture behavior grade. Each suite runs the shipped kernel module and grades what it returned.

Capability matrix — an absence, not a defeat

A coding harness is a real agent scaffold: it loops, calls tools, manages context. What it lacks for operator work is the domainscaffold. Tool-permission prompts in Claude Code and Codex protect the developer machine; they are not an approval queue for money, email, and publishing under a client’s name. So we report the difference as a matrix, not a score. Peers implement 1 of 7 domain-scaffold capabilities — an absence by design.

Capability matrix across harnesses — Figure 4 — Capability matrix. Only Mycel encodes packs, risk, consequence gates, promises, autonomy, and a generation floor.

Capability	Mycel	OpenCode bare	Claude Code	Codex CLI
Agent tool loop	yes	yes	yes	yes
Pinned verified primitives	yes	absent	absent	absent
Computed risk engine	yes	absent	absent	absent
Consequence gate (money/send/publish)	yes	absent	absent	absent
Injection-resistant gate	yes	absent	absent	absent
Unforgeable promise ledger	yes	absent	absent	absent
Ratcheting autonomy envelope	yes	absent	absent	absent
Generation quality floor	yes	absent	absent	absent

What this proves

Mycel’s verified primitives are correct on the exact adversarial business cases where a wrong call is expensive — behavior-graded, deterministic, byte-stable.
The consequence gate, injection resistance, promise ledger, autonomy envelope, generation floor, and composability are properties of the scaffold, not the model — a bare coding harness of the same class (verified against the matrix) has none of them.
The judgment head-to-head is fair: a peer told the exact policy can attempt it; the pinned primitive removes the drift that remains.

What this does NOT prove

It does not run a live end-to-end agent loop against a paying customer. MYCEL_RUNTIME=mock— this verifies the machinery around the model, not the model’s judgment in a live engagement. That is the largest gap between this study and a shipped product.
The naive-unscaffolded baseline is a documented lower bound, not a frontier model. The real model comparison needs --live with a key, and those numbers depend on the model, temperature, and date.
It makes no claim about SWE-bench or general coding ability. We measure operator deployability, not IQ.

Reproduce

Deterministic path — no network, no keys, byte-stable, the numbers on this page:

cd evals
npm install
npm run ci                          # Mycel behavior grades + naive lower bound + report
npm run eval -- --live --ablation   # adds real-model judgment columns (needs OPENAI_API_KEY)

Machine-readable run: evals/results/latest.json (synced to landing/public/blog/results.json). Full write-up: evals/report/HARVEST.md. A skeptical reviewer can re-grade every case offline — the graders are exact-match on real module output, and the modules are the shipped ones.