The eval harness, evaluated honestly.

The live target is the Equity Research Copilot public demo — a keyless deployment whose corpus seeds two companies (NVDA, MSFT) with 10-K and transcript excerpts. The eval suite was authored against a six-company universe (adding AAPL, JPM, XOM) with document ids like nvda_2025_10k.

The harness surfaced two distinct problems, case by case. First, the coverage gap: questions about unseeded companies error out (error_rate 0.60–0.80 across categories), and citation rules never match because the demo's document ids differ from the suite's expected labels — hence citation precision 0.00 against 1.00 on the mock.

Second — and more damning — a real behavioral bug: on the five cases designed to be unanswerable, the demo's deterministic fallback answered anyway, synthesizing from whatever chunks it retrieved. Refusal accuracy on that category: 0.00, with all five flagged as severe hallucinations. The system refuses correctly when asked about companies it doesn't know (refusal accuracy 1.00 elsewhere), but not when asked unanswerable questions about companies it does know. That is precisely the "fluent but wrong" failure mode this harness exists to catch.

Why publish a failing report? Because a harness that only ever produces green dashboards is marketing. This one produced a precise, reproducible diagnosis of a real deployment — a coverage gap and a refusal bug — and both are now roadmap items with a regression test attached: seed the remaining companies, align document ids, gate the fallback on evidence strength, re-run.