SYSTEM 01
No LLM
Equity Copilot · local
Existing local provider · deterministic API
0 / 50 passed
0% strict case pass rate
- Composite score
- 0.5911
- Citation precision
- 59.83%
Deterministic · No LLMCaptured 11 September 2026
Financial answers,
put to the test.
Two retrieval systems. The same 50 source-grounded questions. A transparent look at what passes, what fails, and what the scores actually tell us.
50 cases per system
6 companies
16 primary documents
100 completed calls
01A shared quality gap
Strict passes require every case-level check
SYSTEM 02
No LLM
Lexical retrieval baseline
Query-token overlap · top three chunks
2 / 50 passed
4% strict case pass rate
- Composite score
- 0.6387
- Citation precision
- 73.33%
Both systems need better answer quality. The Copilot passed 0 of 50 strict cases; the baseline passed 2. Both missed all seven expected refusals. Every call returned a nonempty response without a transport error. This local comparison uses no LLM and does not establish production readiness.
02Measured results
Same questions, different retrieval approaches
| Metric | Equity Copilot · local | Lexical baseline |
|---|---|---|
| Strict case passes | 0 / 50 | 2 / 50 |
| Composite score (not accuracy) | 0.5911 | 0.6387 |
| Factual-point recall · 43 answerable cases | 17.44% | 17.05% |
| Citation precision · per-case mean | 59.83% | 73.33% |
| Required-document recall · per-case mean | 74.00% | 84.00% |
| Correct expected refusals | 0 / 7 | 0 / 7 |
| Transport errors / empty answers | 0 / 0 | 0 / 0 |
| Median / p95 answer call | 670 / 1614 ms | 0 / 2 ms |
The API includes loopback HTTP and persistence; the baseline runs in-process. Its 0 ms median is integer rounding of sub-millisecond calls, not zero work. These timings do not establish a service speedup.
How to read the scores
Read the limitations alongside the numbers.
The baseline has better citation-ID matching in this run. A higher composite score does not establish a more accurate financial answer.
- A strict pass requires factual coverage, citation thresholds and correct refusal behavior together. Partial, relevant answers can still fail.
- The raw harness reports 86% behavior accuracy for both targets: 43 correct non-refusals out of 50 answered cases. Actual expected-refusal success is 0 of 7.
- The scorer uses word/number overlap and document identity, not semantic entailment. Seven severe flags per target reflect missing refusal; they are not seven independently verified fabrications.
- Some questions omit the fiscal period that their required-document metadata assumes. Exact-period scoring therefore has a question-design limitation.
03What went wrong
Three patterns from the selected-response review.
Relevant text, incomplete answers
The NVIDIA growth answer mentions AI demand but omits the annual growth figure and introduces quarterly data. The annual/quarterly bridge misses the annual citation.
Refusal and relevance
For an unavailable price target, both systems return unrelated filing text instead of explicitly stating that the sources cannot answer the question.
The metric has blind spots
An AI infrastructure comparison receives lexical credit for risk language that does not provide the requested investment comparison. Keyword overlap can over-credit an answer.
Read the selected-response review and full limitations. This review is AI-assisted; independent human adjudication remains outstanding.
04Sources & execution
Six companies, 16 complete primary documents, and 1,311 API chunks. Raw files, normalized source texts and all 40 pinned evidence contexts match the suite hashes. Actual ingestion uses the Copilot’s production HTML/PDF extraction with paragraph boundaries. Both targets receive complete extracted source content; no reference answers, rubric text, or evaluator-selected documents enter the target request.
The API uses its existing deterministic local provider. The baseline selects the top three 1,200-character chunks by query-token overlap and returns 400 characters per chunk. No LLM was available at the local Ollama endpoint, and no paid provider was used.
Capture provenance & original artifact links
Captured: 2026-09-11T04:08:22.238427+00:00
Harness commit
b0a50bfb32b899a4a67db8d86d71a48e3d1d07b4
Copilot commit
9f1fc77a76da3b6b0761c53f82e7c18b1c96fb34
Suite SHA-256
1eaf08f68d6df324e95df38f6b6ac87850d2125c0602f61249ea467caaf9e0cb
Configuration · Source provenance · Ingestion and extraction hashes · 100 redacted capture records · Capture checksums
Public records omit extractive answer text and source excerpts to avoid reproducing substantial filing/transcript prose. Scores are unchanged; original full-response hashes remain available. Public redacted records cannot independently rescore text without the original local or newly reproduced captures. Export policy · Original artifact hashes
Inspect & reproduce
Equity Copilot · local
Lexical retrieval baseline
Methodology and reproduction commands. Display rebuilding is offline; a new evaluation requires new target calls and a new output directory.