A research system returns 100 rows. Every row has a company name, a role and a source link. The table looks finished. How many rows can a seller actually use?
That question requires more than checking whether the company exists. A real person at the wrong employer is an incorrect prospect. A correct account missing the fact that determines qualification is incomplete. Three copies of the same company are one finding and two duplicates.
Fama's research work treats these as separate evaluation problems. The implementation and test definitions discussed here were inspected in development. This article does not report a live evaluation, a production accuracy score, or a finished customer capability.
Define the Unit Before the Score
For account research, the unit might be a company with a stable domain and the facts required by the brief. For contact research, it includes the relationship between the person and the relevant employer. For an event list, the edition or date may be part of the identity.
This changes what counts as correct. Finding the right conference from the wrong year should not pass an upcoming-events request. Finding an operations leader who left the target company should not pass a current-contact request.
Fama's development evaluation code builds keys for these different units and checks required facts separately from identity. That is a concrete design choice: a recognizable name alone cannot earn a fully correct result.
The reference set also needs a version and a date. A source can change between two runs. If the expected answer changes silently, a score may reflect the reference update rather than an improvement in research.
Keep the Denominators Visible
Consider this deliberately invented evaluation result. The system outputs 100 accepted rows. Ten are duplicate copies of other rows. Among the 90 unique rows, 72 meet the reference requirements, eight contain a wrong required fact, five omit a required fact, and five have not yet been adjudicated.
The immediately usable yield is 72 out of 100 displayed rows. Among the 85 unique rows that have been adjudicated, 72 satisfy the full requirements. The five unjudged rows remain unknown; treating them as correct would inflate the result, while calling them false would invent a failure.
If the frozen reference contains 120 eligible units and the system found 72 correctly, recall against that reference is 60%. That is not recall against the whole market unless the reference itself is complete for the stated scope.
The calculations reveal different problems. Duplicates waste review effort. Wrong facts can create bad outreach. Missing facts leave qualification unfinished. Missed eligible units reduce coverage. One blended score hides which repair matters.
Test Rejection and Uncertainty
A system evaluated only on obvious positive examples can learn to accept too easily. Include accounts that look attractive but violate a decisive criterion, and cases where the right answer is “not enough evidence.”
For a seller targeting centrally managed service businesses, an eligible case could have a confirmed group operations function. A negative case could have the same industry and size but independent local purchasing. An uncertain case could have several branches with no evidence about decision ownership.
Each case needs a different decision, supported by the available information.
Also inspect false rejections. A system that accepts almost nothing can look precise while being commercially useless. Evaluation needs to count relevant accounts held back or rejected, not only errors in the accepted list.
Verify the Claim Behind the Citation
Use difficult evidence shapes: two people in neighboring rows, an old job alongside a current role, a funding amount quoted in different units, a press release copied across domains, and a source that contradicts an earlier version.
The claim should stay attached to the right entity and time. A thousand-fold amount error should not pass because the number contains the same digits. A second copy of a release should not earn independent-source credit.
Fama's inspected evaluation code contains required-fact checks and amount normalization; its test definitions include source-lineage and evidence-strength cases. These checks are narrower than a complete guarantee. They should be examined for false positives, false negatives and disagreement with expert review.
Evaluate the Final Answer Too
The table may preserve uncertainty while the summary erases it. A “reported” value becomes a definitive claim; a partial search becomes “all matching accounts”; an assumption becomes a verified fact.
Anthropic's agent-evaluation guidance distinguishes the agent's narrative from the actual outcome. For prospect research, inspect both the stored records and the answer the seller reads. The answer should carry the same limits as the underlying work.
Different checks have different jobs. Code is useful for identity keys, duplicate counts, required fields and arithmetic. A model can help assess whether a passage supports a sentence, but its judgments need calibration. A human reviewer should settle ambiguous commercial fit and inspect cases where the automated score appears surprising.
Measure the Work Left for the Seller
Two systems can produce the same number of usable prospects while imposing different review costs. Record how long it takes to inspect the evidence, correct the identity and prepare the record for handoff. Keep latency and research spend alongside quality, rather than folding everything into one opaque grade.
For a buying decision, repeat the evaluation across more than one brief and preserve the awkward cases. A single attractive demonstration tells you little about reliability when the sources are messy or the requirements change.
The test is whether a seller can inspect the result and decide what to do next. A finished-looking table still needs honest coverage and manageable correction work.