The target quantity
Consider a fixed pool of tasks with independent reference labels, so that answer correctness can be determined for every output. Write A for correctness and E for whether the evidence meets a specified sufficiency standard. For a task attribute or predefined evidence condition G, the quantity of interest is:
This probability initially refers to a proportion in the fixed pool: how many correct outputs in group g lack sufficient evidence. Applying it to deployment requires an account of how the tasks were sampled and how the deployment distribution differs. A group with no correct answers has undefined conditional risk. There is no correct-answer denominator to divide by.
The sufficiency standard needs a finite reference corpus, a time cutoff, and an audit rubric. It must identify the claims that require support, the support each requires, the counterevidence that blocks acceptance, and the rules for resolving conflicts. Material outside the corpus may change the assessment. A passing label therefore applies within the stated reference scope.
For an open-ended task, even answer correctness may require expensive human review. A would then be unknown outside the audit sample. Both the failure count and the number of correct answers would need estimation, with uncertainty in the answer labels handled as well. Assuming known A lets me isolate the evidence problem first.
Four combinations of answer and evidence
Answer correctness and evidence sufficiency produce four cells. The two can both pass, both fail, or disagree. The case A = 0, E = 1 deserves attention: here, E measures whether the evidence package supports specified claims. It does not certify the full derivation of the final answer. An agent can retrieve adequate material and then make a mistake when combining it, calculating a result, or writing the answer.
A different definition of E would change the table. If E means a complete proof that entails the final answer, A = 0, E = 1 should be empty whenever the rubric and reference answer are sound and consistent. An occupied cell would then be a reason to check the definitions or labels. The interpretation has to be settled before using the diagram.
Only outputs with resolved evidence status enter the cells. Unresolved cases remain outside, and the table needs a resolution-coverage figure beside it. Let D indicate a resolved status and pae,g be the joint proportions among resolved records in group g. Then:
A lower conditional failure rate need not mean the evidence improved. Two agent versions can answer different tasks correctly; one could lower its rate simply by losing the difficult correct answers. I would report each cell's share of the original pool alongside answer accuracy and the unresolved rate. A paired comparison on tasks both versions answer correctly can then compare evidence within that common subset, but its result applies only to that subset.
How unresolved cases affect risk
The audit record should connect each claim to a source location and material version, state the alleged support relation, and record the scope of the counterevidence search. A working citation lets a reviewer retrieve material. A matching quotation confirms that the words occur there. The reviewer still has to check what those words support. Increasing the number of retrieved documents also leaves completeness unproven.
I use three review states: supported, insufficient, and unresolved. Supported and insufficient map to E = 1 and E = 0. Unresolved records a judgment the audit cannot yet make. Finding no counterevidence is too weak a reason to assign supported; taking a long time to review a case is too weak a reason to assign insufficient. Confirming that required support is missing can justify insufficient. Temporarily inaccessible material or an unfinished assessment remains unresolved.
The following bounds require the fixed rubric to define a latent binary evidence status for every record, with resolved labels treated as reference judgments. Unresolved means that this status has not yet been measured. If the rubric itself cannot define the binary distinction, the target needs revision before applying the bounds. In a complete audit of the group A = 1, G = g, suppose F cases fail, S have sufficient evidence, and U remain unresolved. Then:
The interval expresses what unresolved cases leave unidentified. It contains no allowance for sampling error. With a probability sample, F, S, and U must also be estimated using the sampling design, and uncertainty added to the bounds. Dropping U changes the target to resolved cases. Counting unresolved cases as sufficient evidence can understate the risk.
Evidence sufficiency and causal faithfulness also require separate checks. A model could guess the right answer and attach supporting material afterward. Testing whether evidence drove the answer would require interventions, such as changing relevant material while holding irrelevant presentation features fixed and checking the output's response. The interpretation would depend on those intervention assumptions. Static citation review cannot supply it.
What review signals tell us
Judge scores, reviewer disagreement, citation matches, source overlap, and retrieval-coverage measures are possible audit signals. Record them as Z for every output. A threshold may be useful for allocating review effort, but it needs validation against evidence judgments before treating it as evidence of sufficiency.
Several judges can read the same summary and miss the same counterevidence. They may also share a model family. Their agreement then carries less information than the number of reviewers suggests. Disagreement has ambiguities of its own: an unclear rubric can produce conflicting judgments even when the evaluated agent has made no error.
I am interested in how many failures human review can still find among outputs with high judge agreement. Citation matching has a similar limitation worth measuring: it may help find broken references while offering little information about references that exist but fail to support a claim. Once tested against human judgments, these signals can guide the audit, identify outputs needing attention, and potentially improve the precision of the estimates.
Retrieval coverage is harder to observe than its name suggests. The system knows what it retrieved; it usually does not know every document it should have retrieved. Different queries or tools can extend the search without proving it complete. The record needs both the gaps that were found and the boundaries of the search, so that a later reviewer can interpret what "not found" means.
Auditing on a fixed budget
Freeze the task pool, outputs, rubric, profile definitions, and budget before reviewing labels. Profiles should use a small number of reproducible attributes, such as task category, a precomputed evidence-conflict flag, or source availability. Task attributes make group membership stable when comparing agent versions. For groups defined by output signals, report membership changes. A group discovered from unusually poor audit results needs confirmation on independent data or an appropriate post-selection analysis; an unadjusted interval on those same results ignores how the group was chosen.
My starting design reserves some audit places for every nonempty stratum, then assigns the remaining places to strata expected to be difficult. Once the allocations are fixed, draw a simple random sample without replacement in each stratum. Save the allocations, stratum sizes, and random seed. A record in a stratum of size Nh with nh audit places has inclusion probability nh/Nh.
The figure assigns failure rates to a finite population for illustration. Oversampling a risk stratum changes the mix of records in the audit. The unweighted average follows that mix; weighting by known inclusion probabilities recovers the original pool's proportions in design expectation. Whether judge disagreement identifies a risk stratum in practice would still need to be checked.
Every target record must have a positive inclusion probability. If the audit covers only low-scoring or disputed outputs, high-agreement records outside that coverage have no chance of being selected. Any shared errors in those excluded records cannot be recovered by later weighting. When the budget cannot cover many fine strata, merging strata is preferable to silently leaving some out of a claimed full-pool estimate.
Weighted and model-assisted estimation
Suppose the rubric defines a binary status for every record and every record would receive a resolved label if audited. Let Bi indicate membership in A = 1, G = g, Fi = Bi·1(Ei = 0), and πi be the known inclusion probability. Because A and G are known across the pool, the group size Ng = Σi∈pool Bi is fixed. For Ng > 0:
The numerator is a Horvitz-Thompson estimate of the failure total. Dividing by the known group size gives an unbiased risk estimate under the stated probability-sampling design. If group size must also be estimated, the ratio of the weighted failure and group totals is a Hájek-type estimate. Each total can be unbiased while their ratio is biased in finite samples. Sparse audits of small groups can produce unstable estimates in either case, so the point estimate needs an uncertainty assessment.
Judge signals can enter through a failure predictor trained on an independent development set and frozen before the audit. Let mi be its prediction for each record, with predictions outside the target group set to zero. Start with the predicted failure total, then correct it using the sampled residuals:
This standard model-assisted difference estimator corrects prediction error in design expectation when the predictor and population are fixed, the sampling design is correct, and labels are resolved. It still estimates failures under the human rubric. Divide by the known group size to obtain risk. The predictor can be imperfect; whether it improves precision depends on residual variation and the sampling design. Weak predictions may increase variance. A realized estimate can also fall outside its natural range, and clipping the risk to [0, 1] sacrifices the original estimator's finite-sample unbiasedness.
Training on the current audit labels changes this argument because the predictor is no longer fixed independently of the sample. Sample splitting, suitable cross-fitting, or a separate design justification would be needed. When some cases remain unresolved, estimate confirmed failures and unresolved totals separately and report risk bounds. Predictions should not be used to fill unresolved cases as though they were reference labels.
Audit records and uncertainty
Keep the pre-sampling signals, inclusion probabilities, material shown to reviewers, and final adjudications as separate records. This makes it possible to recompute results and inspect what a reviewer could see. Reviewers apply the fixed rubric independently before adjudicating disagreements. If the rubric changes during review, record the version and review affected cases again.
pool = freeze(tasks, outputs, reference_labels)
rubric = freeze(reference_corpus, cutoff, evidence_rules)
profiles, signals = compute_before_audit(pool)
design = allocate_with_positive_probability(profiles, budget)
sample = random_sample_within_strata(pool, design)
for item in sample:
packet = build_blinded_packet(item, rubric)
review = independent_review(packet)
result = adjudicate_or_keep_unresolved(review, rubric)
save(item.id, design.pi(item), result, review.trace)
report_joint_outcomes_and_resolution_coverage()
report_weighted_risk_or_identification_bounds()
report_uncertainty_using_the_actual_sampling_design()Blinding can hide system identity and judge scores, but the packet must retain context needed to assess support. Excerpt selection can affect the judgment too. Reviewers need access to original material within the fixed reference scope. The pseudocode describes the proposed audit procedure; it has not been validated as an evaluation implementation.
For a fixed pool, the uncertainty calculation follows the actual sampling units and design. Record-level stratified sampling without replacement needs its corresponding variance estimate. Sampling task or source clusters requires treating those clusters as the sampling units. Shared origins alone do not invalidate finite-population design-based inference. Generalizing beyond the pool is a separate problem and requires assumptions about dependence between tasks, sources, and repeated outputs. Small groups and unresolved-case bounds need particular attention.
Test the estimator before comparing systems
A small pool that can be audited relatively completely would provide the reference population. After a pilot, freeze the rubric and retain unresolved cases. Repeated probability samples from that population can then compare simple random sampling, risk stratification, and estimation with auxiliary signals at a fixed human-label budget. The predictor's development labels have a cost too, which should be reported separately.
The comparisons need bias, mean squared error, interval coverage, and interval width, both overall and within each prespecified group. An average can improve while the estimates for a small group deteriorate. If the reference audit still has unresolved cases, evaluate the bounds and their uncertainty. Auditing every record does not remove ambiguity from its judgment.
Agent-version comparisons require paired outputs on fixed tasks and reference material, with model settings and budget accounting specified. Report answer quality and evidence outcomes together with unresolved rates and model and tool costs. Full-system comparisons should use equal execution budgets. A component study should hold other conditions fixed and account for any extra computation.
Controlled defects can test how the evaluator responds: remove a necessary support span, substitute an irrelevant citation, or add conflicting material. These tests probe particular weaknesses; the natural failure distribution still needs its own evaluation. I have not run these experiments. I would first use them to check the evaluation target and estimation procedure, before asking whether they can help improve an agent.
Evidence gaps in the high-agreement region
The evidence evaluation I want to develop from MARS-Gov would list the claims needing support, check what the materials actually support, and retain the judgments still unresolved. Those records would establish what a reliability score counts.
I am most interested in outputs with high answer accuracy and high reviewer agreement. The existing signals may be near their ceiling there, but the frequency of evidence failures and the difficulty of estimating it remain open questions. Which additional information can predict gaps found by human review without repeating the blind spots of the existing signals?
A first study could compare estimation error and interval coverage with and without auxiliary signals on a fixed task pool. If the same human-review budget yields no precision gain, the additional judge cost is not justified by improved estimation precision. If the pilot reveals that the rubric lacks an operational sufficiency criterion, revise it and review affected records again before comparing systems.