Harness Anything

1. Turn Domain Tasks into Execution Contracts

The Deliverable Determines What the System Must Store

“Analyze this company” is difficult to use as an executable product specification. “Produce an operating analysis as of a specified date, covering specified entities, with every key figure traceable to original material” already contains requirements that can be checked.

A task specification should at least define the deliverable, scope, data date, permitted actions, and completion criteria. If any of these are missing, the agent may supply a plausible interpretation during execution. Standardizing the output format is often easy; standardizing what the format means takes more work.

JobSpec:
    deliverable: report | proposed_change | committed_operation
    scope: entities, account_or_document, jurisdiction
    time: effective_at, knowledge_cutoff
    constraints: metric_basis, policy_version, allowed_actions
    acceptance: required_evidence, checks, reviewer
    limits: cost, elapsed_time, tool_calls
    escalation: missing_inputs, conflict, uncertain_outcome

The user need not fill in this entire specification at once. The system can obtain some of it from the existing identity, business object, and application configuration, asking the user only about ambiguities that materially affect the conclusion. Whether “revenue” means gross or net requires clarification; an unspecified font size can usually use the product default.

Required and optional evidence should also be separate. If a key entity cannot be identified, the system cannot proceed with a conclusion that assumes its identity is settled. If supplementary material is missing, it can narrow the deliverable and state what remains uncovered. Degradation should change the task commitment and make that change visible to the user, rather than merely relabeling a failure as ignorable internally.

The task contract also changes storage design. For a draft intended only for human reading, retaining its text and sources may be sufficient. For a committed business operation, the system must also retain the approved object, operation identifier, remote receipt, and final outcome. A single “complete” field cannot represent these different commitments.

Separate Professional Requirements into Different Kinds of Checks

Domain experts often express a requirement in one sentence: “These two measurement bases must not be mixed.” Implementation requires further questions: which fields represent the basis? What does a missing field mean? Even with matching currencies, must units and reporting periods be checked? Who adjudicates exceptions?

Requirements that can be computed explicitly should become programmatic checks. A refund must not exceed the refundable balance; calculation inputs must use compatible currencies; contract edits must target the specified document version. Requirements involving semantic judgment can be assessed by a model or expert, such as whether evidence is sufficient for a causal explanation. Both kinds of checks need a state for insufficient information.

JSON Schema can guarantee that fields exist and that types and enumerations are valid. It cannot prove that the correct company was selected or that a number has the correct business meaning. A rule engine likewise covers only the rules that have been written. Acceptance records should include the check, input versions, and result, so that “passed the check” can be distinguished from “reported no error.”

Domain knowledge should also be placed according to its purpose. Stable concepts belong in data types and tool interfaces. Explicitly decidable constraints belong in rules. Frequently updated material that needs interpretation should remain versioned evidence. Mandatory steps belong in control flow. Errors discovered by experts belong in evaluation cases. Putting all of this into one long prompt loses the distinct update, permission, and validation mechanisms each kind of knowledge needs.

2. Decide Which Decisions Belong to the Model

Fix the Business Steps; Allow Local Exploration

Business workflows often include steps that must run every time: verify identity, load current policy, check measurement conventions, and validate before submission. Letting the model decide whether to perform these steps turns a definite business requirement into a probabilistic event.

A better division of responsibility is for code to guarantee the steps and their order, while the model handles local work that cannot yet be laid out in advance: which material to find, how to explain an anomaly, what to search next, or how to repair a failed draft. Use a fixed workflow when the steps and transition conditions are known. Open an agent loop when the next step depends on information obtained during execution. Greater autonomy also brings additional latency, cost, and error propagation. Building effective agents

Stripe's Minions place mandatory checks in deterministic nodes and give exploration and repair to agent nodes. Their development environment is isolated and permission-constrained. This execution structure is useful to borrow, while production business writes still need a separate authorization boundary. Stripe Minions: workflow design

spec = resolve_job_spec(request, trusted_context)
require(check_scope_and_permission(spec))
evidence = gather_within_scope(spec)
candidate = agent.solve(spec, evidence, read_and_prepare_tools)
checks = validate(candidate, spec, evidence.versions)
return deliver_or_escalate(candidate, checks)

A fixed skeleton does not mean every task needs the same long pipeline. Low-risk material organization may require only a few checks; external writes can add proposals, approval, and submission. Keep a mechanism when it reduces a real class of failures enough to justify the burden it adds to each task.

Split Agents along Context, Permission, and Conflict Boundaries

Giving separate role prompts the names “analyst,” “researcher,” and “reviewer” may help organize work, but role names do not themselves create independent checking capabilities. Several agents using the same material and the same mistaken measurement basis can unanimously reach the wrong conclusion.

A split should have a concrete benefit: a subtask needs only a small domain context; an executor should hold restricted permissions; tasks can compute independently; or a local result has clear acceptance criteria. Small, focused agents make context and evaluation easier to control, but there is no fixed step limit suitable for every task. 12-Factor Agents: focused agents

After splitting, merging must be designed too. Harvey's contract review system has workers for different rules produce edits on versioned document branches, labels edits with their rule ownership, and sends conflicts back to a coordination layer. The important design elements are branches, ownership, and a conflict protocol. Multiple workers editing the same text do not naturally produce a correct result. Harvey: rebuilding contract review

A general domain system can have workers return proposed edits, supporting evidence, a base version, and an impact scope. The coordinator first checks whether they overlap, then decides to merge, recompute, or seek human review. Non-overlapping text positions do not guarantee compatible business meaning: one edit might change a definition while another changes an exception. The merged artifact still needs cross-rule consistency checks. Parallelism from multiple agents must be weighed against extra model calls, duplicate retrieval, and coordination cost.

3. Turn Domain Knowledge into Checkable Evidence

Define the Search Space before Optimizing Recall

Domain knowledge is usually scattered across documents, table schemas, past operations, policies, and people's experience. Chunking everything into a vector database may fail to preserve the relationships that matter: which business line owns a term, whether a table measures orders or fulfillment, or which jurisdiction a rule covers.

Organizing table schemas and example queries into domain workspaces, followed by intent routing and relevant-table selection, narrows a candidate space whose items may be semantically similar but operationally different. Uber QueryGPT uses this design rather than relying directly on full-database matching across a large number of tables. Uber: QueryGPT

A domain harness can adopt the same approach: identify the entity, task category, and applicable rules first, then choose the databases, indexes, and tools to query. Routing decisions must also be correctable. If the initial domain is wrong, excellent local retrieval cannot recover material that was excluded from the search space.

Knowledge organization should preserve domain structure. Clause hierarchies, product-to-fault-code mappings, and financial metric dependencies may be better served by relational queries, directory navigation, or graph structures. Vector retrieval can supplement semantic matching. Whether to add query rewriting, hybrid retrieval, and reranking should depend on task outcomes and latency. Patterns for building GenAI applications

Permission filtering belongs in the retrieval path, not only in final-answer review. Once material enters model context, it has already been processed. Caches also need tenant, identity, or authorization boundaries; matching question text alone is not sufficient for a shared-cache hit.

Retrieval results should support the next action: applied filters, sources and versions, truncation markers, and an entry point for fetching full context. When results are numerous, distributions over authorized metadata can help narrow the scope. Metadata that cannot change a subsequent decision may simply burden context. Jason Liu: Beyond Chunks

Distinguish the Period a Fact Describes from When It Became Known

A financial report might describe last year's fourth quarter, be published in February, revised in March, and ingested by the system in April. At least four temporal concepts are involved: the period described, publication time, the revision version, and system recording time. A single timestamp field conflates their meanings.

Martin Fowler's discussion of bitemporal history distinguishes when a fact is valid from when the system records learning it. This distinction helps with retroactive revisions. Publication time should also be recorded separately: late ingestion does not mean the information was previously unavailable to the outside world. Bitemporal History

Fact:
    entity_id, metric, value, unit, currency, basis
    period_start, period_end
    published_at, recorded_at, revision_id
    source_id, source_version, locator

A task reconstructing a decision at a historical date should use only information available then. A task reconstructing history as currently understood may use later revisions. Both are legitimate, but answer different questions. Mixing these views in evaluation leaks hindsight into historical tasks.

Not every system needs a full bitemporal database. Fixed input snapshots and preserved decision-time versions may suffice. More complex storage and querying become worthwhile when the system frequently needs to ask “what was known then?” or trace the effects of retroactive revisions.

Check Citation Existence and Claim Support Separately

Attaching a source to a sentence involves at least three questions: does the cited passage actually exist? Does it apply to the current entity, time, and question? Does it support the conclusion? String matching can address the first, but cannot replace the other two.

This is also where structured extraction is easily overestimated. Extracted fields can fully satisfy a schema while their source describes only a subsidiary. A passage containing “growth” need not support a claim about profit growth. CitationMixin's documentation explicitly distinguishes validation of quoted passages from the accuracy of extracted facts; users must still check the relationship between evidence and claims. Instructor: CitationMixin

An important conclusion should ideally be traceable through its dependencies: source version, normalized facts, calculation, and final claim. Use calculation tools for unit conversions, period aggregation, and formulas. For interpretations, preserve key premises and their supporting material. This makes it possible to find the work that must be redone when a premise changes.

Conflicting evidence should also be an explicit result. When two sources give different figures, first check the entity, period, currency, statistical scope, and revision version. If the difference remains unexplained, retain the conflict and state its impact. Asking the model to choose the more plausible number discards information that should enter human judgment.

Withdraw Invalid Conclusions from Every Consumption Path

Financial auditing can treat currency, period, and source as parts of a numerical claim, then check the business basis of its calculation inputs. FinRobot stores these details in numerical claims and checks for mixed currencies in valuation-related audits. Acceptance can then go beyond numerical ranges to examine what the inputs mean. Numerical claim structure, currency-basis checks

The same implementation shows the boundary of check coverage. Its period audit returns empty findings when dates are missing, and the status aggregator classifies empty findings as a pass. An absence of recorded problems therefore does not establish that period completeness was verified. A stricter delivery contract should separately record which checks actually ran. Empty-input path in period auditing, status aggregation

Withdrawal propagation is even easier to overlook. If a conclusion has entered summaries, charts, caches, and later reports, marking the original field invalid is insufficient. FinRobot scans alternative display slots for a withdrawn target price so that another read path cannot show the old value again. Withdrawal must cover derived results and consumption paths. Alternative-field checks, clearing display fields together

on_input_revision_changed(input_id, new_version):
    affected = dependency_index.descendants(input_id)
    mark(affected, status=NEEDS_REVALIDATION)
    prevent_new_delivery(affected)
    invalidate_dependent_caches(affected)
    enqueue_required_checks(affected, new_version)

Dependency indexes and invalidation notifications locate affected work and trigger recomputation. The delivery boundary must also check whether the versions referenced by an artifact still satisfy the task contract. Otherwise, a concurrent task may publish an old conclusion before an asynchronous notification arrives. If delivery is based on a frozen historical snapshot, check against that snapshot. If it promises currently valid information, check current dependencies at a controlled publication boundary.

This is a suggested approach to dependency management, not a requirement to turn the entire knowledge base into a complex graph database. An early system can store dependency IDs and versions with its artifacts and invalidate them by task batch. Finer indexes become worthwhile only when recomputation costs and shared dependencies grow enough. Reports already delivered also need correction records; deleting a current page does not erase a past delivery.

Acceptance should run once more on the final assembled artifact. The body, summary, tables, and template can each be correct in isolation yet combine conclusions from different versions. Before delivery, check that key values and states agree and prevent unreviewed dependencies from entering new artifacts. Checking only one generation step misses later assembly and fallback paths.

4. Express Business Semantics through Tool Interfaces

Tool Results Should Include the Facts Needed to Continue

General SQL, shell, and HTTP tools are expressive, but also leave table relationships, valid parameter combinations, and business prerequisites to the model. Domain tools can narrow these choices: “retrieve this entity's revenue series on a specified basis,” for example, instead of making the model reconstruct a data provider's raw response every time.

An interface should return both the result and its boundaries: canonical entity ID, data version, units, covered periods, missing data, and usable source locations. Errors should also enable a next action, distinguishing MISSING_PERIOD, CURRENCY_MISMATCH, and temporary unavailability. A generic “tool failed” cannot tell the system whether to retry, obtain more material, or change its approach.

Research tools have the same requirements. An experimental result should at least be associated with its data split, code and configuration versions, environment, and run identifier. If the conclusion depends on randomized trials, preserve the corresponding randomness settings and statistical method as well. Returning a performance number alone leaves later analysis unable to determine whether it is comparable with a baseline.

Narrower domain tools are not always better. Defining a separate function for every possible query causes interface proliferation and frequent code changes as the business evolves. Stable interfaces can cover frequent operations, complex measurement conventions, and high-impact actions, while open exploration remains in a restricted, read-only query environment. Tool design should fix error-prone semantics without eliminating every model choice.

Model Analysis, Proposals, and Submission Separately

“A refund is recommended” and “a refund has occurred” need different objects. The model can generate candidates; the business service checks eligibility, freezes a proposal, and submits it once execution conditions hold. If preparation locks inventory, reserves capacity, or sends a notification, those effects must be recorded explicitly. A function named prepare cannot be assumed to have no side effects.

proposal = prepare_refund(
    order_id=canonical_order_id,
    amount=Money(value, currency),
    reason=allowed_reason,
    expected_order_version=observed_version
)
# In this example, prepare only reads, validates, and saves a proposal
decision = obtain_required_approval(proposal)
receipt = commit_refund(proposal.id, decision.id)

Preparation checks can catch obvious errors early and show reviewers the specific object and its impact. Submission checks must run again against the current state. The two checks serve different purposes: the former reduces invalid proposals, while the latter handles changes between proposal and execution.

Tool adapters should also declare a failure contract: whether an operation can be retried, how to query its result, how to identify the same business intent, whether partial completion is possible, and who reconciles the outcome. For a legacy write interface that supports neither result queries nor reliable deduplication, limiting the agent to drafts or human submission may be more effective than adding model self-checks.

5. Route Authorization, Approval, and Submission through One Control Path

Bind Approval to a Specific Proposal and Its Expiry

“Allow the refund tool” does not specify the order, amount, recipient, or duration of permission. Approval should target an immutable proposal bound, at minimum, to the tenant, initiating identity, target object, normalized parameters, business version, rule version, and expiry. A material parameter change requires a new proposal.

Tool approval in the OpenAI Agents SDK uses a specific call identity and checks the pending run state. Its server-side example keeps the full RunState on the server, sending the client only display details and an opaque decision ID. Call matching answers “which call does this decision belong to?” User authentication and business permissions remain the application's responsibility. Approval matching, server-side approval example

A proposal hash can help detect content changes, but cannot establish who approved it. Approval records must come from an authenticated, authorized server-side control path. Production systems also need to consume pending decisions atomically, preventing two requests from approving and dispatching the same work concurrently.

with local_db.transaction():
    pending = lock_pending_decision(decision_id)
    proposal = load_frozen_proposal(pending.proposal_id)
    require(can_review(authenticated_user, pending))
    require(pending.is_pending and not pending.expired)
    require(pending.proposal_hash == proposal.hash)
    pending.status = APPROVED
    record_approval(pending, authenticated_user)
    outbox.insert_unique(pending.operation_id, proposal)

This local transaction guarantees only that the approval record and the pending delivery are persisted together. The remote business API is outside the transaction, and the outbox consumer may deliver repeatedly. The subsequent business submission still needs its own idempotency and concurrency control.

Recheck Real-World Conditions at Execution Time

While approval is pending, an order may be canceled, another operation may consume the balance, or the initiator's permissions may be revoked. Restoring the same run state does not restore the business world as it was.

Before execution, recheck current identity permissions, approval validity, and applicable rules. For objects subject to concurrent modification, the business service must also check expected_version or an equivalent condition at the commit boundary. Reading a version in the application and then issuing an ordinary write still leaves a race between the two requests.

Approval determines whether execution is permitted; idempotency handles duplicate arrivals; conditional submission determines whether execution is still valid now. These solve different problems. The same approval may cover a retry of an unfinished operation, but cannot conceal changed parameters. The same idempotency key cannot prevent the first request from executing against stale business state.

Normal execution, resumption after human approval, subagent delegation, and automatic retries should all pass through the same submission adapter. Otherwise, even complete checks on the main path can be bypassed by a recovery path. If the downstream service lacks the required conditional-write support, the system must state its weaker guarantee and choose an effective constraint, such as serialization, an additional business lock, or human submission.

Data Movement Is Also a Permission Question

External documents, webpages, emails, and retrieved passages may contain instructions directed at the agent. They can serve as analytical material. Entering context does not grant them permission to change system rules, invoke sensitive tools, or send data outside the system.

Simon Willison describes a specific risk combination: a system that can read private data, encounter untrusted content, and communicate externally may be induced by injected content to exfiltrate data. Constraining combinations of data, capabilities, and destinations creates a more checkable boundary than merely asking the model to “ignore malicious instructions.” Prompt injection and tool combinations

Labeling tools “read” or “write” is insufficient. External search is usually a read operation, yet the query may carry internal information. Harvey's MCP policy engine addresses this directly by checking each invocation, verifying approved tool metadata, and considering the historical trace. Its fine-grained mutation capabilities that remain under development should not be treated as complete existing guarantees. Harvey: MCP policy engine

At the application layer, calls can be restricted by target system, data sensitivity, allowed fields, and outbound channel. Execution credentials stay in the adapter rather than being assembled by the model. Model-based detectors can supplement anomaly detection, but executable policies should still govern whether a class of data may leave the system. Tighter permissions, egress controls, and isolation also incur maintenance costs and must cover every actual tool path.

The timing of output review matters too. The Agents SDK runs a function tool's output guardrail after the real call. Blocking returned text cannot undo a remote operation that has already happened. Checks that serve as action prerequisites must precede the effect. Tool execution and post-call checks

6. Preserve Both Execution State and Business Outcomes for Long Tasks

Checkpoints and Operation Records Answer Different Questions

An execution checkpoint answers “where did execution stop, and what does the next step need?” A business operation record answers “what intent was sent to which system, and did it actually take effect?” They can live in the same database, but both meanings must be retained.

A long approval wait need not occupy an HTTP request or an in-memory coroutine. Persisting the proposal, decision, and wake-up condition lets the task lifecycle survive process restarts. Temporal's durable execution separates control flow from the process lifecycle; effects in external systems still depend on each Activity interface's own contract. Durable Execution, Idempotency and durable execution

WorkflowState: job_spec, step, artifact_refs, pending_decision
Operation: operation_id, proposal_id, target, frozen_args
           expected_version, provider_key, outcome, receipt
Evidence: source_versions, derived_artifacts, validation_records

The same operation_id should span approval, delivery, remote deduplication, and reconciliation. Record framework attempts separately from business intent: two attempts may belong to one refund, while two refunds with identical amounts may represent different business operations. Deduplicating solely by a parameter hash can suppress legitimate repeated business.

Resumption Can Re-enter Code and Invoke Tools Again

When LangGraph resumes an interrupt, it reruns the node from its beginning and retrieves the resume value at the corresponding interruption point. Python code before approval may therefore execute again. If that code performs an external write, adding a checkpointer does not automatically make it run once. interrupt resumption semantics

Separate proposal generation, approval, and submission, so that the approval node repeatedly reads the same frozen proposal and all submissions use the business adapter. This narrows the scope of re-entry, but a failure can still occur after a tool succeeds and before its result is persisted. Receiver-side deduplication and result queries remain necessary.

Distinguish three kinds of “replay”: reading an existing conversation and its results, calling the model again to generate an answer, and re-executing external operations. The first restores a view, the second introduces new candidate results, and the third creates real effects. A debugging interface's “rerun” should select its scope explicitly rather than defaulting to repeating every business action in the trace.

Context compaction should likewise retain source locations and versions, unresolved conflicts, approved objects, and operation states. Compressing “proposal P17 approved; remote outcome not yet confirmed” into “refund completed” directly breaks recovery semantics. Keep important fields in structured state; use summaries for reading and retrieval.

Reconcile Unknown Outcomes Instead of Relabeling Them as Failures

A network timeout only establishes that the client did not receive a definite result. The remote service may not have executed, may have succeeded, or may still be executing. Classifying all of these as FAILED makes it easy for the agent to try again with different parameters or a new operation identifier.

Stripe's error documentation explicitly treats some server errors as indeterminate outcomes. Under its API v1 contract, the same key can also keep returning a cached error. “Make another request” and “establish the business outcome” are therefore different tasks. Stripe: Advanced error handling

if operation.outcome == UNKNOWN:
    result = reconcile(operation, provider_contract)
    if result.proves_success:
        record(SUCCEEDED, result.receipt)
    elif result.proves_no_effect_and_no_attempt_in_flight:
        continue_under_same_operation_contract()
    else:
        suspend_further_submission()
        escalate_with_operation_record()

A single query returning no object does not always prove that no effect occurred: read lag, permission differences, or an earlier request still in flight may explain it. Every write tool must define what evidence confirms the outcome, how long to wait, and who continues reconciliation. If those capabilities are missing, narrow the scope of automatic execution accordingly.

Cross-system workflows may also complete only partially. Compensation should follow business rules and retain its own state and failure handling. For steps supporting conditional compensation, register the responsibility before the forward call so that an effect followed by a timeout does not leave compensation unrecorded. The compensation must still handle the case where the original effect never occurred. Temporal: designing compensating actions

Compensation does not restore a database snapshot. Sent notifications, resources already used by others, and completed trades may not return to a state in which they “never happened.” Task acceptance must specify permitted intermediate states, acceptable remediation outcomes, and takeover responsibility.

7. Define Evaluation and Release Policies through Domain Failures

Ask Experts to Explain Errors before Choosing Evaluators

“Answer quality is poor” does not guide a fix. The system needs to know whether it selected the wrong entity, cited evidence that did not support the claim, omitted an exception, mixed calculation bases, or executed an otherwise correct action without permission. These failures occur at different points and require different checks and repairs.

Domain experts should review complete task traces and deliverables and record why a result is unusable. Open-ended analysis of real errors, followed by a stable failure taxonomy and rubric, localizes problems more effectively than beginning with many abstract dimensions. Mark the first departure from requirements as well, avoiding repeated counts of downstream consequences from one upstream error. Hamel Husain and Shreya Shankar: error analysis

If experts must repeatedly read long free-form text from scratch, the product can improve the deliverable: show differences from a known template, separate facts, inferences, and unresolved items, and provide checkable fields and sources. Hamel Husain connects difficulty of evaluation with product design. The observation also applies to domain systems: a clear review path directly affects how much work is actually saved. “It's Hard to Eval” Is a Product Smell

for trace in representative_traces:
    review = domain_expert.review(trace, trace.delivered_artifact)
    record(review.verdict, review.reason, review.evidence)
    if review.failure is None: continue
    if review.failure.can_be_checked_reliably:
        add_versioned_check_and_regression_case(review.failure)
    else:
        define_review_or_escalation_rule(review.failure)

A regression set prevents known failures from returning. Independent, stable held-out sets and continually sampled new tasks are also needed. Repeatedly adjusting prompts against the same small group of examples gradually separates regression performance from real-world performance.

Model judges should also be validated on an uncontaminated, expert-labeled set, measuring false acceptance and false rejection separately. When most examples pass, an overall agreement rate can conceal missed rare but serious failures. Changes to the model, domain rules, or rubric require recalibration; scores under different standards are not directly comparable. Using LLM-as-a-Judge For Evaluation

Evaluate Outcomes, Execution, and Policy Compliance Separately

A refund task can leave an order in the correct final state while bypassing confirmation. An identical final database does not prove policy-compliant execution. The presence of a correct action in a trace does not prove that no impermissible action preceded it.

Retail policies in the τ-bench series require confirmation before database updates, yet the cancellation tool itself takes no confirmation credential. This gap tests whether an agent follows policy; it should not be copied directly as a production tool boundary. Retail policy, order cancellation tool

In the same pinned implementation, environment scoring compares final database states through replay. Action scoring looks for expected calls; it does not prove strict ordering or the absence of extra actions. The final reward combines only the components specified by the task, so score reports must retain the scope of the checks. Environment scoring, action scoring, reward composition

Production acceptance can separate artifact correctness, operation prerequisites, permitted action order, data leakage, and final business-state consistency. Enforce violations before execution wherever possible. Offline scoring finds omissions but cannot repair effects that have already occurred.

Component evaluation helps localize errors. QueryGPT tests intent, table selection, and SQL separately, supplying correct upstream inputs to downstream components. This identifies where a failure originates. Full end-to-end tasks must remain, however, to reveal problems that appear when components connect. QueryGPT: evaluation design

Passing components do not imply a passing user task. A correctly retrieved clause can still be misinterpreted, and executable SQL need not use the correct business basis. Evaluation reports should preserve check scope rather than promote a local proxy metric into a complete business conclusion.

Track Both Automatic Coverage and Errors after Release

A system can lower errors in automatic outputs by sending many tasks to humans, or raise automatic completion by relaxing checks. Reporting only one metric makes progress difficult to assess.

The risk–coverage perspective from selective prediction can help organize these operational metrics. This is an evaluation approach; it does not directly confer statistical guarantees on an agent. SelectiveNet

automatic_coverage = automatically_released / eligible_tasks
released_error_rate = incorrect_releases / automatically_released
cost_per_accepted_task = total_operating_cost / accepted_tasks

For example, of 1,000 tasks within a predefined scope, suppose 600 are released automatically and review finds 30 errors among them. Coverage is 60%, and the error rate after release is 5%. These numbers only illustrate the calculation. If production review samples only some tasks, report the sampling method, estimation uncertainty, and task distribution. Model-reported confidence is not the actual error rate.

Errors also differ in consequence. Omitting a background citation and submitting the wrong amount to a business system should not simply count as two equivalent “failed tasks.” Track task categories and impacts separately, apply stricter release conditions to high-impact actions, and record human handoff volume, queueing time, and review quality.

Define the eligible-task denominator before evaluation. Removing difficult tasks after failures does not establish an improved success rate. Scope can change, but report scope changes separately from changes in system capability.

8. Evolve from a Constrained Workflow

Expand Execution Authority According to Action Risk

Start with a frequent task whose inputs have clear boundaries and whose results can be checked. Reproduce real examples offline, then run in shadow mode, then provide human-reviewable drafts or proposals. Only when evidence supports it should qualifying actions be submitted automatically.

Shadow runs can measure retrieval, analysis, and proposal quality, but cannot adequately validate concurrency, timeouts, and partial completion in real submissions. The submission path also needs controlled-environment validation and a limited rollout. Read-only modes still require permission, privacy, and cost constraints.

Configure autonomy per action. The same system can collect material and calculate automatically, ask a human to approve changes, and let a business service perform submission. A single global switch need not decide whether the entire agent is “automatic” or “manual.”

Human takeover also needs a handoff format: the current task, completed work, evidence and conflicts, frozen proposals, actions known to have occurred, actions still uncertain, and recommended next steps. A popup saying only “human help required” transfers the full cost of reconstructing context to the operator.

A handoff must also establish who has authority to continue acting. Once a human starts handling the same object, the agent should stop the corresponding automatic actions. Returning control to automation requires updated permissions and business versions. Handoff transfers control as well as information; otherwise, the human and agent may both write from the same stale state.

Calculate the Total Cost per Acceptable Result

Model-call charges are only one part of operating cost. Repeated retrieval, ineffective repairs, tool latency, human review, and remediation all affect whether a task is worth automating. Additional checks may reduce incidents while slowing every task. Additional agents may shorten a local step while increasing coordination overhead.

Stripe Minions run faster checks after local edits, followed by more expensive full validation, and limit repair rounds. This suggests an ordering principle: begin with cheap, decisive checks that prevent further wasted work, then perform semantic review or full validation. Stripe Minions: validation and repair

Execution budgets should also distinguish progress from repeated attempts. Repeatedly finding the same missing data or measurement conflict is not necessarily helped by more search rounds. The system can track new evidence, resolved issues, and remaining blockers to trigger a clarification, a different tool, or takeover. Stopping conditions should amount to more than a very large maximum step count.

Release Changes as a Traceable Set of Versions

A domain system's behavior depends jointly on the model, prompts, tool schemas, retrieval indexes, data snapshots, and business rules. Evaluation rubrics and judges also need versions. Recording only the model version cannot explain why a task was allowed today but rejected yesterday.

Store a version manifest for each run and connect releases to evaluation results. Pending proposals also need a compatibility policy: can old execution code still resume? Have tool parameters changed meaning? Is an approval under the old rules still usable? Check code compatibility and business validity separately. Being able to deserialize old state does not mean an old action should still be submitted.

Log retention should support reproduction and diagnosis while respecting data scope, redaction, and retention periods. For sensitive tasks, controlled source references, versions, and necessary evidence may be more appropriate than endlessly copying complete material. Observability itself needs permission design.

After adding a mechanism, keep checking whether it reduces the intended errors. If a complex planner does not raise the proportion of acceptable tasks, or another reviewer agent merely repeats existing checks, consider simplifying. Engineering experience includes recognizing which mechanisms no longer justify their maintenance.

Evidence the Design Should Preserve

These objects need not all begin as independent services. A constrained workflow, one database, and clear business adapters can often support a first version. Add mechanisms in response to actual failures as task scope expands. The system can grow more complex over time; the meaning of a task, the object of approval, and the state of an outcome should be explicit from the beginning.