The 12 Rules of Harness Engineering: Cassie Kozyrkov's Guide to Supervising AI Agents

Contents27 sections

AI agents can generate code in a very short time, so the bottleneck shifts from typing speed to intent, context, verification, and maintenance. Cassie Kozyrkov uses Harness Engineering to describe the engineering practice of designing the environment, boundaries, tools, and feedback loops around an agent. A harness is not a longer prompt; it is a system that lets an agent execute, observe, correct mistakes, and escalate to a human when judgment is required.

The Governance Problem Harness Engineering Addresses

Every part of an instruction that remains unspecified becomes a decision the agent fills in on its own. When assumptions are not recorded and outputs are not checked, short-term speed turns into long-term trust debt: the cost of unaudited assumptions and the rework that follows.

The approach separates responsibilities into four layers:

TEXT

Human: intent, priorities, acceptance criteria, and final judgment
  │
  ▼
Harness: context, boundaries, tools, plans, tests, observability, and escalation rules
  │
  ▼
Agent: code, tests, documentation, fixes, review responses, and repeated execution
  │
  ▼
Machine feedback: test results, logs, metrics, traces, UI state, and user feedback

Engineering output increasingly shifts from lines of code to constraint systems and feedback systems; agent autonomy comes from clear boundaries, not ambiguous authorization.

The Complete Map of the Twelve Rules

No.	Rule	Purpose
1	Humans steer, agents execute	Humans own intent and judgment; agents own execution details
2	Map over manual	Use a short entry point that leads to deeper material on demand
3	Knowledge outside context effectively does not exist	Put critical decisions into versioned artifacts the agent can access
4	Code legibility	Make code parsable, predictable, and maintainable for the next agent run
5	Centralize boundaries, decentralize autonomy	Enforce shared architectural and security invariants while allowing local implementation freedom
6	Application legibility	Make UI, logs, metrics, and traces queryable feedback for the agent
7	Treat plans as first-class citizens	Make plans, progress, and decision logs auditable state
8	Correction is cheap; waiting is expensive	Use fast generation and verification loops instead of unnecessary waiting
9	Continuous garbage collection	Continuously remove drift, redundancy, and AI slop
10	Encode taste into the system	Turn implicit preferences into lint, structural tests, and tool rules
11	Prefer boring, composable technology	Prefer stable, transparent technology that agents can model easily
12	Escalate to humans only when judgment is needed	Keep priority, user feedback, and outcome-value judgment with humans

From Intent to Context: Who Owns What

Rule 1: Humans steer, agents execute

Humans decide what to build, why it matters, what counts as done, and which risks are unacceptable. Agents turn those requirements into code, tests, documentation, CI configuration, and repair actions.

When an agent is stuck, first inspect what the environment lacks: a tool, a document, an assertion, permission, or observability. Humans improve the harness, and the agent continues the code-level repair. A failure should make the environment more complete instead of pulling humans back into line-by-line coding.

Rule 2: Map over manual

An oversized document that combines every rule, background detail, and exception consumes the context needed for the task, code, and tests. The entry file should be a map: it defines the project boundary, lists sources of truth, and points to domain documents, run commands, and verification scripts.

Progressive disclosure can divide context into layers: a short, stable AGENTS.md or project entry point; domain-specific documents; and reference material loaded only when the task needs it. The map should stay short, stable, and verifiable, while deeper documents need clear paths and ownership.

Rule 3: Knowledge outside context effectively does not exist

An architectural agreement that exists only in Slack, Google Docs, or a senior engineer's memory cannot be used reliably by the next agent run. Critical intent, data schemas, directory conventions, acceptance criteria, known limitations, and technical debt should become repository-local Markdown, code, configuration, or executable plans under version control.

The same rule applies to cross-session state. Writing current progress, completed work, failure reasons, and next steps to files lets a new agent resume after a context reset without guessing what happened in the previous session.

Making Code and Applications Legible and Verifiable

Rule 4: Code legibility

The audience for code legibility includes the next agent run. Clear module boundaries, stable APIs, explicit data contracts, predictable naming, and narrow changes are easier to maintain continuously than implicit conventions.

This does not require every implementation to match human aesthetic preferences. It requires behavior to be explainable, tests to provide coverage, and failures to be attributable. Complex abstractions, hidden side effects, and opaque dependencies enlarge the agent's reasoning space and increase the chance of reproducing a bad pattern.

Rule 5: Centralize boundaries, decentralize autonomy

Architecture layers, data boundaries, permissions, error handling, logging formats, and critical performance constraints should be defined centrally and enforced mechanically through lint, structural tests, or CI. Within those boundaries, agents can choose local function decomposition, algorithms, and implementation style.

This division focuses human attention on a small set of non-negotiable invariants while preserving agent throughput. Micromanaging every line becomes a human bottleneck; abandoning boundaries entirely lets the codebase fork rapidly.

Rule 6: Application legibility

Static code review cannot cover every UI and runtime problem. Agents need to start isolated application instances, read DOM snapshots or screenshots, query structured logs, metrics, and traces, and reproduce user paths through tests or tools.

Legibility is therefore a runtime property. The application should tell the agent what happened, which constraint failed, and in which request or interaction step the failure occurred. Independent worktrees, ephemeral observability environments, and stable query interfaces can turn QA work that humans once performed manually into executable machine feedback.

Making Long Tasks Recoverable and Correctable

Rule 7: Treat plans as first-class citizens

Long tasks cannot rely only on conversation history. An execution plan should record goals, scope, dependencies, acceptance conditions, current state, decisions, and known technical debt, and it should live under version control with the code.

A plan can drive an incremental loop: the agent selects one unfinished feature with a clear priority, implements it, runs verification, and updates the plan state. The plan then serves as task decomposition, cross-session handoff, and audit evidence at the same time.

Rule 8: Correction is cheap; waiting is expensive

When agent generation is much faster than human review, making every change wait for item-by-item approval turns people into a queue. Short branches, automated checks, quick reruns, and small fixes are often more effective than waiting a long time for one perfect attempt.

This rule depends on risk stratification. Low-risk code can be iterated quickly; actions involving data destruction, permissions, payments, production releases, or compliance should retain hard gates and human confirmation. High throughput does not remove governance; it moves governance to boundaries and feedback.

Rule 9: Continuous garbage collection

Agents reproduce patterns already present in a repository, including obsolete, duplicated, or low-quality patterns. A weekly cleanup easily lets problems accumulate; a more stable approach schedules continuous scans so small refactors, documentation fixes, and rule upgrades enter the feedback loop quickly.

Garbage collection can inspect dead code, duplicate utilities, stale documentation, architectural drift, failed retry patterns, and test-coverage gaps. Handling one verifiable, narrow deviation at a time reduces review cost and keeps new agents from learning bad patterns.

Encoding Taste and Technology Choices in the System

Rule 10: Encode taste into the system

Team taste often hides in review comments, refactoring habits, and verbal agreements. A sentence such as “keep it clean” does not give an agent a stable rule; the preference needs to become a detectable invariant, such as a naming constraint, file-size limit, dependency direction, structured logging rule, or boundary schema.

Lint error messages are part of the harness too. If an error explains what was violated, why it is disallowed, and how to fix it, the agent can turn one failure into context for the next run. Repeated review feedback should trigger a decision about whether to promote it into a tool rule.

Rule 11: Prefer boring, composable technology

Here, boring means technology with stable APIs, adequate documentation, transparent boundaries, a mature ecosystem, and easy composition. Such technology usually has more training data and more reference implementations, making it easier for an agent to build a reliable working model.

Technology selection still has to follow real constraints; popularity is not proof of quality. A small, transparent local implementation can sometimes be easier to verify than an opaque third-party black box, but the decision must account for maintenance cost, security updates, and long-term compatibility.

Reserve Human Escalation for Judgment

Rule 12: Escalate to humans only when judgment is needed

Human intervention should focus on judgment agents cannot replace: setting priorities, interpreting user feedback and real-world experience, defining acceptance criteria, deciding whether the final outcome has value, and approving high-risk actions. Other execution details should remain with the agent inside the harness boundaries as much as possible.

This rule changes how failures are handled. After finding a bug, humans first decide whether the missing piece is a product decision or an environment capability. If the gap is only a tool, test, or context, the harness is improved and the agent still writes and verifies the fix.

An Implementation Checklist for a Minimal Harness

1. Turn goals into acceptance criteria

Rewrite a product wish as verifiable behavior, boundaries, inputs, outputs, performance targets, and failure conditions. For a vague task such as “build a dashboard,” specify at least the users, data sources, time range, interactions, empty states, permissions, and evidence of completion.

2. Establish a short entry point and deep sources of truth

Keep a small project entry point that states the directory map, startup method, verification commands, and document index. Split architecture, domain rules, design standards, and security requirements into independently maintainable files, and link to them from the entry point.

3. Mechanize important rules

Put dependency direction, schemas, naming, file boundaries, tests, and security constraints into lint, structural tests, CI, or runtime assertions first. A rule that can be checked automatically should not live only in review habits.

4. Expose application state

Let the agent start the application in isolation and access logs, metrics, traces, DOM, screenshots, and test results. Each feedback item should contain enough object, time, and failure-location information for the next run to diagnose it.

5. Externalize plans and progress

Write task lists, current state, decision records, failure reasons, and known technical debt to versioned files. Advance one verifiable small goal per loop and update the state after verification.

6. Build a continuous correction loop

Have the agent self-check, request independent review, process feedback, and verify repeatedly; schedule periodic work to remove drift and redundancy as well. Re-examine the harness after a model upgrade, remove scaffolding that is no longer load-bearing, and fill capability gaps exposed by the new model.

Boundaries and Decision Criteria

The value of a harness grows with task complexity, duration, agent throughput, and the cost of failure. A simple one-off script may need only a clear prompt and basic tests; long-running, collaborative, production-facing, or tool-rich systems need more complete context, plans, observability, permissions, and escalation rules.

OpenAI's zero-manual-code Codex experiment and Anthropic's long-running harness experiments show that environment design can materially change agent outcomes, but they used specific models, tools, and repository structures. Their experimental numbers should not be extrapolated directly to every team. A harness also adds call cost, latency, and maintenance work, so real tasks and regression evaluations must determine which components remain useful.

A harness can be judged by four outcomes: whether the agent finds the right context, executes within boundaries, uses machine feedback to find errors, and wakes a human only when judgment is actually needed. Reliability comes from the combination of intent, constraints, observability, and continuous correction, not from one longer rule file.

REFERENCES