Measuring Outcomes by What Survives

"Did you accept it?" is weak evidence of an outcome. Acceptance is a judgment made at the moment of receiving, and if measurement stops there, changes you later deleted keep counting as successes. Counting what is still there at commit closes that mistake.

Problem

We count whether an agent's output was any good by whether we accepted it. Take the diff and it succeeded; throw it away and it failed. That is easy to count and settles the instant you act, which makes it a convenient metric.

But acceptance is only a judgment made at the moment of receiving. All you have in hand at that moment is the diff itself and the few dozen seconds of understanding you got from reading it. You have not run it, and you have not looked at the callers next door. Nothing in the acceptance record says what became of the change ten minutes later. Rewritten, deleted, or passed through untouched — the record cannot tell them apart.

Two errors then occur at once. A change you accepted and immediately rewrote keeps counting as a success. Conversely, a suggestion you declined out of unease that was in fact correct stays recorded as a failure and is never revisited. Both arise because the metric stops at the moment of receipt.

Worse, acceptance is a quantity the receiving side can steer. A diff that is short, narrow, and traces existing style reads fast and gets taken. A change that goes further takes longer to read and gets taken less often. Raising the acceptance rate does not require producing better changes. That loophole is open to the producing side and to us, in how we frame the request.

Context

When it applies:

• You are doing the same kind of work repeatedly — when you want a trend rather than the quality of one attempt. Survival takes elapsed time to count, so without repetition the denominator never accumulates
• You want to compare a change in how you delegate, or in model choice — when you need to say what actually improved between before and after
• Outcomes land as commits — what you count has to be in version control

When it does not:

• One-off work. Survival takes time to settle. It cannot inform the decision of whether to accept this diff right now
• Outcomes that never reach a commit. The conclusion of an investigation, a review comment, a design decision — none of these give survival anything to count
• When trying things out is the point. An exploratory implementation written to be thrown away has not failed by disappearing

Boundaries with adjacent patterns:

• Diff Review and Incremental Adoption — how you receive a diff is that pattern; what you count afterwards is this one. They are adjacent on the time axis, and this pattern's question survives even for diffs adopted in fine grain
• Making the Verification Target Explicit — handing over the standard for "fixed" as a contract beforehand is that pattern. This one is about what you count afterwards, and holds whether or not such a contract exists
• Measuring Revisions Before Adopting Them — using measurements to accept or reject a revision to your instructions is that pattern. This one is the what to measure side, and feeds into it

Solution

Why it works: acceptance and survival are decided by different parties.

Acceptance is decided by us, mid-read. The material is the diff and the understanding we got from it. That judgment sits close to the producing side — write it to read easily and it goes through more often. As a metric, acceptance carries exactly that much room to be steered.

Survival is decided by us as we worked afterwards. We ran it and it failed; it did not mesh with the callers next door; a reviewer objected; or we simply reread it and found it unnecessary. None of that material was in hand at the moment of acceptance. Survival is a verdict rendered by material that arrived later, and it sits far from the producing side. Being far, it cannot be traced through.

So what this pattern does is move the metric toward the side that is harder to steer. It does not discard acceptance. Acceptance remains fast and usable on the spot. But watching only acceptance means that the acceptance rate went up and nothing actually got better is a state the metric cannot detect. Place survival alongside it, and you notice when the two point in opposite directions.

⚠ State plainly that this is this catalog's judgment. Ranking survival above acceptance is written nowhere in the primary sources we checked — the source material places both side by side as quality metrics (see the practice section). The ranking is a judgment this catalog draws, and is not Official. What the primary source supports is the definitions and the measurements, plus the framing that treats reaching a durable result as the goal of evaluation.

Keep the counting to one thing. Count only whether it is still there at commit. Version control is everywhere and can be recounted later. There is also a way to count over a window of "so many minutes after acceptance," but that requires a record of the editing operations themselves, which depends on instrumentation in the tool (there is a real example in the practice section). This pattern holds using only the side that does not depend on it.

Trade-offs

Benefits

• It moves the metric to a side that cannot be traced through. Unlike acceptance, which rewards being easy to read, survival is decided by material that arrived later. That difference is what matters when comparing changes in how you delegate
• You notice when acceptance and survival diverge. Watching only one of them, that state looks like improvement. Placing them side by side is the detection

Costs

• It is slow. Survival does not settle until time has passed. It cannot inform whether to accept this diff now; it is only a metric for trends
• Having survived is not the same as being right. A change nobody ever reread also survives. Survival says only "was not deleted later," and is not proof of quality. Make the survival rate a target and it becomes the next thing to be steered

GitHub Copilot in Practice

VS Code — a measurement reported in three metrics

Source: MAI-Code-1-Flash: early results from real developer workflows (retrieved: 2026-07-30) Official. The surface is VS Code — the population is developers who selected each model in VS Code Copilot Chat, and there is no description covering github.com, Copilot CLI, or cloud agent.

What these numbers do and do not say

The baseline is the article's own vendor's new model. It is a self-comparison with that model placed at 0%, and absolute values are not published. All figures are relative to that baseline, and they are medians, not means. The measurement period is 2026-06-02 to 2026-07-24.

The article measures quality with three metrics Official:

Metric	Definition
Accept rate	The share of AI-generated suggestions the user accepted
Code survival rate	The share of AI-generated code still present after 10 minutes of active editing
Commit survival rate	The share of AI-generated code still present at the git commit

⚠ The ten-minute clock does not start at acceptance. The original definition is ten minutes of active editing — bounded by the amount of editing, not by elapsed time. That layer requires a record of editing operations, so version control alone cannot reproduce it. This is why this pattern places its core at the commit point.

The article's framing of what it measures is itself citable as Official — it looks at the quality of the generated code and how efficiently developers reached a durable result. Putting "a durable result" up as the goal is the article's own phrasing.

A row where acceptance and survival point in opposite directions sits in that same table — Kimi K2.7 Code has an accept rate 6% below baseline, while its code survival rate is 4% higher and its commit survival rate 6% higher.

⚠ The article does not leave that row unjudged. Despite the lower accept rate, it rates the two models including this row as ahead on quality (noting their higher consumption as the cost) Official. So the article itself, in its net judgment of that row, weights survival above acceptance. ⚠ But that is a net judgment about one row, not a general ranking that "survival beats acceptance as a metric." The article does not write that generalization, and says nothing about why the two diverge. The generalization is this catalog's judgment.

⚠ The same article carries a second table over a different population (a comparison under automatic model assignment rather than user selection). Because the metrics and the basis of comparison differ, figures from the two tables must not be mixed or combined — there is a place where the same model's consumption looks lower than baseline in one and higher in the other. This page cites only the first table.

The other vendor — no description that counts survival

We retrieved the full raw HTML of seven Anthropic primary sources and searched them, and found zero metrics that count the survival of a deliverable (all retrieved 2026-07-30). The documents covered were the Claude Code analytics API reference, an article on scaling agentic coding, research on how Claude Code is used in practice, an article on code review, economic-index research on the impact on software development, research on measuring agent autonomy, and Claude Code best practices. The search terms were acceptance rate / accept / retain / retention / survive / revert / commit / still, plus the stem surviv.

⚠ The zero applies only to the survival side. The acceptance side is there — the analytics API carries per-tool accepted/rejected counts and the formula for an acceptance rate Official. Descriptions that use commit counts, PR counts, and passing test suites as hard evidence of an outcome are also there Official. What is absent is a definition that follows an accepted diff to see what became of it. The retain hits concern data retention periods and retaining a plan, not the survival of a deliverable.

What is absent is the description, not the capability. We checked the range of these seven documents; it may exist in other documents or on other product surfaces.

Putting it into practice

A tool measuring all three layers is a matter of aggregate telemetry on the VS Code surface. To do the equivalent yourself, use version control. Whether something is still there at commit can be counted with any tool — put the size of the diff the agent produced next to the size of the diff that actually landed, per branch. Counting the amount at the commit rather than immediately after acceptance is what this pattern is.

Also Known As

Name	Source
Commit survival rate / Code survival rate	Primary — the names the VS Code surface blog gives its metrics (source in the practice section). ⚠ These name the metrics, not this pattern (a practice)
Accept rate	Primary — the name the same source gives the paired metric. It is the metric this pattern argues against relying on alone (source in the practice section)
Measuring Outcomes by What Survives	Descriptive — this catalog's descriptive name; it has no source in the primary literature

Do not use the name of a metric as the name of a practice. commit survival rate names a quantity measured, while the choice of how to measure — to stop measuring at the moment of receipt, or to count after time has passed — has no name. Borrow the quantity's name for the choice, and in any environment lacking a tool that computes that quantity, the choice itself goes unnoticed.

Related

• Diff Review and Incremental Adoption — adjacent on the time axis. That pattern covers the act of adopting a diff; this one covers the amount that remained afterwards. Adopting in fine grain does not dissolve this pattern's question
• Making the Verification Target Explicit — the contract beforehand is that pattern, the counting afterwards is this one. Even with a contract handed over, whether it remained has to be counted separately
• Measuring Revisions Before Adopting Them — this pattern is what to measure; that one is what to accept or reject based on the measurement. This pattern's output is that pattern's input
• Recording by Provenance — this page is itself written in that shape. The three definitions and the figures are Official; ranking survival above acceptance is this catalog's judgment, and the two are not placed under the same label
• The Break-Even Point of Delegation — the same table in the same source is also used on the consumption side. That the efficiency metric splits into "generated tokens per turn" and "turns to reach a commit" is handled there
• Read next Diff Review and Incremental Adoption — once you have decided what to count, look at the act of receiving itself