Measuring Revisions Before Adopting Them

Revisions to an instruction file survive on the author's impression. Without a place to measure whether a revision helped, the ones that made things worse survive too — and become the foundation the next revision builds on.

Problem

Instruction files and procedure documents grow, and they grow the same way. A result comes back wrong, a likely cause suggests itself, a line is added. The next result is wrong too, so another line is added.

There is no accept-or-reject decision on any of it. A line survives because its author felt it should have helped.

Three things make that absence hard to notice.

First, the effect of a revision is invisible unless the same input is run twice. What gets run after a revision is almost always the next request, not the previous one. Nothing is compared, so "better" can only be asserted from impression.

Second, output varies every time. One success is not evidence that a revision worked; the second run might have succeeded without it. A single success or failure can always be misread as the revision's effect.

Third, revisions accumulate. A line that made things worse becomes the foundation for every revision that follows. Nobody notices the foundation is tilted, and fine-tuning continues on top of it.

The document grows. Nobody can say which lines in it are doing any work. Pruning becomes impossible for lack of grounds — not one line has ever been shown to be ineffective.

Context

Apply when:

• The instruction document keeps growing — revisions have accumulated and the text has been rewritten many times
• The same kind of work repeats — past requests can be set aside and reused for judging
• You can decide success yourself — tests pass, output matches an expected shape, a specified procedure was followed
• A bad revision is expensive — the document affects a lot of work, so damage spreads

Do not apply when:

• The work is one-time. There is nothing to measure against later
• The instructions are short enough to see whole. A tilted foundation has nowhere to hide
• Success cannot be judged. This pattern has no place in matters of taste or work with no definable correct answer. You cannot gate what you cannot measure
• The failure can be caught mechanically. That is the job of a mechanical gate, not of an instruction revision

Boundaries with adjacent patterns:

• Failure as Institutional Memory — the closest. That pattern is a filter on what to write when recording a failure, and it applies before writing. This one asks how you judge whether it helped, and applies after. Where that page notes of its own core claim that "this is a judgment, not a measurement," it is naming exactly the gap this pattern tries to fill
• Resident Context Inventory — the trigger to cut differs. There the trigger is total size; here it is score. ⚠ The total size of a document and the size of a single revision are different things — conflating them turns "I made it shorter" into "I measured it"
• Making the Verification Target Explicit — that pattern concerns what to verify within a single piece of work. This one compares versions of the instruction document. The difference is whether the object under verification is the artifact or the instructions
• Separating Mechanical Gates from Semantic Gates — the gate applies to different things. There it applies to artifacts (code, documents); here it applies to the instructions themselves
• Auditing Past Decisions — a discarded revision and its reason are a kind of decision record. That pattern keeps records for later review; this one feeds them back into the next revision

Solution

Do not adopt a revision at the moment you write it. Hold it as a candidate, measure, then decide.

There are four parts.

1. Do not look at the judging tasks while writing the revision. This is the crux. A revision can be written to fit the cases you looked at. Improving on cases you consulted is not evidence that the revision worked — you wrote it with the answers in view. The cases used for judging must be ones you had not seen.

2. The baseline is the immediately preceding version. No absolute standard of quality is needed. The only question is whether it is better than before. Framed this way, the comparison keeps its shape no matter how far the document grows.

3. Discard anything that does not beat the baseline. Make discarding the default and put the burden of proof on keeping. That asymmetry is where this pattern does its work. Allowing "it probably isn't hurting, so keep it" returns you to having no decision at all.

4. Keep the discarded revision and its reason. Without that record, six months later you will conceive the same revision, pay the same cost, and arrive at the same conclusion. The record is negative feedback for the next attempt.

Two supporting disciplines:

• Cap how much changes at once. A large rewrite makes it impossible to isolate which change was at fault when the result does not beat the baseline. The cap exists for resolution of judgment. ⚠ This is about the size of one revision, not the total size of the document
• Draw material from multiple runs. A revision fitted to a single failure works only for that failure. Fold back into the procedure only the failures that recurred across runs

Why it works

The reason is not a tool feature. It is a structural asymmetry in instruction documents.

Addition has a motive: the moment a failure is visible, there is a reason to write a line. Deletion and retraction have no motive — nobody notices that "this would have worked without that line," because an ineffective line never shows up as a failure.

Left alone, that asymmetry always accumulates on one side. The document grows monotonically, and everything else in it is diluted by that much. A place to measure is a device that charges a cost only to the addition side, cancelling the asymmetry. The goal is to stop "it feels better" from passing, not to obtain a precise measurement.

⚠ This catalog has not measured the effect of running this pattern by hand. How far the evidence reaches is set out in the practice section.

Trade-offs

Maintaining the held-aside task set costs something. Cases must be gathered, kept, and replaced when they go stale. That can be more work than the document itself.

Revision slows down. If every single line requires several runs, small revisions stop being worth it. That is not necessarily a loss — stopping revisions that are not worth it is the point. But the good habit of fixing things as you notice them stops along with it, so decide in advance where small revisions are exempt from measurement.

A small held-aside set lets through revisions that only fit it. A revision passes judgment and then improves nothing outside the cases used. The number and diversity of those cases is the credibility of the judgment.

Qualities you cannot measure get dropped. Readability, clarity for whoever inherits the document, future maintainability: none of these show up in a score. Do not read "did not show up" as "worthless." This pattern can say only that improvement was not demonstrated, never that something has no value.

Variance gets mistaken for effect. Deciding on a single run passes the model's non-determinism straight into the verdict. Either run several times, or accept only differences larger than the observed spread.

The lightest version barely counts as measurement. Assembling the whole apparatus is heavy. Simply running the same request once before you change anything, and keeping the result, gives you something to compare against. The asymmetry does not disappear, but it bends a little.

GitHub Copilot in Practice

The weak form — "try it, look at the result, revise" — exists in primary sources on both sides. Everything below was fetched as raw body text with curl on 2026-07-27 and searched in full. Byte counts record that the whole document was retrieved; they are not proof of identity.

VS Code surface — try a prompt file and refine it Official

Source: Prompt files (80,378 bytes; HTML searched after tag removal).

Use the editor play button to test your prompts and refine them based on the results.

This option is useful for quickly testing and iterating on your prompt files.

github.com surface — duplicate a custom agent to try updates Official

Source: Testing and releasing custom agents (4,876 bytes).

You can create a net-new profile or duplicate an existing profile to test potential updates.

Continue making changes and testing your custom agent as needed until you are satisfied with its performance.

This is where the difference from this pattern shows. What the documentation states is "until you are satisfied," and what counts as satisfied is left to the author. There is no comparison against the previous version and no condition for discarding a revision that falls short.

The Anthropic counterpart — the closest single sentence Official

Source: Claude Code Best practices (39,582 bytes).

Treat CLAUDE.md like code: review it when things go wrong, prune it regularly, and test changes by observing whether Claude's behavior actually shifts.

Within the range checked, this is the closest statement that testing an instruction document's revisions is worthwhile. But it says to observe whether behavior shifted; it does not say to measure whether the shift was an improvement against a held-aside set of cases.

The same source is explicit about closing the verification loop on artifacts:

Claude does the work, runs the check, reads the result, and iterates until the check passes.

⚠ That is verification of an artifact, not of an instruction revision. Do not confuse the objects (the boundary is Separating Mechanical Gates from Semantic Gates).

The vocabulary of held-aside sets appears, in a different context Official

Source: Anthropic, Create strong empirical evaluations (137,547 bytes).

Our sentiment analysis model should achieve an F1 score of at least 0.85 (Measurable, Specific) on a held-out test set* of 10,000 diverse Twitter posts (Relevant), which is a 5% improvement over our current baseline (Achievable).

Design evals that mirror your real-world task distribution. Don't forget to factor in edge cases!

⚠ The scope is different. What is being measured is the performance of an application, not the acceptance of a revision to an instruction document. Both "held-out test set" and "baseline" appear, which makes this tempting to cite as support, but the same words are not being used for the same purpose. This pattern is what that thinking looks like when turned on instruction revisions.

Absence Confirmed — the strong form appears nowhere in the range checked

In addition to the four sources above, the full body text of seven more was searched: repository custom instructions on github.com (22,848 bytes), your first prompt file (3,087 bytes), reviewing output (2,472 bytes), VS Code custom instructions (114,884 bytes) and agents overview (64,541 bytes), and Anthropic's Agent Skills best practices (41,420 bytes) — eleven documents in total. Retrieval and verification were performed by a different agent from the one that ingested the source material.

What was searched for	Principal search terms	Result
Compare against the previous version on a held-aside set, adopt only if it wins	held-out test set / baseline / earlier version	Not found (the terms occur, but the object is not an instruction revision)
Keep discarded revisions and their reasons for reuse	reject / rejected / discard / negative feedback	Not found (the object is agent output, or a rejected proposal)
Cap the size of a single revision	bounded / limit / one change	Not found
Extract failures common to several runs	multiple / common / diverse / edge cases	Not found (these concern eval diversity and testing across models)
Pay the cost at revision time, require no extra calls at deployment	static / additional model / deploy	Not found

"Absent from primary sources" is not "does not exist." It may lie outside the range checked. The claim here is only that these eleven documents do not contain it.

The published size guidance is about totals, not about revisions Official

Source: Adding repository custom instructions (github.com surface, 22,848 bytes).

Instructions must be no longer than 2 pages.

⚠ That is a cap on the size of the document, not the "cap on one revision" this pattern describes. Total size is the subject of Resident Context Inventory and Instruction Layering and Scoping. Do not let one word, "limit," merge the two.

Research — only one source in the range checked measures the strong form

arXiv:2605.23904 (v2, updated 2026-05-25; PDF of 879,864 bytes, retrieved 2026-07-27) proposes a method that revises a procedure document iteratively as external state of a frozen model, and places all of the following as stabilizing devices: an acceptance gate on a held-aside set, reuse of discarded revisions, a cap on revision size, and extraction from multiple runs. Its authors include Microsoft and several universities.

⚠ Keep all three limits attached.

• This is not vendor product documentation. It is an academic preprint, and the original states nothing about peer review. Microsoft being among the authors does not make it an official description of Copilot — the paper describes none of the VS Code, github.com, Copilot CLI, or cloud agent surfaces
• What was measured is the performance of an automated optimizer, not revisions made by hand. "A person who measures and revises their own instruction file will get better results" lies outside this paper's scope
• This catalog does not quote the paper's improvement figures. They compare automated optimizers against each other, not human revision. Only the direction is taken

Also Known As

Name	Source
Measuring Revisions Before Adopting Them	Descriptive — this catalog's descriptive name; it has no primary-source use
Prompt optimization	[Common] — used mainly in the context of automation, not for the practice of a person deciding whether to accept a revision
Regression testing	[Common] — an established term in the context of code. No primary source has been confirmed applying it to instruction documents

The research this page draws on gives its method a proper name, but this catalog does not use it as the name of the pattern. Naming a type after a method implies that using the method makes the type hold. In reality you can satisfy this pattern without the method, and you can use the method while failing to keep the judging cases separate.

Related

• Failure as Institutional Memory — the filter before writing is that pattern; the gate after writing is this one. Where it reserves that "this is a judgment, not a measurement" is where this pattern starts
• Resident Context Inventory — the trigger to cut is total size there and score here. ⚠ Do not discuss document totals and single-revision size under one word, "limit"
• Making the Verification Target Explicit — the verification target within one piece of work is that pattern; comparison between versions of the instruction document is this one
• Separating Mechanical Gates from Semantic Gates — the gate applies to artifacts there and to the instructions themselves here. Do not try to fix with an instruction revision what a gate on the artifact would catch
• Auditing Past Decisions — a discarded revision and its reason are a decision record, and become a target for later review
• One Authoritative Source for Instructions — when the authoritative copy is split, which version you measured is undefined. It is a precondition for measuring at all