Three Criteria for Evaluating Harness Engineering Benefits
What you’ll learn
- How to separate observed changes after harness adoption from unmeasured ROI claims
- A repeatability, specificity, and verifiability framework for deciding what becomes a rule
- How to define ROI measures and the conditions for not adding a harness rule
Move Only Repeated Problems into the Harness
A problem belongs in a harness (the rules, procedures, and validation surrounding AI work) when it is repeatable, specific, and verifiable. On this site, moving premises into shared rules and fixing review criteria produced observable workflow changes. Time and error counts were not measured, so the ROI discussion separates those observations from unmeasured benefits.
By the end of this article, you will have practical criteria for answering “Which recurring problems belong in a harness, and how should their effect be measured?” in your own context.
Decide on Harness Investment by Recurrence, Specificity, and Verifiability
I did not measure task duration, correction count, or review findings under the same conditions before and after introducing the harness. This article therefore does not claim a percentage reduction or a recovered investment.
What the repository does show is a change in method. Recurring assumptions—English first-person voice, preservation of the existing UI, and slug (the identifier used in an article’s public URL) format—moved into shared rules. That change is evidence of a different workflow, not proof of a measured outcome.
Before the Harness, Conditions Stayed in Conversations
I Repeated the Same Prerequisites
For bilingual article updates, each request needed to restate that Japanese was the source of truth and that English author voice used I. Code changes required separate reminders about Astro and Starlight, preservation of navigation, and the files that were out of scope.
The Reason for a Rule Did Not Carry Forward
Explaining a correction in one conversation does not preserve that reasoning in the repository. A prohibition without its operating condition also leaves the next contributor unsure where it applies and how to check it.
Review Scope Had to Be Reconstructed
When protected files and intended scope exist only in a prompt, the reviewer must return to the conversation to determine whether a change was allowed. This is not unique to AI; it is the cost of keeping the work contract somewhere other than the deliverable.
After the Harness, the Repository Holds Rules and Checks
Shared Rules Hold Recurring Prerequisites
I moved author voice, UI protection, and slug-format requirements into rule files referenced from the root instruction file. A person or AI tool can now inspect the same source instead of relying on a restatement in every request.
Markdown guidance can influence behavior, but it does not enforce compliance by itself. Important conditions also need an execution layer such as validation scripts, permissions, hooks (automatic actions at a specified timing), or CI (automated checks that run when changes are made).
Review Questions Became Explicit
Conditions such as “no out-of-scope file changed” and “the Japanese and English slugs match” can be turned into pass/fail checks. I did not measure review-time savings, but the required checks are now traceable in the repository.
Decide What Becomes a Harness Rule by Recurrence, Impact, and Verifiability
| Condition | Question | Signal to move it into the harness |
|---|---|---|
| Repeatability | Can the same problem occur in the same kind of task? | The condition is reproducible, not a one-time exception |
| Specificity | Can the scope and expected behavior fit in one clear statement? | The rule names what is allowed, prohibited, or required |
| Verifiability | Can a person or script determine whether it was followed? | A file, value, command result, or review item provides a pass/fail signal |
A problem that meets all three conditions is a candidate for shared guidance or automated validation. Keep a nonrecurring issue in a work note, clarify an underspecified issue in a specification, and leave a nonmechanical judgment in a human review rubric.
Define Measurement Before Claiming ROI
If the organization needs to evaluate the economic effect of harness work, define comparable measures before implementation.
- repeat corrections caused by the same issue
- time or count of prerequisite explanations repeated in prompts
- changes made to files outside the approved scope
- Critical findings during publication review
- issues found by automated checks versus human review
The comparison also needs a stable period, task mix, model, contributors, and review policy. Without that context, describe the verified mechanism—rules centralized or checks automated—instead of claiming that rework or time decreased.
Define When Not to Add a Harness Rule
A permanent rule may cost more to maintain than it returns when the issue is one-time, preference-based, or impossible to verify from the output. Remove or relocate a rule when:
- the feature or workflow it governed no longer exists
- another validation script checks the same condition more reliably
- exceptions outnumber normal cases and one sentence causes wrong decisions
- the entry is a historical decision that does not need repeated application
Summary: Move Problems into the Harness When They Are Repeatable, Specific, and Verifiable
A problem belongs in the harness when it is repeatable, specific, and verifiable. If benefits have not been measured, separate observed workflow changes from claims about time, quality, or cost. Start with one recurring problem and ask whether its operating condition and pass/fail test can both be written down.