Building a Validation Script with AI: Automating Consistency Checks for Harness Configuration
The limits of manual checking
As I built this site, the number of harness files (the configuration files that instruct AI) kept growing. Once the count of files with cross-references — CLAUDE.md, rule files, skill files, agent definition files — exceeded ten, manual consistency checks became impractical.
For example, confirming “does CLAUDE.md reference a rule file under shared/rules/ that actually exists?” or “is the script path listed in the command definition still valid?” by reading each file by eye takes time and leads to oversights.
To address this, I decided to build a validation script.
I cover the underlying configuration drift and prevention policy in What Harness Drift Is. This article is limited to turning detection requirements into an executable validation script.
What a validation script is
A validation script is a program that automatically checks whether configuration is correct.
A familiar analogy: instead of a person visually verifying one by one that every item on a packing list is in the box before a shipment goes out, a machine does the check. That is the idea here.
The script for this site (npm run harness:check) confirms things such as:
- Whether paths referenced in harness files actually exist
- Whether symbolic link targets are valid
- Whether required fields are present in rule files
- Whether commands listed in CLAUDE.md are actually defined
The process of building the script with AI
When building the script, I did not start by telling AI to “write the code.” I started by defining the specification — what should be checked.
Step 1: Identifying what to check
First, I listed in my own words what “broken” would look like: “a referenced file does not exist,” “a slug does not match the format specification,” “a symbolic link is broken.” I wrote these as a bulleted list.
Step 2: AI writes the implementation
I handed this list to AI and asked it to write a Node.js script that checks each item. AI translated each check into code.
Step 3: Testing and requesting revisions
I ran the script and reviewed the output. I gave specific feedback — “this error is not actually a problem,” “this item is not being detected” — and iterated on fixes.
Overall, my role was to define the specification and verify the behavior. AI handled nearly all of the code writing.
The effect of automated detection
Once the script was ready, I incorporated running npm run harness:check into the procedure after every configuration update.
As a result, problems such as “AI rewrote a rule file to reference a different path” or “AI accidentally removed a symbolic link” were detected at the next command execution.
The kind of problem that previously went unnoticed for a while was now caught immediately after the change occurred.
Why defining the specification first matters
In retrospect, the quality of the script depended on how well I specified what to check. Asking AI to “build a validation script” with a vague spec leads to missing important checks or including irrelevant ones.
AI is good at writing code, but deciding “what to check and to what extent” requires understanding the project context — and that is something the human must supply. This division of responsibility determines how useful the script ends up being.
Summary
Manually verifying harness configuration consistency has limits. Building a validation script makes it possible to detect configuration inconsistencies automatically. AI handles the implementation, but the human must define the specification — deciding what to check. That division of responsibility is what makes the script practical.