Harness Testing and Validation — Drift Detection and CI Validation

Since a harness is a collection of configuration files, it is difficult to write unit tests for it, unlike application code. However, if symlinks break or settings drift, Claude will not behave as expected. This page covers drift detection methods, how the harness:check command works, and how to incorporate it into CI — in order.

---

Why Harness Testing Is Necessary

Drift refers to when the actual state of a harness deviates from the expected state. This is the equivalent of a build error or type error in code, but in the case of a harness, an error does not immediately occur when drift happens. Symptoms appear in the form of Claude “vaguely not following rules” or “a specific skill not functioning.”

Without an explicit validation step, the following problems cannot be isolated.

• Whether it is a model problem or a configuration problem
• Whether a specific skill is correctly registered, or a symlink is simply broken
• Whether the paths referenced in CLAUDE.md are pointing to correct paths

Harness validation is necessary to “maintain a state where you can control whether Claude’s behavior is as intended.”

---

Common Drift Patterns

Problem	Cause	Detection Method
Broken symlinks	shared/ was moved or deleted without updating .claude/	ls -la .claude/
Skills not reflected	Added to shared/skills/ without running harness:sync	npm run harness:check
Rules not working	The link from CLAUDE.md to the referenced rule file is broken	npm run verify-links
Outdated permission settings	The allow list in settings.json has drifted from actually needed commands	Manual check

Drift frequently manifests not “at the time settings were changed” but “after using for a while.” Regular validation functions as preventive maintenance.

---

How It Works: Details of the harness:check Command

npm run harness:check checks the following items.

Symlink Verification

Verifies that .claude/agents, .claude/commands, and .claude/workflow exist as symlinks and that the reference destinations (shared/agents, shared/commands, shared/workflow) actually exist. Even if a symlink exists, if the reference destination has been deleted, it is detected as a “broken symlink.”

File Path Existence Verification

Checks whether paths such as shared/rules/*.md and shared/skills/*/SKILL.md listed in CLAUDE.md actually exist on the filesystem. This can detect cases where paths in CLAUDE.md have been left outdated.

JSON Syntax Check

Checks whether .claude/settings.json can be parsed as valid JSON. This early detects syntax errors that tend to occur when manually editing settings.

Detection of Known Dangerous Configuration Patterns

Detects configuration patterns that affect safety, such as whether npm run build is not included in the allow list in settings.json. This prevents the risk of production deployment being accidentally executed at the system level.

Division of Roles with verify-links

npm run verify-links checks whether links (in the Markdown [text](url) format) in documents point to valid URLs or file paths. While harness:check validates the structural integrity of harness settings, verify-links detects broken links in content — these are their respective roles.

---

Best Practices

Run harness:check for Every PR

Manage harness setting changes with PRs and automatically run harness:check in CI. This prevents the “it was working locally” state from entering main.

Always Run harness:sync After Editing shared/

Instead of manually checking symlinks, habitually run npm run harness:sync. Visually confirming every time that changes to shared/ are reflected in .claude/ is a source of human error.

# Pattern to run every time shared/ is edited
npm run harness:sync && npm run harness:check

Have Configuration Changes Reviewed via PR

Rather than committing changes to settings.json permission settings and rule files directly to main, put them through a PR so others can review them. Security-related configuration mistakes can be caught early.

Conduct an Exercise of “Intentionally Creating a Broken State”

To be prepared to quickly diagnose when an actual problem occurs, experience the detection flow by intentionally deleting symlinks. It is recommended to experience this once through the following hands-on exercise.

---

Hands-on Exercise

Step 1: Check the Current State

Run the following command from the project root to check the current state of the harness.

npm run harness:check

Example output:

✓ symlinks valid
✓ CLAUDE.md paths resolved
✓ settings.json valid JSON
✓ no dangerous patterns found
✓ harness:check passed

✅ Verify: If harness:check passed is displayed, it has been confirmed that there is no drift in the current harness. If errors are displayed, fix the problems according to the output messages before proceeding to the next step.

---

Step 2: Intentionally Break a Symlink and Experience Detection

To confirm that broken symlinks are correctly detected, temporarily delete a symlink.

# Rename the symlink with a backup name for testing
mv .claude/commands .claude/commands-backup

# Run harness:check to check for errors
npm run harness:check

Expected output example:

✗ symlink missing: .claude/commands
harness:check failed

After confirming the error was detected, restore the symlink.

# Restore the symlink to its original name
mv .claude/commands-backup .claude/commands

# Confirm it returns to normal after repair
npm run harness:check

✅ Verify: If harness:check failed is displayed, then after restoring, harness:check passed is displayed, the detection flow is working correctly.

---

Step 3: Check for Broken Links

npm run verify-links

If the output contains broken links, fix the file paths or URLs being referenced in the documents under src/content/docs/.

✅ Verify: If verify-links completes without errors, all links in the documents are valid.

---

Step 4: Create a CI Workflow

Create .github/workflows/harness-check.yml with the following content.

# .github/workflows/harness-check.yml
name: Harness Check
on: [push, pull_request]

jobs:
  harness:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: '22'
      - run: npm ci
      - run: npm run harness:check
      - run: npm run verify-links

After creating the file, commit it and push. From the next PR, validation will run automatically.

✅ Verify: If the Harness Check workflow is displayed in the GitHub Actions tab with a green check, the CI integration is complete.

---

Summary

• Drift is when harness settings deviate from the expected state, manifesting as Claude’s behavioral failures
• npm run harness:check checks symlinks, file paths, JSON syntax, and dangerous patterns all at once
• npm run verify-links detects broken links in documents. Its role differs from harness:check
• After editing shared/, always run npm run harness:sync && npm run harness:check
• Incorporating harness:check into CI continuously prevents the risk of configuration mistakes entering main

---

Next Steps

• Marketplace Publishing — Move on to how to share the harness you have built with the community

---

Frequently Asked Questions

Q: How often should harness:check be run?

A: The recommended pattern is after editing the shared/ directory, and automatically for each PR in CI. You don’t need to run it every time during normal Claude Code use, but having the habit of running it as the first step when you feel “Claude is behaving strangely” speeds up diagnosis.

Q: harness:check shows errors but Claude is operating normally. What should I do?

A: Check the content of the errors. The detection of broken symlinks or dangerous configuration patterns represents potential risks even if they don’t immediately affect Claude’s operation. Even in a “working now” state, it is recommended to fix the issues to maintain configuration integrity. If left unresolved, problems may manifest the next time harness:sync or dependency updates are run.

Q: How should I fix a harness:check failure in CI?

A: Check the error message output in the CI log and make the corresponding fixes locally. For broken symlinks, try repair with npm run harness:sync. For CLAUDE.md path reference errors, fix the corresponding file path. After fixing, run npm run harness:check locally to confirm the error has been resolved before pushing.

Q: Is it necessary to include both verify-links and harness:check in CI?

A: It is recommended to include both. harness:check validates the structural integrity of harness settings, and verify-links detects broken links in document content. Since their purposes differ, there are problems that either one alone cannot detect.

See the references for the external specifications and background sources used on this page.[1][2]

References

1. Anthropic, Claude Code documentation
2. Anthropic, Claude API documentation