Designing an AI Review Loop for AI Output: What Automated Checks Can and Cannot Prevent
What This Article Covers
This site uses AI to help write articles. As part of that process, I run a separate AI to review what the writing AI produces — essentially separating the role of the writer and the reviewer, both handled by AI.
This article describes what this setup reliably catches and what it does not.
How the Setup Works
An AI in the writer role generates article content. A separate AI in the reviewer role then checks that output against a list of items and criteria provided to it.
Separating these roles makes it easier to detect formatting errors and configuration issues than when the same AI both writes and reviews its own output.
What Automated Checks Can Prevent
Detecting Broken Links
The reviewer checks whether links in the article body function correctly. It detects cases where a link points to a nonexistent page or the URL format is incorrect.
Verifying Reference Number Consistency
When the body contains citations such as [1], the reviewer checks whether those numbers correspond to entries in the references section at the end of the article. It detects cases where a number is missing from the list or where an entry in the list is not referenced in the body.
Checking for Prohibited Expressions
The reviewer checks whether the article contains expressions the project rules define as off-limits. Specific terms such as “AI transformation” (in its prohibited variant) or “revolutionary” are checked mechanically.
Verifying Frontmatter Format
The reviewer checks whether required fields are present and whether values conform to the expected format. This covers date format, slug structure, and the presence of required tags.
What Automated Checks Cannot Prevent
Content That Is Outdated
Automated review cannot determine whether the information in an article is still accurate at the time of reading. Content that was correct when written may become outdated due to product specification changes or evolving circumstances. Periodic review for currency requires human judgment.
A Source That Exists But Does Not Support the Claim
Even when a reference link is functional, the reviewer cannot verify whether the linked document actually supports the claim it is attached to. Confirming that a source supports an argument requires reading both the source and the claim.
Content That Does Not Match the Author’s Actual Experience
For articles grounded in personal experience, whether the content accurately reflects what actually happened can only be confirmed by the author. AI in a reviewer role has no way to verify alignment with the author’s firsthand account.
How I Combine Automated and Human Review
My current approach divides the work as follows:
- Format, structure, and link checks are handled by automated review.
- Factual accuracy — especially for claims that rely on external information — is checked by me.
- Content grounded in personal experience is verified by me before publication.
Automated review is well-suited to finding mechanical, structural problems. It is not a mechanism for guaranteeing content accuracy. Understanding that boundary and applying the appropriate method to each type of check is the basis for a practical quality management process.
Summary
AI-assisted automated review is an effective tool for detecting formatting errors and configuration issues. For factual accuracy, correspondence between claims and their sources, and consistency with the author’s actual experience, human review remains necessary. Designing a quality process means understanding what can be automated and what cannot, and assigning each type of check to the appropriate method.