Capability CI/CD: Verifying Agent Skills Before Distribution
What you’ll learn
- Why an Agent Skill should be managed as an asset whose changes and distribution destinations are tracked
- The different roles of static checks, overlap detection, and live evaluation
- What cryptographic signatures and an internal Skill catalog can confirm—and what they cannot
- Which records and approvals an organization needs when deciding whether to distribute a Skill to production
The Role of Capability CI/CD and Pre-Distribution Verification
An Agent Skill is a package of instructions, procedures, and supporting scripts that helps an AI perform a particular task. Creating one is only the beginning. Before using a Skill for work, an organization must check whether its contents are acceptable, whether it is actually useful, and which version each person may use.
Capability CI/CD applies two software delivery practices to Skills. CI (continuous integration) checks a Skill whenever it changes, while CD (continuous delivery) distributes approved versions to use environments. Together, they manage Skill quality and safety from change review through distribution.
On August 19, 2026, NVIDIA released SkillEvaluator, a tool for evaluating Skill quality and impact. It checks structure, safety, overlap with existing Skills, and practical effectiveness as separate concerns.[1][2]
Organizations that distribute Skills across multiple AI runtimes—the environments where AI systems load Skills and perform work—also need to connect evaluation results with required permissions and approval conditions. This article focuses on what evidence should qualify a Skill and who should approve its production distribution. It separates what public sources confirm from what each organization must decide for itself.
Structure, Safety, Overlap, and Practical Effectiveness Are Different Questions
A Skill evaluation should distinguish correct structure, safe use, and practical value. SkillEvaluator organizes these concerns into three tiers.[1][2]
| Evaluation tier | What it checks | Decision it supports |
|---|---|---|
| Tier 1: Structure and safety | Format, secrets, personal data, license terms, scripts that execute processing, and safety | Whether the Skill’s contents and organization have a problem |
| Tier 2: Overlap with existing Skills | Whether another Skill has the same role or meaning despite different wording | Whether a separate Skill is necessary |
| Tier 3: Live comparison | Results from running the same task with and without the Skill in an isolated environment that limits effects on production data and systems | Whether the Skill actually improves results |
A well-formed Skill is not necessarily useful. A Skill that passes safety checks may still reduce answer accuracy. Conversely, a Skill may improve accuracy while reading secrets or requiring more permission than the task needs. Such a Skill is not suitable for production use.
NVIDIA reported higher average Correctness and Effectiveness scores after evaluating more than 300 Verified Skills across over 30 product families. Effectiveness measures whether the agent achieved the goal and followed the expected workflow.[1]
However, 85% of the published results used one attempt per task and 15% used two, and the post did not report confidence intervals.[1]
Those catalog-wide averages do not prove that a particular Skill will improve results in another organization. Each organization should compare runs with and without the Skill using its own inputs, expected results, and permissions.
Cryptographic Signatures and an Internal Catalog Track Source, Version, and Approval State
An evaluated Skill should be tied to its author, version, purpose, evaluation conditions, and approved environments. This lets users determine who created it and under which conditions it was checked.
A cryptographic signature is a mechanism for confirming the signer and whether content changed after signing. For NVIDIA’s published Skills, successful signature verification confirms that the contents have not been modified since NVIDIA signed them.[3] A valid signature does not prove that the Skill’s business decisions are correct. It also cannot guarantee safety for every possible input.
An internal catalog is both a place to find available Skills and a ledger of approval state. At minimum, it should record:
- The owner responsible for maintenance
- Version and business use
- Permitted AI runtimes
- Required permissions
- Last evaluation date and expiry date
- Retirement status
The goal is to let users choose an approved version from the internal catalog instead of installing from a GitHub URL alone.
The earlier SkillOps article treats creation, publishing, evaluation, and maintenance as one operating lifecycle. Capability CI/CD focuses specifically on the quality gates—the checks and approvals that decide whether work may proceed—and the evidence needed to authorize production distribution.
Records and Approval Information Needed for Distribution
When results are split across several screens, it is difficult to decide which version of a Skill should be approved. A distribution decision should connect the Skill with its evaluation and approval evidence in one place.
This article calls that set of records a Capability Release Packet. It is not a product name. It is a template for organizing the information needed for approval.
| Item | Record | Stop distribution when |
|---|---|---|
| Purpose and maintenance owner | Business task, users, and person responsible for maintenance | The purpose or maintenance owner is unknown |
| Change | Version, before-and-after differences, and required tools or services | An executable file or connection destination is unreviewed |
| Static checks, which inspect contents without running the Skill | Format, secrets, personal data, license terms, and safety | A finding rated Critical remains unresolved |
| Overlap review | Difference in role from existing Skills | The content should be merged into an existing Skill |
| Live evaluation | With/without-Skill accuracy, workflow compliance, cost, and processing time | An important metric falls below its threshold |
| Permissions | Data read or written, external connections, and actions requiring approval | Permissions exceed what the purpose requires |
| Approval and expiry | Approver, target environment, reevaluation date, and expiry date | Approval or an expiry date is missing |
The most useful metrics are those that match the task, not simply a large number of metrics. For a weekly reporting Skill, the review may focus on whether it uses appropriate sources, omits important items, reproduces numbers correctly, follows the required format, and completes in an acceptable time.
For a Skill that changes user actions or external systems, output quality alone is insufficient. Its acceptance criteria should also cover appropriate permissions, a way to reverse actions, and human approval where needed.
Updates, Suspension, and Rollback After Distribution
A Skill’s behavior can change even when the Skill itself does not. An API—the access point used to call an external system or service—a source document, an AI model, or an internal policy may change.
Evaluation only at first release will not detect later degradation. When a version changes, update the distribution record and rerun the evaluations affected by that change.
These states make operating decisions easier to distinguish:
| State | Meaning and permitted use |
|---|---|
| Draft | Under development and not distributed to a use environment |
| Evaluating | Undergoing checks and comparisons in isolation |
| Approved | Available in named environments with named permissions |
| Suspended | Blocked from new runs during an investigation |
| Deprecated | Moving to a successor and unavailable for new adoption |
| Revoked | Withdrawn because of a serious issue |
A rollback—returning to an earlier version after a problem—needs more than the previous Skill files. It also needs the tools, permissions, and evaluation evidence that version assumed. A stored signed file cannot restore the same behavior if the API it depends on has changed.
Review Depth and Evaluation Limits Should Follow Risk
Not every Skill needs the same review depth. A Skill that only formats text and neither connects externally nor changes data has a different potential impact from one that reads customer records and sends them to an external API. The depth of review should reflect that difference.
An organization can start with one representative Skill and establish static checks, a with/without-Skill comparison, approval by the maintenance owner, and a fixed version.
Review should be deeper when a Skill has broad permissions, sends data externally, performs hard-to-reverse actions, or handles regulated data. Stronger isolation, security review, more attempts, and a shorter expiry can be combined based on the risk.
A scanner, a tool that searches for problematic wording or structure, cannot prove absolute safety on its own. Static checks cannot reproduce every runtime input, external-service response, and permission combination. Combine scan results with live evaluation, and state who is accountable for the final decision.
Summary: Evidence and Operational Responsibility Support Skill Distribution
Capability CI/CD checks a Skill’s structure, safety, overlap, and practical effectiveness, then distributes it with a version and approval conditions. SkillEvaluator turns static checks, overlap detection, and with/without-Skill live comparisons into one evaluation flow.
A distribution decision depends on more than a high evaluation score. An organization should be able to trace who approved which version, on what evidence, and for use in which environment. A Capability Release Packet puts that information in one place.
A practical first implementation is to assemble this record for one Skill. Including stop conditions and a reevaluation date lets the organization use the same record both when adding a Skill and when withdrawing it after a problem.
This article is a general information summary and is not legal advice. Confirm practical decisions with a qualified specialist.
References
- NVIDIA Technical Blog, Evaluating AI Agent Skill Performance with NVIDIA SkillEvaluator, August 19, 2026
- NVIDIA, SkillEvaluator
- NVIDIA, NVIDIA Agent Skills