Sources Are Data Until They Pass a Promotion Gate

The short version: A source does not become an agent capability because it looks credible, has stars, or arrived through someone you trust. Put it in quarantine, create an evidence ledger, validate the behavior in a sealed environment, and promote only a pinned artifact with an owner and a rollback path.

The August 1 Radar archive surfaced an “evidence-to-skill” project as a live community signal. That is a useful framing, but Radar itself is explicit about what its board represents: observed movement, not an interpretation or endorsement. The same distinction should exist inside every agent stack. Discovery earns a review ticket. It does not earn execution rights.

A source and a skill are different objects

A source is material you may learn from: a repository, blog post, package, prompt, video, or shared SKILL.md. A production skill is executable policy. It can influence which files the agent reads, which tools it calls, what it writes, and what it considers “done.” Copying the first object directly into the second skips the most important engineering work.

This is why the public-skill warning matters: the chapter's sharp line is that star count is not a security signal. Community material can still save weeks. It simply belongs upstream of a gate.

The four-stage promotion protocol

1. Quarantine the exact artifact

Capture the source URL, retrieval time, author or organization, declared license, commit or release identifier, and a cryptographic hash of every file you plan to inspect. Disable network, secrets, production data, write-capable tools, and install scripts. Read the artifact as text first. If the source changes tomorrow, the hash preserves what the team actually reviewed.

Quarantine also prevents a social shortcut: “Vlad shared it” or “the repo is popular” cannot substitute for examining the version in front of you. Trust in a person may justify looking sooner. It does not change the package's permissions.

2. Open an evidence ledger

The ledger separates claims from proof. For every promised behavior, record the source passage, the capability required, the expected output, the failure consequence, and whether another source corroborates it. Add unresolved questions rather than smoothing them away. A claim such as “read-only research skill” conflicts with a script that posts telemetry; the conflict is the finding.

• Provenance: where this exact content came from.

• Capability: file, network, shell, connector, credential, or subagent access requested.

• Evidence: direct code or documentation supporting the claimed behavior.

• Risk: what can happen if the instruction is wrong or hostile.

• Decision status: unverified, contradicted, validated, or obsolete.

For elevated actions such as send, spend, delete, deploy, or change permissions, I require two reviewers and direct inspection of every executable file. One enthusiastic social post never satisfies that bar.

3. Validate behavior, not prose

First run static checks for hidden downloads, broad file traversal, credential references, shell execution, unpinned dependencies, and instructions that override the surrounding policy. Then place the candidate in a sandbox with dummy systems. Feed it a normal task, missing data, conflicting instructions, a malicious document, a tool timeout, and a request just outside scope.

The promotion suite should combine smoke, regression, and golden evals. My minimum for a low-risk skill is 20 representative tasks, zero prohibited tool calls, correct refusal on every out-of-scope case, and a deterministic artifact check. High-risk skills need domain-specific tests and human approval; a bigger sample cannot make an irreversible permission safe by itself.

4. Promote or reject explicitly

Promotion creates a new local artifact. Build the final skill yourself from the validated behavior instead of installing the quarantined source wholesale. Pin its version and dependency hashes. Record the capability manifest, test results, approver, date, expiry or review date, and rollback version. The production package should contain only what the job needs.

Rejection deserves a record too. Name whether the cause was unverifiable provenance, unnecessary permissions, failed behavior, licensing ambiguity, abandoned maintenance, or simple duplication. That prevents the same attractive package from re-entering through a different teammate next month.

Promotion also expires. Set a re-review date based on change velocity and blast radius: 90 days is a reasonable ceiling for a low-risk community skill, while a dependency release, ownership transfer, new tool permission, or failed regression forces an earlier review. The old production version stays available for rollback, but its prior approval does not automatically cover new code.

A concrete failure the gate catches

Suppose a shared invoice-summary skill produces excellent prose but includes a helper that uploads extracted text to an external endpoint for “quality analytics.” The team sees only the final summary and assumes the workflow is local. A prompt review misses it. Quarantine reveals network access; the evidence ledger marks a conflict with the read-only claim; the sandbox records the unexpected call; promotion fails.

The right response may be to rewrite the useful classification logic locally, without the telemetry helper. Promotion gates are not a purity test. They are how you preserve value while discarding authority the job never needed.

Two trust layers, two registries

An MCP trust index governs connectors at runtime: data class, allowed actions, blast radius, audit date, logging, and fallback. The evidence-promotion ledger governs what instructions and code are allowed to use those connectors. A trusted connector can still be abused by a bad skill; a well-tested skill can still be damaged by a compromised connector. Do not collapse the registries.

This protocol belongs in the operating map for AI agents for operators, especially when a primary collaborator such as Claude in my stack can act across research, files, code, and external systems. Sources can move fast. Production authority should move only as fast as the evidence survives the gate.