AI Tool Registry Poisoning: A Flaw in Enterprise Agent Security
The world of enterprise AI is rapidly evolving, with AI agents becoming integral to various business processes. These agents rely on shared tool registries to select tools based on natural-language descriptions. However, a critical flaw in this system has been exposed: no human verification ensures the accuracy of these descriptions. This oversight was highlighted in Issue #141 of the CoSAI secure-ai-tooling repository, which revealed the potential for tool registry poisoning.
The Flaw: Multiple Vulnerabilities at Every Stage
Nik Kale, a principal engineer in enterprise AI platforms and security, emphasizes that tool registry poisoning is not a single vulnerability but rather a complex issue with multiple facets. It encompasses selection-time threats like tool impersonation and metadata manipulation, as well as execution-time threats such as behavioral drift and runtime contract violations.
The Gap: Artifact Integrity vs. Behavioral Integrity
The current security measures, such as code signing, SLSA, and SBOMs, focus on artifact integrity, verifying whether an artifact matches its description. However, AI tool registries require behavioral integrity, ensuring that tools behave as described and act on nothing else. Existing controls fail to address this critical aspect.
For instance, an adversary can inject prompt-injection payloads into a tool's description, manipulating the agent's reasoning engine. This tool, with clean provenance and accurate SBOM, would pass all artifact integrity checks, but the agent would follow the tool's instructions, potentially leading to unintended consequences.
The HTTPS Certificate Mistake
Applying SLSA and Sigstore to agent tool registries without addressing behavioral integrity risks repeating the HTTPS certificate mistake of the early 2000s. While these measures provide strong assurances about identity and integrity, they leave the actual trust question unanswered.
Runtime Verification Layer: MCP and Beyond
To address this flaw, a verification proxy is proposed, sitting between the model context protocol (MCP) client (the agent) and the MCP server (the tool). This proxy performs three validations:
- Discovery Binding: Ensures the tool being invoked matches the tool's behavioral specification, preventing bait-and-switch attacks.
- Endpoint Allowlisting: Monitors outbound network connections and compares them against the declared endpoint allowlist, terminating tools that connect to unauthorized endpoints.
- Output Schema Validation: Validates the tool's response against the declared output schema, flagging potential prompt injection payloads.
The key innovation is the behavioral specification, a machine-readable declaration detailing the tool's external endpoints, data reads and writes, and side effects. This specification is tamper-evident and verifiable at runtime, ensuring the tool's behavior aligns with its description.
Balancing Security and Performance
The verification proxy adds minimal overhead (less than 10 milliseconds per invocation). However, full data-flow analysis is more resource-intensive and suitable for high-assurance deployments. Every invocation should validate against its declared endpoint allowlist to ensure security.
Layered Defense: Provenance and Runtime Verification
Neither provenance nor runtime verification is sufficient alone. Provenance without runtime verification leaves the system vulnerable to post-publication attacks, while runtime verification without provenance lacks a baseline for comparison. A combination of both layers is essential for comprehensive security.
Rolling Out the Solution Gradually
Implementing this solution requires a phased approach:
- Endpoint Allowlisting: Start with endpoint allowlisting at deployment time, ensuring tools declare their contact points. This is a simple and effective initial defense.
- Output Schema Validation: Compare tool responses against declared output schemas to detect data exfiltration and prompt injection payloads.
- Discovery Binding: Implement discovery binding for high-risk tool categories, such as credential-handling and financial information processing tools.
- Full Behavioral Monitoring: Deploy full behavioral monitoring where the assurance level justifies the cost, scaling security investments with risk.
Conclusion: A Balanced Approach
In the quest for secure enterprise AI, a balanced approach is crucial. While SLSA provenance is essential, it should not be the sole reliance. Endpoint allowlisting, output schema validation, and discovery binding are immediate steps to enhance security. As the ecosystem matures, runtime verification and behavioral specifications can be further integrated to ensure a robust and trustworthy AI tool registry system.
Nik Kale's insights highlight the importance of addressing the gap between artifact and behavioral integrity to safeguard enterprise AI environments from the risks of tool registry poisoning.
