Repository navigation
Proposed revisions to CNA rules to clarify inclusion rules for AI vulns #46
Description
Activity
See pdf in the orgiginal post for full descriptions.
In regard to the discussion about whether CVE/CWE should consider model outputs when cataloging AI vulns, if the decision is to ignore them (based on the notion that the model itself is at no point being compromised), then I think this would automatically exclude all prompt injection/jailbreak attacks.
If we do choose to consider model outputs, then this allows for dicsusion of whether or not to consider hallucinations, or any other form of aligment problem, such as scheming, lying, deception.
Possibly another clarification would be whether or not a vulnerability need to be triggered maliciously in order to be catalogued by cve or cwe. With regular software vulns, this is a foregone conclusion, but here, some of these problems are expected behaviors of LLMs (e.g. hallucinations), even if they are unwanted.
Thanks for putting this together, Sasha. The document is a real step forward, especially the way items 1 through 5 tie the AI cases back to existing precedent (CWE-1278, CWE-1261, CWE-1428, default passwords). That framing makes clear these rules restore consistency rather than carve out an AI exception, which is the right posture.
I want to focus on the two open items, since that is where the remaining work sits.
On item 7 (integrity). I would define integrity here as requiring an attacker-influenced deviation from intended behavior, not mere untrustworthiness of the output. If integrity absorbs the broad "correctness integrity" reading from RFC 4949, then every hallucination becomes a candidate, which pulls item 1 back in through the definitional side door. Anchoring integrity to an adversary steering the system away from its intended behavior preserves the line the rest of the document draws. Baseline model fallibility is a quality problem. An induced deviation is a security problem. The definition should keep those separate.
On item 8 (output). I would put output in scope only when it crosses a trust boundary into another component without a human evaluating it in between. The strongest inclusion cases in the discussion notes all share this property: generated code that executes, a tool call, a config change, or a hallucinated package name that resolves to something installable. In each the output is a vector into another system, not just a wrong answer read by a person who can catch it. Output that is merely incorrect, biased, or deceptive, consumed by a human with judgment in the loop, should be addressed for safety and quality rather than CVE. I recognize the harder cases sit right on this line. Deception in a regulated context, or output that silently guides a security-critical decision, are where reasonable people in the WG disagree, and the notes reflect that. I would still hold the boundary at the trust-boundary crossing, because "does the output reach another component unmediated" is a testable question, while "is this output trustworthy enough" is not.
One seam worth making explicit before these finalize. Item 6 keeps evasion attacks (prompt injection, jailbreaks) out of CVE, consistent with 4.1.7 and the Program's own AI blog treatment of prompt injection as detection bypass. Item 8's strongest cases, a model steered into emitting a malicious tool call or a bad config, are typically reached through prompt injection. The Program has addressed the technique under 6 and the impact under 8 separately, but I do not think it has published a compositional rule for when a single incident implicates both. Absent that, CNAs will apply the two rules to the same artifact and reach different answers. My suggestion: state explicitly that the evasion technique itself stays out under 6, while the downstream trust-boundary crossing it produces is what gets evaluated under 8. That keeps 6 intact and gives CNAs a consistent way to handle the overlap, which is exactly the ambiguity that otherwise feeds the volume-panic narrative.
- Definition of integrity and whether it includes “trustworthiness” of model output.
- Whether or not CVE or CWE assignment should consider the output of a generative
AI model (as opposed to just data inputs).
My opinion is that "the inherent (or common) non-deterministic behavior of models" is not, by itself, a vulnerability. As others have noted, this is similar reasonging to physical attacks (4.1.5) and detection bypass (4.1.7). So perhaps a rule change (clarification) will end up noting that AI model prompt injection and any other cause of an incorrect output is not, by itself, a vulnerability?
(IIUC one could make model behavior deterministic by setting the same seed and using the same local model and prompts and no RAG, but in most practical cases, there is variability, and even being deterministic doesn't make the answer accurate or outright prevent injections.)
So, a software system (Product) might include a model. The model itself doesn't imply vulnerability, but how the Product uses the model is where we can identify vulnerabilities. Many models will handle untrusted input in various forms of prompt injection. The securitiy boundary (explicit or implicit policy, and policy violation) takes place when the Product decides to take insecure action based on model output. IOW, the Supplier of the Product is responsible for knowing or defining that model output may be incorrect, possibly tainted by untrusted input containing malicious prompt injection, and taking reasonable security measures. "Reasonable" will surely be subjective, but we might be able to agree on some of the extreme cases, for example, if the Product simply executes model output, then vulnerability. If the Product makes reasonable attempts to detect badness, executes in a sandbox first, provides secure defaults, and requires human intervention to perform possibly insecure actions, no vulnerability. (As covered by 4.1.3 and 4.1.4, if a user enables yolo vibe mode, no vulnerability.)
Even with this framing we probably still have a fundamental detection problem. If we accept that the model can't be responsible for correct or secure ouput, then the Product still has to somehow detect all bad outputs from the model, which slips back into 4.1.7. It remains computer science and philosophically impossible to identify any and all "bad" things.
Jerry, to your first response regarding Item 7 definition of integrity, you speak about "attacker-influenced." If malicious triggering is, in fact, a precondition for CVE or CWE assignment, that should be explicity stated in a rule. And if so, then hallucinations or any other kind of vuln that leads to a misalignment problem (scheming, deception, lying) would automatically be excluded from consideration. I'm not trying to argue one way or another, I'm just seeking clarity.
In regard to your response to Item 8 (output): if we are omitting vulns that aren't triggered maliciously, then that would rule out a bunch of your examples, I believe, such as hallucinated package. You then mention that vulns can be CVE-able when "the output reach[es] another component unmediated." Maybe that is testable, but that seems like a weird property to require of a vulnerability. Isn't that more of a property of larger system or environment in which the vuln exists?
I'm not sure I follow the explanation in your last paragraph. Per your arguments, l don't see why a prompt injection that results in a hallucinated package needs a compositional rule. The vuln is in the prompt injection (if at all), not in the hallucinated package (which would already be excluded).
My opinion is that "the inherent (or common) non-deterministic behavior of models" is not, by itself, a vulnerability. As others have noted, this is similar reasonging to physical attacks (4.1.5) and detection bypass (4.1.7). So perhaps a rule change (clarification) will end up noting that AI model prompt injection and any other cause of an incorrect output is not, by itself, a vulnerability?
yes, agree. randomness is not a criteria for exclusion. i.e. race condition vuln already exist.
Even with this framing we probably still have a fundamental detection problem. If we accept that the model can't be responsible for correct or secure ouput, then the Product still has to somehow detect all bad outputs from the model, which slips back into 4.1.7. It remains computer science and philosophically impossible to identify any and all "bad" things.
Wouldn't the vuln then just be related to the product ingesting untrusted content, and not the prompt injection (per 4.1.7)?
The purpose of this document is to proposed revisions to CNA rules regarding assignment of CVE-IDs or CWEs for vulnerabilities in AI systems. (edited) Specifically, these commments are designed to provide clarity when determining whether AI vulns warrant CVE or CWE assignment.
In determining whether to assign a new CVE-ID or CWE, the following criteria NEED NOT be taken into consideration:
There are still 2 issues which require further discussion:
7. Definition of integrity and whether it includes “trustworthiness” of model output.
8. Whether or not CVE or CWE assignment should consider the output of a generative AI model (as opposed to just data inputs).
Romanosky - Criteria for AI vuln inclusion as CWE or CVE - 7-10-2026.pdf