Remove all "supportingMedia" sections from the JSON

[jayjacobs]

I was pulling some data to provide data-support around issue #200. I found that not only is supportingMedia not used much at all, but when it is used, it is misused and generally just adds noise and parsing effort.

At the time I opened this, there are 256,421 CVEs in a published state, a grand total of 18,123 CVEs have used any of the supportingMedia fields, which is 7% of all published CVEs. Across the 18,123 CVEs, there are 26,046 references to supportMedia with a value and this is the count of unique CVEs with each field:

  field                                                cves
1 containers.cna.descriptions.supportingMedia.value   18099
2 containers.cna.solutions.supportingMedia.value       6641
3 containers.cna.workarounds.supportingMedia.value      728
4 containers.cna.exploits.supportingMedia.value         332
5 containers.cna.configurations.supportingMedia.value   246

Digging into it a bit further, the type of supportingMedia is always "text/html" with 26,117 fields specifying that value. If you notice the counts, there are 71 more definitions of the supportingMedia.type than defined values.

Zero records had base64 values.

There is also a "base64" field to accompany the supportingMedia section with 26,117 records setting a boolean value and 162 records are set to TRUE, 146 are on the "containers.cna.descriptions.supportingMedia" section. Manual inspection of all 146 showed zero with base64 encoded values. This supports closing #200 as a non-issue in practice.

Zero CVEs used supportingMedia to share supporting media.

For those that have text/html types defined (all of them), I focused on the supportingMedia for the description. I noticed that many of them looked to just repeat the text in the main description field. It was a little difficult to check since I was comparing html to plain text, all but a small handful matched the description once I converted from html to plain text, manual inspection of the rest showed any divergence was minor and in only three cases (CVE-2024-6442, CVE-2024-6443, CVE-2024-6444) was the text completely different, but the "supporting media" was just a summary. In other words, ZERO CVEs used the supportingMedia field for supporting media

I did not look at the other supportingMedia fields, I can if people think it would be helpful.

[boblord]

Looks like a candidate for removal from the schema.

[MrMegaZone]

I'm on board with deprecating supportingMedia with an eye toward future deletion.

[prabhu]

My vulnerability-db project uses supportingMedia in a few places. But happy to rewrite in a future iteration.

[ElectricNroff]

My understanding is that the value proposition for supportingMedia is to allow a CNA to edit a single document that has derived output of both a CVE Record (e.g., with the data that is used by the cve.org website) and a security advisory that is published on the CNA's own website. They are different because the cve.org website does not want to cede full control over layout (e.g., it needs the flexibility to support both desktop and mobile browsers) but a CNA's publication preferences may be different (e.g., complete control over layout as well as branding).

Although today not many CNAs are composing their advisories in this way, doing so is very helpful to CVE consumers, who are often opposed to any delay between when the security advisory is ready and when the CVE Record is ready. This is becoming more and more important as the time window before exploitation becomes shorter and shorter. Instead of eliminating supportingMedia, we should more strongly recommend it (and perhaps other options) as a way to ensure that publishing a CVE Record is a core part of each CNA's disclosure process, and is never an afterthought that may be delayed for days.

[MrMegaZone]

That wasn't the initial purpose of supportingMedia.

And the CVE text fields support formatting - it is only the CVE.org website that currently strips the formatting from what it displays. The formatting is still there in the JSON for consumers.

As a CNA I can say we will never use supportingMedia in this way. It would be yet another copy of the same information to get out of sync. No way, no how, never gonna happen.

I'd rather see this field go away as it has no official structure and it's pretty much useless in operation today.

[chandanbn]

supportingMedia is meant increasing legibility of text in CVE records for humans - something better than the wall of text you see now on the current CVE.org rendering.

Allowing new lines, paragraphs, bulleted lists, code font, bold, hyperlinks, tables, diagrams, screenshots etc., considered essential in technical text improves legibility and utility of CVE records.

We did publish several CVEs with paragraph breaks, hyperlinks, bulleted lists and tables and also embedded screenshots as it makes it quite easy to understand where or how a vulnerability manifests.

Please keep this field to evolve the CVE record display from looking like ancient Egyptian Hieroglyphs to something of a modern day technical brief.

Ancient scribes saved on screen (or sandstone or papyrus) real estate by omitting spaces, line breaks etc., I don't understand how much screen real estate are we saving on by not displaying html or markdown content from supportingMedia.

The HTML should be and can be easily DOM sanitized and filtered for colors and sizes or other styling to retain just the rich-text structure.

Please see
CVEProject/cve-website#2557

[MrMegaZone]

We can already do all of the nice markup in the other fields, so we do not need supportingMedia for that. The thing to fix is the website - it isn't going to render supportingMedia either so that's not going to solve the website problem.

And if the goal is for those not using the website, but reading the JSON, the other fields can be formatted.

Using it for images might be a valid reason to keep it. But maybe we should restrict the element more than it is. Today it is kind of an arbitrary element which means clients tend to ignore it rather than deal with it. If it was restricted to something like images it would probably be more widely supported.

[kernelsmith]

I also use supportingMedia in a manner similar to @prabhu, but I don't submit CVEs for publishing w/that field populated. I'm intentionally abusing the field to add "attachments" so I can share a single file (the JSON). This of course is not a reason in favor of, or against, removing the field. I also run into the problem @jayjacobs mentions, where parsing is slowed considerably because I also always base64 the files. This can be overcome by using a "fast JSON parser", one that will skip fields you don't want it to parse. Most "regular" JSON parsers will parse every field first, then return whatever you ask for. A fast parser will only parse what you ask for essentially. For most languages, the default standard lib for JSON parsing is not a "fast" one, so it's a little extra work. 1 additional detail, not really pertinent to the discussion at hand, if I don't want to purely embed an "attachment" in this way, usually for this slow parsing reason, I still use the supportingMedia, but the "value": is the filename instead of the file contents, usually in the forma of "./some_colocated_attachment.ext"

Coincidentally, I actually came to the issues because I was considering creating one that asks if we could add an optional "filename:" or "hint:" attribute to the media, but when searching to see if the issue already existed, I found this issue. Given this issue goes in the other direction, I won't bother adding any noise, but I was going to suggest:

              {
                "type": "application/vnd.tcpdump.pcap",
                "base64": true,
                "value": "Base64PCAPdata..."
                "hint": "CVE-9999-00000.pcap"
              },

[alilleybrinker]

Per the QWG discussion yesterday, this could be a candidate for an "unused" marking, and then a future removal. See #418 (comment)

[darakian]

@jayjacobs since you're the author of this issue would you mind making a PR w/ small RFD to mark this field as "unused"/"deprecated"?

@alilleybrinker I forget if we came to a concrete decision about how to mechanically mark fields for removal. Was it via the deprecated jsonschema keyword?
https://json-schema.org/understanding-json-schema/reference/annotations