Skip to content

Add benchmarking information #79

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
domguinard wants to merge 5 commits into
base: main
Choose a base branch
from
Open

Add benchmarking information #79

Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
64a2e76
Add benchmarking information to schema
domguinard Sep 30, 2026
e76f963
Adding benchmarking info, adding a sample entry
domguinard Sep 30, 2026
490cb89
Merge branch 'main' into add-benchmarking-information
dcondrey Oct 5, 2026
abb3782
Adding suggested improvements
domguinard Oct 6, 2026
01f4a79
Adding description for benchmarks
domguinard Oct 6, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
75 changes: 74 additions & 1 deletion softbinding-algorithm-list.schema.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,20 @@
],
"dateEntered": "2025-01-01T00:00:00Z",
"contact": "contact@example.com",
"informationalUrl": "https://example.com/softbinding/alg1"
"informationalUrl": "https://example.com/softbinding/alg1",
"benchmarkingInformation": [

@mrappard mrappard Sep 30, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
"benchmarkingInformation": [
"externalValidation": [
{
"url": "https://example.com/softbinding/alg1/benchmark-results",
"sha256": "3a7bd3e2360a3d29eea436fcfb7e44c735d117c42d1c1835420b6b9942dd4f1b",
"type": "benchmark"
}
]

@domguinard domguinard Oct 6, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Calling it externalValidation could be confusing (because of the term "validation" which has a defined meaning in C2PA).
I also think there is virtue in calling it a benchmark since it is what it actually is. Either we provide benchmarking information or we do not I would argue.

{
"description": "Robustness benchmark covering common image transformations",
"coveredMediaTypes": [
"image"
],
"benchmarkDate": "2026-07-10",
"results": {
"url": "https://example.com/softbinding/alg1/benchmark-results.json",
"sha256": "0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef"
}
}
]
},
"softBindingResolutionApis": [
"https://resolver.example.com/alg1"
Expand Down Expand Up @@ -106,6 +119,66 @@
"type": "string",
"format": "uri",
"description": "A web page containing more details about the algorithm"
},
"benchmarkingInformation": {
"type": "array",
"minItems": 1,
"description": "A list of benchmarking records that help explain the algorithm's measured capabilities",
"items": {
"type": "object",
"properties": {
"description": {
"type": "string",
"description": "Human readable description of the algorithm"
},
"coveredMediaTypes": {
"type": "array",
"minItems": 1,
"items": {
"enum": [
"application",
"audio",
"image",
"model",
"text",
"video"
],
"description": "IANA top level media type addressed by this benchmark"
}
},
"benchmarkDate": {
"type": "string",
"format": "date",
"description": "The date the benchmark was performed"
},
"results": {
"type": "object",
"description": "A link to the benchmark results and its SHA-256 content digest",
"properties": {
"url": {
"type": "string",
"format": "uri",
"description": "A URL containing the benchmark results"
},
"sha256": {
"type": "string",
"pattern": "^[A-Fa-f0-9]{64}$",
"description": "SHA-256 digest of the benchmark results content, encoded as 64 hexadecimal characters"
}
},
"required": [
"url",
"sha256"
]
}
},
"required": [
"description",
"coveredMediaTypes",
"benchmarkDate",
"results"
]
}
}
},

@mrappard mrappard Sep 30, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
},
},
"externalValidation": {
"type": "array",
"minItems": 1,
"description": "A list of externally hosted validation records (e.g., benchmark results, audits, certifications) that help explain the algorithm's measured capabilities. Each record is bound to its content by a SHA-256 hash",
"items": {
"type": "object",
"properties": {
"url": {
"type": "string",
"format": "uri",
"description": "A URL to the external validation record"
},
"sha256": {
"type": "string",
"pattern": "^[A-Fa-f0-9]{64}$",
"description": "Hex-encoded SHA-256 hash of the content retrieved from url"
},
"type": {
"type": "string",
"description": "The kind of external validation, e.g., benchmark, audit, certification"
}
},
"required": [
"url",
"sha256",
"type"
]
}
}

@domguinard domguinard Oct 6, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks @mrappard I took the sha256 suggestion and implemented them in the current version

"required": [
Expand Down
Loading