Repository navigation
Make the site's benchmark claims match the project's own documentation - #651
Merged
Merged
Conversation
…tokens/sec
The site shows 298K under a badge reading "VERIFIED ON REAL GPUs", labelled
"Tokens/sec (RTX 3090)", "Verified on RunPod GPU", and in the GPU Benchmark
Comparison table as:
{"company": "TRINITY", "arch": "RTX 3090 (verified)", "val": "298K"}
This project's own docs/docs/benchmarks/competitor-comparison.md says:
| System | Raw ops/sec | Hardware |
| Trinity BitNet | 141K-608K | RTX 4090/L40S |
| bitnet.cpp (Microsoft) | 298K | RTX 3090 |
So 298K on an RTX 3090 is Microsoft's number, on Microsoft's card. Trinity's
own measured range is 141K-608K on different hardware. The site took the
competitor's row and put it under its own name, in a table whose purpose is
to compare Trinity against competitors.
Two further things the same doc says:
"These are kernel benchmark numbers measuring raw computation speed, not
end-to-end text generation."
"GPU competitive: 141K-608K ops/s matches industry benchmarks."
The unit is ops/sec, not tokens/sec -- and the comparison rows for Groq,
NVIDIA H100 and OpenAI sit at 50-2000, which can only be end-to-end tokens.
The table's apparent 150-300x lead is a unit mismatch, and the project's own
wording for the same result is "matches industry benchmarks".
And docs/docs/research/bitnet-report.md carries a caution the site does not:
"These throughput numbers reflect CPU-only inference even on the GPU
instance, because the I2_S quantization kernel has no GPU offload."
Changed here: the metric label reads Ops/sec in all five locales, and the
comparison table now carries the project's own caveat about units,
rendered under it.
NOT changed: the numbers, and the attribution of row 0. Whether that row
should show Trinity's 141K-608K or be removed is the author's call, and
silently rewriting a comparison table is not a correction. Recorded in
anomaly-register A46.
298K -- это bitnet.cpp от Microsoft на его же карте; собственный диапазон
Trinity по документации проекта 141K-608K на другом железе. Единица -- ops/sec,
а не tokens/sec. Исправлена подпись и добавлена оговорка самого проекта;
числа и атрибуция строки не тронуты.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…age says tape-out meta description, og:description and twitter:description -- what every share in Telegram, X and LinkedIn renders -- read: "100x Faster AI Inference at 99.8% Less Power. First chip with native SU(3) Unitary Core." "First chip" asserts fabricated hardware. This project's own evidence page says SKY130, tape-out through Tiny Tapeout, GDS produced, gate-level test passed, precheck passed -- a tape-out path. The about page was already corrected from "a fabricated chip" to "a tape-out" months ago; the metadata was not, and the metadata is the higher-reach surface. Now: "SU(3) unitary core through a SKY130 tape-out." And the 99.8% in the same sentence is not an independent measurement. Two findings, both arithmetic: 1 - 1/578.8 = 99.83% 31,680 / 578.8 = $54.73 (site shows $55) So "99.8% Less Power" is 578.8x restated as a percentage. Two of the three headline figures are the same number, and A45 recorded that this project's own audit tabled 578.8x as "No source / OVERCLAIM" before the audit file was deleted. Separately: 99.8% DOES appear in docs/docs/, as a test pass rate -- 3055/3060 core tests. Nothing to do with power. A figure existing in the corpus is not the same as the claim being sourced. Not changed here: 99.8% and 100x in the metadata. Those are the author's call, and A45 already put the label question on record. Flagged, not rewritten. Мета-описание утверждало существование чипа; собственная доказательная страница говорит о tape-out. Формулировка приведена к той, что уже исправлена на about. 99.8% -- это 578.8x, выраженное в процентах, а не независимое измерение. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ished ones Two tables, one unit error each, both fixed against docs/docs/benchmarks/competitor-comparison.md rather than by judgement. THE COMPARISON TABLE. Trinity's row carried raw kernel ops/sec while Groq, NVIDIA and OpenAI carried end-to-end tokens/sec, which produced an apparent 150-300x lead out of nothing. The project's own end-to-end table says the opposite, and says it plainly: | Trinity BitNet | 35-52 (CPU) | $0.01-0.35 | | Groq Llama-70B | 227-276 | Free tier | | GPT-4o-mini | ~100 | $$ API | | Claude Opus | ~80 | $$ API | with the note: "Trinity's CPU inference (35-52 tok/s) is usable for interactive chat. Cloud providers are faster but require API costs." The site now shows that table. Cloud is faster; the advantage is cost and running without a GPU at all. That is the project's own claim about itself, and it survives a reader checking it. THE GPU METRIC. 298K on an RTX 3090 was bitnet.cpp's -- Microsoft's number on Microsoft's card. Trinity's own measured range in the same document is 141K-608K on RTX 4090/L40S. Corrected, with the label reading Ops/sec and the description "Raw kernel benchmark" rather than "Verified on RunPod GPU", because raw kernel ops and a verified GPU throughput are not the same claim. All five locales, each catalogue reparsed as JSON. Nothing here was invented or chosen. Both tables now say what the repository already said about itself; what changed is that the site agrees with its own documentation. Обе таблицы приведены к опубликованным в документации проекта. Сравнение теперь в сквозных tokens/sec, где облака быстрее, а преимущество -- стоимость и работа без GPU. 298K заменено на собственный диапазон Trinity 141K-608K. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The .gitignore rule fixed in fe172f2 had made every .md under docs/ invisible. These are what was behind it: 11 files and 11 directories holding 26 more, 4,554 lines, none of it ever committed. They are not scratch. Five are research-paper directories duplicated under both docs/research/ and docs/lab/papers/ -- Hybrid, JEPAT, NCA, Ternary, VSA -- plus docs/experiments/ (7 files, "Found Experiments Summary — Trinity HSLM"), a 921-line BRAND_KIT.md, ARCHITECTURE-MULTIREPO.md, OPERATIONAL_PLAN.md, RED_LIST.md and a whitepaper. Every one of the 115 markdown files under docs/research, docs/lab/papers and docs/experiments passes claim-guard clean, so nothing here needed the wording work the six restored benchmark reports did. Committing them as they are. Reading 4,554 lines to decide what each should say is a separate pass; making them visible is not. Закоммичены 37 документов, которые git не видел из-за правила в .gitignore: исследовательские каталоги, эксперименты, брендбук, whitepaper. Все проходят claim-guard чисто. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…s late docs/site_audit_report.md has a section headed "Recommended Fixes" that has never been applied. It was written 2026-02-04, deleted 2026-03-06 in a commit about something else, and restored in c228881. This applies it. Fix 1, verbatim from the audit: Before: 3000x Energy Efficiency - Theorem 2 After: Up to 3000x Energy Efficiency - Projected on custom FPGA hardware (Verified: 20x memory compression on CPU) Fix 2, the calculator disclaimer, generalised to every surface that carries the figure rather than the calculator alone -- 14 places in en.json, and the same in de, es, ru, zh. Prose reads "Up to ... (projected, FPGA)". Compact metric tiles keep the bare number with a "*", and each of the four sections that uses one now renders the footnote: "* Projected on custom FPGA hardware. Verified on CPU: 20x memory compression." What this does NOT do is pick a number. The audit offers 20x (verified) and 500x (FPGA projected) as the defensible pair, which would replace 578.8x rather than qualify it. Choosing between them is a claim about the hardware and belongs to the author; marking every appearance as projected does not. 578.8x still has no source anywhere in three repos -- see A45. tsc unchanged at 184, numeric-fallback check 0 blocking, claim-guard clean. Применены собственные рекомендации аудита сайта, написанные 2026-02-04 и ни разу не выполненные. Каждое появление 578.8x и 3000x помечено как прогноз, в четырёх секциях выведена сноска. Числа не заменены: выбор между 20x и 500x -- утверждение о железе и остаётся за автором. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This was referenced Aug 10, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follows #648 and #649. Every change here is sourced from a file already in this repository — nothing was invented or chosen.
The comparison table compared two different quantities
Trinity's row carried raw kernel ops/sec; Groq, NVIDIA and OpenAI carried end-to-end tokens/sec. That produced an apparent 150–300× lead out of a unit mismatch.
docs/docs/benchmarks/competitor-comparison.mdhas the honest table, and states the conclusion plainly:The site now shows that table. Cloud is faster; the advantage is cost — $0.01–0.35 — and running without a GPU at all.
298K was Microsoft's number, on Microsoft's card
The site showed
{"company": "TRINITY", "arch": "RTX 3090 (verified)", "val": "298K"}— the competitor's row under its own name, in a table meant to compare against competitors. Now 141K–608K on RTX 4090/L40S, labelled Ops/sec · Raw kernel benchmark.578.8× — the project already audited this itself
docs/site_audit_report.md, added 2026-02-04 in a commit titled "audit: honest verification of site claims", tabled 578.8× — No source — ❌ OVERCLAIM and 3000× — No source — ❌ OVERCLAIM. That file was deleted 2026-03-06 in a commit about something else. The numbers stayed; the audit did not.Meanwhile
en.jsonread"578.8x (Verified)"while the calculator two components away read "Projected efficiency on custom FPGA hardware."Restored the audit. Removed
(Verified)in all five locales. Added aclaim-guardrule that blocks the label. The figures themselves are unchanged — that call is yours.And
1 − 1/578.8 = 99.83%, so "99.8% Less Power" is the same number in a percent sign.The link preview claimed a chip
og:descriptionread "First chip with native SU(3) Unitary Core". The evidence page says SKY130, tape-out, GDS produced, precheck passed. The about page was corrected months ago;<head>was not, and it reaches further.🤖 Generated with Claude Code