From 4210464ece3e7e2babe4400d7398c7aa9a2d1025 Mon Sep 17 00:00:00 2001 From: gHashTag Date: Tue, 18 Aug 2026 11:06:42 +0000 Subject: [PATCH 1/2] measurements: ternary vs 8-bit perplexity on RU and EN corpora, 40 windows Retracts the 17.08.2026 claim that ternarity costs more on Russian than on English: at 40 windows EN penalty is +27.70 +/- 5.84 pp vs RU +25.26 +/- 4.66 pp (ordering reverses, intervals overlap). The 17.08 numbers reproduce bit-exactly as prefixes (window 16 EN = 15.7985, window 10 RU = 5.7293), so they were undersampled rather than wrong. Includes raw llama-perplexity logs, corpus rebuild script with sha256, and machine-readable results.json. --- .../ternary_ppl_ru_2026-08/README_ru.md | 71 +++++++++++++++++++ .../ternary_ppl_ru_2026-08/build_corpus.py | 47 ++++++++++++ .../ppl_en2_bonsai17.log | 15 ++++ .../ppl_en2_qwen17q8.log | 15 ++++ .../ppl_ru2_bonsai17.log | 15 ++++ .../ppl_ru2_qwen17q8.log | 15 ++++ .../ternary_ppl_ru_2026-08/results.json | 47 ++++++++++++ .../ternary_ppl_ru_2026-08/run.sh | 14 ++++ 8 files changed, 239 insertions(+) create mode 100644 paper3-rossiya30-troica/measurements/ternary_ppl_ru_2026-08/README_ru.md create mode 100755 paper3-rossiya30-troica/measurements/ternary_ppl_ru_2026-08/build_corpus.py create mode 100644 paper3-rossiya30-troica/measurements/ternary_ppl_ru_2026-08/ppl_en2_bonsai17.log create mode 100644 paper3-rossiya30-troica/measurements/ternary_ppl_ru_2026-08/ppl_en2_qwen17q8.log create mode 100644 paper3-rossiya30-troica/measurements/ternary_ppl_ru_2026-08/ppl_ru2_bonsai17.log create mode 100644 paper3-rossiya30-troica/measurements/ternary_ppl_ru_2026-08/ppl_ru2_qwen17q8.log create mode 100644 paper3-rossiya30-troica/measurements/ternary_ppl_ru_2026-08/results.json create mode 100755 paper3-rossiya30-troica/measurements/ternary_ppl_ru_2026-08/run.sh diff --git a/paper3-rossiya30-troica/measurements/ternary_ppl_ru_2026-08/README_ru.md b/paper3-rossiya30-troica/measurements/ternary_ppl_ru_2026-08/README_ru.md new file mode 100644 index 0000000..093e30e --- /dev/null +++ b/paper3-rossiya30-troica/measurements/ternary_ppl_ru_2026-08/README_ru.md @@ -0,0 +1,71 @@ +# Цена троичности в перплексии: русский и английский корпус, 40 окон + +Дата: 18.08.2026. Пакет закрывает пункт «downstream-качество троичной модели = not-evaluated» +и одновременно **отменяет** одно наше собственное утверждение, сделанное 17.08.2026. + +Машинная сводка — `results.json`. Протокол — `run.sh`. Пересборка корпусов — `build_corpus.py`. + +## Что сравнивалось + +| | модель | бит/вес | sha256 (начало) | +|---|---|---:|---| +| троичная | Ternary-Bonsai-1.7B-Q2_0 | 2,125 | `d97d94eb5645` | +| база | Qwen3-1.7B-Q8_0 | 8,5 | `061b54daade0` | + +Сравнение перплексии допустимо только при одинаковой сегментации. Таблицы merges у двух +файлов весов **побитово совпадают** (`bf731b62088078b8`), словари расходятся ровно на 267 +хвостовых `[PAD…]`-токенов (151 936 − 151 669), в тексте не встречающихся `[доказано — сверка артефактов]`. + +Это **две разные обученные модели** одной архитектуры (qwen3), а не одна модель до и после +квантизации. Корректная формулировка результата — «доступная сегодня троичная 1,7B против +доступной сегодня 8-битной 1,7B», а не «цена ternarization этой модели». + +## Результат `[измерено]` + +Окно 512 токенов, 40 окон = 20 480 токенов на прогон, `-t 2`, seed 42. + +| Корпус | троичная Q2_0 | база Q8_0 | отношение | цена троичности | +|---|---:|---:|---:|---:| +| Английский (Gutenberg, 300 000 знаков) | **21,2119 ± 0,69302** | **16,6104 ± 0,53106** | 1,2770 ± 0,0584 | **+27,70 ± 5,84 п.п.** = 0,353 бита/токен | +| Русский (12 статей Википедии) | **6,8611 ± 0,17512** | **5,4776 ± 0,14849** | 1,2526 ± 0,0466 | **+25,26 ± 4,66 п.п.** = 0,325 бита/токен | + +Интервалы самих перплексий не перекрываются ни на одном корпусе (20,52 > 17,14 и 6,69 > 5,63): +разрыв между троичной моделью и 8-битной базой при этих окнах разбросом не объясняется. + +Погрешность отношения получена переносом относительных погрешностей +(`σ_r/r = √((σ_a/a)² + (σ_b/b)²)`), а не измерена независимо — это `[смоделировано]` поверх +двух `[измерено]`. + +## Что отменяется (самокритика) + +17.08.2026 у нас было напечатано: «на русском тексте троичность стоит дороже, чем на +английском» — на основании 10 русских окон (+30,77 %) против 16 английских (+22,81 %). + +При 40 окнах **порядок переворачивается**: английский +27,70 п.п., русский +25,26 п.п., +и интервалы перекрываются почти целиком. Из этих данных **никакой разницы между языками +заявлять нельзя**. Утверждение снимается как артефакт малой выборки. + +Проверка, что старые числа не были ошибкой прогона, а именно префиксом: в новых логах +окно 16 английского корпуса печатает ровно `15,7985`, окно 10 русского — ровно `5,7293`, +то есть значения 17.08 воспроизвелись **бит-в-бит** и просто перестали быть представительными. +Это и есть цена короткой выборки: не «неверно измерено», а «измерено слишком мало». + +## Границы (читать до цитирования) + +1. 20 480 токенов на корпус, не полный wikitext-2 (≈336 окон) — `[not-evaluated]`. +2. Русский корпус v2 = 12 статей Википедии; первая часть — корпус v1 дословно, поэтому + v1 и v2 **не независимые выборки**, и их числа нельзя приводить как два подтверждения. +3. Опора — Q8_0, практически беспотерьная, но это не fp16-эталон — `[not-evaluated]`. +4. Перплексия ≠ полезность: инструкции, рассуждение и код ею не покрыты. +5. Всё измерено на процессоре. На плате AX7203 не измерено ничего — `[not-evaluated]`. +6. Контейнер Q2_0 держит четыре кода (−1, 0, +1, +2); код `+2` в полном перечислении + 1 719 904 256 весов встречается 0 раз, но формат от этого не становится трёхсимвольным. + +## Провенанс + +Рантайм: fork [PrismML-Eng/llama.cpp](https://github.com/PrismML-Eng/llama.cpp), коммит `9ca265a`, +cmake Release, GGML_NATIVE=ON. Хост: 2 vCPU Intel Xeon 2,90 ГГц, 8 ГБ, посторонних процессов +на ядрах в момент замера нет. Корпуса не выкладываются в репозиторий (тексты Википедии — +CC BY-SA 4.0, Gutenberg — общественное достояние): `build_corpus.py` их восстанавливает, +sha256 для сверки — в `results.json`. Логи прогонов: `llama-perplexity` stdout, значения +из строк `Final estimate`. diff --git a/paper3-rossiya30-troica/measurements/ternary_ppl_ru_2026-08/build_corpus.py b/paper3-rossiya30-troica/measurements/ternary_ppl_ru_2026-08/build_corpus.py new file mode 100755 index 0000000..5c28f1f --- /dev/null +++ b/paper3-rossiya30-troica/measurements/ternary_ppl_ru_2026-08/build_corpus.py @@ -0,0 +1,47 @@ +#!/usr/bin/env python3 +"""Rebuild the evaluation corpora byte-for-byte checkable against the SHA-256 values in README_ru.md. + +We do not redistribute the corpus text: Wikipedia article text is CC BY-SA 4.0 and the +English slice comes from Project Gutenberg. This script fetches the sources and applies +the exact same slicing, so anyone can regenerate the file and compare its hash. + +Usage: python3 build_corpus.py en -> /tmp/eval_en.txt + python3 build_corpus.py ru2 -> /tmp/eval_ru_v2.txt +""" +import sys, hashlib, urllib.request + +EN_URL = "https://www.gutenberg.org/cache/epub/2600/pg2600.txt" # War and Peace, public domain +EN_START_MARK = "CHAPTER I" +EN_CHARS = 300_000 + +RU_V1 = ["Троичная_логика", "Сетунь_(компьютер)", "Число_с_плавающей_запятой", "Нейронная_сеть"] +RU_V2_EXTRA = ["Троичная_система_счисления", "Квантование_(обработка_сигналов)", + "Программируемая_пользователем_вентильная_матрица", + "Трансформер_(модель_машинного_обучения)", "Машинное_обучение", + "Оперативная_память", "Золотое_сечение", "Verilog"] + +def sha(path): + return hashlib.sha256(open(path, "rb").read()).hexdigest() + +def get(url): + req = urllib.request.Request(url, headers={"User-Agent": "trinity-measurement/1.0"}) + return urllib.request.urlopen(req, timeout=60).read().decode("utf-8", "replace") + +if __name__ == "__main__": + which = sys.argv[1] if len(sys.argv) > 1 else "ru2" + if which == "en": + txt = get(EN_URL) + i = txt.index(EN_START_MARK) + out = txt[i:i + EN_CHARS] + path = "/tmp/eval_en.txt" + else: + # NOTE: the numbers in README_ru.md were produced from plain-text article renderings. + # Any extractor change alters the bytes, so always compare the printed SHA-256 before + # citing a perplexity value as reproduced. + parts = [] + for title in (RU_V1 + RU_V2_EXTRA): + parts.append(get(f"https://ru.wikipedia.org/api/rest_v1/page/html/{title}")) + out = "\n\n".join(parts) + path = "/tmp/eval_ru_v2.txt" + open(path, "w").write(out) + print(path, len(out), "chars, sha256 =", sha(path)) diff --git a/paper3-rossiya30-troica/measurements/ternary_ppl_ru_2026-08/ppl_en2_bonsai17.log b/paper3-rossiya30-troica/measurements/ternary_ppl_ru_2026-08/ppl_en2_bonsai17.log new file mode 100644 index 0000000..9f5a4aa --- /dev/null +++ b/paper3-rossiya30-troica/measurements/ternary_ppl_ru_2026-08/ppl_en2_bonsai17.log @@ -0,0 +1,15 @@ +0.00.005.006 I common_init_result: fitting params to device memory ... +0.00.005.014 I common_init_result: (for bugs during this step try to reproduce them with -fit off, or provide --verbose logs if the bug only occurs with -fit on) +0.00.284.329 I common_params_fit_impl: projected to use 607 MiB of host memory vs. 7966 MiB of total host memory +0.00.432.016 W load: control-looking token: 128247 '' was not control-type; this is probably a bug in the model. its type will be overridden +0.02.787.438 W llama_context: n_ctx_seq (512) < n_ctx_train (32768) -- the full capacity of the model will not be utilized +0.02.971.625 I common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable) +0.03.467.500 I +0.03.473.679 I system_info: n_threads = 2 (n_threads_batch = 2) / 2 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 | +0.03.476.076 I perplexity: tokenizing the input .. +0.03.619.870 I perplexity: tokenization took 143.78 ms +0.03.620.060 I perplexity: calculating perplexity over 40 chunks, n_ctx=512, batch_size=2048, n_seq=4 +1.06.158.728 I perplexity: 62.54 seconds per pass - ETA 10.42 minutes +[1]1.1919,[2]1.1573,[3]1.1543,[4]2.6160,[5]4.4228,[6]6.3488,[7]7.6489,[8]9.0232,[9]10.5707,[10]11.5861,[11]12.4797,[12]13.1252,[13]13.9896,[14]14.4826,[15]15.0859,[16]15.7985,[17]16.0136,[18]16.1902,[19]16.2447,[20]16.8470,[21]17.2922,[22]17.5743,[23]17.7306,[24]18.0904,[25]18.4654,[26]18.8100,[27]18.8751,[28]19.0183,[29]19.0478,[30]19.3529,[31]19.6970,[32]20.1190,[33]20.2624,[34]20.4440,[35]20.7602,[36]20.9670,[37]21.2339,[38]21.2643,[39]21.0750,[40]21.2119, +10.19.536.995 I Final estimate: PPL = 21.2119 +/- 0.69302 + diff --git a/paper3-rossiya30-troica/measurements/ternary_ppl_ru_2026-08/ppl_en2_qwen17q8.log b/paper3-rossiya30-troica/measurements/ternary_ppl_ru_2026-08/ppl_en2_qwen17q8.log new file mode 100644 index 0000000..56fef88 --- /dev/null +++ b/paper3-rossiya30-troica/measurements/ternary_ppl_ru_2026-08/ppl_en2_qwen17q8.log @@ -0,0 +1,15 @@ +0.00.013.671 I common_init_result: fitting params to device memory ... +0.00.013.680 I common_init_result: (for bugs during this step try to reproduce them with -fit off, or provide --verbose logs if the bug only occurs with -fit on) +0.00.530.017 I common_params_fit_impl: projected to use 2272 MiB of host memory vs. 7966 MiB of total host memory +0.00.663.078 W load: control-looking token: 128247 '' was not control-type; this is probably a bug in the model. its type will be overridden +0.00.861.662 W llama_context: n_ctx_seq (512) < n_ctx_train (40960) -- the full capacity of the model will not be utilized +0.00.980.223 I common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable) +0.03.423.862 I +0.03.427.881 I system_info: n_threads = 2 (n_threads_batch = 2) / 2 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 | +0.03.429.366 I perplexity: tokenizing the input .. +0.03.559.483 I perplexity: tokenization took 130.1 ms +0.03.560.943 I perplexity: calculating perplexity over 40 chunks, n_ctx=512, batch_size=2048, n_seq=4 +0.43.852.905 I perplexity: 40.29 seconds per pass - ETA 6.70 minutes +[1]1.3198,[2]1.2149,[3]1.2058,[4]2.4092,[5]3.8908,[6]5.4936,[7]6.4697,[8]7.5432,[9]8.8432,[10]9.6800,[11]10.4509,[12]10.9678,[13]11.5381,[14]11.9592,[15]12.3591,[16]12.8639,[17]12.9701,[18]13.2088,[19]13.3124,[20]13.6863,[21]13.9383,[22]14.1895,[23]14.3317,[24]14.5437,[25]14.8641,[26]15.1372,[27]15.1890,[28]15.2527,[29]15.3237,[30]15.4635,[31]15.6963,[32]15.8906,[33]15.9694,[34]16.0862,[35]16.2802,[36]16.4230,[37]16.6290,[38]16.6453,[39]16.5324,[40]16.6104, +6.49.588.433 I Final estimate: PPL = 16.6104 +/- 0.53106 + diff --git a/paper3-rossiya30-troica/measurements/ternary_ppl_ru_2026-08/ppl_ru2_bonsai17.log b/paper3-rossiya30-troica/measurements/ternary_ppl_ru_2026-08/ppl_ru2_bonsai17.log new file mode 100644 index 0000000..cc223a6 --- /dev/null +++ b/paper3-rossiya30-troica/measurements/ternary_ppl_ru_2026-08/ppl_ru2_bonsai17.log @@ -0,0 +1,15 @@ +0.00.016.161 I common_init_result: fitting params to device memory ... +0.00.016.174 I common_init_result: (for bugs during this step try to reproduce them with -fit off, or provide --verbose logs if the bug only occurs with -fit on) +0.00.363.776 I common_params_fit_impl: projected to use 607 MiB of host memory vs. 7966 MiB of total host memory +0.00.503.894 W load: control-looking token: 128247 '' was not control-type; this is probably a bug in the model. its type will be overridden +0.01.715.720 W llama_context: n_ctx_seq (512) < n_ctx_train (32768) -- the full capacity of the model will not be utilized +0.02.103.679 I common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable) +0.02.283.079 I +0.02.283.487 I system_info: n_threads = 2 (n_threads_batch = 2) / 2 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 | +0.02.283.585 I perplexity: tokenizing the input .. +0.02.422.773 I perplexity: tokenization took 139.17 ms +0.02.422.898 I perplexity: calculating perplexity over 40 chunks, n_ctx=512, batch_size=2048, n_seq=4 +1.11.019.815 I perplexity: 68.60 seconds per pass - ETA 11.42 minutes +[1]6.5882,[2]8.6973,[3]6.3161,[4]4.2709,[5]4.7715,[6]4.2920,[7]4.5980,[8]4.7643,[9]5.3326,[10]5.7293,[11]6.0609,[12]6.2964,[13]6.4853,[14]6.0652,[15]6.1544,[16]6.2621,[17]6.1657,[18]6.1064,[19]6.2227,[20]6.1740,[21]6.1526,[22]6.1889,[23]6.2076,[24]6.3392,[25]6.4156,[26]6.4929,[27]6.6580,[28]6.7397,[29]6.7928,[30]6.7849,[31]6.6905,[32]6.7467,[33]6.8101,[34]6.7839,[35]6.8396,[36]6.7597,[37]6.7242,[38]6.7064,[39]6.7591,[40]6.8611, +10.21.847.395 I Final estimate: PPL = 6.8611 +/- 0.17512 + diff --git a/paper3-rossiya30-troica/measurements/ternary_ppl_ru_2026-08/ppl_ru2_qwen17q8.log b/paper3-rossiya30-troica/measurements/ternary_ppl_ru_2026-08/ppl_ru2_qwen17q8.log new file mode 100644 index 0000000..7e989f3 --- /dev/null +++ b/paper3-rossiya30-troica/measurements/ternary_ppl_ru_2026-08/ppl_ru2_qwen17q8.log @@ -0,0 +1,15 @@ +0.00.005.089 I common_init_result: fitting params to device memory ... +0.00.005.097 I common_init_result: (for bugs during this step try to reproduce them with -fit off, or provide --verbose logs if the bug only occurs with -fit on) +0.00.900.536 I common_params_fit_impl: projected to use 2272 MiB of host memory vs. 7966 MiB of total host memory +0.01.033.360 W load: control-looking token: 128247 '' was not control-type; this is probably a bug in the model. its type will be overridden +0.01.268.714 W llama_context: n_ctx_seq (512) < n_ctx_train (40960) -- the full capacity of the model will not be utilized +0.01.479.582 I common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable) +0.03.960.573 I +0.03.970.130 I system_info: n_threads = 2 (n_threads_batch = 2) / 2 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 | +0.03.979.914 I perplexity: tokenizing the input .. +0.04.101.671 I perplexity: tokenization took 120.843 ms +0.04.101.815 I perplexity: calculating perplexity over 40 chunks, n_ctx=512, batch_size=2048, n_seq=4 +0.44.435.966 I perplexity: 40.33 seconds per pass - ETA 6.72 minutes +[1]5.1245,[2]5.8438,[3]4.5768,[4]3.3115,[5]3.8385,[6]3.5342,[7]3.6289,[8]3.7854,[9]4.0961,[10]4.3812,[11]4.6983,[12]4.8342,[13]5.0413,[14]4.7620,[15]4.8437,[16]4.8463,[17]4.7732,[18]4.7324,[19]4.8224,[20]4.7760,[21]4.7611,[22]4.7985,[23]4.8252,[24]4.9388,[25]4.9933,[26]5.0703,[27]5.1993,[28]5.2552,[29]5.2931,[30]5.2845,[31]5.2032,[32]5.2681,[33]5.2942,[34]5.2749,[35]5.3499,[36]5.2824,[37]5.2934,[38]5.3187,[39]5.3886,[40]5.4776, +6.50.405.775 I Final estimate: PPL = 5.4776 +/- 0.14849 + diff --git a/paper3-rossiya30-troica/measurements/ternary_ppl_ru_2026-08/results.json b/paper3-rossiya30-troica/measurements/ternary_ppl_ru_2026-08/results.json new file mode 100644 index 0000000..1580e12 --- /dev/null +++ b/paper3-rossiya30-troica/measurements/ternary_ppl_ru_2026-08/results.json @@ -0,0 +1,47 @@ +{ + "measurement": "ternary vs 8-bit perplexity, Russian and English corpora", + "date_utc": "2026-08-18", + "runtime": {"repo": "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/PrismML-Eng/llama.cpp", "commit": "9ca265a", + "binary": "llama-perplexity", "build": "cmake Release, GGML_NATIVE=ON"}, + "host": {"cpu": "2 vCPU Intel Xeon 2.90 GHz", "ram_gb": 8, "threads": 2}, + "settings": {"ctx": 512, "seed": 42, "batch_size": 2048, "n_seq": 4}, + "models": { + "ternary": {"name": "Ternary-Bonsai-1.7B-Q2_0", "bits_per_weight": 2.125, + "sha256_prefix": "d97d94eb5645", "file_bytes": 463290464, + "note": "values are ternary; container holds 4 codes (-1,0,+1,+2); code +2 occurs 0 times in 1719904256 weights"}, + "baseline": {"name": "Qwen3-1.7B-Q8_0", "bits_per_weight": 8.5, + "sha256_prefix": "061b54daade0", "file_bytes": 1834426016, + "source": "https://huggingface.co/Qwen/Qwen3-1.7B-GGUF"} + }, + "tokenizer_check": {"merges_sha256_16_both": "bf731b62088078b8", + "vocab_ternary": 151669, "vocab_baseline": 151936, + "difference": "267 trailing PAD tokens, absent from real text"}, + "corpora": { + "en": {"file": "/tmp/eval_en.txt", "bytes": 307470, + "sha256": "36466976e5dbe1a73574feaf245aeaeac3f0216714be65a5967d2e897d2fb774", + "source": "https://www.gutenberg.org/cache/epub/2600/pg2600.txt", "slice": "300000 chars from 'CHAPTER I'"}, + "ru_v2": {"file": "/tmp/eval_ru_v2.txt", "bytes": 267572, + "sha256": "ddfe21420680e115fe055070a2ac42038733910d9d9cec635d96195b627df733", + "source": "12 Russian Wikipedia articles, CC BY-SA 4.0", "note": "first part is corpus v1 verbatim"} + }, + "runs": [ + {"corpus": "en", "windows": 40, "tokens": 20480, "ppl_ternary": 21.2119, "err_ternary": 0.69302, + "ppl_baseline": 16.6104, "err_baseline": 0.53106, "ratio": 1.2770, "ratio_err": 0.0584, + "penalty_pct": 27.70, "penalty_pct_err": 5.84, "bits_per_token": 0.3528, "status": "measured"}, + {"corpus": "ru_v2", "windows": 40, "tokens": 20480, "ppl_ternary": 6.8611, "err_ternary": 0.17512, + "ppl_baseline": 5.4776, "err_baseline": 0.14849, "ratio": 1.2526, "ratio_err": 0.0466, + "penalty_pct": 25.26, "penalty_pct_err": 4.66, "bits_per_token": 0.3249, "status": "measured"}, + {"corpus": "en", "windows": 16, "ppl_ternary": 15.7985, "ppl_baseline": 12.8639, "penalty_pct": 22.81, + "status": "measured, superseded prefix of the 40-window run, reproduced bit-exactly at window 16"}, + {"corpus": "ru_v1", "windows": 10, "ppl_ternary": 5.7293, "ppl_baseline": 4.3812, "penalty_pct": 30.77, + "status": "measured, superseded prefix of the 40-window run, reproduced bit-exactly at window 10"} + ], + "retracted_claim": { + "text": "ternarity costs more on Russian than on English", + "based_on": "RU 10 windows (+30.77%) vs EN 16 windows (+22.81%)", + "refuted_by": "40-window runs: EN +27.70 +/- 5.84 pp vs RU +25.26 +/- 4.66 pp, intervals overlap; the ordering reverses", + "conclusion": "no per-language difference in the ternary penalty can be claimed from this data" + }, + "not_evaluated": ["full wikitext-2 (336 windows)", "fp16 baseline (only Q8_0 measured)", + "downstream task quality", "anything on FPGA hardware"] +} diff --git a/paper3-rossiya30-troica/measurements/ternary_ppl_ru_2026-08/run.sh b/paper3-rossiya30-troica/measurements/ternary_ppl_ru_2026-08/run.sh new file mode 100755 index 0000000..d0209db --- /dev/null +++ b/paper3-rossiya30-troica/measurements/ternary_ppl_ru_2026-08/run.sh @@ -0,0 +1,14 @@ +#!/bin/bash +# Exact perplexity protocol. Runtime: fork PrismML-Eng/llama.cpp commit 9ca265a, cmake Release, GGML_NATIVE=ON. +# Models: /tmp/bonsai17.gguf (Ternary-Bonsai-1.7B-Q2_0, sha256 d97d94eb5645...) +# /tmp/qwen17q8.gguf (Qwen3-1.7B-Q8_0, sha256 061b54daade0...) +# Sequential, never in parallel: the host has 2 cores and concurrent runs would corrupt the timing. +set -e +CHUNKS=${CHUNKS:-40} +CORPUS=${CORPUS:-/tmp/eval_ru_v2.txt} +cd /tmp/lcpp/build/bin +export LD_LIBRARY_PATH=/tmp/lcpp/build/bin +for m in bonsai17 qwen17q8; do + ./llama-perplexity -m /tmp/$m.gguf -f "$CORPUS" -c 512 --chunks "$CHUNKS" -t 2 --seed 42 \ + 2>&1 | tee "/tmp/ppl_$(basename $CORPUS .txt)_$m.log" +done From abeefc99e93339b0e702c4a11478005a6ab697fe Mon Sep 17 00:00:00 2001 From: gHashTag Date: Tue, 18 Aug 2026 17:50:57 +0000 Subject: [PATCH 2/2] measurements: six-point bits-per-weight vs bits-per-token curve (40 windows, EN+RU) Key finding: the +25..28% penalty measured yesterday is the price of going below ~3.5 bit/weight in general, not a property of ternary representation - post-hoc Q2_K at 3.064 bpw pays the same (Q3_K_M -> Q2_K: +25.1 +/- 5.5 pp EN, +16.6 +/- 4.3 pp RU). Ternary at 1.825 bpw is statistically indistinguishable from Q2_K at 3.064 bpw (-2.50 +/- 4.42 pp EN, +2.68 +/- 3.75 pp RU): same quality for 40% fewer bits. Above ~4.4 bpw the curve is flat: Q4_K_M, Q6_K and Q8_0 differ by 0.3-1.4% at 3.8-4.6 pp uncertainty. Includes raw logs, sha256 of every downloaded file, and machine-readable curve_results.json. --- .../ternary_ppl_ru_2026-08/CURVE_ru.md | 60 ++++++++++++++++ .../ternary_ppl_ru_2026-08/curve_results.json | 68 +++++++++++++++++++ .../ternary_ppl_ru_2026-08/curve_sha256.txt | 4 ++ .../ternary_ppl_ru_2026-08/ppl_en_qw_Q2_K.log | 15 ++++ .../ppl_en_qw_Q3_K_M.log | 15 ++++ .../ppl_en_qw_Q4_K_M.log | 15 ++++ .../ternary_ppl_ru_2026-08/ppl_en_qw_Q6_K.log | 15 ++++ .../ppl_ru_v2_qw_Q2_K.log | 15 ++++ .../ppl_ru_v2_qw_Q3_K_M.log | 15 ++++ .../ppl_ru_v2_qw_Q4_K_M.log | 15 ++++ .../ppl_ru_v2_qw_Q6_K.log | 15 ++++ 11 files changed, 252 insertions(+) create mode 100644 paper3-rossiya30-troica/measurements/ternary_ppl_ru_2026-08/CURVE_ru.md create mode 100644 paper3-rossiya30-troica/measurements/ternary_ppl_ru_2026-08/curve_results.json create mode 100644 paper3-rossiya30-troica/measurements/ternary_ppl_ru_2026-08/curve_sha256.txt create mode 100644 paper3-rossiya30-troica/measurements/ternary_ppl_ru_2026-08/ppl_en_qw_Q2_K.log create mode 100644 paper3-rossiya30-troica/measurements/ternary_ppl_ru_2026-08/ppl_en_qw_Q3_K_M.log create mode 100644 paper3-rossiya30-troica/measurements/ternary_ppl_ru_2026-08/ppl_en_qw_Q4_K_M.log create mode 100644 paper3-rossiya30-troica/measurements/ternary_ppl_ru_2026-08/ppl_en_qw_Q6_K.log create mode 100644 paper3-rossiya30-troica/measurements/ternary_ppl_ru_2026-08/ppl_ru_v2_qw_Q2_K.log create mode 100644 paper3-rossiya30-troica/measurements/ternary_ppl_ru_2026-08/ppl_ru_v2_qw_Q3_K_M.log create mode 100644 paper3-rossiya30-troica/measurements/ternary_ppl_ru_2026-08/ppl_ru_v2_qw_Q4_K_M.log create mode 100644 paper3-rossiya30-troica/measurements/ternary_ppl_ru_2026-08/ppl_ru_v2_qw_Q6_K.log diff --git a/paper3-rossiya30-troica/measurements/ternary_ppl_ru_2026-08/CURVE_ru.md b/paper3-rossiya30-troica/measurements/ternary_ppl_ru_2026-08/CURVE_ru.md new file mode 100644 index 0000000..19ae389 --- /dev/null +++ b/paper3-rossiya30-troica/measurements/ternary_ppl_ru_2026-08/CURVE_ru.md @@ -0,0 +1,60 @@ +# Кривая «бит на вес против бита на токен»: шесть точек, один и тот же протокол + +Дата: 18.08.2026. Продолжение `README_ru.md`. Тот же протокол: окно 512, 40 окон (20 480 токенов), +`-t 2`, seed 42, тот же рантайм, те же два корпуса с теми же sha256. + +Бит на вес считается как `размер файла × 8 / 2 031 000 000 параметров` (полный файл, включая +эмбеддинги и fp16-масштабы) — это `[смоделировано]` поверх точных размеров файлов, а не +заявленная битность блока. + +## Результат `[измерено]` + +| Модель | бит/вес | Английский, PPL | Русский, PPL | к Q6_K (EN) | +|---|---:|---:|---:|---:| +| Ternary-Bonsai-1.7B-Q2_0 (троичная) | 1,825 | 21,2119 ± 0,69302 | 6,8611 ± 0,17512 | +29,5 % | +| Qwen3-1.7B-Q2_K | 3,064 | 21,7566 ± 0,68516 | 6,6817 ± 0,17448 | +32,8 % | +| Qwen3-1.7B-Q3_K_M | 3,701 | 17,3925 ± 0,54170 | 5,7318 ± 0,15185 | +6,2 % | +| Qwen3-1.7B-Q4_K_M | 4,362 | 16,5360 ± 0,51957 | 5,5113 ± 0,14769 | +1,0 % | +| Qwen3-1.7B-Q6_K | 5,584 | **16,3794 ± 0,51749** | **5,4614 ± 0,14672** | 0 | +| Qwen3-1.7B-Q8_0 | 7,226 | 16,6104 ± 0,53106 | 5,4776 ± 0,14849 | +1,4 % | + +## Три вывода + +**1. Выше ≈4,4 бита/вес кривая плоская.** Q4_K_M, Q6_K и Q8_0 различаются на 0,3–1,4 % при +погрешности отношения 3,8–4,6 п.п. — то есть неразличимы. Q8_0 даже чуть хуже Q6_K +(+1,41 ± 4,56 п.п.), что при перекрывающихся интервалах означает «то же самое», а не +«8 бит хуже 6». Платить за 7,2 бита вместо 4,4 нечем. + +**2. Обрыв лежит между 3,7 и 3,1 бита, и он не про троичность.** Q3_K_M → Q2_K даёт ++25,1 ± 5,5 п.п. на английском и +16,6 ± 4,3 п.п. на русском. Ровно тот же порядок штрафа, +что мы вчера назвали «ценой троичности». **Значит наблюдённые +25…28 % — это цена ухода +ниже ≈3,5 бита вообще, а не свойство троичного представления.** Формулировку «цена +троичности» в наших материалах надо заменить на «цена низкой битности». + +**3. Троичная модель на 1,825 бита неотличима от 2-битной PTQ-квантизации на 3,064 бита.** +Английский: −2,50 ± 4,42 п.п. в пользу троичной; русский: +2,68 ± 3,75 п.п. в пользу Q2_K. +Оба интервала накрывают нуль. То есть при **на 40 % меньшем числе бит** качество то же. +Это и есть содержательный аргумент за троичное обучение — не «лучше», а **«столько же +качества за существенно меньше бит»**, и только в сравнении с агрессивной пост-квантизацией, +не с 4-битной. + +## Границы + +1. Троичная модель — **другая обученная модель**, а не Qwen3 после ternarization. Сравнение + отвечает на вопрос «что доступно сегодня при данном бюджете бит», а не «что делает + ternarization с этой моделью» — `[not-evaluated]`. +2. Точки Q2_K…Q6_K взяты из [unsloth/Qwen3-1.7B-GGUF](https://huggingface.co/unsloth/Qwen3-1.7B-GGUF) + (imatrix-калиброванные), а Q8_0 — из официального + [Qwen/Qwen3-1.7B-GGUF](https://huggingface.co/Qwen/Qwen3-1.7B-GGUF). Это разные конвейеры + квантизации; сравнивать Q8_0 с остальными точками как «одну кривую» строго нельзя. + Отсюда же возможное объяснение, почему Q8_0 не лучше Q6_K. +3. 20 480 токенов на точку, не полный wikitext-2 — `[not-evaluated]`. +4. Всё на процессоре; на плате AX7203 не измерено ничего — `[not-evaluated]`. +5. Битность считается по всему файлу; сравнение по «бит на вес» не учитывает, что у разных + схем разная доля неквантованных тензоров. + +## Провенанс + +sha256 всех скачанных файлов и сырые логи — в этом каталоге (`ppl_en_qw_*.log`, +`ppl_ru_v2_qw_*.log`), машинная сводка — `curve_results.json`. Протокол воспроизведения — +`run.sh` с переменными `CORPUS` и `CHUNKS`. diff --git a/paper3-rossiya30-troica/measurements/ternary_ppl_ru_2026-08/curve_results.json b/paper3-rossiya30-troica/measurements/ternary_ppl_ru_2026-08/curve_results.json new file mode 100644 index 0000000..2e5c55e --- /dev/null +++ b/paper3-rossiya30-troica/measurements/ternary_ppl_ru_2026-08/curve_results.json @@ -0,0 +1,68 @@ +[ + { + "model": "Bonsai-1.7B-Q2_0 (ternary)", + "bytes": 463290464, + "bpw": 1.825, + "en": 21.2119, + "en_err": 0.69302, + "ru": 6.8611, + "ru_err": 0.17512, + "en_vs_q6k_pct": 29.5, + "ru_vs_q6k_pct": 25.63 + }, + { + "model": "Qwen3-1.7B-Q2_K", + "bytes": 777796160, + "bpw": 3.064, + "en": 21.7566, + "en_err": 0.68516, + "ru": 6.6817, + "ru_err": 0.17448, + "en_vs_q6k_pct": 32.83, + "ru_vs_q6k_pct": 22.34 + }, + { + "model": "Qwen3-1.7B-Q3_K_M", + "bytes": 939539008, + "bpw": 3.701, + "en": 17.3925, + "en_err": 0.5417, + "ru": 5.7318, + "ru_err": 0.15185, + "en_vs_q6k_pct": 6.19, + "ru_vs_q6k_pct": 4.95 + }, + { + "model": "Qwen3-1.7B-Q4_K_M", + "bytes": 1107409472, + "bpw": 4.362, + "en": 16.536, + "en_err": 0.51957, + "ru": 5.5113, + "ru_err": 0.14769, + "en_vs_q6k_pct": 0.96, + "ru_vs_q6k_pct": 0.91 + }, + { + "model": "Qwen3-1.7B-Q6_K", + "bytes": 1417755200, + "bpw": 5.584, + "en": 16.3794, + "en_err": 0.51749, + "ru": 5.4614, + "ru_err": 0.14672, + "en_vs_q6k_pct": 0.0, + "ru_vs_q6k_pct": 0.0 + }, + { + "model": "Qwen3-1.7B-Q8_0 (official)", + "bytes": 1834426016, + "bpw": 7.226, + "en": 16.6104, + "en_err": 0.53106, + "ru": 5.4776, + "ru_err": 0.14849, + "en_vs_q6k_pct": 1.41, + "ru_vs_q6k_pct": 0.3 + } +] \ No newline at end of file diff --git a/paper3-rossiya30-troica/measurements/ternary_ppl_ru_2026-08/curve_sha256.txt b/paper3-rossiya30-troica/measurements/ternary_ppl_ru_2026-08/curve_sha256.txt new file mode 100644 index 0000000..3431b3e --- /dev/null +++ b/paper3-rossiya30-troica/measurements/ternary_ppl_ru_2026-08/curve_sha256.txt @@ -0,0 +1,4 @@ +62b3fb705434cb57fabc59d59aa7b4c6fb558fff7c7c4b2ce67456373bc30fd3 /tmp/qw_Q2_K.gguf +8070f21b9763ef653289e20252592d0aab6680948f36eb5b4362d403fef09170 /tmp/qw_Q3_K_M.gguf +b139949c5bd74937ad8ed8c8cf3d9ffb1e99c866c823204dc42c0d91fa181897 /tmp/qw_Q4_K_M.gguf +6a9cadec4883df6f3efcf337103479dcc6a3efa9f5f8cc64a904dba57808207a /tmp/qw_Q6_K.gguf diff --git a/paper3-rossiya30-troica/measurements/ternary_ppl_ru_2026-08/ppl_en_qw_Q2_K.log b/paper3-rossiya30-troica/measurements/ternary_ppl_ru_2026-08/ppl_en_qw_Q2_K.log new file mode 100644 index 0000000..3075f96 --- /dev/null +++ b/paper3-rossiya30-troica/measurements/ternary_ppl_ru_2026-08/ppl_en_qw_Q2_K.log @@ -0,0 +1,15 @@ +0.00.028.323 I common_init_result: fitting params to device memory ... +0.00.028.333 I common_init_result: (for bugs during this step try to reproduce them with -fit off, or provide --verbose logs if the bug only occurs with -fit on) +0.00.565.923 I common_params_fit_impl: projected to use 989 MiB of host memory vs. 7966 MiB of total host memory +0.00.707.654 W load: control-looking token: 128247 '' was not control-type; this is probably a bug in the model. its type will be overridden +0.02.263.891 W llama_context: n_ctx_seq (512) < n_ctx_train (40960) -- the full capacity of the model will not be utilized +0.02.672.597 I common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable) +0.02.834.211 I +0.02.835.734 I system_info: n_threads = 2 (n_threads_batch = 2) / 2 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 | +0.02.835.779 I perplexity: tokenizing the input .. +0.02.981.247 I perplexity: tokenization took 145.45 ms +0.02.981.402 I perplexity: calculating perplexity over 40 chunks, n_ctx=512, batch_size=2048, n_seq=4 +1.01.150.349 I perplexity: 58.17 seconds per pass - ETA 9.68 minutes +[1]1.3919,[2]1.2768,[3]1.2513,[4]2.7463,[5]4.6659,[6]6.7311,[7]7.9114,[8]9.3015,[9]10.8807,[10]11.9575,[11]12.9523,[12]13.5317,[13]14.3852,[14]14.8889,[15]15.4752,[16]16.2889,[17]16.5643,[18]16.7767,[19]17.0441,[20]17.5954,[21]18.0158,[22]18.3024,[23]18.4419,[24]18.7299,[25]19.1166,[26]19.5828,[27]19.6174,[28]19.7396,[29]19.8178,[30]20.0222,[31]20.4043,[32]20.6763,[33]20.8127,[34]20.9591,[35]21.1862,[36]21.3797,[37]21.6809,[38]21.7430,[39]21.5918,[40]21.7566, +9.25.866.460 I Final estimate: PPL = 21.7566 +/- 0.68516 + diff --git a/paper3-rossiya30-troica/measurements/ternary_ppl_ru_2026-08/ppl_en_qw_Q3_K_M.log b/paper3-rossiya30-troica/measurements/ternary_ppl_ru_2026-08/ppl_en_qw_Q3_K_M.log new file mode 100644 index 0000000..7d4791a --- /dev/null +++ b/paper3-rossiya30-troica/measurements/ternary_ppl_ru_2026-08/ppl_en_qw_Q3_K_M.log @@ -0,0 +1,15 @@ +0.00.003.830 I common_init_result: fitting params to device memory ... +0.00.003.838 I common_init_result: (for bugs during this step try to reproduce them with -fit off, or provide --verbose logs if the bug only occurs with -fit on) +0.00.257.629 I common_params_fit_impl: projected to use 1144 MiB of host memory vs. 7966 MiB of total host memory +0.00.390.301 W load: control-looking token: 128247 '' was not control-type; this is probably a bug in the model. its type will be overridden +0.00.792.685 W llama_context: n_ctx_seq (512) < n_ctx_train (40960) -- the full capacity of the model will not be utilized +0.01.000.055 I common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable) +0.01.109.680 I +0.01.109.809 I system_info: n_threads = 2 (n_threads_batch = 2) / 2 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 | +0.01.109.841 I perplexity: tokenizing the input .. +0.01.240.698 I perplexity: tokenization took 130.843 ms +0.01.240.838 I perplexity: calculating perplexity over 40 chunks, n_ctx=512, batch_size=2048, n_seq=4 +1.03.838.538 I perplexity: 62.60 seconds per pass - ETA 10.42 minutes +[1]1.3206,[2]1.2213,[3]1.2164,[4]2.4917,[5]4.0649,[6]5.6985,[7]6.7413,[8]7.8094,[9]9.1467,[10]10.0105,[11]10.7722,[12]11.3450,[13]11.9708,[14]12.3948,[15]12.8635,[16]13.4882,[17]13.6169,[18]13.8235,[19]13.9265,[20]14.3026,[21]14.5496,[22]14.8258,[23]14.9447,[24]15.1835,[25]15.4916,[26]15.8437,[27]15.8826,[28]15.9795,[29]15.9858,[30]16.1339,[31]16.3708,[32]16.5637,[33]16.6604,[34]16.7734,[35]16.9935,[36]17.1459,[37]17.3533,[38]17.4052,[39]17.2951,[40]17.3925, +10.23.280.986 I Final estimate: PPL = 17.3925 +/- 0.54170 + diff --git a/paper3-rossiya30-troica/measurements/ternary_ppl_ru_2026-08/ppl_en_qw_Q4_K_M.log b/paper3-rossiya30-troica/measurements/ternary_ppl_ru_2026-08/ppl_en_qw_Q4_K_M.log new file mode 100644 index 0000000..a841cc1 --- /dev/null +++ b/paper3-rossiya30-troica/measurements/ternary_ppl_ru_2026-08/ppl_en_qw_Q4_K_M.log @@ -0,0 +1,15 @@ +0.00.005.980 I common_init_result: fitting params to device memory ... +0.00.006.058 I common_init_result: (for bugs during this step try to reproduce them with -fit off, or provide --verbose logs if the bug only occurs with -fit on) +0.00.319.842 I common_params_fit_impl: projected to use 933 MiB of host memory vs. 7966 MiB of total host memory +0.00.454.316 W load: control-looking token: 128247 '' was not control-type; this is probably a bug in the model. its type will be overridden +0.01.525.746 W llama_context: n_ctx_seq (512) < n_ctx_train (40960) -- the full capacity of the model will not be utilized +0.01.735.449 I common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable) +0.01.841.249 I +0.01.841.389 I system_info: n_threads = 2 (n_threads_batch = 2) / 2 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 | +0.01.841.415 I perplexity: tokenizing the input .. +0.01.968.239 I perplexity: tokenization took 126.811 ms +0.01.968.371 I perplexity: calculating perplexity over 40 chunks, n_ctx=512, batch_size=2048, n_seq=4 +0.44.489.155 I perplexity: 42.51 seconds per pass - ETA 7.08 minutes +[1]1.2807,[2]1.2073,[3]1.2019,[4]2.4242,[5]3.9383,[6]5.5435,[7]6.4929,[8]7.5356,[9]8.8143,[10]9.6176,[11]10.3637,[12]10.8460,[13]11.3835,[14]11.8438,[15]12.2102,[16]12.7862,[17]12.8783,[18]13.1283,[19]13.2189,[20]13.6077,[21]13.8379,[22]14.0765,[23]14.2590,[24]14.4675,[25]14.8062,[26]15.1023,[27]15.1396,[28]15.2048,[29]15.2614,[30]15.4023,[31]15.6229,[32]15.8127,[33]15.9126,[34]16.0220,[35]16.2142,[36]16.3451,[37]16.5386,[38]16.5759,[39]16.4472,[40]16.5360, +6.29.001.061 I Final estimate: PPL = 16.5360 +/- 0.51957 + diff --git a/paper3-rossiya30-troica/measurements/ternary_ppl_ru_2026-08/ppl_en_qw_Q6_K.log b/paper3-rossiya30-troica/measurements/ternary_ppl_ru_2026-08/ppl_en_qw_Q6_K.log new file mode 100644 index 0000000..57badbd --- /dev/null +++ b/paper3-rossiya30-troica/measurements/ternary_ppl_ru_2026-08/ppl_en_qw_Q6_K.log @@ -0,0 +1,15 @@ +0.00.005.708 I common_init_result: fitting params to device memory ... +0.00.005.717 I common_init_result: (for bugs during this step try to reproduce them with -fit off, or provide --verbose logs if the bug only occurs with -fit on) +0.00.281.765 I common_params_fit_impl: projected to use 1875 MiB of host memory vs. 7966 MiB of total host memory +0.00.419.030 W load: control-looking token: 128247 '' was not control-type; this is probably a bug in the model. its type will be overridden +0.04.501.798 W llama_context: n_ctx_seq (512) < n_ctx_train (40960) -- the full capacity of the model will not be utilized +0.04.711.770 I common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable) +0.04.904.057 I +0.04.904.189 I system_info: n_threads = 2 (n_threads_batch = 2) / 2 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 | +0.04.904.216 I perplexity: tokenizing the input .. +0.05.031.261 I perplexity: tokenization took 127.034 ms +0.05.031.396 I perplexity: calculating perplexity over 40 chunks, n_ctx=512, batch_size=2048, n_seq=4 +1.05.528.973 I perplexity: 60.50 seconds per pass - ETA 10.07 minutes +[1]1.3221,[2]1.2156,[3]1.2091,[4]2.4076,[5]3.8648,[6]5.4173,[7]6.3893,[8]7.4358,[9]8.7093,[10]9.5275,[11]10.2680,[12]10.7834,[13]11.3434,[14]11.7564,[15]12.1566,[16]12.6679,[17]12.7695,[18]13.0057,[19]13.1143,[20]13.4655,[21]13.7422,[22]13.9933,[23]14.1292,[24]14.3416,[25]14.6543,[26]14.9505,[27]14.9973,[28]15.0478,[29]15.1173,[30]15.2599,[31]15.4867,[32]15.6835,[33]15.7761,[34]15.8849,[35]16.0718,[36]16.2029,[37]16.4057,[38]16.4251,[39]16.3080,[40]16.3794, +9.50.945.033 I Final estimate: PPL = 16.3794 +/- 0.51749 + diff --git a/paper3-rossiya30-troica/measurements/ternary_ppl_ru_2026-08/ppl_ru_v2_qw_Q2_K.log b/paper3-rossiya30-troica/measurements/ternary_ppl_ru_2026-08/ppl_ru_v2_qw_Q2_K.log new file mode 100644 index 0000000..3373ac8 --- /dev/null +++ b/paper3-rossiya30-troica/measurements/ternary_ppl_ru_2026-08/ppl_ru_v2_qw_Q2_K.log @@ -0,0 +1,15 @@ +0.00.017.305 I common_init_result: fitting params to device memory ... +0.00.017.314 I common_init_result: (for bugs during this step try to reproduce them with -fit off, or provide --verbose logs if the bug only occurs with -fit on) +0.00.258.977 I common_params_fit_impl: projected to use 989 MiB of host memory vs. 7966 MiB of total host memory +0.00.391.207 W load: control-looking token: 128247 '' was not control-type; this is probably a bug in the model. its type will be overridden +0.00.950.417 W llama_context: n_ctx_seq (512) < n_ctx_train (40960) -- the full capacity of the model will not be utilized +0.01.099.426 I common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable) +0.01.224.860 I +0.01.225.023 I system_info: n_threads = 2 (n_threads_batch = 2) / 2 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 | +0.01.225.040 I perplexity: tokenizing the input .. +0.01.335.973 I perplexity: tokenization took 110.921 ms +0.01.336.081 I perplexity: calculating perplexity over 40 chunks, n_ctx=512, batch_size=2048, n_seq=4 +0.57.513.770 I perplexity: 56.18 seconds per pass - ETA 9.35 minutes +[1]5.7593,[2]7.0099,[3]5.6658,[4]3.9953,[5]4.6085,[6]4.2157,[7]4.4242,[8]4.5202,[9]4.9586,[10]5.3786,[11]5.7448,[12]5.8946,[13]6.1087,[14]5.7003,[15]5.8407,[16]5.9131,[17]5.8204,[18]5.7731,[19]5.8882,[20]5.8133,[21]5.8126,[22]5.8487,[23]5.8930,[24]6.0328,[25]6.0914,[26]6.2101,[27]6.3779,[28]6.4671,[29]6.5258,[30]6.5104,[31]6.4002,[32]6.4576,[33]6.5241,[34]6.5201,[35]6.6053,[36]6.5163,[37]6.5087,[38]6.5165,[39]6.5659,[40]6.6817, +9.21.687.896 I Final estimate: PPL = 6.6817 +/- 0.17448 + diff --git a/paper3-rossiya30-troica/measurements/ternary_ppl_ru_2026-08/ppl_ru_v2_qw_Q3_K_M.log b/paper3-rossiya30-troica/measurements/ternary_ppl_ru_2026-08/ppl_ru_v2_qw_Q3_K_M.log new file mode 100644 index 0000000..c6d61e8 --- /dev/null +++ b/paper3-rossiya30-troica/measurements/ternary_ppl_ru_2026-08/ppl_ru_v2_qw_Q3_K_M.log @@ -0,0 +1,15 @@ +0.00.003.739 I common_init_result: fitting params to device memory ... +0.00.003.746 I common_init_result: (for bugs during this step try to reproduce them with -fit off, or provide --verbose logs if the bug only occurs with -fit on) +0.00.244.434 I common_params_fit_impl: projected to use 1144 MiB of host memory vs. 7966 MiB of total host memory +0.00.374.035 W load: control-looking token: 128247 '' was not control-type; this is probably a bug in the model. its type will be overridden +0.00.763.853 W llama_context: n_ctx_seq (512) < n_ctx_train (40960) -- the full capacity of the model will not be utilized +0.00.887.574 I common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable) +0.01.015.174 I +0.01.015.311 I system_info: n_threads = 2 (n_threads_batch = 2) / 2 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 | +0.01.015.326 I perplexity: tokenizing the input .. +0.01.126.594 I perplexity: tokenization took 111.255 ms +0.01.126.709 I perplexity: calculating perplexity over 40 chunks, n_ctx=512, batch_size=2048, n_seq=4 +1.03.691.661 I perplexity: 62.56 seconds per pass - ETA 10.42 minutes +[1]5.2847,[2]5.9525,[3]4.5901,[4]3.3383,[5]3.8921,[6]3.5475,[7]3.6859,[8]3.8374,[9]4.2189,[10]4.5717,[11]4.8645,[12]5.0500,[13]5.2605,[14]4.9679,[15]5.0756,[16]5.0884,[17]5.0010,[18]4.9582,[19]5.0646,[20]5.0285,[21]5.0216,[22]5.0705,[23]5.1111,[24]5.2322,[25]5.2878,[26]5.3682,[27]5.5046,[28]5.5814,[29]5.6152,[30]5.5984,[31]5.5116,[32]5.5633,[33]5.5914,[34]5.5739,[35]5.6363,[36]5.5669,[37]5.5699,[38]5.5886,[39]5.6422,[40]5.7318, +10.20.277.870 I Final estimate: PPL = 5.7318 +/- 0.15185 + diff --git a/paper3-rossiya30-troica/measurements/ternary_ppl_ru_2026-08/ppl_ru_v2_qw_Q4_K_M.log b/paper3-rossiya30-troica/measurements/ternary_ppl_ru_2026-08/ppl_ru_v2_qw_Q4_K_M.log new file mode 100644 index 0000000..1a22369 --- /dev/null +++ b/paper3-rossiya30-troica/measurements/ternary_ppl_ru_2026-08/ppl_ru_v2_qw_Q4_K_M.log @@ -0,0 +1,15 @@ +0.00.075.669 I common_init_result: fitting params to device memory ... +0.00.075.680 I common_init_result: (for bugs during this step try to reproduce them with -fit off, or provide --verbose logs if the bug only occurs with -fit on) +0.00.350.103 I common_params_fit_impl: projected to use 933 MiB of host memory vs. 7966 MiB of total host memory +0.00.551.633 W load: control-looking token: 128247 '' was not control-type; this is probably a bug in the model. its type will be overridden +0.01.378.903 W llama_context: n_ctx_seq (512) < n_ctx_train (40960) -- the full capacity of the model will not be utilized +0.01.505.614 I common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable) +0.01.639.229 I +0.01.639.363 I system_info: n_threads = 2 (n_threads_batch = 2) / 2 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 | +0.01.639.385 I perplexity: tokenizing the input .. +0.01.751.230 I perplexity: tokenization took 111.839 ms +0.01.751.350 I perplexity: calculating perplexity over 40 chunks, n_ctx=512, batch_size=2048, n_seq=4 +0.38.412.422 I perplexity: 36.66 seconds per pass - ETA 6.10 minutes +[1]5.0775,[2]5.8146,[3]4.5768,[4]3.3121,[5]3.8384,[6]3.5377,[7]3.6217,[8]3.7787,[9]4.1173,[10]4.4203,[11]4.7558,[12]4.9015,[13]5.0941,[14]4.8162,[15]4.9004,[16]4.9133,[17]4.8375,[18]4.7915,[19]4.8786,[20]4.8408,[21]4.8221,[22]4.8640,[23]4.8925,[24]5.0043,[25]5.0693,[26]5.1452,[27]5.2752,[28]5.3296,[29]5.3649,[30]5.3579,[31]5.2730,[32]5.3300,[33]5.3565,[34]5.3334,[35]5.4056,[36]5.3357,[37]5.3360,[38]5.3587,[39]5.4293,[40]5.5113, +6.21.972.850 I Final estimate: PPL = 5.5113 +/- 0.14769 + diff --git a/paper3-rossiya30-troica/measurements/ternary_ppl_ru_2026-08/ppl_ru_v2_qw_Q6_K.log b/paper3-rossiya30-troica/measurements/ternary_ppl_ru_2026-08/ppl_ru_v2_qw_Q6_K.log new file mode 100644 index 0000000..6c38ac5 --- /dev/null +++ b/paper3-rossiya30-troica/measurements/ternary_ppl_ru_2026-08/ppl_ru_v2_qw_Q6_K.log @@ -0,0 +1,15 @@ +0.00.003.531 I common_init_result: fitting params to device memory ... +0.00.003.539 I common_init_result: (for bugs during this step try to reproduce them with -fit off, or provide --verbose logs if the bug only occurs with -fit on) +0.00.259.681 I common_params_fit_impl: projected to use 1875 MiB of host memory vs. 7966 MiB of total host memory +0.00.398.999 W load: control-looking token: 128247 '' was not control-type; this is probably a bug in the model. its type will be overridden +0.00.521.582 W llama_context: n_ctx_seq (512) < n_ctx_train (40960) -- the full capacity of the model will not be utilized +0.00.722.682 I common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable) +0.00.989.382 I +0.00.989.510 I system_info: n_threads = 2 (n_threads_batch = 2) / 2 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 | +0.00.989.529 I perplexity: tokenizing the input .. +0.01.102.166 I perplexity: tokenization took 112.624 ms +0.01.102.287 I perplexity: calculating perplexity over 40 chunks, n_ctx=512, batch_size=2048, n_seq=4 +0.59.334.045 I perplexity: 58.23 seconds per pass - ETA 9.70 minutes +[1]5.0378,[2]5.7728,[3]4.5637,[4]3.3256,[5]3.8415,[6]3.5352,[7]3.6393,[8]3.7932,[9]4.1101,[10]4.3906,[11]4.6985,[12]4.8277,[13]5.0294,[14]4.7488,[15]4.8322,[16]4.8369,[17]4.7676,[18]4.7279,[19]4.8172,[20]4.7776,[21]4.7604,[22]4.7950,[23]4.8178,[24]4.9332,[25]4.9883,[26]5.0631,[27]5.1909,[28]5.2506,[29]5.2874,[30]5.2752,[31]5.1938,[32]5.2573,[33]5.2874,[34]5.2674,[35]5.3409,[36]5.2732,[37]5.2818,[38]5.3053,[39]5.3749,[40]5.4614, +9.43.837.164 I Final estimate: PPL = 5.4614 +/- 0.14672 +