A hand-rolled, from-scratch mini decision engine in the spirit of Jev — ~680K trainable parameters, trains in minutes on a laptop CPU, serves a System One–compatible API with sub-2 ms latency.
This is a learning project. MiniJev was built to understand, from first principles, what a "System One" decision model is: states and questions in, complete probability distributions out, zero output-token decoding. It is small on purpose. It is not production software, and its out-of-distribution behavior has real, documented limitations (see Known limitations).
Jev-style decision models take a state (a ticket, a message, a JSON record) plus named questions, and answer with a full probability distribution over candidates — no autoregressive token generation. usage.output_tokens is always 0.
MiniJev implements this with everything reduced to candidate scoring:
| System One question type | Candidate set | Output |
|---|---|---|
noul (yes/no) |
[false, true], 2 candidates |
noul = p(true) |
choice (single select) |
criteria keys (2–255) | choice + confidence + probabilities |
score (ordinal) |
ordered criteria levels (2–10) | score + confidence + legend + probabilities |
(state, question, candidate) --concatenate--> single sequence
|
v
word-level tokenizer (lowercase + regex split; trained vocab: 473 words)
|
v
tiny Transformer encoder (3 layers, d_model=128, 4 heads, masked mean pooling)
|
v
scalar scoring head: one logit per candidate path
|
v
candidates of the same question -> softmax -> probability distribution
Training: candidate-level cross-entropy + label smoothing 0.05 (same protocol as jev-eval's DistilBERT baseline), followed by temperature calibration on dev.
# 1. Generate the synthetic dataset (8,700 states / ~18.6K questions)
python -m minijev.data
# 2. Train on CPU (~3M-param model, minutes)
python -m minijev.train --epochs 8
# 3. Evaluate (test + ood, three systems compared)
python -m minijev.evaluate
# 4. Serve the API
python -m minijev.server --port 8088Dependencies: torch + numpy. The API server itself is zero-dependency (stdlib http.server).
POST http://127.0.0.1:8088/v1/systemone
GET http://127.0.0.1:8088/v1/models
Request body (same top-level fields as the official API): model (required, minijev-latest), state (required), questions (required map of {type, instructions, criteria}).
curl -X POST http://127.0.0.1:8088/v1/systemone \
-H "Content-Type: application/json" \
-d '{
"model": "minijev-latest",
"state": "My card was charged twice for order A-104 and nobody replied in three days.",
"questions": {
"needs_human": {
"type": "noul",
"instructions": "Does this ticket need a human agent?",
"criteria": {"true": "money, legal, or an unanswered complaint",
"false": "a routine question a bot can close"}
},
"route": {
"type": "choice",
"instructions": "Which team should handle this?",
"criteria": {"billing": "payment or charge problems",
"shipping": "delivery problems",
"technical": "application bugs",
"account": "login and profile management"}
},
"urgency": {
"type": "score",
"instructions": "How urgent is this ticket?",
"criteria": ["low", "medium", "high"]
}
}
}'Response (abridged):
{
"model": "minijev-1.0.0",
"answers": {
"needs_human": {"type": "noul", "noul": 0.94},
"route": {"type": "choice", "choice": "billing", "confidence": 0.9, "probabilities": {"...": "..."}},
"urgency": {"type": "score", "score": "high", "confidence": 0.8, "legend": ["low", "medium", "high"], "probabilities": {"...": "..."}}
},
"usage": {"input_tokens": 612, "output_tokens": 0}
}Error semantics follow the official API: 403 authentication_error (only if MINIJEV_API_KEY is set), 400 invalid_request_error, 404 model_not_found.
Protocol aligned with jev-eval, plus two calibration metrics that jev-eval lacks:
| Metric | Definition | Why |
|---|---|---|
| ROC-AUC macro-OvR | macro one-vs-rest AUC over tasks (probability matrix, not argmax) | ranking quality |
| Accuracy / Macro-F1 | argmax hit rate / per-class macro F1 | decision quality |
| ECE (15 bins) | confidence vs. empirical accuracy gap | can confidence be used as a threshold directly? |
| Brier (multiclass) | mean over samples of mean_c (p_c − y_c)² | overall probability quality |
| Latency quantiles + throughput | end-to-end per state | the point of a tiny model |
Fail-first principle (inherited from jev-eval): NaN, probabilities not summing to 1, or sample-count mismatch abort the evaluation with an exception — no report is produced.
Baselines: majority (constant class-frequency distribution, lower bound) and keyword (hand-written keyword rules covering ~70% of the generation rules, simulating a hand-rolled rule engine).
Full report: results/eval_report.md (machine-readable: results/eval_report.json).
| System | Acc | Macro-F1 | ROC-AUC | ECE | Brier | p50 (ms) | Throughput/s |
|---|---|---|---|---|---|---|---|
| minijev | 1.0000 | 1.0000 | 1.0000 | 0.0015 | 0.0001 | 1.57 | 630 |
| keyword | 0.9639 | 0.9609 | 0.9764 | 0.1139 | 0.0211 | 0.005 | 182K |
| majority | 0.3688 | 0.1648 | 0.5000 | 0.0282 | 0.1802 | 0.002 | 548K |
| System | Acc | Macro-F1 | ROC-AUC | ECE | Brier |
|---|---|---|---|---|---|
| minijev | 0.8592 | 0.8881 | 0.9465 | 0.0737 | 0.0503 |
| keyword | 0.8571 | 0.8704 | 0.9144 | 0.0071 | 0.0607 |
| majority | 0.3340 | 0.1516 | 0.5000 | 0.0355 | 0.1844 |
Takeaways: in-distribution the model is perfect and well calibrated (ECE 0.0015 — confidence is directly usable as a threshold, which is the core Jev use case); latency is 1.6 ms p50 on a single CPU thread. The keyword baseline's ECE of 0.11 shows why rule engines have no usable confidence.
A hand-written holdout of 45 novel cases / 102 questions (holdout_eval.py, never seen in any split) exposes the ceiling of a word-level model:
- Holdout accuracy: 58.8% (60/102) — e.g. route 36%, intent 40%, satisfaction 40%.
- The trained vocabulary is only 473 words (the synthetic templates never exceed it); on hand-written phrasing, 42% of tokens become UNK, and per-case UNK rate correlates with errors (r = −0.47).
- Confidence breaks down out of distribution: mean confidence on errors (0.76) exceeds that on correct answers (0.71) — the model does not know what it does not know.
- Root causes — token coverage and semantic generalization — cannot be fixed by more training on the same data; they require a pretrained backbone. That is exactly what the follow-up project JevNext does (Qwen3-0.6B backbone + LoRA: same holdout, 86–91%).
This failure mode is documented deliberately: it is the most instructive part of the project.
All data is synthetic and rule-generated; gold labels are derived from the generation rules themselves (no human labeling of generated data).
| Set | Size | Purpose |
|---|---|---|
train / dev / test |
data/decisions.jsonl splits |
rule-generated ticket / message / review states; test = 1,000 held-out states from the same templates |
ood |
700 states | different phrasing templates + 35% adversarial distractor sentences (markers of other classes that do not change the gold answer) |
holdout |
45 cases / 102 questions in holdout_eval.py |
fully hand-written: synonym paraphrases, novel urgency/needs-human markers, adversarial distractors, far-OOD (German) |
The holdout is shared verbatim with JevNext so the two projects are directly comparable.
# Unit tests (19 tests, stdlib unittest — no pytest needed)
python -m unittest discover -s tests -v
# Holdout evaluation (requires the server on 8088)
python -m minijev.server --port 8088 &
python holdout_eval.py --url http://127.0.0.1:8088/v1/systemone \
--model minijev-latest --out results/holdout_report.json- Jev — System One models (TypeSafe) — the conceptual origin of "decision models: full distributions, zero output tokens". MiniJev's API format follows this spec. MiniJev is an independent learning project and is not affiliated with or endorsed by TypeSafe.
- NanoJev — a 0.6B nano replica of Jev (Qwen3-0.6B backbone); its scoring-head design informed this project.
- jev-eval — evaluation harness whose metric protocol (ROC-AUC / Macro-F1 / latency quantiles / fail-fast) this project follows.
- The follow-up project JevNext (same author) replaces the hand-rolled backbone with Qwen3-0.6B + LoRA.
This software is provided "as is" for educational purposes only, without warranty of any kind. The dataset is synthetic; the model has known out-of-distribution failure modes (see above). Do not use MiniJev to make or automate real-world decisions (routing, escalation, urgency triage, etc.) without human oversight. The authors accept no liability for any use of this software.
MIT — do whatever you want, attribution appreciated.