Scale the corpus
enwik8/enwik9 tests with vocabulary pressure, sideband cost, speed, memory and failure modes.
Spearhead project / public RFC
Rare1
reversible sidebands.
Rare1 separates rare or singleton token IDs into a sideband while preserving exact reconstruction. The purpose is structural: test whether reducing common-vocabulary pressure makes downstream streams easier to inspect or predict.
$ okc rare1 validate enwik7
PASS: exact reconstruction
PASS: rare id sideband roundtrip
rare/singleton ids → sideband
common stream → lower active pressure
decoder → original ID sequence
STATUS: RFC, not universal claim.
Validated context
Evidence with its boundary attached.
These are reproducible experiment notes, not a claim that Rare1 replaces BPE, SentencePiece, zstd or production tokenizers.
| Area | Status | Meaning |
|---|---|---|
| DUMB / Redumb u32 roundtrip | PASS | Encoded streams can be reconstructed exactly in the current validation setup. |
| Rare1 ID reconstruction | PASS | Rare-token sideband reconstruction works on the current RFC corpus target. |
| Tiny TCN benchmark | INITIAL | Useful benchmark context; not a final performance or tokenizer claim. |
Observed experiment
Small signal. Exact restore.
In the controlled enwik7 tiny-TCN comparison, the same active vocabulary was used for the tested paths.
Rare1
Approximately 1.939 bits/byte
Observed under the documented tiny-TCN configuration, with exact ID-stream reconstruction.
SentencePiece BPE
Approximately 2.243 bits/byte
The comparison baseline for that specific benchmark, not a universal tokenizer ranking.
Approximate difference observed: 13.6% favourable to Rare1 under this metric and configuration. Independent reproduction remains the point.
Open validation targets
What still needs testing.
Scale, compare and measure the costs that a ratio alone hides.
Compare tokenizers
BPE and Unigram comparisons across token count, vocabulary, exactness, speed and memory.
Profile local hardware
ARM, edge and local AI measurements where bandwidth and memory matter more than raw storage.
Claim boundary
What Rare1 is — and is not.
Allowed
- Reversible RFC and validation target.
- Exact reconstruction in the tested setup.
- Rare-token sideband experiment.
- Vocabulary-pressure hypothesis for small/local regimes.
Not claimed
- No universal win over BPE or SentencePiece.
- No production-readiness claim.
- No SOTA tokenizer claim.
- No replacement claim for gzip, xz or zstd.
Public record