cl100k_base
273,662,103 canonical BPE tokens.
Encode wall: 76.098723388 s.
Decode + hash: 12.579615231 s.
Whole-process RSS: 6,015,284 KiB.
Exact reconstruction: PASS.
NEREIDS / PHASE 3 / RECEIPT-BACKED
Nereids
exact structural interoperability.
One billion bytes. One fixed target vocabulary. Exact reconstruction.
Nereids projects an existing DU3 structural stream into IDs from the full
cl100k_base mergeable vocabulary while preserving the original
byte stream exactly.
1 GB EXACT
0 UNMAPPABLE
CL100K VIEW
LNCL 1.2 SOURCE-AVAILABLE
raw bytes
↓
DU3 structural representation
↓
canonical DU lexeme bytes
↓
precomputed DU GID → target ID[]
↓
cl100k_base mergeable IDs
↓
exact original bytes
raw SHA == DU SHA == target SHA
EXACT
1 GB RESULT
A tokenizer view without surrendering byte identity.
The tested target space is the complete cl100k_base mergeable
vocabulary from tiktoken 0.14.0. Nereids does not claim that its segmentation
equals canonical cl100k BPE segmentation.
| Measurement | Phase 3 receipt |
|---|---|
| Dataset | enwik9 |
| Input | 1,000,000,000 bytes |
| DU vocabulary entries | 2,229,308 |
| Target vocabulary | cl100k_base / tiktoken 0.14.0 |
| Target vocabulary entries | 100,256 |
| DU positions consumed | 402,990,603 |
| Target IDs emitted | 461,761,619 |
| Unmappable DU entries | 0 |
| Target-decoded bytes | 1,000,000,000 |
| Exact reconstruction | PASS |
Input SHA-256: 159b85351e5f76e60cbe32e04c677847a9ecba3adc79addab6f4c6c7aa3744bc
PHASE 3 RUNTIME
Precompute once.
Stream the view.
The DU dictionary → target-vocabulary map is built once for a given DU dictionary and target vocabulary pair. The warm path performs ID lookup and expansion over the existing DU structural stream.
map build 12.653240804 s
warm remap 17.628295101 s
throughput 56,726,983 B/s
logical chunks 10 × ~100 MB
peak RSS 234,464 KiB
whole process 31.85 s
warm remap 17.628295101 s
throughput 56,726,983 B/s
logical chunks 10 × ~100 MB
peak RSS 234,464 KiB
whole process 31.85 s
COMPARISON BOUNDARY
Same corpus.
Different starting point.
The recorded canonical tokenizer baseline starts from text. Nereids warm remap starts from an existing DU structural stream. These numbers are shown as separate receipts, not as a direct tokenizer speedup claim.
DU3 → cl100k IDs
461,761,619 target-vocabulary IDs emitted.
Warm remap wall: 17.628295101 s.
Peak RSS: 234,464 KiB.
Exact reconstruction: PASS.
No direct “4.3× faster tokenizer” claim is made. Matched end-to-end methodology is a separate future gate.
ARTIFACTS
Explicit state.
Measured size.
The target model is tokenizer-global and corpus-independent. The DU→target map depends on the specific DU dictionary and target vocabulary pair.
| Artifact | Bytes | Input share |
|---|---|---|
| enwik9 raw input | 1,000,000,000 | 100.0000% |
| DU3 merged.dict | 20,856,995 | 2.0857% |
| cl100k model.bin | 1,044,858 | 0.1045% |
| precomputed map ID payload | 26,518,400 | 2.6518% |
| current full CSR map | 44,352,872 | 4.4353% |
EXACTNESS BOUNDARY
Exact bytes ≠ canonical segmentation.
Nereids separates information identity from tokenizer segmentation. A tokenizer chooses an operational factorization of the byte stream; that factorization is not treated as the canonical identity underneath it.
Structural identity
Byte reconstruction.
DU lexeme reconstruction.
DU dictionary → target-vocabulary mapping.
Valid cl100k mergeable IDs.
Streamed 1 GB translation.
Raw / DU / target SHA equality.
Model semantics
Canonical cl100k BPE segmentation equality.
Drop-in native tokenizer equivalence.
KV-cache portability.
Hidden-state portability.
Learned projection exactness.
Rare1, rengrams or zRank integration.
ARCHITECTURE
Information first.
Tokenizer view second.
DU3 carries structural identity. Nereids restores each DU dictionary lexeme to canonical bytes before projection into the target vocabulary.
DU3 control representations are valid serialization, not corruption. Nereids decodes them back to canonical lexeme bytes before target-vocabulary mapping.
RAW BYTES
↓
DU3
↓
canonical structural identity
↓
NEREIDS
↓
tokenizer / model vocabulary views
↓
DU3
↓
canonical structural identity
↓
NEREIDS
↓
tokenizer / model vocabulary views
THESIS
BPE is a view over the bytes,
not the bytes.
The representation required to preserve information exactly is not identical to the segmentation convention selected by a tokenizer.
Receipts before claims.
Exact where exactness is possible. Approximate where learning begins. Nereids is source-available under LNCL 1.2 and is not presented as OSI open-source software.
NEXT GATES
Compatibility before projection hype.
Each new adapter or structural layer gets its own exact reconstruction receipt before it becomes a public compatibility claim.
Matched DU3 methodology
Fresh DU3-v1g enwik9 measurement and matched end-to-end comparison.
Mistral / Qwen
Local model-I/O validation with explicit tokenizer-family receipts.
Proteus
Snapshots, deltas, bounded repair and model-I/O orchestration above the exact structural core.