commit 7c33aa94355c6ba4feb71d3b2c13c73c34adee29
parent acedb457be1a08f21d74c95ab53cc0825455b2b6
Author: Ryan Sepassi <rsepassi@gmail.com>
Date: Sun, 14 Jun 2026 06:23:54 -0700
doc(plan): FRONTEND-SHAPE §D.6 — P1 lean-lexer NEGATIVE (1.06x); representation isn't the gap; P1b (measure tcc's lexer) is the decisive next probe
Diffstat:
1 file changed, 32 insertions(+), 0 deletions(-)
diff --git a/doc/plan/FRONTEND-SHAPE.md b/doc/plan/FRONTEND-SHAPE.md
@@ -117,6 +117,38 @@ floor for a modular C compiler** and we stop there. This validation gates the sp
- **Fast/slow split + gate:** which constructs take the lean lane vs fall back, and
how the gate proves the two lanes emit **byte-identical** output.
+### D.6 Results — P1 negative; the decisive probe P1 left unmeasured
+
+**P1 (lean-lexer head-to-head, 2026-06-14) — NEGATIVE.** A throwaway tcc-shape lean
+lexer (int kind + small value, byte-offset loc only, interns identifiers only),
+faithful in lexical coverage (byte-identical token histogram to the real lexer —
+1,150,591 tokens over `sqlite3.c`), measured against the real `lex_next`:
+
+| lexer | instr/token |
+|---|--:|
+| real (28-byte `Tok`, per-token `SrcLoc`, interns spellings) | 425.7 |
+| lean (int + slice, byte-offset loc, idents-only intern) | 401.3 |
+| **ratio** | **1.06×** |
+
+1.19× even at the physically-impossible "intern *nothing*" ceiling — both well
+below the 1.3× floor. **Representation overhead is only ~6 % of the lexer's cost;**
+the rest is inherent scanning + per-token dispatch that both representations pay
+equally. This **falsifies "wholesale lean representation compounds" on the biggest,
+most self-contained phase** and, with the Path-A spikes (§D.2), strongly indicates
+Path B is also marginal. (Throwaway: branch `worktree-agent-a2c47ce2c0d4ab45f`,
+`experiments/p1_leanlex/`, unmerged.)
+
+**The number P1 did NOT measure — P1b, the actually-decisive probe.** P1 compared
+lean-kit vs real-kit; it never measured **tcc's** lexer. kit's lexer is ~425
+instr/token *regardless of representation*. Whether that is **above tcc's floor**
+(→ the lever is the scanning/dispatch **algorithm** — word-at-a-time / computed-goto
+— an axis *no* experiment has tested) or **at it** (→ the lexer is not the gap; the
+2.9× lives in pp/parse/types) is unknown and decides where to look next. For
+context, kit's raw lexer alone (~0.49 B over sqlite) already exceeds tcc's *entire*
+lex+pp (~0.33 B), so the lexer is implicated — but only P1b says whether the cause
+is algorithm or is mis-attributed. **P1b = add a tcc-lexer (`next_nomacro`) variant
+to the same micro-harness; measure its instr/token.** Cheap; completes the triangle.
+
---
## 0. Verified current state (what §4.3 already bought)
