commit b792987a36a8ed5e352b0f523bf89b451c9d7ddc
parent 7c33aa94355c6ba4feb71d3b2c13c73c34adee29
Author: Ryan Sepassi <rsepassi@gmail.com>
Date: Sun, 14 Jun 2026 07:33:23 -0700
doc(plan): FRONTEND-SHAPE §D.6/D.7 — P1b POSITIVE (kit lexer 2.69x tcc/byte, algorithmic not representation); lever real but ~-12% bounded; P1c (tcc -E) next to locate the rest of the 2.9x
Diffstat:
1 file changed, 45 insertions(+), 0 deletions(-)
diff --git a/doc/plan/FRONTEND-SHAPE.md b/doc/plan/FRONTEND-SHAPE.md
@@ -149,6 +149,51 @@ lex+pp (~0.33 B), so the lexer is implicated — but only P1b says whether the c
is algorithm or is mis-attributed. **P1b = add a tcc-lexer (`next_nomacro`) variant
to the same micro-harness; measure its instr/token.** Cheap; completes the triangle.
+**P1b (tcc lexer, 2026-06-14) — POSITIVE; the lever is real but bounded.** Same
+micro-harness, three lexers over `sqlite3.c` (9,281,384 B), back-to-back, best-of-7,
+**instr/byte** (robust to token-segmentation differences between lexers):
+
+| lexer | instr/byte | instr/token |
+|---|--:|--:|
+| kit real `lex_next` | 52.8 | 425 |
+| kit lean (P1) | 49.7 | 401 |
+| **tcc `next_nomacro` (raw)** | **19.6** | **160** |
+
+**kit/tcc = 2.69× per byte** (2.53× even for *lean*-kit; stable across 3 sessions).
+The lexer carries a real ~2.7× excess that is **not** representation (P1: lean-vs-real
+is 6 %) and **not** deferred number/string decode (eager tcc decode is only +9.7 %).
+Decisive reconciliation: **lean-kit (401) is still 2.5× tcc (160)** despite a lean
+rep → the bulk is the **scan/dispatch algorithm**, not the token record — tcc's
+one-switch-into-per-class-inner-loops vs kit's cclass-table + a 28-byte `Tok` and a
+per-token `SrcLoc` built every token. **The lever: a word-at-a-time / computed-goto
+scanner that also drops per-token loc + punctuator spelling-intern** (puncts are
+53 % of the stream, interned for nothing — the parser switches on the code).
+**Bounded:** the lexer is ~24 % of `-c`, so a tcc-class scanner caps at ≈ **−12 %
+of total** — real and worth doing, but *not* a halving on its own. Throwaway: branch
+`worktree-agent-a99ab2852f8e4e000`, `experiments/p1b_tcclex/`.
+
+### D.7 Synthesis (post-P1b) and the next probe
+
+Three experiments trilaterate the gap: representation (P1, 1.06×), per-item caching
+(Path-A spikes, marginal), and copy elimination (tiers, ~1.2 %) are **all marginal**;
+the lexer carries a real **2.7× algorithmic** excess (P1b). So the 2.9× is **pervasive
+per-operation instruction density** — kit emits ~2.5–2.7× the instructions per unit
+work as tcc's hyper-tuned monolith — *not* one fixable layer, and *not* fixable by a
+leaner representation. The lexer is simply the first lever **measured** to be real
+(≤ −12 % ceiling).
+
+**The §D goal now hinges on one unanswered question:** does the same ~2.5× density
+excess exist — and is it *tunable* — in **pp + parse + types** (the other ~76 %)? If
+yes, a coordinated per-phase re-tuning campaign (scanner first) can approach the goal.
+If those phases' cost is **structural generality** (the CG type universe, the
+value-stack API, context-struct indirection) rather than tunable code density, then
+~2.5× is the floor and no amount of scanner work changes the total. **P1c (next
+probe): a kit-vs-tcc per-phase instruction breakdown** — `tcc -E` is the one missing
+number; combined with kit `-E`/`-c` and the P1/P1b raw-lex isolations it splits the
+gap into lex / pp / parse+emit and says where the remaining excess lives and whether
+it is attackable. Until P1c, do **not** invest in the scanner build — a −12 % lever is
+only worth it if the other phases are also tunable; otherwise the floor is ~2.5×.
+
---
## 0. Verified current state (what §4.3 already bought)
