commit 0f9b477324a56ef9ee19c54b592b703c5d5796e6
parent 51c694af85e1ba3b7eae6763b22431cd5802087b
Author: Ryan Sepassi <rsepassi@gmail.com>
Date: Mon, 15 Jun 2026 10:03:27 -0700
docs: define frontend shared cell shape
Diffstat:
1 file changed, 408 insertions(+), 706 deletions(-)
diff --git a/doc/plan/FRONTEND-SHAPE.md b/doc/plan/FRONTEND-SHAPE.md
@@ -1,731 +1,433 @@
-# Frontend shape: closing the gap to tcc's frontend (PERF.md §4.3)
-
-**Goal.** Reach tcc's *frontend shape* — the structural end-state PERF.md §4.3
-names: a shared mutable token slot fed by both scanner and macro replayer,
-identifiers resolved through symbol pointers cached on the interned token, and a
-thin `SValue[]` the parser drives by calling emit directly. The `CgTarget` →
-`NativeTarget` vtable polymorphism **stays** (the two indirect calls per
-primitive are the price of the seven-backend modularity, per §4.3); this plan is
-about the *front half and the value-stack seam*, not the backend vtable collapse.
-
-This doc is the sequenced campaign. It is grounded in a code audit (2026-06-13)
-whose headline finding reframes the work: **most of §4.3 is already landed.** The
-"large refactor" is, in reality, one finished pillar, one ~70%-done pillar, and
-one genuine-but-bounded structural change. Each tier below is independently
-landable and individually gated.
-
-> **Superseded for the instruction goal (2026-06-14).** The §4.3 tiers below were
-> built and measured (three parallel worktree branches, all byte-identical) — they
-> are real but **marginal**: Tier 1 (1A+1B) −0.70% on `-c`, Tier 2 −0.49%, Tier 3
-> instruction-neutral; ≈ **−1.2% combined**. They remain worthwhile *cleanup*, but
-> they do not move the real number. The committed direction for the actual target —
-> **halving `-c` instructions (within 1.5× of tcc)** — is **§D** immediately below.
-> Treat §0 and the Tier plan as the landed/landable cleanup record beneath it.
-
----
-
-## D. Direction (2026-06-14): halving instructions — Path B, no layer fusion
-
-The goal is no longer "reach §4.3's shape" (mostly landed, and marginal). It is
-**halve `-c` retired instructions** — 1.91 B → ~0.95 B, i.e. within 1.5× of tcc
-(0.665 B). This section records what governs that, what was ruled out, and the
-committed bet.
-
-### D.1 The arithmetic that governs everything
-
-| phase | kit `-c` | share | vs tcc |
-|---|--:|--:|--:|
-| **lex + pp** | **~0.93 B** | **49%** | ~2.8× |
-| parse + sema + types + CG-drive | ~0.71 B | 37% | ~3.5× |
-| native emit + objwrite | ~0.27 B | 14% | ~7× |
-| **total** | **1.91 B** | | **2.87×** |
-
-**kit's preprocessor alone (0.93 B) is 1.4× tcc's entire compile (0.665 B).** So
-the target is unreachable by improving parse/types/codegen alone — even zeroing
-emit leaves 1.64 B (2.5×). Halving the total means **roughly halving every phase**;
-the cost is **uniform ~2.9× with no single dominant hot function** (the doc's
-long-standing observation). That uniformity is the whole problem: there is nothing
-to "crush," only a pervasive per-token / per-operation tax to remove everywhere.
-
-### D.2 What was ruled out (measured, not assumed)
-
-- **Micro-opts / seam-preserving copy elimination — exhausted.** The three §4.3
- tiers move `-c` ~1.2% combined. This *is* the data point: the incremental surface
- is spent.
-- **Path A (precompute / cache) — DEAD, spiked marginal.** Two validation spikes:
- *lever 1* — kill per-token `SrcLoc`, recover location lazily — **marginal**;
- *lever 3* — classify-once type facts (kill the `unalias`/`pred_bits`/`class`/
- `size` re-derivation), carried inline — **marginal**. The spikes prove the cost
- is **not the per-item work** (which caching removes) but the **operation count
- and the dispatch structure itself**. Making each operation cheaper is futile when
- the number of operations is the tax.
-- **Layer fusion — ruled out by decision** (see D.3). Collapsing lex→pp→parse→emit
- into one mega-pass would buy the uniform inner-loop win, but at the cost of the
- architecture's legibility and reuse (the shared cpp backing `cc -E`/`cpp`, the
- separable parser, the CgTarget seam + seven backends). Not pursued.
-
-### D.3 The committed bet: Path B = a lean hot-path *representation*, not control fusion
-
-**Decision: Path B only, and explicitly without layer fusion.** Path B is a
-**representation pivot, not a control-flow pivot.**
-
-- **Lean wholesale representation.** The C `-O0` hot path stops paying for
- generality: a **lean token** (no per-token heft/loc-stamp), an **int-bitmask
- type** carried directly on a **lean value slot**, bypassing the general CG type
- universe and the public `ApiSValue` value-stack API. The wager is that the lean
- representation's savings **compound when adopted wholesale** — the inner loop gets
- uniformly cheaper across *all* phases at once — in a way the incremental spikes
- (a lean *cache layered on top of* the heavy objects) structurally could not show.
-- **No layer fusion — but APIs and handoffs are fair game.** The pipeline stays
- factored: **lexer | preprocessor | parser | CG | emit remain distinct,
- separately-reusable modules** — the test is that the cpp still stands alone for
- `cc -E`/`cpp`, the parser stays separable, and the `CgTarget` seam + seven
- backends stay. What is preserved is module **separability**, *not* the current
- interfaces: the **inter-module APIs and per-token handoffs are explicitly in scope
- to redesign** (e.g. call/return-per-token → a shared cursor over a lean token
- buffer, or a leaner pull). We do **not** merge the stages' control flow into one
- pass. **The win comes from the data that flows and how it is handed off, not from
- merging the loops.**
-
-In one line: **replace the heavy general data objects with lean ones, and the
-per-token handoffs with cheap ones; leave the modules themselves as separable as
-they are.**
-
-### D.4 The load-bearing risk — validate before building the full lane
-
-The spikes were marginal; Path B bets that **wholesale adoption compounds where
-incremental did not.** That bet is **unproven**, and Path B is expensive (a parallel
-lean lane plus a fast/slow split to gate). So the next step is **not** to build the
-lane — it is to **build a minimal end-to-end lean prototype** (a reduced-grammar
-path carrying lean token + int-type + lean value slot from bytes to emitted bytes)
-and measure whether the compounding actually appears on a representative slice. If
-it does not, Path B is also marginal and the honest conclusion is that **~2× is the
-floor for a modular C compiler** and we stop there. This validation gates the spend.
-
-### D.5 Open design questions for the lean lane (after D.4 validates)
-
-- **Inter-layer handoff (sanctioned lever, not just open):** redesign how separate
- modules pass the lean representation — kill the per-token call/copy tax
- (call/return-per-token → a shared cursor over a lean token buffer, or a leaner
- pull) *without* collapsing the loops. Module **separability** is the only
- invariant; the interfaces between modules are free to change.
-- **Int-bitmask type on the hot path:** the encoding, and how it **falls back** to
- the full `Type` / CG-type universe for what a bitmask can't carry (aggregates,
- bitfields, VLAs, `_Atomic`, wide/`__int128`).
-- **Lean value slot:** what the parser drives that still resolves to the **kept**
- `CgTarget` seam for emission (the seam is cheap — indirection was never the cost).
-- **Fast/slow split + gate:** which constructs take the lean lane vs fall back, and
- how the gate proves the two lanes emit **byte-identical** output.
-
-### D.6 Results — P1 negative; the decisive probe P1 left unmeasured
-
-**P1 (lean-lexer head-to-head, 2026-06-14) — NEGATIVE.** A throwaway tcc-shape lean
-lexer (int kind + small value, byte-offset loc only, interns identifiers only),
-faithful in lexical coverage (byte-identical token histogram to the real lexer —
-1,150,591 tokens over `sqlite3.c`), measured against the real `lex_next`:
-
-| lexer | instr/token |
-|---|--:|
-| real (28-byte `Tok`, per-token `SrcLoc`, interns spellings) | 425.7 |
-| lean (int + slice, byte-offset loc, idents-only intern) | 401.3 |
-| **ratio** | **1.06×** |
-
-1.19× even at the physically-impossible "intern *nothing*" ceiling — both well
-below the 1.3× floor. **Representation overhead is only ~6 % of the lexer's cost;**
-the rest is inherent scanning + per-token dispatch that both representations pay
-equally. This **falsifies "wholesale lean representation compounds" on the biggest,
-most self-contained phase** and, with the Path-A spikes (§D.2), strongly indicates
-Path B is also marginal. (Throwaway: branch `worktree-agent-a2c47ce2c0d4ab45f`,
-`experiments/p1_leanlex/`, unmerged.)
-
-**The number P1 did NOT measure — P1b, the actually-decisive probe.** P1 compared
-lean-kit vs real-kit; it never measured **tcc's** lexer. kit's lexer is ~425
-instr/token *regardless of representation*. Whether that is **above tcc's floor**
-(→ the lever is the scanning/dispatch **algorithm** — word-at-a-time / computed-goto
-— an axis *no* experiment has tested) or **at it** (→ the lexer is not the gap; the
-2.9× lives in pp/parse/types) is unknown and decides where to look next. For
-context, kit's raw lexer alone (~0.49 B over sqlite) already exceeds tcc's *entire*
-lex+pp (~0.33 B), so the lexer is implicated — but only P1b says whether the cause
-is algorithm or is mis-attributed. **P1b = add a tcc-lexer (`next_nomacro`) variant
-to the same micro-harness; measure its instr/token.** Cheap; completes the triangle.
-
-**P1b (tcc lexer, 2026-06-14) — POSITIVE; the lever is real but bounded.** Same
-micro-harness, three lexers over `sqlite3.c` (9,281,384 B), back-to-back, best-of-7,
-**instr/byte** (robust to token-segmentation differences between lexers):
-
-| lexer | instr/byte | instr/token |
-|---|--:|--:|
-| kit real `lex_next` | 52.8 | 425 |
-| kit lean (P1) | 49.7 | 401 |
-| **tcc `next_nomacro` (raw)** | **19.6** | **160** |
-
-**kit/tcc = 2.69× per byte** (2.53× even for *lean*-kit; stable across 3 sessions).
-The lexer carries a real ~2.7× excess that is **not** representation (P1: lean-vs-real
-is 6 %) and **not** deferred number/string decode (eager tcc decode is only +9.7 %).
-Decisive reconciliation: **lean-kit (401) is still 2.5× tcc (160)** despite a lean
-rep → the bulk is the **scan/dispatch algorithm**, not the token record — tcc's
-one-switch-into-per-class-inner-loops vs kit's cclass-table + a 28-byte `Tok` and a
-per-token `SrcLoc` built every token. **The lever: a word-at-a-time / computed-goto
-scanner that also drops per-token loc + punctuator spelling-intern** (puncts are
-53 % of the stream, interned for nothing — the parser switches on the code).
-**Bounded:** the lexer is ~24 % of `-c`, so a tcc-class scanner caps at ≈ **−12 %
-of total** — real and worth doing, but *not* a halving on its own. Throwaway: branch
-`worktree-agent-a99ab2852f8e4e000`, `experiments/p1b_tcclex/`.
-
-### D.7 Synthesis (post-P1b) and the next probe
-
-Three experiments trilaterate the gap: representation (P1, 1.06×), per-item caching
-(Path-A spikes, marginal), and copy elimination (tiers, ~1.2 %) are **all marginal**;
-the lexer carries a real **2.7× algorithmic** excess (P1b). So the 2.9× is **pervasive
-per-operation instruction density** — kit emits ~2.5–2.7× the instructions per unit
-work as tcc's hyper-tuned monolith — *not* one fixable layer, and *not* fixable by a
-leaner representation. The lexer is simply the first lever **measured** to be real
-(≤ −12 % ceiling).
-
-**The §D goal now hinges on one unanswered question:** does the same ~2.5× density
-excess exist — and is it *tunable* — in **pp + parse + types** (the other ~76 %)? If
-yes, a coordinated per-phase re-tuning campaign (scanner first) can approach the goal.
-If those phases' cost is **structural generality** (the CG type universe, the
-value-stack API, context-struct indirection) rather than tunable code density, then
-~2.5× is the floor and no amount of scanner work changes the total. **P1c (next
-probe): a kit-vs-tcc per-phase instruction breakdown** — `tcc -E` is the one missing
-number; combined with kit `-E`/`-c` and the P1/P1b raw-lex isolations it splits the
-gap into lex / pp / parse+emit and says where the remaining excess lives and whether
-it is attackable. Until P1c, do **not** invest in the scanner build — a −12 % lever is
-only worth it if the other phases are also tunable; otherwise the floor is ~2.5×.
-
-### D.8 P1c result (2026-06-14): the gap is a UNIFORM constant factor — floor confirmed
-
-`-E` is the **wrong instrument**: it is dominated by **text serialization**, and kit's
-serializer is *cheaper* than tcc's — kit `-E` 0.93 B **<** tcc `-E` 1.15 B — which
-masks the compile-path gap entirely. **Discard the §2 `-E`-derived phase ratios
-("lex+pp 2.8×, emit 7×") — serialization-contaminated.** Using only the clean
-isolations (raw-lex harnesses + full `-c`, no serialization; tcc `-c` re-measured =
-0.6635 B):
-
-| slice | kit | tcc | ratio | share of kit |
-|---|--:|--:|--:|--:|
-| raw lex (harness) | 0.489 B | 0.182 B | **2.69×** | 26 % |
-| everything else (`-c` − raw-lex) | 1.42 B | 0.482 B | **2.95×** | 74 % |
-| **whole `-c`** | **1.91 B** | **0.664 B** | **2.88×** | 100 % |
-
-**The gap is a uniform ~2.8× constant factor.** kit and tcc spend their instructions
-in nearly identical *proportions* (~26 % lex / ~74 % rest in **both**); kit is simply
-~2.8× denser **everywhere**. There is **no disproportionately-bad phase to crush** —
-"emit 7×" was a serialization artifact. This is the signature of **pervasive
-engineering-style overhead** (context-struct indirection, bounds checks, composable
-functions + call overhead, growable containers, the public-API / value-stack / type-
-universe layers, the kept vtable seam) vs tcc's hyper-tuned monolith — *not* an
-algorithm or representation localizable to one place.
-
-**Conclusion — the answer to "what halves `-c`":** nothing structural does. Halving
-the total means roughly halving kit's instruction density **pervasively, in every
-phase** — i.e. tcc's monolithic engineering — incompatible with the standing
-constraints (keep module separability, the seven backends, the `CgTarget` seam; no
-fusion). What *is* available:
-- **One concrete one-time lever:** a tcc-class scanner (word-at-a-time / computed-goto;
- drop per-token loc + punctuator spelling-intern) → lexer 2.69×→~1×, i.e. **≈ −12 %**
- of `-c`. Real, worth banking; not a halving.
-- **The incremental touch-once grind** (the PERF campaign) is the only thing shown to
- move the *constant factor*: 3.08×→2.64× tcc over ~22 commits (~−14 % Ir). Reaching
- 1.5× that way is **many more multi-week waves** — a long-horizon destination, not a
- single refactor.
-
-**1.5× tcc is a long-horizon incremental goal, not a structural one.** Recommended
-posture: bank the scanner lever (−12 %), continue the touch-once grind, treat "within
-1.5× of tcc" as a multi-campaign destination — unless the modularity constraints are
-relaxed (a tcc-style monolithic hot path), which is explicitly out of scope. **The
-structural hunt (§D) is closed; the path forward is incremental.** *(Reopened
-2026-06-15 — §E: direction to relax the no-fusion constraint, the one lever §D.8
-named as able to move the uniform-density factor.)*
-
----
-
-## E. Direction (2026-06-15): front-half fusion around a shared tagged cell
-
-§D.8 is the premise, not a contradiction: the gap is a **uniform ~2.8×
-per-operation density** factor — kit and tcc spend instructions in nearly the same
-*proportions*, kit is ~2.8× denser **everywhere** — so there is no one phase to
-crush, and the only lever that reduces density *across every phase at once* is the
-one §D.2 named and then declined: **fusion**. It removes the per-token / per-op
-*overhead* — inter-stage call/return per token, the 32 B token copied through
-slots, the parallel value stacks, the public value-stack API layer, the CG
-type-universe re-query, growable-container bounds checks — which is exactly the
-pervasive tax §D.8 measured. The landed lean-token boundary (LEX-PP-API.md) is the
-confirming negative: a lean *representation that still copies through separate
-layers* nets ≈ −1.3 % (drain-measured lex+pp still **2.1× tcc**), precisely as
-§D.6/§D.8 predicted a representation-only change would. **Representation was never
-the lever; the handoff and the per-op layering are.** This section relaxes the
-no-fusion constraint under explicit direction and specifies the fused design.
-
-### E.1 What is relaxed; what stays invariant
-
-**Relaxed.** The front half — **lexer + preprocessor + parser** — fuses around one
-shared, in-place **cell** and one value stack. The parser stops being a module that
-consumes an abstract token stream *by value*; it shares the cell representation with
-the pp output and drives emit off it directly. The standing "leave the modules as
-separable as they are" rule (§D.3) is dropped *for the front half*.
-
-**Invariant — two hard constraints the fusion must preserve:**
-1. **The preprocessor stays independently drainable.** `cpp` / `cc -E` /
- `KIT_PP_DRAIN` must run lexer+pp to a token stream with **no parser attached**.
- Mechanically: the cell's *lexeme* fields are self-contained, the pp reads/writes
- only those, and `pp_next` remains the standalone pull. The parser's *semantic*
- tags live in a region the pp never touches.
-2. **The `CgTarget`→`NativeTarget` backend seam + seven backends stay** (§4.3). The
- fused value stack drives emission *through* the kept seam (its two indirect calls
- per primitive are not the cost — §D.8). We fuse the *front half*, not the backend.
-
-One line: **one record flows bytes→token→value with no copy and no re-derivation;
-the pp can still be drained at the token stage; the backend seam is untouched.**
-
-### E.2 The shared cell
-
-One record serves the whole front half. The lexer/pp fill the **lexeme** view; the
-parser resolves a primary **in place** into the **value** view, at which point the
-cell *is* a value-stack entry (kit's `ApiSValue`, tcc's `SValue`). The two views
-overlap — a resolved value no longer needs spelling/`aux` — so the record stays
-small; `loc` and `flags` survive both. Exact layout is an implementation detail to
-size against the < 64 B value-node discipline; the shape:
+# Frontend shared tagged cell
+
+**Goal.** Define the end-state shape for the C front half around one shared,
+compact cell used by the lexer, preprocessor, and parser input ring. This is a
+design document, not a phasing plan.
+
+The value-stack redesign lives in `doc/plan/CG-STACK-API.md`: the C parser drives
+`KitCg` directly, `KitCg` owns the single liveness-authoritative value stack, C
+type/value facts live in the CG language sidecar, and constant values live in the
+CG constant payload. This document deliberately stops at the front-half cell and
+the handoff from parser primary to CG stack.
+
+In one line: **one compact lexeme cell is shared by lexer, preprocessor, and the
+parser input ring; parser-only semantic tagging happens only on parser-owned
+cells; the preprocessor remains independently drainable; the CG stack remains the
+only live expression stack.**
+
+## Non-Goals
+
+- No parser-owned shadow value stack. `Parser.cg_slot_stack`, `PcgSlot`, and the
+ `pcg_*` adapter are removed by `CG-STACK-API.md`, not replaced here.
+- No `ApiSValue` stored inside the frontend cell. A tagged cell is a short-lived
+ lowering seed, not a durable value.
+- No scalar int-bitmask type lane in lex/pp cells. If scalar classification needs
+ caching after measurement, the cache belongs in the CG language sidecar or a
+ C type cache keyed by `const Type*`, not in every token.
+- No C parser dependency from the lexer or preprocessor headers. Shared cell
+ helpers use `void*` and integer ids; C-specific wrappers live under `lang/c`.
+- No global state. Source registries, macro tables, macro-disabled state, parser
+ rings, and CG side payloads hang off `Pp`, `Parser`, or `KitCg`.
+
+## Ownership Boundaries
+
+1. **Lexer and pp own the lexeme contract.** A lexeme cell is complete enough for
+ macro expansion, directives, `-E` serialization, diagnostics, and replay. It
+ carries lazy `LocRef`/`TextRef`, eagerly interned identifiers, punctuator codes
+ in `aux`, and no C parser state.
+2. **The pp is independently drainable.** `cpp`, `cc -E`, and `KIT_PP_DRAIN`
+ pull cells into caller-owned storage with no parser attached. The pp reads and
+ writes only the lexeme view.
+3. **The parser owns semantic tagging.** The parser may tag only cells in its own
+ input ring or short-lived scratch. It must never tag macro-definition storage,
+ directive buffers, or pp-owned source buffers.
+4. **Macro replay copies before tagging.** Macro bodies are immutable lexeme-cell
+ arrays. When replay emits a token, pp copies one lexeme into the caller's
+ output cell, applies loc/first-token flag overrides, and returns. The parser
+ may then tag that caller-owned copy.
+5. **`KitCg` owns values.** Once a primary is accepted, the parser lowers through
+ `kit_cg_*`, stamps the CG top with `kit_cg_retag_*`, and advances. The cell can
+ carry debug/lowering facts until the next `advance`, but expression lifetime is
+ the CG stack slot.
+
+## Shared Cell
+
+The durable stream/storage unit stays the current lean-token shape: 32 bytes on
+LP64, trivially copyable, and without heap ownership. If the implementation
+renames the type, today's `Tok` should become an alias of this cell rather than a
+second representation.
+
+The shared header must not include C parser or CG internals.
```c
-typedef struct Cell {
- u16 kind; /* token kind in lexeme view; value-node kind once resolved */
- u16 flags; /* lexeme: BOL/space/suffix/encoding · value: lvalue/const/...
- * (disjoint bit ranges — the two views do not co-live) */
- u32 loc; /* byte offset; kept through to emit for diagnostics + DWARF */
+typedef enum CFeKind {
+ /* 0..0x11ff remain TokKind / pp-internal token kinds. */
+ CFE_SEM_FIRST = 0x8000,
+ CFE_SEM_VALUE = CFE_SEM_FIRST, /* parser-only: current token lowered as value */
+ CFE_SEM_PLACE, /* parser-only: current token lowered as place */
+} CFeKind;
+
+typedef struct CFeSem {
+ const void* lang_type; /* C parser: const Type*; opaque to lex/pp */
+ u32 cg_type; /* KitCgTypeId value, represented without cg includes */
+ u32 lang_flags; /* same C value flags passed to kit_cg_retag_* */
+} CFeSem;
+
+typedef struct CFeCell {
+ u16 kind; /* TokKind in lexeme view; CFeKind >= CFE_SEM_FIRST after tagging */
+ u16 flags; /* lexeme: TF_*; semantic: parser-private flags */
+ u32 aux; /* lexeme: Sym/Punct/PP_PARAM; semantic: small discriminator */
+ LocRef loc; /* survives both views; line/col materialize lazily through Pp */
union {
- struct { /* LEXEME — lexer/pp own; the pp drain reads ONLY this */
- u32 aux; /* IDENT: interned Sym · PUNCT/#/##: code · PARAM: idx */
- u32 text; /* spelling ref: (len @loc) for source, or a Sym, tagged */
- } t;
- struct { /* VALUE — parser owns; the pp NEVER reads/writes this */
- u32 type; /* int-bitmask scalar type, or a CG type-id (fallback) */
- u32 r; /* storage class / value-stack location (local/reg/const)*/
- i64 c; /* immediate when constant-known (parse-time fold) */
- } v;
+ TextRef text; /* lexeme spelling ref; lexer/pp/drain view */
+ CFeSem sem; /* parser-only semantic seed; pp never reads this */
} u;
-} Cell;
+} CFeCell;
+
+_Static_assert(sizeof(CFeSem) <= sizeof(TextRef),
+ "semantic overlay must not widen the lexeme cell");
+_Static_assert(sizeof(CFeCell) == 32,
+ "cell must stay the compact token-stream storage unit");
+```
+
+Cell rules:
+
+- `kind < CFE_SEM_FIRST` means lexeme view. All pp helpers assert this when
+ reading a cell.
+- `kind >= CFE_SEM_FIRST` means parser-private semantic view. Such a cell is
+ illegal to feed back into pp, macro storage, directive evaluation, or replay.
+- The lexeme and semantic views do not co-live. The parser must finish all
+ spelling, suffix, and encoding reads before overwriting `u.text`.
+- `loc` survives tagging. Diagnostics and debug locations can materialize full
+ `SrcLoc` lazily through `Pp`.
+- Wide integer constants, float payloads, aggregate initializers, static
+ relocation constants, and expression lifetime never live in the cell. They go
+ to the CG constant payload or the existing initializer/static-data machinery.
+
+## Lexer Contract
+
+The lexer writes one caller-owned cell and returns nothing by value:
+
+```c
+void lex_next(Lexer*, CFeCell* out);
```
-- **`type` is an int-bitmask for the scalar common case** (signedness × width ×
- float/ptr/bool), carried on the cell so the hot path never calls
- `api_unalias_type`/`pred_bits`/`class`/`size` — the §D.2 "classify-once" facts,
- now *inline on the flowing value* rather than a cache beside the heavy `Type`.
- Anything a bitmask can't carry (aggregate, bitfield, VLA, `_Atomic`, wide /
- `__int128`) sets a `FALLBACK` bit and `type` indexes the full CG type universe;
- the slow path is unchanged. This is the §D "int-bitmask type on a lean value slot,
- with fallback" made concrete on the unified cell.
-- **No separate `PcgSlot` + `ApiSValue`.** Today the front half keeps two parallel
- value stacks in lockstep (§0 Pillar 3). The cell *is* the single stack entry;
- Tier 3's "CG-owns-one-stack-with-an-opaque-aux-slot" is subsumed — the aux *is*
- the value view of the cell.
-
-### E.3 Data flow (explicit, end to end)
+Lexeme shape:
+
+- `TOK_IDENT`: `aux = Sym`, `u.text = TEXT_SRC` for exact spelling. Identifiers
+ remain eagerly interned because `Sym` is the macro, keyword, and binding key.
+- `TOK_PUNCT`, `TOK_PP_HASH`, `TOK_PP_PASTE`: `aux = Punct` or character code.
+ Canonical punctuators use `TEXT_NONE`; digraphs may use `TEXT_SRC`.
+- `TOK_NUM`, `TOK_FLT`, `TOK_STR`, `TOK_CHR`, `TOK_HEADER`: `u.text` is a source
+ span or synthetic symbol. Literal suffix and encoding facts stay in `flags`
+ until the parser decodes the literal.
+- `TOK_NEWLINE`: produced only on raw/preprocessor-drain paths that need it for
+ text reconstruction. Parser-feed mode uses `TF_AT_BOL` plus directive state.
+- `loc`: `(file_id, byte_off)` into the pp source registry. The lexer does not
+ stamp line/column per token.
+- `TF_NO_EXPAND` is clear on lexer-produced identifiers. Only the pp sets it
+ when an identifier token becomes unavailable for macro expansion.
+
+Materialization remains pp-owned because only `Pp` retains source buffers, splice
+tables, and `#line` overlays:
+```c
+SrcLoc pp_materialize_loc(Pp*, LocRef loc);
+KitSlice pp_text_slice(Pp*, const CFeCell* cell);
+Sym pp_text_intern(Pp*, const CFeCell* cell);
+int pp_text_eq_cstr(Pp*, const CFeCell* cell, const char* s);
```
- source bytes
- │ scan (cclass / word-at-a-time) ── LEXER ──
- ▼
- Cell.t ← {kind, flags(BOL/space/suffix), loc=off, aux=Sym|punct, text=span}
- │ written into a small ring slot (cur / next / pending); NOT returned by value
- ▼
- ── PREPROCESSOR ── reads/writes only Cell.t
- │ · directive recognition, #if skip, include push (source stack)
- │ · macro expansion: bodies are Cell[] streams, replayed by pointer;
- │ hidesets ride a side-channel keyed by stream position (never on the cell)
- │ · __LINE__/__FILE__/paste/stringize synthesize Cell.t (text = interned Sym)
- ▼
- pp_next(pp, &slot) ── the drain boundary ──
- ├───────────────► cpp / cc -E / KIT_PP_DRAIN : consume Cell.t, serialize/discard.
- │ No parser, no Cell.v ever touched.
- ▼
- ── PARSER ── tags the slot IN PLACE, then it becomes a value-stack entry
- │ primary IDENT: resolve Sym → binding (BindingTab, O(1)) → write Cell.v
- │ {type(bitmask|fallback), r=storage, c=const?}; push.
- │ primary literal: decode text → Cell.v {type, c}; push.
- │ operator: reduce the top Cell.v's on the value stack, driving emit.
- ▼
- ── CG-DRIVE ── the value-stack Cell.v's call the CgTarget seam directly
- │ (no public ApiSValue push/pop API in between; the cell carries r/type/c)
- ▼
- CgTarget → NativeTarget → MCEmitter (KEPT seam; emits bytes)
+
+## Preprocessor Contract
+
+The pp exposes two drainable pulls, both out-pointer only:
+
+```c
+void pp_next_parse(Pp*, CFeCell* out); /* expanded, directives consumed,
+ * non-directive newlines suppressed */
+void pp_next_raw(Pp*, CFeCell* out); /* expanded, directives consumed,
+ * TOK_NEWLINE preserved for -E/cpp */
+void pp_emit_text(Pp*, Writer* out); /* drains pp_next_raw */
```
-The copies that exist today and disappear: (a) `lex_next` sret → `pp` slot → parser
-`fetch_tok` by-value return → `p->cur` (three hops → one in-place write); (b) token
-→ separate `PcgSlot` *and* `ApiSValue` derivation (→ the primary's cell *is* the
-value entry); (c) the per-op `Type`-universe queries (→ bitmask on the cell). The
-remaining unavoidable move is `next → cur` promotion for LL(2) lookahead (one struct
-move), as today.
-
-### E.4 Tagging: token → value, in place
-
-This is the fused step the boundary rewrite set up but did not take. When the parser
-accepts a **primary**:
-
-- **Identifier.** `kind==IDENT`, `u.t.aux` is the interned `Sym`. Resolve once:
- keyword via `kw_map`; else binding via `BindingTab` (already O(1), §0 Pillar 2 —
- the kit-native `TokenSym.sym_identifier`). Overwrite the cell's value view:
- `u.v.type` = the binding's int-bitmask type (or fallback id), `u.v.r` = its storage
- (local frame slot / global sym / enum const → `c`). The cell is now an SValue;
- push it. **No second lookup, no `PcgSlot`+`ApiSValue` pair, no `Type`-universe
- call** for the scalar case.
-- **Literal.** Decode `u.t.text` (via the text helper) into `u.v.c` + `u.v.type`
- from the suffix/encoding `flags`; push. (Decode stays lazy — only literals that
- reach a value are decoded, as now.)
-
-Operators never "tag a token" — they consume value-cells off the stack and call the
-seam. So "the parser tags the token with symbol/type" is precisely: **a primary's
-lexeme cell is resolved in place into the value cell it becomes**, fusing the
-token stream and the value stack into one representation with one resolution.
-
-Diagnostics/DWARF: `loc` rides the cell into `u.v`, so `pcg_set_loc` reads it off the
-live value entry; line/col still materialize lazily through the pp (LEX-PP-API.md).
-
-### E.5 How pp drainability survives the fusion
-
-- The **lexeme view (`Cell.t`) is closed under lex+pp**: every pp operation
- (directives, macro expansion, paste/stringize, `__LINE__`) produces and consumes
- only `Cell.t`. Nothing in the pp reads `Cell.v`.
-- `pp_next(pp, &slot)` is the **single standalone entry**: `cpp`, `cc -E`, and
- `KIT_PP_DRAIN` call it in a loop over a slot they own and never tag. The parser is
- just *another* caller that happens to then tag the slot. Fusing the pp's internal
- pull chain (`lex_next`→`src_next_raw_into`→`pp_pull_into` → one tight `pp_next`,
- tcc's `next_nomacro`+`next` shape) is **internal** to the pp module and does not
- touch this boundary.
-- So "fused" here means **shared representation + in-place tagging + a collapsed
- internal pull**, *not* one mega-function. The pp stays a module with a drain entry;
- the parser stays a module that drives the seam. What's gone is the *by-value
- handoff and the duplicate value representation* between them.
-
-### E.6 Sequenced build (each byte-identity-gated)
-
-Order matters: land the cheap copy/handoff collapses first (they're the Tier 1/2/3
-work below, now repurposed as fusion precursors), then the two genuinely-fused steps.
+Internal pp storage is lexeme-only:
+```c
+typedef struct CFeReplay {
+ const CFeCell* cells;
+ u32 n;
+ Macro* disabled_owner; /* non-NULL while rescanning this macro replacement */
+ LocRef loc_override;
+ u16 first_flags_or;
+} CFeReplay;
```
-E-pre Tier 1B + 1A (macro-arg copies; newline→BOL) — relay slimming, byte-id
- Tier 2 (out-pointer pull into parser ring) — kills the by-value copy
-E-a collapse the pp internal pull chain into one pp_next (next_nomacro shape);
- pp stays drainable. Internal-only; byte-id.
-E-b introduce Cell; lexer/pp fill Cell.t; parser ring holds Cells; pp drain
- reads Cell.t. (Still derives PcgSlot/ApiSValue separately — no value fusion
- yet.) Byte-id; this is the representation swap, isolated.
-E-c value fusion: a primary's Cell.t resolves in place to Cell.v; Cell IS the
- single value-stack entry; delete cg_slot_stack + the ApiSValue duplicate;
- drive the CgTarget seam off Cell.v. Byte-id; the big structural step.
-E-d int-bitmask type on Cell.v for scalars + fallback bit to the CG type
- universe. Byte-id; this is the type-density lever.
+
+Requirements:
+
+- Macro bodies, directive-line buffers, replay buffers, and argument slices store
+ `CFeCell[]` in lexeme view only.
+- A live lexer source should write directly into the caller's output cell where
+ possible. A replay source copies one immutable lexeme into the caller's output
+ cell.
+- Directive processing, `#if` expression evaluation, paste, stringize,
+ `__LINE__`, `__FILE__`, `_Pragma`, and `#include` consume and produce lexeme
+ cells only.
+- Synthesized tokens use `TEXT_SYM` or `TEXT_NONE`; they never point at storage
+ whose lifetime is shorter than `pp_free`.
+- The parser-feed path must not require newline tokens. `TF_AT_BOL` plus
+ directive state is the line-boundary contract; raw `-E` keeps `TOK_NEWLINE` for
+ text reconstruction.
+
+## Macro Expansion Availability
+
+Use the simpler cpplib-style availability model instead of a per-token hideset
+table: a macro is disabled while its own replacement-list frame is being
+rescanned, and an identifier token has one permanent unavailable bit.
+
+The unavailable bit is `TF_NO_EXPAND` in lexeme view. It means "this identifier
+token must not be macro-expanded if it is seen again." One bit is sufficient
+because the token's spelling names at most one macro at the time it is tested;
+the bit does not need to encode a set of macro names.
+
+Suggested pp-owned state:
+
+```c
+typedef struct Macro {
+ Sym name;
+ u16 disabled_depth;
+ /* existing macro definition fields... */
+} Macro;
+
+typedef struct TokSrc {
+ u8 kind;
+ CFeCell* cells;
+ u32 i;
+ u32 n;
+ Macro* disabled_owner; /* non-NULL for macro replacement-list frames */
+ /* existing source/replay fields... */
+} TokSrc;
```
-Every step keeps the emitted object **bit-for-bit identical** (it changes how data
-flows, not what is emitted) — same gate as the boundary rewrite: `make perf-golden`
-→ edit → `make perf-gate` (incl. the splice/`#line`/diagnostic battery), full
-pp/parse suites, and verify on **x64 + rv64** for the value-stack steps (shared NDT
-infra, §5). `KIT_PP_DRAIN` + the tcc `-bench` drain (PERF.md §3) measure lex+pp after
-each step to confirm the gap is actually closing.
-
-### E.7 Prototype-first; payoff; the floor
-
-Per §D.4's discipline, **validate the compounding before building the full lane.**
-The decisive prototype is **E-c on a reduced grammar slice** (scalar locals + arith +
-calls, end to end bytes→emit on the unified cell, no fallback paths) measured
-instr/byte against tcc — because §D.6 proved representation alone is ~6 %, the
-*fusion* (removed handoff + removed dual stack + removed type re-query) is the
-untested variable. If a fused scalar slice does not move materially toward tcc's
-density, fusion is also marginal and ~2.5× is the floor for kit's engineering style
-(the honest §D.4 stop). If it does, build E-a..E-d in order.
-
-Payoff and floor, stated honestly:
-- This attacks the **pervasive** factor §D.8 found, so unlike the −12 % scanner lever
- it is *not* bounded to one phase — it is the credible path toward 1.5× tcc.
-- It is **unproven and expensive** (a front-half rewrite, larger than the boundary
- rewrite), and it **spends the front-half modularity** (the parser fuses with the
- pp output; only pp-drain + the backend seam remain as boundaries).
-- The **kept floor**: the `CgTarget` seam's two indirect calls per primitive and the
- seven-backend generality remain by §4.3 decision — full tcc parity (a single
- monolithic hot path through emission) is still out of scope; the target is *1.5×*,
- not parity.
-
----
-
-## 0. Verified current state (what §4.3 already bought)
-
-### Pillar 2 — "symbol pointers cached on the interned token" — **DONE**
-
-PERF.md §4.1 #1 (the "highest-value front-half brick," describing `scope_lookup`
-as "an N-scope chain walk per identifier") is **stale**. Resolution is already
-one array load:
+Frame rules:
+
+- Pushing a macro replacement-list frame increments
+ `disabled_owner->disabled_depth`. Popping that frame decrements it.
+- The invoking macro is **not** disabled while its arguments are collected or
+ pre-expanded. It becomes disabled only while the substituted replacement list
+ is rescanned.
+- Nested expansions stack naturally: parent macro frames remain on the source
+ stack while child macro frames are scanned, so parent macros remain disabled
+ through nested replacement-list rescans.
+- Object-like no-copy replay, function-like substituted bodies, and paste
+ results all use the same frame rule. Empty replacement lists need no frame.
+
+Identifier test in `pp_next_*`:
```c
-// lang/c/parse/parse.c:388
-SymEntry* scope_lookup(Parser* p, Sym name) {
- return name ? BindingTab_get(&p->bind, name) : NULL; // O(1)
+if (cell.kind == TOK_IDENT && (cell.flags & TF_NO_EXPAND) == 0) {
+ Macro* m = mt_get(pp, tok_ident(&cell));
+ if (m && m->disabled_depth != 0) {
+ cell.flags |= TF_NO_EXPAND;
+ *out = cell;
+ return;
+ }
+ if (m && macro_invocation_is_present(pp, m, &cell)) {
+ push_replacement_frame(pp, m, &cell);
+ continue;
+ }
}
```
-`BindingTab` (`KIT_SYMTAB_DEFINE`, a Sym-indexed dense array;
-`parse_priv.h:360`) is the kit-native form of tcc's `TokenSym.sym_identifier`.
-`SymEntry.shadowed` (`parse.c:363`) is the push/restore stack; `scope_pop`
-(`parse.c:308-321`) unwinds it. `kit/support/symtab.h`'s own header comment
-explains why kit's *per-consumer* table is **better** than tcc's field-on-the-
-token: the pp keeps `Sym→Macro*`, the parser keeps `Sym→SymEntry*`, with no
-shared per-symbol struct and no global state — the module boundary stays intact.
-**Nothing structural remains here.** (Action: correct §4.1 #1 / §4.3 — see Tier 0.)
-
-### Pillar 1 — "shared mutable token slot fed by scanner + replayer" — **~70%**
-
-- One 28-byte `Tok` (`lang/cpp/lex/lex.h:67`), **no per-stage variants**.
-- `src_next_raw_into(Pp*, Tok* out, …)` already writes `lex_next`'s sret straight
- into the caller's slot (`pp.c:72`, see the out-pointer rationale comment at
- `pp.c:61-71`).
-- No-`##` object macros already replay **by pointer** — `push_buf_replay`
- (`pp.c:193`) const-casts the immutable body; the two read sites load by value
- and never write back (`pp.c:187-192`).
-
-Three copy-sources remain (the Tier 1 / Tier 2 work):
-1. **Newlines** (~51 % of lexer outputs) are materialized as `TOK_NEWLINE` then
- drained (`pp_expand.c:1040`). → Tier 1.
-2. **Function-macro args** still cost 3–4 copies/token (raw-collect →
- pre-expand → phase-1 subst → phase-2 paste → read). → Tier 1.
-3. The **parser pulls tokens by value** into `p->cur`/`p->next`/`p->pending`
- (`fetch_tok`/`advance`/`peek1`, `parse.c:165-210`). → Tier 2.
-
-### Pillar 3 — "thin SValue the parser drives by calling emit" — **partial / forked**
-
-`ApiSValue` is already 40 B (`src/cg/internal.h:99`, enforced < 64). The parser
-drives codegen inline (no AST). **But there are two parallel value stacks**, kept
-in lockstep:
-
-- Frontend: `PcgSlot cg_slot_stack[]` (`cg_adapter.h:73`, `parse_priv.h:250`) —
- `{const Type*, PcgLvAux, KitCgTypeId cg_id, u8 flags}`.
-- CG API: `ApiSValue stack[]` inside `KitCg` (`internal.h:189`).
-
-tcc has *one* `SValue[]`. Unifying them is the Tier 3 work, using the user's
-design: **the CG API exposes an opaque per-entry aux slot the frontend tags**, so
-the language-neutral seam is preserved while the parallel stack disappears.
-
-> Note: `doc/FRONTENDS.md` is also stale here — it still describes three parallel
-> arrays `cg_type_stack` / `cg_value_flags` / `cg_lv_aux`; those were merged into
-> the single `PcgSlot` array. Fix in Tier 0.
-
----
-
-## Tier 0 — re-baseline (prerequisite, ~½ day)
-
-Cheap, do first; everything else re-ranks off it.
-
-1. **Re-profile** per PERF.md §3 (`scripts/perf_callgrind.sh run shape-base`) and
- `make perf-golden` for the byte-identity gate snapshot. Record the fresh
- self-`Ir` table; the §2 numbers predate recent landings.
-2. **Correct the stale docs** (no code): PERF.md §4.1 #1 (mark symbol-binding
- cache *landed*, point at `BindingTab`), §4.3 (note Pillar 2 done, Pillar 1
- ~70 %), and `doc/FRONTENDS.md` §"Parser" (the `PcgSlot` single stack, not the
- three-array description).
-
-Gate: docs only + golden snapshot exists. No behavior change.
-
----
-
-## Tier 1 — finish the relay (seam-preserving copy elimination)
-
-Two levers, **byte-identical-gated** (`make perf-golden` → edit → `make perf-gate`,
-plus the line-splice battery for any lexer touch, §5). Order 1B before 1A (1B is
-lower-risk and de-risks the gate harness; 1A is the riskiest lever in the plan).
-
-### 1B — cut function-macro argument copies
-
-**Obstacle.** A function-macro invocation copies each argument token 3–4×:
-raw-collect (`read_invocation_args`, `pp_expand.c:566/586` `tv_push`), per-arg
-`memcpy` into a pre-expansion slice (`preexpand_args`, `pp_expand.c:648`),
-pre-expand output (`expand_arg_to_eof`, `pp_expand.c:475`), then phase-1 subst
-(`pp_expand.c:790…`) and phase-2 paste (`pp_expand.c:866`).
-
-**Change (incremental, each independently gated):**
-- Extend the no-`##` pointer-replay idea (already used for object bodies) to the
- *substituted body*: when `m->has_paste == 0` **and** no argument needs both raw
- and pre-expanded forms, build the result by **pushing argument slices by
- reference** rather than copying each token into one flat buffer. The
- `has_paste` flag is already set at definition time (`pp_expand.c:264`), so the
- branch is free.
-- Where a copy is unavoidable (paste/stringize present), collapse the
- phase-1→phase-2 hand-off so a token is written once, not staged then re-walked
- (today phase 2 re-copies every non-`##` token, `pp_expand.c:870`).
-- The pre-expansion `memcpy` (`pp_expand.c:648`) exists to give
- `expand_arg_to_eof` a stable buffer; check whether it can read the raw range
- in place (the raw buffer outlives the pre-expand for that arg).
-
-**Gate:** byte-identical on the full pp corpus + sqlite `-E`/`-c`. Macro-heavy
-inputs (`#define` batteries, nested/variadic, `##`/`#`) must be in the diff set.
-**Payoff:** bounded by macro density; real on sqlite (declaration/macro-heavy).
-`pp_pull_into` 4.4 % + the arg machinery feed it. **Risk:** medium — argument
-lifetime + hideset (`s->hs`/`hs_uniform`) must ride along with by-reference
-slices; the Prosser rescan correctness is the thing to protect.
-
-### 1A — stop materializing non-directive newlines on the cc path
-
-**Obstacle.** The lexer emits a `TOK_NEWLINE` per physical line
-(`lex.c:704-717`) so the pp can (a) delimit directive lines (`collect_line`,
-`pp_directive.c:36-48`; the `do…while(kind!=TOK_NEWLINE)` scans,
-`pp_directive.c:533`) and (b) recognize `#`-at-BOL. But **`TF_AT_BOL` already
-carries the same signal** on the next real token (`pp.c:124-125`,
-`pp_directive.c:553`). The parser never sees newlines (drained, `pp_expand.c:1040`).
-So the newline token is redundant for cc *except* as a line-end sentinel inside
-directive parsing, and for `-E` (which reconstructs line structure,
-`pp_emit_text`, `pp.c:289-316`).
-
-**Change.** Make `TOK_NEWLINE` an `-E`-only product:
-- Add a lexer mode flag (cc vs `-E`). In cc mode, **do not emit `TOK_NEWLINE`**;
- instead the first token of each line is tagged `TF_AT_BOL` (already happens) and
- the lexer threads "saw a line boundary" into that flag.
-- Rewrite directive line-termination to **"read until the next `TF_AT_BOL`
- token (or `TOK_EOF`)"** instead of "until `TOK_NEWLINE`." The `#if`/`#define`/
- `#include`/`#line`/`#pragma`/`#error`/`#embed` collectors in `pp_directive.c`
- all funnel through `collect_line`; convert that one helper and audit the few
- direct `TOK_NEWLINE` checks (`pp_directive.c:533/1441`, `pp.c:243`,
- `pp_expand.c:424/470/524` — macro-arg newline-as-whitespace, which simply
- disappears in cc mode).
-- Keep `-E` exactly as today (emit newlines): the path already splits on
- `skip_nl` / `pp_next_raw_into`, so this is a mode flag, not a fork of the
- source.
-
-**Gate:** byte-identical on `-c` **and** `-E` (the `-E` path must be untouched);
-full pp/parse corpora; the line-splice battery + a diagnostic whose line number
-falls across a splice (line tracking must survive losing the newline token —
-verify `loc.line` still advances via `lex_catchup_splices`/`l->line`, not via the
-consumer counting newlines). **Payoff:** the largest Tier-1 instruction win —
-removes ~51 % of `lex_next` outputs (16.9 % self-`Ir`) and the cc drain loop.
-**Risk:** **high** — the directive-line contract is subtle (continuation,
-empty directives, `#` alone on a line, `_Pragma`). This is the one lever to land
-last and gate hardest.
-
----
-
-## Tier 2 — single shared mutable token slot at the parser boundary
-
-**End-state.** The pp writes each preprocessed token **directly into the
-parser's current-token slot** (out-pointer all the way through), eliminating the
-by-value `pp_next` return and minimizing the lookahead shuffle — the literal
-"shared mutable slot fed by scanner and replayer," extended to the consumer.
-
-**Obstacles (what the slot must keep doing).**
-- **`pp_next` returns `Tok` by value** (`pp.h:31`); `fetch_tok` stores it into
- `p->cur`/`p->next`/`p->pending` (`parse.c:165-210`). The by-value return is the
- copy to kill — `pp_next_into(pp, Tok* out)` already exists (`pp_expand.c:1199`).
-- **LL(2) lookahead:** `peek1` needs one token ahead (`p->next`/`has_next`); a few
- sites need two. The slot model keeps a tiny ring (`cur` + `next`), not a single
- global like tcc — kit's grammar genuinely needs the second token.
-- **String-literal fusion:** `fetch_tok` collapses adjacent `TOK_STR` runs
- (`parse.c:164-183`), using `p->pending` as a one-token pushback. Must survive.
-- **Replay sources:** `advance` already sources `p->cur` from *either* the replay
- buffer *or* pp (`parse.c:186-198`). Replay backs initializer re-parse
- (`parse_init.c:71-151`) and a `_Generic`/builtin save-restore
- (`parse_expr.c:2785-2805`). The slot model must treat replay as just another
- source feeding the slot (it already is).
-
-**Change.**
-1. Convert `fetch_tok`'s pp pull to the out-pointer form: pull straight into the
- destination slot (`pp_next_into(p->pp, &slot)`), dropping the by-value return
- and its 28-byte sret copy. Keep string fusion by fusing in place into the slot.
-2. Make the lookahead shuffle move *slots*, not re-pull: `advance` promotes
- `next → cur` by struct move (unavoidable for LL(2)); ensure no redundant copy
- beyond that one.
-3. Leave replay as a source that fills the same slot. No new buffering.
-
-This is **not** tcc's single global `tok` — kit's LL(2) + string fusion + replay
-make a one-word slot wrong. The faithful kit form is "one out-pointer relay from
-lexer through pp into a 2-slot parser ring." That is the achievable shape; pursue
-no further collapse here.
-
-**Gate:** byte-identical, full parse corpus + the string-fusion and replay cases
-(`_Generic`, designated/array-string initializers) explicitly in the diff set.
-**Payoff:** removes the per-token by-value copy on the hottest path (the whole
-parse cluster, ~37 % of `-c`); modest but uniform. **Risk:** medium — lookahead
-and pushback are easy to get subtly wrong; the replay save/restore
-(`parse_expr.c:2785`) must round-trip identically.
-
----
-
-## Tier 3 — unify the value stacks via a CG-provided aux hook
-
-**End-state (user's design).** One value stack, owned by the CG API. The CG
-exposes an **opaque per-entry aux slot** that the C frontend tags with its
-ancillary info (`const Type*`, `PcgLvAux`, value flags). The frontend stops
-keeping `cg_slot_stack` in lockstep; structural ops (`dup`/`swap`/`drop`/`rot`)
-carry the aux **for free** because the CG owns the single stack. The seam stays
-intact: the CG never interprets the aux bytes.
-
-**Why it's worth doing.** It deletes an entire bug class (the two stacks
-"drifting") and the lockstep mirroring code in `cg_adapter.c`
-(`pcg_dup`/`pcg_swap`/`pcg_rot3`, `cg_adapter.c:113-142`), and turns
-`pcg_top_type` / `pcg_aux_top` (`cg_adapter.c:174-204`) into a direct read of the
-live entry instead of a parallel-array index. The instruction payoff is the
-removed mirroring; the larger payoff is structural cleanliness (the project's
-stated preference, `[[clean-structural-redesign-pref]]`).
-
-**API additions (new public CG surface, kept opaque):**
+Details that matter:
+
+- A token returned because its macro is disabled keeps `TF_NO_EXPAND` if it is
+ later captured as an argument, substituted into another macro, or replayed.
+ This is what makes `foo(foo)(1)` and `id(foo(foo)(1))` stay `foo(1)`.
+- A function-like macro name that is not followed by `(` is returned as a plain
+ identifier **without** setting `TF_NO_EXPAND`. If it is later rescanned in a
+ context where another macro has supplied the `(`, it may expand. This keeps
+ cases like `id(f L 1))`, with `#define L (`, able to become `1`.
+- `defined` operands in `#if` expansion can reuse `TF_NO_EXPAND`: the source of
+ the mark is different, but the effect is the same.
+- Tokens created by `##` are fresh lexeme cells with `TF_NO_EXPAND` clear. They
+ are then rescanned under the active disabled-frame stack; if a pasted token
+ spells a currently disabled macro, the normal identifier test marks it
+ unavailable before returning it.
+- This removes the canonical hideset table and the per-token `HidesetId` side
+ arrays from replay sources. The only per-token availability state that flows
+ through arguments and replay is the `TF_NO_EXPAND` bit already in the cell.
+
+## Parser Input Ring
+
+The parser consumes pp output through a small fixed ring of parser-owned cells.
+APIs return pointers to parser-owned cells, never cells by value:
+
```c
-// configured once at cg-open: the frontend declares its per-entry aux size.
-KitCg* kit_cg_open(..., uint32_t sv_aux_size, uint32_t sv_aux_align);
-// returns a pointer to the live aux bytes for the entry `depth` from TOS.
-void* kit_cg_sv_aux(KitCg*, uint32_t depth);
+typedef struct ParserInput {
+ CFeCell cells[3]; /* fixed cur + LL(2) lookahead ring; no VLA */
+ u8 cur; /* index of the current token in cells[] */
+ u8 nvalid; /* number of valid slots from cur, 0..3 */
+ CFeCell pending; /* string-literal fusion / one-token pushback */
+ u8 has_pending;
+} ParserInput;
+
+void parser_advance(Parser*);
+void parser_fill_to(Parser*, u32 depth); /* cold path: fills through depth 0..2 */
+
+static inline u8 parser_ring_idx(const ParserInput* in, u32 depth) {
+ return (u8)((in->cur + depth) % 3u); /* depth 0 == current */
+}
+
+static inline CFeCell* parser_cur(ParserInput* in) {
+ return &in->cells[in->cur];
+}
+
+static inline const CFeCell* parser_cur_const(const ParserInput* in) {
+ return &in->cells[in->cur];
+}
+
+static inline const CFeCell* parser_peek(Parser* p, u32 depth) {
+ ParserInput* in = &p->input;
+ if (in->nvalid <= depth) parser_fill_to(p, depth);
+ return &in->cells[parser_ring_idx(in, depth)]; /* depth 1..2 */
+}
```
-The CG widens its stack node to `{ApiSValue core; <aux stride>}` and `memcpy`s
-the aux on every structural op. The frontend reads/writes `*(PcgSlot_payload*)
-kit_cg_sv_aux(g, 0)`.
-
-**Key design decisions to settle during implementation:**
-- **Aux storage: inline stride vs `void*` cookie.** Inline (the node grows by the
- ~40-byte C payload) keeps the stack contiguous and dup/swap a single `memcpy`,
- but widens the hot `ApiSValue` node — measure against the < 64 B discipline
- (the node is currently 40 B specifically to keep `delayed` off-node). A `void*`
- cookie keeps the core node small but adds an indirection and per-push aux
- allocation. **Recommendation:** inline stride, frontend-declared, because it
- collapses two stacks (40 B + ~48 B) into one and removes the second growth/
- allocation entirely; gate the node-size impact on the value-stack microbench.
-- **Suppressed-codegen mode.** The frontend runs sizeof/constant contexts with
- emit disabled (`suppress_codegen`, `parse_priv.h:390`; `pcg_emit_enabled`)
- while still tracking types on `cg_slot_stack`. With a unified stack the CG must
- still maintain the *aux* layer (and a stack-depth counter) when codegen is
- suppressed — i.e. push/pop the entry and its aux without emitting. Define a
- "type-only" push on the CG stack, or have suppression gate emission only, never
- the stack bookkeeping.
-- **lvalue-aux folding without an emit.** `pcg_lv_member`/`pcg_lv_subscript`
- mutate the TOS aux *without* a CG op (`cg_adapter.c`, the `field`/`index`
- folding). Under the unified stack these become in-place writes through
- `kit_cg_sv_aux(g, 0)` — same effect, no parallel array.
-
-**Migration (mechanical, large surface):** rewrite every `pcg_*` helper to read/
-write aux through the hook instead of `cg_slot_stack`; delete `cg_slot_stack` /
-`cg_type_sp` and the mirroring. The pcg API to the parser is unchanged, so
-`parse_expr.c` / `parse_stmt.c` / etc. are untouched — the churn is confined to
-`cg_adapter.c` + the CG stack internals.
-
-**Gate:** **byte-identical** — this is a representational change, not an emitted-
-byte change, so the strict gate applies (it must not perturb output). Full
-suite + sqlite `-E`/`-c` identical to golden. Verify on **x64 and rv64**, not
-just aa64 (the value stack is shared NDT infrastructure; §5). **Risk:** medium-
-high plumbing, low semantic — the emitted code should be identical; the risk is a
-missed mirror site or an aux-not-carried-on-rot bug, which the byte gate catches.
-
----
-
-## Sequencing & recommendation
+Rules:
+
+- `parser_advance` rotates `cur = (cur + 1) % 3` and fills the newly-free tail
+ slot when needed. It does not copy or move live `CFeCell` values between
+ current/lookahead slots.
+- `parser_peek(p, depth)` does the cheap inline validity check
+ (`nvalid <= depth`) and calls `parser_fill_to` only on miss. `parser_fill_to`
+ fills the ring tail with `pp_next_parse(p->pp, slot)`.
+- `parser_peek` returns a `const CFeCell*` valid until the next `advance` or replay
+ source switch. Callers that need persistence copy the cell explicitly.
+- Adjacent string literal fusion writes the fused result into the destination
+ parser cell in place: `TEXT_SYM`, combined encoding flags, and the preserved
+ location of the first literal.
+- Parser replay sources store lexeme cells only. Before a cell is recorded for
+ replay, it must be in lexeme view; tagged semantic cells are not replayable.
+- There are no VLAs. Fixed lookahead stays fixed; any variable replay buffer uses
+ the parser arena or an existing vector helper.
+
+## Parser Tagging
+
+Tagging is local to a consumed primary. It is not a pp feature and it is not an
+expression stack.
+
+The shared-cell helper has no C or CG header dependency:
+
+```c
+static inline void cfe_tag_primary_raw(CFeCell* c, u16 sem_kind,
+ const void* lang_type,
+ u32 cg_type,
+ u32 lang_flags) {
+ c->kind = sem_kind; /* CFE_SEM_VALUE or CFE_SEM_PLACE */
+ c->flags = 0; /* parser-private flags if needed */
+ c->u.sem.lang_type = lang_type;
+ c->u.sem.cg_type = cg_type;
+ c->u.sem.lang_flags = lang_flags;
+}
```
-Tier 0 re-baseline + doc fixes (½ day, no behavior change) ← do first
-Tier 1B macro-arg copy reduction (byte-identical) ← lowest risk
-Tier 1A newline → BOL flag, cc path (byte-identical, hard gate) ← riskiest; land last in T1
-Tier 2 single token slot at parser (byte-identical)
-Tier 3 unify value stacks via aux hook(byte-identical, x64+rv64) ← biggest structural payoff
+
+The C parser may wrap it with C/CG-specific types:
+
+```c
+static inline void c_tag_primary(CFeCell* c, u16 sem_kind,
+ const Type* type,
+ KitCgTypeId cg_type,
+ u32 c_flags) {
+ cfe_tag_primary_raw(c, sem_kind, type, (u32)cg_type, c_flags);
+}
+```
+
+Identifier primary:
+
+1. `TOK_IDENT` carries `Sym` in `aux`.
+2. The parser checks `kw_map`; otherwise it reads `BindingTab` once.
+3. The parser emits the corresponding CG seed: local place, global/function
+ address, enum constant, integer constant, or other binding-specific form.
+4. The parser calls `kit_cg_retag_top(p->cg, type, flags)` with the same semantic
+ facts it wrote into the cell.
+5. The parser advances; the tagged cell is no longer live.
+
+Literal primary:
+
+1. The parser reads spelling, suffix, and encoding from lexeme view.
+2. The parser decodes only when the literal becomes a value.
+3. The parser pushes through the CG constant/value API. Width-complete integer
+ constants use `KitCgConstInt` from `CG-STACK-API.md`.
+4. The parser retags the CG top with C type/value flags.
+5. The parser may tag the current cell as a debug/lowering seed until `advance`.
+
+Operators never tag tokens. They consume CG stack entries via `kit_cg_*`, query
+`KitCgSlotInfo`, and retag results according to `CG-STACK-API.md`. The parser
+input cell can be tagged; the live value is still the CG stack slot.
+
+## CG-Facing Dependencies
+
+This design depends on the CG stack contract from `CG-STACK-API.md`:
+
+```c
+void kit_cg_lang_enable(KitCg*);
+KitCgSlotInfo kit_cg_slot_info(KitCg*, u32 depth_from_top);
+void kit_cg_retag_top(KitCg*, const void* lang_type, u32 lang_flags);
+void kit_cg_retag_at(KitCg*, u32 depth_from_top,
+ const void* lang_type, u32 lang_flags);
+void kit_cg_set_top_flags(KitCg*, u32 set, u32 clear);
+
+int kit_cg_top_const_int_ex(KitCg*, KitCgConstInt* out);
+void kit_cg_push_const_int(KitCg*, KitCgTypeId type,
+ const KitCgConstInt* value);
+```
+
+`CFeSem` deliberately mirrors the non-constant part of `KitCgSlotInfo`:
+`{lang_type, cg_type, lang_flags}`. That keeps parser tagging and CG retagging
+the same shape without making the pp pay for parser-only facts.
+
+## End-State Data Flow
+
+```text
+source bytes
+ -> lex_next(Lexer*, CFeCell*) writes lexeme view
+ -> pp source/replay stack reads/writes lexeme view only
+ -> pp_next_parse/raw(Pp*, CFeCell*) fills caller-owned cell
+ -> cpp / cc -E / KIT_PP_DRAIN drains lexeme cells; no parser
+ -> ParserInput ring parser-owned cells
+ -> primary resolution optional in-place semantic tag
+ -> kit_cg_* push/op + retag CG stack owns values
+ -> CgTarget -> NativeTarget -> MC backend seam unchanged
```
-Each tier is independently landable and revert-safe; re-profile and re-rank after
-each (§5 — the highest lever moves as work is removed). The honest expectation,
-consistent with PERF.md §2 (the frontend gap is ~uniform 3× with no single hot
-function): **no tier is a silver bullet.** Tier 1A is the largest single
-instruction win; Tier 3 is the largest *structural* win (deletes the dual-stack
-bug class) for a modest instruction gain. The ~2× headroom a full tcc-shape
-collapse *including the vtable* might reach stays out of scope by §4.3's standing
-decision — the seven-backend modularity is worth its two indirect calls.
+The fusion point is the front-half representation and handoff: eliminate
+by-value token returns and duplicate token shapes, preserve pp drainability, and
+do not create a second value representation beside `KitCg`.
+
+## Correctness Checklist
+
+- A pp drain can run with no parser and never observes `CFE_SEM_*`.
+- Macro bodies and replay buffers are immutable lexeme-cell streams.
+- Parser tagging is limited to caller-owned parser cells.
+- Tagged cells are never recorded for replay or passed back to pp helpers.
+- Text and location materialization work after source pop and macro replay.
+- String-literal fusion produces a single lexeme cell without a second token type.
+- Parser current/lookahead is a three-cell ring with index rotation; `advance`
+ does not copy or move live `CFeCell` values.
+- `TF_AT_BOL` is sufficient for parser-feed directive recognition; raw mode keeps
+ `TOK_NEWLINE` for `-E`.
+- Macro replacement frames disable their owner only while the replacement list is
+ being rescanned, and re-enable it exactly when that frame is popped.
+- A token skipped because its macro is disabled is returned with `TF_NO_EXPAND`,
+ and that bit survives argument pre-expansion, substitution, replay, and later
+ rescans.
+- A function-like macro name not followed by `(` is not marked `TF_NO_EXPAND`;
+ it can expand if a later rescan sees a `(` supplied by another macro.
+- Pasted identifier tokens start with `TF_NO_EXPAND` clear and are judged by the
+ active disabled-frame stack during normal rescan.
+- Identifier resolution does one `Sym`-keyed lookup after keyword classification.
+- Literal decoding is lazy and occurs before `u.text` is overwritten.
+- The CG stack, not `CFeCell`, owns expression lifetime, constants, liveness, and
+ C type/value flags after primary lowering.
+- No lexer/pp header includes C parser or CG internals.
+- No VLAs and no global state.
