commit 085571a2844d40bee4bcfe3a41944c0bb0d2c87a
parent 5d4228cb2de9464069e87e01ca86b62df9ca2b0a
Author: Ryan Sepassi <rsepassi@gmail.com>
Date: Sun, 14 Jun 2026 13:17:31 -0700
doc: add frontend API roadmaps
Document the planned CG stack-sidecar API and lean lexer/preprocessor token boundary.
Diffstat:
3 files changed, 1068 insertions(+), 0 deletions(-)
diff --git a/doc/plan/CG-STACK-API.md b/doc/plan/CG-STACK-API.md
@@ -0,0 +1,495 @@
+# CG stack API: parser-driven codegen without a shadow stack
+
+**Goal.** Make the C parser drive `KitCg` directly, in the same single-pass
+shape it has today, without maintaining a duplicate `pcg` value stack. The
+codegen stack stays the liveness authority: a value/place is live exactly while
+it is represented by a stack slot, and popping that slot is the cheap
+single-pass death signal the native direct target and temporary reclaim already
+use.
+
+This is not a plan to replace the stack API with free-floating value handles. It
+is a plan to (a) make stack slots rich enough for a language frontend to use them
+as its expression stack, (b) move all effective-address folding behind the CG
+API, (c) make codegen suppression a CG mode that also folds constants, so the
+parser has one uniform path and one constant evaluator, and (d) **delete the
+`pcg` adapter layer** so the parser drives the CG API directly.
+
+This is a **clean-break rewrite, no backcompat**. The spec below describes the
+end state; §7 is a direct cutover, not an incremental dual-stack migration.
+
+---
+
+## 1. Decisions (resolved)
+
+| # | Decision | Choice |
+|---|---|---|
+| D1 | Keep the stack API, add a stack-owned frontend slot sidecar | yes |
+| D2 | Free `KitCgValue`/`KitCgPlace` handles independent of stack lifetime | **rejected** — loses the "on stack == live" invariant |
+| D3 | Where the type/flags stack lives during suppression | **CG unevaluated mode** — KitCg maintains a valueless type+sidecar stack; the parser drives CG uniformly with no `if (emit)` forks |
+| D4 | Effective-address folding | **Automatic behind the CG API.** The parser builds places naively (`deref`, `field_at`, `elem`); CG decides whether to fold into the fused `[base + index*scale + offset]` operand or materialize. **No deferral or folding logic in the parser.** |
+| D5 | Success gate | **Correctness + determinism** (full suites pass, output deterministic). Emitted bytes MAY change vs today; byte-identity is *not* a gate. |
+| D6 | Fused `store_void` / `store_keep` as CG ops | **in scope** |
+| D7 | The `cg_adapter.c` / `pcg_*` layer | **deleted entirely.** The parser drives `kit_cg_*` directly and applies C facts via `kit_cg_retag_top`/`set_top_flags`. C-semantic operations survive as plain helper *functions* (op-enum maps, the conversion lattice, atomic/inline-asm/inc-dec shaping) — not a stack-mirroring layer, and not under a `pcg_` prefix. |
+| D8 | Call-arg stack-mark API (old §4.4) | **dropped** — C already passes args on the CG stack (`kit_cg_call`), so a mark API adds nothing |
+| D9 | `KitCgSlotInfo` + new ops are public API | **forced** — `lang/` builds only against `include/kit/cg.h` (no `src/cg` includes) |
+| D10 | Sidecar is a parallel array, not inlined into `ApiSValue` | **forced** — `ApiSValue` is `_Static_assert`'d `<= 64` bytes and is near the limit (`src/cg/value.c:14`) |
+| D11 | C integer-constant-expression evaluation | **moves into CG.** Unevaluated mode also **folds constants** (not just types); the parser evaluates an ICE by parsing it once through the normal path under unevaluated mode and reading the folded value. The parser's duplicate `cexpr_*` grammar and `cint_*` arithmetic engine are **deleted**. |
+
+---
+
+## 2. Current state (what we are removing)
+
+The C frontend currently runs two stacks in lockstep:
+
+- **The real CG stack:** `KitCg.stack` of `ApiSValue` (`src/cg/internal.h:99`).
+ `api_push`/`api_pop` (`src/cg/value.c:161/187`) maintain `-O0` transient
+ reference counts that feed cheap single-pass liveness and temp reclaim. The
+ node already carries `type` (CG type id), an `lvalue` byte (PLACE vs VALUE),
+ and an `ApiBitField` (bit-field geometry of a PLACE).
+- **The parser shadow stack:** `Parser.cg_slot_stack` of `PcgSlot`
+ (`lang/c/parse/parse_priv.h:250`, `cg_adapter.h:73`). Each slot carries
+ `const Type*`, a cached `cg_id`, C value flags, and a `PcgLvAux`
+ effective-address record (`cg_adapter.h:50`).
+
+`PcgSlot` duplicates four things; only one is genuinely C-only:
+
+| `PcgSlot` field | Status |
+|---|---|
+| `const Type*` (C type) | **C-only**; opaque to CG; must live somewhere |
+| `cg_id` (cached CG id) | **redundant** — the CG node already has `type` |
+| flags: LVALUE / MODIFIABLE / BITFIELD / NULL_PTR_CONST / REGISTER | LVALUE≈node `lvalue`, BITFIELD≈node bit-field; MODIFIABLE/NULL_PTR_CONST/REGISTER are **C-only** |
+| `PcgLvAux` offset/scale/base_kind/bit_* | **redundant** with the CG place operand (`OPK_INDIRECT` base+index*scale+ofs) + node bit-field — kept only to *defer* the deref and avoid re-crossing C-layout |
+
+CG already has a complete PLACE/VALUE addressing model (`include/kit/cg.h:668`):
+`push_local` / `deref(offset)` / `field(index)` / `elem` / `elem_scaled` /
+`addr` / `load` / `store`, with offset/index/scale folded into the `OPK_INDIRECT`
+operand (`kit_cg_field`, `src/cg/control.c:1157`, folds the field offset into the
+displacement and preserves any index/scale). **toy and wasm use this model
+directly.** The C frontend bypasses `kit_cg_field`/`kit_cg_elem` and reimplements
+the same folding in `PcgLvAux` (`pcg_materialize_lv_to_ptr`,
+`pcg_lv_to_memop_place`, `cg_adapter.c:585/647`) purely to (1) defer the deref so
+`*p`/`p->f` stay a pointer rvalue, and (2) avoid re-deriving an offset it already
+knows. D4 deletes this: the folding moves behind the CG API.
+
+The sqlite layer profile makes the cost visible: ~1.43 M `kit_cg_*` calls per
+`sqlite3.c`, CG API ~19% of compile ticks, of which the stack/memory family is
+~49%. The hot expression seam is paid twice — once per CG stack edit, once per
+shadow-stack edit.
+
+---
+
+## 3. Invariants (load-bearing)
+
+1. **Stack lifetime stays authoritative.** A transient operand is live iff a live
+ `ApiSValue` stack entry references it. `api_push`/`api_pop` remain the lifetime
+ hooks. The sidecar never owns lifetime.
+2. **No free expression handles.** Every API returns either a
+ local/symbol/label handle with existing lifetime rules, or a short-lived
+ stack depth — never a separately-live expression value.
+3. **The frontend can inspect and retag slots.** Read top/top2/depth-N lang type
+ + flags; retag the result slot after an op.
+4. **CG does not depend on C frontend types.** `src/cg` must not include `lang/c`
+ headers or dereference the lang payload. The lang payload is an opaque
+ `const void*` copied/dropped with the slot.
+5. **Backend boundaries stay intact.** `CgTarget` / NDT / `NativeTarget` / MC
+ receive no C frontend state. The new place ops lower through the *existing*
+ CgTarget contract — **no backend changes**.
+6. **The hot `ApiSValue` node stays `<= 64` bytes.** The sidecar is a parallel
+ array (D10).
+7. **No global state.** The sidecar and the unevaluated counter hang off `KitCg`.
+8. **The sidecar is opt-in.** It is allocated only for a frontend that enables
+ it; toy/wasm leave it off and pay nothing.
+
+---
+
+## 4. The new architecture
+
+### 4.1 One stack + an opaque lang sidecar
+
+A parallel array `KitCg.lang_side`, the same length as `KitCg.stack`, allocated
+lazily when the frontend calls `kit_cg_lang_enable`. Entry *i* shadows
+`stack[i]`. CG moves sidecar entries in exact lockstep with the value stack
+(§6). NULL `lang_side` means "no sidecar"; all sidecar ops degrade to no-ops /
+zeroed reads.
+
+```c
+typedef struct KitCgSlotInfo {
+ KitCgTypeId cg_type; /* read-only echo of the node's CG type, for queries */
+ const void* lang_type; /* opaque to CG; lang/c stores a const Type* */
+ uint32_t lang_flags; /* frontend-defined (C: LVALUE/MODIFIABLE/...) */
+} KitCgSlotInfo;
+```
+
+The sidecar stores only `{lang_type, lang_flags}`; `cg_type` in the returned
+struct is filled from `stack[i].type` at query time. The C flags are
+frontend-private bits; suggested C layout:
+
+```
+C_LVALUE /* a C lvalue (distinct from CG place-ness; e.g. a const lvalue) */
+C_MODIFIABLE /* modifiable lvalue (not const/array/func/void) */
+C_BITFIELD /* C-semantic bit-field marker (geometry lives on the CG place) */
+C_NULL_PTR_CONST/* integer 0 / null-pointer-constant */
+C_REGISTER /* `register` storage class — forbids `&` */
+```
+
+`PcgLvAux.is_subobject` disappears: with eager places (4.3) the place operand
+*is* the exact sub-object, so the to_rvalue struct-materialize heuristic
+(`parse_expr.c:1040`) collapses to "leave the place as-is."
+
+### 4.2 Slot queries and retag
+
+```c
+void kit_cg_lang_enable(KitCg*); /* alloc + track sidecar */
+KitCgSlotInfo kit_cg_slot_info(KitCg*, uint32_t depth_from_top); /* depth 0 == top */
+void kit_cg_retag_top(KitCg*, const void* lang_type, uint32_t lang_flags);
+void kit_cg_retag_at(KitCg*, uint32_t depth_from_top,
+ const void* lang_type, uint32_t lang_flags);
+void kit_cg_set_top_flags(KitCg*, uint32_t set, uint32_t clear);
+```
+
+`kit_cg_stack_depth` already exists (`include/kit/cg.h:706`). Retag touches only
+the sidecar — never the node's CG type (the producing op sets that). C-type
+changes that carry no CG op (qualifier strip on a struct lvalue; the deref
+retype) become a retag. Today's C queries — `pcg_top_type`, `pcg_top2_type`,
+`pcg_top_is_lvalue`/`_modifiable_lvalue`/`_null_ptr_const`/`_bitfield`/
+`_register`, and the `pcg_retag_*` family — are replaced by direct
+`kit_cg_slot_info` reads and `kit_cg_retag_*` calls at the parser sites (small
+static inline accessors over the flag bits are fine; there is no `pcg` layer).
+
+**Producer rule:** an op that produces a *fresh* result slot (load, call result,
+binop/cmp result, convert, addr, push_*) clears that slot's sidecar
+(`lang_type=NULL, lang_flags=0`). The parser stamps the C facts immediately after
+via `kit_cg_retag_top`. Structural ops (`dup`/`dup2`/`swap`/`rot3`/`drop`)
+move/copy sidecar entries so `lang_side[i]` always tracks `stack[i]`. This is
+exactly today's "emit op, then `pcg_retag_top`" idiom, with the structural half
+done automatically inside CG.
+
+### 4.3 Eager places; folding behind the CG API (D4)
+
+The parser builds places naively and immediately; CG folds. One new op is
+needed; the rest already exist.
+
+```c
+/* PLACE -> PLACE: project to a sub-object at a known byte offset and field type.
+ * Folds the offset into the place operand (no record-layout lookup — the
+ * frontend already knows the offset from its own C layout). Errors if TOS is not
+ * a PLACE. Bit-field geometry, when present, is attached separately via
+ * kit_cg_field_bits. */
+void kit_cg_field_at(KitCg*, int64_t byte_offset, KitCgTypeId field_type);
+```
+
+Composition (all folding decided inside CG, as `kit_cg_field` already does):
+
+| C expression | CG op sequence | result |
+|---|---|---|
+| `x` (local) | `push_local x` | PLACE |
+| `*p` | `deref 0` | PLACE |
+| `s.f` | … `field_at(off_f, ty_f)` | PLACE |
+| `p->f` | `deref 0; field_at(off_f, ty_f)` | PLACE |
+| `a[i]` | decay `a`→`elem*` (`addr; bitcast`); push `i`; `elem 0` | PLACE |
+| `a[i].f` | …`elem 0; field_at(off_f, ty_f)` | PLACE |
+| `&e` | build place; `addr` | VALUE(ptr) |
+| read of any lvalue `e` | build place; `load` | VALUE |
+
+`deref(0); field_at(off)` collapses inside CG to one
+`OPK_INDIRECT(base=p, ofs=off)` memop; `elem; field_at` keeps the fused
+`[base + i*scale + off]` form (the `kit_cg_field` INDIRECT branch already does
+this). The "stride differs from access type" case (`arr[i].f`) is handled by the
+natural `elem` (scales by `sizeof(elem)`) then `field_at` (adds `off_f`)
+composition — so the C frontend no longer needs `kit_cg_elem_scaled` for it.
+
+Folding intelligence that currently lives in the parser moves into the CG place
+ops, where it belongs:
+
+- constant offset → `OPK_INDIRECT` displacement (no extra add);
+- dynamic index → fused `[base + i*scale + off]`;
+- out-of-int32 displacement → materialize an explicit `base+ofs` pointer;
+- the widening-signed-load hint (`KIT_CG_MEM_SEXT_LOAD`, today set in
+ `pcg_load`, `cg_adapter.c:540`) is decided inside `kit_cg_load` from the
+ access type;
+- bit-field geometry rides the place via `kit_cg_field_bits`
+ (`src/cg/control.c:1253`), set by the frontend right after `field_at`.
+
+`pcg_materialize_lv_to_ptr`, `pcg_lv_to_memop_place`, `pcg_lv_member`,
+`pcg_lv_subscript`, `pcg_decay_array`, and the whole `PcgLvAux` struct are
+**deleted**. `kit_cg_field` (by index) and `kit_cg_elem`/`kit_cg_elem_scaled`
+stay for toy/wasm.
+
+### 4.4 Fused load/store (D6)
+
+```c
+void kit_cg_load(KitCg*, KitCgMemAccess); /* [place] -> [value] (exists) */
+void kit_cg_store(KitCg*, KitCgMemAccess); /* [place, value] -> [] (exists) */
+void kit_cg_store_keep(KitCg*, KitCgMemAccess); /* [place, value] -> [value] */
+```
+
+`kit_cg_store` already is the `[place,value] -> []` primitive; the parser names
+the discard intent through it (`store_void` == `store`, kept as the documented
+spelling). `kit_cg_store_keep` adds the C assignment-expression value
+preservation (`a = b`, `if ((x=f()))`) as one op, so the parser drops the
+`dup`/`rot3`/`swap` choreography it does today (`cg_adapter.c:769-773`). CG keeps
+the delayed-RHS-into-local fast path (`kit_cg_store`, `src/cg/memory.c:506-551`)
+in both forms.
+
+### 4.5 Unevaluated mode: fold + types, zero emission (D3, D11)
+
+Unevaluated mode serves two needs that share one machine: (1) compute C result
+types/flags while emitting nothing (`sizeof` operand, `_Generic` controlling
+expr, statically-dead `&&`/`||`/`?:` arms, `extern inline` bodies, constant
+initializer paths), and (2) **fold a constant expression** so the parser can read
+its value (case labels, array sizes incl. file scope, enum values, bit-field
+widths, `alignas`, `_Static_assert`; see §4.7). It is the single mechanism that
+replaces both `suppress_codegen` and the parser's `cexpr_*`/`cint_*` evaluator.
+
+```c
+void kit_cg_unevaluated_push(KitCg*); /* enter fold+type-only mode (nestable) */
+void kit_cg_unevaluated_pop(KitCg*);
+```
+
+Contract while `unevaluated > 0`:
+
+- **Maintained:** stack depth, the sidecar (lang_type/lang_flags), and
+ **constant folding** of immediate operands — `push_int`/`binop`/`unop`/`cmp`
+ fold through the existing C-correct core (`api_try_fold_int_binop/unop/cmp`,
+ `api_fold_result`, `src/cg/fold.c`), so `kit_cg_top_const_int` answers
+ correctly. Structural ops move sidecar entries normally.
+- **Skipped:** every `CgTarget`/emission call, temp/local allocation, const-local
+ store/load tracking, `-O0` ref accounting, and the aggregate-place validation
+ in `api_push`. A producing op that *cannot* fold to an immediate pushes a
+ value-less node (operand = none/poison) — its `kit_cg_top_const_int` is simply
+ false, which is the "not a constant expression" signal (§4.7).
+- **No forced CG-type lowering for the type-only need.** The parser reads C
+ facts from the sidecar, not the node `type`; push ops tolerate
+ `KIT_CG_TYPE_NONE`. (Constant folding *does* use the supplied CG type for
+ width/signedness, so an ICE parse passes real types — cheap, cached.)
+- **Function-independent.** Folding allocates nothing and touches no function
+ state, so the mode is valid at file scope (no open function). Handle-returning
+ control ops (`label_new`, `scope_begin`, `scope_*_label`) return benign
+ dummies; `label_place`/`jump`/`branch_*`/`scope_end` tolerate them. The
+ `&&`/`||`/`?:` parsers fold their all-constant cases via `kit_cg_top_const_int`
+ (which now works here) and their branch/temp emission no-ops away.
+
+This generalizes suppression from "skip the value op" to "skip *all* of CG,"
+letting the parser **delete its `if (emit)` forks** entirely (e.g. the dual-path
+loop in `parse_stmt.c:85-118` collapses to one path). `suppress_codegen` moves
+out of `Parser` into `KitCg`. Parser-side semantic side-effects in suppressed
+regions (VLA-size slot recording for `sizeof`, `_Generic` association
+collection, diagnostics) still run, because the parser code path is unchanged.
+
+### 4.6 Calls (D8)
+
+C call arguments already flow through the CG stack: the C call path drives
+`kit_cg_call(nargs, fn_type, attrs)`, which consumes the callee+args segment.
+There is no second frontend arg list to remove. **No mark API is added.** The
+parser stamps the result slot's C type via retag.
+
+### 4.7 Constant-expression evaluation on CG (D11)
+
+C integer-constant-expression evaluation stops being a parser subsystem. Today
+`eval_const_int` → `cexpr_cond` (`parse_expr.c`) is a *complete second expression
+grammar* (`cexpr_mul`/`add`/`shift`/`rel`/`eq`/`band`/`bxor`/`bor`/`land`/…) plus
+a *second constant-arithmetic engine* (`cint_*`, with 128-bit lo/hi, casts,
+conversions) — and it re-implements `offsetof` / `__builtin_constant_p` /
+enum-constant handling separately from the real parser, a standing drift hazard.
+All of it is **deleted**.
+
+The replacement is one helper over the normal path:
+
+```
+i64 / CConstInt eval_const_int(p, loc):
+ kit_cg_unevaluated_push(cg);
+ parse_conditional(p); /* the REAL grammar, folding as it goes */
+ ok = kit_cg_top_const_int(cg, &v); /* (or the i128 variant below) */
+ info = kit_cg_slot_info(cg, 0); /* C result type from the sidecar */
+ kit_cg_drop(cg);
+ kit_cg_unevaluated_pop(cg);
+ if (!ok) perr("constant expression required");
+ return { v, info.lang_type };
+```
+
+Consequences:
+
+- One grammar, one fold engine. The constant value comes from the same
+ C-correct fold core used at runtime (`src/cg/fold.c`); the C *type* of the
+ result comes from the sidecar (the normal path ran usual-arithmetic
+ conversions). `sizeof`, `_Alignof`, `offsetof`, `_Generic`, and enum constants
+ are handled once, by the real parser — a correctness win.
+- **New API:** a width-complete constant query for i128 ICE, since
+ `kit_cg_top_const_int` is 64-bit:
+ ```c
+ int kit_cg_top_const_i128(KitCg*, uint64_t* lo, uint64_t* hi); /* or a bytes form */
+ ```
+- **Diagnostics tradeoff:** failure to fold yields a generic "constant
+ expression required" rather than `cexpr_*`'s targeted messages, and ICE
+ constraints are enforced semantically (non-foldable → reject) rather than
+ syntactically (e.g. the comma operator is foldable yet is not an ICE). A light
+ syntactic guard can be added where a precise diagnostic is required.
+- Address/static-initializer constants (`&x + 4`, string-literal addresses,
+ designated-init relocations) are a *separate* constant category handled in
+ `parse_init.c`; they are **out of scope** here (a later unification could fold
+ them onto a CG constant-data path, but not in this plan).
+
+---
+
+## 5. What moves where
+
+| Today in `pcg` / `Parser` | Target owner |
+|---|---|
+| `PcgSlot.type` (`const Type*`) | CG sidecar `lang_type` (opaque) |
+| `PcgSlot.cg_id` | gone — read the node `type` via `kit_cg_slot_info` |
+| `PcgSlot.flags` (C value flags) | CG sidecar `lang_flags` |
+| `Parser.cg_slot_stack` / `cg_type_sp` / `cg_type_cap` | gone — the CG stack is the typed stack |
+| `PcgLvAux` (offset/scale/base_kind/bit_*) + `pcg_materialize_lv_to_ptr` / `pcg_lv_to_memop_place` / `pcg_lv_member` / `pcg_lv_subscript` / `pcg_decay_array` | gone — CG place ops (`deref`/`field_at`/`elem`/`addr`/`field_bits`) fold automatically |
+| `Parser.suppress_codegen` + `pcg_emit_enabled` forks | CG unevaluated mode |
+| `pcg_dup`/`swap`/`drop` (mirror onto shadow stack) | direct `kit_cg_dup`/`swap`/`drop` (sidecar moves automatically) |
+| assignment `dup`/`rot3`/`swap` sequences | `kit_cg_store` / `kit_cg_store_keep` |
+| `cg_adapter.c` / `cg_adapter.h` / the `pcg_*` layer | **deleted** — parser drives `kit_cg_*` directly |
+| `cexpr_*` grammar + `cint_*` constant-arith engine (`parse_expr.c`) | **deleted** — ICE folds on CG under unevaluated mode (§4.7) |
+| C conversions, usual-arithmetic-conversions, binop/cmp/atomic op maps, lvalue-legality, null-ptr/register/bitfield C rules | plain helper *functions* the parser calls (no stack mechanics, no `pcg_` prefix) |
+
+After the cutover there is **no adapter layer**. The parser calls `kit_cg_*`
+directly and stamps C facts with `kit_cg_retag_top`/`set_top_flags`. The
+genuinely C-semantic operations survive as ordinary functions in the parse module
+— the op-enum maps, the conversion lattice (today's `pcg_convert`), inc/dec,
+call/atomic/intrinsic/inline-asm shaping — each driving `kit_cg_*` + retag
+directly, with no parallel stack and no EA state.
+
+---
+
+## 6. Op-by-op: the sidecar lockstep set
+
+Every stack-mutating CG op maintains `lang_side` so depths stay in sync. Grouped
+by effect:
+
+- **Producers (push a fresh slot, clear its sidecar):** `push_int`, `push_float`,
+ `push_null`, `push_local`, `push_local_addr`, `push_symbol_addr`,
+ `push_label_addr`, `load`, `addr`, `deref`, `field`/`field_at`/`elem`/
+ `elem_scaled` (consume then push a place — clear), `alloca`, `vararg_next`,
+ call result, `atomic_load`/`atomic_rmw`/`atomic_cmpxchg` result,
+ `intrinsic` result, overflow-builtin result, `inline_asm` outputs.
+- **Retypers (1→1, keep depth):** `trunc`/`sext`/`zext`/`bitcast`/`fpext`/
+ `fptrunc`/`int<->float`/`ptr<->int`, `int_unop`/`fp_unop`. Result sidecar
+ cleared; parser retags.
+- **Combiners (N→1):** `int_binop`/`fp_binop`/`int_cmp`/`fp_cmp` (2→1),
+ `field`/`elem` (consume base/index), `store_keep` (2→1), `call`/`call_symbol`
+ (N→0/1), `va_copy` (2→0), `atomic_store` (2→0), `atomic_cmpxchg` (3→…).
+- **Pure structural:** `dup` (copy top sidecar), `dup2`, `swap`, `rot3`, `drop`,
+ `store`/`store_void` (2→0). These move sidecar entries with no semantic change.
+- **Consumers (→0):** `branch_true`/`branch_false`/`switch`/`computed_goto`,
+ `ret`.
+- **Scope edges:** `api_scope_store_results`/`api_scope_push_results` move the
+ sidecar with carried results when `lang_side` is active. (C uses only void
+ scopes, so this is inert for C, but must be correct for any future
+ result-carrying frontend that enables the sidecar.)
+
+Implementation choke points: `api_push`/`api_pop` (`src/cg/value.c:161/187`) and
+the structural ops `kit_cg_dup`/`dup2`/`swap`/`drop`/`rot3`
+(`src/cg/memory.c:598-735`). Routing the sidecar move through `api_push`/`api_pop`
+(which every producer/consumer already calls) covers most ops automatically; the
+structural ops and scope-edge movers need explicit sidecar handling.
+
+---
+
+## 7. Cutover plan (clean break, gate = correctness + determinism)
+
+No dual-stack interim, no byte-identity gate. Land in dependency order:
+
+1. **CG sidecar + queries.** Add `KitCgSlotInfo`, `lang_side`,
+ `kit_cg_lang_enable`, `kit_cg_slot_info`, `kit_cg_retag_top`/`_at`,
+ `kit_cg_set_top_flags`; wire lockstep through `api_push`/`api_pop` + the
+ structural ops + scope-edge movers. No frontend change yet.
+2. **CG unevaluated mode.** Add `kit_cg_unevaluated_push`/`pop` with the
+ fold-constants + maintain-stack + skip-emission/skip-alloc contract (4.5),
+ valid at file scope.
+3. **CG place + store + const ops.** Add `kit_cg_field_at`, `kit_cg_store_keep`,
+ and the i128 constant query (`kit_cg_top_const_i128`); confirm
+ `deref`/`elem`/`field_bits` compose to the fused operand without parser help.
+4. **Rewrite the C frontend onto the new API.** Delete `cg_slot_stack` /
+ `cg_type_sp` / `cg_type_cap` / `PcgSlot` / `PcgLvAux`; drive `kit_cg_*`
+ directly + retag; replace EA folding with eager places; replace assignment
+ choreography with `store`/`store_keep`; call `kit_cg_lang_enable` at
+ function/unit setup; move `suppress_codegen` to the CG unevaluated counter and
+ drop the parser `if (emit)` forks.
+5. **Delete the adapter.** Remove `cg_adapter.c`/`.h` and the `pcg_` prefix;
+ relocate the C-semantic helpers (op maps, conversion lattice, inc/dec,
+ atomic/intrinsic/inline-asm shaping) as ordinary parse-module functions
+ (D7). Update `doc/FRONTENDS.md` to describe the durable parser → CG contract.
+6. **Move ICE onto CG (last).** Reimplement `eval_const_int` over the normal
+ parse path under unevaluated mode (§4.7); delete the `cexpr_*` grammar and the
+ `cint_*` engine. Sequenced last so the stack rewrite is validated before the
+ constant subsystem is removed. Re-run every ICE context: case labels, array
+ sizes (block + file scope), enum values, bit-field widths, `alignas`,
+ `_Static_assert`, designated-init indices, and i128 constant expressions.
+
+Gate at each step:
+
+- `make test-cg-api test-parse test-pp test-toy` (frontend + CG API),
+- `make test-cross` / targeted smoke for codegen correctness across arches,
+- determinism: same input → identical output across repeated runs,
+- toy/wasm unaffected (sidecar off): `test-toy`, wasm front suite.
+
+Targeted codegen tests for the place rewrite: local/global load+store; `s.f`,
+`p->f`, `a[i]`, `a[i].f`; bit-field load/store; assignment-expression value
+preservation (`a=b=c`, `if((x=f()))`); `++`/`--` and compound assignment;
+`&` of each lvalue form; struct lvalue / sub-object rvalue; volatile and atomic
+accesses; `sizeof`/`_Generic`/dead-arm suppression returning the right types.
+
+---
+
+## 8. Requirements checklist (the agent verifies against this)
+
+- R1 Single liveness-authoritative value stack; `api_push`/`api_pop` unchanged as
+ lifetime hooks. (§3.1)
+- R2 No free expression handles; only handles + stack depth escape. (§3.2)
+- R3 Read/retag lang type+flags at top/top2/depth-N. (§4.2)
+- R4 CG never dereferences the lang payload; it is `const void*`. (§3.4)
+- R5 No backend (`CgTarget`/NDT/NT/MC) change; place ops lower through existing
+ contracts. (§3.5)
+- R6 `ApiSValue` stays `<= 64` bytes; sidecar is a parallel array. (§3.6, D10)
+- R7 No global state; sidecar + unevaluated counter hang off `KitCg`. (§3.7)
+- R8 Sidecar opt-in; toy/wasm pay nothing. (§3.8)
+- R9 Suppression: type+flags maintained, zero emission, zero temp/local alloc, no
+ forced CG-type lowering; parser loses its `if (emit)` forks. (§4.5)
+- R10 Eager places; all EA folding (const offset, dynamic index, displacement
+ overflow, SEXT-load hint) decided inside CG; `PcgLvAux` deleted. (§4.3, D4)
+- R11 Bit-field geometry rides the CG place (`field_bits`); C keeps a semantic
+ `C_BITFIELD` flag for `sizeof`/`&` rejection. (§4.1, §4.3)
+- R12 C value flags (MODIFIABLE / NULL_PTR_CONST / REGISTER / LVALUE) in
+ `lang_flags`. (§4.1)
+- R13 `store` / `store_keep` replace assignment dup/rot3/swap. (§4.4)
+- R14 No call-mark API; C args stay on the CG stack. (§4.6, D8)
+- R15 ICE evaluation stays parser-side and suppression-independent. (§4.5)
+- R16 Gate = correctness + determinism, not byte-identity. (D5)
+- R17 `cg_adapter.c`/`.h` and the `pcg_` prefix deleted; parser drives `kit_cg_*`
+ directly; C-semantic helpers survive as plain functions. (§5, D7)
+- R18 inc/dec, compound assignment, va_arg, atomics, intrinsics, inline-asm,
+ calls, returns all re-expressed on the new place ops + slot queries. (§6)
+- R19 Unevaluated mode folds constants (not just types) and is valid at file
+ scope; `&&`/`||`/`?:` fold their all-constant cases under it. (§4.5)
+- R20 `eval_const_int` reimplemented over the normal parse path; the `cexpr_*`
+ grammar and `cint_*` engine deleted; all ICE contexts re-tested. (§4.7, §7.6)
+- R21 i128 constant query (`kit_cg_top_const_i128`) added; ICE diagnostics
+ preserved adequately ("constant expression required" + optional syntactic
+ guards). (§4.7)
+
+---
+
+## 9. Expected payoff
+
+CG API self time is ~19% of compile, half of it the stack/memory family, so the
+directly visible ceiling of this cleanup is ~9-10% of compile. The real win is
+larger only insofar as deleting the parser's duplicate stack + EA bookkeeping
+also removes frontend work (one push/pop instead of two; no `PcgLvAux`
+maintenance; no `if (emit)` branching). A realistic target is **5-10% total
+compile improvement**, plus the qualitative payoff: one expression stack, no
+fragile choreography, "parse expression, drive CG," and a clean home for future
+frontend instrumentation — with single-pass liveness and temp reclaim preserved.
+
+---
+
+## 10. Non-goals
+
+- No AST pass; no handle-based expression IR.
+- No collapse of parser / CG / NDT / NT / MC into one module.
+- No C frontend type dependency from `src/cg`.
+- No attempt to close the whole ~2.8× tcc compile-speed gap with this refactor.
diff --git a/doc/plan/LEX-PP-API.md b/doc/plan/LEX-PP-API.md
@@ -0,0 +1,571 @@
+# Lexer and preprocessor API: a lean token boundary for the C frontend
+
+**Goal.** Redesign the lexer -> preprocessor -> parser boundary so a wholesale
+lexer/preprocessor rewrite can be high-performance without fusing the frontend
+layers. The lexer, preprocessor, and parser remain separately reusable modules:
+`cpp` / `cc -E` still use the preprocessor as a standalone component, and the C
+parser still consumes a preprocessed token stream. What changes is the data and
+the handoff contract.
+
+This is a design note for the new API and boundary types. It is not yet an
+implementation plan for the scanner algorithm itself.
+
+---
+
+## 1. Current boundary
+
+Today the frontend has a simple but heavy token contract.
+
+### 1.1 C compile path
+
+`lang/c/c.c` wires the C frontend as:
+
+```text
+source bytes
+ -> lex_open_mem / lex_skip_shebang
+ -> pp_new / pp_push_input
+ -> parse_c
+ -> pcg adapter
+ -> KitCg
+ -> CgTarget / NativeDirectTarget
+ -> NativeTarget
+ -> MC emit
+```
+
+The current parser-feed path sets `pp_set_suppress_lexer_newlines(pp, 1)` before
+the primary lexer is pushed. Include lexers inherit the same policy. This avoids
+materializing non-directive newline tokens for the parser, while still surfacing
+directive-terminating newlines so directive reading works.
+
+### 1.2 Lexer input
+
+`lex_open_mem` takes:
+
+- `Compiler*`
+- source name as `const char*` or pre-interned `Sym`
+- borrowed source bytes and byte length
+
+Opening a lexer also registers a fresh compiler source file id. The lexer owns
+phase-2 line-splice folding. If a file contains no backslash-newline splices, it
+borrows the original bytes. If splices exist, it builds a folded copy and records
+fold points for physical line accounting.
+
+That means the true lexer input boundary is not just `(char*, len)`. It is:
+
+- source identity
+- source bytes
+- ownership/lifetime rule for those bytes
+- source registry side effect
+- newline policy
+- phase-2 splice policy
+- optional primary-file shebang handling
+
+### 1.3 Lexer output
+
+The lexer returns `Tok` by value:
+
+```c
+typedef struct Tok {
+ u16 kind;
+ u16 flags;
+ SrcLoc loc;
+ Sym spelling;
+ union {
+ Sym ident;
+ Sym str;
+ u32 punct;
+ } v;
+} Tok;
+```
+
+Important properties:
+
+- Every content token carries a full `SrcLoc` (`file_id`, `line`, `col`).
+- Every token that needs text carries an interned exact `spelling`.
+- Identifiers are represented by interned `Sym`.
+- Punctuators carry both an integer punctuator code and an interned spelling.
+- Number/string/char literals carry interned spelling even though parsing later
+ reads the spelling bytes again.
+- `TOK_NEWLINE` is a real token for PP mode and directive termination.
+- `TOK_PP_HASH`, `TOK_PP_PASTE`, and `TOK_HEADER` are lexer-visible PP concepts.
+
+### 1.4 PP internal source stack
+
+PP reads from a stack of token sources:
+
+- `SRC_LEX`: a `Lexer*`
+- `SRC_BUF`: a `Tok[]` replay buffer
+
+`SRC_BUF` is used for macro bodies, argument prescan, pushback, `#pragma`
+forwarding, and synthetic token sequences. Hidesets are internal to PP and are
+carried either as a per-token side array or as one uniform hideset id for a
+whole replay buffer.
+
+Current macro bodies store full `Tok` arrays. A no-`##` body can be pointer
+replayed with per-invocation loc/first-flag overrides; bodies with paste still
+copy/substitute/paste through scratch buffers.
+
+### 1.5 PP output to parser
+
+`pp_next()` returns a macro-expanded `Tok` by value. The parser wraps it with:
+
+- a `pending` token slot for string-literal fusion
+- a one-token lookahead slot
+- a replay buffer for recorded braced blocks
+
+The parser should not see non-directive newlines or directives. It does still
+see forwarded non-pragma `#` tokens in invalid cases, and it explicitly swallows
+forwarded pragmas.
+
+There are already out-pointer paths inside PP (`pp_next_raw_into`,
+`pp_next_into`) because returning a 24-byte `Tok` by value costs a hidden sret
+copy. The public boundary has not caught up to that shape.
+
+---
+
+## 2. Design decision
+
+Use a lean token as the canonical internal token for the C preprocessor lane:
+
+- identifiers are interned eagerly
+- punctuators are represented by integer code only
+- exact spelling is a text reference, not always an interned `Sym`
+- source location is a compact location reference, with line/column recovered
+ lazily when diagnostics or `__LINE__` need it
+- PP writes into caller-owned token slots
+- PP owns hidesets; hidesets do not cross to the parser
+
+This does not require control-flow fusion. The scanner can be rewritten as a
+tcc-class high-throughput scanner, the PP can keep a source stack and macro
+engine, and the parser can remain recursive descent. The boundary becomes cheap
+enough that keeping the layers separate is not itself the hot cost.
+
+---
+
+## 3. Proposed core types
+
+The names here are provisional. The point is the contract.
+
+### 3.1 Source input
+
+```c
+typedef uint32_t CppSourceId;
+
+typedef enum CppSourceFlag {
+ CPP_SRC_PRIMARY = 1u << 0, /* allow shebang skip */
+ CPP_SRC_SYSTEM = 1u << 1, /* diagnostics/deps source property */
+ CPP_SRC_PARSER_FEED = 1u << 2, /* suppress non-directive newlines */
+ CPP_SRC_NO_SPLICES = 1u << 3, /* caller proves no backslash-newline */
+} CppSourceFlag;
+
+typedef struct CppSourceSpec {
+ KitSlice name;
+ const char* bytes;
+ uint32_t len;
+ uint32_t flags;
+} CppSourceSpec;
+```
+
+The lexer open API should make source identity explicit. It should return, or
+store in the lexer, the compiler file id registered for this source. Include
+resolution remains PP-owned; the lexer should not do filesystem work.
+
+`CPP_SRC_NO_SPLICES` is the safe version of today's paste fast path. It is only
+valid for synthetic buffers whose construction proves no line splice can exist.
+Normal source files and command-line definitions stay on the safe splice scan.
+
+### 3.2 Location reference
+
+```c
+typedef struct CppLocRef {
+ uint32_t file_id;
+ uint32_t byte_off;
+} CppLocRef;
+```
+
+The hot token carries a byte offset in the logical source stream, not an eager
+`line`/`col` pair. The source registry or lexer source object must have enough
+line-map data to turn `(file_id, byte_off)` into `KitSrcLoc` on demand.
+
+`#line` complicates this. The PP should keep the current file/line overlay on
+the source stack, as it does today, and provide a helper:
+
+```c
+KitSrcLoc cpp_pp_materialize_loc(CppPP* pp, CppLocRef loc);
+```
+
+Dynamic `__LINE__` expansion should use the overlay-aware helper. Diagnostics
+from parser/PP should also route through it. The parser should not know the
+details of `#line` state.
+
+### 3.3 Text reference
+
+```c
+typedef enum CppTextKind {
+ CPP_TEXT_NONE = 0,
+ CPP_TEXT_SOURCE = 1,
+ CPP_TEXT_SYM = 2,
+} CppTextKind;
+
+typedef struct CppTextRef {
+ uint32_t kind;
+ uint32_t source_id;
+ uint32_t off;
+ uint32_t len_or_sym;
+} CppTextRef;
+```
+
+For source tokens, `CPP_TEXT_SOURCE` names a byte span in the source's logical
+post-splice buffer. For synthetic tokens, `CPP_TEXT_SYM` carries an interned
+symbol id in `len_or_sym`.
+
+Helpers:
+
+```c
+KitSlice cpp_text_slice(CppPP* pp, CppTextRef text);
+Sym cpp_text_intern(CppPP* pp, CppTextRef text);
+int cpp_text_eq_cstr(CppPP* pp, CppTextRef text, const char* s);
+```
+
+This keeps exact spelling available for `-E`, diagnostics, stringize, paste,
+literal decode, and macro identity checks without forcing every punctuator and
+literal spelling through the global symbol table on first lex.
+
+### 3.4 Token
+
+```c
+typedef struct CppTok {
+ uint16_t kind;
+ uint16_t flags;
+ uint32_t aux;
+ CppLocRef loc;
+ CppTextRef text;
+} CppTok;
+```
+
+Field meaning:
+
+- `kind`: token kind (`EOF`, `IDENT`, `NUM`, `FLT`, `STR`, `CHR`, `PUNCT`,
+ `PP_HASH`, `PP_PASTE`, `HEADER`, `NEWLINE`, plus PP-internal kinds).
+- `flags`: BOL, leading-space, no-expand, literal suffix/encoding flags,
+ literal-bad.
+- `aux`: identifier `Sym`, punctuator code, macro parameter index, or small
+ synthetic value depending on kind.
+- `loc`: compact source location.
+- `text`: exact spelling reference if the token has one.
+
+For identifiers, `aux` is the interned `Sym`. `text` may also be present as the
+exact spelling span, but normal identifier equality and macro lookup use `aux`.
+
+For punctuators, `aux` is the `Punct` code. `text` is optional for common
+canonical punctuators on the parser-feed path. The `-E`, stringize, and paste
+paths can still recover spelling by either source span or canonical punctuator
+table. Digraphs need an exact source span.
+
+For literals, `text` is the source or synthetic spelling. Numeric and string
+decoding should read via `cpp_text_slice` and only intern if a later operation
+requires a `Sym`.
+
+---
+
+## 4. New API shape
+
+### 4.1 Lexer
+
+```c
+typedef struct CppLexer CppLexer;
+
+int cpp_lex_open(CppLexer* lx, KitCompiler* c, const CppSourceSpec* src);
+void cpp_lex_reset(CppLexer* lx, const CppSourceSpec* src);
+void cpp_lex_close(CppLexer* lx);
+
+void cpp_lex_set_mode(CppLexer* lx, uint32_t flags);
+void cpp_lex_next(CppLexer* lx, CppTok* out);
+
+uint32_t cpp_lex_file_id(const CppLexer* lx);
+KitSrcLoc cpp_lex_materialize_loc(const CppLexer* lx, CppLocRef loc);
+KitSlice cpp_lex_text_slice(const CppLexer* lx, CppTextRef text);
+```
+
+The lexer should fill a caller-provided token slot. It should not return token
+structs by value.
+
+The lexer should not track include keyword state as a hidden state machine.
+Instead PP should request header-name lexing while reading an include/embed
+directive. That can be a mode bit scoped to the next token, or an explicit
+entry point:
+
+```c
+void cpp_lex_next_header_name(CppLexer* lx, CppTok* out);
+```
+
+This removes `#include` knowledge from the normal scanner hot path.
+
+### 4.2 Preprocessor
+
+```c
+typedef struct CppPP CppPP;
+
+int cpp_pp_open(CppPP* pp, KitCompiler* c, const KitPreprocessOptions* opts);
+void cpp_pp_close(CppPP* pp);
+
+int cpp_pp_push_source(CppPP* pp, const CppSourceSpec* src);
+void cpp_pp_add_include_dir(CppPP* pp, const char* dir, int system);
+void cpp_pp_define(CppPP* pp, const char* name, const char* body);
+void cpp_pp_undef(CppPP* pp, const char* name);
+
+void cpp_pp_next_raw(CppPP* pp, CppTok* out);
+void cpp_pp_next_parse(CppPP* pp, CppTok* out);
+void cpp_pp_emit_text(CppPP* pp, KitWriter* out);
+
+KitSrcLoc cpp_pp_materialize_loc(CppPP* pp, CppLocRef loc);
+KitSlice cpp_pp_text_slice(CppPP* pp, CppTextRef text);
+Sym cpp_pp_text_intern(CppPP* pp, CppTextRef text);
+```
+
+`cpp_pp_next_raw` is the `-E` stream: macro-expanded, directives consumed,
+newlines preserved as needed for text reconstruction.
+
+`cpp_pp_next_parse` is the parser stream: macro-expanded, directives consumed,
+non-directive newlines absent, forwarded pragmas swallowed or represented by a
+parser-ignored event.
+
+Both write into caller-owned slots.
+
+### 4.3 Parser
+
+The parser should own a small token cursor:
+
+```c
+typedef struct CTokenCursor {
+ CppPP* pp;
+ CppTok cur;
+ CppTok next;
+ CppTok pending;
+ uint8_t has_next;
+ uint8_t has_pending;
+} CTokenCursor;
+```
+
+The parser's `advance`, `peek1`, pending string-literal fusion, and braced-block
+replay stay local to the parser. The difference is that they copy `CppTok`, not
+the old eager-interned `Tok`, and they ask PP/text helpers only when a spelling
+or materialized location is actually needed.
+
+---
+
+## 5. Layer responsibilities
+
+### 5.1 Lexer owns
+
+- raw byte scanning
+- phase-2 splice folding and source line maps
+- physical BOL / leading-space flags
+- token kind recognition
+- identifier interning
+- numeric/string/char literal extent and suffix/encoding flags
+- punctuator code classification
+- header-name tokenization only when PP explicitly requests it
+
+### 5.2 PP owns
+
+- include search, include cache, include graph edges
+- source stack and source lifetime
+- directive recognition and directive-line reading
+- conditional inclusion stack and skipped-section scanning
+- macro table
+- hidesets
+- macro argument collection and prescan
+- macro replay buffers
+- token paste and stringize
+- dynamic predefined macros
+- `#line` overlays
+- raw stream vs parser stream policy
+- text emission for `-E`
+
+### 5.3 Parser owns
+
+- C keyword classification
+- token lookahead
+- string-literal fusion
+- braced-block replay used by current parser logic
+- literal value decoding
+- diagnostics through PP materialization helpers
+- semantic/type resolution and CG driving
+
+Hidesets must not cross into the parser. Include state must not cross into the
+lexer. C semantic keyword/type state must not cross into PP.
+
+---
+
+## 6. Important invariants
+
+1. **Exact spelling remains available.**
+ `#`, `##`, `-E`, diagnostics, literal decode, macro redefinition checks, and
+ header parsing all require exact spelling. The change is lazy access, not
+ lossy tokens.
+
+2. **Parser-feed tokens do not require newline materialization.**
+ The parser stream should never allocate or return non-directive newline
+ tokens. Directive termination remains an internal PP/lexer concern.
+
+3. **Identifier lookup stays `Sym`-based.**
+ Macro lookup and parser binding/keyword lookup should be one indexed or
+ table lookup on an interned identifier.
+
+4. **Punctuator spelling is not a hot-path symbol-table operation.**
+ Punctuators are roughly half the stream on large C files. The parser needs
+ the punctuator code, not a symbol table entry. Exact text is only needed for
+ `-E`, stringize, paste, and diagnostics.
+
+5. **Line/column is lazy.**
+ Hot tokens should not pay column arithmetic and `SrcLoc` construction unless
+ a consumer needs a materialized location.
+
+6. **Synthetic tokens have stable text.**
+ Macro expansions, dynamic predefined macros, paste output, stringize output,
+ and command-line definitions must have text refs with lifetime at least until
+ the returned token is consumed or until the owning PP arena is reset.
+
+7. **Macro replay can remain pointer-based.**
+ Macro bodies should store `CppTok[]` plus compact metadata. Object-like
+ no-paste bodies can still be pointer-replayed with loc/first-flag override.
+
+8. **The public compatibility API can exist during migration.**
+ Old `Tok pp_next(Pp*)` can be implemented as an adapter while parser code is
+ converted. This allows byte-identical gates at each step.
+
+---
+
+## 7. Migration plan
+
+### Step 1: introduce the types and adapters
+
+Add `CppTok`, `CppTextRef`, `CppLocRef`, and text/location helper APIs next to
+the current `Tok`. Keep old `Tok` as the public compatibility type.
+
+Build adapter helpers:
+
+```c
+void cpp_tok_to_old_tok(CppPP* pp, const CppTok* in, Tok* out);
+void old_tok_to_cpp_tok(CppPP* pp, const Tok* in, CppTok* out);
+```
+
+The old adapter will eagerly intern text and materialize locations. That is
+acceptable as a temporary compatibility layer.
+
+### Step 2: make PP public pull slot-based
+
+Add slot-based public APIs over the existing implementation:
+
+```c
+void pp_next_into_public(Pp* pp, Tok* out);
+void pp_next_raw_into_public(Pp* pp, Tok* out);
+```
+
+Convert parser fetch to use the slot API before changing representation. This
+is a low-risk cleanup because PP already has internal out-pointer machinery.
+
+### Step 3: rewrite lexer behind a compatibility adapter
+
+Implement the new scanner to fill `CppTok`. Convert to old `Tok` at the lexer
+or PP boundary. Gate on:
+
+- PP golden tests
+- parser tests
+- sqlite object byte identity
+- sqlite `-E` token/text identity
+- diagnostic location parity on focused cases
+
+At this stage performance will understate the final win because the adapter
+still eagerly interns/materializes.
+
+### Step 4: convert PP macro storage and source stack to `CppTok`
+
+Move macro bodies, replay buffers, argument vectors, and directive lines to
+`CppTok`. Keep `pp_next()` compatibility at the outside edge.
+
+This is where lazy spelling starts to matter: macro expansion should carry
+source text refs until a PP operation demands interned text.
+
+### Step 5: convert parser cursor to `CppTok`
+
+Replace parser `Tok` slots with `CppTok` slots. Parser helpers should use:
+
+- `tok_ident(t)` for identifier `Sym`
+- `tok_punct(t)` for punctuator code
+- `cpp_pp_text_slice` for literal spelling
+- `cpp_pp_materialize_loc` for diagnostics and CG source loc
+
+After this step, the old `Tok` adapter is no longer on the compile hot path.
+
+### Step 6: delete or narrow the old token contract
+
+Keep an old-style token only where external API compatibility requires it, or
+delete it if no public API promises it. The C frontend hot path should be
+`CppTok` end to end through lexer, PP, and parser.
+
+---
+
+## 8. Open questions
+
+1. **Line-map owner.**
+ Should line maps live in the compiler source registry, the lexer source
+ object, or PP's source cache? The answer affects how parser diagnostics
+ materialize `CppLocRef` after the lexer for an include has been popped.
+
+2. **Text lifetime for folded source.**
+ If a source needed splice folding, `CppTextRef` must point at the folded
+ logical buffer, not the original bytes. That folded buffer must live long
+ enough for any token text refs in macro bodies that outlive the source file.
+
+3. **Macro body text retention.**
+ A macro body defined in an include may survive after that include source is
+ popped. Either macro-body tokens must intern/copy their text at definition
+ time, or source buffers referenced by macro bodies must be retained until
+ `pp_free`.
+
+4. **Canonical punctuator text.**
+ For canonical punctuators, can `CppTextRef` be omitted and reconstructed from
+ the punctuator code? That is attractive for parser-feed and most PP paths,
+ but `-E` must preserve digraph spelling where relevant.
+
+5. **Header-name mode.**
+ The current lexer has directive state to emit `TOK_HEADER`. The new design
+ should move this to PP, but the exact API should be chosen when directive
+ reading is rewritten.
+
+6. **Token-paste validation.**
+ Paste currently concatenates spellings and re-lexes a tiny buffer. The new
+ scanner should preserve that validation path but route it through
+ `CPP_SRC_NO_SPLICES` and a reusable lexer object.
+
+7. **Diagnostics parity.**
+ Lazy location must preserve user-visible line/column behavior, including
+ `#line`, includes, splices, comments, and shebang handling.
+
+8. **Public names.**
+ The final names should probably not expose `Cpp` if this becomes the common
+ C-family preprocessor substrate. The design uses `Cpp*` only to avoid
+ colliding with current `Tok`/`Pp` names.
+
+---
+
+## 9. Initial right-sized target
+
+The first useful deliverable is not the full PP rewrite. It is a measured
+compatibility lane:
+
+1. New source spec, loc ref, text ref, and token definitions.
+2. Slot-based public PP pull API over the current code.
+3. New scanner filling the lean token.
+4. Adapter to old `Tok`.
+5. Head-to-head sqlite measurement:
+ - raw lex instructions per byte
+ - compile `-c` instructions
+ - `-E` byte identity
+ - object byte identity
+
+If the adapter lane cannot produce a scanner win in isolation, the scanner
+algorithm is not yet good enough. If it does win in isolation but not through
+compile, the next bottleneck is PP token storage/replay and parser conversion.
diff --git a/doc/plan/README.md b/doc/plan/README.md
@@ -11,6 +11,8 @@ shrinks to whatever remains open.
| [RELEASE.md](RELEASE.md) | Cross-cutting initial-release punchlist: release scope, deferred features, and per-subsystem completion/validation items. | — |
| [OPTIMIZER.md](OPTIMIZER.md) | Completing the O2 SSA mid-end, expanded inlining, -O0/-O1 performance work, machine register-constraint improvements. | [../OPT.md](../OPT.md) |
| [PERF.md](PERF.md) | Making kit the fastest `-O0` C compiler with code as dense as tcc: current compile-speed + code-size standings, how to reproduce them (macOS instruction counts, Linux callgrind, the `make bench-cc` scaling guard), and the ranked forward-looking levers on both axes. | [../ARCH.md](../ARCH.md) |
+| [CG-STACK-API.md](CG-STACK-API.md) | Refactoring the parser-to-CG expression seam: keep stack-based liveness, move the C frontend's duplicated `pcg` slot state onto CG-owned stack slots, and add parser-shaped place/load/store operations without backend fusion. | [../FRONTENDS.md](../FRONTENDS.md) |
+| [LEX-PP-API.md](LEX-PP-API.md) | Redesigning the lexer -> preprocessor -> parser boundary around lean tokens, lazy spelling/location materialization, and slot-based handoffs for a high-performance lexer/preprocessor rewrite without frontend layer fusion. | [../FRONTENDS.md](../FRONTENDS.md) |
| [LINKER.md](LINKER.md) | Incremental linking: the file-based object-link redesign and remaining non-ELF format coverage. | [../LINK.md](../LINK.md) |
| [LINKER-COMPAT.md](LINKER-COMPAT.md) | Completing system-linker compatibility across the support set: ordered DSO selection, ELF TLS/TLSDESC, shared libraries, relocatable links, runtime/sysroot interoperability, and Rust/toolchain validation. | [../LINK.md](../LINK.md), [../OBJ.md](../OBJ.md), [../DRIVER.md](../DRIVER.md) |
| [JIT.md](JIT.md) | Function-level hot reload, Go-runtime-style codegen support, and remaining JIT host-portability work. | [../JIT.md](../JIT.md) |
