commit 54cd542c9e1c6acb458dfe3ff604e99f1926554a
parent 94eaef5e27c8ca1c0e3f9dcc5556f29f7b6503e2
Author: Ryan Sepassi <rsepassi@gmail.com>
Date: Mon, 15 Jun 2026 08:30:26 -0700
lex/pp: lean-token boundary rewrite (byte-identical, -1.3% -c)
Clean-break rewrite of the C lexer/preprocessor/parser boundary to the
doc/plan/LEX-PP-API.md "lean token" design:
- Tok carries a compact LocRef (file_id + byte offset) and a TextRef
(source span / interned Sym / canonical-punct), not an eager SrcLoc +
interned spelling; line/col and spelling materialize on demand.
- Slot-based pulls (lex_next / pp_next_raw / pp_next_parse fill caller
slots); PP creates and owns lexers via pp_push_source.
- PP owns a SrcInfo registry (retained source buffers + lazy memchr line
index + #line overlay segments) for loc/text materialization; hidesets
stay PP-internal.
Byte-identical: sqlite -E / -c / -c -g / -S / diagnostics bit-for-bit vs the
pre-rewrite snapshot; test-pp, test-parse, test-toy, test-cg-api, test-dwarf,
test-debug and perf-gate (incl. the splice/#line/diagnostic battery) green.
Net -24M -c instructions (-1.3%) vs baseline, after the loc-materialization
fix (memchr index build + cursored pp_materialize_loc).
This is the boundary redesign only. Drain-measured lex+pp is still ~2.1x tcc
(per-token struct copy, Prosser hidesets, and layered dispatch are unchanged);
the structural per-token work is a follow-on (PERF.md section 4.1). KIT_PP_DRAIN
(lang/c/c.c) isolates lex+pp for that measurement.
Diffstat:
16 files changed, 1650 insertions(+), 1157 deletions(-)
diff --git a/doc/plan/PERF.md b/doc/plan/PERF.md
@@ -39,19 +39,36 @@ readings (`sysctl -n vm.loadavg`) shown only for context.
| compiler | instructions | cycles † | wall † | object |
|---|--:|--:|--:|--:|
| **tcc 0.9.28** | 0.66 B | 0.19 B | 0.06 s | 2.11 MB |
-| **kit** | 1.87 B | 0.52 B | 0.16 s | 1.83 MB |
+| **kit** | 1.84 B | 0.53 B | 0.16 s | 1.83 MB |
| clang 22 | 8.73 B | 2.57 B | 0.81 s | 1.50 MB |
† low-load; instructions is the figure to trust.
kit beats clang and is the fastest *general* backend, but **tcc is ~2.8×
(instructions) ahead** — this is the whole game. **The gap is instructions, not
-cache:** kit's IPC (~3.7) is *higher* than tcc's (~3.5), so both are compute-bound
+cache:** kit's IPC (~3.5) is on par with tcc's (~3.5), so both are compute-bound
on a wide core and there is no hidden cache-miss penalty to claw back. The payoff
mechanism is **fewer retired instructions** (fewer copies, indirect calls,
redundant recomputations) — denser structures help only because they cost fewer
load/store/move *instructions*.
+> **Lean-token lexer/PP rewrite (landed, byte-identical, net win).** The
+> lexer/preprocessor boundary was rewritten to the `doc/plan/LEX-PP-API.md` "lean
+> token" design (compact `LocRef` byte-offset locations materialized on demand;
+> `TextRef` spellings; slot-based pulls). sqlite `-E`, `-c` object, `-g`, `-S`,
+> and diagnostics are **bit-for-bit identical** to the pre-rewrite snapshot
+> (`perf-gate` green incl. the splice/`#line`/diagnostic battery), so **code size
+> is unchanged**. The boundary-only landing was initially a −3.35 % `-c`
+> regression — lazy line/col was a *loss* because line numbers are needed in the
+> happy path anyway (sqlite expands `__LINE__` pervasively, and the parser stamps
+> a CG loc per statement) and the first cut built the line index with a scalar
+> `\n` rescan + a per-loc binary search. Fixing the *implementation* of laziness
+> (build the index with memchr; cursor the materializer — §4.1) turned it into a
+> net **−24 M `-c` win** (1.865 B → 1.841 B), since the scanner no longer builds a
+> `SrcLoc` per token. The remaining ~+19 M parse+CG residual (per-statement loc
+> lookups the old eager-on-token loc got for free) is the optional Stage-2 target
+> in §4.1.
+
### Code size — ≈ parity (1.044× tcc)
The honest metric is **`.text` machine code** (the object file is format-skewed —
@@ -79,44 +96,83 @@ Net +14,972.
## 2. Where the cost is (current profile)
-### Compile is frontend-bound, ~uniform ~3× across phases
+### Compile is frontend-bound — lex+pp ~2.1× tcc, post-PP ~3.8× tcc
-Phase decomposition by subtraction (instructions; `-fsyntax-only` drives the full
-CG value-stack + type lowering routed to the no-op check backend, so `(-c) −
-(-fsyntax-only)` isolates native-emit + object-write):
+Phase split measured directly (instructions, best-of-7), not estimated from `-E`.
+`-E` is a **bad** lex+pp proxy: it re-serializes the token stream to text, work
+neither compiler does at `-c`, and the two serialize at wildly different cost
+(tcc's `-E` 1.17 B is ~0.79 B *serialization* via `get_tok_str` re-stringify +
+whitespace logic, so tcc `-E` even exceeds tcc `-c`; kit's `-E` serializes for
+~0.06 B since tokens carry `TEXT_SRC` spans → emit is a `memcpy`). To isolate
+lex+pp we **drain the token stream to EOF with no parse and no output** in both
+compilers (tcc: `-bench` hook, patched to use `-c` `parse_flags`; kit:
+`KIT_PP_DRAIN`, see §3):
-| phase | kit `-c` | share | ratio to tcc |
+| phase | kit `-c` | tcc `-c` | kit / tcc |
|---|--:|--:|--:|
-| lex + pp | ~0.93 B | ~49 % | ~2.8× |
-| parse + sema + types + CG-drive | ~0.71 B | ~37 % | ~3.5× |
-| native emit + object write | ~0.27 B | ~14 % | **~7×** (worst ratio, smallest slice) |
-| **total** | **1.91 B** | 100 % | **~2.9×** |
-
-Two facts govern strategy: (1) the gap is **roughly uniform across phases** — no
-single hot function to crush; closing it is a campaign across the pipeline. (2)
-**Post-PP work alone (`-c` − `-E` ≈ 0.98 B) is already ~1.5× tcc's entire
-compile** — a *free* lexer would still leave kit above tcc, so the lexer is not
-the sole frontier.
+| **lex + pp** | **0.82 B** | **0.39 B** | **~2.1×** |
+| parse + sema + types + CG-drive | 0.76 B | ┐ 0.27 B | — |
+| native emit + object write | 0.26 B | ┘ (post-PP) | — |
+| **post-PP total** | **1.02 B** | **0.27 B** | **~3.8×** |
+| **total** | **1.84 B** | **0.66 B** | **~2.8×** |
+
+So **both halves are real frontiers** — lex+pp is *not* at parity (an earlier
+claim from the misleading `-E` proxy was wrong). The post-PP ratio is larger, but
+lex+pp is ~44 % of kit's `-c` and a clean 2.1× behind. Note kit's lex+pp drain
+does *less* than tcc's — kit defers number/string decode to the parser (tcc decodes
+in the lexer; bare tcc scan+expand is 0.38 B, +decode 0.39 B) — yet is still ~2.1×
+heavier, so the gap is pure per-token engine overhead, not extra work:
+
+- **tcc: one mutable global token** (`int tok` + `CValue tokc` + `tok_flags`) —
+ the lexer and macro replayer write it in place; no per-token struct is built or
+ copied. kit writes a 32-byte `Tok` (kind/flags/aux/`LocRef`/`TextRef`) per token
+ and copies it lex→pp→parser through slots.
+- **tcc: no per-token location** (diagnostics read a `file->line_num` counter);
+ kit stores a `LocRef` per token (cheap, but ≠ free) and the pp checks it.
+- **tcc: macro bodies are `int[]` token streams** replayed via a `macro_ptr`
+ cursor with **no hidesets** (a cheap nested-macro guard); kit replays `Tok[]`
+ with a Prosser hideset side-channel (hash-consed ids, union/dedup per expansion).
+- **tcc: `TokenSym` caches direct `Sym*` pointers** (define/ident/struct/label) on
+ the interned token, so define-lookup is a pointer-follow; kit re-probes
+ (`pool_intern_slice` 3.9 %, the Sym-indexed macro table 1 load).
+- **tcc: one `next()`** does lex + pp + feeds the parser — no inter-stage boundary;
+ kit layers lex → pp (source stack, `src_next_raw_into`→`pp_pull_into`) →
+ `pp_next_parse` → parser, and the pp re-checks directive/`defined`/dynamic-macro
+ state per token. This is the [[frontend-instruction-halving-pathb]] lever: fewer
+ per-token ops, not less work.
+
+The post-PP 3.8× is the bigger slice: tcc drives a thin `SValue[]` straight into a
+one-pass emitter, while kit routes through the `CgTarget`→`NativeTarget`→`MCEmitter`
+seam and a richer type/ABI layer (§4). Closing either is a campaign, not one hot
+function.
+
+Net of the lean-token rewrite + its loc-materialization fix (golden→now,
+best-of-7): **lex+pp −44 M** (the scanner no longer builds a `SrcLoc` per token —
+`loc` is a byte offset), **parse+sema+CG +19 M** (the residual: a per-statement
+cursor lookup at `pcg_set_loc` the old eager-on-token loc got for free), emit
+flat → **−24 M total (−1.3 %)**.
**Linux callgrind** (inclusive, instruction-grounded — the tool that sees what
-wall-clock `sample` hides; total **1.746 B `Ir`**, glibc/ELF):
+wall-clock `sample` hides; total **1.652 B `Ir`**, glibc/ELF; self-`Ir` summed
+across callgrind's `'2` symbol splits):
| self % | function(s) | subsystem |
|--:|---|---|
-| **16.9** | `lex_next` | scanner |
-| 5.0 / 4.5 / 2.0 | `src_next_raw_into` / `lex_point_at` / `finish_ident` | scanner+pp |
-| 4.4 | `pp_pull_into` | preprocessor |
-| 4.1 | `pool_intern_slice` | interning |
-| 3.1 / 2.8 | `api_unalias_type` / `api_type_pred_bits` | types |
-| 1.5×4 / 1.3 / 1.2 | `resolve_type`,`cg_type_get`,`type_cg_lower`,`api_type_class` / `abi_cg_type_info` / `cg_type_size` | types |
+| **11.7** | `lex_next` | scanner |
+| 4.5 / 3.9 | `src_next_raw_into` / `pool_intern_slice` | pp + interning |
+| 3.8 | `pp_pull_into` | preprocessor |
+| 3.0 / 3.0 | `api_unalias_type` / `api_type_pred_bits` | types |
+| 2.6 | `finish_ident` | scanner |
+| 1.4 / 1.0×2 | `cg_type_get` / `resolve_type`,`api_type_class` | types |
| 1.0 | `aa_emit_mem` | codegen |
+| **0.9** | `pp_materialize_loc` | lazy-loc lookup (was 2.6 % — now cursored) |
+| **0.6** | `srcinfo_build_lines` | lazy-loc index build (was 5.3 % — now memchr) |
-**Subsystem rollup:** scanner ~24 %, **types ~14 %**, preprocessor ~12 %,
-interning ~4 %, codegen/emit ~3 %. The macOS wall-clock `sample` makes the scanner
-*look* like the lone frontier (`lex_next` ~72 %) and renders the type subsystem
-nearly invisible — callgrind reveals **types as the #2 self-`Ir` cluster** (a
-dozen O(1)-but-very-frequent per-query helpers). Trust callgrind for *where*;
-trust macOS instructions for *how much*.
+**Subsystem rollup:** scanner ~18 %, types ~13 %, preprocessor ~11 %, interning
+~4 %, lazy-loc (`pp_materialize_loc` + `srcinfo_build_lines`) ~1.5 %, codegen/emit
+~3 %. The scanner is leaner than pre-rewrite (line tracking + eager literal/punct
+interning left `lex_next`); the lazy-loc machinery that briefly dominated is now
+sub-2 %. Trust callgrind for *where*; trust macOS instructions for *how much*.
### Code size is spill-bound
@@ -158,12 +214,20 @@ m tmp/tinycc/tcc -c sqlite3.c -o /tmp/t.o
Caveat: Apple clang spawns a `cc1` that `/usr/bin/time` does **not** count —
measure it with `cc -fintegrated-cc1 -c …`. kit and tcc are single-process.
-**Phase decomposition** (subtraction; instruction counts are cleanest):
+**Phase decomposition.** Do **not** use `-E` as the lex+pp number — it adds
+token→text serialization neither compiler does at `-c`, and tcc/kit serialize at
+very different cost (§2). Instead **drain the token stream to EOF** (lex+pp only,
+no parse, no output) in each compiler:
```sh
-m build/release/kit cc -E -o /dev/null sqlite3.c --sysroot "$SDK" # lex+pp
-m build/release/kit cc -fsyntax-only sqlite3.c --sysroot "$SDK" # +parse/sema/types/CG-drive
+# kit: KIT_PP_DRAIN runs pp_next_parse to EOF then returns (lang/c/c.c)
+m env KIT_PP_DRAIN=1 build/release/kit cc -c sqlite3.c --sysroot "$SDK" -o /dev/null # lex+pp
+m build/release/kit cc -fsyntax-only sqlite3.c --sysroot "$SDK" # lex+pp +parse/sema/types/CG
m build/release/kit cc -c -o /tmp/k.o sqlite3.c --sysroot "$SDK" # +emit+objwrite
+# tcc: the -bench hook drains next() to EOF; patched (tccpp.c tcc_preprocess)
+# to use -c parse_flags. TCC_DRAIN_FLAGS=c also decodes literals in the lexer.
+m env TCC_DRAIN_FLAGS=c tmp/tinycc/tcc -E -bench sqlite3.c -o /dev/null # tcc lex+pp
+m tmp/tinycc/tcc -c sqlite3.c -o /tmp/t.o # tcc full -c
```
**macOS hotspot sample** (self-time leaves only — see the callgrind caveat). One
@@ -243,7 +307,33 @@ not the swappable vtable), or the code-size track (which *bytes* the NDT emits).
### 4.1 Compile speed (frontend-bound)
-**Status (this campaign).** Most of §4.1 has now landed, byte-identical, for a
+**Lean-token loc-materialization fix — Stage 1 DONE (`-c` −24 M net win).** The
+boundary rewrite's initial +62 M `-c` regression was two lazy-loc implementation
+costs, both now fixed byte-identically (frontend-only, no CG-API change):
+- ✅ **memchr the line-index build.** `srcinfo_build_lines` built the line-start
+ index with a scalar `\n` byte loop; replaced with a memchr sweep (5.3 % → 0.6 %
+ self-`Ir`). The build is genuinely needed in the happy path — `__LINE__` (which
+ sqlite expands pervasively) and diagnostics need line numbers even without
+ `-g` — so the lever was to make it cheap, not to skip it.
+- ✅ **cursor `pp_materialize_loc`.** Per-loc lookup was a fresh binary search;
+ now a `SrcInfo.line_cursor` caches the last line index, so the near-monotonic
+ access pattern (parser advances forward) is O(1), binary search only on a
+ backward jump (2.6 % → 0.9 %).
+
+Result: lex+pp −44 M (the scanner no longer builds a `SrcLoc` per token),
+parse+CG +19 M residual, **net −24 M** (§1, §2), `perf-gate` byte-identical.
+
+**Stage 2 (optional, ~+19 M parse+CG residual).** The residual is the
+per-statement loc lookup at `pcg_set_loc` (CG source-loc) that the old
+eager-on-token loc got for free. `g->cur_loc` is consumed only by DWARF (gated on
+`g->debug`), descriptor locs, and cold CG `compiler_panic`s — so with `-g` off and
+no error it is discarded. Carrying it as a `LocRef` (a `kit_cg_set_locref` +
+a "resolve `LocRef`→`SrcLoc`" callback the frontend registers, so cold CG
+diagnostics still materialize exactly) would skip those per-statement lookups
+entirely. Smaller payoff than Stage 1, needs a contained CG-API addition + the
+parser loc-plumbing; gate: byte-identical (`-g` DWARF + diag battery unchanged).
+
+**Status (prior campaign).** Most of §4.1 has now landed, byte-identical, for a
combined **−43.7 M `-c` instructions (−2.29 %)** on sqlite vs the pre-campaign
snapshot (sqlite `.o` + `-E` bit-for-bit identical; 60/60 gate; full parse/pp/cg
/toy/smoke-x64/rv64/debug/dwarf suites green). Item-by-item below: #1 ✅ (landed
diff --git a/lang/c/c.c b/lang/c/c.c
@@ -54,7 +54,8 @@ static void c_profile_define(KitCompiler* c) {
"compile.c.cleanup");
kit_frontend_profile_define_counter(c, C_PROFILE_COUNTER_PP_INCLUDE_DIRS,
"compile.c.pp_include_dirs");
- kit_frontend_profile_define_counter(c, C_PROFILE_COUNTER_PP_SYSTEM_INCLUDE_DIRS,
+ kit_frontend_profile_define_counter(c,
+ C_PROFILE_COUNTER_PP_SYSTEM_INCLUDE_DIRS,
"compile.c.pp_system_include_dirs");
kit_frontend_profile_define_counter(c, C_PROFILE_COUNTER_PP_DEFINES,
"compile.c.pp_defines");
@@ -118,9 +119,9 @@ static KitStatus c_frontend_compile_cg(KitFrontendState* frontend,
* KitFrontendCompileOptions; the C frontend uses no language_options. */
const KitSlice* bytes;
Pool* pool;
- Lexer* lex;
Pp* pp;
DeclTable* decls;
+ SourceSpec spec;
if (!fe || !fe->c) return KIT_INVALID;
c = fe->c;
@@ -133,15 +134,10 @@ static KitStatus c_frontend_compile_cg(KitFrontendState* frontend,
pool = c_pool_new(c);
kit_frontend_profile_scope_end(c, C_PROFILE_SCOPE_POOL_NEW);
if (!pool) compiler_panic(c, c_no_loc(), "C compiler out of memory");
- kit_frontend_profile_scope_begin(c, C_PROFILE_SCOPE_LEX_OPEN);
- lex = lex_open_mem(c, input->name.s, bytes->s, bytes->len);
- if (lex) lex_skip_shebang(lex);
- kit_frontend_profile_scope_end(c, C_PROFILE_SCOPE_LEX_OPEN);
kit_frontend_profile_scope_begin(c, C_PROFILE_SCOPE_PP_NEW);
pp = pp_new(c);
kit_frontend_profile_scope_end(c, C_PROFILE_SCOPE_PP_NEW);
- if (!lex || !pp || !cg)
- compiler_panic(c, c_no_loc(), "C compiler out of memory");
+ if (!pp || !cg) compiler_panic(c, c_no_loc(), "C compiler out of memory");
kit_frontend_profile_scope_begin(c, C_PROFILE_SCOPE_DECL_NEW);
decls = decl_new(c, pool, cg);
kit_frontend_profile_scope_end(c, C_PROFILE_SCOPE_DECL_NEW);
@@ -158,16 +154,33 @@ static KitStatus c_frontend_compile_cg(KitFrontendState* frontend,
fe_opts->preprocess.nundefines);
c_apply_pp_options(pp, &fe_opts->preprocess);
kit_frontend_profile_scope_end(c, C_PROFILE_SCOPE_PP_OPTIONS);
- /* The C parser drops preprocessor newlines, so suppress non-directive
- * newline tokens at the lexer (primary source + #includes) — ~51% of lexer
- * outputs. Directive-terminating newlines are still emitted, so the PP's
- * directive handling is unaffected. Must be set before pp_push_input so the
- * primary lexer inherits the mode. */
- pp_set_suppress_lexer_newlines(pp, 1);
+ /* SRC_PARSER_FEED: the C parser drops preprocessor newlines, so non-directive
+ * newline tokens are suppressed at every lexer (primary + #includes) — ~51%
+ * of lexer outputs. Directive-terminating newlines are still emitted, so the
+ * PP's directive handling is unaffected. SRC_PRIMARY enables shebang skip. */
kit_frontend_profile_scope_begin(c, C_PROFILE_SCOPE_PP_PUSH_INPUT);
- pp_push_input(pp, lex);
+ memset(&spec, 0, sizeof(spec));
+ spec.name = kit_slice_cstr(input->name.s);
+ spec.bytes = bytes->s;
+ spec.len = (u32)bytes->len;
+ spec.flags = SRC_PRIMARY | SRC_PARSER_FEED;
+ pp_push_source(pp, &spec);
kit_frontend_profile_scope_end(c, C_PROFILE_SCOPE_PP_PUSH_INPUT);
+ /* Perf instrumentation: drain the preprocessed token stream to EOF without
+ * parsing or codegen, to isolate lex+pp instructions (the parser-feed stream
+ * matches -c exactly — newlines suppressed, no text serialization). */
+ if (kit_debug_getenv("KIT_PP_DRAIN")) {
+ Tok t;
+ do {
+ pp_next_parse(pp, &t);
+ } while (t.kind != TOK_EOF);
+ decl_free(decls);
+ pp_free(pp);
+ c_pool_free(pool);
+ return KIT_OK;
+ }
+
kit_frontend_profile_scope_begin(c, C_PROFILE_SCOPE_PARSE_CODEGEN);
parse_c(c, pool, pp, decls, cg, (KitSymVis)fe_opts->code.default_visibility,
(int)fe_opts->code.trivial_auto_var_init);
diff --git a/lang/c/parse/parse.c b/lang/c/parse/parse.c
@@ -79,11 +79,13 @@ static const char* const kw_names[KW_COUNT] = {
* Diagnostics
* ============================================================ */
-static SrcLoc tok_loc(const Tok* t) { return t->loc; }
+static SrcLoc tok_loc(Parser* p, const Tok* t) {
+ return pp_materialize_loc(p->pp, t->loc);
+}
_Noreturn void perr(Parser* p, const char* fmt, ...) {
va_list ap;
- SrcLoc loc = tok_loc(&p->cur);
+ SrcLoc loc = tok_loc(p, &p->cur);
va_start(ap, fmt);
compiler_panicv(p->c, loc, fmt, ap);
}
@@ -107,8 +109,8 @@ static Tok fuse_string_lits(Parser* p, Tok a, Tok b) {
u16 ae = (u16)(a.flags & STR_ENC_MASK);
u16 be = (u16)(b.flags & STR_ENC_MASK);
u16 fused_enc;
- KitSlice a_sl = kit_sym_str(p->pool->c, a.spelling);
- KitSlice b_sl = kit_sym_str(p->pool->c, b.spelling);
+ KitSlice a_sl = pp_text_slice(p->pp, &a);
+ KitSlice b_sl = pp_text_slice(p->pp, &b);
size_t alen = a_sl.len, blen = b_sl.len;
const char* as = a_sl.s;
const char* bs = b_sl.s;
@@ -155,7 +157,8 @@ static Tok fuse_string_lits(Parser* p, Tok a, Tok b) {
}
buf[k++] = '"';
out = a;
- out.spelling = kit_sym_intern(p->pool->c, (KitSlice){.s = buf, .len = k});
+ out.text =
+ text_sym_ref(kit_sym_intern(p->pool->c, (KitSlice){.s = buf, .len = k}));
out.flags = (u16)((a.flags & ~STR_ENC_MASK) | fused_enc);
h->free(h, buf, 0);
return out;
@@ -168,11 +171,12 @@ static Tok fetch_tok(Parser* p) {
t = p->pending;
p->has_pending = 0;
} else {
- t = pp_next(p->pp);
+ pp_next_parse(p->pp, &t);
}
if (t.kind != TOK_STR) return t;
for (;;) {
- Tok n = pp_next(p->pp);
+ Tok n;
+ pp_next_parse(p->pp, &n);
if (n.kind != TOK_STR) {
p->pending = n;
p->has_pending = 1;
@@ -293,7 +297,6 @@ u32 count_recorded_top_level_items(const Tok* vec, u32 len) {
* O(1) per lookup. */
#define SCOPE_INDEX_THRESHOLD 12u
-
Scope* scope_new(Parser* p, Scope* parent) {
Scope* s = arena_new(p->pool->arena, Scope);
if (!s) perr(p, "out of memory in scope_new");
@@ -1304,7 +1307,8 @@ static void parse_function_body(Parser* p, ObjSymId fsym, const Type* fn_ty,
/* Record whether this body emits before func_begin (which is itself gated on
* emit-enabled): goto-label allocation keys off this, not the momentary
* suppress_codegen depth, so a label first referenced inside a constant-false
- * region still gets a real CG-label id rather than the suppression sentinel. */
+ * region still gets a real CG-label id rather than the suppression sentinel.
+ */
p->cur_func_emits = (u8)pcg_emit_enabled(p);
pcg_func_begin(p, &fd);
@@ -1575,7 +1579,8 @@ static void parse_file_scope_asm(Parser* p) {
size_t nbytes;
advance(p); /* asm / __asm__ */
for (;;) {
- if (is_kw(p, &p->cur, KW_VOLATILE)) { /* matches `volatile` and `__volatile__` */
+ if (is_kw(p, &p->cur,
+ KW_VOLATILE)) { /* matches `volatile` and `__volatile__` */
advance(p);
continue;
}
@@ -1752,7 +1757,8 @@ void parse_c(Compiler* c, Pool* pool, Pp* pp, DeclTable* decls, CG* cg,
p.sym_thread_alias = kit_sym_intern(p.pool->c, KIT_SLICE_LIT("__thread"));
/* GNU alias spellings -> their canonical CKw. Registered after the canonical
* keywords above so the canonical mapping always wins (matching the old
- * ident_kw_inline fall-through, which scanned kw_sym[] before the aliases). */
+ * ident_kw_inline fall-through, which scanned kw_sym[] before the aliases).
+ */
(void)KwTab_set(&p.kw_map, p.sym_alignof_alias, (u8)KW_ALIGNOF);
(void)KwTab_set(&p.kw_map, p.sym_asm_alias, (u8)KW_BUILTIN_ASM);
(void)KwTab_set(&p.kw_map, p.sym_inline_alias, (u8)KW_INLINE);
diff --git a/lang/c/parse/parse_expr.c b/lang/c/parse/parse_expr.c
@@ -102,7 +102,7 @@ static u32 cint_bits(Parser* p, const Type* ty);
static int cint_signed(Parser* p, const Type* ty);
static u64 parse_int_literal_u64(Parser* p, const Tok* t, int* decimal_out) {
- KitSlice spell_sl = kit_sym_str(p->pool->c, t->spelling);
+ KitSlice spell_sl = pp_text_slice(p->pp, t);
size_t len = spell_sl.len;
const char* s = spell_sl.s;
size_t i = 0;
@@ -222,7 +222,7 @@ static const Type* int_literal_type(Parser* p, const Tok* t) {
}
double parse_float_literal(Parser* p, const Tok* t) {
- KitSlice spell_sl = kit_sym_str(p->pool->c, t->spelling);
+ KitSlice spell_sl = pp_text_slice(p->pp, t);
size_t len = spell_sl.len;
const char* s = spell_sl.s;
size_t i = 0;
@@ -365,7 +365,7 @@ int string_literal_initializes_array(Parser* p, const Type* elem,
}
i64 decode_char_literal(Parser* p, const Tok* t) {
- KitSlice spell_sl = kit_sym_str(p->pool->c, t->spelling);
+ KitSlice spell_sl = pp_text_slice(p->pp, t);
size_t len = spell_sl.len;
const char* s = spell_sl.s;
size_t i = 0;
@@ -388,12 +388,12 @@ i64 decode_char_literal(Parser* p, const Tok* t) {
if (i >= len || s[i] == '\'') perr(p, "empty character literal");
if (!c_lit_decode_unit(s, len, &i, &unit, &err)) {
compiler_panic(
- p->c, t->loc, "%.*s",
+ p->c, pp_materialize_loc(p->pp, t->loc), "%.*s",
KIT_SLICE_ARG(kit_slice_cstr(err ? err : "bad character literal")));
}
if (!c_lit_encode_char_unit(enc, bits, unit, &v, &err)) {
compiler_panic(
- p->c, t->loc, "%.*s",
+ p->c, pp_materialize_loc(p->pp, t->loc), "%.*s",
KIT_SLICE_ARG(kit_slice_cstr(err ? err : "bad character literal")));
}
if (i >= len || s[i] != '\'') {
@@ -403,7 +403,7 @@ i64 decode_char_literal(Parser* p, const Tok* t) {
}
u8* decode_string_literal(Parser* p, const Tok* t, size_t* nlen_out) {
- KitSlice spell_sl = kit_sym_str(p->pool->c, t->spelling);
+ KitSlice spell_sl = pp_text_slice(p->pp, t);
size_t len = spell_sl.len;
const char* s = spell_sl.s;
size_t i = 0;
@@ -429,12 +429,12 @@ u8* decode_string_literal(Parser* p, const Tok* t, size_t* nlen_out) {
CLitUnit unit;
if (!c_lit_decode_unit(s, len, &i, &unit, &err)) {
compiler_panic(
- p->c, t->loc, "%.*s",
+ p->c, pp_materialize_loc(p->pp, t->loc), "%.*s",
KIT_SLICE_ARG(kit_slice_cstr(err ? err : "bad string literal")));
}
if (!c_lit_append_string_unit(buf, &k, enc, elem_size, unit, &err)) {
compiler_panic(
- p->c, t->loc, "%.*s",
+ p->c, pp_materialize_loc(p->pp, t->loc), "%.*s",
KIT_SLICE_ARG(kit_slice_cstr(err ? err : "bad string literal")));
}
}
@@ -455,7 +455,8 @@ KitCgSym emit_string_literal_to_rodata(Parser* p, const u8* bytes,
u32 elem_size = c_abi_sizeof(p->abi, p->pool, elem_ty);
u32 count = elem_size ? (u32)(nbytes / elem_size) : 0;
const Type* arr_ty = type_array(p->pool, elem_ty, count, 0);
- return kit_cg_const_data(p->cg, bytes, nbytes, c_abi_alignof(p->abi, p->pool, elem_ty),
+ return kit_cg_const_data(p->cg, bytes, nbytes,
+ c_abi_alignof(p->abi, p->pool, elem_ty),
pcg_tid(p, arr_ty));
}
@@ -890,7 +891,8 @@ static CConstInt cexpr_unary(Parser* p, SrcLoc loc) {
const Type* t = parse_type_name(p);
expect_punct(p, ')', "')' after sizeof type-name");
require_sizeof_type(p, t);
- return cint_make_u64(p, ty_size_t(p), c_abi_sizeof(p->abi, p->pool, t));
+ return cint_make_u64(p, ty_size_t(p),
+ c_abi_sizeof(p->abi, p->pool, t));
}
}
}
@@ -912,7 +914,8 @@ static CConstInt cexpr_unary(Parser* p, SrcLoc loc) {
{
const Type* t = parse_type_name(p);
expect_punct(p, ')', "')' after _Alignof type-name");
- return cint_make_u64(p, ty_size_t(p), c_abi_alignof(p->abi, p->pool, t));
+ return cint_make_u64(p, ty_size_t(p),
+ c_abi_alignof(p->abi, p->pool, t));
}
}
}
@@ -962,7 +965,7 @@ static CConstInt cexpr_unary(Parser* p, SrcLoc loc) {
return cint_make_u64(p, ty, (u64)v);
}
if (p->cur.kind == TOK_IDENT) {
- Sym name = p->cur.v.ident;
+ Sym name = tok_ident(&p->cur);
if (name == p->sym_b_offsetof) {
u32 off = 0;
const Type* root;
@@ -1262,7 +1265,7 @@ static int parse_builtin_clear_cache_call(Parser* p, Sym name, SrcLoc loc) {
static MemOrder parse_atomic_mem_order(Parser* p) {
if (p->cur.kind == TOK_NUM) {
- return (MemOrder)eval_const_int(p, p->cur.loc);
+ return (MemOrder)eval_const_int(p, pp_materialize_loc(p->pp, p->cur.loc));
}
parse_assign_expr(p);
to_rvalue(p);
@@ -1310,12 +1313,13 @@ static int offsetof_find_member(Parser* p, const Type* rec_ty, Sym mname,
static const Type* offsetof_designator(Parser* p, const Type* base, u32* off) {
const Type* cur = base;
- if (p->cur.kind != TOK_IDENT || ident_kw_inline(p, p->cur.v.ident) != KW_NONE) {
+ if (p->cur.kind != TOK_IDENT ||
+ ident_kw_inline(p, tok_ident(&p->cur)) != KW_NONE) {
perr(p, "expected member name in __builtin_offsetof");
}
for (;;) {
if (cur->kind == TY_STRUCT || cur->kind == TY_UNION) {
- Sym mname = p->cur.v.ident;
+ Sym mname = tok_ident(&p->cur);
const Type* mty = NULL;
u32 moff = 0;
if (!offsetof_find_member(p, cur, mname, &mty, &moff))
@@ -1330,14 +1334,15 @@ static const Type* offsetof_designator(Parser* p, const Type* base, u32* off) {
}
if (is_punct(&p->cur, '.')) {
advance(p);
- if (p->cur.kind != TOK_IDENT || ident_kw_inline(p, p->cur.v.ident) != KW_NONE) {
+ if (p->cur.kind != TOK_IDENT ||
+ ident_kw_inline(p, tok_ident(&p->cur)) != KW_NONE) {
perr(p, "expected member name after '.'");
}
continue;
}
if (is_punct(&p->cur, '[')) {
advance(p);
- i64 idx = eval_const_int(p, p->cur.loc);
+ i64 idx = eval_const_int(p, pp_materialize_loc(p->pp, p->cur.loc));
expect_punct(p, ']', "']' in __builtin_offsetof");
if (cur->kind != TY_ARRAY) {
perr(p, "__builtin_offsetof '[' on non-array");
@@ -1772,8 +1777,8 @@ static int parse_kit_syscall_call(Parser* p, Sym name, SrcLoc loc) {
}
static int try_parse_builtin_call(Parser* p) {
- Sym name = p->cur.v.ident;
- SrcLoc loc = p->cur.loc;
+ Sym name = tok_ident(&p->cur);
+ SrcLoc loc = pp_materialize_loc(p->pp, p->cur.loc);
if (parse_kit_syscall_call(p, name, loc)) return 1;
@@ -1798,17 +1803,16 @@ static int try_parse_builtin_call(Parser* p) {
name != p->sym_b_trap && name != p->sym_b_unreachable &&
name != p->sym_b_return_address && name != p->sym_b_frame_address &&
name != p->sym_b_readcyclecounter && name != p->sym_b_expect &&
- name != p->sym_b_offsetof &&
- name != p->sym_b_va_start && name != p->sym_b_va_arg &&
- name != p->sym_b_va_end && name != p->sym_b_va_copy &&
- name != p->sym_a_load_n && name != p->sym_a_store_n &&
- name != p->sym_a_exchange_n && name != p->sym_a_fetch_add &&
- name != p->sym_a_fetch_sub && name != p->sym_a_fetch_and &&
- name != p->sym_a_fetch_or && name != p->sym_a_fetch_xor &&
- name != p->sym_a_fetch_nand && name != p->sym_a_cas_n &&
- name != p->sym_a_always_lock_free && name != p->sym_a_is_lock_free &&
- name != p->sym_a_thread_fence && name != p->sym_a_signal_fence &&
- name != p->sym_sync_synchronize) {
+ name != p->sym_b_offsetof && name != p->sym_b_va_start &&
+ name != p->sym_b_va_arg && name != p->sym_b_va_end &&
+ name != p->sym_b_va_copy && name != p->sym_a_load_n &&
+ name != p->sym_a_store_n && name != p->sym_a_exchange_n &&
+ name != p->sym_a_fetch_add && name != p->sym_a_fetch_sub &&
+ name != p->sym_a_fetch_and && name != p->sym_a_fetch_or &&
+ name != p->sym_a_fetch_xor && name != p->sym_a_fetch_nand &&
+ name != p->sym_a_cas_n && name != p->sym_a_always_lock_free &&
+ name != p->sym_a_is_lock_free && name != p->sym_a_thread_fence &&
+ name != p->sym_a_signal_fence && name != p->sym_sync_synchronize) {
return 0;
}
advance(p); /* IDENT */
@@ -1880,7 +1884,7 @@ static int try_parse_builtin_call(Parser* p) {
if (name == p->sym_b_return_address || name == p->sym_b_frame_address) {
/* GCC requires the level to be an integer constant expression. */
int is_return = (name == p->sym_b_return_address);
- i64 level = eval_const_int(p, p->cur.loc);
+ i64 level = eval_const_int(p, pp_materialize_loc(p->pp, p->cur.loc));
expect_punct(p, ')',
"')' after __builtin_return_address/__builtin_frame_address");
if (level < 0)
@@ -1984,7 +1988,8 @@ static int try_parse_builtin_call(Parser* p) {
}
/* __sync_synchronize(): the legacy GCC full barrier. No operands; always a
- * sequentially-consistent fence (the __sync_* family is implicitly seq-cst). */
+ * sequentially-consistent fence (the __sync_* family is implicitly seq-cst).
+ */
if (name == p->sym_sync_synchronize) {
expect_punct(p, ')', "')' after __sync_synchronize");
pcg_set_loc(p, loc);
@@ -1994,7 +1999,7 @@ static int try_parse_builtin_call(Parser* p) {
}
if (name == p->sym_a_always_lock_free || name == p->sym_a_is_lock_free) {
- i64 size = eval_const_int(p, p->cur.loc);
+ i64 size = eval_const_int(p, pp_materialize_loc(p->pp, p->cur.loc));
expect_punct(p, ',', "',' in atomic lock-free builtin");
parse_assign_expr(p);
to_rvalue(p);
@@ -2044,7 +2049,7 @@ static int try_parse_builtin_call(Parser* p) {
coerce_top_to_type(p, val_ty);
expect_punct(p, ',', "',' in __atomic_compare_exchange_n");
- (void)eval_const_int(p, p->cur.loc); /* weak */
+ (void)eval_const_int(p, pp_materialize_loc(p->pp, p->cur.loc)); /* weak */
expect_punct(p, ',', "',' in __atomic_compare_exchange_n");
MemOrder succ = parse_atomic_mem_order(p);
expect_punct(p, ',', "',' in __atomic_compare_exchange_n");
@@ -2157,7 +2162,7 @@ static void parse_primary(Parser* p) {
}
if (t.kind == TOK_IDENT) {
SymEntry* e;
- if (ident_kw_inline(p, t.v.ident) != KW_NONE) {
+ if (ident_kw_inline(p, tok_ident(&t)) != KW_NONE) {
perr(p, "unexpected keyword in expression");
}
{
@@ -2174,15 +2179,16 @@ static void parse_primary(Parser* p) {
* the same value. We synthesize the string lazily — the symbol
* lives in .rodata and the resulting type is `char[N+1]` (with the
* trailing NUL). */
- if (t.v.ident == p->sym_func || t.v.ident == p->sym_func_gcc ||
- t.v.ident == p->sym_pretty_func_gcc) {
+ if (tok_ident(&t) == p->sym_func || tok_ident(&t) == p->sym_func_gcc ||
+ tok_ident(&t) == p->sym_pretty_func_gcc) {
if (p->cur_func_name == 0) {
compiler_panic(
- p->c, t.loc, "'%.*s' used outside a function",
- KIT_SLICE_ARG(kit_slice_cstr(t.v.ident == p->sym_func ? "__func__"
- : t.v.ident == p->sym_func_gcc
- ? "__FUNCTION__"
- : "__PRETTY_FUNCTION__")));
+ p->c, pp_materialize_loc(p->pp, t.loc),
+ "'%.*s' used outside a function",
+ KIT_SLICE_ARG(kit_slice_cstr(
+ tok_ident(&t) == p->sym_func ? "__func__"
+ : tok_ident(&t) == p->sym_func_gcc ? "__FUNCTION__"
+ : "__PRETTY_FUNCTION__")));
}
KitSlice fn_name_sl = kit_sym_str(p->pool->c, p->cur_func_name);
size_t nlen = fn_name_sl.len;
@@ -2199,13 +2205,13 @@ static void parse_primary(Parser* p) {
pcg_push_global(p, sym, arr_ty);
return;
}
- e = scope_lookup(p, t.v.ident);
+ e = scope_lookup(p, tok_ident(&t));
if (!e) {
- KitSlice ident_sl = kit_sym_str(p->pool->c, t.v.ident);
+ KitSlice ident_sl = kit_sym_str(p->pool->c, tok_ident(&t));
size_t nlen = ident_sl.len;
const char* nm = ident_sl.s;
- compiler_panic(p->c, t.loc, "undeclared identifier '%.*s'", (int)nlen,
- nm ? nm : "?");
+ compiler_panic(p->c, pp_materialize_loc(p->pp, t.loc),
+ "undeclared identifier '%.*s'", (int)nlen, nm ? nm : "?");
}
advance(p);
switch (e->kind) {
@@ -2440,10 +2446,11 @@ static void parse_postfix(Parser* p) {
perr(p,
"request for member in something that is not a struct or union");
}
- if (p->cur.kind != TOK_IDENT || ident_kw_inline(p, p->cur.v.ident) != KW_NONE) {
+ if (p->cur.kind != TOK_IDENT ||
+ ident_kw_inline(p, tok_ident(&p->cur)) != KW_NONE) {
perr(p, "expected member name after '.'");
}
- mname = p->cur.v.ident;
+ mname = tok_ident(&p->cur);
advance(p);
lt = type_unqual(p->pool, lt);
if (!find_record_member_path(p, lt, mname, &mty, &off, &bf_off, &bf_w,
@@ -2470,10 +2477,11 @@ static void parse_postfix(Parser* p) {
if (!rec_ty || (rec_ty->kind != TY_STRUCT && rec_ty->kind != TY_UNION)) {
perr(p, "'->' on pointer to non-struct/union");
}
- if (p->cur.kind != TOK_IDENT || ident_kw_inline(p, p->cur.v.ident) != KW_NONE) {
+ if (p->cur.kind != TOK_IDENT ||
+ ident_kw_inline(p, tok_ident(&p->cur)) != KW_NONE) {
perr(p, "expected member name after '->'");
}
- mname = p->cur.v.ident;
+ mname = tok_ident(&p->cur);
advance(p);
if (!find_record_member_path(p, rec_ty, mname, &mty, &off, &bf_off, &bf_w,
&bf_ss))
@@ -2572,11 +2580,12 @@ void parse_unary(Parser* p) {
Sym name;
SrcLoc loc;
advance(p); /* '&&' */
- if (p->cur.kind != TOK_IDENT || ident_kw_inline(p, p->cur.v.ident) != KW_NONE) {
+ if (p->cur.kind != TOK_IDENT ||
+ ident_kw_inline(p, tok_ident(&p->cur)) != KW_NONE) {
perr(p, "expected label name after '&&'");
}
- name = p->cur.v.ident;
- loc = p->cur.loc;
+ name = tok_ident(&p->cur);
+ loc = pp_materialize_loc(p->pp, p->cur.loc);
advance(p);
pcg_push_label_addr(p, take_label_addr(p, name, loc));
return;
@@ -2760,7 +2769,7 @@ void parse_unary(Parser* p) {
}
memset(&buf[len], 0, sizeof(Tok));
buf[len].kind = TOK_PUNCT;
- buf[len].v.punct = ',';
+ buf[len].aux = ',';
++len;
default_buf = buf;
default_len = len;
@@ -3076,8 +3085,7 @@ static void parse_shift(Parser* p) {
const Type* lp = cint_promote_type(p, lt);
if (!type_is_int(lt)) perr(p, "shift operator requires integer operands");
if (pcg_top_type(p) != lp) pcg_convert(p, lp);
- if (bop == BO_SHR_S && !type_is_signed_integer(lp))
- bop = BO_SHR_U;
+ if (bop == BO_SHR_S && !type_is_signed_integer(lp)) bop = BO_SHR_U;
}
parse_add(p);
to_rvalue(p);
@@ -3490,8 +3498,7 @@ void parse_assign_expr(Parser* p) {
}
advance(p);
const Type* lhs = pcg_top_type(p);
- if (compound == BO_SHR_S && !type_is_signed_integer(lhs))
- compound = BO_SHR_U;
+ if (compound == BO_SHR_S && !type_is_signed_integer(lhs)) compound = BO_SHR_U;
{
if (lhs && (lhs->qual & Q_CONST)) {
perr(p, "assignment to const-qualified object");
diff --git a/lang/c/parse/parse_init.c b/lang/c/parse/parse_init.c
@@ -16,7 +16,9 @@
* File-local helpers
* ============================================================ */
-static SrcLoc tok_loc_init(const Tok* t) { return t->loc; }
+static SrcLoc tok_loc_init(Parser* p, const Tok* t) {
+ return pp_materialize_loc(p->pp, t->loc);
+}
static const Type* init_field_type_at(const Type* ty, u16 i) {
const Field* f = &ty->rec.fields[i];
@@ -384,7 +386,7 @@ static void parse_designator_chain(Parser* p, const Type* outer_ty,
if (is_punct(&p->cur, '[')) {
i64 idx;
u32 esz;
- SrcLoc cloc = tok_loc_init(&p->cur);
+ SrcLoc cloc = tok_loc_init(p, &p->cur);
const Type* parent_ty = cur_ty;
u32 parent_off = cur_off;
advance(p);
@@ -416,10 +418,10 @@ static void parse_designator_chain(Parser* p, const Type* outer_ty,
u32 parent_off = cur_off;
advance(p);
if (p->cur.kind != TOK_IDENT ||
- ident_kw_inline(p, p->cur.v.ident) != KW_NONE) {
+ ident_kw_inline(p, tok_ident(&p->cur)) != KW_NONE) {
perr(p, "expected field name after '.'");
}
- fname = p->cur.v.ident;
+ fname = tok_ident(&p->cur);
advance(p);
if (!cur_ty || (cur_ty->kind != TY_STRUCT && cur_ty->kind != TY_UNION)) {
perr(p, "field designator on non-record type");
@@ -1183,18 +1185,20 @@ static int try_parse_static_address_const(Parser* p, CStaticConst* out) {
advance(p);
lit_ty = parse_type_name(p);
expect_punct(p, ')', "')' after compound literal type-name");
- *out = parse_static_compound_literal_after_type(p, lit_ty, t.loc);
+ *out = parse_static_compound_literal_after_type(
+ p, lit_ty, pp_materialize_loc(p->pp, t.loc));
return 1;
}
}
if (p->cur.kind != TOK_IDENT ||
- ident_kw_inline(p, p->cur.v.ident) != KW_NONE) {
+ ident_kw_inline(p, tok_ident(&p->cur)) != KW_NONE) {
perr(p, "expected identifier after '&' in static initializer");
}
- name = p->cur.v.ident;
+ name = tok_ident(&p->cur);
advance(p);
- } else if (t.kind == TOK_IDENT && ident_kw_inline(p, t.v.ident) == KW_NONE) {
- name = t.v.ident;
+ } else if (t.kind == TOK_IDENT &&
+ ident_kw_inline(p, tok_ident(&t)) == KW_NONE) {
+ name = tok_ident(&t);
advance(p);
} else {
return 0;
@@ -1208,7 +1212,7 @@ static int try_parse_static_address_const(Parser* p, CStaticConst* out) {
if (saw_amp && is_punct(&p->cur, '[')) {
SrcLoc cloc;
advance(p);
- cloc = tok_loc_init(&p->cur);
+ cloc = tok_loc_init(p, &p->cur);
element_addend = eval_const_int(p, cloc);
expect_punct(p, ']', "']' after array-subscript constant");
if (tgt_ty && tgt_ty->kind == TY_ARRAY) {
@@ -1223,13 +1227,14 @@ static int try_parse_static_address_const(Parser* p, CStaticConst* out) {
SrcLoc cloc;
i64 v;
advance(p);
- cloc = tok_loc_init(&p->cur);
+ cloc = tok_loc_init(p, &p->cur);
v = eval_const_int(p, cloc);
if (neg) v = -v;
if (tgt_ty && tgt_ty->kind == TY_ARRAY) {
byte_addend += v * (i64)c_abi_sizeof(p->abi, p->pool, tgt_ty->arr.elem);
} else if (tgt_ty && tgt_ty->kind == TY_PTR) {
- byte_addend += v * (i64)c_abi_sizeof(p->abi, p->pool, tgt_ty->ptr.pointee);
+ byte_addend +=
+ v * (i64)c_abi_sizeof(p->abi, p->pool, tgt_ty->ptr.pointee);
} else if (saw_amp) {
byte_addend += v * (i64)c_abi_sizeof(p->abi, p->pool, tgt_ty);
} else {
@@ -1254,11 +1259,11 @@ static CStaticConst parse_static_const(Parser* p, const Type* ty, SrcLoc loc) {
SrcLoc lloc;
advance(p); /* '&&' */
if (p->cur.kind != TOK_IDENT ||
- ident_kw_inline(p, p->cur.v.ident) != KW_NONE) {
+ ident_kw_inline(p, tok_ident(&p->cur)) != KW_NONE) {
perr(p, "expected label name after '&&' in static initializer");
}
- lname = p->cur.v.ident;
- lloc = tok_loc_init(&p->cur);
+ lname = tok_ident(&p->cur);
+ lloc = tok_loc_init(p, &p->cur);
advance(p);
r.kind = C_STATIC_CONST_LABEL_ADDR;
r.label = take_label_addr(p, lname, lloc);
@@ -1303,7 +1308,7 @@ static CStaticConst parse_static_const(Parser* p, const Type* ty, SrcLoc loc) {
if (cast_unqual->kind == TY_PTR &&
(p->cur.kind == TOK_STR || is_punct(&p->cur, '&') ||
(p->cur.kind == TOK_IDENT &&
- ident_kw_inline(p, p->cur.v.ident) == KW_NONE)) &&
+ ident_kw_inline(p, tok_ident(&p->cur)) == KW_NONE)) &&
try_parse_static_address_const(p, &r)) {
return r;
}
@@ -1336,7 +1341,7 @@ static CStaticConst parse_static_const(Parser* p, const Type* ty, SrcLoc loc) {
static void parse_static_bitfield_at(Parser* p, u8* buf, u32 buflen,
u32 rec_offset, const ABIFieldLayout* fl,
const Type* field_ty) {
- SrcLoc cloc = tok_loc_init(&p->cur);
+ SrcLoc cloc = tok_loc_init(p, &p->cur);
CStaticConst parsed = parse_static_const(p, field_ty, cloc);
u32 storage_off = rec_offset + fl->offset;
u32 storage_size = fl->storage_size;
@@ -1564,7 +1569,7 @@ void parse_static_init_at(Parser* p, u8* buf, u32 buflen, u32 offset,
/* Scalar / pointer. */
{
int had_brace = accept_punct(p, '{');
- SrcLoc cloc = tok_loc_init(&p->cur);
+ SrcLoc cloc = tok_loc_init(p, &p->cur);
u32 sz = c_abi_sizeof(p->abi, p->pool, ty);
CStaticConst cv;
if (offset + sz > buflen) perr(p, "initializer overflows object");
diff --git a/lang/c/parse/parse_priv.h b/lang/c/parse/parse_priv.h
@@ -310,7 +310,7 @@ typedef struct Parser {
const Type* cur_func_ret;
Sym sym_b_expect;
Sym sym_b_offsetof;
- Sym sym_b_constant_p; /* __builtin_constant_p */
+ Sym sym_b_constant_p; /* __builtin_constant_p */
Sym sym_b_va_list;
/* Cached singleton for __builtin_va_list — built lazily on first
* mention so every occurrence resolves to the same Type* (and the
@@ -323,10 +323,10 @@ typedef struct Parser {
Sym sym_b_va_arg;
Sym sym_b_va_end;
Sym sym_b_va_copy;
- Sym sym_b_return_address; /* __builtin_return_address */
- Sym sym_b_frame_address; /* __builtin_frame_address */
- Sym sym_b_readcyclecounter; /* __builtin_readcyclecounter */
- Sym sym_kit_syscall[7]; /* __kit_syscall0 .. __kit_syscall6 */
+ Sym sym_b_return_address; /* __builtin_return_address */
+ Sym sym_b_frame_address; /* __builtin_frame_address */
+ Sym sym_b_readcyclecounter; /* __builtin_readcyclecounter */
+ Sym sym_kit_syscall[7]; /* __kit_syscall0 .. __kit_syscall6 */
Sym sym_attribute;
Sym sym_volatile_alias;
Sym sym_alignof_alias;
@@ -353,7 +353,7 @@ typedef struct Parser {
Sym sym_a_is_lock_free;
Sym sym_a_thread_fence;
Sym sym_a_signal_fence;
- Sym sym_sync_synchronize; /* __sync_synchronize (legacy full barrier) */
+ Sym sym_sync_synchronize; /* __sync_synchronize (legacy full barrier) */
Scope* scope;
/* Sym -> innermost-visible binding cache; the O(1) read side of scope_lookup.
@@ -469,18 +469,18 @@ TagEntry* tag_lookup_local(Parser* p, Sym name);
* ============================================================ */
static inline int is_punct(const Tok* t, u32 punct) {
- return t->kind == TOK_PUNCT && t->v.punct == punct;
+ return t->kind == TOK_PUNCT && tok_punct(t) == punct;
}
static inline int is_pp_hash(const Tok* t) { return t->kind == TOK_PP_HASH; }
/* THE keyword classifier — the single canonical way keywordness is decided.
* Interned Sym -> CKw via kw_map; KW_NONE if the Sym is not a keyword. The map
- * holds the canonical keyword spellings AND the GNU alias spellings (`__inline__`
- * etc.), each mapping to its CKw, registered once in the kw_map population in
- * parse_c — so aliases are classified identically to their canonical keyword,
- * with no separate "alias-aware" path. Everything below (and the per-token
- * classify_kw / is_kw adapters) routes through this. */
+ * holds the canonical keyword spellings AND the GNU alias spellings
+ * (`__inline__` etc.), each mapping to its CKw, registered once in the kw_map
+ * population in parse_c — so aliases are classified identically to their
+ * canonical keyword, with no separate "alias-aware" path. Everything below (and
+ * the per-token classify_kw / is_kw adapters) routes through this. */
static inline CKw ident_kw_inline(const Parser* p, Sym name) {
return name ? (CKw)KwTab_get(&p->kw_map, name) : KW_NONE;
}
@@ -488,7 +488,7 @@ static inline CKw ident_kw_inline(const Parser* p, Sym name) {
/* A token's keyword identity (KW_NONE for a non-identifier or non-keyword). The
* one place to classify a Tok: classify once, then compare CKw. */
static inline CKw classify_kw(const Parser* p, const Tok* t) {
- return t->kind == TOK_IDENT ? ident_kw_inline(p, t->v.ident) : KW_NONE;
+ return t->kind == TOK_IDENT ? ident_kw_inline(p, tok_ident(t)) : KW_NONE;
}
/* Is token t the keyword k? Thin boolean shape-adapter over classify_kw. */
diff --git a/lang/c/parse/parse_stmt.c b/lang/c/parse/parse_stmt.c
@@ -11,7 +11,9 @@
* File-local helpers
* ============================================================ */
-static SrcLoc tok_loc_stmt(const Tok* t) { return t->loc; }
+static SrcLoc tok_loc_stmt(Parser* p, const Tok* t) {
+ return pp_materialize_loc(p->pp, t->loc);
+}
static int accept_kw_stmt(Parser* p, CKw k) {
if (!is_kw(p, &p->cur, k)) return 0;
@@ -255,8 +257,9 @@ GotoLabel* label_get_or_create(Parser* p, Sym name, SrcLoc loc) {
* `LABEL:` placement and its `goto LABEL` references can straddle a
* constant-false (suppressed) region. Key allocation off whether the function
* emits at all, not the transient suppress depth — otherwise a label first
- * mentioned inside a suppressed `goto` would cache pcg_label_new's suppression
- * sentinel and later alias the function's first real label ("placed twice"). */
+ * mentioned inside a suppressed `goto` would cache pcg_label_new's
+ * suppression sentinel and later alias the function's first real label
+ * ("placed twice"). */
gl->label = p->cur_func_emits ? kit_cg_label_new(p->cg) : pcg_label_new(p);
gl->placed = 0;
gl->first_use = loc;
@@ -321,11 +324,12 @@ static void parse_goto_stmt(Parser* p) {
parse_computed_goto(p);
return;
}
- if (p->cur.kind != TOK_IDENT || ident_kw_inline(p, p->cur.v.ident) != KW_NONE) {
+ if (p->cur.kind != TOK_IDENT ||
+ ident_kw_inline(p, tok_ident(&p->cur)) != KW_NONE) {
perr(p, "expected label name after 'goto'");
}
- name = p->cur.v.ident;
- loc = tok_loc_stmt(&p->cur);
+ name = tok_ident(&p->cur);
+ loc = tok_loc_stmt(p, &p->cur);
advance(p);
expect_punct(p, ';', "';' after goto");
gl = label_get_or_create(p, name, loc);
@@ -340,8 +344,8 @@ static void parse_goto_stmt(Parser* p) {
}
static void parse_label_stmt(Parser* p) {
- Sym name = p->cur.v.ident;
- SrcLoc loc = tok_loc_stmt(&p->cur);
+ Sym name = tok_ident(&p->cur);
+ SrcLoc loc = tok_loc_stmt(p, &p->cur);
GotoLabel* gl;
advance(p); /* IDENT */
advance(p); /* ':' */
@@ -360,7 +364,7 @@ static void parse_case_stmt(Parser* p) {
i64 v;
CGLabel L;
CaseEntry* ce;
- SrcLoc loc = tok_loc_stmt(&p->cur);
+ SrcLoc loc = tok_loc_stmt(p, &p->cur);
if (!p->cur_switch) perr(p, "'case' label not in switch statement");
v = eval_const_int(p, loc);
for (ce = p->cur_switch->cases; ce; ce = ce->next) {
@@ -529,13 +533,13 @@ static void parse_switch_stmt(Parser* p) {
}
void parse_static_assert(Parser* p) {
- SrcLoc loc = tok_loc_stmt(&p->cur);
+ SrcLoc loc = tok_loc_stmt(p, &p->cur);
i64 v;
if (!accept_kw_stmt(p, KW_STATIC_ASSERT)) {
perr(p, "expected _Static_assert");
}
expect_punct(p, '(', "'(' after _Static_assert");
- v = eval_const_int(p, tok_loc_stmt(&p->cur));
+ v = eval_const_int(p, tok_loc_stmt(p, &p->cur));
expect_punct(p, ',', "',' separating _Static_assert args");
if (p->cur.kind != TOK_STR) {
perr(p, "expected string literal as _Static_assert message");
@@ -546,7 +550,7 @@ void parse_static_assert(Parser* p) {
expect_punct(p, ')', "')' after _Static_assert");
expect_punct(p, ';', "';' after _Static_assert");
if (!v) {
- KitSlice msg_sl = kit_sym_str(p->pool->c, msg.spelling);
+ KitSlice msg_sl = pp_text_slice(p->pp, &msg);
size_t mlen = msg_sl.len;
const char* mstr = msg_sl.s;
compiler_panic(p->c, loc, "static assertion failed: %.*s", (int)mlen,
@@ -588,7 +592,7 @@ static Sym parse_asm_operand_name(Parser* p) {
if (p->cur.kind != TOK_IDENT) {
perr(p, "expected identifier inside '[name]' on asm operand");
}
- name = p->cur.v.ident;
+ name = tok_ident(&p->cur);
advance(p);
expect_punct(p, ']', "']' after asm operand name");
return name;
@@ -629,7 +633,7 @@ static Sym asm_operand_pinned_reg(Parser* p, FrameSlot* slot_out) {
if (p->cur.kind != TOK_IDENT) return 0;
nxt = peek1(p);
if (!is_punct(&nxt, ')')) return 0;
- e = scope_lookup(p, p->cur.v.ident);
+ e = scope_lookup(p, tok_ident(&p->cur));
if (!e || e->kind != SEK_LOCAL) return 0;
if (e->reg_asm_name && slot_out) *slot_out = e->v.slot;
return e->reg_asm_name;
@@ -644,10 +648,11 @@ static void parse_asm_stmt(Parser* p) {
u32 nout = 0, nin = 0, nclob = 0;
u32 cap_out = 0, cap_in = 0, cap_clob = 0;
int saw_goto = 0;
- SrcLoc loc = tok_loc_stmt(&p->cur);
+ SrcLoc loc = tok_loc_stmt(p, &p->cur);
for (;;) {
- if (accept_kw_stmt(p, KW_VOLATILE)) continue; /* `volatile` or `__volatile__` */
+ if (accept_kw_stmt(p, KW_VOLATILE))
+ continue; /* `volatile` or `__volatile__` */
break;
}
if (accept_kw_stmt(p, KW_GOTO)) saw_goto = 1;
@@ -878,8 +883,9 @@ void parse_compound_stmt(Parser* p) {
}
void parse_stmt(Parser* p) {
- pcg_set_loc(p, tok_loc_stmt(&p->cur));
- if (p->cur.kind == TOK_IDENT && ident_kw_inline(p, p->cur.v.ident) == KW_NONE) {
+ pcg_set_loc(p, tok_loc_stmt(p, &p->cur));
+ if (p->cur.kind == TOK_IDENT &&
+ ident_kw_inline(p, tok_ident(&p->cur)) == KW_NONE) {
Tok n = peek1(p);
if (is_punct(&n, ':')) {
parse_label_stmt(p);
diff --git a/lang/c/parse/parse_type.c b/lang/c/parse/parse_type.c
@@ -68,7 +68,9 @@ static const struct {
{"import_name", ATTR_IMPORT_NAME, AS_STRING},
};
-static SrcLoc tok_loc(const Tok* t) { return t->loc; }
+static SrcLoc tok_loc(Parser* p, const Tok* t) {
+ return pp_materialize_loc(p->pp, t->loc);
+}
static void attr_canon_range(const char* s, size_t len, const char** out_p,
size_t* out_len);
@@ -98,8 +100,7 @@ static const Type* attrs_apply_type_mode(Parser* p, const Type* base,
if (a->kind != ATTR_MODE || a->nargs == 0) continue;
if (attr_sym_canon_eq(p, a->v.sym, "TI")) {
const Type* u = type_unqual(p->pool, base);
- int is_unsigned =
- u && type_is_int(u) && type_is_signed_integer(u) == 0;
+ int is_unsigned = u && type_is_int(u) && type_is_signed_integer(u) == 0;
if (!target_has_int128(p)) {
perr(p, "__int128 is not supported on the target architecture");
}
@@ -110,7 +111,7 @@ static const Type* attrs_apply_type_mode(Parser* p, const Type* base,
}
int starts_attr(const Parser* p) {
- return p->cur.kind == TOK_IDENT && p->cur.v.ident == p->sym_attribute;
+ return p->cur.kind == TOK_IDENT && tok_ident(&p->cur) == p->sym_attribute;
}
static int starts_asm_label(const Parser* p) {
@@ -270,7 +271,7 @@ static void parse_attr_args(Parser* p, Attr* a, AttrArgShape shape,
advance(p);
return;
}
- loc = tok_loc(&p->cur);
+ loc = tok_loc(p, &p->cur);
a->v.i = eval_const_int(p, loc);
if (a->kind == ATTR_ALIGNED && a->v.i > 0 &&
(((u64)a->v.i & ((u64)a->v.i - 1u)) != 0)) {
@@ -307,7 +308,7 @@ static void parse_attr_args(Parser* p, Attr* a, AttrArgShape shape,
perr(p, "attribute '%.*s' expects an identifier",
KIT_SLICE_ARG(kit_slice_cstr(attr_diag_name)));
}
- a->v.sym = p->cur.v.ident;
+ a->v.sym = tok_ident(&p->cur);
a->nargs = 1;
advance(p);
expect_punct(p, ')', "')' after attribute identifier argument");
@@ -322,10 +323,10 @@ static void parse_attr_args(Parser* p, Attr* a, AttrArgShape shape,
}
advance(p);
expect_punct(p, ',', "',' after format archetype");
- mloc = tok_loc(&p->cur);
+ mloc = tok_loc(p, &p->cur);
mv = eval_const_int(p, mloc);
expect_punct(p, ',', "',' after format string-index");
- nloc = tok_loc(&p->cur);
+ nloc = tok_loc(p, &p->cur);
nv = eval_const_int(p, nloc);
if (mv < 0 || mv > 0xFFFF || nv < 0 || nv > 0xFFFF) {
perr(p, "attribute 'format' indices out of range");
@@ -348,7 +349,7 @@ Attr* parse_attribute_spec_list(Parser* p) {
Attr* head = NULL;
Attr* tail = NULL;
while (starts_attr(p)) {
- SrcLoc kw_loc = tok_loc(&p->cur);
+ SrcLoc kw_loc = tok_loc(p, &p->cur);
advance(p); /* __attribute__ */
expect_punct(p, '(', "'(' after __attribute__");
expect_punct(p, '(', "'((' after __attribute__");
@@ -366,11 +367,11 @@ Attr* parse_attribute_spec_list(Parser* p) {
if (p->cur.kind != TOK_IDENT) {
perr(p, "expected attribute name");
}
- aname = p->cur.v.ident;
+ aname = tok_ident(&p->cur);
a = arena_new(p->pool->arena, Attr);
if (!a) perr(p, "out of memory in parse_attribute_spec_list");
memset(a, 0, sizeof *a);
- a->loc = tok_loc(&p->cur);
+ a->loc = tok_loc(p, &p->cur);
a->name = aname;
a->kind = (u16)classify_attr(p, aname, &shape);
advance(p);
@@ -607,7 +608,7 @@ int parse_decl_specs(Parser* p, DeclSpecs* out) {
out->vla_byte_slot = FRAME_SLOT_NONE;
out->vla_bounds = NULL;
out->attrs = NULL;
- loc = tok_loc(&p->cur);
+ loc = tok_loc(p, &p->cur);
for (;;) {
Tok t = p->cur;
/* Classify the token's keyword identity exactly once per iteration; the
@@ -704,18 +705,18 @@ int parse_decl_specs(Parser* p, DeclSpecs* out) {
acc.saw_explicit_type = 1;
advance(p);
seen = 1;
- } else if (t.kind == TOK_IDENT && t.v.ident == p->sym_int128) {
+ } else if (t.kind == TOK_IDENT && tok_ident(&t) == p->sym_int128) {
acc.saw_int128 = 1;
acc.saw_explicit_type = 1;
advance(p);
seen = 1;
- } else if (t.kind == TOK_IDENT && t.v.ident == p->sym_int128_t) {
+ } else if (t.kind == TOK_IDENT && tok_ident(&t) == p->sym_int128_t) {
acc.saw_int128 = 1;
acc.saw_signed = 1;
acc.saw_explicit_type = 1;
advance(p);
seen = 1;
- } else if (t.kind == TOK_IDENT && t.v.ident == p->sym_uint128_t) {
+ } else if (t.kind == TOK_IDENT && tok_ident(&t) == p->sym_uint128_t) {
acc.saw_int128 = 1;
acc.saw_unsigned = 1;
acc.saw_explicit_type = 1;
@@ -782,7 +783,7 @@ int parse_decl_specs(Parser* p, DeclSpecs* out) {
const Type* tn = parse_type_name(p);
a = c_abi_alignof(p->abi, p->pool, tn);
} else {
- i64 v = eval_const_int(p, tok_loc(&p->cur));
+ i64 v = eval_const_int(p, tok_loc(p, &p->cur));
if (v < 0) perr(p, "_Alignas requires a non-negative alignment");
a = (u32)v;
}
@@ -821,8 +822,9 @@ int parse_decl_specs(Parser* p, DeclSpecs* out) {
} else if (!acc.saw_explicit_type && !tagged_ty && t.kind == TOK_IDENT &&
tkw == KW_NONE) {
/* tkw == KW_NONE here is exactly ident_kw_inline(t)==KW_NONE: this is the
- * terminal else-if, so every keyword/alias branch above already failed. */
- if (t.v.ident == p->sym_b_va_list) {
+ * terminal else-if, so every keyword/alias branch above already failed.
+ */
+ if (tok_ident(&t) == p->sym_b_va_list) {
if (!p->type_b_va_list)
p->type_b_va_list = c_abi_va_list_type(p->abi, p->pool);
tagged_ty = p->type_b_va_list;
@@ -831,7 +833,7 @@ int parse_decl_specs(Parser* p, DeclSpecs* out) {
seen = 1;
continue;
}
- SymEntry* e = scope_lookup(p, t.v.ident);
+ SymEntry* e = scope_lookup(p, tok_ident(&t));
if (e && e->kind == SEK_TYPEDEF) {
tagged_ty = e->type;
if (e->vla_byte_slot != FRAME_SLOT_NONE) {
@@ -975,7 +977,7 @@ static void parse_member_decls(Parser* p, TypeRecordBuilder* b) {
}
for (;;) {
Sym mname = 0;
- SrcLoc mloc = tok_loc(&p->cur);
+ SrcLoc mloc = tok_loc(p, &p->cur);
const Type* mty;
Field f;
memset(&f, 0, sizeof f);
@@ -1044,9 +1046,10 @@ const Type* parse_struct_or_union(Parser* p, TypeKind kind,
TagDeclKind tdk = (kind == TY_STRUCT) ? TAG_STRUCT : TAG_UNION;
Attr* rec_attrs = NULL;
parse_attrs_into(p, &rec_attrs);
- tag_loc = tok_loc(&p->cur);
- if (p->cur.kind == TOK_IDENT && ident_kw_inline(p, p->cur.v.ident) == KW_NONE) {
- tag_name = p->cur.v.ident;
+ tag_loc = tok_loc(p, &p->cur);
+ if (p->cur.kind == TOK_IDENT &&
+ ident_kw_inline(p, tok_ident(&p->cur)) == KW_NONE) {
+ tag_name = tok_ident(&p->cur);
advance(p);
}
int has_body = is_punct(&p->cur, '{');
@@ -1138,9 +1141,10 @@ const Type* parse_enum(Parser* p, Attr** anon_attrs_out) {
SrcLoc tag_loc;
Attr* rec_attrs = NULL;
parse_attrs_into(p, &rec_attrs);
- tag_loc = tok_loc(&p->cur);
- if (p->cur.kind == TOK_IDENT && ident_kw_inline(p, p->cur.v.ident) == KW_NONE) {
- tag_name = p->cur.v.ident;
+ tag_loc = tok_loc(p, &p->cur);
+ if (p->cur.kind == TOK_IDENT &&
+ ident_kw_inline(p, tok_ident(&p->cur)) == KW_NONE) {
+ tag_name = tok_ident(&p->cur);
advance(p);
}
/* C23 §6.7.2.2: an optional fixed underlying type — `enum [tag] : T` —
@@ -1188,12 +1192,13 @@ const Type* parse_enum(Parser* p, Attr** anon_attrs_out) {
i64 next_val = 0;
for (;;) {
Sym name;
- SrcLoc nloc = tok_loc(&p->cur);
+ SrcLoc nloc = tok_loc(p, &p->cur);
SymEntry* e;
- if (p->cur.kind != TOK_IDENT || ident_kw_inline(p, p->cur.v.ident) != KW_NONE) {
+ if (p->cur.kind != TOK_IDENT ||
+ ident_kw_inline(p, tok_ident(&p->cur)) != KW_NONE) {
perr(p, "expected enumerator name");
}
- name = p->cur.v.ident;
+ name = tok_ident(&p->cur);
advance(p);
i64 val = next_val;
if (accept_punct(p, '=')) {
@@ -1238,7 +1243,7 @@ const Type* parse_enum(Parser* p, Attr** anon_attrs_out) {
int starts_type_name(const Parser* p, const Tok* t) {
if (t->kind != TOK_IDENT) return 0;
- CKw k = ident_kw_inline(p, t->v.ident);
+ CKw k = ident_kw_inline(p, tok_ident(t));
switch (k) {
case KW_VOID:
case KW_CHAR:
@@ -1269,11 +1274,11 @@ int starts_type_name(const Parser* p, const Tok* t) {
case KW_THREAD_LOCAL:
return 1;
case KW_NONE: {
- if (t->v.ident == p->sym_b_va_list) return 1;
- if (t->v.ident == p->sym_int128 || t->v.ident == p->sym_int128_t ||
- t->v.ident == p->sym_uint128_t)
+ if (tok_ident(t) == p->sym_b_va_list) return 1;
+ if (tok_ident(t) == p->sym_int128 || tok_ident(t) == p->sym_int128_t ||
+ tok_ident(t) == p->sym_uint128_t)
return 1;
- SymEntry* e = scope_lookup((Parser*)p, t->v.ident);
+ SymEntry* e = scope_lookup((Parser*)p, tok_ident(t));
return e && e->kind == SEK_TYPEDEF;
}
default:
@@ -1376,7 +1381,7 @@ static void parse_param_array_bound(Parser* p, DeclSuffix* out) {
has_expr = 1;
advance(p);
}
- if (ntoks == 1 && toks[0].kind == TOK_PUNCT && toks[0].v.punct == '*') {
+ if (ntoks == 1 && toks[0].kind == TOK_PUNCT && tok_punct(&toks[0]) == '*') {
has_expr = 0;
}
if (ntoks == 1 && toks[0].kind == TOK_NUM) {
@@ -1431,15 +1436,15 @@ int parse_decl_suffix(Parser* p, DeclSuffix* out) {
perr(p, "array bound requires integer type");
}
if (!is_const_start && t.kind == TOK_IDENT) {
- SymEntry* e = scope_lookup(p, t.v.ident);
+ SymEntry* e = scope_lookup(p, tok_ident(&t));
if (e && e->kind == SEK_ENUM_CST) is_const_start = 1;
if (!is_const_start) {
- CKw k = ident_kw_inline(p, t.v.ident);
+ CKw k = ident_kw_inline(p, tok_ident(&t));
if (k == KW_SIZEOF || k == KW_ALIGNOF) is_const_start = 1;
}
}
if (is_const_start) {
- SrcLoc cloc = tok_loc(&p->cur);
+ SrcLoc cloc = tok_loc(p, &p->cur);
i64 v = eval_const_int(p, cloc);
if (v < 0) perr(p, "negative array size");
out->count = (u32)v;
@@ -1554,8 +1559,9 @@ const Type* parse_declarator_full_info(Parser* p, const Type* base,
int is_inner = 0;
if (is_punct(&n, '*')) {
is_inner = 1;
- } else if (n.kind == TOK_IDENT && ident_kw_inline(p, n.v.ident) == KW_NONE) {
- SymEntry* e = scope_lookup(p, n.v.ident);
+ } else if (n.kind == TOK_IDENT &&
+ ident_kw_inline(p, tok_ident(&n)) == KW_NONE) {
+ SymEntry* e = scope_lookup(p, tok_ident(&n));
if (!(e && e->kind == SEK_TYPEDEF)) is_inner = 1;
}
if (is_inner) {
@@ -1589,9 +1595,10 @@ const Type* parse_declarator_full_info(Parser* p, const Type* base,
}
inner_quals[nptrs_inner++] = q;
}
- if (p->cur.kind == TOK_IDENT && ident_kw_inline(p, p->cur.v.ident) == KW_NONE) {
- name = p->cur.v.ident;
- nloc = tok_loc(&p->cur);
+ if (p->cur.kind == TOK_IDENT &&
+ ident_kw_inline(p, tok_ident(&p->cur)) == KW_NONE) {
+ name = tok_ident(&p->cur);
+ nloc = tok_loc(p, &p->cur);
advance(p);
} else if (is_punct(&p->cur, '(')) {
Tok nn = peek1(p);
@@ -1629,9 +1636,9 @@ const Type* parse_declarator_full_info(Parser* p, const Type* base,
nested_quals[nptrs_nested++] = q;
}
if (p->cur.kind == TOK_IDENT &&
- ident_kw_inline(p, p->cur.v.ident) == KW_NONE) {
- name = p->cur.v.ident;
- nloc = tok_loc(&p->cur);
+ ident_kw_inline(p, tok_ident(&p->cur)) == KW_NONE) {
+ name = tok_ident(&p->cur);
+ nloc = tok_loc(p, &p->cur);
advance(p);
} else if (!allow_abstract) {
perr(p, "expected declarator name");
@@ -1653,9 +1660,10 @@ const Type* parse_declarator_full_info(Parser* p, const Type* base,
}
if (!has_inner_parens) {
- if (p->cur.kind == TOK_IDENT && ident_kw_inline(p, p->cur.v.ident) == KW_NONE) {
- name = p->cur.v.ident;
- nloc = tok_loc(&p->cur);
+ if (p->cur.kind == TOK_IDENT &&
+ ident_kw_inline(p, tok_ident(&p->cur)) == KW_NONE) {
+ name = tok_ident(&p->cur);
+ nloc = tok_loc(p, &p->cur);
advance(p);
} else if (!allow_abstract) {
perr(p, "expected declarator name");
diff --git a/lang/cpp/cpp.c b/lang/cpp/cpp.c
@@ -52,8 +52,8 @@ typedef struct CppRun {
static KitStatus cpp_preprocess_body(KitCompiler* c, void* user) {
CppRun* r = (CppRun*)user;
- Lexer* lex;
Pp* pp;
+ SourceSpec spec;
const KitPreprocessOptions* opts = r->opts;
const KitSlice* input = r->input;
@@ -67,13 +67,16 @@ static KitStatus cpp_preprocess_body(KitCompiler* c, void* user) {
cpp_bad_options(c, "input data is NULL but len > 0");
}
- lex = lex_open_mem(c, r->name.s, input->s, input->len);
- if (lex) lex_skip_shebang(lex);
pp = pp_new(c);
- if (!lex || !pp)
- compiler_panic(c, cpp_no_loc(), "C preprocessor out of memory");
+ if (!pp) compiler_panic(c, cpp_no_loc(), "C preprocessor out of memory");
cpp_apply_options(pp, opts);
- pp_push_input(pp, lex);
+ /* -E / cpp: NOT parser-feed — newlines surface for text reconstruction. */
+ memset(&spec, 0, sizeof(spec));
+ spec.name = kit_slice_cstr(r->name.s);
+ spec.bytes = input->s;
+ spec.len = (u32)input->len;
+ spec.flags = SRC_PRIMARY;
+ pp_push_source(pp, &spec);
pp_emit_text(pp, out);
pp_free(pp);
return KIT_OK;
diff --git a/lang/cpp/lex/lex.c b/lang/cpp/lex/lex.c
@@ -1,4 +1,4 @@
-/* C11 lexer (§6.4). Streams tokens out of a borrowed source buffer.
+/* C11 lexer (§6.4). Streams lean tokens out of a borrowed source buffer.
*
* Tokens are recognized per the standard's lexical grammar:
* - identifiers (§6.4.2) — keyword bucketing happens later in parse_c
@@ -6,19 +6,21 @@
* - string literals (§6.4.5) and character constants (§6.4.4.4)
* including the L/u/u8/U encoding prefixes
* - punctuators (§6.4.6), longest-match, including digraphs
- * - `#` and `##` surface as TOK_PP_HASH / TOK_PP_PASTE so the
- * preprocessor can recognize directives and the paste operator
+ * - `#` and `##` (and their digraph forms `%:` and `%:%:`) become
+ * TOK_PP_HASH / TOK_PP_PASTE so PP can recognize directives and paste
*
- * Comments (§6.4.9) are consumed as whitespace; physical newlines surface
- * as TOK_NEWLINE so PP can implement directive-line semantics.
+ * Comments (§6.4.9) are consumed as whitespace; physical newlines surface as
+ * TOK_NEWLINE so PP can implement directive-line semantics.
*
- * The scanner is a raw-cursor walk: the hot path holds `cur`/`end` as
- * pointers into the (line-splice-folded) buffer and classifies each byte with
- * one load from `cclass[256]`. There is no per-byte struct round-trip, no
- * per-byte column arithmetic, and — on the overwhelmingly common splice-free
- * input — no per-byte splice test. The physical column is *deferred*: it is
- * not tracked per byte but computed once per token from a `line_start` pointer
- * (col = cur - line_start + 1) when a SrcLoc is built. */
+ * Lean-token shape: each token carries a compact LocRef (file_id + byte offset
+ * into the folded buffer) and a TextRef (a source span for source tokens, an
+ * interned Sym for synthetic tokens, or nothing for canonical punctuators). The
+ * scanner does NOT track a running line number — line/col are reconstructed on
+ * demand from the source's retained buffer + splice table (see the PP SrcInfo
+ * registry). The hot path holds `cur`/`end` as pointers into the
+ * (line-splice-folded) buffer and classifies each byte with one load from
+ * `cclass[256]`; because phase-2 splices are folded out up front, the scan
+ * walks a clean buffer with a single code path (no per-byte splice test). */
#include "lex/lex.h"
@@ -93,114 +95,153 @@ static const u8 cclass[256] = {
};
/* clang-format on */
-/* Dense slot for a punctuator's spelling cache. A Punct value indexes
- * `punct_slot[]` to a compact 1-based slot; the few dozen distinct
- * punctuators map to slots 1..PUNCT_SLOTS-1, and every other value (the
- * `default`/unknown-byte path, the digraph-aliased forms) maps to slot 0,
- * which is never cached (those intern their bytes directly). This keeps the
- * per-lexer cache to PUNCT_SLOTS entries instead of one per Punct value
- * (P_HASH_HASH+1 = 279), shrinking `struct Lexer` and its per-open memset by
- * ~1 KB while preserving the per-spelling caching. Slot assignment is
- * arbitrary but stable; only distinctness matters. */
-#define PUNCT_SLOTS 49u
+/* Canonical spelling bytes for the single-char punctuators: punct1[c] == c for
+ * every punctuator codepoint, so a single-char Punct code reconstructs its
+ * spelling as the one byte at &punct1[code]. */
/* clang-format off */
-static const u8 punct_slot[P_HASH_HASH + 1] = {
- /* single-char punctuators (indexed by ASCII codepoint) */
- ['!'] = 1, ['#'] = 2, ['%'] = 3, ['&'] = 4, ['('] = 5, [')'] = 6,
- ['*'] = 7, ['+'] = 8, [','] = 9, ['-'] = 10, ['.'] = 11, ['/'] = 12,
- [':'] = 13, [';'] = 14, ['<'] = 15, ['='] = 16, ['>'] = 17, ['?'] = 18,
- ['['] = 19, [']'] = 20, ['^'] = 21, ['{'] = 22, ['|'] = 23, ['}'] = 24,
- ['~'] = 25,
- /* multi-char punctuators (P_ARROW..P_HASH_HASH) */
- [P_ARROW] = 26, [P_INC] = 27, [P_DEC] = 28, [P_SHL] = 29, [P_SHR] = 30,
- [P_LE] = 31, [P_GE] = 32, [P_EQ] = 33, [P_NE] = 34, [P_AND] = 35,
- [P_OR] = 36, [P_ADD_ASSIGN] = 37, [P_SUB_ASSIGN] = 38, [P_MUL_ASSIGN] = 39,
- [P_DIV_ASSIGN] = 40, [P_MOD_ASSIGN] = 41, [P_AND_ASSIGN] = 42,
- [P_OR_ASSIGN] = 43, [P_XOR_ASSIGN] = 44, [P_SHL_ASSIGN] = 45,
- [P_SHR_ASSIGN] = 46, [P_ELLIPSIS] = 47, [P_HASH_HASH] = 48,
+static const char punct1[256] = {
+ ['!']='!', ['#']='#', ['%']='%', ['&']='&', ['(']='(', [')']=')',
+ ['*']='*', ['+']='+', [',']=',', ['-']='-', ['.']='.', ['/']='/',
+ [':']=':', [';']=';', ['<']='<', ['=']='=', ['>']='>', ['?']='?',
+ ['[']='[', [']']=']', ['^']='^', ['{']='{', ['|']='|', ['}']='}',
+ ['~']='~',
};
/* clang-format on */
+const char* punct_canon(u32 punct, u32* len) {
+ if (punct < 256u) {
+ *len = 1u;
+ return &punct1[punct];
+ }
+ switch (punct) {
+ case P_ARROW:
+ *len = 2;
+ return "->";
+ case P_INC:
+ *len = 2;
+ return "++";
+ case P_DEC:
+ *len = 2;
+ return "--";
+ case P_SHL:
+ *len = 2;
+ return "<<";
+ case P_SHR:
+ *len = 2;
+ return ">>";
+ case P_LE:
+ *len = 2;
+ return "<=";
+ case P_GE:
+ *len = 2;
+ return ">=";
+ case P_EQ:
+ *len = 2;
+ return "==";
+ case P_NE:
+ *len = 2;
+ return "!=";
+ case P_AND:
+ *len = 2;
+ return "&&";
+ case P_OR:
+ *len = 2;
+ return "||";
+ case P_ADD_ASSIGN:
+ *len = 2;
+ return "+=";
+ case P_SUB_ASSIGN:
+ *len = 2;
+ return "-=";
+ case P_MUL_ASSIGN:
+ *len = 2;
+ return "*=";
+ case P_DIV_ASSIGN:
+ *len = 2;
+ return "/=";
+ case P_MOD_ASSIGN:
+ *len = 2;
+ return "%=";
+ case P_AND_ASSIGN:
+ *len = 2;
+ return "&=";
+ case P_OR_ASSIGN:
+ *len = 2;
+ return "|=";
+ case P_XOR_ASSIGN:
+ *len = 2;
+ return "^=";
+ case P_SHL_ASSIGN:
+ *len = 3;
+ return "<<=";
+ case P_SHR_ASSIGN:
+ *len = 3;
+ return ">>=";
+ case P_ELLIPSIS:
+ *len = 3;
+ return "...";
+ case P_HASH_HASH:
+ *len = 2;
+ return "##";
+ default:
+ *len = 0;
+ return "";
+ }
+}
+
struct Lexer {
Compiler* c;
Heap* heap;
/* `src`/`len` are the post-phase-2 (line-splice-folded) logical bytes the
* scanner walks. When the input contains no `\<newline>` (the common case)
* this borrows the input verbatim (owns_src = 0, zero-copy); otherwise it is
- * a folded heap copy. `cur` is the live cursor and `end` == src + len; both
- * stay pointers so the hot scan never reloads pos/len from the struct. */
+ * a folded heap copy. `cur` is the live cursor and `end` == src + len. */
const char* src;
const char* end;
const char* cur;
- size_t len;
- /* Deferred column: the column of a byte at `p` is `(p - line_start) + 1`.
- * `line_start` points just past the last physical line break the cursor has
- * crossed (the byte after a '\n', or the fold point of a `\<newline>`), so
- * no per-byte column arithmetic is needed — only a subtraction per built
- * SrcLoc. */
- const char* line_start;
+ u32 len;
u32 file_id;
- u32 line;
u8 at_bol;
u8 had_space;
/* §5.1.1.2 phase 4 directive context for header-name lexing.
* 0 = none, 1 = saw pp-hash, 2 = saw `#include`/etc and the next
* token may be a header-name. */
u8 dstate;
- u8 owns_src; /* src is a heap-allocated folded copy to free at close */
- /* Newline-token policy. When emit_newlines (the default) is set, every
- * physical newline surfaces as a TOK_NEWLINE (the -E / cpp path needs them
- * for line reconstruction). When cleared (the cc parser-feed path), a
- * NON-directive newline is consumed silently — the line/at_bol/dstate
- * bookkeeping still runs, but no token is returned; the parser never wanted
- * newlines and PP's downstream skip_nl drain becomes a no-op. The newline
- * that TERMINATES a directive line is ALWAYS emitted regardless, so
- * read_directive_line / #if-skip still see their terminator (Option B2). */
+ u8 owns_src; /* src is a heap-allocated folded copy this lexer must free */
+ /* Newline-token policy. When set (default), every physical newline surfaces
+ * as a TOK_NEWLINE (the -E / cpp path needs them). When clear (cc parser
+ * feed), a NON-directive newline is consumed silently; the directive-line
+ * terminating newline is always emitted (Option B2). */
u8 emit_newlines;
/* Set when a directive-introducing `#` (TOK_PP_HASH at BOL) was emitted on
- * the current physical line; persists across the directive's tokens and is
- * cleared by the line-terminating newline. Used to keep emitting that one
- * newline even when emit_newlines is off. */
+ * the current physical line; persists across the directive's tokens, cleared
+ * by the terminating newline. Forces that one newline to surface even when
+ * emit_newlines is off. */
u8 directive_line;
- /* Sorted logical offsets at which a `\<newline>` splice was folded out, so
- * line tracking can still advance the physical line as the cursor passes
- * each. Empty (and NULL) for splice-free input — the common case pays
- * nothing for splices. */
+ /* Sorted logical offsets at which a `\<newline>` splice was folded out. The
+ * scanner never consults these (the folded buffer is splice-free) — they are
+ * retained metadata so the PP SrcInfo registry can reconstruct physical line
+ * numbers from a byte offset. NULL/empty for splice-free input. */
u32* splices;
u32 nsplices;
- u32 next_splice;
- /* Lazily-interned spelling Sym per punctuator, cached so the same ~60
- * distinct punctuator spellings are not re-interned on every occurrence.
- * The table is a compact, densely-indexed array (see PUNCT_SLOTS /
- * punct_slot): a punctuator's Punct value maps through the static
- * `punct_slot[]` table to a small slot, keeping the per-lexer cache to a
- * few hundred bytes instead of the ~1.1 KB a Punct-value-indexed array
- * cost — the bulk of `struct Lexer` and of lex_open_mem's per-open memset.
- * Entry value 0 is the reserved pool "none" sym, an unambiguous
- * not-yet-interned sentinel. The digraph spellings (<: <% %> :> %: %:%:)
- * are NOT cached here: they share a Punct value with their canonical form
- * but spell differently, so they intern their source bytes verbatim each
- * time (see the punct scanner). */
- Sym punct_sym[PUNCT_SLOTS];
+ /* Logical offset of the shebang line's terminating newline (the column origin
+ * of line 1 when a shebang was skipped); 0 when no shebang. */
+ u32 shebang_off;
};
/* §5.1.1.2 translation phase 2: fold every `\<newline>` line splice out of the
- * input once, up front, so the scanner never tests for splices per byte. The
- * common (splice-free) input is borrowed verbatim; otherwise a heap copy holds
- * the folded text and `splices[]` records each fold point (a logical offset) so
- * line numbering still advances across the removed physical newline. */
-static void lex_fold_splices(Lexer* l, const char* src, size_t len) {
- size_t i;
- size_t nspl = 0;
+ * input once, up front, so the scanner walks a clean buffer. The common
+ * (splice-free) input is borrowed verbatim; otherwise a heap copy holds the
+ * folded text and `splices[]` records each fold point (a logical offset) so the
+ * SrcInfo line-map can still account for the removed physical newline. */
+static void lex_fold_splices(Lexer* l, const char* src, u32 len) {
+ u32 i;
+ u32 nspl = 0;
char* buf;
u32* spl;
- size_t w, s, wcap;
- l->next_splice = 0;
- /* Count `\<newline>` splices without a scalar whole-buffer pass: backslash is
- * rare in C, so a memchr-driven sweep skips non-backslash spans at glibc-NEON
- * speed (~1 instruction per 16-32 bytes) instead of testing every byte. The
- * `-1` keeps bs[1] in range, so a trailing lone `\\` at src[len-1] is never
- * counted as a splice — bit-exact with the old scalar `i + 1 < len` loop. */
+ u32 w, s, wcap;
+ /* Count `\<newline>` splices via a memchr-driven sweep over backslashes. The
+ * `-1` keeps bs[1] in range so a trailing lone `\` is never counted. */
if (len) {
const char* p = src;
const char* e = src + len;
@@ -219,14 +260,10 @@ static void lex_fold_splices(Lexer* l, const char* src, size_t len) {
l->nsplices = 0;
return;
}
- /* Each folded splice removes exactly 2 bytes, so the folded length is known
- * up front; allocate it exactly so the close-time free size matches. */
wcap = len - 2u * nspl;
spl = (u32*)l->heap->alloc(l->heap, nspl * sizeof(u32), _Alignof(u32));
buf = wcap ? (char*)l->heap->alloc(l->heap, wcap, 1) : NULL;
if (!spl || (wcap && !buf)) {
- /* OOM: fall back to the raw input. The (rare) splice then survives into a
- * token spelling, but lexing stays memory-safe. */
if (buf) l->heap->free(l->heap, buf, wcap);
if (spl) l->heap->free(l->heap, spl, nspl * sizeof(u32));
l->src = src;
@@ -240,7 +277,7 @@ static void lex_fold_splices(Lexer* l, const char* src, size_t len) {
s = 0;
for (i = 0; i < len;) {
if (i + 1 < len && src[i] == '\\' && src[i + 1] == '\n') {
- spl[s++] = (u32)w; /* fold point = logical offset of the next byte */
+ spl[s++] = w; /* fold point = logical offset of the next byte */
i += 2;
continue;
}
@@ -250,37 +287,12 @@ static void lex_fold_splices(Lexer* l, const char* src, size_t len) {
l->len = w; /* == wcap */
l->owns_src = wcap ? 1u : 0u;
l->splices = spl;
- l->nsplices = (u32)s;
+ l->nsplices = s;
}
-/* Catch line tracking up to the cursor: process every splice fold point at or
- * before `cur` (a `\<newline>` removed there was a physical line break, so it
- * bumps the line and starts a fresh column origin at the fold). Called after a
- * fast scan that may have jumped over fold points; cheap no-op when no splices
- * remain. Only ever called when l->splices != NULL.
- *
- * Bit-exactness: a token body (identifier/number/string/punct/header) contains
- * no '\n', so the only line-break event within it is a fold; processing them
- * in order leaves `line` advanced by one per fold and `line_start` at the last
- * fold — identical to the old per-byte advance, which is observable only at the
- * next SrcLoc (no column is read mid-body). The inclusive `<= cur` bound makes
- * a fold sitting exactly at the cursor (a token that starts on a freshly
- * spliced line) fire here, matching the old "sync when pos == fold". */
-static void lex_catchup_splices(Lexer* l) {
- const char* base = l->src;
- u32 ns = l->next_splice;
- u32 n = l->nsplices;
- while (ns < n && base + l->splices[ns] <= l->cur) {
- l->line++;
- l->line_start = base + l->splices[ns];
- ns++;
- }
- l->next_splice = ns;
-}
-
-/* The off-th logical byte at the cursor, or -1 at end of input. Used only on
- * cold lookahead paths (encoding-prefix detection, punctuator longest-match,
- * UCNs); the hot scan walks `cur`/`end` directly. */
+/* The off-th logical byte at the cursor, or -1 at end of input. Cold lookahead
+ * only (encoding-prefix detection, punctuator longest-match, UCNs); the hot
+ * scan walks `cur`/`end` directly. */
static int peek(const Lexer* l, size_t off) {
const char* p = l->cur + off;
return p < l->end ? (unsigned char)*p : -1;
@@ -292,25 +304,16 @@ static int is_hex_digit(int c) {
(c >= 'A' && c <= 'F');
}
-/* Consume a maximal run of identifier-continuation bytes (CC_IDCONT) from the
- * cursor in a tight raw-pointer loop: one table load per byte, no struct
- * reload, no per-byte column or splice work. No CC_IDCONT byte is '\n', so the
- * line is untouched here; any fold points crossed are reconciled once at the
- * end. Stops at the first non-CC_IDCONT byte (a '\\' UCN lead, a punctuator, or
- * end), which the caller re-examines (e.g. for a UCN continuation). */
+/* Consume a maximal run of identifier-continuation bytes (CC_IDCONT). */
static void scan_ident_run(Lexer* l) {
const char* p = l->cur;
const char* end = l->end;
while (p < end && (cclass[(unsigned char)*p] & CC_IDCONT)) p++;
l->cur = p;
- if (l->splices) lex_catchup_splices(l);
}
-/* Match a UCN at offset `off` from the current position. Returns the total
- * length (6 for \uXXXX, 10 for \UXXXXXXXX), or 0 if no UCN matches. The
- * range constraints from §6.4.3 (no UCN < 00A0 except $/@/`, and none in
- * D800–DFFF) are not enforced here — the lexical form is matched and any
- * downstream phase that cares can diagnose. */
+/* Match a UCN at offset `off`. Returns total length (6 for \uXXXX, 10 for
+ * \UXXXXXXXX), or 0 if no UCN matches. */
static int ucn_len(const Lexer* l, size_t off) {
int n, i;
if (peek(l, off) != '\\') return 0;
@@ -326,182 +329,124 @@ static int ucn_len(const Lexer* l, size_t off) {
return 2 + n;
}
-static SrcLoc lex_here(const Lexer* l) {
- SrcLoc loc;
+static LocRef loc_at(const Lexer* l, const char* p) {
+ LocRef loc;
loc.file_id = l->file_id;
- loc.line = l->line;
- loc.col = (u32)(l->cur - l->line_start) + 1u;
+ loc.off = (u32)(p - l->src);
return loc;
}
-/* Fold splices on (src, len) and reset the scanner cursor/line state to the
- * start of the buffer. Shared by lex_open_mem_sym (fresh lexer) and
- * lex_reset_mem (re-pointed lexer). Does NOT touch file_id, which the caller
- * sets, nor the punct_sym cache, which stays valid across resets (the interned
- * Sym for a spelling is stable). */
-static void lex_point_at(Lexer* l, const char* src, size_t len) {
- lex_fold_splices(l, src ? src : "", src ? len : 0);
- l->cur = l->src;
- l->end = l->src + l->len;
- l->line = 1;
- l->line_start = l->src;
- l->dstate = 0;
- l->directive_line = 0;
- if (l->splices) lex_catchup_splices(l); /* a splice folded at offset 0 -> line 2 */
- l->at_bol = 1;
- l->had_space = 0;
+/* A TEXT_SRC reference to [a,b) in this lexer's logical buffer. */
+static TextRef text_src(const Lexer* l, const char* a, const char* b) {
+ TextRef t;
+ t.kind = TEXT_SRC;
+ t.file_id = l->file_id;
+ t.off = (u32)(a - l->src);
+ t.len_or_sym = (u32)(b - a);
+ return t;
}
-/* Splice-free variant of lex_point_at for buffers PROVABLY free of `\<newline>`
- * splices. Does exactly what lex_point_at does, but takes the nspl==0 branch of
- * lex_fold_splices unconditionally (borrow src verbatim, no splice table)
- * WITHOUT the per-byte memchr scan that, for such a buffer, would only ever
- * conclude "no splices". The cursor/line resets are identical to lex_point_at's
- * (and to the nspl==0 path it inlines: owns_src=0, splices=NULL, no
- * lex_catchup_splices since l->splices stays NULL).
- *
- * PRECONDITION (caller-guaranteed, NOT checked here): (src, len) contains no
- * `\<newline>` line-splice. The ONLY caller is the token-paste path
- * (lex_reset_mem ← paste_tokens): paste content is interned, already-folded
- * token spellings concatenated with one appended '\n', so a lone `\` can never
- * sit immediately before a '\n'. Do NOT route <command-line>/<_Pragma>/file
- * opens through here — those stay on the safe lex_point_at path (a -D value
- * could in theory carry a splice). */
-static void lex_point_at_nosplice(Lexer* l, const char* src, size_t len) {
- l->src = src ? src : "";
- l->len = src ? len : 0;
- l->owns_src = 0;
- l->splices = NULL;
- l->nsplices = 0;
- l->next_splice = 0;
+/* Reset the scanner cursor over (src, len). Shared by lex_open and lex_reset.
+ * Does NOT touch file_id (caller sets it). */
+static void lex_point_at(Lexer* l, const char* src, u32 len, int no_splices) {
+ if (no_splices) {
+ l->src = src ? src : "";
+ l->len = src ? len : 0;
+ l->owns_src = 0;
+ l->splices = NULL;
+ l->nsplices = 0;
+ } else {
+ lex_fold_splices(l, src ? src : "", src ? len : 0);
+ }
l->cur = l->src;
l->end = l->src + l->len;
- l->line = 1;
- l->line_start = l->src;
l->dstate = 0;
l->directive_line = 0;
l->at_bol = 1;
l->had_space = 0;
+ l->shebang_off = 0;
+}
+
+static u32 spec_file_id(Compiler* c, const SourceSpec* spec) {
+ u32 id = 0;
+ Sym name = spec->name_sym ? spec->name_sym : kit_sym_intern(c, spec->name);
+ (void)kit_source_add_memory_sym(c, name, &id);
+ return id;
}
-Lexer* lex_open_mem_sym(Compiler* c, Sym name, const char* src, size_t len) {
+Lexer* lex_open(Compiler* c, const SourceSpec* spec) {
Heap* h = (Heap*)kit_compiler_context(c)->heap;
Lexer* l = (Lexer*)h->alloc(h, sizeof(*l), _Alignof(Lexer));
if (!l) return NULL;
- /* Right-sized zero-init: only punct_sym genuinely needs zeroing (slot value 0
- * is the not-yet-interned sentinel; see punct_spelling). Every other field is
- * assigned before its first read below — c/heap/file_id explicitly here, then
- * src/len/owns_src/splices/nsplices/next_splice via lex_fold_splices and
- * cur/end/line/line_start/dstate/at_bol/had_space via lex_point_at. This
- * replaces a full ~288B struct memset (run on ~6,000 opens) with a ~196B
- * punct_sym clear. */
- memset(l->punct_sym, 0, sizeof l->punct_sym);
l->c = c;
l->heap = h;
- l->file_id = 0;
- /* Default: surface newline tokens (the -E / cpp / paste / _Pragma callers
- * rely on them). The cc parser-feed path opts out via lex_set_emit_newlines.
- * directive_line is (re)initialized by lex_point_at below. */
- l->emit_newlines = 1;
- (void)kit_source_add_memory_sym(c, name, &l->file_id);
- lex_point_at(l, src, len);
+ l->file_id = spec_file_id(c, spec);
+ l->emit_newlines = (spec->flags & SRC_PARSER_FEED) ? 0u : 1u;
+ lex_point_at(l, spec->bytes, spec->len, (spec->flags & SRC_NO_SPLICES) != 0);
return l;
}
-Lexer* lex_open_mem(Compiler* c, const char* name, const char* src,
- size_t len) {
- Sym sym = kit_sym_intern(c, kit_slice_cstr(name));
- return lex_open_mem_sym(c, sym, src, len);
-}
-
-/* Re-point an existing memory lexer at a fresh buffer for reuse across many
- * tiny opens (e.g. `<paste>` buffers), avoiding a fresh alloc(sizeof Lexer) +
- * lex_close free pair per open. Frees the previous folded buffer/splices (if
- * any), re-registers a fresh sequential file_id (the count is byte-observable
- * via DWARF, so it must still be bumped per open), then re-folds + resets the
- * scanner. The borrowed (src, len) must outlive the next use of the lexer. */
-void lex_reset_mem(Lexer* l, Sym name, const char* src, size_t len) {
+void lex_reset(Lexer* l, const SourceSpec* spec) {
if (l->owns_src) l->heap->free(l->heap, (char*)l->src, l->len);
- if (l->splices)
- l->heap->free(l->heap, l->splices, l->nsplices * sizeof(u32));
+ if (l->splices) l->heap->free(l->heap, l->splices, l->nsplices * sizeof(u32));
l->owns_src = 0;
l->src = NULL;
l->splices = NULL;
l->nsplices = 0;
- l->file_id = 0;
- (void)kit_source_add_memory_sym(l->c, name, &l->file_id);
- /* The paste path is the sole caller, and its buffer (concatenated interned
- * token spellings + one trailing '\n') provably has no `\<newline>` splice,
- * so skip the memchr splice scan entirely (see lex_point_at_nosplice). */
- lex_point_at_nosplice(l, src, len);
+ l->file_id = spec_file_id(l->c, spec);
+ l->emit_newlines = (spec->flags & SRC_PARSER_FEED) ? 0u : 1u;
+ lex_point_at(l, spec->bytes, spec->len, (spec->flags & SRC_NO_SPLICES) != 0);
}
void lex_close(Lexer* l) {
if (!l) return;
if (l->owns_src) l->heap->free(l->heap, (char*)l->src, l->len);
- if (l->splices)
- l->heap->free(l->heap, l->splices, l->nsplices * sizeof(u32));
+ if (l->splices) l->heap->free(l->heap, l->splices, l->nsplices * sizeof(u32));
l->heap->free(l->heap, l, sizeof(*l));
}
-/* Skip a script "shebang" line: a `#!` at the very start of the source.
- * The kernel-level `#!/path interpreter` mechanism (used to make a C file
- * executable via `kit run`) leaves the interpreter line as the first line of
- * the file, which is not valid C — `#!` would otherwise be lexed as a `#`
- * directive introducer. We only recognize it at byte 0, so a `#!` anywhere
- * else is left untouched. The line's trailing newline is left in place so the
- * lexer emits its TOK_NEWLINE and line numbering stays accurate (the shebang
- * remains line 1). No-op unless the buffer begins with the two bytes `#!`.
- * Apply only to a primary source file, never to includes/paste buffers. */
+/* Skip a leading `#!` shebang line. The shebang stays logical line 1; record
+ * the terminating newline's offset as the column origin so the materializer
+ * reports column 1 for it (matching the historical column-untracked skip). */
void lex_skip_shebang(Lexer* l) {
if (!l || l->cur != l->src) return;
if (l->len < 2 || l->src[0] != '#' || l->src[1] != '!') return;
while (l->cur < l->end && *l->cur != '\n') l->cur++;
- /* The shebang stays line 1; rebase line_start so the trailing newline still
- * reports column 1 (matching the old, column-untracked skip). */
- l->line_start = l->cur;
+ l->shebang_off = (u32)(l->cur - l->src);
}
-SrcLoc lex_loc(const Lexer* l) { return lex_here(l); }
+LocRef lex_here(const Lexer* l) { return loc_at(l, l->cur); }
u32 lex_file_id(const Lexer* l) { return l->file_id; }
-/* Set the newline-token policy (see the emit_newlines field). Off = suppress
- * non-directive newlines (cc parser feed); on (default) = surface every
- * newline (-E / cpp). Directive-terminating newlines are emitted either way. */
-void lex_set_emit_newlines(Lexer* l, int on) {
- if (l) l->emit_newlines = on ? 1u : 0u;
+const char* lex_buf(const Lexer* l) { return l->src; }
+u32 lex_buf_len(const Lexer* l) { return l->len; }
+int lex_owns_buf(const Lexer* l) { return l->owns_src; }
+u32 lex_shebang_off(const Lexer* l) { return l->shebang_off; }
+const u32* lex_splices(const Lexer* l, u32* n_out) {
+ *n_out = l->nsplices;
+ return l->splices;
}
-
-/* Intern a token's spelling [a, b) straight from the folded buffer (it is
- * already post-phase-2 logical text — splices were removed at open). */
-static Sym lex_intern(Lexer* l, const char* a, const char* b) {
- return kit_sym_intern(l->c,
- (KitSlice){.s = a, .len = (size_t)(b - a)});
+void lex_disown_buf(Lexer* l) {
+ l->owns_src = 0;
+ l->splices = NULL; /* PP now owns buffer + splice table */
+ l->nsplices = 0;
}
-/* Spelling Sym for a non-digraph punctuator, interned once per lexer and then
- * reused for every later occurrence of the same punctuator. The Punct value
- * uniquely identifies the source spelling for every non-digraph form (all '+'
- * tokens are "+", all P_ARROW are "->", ...), so caching by it is exact. The
- * first occurrence interns the exact source bytes [a,b), making the Sym
- * bit-identical to what an unconditional lex_intern would have produced. Pool
- * entry 0 is the reserved "none" sym, so a cached spelling is always nonzero
- * and 0 is an unambiguous not-yet-interned sentinel. Digraphs are excluded by
- * the caller: they share a Punct value with their canonical form but spell
- * differently, so they must intern their source bytes verbatim. */
-static Sym punct_spelling(Lexer* l, u32 punct, const char* a, const char* b) {
- u32 slot = punct_slot[punct];
- Sym s;
- if (!slot) return lex_intern(l, a, b); /* uncached value: intern directly */
- s = l->punct_sym[slot];
- if (!s) s = l->punct_sym[slot] = lex_intern(l, a, b);
+KitSlice lex_text_slice(const Lexer* l, TextRef text) {
+ KitSlice s;
+ if (text.kind == TEXT_SRC) {
+ s.s = l->src + text.off;
+ s.len = text.len_or_sym;
+ return s;
+ }
+ if (text.kind == TEXT_SYM) return kit_sym_str(l->c, (Sym)text.len_or_sym);
+ s.s = "";
+ s.len = 0;
return s;
}
-/* §6.4.7 header-name lookahead: in include-directive context, a `<` or `"`
- * starts a header-name that runs to the matching `>` or `"`. The lexer
- * recognizes only header-name forms (whose contents are implementation
- * defined), not q-char-sequence escape rules. */
+/* §6.4.7 header-name lookahead: include-family keyword set, matched on the raw
+ * source bytes of the directive identifier following a `#`. */
static int matches_include_kw(const char* s, size_t n) {
if (n == 7 && memcmp(s, "include", 7) == 0) return 1;
if (n == 12 && memcmp(s, "include_next", 12) == 0) return 1;
@@ -510,35 +455,11 @@ static int matches_include_kw(const char* s, size_t n) {
return 0;
}
-/* Splice-present per-byte advance: walk one byte, advancing line/line_start
- * across a '\n' and across any fold point now at the cursor exactly as the old
- * bump()+sync did. Used only by the splice-present whitespace/comment skip,
- * where '\n' bytes and fold points interleave and must be processed in order.
- * Cursor must be < end. */
-static void lex_bump_sp(Lexer* l) {
- char ch = *l->cur++;
- if (ch == '\n') {
- l->line++;
- l->line_start = l->cur;
- }
- {
- const char* base = l->src;
- u32 ns = l->next_splice;
- u32 n = l->nsplices;
- while (ns < n && base + l->splices[ns] == l->cur) {
- l->line++;
- l->line_start = l->cur;
- ns++;
- }
- l->next_splice = ns;
- }
-}
-
-/* Fast whitespace + comment skip (splice-free input): a raw-pointer table walk.
- * A token with no leading whitespace pays only the failed CC_SPACE test. '\n'
- * is never CC_SPACE (it surfaces as TOK_NEWLINE), so only a block comment can
- * advance the physical line here. Sets had_space when anything is consumed. */
-static void skip_ws_fast(Lexer* l) {
+/* Fast whitespace + comment skip. The folded buffer is splice-free, so this is
+ * the single skip path. '\n' is never CC_SPACE (it surfaces as TOK_NEWLINE), so
+ * only a block comment can contain a newline here — and line tracking is lazy,
+ * so even those are just skipped. Sets had_space when anything is consumed. */
+static void skip_ws_and_comments(Lexer* l) {
const char* p = l->cur;
const char* end = l->end;
int adv = 0;
@@ -560,10 +481,6 @@ static void skip_ws_fast(Lexer* l) {
p += 2;
break;
}
- if (*p == '\n') {
- l->line++;
- l->line_start = p + 1;
- }
p++;
}
adv = 1;
@@ -575,57 +492,13 @@ static void skip_ws_fast(Lexer* l) {
if (adv) l->had_space = 1;
}
-/* Whitespace + comment skip, splice-present path. Mirrors the old per-byte
- * loop (peek/bump) exactly so line/line_start stay bit-identical when a fold
- * point lands inside a comment between physical newlines. The rare path. */
-static void skip_ws_spliced(Lexer* l) {
- for (;;) {
- int ch = (l->cur < l->end) ? (unsigned char)*l->cur : -1;
- if (ch == ' ' || ch == '\t' || ch == '\r' || ch == '\v' || ch == '\f') {
- lex_bump_sp(l);
- l->had_space = 1;
- continue;
- }
- if (ch == '/' && l->cur + 1 < l->end && l->cur[1] == '/') {
- lex_bump_sp(l);
- lex_bump_sp(l);
- while (l->cur < l->end && *l->cur != '\n') lex_bump_sp(l);
- l->had_space = 1;
- continue;
- }
- if (ch == '/' && l->cur + 1 < l->end && l->cur[1] == '*') {
- lex_bump_sp(l);
- lex_bump_sp(l);
- while (l->cur < l->end) {
- if (*l->cur == '*' && l->cur + 1 < l->end && l->cur[1] == '/') {
- lex_bump_sp(l);
- lex_bump_sp(l);
- break;
- }
- lex_bump_sp(l);
- }
- l->had_space = 1;
- continue;
- }
- break;
- }
-}
-
-static void skip_ws_and_comments(Lexer* l) {
- if (l->splices)
- skip_ws_spliced(l);
- else
- skip_ws_fast(l);
-}
-
-/* Consume a pp-number per §6.4.8. The cursor is positioned at the leading
- * digit (or `.` followed by a digit) on entry. No '\n' is consumable here, so
- * line tracking only reconciles fold points crossed (splice-present path). */
+/* Consume a pp-number per §6.4.8; cursor at the leading digit (or `.`+digit).
+ */
static void scan_pp_number(Lexer* l) {
const char* p = l->cur;
const char* end = l->end;
if (p < end && *p == '.') p++;
- if (p < end) p++; /* first digit (dispatch guarantees it is present) */
+ if (p < end) p++; /* first digit (dispatch guarantees it) */
while (p < end) {
unsigned char c = (unsigned char)*p;
if ((c == 'e' || c == 'E' || c == 'p' || c == 'P') && p + 1 < end &&
@@ -638,11 +511,9 @@ static void scan_pp_number(Lexer* l) {
}
}
l->cur = p;
- if (l->splices) lex_catchup_splices(l);
}
-/* 1 if the pp-number text is a floating constant (§6.4.4.2): contains a
- * radix `.`, a hex `p`/`P` exponent, or a decimal `e`/`E` exponent. */
+/* 1 if the pp-number text is a floating constant (§6.4.4.2). */
static int pp_number_is_float(const char* s, size_t n) {
int is_hex = 0;
size_t i = 0;
@@ -664,11 +535,8 @@ static int pp_number_is_float(const char* s, size_t n) {
return 0;
}
-/* Consume a quoted body — string ('"') or character ('\''). The cursor is
- * positioned at the opening quote on entry. Returns 1 on an unterminated or
- * newline-broken literal, 0 on a clean close. A '\n' terminates the literal
- * without being consumed (it surfaces as the next TOK_NEWLINE), so line
- * tracking only reconciles fold points crossed. */
+/* Consume a quoted body — string ('"') or character ('\''); cursor at the open
+ * quote. Returns 1 on an unterminated/newline-broken literal. */
static int scan_quoted(Lexer* l, int quote) {
const char* p = l->cur;
const char* end = l->end;
@@ -699,26 +567,23 @@ static int scan_quoted(Lexer* l, int quote) {
p++;
}
l->cur = p;
- if (l->splices) lex_catchup_splices(l);
return bad;
}
-/* Finish a string/character literal token: skip `sp_len` prefix bytes, consume
- * the quoted body, classify, and intern the full spelling [tok_start, cur). */
+/* Finish a string/character literal: skip `sp_len` prefix bytes, consume the
+ * quoted body, classify, and record the full spelling span [tok_start, cur). */
static void finish_str_lit(Lexer* l, Tok* t, const char* tok_start, int sp_len,
int is_char, u32 encf) {
l->cur += sp_len;
if (scan_quoted(l, is_char ? '\'' : '"')) t->flags |= TF_LITERAL_BAD;
t->kind = (u16)(is_char ? TOK_CHR : TOK_STR);
- t->flags |= encf;
- t->spelling = lex_intern(l, tok_start, l->cur);
- t->v.str = t->spelling;
+ t->flags |= (u16)encf;
+ t->aux = 0;
+ t->text = text_src(l, tok_start, l->cur);
l->dstate = 0;
}
-/* Finish an identifier token (§6.4.2). On entry the cursor is at the first
- * identifier byte (an ident-start byte or a UCN lead). Consume the first
- * element, then alternate maximal CC_IDCONT runs with UCN continuations. */
+/* Finish an identifier token (§6.4.2). Cursor at the first identifier byte. */
static void finish_ident(Lexer* l, Tok* t, const char* tok_start) {
int u = ucn_len(l, 0);
if (u)
@@ -726,81 +591,61 @@ static void finish_ident(Lexer* l, Tok* t, const char* tok_start) {
else
l->cur += 1;
for (;;) {
- scan_ident_run(l); /* consume the maximal CC_IDCONT run in one go */
+ scan_ident_run(l);
u = ucn_len(l, 0);
if (!u) break;
l->cur += u;
}
t->kind = TOK_IDENT;
- t->spelling = lex_intern(l, tok_start, l->cur);
- t->v.ident = t->spelling;
+ t->aux = (u32)kit_sym_intern(
+ l->c, (KitSlice){.s = tok_start, .len = (size_t)(l->cur - tok_start)});
+ t->text = text_src(l, tok_start, l->cur);
if (l->dstate == 1) {
- KitSlice s = kit_sym_str(l->c, t->spelling);
- l->dstate = (s.s && matches_include_kw(s.s, s.len)) ? 2 : 0;
+ l->dstate =
+ matches_include_kw(tok_start, (size_t)(l->cur - tok_start)) ? 2 : 0;
} else {
l->dstate = 0;
}
}
-Tok lex_next(Lexer* l) {
+void lex_next(Lexer* l, Tok* out) {
Tok t;
- SrcLoc tloc;
const char* tok_start;
const char* end = l->end;
unsigned cc;
int ch;
- /* No per-token memset: every content-token path assigns kind/loc/spelling/v,
- * and flags is the only field accumulated via |=, so a deterministic zero
- * base for flags (and the v union, which the number path leaves unwritten) is
- * all the hot path needs. The cold EOF/NEWLINE early returns explicitly clear
- * spelling so the returned token stays byte-identical to the old
- * memset-then-fill token. */
t.flags = 0;
- t.v.ident = 0;
+ t.aux = 0;
- /* Skip whitespace and comments. A newline token is emitted before any
- * subsequent content tokens for the line that follows. */
+ /* Skip whitespace and comments; surface newlines per policy. */
for (;;) {
skip_ws_and_comments(l);
if (l->cur >= end) {
t.kind = TOK_EOF;
t.loc = lex_here(l);
- t.spelling = 0;
- return t;
+ t.text = text_none_ref();
+ *out = t;
+ return;
}
if (*l->cur == '\n') {
- /* Whether this newline must surface: always under emit_newlines (-E /
- * cpp), and on the cc path only when it terminates a directive line
- * (Option B2 — read_directive_line / #if-skip rely on that one). */
int emit_nl = l->emit_newlines || l->directive_line;
- tloc = lex_here(l);
- /* consume the newline */
+ LocRef nloc = lex_here(l);
l->cur++;
- l->line++;
- l->line_start = l->cur;
- if (l->splices) lex_catchup_splices(l);
l->at_bol = 1;
l->had_space = 0;
l->dstate = 0;
- /* The directive line (if any) ends at its terminating newline. */
l->directive_line = 0;
- if (!emit_nl) {
- /* Suppressed non-directive newline: bookkeeping done above (line /
- * line_start / at_bol / dstate / splices), drop the token and keep
- * scanning. The next content token carries TF_AT_BOL via l->at_bol, so
- * BOL detection is unaffected by not materializing the newline. */
- continue;
- }
+ if (!emit_nl) continue;
t.kind = TOK_NEWLINE;
- t.loc = tloc;
- t.spelling = 0;
- return t;
+ t.loc = nloc;
+ t.text = text_none_ref();
+ *out = t;
+ return;
}
break;
}
- tloc = lex_here(l);
tok_start = l->cur;
ch = (unsigned char)*l->cur;
cc = cclass[(unsigned char)ch];
@@ -809,10 +654,10 @@ Tok lex_next(Lexer* l) {
if (l->had_space) t.flags |= TF_HAS_SPACE;
l->at_bol = 0;
l->had_space = 0;
- t.loc = tloc;
+ t.loc = loc_at(l, tok_start);
- /* §6.4.7 header-name: only valid in #include / #embed argument context.
- * Takes precedence over the string-literal reading of a leading `"`. */
+ /* §6.4.7 header-name in #include / #embed argument context. Takes precedence
+ * over the string-literal reading of a leading `"`. */
if (l->dstate == 2 && (ch == '<' || ch == '"')) {
int closer = (ch == '<') ? '>' : '"';
const char* p = l->cur + 1;
@@ -828,75 +673,74 @@ Tok lex_next(Lexer* l) {
p++;
}
l->cur = p;
- if (l->splices) lex_catchup_splices(l);
t.kind = TOK_HEADER;
- t.spelling = lex_intern(l, tok_start, l->cur);
- t.v.str = t.spelling;
+ t.text = text_src(l, tok_start, l->cur);
l->dstate = 0;
- return t;
+ *out = t;
+ return;
}
- /* Identifier (§6.4.2), or an encoding-prefixed string/char literal whose
- * prefix (L/u/u8/U) is itself an identifier-start byte. The prefix+quote
- * forms are matched first since `L"`/`u'`/... is a literal, not the
- * identifier `L`/`u`. */
+ /* Identifier (§6.4.2), or an encoding-prefixed string/char literal. */
if (cc & CC_IDST) {
if (ch == 'L') {
if (peek(l, 1) == '"') {
finish_str_lit(l, &t, tok_start, 1, 0, TF_STR_WIDE);
- return t;
+ *out = t;
+ return;
}
if (peek(l, 1) == '\'') {
finish_str_lit(l, &t, tok_start, 1, 1, TF_STR_WIDE);
- return t;
+ *out = t;
+ return;
}
} else if (ch == 'u') {
if (peek(l, 1) == '8' && peek(l, 2) == '"') {
finish_str_lit(l, &t, tok_start, 2, 0, TF_STR_U8);
- return t;
+ *out = t;
+ return;
}
if (peek(l, 1) == '"') {
finish_str_lit(l, &t, tok_start, 1, 0, TF_STR_U16);
- return t;
+ *out = t;
+ return;
}
if (peek(l, 1) == '\'') {
finish_str_lit(l, &t, tok_start, 1, 1, TF_STR_U16);
- return t;
+ *out = t;
+ return;
}
} else if (ch == 'U') {
if (peek(l, 1) == '"') {
finish_str_lit(l, &t, tok_start, 1, 0, TF_STR_U32);
- return t;
+ *out = t;
+ return;
}
if (peek(l, 1) == '\'') {
finish_str_lit(l, &t, tok_start, 1, 1, TF_STR_U32);
- return t;
+ *out = t;
+ return;
}
}
finish_ident(l, &t, tok_start);
- return t;
+ *out = t;
+ return;
}
/* Bare string / character literal (no encoding prefix). */
if (ch == '"' || ch == '\'') {
finish_str_lit(l, &t, tok_start, 0, ch == '\'', 0);
- return t;
+ *out = t;
+ return;
}
- /* pp-number (§6.4.8), then classified to TOK_NUM / TOK_FLT. */
+ /* pp-number (§6.4.8), classified to TOK_NUM / TOK_FLT. */
if ((cc & CC_DIGIT) || (ch == '.' && is_digit(peek(l, 1)))) {
const char* text;
size_t k;
scan_pp_number(l);
- /* The spelling is contiguous logical text in the folded buffer (splices
- * were removed at open), so classify and intern straight from there. */
text = tok_start;
k = (size_t)(l->cur - tok_start);
t.kind = (u16)(pp_number_is_float(text, k) ? TOK_FLT : TOK_NUM);
- /* Suffix flags for §6.4.4.1 / §6.4.4.2. The parser dispatches on
- * TF_INT_U/L/LL and TF_FLT_F/L to pick a TY_* tag for the literal,
- * so missing flags would silently coerce `42U`/`42.0f` to plain
- * int/double. */
if (t.kind == TOK_FLT) {
size_t j = k;
while (j > 0) {
@@ -935,21 +779,22 @@ Tok lex_next(Lexer* l) {
break;
}
}
- t.spelling = kit_sym_intern(l->c, (KitSlice){.s = text, .len = k});
+ t.aux = 0;
+ t.text = text_src(l, tok_start, l->cur);
l->dstate = 0;
- return t;
+ *out = t;
+ return;
}
- /* Identifier introduced by a UCN (§6.4.3): `\uXXXX`/`\UXXXXXXXX` as the
- * first element. The '\\' lead is not CC_IDST, so it is matched here. */
+ /* Identifier introduced by a UCN (§6.4.3). */
if (ch == '\\' && ucn_len(l, 0)) {
finish_ident(l, &t, tok_start);
- return t;
+ *out = t;
+ return;
}
- /* Punctuator (§6.4.6) — longest match. `#` and `##` (and their digraph
- * forms `%:` and `%:%:`) become TOK_PP_HASH / TOK_PP_PASTE so PP can
- * recognize directives and the paste operator. */
+ /* Punctuator (§6.4.6) — longest match. `#`/`##` (and digraphs `%:`/`%:%:`)
+ * become TOK_PP_HASH / TOK_PP_PASTE. */
{
int n0 = ch;
int n1 = peek(l, 1);
@@ -958,9 +803,8 @@ Tok lex_next(Lexer* l) {
int adv = 1;
u32 punct = P_NONE;
u16 kind = TOK_PUNCT;
- /* Set in the six digraph branches (<: <% %> :> %: %:%:): their source
- * spelling differs from the canonical punct, so they bypass the punct_sym
- * cache and intern verbatim rather than aliasing the canonical spelling. */
+ /* A digraph spells differently from its canonical punct, so it keeps an
+ * exact source span; canonical puncts reconstruct from the code. */
int digraph = 0;
switch (n0) {
@@ -1025,13 +869,11 @@ Tok lex_next(Lexer* l) {
adv = 2;
punct = '[';
digraph = 1;
- } /* digraph */
- else if (n1 == '%') {
+ } else if (n1 == '%') {
adv = 2;
punct = '{';
digraph = 1;
- } /* digraph */
- else {
+ } else {
adv = 1;
punct = '<';
}
@@ -1138,8 +980,7 @@ Tok lex_next(Lexer* l) {
adv = 2;
punct = '}';
digraph = 1;
- } /* digraph */
- else {
+ } else {
adv = 1;
punct = '%';
}
@@ -1149,8 +990,7 @@ Tok lex_next(Lexer* l) {
adv = 2;
punct = ']';
digraph = 1;
- } /* digraph */
- else {
+ } else {
adv = 1;
punct = ':';
}
@@ -1169,30 +1009,25 @@ Tok lex_next(Lexer* l) {
punct = (u32)n0;
break;
default:
- /* Unknown byte. Surface as a single-char punct so the token
- * stream still progresses; PP/parse may diagnose. */
+ /* Unknown byte: surface as a single-char punct so the stream still
+ * progresses; the digraph path keeps its exact source span. */
adv = 1;
punct = (u32)n0;
+ digraph = 1;
break;
}
l->cur += adv;
- if (l->splices) lex_catchup_splices(l);
t.kind = kind;
- t.v.punct = punct;
- t.spelling = digraph ? lex_intern(l, tok_start, l->cur)
- : punct_spelling(l, punct, tok_start, l->cur);
+ t.aux = punct;
+ t.text = digraph ? text_src(l, tok_start, l->cur) : text_none_ref();
if (kind == TOK_PP_HASH) {
l->dstate = 1;
- /* A `#` at beginning-of-line introduces a directive (the PP only treats
- * a TF_AT_BOL, lex-sourced `#` as a directive). Mark the line so its
- * terminating newline is emitted even when emit_newlines is off, keeping
- * read_directive_line's contract intact. A non-BOL `#` (e.g. `a # b`) is
- * not a directive and leaves directive_line untouched. */
if (t.flags & TF_AT_BOL) l->directive_line = 1;
} else {
l->dstate = 0;
}
- return t;
+ *out = t;
+ return;
}
}
diff --git a/lang/cpp/lex/lex.h b/lang/cpp/lex/lex.h
@@ -3,20 +3,40 @@
#include "cpp_support.h"
+/* C11 lexer boundary (§6.4) — the "lean token" contract.
+ *
+ * The scanner streams tokens out of a borrowed, line-splice-folded source
+ * buffer into a caller-provided Tok slot. Unlike the historical token, the lean
+ * token defers the two expensive per-token operations:
+ *
+ * - SOURCE LOCATION is a compact (file_id, byte_off) reference, not an eager
+ * (file_id, line, col) triple. line/col are recovered on demand from the
+ * owning source's retained buffer + splice table (see TextRef / the PP
+ * SrcInfo registry). The scanner no longer tracks a running line number.
+ *
+ * - EXACT SPELLING is a text reference, not an eagerly interned Sym. Source
+ * tokens name a byte span in the post-splice buffer; synthetic tokens carry
+ * an interned Sym; canonical punctuators carry neither and reconstruct
+ * their spelling from the punctuator code. Identifiers are still interned
+ * eagerly (their Sym is the macro/keyword lookup key) and that Sym lives in
+ * `aux`.
+ */
+
typedef enum TokKind {
TOK_EOF = 0,
- TOK_IDENT, /* v.ident */
- TOK_NUM, /* lit */
- TOK_FLT, /* lit */
- TOK_STR, /* lit; v.str is decoded bytes if target-independent */
- TOK_CHR, /* lit */
- TOK_PUNCT, /* v.punct */
- TOK_PP_HASH, /* # */
- TOK_PP_PASTE, /* ## */
- TOK_HEADER, /* header-name in #include / #embed */
- TOK_NEWLINE, /* visible to PP only */
+ TOK_IDENT, /* aux = interned Sym; text = source span */
+ TOK_NUM, /* text = spelling; flags carry TF_INT_* */
+ TOK_FLT, /* text = spelling; flags carry TF_FLT_* */
+ TOK_STR, /* text = spelling; flags carry TF_STR_* / TF_LITERAL_BAD */
+ TOK_CHR, /* text = spelling; flags carry TF_STR_* / TF_LITERAL_BAD */
+ TOK_PUNCT, /* aux = Punct code; text = NONE (canonical) or span (digraph) */
+ TOK_PP_HASH, /* # — aux = '#' */
+ TOK_PP_PASTE, /* ## — aux = P_HASH_HASH */
+ TOK_HEADER, /* header-name in #include / #embed; text = source span */
+ TOK_NEWLINE, /* visible to PP only */
TOK_KW_FIRST,
- /* C11 keywords are inserted into this range by parse_c via pool */
+ /* Historical keyword range, unused by the parser (keyword classification is
+ * Sym-based). Kept for value stability. */
TOK_KW_LAST = 0x1000,
} TokKind;
@@ -64,57 +84,157 @@ typedef enum Punct {
P_HASH_HASH,
} Punct;
+/* ============================================================
+ * Location reference
+ * ============================================================ */
+
+/* A compact source location: a byte offset into the owning source's logical
+ * (post-line-splice-fold) buffer. line/col are materialized on demand via
+ * pp_materialize_loc / lex_materialize_loc, which consult the source's retained
+ * buffer + splice table and the active #line overlay. file_id == 0 is the
+ * "no location" sentinel (materializes to {0,0,0}). */
+typedef struct LocRef {
+ u32 file_id;
+ u32 off;
+} LocRef;
+
+/* ============================================================
+ * Text reference
+ * ============================================================ */
+
+typedef enum TextKind {
+ TEXT_NONE = 0, /* no spelling (EOF/NEWLINE/placemarker); a canonical
+ * punctuator reconstructs its spelling from `aux` instead */
+ TEXT_SRC = 1, /* a byte span in source `file_id`'s logical buffer */
+ TEXT_SYM = 2, /* an interned Sym (synthetic / predefined tokens) */
+} TextKind;
+
+/* Exact spelling reference. For TEXT_SRC, (file_id, off, len) names a span in
+ * the retained post-splice buffer of that source — valid until pp_free, because
+ * the PP SrcInfo registry retains every source buffer. For TEXT_SYM, `sym`
+ * holds the interned spelling. The two share storage in a small flat struct so
+ * the lean token stays trivially copyable. */
+typedef struct TextRef {
+ u32 kind; /* TextKind */
+ u32 file_id;
+ u32 off;
+ u32 len_or_sym; /* TEXT_SRC: byte length; TEXT_SYM: Sym */
+} TextRef;
+
+/* ============================================================
+ * Token
+ * ============================================================ */
+
typedef struct Tok {
- u16 kind;
- u16 flags;
- SrcLoc loc;
- Sym spelling; /* exact token spelling for diagnostics/#/## */
- union {
- Sym ident;
- Sym str;
- u32 punct;
- } v;
+ u16 kind; /* TokKind (+ PP-internal kinds from pp_priv.h) */
+ u16 flags; /* TokFlag bits */
+ u32 aux; /* IDENT: Sym; PUNCT/PP_HASH/PP_PASTE: Punct code;
+ * PP_PARAM: parameter index; otherwise unused (0) */
+ LocRef loc;
+ TextRef text;
} Tok;
+/* Lean accessors. The parser/PP read identifier identity and punctuator code
+ * straight off `aux` — never a symbol-table probe for punctuators. */
+static inline Sym tok_ident(const Tok* t) { return (Sym)t->aux; }
+static inline u32 tok_punct(const Tok* t) { return t->aux; }
+
+/* Lean-token reference constructors (shared by lexer, PP, and parser). */
+static inline TextRef text_none_ref(void) {
+ TextRef t;
+ t.kind = TEXT_NONE;
+ t.file_id = 0;
+ t.off = 0;
+ t.len_or_sym = 0;
+ return t;
+}
+static inline TextRef text_sym_ref(Sym s) {
+ TextRef t;
+ t.kind = TEXT_SYM;
+ t.file_id = 0;
+ t.off = 0;
+ t.len_or_sym = (u32)s;
+ return t;
+}
+static inline LocRef locref_none(void) {
+ LocRef l;
+ l.file_id = 0;
+ l.off = 0;
+ return l;
+}
+
+/* Canonical spelling for a non-digraph punctuator code (incl. '#'/P_HASH_HASH).
+ * Returns a static NUL-terminated string and writes its length to *len. Used to
+ * reconstruct punctuator text for -E / stringize / paste / diagnostics without
+ * a per-lexer spelling cache. Returns "" (len 0) for an unknown code. */
+const char* punct_canon(u32 punct, u32* len);
+
+/* ============================================================
+ * Source spec
+ * ============================================================ */
+
+typedef enum SourceFlag {
+ SRC_PRIMARY = 1u << 0, /* primary translation unit (allows shebang skip) */
+ SRC_SYSTEM = 1u << 1, /* system header (diagnostics/deps property) */
+ SRC_PARSER_FEED = 1u << 2, /* suppress non-directive newline tokens */
+ SRC_NO_SPLICES = 1u << 3, /* caller proves no backslash-newline splice */
+} SourceFlag;
+
+typedef struct SourceSpec {
+ KitSlice name; /* source name; interned unless name_sym != 0 */
+ Sym name_sym; /* pre-interned name (0 = intern `name`) */
+ const char* bytes;
+ u32 len;
+ u32 flags; /* SourceFlag bits */
+} SourceSpec;
+
+/* ============================================================
+ * Lexer
+ * ============================================================ */
+
typedef struct Lexer Lexer;
-/* lex_open_mem borrows (src, len). The lexer does not copy source bytes;
- * tokens carry SrcLoc + Sym spellings into the global pool, but diagnostics
- * and the preprocessor's directive scanner read from the borrowed buffer.
- *
- * Ownership: a Lexer that has been handed to pp_push_input is owned by PP
- * thereafter — PP closes it on EOF-pop or in pp_free. Callers only call
- * lex_close on a Lexer they have not pushed (e.g. standalone .s assembly).
+/* Open a memory lexer over the borrowed (spec->bytes, spec->len). Registers a
+ * fresh sequential compiler file_id for the source. Phase-2 line splices are
+ * folded up front: with splices the lexer builds a folded copy (lex_owns_buf
+ * true) and records each fold offset; without, it borrows the bytes verbatim.
+ * SRC_NO_SPLICES skips the fold scan (caller-guaranteed splice-free buffer).
*
- * The borrowed (src, len) buffer must outlive the Lexer, which for a pushed
- * Lexer means outliving pp_free. */
-Lexer* lex_open_mem(Compiler*, const char* name, const char* src, size_t len);
-/* As lex_open_mem but takes a pre-interned name Sym, so a fixed name reused
- * across many opens (e.g. "<paste>") is interned just once by the caller. */
-Lexer* lex_open_mem_sym(Compiler*, Sym name, const char* src, size_t len);
-/* Re-point an existing memory lexer at a fresh buffer, reusing its allocation
- * (no per-open alloc/free of the Lexer). Still registers a fresh sequential
- * file_id per call, so the file_id order is identical to lex_open_mem_sym. */
-void lex_reset_mem(Lexer*, Sym name, const char* src, size_t len);
+ * Ownership: a Lexer pushed onto PP is owned by PP (closed at EOF-pop or
+ * pp_free). The borrowed bytes — and, once PP adopts it, the folded buffer —
+ * must outlive pp_free. */
+Lexer* lex_open(Compiler*, const SourceSpec* spec);
+/* Re-point an existing lexer at a fresh buffer, reusing the Lexer allocation.
+ * Registers a fresh file_id. Used by the token-paste re-lex (splice-free). */
+void lex_reset(Lexer*, const SourceSpec* spec);
void lex_close(Lexer*);
-/* Skip a leading `#!` script-interpreter ("shebang") line so an executable
- * C file run via `kit run` lexes cleanly. Call only on a freshly-opened
- * primary source lexer, before any token is pulled; no-op otherwise. */
+/* Skip a leading `#!` shebang line on a freshly-opened primary lexer (no-op
+ * otherwise). Must be called before the first token is pulled. */
void lex_skip_shebang(Lexer*);
-/* Newline-token policy. Default ON: every physical newline surfaces as a
- * TOK_NEWLINE (required by the -E / cpp text reconstruction and the paste /
- * _Pragma sub-lexers). Set OFF on the cc parser-feed lexers (primary source +
- * includes) to suppress NON-directive newlines at the source — the parser
- * never wanted them, so the ~51% of lexer outputs that are newlines are never
- * materialized. The newline that TERMINATES a directive line is emitted in
- * either mode so read_directive_line / #if-skip keep their terminator. */
-void lex_set_emit_newlines(Lexer*, int on);
-
-/* Streaming. Returns TOK_EOF repeatedly at end of input. */
-Tok lex_next(Lexer*);
-SrcLoc lex_loc(const Lexer*);
+/* Streaming. Fills the caller's slot; returns TOK_EOF repeatedly at end. */
+void lex_next(Lexer*, Tok* out);
+
+/* The location of the current cursor (used by PP for directive/diagnostic
+ * positions). */
+LocRef lex_here(const Lexer*);
u32 lex_file_id(const Lexer*);
+/* Buffer adoption — the PP SrcInfo registry takes over the lexer's logical
+ * buffer + splice metadata so line/col and TEXT_SRC spellings remain
+ * materializable after the lexer is popped. After lex_disown_buf the lexer no
+ * longer frees the buffer at close. */
+const char* lex_buf(const Lexer*);
+u32 lex_buf_len(const Lexer*);
+int lex_owns_buf(const Lexer*);
+const u32* lex_splices(const Lexer*, u32* nsplices_out);
+u32 lex_shebang_off(
+ const Lexer*); /* logical offset of the shebang newline, or 0 */
+void lex_disown_buf(Lexer*);
+
+/* Resolve a TEXT_SRC reference into this lexer's own live buffer (used by the
+ * paste/_Pragma re-lex, before the result is interned). */
+KitSlice lex_text_slice(const Lexer*, TextRef text);
+
#endif
diff --git a/lang/cpp/pp/pp.c b/lang/cpp/pp/pp.c
@@ -13,6 +13,7 @@
* pp_new/free, predefined macros, lifecycle, keyword interning. */
#include <kit/compile.h>
+#include <kit/source.h>
#include "pp/pp_priv.h"
@@ -58,17 +59,21 @@ void src_pop(Pp* pp) {
--pp->nsources;
}
+/* Synthesize a lean EOF token into *out. */
+static void tok_eof(Tok* out) {
+ out->kind = TOK_EOF;
+ out->flags = 0;
+ out->aux = 0;
+ out->loc = (LocRef){0, 0};
+ out->text = (TextRef){TEXT_NONE, 0, 0, 0};
+}
+
/* Read next raw token from the top source, writing it through `out`. Sets
* *out to TOK_EOF when the stack is empty. Pops empty buffer/lexer sources
* as it descends. `src_kind_out`, if non-NULL, receives the kind of the
* source the token came from (SRC_LEX vs SRC_BUF). Used by pp_next_raw to
* gate directive recognition to lex-sourced tokens only — a `#` produced by
- * macro expansion never starts a directive (§6.10.3.4 ¶3, covered by
- * `63_rescan_not_directive`).
- *
- * This is the out-pointer form: lex_next already returns via sret, so it
- * writes straight into the caller's slot with no inter-frame copy, and the
- * EOF/empty-stack fallthrough is the only path that has to zero *out. */
+ * macro expansion never starts a directive (§6.10.3.4 ¶3). */
void src_next_raw_into(Pp* pp, Tok* out, HidesetId* hs_out, u8* src_kind_out) {
TokSrc* s;
while ((s = src_top(pp)) != NULL) {
@@ -90,13 +95,7 @@ void src_next_raw_into(Pp* pp, Tok* out, HidesetId* hs_out, u8* src_kind_out) {
return;
}
if (s->scope_top) {
- /* Explicit field zeroing in lieu of memset: byte-identical to the
- * old memset-then-kind EOF token, minus the per-EOF libcall. */
- out->kind = TOK_EOF;
- out->flags = 0;
- out->loc = (SrcLoc){0};
- out->spelling = 0;
- out->v.ident = 0;
+ tok_eof(out);
if (hs_out) *hs_out = HS_EMPTY;
if (src_kind_out) *src_kind_out = SRC_BUF;
return;
@@ -105,7 +104,7 @@ void src_next_raw_into(Pp* pp, Tok* out, HidesetId* hs_out, u8* src_kind_out) {
continue;
}
/* SRC_LEX */
- *out = lex_next(s->lex);
+ lex_next(s->lex, out);
if (out->kind == TOK_EOF) {
if (pp->nsources > 1) {
src_pop(pp);
@@ -125,23 +124,14 @@ void src_next_raw_into(Pp* pp, Tok* out, HidesetId* hs_out, u8* src_kind_out) {
!(out->kind == TOK_PP_HASH && (out->flags & TF_AT_BOL))) {
s->guard_state = GUARD_FAILED;
}
- /* Apply #line line-number delta on the way out so the rest of
- * the pipeline sees user-visible line numbers (matters for
- * __LINE__ expansion and for line-tracking output cursors). */
- if (s->line_delta) {
- out->loc.line = (u32)((i32)out->loc.line + s->line_delta);
- }
+ /* #line numbering is no longer applied here: a lean token carries only a
+ * byte offset, and the #line delta is recorded as a positional overlay
+ * segment (see pp_add_line_seg) applied lazily by pp_materialize_loc. */
if (hs_out) *hs_out = HS_EMPTY;
if (src_kind_out) *src_kind_out = SRC_LEX;
return;
}
- /* Explicit field zeroing in lieu of memset: byte-identical to the old
- * memset-then-kind EOF token, minus the per-EOF libcall. */
- out->kind = TOK_EOF;
- out->flags = 0;
- out->loc = (SrcLoc){0};
- out->spelling = 0;
- out->v.ident = 0;
+ tok_eof(out);
if (hs_out) *hs_out = HS_EMPTY;
if (src_kind_out) *src_kind_out = SRC_LEX;
}
@@ -191,7 +181,7 @@ void push_buf_uniform(Pp* pp, Tok* toks, u32 n, HidesetId hs_uniform) {
* (has_loc_override) reproduces the mutations the old fresh-copy path baked in,
* so the shared body is surfaced byte-for-byte without copying it. */
void push_buf_replay(Pp* pp, const Tok* toks, u32 n, HidesetId hs_uniform,
- SrcLoc loc_override, u16 first_flags_or) {
+ LocRef loc_override, u16 first_flags_or) {
TokSrc s;
memset(&s, 0, sizeof(s));
s.kind = SRC_BUF;
@@ -210,36 +200,30 @@ void push_buf_replay(Pp* pp, const Tok* toks, u32 n, HidesetId hs_uniform,
* Public streaming entries
* ============================================================ */
-/* Defined in pp_expand.c (deliberately not in pp_priv.h): the §C.3a
- * newline-filtering reader. Identical to pp_next_raw_into but drops
- * non-directive newlines one frame lower, so the ~51% of produced tokens that
- * this loop would otherwise read-and-discard never round-trip through the call. */
-void pp_next_into(Pp* pp, Tok* out);
-
-Tok pp_next(Pp* pp) {
- /* Public: filter newlines so consumers like the C parser don't need
- * to handle them. pp_emit_text uses pp_next_raw via its own loop.
+void pp_next_parse(Pp* pp, Tok* out) {
+ /* Public parser-feed stream: filter newlines so consumers like the C parser
+ * don't need to handle them. pp_emit_text uses pp_next_raw via its own loop.
*
* The top-of-loop read uses pp_next_into, which already drops non-directive
* newlines internally; the peek-after-hash (t2) and pragma-swallow reads
- * below stay on pp_next_raw_into because they must still see a TOK_NEWLINE
- * (as a non-pragma push-back token, and as the swallow terminator).
+ * below stay on pp_next_raw because they must still see a TOK_NEWLINE (as a
+ * non-pragma push-back token, and as the swallow terminator).
*
- * Also drop forwarded `#pragma` lines: do_pragma pushes the directive
- * back onto the source stack so pp_emit_text can re-emit it verbatim
- * in cpp mode, but the C parser (cc mode) would see the trailing
- * tokens as stray identifiers. When we see TOK_PP_HASH followed by
- * `pragma`, swallow tokens through the next NEWLINE. */
+ * Also drop forwarded `#pragma` lines: do_pragma pushes the directive back
+ * onto the source stack so pp_emit_text can re-emit it verbatim in cpp mode,
+ * but the C parser (cc mode) would see the trailing tokens as stray
+ * identifiers. When we see TOK_PP_HASH followed by `pragma`, swallow tokens
+ * through the next NEWLINE. */
for (;;) {
Tok t;
pp_next_into(pp, &t);
if (t.kind == TOK_PP_HASH) {
Tok t2;
- pp_next_raw_into(pp, &t2);
- if (t2.kind == TOK_IDENT && t2.v.ident == pp->sym_pragma) {
+ pp_next_raw(pp, &t2);
+ if (t2.kind == TOK_IDENT && tok_ident(&t2) == pp->sym_pragma) {
for (;;) {
Tok tt;
- pp_next_raw_into(pp, &tt);
+ pp_next_raw(pp, &tt);
if (tt.kind == TOK_NEWLINE || tt.kind == TOK_EOF) break;
}
continue;
@@ -251,9 +235,11 @@ Tok pp_next(Pp* pp) {
keep[0] = t2;
hs[0] = HS_EMPTY;
push_buf(pp, keep, hs, 1);
- return t;
+ *out = t;
+ return;
}
- return t;
+ *out = t;
+ return;
}
}
@@ -292,7 +278,8 @@ void pp_emit_text(Pp* pp, Writer* out) {
int at_bol = 1;
for (;;) {
Tok t;
- pp_next_raw_into(pp, &t);
+ KitSlice s;
+ pp_next_raw(pp, &t);
if (t.kind == TOK_EOF) break;
if (t.kind == TOK_NEWLINE) {
pp_emit_stage(out, obuf, sizeof obuf, &on, "\n", 1);
@@ -306,16 +293,248 @@ void pp_emit_text(Pp* pp, Writer* out) {
* don't run together. */
pp_emit_stage(out, obuf, sizeof obuf, &on, " ", 1);
}
- if (t.spelling) {
- KitSlice s = kit_sym_str(pp->pool->c, t.spelling);
- pp_emit_stage(out, obuf, sizeof obuf, &on, s.s, s.len);
- }
+ s = pp_text_slice(pp, &t);
+ if (s.len) pp_emit_stage(out, obuf, sizeof obuf, &on, s.s, s.len);
at_bol = 0;
}
if (on) (void)kit_writer_write(out, obuf, on);
}
/* ============================================================
+ * SrcInfo registry — lazy loc + retained text
+ * ============================================================ */
+
+SrcInfo* pp_srcinfo(Pp* pp, u32 file_id) {
+ if (file_id >= pp->srcinfo_cap) {
+ u32 nc = pp->srcinfo_cap ? pp->srcinfo_cap * 2 : 16;
+ while (nc <= file_id) nc *= 2;
+ pp->srcinfo = (SrcInfo*)pp_xrealloc(
+ pp, pp->srcinfo, sizeof(SrcInfo) * pp->srcinfo_cap,
+ sizeof(SrcInfo) * nc, _Alignof(SrcInfo));
+ memset(pp->srcinfo + pp->srcinfo_cap, 0,
+ sizeof(SrcInfo) * (nc - pp->srcinfo_cap));
+ pp->srcinfo_cap = nc;
+ }
+ return &pp->srcinfo[file_id];
+}
+
+void pp_register_srcinfo(Pp* pp, Lexer* lex) {
+ u32 fid = lex_file_id(lex);
+ SrcInfo* si = pp_srcinfo(pp, fid);
+ u32 nspl = 0;
+ si->buf = lex_buf(lex);
+ si->len = lex_buf_len(lex);
+ si->owns_buf = (u8)lex_owns_buf(lex);
+ si->splices = (u32*)lex_splices(lex, &nspl);
+ si->nsplices = nspl;
+ si->shebang_off = lex_shebang_off(lex);
+ si->line_built = 0;
+ si->line_off = NULL;
+ si->nlines = 0;
+ si->line_cursor = 0;
+ si->segs = NULL;
+ si->nsegs = 0;
+ si->segs_cap = 0;
+ /* PP now owns the buffer + splice table; the lexer must not free them. */
+ lex_disown_buf(lex);
+}
+
+void pp_add_line_seg(Pp* pp, u32 file_id, u32 off, i32 delta, Sym file) {
+ SrcInfo* si = pp_srcinfo(pp, file_id);
+ if (si->nsegs == si->segs_cap) {
+ u32 nc = si->segs_cap ? si->segs_cap * 2 : 4;
+ si->segs =
+ (LineSeg*)pp_xrealloc(pp, si->segs, sizeof(LineSeg) * si->segs_cap,
+ sizeof(LineSeg) * nc, _Alignof(LineSeg));
+ si->segs_cap = nc;
+ }
+ si->segs[si->nsegs].off = off;
+ si->segs[si->nsegs].delta = delta;
+ si->segs[si->nsegs].file = file;
+ ++si->nsegs;
+}
+
+/* Build the physical line-start index for a source by replaying the lexer's
+ * line accounting over the (immutable, fully-folded) buffer. A line-break event
+ * has a "trigger" offset equal to the line_start it establishes: a '\n' at
+ * offset p triggers at p+1; a folded splice at offset s triggers at s. The
+ * lexer processes both in scan order, so the sorted multiset of triggers,
+ * prefixed by line 1's origin (shebang_off, else 0), reproduces line/col
+ * exactly: line(off) = 1 + count(triggers <= off); col = off - origin + 1. */
+static void srcinfo_build_lines(Pp* pp, SrcInfo* si) {
+ const char* b = si->buf;
+ const char* e = b + si->len;
+ const char* p;
+ const char* nl;
+ u32 nnl = 0, k, sp;
+ u32* lo;
+ /* Count newlines via memchr (NEON-accelerated) rather than a scalar byte
+ * loop — this build is the one-time cost paid the first time a loc in this
+ * source is materialized. */
+ for (p = b; (nl = (const char*)memchr(p, '\n', (size_t)(e - p))) != NULL;)
+ ++nnl, p = nl + 1;
+ {
+ u32 cap = 1u + nnl + si->nsplices;
+ lo = (u32*)pp_xrealloc(pp, NULL, 0, sizeof(u32) * cap, _Alignof(u32));
+ }
+ k = 0;
+ lo[k++] = si->shebang_off;
+ sp = 0;
+ for (p = b; (nl = (const char*)memchr(p, '\n', (size_t)(e - p))) != NULL;) {
+ u32 nlv = (u32)(nl - b) + 1u;
+ while (sp < si->nsplices && si->splices[sp] < nlv)
+ lo[k++] = si->splices[sp++];
+ lo[k++] = nlv;
+ p = nl + 1;
+ }
+ while (sp < si->nsplices) lo[k++] = si->splices[sp++];
+ si->line_off = lo;
+ si->nlines = k;
+ si->line_cursor = 0;
+ si->line_built = 1;
+}
+
+/* Largest index k in line_off[0..n) with line_off[k] <= off (n >= 1). */
+static u32 line_index(const u32* lo, u32 n, u32 off) {
+ u32 a = 0, b = n;
+ while (a < b) {
+ u32 m = a + (b - a) / 2u;
+ if (lo[m] <= off)
+ a = m + 1u;
+ else
+ b = m;
+ }
+ return a ? a - 1u : 0u;
+}
+
+/* Cursored line lookup: loc materialization walks the source in near-monotonic
+ * byte order (the parser advances forward), so the previously-returned line is
+ * almost always the answer or one step behind. Hit-in-line and short forward
+ * runs are O(1); a far forward jump or any backward jump falls back to the
+ * binary search. Updates the cursor for the next call. */
+static u32 line_index_cursored(SrcInfo* si, u32 off) {
+ const u32* lo = si->line_off;
+ u32 n = si->nlines;
+ u32 c = si->line_cursor;
+ if (c >= n) c = 0;
+ if (lo[c] <= off) {
+ if (c + 1u >= n || off < lo[c + 1u])
+ return c; /* same line — the hot case */
+ {
+ u32 step = 0;
+ while (c + 1u < n && lo[c + 1u] <= off && step < 8u) ++c, ++step;
+ if (c + 1u < n && lo[c + 1u] <= off) c = line_index(lo, n, off);
+ }
+ } else {
+ c = line_index(lo, n, off); /* backward jump */
+ }
+ si->line_cursor = c;
+ return c;
+}
+
+static i32 seg_delta_at(const SrcInfo* si, u32 off) {
+ i32 d = 0;
+ u32 i;
+ for (i = 0; i < si->nsegs; ++i) {
+ if (si->segs[i].off <= off)
+ d = si->segs[i].delta;
+ else
+ break;
+ }
+ return d;
+}
+
+static Sym seg_file_at(const SrcInfo* si, u32 off) {
+ Sym f = 0;
+ u32 i;
+ for (i = 0; i < si->nsegs; ++i) {
+ if (si->segs[i].off <= off)
+ f = si->segs[i].file;
+ else
+ break;
+ }
+ return f;
+}
+
+SrcLoc pp_materialize_loc(Pp* pp, LocRef loc) {
+ SrcLoc r;
+ SrcInfo* si;
+ u32 k;
+ r.file_id = loc.file_id;
+ r.line = 0;
+ r.col = 0;
+ if (!loc.file_id) return r;
+ si = pp_srcinfo(pp, loc.file_id);
+ if (!si->buf) return r;
+ if (!si->line_built) srcinfo_build_lines(pp, si);
+ k = line_index_cursored(si, loc.off);
+ r.line = (u32)((i32)(k + 1u) + seg_delta_at(si, loc.off));
+ r.col = loc.off - si->line_off[k] + 1u;
+ return r;
+}
+
+u32 pp_phys_line(Pp* pp, LocRef loc) {
+ SrcInfo* si;
+ if (!loc.file_id) return 0;
+ si = pp_srcinfo(pp, loc.file_id);
+ if (!si->buf) return 0;
+ if (!si->line_built) srcinfo_build_lines(pp, si);
+ return line_index(si->line_off, si->nlines, loc.off) + 1u;
+}
+
+Sym pp_materialize_file(Pp* pp, LocRef loc) {
+ SrcInfo* si;
+ Sym f;
+ KitSourceFile sf;
+ if (!loc.file_id) return 0;
+ si = pp_srcinfo(pp, loc.file_id);
+ f = si->buf ? seg_file_at(si, loc.off) : 0;
+ if (f) return f;
+ if (kit_source_file(pp->c, loc.file_id, &sf) == KIT_OK) return sf.name;
+ return 0;
+}
+
+/* ============================================================
+ * Text materialization
+ * ============================================================ */
+
+KitSlice pp_text_slice(Pp* pp, const Tok* t) {
+ KitSlice s;
+ switch (t->text.kind) {
+ case TEXT_SRC: {
+ SrcInfo* si = pp_srcinfo(pp, t->text.file_id);
+ s.s = si->buf + t->text.off;
+ s.len = t->text.len_or_sym;
+ return s;
+ }
+ case TEXT_SYM:
+ return kit_sym_str(pp->pool->c, (Sym)t->text.len_or_sym);
+ default: /* TEXT_NONE: a canonical punctuator reconstructs from its code */
+ if (t->kind == TOK_PUNCT || t->kind == TOK_PP_HASH ||
+ t->kind == TOK_PP_PASTE) {
+ u32 n;
+ s.s = punct_canon(t->aux, &n);
+ s.len = n;
+ return s;
+ }
+ s.s = "";
+ s.len = 0;
+ return s;
+ }
+}
+
+Sym pp_text_intern(Pp* pp, const Tok* t) {
+ if (t->text.kind == TEXT_SYM) return (Sym)t->text.len_or_sym;
+ return kit_sym_intern(pp->pool->c, pp_text_slice(pp, t));
+}
+
+int pp_text_eq_cstr(Pp* pp, const Tok* t, const char* s) {
+ KitSlice sl = pp_text_slice(pp, t);
+ size_t n = s ? strlen(s) : 0;
+ return sl.len == n && (n == 0 || memcmp(sl.s, s, n) == 0);
+}
+
+/* ============================================================
* Lifecycle and configuration
* ============================================================ */
@@ -935,6 +1154,19 @@ void pp_free(Pp* pp) {
/* The reused token-paste lexer is never pushed as a source, so close it
* directly here (it is lazily opened on the first `##`). */
if (pp->paste_lex) lex_close(pp->paste_lex);
+ /* Release the SrcInfo registry: each entry's folded buffer (if PP owns it),
+ * splice table, lazy line index, and #line overlay segments. */
+ if (pp->srcinfo) {
+ u32 i;
+ for (i = 0; i < pp->srcinfo_cap; ++i) {
+ SrcInfo* si = &pp->srcinfo[i];
+ if (si->owns_buf && si->buf) pp_xfree(pp, (char*)si->buf, si->len);
+ if (si->splices) pp_xfree(pp, si->splices, si->nsplices * sizeof(u32));
+ if (si->line_off) pp_xfree(pp, si->line_off, si->nlines * sizeof(u32));
+ if (si->segs) pp_xfree(pp, si->segs, si->segs_cap * sizeof(LineSeg));
+ }
+ pp_xfree(pp, pp->srcinfo, sizeof(SrcInfo) * pp->srcinfo_cap);
+ }
pp_xfree(pp, pp->sources, sizeof(TokSrc) * pp->sources_cap);
MacroTab_fini(&pp->macros);
pp_xfree(pp, pp->hsets, sizeof(Hideset*) * pp->hsets_cap);
@@ -947,20 +1179,20 @@ void pp_free(Pp* pp) {
h->free(h, pp, sizeof(*pp));
}
-void pp_set_suppress_lexer_newlines(Pp* pp, int on) {
- if (pp) pp->suppress_lexer_newlines = on ? 1u : 0u;
-}
-
-void pp_push_input(Pp* pp, Lexer* lex) {
+void pp_push_source(Pp* pp, const SourceSpec* spec) {
TokSrc s;
+ Lexer* lex = lex_open(pp->c, spec);
+ if (!lex) compiler_panic(pp->c, (SrcLoc){0, 0, 0}, "pp: out of memory");
+ if (spec->flags & SRC_PRIMARY) lex_skip_shebang(lex);
+ /* Remember parser-feed mode so #include'd lexers inherit it (they build their
+ * own SourceSpec in do_include and OR in SRC_PARSER_FEED from this flag). */
+ if (spec->flags & SRC_PARSER_FEED) pp->parser_feed = 1;
+ /* Adopt the source buffer + splice table for lazy loc/text materialization.
+ */
+ pp_register_srcinfo(pp, lex);
memset(&s, 0, sizeof(s));
s.kind = SRC_LEX;
s.lex = lex;
- /* cc parser-feed mode: drop non-directive newlines at the lexer (see
- * pp_set_suppress_lexer_newlines). pp_push_input is the entry for the primary
- * source; #include'd lexers apply the same mode at their open site in
- * do_include. */
- if (pp->suppress_lexer_newlines) lex_set_emit_newlines(lex, 0);
src_push(pp, s);
}
@@ -978,21 +1210,23 @@ void pp_add_include_dir(Pp* pp, const char* dir, int system) {
}
void pp_define(Pp* pp, const char* name, const char* body) {
- /* Stage 1+2: build a synthetic source line "name body\n" and run it
- * through the lexer + define machinery so command-line -D matches the
- * normal #define path. */
+ /* Build a synthetic source line "name body\n" and run it through the lexer +
+ * define machinery so command-line -D matches the normal #define path. The
+ * buffer is allocated in pp->arena (retained to pp_free) because the
+ * resulting macro-body tokens carry TEXT_SRC spans into it. */
size_t nlen = name ? kit_slice_cstr(name).len : 0;
size_t blen = body ? kit_slice_cstr(body).len : 0;
- Heap* h = pp_heap(pp);
char* buf;
size_t pos = 0;
+ SourceSpec spec;
+ TokSrc s;
Lexer* lex;
Tok* line;
u32 lineN;
if (!name || !*name) return;
/* "name" + " " + "body" + "\n" */
- buf = (char*)h->alloc(h, nlen + 1 + blen + 1 + 1, 1);
+ buf = (char*)arena_alloc(pp->arena, nlen + 1 + blen + 1, 1);
memcpy(buf + pos, name, nlen);
pos += nlen;
buf[pos++] = ' ';
@@ -1001,21 +1235,21 @@ void pp_define(Pp* pp, const char* name, const char* body) {
pos += blen;
}
buf[pos++] = '\n';
- buf[pos] = 0;
- lex = lex_open_mem(pp->c, "<command-line>", buf, pos);
- {
- TokSrc s;
- memset(&s, 0, sizeof(s));
- s.kind = SRC_LEX;
- s.lex = lex;
- src_push(pp, s);
- }
+ memset(&spec, 0, sizeof(spec));
+ spec.name = KIT_SLICE_LIT("<command-line>");
+ spec.bytes = buf;
+ spec.len = (u32)pos;
+ lex = lex_open(pp->c, &spec);
+ pp_register_srcinfo(pp, lex);
+ memset(&s, 0, sizeof(s));
+ s.kind = SRC_LEX;
+ s.lex = lex;
+ src_push(pp, s);
read_directive_line(pp, &line, &lineN);
do_define(pp, line, lineN);
/* Drain anything trailing (shouldn't be any) and pop the lexer. */
src_pop(pp);
- h->free(h, buf, nlen + 1 + blen + 1 + 1);
}
void pp_undef(Pp* pp, const char* name) {
@@ -1027,10 +1261,10 @@ void pp_undef(Pp* pp, const char* name) {
uint32_t pp_pack_alignment(const Pp* pp) { return pp ? pp->pack_align : 0; }
-void pp_add_include_edge(Pp* pp, u32 includer, u32 included, SrcLoc include_loc,
+void pp_add_include_edge(Pp* pp, u32 includer, u32 included, LocRef include_loc,
int system) {
/* This generic edge-recording entry point has no resolved-dir context, so it
* can only fall back to the spelling form for the resolved-system flag. */
- kit_source_add_include(pp->c, includer, included, include_loc, system,
- system);
+ kit_source_add_include(pp->c, includer, included,
+ pp_materialize_loc(pp, include_loc), system, system);
}
diff --git a/lang/cpp/pp/pp.h b/lang/cpp/pp/pp.h
@@ -17,30 +17,51 @@ void pp_define(Pp*, const char* name, const char* body); /* -D */
void pp_undef(Pp*, const char* name); /* -U */
uint32_t pp_pack_alignment(const Pp*);
-/* Pushes a Lexer onto the include stack. PP takes ownership of the Lexer:
- * it is closed when the input hits EOF and is popped, or in pp_free if it
- * is still on the stack. Callers must not call lex_close on a pushed
- * Lexer. The borrowed source buffer (see lex_open_mem) must outlive
- * pp_free. */
-void pp_push_input(Pp*, Lexer*);
+/* Open `spec` as a new top-of-stack source: PP creates the lexer, registers the
+ * source in the SrcInfo registry (so its buffer + line map outlive the lexer
+ * for lazy loc/text materialization), and pushes it. PP owns the lexer and its
+ * buffer thereafter (freed at EOF-pop / pp_free). The borrowed spec->bytes must
+ * outlive pp_free.
+ *
+ * The SRC_PARSER_FEED flag on the primary source selects cc parser-feed mode:
+ * non-directive newline tokens are suppressed at every lexer (primary + each
+ * #include), since the C parser drops them. The -E / cpp path leaves the flag
+ * clear so newlines surface for text reconstruction. SRC_PRIMARY additionally
+ * applies shebang skipping. */
+void pp_push_source(Pp*, const SourceSpec* spec);
void pp_add_include_edge(Pp*, u32 includer_file_id, u32 included_file_id,
- SrcLoc include_loc, int system);
-
-/* cc parser-feed mode: suppress non-directive newline tokens at every
- * SRC_LEX lexer (primary source + #includes). The C parser drops newlines, so
- * materializing them is pure overhead (~51% of lexer outputs). The -E / cpp
- * path must NOT call this (it reconstructs text from newline tokens). Set once
- * after pp_new, before pushing the primary input. Directive-terminating
- * newlines are still emitted, so directive parsing is unaffected. */
-void pp_set_suppress_lexer_newlines(Pp*, int on);
-
-/* Streaming. Yields preprocessed tokens (macro-expanded, directives consumed).
- */
-Tok pp_next(Pp*);
-
-/* Drains pp_next into `out` as preprocessed C source text: token spellings
- * separated by single spaces where TF_HAS_SPACE is set, with newlines for
- * TF_AT_BOL transitions. Stops on TOK_EOF. Used by the C driver. */
+ LocRef include_loc, int system);
+
+/* Streaming. Both fill a caller-owned slot (no by-value sret).
+ *
+ * pp_next_parse: the C parser stream — macro-expanded, directives consumed,
+ * non-directive newlines absent, forwarded #pragma lines swallowed.
+ * pp_next_raw: the -E / cpp stream — macro-expanded, directives consumed,
+ * TOK_NEWLINE preserved for text reconstruction. */
+void pp_next_parse(Pp*, Tok* out);
+void pp_next_raw(Pp*, Tok* out);
+
+/* Drains pp_next_raw into `out` as preprocessed C source text: token spellings
+ * separated by single spaces where TF_HAS_SPACE/TF_AT_BOL is set, with newlines
+ * for TOK_NEWLINE. Stops on TOK_EOF. Used by the C driver. */
void pp_emit_text(Pp*, Writer* out);
+/* ============================================================
+ * Lazy materialization helpers (loc + text)
+ * ============================================================ */
+
+/* Materialize a LocRef into a full (file_id, line, col) SrcLoc, consulting the
+ * source's retained buffer/splice table and the active #line overlay. */
+SrcLoc pp_materialize_loc(Pp*, LocRef loc);
+
+/* Resolve a token's exact spelling. TEXT_SRC reads the retained source buffer;
+ * TEXT_SYM reads the interned symbol; a TEXT_NONE punctuator reconstructs its
+ * canonical spelling from the punctuator code; otherwise returns an empty
+ * slice. The returned bytes are valid until pp_free. */
+KitSlice pp_text_slice(Pp*, const Tok* t);
+/* As pp_text_slice but interns the result and returns the Sym. */
+Sym pp_text_intern(Pp*, const Tok* t);
+/* True iff the token's spelling equals the NUL-terminated cstr. */
+int pp_text_eq_cstr(Pp*, const Tok* t, const char* s);
+
#endif
diff --git a/lang/cpp/pp/pp_directive.c b/lang/cpp/pp/pp_directive.c
@@ -101,23 +101,23 @@ static i64 parse_pp_int(const char* s, size_t n) {
static void prepass_defined(Pp* pp, const Tok* in, u32 nin, TokVec* out) {
u32 i;
for (i = 0; i < nin; ++i) {
- if (in[i].kind == TOK_IDENT && in[i].v.ident == pp->sym_defined) {
+ if (in[i].kind == TOK_IDENT && tok_ident(&in[i]) == pp->sym_defined) {
int has_paren = 0;
Sym ident = 0;
u32 j = i + 1;
- if (j < nin && in[j].kind == TOK_PUNCT && in[j].v.punct == '(') {
+ if (j < nin && in[j].kind == TOK_PUNCT && tok_punct(&in[j]) == '(') {
has_paren = 1;
++j;
}
if (j >= nin || in[j].kind != TOK_IDENT) {
- compiler_panic(pp->c, in[i].loc,
+ compiler_panic(pp->c, pp_materialize_loc(pp, in[i].loc),
"operand of 'defined' must be an identifier");
}
- ident = in[j].v.ident;
+ ident = tok_ident(&in[j]);
++j;
if (has_paren) {
- if (j >= nin || in[j].kind != TOK_PUNCT || in[j].v.punct != ')') {
- compiler_panic(pp->c, in[i].loc,
+ if (j >= nin || in[j].kind != TOK_PUNCT || tok_punct(&in[j]) != ')') {
+ compiler_panic(pp->c, pp_materialize_loc(pp, in[i].loc),
"expected ')' after 'defined' operand");
}
++j;
@@ -136,8 +136,9 @@ static void prepass_defined(Pp* pp, const Tok* in, u32 nin, TokVec* out) {
t.kind = TOK_NUM;
t.flags = in[i].flags & (TF_AT_BOL | TF_HAS_SPACE);
t.loc = in[i].loc;
- t.spelling = kit_sym_intern(
- pp->pool->c, is_defined ? KIT_SLICE_LIT("1") : KIT_SLICE_LIT("0"));
+ t.aux = 0;
+ t.text = text_intern_ref(
+ pp, is_defined ? kit_slice_cstr("1") : kit_slice_cstr("0"));
tv_push(pp, out, t);
}
i = j - 1;
@@ -174,11 +175,12 @@ static void expand_for_if(Pp* pp, const Tok* in, u32 nin, TokVec* out) {
* has been handled. */
static void replace_remaining_if_identifiers(Pp* pp, TokVec* toks) {
u32 i;
- Sym zero = kit_sym_intern(pp->pool->c, KIT_SLICE_LIT("0"));
+ TextRef zero = text_intern_ref(pp, kit_slice_cstr("0"));
for (i = 0; i < toks->n; ++i) {
if (toks->data[i].kind == TOK_IDENT) {
toks->data[i].kind = TOK_NUM;
- toks->data[i].spelling = zero;
+ toks->data[i].aux = 0;
+ toks->data[i].text = zero;
}
}
}
@@ -201,7 +203,7 @@ static const Tok* ee_peek(EE* e) {
static int ee_match_punct(EE* e, u32 p) {
const Tok* t = ee_peek(e);
- if (t && t->kind == TOK_PUNCT && t->v.punct == p) {
+ if (t && t->kind == TOK_PUNCT && tok_punct(t) == p) {
++e->pos;
return 1;
}
@@ -212,7 +214,7 @@ static i64 ee_primary(EE* e) {
const Tok* t = ee_peek(e);
if (!t) compiler_panic(e->pp->c, e->loc, "#if: missing operand");
if (t->kind == TOK_NUM) {
- KitSlice s = kit_sym_str(e->pp->pool->c, t->spelling);
+ KitSlice s = pp_text_slice(e->pp, t);
++e->pos;
return parse_pp_int(s.s, s.len);
}
@@ -220,28 +222,30 @@ static i64 ee_primary(EE* e) {
/* Treat as the codepoint of the first character (post-decoding
* not implemented; cover the common case of a single ASCII
* char). */
- KitSlice s = kit_sym_str(e->pp->pool->c, t->spelling);
+ KitSlice s = pp_text_slice(e->pp, t);
++e->pos;
if (s.len >= 3 && s.s[0] == '\'') return (unsigned char)s.s[1];
return 0;
}
- if (t->kind == TOK_PUNCT && t->v.punct == '(') {
+ if (t->kind == TOK_PUNCT && tok_punct(t) == '(') {
i64 v;
++e->pos;
v = ee_ternary(e);
if (!ee_match_punct(e, ')')) {
- compiler_panic(e->pp->c, t->loc, "#if: expected ')'");
+ compiler_panic(e->pp->c, pp_materialize_loc(e->pp, t->loc),
+ "#if: expected ')'");
}
return v;
}
- compiler_panic(e->pp->c, t->loc, "#if: unexpected token in expression");
+ compiler_panic(e->pp->c, pp_materialize_loc(e->pp, t->loc),
+ "#if: unexpected token in expression");
return 0;
}
static i64 ee_unary(EE* e) {
const Tok* t = ee_peek(e);
if (t && t->kind == TOK_PUNCT) {
- u32 p = t->v.punct;
+ u32 p = tok_punct(t);
if (p == '!' || p == '-' || p == '+' || p == '~') {
i64 v;
++e->pos;
@@ -359,7 +363,7 @@ static const EeOp* ee_lookup_op(const Tok* t) {
size_t i;
if (!t || t->kind != TOK_PUNCT) return NULL;
for (i = 0; i < sizeof(ee_ops) / sizeof(ee_ops[0]); ++i) {
- if (ee_ops[i].punct == t->v.punct) return &ee_ops[i];
+ if (ee_ops[i].punct == tok_punct(t)) return &ee_ops[i];
}
return NULL;
}
@@ -373,7 +377,7 @@ static i64 ee_binary(EE* e, int min_prec) {
SrcLoc op_loc;
i64 rhs;
if (!op || op->prec < min_prec) break;
- op_loc = t->loc;
+ op_loc = pp_materialize_loc(e->pp, t->loc);
++e->pos;
/* Left-associative: parse the RHS with strictly higher precedence so
* same-prec operators fold left-to-right. */
@@ -400,10 +404,10 @@ static i64 ee_ternary(EE* e) {
/* Header-resolution helpers used by the __has_include pre-pass below; the
* definitions live further down with the #include machinery. */
-static void parse_include_path(Pp* pp, const Tok* line, u32 n, SrcLoc loc,
+static void parse_include_path(Pp* pp, const Tok* line, u32 n, LocRef loc,
char* path_out, size_t cap, int* system_out);
static int find_and_open_include(Pp* pp, const char* path, int system,
- SrcLoc loc, const u8** data, size_t* size,
+ LocRef loc, const u8** data, size_t* size,
char* resolved, size_t resolved_cap,
int* resolved_system_out, u32* next_start_out);
static int find_and_open_include_next(Pp* pp, const char* path, u32 start,
@@ -436,9 +440,9 @@ static void prepass_has_include(Pp* pp, const Tok* in, u32 nin, TokVec* out) {
const char* what;
if (in[i].kind == TOK_IDENT) {
- if (in[i].v.ident == pp->sym_has_include)
+ if (tok_ident(&in[i]) == pp->sym_has_include)
is_has = 1;
- else if (in[i].v.ident == pp->sym_has_include_next)
+ else if (tok_ident(&in[i]) == pp->sym_has_include_next)
is_has = is_next = 1;
}
if (!is_has) {
@@ -448,21 +452,24 @@ static void prepass_has_include(Pp* pp, const Tok* in, u32 nin, TokVec* out) {
what = is_next ? "__has_include_next" : "__has_include";
j = i + 1;
- if (j >= nin || in[j].kind != TOK_PUNCT || in[j].v.punct != '(') {
- compiler_panic(pp->c, in[i].loc, "expected '(' after %s", what);
+ if (j >= nin || in[j].kind != TOK_PUNCT || tok_punct(&in[j]) != '(') {
+ compiler_panic(pp->c, pp_materialize_loc(pp, in[i].loc),
+ "expected '(' after %s", what);
}
++j; /* past '(' */
op_first = &in[j];
depth = 1;
op_n = 0;
for (; j < nin; ++j, ++op_n) {
- if (in[j].kind == TOK_PUNCT && in[j].v.punct == '(') {
+ if (in[j].kind == TOK_PUNCT && tok_punct(&in[j]) == '(') {
++depth;
- } else if (in[j].kind == TOK_PUNCT && in[j].v.punct == ')') {
+ } else if (in[j].kind == TOK_PUNCT && tok_punct(&in[j]) == ')') {
if (--depth == 0) break;
}
}
- if (j >= nin) compiler_panic(pp->c, in[i].loc, "unterminated %s", what);
+ if (j >= nin)
+ compiler_panic(pp->c, pp_materialize_loc(pp, in[i].loc),
+ "unterminated %s", what);
/* in[j] is the matching ')'. */
parse_include_path(pp, op_first, op_n, in[i].loc, path, sizeof(path),
@@ -488,14 +495,15 @@ static void prepass_has_include(Pp* pp, const Tok* in, u32 nin, TokVec* out) {
t.kind = TOK_NUM;
t.flags = in[i].flags & (TF_AT_BOL | TF_HAS_SPACE);
t.loc = in[i].loc;
- t.spelling = kit_sym_intern(
- pp->pool->c, present ? KIT_SLICE_LIT("1") : KIT_SLICE_LIT("0"));
+ t.aux = 0;
+ t.text = text_intern_ref(
+ pp, present ? kit_slice_cstr("1") : kit_slice_cstr("0"));
tv_push(pp, out, t);
i = j; /* resume past the matching ')' */
}
}
-i64 eval_if_expr(Pp* pp, const Tok* line, u32 n, SrcLoc loc) {
+i64 eval_if_expr(Pp* pp, const Tok* line, u32 n, LocRef loc) {
TokVec defs = {0};
TokVec hasinc = {0};
TokVec exp = {0};
@@ -513,7 +521,7 @@ i64 eval_if_expr(Pp* pp, const Tok* line, u32 n, SrcLoc loc) {
e.toks = defs2.data;
e.n = defs2.n;
e.pos = 0;
- e.loc = loc;
+ e.loc = pp_materialize_loc(pp, loc);
v = ee_ternary(&e);
if (e.pos != e.n) {
compiler_panic(pp->c, e.loc,
@@ -548,7 +556,8 @@ static void skip_until_active(Pp* pp) {
if (top->state == IF_INCLUDE && local_depth == 0) return;
t = src_next_raw(pp, NULL, NULL);
if (t.kind == TOK_EOF) {
- compiler_panic(pp->c, top->loc, "unterminated #if / #ifdef");
+ compiler_panic(pp->c, pp_materialize_loc(pp, top->loc),
+ "unterminated #if / #ifdef");
}
if (t.kind != TOK_PP_HASH || (t.flags & TF_AT_BOL) == 0) continue;
@@ -561,7 +570,7 @@ static void skip_until_active(Pp* pp) {
consume_to_newline(pp);
continue;
}
- name = nt.v.ident;
+ name = tok_ident(&nt);
if (name == pp->sym_if || name == pp->sym_ifdef ||
name == pp->sym_ifndef) {
++local_depth;
@@ -581,7 +590,8 @@ static void skip_until_active(Pp* pp) {
consume_to_newline(pp);
if (local_depth > 0) continue;
if (top->has_else) {
- compiler_panic(pp->c, t.loc, "duplicate #else");
+ compiler_panic(pp->c, pp_materialize_loc(pp, t.loc),
+ "duplicate #else");
}
top->has_else = 1;
if (top->state == IF_SEEK_TRUE) {
@@ -638,16 +648,16 @@ static int is_predefined_macro_name(Pp* pp, Sym name) {
* #ifdef / #if / #elif / #else / #endif
* ============================================================ */
-static void do_ifdef(Pp* pp, const Tok* line, u32 n, int negate, SrcLoc loc) {
+static void do_ifdef(Pp* pp, const Tok* line, u32 n, int negate, LocRef loc) {
int defined;
IfFrame f;
if (n < 1 || line[0].kind != TOK_IDENT) {
- compiler_panic(pp->c, loc,
+ compiler_panic(pp->c, pp_materialize_loc(pp, loc),
negate ? "#ifndef: expected identifier"
: "#ifdef: expected identifier");
}
- defined = (mt_get(pp, line[0].v.ident) != NULL) ||
- is_predefined_macro_name(pp, line[0].v.ident);
+ defined = (mt_get(pp, tok_ident(&line[0])) != NULL) ||
+ is_predefined_macro_name(pp, tok_ident(&line[0]));
if (negate) defined = !defined;
memset(&f, 0, sizeof(f));
f.state = defined ? IF_INCLUDE : IF_SEEK_TRUE;
@@ -656,7 +666,7 @@ static void do_ifdef(Pp* pp, const Tok* line, u32 n, int negate, SrcLoc loc) {
if (!defined) skip_until_active(pp);
}
-static void do_if_directive(Pp* pp, const Tok* line, u32 n, SrcLoc loc) {
+static void do_if_directive(Pp* pp, const Tok* line, u32 n, LocRef loc) {
i64 v = eval_if_expr(pp, line, n, loc);
IfFrame f;
memset(&f, 0, sizeof(f));
@@ -666,27 +676,30 @@ static void do_if_directive(Pp* pp, const Tok* line, u32 n, SrcLoc loc) {
if (!v) skip_until_active(pp);
}
-static void do_elif(Pp* pp, SrcLoc loc) {
+static void do_elif(Pp* pp, LocRef loc) {
/* We only reach do_elif from the active branch — meaning the
* preceding group emitted code. So we must skip the rest. */
IfFrame* top = if_top(pp);
- if (!top) compiler_panic(pp->c, loc, "stray #elif");
- if (top->has_else) compiler_panic(pp->c, loc, "#elif after #else");
+ if (!top) compiler_panic(pp->c, pp_materialize_loc(pp, loc), "stray #elif");
+ if (top->has_else)
+ compiler_panic(pp->c, pp_materialize_loc(pp, loc), "#elif after #else");
top->state = IF_DONE;
skip_until_active(pp);
}
-static void do_else(Pp* pp, SrcLoc loc) {
+static void do_else(Pp* pp, LocRef loc) {
IfFrame* top = if_top(pp);
- if (!top) compiler_panic(pp->c, loc, "stray #else");
- if (top->has_else) compiler_panic(pp->c, loc, "duplicate #else");
+ if (!top) compiler_panic(pp->c, pp_materialize_loc(pp, loc), "stray #else");
+ if (top->has_else)
+ compiler_panic(pp->c, pp_materialize_loc(pp, loc), "duplicate #else");
top->has_else = 1;
top->state = IF_DONE;
skip_until_active(pp);
}
-static void do_endif(Pp* pp, SrcLoc loc) {
- if (!if_top(pp)) compiler_panic(pp->c, loc, "stray #endif");
+static void do_endif(Pp* pp, LocRef loc) {
+ if (!if_top(pp))
+ compiler_panic(pp->c, pp_materialize_loc(pp, loc), "stray #endif");
if_pop(pp);
}
@@ -792,7 +805,7 @@ static int open_include_cached(Pp* pp, const char* path, const u8** data_out,
/* Return the includer's directory for resolving a quoted include, or "."
* for in-memory/builtin sources (where CWD is the natural fallback, like
* gcc treats stdin). `dir_out` must point to a buffer of size >= cap. */
-static int includer_dir(Pp* pp, SrcLoc loc, char* dir_out, size_t cap) {
+static int includer_dir(Pp* pp, LocRef loc, char* dir_out, size_t cap) {
KitSourceFile sf;
const char* p = NULL;
size_t plen = 0;
@@ -843,7 +856,7 @@ static int includer_dir(Pp* pp, SrcLoc loc, char* dir_out, size_t cap) {
* skips the includer step, so its key omits the dir. Returns 0 only if
* the key would overflow the scratch buffer (memo simply not used). */
static Sym inc_resolve_key(Pp* pp, const char* path, size_t plen, int system,
- SrcLoc loc) {
+ LocRef loc) {
char key[4096 + 4096 + 8];
size_t pos = 0;
key[pos++] = system ? '<' : '"';
@@ -862,8 +875,7 @@ static Sym inc_resolve_key(Pp* pp, const char* path, size_t plen, int system,
}
memcpy(key + pos, path, plen);
pos += plen;
- return kit_sym_intern(pp->pool->c,
- (KitSlice){.s = key, .len = pos});
+ return kit_sym_intern(pp->pool->c, (KitSlice){.s = key, .len = pos});
}
/* Record a successful dir-search resolution under its spelling key, so the
@@ -895,10 +907,9 @@ static void inc_resolve_record(Pp* pp, Sym rkey, const char* resolved,
* flag, and sets *next_start to one past the winning dir (the resume point
* for a subsequent #include_next from the resolved file). Returns 0 if no
* configured dir holds the header. */
-static int search_inc_dirs(Pp* pp, const char* path, u32 start,
- const u8** data, size_t* size, char* resolved,
- size_t resolved_cap, int* resolved_system_out,
- u32* next_start) {
+static int search_inc_dirs(Pp* pp, const char* path, u32 start, const u8** data,
+ size_t* size, char* resolved, size_t resolved_cap,
+ int* resolved_system_out, u32* next_start) {
char buf[4096];
size_t plen = kit_slice_cstr(path).len;
u32 i;
@@ -922,9 +933,10 @@ static int search_inc_dirs(Pp* pp, const char* path, u32 start,
}
static int find_and_open_include(Pp* pp, const char* path, int system,
- SrcLoc loc, const u8** data, size_t* size,
+ LocRef loc, const u8** data, size_t* size,
char* resolved, size_t resolved_cap,
- int* resolved_system_out, u32* next_start_out) {
+ int* resolved_system_out,
+ u32* next_start_out) {
char buf[4096];
size_t plen = kit_slice_cstr(path).len;
Sym rkey = 0;
@@ -1011,17 +1023,22 @@ static int find_and_open_include_next(Pp* pp, const char* path, u32 start,
* capacity, and writes the unwrapped contents to out. `what` is the
* directive name used in panic messages (e.g. "#include", "#embed"). */
static void header_name_to_path(Pp* pp, KitSlice slc, char* out, size_t cap,
- int* system_out, SrcLoc loc, const char* what) {
+ int* system_out, LocRef loc, const char* what) {
const char* s = slc.s;
size_t slen = slc.len;
- if (slen < 2) compiler_panic(pp->c, loc, "%s: malformed header name", what);
+ if (slen < 2)
+ compiler_panic(pp->c, pp_materialize_loc(pp, loc),
+ "%s: malformed header name", what);
if (s[0] == '<' && s[slen - 1] == '>')
*system_out = 1;
else if (s[0] == '"' && s[slen - 1] == '"')
*system_out = 0;
else
- compiler_panic(pp->c, loc, "%s: malformed header name", what);
- if (slen - 2 + 1 > cap) compiler_panic(pp->c, loc, "%s: path too long", what);
+ compiler_panic(pp->c, pp_materialize_loc(pp, loc),
+ "%s: malformed header name", what);
+ if (slen - 2 + 1 > cap)
+ compiler_panic(pp->c, pp_materialize_loc(pp, loc), "%s: path too long",
+ what);
memcpy(out, s + 1, slen - 2);
out[slen - 2] = 0;
}
@@ -1030,12 +1047,14 @@ static void header_name_to_path(Pp* pp, KitSlice slc, char* out, size_t cap,
* - directly-lexed TOK_HEADER: < ... > or " ... "
* - macro-replaced form: line is macro-expanded, then expected to
* produce either a TOK_STR ("...") or a < ... > sequence. */
-static void parse_include_path(Pp* pp, const Tok* line, u32 n, SrcLoc loc,
+static void parse_include_path(Pp* pp, const Tok* line, u32 n, LocRef loc,
char* path_out, size_t cap, int* system_out) {
- if (n == 0) compiler_panic(pp->c, loc, "#include: missing path");
+ if (n == 0)
+ compiler_panic(pp->c, pp_materialize_loc(pp, loc),
+ "#include: missing path");
if (line[0].kind == TOK_HEADER) {
- KitSlice sl = kit_sym_str(pp->pool->c, line[0].spelling);
+ KitSlice sl = pp_text_slice(pp, &line[0]);
header_name_to_path(pp, sl, path_out, cap, system_out, loc, "#include");
return;
}
@@ -1048,36 +1067,41 @@ static void parse_include_path(Pp* pp, const Tok* line, u32 n, SrcLoc loc,
expand_arg_to_eof(pp, slice, NULL, n, &exp);
if (exp.n == 0) {
- compiler_panic(pp->c, loc, "#include: empty after macro replacement");
+ compiler_panic(pp->c, pp_materialize_loc(pp, loc),
+ "#include: empty after macro replacement");
}
if (exp.data[0].kind == TOK_STR) {
- KitSlice sl = kit_sym_str(pp->pool->c, exp.data[0].spelling);
+ KitSlice sl = pp_text_slice(pp, &exp.data[0]);
const char* s = sl.s;
size_t slen = sl.len;
if (slen < 2 || s[0] != '"' || s[slen - 1] != '"') {
- compiler_panic(pp->c, loc, "#include: malformed string");
+ compiler_panic(pp->c, pp_materialize_loc(pp, loc),
+ "#include: malformed string");
}
if (slen - 2 + 1 > cap) {
- compiler_panic(pp->c, loc, "#include: path too long");
+ compiler_panic(pp->c, pp_materialize_loc(pp, loc),
+ "#include: path too long");
}
memcpy(path_out, s + 1, slen - 2);
path_out[slen - 2] = 0;
*system_out = 0;
return;
}
- if (exp.data[0].kind == TOK_PUNCT && exp.data[0].v.punct == '<') {
+ if (exp.data[0].kind == TOK_PUNCT && tok_punct(&exp.data[0]) == '<') {
size_t pos = 0;
u32 i;
for (i = 1; i < exp.n; ++i) {
size_t slen = 0;
const char* s = NULL;
- if (exp.data[i].kind == TOK_PUNCT && exp.data[i].v.punct == '>') {
+ if (exp.data[i].kind == TOK_PUNCT && tok_punct(&exp.data[i]) == '>') {
break;
}
- if (exp.data[i].spelling) {
- KitSlice sl = kit_sym_str(pp->pool->c, exp.data[i].spelling);
- s = sl.s;
- slen = sl.len;
+ {
+ KitSlice sl = pp_text_slice(pp, &exp.data[i]);
+ if (sl.len) {
+ s = sl.s;
+ slen = sl.len;
+ }
}
if (s && pos + slen + 1 <= cap) {
memcpy(path_out + pos, s, slen);
@@ -1088,13 +1112,13 @@ static void parse_include_path(Pp* pp, const Tok* line, u32 n, SrcLoc loc,
*system_out = 1;
return;
}
- compiler_panic(pp->c, loc,
+ compiler_panic(pp->c, pp_materialize_loc(pp, loc),
"#include: expected \"...\" or <...> after expansion");
}
}
/* Shared core for #include and #include_next (is_next == 1). */
-static void do_include(Pp* pp, const Tok* line, u32 n, SrcLoc loc,
+static void do_include(Pp* pp, const Tok* line, u32 n, LocRef loc,
int is_next) {
char path[4096];
char resolved[4096];
@@ -1135,7 +1159,8 @@ static void do_include(Pp* pp, const Tok* line, u32 n, SrcLoc loc,
resolved, sizeof(resolved), &resolved_system,
&next_start);
if (!found) {
- compiler_panic(pp->c, loc, "%s: file not found: %.*s",
+ compiler_panic(pp->c, pp_materialize_loc(pp, loc),
+ "%s: file not found: %.*s",
is_next ? "#include_next" : "#include",
KIT_SLICE_ARG(kit_slice_cstr(path)));
}
@@ -1152,12 +1177,23 @@ static void do_include(Pp* pp, const Tok* line, u32 n, SrcLoc loc,
if (e && (e->once || (e->guard && mt_get(pp, e->guard) != NULL))) return;
}
- lex = lex_open_mem(pp->c, resolved, (const char*)data, size);
- included_id = lex_file_id(lex);
/* Inherit the parser-feed newline policy from the primary source: in cc mode
* an #include'd file's non-directive newlines are dropped at the lexer too
* (the parser never sees them); -E / cpp leaves them on. */
- if (pp->suppress_lexer_newlines) lex_set_emit_newlines(lex, 0);
+ {
+ SourceSpec spec;
+ memset(&spec, 0, sizeof spec);
+ spec.name = kit_slice_cstr(resolved);
+ spec.bytes = (const char*)data;
+ spec.len = (u32)size;
+ spec.flags = (pp->parser_feed ? SRC_PARSER_FEED : 0u) |
+ (resolved_system ? SRC_SYSTEM : 0u);
+ lex = lex_open(pp->c, &spec);
+ }
+ included_id = lex_file_id(lex);
+ /* Adopt the source buffer + splice table for lazy loc/text materialization
+ * (must happen before the source is pushed). */
+ pp_register_srcinfo(pp, lex);
memset(&s, 0, sizeof(s));
s.kind = SRC_LEX;
@@ -1167,7 +1203,8 @@ static void do_include(Pp* pp, const Tok* line, u32 n, SrcLoc loc,
s.guard_if_base = pp->ifstk_n;
src_push(pp, s);
- kit_source_add_include(pp->c, includer_id, included_id, loc, system_form,
+ kit_source_add_include(pp->c, includer_id, included_id,
+ pp_materialize_loc(pp, loc), system_form,
resolved_system);
}
@@ -1186,7 +1223,7 @@ TokSrc* current_lex_src(Pp* pp) {
return NULL;
}
-static void do_line(Pp* pp, const Tok* line, u32 n, SrcLoc loc) {
+static void do_line(Pp* pp, const Tok* line, u32 n, LocRef loc) {
/* Macro-replace arguments first (a2). */
TokVec exp = {0};
Tok* slice;
@@ -1194,26 +1231,30 @@ static void do_line(Pp* pp, const Tok* line, u32 n, SrcLoc loc) {
i64 target_line;
Sym target_file = 0;
- if (n == 0) compiler_panic(pp->c, loc, "#line: missing arguments");
+ if (n == 0)
+ compiler_panic(pp->c, pp_materialize_loc(pp, loc),
+ "#line: missing arguments");
slice = arena_array(pp->arena, Tok, n);
memcpy(slice, line, sizeof(Tok) * n);
expand_arg_to_eof(pp, slice, NULL, n, &exp);
if (exp.n == 0 || exp.data[0].kind != TOK_NUM) {
- compiler_panic(pp->c, loc, "#line: expected line number");
+ compiler_panic(pp->c, pp_materialize_loc(pp, loc),
+ "#line: expected line number");
}
{
- KitSlice s = kit_sym_str(pp->pool->c, exp.data[0].spelling);
+ KitSlice s = pp_text_slice(pp, &exp.data[0]);
target_line = parse_pp_int(s.s, s.len);
}
if (exp.n >= 2) {
if (exp.data[1].kind != TOK_STR) {
- compiler_panic(pp->c, loc, "#line: file argument must be a string");
+ compiler_panic(pp->c, pp_materialize_loc(pp, loc),
+ "#line: file argument must be a string");
}
{
- KitSlice s = kit_sym_str(pp->pool->c, exp.data[1].spelling);
+ KitSlice s = pp_text_slice(pp, &exp.data[1]);
if (s.len >= 2 && s.s[0] == '"' && s.s[s.len - 1] == '"') {
- /* Destringize to logical bytes (undo \" and \\): file_override is
+ /* Destringize to logical bytes (undo \" and \\): the overlay file is
* stored unescaped, like a real source path, and __FILE__ re-escapes
* it uniformly when expanded. */
char* fbuf = (char*)arena_alloc(pp->arena, s.len, 1);
@@ -1226,14 +1267,18 @@ static void do_line(Pp* pp, const Tok* line, u32 n, SrcLoc loc) {
}
lex_src = current_lex_src(pp);
- if (!lex_src) compiler_panic(pp->c, loc, "#line outside any file");
+ if (!lex_src)
+ compiler_panic(pp->c, pp_materialize_loc(pp, loc),
+ "#line outside any file");
{
- /* The next token (post-directive-NL) currently has lex.line ==
- * <lex's line counter>. Set delta so its user-visible line ==
- * target_line. */
- SrcLoc here = lex_loc(lex_src->lex);
- lex_src->line_delta = (i32)target_line - (i32)here.line;
- if (target_file) lex_src->file_override = target_file;
+ /* Record a positional #line overlay segment: from the cursor offset where
+ * the directive takes effect onward, the reported line is the physical line
+ * plus `delta`, and __FILE__ reports `target_file` (0 = none given). */
+ LocRef effect = lex_here(lex_src->lex);
+ u32 phys = pp_phys_line(pp, effect);
+ i32 delta = (i32)target_line - (i32)phys;
+ pp_add_line_seg(pp, lex_file_id(lex_src->lex), effect.off, delta,
+ target_file);
}
}
@@ -1244,7 +1289,7 @@ static void do_line(Pp* pp, const Tok* line, u32 n, SrcLoc loc) {
/* Push the unmodified directive line back onto the source stack as a
* buffer, so pp_emit_text writes it as-is. SRC_BUF gates directive
* recognition off, so this won't recurse. */
-void emit_pragma_line(Pp* pp, const Tok* line, u32 n, SrcLoc loc) {
+void emit_pragma_line(Pp* pp, const Tok* line, u32 n, LocRef loc) {
TokVec out = {0};
HidesetId* hids;
u32 i;
@@ -1254,15 +1299,16 @@ void emit_pragma_line(Pp* pp, const Tok* line, u32 n, SrcLoc loc) {
hash.kind = TOK_PP_HASH;
hash.flags = TF_AT_BOL;
hash.loc = loc;
- hash.spelling = kit_sym_intern(pp->pool->c, KIT_SLICE_LIT("#"));
+ hash.aux = '#';
+ hash.text = text_none_ref();
tv_push(pp, &out, hash);
memset(&ident, 0, sizeof(ident));
ident.kind = TOK_IDENT;
ident.flags = 0;
ident.loc = loc;
- ident.spelling = pp->sym_pragma_kw;
- ident.v.ident = pp->sym_pragma_kw;
+ ident.aux = pp->sym_pragma_kw;
+ ident.text = text_sym_ref(pp->sym_pragma_kw);
tv_push(pp, &out, ident);
for (i = 0; i < n; ++i) {
@@ -1291,7 +1337,7 @@ static int pragma_num_u32(Pp* pp, const Tok* t, u32* out) {
u32 v = 0;
KitSlice sl;
if (!t || t->kind != TOK_NUM || !out) return 0;
- sl = kit_sym_str(pp->pool->c, t->spelling);
+ sl = pp_text_slice(pp, t);
s = sl.s;
len = sl.len;
if (!s || len == 0) return 0;
@@ -1303,44 +1349,36 @@ static int pragma_num_u32(Pp* pp, const Tok* t, u32* out) {
return 1;
}
-static void handle_pragma_pack(Pp* pp, const Tok* line, u32 n, SrcLoc loc) {
+static void handle_pragma_pack(Pp* pp, const Tok* line, u32 n, LocRef loc) {
u32 i = 0;
if (n < 3 || line[0].kind != TOK_IDENT) return;
- {
- KitSlice sl = kit_sym_str(pp->pool->c, line[0].v.ident);
- const char* s = sl.s;
- size_t len = sl.len;
- if (!s || len != 4 || memcmp(s, "pack", 4) != 0) return;
- }
- if (line[1].kind != TOK_PUNCT || line[1].v.punct != '(') return;
+ if (!pp_text_eq_cstr(pp, &line[0], "pack")) return;
+ if (line[1].kind != TOK_PUNCT || tok_punct(&line[1]) != '(') return;
i = 2;
- if (i < n && line[i].kind == TOK_PUNCT && line[i].v.punct == ')') {
+ if (i < n && line[i].kind == TOK_PUNCT && tok_punct(&line[i]) == ')') {
pp->pack_align = 0;
return;
}
if (i < n && line[i].kind == TOK_IDENT) {
- KitSlice sl = kit_sym_str(pp->pool->c, line[i].v.ident);
- const char* s = sl.s;
- size_t len = sl.len;
- if (s && len == 4 && memcmp(s, "push", 4) == 0) {
+ if (pp_text_eq_cstr(pp, &line[i], "push")) {
if (pp->pack_stack_n <
(u32)(sizeof pp->pack_stack / sizeof pp->pack_stack[0])) {
pp->pack_stack[pp->pack_stack_n++] = pp->pack_align;
} else {
pp_warn(
- pp, loc,
+ pp, pp_materialize_loc(pp, loc),
"#pragma pack(push): pack stack overflow (max %u); push dropped",
(u32)(sizeof pp->pack_stack / sizeof pp->pack_stack[0]));
}
++i;
- if (i < n && line[i].kind == TOK_PUNCT && line[i].v.punct == ',') {
+ if (i < n && line[i].kind == TOK_PUNCT && tok_punct(&line[i]) == ',') {
u32 v = 0;
++i;
if (i < n && pragma_num_u32(pp, &line[i], &v)) pp->pack_align = v;
}
return;
}
- if (s && len == 3 && memcmp(s, "pop", 3) == 0) {
+ if (pp_text_eq_cstr(pp, &line[i], "pop")) {
if (pp->pack_stack_n) pp->pack_align = pp->pack_stack[--pp->pack_stack_n];
return;
}
@@ -1351,10 +1389,12 @@ static void handle_pragma_pack(Pp* pp, const Tok* line, u32 n, SrcLoc loc) {
}
}
-static void do_pragma(Pp* pp, const Tok* line, u32 n, SrcLoc loc) {
+static void do_pragma(Pp* pp, const Tok* line, u32 n, LocRef loc) {
/* #pragma once: mark the current file include-once for the multiple-include
- * optimization (see do_include). Still forwarded to output like any pragma. */
- if (n >= 1 && line[0].kind == TOK_IDENT && line[0].v.ident == pp->sym_once) {
+ * optimization (see do_include). Still forwarded to output like any pragma.
+ */
+ if (n >= 1 && line[0].kind == TOK_IDENT &&
+ tok_ident(&line[0]) == pp->sym_once) {
TokSrc* gls = current_lex_src(pp);
if (gls) gls->once = 1;
}
@@ -1370,12 +1410,12 @@ static void do_pragma(Pp* pp, const Tok* line, u32 n, SrcLoc loc) {
* does its own escape handling for any string literals nested inside. */
static void destringize(Pp* pp, const Tok* str_tok, char* out, size_t cap,
size_t* out_len) {
- KitSlice sl = kit_sym_str(pp->pool->c, str_tok->spelling);
+ KitSlice sl = pp_text_slice(pp, str_tok);
const char* s = sl.s;
size_t slen = sl.len;
size_t i, w = 0;
if (slen < 2 || s[0] != '"' || s[slen - 1] != '"') {
- compiler_panic(pp->c, str_tok->loc,
+ compiler_panic(pp->c, pp_materialize_loc(pp, str_tok->loc),
"_Pragma: argument must be a string literal");
}
for (i = 1; i + 1 < slen; ++i) {
@@ -1385,7 +1425,8 @@ static void destringize(Pp* pp, const Tok* str_tok, char* out, size_t cap,
c = s[i];
}
if (w + 1 >= cap)
- compiler_panic(pp->c, str_tok->loc, "_Pragma: payload too long");
+ compiler_panic(pp->c, pp_materialize_loc(pp, str_tok->loc),
+ "_Pragma: payload too long");
out[w++] = c;
}
out[w] = 0;
@@ -1417,14 +1458,16 @@ int try_expand_pragma_op(Pp* pp, const Tok* invoke) {
str = src_next_raw(pp, &hs, NULL);
}
if (str.kind != TOK_STR) {
- compiler_panic(pp->c, invoke->loc, "_Pragma: expected string literal");
+ compiler_panic(pp->c, pp_materialize_loc(pp, invoke->loc),
+ "_Pragma: expected string literal");
}
{
HidesetId hs;
rp = src_next_raw(pp, &hs, NULL);
}
- if (rp.kind != TOK_PUNCT || rp.v.punct != ')') {
- compiler_panic(pp->c, invoke->loc, "_Pragma: expected ')'");
+ if (rp.kind != TOK_PUNCT || tok_punct(&rp) != ')') {
+ compiler_panic(pp->c, pp_materialize_loc(pp, invoke->loc),
+ "_Pragma: expected ')'");
}
(void)lp;
@@ -1436,13 +1479,23 @@ int try_expand_pragma_op(Pp* pp, const Tok* invoke) {
/* Re-lex into args. Bytes need to live until lex_close; copy into
* arena. */
{
+ SourceSpec spec;
char* arena_buf = (char*)arena_alloc(pp->arena, buf_n + 1, 1);
memcpy(arena_buf, buf, buf_n + 1);
- lex = lex_open_mem(pp->c, "<_Pragma>", arena_buf, buf_n);
+ memset(&spec, 0, sizeof spec);
+ spec.name = kit_slice_cstr("<_Pragma>");
+ spec.bytes = arena_buf;
+ spec.len = (u32)buf_n;
+ lex = lex_open(pp->c, &spec);
}
for (;;) {
- Tok t = lex_next(lex);
+ Tok t;
+ lex_next(lex, &t);
if (t.kind == TOK_EOF || t.kind == TOK_NEWLINE) break;
+ /* These tokens outlive `lex`; a TEXT_SRC spelling points into the lexer's
+ * own buffer, so intern it now while the lexer is still open. */
+ if (t.text.kind == TEXT_SRC)
+ t.text = text_intern_ref(pp, lex_text_slice(lex, t.text));
tv_push(pp, &args, t);
}
lex_close(lex);
@@ -1458,8 +1511,7 @@ int try_expand_pragma_op(Pp* pp, const Tok* invoke) {
static void directive_message(Pp* pp, const Tok* line, u32 n, CharBuf* cb) {
u32 i;
for (i = 0; i < n; ++i) {
- KitSlice slc = line[i].spelling ? kit_sym_str(pp->pool->c, line[i].spelling)
- : KIT_SLICE_NULL;
+ KitSlice slc = pp_text_slice(pp, &line[i]);
const char* s = slc.s;
size_t sl = slc.len;
if (i > 0) cb_putc(pp, cb, ' ');
@@ -1479,17 +1531,17 @@ static void pp_warn(Pp* pp, SrcLoc loc, const char* fmt, ...) {
if (sink) sink->warnings++;
}
-static void do_error(Pp* pp, const Tok* line, u32 n, SrcLoc loc) {
+static void do_error(Pp* pp, const Tok* line, u32 n, LocRef loc) {
CharBuf cb = {0};
directive_message(pp, line, n, &cb);
- compiler_panic(pp->c, loc, "#error: %.*s",
+ compiler_panic(pp->c, pp_materialize_loc(pp, loc), "#error: %.*s",
KIT_SLICE_ARG(kit_slice_cstr(cb.data ? cb.data : "")));
}
-static void do_warning(Pp* pp, const Tok* line, u32 n, SrcLoc loc) {
+static void do_warning(Pp* pp, const Tok* line, u32 n, LocRef loc) {
CharBuf cb = {0};
directive_message(pp, line, n, &cb);
- pp_warn(pp, loc, "#warning: %.*s",
+ pp_warn(pp, pp_materialize_loc(pp, loc), "#warning: %.*s",
KIT_SLICE_ARG(kit_slice_cstr(cb.data ? cb.data : "")));
}
@@ -1497,7 +1549,7 @@ static void do_warning(Pp* pp, const Tok* line, u32 n, SrcLoc loc) {
* #embed (C23, §6.10.* per N3033)
* ============================================================ */
-static void do_embed(Pp* pp, const Tok* line, u32 n, SrcLoc loc) {
+static void do_embed(Pp* pp, const Tok* line, u32 n, LocRef loc) {
char path[4096];
char resolved[4096];
int system_form = 0;
@@ -1511,40 +1563,45 @@ static void do_embed(Pp* pp, const Tok* line, u32 n, SrcLoc loc) {
/* Header-name path: first token. */
u32 arg_start = 0;
- if (n == 0) compiler_panic(pp->c, loc, "#embed: missing path");
+ if (n == 0)
+ compiler_panic(pp->c, pp_materialize_loc(pp, loc), "#embed: missing path");
if (line[0].kind == TOK_HEADER) {
- KitSlice slc = kit_sym_str(pp->pool->c, line[0].spelling);
+ KitSlice slc = pp_text_slice(pp, &line[0]);
header_name_to_path(pp, slc, path, sizeof(path), &system_form, loc,
"#embed");
arg_start = 1;
} else {
- compiler_panic(pp->c, loc, "#embed: header-name argument required");
+ compiler_panic(pp->c, pp_materialize_loc(pp, loc),
+ "#embed: header-name argument required");
}
/* Parse trailing parameters: limit(N), if_empty(...). */
j = arg_start;
while (j < n) {
if (line[j].kind == TOK_IDENT) {
- KitSlice slc = kit_sym_str(pp->pool->c, line[j].v.ident);
+ KitSlice slc = pp_text_slice(pp, &line[j]);
const char* s = slc.s;
size_t sl = slc.len;
if (sl == 5 && memcmp(s, "limit", 5) == 0) {
if (j + 1 >= n || line[j + 1].kind != TOK_PUNCT ||
- line[j + 1].v.punct != '(') {
- compiler_panic(pp->c, loc, "#embed: expected '(' after limit");
+ tok_punct(&line[j + 1]) != '(') {
+ compiler_panic(pp->c, pp_materialize_loc(pp, loc),
+ "#embed: expected '(' after limit");
}
j += 2;
if (j >= n || line[j].kind != TOK_NUM) {
- compiler_panic(pp->c, loc, "#embed: limit() expects an integer");
+ compiler_panic(pp->c, pp_materialize_loc(pp, loc),
+ "#embed: limit() expects an integer");
}
{
- KitSlice s2 = kit_sym_str(pp->pool->c, line[j].spelling);
+ KitSlice s2 = pp_text_slice(pp, &line[j]);
limit_n = parse_pp_int(s2.s, s2.len);
}
++j;
- if (j >= n || line[j].kind != TOK_PUNCT || line[j].v.punct != ')') {
- compiler_panic(pp->c, loc, "#embed: expected ')' to close limit");
+ if (j >= n || line[j].kind != TOK_PUNCT || tok_punct(&line[j]) != ')') {
+ compiler_panic(pp->c, pp_materialize_loc(pp, loc),
+ "#embed: expected ')' to close limit");
}
++j;
continue;
@@ -1553,16 +1610,17 @@ static void do_embed(Pp* pp, const Tok* line, u32 n, SrcLoc loc) {
u32 depth = 0;
u32 start;
if (j + 1 >= n || line[j + 1].kind != TOK_PUNCT ||
- line[j + 1].v.punct != '(') {
- compiler_panic(pp->c, loc, "#embed: expected '(' after if_empty");
+ tok_punct(&line[j + 1]) != '(') {
+ compiler_panic(pp->c, pp_materialize_loc(pp, loc),
+ "#embed: expected '(' after if_empty");
}
j += 2;
start = j;
while (j < n) {
if (line[j].kind == TOK_PUNCT) {
- if (line[j].v.punct == '(')
+ if (tok_punct(&line[j]) == '(')
++depth;
- else if (line[j].v.punct == ')') {
+ else if (tok_punct(&line[j]) == ')') {
if (depth == 0) break;
--depth;
}
@@ -1570,7 +1628,8 @@ static void do_embed(Pp* pp, const Tok* line, u32 n, SrcLoc loc) {
++j;
}
if (j >= n) {
- compiler_panic(pp->c, loc, "#embed: unterminated if_empty");
+ compiler_panic(pp->c, pp_materialize_loc(pp, loc),
+ "#embed: unterminated if_empty");
}
if_empty_toks = arena_array(pp->arena, Tok, j - start ? j - start : 1);
if_empty_n = j - start;
@@ -1579,7 +1638,8 @@ static void do_embed(Pp* pp, const Tok* line, u32 n, SrcLoc loc) {
continue;
}
}
- compiler_panic(pp->c, loc, "#embed: unexpected token in parameter list");
+ compiler_panic(pp->c, pp_materialize_loc(pp, loc),
+ "#embed: unexpected token in parameter list");
}
{
@@ -1590,7 +1650,8 @@ static void do_embed(Pp* pp, const Tok* line, u32 n, SrcLoc loc) {
if (!find_and_open_include(pp, path, system_form, loc, &data, &size,
resolved, sizeof(resolved),
&embed_resolved_system, &embed_next_start)) {
- compiler_panic(pp->c, loc, "#embed: file not found: %.*s",
+ compiler_panic(pp->c, pp_materialize_loc(pp, loc),
+ "#embed: file not found: %.*s",
KIT_SLICE_ARG(kit_slice_cstr(path)));
}
}
@@ -1635,8 +1696,9 @@ static void do_embed(Pp* pp, const Tok* line, u32 n, SrcLoc loc) {
memset(&t, 0, sizeof(t));
t.kind = TOK_NUM;
t.loc = loc;
- t.spelling = kit_sym_intern(
- pp->pool->c, (KitSlice){.s = numbuf, .len = (size_t)nl});
+ t.aux = 0;
+ t.text =
+ text_intern_ref(pp, (KitSlice){.s = numbuf, .len = (size_t)nl});
if (i == 0) t.flags = TF_AT_BOL;
/* Bytes after a comma get a leading space to match
* clang's `, ` separator format. */
@@ -1648,9 +1710,9 @@ static void do_embed(Pp* pp, const Tok* line, u32 n, SrcLoc loc) {
Tok comma;
memset(&comma, 0, sizeof(comma));
comma.kind = TOK_PUNCT;
- comma.v.punct = ',';
+ comma.aux = ',';
comma.loc = loc;
- comma.spelling = kit_sym_intern(pp->pool->c, KIT_SLICE_LIT(","));
+ comma.text = text_none_ref();
tv_push(pp, &out, comma);
}
}
@@ -1668,7 +1730,7 @@ static void do_embed(Pp* pp, const Tok* line, u32 n, SrcLoc loc) {
* Directive dispatch
* ============================================================ */
-void process_directive(Pp* pp, SrcLoc hash_loc) {
+void process_directive(Pp* pp, LocRef hash_loc) {
Tok* line;
u32 n;
Sym name;
@@ -1679,9 +1741,10 @@ void process_directive(Pp* pp, SrcLoc hash_loc) {
return;
}
if (line[0].kind != TOK_IDENT) {
- compiler_panic(pp->c, line[0].loc, "expected directive name after '#'");
+ compiler_panic(pp->c, pp_materialize_loc(pp, line[0].loc),
+ "expected directive name after '#'");
}
- name = line[0].v.ident;
+ name = tok_ident(&line[0]);
/* Multiple-include-guard state machine, run on the current file source
* before dispatch. The only directive that may open a whole-file guard is
@@ -1695,7 +1758,7 @@ void process_directive(Pp* pp, SrcLoc hash_loc) {
if (gls->guard_state == GUARD_START) {
if (name == pp->sym_ifndef && n >= 2 && line[1].kind == TOK_IDENT &&
pp->ifstk_n == gls->guard_if_base) {
- gls->guard_macro = line[1].v.ident;
+ gls->guard_macro = tok_ident(&line[1]);
gls->guard_state = GUARD_IN;
} else {
gls->guard_state = GUARD_FAILED;
@@ -1737,7 +1800,8 @@ void process_directive(Pp* pp, SrcLoc hash_loc) {
else if (name == pp->sym_embed)
do_embed(pp, line + 1, n - 1, hash_loc);
else {
- compiler_panic(pp->c, line[0].loc, "unsupported directive");
+ compiler_panic(pp->c, pp_materialize_loc(pp, line[0].loc),
+ "unsupported directive");
}
/* The controlling #endif (the one returning the if-stack to the file's base
diff --git a/lang/cpp/pp/pp_expand.c b/lang/cpp/pp/pp_expand.c
@@ -3,8 +3,9 @@
#include "pp/pp_priv.h"
-static int body_tokens_equal(const Tok* a, u32 na, const Tok* b, u32 nb);
-static int macros_equal(const Macro* a, const Macro* b);
+static int body_tokens_equal(Pp* pp, const Tok* a, u32 na, const Tok* b,
+ u32 nb);
+static int macros_equal(Pp* pp, const Macro* a, const Macro* b);
/* ============================================================
* Hideset table
@@ -137,14 +138,15 @@ void do_define(Pp* pp, const Tok* line, u32 n) {
Macro* m;
u32 i = 0;
Sym name;
- SrcLoc def_loc;
+ LocRef def_loc;
Macro* existing;
if (i >= n || line[i].kind != TOK_IDENT) {
- compiler_panic(pp->c, n ? line[0].loc : (SrcLoc){0, 0, 0},
+ compiler_panic(pp->c,
+ n ? pp_materialize_loc(pp, line[0].loc) : (SrcLoc){0, 0, 0},
"#define: expected macro name");
}
- name = line[i].v.ident;
+ name = tok_ident(&line[i]);
def_loc = line[i].loc;
++i;
@@ -154,21 +156,21 @@ void do_define(Pp* pp, const Tok* line, u32 n) {
/* Function-like vs object-like: '(' immediately after the name with no
* intervening whitespace. */
- if (i < n && line[i].kind == TOK_PUNCT && line[i].v.punct == '(' &&
+ if (i < n && line[i].kind == TOK_PUNCT && tok_punct(&line[i]) == '(' &&
(line[i].flags & TF_HAS_SPACE) == 0) {
Sym* params = NULL;
u32 pcap = 0, pn = 0;
++i;
m->is_func = 1;
- if (i < n && line[i].kind == TOK_PUNCT && line[i].v.punct == ')') {
+ if (i < n && line[i].kind == TOK_PUNCT && tok_punct(&line[i]) == ')') {
++i;
} else {
for (;;) {
if (i >= n) {
- compiler_panic(pp->c, def_loc,
+ compiler_panic(pp->c, pp_materialize_loc(pp, def_loc),
"#define: unterminated parameter list");
}
- if (line[i].kind == TOK_PUNCT && line[i].v.punct == P_ELLIPSIS) {
+ if (line[i].kind == TOK_PUNCT && tok_punct(&line[i]) == P_ELLIPSIS) {
/* Append a synthetic __VA_ARGS__ param so body-rewrite
* matches the standard identifier directly. */
if (pn == pcap) {
@@ -189,7 +191,7 @@ void do_define(Pp* pp, const Tok* line, u32 n) {
params = nb;
pcap = nc;
}
- params[pn++] = line[i].v.ident;
+ params[pn++] = tok_ident(&line[i]);
++i;
/* GNU named variadic: `args...` — the named parameter itself collects
* the trailing arguments (the body refers to it by name rather than
@@ -199,30 +201,32 @@ void do_define(Pp* pp, const Tok* line, u32 n) {
* follows. Linux UAPI headers use this (e.g. <linux/stddef.h>'s
* __struct_group). */
if (i < n && line[i].kind == TOK_PUNCT &&
- line[i].v.punct == P_ELLIPSIS) {
+ tok_punct(&line[i]) == P_ELLIPSIS) {
m->is_variadic = 1;
++i;
}
} else {
- compiler_panic(pp->c, line[i].loc, "#define: bad parameter list");
+ compiler_panic(pp->c, pp_materialize_loc(pp, line[i].loc),
+ "#define: bad parameter list");
}
if (i >= n) {
- compiler_panic(pp->c, def_loc,
+ compiler_panic(pp->c, pp_materialize_loc(pp, def_loc),
"#define: unterminated parameter list");
}
- if (line[i].kind == TOK_PUNCT && line[i].v.punct == ')') {
+ if (line[i].kind == TOK_PUNCT && tok_punct(&line[i]) == ')') {
++i;
break;
}
if (m->is_variadic) {
- compiler_panic(pp->c, line[i].loc,
+ compiler_panic(pp->c, pp_materialize_loc(pp, line[i].loc),
"#define: '...' must be last parameter");
}
- if (line[i].kind == TOK_PUNCT && line[i].v.punct == ',') {
+ if (line[i].kind == TOK_PUNCT && tok_punct(&line[i]) == ',') {
++i;
continue;
}
- compiler_panic(pp->c, line[i].loc, "#define: expected ',' or ')'");
+ compiler_panic(pp->c, pp_materialize_loc(pp, line[i].loc),
+ "#define: expected ',' or ')'");
}
}
m->params = params;
@@ -234,7 +238,7 @@ void do_define(Pp* pp, const Tok* line, u32 n) {
if (name == pp->sym_defined || name == pp->sym_line__ ||
name == pp->sym_file__ || name == pp->sym_date__ ||
name == pp->sym_time__) {
- compiler_panic(pp->c, def_loc,
+ compiler_panic(pp->c, pp_materialize_loc(pp, def_loc),
"#define of a reserved / predefined name is not allowed");
}
/* Static predefineds are already in the macro table; redefining
@@ -248,7 +252,7 @@ void do_define(Pp* pp, const Tok* line, u32 n) {
* by checking whether it's already in the table — at pp_new the
* first call goes through cleanly. */
if (mt_get(pp, name)) {
- compiler_panic(pp->c, def_loc,
+ compiler_panic(pp->c, pp_materialize_loc(pp, def_loc),
"#define of a mandatory predefined macro is not allowed");
}
}
@@ -265,9 +269,12 @@ void do_define(Pp* pp, const Tok* line, u32 n) {
if (m->is_func && t.kind == TOK_IDENT) {
u32 p;
for (p = 0; p < m->n_params; ++p) {
- if (m->params[p] == t.v.ident) {
+ if (m->params[p] == tok_ident(&t)) {
+ /* Rewrite in place to TOK_PP_PARAM; leave `text` as the original
+ * TEXT_SRC span (the param-name spelling, needed by the
+ * redefinition compare). The parameter index lives in `aux`. */
t.kind = TOK_PP_PARAM;
- t.v.punct = p;
+ t.aux = p;
break;
}
}
@@ -275,8 +282,8 @@ void do_define(Pp* pp, const Tok* line, u32 n) {
/* §6.10.3 ¶5: __VA_ARGS__ outside a variadic macro is
* undefined behavior; we diagnose. */
if (!m->is_variadic && t.kind == TOK_IDENT &&
- t.v.ident == pp->sym_va_args) {
- compiler_panic(pp->c, t.loc,
+ tok_ident(&t) == pp->sym_va_args) {
+ compiler_panic(pp->c, pp_materialize_loc(pp, t.loc),
"__VA_ARGS__ may only appear in a variadic macro body");
}
m->body[j] = t;
@@ -289,8 +296,8 @@ void do_define(Pp* pp, const Tok* line, u32 n) {
existing = mt_get(pp, name);
if (existing) {
- if (!macros_equal(existing, m)) {
- compiler_panic(pp->c, def_loc,
+ if (!macros_equal(pp, existing, m)) {
+ compiler_panic(pp->c, pp_materialize_loc(pp, def_loc),
"macro redefined with different replacement");
}
return;
@@ -301,15 +308,16 @@ void do_define(Pp* pp, const Tok* line, u32 n) {
void do_undef(Pp* pp, const Tok* line, u32 n) {
Sym name;
if (!n || line[0].kind != TOK_IDENT) {
- compiler_panic(pp->c, n ? line[0].loc : (SrcLoc){0, 0, 0},
+ compiler_panic(pp->c,
+ n ? pp_materialize_loc(pp, line[0].loc) : (SrcLoc){0, 0, 0},
"#undef: expected identifier");
}
- name = line[0].v.ident;
+ name = tok_ident(&line[0]);
if (name == pp->sym_defined || name == pp->sym_line__ ||
name == pp->sym_file__ || name == pp->sym_date__ ||
name == pp->sym_time__ || name == pp->sym_stdc__ ||
name == pp->sym_stdc_hosted__ || name == pp->sym_stdc_version__) {
- compiler_panic(pp->c, line[0].loc,
+ compiler_panic(pp->c, pp_materialize_loc(pp, line[0].loc),
"#undef of a mandatory predefined name is not allowed");
}
mt_del(pp, name);
@@ -319,12 +327,23 @@ void do_undef(Pp* pp, const Tok* line, u32 n) {
* Body comparison helpers
* ============================================================ */
-static int body_tokens_equal(const Tok* a, u32 na, const Tok* b, u32 nb) {
+static int body_tokens_equal(Pp* pp, const Tok* a, u32 na, const Tok* b,
+ u32 nb) {
u32 i;
if (na != nb) return 0;
for (i = 0; i < na; ++i) {
+ KitSlice as, bs;
if (a[i].kind != b[i].kind) return 0;
- if (a[i].spelling != b[i].spelling) return 0;
+ /* Two body tokens are equal iff their exact spellings match (§6.10.3 ¶1).
+ * Compare the resolved bytes directly: a TOK_PP_PARAM keeps its original
+ * param-name span, so identical parameters compare equal byte-for-byte,
+ * and interned/source spellings compare the same way the old Sym-equality
+ * did (identical interned spelling == identical bytes). */
+ as = pp_text_slice(pp, &a[i]);
+ bs = pp_text_slice(pp, &b[i]);
+ if (as.len != bs.len || (as.len && memcmp(as.s, bs.s, as.len) != 0)) {
+ return 0;
+ }
/* Whitespace separation must match (§6.10.3 ¶2). The first body
* token's leading-space bit is meaningless (it's whatever was
* between macro name and body); skip i==0 for that bit. */
@@ -337,7 +356,7 @@ static int body_tokens_equal(const Tok* a, u32 na, const Tok* b, u32 nb) {
return 1;
}
-static int macros_equal(const Macro* a, const Macro* b) {
+static int macros_equal(Pp* pp, const Macro* a, const Macro* b) {
if (a->is_func != b->is_func) return 0;
if (a->is_variadic != b->is_variadic) return 0;
if (a->n_params != b->n_params) return 0;
@@ -347,7 +366,7 @@ static int macros_equal(const Macro* a, const Macro* b) {
if (a->params[i] != b->params[i]) return 0;
}
}
- return body_tokens_equal(a->body, a->body_len, b->body, b->body_len);
+ return body_tokens_equal(pp, a->body, a->body_len, b->body, b->body_len);
}
/* ============================================================
@@ -434,7 +453,7 @@ int peek_for_invoke_paren(Pp* pp, int* ws_has_space_out) {
return 0;
}
if (t.flags & TF_HAS_SPACE) saw_ws = 1;
- if (t.kind == TOK_PUNCT && t.v.punct == '(') {
+ if (t.kind == TOK_PUNCT && tok_punct(&t) == '(') {
/* Consumed. The newlines we walked past are whitespace and
* dropped (per spec); they don't go back on the stack. */
*ws_has_space_out = saw_ws;
@@ -465,7 +484,7 @@ void expand_arg_to_eof(Pp* pp, Tok* in, HidesetId* hs, u32 nin, TokVec* out) {
src_push(pp, src);
for (;;) {
- pp_next_raw_into(pp, &t); /* drives macro expansion within this scope */
+ pp_next_raw(pp, &t); /* drives macro expansion within this scope */
if (t.kind == TOK_EOF) break;
if (t.kind == TOK_NEWLINE) {
/* Newlines inside an arg act as whitespace; convert to
@@ -496,7 +515,7 @@ typedef struct ArgList {
/* Collect arguments. Caller has just consumed the opening `(`. Returns the
* close-paren's token (used as the invocation's last source location). */
-static Tok read_invocation_args(Pp* pp, const Macro* m, SrcLoc invoke_loc,
+static Tok read_invocation_args(Pp* pp, const Macro* m, LocRef invoke_loc,
ArgList* out) {
TokVec raw = {0};
HsVec raw_hs = {0};
@@ -518,7 +537,7 @@ static Tok read_invocation_args(Pp* pp, const Macro* m, SrcLoc invoke_loc,
for (;;) {
t = src_next_raw(pp, &hs, NULL);
if (t.kind == TOK_EOF) {
- compiler_panic(pp->c, invoke_loc,
+ compiler_panic(pp->c, pp_materialize_loc(pp, invoke_loc),
"unterminated function-like macro invocation");
}
if (t.kind == TOK_NEWLINE) {
@@ -529,7 +548,7 @@ static Tok read_invocation_args(Pp* pp, const Macro* m, SrcLoc invoke_loc,
}
if (t.kind == TOK_PUNCT) {
- u32 p = t.v.punct;
+ u32 p = tok_punct(&t);
if (p == '(') {
++depth;
} else if (p == ')') {
@@ -598,7 +617,7 @@ done:
if (n_args + 1 == (expected ? expected - 1 : 0)) {
/* off by one — fall through to error */
}
- compiler_panic(pp->c, invoke_loc,
+ compiler_panic(pp->c, pp_materialize_loc(pp, invoke_loc),
"too few arguments to variadic macro invocation");
}
/* Synthesize an empty __VA_ARGS__ if caller passed exactly
@@ -618,7 +637,7 @@ done:
if (n_args != expected) {
/* Spec: arity-0 macro `M()` invoked as `M()` is allowed and
* has 0 args. Above logic produces 0 in that case. */
- compiler_panic(pp->c, invoke_loc,
+ compiler_panic(pp->c, pp_materialize_loc(pp, invoke_loc),
"wrong number of arguments to function-like macro");
}
}
@@ -660,17 +679,15 @@ static void preexpand_args(Pp* pp, ArgList* a) {
* `arg[lo..hi)`. The first token's leading-space flag is ignored (leading
* whitespace stripped). Inside string/char-literal spellings, '"' and '\'
* are escaped. */
-static Tok make_stringize(Pp* pp, const Tok* arg, u32 lo, u32 hi, SrcLoc loc) {
+static Tok make_stringize(Pp* pp, const Tok* arg, u32 lo, u32 hi, LocRef loc) {
CharBuf b = {0};
u32 i;
Tok t;
- Sym sp;
cb_putc(pp, &b, '"');
for (i = lo; i < hi; ++i) {
const Tok* at = &arg[i];
- KitSlice sl =
- at->spelling ? kit_sym_str(pp->pool->c, at->spelling) : KIT_SLICE_NULL;
+ KitSlice sl = pp_text_slice(pp, at);
const char* s = sl.s;
size_t slen = sl.len;
if (i > lo && (at->flags & TF_HAS_SPACE)) cb_putc(pp, &b, ' ');
@@ -686,18 +703,17 @@ static Tok make_stringize(Pp* pp, const Tok* arg, u32 lo, u32 hi, SrcLoc loc) {
}
cb_putc(pp, &b, '"');
- sp = kit_sym_intern(pp->pool->c, (KitSlice){.s = b.data, .len = b.len});
memset(&t, 0, sizeof(t));
t.kind = TOK_STR;
+ t.aux = 0;
t.loc = loc;
- t.spelling = sp;
- t.v.str = sp;
+ t.text = text_intern_ref(pp, (KitSlice){.s = b.data, .len = b.len});
return t;
}
/* Concatenate two token spellings and re-lex into a single token. Empty
* (placemarker) sides collapse to the other side per §6.10.3.3 ¶2. */
-static Tok paste_tokens(Pp* pp, Tok lhs, Tok rhs, SrcLoc loc) {
+static Tok paste_tokens(Pp* pp, Tok lhs, Tok rhs, LocRef loc) {
char buf[1024];
size_t alen = 0, blen = 0;
const char* a;
@@ -708,33 +724,38 @@ static Tok paste_tokens(Pp* pp, Tok lhs, Tok rhs, SrcLoc loc) {
if (lhs.kind == TOK_PP_PLACEMARKER) return rhs;
if (rhs.kind == TOK_PP_PLACEMARKER) return lhs;
- if (lhs.spelling) {
- KitSlice s = kit_sym_str(pp->pool->c, lhs.spelling);
- a = s.s;
- alen = s.len;
- } else {
- a = "";
+ {
+ KitSlice s = pp_text_slice(pp, &lhs);
+ if (s.len) {
+ a = s.s;
+ alen = s.len;
+ } else {
+ a = "";
+ }
}
- if (rhs.spelling) {
- KitSlice s = kit_sym_str(pp->pool->c, rhs.spelling);
- b = s.s;
- blen = s.len;
- } else {
- b = "";
+ {
+ KitSlice s = pp_text_slice(pp, &rhs);
+ if (s.len) {
+ b = s.s;
+ blen = s.len;
+ } else {
+ b = "";
+ }
}
if (alen + blen + 2 > sizeof(buf)) {
- compiler_panic(pp->c, loc, "token paste: spelling too long");
+ compiler_panic(pp->c, pp_materialize_loc(pp, loc),
+ "token paste: spelling too long");
}
if (alen) memcpy(buf, a, alen);
if (blen) memcpy(buf + alen, b, blen);
buf[alen + blen] = '\n';
buf[alen + blen + 1] = 0;
- /* INVARIANT for lex_reset_mem's no-splice fast path: this buffer is two
- * already-folded interned token spellings (a, b) followed by exactly one
- * '\n'. A token spelling never ends in a lone '\\' immediately before that
- * '\n' (a backslash-newline would have been folded out at the original lex),
- * so the buffer is free of `\<newline>` line-splices and the paste lexer
- * (lex_reset_mem → lex_point_at_nosplice) can skip the splice scan. */
+ /* INVARIANT for the re-lex's no-splice fast path: this buffer is two
+ * already-folded token spellings (a, b) followed by exactly one '\n'. A
+ * token spelling never ends in a lone '\\' immediately before that '\n' (a
+ * backslash-newline would have been folded out at the original lex), so the
+ * buffer is free of `\<newline>` line-splices and the paste lexer is opened
+ * with SRC_NO_SPLICES to skip the splice scan. */
/* Intern the "<paste>" name once and reuse a single lexer across pastes: the
* lexer is re-pointed (not reallocated) per paste, but still draws a fresh
@@ -743,21 +764,35 @@ static Tok paste_tokens(Pp* pp, Tok lhs, Tok rhs, SrcLoc loc) {
if (!pp->paste_name_sym) {
pp->paste_name_sym = kit_sym_intern(pp->c, kit_slice_cstr("<paste>"));
}
- if (pp->paste_lex) {
- lex_reset_mem(pp->paste_lex, pp->paste_name_sym, buf, alen + blen + 1);
- } else {
- pp->paste_lex =
- lex_open_mem_sym(pp->c, pp->paste_name_sym, buf, alen + blen + 1);
+ {
+ SourceSpec spec;
+ memset(&spec, 0, sizeof spec);
+ spec.name_sym = pp->paste_name_sym;
+ spec.bytes = buf;
+ spec.len = (u32)(alen + blen + 1);
+ spec.flags = SRC_NO_SPLICES;
+ if (pp->paste_lex) {
+ lex_reset(pp->paste_lex, &spec);
+ } else {
+ pp->paste_lex = lex_open(pp->c, &spec);
+ }
}
lex = pp->paste_lex;
- t1 = lex_next(lex);
- t2 = lex_next(lex);
+ lex_next(lex, &t1);
+ lex_next(lex, &t2);
if (t1.kind == TOK_EOF) {
/* Both empty (shouldn't reach here since we handled placemarkers). */
return lhs;
}
if (t2.kind != TOK_NEWLINE && t2.kind != TOK_EOF) {
- compiler_panic(pp->c, loc, "token pasting yields multiple tokens, invalid");
+ compiler_panic(pp->c, pp_materialize_loc(pp, loc),
+ "token pasting yields multiple tokens, invalid");
+ }
+
+ /* The paste buffer is transient (a stack array re-pointed per paste), so a
+ * source-derived spelling must be interned before the buffer is reused. */
+ if (t1.text.kind == TEXT_SRC) {
+ t1.text = text_intern_ref(pp, lex_text_slice(lex, t1.text));
}
/* Inherit positional flags from LHS (it sat in the same slot). */
@@ -794,11 +829,11 @@ static void subst_phase1_impl(Pp* pp, const Macro* m, ArgList* a,
if (bt->kind == TOK_PP_HASH) {
/* §6.10.3.2: # must be followed by a parameter. */
if (j + 1 >= m->body_len || m->body[j + 1].kind != TOK_PP_PARAM) {
- compiler_panic(pp->c, bt->loc,
+ compiler_panic(pp->c, pp_materialize_loc(pp, bt->loc),
"'#' is not followed by a macro parameter");
}
{
- u32 p = m->body[j + 1].v.punct;
+ u32 p = m->body[j + 1].aux;
u32 lo = a->raw_start[p];
u32 hi = a->raw_start[p + 1];
Tok s = make_stringize(pp, a->raw, lo, hi, invoke->loc);
@@ -814,7 +849,7 @@ static void subst_phase1_impl(Pp* pp, const Macro* m, ArgList* a,
}
}
if (bt->kind == TOK_PP_PARAM) {
- u32 p = bt->v.punct;
+ u32 p = bt->aux;
int adj_paste =
(j > 0 && m->body[j - 1].kind == TOK_PP_PASTE) ||
(j + 1 < m->body_len && m->body[j + 1].kind == TOK_PP_PASTE);
@@ -903,7 +938,7 @@ static void subst_phase2(Pp* pp, const Tok* in, u32 nin, const Tok* invoke,
if (t.kind == TOK_PP_PASTE) {
Tok lhs, rhs, pasted;
if (out->n == 0 || i + 1 >= nin) {
- compiler_panic(pp->c, invoke->loc,
+ compiler_panic(pp->c, pp_materialize_loc(pp, invoke->loc),
"'##' at start or end of replacement list");
}
lhs = out->data[--out->n];
@@ -1006,7 +1041,7 @@ static int try_expand_func_macro(Pp* pp, const Macro* m, const Tok* invoke,
* Defined here; also declared in pp_priv.h so pp.c can call it.
* ============================================================ */
-/* pp_pull_into: shared core of pp_next_raw_into / pp_next_into. Reads from the
+/* pp_pull_into: shared core of pp_next_raw / pp_next_into. Reads from the
* top source into *out, applies macro expansion when an identifier names a
* macro that isn't blue-painted, and consumes directives in-place.
*
@@ -1049,8 +1084,7 @@ static void pp_pull_into(Pp* pp, Tok* out, int skip_nl) {
* guard, so the dead reset on plain-token runs is elided -- byte-identical
* since resetting an already-pristine arena does nothing. */
if (!pp->in_if_expansion &&
- (pp->nsources == 0 ||
- pp->sources[pp->nsources - 1].kind == SRC_LEX)) {
+ (pp->nsources == 0 || pp->sources[pp->nsources - 1].kind == SRC_LEX)) {
if (!kit_arena_is_empty(pp->xarena)) kit_arena_reset(pp->xarena);
}
/* Fast path: top source is a non-exhausted SRC_BUF (the dominant
@@ -1059,8 +1093,7 @@ static void pp_pull_into(Pp* pp, Tok* out, int skip_nl) {
* scope_top-EOF and #line-delta cases never apply to a plain in-bounds
* SRC_BUF read, so they stay on the general (cold) path. */
if (pp->nsources != 0 &&
- (s = &pp->sources[pp->nsources - 1])->kind == SRC_BUF &&
- s->i < s->n) {
+ (s = &pp->sources[pp->nsources - 1])->kind == SRC_BUF && s->i < s->n) {
*out = s->toks[s->i];
hs = s->hs ? s->hs[s->i] : s->hs_uniform;
src_kind = SRC_BUF;
@@ -1120,7 +1153,7 @@ static void pp_pull_into(Pp* pp, Tok* out, int skip_nl) {
* the `defined_skip` field comment in pp_priv.h. */
if (pp->in_if_expansion) {
if (pp->defined_skip == 2) {
- if (out->kind == TOK_PUNCT && out->v.punct == '(') {
+ if (out->kind == TOK_PUNCT && tok_punct(out) == '(') {
pp->defined_skip = 3;
} else if (out->kind == TOK_IDENT) {
/* `defined IDENT` (no parens): mark the operand and reset. */
@@ -1133,19 +1166,19 @@ static void pp_pull_into(Pp* pp, Tok* out, int skip_nl) {
if (out->kind == TOK_IDENT) {
out->flags |= TF_NO_EXPAND;
pp->defined_skip = 4;
- } else if (out->kind == TOK_PUNCT && out->v.punct == ')') {
+ } else if (out->kind == TOK_PUNCT && tok_punct(out) == ')') {
pp->defined_skip = 0;
}
} else if (pp->defined_skip == 4) {
- if (out->kind == TOK_PUNCT && out->v.punct == ')') {
+ if (out->kind == TOK_PUNCT && tok_punct(out) == ')') {
pp->defined_skip = 0;
}
- } else if (out->kind == TOK_IDENT && out->v.ident == pp->sym_defined) {
+ } else if (out->kind == TOK_IDENT && tok_ident(out) == pp->sym_defined) {
pp->defined_skip = 2;
}
}
if (out->kind == TOK_IDENT && (out->flags & TF_NO_EXPAND) == 0) {
- Sym id = out->v.ident;
+ Sym id = tok_ident(out);
/* Dynamic predefined macros: __LINE__ / __FILE__ /
* __DATE__ / __TIME__. Always expand, ignoring the macro
@@ -1153,7 +1186,7 @@ static void pp_pull_into(Pp* pp, Tok* out, int skip_nl) {
if (id == pp->sym_line__) {
char tmp[16], buf[16];
int k = 0, j = 0;
- u32 ln = out->loc.line;
+ u32 ln = pp_materialize_loc(pp, out->loc).line;
if (ln == 0)
buf[k++] = '0';
else {
@@ -1164,25 +1197,17 @@ static void pp_pull_into(Pp* pp, Tok* out, int skip_nl) {
while (j > 0) buf[k++] = tmp[--j];
}
out->kind = TOK_NUM;
- out->spelling =
- kit_sym_intern(pp->pool->c, (KitSlice){.s = buf, .len = (size_t)k});
+ out->aux = 0;
+ out->text = text_intern_ref(pp, (KitSlice){.s = buf, .len = (size_t)k});
return;
}
if (id == pp->sym_file__) {
- TokSrc* ls = current_lex_src(pp);
- Sym name = 0;
+ /* pp_materialize_file resolves the __FILE__ name at this location,
+ * applying any active #line overlay; no current-lexer probe needed. */
+ Sym name = pp_materialize_file(pp, out->loc);
size_t nlen = 0;
const char* nstr = NULL;
char* buf;
- if (ls && ls->file_override) {
- name = ls->file_override;
- } else if (ls) {
- KitSourceFile sf;
- memset(&sf, 0, sizeof(sf));
- if (kit_source_file(pp->c, lex_file_id(ls->lex), &sf) == 0) {
- name = sf.name;
- }
- }
if (name) {
KitSlice s = kit_sym_str(pp->pool->c, name);
nstr = s.s;
@@ -1206,22 +1231,21 @@ static void pp_pull_into(Pp* pp, Tok* out, int skip_nl) {
}
buf[bn++] = '"';
out->kind = TOK_STR;
- out->spelling =
- kit_sym_intern(pp->pool->c, (KitSlice){.s = buf, .len = bn});
- out->v.str = out->spelling;
+ out->aux = 0;
+ out->text = text_intern_ref(pp, (KitSlice){.s = buf, .len = bn});
}
return;
}
if (id == pp->sym_date__) {
out->kind = TOK_STR;
- out->spelling = pp->val_date_str;
- out->v.str = out->spelling;
+ out->aux = 0;
+ out->text = text_sym_ref(pp->val_date_str);
return;
}
if (id == pp->sym_time__) {
out->kind = TOK_STR;
- out->spelling = pp->val_time_str;
- out->v.str = out->spelling;
+ out->aux = 0;
+ out->text = text_sym_ref(pp->val_time_str);
return;
}
if (id == pp->sym__pragma) {
@@ -1247,20 +1271,14 @@ static void pp_pull_into(Pp* pp, Tok* out, int skip_nl) {
}
}
-/* pp_next_raw_into: out-pointer form, every token surfaced (TOK_NEWLINE
- * preserved for pp_emit_text and for directive/pragma line termination). */
-void pp_next_raw_into(Pp* pp, Tok* out) { pp_pull_into(pp, out, /*skip_nl=*/0); }
+/* pp_next_raw: public out-pointer form (pp.h), every token surfaced
+ * (TOK_NEWLINE preserved for pp_emit_text and for directive/pragma line
+ * termination). The mutual-recursion entry: expand_arg_to_eof and the -E loop
+ * call it; it drives directives and expansion. */
+void pp_next_raw(Pp* pp, Tok* out) { pp_pull_into(pp, out, /*skip_nl=*/0); }
-/* pp_next_into: out-pointer form for the C-parser feed (pp_next). Identical to
- * pp_next_raw_into but drops non-directive newlines internally, so the 51% of
- * produced tokens that pp_next would otherwise discard never escape this frame.
- * Declared locally (extern) in pp.c — kept out of pp_priv.h on purpose. */
+/* pp_next_into: out-pointer form for the C-parser feed (pp_next_parse).
+ * Identical to pp_next_raw but drops non-directive newlines internally, so the
+ * 51% of produced tokens that pp_next_parse would otherwise discard never
+ * escape this frame. Declared in pp_priv.h. */
void pp_next_into(Pp* pp, Tok* out) { pp_pull_into(pp, out, /*skip_nl=*/1); }
-
-/* Thin by-value shim: the mutual-recursion entry (expand_arg_to_eof) and any
- * external caller use this; the hot -E loops call pp_next_raw_into directly. */
-Tok pp_next_raw(Pp* pp) {
- Tok t;
- pp_next_raw_into(pp, &t);
- return t;
-}
diff --git a/lang/cpp/pp/pp_priv.h b/lang/cpp/pp/pp_priv.h
@@ -5,11 +5,10 @@
#ifndef KIT_PP_PRIV_H
#define KIT_PP_PRIV_H
+#include <kit/support/symtab.h>
#include <stdlib.h>
#include <string.h>
-#include <kit/support/symtab.h>
-
#include "cpp_support.h"
#include "pp/pp.h"
@@ -27,7 +26,7 @@
typedef struct Macro {
Sym name;
- SrcLoc def_loc;
+ LocRef def_loc;
u8 is_func;
u8 is_variadic;
/* Pure cache of the body (NOT part of macro identity): 1 iff the body
@@ -37,7 +36,7 @@ typedef struct Macro {
u8 pad[1];
u32 n_params;
Sym* params; /* parameter names */
- Tok* body; /* body tokens; TOK_PP_PARAM kind + v.punct=idx */
+ Tok* body; /* body tokens; TOK_PP_PARAM kind + aux=param idx */
u32 body_len;
} Macro;
@@ -51,7 +50,8 @@ typedef u32 HidesetId;
typedef struct Hideset {
u32 n;
- u32 hash; /* content hash of names[0..n) — dedup index key (see hs_register) */
+ u32 hash; /* content hash of names[0..n) — dedup index key (see hs_register)
+ */
Sym names[1]; /* flexible; allocated with extra trailing slots */
} Hideset;
@@ -102,15 +102,12 @@ typedef struct TokSrc {
HidesetId* hs;
HidesetId hs_uniform;
/* Per-token loc applied when has_loc_override (see scope_top block). */
- SrcLoc loc_override;
+ LocRef loc_override;
u32 i;
u32 n;
- /* #line state (SRC_LEX only). line_delta is added to every emitted
- * token's loc.line on its way out so __LINE__ and the output cursor
- * see user-visible numbering. file_override is the Sym (without
- * surrounding quotes) used by __FILE__ when set. */
- i32 line_delta;
- Sym file_override;
+ /* #line state is recorded positionally in the source's SrcInfo overlay
+ * segments (see SrcInfo / pp_materialize_loc), not on the frame: a lean
+ * token carries only a byte offset, and line/file are resolved on demand. */
/* SRC_LEX only: the index into pp->inc_dirs from which a `#include_next`
* (or `__has_include_next`) appearing in this file begins its search —
* i.e. one past the search dir this file was itself found in. 0 for the
@@ -139,15 +136,52 @@ typedef struct IfFrame {
u8 state;
u8 has_else;
u8 pad[2];
- SrcLoc loc;
+ LocRef loc;
} IfFrame;
+/* ============================================================
+ * Source-info registry (lazy loc + retained text)
+ * ============================================================ */
+
+/* A #line overlay segment: from byte offset `off` onward (until the next
+ * segment), the reported line number is the physical line plus `delta`, and
+ * __FILE__ reports `file` (0 = the source's registry name). Recorded by do_line
+ * in offset order; consulted by pp_materialize_loc / pp_materialize_file. */
+typedef struct LineSeg {
+ u32 off;
+ i32 delta;
+ Sym file;
+} LineSeg;
+
+/* Per-source retained metadata, keyed by file_id. The folded logical buffer and
+ * splice table are retained until pp_free so that (a) TEXT_SRC spellings stay
+ * resolvable after the lexer is popped (macro bodies, parser lookahead/replay),
+ * and (b) line/col can be reconstructed lazily from a byte offset. The line
+ * index is built on first materialization by scanning the buffer for '\n' and
+ * merging the splice fold points. */
+typedef struct SrcInfo {
+ const char* buf; /* folded logical buffer (borrowed/arena/owned) */
+ u32 len;
+ u8 owns_buf; /* PP frees buf at pp_free (a folded heap copy) */
+ u8 line_built; /* line_off has been computed */
+ u32 shebang_off;
+ u32* splices; /* fold offsets; PP frees at pp_free */
+ u32 nsplices;
+ u32* line_off; /* line_off[k] = start offset of physical line k+1 */
+ u32 nlines;
+ u32 line_cursor; /* last line index returned — loc materialization is
+ * near-monotonic, so this makes the common lookup O(1) */
+ LineSeg* segs; /* #line overlay, offset-ordered */
+ u32 nsegs;
+ u32 segs_cap;
+} SrcInfo;
+
/* Sym-keyed hashmaps below (include caches). See core/hashmap.h. */
#include <kit/support/hashmap.h>
/* IncCache = Sym(resolved-path) -> cached header bytes. The same guarded
* header included many times across a TU is read from disk once; later
- * inclusions reuse the arena-resident bytes (lex_open_mem only borrows
+ * inclusions reuse the arena-resident bytes (lex_open only borrows
* them and the pp arena keeps them alive until pp_free), saving the
* open+fstat+read+close + malloc/memcpy/free per repeat. Keyed strictly
* on the resolved-path STRING so distinct spellings/symlinks never
@@ -172,8 +206,8 @@ KIT_HASHMAP_DEFINE(IncCache, Sym, IncEntry, inc_hash_);
* exact one a fresh search would produce — byte-identical resolved path
* and resolved_system flag feeding source_add_include / DWARF / -M. */
typedef struct IncResolved {
- Sym path; /* interned resolved path string */
- u8 system; /* the resolved_system flag (winning -isystem) */
+ Sym path; /* interned resolved path string */
+ u8 system; /* the resolved_system flag (winning -isystem) */
u32 next_start; /* inc_dirs index a #include_next from the */
/* resolved file starts at (see TokSrc) */
} IncResolved;
@@ -193,6 +227,12 @@ struct Pp {
u32 nsources;
u32 sources_cap;
+ /* Per-file retained metadata, indexed by compiler file_id (dense from 1).
+ * Holds the retained source buffer + splice table + lazy line index + #line
+ * overlay so loc/text materialize after the lexer is popped. */
+ SrcInfo* srcinfo;
+ u32 srcinfo_cap;
+
/* Macro table (Sym-indexed dense; value = Macro*, NULL = not a macro). */
MacroTab macros;
@@ -325,12 +365,12 @@ struct Pp {
* doesn't count directive-internal tokens (e.g. the `ifndef` / macro name of
* the guard itself) as file content. */
u8 reading_directive;
- /* cc parser-feed mode: when set, every SRC_LEX lexer (the primary source and
- * each #include'd file) is opened with lex_set_emit_newlines(lex, 0) so
- * non-directive newlines are suppressed at the lexer (the parser drops them
- * anyway). The -E / cpp path leaves this 0 so newlines surface for text
- * reconstruction. Set via pp_set_suppress_lexer_newlines after pp_new. */
- u8 suppress_lexer_newlines;
+ /* cc parser-feed mode: set when the primary source is pushed with
+ * SRC_PARSER_FEED, so every SRC_LEX lexer (primary + each #include) is opened
+ * with newline suppression (the parser drops non-directive newlines). The
+ * -E / cpp path leaves this clear so newlines surface for text
+ * reconstruction. */
+ u8 parser_feed;
};
/* ============================================================
@@ -354,6 +394,16 @@ static inline void pp_xfree(Pp* pp, void* p, size_t n) {
}
/* ============================================================
+ * Lean-token constructors (synthetic tokens)
+ * ============================================================ */
+
+/* text_none_ref / text_sym_ref / locref_none live in lex.h (shared with the
+ * parser). text_intern_ref additionally interns, so it needs the Pp pool. */
+static inline TextRef text_intern_ref(Pp* pp, KitSlice sl) {
+ return text_sym_ref(kit_sym_intern(pp->pool->c, sl));
+}
+
+/* ============================================================
* Token-vector helpers
* ============================================================ */
@@ -436,8 +486,8 @@ static inline void cb_putc(Pp* pp, CharBuf* b, char c) {
* ============================================================ */
/* --- pp.c (source stack) → pp_expand.c, pp_directive.c --- */
-/* Out-pointer form (hot path, no 24B sret round-trip); src_next_raw is the
- * by-value shim over it for the cold/general callers. */
+/* Out-pointer form (hot path); src_next_raw is the by-value shim over it for
+ * the cold/general callers. */
void src_next_raw_into(Pp* pp, Tok* out, HidesetId* hs_out, u8* src_kind_out);
Tok src_next_raw(Pp* pp, HidesetId* hs_out, u8* src_kind_out);
void src_push(Pp* pp, TokSrc s);
@@ -446,17 +496,30 @@ void push_buf(Pp* pp, Tok* toks, HidesetId* hs, u32 n);
void push_buf_uniform(Pp* pp, Tok* toks, u32 n, HidesetId hs_uniform);
/* Push an immutable, read-only token buffer (e.g. a no-`##` macro body) for
* pointer-replay. The buffer is never written through s->toks; per-read
- * loc/flags overrides reproduce the mutations the old body-copy path baked in. */
+ * loc/flags overrides reproduce the mutations the old body-copy path baked in.
+ */
void push_buf_replay(Pp* pp, const Tok* toks, u32 n, HidesetId hs_uniform,
- SrcLoc loc_override, u16 first_flags_or);
-
-/* pp_next_raw is the mutual-recursion entry: expand_arg_to_eof calls it,
- * and pp_next_raw drives directives and expansion. Declared non-static so
- * pp_expand.c can call it without a forward decl each time. pp_next_raw_into
- * is the out-pointer form the hot -E loops call directly; pp_next_raw is the
- * by-value shim. */
-void pp_next_raw_into(Pp* pp, Tok* out);
-Tok pp_next_raw(Pp* pp);
+ LocRef loc_override, u16 first_flags_or);
+
+/* pp_next_raw (public, in pp.h) is the mutual-recursion entry:
+ * expand_arg_to_eof and the -E loop call it; it drives directives and
+ * expansion. pp_next_into is the internal newline-dropping pull (skip_nl=1)
+ * used by pp_next_parse. */
+void pp_next_into(Pp* pp, Tok* out);
+
+/* --- SrcInfo registry (pp.c) → pp_directive.c --- */
+/* Get (growing as needed) the SrcInfo for a file_id. */
+SrcInfo* pp_srcinfo(Pp* pp, u32 file_id);
+/* Register a freshly-opened, about-to-be-pushed lexer: adopt its folded buffer
+ * + splice table + shebang offset into the SrcInfo registry (retained to
+ * pp_free) so loc/text stay materializable after the lexer is popped. */
+void pp_register_srcinfo(Pp* pp, Lexer* lex);
+/* Record a #line overlay segment (effect offset, line delta, file name sym). */
+void pp_add_line_seg(Pp* pp, u32 file_id, u32 off, i32 delta, Sym file);
+/* Physical line at a location, ignoring #line overlay (used by do_line). */
+u32 pp_phys_line(Pp* pp, LocRef loc);
+/* __FILE__ name sym at a location (overlay file if set, else registry name). */
+Sym pp_materialize_file(Pp* pp, LocRef loc);
/* --- pp_expand.c → pp.c, pp_directive.c --- */
HidesetId hs_add(Pp* pp, HidesetId id, Sym s);
@@ -477,11 +540,11 @@ static inline void mt_del(Pp* pp, Sym name) {
}
/* --- pp_directive.c → pp_expand.c --- */
-i64 eval_if_expr(Pp* pp, const Tok* line, u32 n, SrcLoc loc);
-void process_directive(Pp* pp, SrcLoc hash_loc);
+i64 eval_if_expr(Pp* pp, const Tok* line, u32 n, LocRef loc);
+void process_directive(Pp* pp, LocRef hash_loc);
/* --- pp_directive.c internal helpers called from pp_expand.c --- */
-void emit_pragma_line(Pp* pp, const Tok* line, u32 n, SrcLoc loc);
+void emit_pragma_line(Pp* pp, const Tok* line, u32 n, LocRef loc);
int peek_for_invoke_paren(Pp* pp, int* ws_has_space_out);
int try_expand_pragma_op(Pp* pp, const Tok* invoke);
