commit f606edf7a5d3cbcd8ce1ac62dd8586940f763c6e
parent fe38baabffe432508c64e96d49fd1beba7370345
Author: Ryan Sepassi <rsepassi@gmail.com>
Date: Fri, 12 Jun 2026 17:37:10 -0700
doc(perf): refresh sqlite phase timings + self-time map after scanner rewrite
Re-profiled -c on a fresh PROFILE=1 build (80-run merged sample, 1467 samples)
and updated the phase split so the investigation map reflects the landed
raw-cursor scanner.
Phase split (instructions, best-of-7): -E 0.98 B, -fsyntax-only 1.85 B,
-c 2.22 B; codegen+emit+object-write is ~17%.
Self-time map (post-rewrite): lex_next 62.2%, pool_intern_slice 14.4%,
_platform_memset 8.2%, pp_next_raw_into 5.0%, finish_ident 3.7%,
src_next_raw_into 3.0%, fsync 1.8% (wall-time I/O, ~0 instructions), codegen
~1.6%, memmove 0.3%. The scanner (lex_next + finish_ident) is ~66% and is now
*more* concentrated than the pre-rewrite 58% — dispatch/classification is one
table load, so the residual is the irreducible per-byte cursor walk + the
per-token interning handoff. Next lever: pool_intern_slice, then parse/codegen
memset.
Diffstat:
1 file changed, 25 insertions(+), 19 deletions(-)
diff --git a/doc/plan/PERF.md b/doc/plan/PERF.md
@@ -137,34 +137,40 @@ build/release/kit cc sqlite3.c shell.c -o /tmp/sq --sysroot "$SDK" -lc
## Current state
**Real-world compilation is frontend-bound, not codegen-bound.** The phase split
-(instructions: `-E` 1.06 B, `-c` 2.43 B) puts native codegen + emit + object-write
-at ~18 %; the frontend — preprocessor, lexer, interner, parser, semantic analysis,
+(instructions: `-E` 0.98 B, `-c` 2.22 B) puts native codegen + emit + object-write
+at ~17 %; the frontend — preprocessor, lexer, interner, parser, semantic analysis,
and the CG value-stack/type-lowering it drives — is the rest. (The synthetic
`bench-cc` axes below over-weight codegen by construction; trust the sqlite profile
for where to spend effort.)
### Where the time goes (self-time, real sqlite `-c`)
-80-run merged `sample` on a `PROFILE=1` build (1809 samples). The profile is
-**single-peaked**: the scanner dominates and everything downstream of it is flat.
+80-run merged `sample` on a `PROFILE=1` build (1467 samples), **after the
+raw-cursor scanner rewrite**. The profile is **single-peaked**: the scanner
+still dominates and everything downstream of it is flat. (`sample` is
+wall-clock self-time, which tracks the instruction metric for CPU-bound code
+but *not* for blocking I/O — see the `fsync` row.)
| function | self % | stage | note |
|---|--:|---|---|
-| `lex_next` | **58.0** | lexer (the scanner) | **the frontier** |
-| `pool_intern_slice` | 14.8 | identifier interning | identifiers (puncts cached) |
-| `_platform_memset` | 7.0 | zeroing | callers are **parse + codegen**, not the lexer |
-| `src_next_raw_into` | 4.0 | preprocessor source stack | |
-| `pp_next_raw_into` | 3.5 | preprocessor token pump | |
-| `_platform_memmove` | 2.7 | buffer moves | callers parse/codegen (`m_emit_bytes`, cg) |
-| `scan_pp_number` | 1.4 | lexer (number scan) | inlined into the scanner lever |
-| codegen (`nd_dst_reg`/`cg_type_get`/`aa_emit_mem`/…) | ~6 total | native emit | each <1.5 % |
-
-`lex_next` plus its inlined helpers (`scan_pp_number`, `bump`, …) is ~60 % of
-self-time; the gap to tcc is now almost entirely *the scanner loop itself*. Three
-things that used to be hot are absent from the profile: the **type system**
-(derived-type + ABI/record dedup are O(1)), the **preprocessor hideset** (O(1)
-content-addressed dedup), and **guarded-header re-lexing** (multiple-include
-optimization).
+| `lex_next` | **62.2** | lexer (the scanner) | dispatch + ws/comment skip + number/string/punct scan + the per-token intern calls (`scan_pp_number`/`scan_quoted`/`skip_ws_fast` inlined in) |
+| `pool_intern_slice` | 14.4 | identifier interning | one per ident/number; puncts cached |
+| `_platform_memset` | 8.2 | zeroing | callers are **parse + codegen**, not the lexer |
+| `pp_next_raw_into` | 5.0 | preprocessor token pump | |
+| `finish_ident` | 3.7 | lexer (identifier scan) | part of the scanner lever (`scan_ident_run` inlined in) |
+| `src_next_raw_into` | 3.0 | preprocessor source stack | |
+| `fsync` | 1.8 | object write | flushing the `.o` to disk — **wall-time I/O, ≈0 retired instructions** (irrelevant to the instruction metric) |
+| codegen (`nd_dst_reg`/`cg_type_is_aggregate`/`api_unalias_type`/…) | ~1.6 total | native emit | each <1 % |
+| `_platform_memmove` | 0.3 | buffer moves | callers parse/codegen |
+
+The scanner (`lex_next` + `finish_ident`) is **~66 %** of self-time — still THE
+lever by a wide margin, and *more* concentrated than before the rewrite (the
+dispatch/classification is now one table load, so the cost is the irreducible
+per-byte cursor walk plus the per-token interning handoff). The next lever is
+`pool_intern_slice` (interning), then the parse/codegen `memset`. Three things
+that used to be hot remain absent: the **type system** (derived-type + ABI/record
+dedup are O(1)), the **preprocessor hideset** (O(1) content-addressed dedup), and
+**guarded-header re-lexing** (multiple-include optimization).
### Resolved — already optimal, don't re-propose
