Benchmarking compile speed and code size

This is the reference for how to measure kit's two performance axes — compile+link throughput and emitted code size — reproducibly. It is the methodology a contributor reuses to attribute a regression, validate a change, or re-rank where the cost is. It is deliberately free of standings and rankings (those move every landing); it describes the tools and the traps.

The two axes here are distinct from the two other doc surfaces nearby. Generated- code quality (runtime speed / density of optimized output) is the optimizer's concern — see OPT.md and plan/OPTIMIZER.md. The correctness gates a perf change must also pass (the byte-identity gate, the toy/ opt/smoke suites, the ecosystem run-and-diff) are in TESTING.md. A static line-count and binary-byte breakdown of kit itself by component is in CODE_SIZE.md; this doc is about measuring the C workload kit compiles.

Two rules

• A superlinear axis is a bug. A compile-speed regression that scales worse than linearly in some input dimension (function count, declaration count, …) means an inner step is O(n²)-ish — a scan run per item, a table that rebuilds, a list re-walked. The synthetic scaling guard below exists to catch these.
• Measure on a RELEASE build. Use make bin RELEASE=1 (binary at build/release/kit), or PROFILE=1 for sampling — never the default make bin, which is RELEASE=0 and links ASan/UBSan. ASan amplifies memory-op cost and dilutes compute cost, which inverts the ranking of where the time goes. Correctness gates are build-mode-independent; timers are not.

A kit binary copied outside the build tree fails with support dir not found — it resolves its rt/support dir relative to its own path. To A/B two builds keep both binaries inside build/release/ (e.g. kit_baseline, kit_fixed), each beside its own support/rt sibling.

The reference workload and the bar

The standing real workload is the sqlite amalgamation — one ~9 MB / 263 K-line C file (tmp/projects/sqlite-amalg/sqlite3.c): real, huge, declaration/macro/ type-heavy, and doubling as an end-to-end correctness test (compile + link the shell, run a query). Commands below assume SDK=$(xcrun --sdk macosx --show-sdk-path) and a release kit at build/release/kit.

The code-size and compile-speed bar is tcc (the fastest mainstream C compiler); clang is recorded for context. Build tcc from the mob mirror as an optimized binary:

git clone --depth 1 https://github.com/tinycc/tinycc tmp/tinycc && cd tmp/tinycc
export SDKROOT=$(xcrun --sdk macosx --show-sdk-path)   # else "stdio.h not found"
sed -i 's|^CFLAGS=.*|CFLAGS=-Wall -O3 -DNDEBUG -Wdeclaration-after-statement|' config.mak
./configure ; make clean ; make CC=cc        # CC=cc = Apple clang (xcrun-aware)
# binary at tmp/tinycc/tcc ; compile with: tcc -c sqlite3.c -o x.o

Compile speed: instruction count (macOS)

instructions retired is the metric to trust — it is load-independent, so it reproduces under noise. /usr/bin/time -l is the macOS perf stat; cycles/wall are low-load readings worth recording only for context. Take best-of-N to filter scheduling noise:

m(){ local bi=9e18 bc=9e18 br=9e18; for i in $(seq 1 7); do
       /usr/bin/time -l "$@" >/dev/null 2>/tmp/m.txt
       local I=$(awk '/instructions retired/{print $1}' /tmp/m.txt)
       local C=$(awk '/cycles elapsed/{print $1}' /tmp/m.txt)
       local R=$(awk '/real/{print $1}' /tmp/m.txt)
       awk "BEGIN{exit !($I<$bi)}"&&bi=$I; awk "BEGIN{exit !($C<$bc)}"&&bc=$C
       awk "BEGIN{exit !($R<$br)}"&&br=$R; done; echo "instr=$bi cycles=$bc real=${br}s"; }
cd tmp/projects/sqlite-amalg
m build/release/kit cc -c sqlite3.c -o /tmp/k.o --sysroot "$SDK"
m tmp/tinycc/tcc -c sqlite3.c -o /tmp/t.o

Caveat: Apple clang spawns a cc1 that /usr/bin/time does not count — measure clang with cc -fintegrated-cc1 -c …. kit and tcc are single-process, so their figures are whole-compile.

Phase decomposition

To split lex+pp from the rest, do not use -E. -E adds a token→text serialization that neither compiler does at -c, and kit and tcc serialize at wildly different cost, so -E is a misleading lex+pp proxy. Instead drain the token stream to EOF (lex+pp only, no parse, no output) in each compiler:

# kit: KIT_PP_DRAIN runs pp_next_parse to EOF then returns (lang/c/c.c)
m env KIT_PP_DRAIN=1 build/release/kit cc -c sqlite3.c --sysroot "$SDK" -o /dev/null  # lex+pp
m build/release/kit cc -fsyntax-only        sqlite3.c --sysroot "$SDK"  # +parse/sema/types/CG
m build/release/kit cc -c -o /tmp/k.o       sqlite3.c --sysroot "$SDK"  # +emit+objwrite
# tcc: the -bench hook drains next() to EOF; patched (tccpp.c tcc_preprocess)
# to use -c parse_flags. TCC_DRAIN_FLAGS=c also decodes literals in the lexer.
m env TCC_DRAIN_FLAGS=c tmp/tinycc/tcc -E -bench sqlite3.c -o /dev/null   # tcc lex+pp
m tmp/tinycc/tcc -c sqlite3.c -o /tmp/t.o                                 # tcc full -c

The three kit invocations are nested (each adds a phase), so successive differences attribute instructions to lex+pp, parse/sema/types/CG, and emit+objwrite respectively.

macOS hotspot sample (self-time leaves)

A single -c is too fast for sample; build a profiling kit (-g + frame pointers, no strip) and merge ~80 short runs. This gives self-time leaves only — for inclusive attribution use Linux callgrind (below).

make RELEASE=1 PROFILE=1 CC=clang BUILD_DIR=build/bench bin   # -g + frame ptrs, no strip
cd tmp/projects/sqlite-amalg ; KIT=build/bench/kit ; rm -f /tmp/p_*.txt
for i in $(seq 1 80); do
  $KIT cc -c sqlite3.c -o /tmp/kk.o --sysroot "$SDK" 2>/dev/null & pid=$!
  sample $pid 5 1 -file /tmp/p_$i.txt -mayDie >/dev/null 2>&1 ; wait $pid; done
for f in /tmp/p_*.txt; do awk '/Sort by top of stack/{f=1;next}/Binary Images/{f=0}f' "$f"; done \
 | sed -E 's/\(in [^)]*\)//' | grep -oE '[A-Za-z_][A-Za-z0-9_]*[[:space:]]+[0-9]+' \
 | awk '{c[$1]+=$2;t+=$2} END{for(k in c) printf "%.1f%% %s\n",100*c[k]/t,k}' | sort -rn | head -25

Compile speed: Linux callgrind (inclusive attribution)

There is no valgrind on macOS, and sample sees only self-time leaves. The inclusive, instruction-grounded profiler — the one that attributes a parent's cost across the functions it calls — is Linux valgrind --tool=callgrind. scripts/perf_callgrind.sh codifies the recipe; one-time scripts/perf_callgrind.sh image bakes the container, then:

scripts/perf_callgrind.sh run <tag>   # builds kit in-container, runs callgrind, prints total Ir
                                      # -> build/linux-prof/cg.<tag>.annot.txt (self Ir + callers)

For an -O1 optimizer-finalize profile of sqlite, the full collection overflows Valgrind's brk segment on the default VM, so give the VM more memory, toggle collection at opt_on_finalize, and force glibc malloc onto mmap:

podman machine stop || true
podman machine set --memory 3072
podman machine start
podman run --rm --platform linux/arm64 -v "$PWD":/work:Z kit-prof sh -c '
  set -eu
  cd /work/tmp/projects/sqlite-amalg
  env MALLOC_MMAP_THRESHOLD_=1 MALLOC_ARENA_MAX=1 \
    valgrind --tool=callgrind --cache-sim=no --branch-sim=no --dump-instr=no \
      --collect-jumps=no --collect-atstart=no --toggle-collect=opt_on_finalize \
      --callgrind-out-file=/work/build/linux-prof/cg.sqlite_o1.opt_finalize.mmap.out \
      /work/build/linux-prof/kit cc -O1 -c sqlite3.c \
        -DSQLITE_THREADSAFE=0 -DSQLITE_OMIT_LOAD_EXTENSION -lc -o /tmp/sqlite_o1.o \
      2>/work/build/linux-prof/cg.sqlite_o1.opt_finalize.mmap.vg.log
  callgrind_annotate --threshold=99.5 \
    /work/build/linux-prof/cg.sqlite_o1.opt_finalize.mmap.out \
    >/work/build/linux-prof/cg.sqlite_o1.opt_finalize.mmap.annot.txt
'
podman machine stop ; podman machine set --memory 2048 ; podman machine start   # restore

Gotchas baked into the script (do not relearn the hard way):

• Pass -lc so kit finds the glibc sysroot even for -c. Use a bookworm base (glibc 2.36), not ubuntu 24.04 (glibc 2.39's bits/math-vector.h uses a vector-typedef attribute the C frontend rejects).
• strip --strip-debug the kit binary in place before profiling — valgrind 3.19 chokes on clang DWARF5, and callgrind needs only .symtab. Strip in place so kit still resolves support/rt relative to its own path.
• Trust self/exclusive Ir, not inclusive %. Recursion in the recursive- descent parser makes --inclusive=yes double-count to absurd numbers. (Self-Ir may be summed across callgrind's '2 symbol splits for a recursive function.)
• The Linux/ELF total is not comparable to the macOS/Mach-O instruction count — different libc, sysroot, and format, and the Linux figure includes glibc + the loader + the -lc probe. Use callgrind for where (distribution); use macOS instructions for how much. An -O1 optimizer-finalize profile excludes frontend recording, so it is a distribution, not a total comparable to a full -O0 compile.

Heap allocation counts (KIT_METRICS)

KIT_METRICS=1 prints the hosted-heap counters (allocs / large allocs ≥32 KiB / reallocs / frees) at exit. The counts are host-independent — kit issues the same h->alloc calls everywhere — so they isolate allocation behavior from the host's malloc cost:

KIT_METRICS=1 build/release/kit cc -c sqlite3.c --sysroot "$SDK" -o /tmp/k.o
# -> kit heap metrics: heap.allocs=... heap.large_allocs=... heap.reallocs=... heap.frees=...

KIT_METRICS=1 is also the way to see where an -O1 compile spends its time: each metrics_scope_* bracket in the optimizer becomes a <scope> <ticks> ticks line, with per-pass counters (opt.funcs, opt.blocks, opt.pregs, the inliner refusal histogram, …). Ticks are the raw host cycle counter; within one run the scopes are directly comparable. See OPT.md §8 for the scope tree and interpretation; the heap counters reuse the same KitProfiler machinery (embedder counter range).

Code size

The honest size metric is .text machine code, not the object-file size: the object is format-skewed (kit emits Mach-O, tcc ELF) and not comparable. kit size reports .text; the per-mnemonic histogram diff localizes where the byte excess (or surplus) is:

build/release/kit cc -c sqlite3.c --sysroot "$SDK" -o /tmp/k.o ; build/release/kit size /tmp/k.o
tmp/tinycc/tcc -c sqlite3.c -o /tmp/t.o                        ; build/release/kit size /tmp/t.o
# per-mnemonic excess (heavier in kit = positive count delta):
hist(){ build/release/kit objdump -d "$1" | grep -E '^[[:space:]]+[0-9a-f]+:' \
  | sed -E 's/.*\t([a-z][a-z0-9._]*).*/\1/' | grep -E '^[a-z]' | sort | uniq -c | sort -rn; }
diff <(hist /tmp/k.o) <(hist /tmp/t.o)

For -O1 generated-code-quality size comparison against clang per ecosystem file, scripts/o1_quality.sh builds the per-file kit (.k.o) and clang (.c.o) objects into build/o1_quality/; the per-file ratio is kit __TEXT ÷ clang __TEXT via size -m. When counting instruction lines from kit objdump -d, note the format is addr:\t bytes \t mnemonic operands — the mnemonic is after the second tab, so a naive grep ':\t\w+' matches the byte column and is wrong. clang on macOS needs -isysroot "$SDK"; kit needs --sysroot "$SDK".

The synthetic scaling guard (make bench-cc)

make bench-cc is the regression guard for the superlinear-axis-is-a-bug rule. It sweeps each input dimension — fn-count, body-size, global-decl, type-decl, locals-per-fn, pp-macro, pp-include, ref-density, obj-count, symbol-count — over a geometric series, fits a scaling exponent (p ≈ 1.0 is the goal; p ≥ 1.4 is flagged SUPERLINEAR and treated as a bug), samples hotspots, and records a clang reference. Output lands in build/bench/cc/ (scaling.md, hotspots.md). Sources of truth: scripts/cc_bench_gen.py (the axis catalog), scripts/cc_bench.sh (harness), scripts/cc_bench_report.py (fit + verdicts).

Use the synthetic sweep to catch a re-introduced O(n²); use the sqlite profile to decide where to optimize — the synthetic axes over-weight codegen relative to real declaration/macro-heavy code.

For an -O1 quality change specifically (which deliberately changes emitted bytes), the relevant linearity check is that sqlite -O1 compile time does not move materially, validated with this sweep plus a before/after timing of the sqlite -O1 compile.

End-to-end correctness as a perf gate

The sqlite workload is also a correctness oracle, run at both opt levels:

build/release/kit cc sqlite3.c shell.c -o /tmp/sq --sysroot "$SDK" -lc
/tmp/sq :memory: "create table t(a,b); insert into t values(2,40),(10,32);
                  select sum(a)+sum(b), count(*) from t;"   # -> 84|2

The full perf-change gate (byte-identity for compile-speed changes, run-correctness + determinism for code-size changes, and the suite list) is in TESTING.md; scripts/perf_identity_gate.sh and scripts/perf_axis_time.py are the gate/timer tooling, driven by make perf-golden / make perf-gate.