Optimizer (OPT)
This document is the design reference for kit's optimizer: the module that
sits between the recording code-generation API and the per-architecture native
backends. The optimizer owns a private, mutable IR; it lowers each recorded
function into that IR, runs analyses and transforms over it, performs register
allocation, builds a physical post-allocation IR (MIR), and finally replays the
MIR into a NativeTarget backend. It is also the source of the function shape
the bytecode interpreter consumes. The focus here is layering, ownership,
representation invariants, and the reasoning behind the boundaries — not API
signatures, which live in the headers. Cross-references: DESIGN.md,
CODEGEN.md, IR.md, ARCH.md, and
INTERPRETER.md.
1. Where the optimizer sits
kit's codegen has two surfaces. The semantic surface is the recording
code-generation API (cg/ir.h, the CgTarget interface): frontends call it to
describe a function. The physical surface is the per-architecture
NativeTarget (arch/native_target.h): it encodes machine code. The optimizer
is the bridge.
frontend --CgTarget calls--> CgIrRecorder --records--> CgIrFunc tape
|
opt_func_from_cg_ir (cg_ir_lower.c)
v
optimizer IR: Func / Block / Inst / Val / PReg
|
passes (analysis + transform)
|
regalloc -> location MFunc (physical MIR)
v
opt_emit_native --NativeTarget calls--> machine code
At -O0 the optimizer is not installed at all: the driver wires the frontend's
CgTarget straight to the backend's NativeDirectTarget, which emits in a
single pass with a small register cache (see CODEGEN.md). At
opt_level >= 1, opt_cgtarget_new (src/opt/opt.c) installs a CgIrRecorder
(cg/ir_recorder.c) as the sink. The recorder captures each function as a
CgIrFunc tape, and on completion fires the optimizer's per-function callback.
OptImpl (in src/opt/opt.c) is the wrapper state: the wrapped real target,
the resolved NativeTarget*, an optional dump writer, and a per-translation-
unit registry of recorded CgIrFuncs (with a parallel lazily-lowered-Func
cache) used for streaming tiny-callee inline lookup.
When each function is processed: finalization-time, all arches
At opt_level >= 1 the optimizer runs in whole-program mode
(o->whole_program = (level >= 1) in opt_cgtarget_new), identically on every
architecture. The per-function callback only registers the recorded
CgIrFunc; it does no lowering (opt_on_func returns early). All processing is
deferred to module finalization (opt_on_finalize -> opt_whole_module_finalize),
where a reachability sweep over the call/data-reloc graph computes the set of
functions actually referenced from a root, prunes the rest, runs the
cross-function inliner over the survivors, and only then lowers and processes
each one.
The per-function callback also carries a dormant eager path: with whole-program
mode off it would lower and fully process each function as it is recorded
(per-function streaming), leaving dead-static elimination to the linker. The
recorder only exists at level >= 1, which always sets whole_program, so no
configuration reaches the eager path; it is a fallback for a non-whole-program
build. The two paths share the same lowering path and backend tail, differing
only in when a function is lowered, whether the cross-function inliner runs,
and whether dead local functions are dropped before lowering or left for the
linker (Section 3.1).
The recording/optimizing boundary
The split between recording (cg/ir) and optimizing (opt/ir) is
deliberate and is the central design decision of this module:
- The recorded
CgIrFuncis a faithful, immutable transcript of the frontend's semantic intent. It speaks inCGLocal/Label/CGCallDescterms and knows nothing about CFGs, dominators, or physical registers. Frontends and ABI lowering own that layer; the optimizer never mutates it. Keeping it immutable is what makes streaming tiny-inline re-lowering cheap and repeatable, and what lets the same recorded tape feed both the native pipeline and the interpreter. opt_func_from_cg_ir(src/opt/cg_ir_lower.c) translates oneCgIrFuncinto the optimizer's own mutableFunc— a real CFG ofBlocks, each holding a linear list ofInsts, plus frame slots, a pseudo-register table, a value table, and the params/locals tables. From here on the optimizer works only onFunc; the recorded tape is a read-only source.
Lowering also performs the first storage-classification decision. In
lower_locals, each semantic CGLocal becomes either register storage
(CG_LOCAL_STORAGE_REG, a fresh PReg, operands of kind OPK_REG) or frame
storage (CG_LOCAL_STORAGE_FRAME, a FrameSlot, operands of kind OPK_LOCAL).
A local is forced to a frame home when it is address-taken / memory-required
(local_needs_home), an aggregate, or larger than a machine word. Everything
else starts in a pseudo-register. Address-taken locals begin in frame storage;
later HIR address-folding and promotion passes recover register storage for
those whose address does not actually escape (Section 4). va_list operands
are lowered as opaque pointer values, never address-taken, so that all
va-layout knowledge stays behind the NativeTarget va hooks.
2. The optimizer IR and its operand model
The optimizer IR lives in src/opt/ir.h / src/opt/ir.c. Its shape:
Funcowns one function: its CFG (blocks,entry,emit_order), frame slots, params, locals, the pseudo-register table, the SSA value table, scope bookkeeping, allocation results, and pass-owned analysis state.Blockis a basic block: a growableInst[], explicitpreds/succedges, and a pre-allocatedMCLabelfor blocks born fromcg_label_new.Instis one recorded operation. TheIROpenum mirrors theCgTargetsurface essentially 1:1 (each recordedCgTargetcall becomes exactly oneInst), plus a few SSA-only ops (IR_PHI,IR_CONST_I,IR_CONST_BYTES). Rich operations (calls, returns, switches, inline asm, atomics, aggregate memory ops, intrinsics, scopes, phis) carry a structuredauxrecord so the full semantic descriptor round-trips to emission.
Virtual vs physical operands; the mode-on-Func invariant
Operand (kind OPK_REG/OPK_IMM/OPK_LOCAL/OPK_GLOBAL/OPK_INDIRECT) is
intentionally not a bare value id. Register operands change meaning across the
pipeline, but the field never changes — the mode is a flag on Func, never
encoded in the numeric id:
- During lowering and the whole O1 path,
OPK_REGcarries aPReg: a mutable pseudo-register id, the persistent storage location of a value. - After
opt_build_reg_ssa(O2 only),OPK_REGcarries aVal: an SSA single-definition value id.Func.opt_reg_ssarecords which namespace is live; shared helpers (opt_reg_count,opt_reg_type,opt_reg_clsinopt_internal.h) consult it rather than guessing from context. - Physical registers never appear in
OPK_REGHIR operands. Allocation results go to a separate location table, and physical operands appear only in the MIR (Section 6).
IR_PARAM_DECL is a def-only marker carrying no operands — the param's storage
lives in the IRParam table, not in a synthetic self-operand. These invariants
(virtual-only HIR operands, single-namespace-at-a-time, def-only param decls)
are what the debug verifier (opt_verify, Section 7) checks at phase
boundaries, so that a stale physical operand or a wrong-namespace use fails at
the nearest checkpoint rather than in the backend encoder.
FrameSlot is the frame-storage currency: locals forced to memory, spill slots,
ABI parameter slots, alloca regions, and outgoing-argument areas.
MIR extends the operand vocabulary with locations rather than hiding locations
behind temporary register ids. OPK_STACK is the scalar value in an FS_SPILL
slot, while OPK_FRAME_ADDR is a rematerializable address of a semantic frame
slot. An OPK_INDIRECT address records the location kind and exact type of its
base and index independently: either a physical register, a spilled scalar, or
(for the base) a frame-address recipe. These kinds never occur in semantic HIR.
They let post-allocation passes preserve the actual allocation result all the
way to emission without manufacturing register lifetimes between instructions.
Token aliasing: optimizer-local names onto NativeTarget types
src/opt/ir.h deliberately reuses the physical backend's data types as the
optimizer's own, via a layer of preprocessor #defines. After including
arch/native_target.h it remaps a set of optimizer-local tokens onto the
Native* types:
FrameSlot→NativeFrameSlot,FrameSlotKind/FS_*→ theNativeFrameSlot*enum,RegClass/RC_*→NativeAllocClass,CGPhysRegInfo→NativePhysRegInfo, the known-frame descriptor, and theCG_REG_*register role flags.- It also re-
#defines the now-removed semantic CG spellings —Operand,CGCallDesc,CGFuncDesc,CGParamDesc,CGScopeDesc,CGLocalStorage,FrameSlotDesc— onto the optimizer's ownOpt*structs, so optimizer code can keep using the short historical names.
The reason is that the optimizer's frame-slot, register-class, and physical-
register vocabulary is the backend's; sharing the structs avoids a translation
layer at the emit boundary, where NativeFrame* is exactly what
opt_emit_native hands the NativeTarget. The cost is a namespace hazard: a
.c file that needs the real semantic cg/ir.h Operand/CG*Desc types (for
example because it reads the recorded tape, or it talks to the live NativeTarget
in Native* terms) must first #undef the aliased tokens. cg_ir_lower.c,
opt.c, and pass_native_emit.c each do exactly this at the top of the file —
they straddle the boundary and must escape the optimizer-local remapping to name
the other side's types. Files that live entirely inside the optimizer IR (the
analysis and transform passes) keep the aliases and never #undef.
3. One lowering path, three consumers
There is a single lowering path through the optimizer IR. The opt level and the consumer choose how far down it the function travels.
opt_func_from_cg_ir
|
+-----------------+------------------+
| | |
O1 native O2 mid-end interpreter tap
opt_run_o1_native opt_cleanup + opt_run_o1_interp
shared lowering (stops before machinize)
| | |
machinize SSA build, |
regalloc value/mem passes, |
MIR + emit conventional SSA, |
undo-SSA, then |
shared lowering |
v v v
NativeTarget NativeTarget interp bytecode
O1 native (opt_run_o1_native)
This is the live optimized path for compiled output, and (with O2 disabled,
below) the only optimization pipeline any -O1/-O2 compile runs. The
per-function driver is the opt_o1_native_prepare + opt_o1_native_finish pair
(opt_run_o1_native is the streaming wrapper around both; the finalize sweep
calls the halves directly so the cross-function inliner can run between them).
Every function is reached the same way today — through the finalize sweep of
Section 1 — and travels the same pipeline entirely in the PReg namespace
(opt_reg_ssa == 0): no SSA construction, no value numbering, no
dominance-frontier phi insertion. That is the deliberate -O1 design point —
fast, reasonable code from local + linear-scan machinery, paying none of the SSA
build/destroy cost. In source order:
build_cfg -> jump_cleanup(CFG) -> build_cfg -> simplify_local
try_tiny_inline (+ cfg/jump_cleanup/cfg if anything inlined)
verify "lowering-cfg"
machinize_native ABI/call/ret/param constraints + machine clobbers
verify "lowering-machinize"
addr_xform_pregs fold ADDR_OF(local) into OPK_LOCAL loads/stores
promote_scalar_locals non-escaped scalar frame slot -> PReg
addr_of_global_cse hoist duplicate ADDR_OF(global) to entry
build_loop_tree
lower_loop_imm_operands / hoist_loop_consts loop-invariant imm materialization
opt_refresh_machine_clobbers rederive effects from final HIR instruction shapes
live_blocks per-block PReg liveness (backward dataflow)
dead_def_elim_with_live pre-RA dead-definition elimination
regalloc_locations PReg -> hard reg / spill slot (no live-range splitting)
verify "post-regalloc"
lower_to_mir deep-clone HIR; rewrite values to physical locations
mir_verify "lower-mir"
mir_combine post-RA peephole / addressing-mode synthesis
mir_dce post-RA dead-code elimination
mir_jump_cleanup(CFG) -> mir_build_cfg -> mir_jump_cleanup(LAYOUT)
emit_native replay MIR into the NativeTarget
Once a function enters this pipeline it runs every stage — there is no per-op
bypass within the pipeline itself. Varargs, inline asm, aggregates/sret/byval are
all handled here. Most stages are bracketed by an opt_verify /
opt_mir_verify checkpoint with a stage tag, and KIT_DUMP=<tag> dumps the
IR at the matching stage (entry before any pass, pre-emit just before emit).
The O1 optimization set. Stripping out the analyses (build_cfg,
build_loop_tree, live_blocks), the verifiers, and pure lowering
(machinize_native, lower_to_mir, emit_native), the transforms that actually
improve the code are exactly these, none of which needs SSA:
simplify_local— local algebraic/addressing canonicalization (the no-SSA-required cleanup, also used by the interpreter tap). Besides the arithmetic identities (x+0,x*1, …) it folds two address/cast idioms to copies so they collapse instead of lowering to register moves:addr_of [base + 0](the&*p/p + 0the frontend records around derefs) and a same-width, same-register-class convert (notably a pointer bitcast likeint* -> char*, where the named types differ but no bits move).- Cross-function inlining —
try_tiny_inlineper function in this pipeline, plus the whole-programopt_inlinethat runs once at finalize before this pipeline (Section 1). This is the only interprocedural transform. addr_xform_pregs— foldADDR_OF(local)into directOPK_LOCALload/store operands and clearFSF_ADDR_TAKENwhen every such def retires.promote_scalar_locals— lift a non-escaped scalar frame slot into a mutable PReg, turning its stores/loads into register copies.addr_of_global_cse— hoist oneADDR_OF(global)compute to the entry block and reuse it.lower_loop_imm_operands+hoist_loop_consts— materialize loop-invariant immediates once in the entry block instead of per iteration.dead_def_elim_with_live— liveness-driven pre-RA dead-definition removal.regalloc_locations— point-bitmap linear-scan allocation, without live-range splitting. A linear move-coalescer now populates the union-find (opt_coalesce_parent) before allocation so copy-related values share a location; the O2-only quality knobs that stay off are live-range splitting and the O(n²) conflict-matrix coalescer (opt_coalesce_ranges).mir_combine— post-RA peephole + addressing-mode synthesis (the sameopt_combineused in O2's SSA combine, here gated on physical-register liveness). Its extension-folding also retires a ZEXT of a plain (zero-extending) narrow load: kit's loads always zero-extend the destination register — sign extension is a flagged load or a separate convert — so the promotion(int)p[0]reproduces bits theldrb/movzbl/lbualready cleared, and the convert collapses to a copy.mir_dce— post-RA dead-code elimination.jump_cleanup/mir_jump_cleanup— unreachable-block drop, jump-chain collapse, and (LAYOUT mode) block reordering for fallthrough + loop rotation.
Everything else under §4 (build_ssa, gvn, dse, licm, copy_prop,
simplify, live-range splitting, the O(n²) matrix coalescer) is O2-only and
never runs. (Linear move coalescing does run at O1 — see regalloc_locations
above; only the splitting/matrix variants are O2-only.) The -O1 code-quality
work (frame layout, rematerialization, switch/cmp immediates, branch cleanup,
inline cap) is landed; its passes are noted inline above.
The reachability decision lives outside this pipeline, in the finalize sweep
(Section 1), identical on every architecture. At module finalization
(opt_on_finalize) file-scope asm blocks captured during recording are replayed
on every target, then the reachability sweep selects which functions are lowered
at all — so dead local functions/data are never lowered or emitted — and the
survivors each run the full pipeline above.
O2 mid-end (opt_cleanup + shared lowering)
The O2 mid-end is the SSA-based optimization schedule defined in opt_cleanup
(src/opt/pass_o2.c). It is the intended mid-end architecture and is fully
implemented, but it is not on the shipped code path: opt_cleanup has no
caller. The finalize sweep (opt_whole_module_finalize) always runs the O1
native pipeline (opt_o1_native_prepare/_finish) regardless of the requested
opt_level, so no compilation ever selects the SSA schedule and every
opt_level >= 1 request runs the O1 native path. Public opt_level == 2 is
normalized to 1, so -O2 produces the same output as -O1 today.
The rationale for keeping it parked is isolation. Keeping the O2 schedule defined and its passes maintained means the SSA representation and its incremental def-use can stabilize against targeted optimizer tests independently, without an SSA-construction or value-numbering bug affecting shipped output. The schedule is documented here because it is the designed mid-end shape that the O1 path is a deliberately reduced subset of; the section describes the intended architecture, not a live code path. The schedule is:
build_cfg / jump_cleanup(CFG) / build_cfg canonicalize control flow
build_reg_ssa PReg -> Val (register SSA)
block_cloning bounded clone of small blocks
build_ssa mem2reg: promote frame locals, insert phis
ssa_dce / copy_cleanup
addr_xform fold address pseudos into mem operands
simplify SSA-aware identity/algebraic cleanup
gvn value numbering, constprop, branch fold,
redundant-load reuse
copy_prop copy + redundant-extension elimination
dse dead store elimination
build_loop_tree / licm hoist loop invariants
pressure_relief sink same-block computes
make_conventional_ssa phis -> edge copies (IRF_NO_COALESCE)
ssa_combine
undo_ssa / copy_cleanup Val -> PReg, allocation-ready
jump_opt
By design an O2 function then re-enters the same backend tail as O1 (machinize
through emit), with the allocator's live-range splitting and move-related
coalescing enabled — the variants that the O1 path leaves off. The SSA
value/memory passes (opt_gvn, opt_dse, opt_licm,
opt_pressure_relief, opt_ssa_combine) live in src/opt/pass_o2.c; SSA
construction and phi destruction in src/opt/pass_ssa.c.
Interpreter tap (opt_run_o1_interp)
The interpreter consumes the optimizer IR directly rather than machine code. The
tap runs the maximal target-independent subset of the O1 pipeline and stops
before machinization: build CFG, jump cleanup, simplify_local, the PReg-level
address folds and scalar-local promotion, addr_of_global_cse, loop tree, and
liveness-driven dead-def elimination. It deliberately stops before
opt_machinize_native, register allocation, MIR lowering, and native emit. The
result is a Func still in the PReg namespace (opt_reg_ssa == 0, no
IR_PHI phis) that src/interp/lower.c lowers into threaded bytecode. The tap
runs the folds even though in the native pipeline they sit after machinize,
because they depend only on the PReg/frame-slot view, not on physical-register
pools — so they are safe and they shrink the interpreter's work. See
INTERPRETER.md.
4. Pass catalog by role
The passes are grouped here by responsibility. Each is one Func-in-place
transform or analysis; the file paths orient the reader.
SSA mid-end (O2)
- Register SSA + mem2reg (
src/opt/pass_ssa.c):opt_build_reg_ssarenames multiply-assigned PRegs into SSAVals;opt_build_ssapromotes eligible frame-backed locals/params to SSA via dominance-frontier phi insertion and rewrites their loads/stores to values.opt_make_conventional_ssalowers phis to edge copies (markedIRF_NO_COALESCE, because coalescing a phi edge copy can collapse a loop-carried value with its successor and miscompile the loop) andopt_undo_ssareturns to the PReg namespace for allocation. - GVN + DSE orchestration (
src/opt/pass_o2.c):opt_gvndoes scalar value numbering, constant propagation, branch folding, and memory-aware redundant load / store-to-load reuse gated by alias-root and version rules;opt_dseremoves stores proven dead or overwritten while preserving observable memory (volatile, atomic, calls that may clobber, escapes).opt_licmandopt_pressure_reliefround out the loop/pressure work, also here. - Peephole combine + addressing-mode synthesis (
src/opt/pass_combine.c):opt_combineis a per-block forward-pass-with-fixpoint that propagates copies, folds address-producing computations into a load/store'sOPK_INDIRECTbase/index/scale/offset where the backend accepts the shape, sinks defs toward their sole use, and folds extension chains (including a ZEXT of a zero-extending narrow load, which the load already performed). It is used in two roles: directly in the O2 SSA combine (opt_ssa_combinewraps it) and as the post-RA MIR combine (Section 6). When run over physical MIR it gates each rewrite on a live-range safety check (Section 5). - Simplify (
src/opt/pass_simplify.c):opt_simplify_localis the no-SSA-required local algebraic/addressing canonicalizer used on every path (arithmetic identities,addr_of [base+0]→ copy, and same-width same-class convert → copy);opt_simplifyis the SSA-aware identity/constant cleanup used in O2. - DCE (
src/opt/pass_dce.c):opt_ssa_dceremoves unused SSA defs;opt_mir_dceremoves post-RA dead physical defs; both preserve side effects, including the subtle case of a value-producing op whose destination is anOPK_LOCAL(a write to an escaped frame-homed local is a memory side effect even when the op is otherwise pure). - Copy cleanup / copy prop (
src/opt/pass_copy.c): redundant-copy removal and copy propagation, including redundant extension/convert-chain elimination. - Inlining (
src/opt/pass_inline.c):opt_try_tiny_inlineis the streaming O1 entry. On the pre-machinize PReg form it resolves each directIR_CALLto a recorded callee via a lookup callback (OptImplowns the registry and the lazily re-lowered callee cache), gates on a tiny straightline-cost cap and a whitelist that excludes calls/control-rich constructs, refuses self/recursive callees, and splices the cloned body in. The whole-program inliner machinery (opt_inline,inline_call_site, and the cost/growth/policy gates) also lives here; it builds anInlineIndexover theFuncSetonce — a symbol→index hash map plus per-function SCC ids (iterative Tarjan) — so callee resolution and the recursion gate (recursive_or_scc, anscc[caller] == scc[callee]compare) are O(1), keeping the pass linear in call sites (Section 8). Both inliners honor the per-function/per-callKitCgInlinePolicy(DEFAULT/HINT/ALWAYS/NEVER), which is how Cinline/always_inline/noinlinereach codegen. - Address folding (
src/opt/pass_addr_fold.c): the always-on O1 HIR folds —opt_addr_xform_pregs(foldADDR_OF(local)into directOPK_LOCALload/store operands and clearFSF_ADDR_TAKENwhen all such defs retire),opt_promote_scalar_locals(promote a non-escaped scalar frame slot to a PReg, turning its stores/loads into copies),opt_addr_of_global_cse(hoist oneADDR_OF(global)compute to the entry block and reuse it), and the loop-invariant constant materialization (opt_hoist_loop_consts/opt_lower_loop_imm_operands).opt_addr_xformis the SSA-namespace counterpart used in O2.
Shared analyses
- CFG (
src/opt/pass_cfg.c):opt_build_cfgderivespreds/succfrom each block's terminator (branches, conditional/fused branches, returns, switches, indirect branches, scope break/continue edges) and validates reciprocity;opt_mir_build_cfgrecomputes them over the physical MIR. - Order + dominators + verify (
src/opt/pass_analysis.c): postorder / reverse-postorder, reachability, immediate dominators, dominator children, dominance frontiers (OptAnalysis), the coarse analysis-validity bits (OPT_ANALYSIS_DEF_USE/DOM/LOOP), and the debug verifieropt_verify. - Liveness (
src/opt/pass_live.c):opt_live_blockssolves per-block PReg liveness by backward dataflow into elastic 64-bit-word bitsets (OptBitset, grown on demand, trailing-zero-trimmed);opt_live_ranges_buildproduces the compressed point-indexed live ranges and per-PReg frequency/spill-cost metrics the allocator consumes. - Hard-register liveness (
src/opt/pass_hard_live.c): physical-register live-in/out over the post-RA MIR. It consumes the same canonical instruction effects (opt_inst_reg_effects) as post-RA combine, DCE, and native emission, so explicit operands, call effects, inline-asm clobbers, and target machine clobbers cannot drift between analyses. This is what makes post-RA combine/DCE safe: a value in a callee-saved register survives a call, while a register in that call's clobber mask is killed by it. - Loop detection (
src/opt/pass_loop.c):opt_build_loop_treecomputes loop nesting depth from dominators; depth feeds the allocator's spill-cost weighting and LICM.
Backend tail
- Type-size lowering (
src/opt/pass_lower.c): the type/size machinery and the allocator that the PReg form needs before MIR (also hosts the allocation and constraint application described below). - Machinize (
src/opt/pass_machinize.c):opt_machinize_nativeis ABI lowering against theNativeTarget. It annotates calls/returns/params with calling-convention constraints (argument/result registers, the call clobber and return masks, callee-save markers), collects and validates the target's register classes (the physical-register table and its O1 allocation flags, the independent O0ndt_allocablecache policy, the O1 emission temporary bank, asm-only temporary bank, caller/callee-saved masks, and cache-preserved subset), resolves inline-asm named-register constraint strings into masks, and records per-instruction fixed-register clobbers (Section 5). The O1 emission bank must be physical, unique, non-allocable, disjoint from the asm-only bank, and large enough for the target's operand-facing hooks. Every target-reported clobber bit is validated against the class's physical-register table before it can enter the allocator/effect side table. - MIR view (
src/opt/pass_mir.c): the post-allocation physical IR. Rather than duplicate the CFG passes,pass_mir.cbuilds a transientFuncview whose block arrays point atFunc.mir, runs the sharedopt_combine,opt_dce,opt_build_cfg, andopt_jump_cleanupover that view, and commits it back. The MIR itself starts as a deep mutable clone, including operand and aux arrays; HIR remains virtual and cannot be changed through an aliased MIR pointer. Theopt_mir_*wrappers are thin shims over this view; the shared passes are written once and reused for both HIR and MIR. - Coalescing / allocation (
src/opt/pass_coalesce.c,src/opt/pass_lower.c):opt_regalloc_locationsis a point-bitmap linear-scan allocator producing the canonicalFunc.preg_locslocation table (hard reg or spill slot per PReg) without mutating HIR operands. Two coalescers feed its union-find: the O1 linear coalesceropt_coalesce_linearmerges copy-related values using bounded per-root member lists + on-demand range-overlap tests (no matrix), andopt_verify_alloctreats same-root PRegs as one value; the O2 matrix coalesceropt_coalesce_ranges, gated on live-range splitting, builds a bounded conflict matrix. Both merge only same-class, same-type values with compatible constraints and no range conflict — never anIRF_NO_COALESCEcopy. - Jump / layout cleanup (
src/opt/pass_jump.c):opt_jump_cleanupin CFG mode drops unreachable blocks and collapses unconditional-jump chains; in LAYOUT mode it reorders blocks for fallthrough, rotates simple single-latch loops, and inverts mis-aligned conditional branches so the per-iteration back-jump disappears. - Native emit (
src/opt/pass_native_emit.c):opt_emit_nativereplays the physical MIR into aNativeTarget, usingNativeLoc(register / frame / imm / address) as the operand currency. It reserves exactly the callee-saved registers the allocator used, pre-maps frame slots, drives the backend's minimal-prologue hook when available, routes scalar call results straight to their destination, uses a hardware zero register for stored zeros where the backend advertises one, and materializes frame/recipe operands in an explicit instruction-scoped temporary lease. The target declares the operand-facingemit_tempsbank separately from its O0scratchpolicy and its asm-onlyasm_temps; emitter temporaries never appear as MIR values. See ARCH.md for the backend contract.
5. Register constraints and canonical effects
Some instructions need registers that are fixed by their encoding, while other
instructions destroy registers as a side effect. Those are instruction-local
machine requirements, not persistent homes for the values involved. For
example, x86-64 division stages its dividend through rax and destroys
rax/rdx, a variable shift stages its count through cl, and atomics and
bitfield stores use fixed implementation registers. TLS descriptor sequences
and syscall intrinsics can also have format- or target-specific effects.
The allocator has one authoritative hard constraint:
OptPRegInfo.forbidden_hard_regs. A hard register present in that mask cannot
be selected for the PReg. preferred_hard_reg is only a placement hint and is
never allowed to override a forbid. There is no persistent fixed-register tie
or pin in the allocation model.
The two instruction-boundary sources populate that constraint differently:
- Inline asm.
pass_machinize.cresolves each register constraint once into an instruction-localIRAsmRegRequirement(register class, optional fixed register, and optional allowed-register mask), and resolves the clobber list into per-class masks. With the current unsplit allocator,pass_lower.cconservatively forbids the candidate or clobbered registers for PRegs that touch or cross that asm instruction. At emission the optimizer stages each operand into a scoped register lease satisfying the requirement, then writes outputs back to their real MIR homes. A generic whole-class constraint needs no reservation because either the allocated register already satisfies it or a spilled value uses an emitter temp. - Generic machine instructions. Machinization initially queries the
target's
machine_op_clobbershook and stores each nonzero result in the dense, per-functionFunc.inst_clobbersside table keyed byInstId.lower_loop_imm_operandscan still change an instruction's fixed effects (for example, by turning an x64 immediate shift into a register shift), andhoist_loop_constsis the final HIR-shape mutator. The explicitopt_refresh_machine_clobbersboundary after both passes replaces the table with effects derived from those final HIR shapes. Allocation forbids those registers for non-def values live after the instruction. The backend still performs the instruction-local staging required by the encoding; a dying operand is free to have any otherwise legal home.
Calls use a separate preservation policy rather than feeding this machine forbid mechanism. Allocation cost prefers callee-saved registers for values that cross calls. If pressure nevertheless puts such a value in a register the call clobbers, MIR lowering inserts the required save and restore. ABI argument and result placement remains a call-marshalling concern.
The corresponding post-allocation authority is OptRegEffects, built only by
opt_inst_reg_effects. It separates explicit uses, explicit defs, and
implicit clobbers; the distinction is intentional because a clobber kills a
live value but does not produce one for DCE. Calls contribute their lowered call
plan and ABI clobber mask, inline asm contributes its bound operands and resolved
clobber mask, and target instructions contribute the machinization side-table
mask. Hard-register liveness, MIR combine/DCE, and the native emitter all query
this one description. Emitter temporary leases are outside this model: they are
owned by one emission scope and are neither MIR defs nor implicit machine
clobbers. machine_op_clobbers is an exhaustive preservation contract for
ordinary NativeTarget operations: apart from explicit defs and reported
clobbers, a hook preserves every register that may hold an allocated or cached
value.
6. Allocation, MIR, and the physical boundary
Allocation does not rewrite HIR. opt_regalloc_locations consumes block
liveness and the compressed live ranges and writes one canonical location per
PReg into Func.preg_locs (OptLoc: hard register or spill slot). HIR operands
stay virtual after allocation — the verifier checks this.
opt_lower_to_mir then builds the physical IR Func.mir (an MFunc): each
virtual OPK_REG is translated through its OptLoc into either a physical
OPK_REG or an OPK_STACK spill value. Spilled indirect bases and indices stay
as typed frame components, and cheap rematerializable values stay as OPK_IMM
or OPK_FRAME_ADDR recipes. Lowering does not surround each spilled use/def
with synthetic reload/store instructions. It inserts explicit location copies
only where the program really crosses a physical boundary, such as preserving
an intentionally caller-clobbered allocation across a call. Call plans are
lowered to the same location vocabulary.
The MIR is a deep mutable clone, not a shallow rewrite of HIR storage. From this
point it is physical and non-SSA (registers may be multiply defined), and stale
HIR definition metadata is cleared. All PReg-to-physical knowledge lives in
this one step; after it, HIR is untouched and MIR is fully physical. The
downstream MIR passes (combine, DCE, jump/layout cleanup) run over the MIR view
and rely on canonical physical-register effects for safety. A pass that grows
the physical graph commits its extended instruction-id namespace with that
graph so later side tables remain correctly sized. Adjacent spill copies and
same-block spill dead stores are normalized over the OPK_STACK location
form. The same canonical operand walker used by liveness and emission accounts
for every stack use, including aux operands and spilled indirect-address
components, before DSE can retire an overwritten store. Then
opt_emit_native replays the MIR.
At emission, each instruction opens a NativeEmitTempScope. The scope derives
its unavailable registers from physical live-after plus the instruction's
canonical uses/defs/clobbers. It prefers target-declared emit_temps for
operand materialization; after that bank is exhausted it may scavenge a
caller-saved O1 allocation register only when physical liveness proves it dead
for the instruction. Multiple references to the same location within the
instruction share a scoped materialization. Calls have a mutating marshalling
phase: marshal_call may stage the callee, store stack arguments, and perform
target-owned argument shuffles while it fills the NativeCallPhase. Each
completed ABI register destination remains phase-owned through the full
multi-move phase.
After emit_call, a phase barrier discards argument/callee staging before
return writeback begins. Inline asm may additionally borrow asm_temps only
under the target-declared constraints; its staging remains local to that asm
instruction.
Frame memory is authoritative. A spill definition is stored to its FS_SPILL
slot even when the emitter retains the clean register value for an adjacent
use; there is no dirty deferred writeback. The optional block-local frame cache
is keyed by exact slot and value type. Persistent entries may use a
target-declared cache-enabled emission temporary in emit_cache_mask, or a
dead O1 allocation register whose preservation is already established:
caller-saved within the current call-free region, or a callee-saved register
already present in the known frame. The O0 ndt_allocable bank is not consulted
by optimized emission. The exhaustive machine-effect contract invalidates any
exceptional fixed clobber, and all entries are cleared at block, call, and
inline-asm boundaries. Thus cache state is a replay optimization, never part of
MIR semantics or a second owner of a spilled value.
The reason allocation results are a separate table rather than rewritten operands is the same mode-clarity principle from Section 2: a post-allocation pass can never accidentally treat a physical register as a PReg, replay can never see a stale virtual operand, and the MIR verifier can assert "no PRegs or Vals here" at a single boundary.
7. Verification and observability
The optimizer is checkpoint-verified in debug builds. opt_verify(Func*, stage)
checks CFG reciprocity, reachable-block shape, emit-order validity, instruction
ids, operand namespaces (no physical registers in HIR; correct PReg-vs-Val
namespace for the current mode), phi consistency, and def-use freshness;
opt_mir_verify checks the physical boundary (no virtual operands, valid frame
slots and location kinds/types, fully physical call plans, no emitter-owned
temporary in MIR, and no stale HIR def metadata). Each pass tags its checkpoint
with the name of the transformation just completed, so a failure localizes to
the nearest boundary. Func.opt_valid_analyses tracks coarse invalidation;
passes that mutate control flow, operands, or instructions rebuild or invalidate
the relevant analysis.
Observability hooks: KIT_DUMP=<tag> dumps the optimizer IR at a named stage,
KIT_DUMPCG=1 dumps the recorded semantic tape before lowering,
KIT_DUMP_INTERP dumps the interpreter-tap Func, and the optimizer emits
scoped timing/count metrics for the frontend, each pass scope, allocation, and
emit. Those metrics surface two ways: through kit run --time (the JIT path),
and — for the AOT path that produces shipped objects (kit cc / kit compile)
— by setting KIT_METRICS=1 in the environment, which wires a process-wide
KitProfiler into the compile and dumps every non-zero scope timer and counter
to stderr at exit. Section 8 walks the output.
8. Profiling the O1 path
KIT_METRICS=1 is the way to see where an -O1 compile spends its time. Each
metrics_scope_* bracket in opt_run_o1_native and the finalize sweep becomes
one <scope> <ticks> ticks (<calls> calls) line; the counters (opt.funcs,
opt.blocks, opt.pregs, the inliner refusal histogram, heap allocs, …) follow.
Ticks are the raw host cycle counter (cntvct_el0 on arm64 ≈ 24 MHz, so divide
by hw.tbfrequency for wall-seconds); within one run the scopes are directly
comparable. The named scopes nest — opt.inline.total and opt.o1.total are the
two roots of the finalize sweep, and compile.tu covers only the frontend
recording, not the finalize sweep that follows it.
$ KIT_METRICS=1 kit cc -O1 -c sqlite3.c --sysroot "$SDK" -o sqlite3.o
kit metrics:
compile.tu ~10M ticks (frontend: lex+pp+parse+CG recording)
opt.inline.total ~6M ticks (whole-program inliner, 1 call)
opt.o1.total ~38M ticks (per-function pipeline, summed over 2580 funcs)
opt.regalloc ~17M ticks (biggest per-function pass)
...
opt.funcs=2580 opt.blocks=122208 opt.tiny_inline.inlined=13 opt.inline.inlined=730
Measured cost and scaling (sqlite amalgamation, arm64/Darwin)
Compiling the 263 K-line sqlite amalgamation (1.6 s — about 10× -c, release kit), -O1 runs in
**-O0** (0.16 s; tcc does the whole file in 0.06 s). The
time is dominated by the per-function pipeline, with the linear-scan allocator
and live-range construction the largest line items — exactly the shape a no-SSA
-O1 should have:
| phase (24 MHz ticks → s) | seconds | share |
|---|---|---|
frontend (compile.tu) |
~0.3 | ~20% |
CG→opt lowering (cg_ir_lower) |
~0.4 | ~25% |
whole-program inliner (opt.inline.total) |
~0.24 | ~15% |
| per-function pipeline (machinize→regalloc→emit) | ~0.7 | ~45% |
| — of which regalloc + live-ranges | ~0.6 |
The cost scales linearly in function count. A synthetic sweep (N small
functions in a dense call graph, best-of-3) — doubling N roughly doubles -O1
time, converging on the same slope as -O0:
| funcs | -O0 |
-O1 |
O1/O0 | -O1 ×/2× funcs |
|---|---|---|---|---|
| 400 | 0.027 | 0.038 | 1.4× | — |
| 800 | 0.030 | 0.049 | 1.6× | 1.3× |
| 1600 | 0.034 | 0.074 | 2.2× | 1.5× |
| 3200 | 0.043 | 0.131 | 3.1× | 1.8× |
| 6400 | 0.062 | 0.264 | 4.3× | 2.0× |
No phase has a superlinear axis. The inliner stays linear in call sites because
its hot gates are O(1) — the InlineIndex (Section 4) resolves callees through a
symbol hash map and answers the recursion check with an scc[caller] == scc[callee] compare over precomputed SCC ids — and per-function lowering plus the
linear-scan allocator are linear in turn.
Inline hints
The inliner honors the frontend's per-function KitCgInlinePolicy, which the C
frontend derives from declaration hints:
| C hint | policy | inliner behavior |
|---|---|---|
| (none) | DEFAULT | inline if cost ≤ 20 |
inline / static inline |
HINT | inline if cost ≤ 40 |
__attribute__((always_inline)) |
ALWAYS | inline regardless of cost (recursion still blocks) |
__attribute__((noinline)) |
NEVER | never inline |
So a static inline body inlines at sizes a plain static one would not,
always_inline ignores the budget entirely, and noinline is always honored.
(The inline keyword is a declaration specifier, distinct from the
__attribute__ flags; the frontend merges it onto the declaration so the HINT
policy reaches codegen.) test/opt/whole_program_inline.sh guards all four
policies on every arch.
Code quality
-O1 produces denser code than -O0 despite doing no SSA optimization, and is
already smaller than tcc — the "reasonable code" half of the goal holds:
| metric (sqlite3.o) | -O0 |
-O1 |
Δ vs O0 |
|---|---|---|---|
__TEXT (code) |
1.41 MB | 1.26 MB | −10.6% |
| total object | 1.75 MB | 1.62 MB | −7.3% |
| (tcc total object, ref) | — | 2.11 MB | — |
Most of the density comes from the cheap per-function transforms
(promote_scalar_locals, the address folds, dead_def_elim, mir_combine,
linear-scan allocation) rather than from the inliner's 730 inlines. The -O1
object links and runs correctly: the ecosystem gate compiles and runs sqlite at
-O0 and -O1 against clang.
The three copy/extension folds above (addr_of [base+0], same-width convert,
ZEXT-of-load) are examples of the local, target-agnostic canonicalizations that
make up the -O1 density work. Each was added after a disassembly audit of the
ecosystem -O1 output found the same redundancy recurring: a register move per
&*p, a move per pointer cast, and a uxtb/uxth after every narrow unsigned
load (C's integer promotions). The folds are not the whole story — -O1 text is
still several times clang's on inline-heavy files, because the wins clang gets
from GVN / DSE / redundant-load elimination and post-inline cleanup are SSA-only
and stay parked in the O2 mid-end (Section 3).
The shape of -O1 quality work
Two design constraints bound everything done to improve -O1 generated code,
and together they define the method:
No SSA — the compile must stay linear.
-O1is the no-SSA pipeline of Section 3: local + linear-scan machinery, no dominance-frontier phi insertion, no value numbering, no interference graph. A-O1density transform must keep that property — a per-functionO(n·log n)slot sort or a bounded per-block side table is fine, but anything that needs SSA, a full interference graph, or an O(n²) analysis belongs in the parked O2 mid-end, not here. The synthetic scaling sweep (Section 8) and sqlite-O1timing are the guard that a quality change did not add a superlinear axis; the correctness gate is the ecosystem run-and-diff vs clang at-O0/-O1plus the toy/opt suites (these changes deliberately alter emitted bytes, so byte-identity is not the gate).The transforms are linear same-block peepholes. Within that budget the realized wins are a family of forward, single-block peepholes over the post-RA MIR (most live in
pass_combine.c, branch-shaped ones inpass_jump.c): store-to-load forwarding across a register mismatch, boolean-into-branch fusion (cset;cbnz→b.cc), redundant-extension drops, single-use copy/shift folding into the consuming op, same-block redundant-load/CSE, and same-block stack dead-store elimination. Each reuses the combiner's existing per-block last-def map and hard-register liveness, so it stays O(1) per instruction. The structural ceiling these cannot reach — the cross-block over-spilling that dominates the residual gap on large, high-pressure functions — needs the SSA register allocator and is out of scope for-O1(Section 3 / the optimizer roadmap).
Frame shape drives density
The largest -O1 density lever is not a peephole but the frame layout, and
it exploits a standing -O1 asset the -O0 single-pass path lacks: at -O1
the frame is fully known before the body is emitted (*_func_begin_known_frame
fixes the final frame size and slot list up front). Two layout choices follow:
Hot-slot-low ordering (all arches). Body frame slots are ordered so the most-frequently-accessed spills get the smallest final displacement from the layout's addressing base — small offsets are cheapest to encode everywhere (
disp8vsdisp32on x64, inside the scaled reach on aa64, inside the ±2 KBimm12window on rv64). The per-slot priority is the allocator's spill-cost metric, aggregated over every PReg that shares a reused slot; ordering is a layout choice with no added analysis (one slot sort per function).Positive-offset spill addressing (aa64). aa64's only wide single-instruction memory form is the unsigned-scaled
ldr/str [base,#pos](reach 0..32760); its signed unscaledldurreaches only ±256. So one-instruction slot access requires the addressing base to sit below the slots. The aa64 known-frame layout therefore anchors the frame pointer x29 at the bottom of the static slots (just above the outgoing-arg area), uniformly, so every slot is a one-instruction positiveldr/str [x29,#k](theadd x16,x29,#hi; ldr [x16,#lo]build only past the 32 KB scaled reach). This replaces the old top-anchored layout'ssub x17,x29,#k; ldurfallback — two-to- four instructions per access, recomputed for each access — which on large high-pressure functions was the single biggest-O1density cost. x29 is the (always reserved, alloca-stable) frame pointer, so the change is contained in the aa64 backend with no regalloc change, and alloca falls out for free: the saved fp/lr pair is co-located at x29 ([x29]/[x29+8]) to preserve the frame-pointer chain kit's unwinder walks, and outgoing args stay sp-relative so calls after an alloca still address their arg area from the current sp. x64 needs none of this —mov [rbp-disp32]already reaches any slot in one instruction; rv64's ±2 KB window covers any realistic hot working set, so it takes the shared ordering only.
The wins compound: hot-slot ordering and positive addressing shrink the cost of each spill, the linear move coalescer (Section 4) lowers the spill count, and rematerialization (recompute an input-less spilled value at its use instead of reloading) cuts spill traffic. None of the three needs SSA.