commit 8d7927db63f2281b5292479137dfb30a9345e263
parent d5e7e1a3f6222191af65cfae4c7b015631d8ce73
Author: Ryan Sepassi <rsepassi@gmail.com>
Date: Sat, 18 Jul 2026 23:54:21 -0700
refactor(opt): make scratch and clobber ownership explicit
Separate semantic HIR from location MIR, centralize register effects, and give native emission and inline asm distinct scoped temp banks across backends.
Add red-green unit, optimizer, corpus, cross-architecture, bootstrap, and backend-pressure coverage for the new boundaries.
Diffstat:
100 files changed, 11982 insertions(+), 3002 deletions(-)
diff --git a/doc/ARCH.md b/doc/ARCH.md
@@ -2,14 +2,14 @@
This document describes kit's arch backend abstraction: how a target
architecture plugs into the compiler, what its responsibilities are, and how
-the three native backends (aa64, x64, rv64) are structured to maximize sharing
-while keeping the ISA-specific seams thin. It also covers the ABI / calling
-convention layer in `src/abi`, which is the single authority for storage layout
-and call classification. The semantic codegen surface a backend sits behind is
-in [CODEGEN.md](CODEGEN.md); the IR the optimizer feeds it is in [IR.md](IR.md);
-the no-SSA O1 and future SSA O2 optimizer paths are in [OPT.md](OPT.md);
-the standalone assembler that shares the ISA tables is in [ASM.md](ASM.md). ABI
-content is canonical here.
+the four native backends (aa64, x64, rv64, arm32) are structured to maximize
+sharing while keeping the ISA-specific seams thin. It also covers the ABI /
+calling convention layer in `src/abi`, which is the single authority for
+storage layout and call classification. The semantic codegen surface a backend
+sits behind is in [CODEGEN.md](CODEGEN.md); the IR the optimizer feeds it is in
+[IR.md](IR.md); the no-SSA O1 and future SSA O2 optimizer paths are in
+[OPT.md](OPT.md); the standalone assembler that shares the ISA tables is in
+[ASM.md](ASM.md). ABI content is canonical here.
## 1. Two layers of "backend": CGBackend and ArchImpl
@@ -53,9 +53,9 @@ struct-prefix subtyping (`src/cg/cgtarget.h`, `src/arch/arch.h`).
debugger op tables, the register file, the predefined macros (`__aarch64__`,
`__x86_64__`, ...), and the DWARF CFI defaults that seed the `.eh_frame` CIE.
-`aa64`, `x64`, and `rv64` expose full `ArchImpl`s (`arch_impl_aa64`,
-`arch_impl_x64`, `arch_impl_rv64`). `wasm` also exposes an `ArchImpl`
-(`arch_impl_wasm`), but it is a *thin* one: its machine-code seams
+`aa64`, `x64`, `rv64`, and `arm32` expose full `ArchImpl`s (`arch_impl_aa64`,
+`arch_impl_x64`, `arch_impl_rv64`, `arch_impl_arm32`). `wasm` also exposes an
+`ArchImpl` (`arch_impl_wasm`), but it is a *thin* one: its machine-code seams
(`asm_new`, `apply_label_fixup`, `link`, register accessors, CFI) are all NULL,
since wasm32 has no native machine encoding, no stack-frame ABI, and no
assembly form in this toolchain — it produces a `WasmModule` attached to the
@@ -105,12 +105,14 @@ answer rather than a crash.
## 3. The NativeTarget contract
`src/arch/native_target.h` defines `NativeTarget`, the physical machine-emission
-contract that all three native backends implement. It is the layer where the
+contract that all four native backends implement. It is the layer where the
generic codegen drivers stop speaking in semantic terms (`CGLocal` ids,
high-level types) and start speaking in physical terms: hard registers, frame
slots, legal immediates, and concrete addressing modes. A `NativeTarget` never
-allocates registers and never decides storage layout — callers hand it
-caller-selected, target-legal physical operands; the target only encodes.
+assigns persistent homes to program values and never decides source storage
+layout. Callers supply target-legal physical locations. A backend may use its
+fixed, backend-private implementation registers while realizing a hook, but
+those temporaries are not MIR values and cannot escape the hook.
It is driven from two directions:
@@ -136,10 +138,21 @@ It is driven from two directions:
- At **-O1**, the optimizer records IR, builds a CFG, runs the no-SSA local
schedule, machinizes, and allocates registers with the linear-scan path (see
[OPT.md](OPT.md)); `src/opt/pass_native_emit.c` then replays the allocated
- program against the *same* `NativeTarget` hooks. By this point every value
- already has a physical home, so the emit pass hands the target hard registers
- and frame slots and the target just encodes. `-O2` currently selects this same
- path; the maintained SSA schedule is not publicly reachable yet.
+ program against the *same* `NativeTarget` hooks. By this point every program
+ value has one canonical MIR home (hard register, frame slot, or a
+ rematerialization recipe). The emit pass leases target-declared
+ instruction-local operand temps as needed, with a liveness-proven dead
+ caller-saved allocation register as a pressure fallback, and hands concrete
+ `NativeLoc`s to the backend. `-O2` currently selects this same path; the
+ maintained SSA schedule is not publicly reachable yet.
+
+ A physical address may retain a frame-resident base or index as
+ `NATIVE_ADDR_BASE_FRAME_VALUE` / `NATIVE_ADDR_INDEX_FRAME_VALUE`. Its
+ `base_type` / `index_type` is the exact scalar type to load, not merely the
+ pointer width. Materializing that component is a `NativeTarget`
+ responsibility and uses the explicit destination or backend-private fixed
+ temps as needed; it does not create a second allocator or change the
+ component's MIR home.
That a single ~35-hook contract serves both paths is what keeps the two code
generators byte-compatible per arch. The hook families:
@@ -158,11 +171,14 @@ generators byte-compatible per arch. The hook families:
`store`, `tls_addr_of`, `copy_bytes`/`set_bytes` (aggregate memcpy/memset),
`bitfield_load`/`bitfield_store`, `spill`/`reload`.
- **Arithmetic.** `binop`, `unop`, `cmp`, `convert`, `alloca_`.
-- **Calls & returns.** A two-phase split: `plan_call` turns a `NativeCallDesc`
- into a `NativeCallPlan` (arg moves, return slots, clobber/return masks,
- outgoing stack size) that the optimizer can inspect during frame planning, and
- `emit_call` realizes it. Symmetrically `plan_ret`/`ret`. `call_stack_bytes`
- and `signature_stack_bytes` are pure pre-pass queries used to size the
+- **Calls & returns.** A marshalling/emission split: `plan_call` is deliberately
+ mutating. It fills a `NativeCallPlan` and performs target-owned pre-call work
+ such as indirect-callee staging, outgoing stack stores, and register
+ parallel copies; the shared caller realizes any remaining plan argument
+ moves. `emit_call` emits the actual control transfer. `plan_ret` describes
+ and may perform target-owned return marshalling, then returns any remaining
+ ordered moves for the caller to realize before `ret`. `call_stack_bytes` and
+ `signature_stack_bytes` are the pure pre-pass queries used to size the
outgoing area and to decide tail-call (sibling) realizability.
- **Atomics & fences.** `atomic_load`/`store`/`rmw`/`cas`, `fence`.
- **Variadics.** `va_start_`/`va_arg_`/`va_end_`/`va_copy_`. All `va_list`
@@ -177,14 +193,20 @@ A handful of small capability flags/queries let the generic drivers specialize
without arch branches: `imm_legal`/`addr_legal` (immediate and addressing-mode
legality), `has_store_zero_reg`/`store_zero_reg` (aa64 `xzr`, rv64 `x0` — store
a constant 0 without materializing it), and the optional `machine_op_clobbers`,
-which reports the fixed registers an encoding clobbers as a side effect (x86
-`idiv` writes rax/rdx, a variable shift uses cl, atomics use rax/rcx/rdx) so the
-allocator keeps values out of them; aa64/rv64 leave it NULL because their
-encodings have no such fixed clobbers.
+which reports the fixed registers an encoding or target sequence destroys so
+the allocator keeps live-across values out of them. x64 reports division,
+variable-shift, atomic, bitfield-store, intrinsic, syscall, and descriptor-TLS
+effects; aa64 reports syscall and descriptor-TLS effects; rv64 reports syscall
+effects. A backend leaves the hook NULL only when none of its operations have
+fixed clobbers. This hook is an exhaustive preservation contract: except for
+explicit destination registers and reported effects, an ordinary
+`NativeTarget` operation preserves every caller-owned register, including one
+holding a clean frame-cache entry. Machinization rejects any reported bit that
+is not a known physical register in that class.
**aa64 is the reference backend.** `src/arch/aa64/native.c` is the most complete
-and most heavily commented implementation; the x64 and rv64 ports are written
-against it. Shared scaffolding extracted across all three lives in
+and most heavily commented implementation; the x64, rv64, and arm32 ports are
+written against it. Shared scaffolding extracted across all four lives in
`src/cg/native_frame.c` (slot-offset arithmetic, the frame-final gate, the
used-callee-save derivation, ABI-driven va-save sizing) and
`src/cg/native_argmove.c` (the parallel-copy register shuffle for call-arg and
@@ -290,31 +312,49 @@ exposes for objdump/asm live separately in `regs.c`). A `NativeRegInfo` is a set
of `NativeAllocClassInfo` (one per `NativeAllocClass`: INT, FP, VEC), each
carrying:
-- an ordered **allocable** list — registers the allocator may assign, ordered by
- preference (aa64 lists caller-saved first so the allocator prefers them and
- avoids prologue saves);
-- a **scratch** list — registers reserved for the backend's own temporaries
- (address materialization, atomic retry loops) and never handed to the
- allocator;
+- an ordered O0 **ndt_allocable** list — the `NativeDirectTarget` value-cache
+ bank and preference order; it is never an O1 allocator input;
+- an O0 **scratch** list — the `NativeDirectTarget` value-cache/acquisition
+ policy only;
+- an O1 **emit_temps** list — non-allocable, operand-facing registers leased for
+ exactly one native-emission instruction; these are MIR materialization
+ locations, not backend-private implementation registers;
+- an **asm_temps** list — a disjoint backend-internal bank the optimized emitter
+ may borrow only while staging a compatible inline-asm block;
+- an **emit_cache_mask** — the subset of `emit_temps` whose clean spill values
+ an emitter may retain across ordinary hooks within one block without an ABI
+ save; a dead O1 allocation register may also become a cache entry when its
+ preservation is already established;
- a `NativePhysRegInfo` row per physical register (class, ABI arg/ret index,
- caller/callee-saved flags, spill/copy costs);
+ caller/callee-saved flags, the independent `NATIVE_REG_ALLOCABLE` O1 flag,
+ and spill/copy costs);
- precomputed caller/callee/arg/ret/reserved bitmasks.
-This one declaration feeds **both** code paths:
+This one declaration describes **both** code paths without making either path's
+register bank an input to the other:
- the **-O0 direct path** resolves `reg_info` and the three `class_info[]`
pointers once at `NativeDirectTarget` construction, so its register cache
- (allocate / evict / scratch-acquire) is an O(1) lookup. It uses the allocable
- order as a simple "next free register" pool with conservative flushes.
-- the **-O1 allocator** (`src/opt`) consumes the same allocable lists, costs, and
- masks as its interference-graph inputs, and reports the callee-saves it
+ (allocate / evict / scratch-acquire) is an O(1) lookup. It uses only the
+ `ndt_allocable` order and O0 scratch policy with conservative flushes.
+- the **-O1 allocator** (`src/opt`) derives its bank from
+ `NativePhysRegInfo.flags & NATIVE_REG_ALLOCABLE`, consumes the physical costs
+ and masks as its interference-graph inputs, and reports the callee-saves it
actually used back through `func_begin_known_frame` / `reserve_callee_saves`
so the backend can reserve save slots and emit the matching prologue/epilogue.
-
-Because incoming arg registers are marked non-allocable, register-destination
-param binds can never alias a live incoming arg, which is what lets `bind_param`
-ordering be unconstrained and lets `bind_params_end` resolve a param permutation
-as a single parallel copy.
+ The optimized emitter independently consumes the validated `emit_temps`,
+ `asm_temps`, and `emit_cache_mask` ownership declarations. It never falls back
+ to the O0 scratch or `ndt_allocable` policies, though it may lease a
+ caller-saved O1 allocation register proven dead at the instruction.
+ Cross-instruction cache retention is limited to `emit_cache_mask` or a dead
+ O1 allocation register whose caller/callee-save preservation is already
+ established, with calls and exhaustive machine effects providing the
+ invalidation boundaries.
+
+The ABI-argument role and persistent allocability are independent target
+choices. Each backend captures incoming parts before body emission can overwrite
+them; deferred register destinations are completed by `bind_params_end` as one
+parallel copy, so a parameter permutation has an explicit ownership boundary.
## 7. The ABI / calling-convention layer
diff --git a/doc/BUILD.md b/doc/BUILD.md
@@ -324,28 +324,33 @@ following approach is the default starting point.
target object — isolating a linker bug from codegen for a specific source file.
- **Inspect MIR around the suspect symbol.** A filtered MIR dump taken after
lowering and the combine pass is usually enough to see the divergence (e.g. a
- call argument referencing a backend scratch register where it should reference
- an allocable one).
+ call argument location aliasing an already-owned ABI destination, or a spill
+ whose frame slot/type changed across a transformation).
- **Avoid `-g` while triaging `-O1` codegen.** Debug info changes object layout
and can create or mask layout-sensitive bugs; triage on the non-`-g` object
first.
The bug classes that have historically broken the `-O1` fixed point all live in
-the interaction between the optimizer's register-level reasoning and the
-backend's scratch-register discipline, with the object/link layer as where the
-symptom surfaces. Keep them in mind when bringing up a new arch or platform:
-
-- **Operand clobber in native emit** — materializing the left operand of a
- binop/compare into a scratch register that already holds the right operand
- (compute the RHS location first and exclude its register when materializing the
- LHS).
-- **Copy propagation across backend scratch registers** — scratch registers may
- appear in lowered MIR but must not be extended across later instructions, since
- native lowering reuses them as transient temporaries.
+the interaction between physical-location MIR, canonical register effects, and
+the emitter's instruction-local temporary leases, with the object/link layer as
+where the symptom surfaces. Keep them in mind when bringing up a new arch or
+platform:
+
+- **Operand clobber in native emit** — leasing a materialization register that
+ already holds another operand or phase-owned ABI destination. The instruction
+ scope derives exclusions from canonical uses/defs/clobbers and holds every
+ destination until its multi-move phase completes.
+- **Emitter temporaries escaping into MIR** — location MIR contains only
+ physical allocation registers, stack values, frame-address recipes, and
+ semantic operands. `emit_temps`/`asm_temps` are scoped emitter-owned leases;
+ the MIR verifier rejects them rather than relying on later passes not to
+ extend their lifetime.
- **Coalesce overlap checks** must use raw range points, not compressed points.
- **Lower-pass hint fallback** must not place values live across a call into
caller-saved hint registers.
-- **Native scratch budget** — a backend needs enough integer scratch registers
- for all-spilled three-operand operations (aa64 needs three).
+- **Native emission-temp budget** — a backend needs enough operand-facing
+ `emit_temps` for all-spilled three-operand operations (aa64 needs three),
+ independently of its O0 `scratch`/`ndt_allocable` policy and backend-private
+ implementation registers.
- **Aggregate copy/set with pointer operands** must not force-home the pointer
local; genuinely frame-backed pointer locals need prematerialized indirect bases.
diff --git a/doc/CODEGEN.md b/doc/CODEGEN.md
@@ -270,9 +270,11 @@ load-bearing invariants are:
- **What is cached:** only scalar locals that fit a register and are neither
address-taken nor memory-required. Aggregates and escaped locals stay
frame-only.
-- **Where:** only caller-saved allocable registers. This sidesteps prologue
- bookkeeping (the direct path reports no clobbered callee-saves) and means the
- blanket flush before any call already covers ABI clobbering.
+- **Where:** only the backend-declared `ndt_allocable` bank, which is independent
+ of the O1 allocator bank. Targets may restrict NDT to caller-saved registers;
+ x64 instead admits callee-saved entries (r13-r15 under SysV, plus rsi/rdi on
+ Win64) and reports every one actually used so the backend reserves and emits
+ the matching save/restore slots.
- **Block scope:** with no CFG/liveness, a cached value cannot cross a control
edge or join. The cache is spilled and emptied at every branch, label, and
return; `func_begin` starts empty.
@@ -283,9 +285,11 @@ load-bearing invariants are:
at that register instead of reloading the frame home. Direct frame-home
accesses to a cached local (e.g. by-value field extraction) flush just that one
local.
-- **Calls and barriers** still flush the whole cache (caller-saved regs die,
- and a memory clobber may observe everything). Address-taking a cached local
- flushes just that local and marks it uncacheable thereafter.
+- **Calls and barriers** still flush the whole cache: caller-saved registers may
+ die and a memory clobber may observe every local. This is deliberately
+ conservative even when an entry occupies a preserved callee-saved register.
+ Address-taking a cached local flushes just that local and marks it
+ uncacheable thereafter.
A monotonic use-tick drives approximate-LRU eviction; cached locals are tracked
in an intrusive insertion-order list so flush-all is O(cached), not O(locals).
@@ -337,9 +341,10 @@ tail-call / entry permutations it is free to rotate). When a true cycle remains
member into a scratch register and redirecting that value's readers.
The scheduling — topological emission, cycle detection, the scratch break — is
-identical across the three arches; only the leaf operations differ (how one move
-is emitted, which register is scratch), supplied through a small ops struct. All
-three native backends plus the entry param-bind path share this one scheduler.
+identical across native architectures; only the leaf operations differ (how one
+move is emitted, which register is scratch), supplied through a small ops
+struct. Every native backend plus the entry param-bind path shares this one
+scheduler.
## Why this shape
diff --git a/doc/INTERFACES.md b/doc/INTERFACES.md
@@ -123,20 +123,20 @@ fixed header count, which becomes stale whenever a public component is added.
The codegen path is **tiered**: each tier is a distinct contract struct (a
vtable) that a backend or layer fills in. A frontend records into the highest
-tier; the bytes come out of the lowest. The native backends — aarch64, x64, and
-rv64 — are all live and all satisfy the same contracts. aarch64 is the reference
-implementation; x64 and rv64 are full peers built on the shared
-`NativeTarget` / `NativeOps` / `NativeFrame` substrate rather than bespoke
-per-arch frame and lowering code.
+tier; the bytes come out of the lowest. The native backends — aarch64, x64,
+rv64, and arm32 — are all live and satisfy the same contracts. aarch64 is the
+reference implementation; the other backends are full peers built on the
+shared `NativeTarget` / `NativeOps` / `NativeFrame` substrate rather than
+bespoke per-arch frame and lowering code.
| Tier | Header | Contract type | What implements it | Role |
|------|--------|---------------|--------------------|------|
-| ABI | `src/abi/abi.h` (+ `abi_internal.h`) | `TargetABI` | per-ABI TUs (`aapcs64`, `aapcs64_windows`, `sysv_x64`, `rv64`, `wasm32`, `apple_arm64`, `apple_x64`, `win64_x64`) | calling-convention + storage-layout queries; `abi_new(Compiler)` selects the implementation by `(arch, obj-format)` |
+| ABI | `src/abi/abi.h` (+ `abi_internal.h`) | `TargetABI` | per-ABI TUs (`aapcs32`, `aapcs64`, `aapcs64_windows`, `sysv_x64`, `rv64`, `wasm32`, `apple_arm64`, `apple_x64`, `win64_x64`) | calling-convention + storage-layout queries; `abi_new(Compiler)` selects the implementation by `(arch, obj-format)` |
| Arch registry | `src/arch/arch.h` | `ArchImpl` | one singleton per arch, via `arch_lookup(kind)` | discovery + dispatch to backend/decode/emu/link/dbg/dwarf surfaces; CFI defaults |
| Semantic CG | `src/cg/cgtarget.h` | `CgTarget` | `native_direct_target` (-O0) or `opt_cgtarget` (-O>=1), wrapping a per-arch `CgTarget` | frontend-facing typed lowering, pre-regalloc |
| -O0 adapter | `src/cg/native_direct_target.h` | `NativeDirectTarget` + `NativeOps` | shared adapter, parameterized by each arch's `NativeOps` | adapts a `NativeTarget` to `CgTarget` for the direct -O0 path |
-| Physical emit | `src/arch/native_target.h` | `NativeTarget` | aa64/x64/rv64 `*_native_target_new()` | hard-register, machine-code emission + frame/CFI |
-| Frame model (shared) | `src/cg/native_frame.h` | `NativeFrame` | shared `native_frame.c`, embedded by all three native backends | arch-neutral frame-slot bookkeeping the `NativeTarget` impls delegate to |
+| Physical emit | `src/arch/native_target.h` | `NativeTarget` | aa64/x64/rv64/arm32 `*_native_target_new()` | hard-register, machine-code emission + frame/CFI |
+| Frame model (shared) | `src/cg/native_frame.h` | `NativeFrame` | shared `native_frame.c`, embedded by all four native backends | arch-neutral frame-slot bookkeeping the `NativeTarget` impls delegate to |
| Machine code | `src/arch/mc.h` | `MCEmitter` | one generic impl, `mc_new(Compiler, ObjBuilder)` | section/label/reloc/CFI byte emission for all machine-code archs |
**Per-arch entry points** — the surface each backend exposes to the rest of the
@@ -147,7 +147,8 @@ its raw word/halfword emit helpers for the assembler path:
|------|--------|--------------|
| aa64 (reference) | `src/arch/aa64/aa64.h` | `aa64_native_target_new`, `aa64_native_direct_ops` |
| x64 | `src/arch/x64/x64.h` | `x64_native_target_new`, `x64_native_direct_ops` |
-| rv64 | `src/arch/rv64/rv64.h` | `rv64_native_target_new`, `rv64_native_direct_ops`, `rv64_emit32/16` |
+| rv64 | `src/arch/riscv/rv64.h` | `rv64_native_target_new`, `rv64_native_direct_ops`, `rv64_emit32/16` |
+| arm32 | `src/arch/arm32/arm32.h` | `arm32_native_target_new`, `arm32_native_direct_ops` |
| c_target | `src/arch/c_target/{c_emit,ir_emit}.h` | C-source emission backend (standalone `CGBackend`, no `ArchImpl`) |
| wasm | `src/arch/wasm/*` | wasm emission backend |
@@ -156,19 +157,37 @@ its raw word/halfword emit helpers for the assembler path:
policy, control flow, data movement, arithmetic/compare/convert, calls
(`plan_call`/`emit_call`/`plan_ret`/`ret`), atomics, variadics, intrinsics, and
inline/file-scope asm — roughly three dozen hooks. The caller (NDT or the
- optimizer) has already selected legal physical operands and run register
- allocation; `NativeTarget` validates and emits but must never allocate
- registers itself. All three native backends implement the full contract.
+ optimizer) owns program-value placement and supplies legal physical
+ locations. A backend may materialize exact-typed `FRAME_VALUE` address
+ components using the explicit destination or fixed backend-private
+ implementation temps, but it must never allocate a persistent program value
+ or borrow a caller-owned register outside that hook. All four native backends
+ implement the full contract.
- The same arch fills both `NativeTarget` (physical) and `NativeOps` (the
semantic shims the -O0 adapter calls). Keep the split clean: semantic decisions
- (operand legality, call planning policy, `va_*`, asm binding, barriers) live in
- `NativeOps`; pure emission lives in `NativeTarget`.
+ and semantic-to-physical adaptation live in `NativeOps`; physical
+ marshalling, validation, and instruction emission live in `NativeTarget`.
+ `plan_call` is an intentionally mutating marshalling phase: it fills the plan
+ while performing target-owned callee/argument staging and stack stores.
+ `plan_ret` is the corresponding mutating return-marshalling phase;
+ `call_stack_bytes` is the pure frame-sizing query.
- A handful of `NativeTarget` hooks are explicitly optional and exist for archs
- whose ISA needs them — `machine_op_clobbers` (x86 idiv/shift fixed-register
- clobbers; NULL on aa64/rv64), `emit_prologue` / `emit_minimal_prologue`,
- `bind_params_end` (for backends that resolve param binds as a parallel copy),
+ whose ISA needs them — `machine_op_clobbers` (including x64 division, shifts,
+ bitfield stores, atomics, and descriptor TLS, plus aa64 syscall and descriptor
+ TLS effects and rv64 syscall effects), `emit_prologue` /
+ `emit_minimal_prologue`, `bind_params_end` (for backends that resolve param
+ binds as a parallel copy),
and the zero-register store fast path (`has_store_zero_reg`). NULL is the
documented "this arch doesn't need it" answer, not an unimplemented gap.
+ The `machine_op_clobbers` contract is exhaustive: NULL asserts zero fixed
+ optimizer-visible clobbers, while an implementation must report every
+ exception, including an `emit_cache_mask` temporary when an operation uses it
+ internally. Backend-private registers are outside that contract. An ordinary
+ hook preserves every caller-owned or cache-eligible register that is neither
+ an explicit destination nor reported by this effect. That contract
+ permits clean frame values to remain in `emit_cache_mask` registers and
+ already-preserved dead O1 allocation registers between invalidation
+ boundaries.
- `CgTarget` likewise carries capability queries so the semantic layer can stay
arch-neutral: `supports_label_table` (false on Wasm, which has no code
addresses in linear memory), `switch_` (overridden only by backends with a
@@ -184,7 +203,7 @@ its raw word/halfword emit helpers for the assembler path:
Every native backend lays out a stack frame the same way at the *bookkeeping*
level, so that bookkeeping was lifted into one shared module. `NativeFrame` owns
the arch-neutral parts; each `NativeTarget` embeds one and keeps the ISA/ABI
-specifics. All three native backends embed a `NativeFrame`.
+specifics. All four native backends embed a `NativeFrame`.
**The split — what `NativeFrame` owns vs. what stays in the backend:**
@@ -198,8 +217,9 @@ specifics. All three native backends embed a `NativeFrame`.
**Why it is shared:** the slot arithmetic, the no-grow-after-prologue gate, and
the derivation of the used-callee-save set from the optimizer's per-class masks
-are identical across the three archs. Consolidating them also folds the three
-per-arch vararg-save magic numbers (rv64 64, x64 176, aa64 64+128) into a single
+are identical across the native archs. Consolidating them also folds the
+per-arch vararg-save sizes (rv64 64, x64 176, aa64 64+128; zero where an ABI
+does not require a register-save area) into a single
ABI-driven `native_frame_va_save_bytes` query, in line with the no-magic-numbers
rule. Frame-slot handles are **1-indexed** with `NATIVE_FRAME_SLOT_NONE` as the
sentinel; callers and the shared allocator must agree on that.
diff --git a/doc/JIT.md b/doc/JIT.md
@@ -240,10 +240,11 @@ A subtlety the access lowering imposes on codegen: the Mach-O TLV sequence
materializes the descriptor in `x0` and (in the AOT form) calls the
resolver thunk via `x16`, clobbering `x0`/`x16`/`x17`/`lr`. Codegen is
shared between AOT and JIT, so the optimizer must model that clobber set
-or a value left live in `x0` across a TLS access is corrupted at `-O1`+.
-The backend reports it via `NATIVE_MOP_TLS_ADDR` from
-`machine_op_clobbers` (ELF Local-Exec, which uses only the destination
-register, reports none).
+at its ownership boundary or a value left live in `x0` across a TLS access is
+corrupted at `-O1`+. The backend reports optimizer-visible `x0` via
+`NATIVE_MOP_TLS_ADDR` from `machine_op_clobbers`; `x16`/`x17`/`lr` are reserved
+backend-private registers and therefore remain outside that effect interface.
+ELF Local-Exec, which uses only the destination register, reports none.
## Symbol and inspector surface
diff --git a/doc/OPT.md b/doc/OPT.md
@@ -29,7 +29,7 @@ frontend --CgTarget calls--> CgIrRecorder --records--> CgIrFunc tape
|
passes (analysis + transform)
|
- regalloc -> MFunc (physical MIR)
+ regalloc -> location MFunc (physical MIR)
v
opt_emit_native --NativeTarget calls--> machine code
```
@@ -103,7 +103,7 @@ The optimizer IR lives in `src/opt/ir.h` / `src/opt/ir.c`. Its shape:
- `Func` owns one function: its CFG (`blocks`, `entry`, `emit_order`), frame
slots, params, locals, the pseudo-register table, the SSA value table, scope
- bookkeeping, allocation results, and per-pass scratch.
+ bookkeeping, allocation results, and pass-owned analysis state.
- `Block` is a basic block: a growable `Inst[]`, explicit `preds`/`succ` edges,
and a pre-allocated `MCLabel` for blocks born from `cg_label_new`.
- `Inst` is one recorded operation. The `IROp` enum mirrors the `CgTarget`
@@ -140,6 +140,15 @@ the nearest checkpoint rather than in the backend encoder.
`FrameSlot` is the frame-storage currency: locals forced to memory, spill slots,
ABI parameter slots, alloca regions, and outgoing-argument areas.
+MIR extends the operand vocabulary with locations rather than hiding locations
+behind temporary register ids. `OPK_STACK` is the scalar value in an `FS_SPILL`
+slot, while `OPK_FRAME_ADDR` is a rematerializable address of a semantic frame
+slot. An `OPK_INDIRECT` address records the location kind and exact type of its
+base and index independently: either a physical register, a spilled scalar, or
+(for the base) a frame-address recipe. These kinds never occur in semantic HIR.
+They let post-allocation passes preserve the actual allocation result all the
+way to emission without manufacturing register lifetimes between instructions.
+
### Token aliasing: optimizer-local names onto NativeTarget types
`src/opt/ir.h` deliberately reuses the physical backend's data types as the
@@ -216,11 +225,12 @@ promote_scalar_locals non-escaped scalar frame slot -> PReg
addr_of_global_cse hoist duplicate ADDR_OF(global) to entry
build_loop_tree
lower_loop_imm_operands / hoist_loop_consts loop-invariant imm materialization
+opt_refresh_machine_clobbers rederive effects from final HIR instruction shapes
live_blocks per-block PReg liveness (backward dataflow)
dead_def_elim_with_live pre-RA dead-definition elimination
regalloc_locations PReg -> hard reg / spill slot (no live-range splitting)
verify "post-regalloc"
-lower_to_mir build physical MFunc; insert spill/reload
+lower_to_mir deep-clone HIR; rewrite values to physical locations
mir_verify "lower-mir"
mir_combine post-RA peephole / addressing-mode synthesis
mir_dce post-RA dead-code elimination
@@ -422,17 +432,19 @@ transform or analysis; the file paths orient the reader.
- **Order + dominators + verify** (`src/opt/pass_analysis.c`): postorder /
reverse-postorder, reachability, immediate dominators, dominator children,
dominance frontiers (`OptAnalysis`), the coarse analysis-validity bits
- (`OPT_ANALYSIS_CFG/DEF_USE/DOM/LOOP`), and the debug verifier `opt_verify`.
+ (`OPT_ANALYSIS_DEF_USE/DOM/LOOP`), and the debug verifier `opt_verify`.
- **Liveness** (`src/opt/pass_live.c`): `opt_live_blocks` solves per-block PReg
liveness by backward dataflow into elastic 64-bit-word bitsets (`OptBitset`,
grown on demand, trailing-zero-trimmed); `opt_live_ranges_build` produces the
compressed point-indexed live ranges and per-PReg frequency/spill-cost metrics
the allocator consumes.
- **Hard-register liveness** (`src/opt/pass_hard_live.c`): physical-register
- live-in/out over the post-RA MIR, plus the per-call clobber mask
- (`opt_call_clobber_mask_for`). This is what makes post-RA combine/DCE safe: a
- value in a callee-saved register survives a call, while caller-saved registers
- are killed by it.
+ live-in/out over the post-RA MIR. It consumes the same canonical instruction
+ effects (`opt_inst_reg_effects`) as post-RA combine, DCE, and native emission,
+ so explicit operands, call effects, inline-asm clobbers, and target machine
+ clobbers cannot drift between analyses. This is what makes post-RA
+ combine/DCE safe: a value in a callee-saved register survives a call, while a
+ register in that call's clobber mask is killed by it.
- **Loop detection** (`src/opt/pass_loop.c`): `opt_build_loop_tree` computes loop
nesting depth from dominators; depth feeds the allocator's spill-cost weighting
and LICM.
@@ -445,16 +457,23 @@ transform or analysis; the file paths orient the reader.
- **Machinize** (`src/opt/pass_machinize.c`): `opt_machinize_native` is ABI
lowering against the `NativeTarget`. It annotates calls/returns/params with
calling-convention constraints (argument/result registers, the call clobber
- and return masks, callee-save markers), collects the target's register classes
- (allocable set, reserved scratch set, caller/callee-saved masks) and checks
- allocable and scratch sets do not overlap, resolves inline-asm
- named-register constraint strings into masks, and records per-instruction
- fixed-register clobbers (Section 5).
+ and return masks, callee-save markers), collects and validates the target's
+ register classes (the physical-register table and its O1 allocation flags,
+ the independent O0 `ndt_allocable` cache policy, the O1 emission temporary
+ bank, asm-only temporary bank, caller/callee-saved masks, and cache-preserved
+ subset), resolves inline-asm named-register constraint strings into masks, and
+ records per-instruction fixed-register clobbers (Section 5). The O1 emission
+ bank must be physical, unique, non-allocable, disjoint from the asm-only bank,
+ and large enough for the target's operand-facing hooks. Every target-reported
+ clobber bit is validated against the class's physical-register table before
+ it can enter the allocator/effect side table.
- **MIR view** (`src/opt/pass_mir.c`): the post-allocation physical IR. Rather
than duplicate the CFG passes, `pass_mir.c` builds a transient `Func` *view*
whose block arrays point at `Func.mir`, runs the shared `opt_combine`,
`opt_dce`, `opt_build_cfg`, and `opt_jump_cleanup` over that view, and commits
- it back. The `opt_mir_*` wrappers are thin shims over this view; the shared
+ it back. The MIR itself starts as a deep mutable clone, including operand and
+ aux arrays; HIR remains virtual and cannot be changed through an aliased MIR
+ pointer. The `opt_mir_*` wrappers are thin shims over this view; the shared
passes are written once and reused for both HIR and MIR.
- **Coalescing / allocation** (`src/opt/pass_coalesce.c`, `src/opt/pass_lower.c`):
`opt_regalloc_locations` is a point-bitmap linear-scan allocator producing the
@@ -477,55 +496,71 @@ transform or analysis; the file paths orient the reader.
registers the allocator used, pre-maps frame slots, drives the backend's
minimal-prologue hook when available, routes scalar call results straight to
their destination, uses a hardware zero register for stored zeros where the
- backend advertises one, and legalizes addresses the backend rejects into a
- reserved scratch register. See [ARCH.md](ARCH.md) for the backend contract.
-
-## 5. Machine register-constraint model
-
-Some target instructions pin operands or results to specific physical registers
-and clobber others as a side effect of their encoding — hardware constraints,
-not allocator choices (x86-64 `idiv`/`div` pinning the dividend to `rax` and
-clobbering `rdx`, variable shifts requiring the count in `cl`, one-operand
-`mul`, `cmpxchg`, and the `va_arg` offset scratch). aarch64 and riscv64 have
-no such instructions — their div/shift/mul are ordinary three-operand forms —
-so on those targets the constraint hooks are inert.
-
-The optimizer models all fixed-register requirements through two allocator
-primitives, and the allocator (`pass_lower.c`) speaks only in physical register
-numbers here:
-
-- **Tied hard register** (`OptPRegInfo.tied_hard_reg`): pin a value to a
- specific physical register. Set for inline-asm operands with a `{reg}`
- constraint and for fixed-input/fixed-output machine operands.
-- **Forbidden / clobbered hard registers**
- (`OptPRegInfo.forbidden_hard_regs` / `clobbered_hard_regs`): for each register
- an instruction clobbers, every value live *across* the instruction (live-after,
- not a use or def of it) is forbidden from that register. The clobbered subset
- is recorded separately so the soft return-register placement hint cannot later
- clear a forbid that came from a real hardware clobber.
-
-Three sources feed these, all unified at allocation time:
-
-1. **Calls** — the call plan's `clobber_mask` (caller-saved by default, or the
- call-specific mask) drives the live-across-forbid loop; argument/result
- registers come from the plan.
-2. **Inline asm** — `pass_machinize.c` resolves named-register constraint
- strings and clobber lists into masks and fixed-register indices on the
- `IRAsmAux`; `pass_lower.c`'s `apply_asm_register_constraints` ties the fixed
- operands and runs the live-across-forbid loop.
-3. **Generic machine instructions** — a binop or convert has no `aux` to hang
- constraints on, so machinization queries the target's machine-clobber hook
- per instruction and stores the result in a per-function side table keyed by
- `InstId` (`Func.inst_clobbers`, built in `machinize_inst_clobbers`). At
- allocation, `apply_machine_reg_clobbers` looks up the instruction's clobber
- mask and applies the same live-across-forbid loop. A NULL hook (aa64/rv64)
- means no entries and zero behavior change.
-
-This is the single place where target ISA register rules enter the allocator,
-and all three sources reuse one mechanism — tie + forbid — rather than patching
-assignments after the fact. A value that merely dies at the instruction needs no
-constraint (the backend stages it into/out of the fixed register itself); only
-values that survive past the instruction are forbidden.
+ backend advertises one, and materializes frame/recipe operands in an explicit
+ instruction-scoped temporary lease. The target declares the operand-facing
+ `emit_temps` bank separately from its O0 `scratch` policy and its asm-only
+ `asm_temps`; emitter temporaries never appear as MIR values. See
+ [ARCH.md](ARCH.md) for the backend contract.
+
+## 5. Register constraints and canonical effects
+
+Some instructions need registers that are fixed by their encoding, while other
+instructions destroy registers as a side effect. Those are instruction-local
+machine requirements, not persistent homes for the values involved. For
+example, x86-64 division stages its dividend through `rax` and destroys
+`rax`/`rdx`, a variable shift stages its count through `cl`, and atomics and
+bitfield stores use fixed implementation registers. TLS descriptor sequences
+and syscall intrinsics can also have format- or target-specific effects.
+
+The allocator has one authoritative hard constraint:
+`OptPRegInfo.forbidden_hard_regs`. A hard register present in that mask cannot
+be selected for the PReg. `preferred_hard_reg` is only a placement hint and is
+never allowed to override a forbid. There is no persistent fixed-register tie
+or pin in the allocation model.
+
+The two instruction-boundary sources populate that constraint differently:
+
+1. **Inline asm.** `pass_machinize.c` resolves each register constraint once
+ into an instruction-local `IRAsmRegRequirement` (register class, optional
+ fixed register, and optional allowed-register mask), and resolves the
+ clobber list into per-class masks. With the current unsplit allocator,
+ `pass_lower.c` conservatively forbids the candidate or clobbered registers
+ for PRegs that touch or cross that asm instruction. At emission the
+ optimizer stages each operand into a scoped register lease satisfying the
+ requirement, then writes outputs back to their real MIR homes. A generic
+ whole-class constraint needs no reservation because either the allocated
+ register already satisfies it or a spilled value uses an emitter temp.
+2. **Generic machine instructions.** Machinization initially queries the
+ target's `machine_op_clobbers` hook and stores each nonzero result in the
+ dense, per-function `Func.inst_clobbers` side table keyed by `InstId`.
+ `lower_loop_imm_operands` can still change an instruction's fixed effects
+ (for example, by turning an x64 immediate shift into a register shift), and
+ `hoist_loop_consts` is the final HIR-shape mutator. The explicit
+ `opt_refresh_machine_clobbers` boundary after both passes replaces the table
+ with effects derived from those final HIR shapes. Allocation forbids those
+ registers for non-def values live after the instruction. The backend still
+ performs the instruction-local staging required by the encoding; a dying
+ operand is free to have any otherwise legal home.
+
+Calls use a separate preservation policy rather than feeding this machine
+forbid mechanism. Allocation cost prefers callee-saved registers for values
+that cross calls. If pressure nevertheless puts such a value in a register the
+call clobbers, MIR lowering inserts the required save and restore. ABI argument
+and result placement remains a call-marshalling concern.
+
+The corresponding post-allocation authority is `OptRegEffects`, built only by
+`opt_inst_reg_effects`. It separates explicit `uses`, explicit `defs`, and
+implicit `clobbers`; the distinction is intentional because a clobber kills a
+live value but does not produce one for DCE. Calls contribute their lowered call
+plan and ABI clobber mask, inline asm contributes its bound operands and resolved
+clobber mask, and target instructions contribute the machinization side-table
+mask. Hard-register liveness, MIR combine/DCE, and the native emitter all query
+this one description. Emitter temporary leases are outside this model: they are
+owned by one emission scope and are neither MIR defs nor implicit machine
+clobbers. `machine_op_clobbers` is an exhaustive preservation contract for
+ordinary `NativeTarget` operations: apart from explicit defs and reported
+clobbers, a hook preserves every register that may hold an allocated or cached
+value.
## 6. Allocation, MIR, and the physical boundary
@@ -535,14 +570,56 @@ PReg into `Func.preg_locs` (`OptLoc`: hard register or spill slot). HIR operands
stay virtual after allocation — the verifier checks this.
`opt_lower_to_mir` then builds the physical IR `Func.mir` (an `MFunc`): each
-virtual `OPK_REG` is translated through its `OptLoc` into a physical register or
-a frame access, spilled values get a reload before each use and a store after
-each def, and call plans are lowered into physical argument/return moves. From
-this point the IR is physical and non-SSA (registers may be multiply defined).
-All PReg-to-physical knowledge lives in this one step; after it, the HIR is
-untouched and the MIR is fully physical. The downstream MIR passes (combine,
-DCE, jump/layout cleanup) run over the MIR view and rely on physical-register
-liveness for their safety checks, then `opt_emit_native` replays the MIR.
+virtual `OPK_REG` is translated through its `OptLoc` into either a physical
+`OPK_REG` or an `OPK_STACK` spill value. Spilled indirect bases and indices stay
+as typed frame components, and cheap rematerializable values stay as `OPK_IMM`
+or `OPK_FRAME_ADDR` recipes. Lowering does not surround each spilled use/def
+with synthetic reload/store instructions. It inserts explicit location copies
+only where the program really crosses a physical boundary, such as preserving
+an intentionally caller-clobbered allocation across a call. Call plans are
+lowered to the same location vocabulary.
+
+The MIR is a deep mutable clone, not a shallow rewrite of HIR storage. From this
+point it is physical and non-SSA (registers may be multiply defined), and stale
+HIR definition metadata is cleared. All PReg-to-physical knowledge lives in
+this one step; after it, HIR is untouched and MIR is fully physical. The
+downstream MIR passes (combine, DCE, jump/layout cleanup) run over the MIR view
+and rely on canonical physical-register effects for safety. A pass that grows
+the physical graph commits its extended instruction-id namespace with that
+graph so later side tables remain correctly sized. Adjacent spill copies and
+same-block spill dead stores are normalized over the `OPK_STACK` location
+form. The same canonical operand walker used by liveness and emission accounts
+for every stack use, including aux operands and spilled indirect-address
+components, before DSE can retire an overwritten store. Then
+`opt_emit_native` replays the MIR.
+
+At emission, each instruction opens a `NativeEmitTempScope`. The scope derives
+its unavailable registers from physical live-after plus the instruction's
+canonical uses/defs/clobbers. It prefers target-declared `emit_temps` for
+operand materialization; after that bank is exhausted it may scavenge a
+caller-saved O1 allocation register only when physical liveness proves it dead
+for the instruction. Multiple references to the same location within the
+instruction share a scoped materialization. Calls have a mutating marshalling
+phase: `plan_call` may stage the callee, store stack arguments, and perform
+target-owned argument shuffles while it fills the plan. Each completed ABI
+register destination remains phase-owned through the full multi-move phase.
+After `emit_call`, a phase barrier discards argument/callee staging before
+return writeback begins. Inline asm may additionally borrow `asm_temps` only
+under the target-declared constraints; its staging remains local to that asm
+instruction.
+
+Frame memory is authoritative. A spill definition is stored to its `FS_SPILL`
+slot even when the emitter retains the clean register value for an adjacent
+use; there is no dirty deferred writeback. The optional block-local frame cache
+is keyed by exact slot and value type. Persistent entries may use a
+target-declared cache-enabled emission temporary in `emit_cache_mask`, or a
+dead O1 allocation register whose preservation is already established:
+caller-saved within the current call-free region, or a callee-saved register
+already present in the known frame. The O0 `ndt_allocable` bank is not consulted
+by optimized emission. The exhaustive machine-effect contract invalidates any
+exceptional fixed clobber, and all entries are cleared at block, call, and
+inline-asm boundaries. Thus cache state is a replay optimization, never part of
+MIR semantics or a second owner of a spilled value.
The reason allocation results are a separate table rather than rewritten
operands is the same mode-clarity principle from Section 2: a post-allocation
@@ -557,11 +634,12 @@ checks CFG reciprocity, reachable-block shape, emit-order validity, instruction
ids, operand namespaces (no physical registers in HIR; correct PReg-vs-Val
namespace for the current mode), phi consistency, and def-use freshness;
`opt_mir_verify` checks the physical boundary (no virtual operands, valid frame
-slots, fully physical call plans). Each pass tags its checkpoint with the name
-of the transformation just completed, so a failure localizes to the nearest
-boundary. `Func.opt_valid_analyses` tracks coarse invalidation; passes that
-mutate control flow, operands, or instructions rebuild or invalidate the
-relevant analysis.
+slots and location kinds/types, fully physical call plans, no emitter-owned
+temporary in MIR, and no stale HIR def metadata). Each pass tags its checkpoint
+with the name of the transformation just completed, so a failure localizes to
+the nearest boundary. `Func.opt_valid_analyses` tracks coarse invalidation;
+passes that mutate control flow, operands, or instructions rebuild or invalidate
+the relevant analysis.
Observability hooks: `KIT_DUMP=<tag>` dumps the optimizer IR at a named stage,
`KIT_DUMPCG=1` dumps the recorded semantic tape before lowering,
diff --git a/doc/plan/ARM32.md b/doc/plan/ARM32.md
@@ -46,9 +46,10 @@ goto scratch), `asm_01_grammar` + `builtin_31_readcyclecounter` skipped (see
"Broad parse-corpus triage" below); the 12 wrong-runtime-result reds were already
CLOSED (2026-06-17). The earlier fix that took the
lane from 33 to 17 fails: the O1 frameless-leaf tier returned through a bare
-`BX lr` while the backend uses `lr` as an emit scratch (global-address staging,
-overflow/popcount/bitfield/byte-copy), so any such leaf clobbered its return
-address (the 16-case O1 globals/statics/attributes `rc 134` cluster). The slim
+`BX lr` while the backend uses LR as a backend-private temporary
+(global-address staging, overflow/popcount/bitfield/byte-copy), so any such
+leaf clobbered its return address (the 16-case O1 globals/statics/attributes
+`rc 134` cluster). The slim
tier now saves `lr` (`PUSH {lr}`/`POP {pc}`).
### Broad parse-corpus triage — ALL reds CLOSED 2026-06-17
@@ -59,21 +60,23 @@ data-model-agnostic test rewrites, 2 data-model skips). The 5 remaining
assembler bug it surfaced, 2 aa64-template / no-cycle-counter skips):
- **Emit/codegen gaps (5) — DONE:**
- [x] `gnu_labels_as_values_threaded` O1 — `opt native emit: no scratch
- register` was a computed-goto store whose address consumed both arm32
- opt-scratch (`ip`+`lr`) as base+index, leaving none to materialize the
- stored value. `collapse_addr_to_reg` (shared `pass_native_emit.c`) now
- reuses the base when it is itself a reserved scratch (a transient dead
- after the access): it folds the index into the base in place, freeing the
- other scratch. Arch-neutral; only triggers when no third scratch exists,
- so it is a no-op on register-rich backends. Runs `35` at O0+O1.
+ register` exposed the old emitter's implicit two-register (`ip`+`lr`)
+ staging policy: a computed-goto store consumed both for base+index and
+ still needed to materialize the value. The immediate fix reused a dead
+ address base in place. The later location-MIR redesign removed that
+ policy entirely: ARM32 now declares r0..r3 as instruction-scoped O1
+ operand temporaries, while IP/LR remain backend-private. Runs `35` at
+ O0+O1.
- [x] `asm_01_grammar` O0+O1 — `.arm32.skip`. The templates are aarch64-specific
(`%w0` modifiers, `mov w0`, `dmb sy`, `x0` clobbers) and never assemble on
arm32, exactly as on rv32/rv64/x64. The underlying error (`optimizer asm
input not in a register`) was a *real* arm32 bug too — a register-
constrained input whose value is a frame-resident local (e.g. a value
that is `+r` inout AND returned) arrives as a frame loc, not a register.
- Fixed in `arm32 native.c` `arm_asm_block_native`: stage such inputs into a
- reserved scratch (IP, then LR) before binding, mirroring the aa64 path.
+ The original ARM backend fix staged such inputs before binding. The
+ current shared asm staging path leases an operand register within the
+ instruction scope, with LR exposed separately as an asm-only temporary;
+ IP remains backend-private.
Surfaced and fixed a second bug in `arm32 asm.c`: `add`/`sub rd, rn,
{#imm | rm}` only had the 2-operand hi-register `add` and the sp-adjust
rows, so `add r4, r4, #1` assembled as `add r4, r4` (×2) and `sub r4, r4,
@@ -96,10 +99,12 @@ assembler bug it surfaced, 2 aa64-template / no-cycle-counter skips):
legality accepted the negate-trick but emit never implemented it. Fixed:
shared `arm_resolve_addsub_imm` for legality + emit. (commit e98cc403)
- [x] `variadic_02_many_ints` O1 — under pressure the spilled constant call args
- rematerialized into the round-robin opt-scratch (ip/lr), aliasing the
- simultaneously-live register args. Fixed: exclude call-arg values from
- remat (keep their spill store) + store register-resident stack args
- directly. (commit 51681b45)
+ exposed the old round-robin temporary policy: rematerialized call args
+ could alias simultaneously-live register args. The immediate fix kept
+ those spill stores and wrote register-resident stack args directly; the
+ location-MIR redesign now prevents the class of bug with scoped leases
+ plus phase-owned completed ABI destinations through the full argument
+ shuffle, followed by a post-call phase barrier. (commit 51681b45)
- [x] `far_slot_large_frame` — was a data-model artifact (`long` is 32-bit on
arm32, so `x`'s 8-byte check failed and the 8-byte far slot wasn't even
exercised). Made the shared case `long long` → tests the 8-byte far slot
@@ -390,9 +395,12 @@ board; the toy corpus went from 355/425 to ~418/425.**
reads at a non-zero element offset, and the switch jump-table crashes.
Globals apply a non-zero byte offset as a separate add (ARM REL MOVW/MOVT
addends are carry-lossy), matching `elf_arm_reloc_field_addend`'s contract.
-- **-O1 native-emit scratch** — `NativeAllocClassInfo.opt_scratch` gives the
- optimizer two scratch (IP+LR) without the NDT taking LR; SREM/UREM no longer
- need a hard LR temp. Fixes the "no scratch register" -O1 panics.
+- **-O1 operand-temp ownership** — `NativeAllocClassInfo.emit_temps` declares
+ r0..r3 as the optimizer emitter's instruction-scoped bank, distinct from the
+ O0 `scratch` policy (IP) and the asm-only bank (LR). IP/LR stay private to
+ backend hook internals, so operand materialization cannot overlap an internal
+ address or arithmetic temporary. This supersedes the earlier shared IP/LR
+ emission policy and fixes the "no register available" pressure failures.
- **Tail/sibling calls** — full AAPCS lowering (direct/indirect/void/sret/
variadic/stack-args), frame-size-independent epilogue, in-window stack args.
- **Frame correctness** — sp kept 8-byte aligned at calls (odd-word saved block
@@ -685,9 +693,9 @@ gated behind a target feature and used only to lower saturating builtins
### Registers (AAPCS)
| reg | role |
|---|---|
-| r0–r3 | argument / result / scratch — **caller-saved**; r0(:r1) returns scalars |
+| r0–r3 | argument / result — **caller-saved**; r0(:r1) returns scalars; O1 instruction-scoped operand-temp bank |
| r4–r11 | **callee-saved** variable registers (r9 is general here — bare-metal, no platform/static-base reservation) |
-| r12 (IP) | intra-procedure scratch — **caller-saved**; kit's codegen scratch for address/immediate materialization (the analog of aa64 x16/x17, rv `t`-regs) |
+| r12 (IP) | intra-procedure temporary — **caller-saved**; O0 scratch policy and backend-private address/immediate staging |
| r13 (SP) | stack pointer — 8-byte aligned at public interfaces (AAPCS) |
| r14 (LR) | link register |
| r15 (PC) | program counter (reads as +4, Thumb) |
@@ -790,7 +798,7 @@ target):
| `disasm.c` | descriptor-driven decode (read first half-word; if `[15:11] ∈ {0b11101,0b11110,0b11111}` it is a 32-bit instruction → read second half-word); IT-state tracking; `.inst`/`.hword` fallback | `riscv/disasm.c` |
| `asm.c` | operand parsing, `ThumbExpandImm`/MOVW-MOVT/branch immediate encoding, IT-block syntax, `.syntax unified` | `aa64/asm.c` |
| `regs.h` / `regs.c` | DWARF index ↔ name table (r0–r15, then s0–s31/d0–d15 for VFP) | `aa64/regs.c` |
-| `native.c` | the `NativeTarget` (`src/arch/native_target.h:459`) + `NativeOps`: register pools (`arm_int_allocable[]` = r0–r3 then r4–r11; scratch = IP/r12, with r0–r3 fronted as arg regs per the rv32 Lever-1 forward-order rule), frame layout (PUSH/POP, prologue/epilogue), all ops, IT-based `cmp`→select, MOVW/MOVT materialization, SDIV/UDIV+MLS, BFI/UBFX bitfields, LDRD/STRD + lane ops for `i64` | `riscv/native.c`, `aa64/native.c` |
+| `native.c` | the `NativeTarget` (defined in `src/arch/native_target.h`) + `NativeOps`: O0 cache pool r0–r3 and `scratch` IP; O1 persistent allocation r4–r11, scoped `emit_temps` r0–r3, and asm-only LR; frame layout (PUSH/POP, prologue/epilogue); all ops; IT-based `cmp`→select; MOVW/MOVT materialization; SDIV/UDIV+MLS; BFI/UBFX bitfields; LDRD/STRD + lane ops for `i64` | `riscv/native.c`, `aa64/native.c` |
| `reloc.c` | `arm32_reloc_desc` table + `arm32_reloc_apply_insn` (the Thumb-2 split-immediate byte patcher) + `link_arch_arm32` | `riscv/reloc.c`, `aa64/reloc.c` |
| `link.c` | `LinkArchDesc` (`src/link/link_arch.h:67`) — static-only, no PLT for v1 | `riscv/link.c` |
| `dbg.c` | breakpoint (BKPT) + displaced-step shim; `min_insn_len=2, max_insn_len=4` | `riscv/dbg.c` |
diff --git a/doc/plan/OPTIMIZER.md b/doc/plan/OPTIMIZER.md
@@ -22,8 +22,7 @@ A few facts about the current code path frame everything below:
- **O1 is the only live optimized path.** `opt_cgtarget_new` normalizes every
`opt_level >= 1` request to `1` (`src/opt/opt.c`), so no compilation selects
O2 today. The O1 pipeline runs entirely in the PReg namespace with the
- non-splitting allocator (`opt_regalloc_locations(..., allow_live_range_split
- = 0)`).
+ non-splitting allocator (`opt_regalloc_locations`).
- **The O2 SSA mid-end (`opt_cleanup`) is fully implemented but unreached.** The
schedule — register SSA, mem2reg, GVN, copy-prop, DSE, LICM, pressure relief,
conventional SSA, undo-SSA — is defined and maintained against targeted opt
@@ -34,12 +33,21 @@ A few facts about the current code path frame everything below:
callees under an 8-op cost cap, always inlines `always_inline`, and refuses
`noinline`/recursive/control-rich callees. The whole-program inliner
(`opt_inline`) also exists but is not wired into a live path.
-- **The generic machine register-constraint mechanism is done (baseline).**
- The tie/forbid/clobber primitives, the per-instruction clobber side table
- (`Func.inst_clobbers`), `machinize_inst_clobbers`, and
- `apply_machine_reg_clobbers` are in tree, fed by the x64 machine-clobber hook
- for div/mod/shift/mul/CAS/RMW/`va_arg`; aarch64/riscv64 leave the hook inert.
- Remaining work is follow-on cleanup, not the core model.
+- **The generic register-constraint/effect mechanism is done (baseline).**
+ `OptPRegInfo.forbidden_hard_regs` is allocation's one hard constraint; a
+ preferred register is only a hint and can never clear it. Machinization
+ resolves inline-asm register requirements at the instruction and builds the
+ initial target-specific implicit-kill table in `Func.inst_clobbers`; after
+ loop-immediate lowering and constant hoisting, the final HIR-shape boundary
+ refreshes that table before liveness and allocation. After allocation,
+ `OptRegEffects` is the one authority for explicit uses/defs and
+ call/asm/machine clobbers, consumed by hard-register liveness, combine, DCE,
+ and native emission. The x64 hook covers fixed effects including division,
+ shifts, atomics, bitfield stores, intrinsics, syscall, and descriptor TLS;
+ aarch64 covers syscall and descriptor TLS; rv64 covers syscall. A hook is
+ inert only for targets and operations with no fixed clobbers. The hook is
+ exhaustive: ordinary backend operations preserve every allocated or cached
+ register not named as an explicit def or machine clobber.
## 1. Complete and turn on the O2 SSA mid-end
@@ -135,7 +143,7 @@ beyond coalescing.
singleton-only, non-call-crossing shape sketched earlier.
- Keep coalescing gated to O2 (matching MIR), preserving the unit-overlap
conflict counting that prevents merging a multiply-defined local with a
- tied-register param.
+ conflicting parameter live range.
## 3. O1 generated-code quality
@@ -242,28 +250,49 @@ they do not change which source variables get stable stack homes.
that clang does not, inflating object text and the linked image and obscuring
measurements. Emit static-inline/header helpers only when referenced.
-## 5. Machine register-constraint model: remaining work
-
-**Goal.** Finish the follow-on cleanup now that the generic fixed-register/
-clobber mechanism is in place, and keep the model the single entry point for
-target ISA register rules into the allocator.
-
-**Rationale.** The core mechanism (tie + forbid + per-instruction clobber side
-table, with the x64 hook covering div/mod/shift/mul/CAS/RMW/`va_arg`) is shipped
-and is what makes the x64 `-O1` div/mod/atomic/overflow/varargs cases correct.
-What remains is removing the now-redundant defensive code the mechanism made
-unnecessary, and keeping the contract clean as new constrained ops appear.
-
-- **Remove defensive backend moves the allocator now guarantees.** With the
- dividend tied to `rax` and nothing live in `rdx`/`rcx`, the `mov rax,
- dividend` in `x64_binop`, the address-staging in `x64_atomic_cas`/`_rmw`, and
- the shift `mov rcx, count` are register-to-itself no-ops. Simplify them away.
-- **Confirm `InstId` density and stability** through `pass_lower` so the
- side-table lookup stays O(1); fall back to a small open-addressed map keyed by
- instruction only if ids ever become sparse.
-- **Add focused regressions** for the live-across-the-constraint shape (e.g. a
- function that divides while keeping an unrelated value live across the divide)
- so the live-across-forbid loop stays covered as new ops are added.
+## 5. Register constraints and effects: maintained contract
+
+**Goal.** Keep target ISA register rules at one allocation boundary and one
+post-allocation effects boundary, with instruction-local staging owned by the
+emitter/backend that performs the operation.
+
+**Rationale.** The allocation mechanism has one hard forbid mask, while
+machinization carries instruction-local asm requirements and the dense machine
+clobber side table. `OptRegEffects` remains the physical authority for distinct
+uses, defs, and clobbers. The target effect hook is also an exhaustive
+preservation contract for clean frame-cache values. This division avoids both
+persistent operand pinning and a second, backend-local liveness model.
+
+- **Keep fixed-encoding staging local.** Do not remove moves into registers
+ such as x64 `rax` or `rcx` merely because live-across values are forbidden
+ there. A dying operand may still begin in any legal home; the backend must
+ stage the current operation's operands and results as its encoding requires.
+- **Keep `Func.inst_clobbers` dense and derived from final HIR.** Machinization
+ builds the initial table. After `lower_loop_imm_operands` and
+ `hoist_loop_consts` (the last HIR-shape mutator),
+ `opt_refresh_machine_clobbers` replaces it before liveness and allocation.
+ Preserve `InstId` density and stability through lowering so lookup stays
+ O(1), and refresh or verify the table whenever instruction identity or effect
+ shape changes.
+- **Keep focused regressions** for live-across fixed effects (division,
+ variable shifts, bitfield stores, atomics, TLS descriptor calls, syscalls)
+ and for inline-asm fixed/subset constraints under spills and pressure.
+- **Resolve inline-asm constraints once.** New constraint forms become
+ `IRAsmRegRequirement`s during machinization and are satisfied by scoped
+ emitter staging at the asm instruction. They must not introduce a
+ whole-live-range hard-register pin or a second allocator in the backend.
+- **Keep frame-cache eligibility derived.** A persistent clean entry may use a
+ target-declared `emit_cache_mask` register or a liveness-proven dead O1
+ allocation register whose ABI preservation is already established. Effects
+ invalidate every exceptional fixed clobber, and block/call/asm boundaries
+ clear the cache;
+ do not add another cache ownership mask.
+- **Keep target hooks declarative.** A new implicit-register operation enters
+ through `machine_op_clobbers` so allocation sees the forbid and
+ `opt_inst_reg_effects` sees the same mask after MIR lowering. The report must
+ include every fixed clobber of a register that can hold an allocated or
+ cached value; backend-local liveness or unreported clobber exceptions are not
+ a supported extension point.
## 6. Inlining beyond tiny streaming callees
diff --git a/doc/plan/TODO.md b/doc/plan/TODO.md
@@ -22,14 +22,6 @@ Add new deferred fixes below as they are discovered.
## Deferred dedups & abstraction cleanups ("use the shared seam")
-- **A.2 — optimizer-path inline-asm dedup.** The three `aa_/x64_/rv_asm_block_native`
- share an out/in/bind/run/store skeleton but diverge by *policy* (staging coverage +
- structurally different scratch-register models: aa64/x64 reuse 2 mem-scratch regs via
- a shared `ntmp`; rv64 uses 4 dedicated stage regs). A hooks dedup would relocate, not
- unify, behavior and add miscompile risk in a hot path. Now that the aa64/x64
- staging gaps are closed, a shared `native_asm_bind_optimizer_operands` can be
- considered. (The -O0 direct path is already shared via
- `native_asm_bind_direct_operands`.)
- **E.3 — OS-neutral env code duplicated** between `driver/env/posix.c` and `windows.c`
(stdio writer + thunks, `env→{context,jit_host,dbg_host}` trio, dir-handle structs +
read/close, `read_stdin` grow/shrink). Move to `driver/env/common.c`. **Blocked on a
diff --git a/driver/env/build_host_posix.c b/driver/env/build_host_posix.c
@@ -239,7 +239,10 @@ static int db_transport_try_accept(void* user, KitBuildListener* lst,
if (!bh || !lst || !out || !ready) return 1;
*out = NULL;
*ready = 0;
- FD_ZERO(&fds);
+ /* Darwin's FD_ZERO expands to the non-C11 __builtin_bzero. Initialize the
+ * public fd_set object directly so the hosted adapter remains bootstrapable
+ * by a strict C11 compiler. */
+ memset(&fds, 0, sizeof fds);
FD_SET(lst->fd, &fds);
tv.tv_sec = 0;
tv.tv_usec = 10000;
diff --git a/lang/c/parse/cg.c b/lang/c/parse/cg.c
@@ -21,14 +21,27 @@ static uint16_t c_cg_lvalue_flags_for_type(const Type* ty) {
return flags;
}
-/* Resolve the CG type id for a value-stack slot: the slot's recorded cg type
- * when present, else map the frontend lang_type. kit_cg_slot_cg_type reads the
- * slot's type directly (no struct copy); the lang_type fallback is taken only
- * when the slot carries no cg type. */
+/* Resolve the operational CG type for a value-stack slot. A complete recorded
+ * type is authoritative. A missing type, or an incomplete nominal record left
+ * by dereferencing a pointer, crosses back through the frontend lang_type so a
+ * by-value operation sees the completed layout. kit_cg_slot_cg_type reads the
+ * slot's id directly (no struct copy). */
static KitCgTypeId c_cg_slot_cg_id(Parser* p, u32 depth) {
+ const Type* lang_ty =
+ (const Type*)kit_cg_slot_lang_type(p->cg, depth);
KitCgTypeId cg = kit_cg_slot_cg_type(p->cg, depth);
- if (cg) return cg;
- return c_cg_tid(p, (const Type*)kit_cg_slot_lang_type(p->cg, depth));
+ KitCgTypeId resolved;
+ if (cg && kit_cg_type_is_complete(p->c, cg)) return cg;
+
+ /* A pointer to a record needs only its nominal CG declaration. Dereferencing
+ * that pointer therefore leaves an incomplete record id on the value stack,
+ * even when the C record itself is complete. A later by-value operation
+ * (load/store/aggregate copy) must cross back through the language type so
+ * its field layout is installed before the memory operation captures a byte
+ * count. Returning the recorded id unconditionally made the first
+ * `*dst = *src` for such a type a zero-byte copy. */
+ resolved = lang_ty ? c_cg_tid(p, lang_ty) : KIT_CG_TYPE_NONE;
+ return resolved ? resolved : cg;
}
static KitCgTypeId c_cg_top_cg_id(Parser* p) { return c_cg_slot_cg_id(p, 0); }
diff --git a/mk/bootstrap.mk b/mk/bootstrap.mk
@@ -41,7 +41,8 @@ endif
BOOTSTRAP_STAMP = $(BOOTSTRAP_DIR)/.stamp
BOOTSTRAP_RT_LIBS = $(addprefix $(RT_BUILD_DIR)/,$(addsuffix /libkit_rt.a,$(RT_DEFAULT_VARIANTS)))
BOOTSTRAP_MAKEFILES = Makefile mk/env.mk mk/config.mk mk/flags.mk \
- mk/lib_srcs.mk mk/driver_srcs.mk mk/rt.mk
+ mk/lib_srcs.mk mk/driver_srcs.mk mk/rt.mk \
+ mk/bootstrap.mk
# BOOTSTRAP_SEED: a pre-built stage1 kit. Empty (default) = the normal native
# flow where stage1 is the host $(BIN). When set (a path), stage1 is copied from
@@ -85,6 +86,9 @@ $(BOOTSTRAP_STAMP): $(BOOTSTRAP_DEPS)
rm -rf $(BOOTSTRAP_DIR)
@mkdir -p $(BOOTSTRAP_STAGE1_DIR)
cp $(BOOTSTRAP_STAGE1_SRC) $(BOOTSTRAP_STAGE1_BIN)
+ # The copied stage1 binary no longer sits beside $(BUILD_DIR)/support. Give
+ # it the same self-contained support/rt layout as every normal bin target.
+ $(MAKE) $(BOOTSTRAP_STAGE1_DIR)/support/rt
@for tool in $(BOOTSTRAP_TOOLS); do ln -sf kit "$(BOOTSTRAP_STAGE1_DIR)/$$tool"; done
$(MAKE) lib bin \
BUILD_DIR='$(abspath $(BOOTSTRAP_STAGE2_DIR))' \
diff --git a/mk/test.mk b/mk/test.mk
@@ -1123,9 +1123,21 @@ test-link-macho-symbols: bin $(LINK_EXE_RUNNER)
OPT_TEST_BIN = build/test/cg_ir_lower_test
TINY_INLINE_TEST_BIN = build/test/tiny_inline_test
-
-test-opt: bin $(OPT_TEST_BIN) test-opt-tiny-inline test-opt-inline test-opt-zero-arg test-opt-static-prune-aa64 test-opt-aa64-tail test-opt-x64-win-tail-sret test-opt-prologue-tier test-opt-whole-program-inline test-opt-lto-phase1 test-opt-redundant-copy-ext test-opt-redundant-frame-sub test-opt-o1-branch-cleanup test-opt-hot-slot-order test-opt-o1-remat test-opt-aa64-x29-bottom test-opt-o1-coalesce test-opt-o1-switch-imm test-opt-o1-inline-cap test-opt-rv64-far-slot test-opt-o1-cmp-imm test-opt-o1-stack-dse test-opt-o1-local-cse test-opt-o1p-aa64 test-opt-o1p-const-divmul test-opt-o1p-combine test-opt-o1p-rider
+REG_EFFECTS_TEST_BIN = build/test/reg_effects_test
+LOCATION_MIR_TEST_BIN = build/test/location_mir_test
+COMBINE_CSE_TEST_BIN = build/test/combine_cse_test
+NATIVE_EMIT_FRAME_DST_TEST_BIN = build/test/native_emit_frame_dst_test
+FRAME_VALUE_BACKEND_TEST_BIN = build/test/frame_value_backend_test
+NATIVE_PART_CHUNK_TEST_BIN = build/test/native_part_chunk_test
+
+test-opt: bin $(OPT_TEST_BIN) $(REG_EFFECTS_TEST_BIN) $(LOCATION_MIR_TEST_BIN) $(COMBINE_CSE_TEST_BIN) $(NATIVE_EMIT_FRAME_DST_TEST_BIN) $(FRAME_VALUE_BACKEND_TEST_BIN) $(NATIVE_PART_CHUNK_TEST_BIN) test-opt-tiny-inline test-opt-inline test-opt-zero-arg test-opt-static-prune-aa64 test-opt-aa64-tail test-opt-aa64-apple-varargs test-opt-x64-win-tail-sret test-opt-prologue-tier test-opt-whole-program-inline test-opt-lto-phase1 test-opt-redundant-copy-ext test-opt-redundant-frame-sub test-opt-o1-branch-cleanup test-opt-macho-block-alt-entry test-opt-hot-slot-order test-opt-o1-remat test-opt-aa64-x29-bottom test-opt-o1-coalesce test-opt-o1-switch-imm test-opt-o1-inline-cap test-opt-rv64-far-slot test-opt-o1-cmp-imm test-opt-o1-stack-dse test-opt-arm32-emit-temp-pressure test-opt-o1-local-cse test-opt-o1p-aa64 test-opt-o1p-const-divmul test-opt-o1p-combine test-opt-o1p-rider
$(OPT_TEST_BIN)
+ $(REG_EFFECTS_TEST_BIN)
+ $(LOCATION_MIR_TEST_BIN)
+ $(COMBINE_CSE_TEST_BIN)
+ $(NATIVE_EMIT_FRAME_DST_TEST_BIN)
+ $(FRAME_VALUE_BACKEND_TEST_BIN)
+ $(NATIVE_PART_CHUNK_TEST_BIN)
# Structural disasm check: the -O1 redundant copy / extension folds
@@ -1146,6 +1158,10 @@ test-opt-redundant-frame-sub: bin
test-opt-o1-branch-cleanup: bin
@KIT=$(abspath $(BIN)) bash test/opt/o1_branch_cleanup.sh
+.PHONY: test-opt-macho-block-alt-entry
+test-opt-macho-block-alt-entry: bin $(ROUNDTRIP_BIN_MACHO)
+ @KIT=$(abspath $(BIN)) bash test/opt/macho_block_alt_entry.sh
+
# Structural disasm check: hot-slot-low frame ordering keeps the hottest spill
# slots in cheap displacement range (x64 disp8, rv64 imm12 single-instruction).
.PHONY: test-opt-hot-slot-order
@@ -1202,6 +1218,13 @@ test-opt-o1-cmp-imm: bin
test-opt-o1-stack-dse: bin
@KIT=$(abspath $(BIN)) bash test/opt/o1_stack_dse.sh
+# Compile-only ARMv7-M pressure guard: location MIR leaves spill values in
+# frame homes, and the native emitter must stage operand-facing values from its
+# explicit r0-r3 bank without borrowing backend-private IP/LR.
+.PHONY: test-opt-arm32-emit-temp-pressure
+test-opt-arm32-emit-temp-pressure: bin
+ @KIT=$(abspath $(BIN)) bash test/opt/arm32_emit_temp_pressure.sh
+
# Behavioral+structural check: same-block redundant-load/expression CSE reuses
# the last load/compute, but never across a may-alias store, call, or volatile.
.PHONY: test-opt-o1-local-cse
@@ -1256,6 +1279,10 @@ test-opt-static-prune-aa64: bin
test-opt-aa64-tail: bin
@KIT=$(abspath $(BIN)) bash test/opt/aa64_tail_call.sh
+.PHONY: test-opt-aa64-apple-varargs
+test-opt-aa64-apple-varargs: bin
+ @KIT=$(abspath $(BIN)) bash test/opt/aa64_apple_varargs.sh
+
.PHONY: test-opt-x64-win-tail-sret
test-opt-x64-win-tail-sret: bin
@KIT=$(abspath $(BIN)) bash test/opt/x64_win_tail_sret.sh
diff --git a/mk/test_unit.mk b/mk/test_unit.mk
@@ -72,7 +72,9 @@ UNIT_TESTS_INTERNAL := \
interp_smoke_test jit_tls_relax_test coff_weak_alias_test \
coff_archive_fixpoint_test elf_version_import_test \
rv64_interp_smoke_test abi_classify_test ir_recorder_test \
- native_direct_target_test x64_dbg_test cg_ir_lower_test tiny_inline_test
+ native_direct_target_test x64_dbg_test cg_ir_lower_test tiny_inline_test \
+ reg_effects_test location_mir_test combine_cse_test \
+ native_emit_frame_dst_test frame_value_backend_test native_part_chunk_test
dwarf_test_SRC := test/dwarf/dwarf_test.c
debug_roundtrip_unit_SRC := test/debug/roundtrip_unit.c
debug_cfi_unit_SRC := test/debug/cfi_unit.c
@@ -97,6 +99,12 @@ native_direct_target_test_SRC := test/cg/native_direct_target_test.c
x64_dbg_test_SRC := test/arch/x64_dbg_test.c
cg_ir_lower_test_SRC := test/opt/cg_ir_lower_test.c
tiny_inline_test_SRC := test/opt/tiny_inline_test.c
+reg_effects_test_SRC := test/opt/reg_effects_test.c
+location_mir_test_SRC := test/opt/location_mir_test.c
+combine_cse_test_SRC := test/opt/combine_cse_test.c
+native_emit_frame_dst_test_SRC := test/opt/native_emit_frame_dst_test.c
+frame_value_backend_test_SRC := test/opt/frame_value_backend_test.c
+native_part_chunk_test_SRC := test/opt/native_part_chunk_test.c
build_pure_test_SRC := test/build/build_pure_test.c
# ---- build rules ------------------------------------------------------------
diff --git a/scripts/cross_test.sh b/scripts/cross_test.sh
@@ -70,14 +70,14 @@ tok_libc() { case "$1" in linux-*) local r="${1#linux-}"; printf '%s' "${r%%-*}"
# Per-arch -march/-mabi a freestanding corpus object must be compiled with so
# its ELF e_flags match the bare-metal stub + runtime it is linked against
-# (test/lib/exec_bare.sh). rv32 is the one with a float-ABI axis: the bare
-# harness (and the toy cross lane) are hard-single (ilp32f), but kit's rv32
-# *default* is now soft-float (ilp32, commit b436d6ef) — without pinning, the
-# corpus object's e_flags (0x1) mismatch the stub's (0x3) and the freestanding
-# link rejects it. Keep in sync with _bare_emit_rv32 / test/toy/run.sh.
+# (test/lib/exec_bare.sh). Both RISC-V widths have a float-ABI axis, so make the
+# contract explicit rather than inheriting a compiler default. rv32's harness
+# is hard-single (ilp32f); rv64 enables D and uses hard-double (lp64d). Keep in
+# sync with _bare_emit_rv32/_bare_emit_rv64 and test/toy/run.sh.
freestanding_abi_flags() { # token -> echoes extra cc flags ("" for most arches)
case "$(tok_arch "$1")" in
rv32) printf '%s' "-march=rv32imafc_zicsr_zifencei -mabi=ilp32f" ;;
+ rv64) printf '%s' "-march=rv64imafd_zicsr_zifencei -mabi=lp64d" ;;
*) printf '' ;;
esac
}
diff --git a/src/arch/aa64/native.c b/src/arch/aa64/native.c
@@ -57,6 +57,11 @@
enum {
AA_X8 = 8u, /* indirect-result (sret) register; usable as a copy base that
aa_copy_bytes (which scratches only x16/x17) never clobbers */
+ /* Tail epilogues restore x30 and may use x16/x17 while rebuilding the frame,
+ * so an indirect tail target is moved here only after argument marshalling.
+ * x9 is an instruction-scoped emitter temp and no later operand phase runs
+ * between plan_call and the tail branch. */
+ AA_TAIL_TARGET = 9u,
AA_TMP0 = 16u,
AA_TMP1 = 17u,
/* Callee-saved frame-base anchor. The single-pass far-slot fast path
@@ -873,8 +878,50 @@ static AANativeSlot* aa_slot(AANativeTarget* a, NativeFrameSlot slot) {
return native_frame_slot_at(&a->frame, slot);
}
-static void aa_addr_base(AANativeTarget* a, NativeAddr addr, u32* base_out,
- i32* off_out) {
+/* x16/x17 are the backend-private integer temporaries available to memory
+ * primitives. Address formation must not silently overwrite a store payload
+ * or a register index that the final memory instruction still consumes. Keep
+ * those conflicts explicit at the address-materialization boundary. */
+typedef u8 AATmpAvoid;
+enum {
+ AA_TMP_AVOID_0 = 1u << 0,
+ AA_TMP_AVOID_1 = 1u << 1,
+};
+
+static AATmpAvoid aa_tmp_avoid_reg(Reg reg) {
+ if (reg == AA_TMP0) return AA_TMP_AVOID_0;
+ if (reg == AA_TMP1) return AA_TMP_AVOID_1;
+ return 0u;
+}
+
+static Reg aa_addr_tmp(AANativeTarget* a, AATmpAvoid avoid) {
+ if (!(avoid & AA_TMP_AVOID_0)) return AA_TMP0;
+ if (!(avoid & AA_TMP_AVOID_1)) return AA_TMP1;
+ aa_panic(a, "address formation has no scratch register");
+ return AA_TMP0;
+}
+
+static Reg aa_addr_adjust_tmp(AANativeTarget* a, Reg base,
+ AATmpAvoid avoid) {
+ AATmpAvoid base_bit = aa_tmp_avoid_reg(base);
+ if (base_bit && !(avoid & base_bit)) return base;
+ return aa_addr_tmp(a, avoid);
+}
+
+static AATmpAvoid aa_mem_addr_avoid(int load, NativeLoc reg,
+ NativeAddr addr) {
+ AATmpAvoid avoid = 0u;
+ /* v16/v17 and x16/x17 are separate register files. Only an integer store
+ * keeps its Rt live across integer address materialization. */
+ if (!load && !native_loc_is_fp(reg))
+ avoid |= aa_tmp_avoid_reg(loc_reg(reg));
+ if (addr.index_kind == NATIVE_ADDR_INDEX_REG)
+ avoid |= aa_tmp_avoid_reg(addr.index.reg);
+ return avoid;
+}
+
+static void aa_addr_base(AANativeTarget* a, NativeAddr addr,
+ AATmpAvoid avoid, u32* base_out, i32* off_out) {
*base_out = AA_TMP0;
*off_out = addr.offset;
switch ((NativeAddrBaseKind)addr.base_kind) {
@@ -887,15 +934,39 @@ static void aa_addr_base(AANativeTarget* a, NativeAddr addr, u32* base_out,
*off_out = aa_fp_off_slot(a, s->off) + addr.offset;
return;
}
+ case NATIVE_ADDR_BASE_FRAME_VALUE: {
+ NativeLoc tmp;
+ NativeAddr base_addr = addr;
+ Reg addr_tmp = aa_addr_tmp(a, avoid);
+ if (!addr.base_type)
+ aa_panic(a, "frame-value base has no exact type");
+ base_addr.index_kind = NATIVE_ADDR_INDEX_NONE;
+ base_addr.index_ext = NATIVE_ADDR_IDX_EXT_NONE;
+ base_addr.log2_scale = 0;
+ memset(&tmp, 0, sizeof tmp);
+ tmp.kind = NATIVE_LOC_REG;
+ tmp.cls = NATIVE_REG_INT;
+ tmp.type = addr.base_type;
+ tmp.v.reg = addr_tmp;
+ a->base.load_addr(&a->base, tmp, base_addr);
+ *base_out = addr_tmp;
+ *off_out = 0;
+ return;
+ }
case NATIVE_ADDR_BASE_GLOBAL: {
NativeLoc tmp;
+ NativeAddr base_addr = addr;
+ Reg addr_tmp = aa_addr_tmp(a, avoid);
+ base_addr.index_kind = NATIVE_ADDR_INDEX_NONE;
+ base_addr.index_ext = NATIVE_ADDR_IDX_EXT_NONE;
+ base_addr.log2_scale = 0;
memset(&tmp, 0, sizeof tmp);
tmp.kind = NATIVE_LOC_REG;
tmp.cls = NATIVE_REG_INT;
tmp.type = builtin_id(KIT_CG_BUILTIN_I64);
- tmp.v.reg = AA_TMP0;
- a->base.load_addr(&a->base, tmp, addr);
- *base_out = AA_TMP0;
+ tmp.v.reg = addr_tmp;
+ a->base.load_addr(&a->base, tmp, base_addr);
+ *base_out = addr_tmp;
*off_out = 0;
return;
}
@@ -915,7 +986,7 @@ static void aa_emit_mem_q(AANativeTarget* a, int load, NativeLoc reg,
MCEmitter* mc = a->base.mc;
if (addr.index_kind != NATIVE_ADDR_INDEX_NONE)
aa_panic(a, "unsupported q-register indexed memory access");
- aa_addr_base(a, addr, &base, &off);
+ aa_addr_base(a, addr, 0u, &base, &off);
rt = loc_reg(reg);
if (off >= 0 && (((u32)off & 15u) == 0) && ((u32)off >> 4) <= 0xfffu) {
aa_emit32(mc, aa_ldst_q_uimm(load, rt, base, (u32)off));
@@ -929,19 +1000,25 @@ static void aa_emit_mem_q(AANativeTarget* a, int load, NativeLoc reg,
aa_emit32(mc, aa_ldst_q_uimm(load, rt, AA_TMP1, 0));
}
-static void aa_emit_mem(AANativeTarget* a, int load, NativeLoc reg,
- NativeAddr addr, MemAccess mem) {
- u32 base, rt, sz, ld_opc;
+static void aa_emit_mem_native(AANativeTarget* a, int load, NativeLoc reg,
+ NativeAddr addr, MemAccess mem) {
+ u32 base, rt, sz, bytes, ld_opc;
+ AATmpAvoid addr_avoid;
i32 off;
MCEmitter* mc = a->base.mc;
rt = loc_reg(reg);
- sz = size_idx(mem.size
- ? mem.size
- : type_size32(&a->base, reg.type ? reg.type : mem.type));
- if (native_loc_is_fp(reg) &&
- (mem.size
- ? mem.size
- : type_size32(&a->base, reg.type ? reg.type : mem.type)) == 16u) {
+ addr_avoid = aa_mem_addr_avoid(load, reg, addr);
+ bytes = mem.size
+ ? mem.size
+ : type_size32(&a->base, reg.type ? reg.type : mem.type);
+ if (native_loc_is_fp(reg)) {
+ if (bytes != 4u && bytes != 8u && bytes != 16u)
+ aa_panic(a, "unsupported native fp memory width");
+ } else if (bytes != 1u && bytes != 2u && bytes != 4u && bytes != 8u) {
+ aa_panic(a, "unsupported native integer memory width");
+ }
+ sz = size_idx(bytes);
+ if (native_loc_is_fp(reg) && bytes == 16u) {
aa_emit_mem_q(a, load, reg, addr);
return;
}
@@ -1020,7 +1097,7 @@ static void aa_emit_mem(AANativeTarget* a, int load, NativeLoc reg,
if (addr.base_kind == NATIVE_ADDR_BASE_GLOBAL &&
addr.index_kind == NATIVE_ADDR_INDEX_NONE) {
i64 addend = addr.base.global.addend + (i64)addr.offset;
- u32 scratch = (!load && rt == AA_TMP0) ? AA_TMP1 : AA_TMP0;
+ u32 scratch = aa_addr_tmp(a, addr_avoid);
u32 pos = mc_pos(mc);
if (aa_use_got_for_sym(&a->base, addr.base.global.sym)) {
aa_emit32(mc, aa64_adrp(scratch, 0, 0));
@@ -1049,7 +1126,7 @@ static void aa_emit_mem(AANativeTarget* a, int load, NativeLoc reg,
addr.base.global.sym, addend, 0, 0);
return;
}
- aa_addr_base(a, addr, &base, &off);
+ aa_addr_base(a, addr, addr_avoid, &base, &off);
if (addr.index_kind != NATIVE_ADDR_INDEX_NONE) {
u32 use_base = base;
u32 scaled = 0;
@@ -1064,7 +1141,7 @@ static void aa_emit_mem(AANativeTarget* a, int load, NativeLoc reg,
else if (addr.index_ext == NATIVE_ADDR_IDX_EXT_UXTW)
option = AA64_LDST_OPTION_UXTW;
if (off) {
- use_base = AA_TMP1;
+ use_base = aa_addr_adjust_tmp(a, base, addr_avoid);
aa_emit_add_imm(a, use_base, base, off);
}
if (addr.log2_scale == 0) {
@@ -1093,10 +1170,93 @@ static void aa_emit_mem(AANativeTarget* a, int load, NativeLoc reg,
: aa_stur_v(sz, native_loc_is_fp(reg), rt, base, off));
return;
}
- aa_emit_add_imm(a, AA_TMP1, base, off);
- aa_emit32(
- mc, load ? aa_ldur_op_v(sz, native_loc_is_fp(reg), ld_opc, rt, AA_TMP1, 0)
- : aa_stur_v(sz, native_loc_is_fp(reg), rt, AA_TMP1, 0));
+ {
+ u32 scratch = aa_addr_adjust_tmp(a, base, addr_avoid);
+ aa_emit_add_imm(a, scratch, base, off);
+ aa_emit32(mc, load ? aa_ldur_op_v(sz, native_loc_is_fp(reg), ld_opc, rt,
+ scratch, 0)
+ : aa_stur_v(sz, native_loc_is_fp(reg), rt, scratch, 0));
+ }
+}
+
+static u32 aa_part_transfer_tmp(AANativeTarget* a, NativeLoc reg,
+ NativeAddr addr) {
+ u32 used = 1u << loc_reg(reg);
+ if (addr.base_kind == NATIVE_ADDR_BASE_REG) used |= 1u << addr.base.reg;
+ if (addr.index_kind == NATIVE_ADDR_INDEX_REG) used |= 1u << addr.index.reg;
+ if (!(used & (1u << AA_TMP0))) return AA_TMP0;
+ if (!(used & (1u << AA_TMP1))) return AA_TMP1;
+ aa_panic(a, "exact ABI part transfer has no scratch register");
+ return AA_TMP0;
+}
+
+/* A register carrier is 8 bytes, but the final part of a small aggregate may
+ * carry any byte count from 1 through 8. Keep the one-instruction native-width
+ * path; decompose 3/5/6/7-byte memory transfers into exact low-to-high chunks.
+ * This is deliberately below the call planner so incoming binds, outgoing
+ * args, and generic NativeCallPlanRet write phases all share the same rule. */
+static void aa_emit_mem(AANativeTarget* a, int load, NativeLoc reg,
+ NativeAddr addr, MemAccess mem) {
+ u32 bytes = mem.size
+ ? mem.size
+ : type_size32(&a->base, reg.type ? reg.type : mem.type);
+ NativePartChunkIter it;
+ NativeLoc lane;
+ u32 chunk_off, chunk_size, regno, tmp;
+ if (bytes == 1u || bytes == 2u || bytes == 4u || bytes == 8u ||
+ (native_loc_is_fp(reg) && bytes == 16u)) {
+ aa_emit_mem_native(a, load, reg, addr, mem);
+ return;
+ }
+ if (native_loc_is_fp(reg) || bytes == 0u || bytes > 8u)
+ aa_panic(a, "unsupported exact ABI part memory width");
+
+ regno = loc_reg(reg);
+ /* A native-width load may overwrite its address register because it is one
+ * instruction (`ldr x0, [x0]`). An exact-width load is a sequence, so retain
+ * the effective address in a backend-private register when dst aliases its
+ * base or index. The other private register remains available for packing
+ * later chunks into dst. */
+ if (load &&
+ ((addr.base_kind == NATIVE_ADDR_BASE_REG && addr.base.reg == regno) ||
+ (addr.index_kind == NATIVE_ADDR_INDEX_REG &&
+ addr.index.reg == regno))) {
+ NativeLoc addr_loc;
+ NativeAddr retained;
+ u32 addr_reg = regno == AA_TMP0 ? AA_TMP1 : AA_TMP0;
+ if (regno == AA_TMP0 || regno == AA_TMP1)
+ aa_panic(a, "exact load destination aliases private address register");
+ addr_loc = native_loc_reg(builtin_id(KIT_CG_BUILTIN_I64),
+ NATIVE_REG_INT, addr_reg);
+ a->base.load_addr(&a->base, addr_loc, addr);
+ memset(&retained, 0, sizeof retained);
+ retained.base_kind = NATIVE_ADDR_BASE_REG;
+ retained.base.reg = addr_reg;
+ retained.base_type = addr_loc.type;
+ addr = retained;
+ }
+ tmp = aa_part_transfer_tmp(a, reg, addr);
+ it = native_part_chunks(bytes, 8u);
+ while (native_part_chunk_next(&it, &chunk_off, &chunk_size)) {
+ NativeAddr chunk_addr = addr;
+ MemAccess chunk_mem = mem;
+ chunk_addr.offset += (i32)chunk_off;
+ chunk_mem.size = chunk_size;
+ if (chunk_mem.align > chunk_size) chunk_mem.align = chunk_size;
+ lane = reg;
+ if (chunk_off == 0u) {
+ lane.v.reg = regno;
+ } else {
+ lane.v.reg = tmp;
+ if (!load)
+ aa_emit32(a->base.mc,
+ aa_ubfm(1u, tmp, regno, chunk_off * 8u, 63u));
+ }
+ aa_emit_mem_native(a, load, lane, chunk_addr, chunk_mem);
+ if (load && chunk_off != 0u)
+ aa_emit32(a->base.mc,
+ aa_logsr_lsl(1u, 1u, regno, regno, tmp, chunk_off * 8u));
+ }
}
static NativeAllocClass aa_class_for_type(NativeTarget* t, KitCgTypeId type) {
@@ -1264,12 +1424,13 @@ static void aa_materialize_frame_index(AANativeTarget* a, NativeAddr* addr,
MemAccess mem;
u32 reg;
if (addr->index_kind != NATIVE_ADDR_INDEX_FRAME_VALUE) return;
+ if (!addr->index_type)
+ aa_panic(a, "frame-value index has no exact type");
reg = avoid_reg == AA_TMP1 ? AA_TMP0 : AA_TMP1;
memset(&load, 0, sizeof load);
load.base_kind = NATIVE_ADDR_BASE_FRAME;
load.base.frame = addr->index.frame;
- load.base_type =
- addr->index_type ? addr->index_type : builtin_id(KIT_CG_BUILTIN_I64);
+ load.base_type = addr->index_type;
memset(&idx, 0, sizeof idx);
idx.kind = NATIVE_LOC_REG;
idx.cls = NATIVE_REG_INT;
@@ -1277,8 +1438,8 @@ static void aa_materialize_frame_index(AANativeTarget* a, NativeAddr* addr,
idx.v.reg = reg;
memset(&mem, 0, sizeof mem);
mem.type = load.base_type;
- mem.size = 8;
- mem.align = 8;
+ mem.size = type_size32(&a->base, load.base_type);
+ mem.align = type_align32(&a->base, load.base_type);
aa_emit_mem(a, 1, idx, load, mem);
addr->index_kind = NATIVE_ADDR_INDEX_REG;
addr->index.reg = reg;
@@ -2504,8 +2665,8 @@ static void aa_load_addr(NativeTarget* t, NativeLoc dst, NativeAddr addr) {
addr.base_type ? addr.base_type : builtin_id(KIT_CG_BUILTIN_I64);
memset(&mem, 0, sizeof mem);
mem.type = load.base_type;
- mem.size = 8;
- mem.align = 8;
+ mem.size = type_size32(t, load.base_type);
+ mem.align = type_align32(t, load.base_type);
aa_emit_mem(a, 1, native_loc_reg(dst.type, NATIVE_REG_INT, base_reg),
load, mem);
aa_load_addr_from_base(a, rd, base_reg, addr.offset, &addr);
@@ -3249,35 +3410,17 @@ static MemAccess aa_mem_for_type(NativeTarget* t, KitCgTypeId type, u32 size) {
/* native_loc_reg / native_loc_stack are shared in native_target.h. */
-static NativeAddr aa_loc_addr(AANativeTarget* a, NativeLoc loc, u32 offset) {
+static NativeAddr aa_storage_addr(AANativeTarget* a, NativeLoc loc,
+ u32 offset) {
NativeAddr addr;
- memset(&addr, 0, sizeof addr);
- switch ((NativeLocKind)loc.kind) {
- case NATIVE_LOC_FRAME:
- addr.base_kind = NATIVE_ADDR_BASE_FRAME;
- addr.base.frame = loc.v.frame;
- addr.base_type = loc.type;
- addr.offset = (i32)offset;
- return addr;
- case NATIVE_LOC_STACK:
- addr.base_kind = NATIVE_ADDR_BASE_FRAME;
- addr.base.frame = loc.v.stack.slot;
- addr.base_type = loc.type;
- addr.offset = loc.v.stack.offset + (i32)offset;
- return addr;
- case NATIVE_LOC_ADDR:
- addr = loc.v.addr;
- addr.offset += (i32)offset;
- return addr;
- default:
- aa_panic(a, "location is not addressable");
- }
+ if (!native_loc_storage_addr(loc, (i32)offset, &addr))
+ aa_panic(a, "location is not storage");
return addr;
}
static void aa_addr_of_loc(NativeTarget* t, NativeLoc dst, NativeLoc src) {
AANativeTarget* a = aa_of(t);
- NativeAddr addr = aa_loc_addr(a, src, 0);
+ NativeAddr addr = aa_storage_addr(a, src, 0);
aa_load_addr(t, dst, addr);
}
@@ -3289,15 +3432,25 @@ static void aa_load_part(NativeTarget* t, NativeLoc dst, NativeLoc src,
aa_move(t, dst, src);
return;
}
- if (src.kind == NATIVE_LOC_FRAME || src.kind == NATIVE_LOC_STACK ||
- src.kind == NATIVE_LOC_ADDR) {
- NativeAddr addr = aa_loc_addr(a, src, offset);
+ if (native_loc_addr_role(src) == NATIVE_LOC_ADDR_ROLE_VALUE) {
+ NativeAddr addr;
+ if (offset != 0u || size != t->c->target.ptr_size ||
+ !native_loc_address_value(src, &addr))
+ aa_panic(a, "split or invalid address-value argument");
+ aa_load_addr(t, dst, addr);
+ return;
+ }
+ if (native_loc_addr_role(src) == NATIVE_LOC_ADDR_ROLE_STORAGE) {
+ NativeAddr addr = aa_storage_addr(a, src, offset);
addr.base_type = dst.type;
aa_emit_mem(a, 1, dst, addr, mem);
return;
}
if (src.kind == NATIVE_LOC_IMM) {
- aa_emit_load_imm(t->mc, loc_is_64(t, dst), loc_reg(dst), src.v.imm);
+ i64 part;
+ if (!native_loc_imm_part(src, offset, size, &part))
+ aa_panic(a, "invalid immediate argument part");
+ aa_emit_load_imm(t->mc, loc_is_64(t, dst), loc_reg(dst), part);
return;
}
aa_panic(a, "unsupported call argument source");
@@ -3307,9 +3460,8 @@ static void aa_store_part(NativeTarget* t, NativeLoc dst, NativeLoc src,
u32 offset, u32 size) {
AANativeTarget* a = aa_of(t);
MemAccess mem = aa_mem_for_type(t, src.type, size);
- if (dst.kind == NATIVE_LOC_FRAME || dst.kind == NATIVE_LOC_STACK ||
- dst.kind == NATIVE_LOC_ADDR) {
- NativeAddr addr = aa_loc_addr(a, dst, offset);
+ if (native_loc_addr_role(dst) == NATIVE_LOC_ADDR_ROLE_STORAGE) {
+ NativeAddr addr = aa_storage_addr(a, dst, offset);
addr.base_type = src.type;
aa_emit_mem(a, 0, src, addr, mem);
return;
@@ -3338,6 +3490,24 @@ static void aa_store_outgoing_part(NativeTarget* t, int tail_call,
aa_emit_mem(aa_of(t), 0, src, addr, mem);
}
+/* Copy only the bytes that belong to a stack-routed value. The caller owns the
+ * separately rounded ABI slot and advances its physical stack cursor itself;
+ * this helper never treats that carrier/padding size as readable source data. */
+static void aa_store_outgoing_value_bytes(NativeTarget* t, int tail_call,
+ u32 stack_off, NativeLoc src,
+ u32 value_size) {
+ NativeLoc tmp =
+ native_loc_reg(src.type, NATIVE_REG_INT, AA_TMP0);
+ u32 off = 0;
+ while (off < value_size) {
+ u32 chunk = value_size - off;
+ if (chunk > 8u) chunk = 8u;
+ aa_load_part(t, tmp, src, off, chunk);
+ aa_store_outgoing_part(t, tail_call, stack_off + off, tmp, chunk);
+ off += chunk;
+ }
+}
+
static const ABIArgInfo* aa_param_abi(NativeTarget* t, const ABIFuncInfo* abi,
const NativeCallDesc* desc, u32 i,
ABIArgInfo* scratch) {
@@ -3552,6 +3722,8 @@ static void aa_plan_call(NativeTarget* t, const NativeCallDesc* desc,
NativeCallPlan* plan) {
NativeCallPlanRet* rets;
const ABIFuncInfo* abi = abi_cg_func_info(t->c->abi, desc->fn_type);
+ NativeLoc saved_indirect_callee;
+ int preserve_indirect_callee = 0;
/* Right-size the result scratch to the exact number of entries the ret loops
* below write: nparts on a DIRECT register return, 1 on the !abi fallback,
* 0 (NULL) otherwise (IGNORE / sret / no results). */
@@ -3568,17 +3740,19 @@ static void aa_plan_call(NativeTarget* t, const NativeCallDesc* desc,
plan->is_variadic = abi && abi->variadic;
plan->stack_arg_size = aa_call_stack_size(t, desc);
native_frame_note_outgoing(&aa_of(t)->frame, plan->stack_arg_size);
- /* Indirect call whose callee lives in x0..x7: the upcoming arg-load loop
- * writes those same registers and would clobber the function pointer
- * before blr reads it. Stash callee into AA_TMP0 (x16) up front and
- * retarget the call. (AA_TMP0 is a backend scratch, never an arg reg.) */
+ /* An indirect target in an argument/result or backend-private register can
+ * be overwritten while this hook fills x0..x7, x8 (sret), or uses x16/x17
+ * for exact-width stack arguments. Preserve it in LR, whose incoming value
+ * is already frame-saved for a non-tail call and is dead on a tail path.
+ * After all argument work, move it to the final call/branch register. */
if (plan->callee.kind == NATIVE_LOC_REG &&
(NativeAllocClass)plan->callee.cls == NATIVE_REG_INT &&
- plan->callee.v.reg < 8u) {
- NativeLoc scratch =
- native_loc_reg(plan->callee.type, NATIVE_REG_INT, AA_TMP0);
- aa_move(t, scratch, plan->callee);
- plan->callee = scratch;
+ (plan->callee.v.reg <= AA_X8 || plan->callee.v.reg == AA_TMP0 ||
+ plan->callee.v.reg == AA_TMP1)) {
+ saved_indirect_callee =
+ native_loc_reg(plan->callee.type, NATIVE_REG_INT, AA_LR);
+ aa_move(t, saved_indirect_callee, plan->callee);
+ preserve_indirect_callee = 1;
}
{
u32 next_int = 0, next_fp = 0, stack = 0, nmoves = 0;
@@ -3623,17 +3797,13 @@ static void aa_plan_call(NativeTarget* t, const NativeCallDesc* desc,
continue;
}
{
- /* Aggregate / multi-word variadic argument: copy its bytes in
- * 8-byte chunks (each chunk's home bytes are all live). */
- NativeLoc tmpreg =
- native_loc_reg(desc->args[i].type, NATIVE_REG_INT, AA_TMP0);
- u32 off = 0;
- while (off < n) {
- u32 chunk = (n - off > 8u) ? 8u : (n - off);
- aa_load_part(t, tmpreg, desc->args[i], off, chunk);
- aa_store_outgoing_part(t, tail_call, stack + off, tmpreg, chunk);
- off += chunk;
- }
+ /* Aggregate / multi-word variadic argument: `n` is the physical ABI
+ * carrier (Apple rounds a 12-byte unnamed aggregate to 16), not a
+ * source access width. Copy the value's exact object bytes, leaving
+ * carrier padding untouched, then advance by the full slot. */
+ u32 value_size = type_size32(t, desc->args[i].type);
+ aa_store_outgoing_value_bytes(t, tail_call, stack, desc->args[i],
+ value_size);
stack += n;
}
continue;
@@ -3696,6 +3866,13 @@ static void aa_plan_call(NativeTarget* t, const NativeCallDesc* desc,
aa_addr_of_loc(t, x8, desc->results[0]);
}
}
+ if (preserve_indirect_callee) {
+ NativeLoc final_callee = native_loc_reg(
+ saved_indirect_callee.type, NATIVE_REG_INT,
+ tail_call ? AA_TAIL_TARGET : AA_TMP0);
+ aa_move(t, final_callee, saved_indirect_callee);
+ plan->callee = final_callee;
+ }
}
if (abi && abi->ret.kind == ABI_ARG_DIRECT && desc->nresults) {
u32 nr = 0, ni = 0, nf = 0;
@@ -3818,7 +3995,7 @@ static void aa_plan_ret(NativeTarget* t, const CGFuncDesc* fd,
dst_addr.base_kind = NATIVE_ADDR_BASE_REG;
dst_addr.base.reg = AA_X8;
dst_addr.base_type = value->type;
- src_addr = aa_loc_addr(a, *value, 0);
+ src_addr = aa_storage_addr(a, *value, 0);
src_addr.base_type = value->type;
memset(&access, 0, sizeof access);
access.type = value->type;
@@ -3831,6 +4008,38 @@ static void aa_plan_ret(NativeTarget* t, const CGFuncDesc* fd,
}
if (value && abi && abi->ret.kind == ABI_ARG_DIRECT) {
u32 ni = 0, nf = 0;
+ int exact_parts = 0;
+ for (u32 p = 0; p < abi->ret.nparts; ++p) {
+ const ABIArgPart* part = &abi->ret.parts[p];
+ if (part->cls == ABI_CLASS_INT && part->size != 1u &&
+ part->size != 2u && part->size != 4u && part->size != 8u) {
+ exact_parts = 1;
+ break;
+ }
+ }
+
+ /* The generic return write phase materializes a memory source using the
+ * scalar carrier type, so an I64 carrier for a 3/5/6/7-byte aggregate part
+ * becomes an eight-byte load before NativeCallPlanRet.mem can narrow the
+ * final move. Once one exact-width part is present, marshal every return
+ * part here in ABI order. aa_load_part retains the exact byte count and
+ * uses only backend-private temporaries, so already-filled x0/x1 or v0..v3
+ * return registers remain live while later parts are loaded. */
+ if (exact_parts) {
+ for (u32 p = 0; p < abi->ret.nparts; ++p) {
+ const ABIArgPart* part = &abi->ret.parts[p];
+ NativeAllocClass cls =
+ part->cls == ABI_CLASS_FP ? NATIVE_REG_FP : NATIVE_REG_INT;
+ KitCgTypeId pty = aa_part_scalar_type(part);
+ Reg rreg = cls == NATIVE_REG_FP ? nf++ : ni++;
+ NativeLoc dst = native_loc_reg(pty, cls, rreg);
+ aa_load_part(t, dst, *value, part->src_offset, part->size);
+ }
+ *out_rets = NULL;
+ *out_nrets = 0;
+ return;
+ }
+
rets = arena_zarray(t->c->tu, NativeCallPlanRet, abi->ret.nparts);
for (u32 p = 0; p < abi->ret.nparts; ++p) {
const ABIArgPart* part = &abi->ret.parts[p];
@@ -4469,12 +4678,11 @@ static int aa_machine_op_clobbers(NativeTarget* t, const NativeMachineOp* op,
/* ELF Local-Exec materializes the address using only the destination
* register (mrs tpidr_el0 + add/add into rd) — no extra clobbers. The
* Mach-O TLV sequence loads the descriptor into x0 and calls the resolver
- * thunk through x16, clobbering x0/x16/x17 and the link register; the JIT
- * relaxation of that same sequence keeps the x0/x16/x17 footprint. Model
- * the descriptor-model clobbers so a value live across a TLS access is not
- * left in one of these registers. */
+ * thunk through backend-private x16, also clobbering private x17/LR. Only
+ * x0 can hold an optimizer-owned value, so it is the sole effect at this
+ * boundary. */
if (!obj_format_tls_via_descriptor(t->c)) return 0;
- mask[NATIVE_REG_INT] = (1u << 0) | (1u << 16) | (1u << 17) | (1u << AA_LR);
+ mask[NATIVE_REG_INT] = 1u << 0;
return 1;
}
if ((NativeMachineOpKind)op->kind != NATIVE_MOP_INTRINSIC ||
@@ -4497,33 +4705,34 @@ static void aa_bind_native_param(NativeTarget* t, const CGParamDesc* p,
/* ABI argument/return registers (x0..x7) come FIRST so the -O0 value cache
* prefers them: an argument expression then computes straight into its ABI arg
* register and the per-call parallel-copy (native_arg_shuffle) collapses to
- * no-ops — the tcc get_reg(0..N) trick (mirrors rv_int_allocable a0..a7 and
+ * no-ops — the tcc get_reg(0..N) trick (mirrors RV's NDT a0..a7 bank and
* x64's leading rsi/rdi). They are caller-saved, so the deferred prologue stays
* callee-save-free. Incoming params are spilled to frame homes at entry before
* any body op (aa_bind_native_param), so caching in x0..x7 cannot clobber a
* live parameter. Next come the remaining caller-saved temps x8/x12..x15;
* callee-saved x19..x28 are appended and only chosen under register pressure,
- * after which the prologue saves/restores them. x9/x10/x11 are the int emit
- * scratch (aa_int_scratch) and stay out of this pool. */
-static const Reg aa_int_allocable[] = {0u, 1u, 2u, 3u, 4u, 5u, 6u, 7u,
- 8u, 12u, 13u, 14u, 15u, 19u, 20u, 21u,
- 22u, 23u, 24u, 25u, 26u, 27u, 28u};
-/* Three int scratch registers, not two: a 3-operand op (e.g. `binop dst, a, b`
- * or `store [base+index], value`) whose dst/sources all spill needs three
- * distinct scratch regs at emit time — the IR spill-rewrite round-robins
- * operands across this pool and the native emitter materializes each into one.
- * With only two, an immediate operand of an all-spilled binop had nowhere to
- * land. x9/x10/x11 are all caller-saved temporaries reserved out of the
- * allocable set above. */
+ * after which the prologue saves/restores them. x9/x10/x11 are the explicit O1
+ * operand-temp bank (also the O0 scratch policy) and stay out of this pool. */
+static const Reg aa_ndt_int_allocable[] = {
+ 0u, 1u, 2u, 3u, 4u, 5u, 6u, 7u, 8u, 12u, 13u, 14u,
+ 15u, 19u, 20u, 21u, 22u, 23u, 24u, 25u, 26u, 27u, 28u};
+/* Three O1 operand temporaries, not two: a 3-operand op (for example
+ * `binop dst, a, b` or `store [base+index], value`) can require three distinct
+ * materializations in one instruction scope. Location MIR leaves spilled
+ * operands in their frame homes until this point; the scope leases x9/x10/x11
+ * without inventing persistent MIR register values. All three are caller-saved
+ * and reserved out of the allocable set above. The O0 direct target happens to
+ * use the same bank through its independent `scratch` policy. */
static const Reg aa_int_scratch[] = {9u, 10u, 11u};
+static const Reg aa_int_asm_temps[] = {AA_TMP0, AA_TMP1};
/* ABI fp argument/return registers (v0..v7) come FIRST (same rationale as the
* int pool: fp args land in place, the shuffle is a no-op). Then the caller-
* saved fp temps v18/v19, v22..v31 — the rest of the -O0 value cache under
- * ndt_caller_saved_only. v16 is the fp emit scratch (the fp AA_TMP0, also the
- * arg-shuffle fp cycle scratch) and v20/v21 are the NDT per-op fp scratch, so
- * all three stay out of the pool. Callee-saved v8..v15 (AAPCS64) are appended
+ * ndt_caller_saved_only. v16 is a backend-private/arg-shuffle cycle temporary;
+ * v20/v21 are the O1 operand-temp bank and the O0 per-op scratch policy, so all
+ * three stay out of the pool. Callee-saved v8..v15 (AAPCS64) are appended
* like the int set; the caller-saved-only NDT never selects them. */
-static const Reg aa_fp_allocable[] = {
+static const Reg aa_ndt_fp_allocable[] = {
0u, 1u, 2u, 3u, 4u, 5u, 6u, 7u, 18u, 19u, 22u, 23u, 24u, 25u,
26u, 27u, 28u, 29u, 30u, 31u, 8u, 9u, 10u, 11u, 12u, 13u, 14u, 15u};
static const Reg aa_fp_scratch[] = {20u, 21u};
@@ -4637,10 +4846,16 @@ static const NativePhysRegInfo aa_fp_phys[] = {
static const NativeAllocClassInfo aa_classes[] = {
{.cls = NATIVE_REG_INT,
- .allocable = aa_int_allocable,
- .nallocable = sizeof aa_int_allocable / sizeof aa_int_allocable[0],
+ .ndt_allocable = aa_ndt_int_allocable,
+ .ndt_allocable_count =
+ sizeof aa_ndt_int_allocable / sizeof aa_ndt_int_allocable[0],
.scratch = aa_int_scratch,
.nscratch = sizeof aa_int_scratch / sizeof aa_int_scratch[0],
+ .emit_temps = aa_int_scratch,
+ .nemit_temps = sizeof aa_int_scratch / sizeof aa_int_scratch[0],
+ .asm_temps = aa_int_asm_temps,
+ .nasm_temps = sizeof aa_int_asm_temps / sizeof aa_int_asm_temps[0],
+ .emit_cache_mask = (1u << 9u) | (1u << 10u) | (1u << 11u),
.phys = aa_int_phys,
.nphys = sizeof aa_int_phys / sizeof aa_int_phys[0],
.caller_saved_mask = 0x0007ffffu,
@@ -4650,10 +4865,14 @@ static const NativeAllocClassInfo aa_classes[] = {
.reserved_mask =
(1u << AA_TMP0) | (1u << AA_TMP1) | (1u << AA_FP) | (1u << AA_LR)},
{.cls = NATIVE_REG_FP,
- .allocable = aa_fp_allocable,
- .nallocable = sizeof aa_fp_allocable / sizeof aa_fp_allocable[0],
+ .ndt_allocable = aa_ndt_fp_allocable,
+ .ndt_allocable_count =
+ sizeof aa_ndt_fp_allocable / sizeof aa_ndt_fp_allocable[0],
.scratch = aa_fp_scratch,
.nscratch = sizeof aa_fp_scratch / sizeof aa_fp_scratch[0],
+ .emit_temps = aa_fp_scratch,
+ .nemit_temps = sizeof aa_fp_scratch / sizeof aa_fp_scratch[0],
+ .emit_cache_mask = (1u << 20u) | (1u << 21u),
.phys = aa_fp_phys,
.nphys = sizeof aa_fp_phys / sizeof aa_fp_phys[0],
/* v8..v15 are callee-saved (low 64 bits per AAPCS64); the rest are
@@ -4747,7 +4966,7 @@ static const NativeRegInfo aa_reg_info = {
/* aa_emit_mem emits ldrsb/ldrsh (opc=10, X form) for an MF_SEXT_LOAD narrow
* integer load, filling the whole register — so the -O0 NDT may drop the
* following CV_SEXT (Lever 4). */
- .ndt_load_sext = 1u,
+ .load_sext = 1u,
.resolve_name = aa_resolve_name,
.asm_operand_reg_ok = aa_asm_operand_reg_ok,
.asm_constraint_reg = aa_asm_constraint_reg,
@@ -4902,7 +5121,7 @@ static void aa_bind_native_param(NativeTarget* t, const CGParamDesc* p,
src = native_loc_reg(p->type, cls, a->next_param_int++);
} else {
/* Stack-passed part: load straight into the dst register when possible,
- * otherwise a scratch for the store-to-frame path. */
+ * otherwise use a backend-private transfer register for the frame store. */
Reg tmp =
reg_dst ? (Reg)dst.v.reg : (cls == NATIVE_REG_FP ? 16u : AA_TMP0);
NativeAddr saddr;
@@ -5033,11 +5252,10 @@ static void aa_load_ap_addr(NativeDirectTarget* d, Operand ap_addr,
d->native->load_addr(d->native, dst, ap);
}
-/* The va cores use only non-allocable registers for their temporaries
- * (scratch x9/x10, reserved x16=TMP0 / x17=TMP1, vector v16) so they never
- * clobber a value the optimizer's register allocator may hold live across the
- * op. The va_list base register is supplied by the caller (ap.base.reg), which
- * the optimizer materializes into a safe register before the call. */
+/* The va cores use caller-provided operand registers plus backend-private
+ * x16/x17 and v16. Direct wrappers choose x9/x10 from the O0 scratch policy;
+ * the optimized emitter supplies an instruction-scoped operand location. None
+ * can alias a persistently allocated live value. */
static u32 aa_va_base_reg(AANativeTarget* a, NativeAddr ap) {
if (ap.base_kind != NATIVE_ADDR_BASE_REG)
compiler_panic(a->base.c, a->func ? a->func->loc : (SrcLoc){0, 0, 0},
@@ -5126,9 +5344,8 @@ static void aa_va_arg_core(AANativeTarget* a, NativeLoc dst, NativeAddr ap,
ABIVaListInfo vai = abi_va_list_layout(t->c->abi);
NativeLoc cur =
native_loc_reg(builtin_id(KIT_CG_BUILTIN_I64), NATIVE_REG_INT, AA_TMP0);
- /* The fetched value is written directly into the caller-provided register
- * `dst`, which the caller guarantees is distinct from the va_list base
- * register. Only TMP0/TMP1 are used as private scratch. */
+ /* The fetched value is written directly into caller-provided `dst`, which is
+ * distinct from the va_list base. Only TMP0/TMP1 are backend-private. */
NativeLoc val = dst;
NativeAddr src;
MemAccess ptr_mem = aa_mem_for_type(t, cur.type, 8);
@@ -5223,8 +5440,8 @@ static void aa_va_copy_core(AANativeTarget* a, NativeAddr dst_ap,
/* ---- Direct-path (NativeDirectTarget) wrappers: convert semantic operands to
* NativeAddr/NativeLoc, then call the shared cores above. ---- */
-/* The cores reserve x16/x17 (TMP0/TMP1) as private scratch and require the
- * va_list base register(s) to be distinct from those. aa_direct_pointer_addr
+/* The cores reserve x16/x17 (TMP0/TMP1) as backend-private temporaries and
+ * require the va_list base register(s) to be distinct. aa_direct_pointer_addr
* returns the pointer in TMP1, so the direct wrappers first relocate it into
* x9/x10 before calling the cores. */
static NativeAddr aa_direct_va_base(NativeDirectTarget* d, Operand ap_addr,
@@ -5547,10 +5764,10 @@ static void aa_direct_asm_block(NativeDirectTarget* d, const char* tmpl,
const Sym* clobbers, u32 nclob,
u32 clobber_abi_sets) {
static const NativeAsmDirectHooks hooks = {
- /* Reserve the two emit scratch regs, x18 (platform reg), fp/lr/sp. */
- .scratch_int = (1u << AA_TMP0) | (1u << AA_TMP1) | (1u << 18u) |
- (1u << AA_FP) | (1u << AA_LR) | (1u << AA_SP),
- .scratch_fp = (1u << 20u) | (1u << 21u),
+ /* Reserve backend-private x16/x17, x18 (platform reg), fp/lr/sp. */
+ .reserved_int_mask = (1u << AA_TMP0) | (1u << AA_TMP1) | (1u << 18u) |
+ (1u << AA_FP) | (1u << AA_LR) | (1u << AA_SP),
+ .reserved_fp_mask = (1u << 20u) | (1u << 21u),
.opk_reg = AA64_INLINE_OPK_REG,
.opcls_fp = AA64_INLINE_OPCLS_FP,
.panic = aa_asm_panic,
@@ -5572,15 +5789,9 @@ static void aa_direct_asm_block(NativeDirectTarget* d, const char* tmpl,
/* ---- NativeTarget (optimizer) asm hook ----
*
- * The optimizer has already allocated every operand register and arranged the
- * surrounding data flow (inputs are live in their registers on entry, outputs
- * are consumed from their registers on exit; the asm's clobber_mask kept the
- * allocator from holding live values in clobbered registers). So unlike the
- * direct path this hook does NOT self-allocate registers and does NOT load
- * inputs / store outputs -- it only binds the pre-allocated registers to the
- * template, materializing memory-operand base addresses into the reserved
- * scratch registers and saving/restoring callee-saved registers the asm
- * clobbers (the only ABI obligation the allocator cannot discharge itself). */
+ * The optimized emitter owns register placement, staging, and output
+ * writeback. This hook binds its concrete locations and only materializes
+ * memory-constraint bases in backend-private registers. */
static NativeAddr aa_asm_loc_to_addr(AANativeTarget* a, SrcLoc loc,
NativeLoc src) {
@@ -5610,8 +5821,8 @@ static NativeAddr aa_asm_loc_to_addr(AANativeTarget* a, SrcLoc loc,
}
/* Resolve a memory-constraint operand to a single base register with zero
- * offset, folding any frame/global/offset into a scratch register. At most the
- * two reserved scratch registers are used across one asm block. */
+ * offset, folding any frame/global/offset into a backend-private register. At
+ * most x16/x17 are used across one asm block. */
static Reg aa_asm_native_mem_base(AANativeTarget* a, SrcLoc loc, NativeLoc src,
u32* ntmp) {
NativeAddr addr = aa_asm_loc_to_addr(a, loc, src);
@@ -5620,7 +5831,7 @@ static Reg aa_asm_native_mem_base(AANativeTarget* a, SrcLoc loc, NativeLoc src,
Reg dst;
if (addr.index_kind != NATIVE_ADDR_INDEX_NONE)
aa_asm_panic_at(a->base.c, loc, "indexed memory asm operand unsupported");
- aa_addr_base(a, addr, &base, &off);
+ aa_addr_base(a, addr, 0u, &base, &off);
if (off == 0) return (Reg)base;
if (*ntmp >= 2u)
aa_asm_panic_at(a->base.c, loc, "too many memory asm operands");
@@ -5630,50 +5841,6 @@ static Reg aa_asm_native_mem_base(AANativeTarget* a, SrcLoc loc, NativeLoc src,
return dst;
}
-static Reg aa_asm_stage_reg(AANativeTarget* a, SrcLoc loc,
- NativeAllocClass cls, u32* nint, u32* nfp) {
- static const Reg int_regs[] = {AA_TMP0, AA_TMP1};
- static const Reg fp_regs[] = {20u, 21u};
- if (cls == NATIVE_REG_FP) {
- if (*nfp >= (u32)(sizeof fp_regs / sizeof fp_regs[0]))
- aa_asm_panic_at(a->base.c, loc, "too many staged fp asm operands");
- return fp_regs[(*nfp)++];
- }
- if (*nint >= (u32)(sizeof int_regs / sizeof int_regs[0]))
- aa_asm_panic_at(a->base.c, loc, "too many staged integer asm operands");
- return int_regs[(*nint)++];
-}
-
-static void aa_asm_load_loc_to_reg(AANativeTarget* a, SrcLoc loc, NativeLoc src,
- NativeLoc dst) {
- NativeTarget* t = &a->base;
- NativeAllocClass cls = (NativeAllocClass)dst.cls;
- if (src.kind == NATIVE_LOC_REG) {
- if (src.v.reg != dst.v.reg || src.cls != dst.cls) t->move(t, dst, src);
- return;
- }
- if (src.kind == NATIVE_LOC_IMM) {
- if (cls != NATIVE_REG_INT)
- aa_asm_panic_at(t->c, loc,
- "floating-point immediate asm input is unsupported");
- t->load_imm(t, dst, src.v.imm);
- return;
- }
- aa_emit_mem(a, 1, dst, aa_asm_loc_to_addr(a, loc, src),
- aa_mem_for_type(t, dst.type, loc_size32(t, dst)));
-}
-
-static void aa_asm_store_reg_to_loc(AANativeTarget* a, SrcLoc loc,
- NativeLoc dst, NativeLoc src) {
- NativeTarget* t = &a->base;
- if (dst.kind == NATIVE_LOC_REG) {
- if (dst.v.reg != src.v.reg || dst.cls != src.cls) t->move(t, dst, src);
- return;
- }
- aa_emit_mem(a, 0, src, aa_asm_loc_to_addr(a, loc, dst),
- aa_mem_for_type(t, src.type, loc_size32(t, src)));
-}
-
static void aa_asm_native_panic(NativeTarget* t, SrcLoc loc, const char* msg) {
aa_asm_panic_at(t->c, loc, msg);
}
@@ -5683,23 +5850,6 @@ static Reg aa_asm_native_mem_base_hook(NativeTarget* t, SrcLoc loc,
return aa_asm_native_mem_base(aa_of(t), loc, src, ntmp);
}
-static Reg aa_asm_native_stage_reg_hook(NativeTarget* t, SrcLoc loc,
- NativeAllocClass cls, u32* ntmp,
- u32* nstage_int, u32* nstage_fp) {
- (void)nstage_int;
- return aa_asm_stage_reg(aa_of(t), loc, cls, ntmp, nstage_fp);
-}
-
-static void aa_asm_native_load_loc_hook(NativeTarget* t, SrcLoc loc,
- NativeLoc src, NativeLoc dst) {
- aa_asm_load_loc_to_reg(aa_of(t), loc, src, dst);
-}
-
-static void aa_asm_native_store_loc_hook(NativeTarget* t, SrcLoc loc,
- NativeLoc dst, NativeLoc src) {
- aa_asm_store_reg_to_loc(aa_of(t), loc, dst, src);
-}
-
static void aa_asm_native_run_template_hook(
NativeTarget* t, const char* tmpl, const AsmConstraint* outs, u32 nout,
Operand* bound_outs, const AsmConstraint* ins, u32 nin, Operand* bound_ins,
@@ -5719,15 +5869,10 @@ static void aa_asm_block_native(NativeTarget* t, const char* tmpl,
AANativeTarget* a = aa_of(t);
SrcLoc loc = a->func ? a->func->loc : (SrcLoc){0, 0, 0};
static const NativeAsmNativeHooks hooks = {
- .opk_reg = AA64_INLINE_OPK_REG,
- .opcls_fp = AA64_INLINE_OPCLS_FP,
.panic = aa_asm_native_panic,
.bound_reg = aa_asm_bound_reg,
.bound_mem = aa_asm_bound_mem,
.mem_base = aa_asm_native_mem_base_hook,
- .stage_reg = aa_asm_native_stage_reg_hook,
- .load_loc_to_reg = aa_asm_native_load_loc_hook,
- .store_reg_to_loc = aa_asm_native_store_loc_hook,
.run_template = aa_asm_native_run_template_hook,
};
native_asm_bind_native_operands(t, loc, tmpl, outs, nout, out_locs, ins, nin,
diff --git a/src/arch/arm32/native.c b/src/arch/arm32/native.c
@@ -15,9 +15,10 @@
* POP {r7, pc} ; restore r7 + return (Thumb bit from stacked lr)
*
* The NDT value-cache pool is the caller-saved arg registers r0..r3 (Lever 1:
- * args materialize directly into their ABI registers); r12 (IP) is the emit
- * scratch; r7 is the frame pointer. Callee-saved r4..r11 are reserved from the
- * -O0 pool (the optimizer allocates over them at -O1). */
+ * args materialize directly into their ABI registers); r12 (IP) is its O0
+ * scratch and a backend-private hook temporary; r7 is the frame pointer.
+ * Callee-saved r4..r11 are reserved from the -O0 pool (the optimizer allocates
+ * over them at -O1). */
#include <string.h>
#include "abi/abi.h"
@@ -80,9 +81,9 @@ typedef struct Arm32NativeTarget {
u8 is_variadic;
NativeFrameSlot sret_ptr_slot;
/* Lazily-allocated 4-byte home for an indirect call target when the call has
- * outgoing stack args. At -O0 the two emit scratch (IP, LR) are both consumed
- * staging stack args (IP = arg value, LR = large-offset address), so the callee
- * cannot survive in a scratch register across staging: it is spilled here and
+ * outgoing stack args. At -O0 the two backend staging registers (IP for the
+ * value, LR for a large-offset address) may both be consumed, so the callee
+ * cannot remain in either register across staging: it is spilled here and
* reloaded just before BLX. NATIVE_FRAME_SLOT_NONE until first needed. */
NativeFrameSlot callee_spill_slot;
u32 func_start;
@@ -212,14 +213,14 @@ static void arm_emit_pop(MCEmitter* mc, u32 reglist) {
* (Lever 1 via api_pack_call_args_in_order). The -O0 cache flushes at every
* call/branch/return, so caller-saved suffices and the prologue needs no
* callee-save spills. */
-static const Reg arm_int_allocable[] = {0u, 1u, 2u, 3u};
+static const Reg arm_ndt_int_allocable[] = {0u, 1u, 2u, 3u};
static const Reg arm_int_scratch[] = {ARM_SCRATCH};
-/* The -O1 optimizer's native-emit pass needs two scratch registers (e.g. to
- * store a spilled value through a spilled-pointer address). The -O0 NDT must NOT
- * take LR as an operand scratch (LR is the backend's dedicated div/staging temp,
- * and arm_emit_global_addr/copy_bytes use it), so LR is exposed only here, on the
- * optimizer path, where operands are allocated registers (never LR). */
-static const Reg arm_int_opt_scratch[] = {ARM_SCRATCH, ARM_TMP};
+/* O1 operand temporaries are the caller-saved low registers. They are excluded
+ * from persistent allocation and every Thumb2 operand hook accepts them. IP
+ * and LR remain backend-private: many hooks use those two internally, so
+ * exposing either as an operand location would couple independent lifetimes. */
+static const Reg arm_int_emit_temps[] = {0u, 1u, 2u, 3u};
+static const Reg arm_int_asm_temps[] = {ARM_TMP};
static const NativePhysRegInfo arm_int_phys[] = {
ARM_PHYS_ARG(0u, 0u), ARM_PHYS_ARG(1u, 1u),
@@ -228,7 +229,7 @@ static const NativePhysRegInfo arm_int_phys[] = {
ARM_PHYS_CALLEE(6u), ARM_PHYS_RESERVED(7u), /* fp */
ARM_PHYS_CALLEE(8u), ARM_PHYS_CALLEE(9u),
ARM_PHYS_CALLEE(10u), ARM_PHYS_CALLEE(11u),
- ARM_PHYS_RESERVED(12u), /* ip = scratch */
+ ARM_PHYS_RESERVED(12u), /* ip = O0/backend temp */
ARM_PHYS_RESERVED(13u), /* sp */
ARM_PHYS_RESERVED(14u), /* lr */
ARM_PHYS_RESERVED(15u), /* pc */
@@ -278,12 +279,16 @@ static int arm_asm_constraint_reg(const NativeRegInfo* ri, const char* body,
static const NativeAllocClassInfo arm_classes[] = {
{.cls = NATIVE_REG_INT,
- .allocable = arm_int_allocable,
- .nallocable = sizeof arm_int_allocable / sizeof arm_int_allocable[0],
+ .ndt_allocable = arm_ndt_int_allocable,
+ .ndt_allocable_count =
+ sizeof arm_ndt_int_allocable / sizeof arm_ndt_int_allocable[0],
.scratch = arm_int_scratch,
.nscratch = sizeof arm_int_scratch / sizeof arm_int_scratch[0],
- .opt_scratch = arm_int_opt_scratch,
- .opt_nscratch = sizeof arm_int_opt_scratch / sizeof arm_int_opt_scratch[0],
+ .emit_temps = arm_int_emit_temps,
+ .nemit_temps = sizeof arm_int_emit_temps / sizeof arm_int_emit_temps[0],
+ .asm_temps = arm_int_asm_temps,
+ .nasm_temps = sizeof arm_int_asm_temps / sizeof arm_int_asm_temps[0],
+ .emit_cache_mask = 0x0000000fu, /* r0..r3; hooks use private ip/lr */
.phys = arm_int_phys,
.nphys = sizeof arm_int_phys / sizeof arm_int_phys[0],
/* r0..r3 + r12 (lr/ip caller-saved; lr reserved) */
@@ -377,6 +382,7 @@ static int arm_addr_legal(NativeTarget* t, const NativeAddr* addr,
(void)mem;
if (!addr) return 0;
/* Base + 12-bit positive or 8-bit signed offset, no scaled index in v1. */
+ if (addr->base_kind == NATIVE_ADDR_BASE_FRAME_VALUE) return 0;
if (addr->index_kind != NATIVE_ADDR_INDEX_NONE) return 0;
return addr->offset >= -255 && addr->offset <= 4095;
}
@@ -477,17 +483,19 @@ static void arm_emit_global_addr(Arm32NativeTarget* a, u32 rd, ObjSymId sym,
/* The unified memory-access address layer (ARM32.md §1) lives here. Every
* load/store/copy resolves a NativeAddr to a legal (base register, in-range
- * displacement) through ONE function, arm_resolve_mem, with ONE scratch-
+ * displacement) through ONE function, arm_resolve_mem, with ONE temporary-
* reservation invariant asserted in one place.
*
- * Scratch budget: two registers are reserved as emit scratch in the function
- * body — IP (r12, ARM_SCRATCH) and LR (r14, ARM_TMP); r7 (FP) holds the frame
- * anchor and is never a scratch. A single access needs at most one
- * address-staging scratch, distinct from the transfer register `rt`. The picker
+ * Backend budget: IP (r12, ARM_SCRATCH) and LR (r14, ARM_TMP) are reserved for
+ * hook-internal staging; they are deliberately not O1 operand locations. r7
+ * (FP) holds the frame anchor and is never temporary storage. A single access
+ * needs at most one address-staging register, distinct from transfer register
+ * `rt`. The picker
* draws it from {IP,LR} minus {rt} minus a caller `pinned` mask (held bases the
* access must preserve) and panics if none is free, so the invariant is explicit
* rather than relying on incidental liveness. */
static void arm_emit_base_off(Arm32NativeTarget* a, u32 rd, u32 base, i32 off);
+static void arm_load_addr(NativeTarget* t, NativeLoc dst, NativeAddr addr);
/* True if every byte of a `span`-byte access at signed displacement `off` fits
* the Thumb-2 single-granule immediate forms (T4 negative -255..0, T3 positive
@@ -684,6 +692,7 @@ static void arm_emit_mem(Arm32NativeTarget* a, int is_load, NativeLoc reg,
NativeAddr addr, MemAccess mem) {
u32 rt = loc_reg(reg);
u32 size = mem.size ? mem.size : native_type_size(&a->base, reg.type);
+ if (size == 0u) arm_panic(a, "zero-width memory access");
if (size > 8u) arm_panic(a, "memory access wider than 8 bytes");
if (size == 8u) {
u32 hi = (rt + 1u) & 0xfu;
@@ -693,6 +702,63 @@ static void arm_emit_mem(Arm32NativeTarget* a, int is_load, NativeLoc reg,
arm_emit_mem_one(a, is_load, hi, r.base, r.off + 4, 4u);
return;
}
+ if (size != 1u && size != 2u && size != 4u) {
+ NativePartChunkIter it = native_part_chunks(size, 4u);
+ KitCgTypeId lane_type = builtin_id(KIT_CG_BUILTIN_I32);
+ u32 carrier_mask = 1u << rt;
+ u32 addr_inputs = 0u;
+ u32 chunk_off, chunk_size;
+ u32 tmp = REG_NONE;
+ ArmMemAddr r;
+ if (size > 4u) carrier_mask |= 1u << ((rt + 1u) & 0xfu);
+ if (addr.base_kind == NATIVE_ADDR_BASE_REG)
+ addr_inputs |= 1u << (addr.base.reg & 0xfu);
+ if (addr.index_kind == NATIVE_ADDR_INDEX_REG)
+ addr_inputs |= 1u << (addr.index.reg & 0xfu);
+ if (is_load && (carrier_mask & addr_inputs)) {
+ /* A single LDR may use the destination as its base (`ldr r0, [r0]`).
+ * An exact part is a sequence, so doing that would destroy the address
+ * before its later chunks. Retain the complete effective address in a
+ * private register first; avoid every address input because load_addr
+ * still needs them while it constructs the indexed form. */
+ u32 addr_reg = arm_pick_scratch(a, 0xffu, carrier_mask | addr_inputs);
+ NativeLoc addr_loc =
+ native_loc_reg(lane_type, NATIVE_REG_INT, (Reg)addr_reg);
+ arm_load_addr(&a->base, addr_loc, addr);
+ r.base = addr_reg;
+ r.off = 0;
+ } else {
+ /* Resolve once for the full span. Besides avoiding repeated global/far
+ * address formation, this gives every chunk one stable base and makes
+ * scratch ownership explicit: a later pack temporary must avoid r.base. */
+ r = arm_resolve_mem(a, &addr, 0xffu, size, carrier_mask);
+ }
+ /* ABIArgPart.size is exact even when the carrier is a 32-bit register.
+ * Split 3/5/6/7-byte parts into bounded 4/2/1-byte accesses and pack each
+ * chunk into the appropriate low/high carrier lane. This boundary is also
+ * used by generic call-result moves, so no caller can silently widen a
+ * semantic part into an adjacent frame object. */
+ while (native_part_chunk_next(&it, &chunk_off, &chunk_size)) {
+ u32 lane_reg = (rt + chunk_off / 4u) & 0xfu;
+ u32 lane_shift = (chunk_off & 3u) * 8u;
+ u32 transfer_reg = lane_reg;
+ if (lane_shift) {
+ if (tmp == REG_NONE)
+ tmp = arm_pick_scratch(a, 0xffu,
+ carrier_mask | (1u << (r.base & 0xfu)));
+ transfer_reg = tmp;
+ if (!is_load)
+ arm_emit_t32(a->base.mc,
+ arm_shift_imm(1u, tmp, lane_reg, lane_shift));
+ }
+ arm_emit_mem_one(a, is_load, transfer_reg, r.base,
+ r.off + (i32)chunk_off, chunk_size);
+ if (is_load && lane_shift)
+ arm_emit_t32(a->base.mc,
+ arm_orr_reg_lsl(lane_reg, lane_reg, tmp, lane_shift));
+ }
+ return;
+ }
{
ArmMemAddr r = arm_resolve_mem(a, &addr, rt, size, 0u);
arm_emit_mem_one(a, is_load, rt, r.base, r.off, size);
@@ -775,7 +841,7 @@ static void arm_emit_base_off(Arm32NativeTarget* a, u32 rd, u32 base, i32 off) {
* base (the resolver guarantees a staged base reg differs from its source);
* only when rd aliases base (load_addr's address-of-an-indexed-REG case)
* fall back to the reserved ARM_TMP, which is dead in that context. */
- u32 tmp = (rd != base) ? rd : ARM_TMP;
+ u32 tmp = (rd != base) ? rd : arm_pick_scratch(a, 0xffu, 1u << base);
arm_emit_load_u32(a, tmp, (u32)(off < 0 ? -off : off));
arm_emit_t32(mc, off < 0 ? arm_sub_reg(rd, base, tmp)
: arm_add_reg(rd, base, tmp));
@@ -800,6 +866,26 @@ static void arm_load_addr(NativeTarget* t, NativeLoc dst, NativeAddr addr) {
} else if (addr.base_kind == NATIVE_ADDR_BASE_FRAME) {
NativeFrameSlotEntry* s = native_frame_slot_at(&a->frame, addr.base.frame);
arm_emit_base_off(a, rd, ARM_FP, -(i32)s->off + addr.offset);
+ } else if (addr.base_kind == NATIVE_ADDR_BASE_FRAME_VALUE) {
+ NativeAddr load;
+ NativeLoc value;
+ MemAccess mem;
+ if (!addr.base_type)
+ arm_panic(a, "frame-value base has no exact type");
+ memset(&load, 0, sizeof load);
+ load.base_kind = NATIVE_ADDR_BASE_FRAME;
+ load.base.frame = addr.base.frame;
+ load.base_type = addr.base_type;
+ memset(&value, 0, sizeof value);
+ value.kind = NATIVE_LOC_REG;
+ value.cls = NATIVE_REG_INT;
+ value.type = addr.base_type;
+ value.v.reg = rd;
+ memset(&mem, 0, sizeof mem);
+ mem.type = value.type;
+ mem = native_mem_for_type(t, value.type, 0);
+ arm_emit_mem(a, 1, value, load, mem);
+ arm_emit_base_off(a, rd, rd, addr.offset);
} else if (addr.base_kind == NATIVE_ADDR_BASE_REG) {
arm_emit_base_off(a, rd, addr.base.reg & 0xfu, addr.offset);
} else {
@@ -808,6 +894,27 @@ static void arm_load_addr(NativeTarget* t, NativeLoc dst, NativeAddr addr) {
if (addr.index_kind == NATIVE_ADDR_INDEX_REG) {
u32 idx = addr.index.reg & 0xfu;
arm_emit_t32(mc, arm_add_reg_lsl(rd, rd, idx, addr.log2_scale));
+ } else if (addr.index_kind == NATIVE_ADDR_INDEX_FRAME_VALUE) {
+ NativeAddr load;
+ NativeLoc value;
+ MemAccess mem;
+ u32 idx = arm_pick_scratch(a, rd, 1u << rd);
+ if (!addr.index_type)
+ arm_panic(a, "frame-value index has no exact type");
+ memset(&load, 0, sizeof load);
+ load.base_kind = NATIVE_ADDR_BASE_FRAME;
+ load.base.frame = addr.index.frame;
+ load.base_type = addr.index_type;
+ memset(&value, 0, sizeof value);
+ value.kind = NATIVE_LOC_REG;
+ value.cls = NATIVE_REG_INT;
+ value.type = addr.index_type;
+ value.v.reg = idx;
+ memset(&mem, 0, sizeof mem);
+ mem.type = value.type;
+ mem = native_mem_for_type(t, value.type, 0);
+ arm_emit_mem(a, 1, value, load, mem);
+ arm_emit_t32(mc, arm_add_reg_lsl(rd, rd, idx, addr.log2_scale));
} else if (addr.index_kind != NATIVE_ADDR_INDEX_NONE) {
arm_panic(a, "unsupported address index kind in load_addr");
}
@@ -1434,15 +1541,16 @@ static void arm_func_begin_known_frame(NativeTarget* t, const CGFuncDesc* fd,
* frameless leaf): a leaf with no callee-saves, no body slots, no outgoing
* args, no sret/variadic and only register params needs no frame record, no r7
* anchor and no SUB sp. It cannot, however, return through a bare `BX lr`:
- * arm32 is register-starved and uses lr (ARM_TMP) pervasively as an emit-time
- * scratch — global-address staging in arm_resolve_mem, the `__builtin_*_overflow`
- * sequences, popcount, bitfield insert, the byte-copy granule engine — any of
- * which a leaf body may hit (e.g. `static int g; return g;` stages &g into lr,
+ * arm32 is register-starved and uses LR (ARM_TMP) pervasively as a
+ * backend-private temporary — global-address staging in arm_resolve_mem,
+ * `__builtin_*_overflow` sequences, popcount, bitfield insert, and the
+ * byte-copy granule engine. A leaf body may hit any of these (e.g.
+ * `static int g; return g;` stages &g into LR,
* after which a bare `BX lr` would branch to &g). So the tier still saves and
* restores the return address with a single-register `PUSH {lr}` / `POP {pc}`,
* leaving the body free to clobber lr; it just skips the r7 frame record and
* the stack reservation. (aa64/rv64 keep their return-address register out of
- * the scratch pool, so their frameless leaves skip even this push; arm32
+ * the backend temporary bank, so their frameless leaves skip even this push; arm32
* cannot.) Inline asm cannot occur here (asm_block is not installed). */
a->slim_prologue = kf && kf->is_leaf && !kf->reads_frame && cs_mask == 0 &&
!a->frame.has_alloca && a->frame.cum_off == 0 &&
@@ -1513,8 +1621,8 @@ static void arm_func_end(NativeTarget* t) {
mc_label_place(mc, a->epilogue_label);
if (a->slim_prologue)
/* Minimal leaf: only lr was saved (PUSH {lr}); pop it straight into pc. This
- * restores the return address the body may have clobbered using lr as an
- * emit scratch — see the func_begin slim_prologue note. */
+ * restores the return address the body may have clobbered using LR as a
+ * backend-private temporary — see the func_begin slim_prologue note. */
arm_emit_pop(mc, 1u << 15u); /* POP {pc} (16-bit, 0xbd00) */
else
arm_emit_frame_restore(a, /*to_pc=*/1);
@@ -1824,26 +1932,30 @@ static u32 arm_call_stack_bytes(NativeTarget* t, const NativeCallDesc* desc) {
static void arm_load_part(NativeTarget* t, NativeLoc dst, NativeLoc src,
u32 src_offset, u32 size) {
+ NativeAddr addr;
/* Load `size` bytes at src_offset of `src` into register `dst`. */
if (src.kind == NATIVE_LOC_REG) {
arm_move(t, dst, src);
return;
}
+ if (src.kind == NATIVE_LOC_IMM) {
+ i64 part;
+ if (!native_loc_imm_part(src, src_offset, size, &part))
+ arm_panic(arm_of(t), "invalid immediate argument part");
+ arm_load_imm(t, dst, part);
+ return;
+ }
+ if (native_loc_addr_role(src) == NATIVE_LOC_ADDR_ROLE_VALUE) {
+ if (src_offset != 0u || size != t->c->target.ptr_size ||
+ !native_loc_address_value(src, &addr))
+ arm_panic(arm_of(t), "split or invalid address-value argument");
+ arm_load_addr(t, dst, addr);
+ return;
+ }
{
- NativeAddr addr;
MemAccess mem;
- memset(&addr, 0, sizeof addr);
- if (src.kind == NATIVE_LOC_FRAME) {
- addr.base_kind = NATIVE_ADDR_BASE_FRAME;
- addr.base.frame = src.v.frame;
- addr.offset = (i32)src_offset;
- } else if (src.kind == NATIVE_LOC_STACK) {
- addr.base_kind = NATIVE_ADDR_BASE_FRAME;
- addr.base.frame = src.v.stack.slot;
- addr.offset = src.v.stack.offset + (i32)src_offset;
- } else {
+ if (!native_loc_storage_addr(src, (i32)src_offset, &addr))
arm_panic(arm_of(t), "unsupported arg source location");
- }
addr.base_type = dst.type;
memset(&mem, 0, sizeof mem);
mem.type = dst.type;
@@ -1877,17 +1989,8 @@ static void arm_store_outgoing(NativeTarget* t, u32 stack_off, NativeLoc src,
* to pass a large INDIRECT aggregate argument by reference). */
static void arm_addr_of_loc(NativeTarget* t, NativeLoc dst, NativeLoc src) {
NativeAddr addr;
- memset(&addr, 0, sizeof addr);
- if (src.kind == NATIVE_LOC_FRAME) {
- addr.base_kind = NATIVE_ADDR_BASE_FRAME;
- addr.base.frame = src.v.frame;
- } else if (src.kind == NATIVE_LOC_STACK) {
- addr.base_kind = NATIVE_ADDR_BASE_FRAME;
- addr.base.frame = src.v.stack.slot;
- addr.offset = src.v.stack.offset;
- } else {
+ if (!native_loc_storage_addr(src, 0, &addr))
arm_panic(arm_of(t), "address-of non-memory location");
- }
addr.base_type = dst.type;
arm_load_addr(t, dst, addr);
}
@@ -1920,6 +2023,7 @@ static void arm_plan_call(NativeTarget* t, const NativeCallDesc* desc,
? abi->ret.nparts
: ((!abi && desc->nresults) ? 1u : 0u);
int tail = (desc->flags & CG_CALL_TAIL) != 0;
+ int tail_callee_saved = 0;
memset(plan, 0, sizeof *plan);
rets = nrets_cap ? arena_zarray(t->c->tu, NativeCallPlanRet, nrets_cap) : NULL;
plan->callee = desc->callee;
@@ -1946,12 +2050,12 @@ static void arm_plan_call(NativeTarget* t, const NativeCallDesc* desc,
* register (r4..r11, or the chosen staging reg) needs no move.
*
* The exception is the -O0 (single-pass) call with outgoing stack args: staging
- * those args consumes BOTH emit scratch (IP for the value, LR for a large-offset
- * store address), so the callee cannot survive in LR. Spill it to its frame
- * home now — before any arg move clobbers its register — and reload just before
- * BLX (arm_emit_call). The known-frame (-O1) path keeps the register form: its
- * optimizer-allocated callee already survives, and its frame is final (no body-
- * time slot allocation). */
+ * those args may consume BOTH backend temporaries (IP for the value, LR for a
+ * large-offset store address), so the callee cannot survive in LR. Spill it
+ * to its frame home now — before any arg move clobbers its register — and
+ * reload just before BLX (arm_emit_call). The known-frame (-O1) path keeps
+ * the register form: its optimizer-allocated callee already survives, and
+ * its frame is final (no body-time slot allocation). */
if (plan->callee.kind == NATIVE_LOC_REG &&
(NativeAllocClass)plan->callee.cls == NATIVE_REG_INT) {
u32 cr = plan->callee.v.reg & 0xfu;
@@ -1976,6 +2080,22 @@ static void arm_plan_call(NativeTarget* t, const NativeCallDesc* desc,
}
}
}
+
+ /* An indirect sibling call lives in IP across argument marshalling because
+ * r0-r11 are restored by the tail epilogue and LR is the stack-part carrier.
+ * A three-byte part needs both private registers: LR carries the part while
+ * IP packs its final byte (and a far outgoing address can need IP as its
+ * resolved base). Preserve the callee with one balanced temporary stack save
+ * for the entire no-call marshalling interval, making both private registers
+ * available to the exact-width boundary. Tail stack arguments are addressed
+ * through FP into the incoming window, never through the temporarily shifted
+ * SP; CFA is FP-anchored. There are no compiler-time exits between this PUSH
+ * and its matching POP below. */
+ if (tail && plan->callee.kind == NATIVE_LOC_REG &&
+ loc_reg(plan->callee) == ARM_SCRATCH) {
+ arm_emit_push(t->mc, 1u << ARM_SCRATCH);
+ tail_callee_saved = 1;
+ }
{
u32 next_int = (abi && abi->has_sret) ? 1u : 0u;
u32 stack = 0, nmoves = 0, i, p;
@@ -2026,12 +2146,10 @@ static void arm_plan_call(NativeTarget* t, const NativeCallDesc* desc,
else stack = align_up_u32(stack, 4u);
if (desc->args[i].kind == NATIVE_LOC_REG) {
/* Source already in a register: store it straight to the outgoing
- * slot. Staging through the arg scratch (ip) would clobber a live
- * value the optimizer parked there — under register pressure it
- * spills/rematerializes arg values into the opt scratch regs (ip/lr,
- * see pass_lower scratch_for), so a register arg whose source is ip
- * was lost when an earlier stack arg's staging overwrote it (the
- * many-int variadic O1 miscompile). */
+ * slot. Routing it through backend-private IP is unnecessary and
+ * would add an independent lifetime while other call operands are
+ * still leased. Keeping the source location intact also preserves
+ * the instruction-scoped ownership contract. */
arm_store_outgoing(t, stack, desc->args[i], part->size, tail);
} else {
NativeLoc tmp = native_loc_reg(lty, NATIVE_REG_INT, arg_stage);
@@ -2058,6 +2176,7 @@ static void arm_plan_call(NativeTarget* t, const NativeCallDesc* desc,
}
}
}
+ if (tail_callee_saved) arm_emit_pop(t->mc, 1u << ARM_SCRATCH);
if (abi && abi->ret.kind == ABI_ARG_DIRECT && desc->nresults) {
u32 nr = 0, ni = 0, p;
for (p = 0; p < abi->ret.nparts; ++p) {
@@ -2122,7 +2241,7 @@ static void arm_emit_call(NativeTarget* t, const NativeCallPlan* plan) {
}
if (plan->callee.kind == NATIVE_LOC_STACK) {
/* -O0 indirect call with stack args: the callee was spilled to a frame home
- * (arm_plan_call) to free the emit scratch for arg staging. Arg staging is
+ * (arm_plan_call) to free the backend staging registers. Arg staging is
* done now, so IP is free; reload the target and BLX it. */
NativeLoc tmp = native_loc_reg(plan->callee.type, NATIVE_REG_INT, ARM_SCRATCH);
NativeAddr home;
@@ -2183,6 +2302,37 @@ static void arm_plan_ret(NativeTarget* t, const CGFuncDesc* fd,
}
if (value && abi && abi->ret.kind == ABI_ARG_DIRECT) {
u32 ni = 0, p;
+ int exact_parts = 0;
+ for (p = 0; p < abi->ret.nparts; ++p) {
+ const ABIArgPart* part = &abi->ret.parts[p];
+ if (part->size != 1u && part->size != 2u && part->size != 4u &&
+ part->size != 8u) {
+ exact_parts = 1;
+ break;
+ }
+ }
+
+ /* A return-plan register destination makes the generic write phase
+ * materialize its source using the scalar carrier type. For a three-byte
+ * AAPCS aggregate tail that carrier is I32, so the exact mem.size=3 on the
+ * final move arrives too late: materialization has already read four bytes.
+ * Future 5/6/7-byte parts have the same mismatch. Once any such part is
+ * present, fill every return register here in ABI order through the shared
+ * exact-width argument boundary. arm_load_part uses only backend-private
+ * temporaries, preserving return registers populated by earlier parts. */
+ if (exact_parts) {
+ for (p = 0; p < abi->ret.nparts; ++p) {
+ const ABIArgPart* part = &abi->ret.parts[p];
+ KitCgTypeId pty = arm_part_scalar_type(part);
+ NativeLoc dst =
+ native_loc_reg(pty, NATIVE_REG_INT, (Reg)(ni++));
+ arm_load_part(t, dst, *value, part->src_offset, part->size);
+ }
+ *out_rets = NULL;
+ *out_nrets = 0;
+ return;
+ }
+
for (p = 0; p < abi->ret.nparts; ++p) {
const ABIArgPart* part = &abi->ret.parts[p];
/* Per-lane type: an i64/double return is two i32 lanes in r0:r1; sizing
@@ -2533,11 +2683,12 @@ static void arm_alloca(NativeTarget* t, NativeLoc dst, NativeLoc size,
* routes 8-byte _Atomic to the spinlock libcall (atomic_lock_free_max=4), so
* only <=4-byte ops reach here.
*
- * Register budget: lr (ARM_TMP) holds the access ADDRESS. The NDT keeps the
- * value operands in the r0..r3 pool OR in ip (ARM_SCRATCH, its materialization
- * scratch), so the atomic sequences must NOT use ip as a temp — they would
- * clobber an operand that happens to live there. STREX additionally needs its
- * status, value and base registers mutually distinct.
+ * Register budget: LR (ARM_TMP) holds the access address. On the shared O0
+ * path, a value operand may occupy IP (ARM_SCRATCH); on O1, operands arrive in
+ * allocated r4..r11 or instruction-scoped r0..r3 locations and IP/LR remain
+ * backend-private. The core must therefore keep IP unavailable for its own
+ * extra state so it is safe on both paths. STREX additionally needs its status,
+ * value and base registers mutually distinct.
*
* The rmw/cas loops therefore borrow callee-saved registers for the private
* temporaries (new value + STREX status). At -O1 the optimizer freely allocates
@@ -2545,8 +2696,8 @@ static void arm_alloca(NativeTarget* t, NativeLoc dst, NativeLoc size,
* across the whole callee-saved set, so a fixed r4/r5 would alias an operand. The
* borrow is therefore OPERAND-AWARE (arm_atomic_borrow): it picks free callee-
* saved registers clear of the live operand+base set, guaranteeing base + all
- * operands + the STREX temporaries are mutually distinct (ARM32.md §2 "operand-
- * aware scratch budget"). The picked registers are push/pop-balanced so the
+ * operands + the STREX temporaries are mutually distinct. The picked
+ * registers are push/pop-balanced so the
* caller's callee-saved values are preserved (free at -O0, safe at -O1). */
/* Pick `n` distinct callee-saved temporaries (r4,r5,r6,r8,r9,r10,r11 — fp/r7 is
@@ -2583,10 +2734,9 @@ static int arm_order_release(KitCgMemOrder o) {
* the LDREX/STREX sequence then uses as the base. */
static u32 arm_atomic_addr_reg(Arm32NativeTarget* a, NativeAddr addr) {
/* Use the pointer's own register when the address already is one (the -O1
- * case: the optimizer collapses &x into a register distinct from the atomic
- * value operands). Materializing into LR would collide with a value operand
- * the optimizer staged in LR (an emit scratch). Otherwise (a FRAME/GLOBAL
- * address, e.g. -O0) materialize into LR, which holds no operand there. */
+ * case: location MIR keeps the allocated address component explicit).
+ * Otherwise (a FRAME/GLOBAL address, e.g. -O0) materialize into
+ * backend-private LR, which holds no operand there. */
if (addr.base_kind == NATIVE_ADDR_BASE_REG && addr.offset == 0 &&
addr.index_kind == NATIVE_ADDR_INDEX_NONE)
return addr.base.reg & 0xfu;
@@ -2847,9 +2997,10 @@ static void arm_va_arg_core(Arm32NativeTarget* a, NativeLoc dst, NativeAddr ap,
MCEmitter* mc = t->mc;
KitCgTypeId i32t = builtin_id(KIT_CG_BUILTIN_I32);
u32 sz = native_type_size(t, type);
- /* The cursor must survive the value load (dst = [cursor]) and the +4 update, so
- * it cannot alias `dst` — at -O1 the result is staged in a reserved scratch
- * (IP, or LR), so pick the other reserved reg for the cursor. */
+ /* The cursor must survive the value load (dst = [cursor]) and the +4 update,
+ * so it cannot alias `dst`. O1 destinations come from the scoped r0..r3 bank;
+ * the O0 direct path may use IP, so choose LR only for that case and IP
+ * otherwise. */
u32 cur_reg = (dst.kind == NATIVE_LOC_REG && loc_reg(dst) == ARM_SCRATCH)
? ARM_TMP
: ARM_SCRATCH;
@@ -2891,7 +3042,8 @@ static NativeAddr arm_va_addr_from_ptr(NativeLoc ap_ptr) {
/* The destination memory of a value-location: a wide (i64/double) va_arg result
* is memory-resident, so the optimizer hands its storage (a frame/stack slot, or
* an explicit address), NOT a pointer in a register. Convert it to a NativeAddr
- * the byte-copy can write through. (Mirrors riscv's rv_loc_addr.) */
+ * the byte-copy can write through. (Mirrors RISC-V's storage-address
+ * resolver.) */
static NativeAddr arm_loc_addr(Arm32NativeTarget* a, NativeLoc loc) {
NativeAddr addr;
memset(&addr, 0, sizeof addr);
@@ -2950,9 +3102,9 @@ static void arm_emit_setcc(MCEmitter* mc, u32 rd, u32 cc) {
/* The 32-bit `__builtin_*_overflow` family. dsts = [value, overflow]; args =
* [a, b]. i64 overflow is lowered to lane ops by the cg layer (arith.c), so the
- * backend only sees the 32-bit forms. lr/ip are the transient scratch (lr is
- * reserved; ip is the materialization scratch, free here since the operands are
- * already in their NDT registers). */
+ * backend only sees the 32-bit forms. LR/IP are backend-private temporaries on
+ * O1; on O0, IP's operand value has already been consumed before these internal
+ * sequences use it. */
/* nd_intrinsic forwards an immediate argument as NATIVE_LOC_IMM (not in a
* register). A value-consuming intrinsic needs it in a register: materialize it
* into `into` (an output register, free to reuse as an input) and return the reg
@@ -3080,7 +3232,7 @@ static void arm_intrinsic(NativeTarget* t, IntrinKind kind,
u32 width = native_type_size(t, dsts[0].type);
NativeLoc a0 = arm_intrin_arg(a, args[0], loc_reg(dsts[0]));
/* Use the 32-bit REV.W/REV16.W forms: the dst may be a high register
- * (the IP codegen scratch at -O0, r8..r11 at -O1) that the 16-bit T1
+ * (IP at -O0 or r4..r11 when allocated at -O1) that the 16-bit T1
* encoding cannot represent. */
if (width == 4u) {
arm_emit_t32(mc, arm_rev_w(loc_reg(dsts[0]), loc_reg(a0)));
@@ -3360,8 +3512,8 @@ static void arm_direct_load_operand_to_reg(NativeDirectTarget* d, Operand op,
static void arm_direct_load_address_to_reg(NativeDirectTarget* d, Operand op,
NativeLoc dst) {
/* For an "m" memory operand: a LOCAL's address is its frame home; an INDIRECT
- * (*(base+ofs)) names an address held in a frame slot, so materialize_addr
- * loads the pointer first (load_addr has no FRAME_VALUE base form). */
+ * (*(base+ofs)) names an address held in a frame slot; load_addr materializes
+ * that typed FRAME_VALUE through backend-owned temporaries. */
d->native->load_addr(d->native, dst, arm_direct_materialize_addr(d, op));
}
static void arm_direct_store_reg_to_operand(NativeDirectTarget* d, Operand op,
@@ -3460,10 +3612,11 @@ static void arm_direct_asm_block(NativeDirectTarget* d, const char* tmpl,
const Sym* clobbers, u32 nclob,
u32 clobber_abi_sets) {
static const NativeAsmDirectHooks hooks = {
- /* Reserve ip (scratch), fp, sp, lr, pc from the operand allocator. */
- .scratch_int = (1u << ARM_SCRATCH) | (1u << ARM_FP) | (1u << 13u) |
- (1u << 14u) | (1u << 15u),
- .scratch_fp = 0u,
+ /* Reserve backend/O0 IP plus FP, SP, LR, and PC from this direct-path
+ * operand allocator. */
+ .reserved_int_mask = (1u << ARM_SCRATCH) | (1u << ARM_FP) |
+ (1u << 13u) | (1u << 14u) | (1u << 15u),
+ .reserved_fp_mask = 0u,
.opk_reg = ARM32_INLINE_OPK_REG,
.opcls_fp = ARM32_INLINE_OPCLS_FP,
.panic = arm_asm_panic,
@@ -3483,9 +3636,7 @@ static void arm_direct_asm_block(NativeDirectTarget* d, const char* tmpl,
&hooks);
}
-/* Convert an inline-asm input location — a computed address, or a frame/stack
- * home — to a NativeAddr. Shared by the "m" (bind the address) and the staged
- * register (load the value) input paths. */
+/* Convert a concrete memory-constraint location to a NativeAddr. */
static NativeAddr arm_asm_in_loc_addr(NativeTarget* t, NativeLoc loc,
KitCgTypeId type) {
NativeAddr ma;
@@ -3504,20 +3655,19 @@ static NativeAddr arm_asm_in_loc_addr(NativeTarget* t, NativeLoc loc,
return ma;
}
-/* Reserve the next scratch register to stage an inline-asm operand into: IP (the
- * operand scratch) first, then LR. The native-emit asm path runs with both free
- * (the optimizer allocates asm operands to r0..r11 and never to IP/LR), so two
- * staged operands are available — enough for the operand shapes the corpus
- * exercises. A third staged operand is a clean diagnostic rather than a clobber. */
-static Reg arm_asm_stage_reg(NativeTarget* t, u32* nstage) {
+/* Reserve the next backend-private memory-base register: IP first, then LR.
+ * The optimized emitter never presents IP/LR as asm operands, so both are free
+ * here. */
+static Reg arm_asm_mem_base_reg(NativeTarget* t, u32* nbase) {
Reg r;
- if (*nstage == 0u)
+ if (*nbase == 0u)
r = (Reg)ARM_SCRATCH;
- else if (*nstage == 1u)
+ else if (*nbase == 1u)
r = (Reg)ARM_TMP;
else
- arm_asm_panic_at(t->c, arm_of(t)->loc, "too many staged inline-asm operands");
- (*nstage)++;
+ arm_asm_panic_at(t->c, arm_of(t)->loc,
+ "too many memory inline-asm operands");
+ (*nbase)++;
return r;
}
@@ -3528,7 +3678,7 @@ static void arm_asm_native_panic(NativeTarget* t, SrcLoc loc,
static Reg arm_asm_native_mem_base_hook(NativeTarget* t, SrcLoc loc,
NativeLoc src, u32* ntmp) {
- Reg base = arm_asm_stage_reg(t, ntmp);
+ Reg base = arm_asm_mem_base_reg(t, ntmp);
NativeAddr ma = arm_asm_in_loc_addr(t, src, src.type);
(void)loc;
arm_load_addr(
@@ -3537,50 +3687,6 @@ static Reg arm_asm_native_mem_base_hook(NativeTarget* t, SrcLoc loc,
return base;
}
-static Reg arm_asm_native_stage_reg_hook(NativeTarget* t, SrcLoc loc,
- NativeAllocClass cls, u32* ntmp,
- u32* nstage_int, u32* nstage_fp) {
- (void)loc;
- (void)nstage_int;
- (void)nstage_fp;
- if (cls != NATIVE_REG_INT)
- arm_asm_panic_at(t->c, arm_of(t)->loc,
- "floating-point asm operands are unsupported");
- return arm_asm_stage_reg(t, ntmp);
-}
-
-static void arm_asm_native_load_loc_hook(NativeTarget* t, SrcLoc loc,
- NativeLoc src, NativeLoc dst) {
- (void)loc;
- if (dst.cls != NATIVE_REG_INT)
- arm_asm_panic_at(t->c, arm_of(t)->loc,
- "floating-point asm operands are unsupported");
- if (src.kind == NATIVE_LOC_REG) {
- if (src.v.reg != dst.v.reg || src.cls != dst.cls) arm_move(t, dst, src);
- return;
- }
- if (src.kind == NATIVE_LOC_IMM) {
- arm_load_imm(t, dst, src.v.imm);
- return;
- }
- arm_load(t, dst, arm_asm_in_loc_addr(t, src, dst.type),
- native_mem_for_type(t, dst.type, native_type_size(t, dst.type)));
-}
-
-static void arm_asm_native_store_loc_hook(NativeTarget* t, SrcLoc loc,
- NativeLoc dst, NativeLoc src) {
- (void)loc;
- if (src.cls != NATIVE_REG_INT)
- arm_asm_panic_at(t->c, arm_of(t)->loc,
- "floating-point asm operands are unsupported");
- if (dst.kind == NATIVE_LOC_REG) {
- if (dst.v.reg != src.v.reg || dst.cls != src.cls) arm_move(t, dst, src);
- return;
- }
- arm_store(t, arm_asm_in_loc_addr(t, dst, src.type), src,
- native_mem_for_type(t, src.type, native_type_size(t, src.type)));
-}
-
static void arm_asm_native_run_template_hook(
NativeTarget* t, const char* tmpl, const AsmConstraint* outs, u32 nout,
Operand* bound_outs, const AsmConstraint* ins, u32 nin, Operand* bound_ins,
@@ -3599,15 +3705,10 @@ static void arm_asm_block_native(NativeTarget* t, const char* tmpl,
const Sym* clobbers, u32 nclob) {
SrcLoc loc = arm_of(t)->loc;
static const NativeAsmNativeHooks hooks = {
- .opk_reg = ARM32_INLINE_OPK_REG,
- .opcls_fp = ARM32_INLINE_OPCLS_FP,
.panic = arm_asm_native_panic,
.bound_reg = arm_asm_bound_reg,
.bound_mem = arm_asm_bound_mem,
.mem_base = arm_asm_native_mem_base_hook,
- .stage_reg = arm_asm_native_stage_reg_hook,
- .load_loc_to_reg = arm_asm_native_load_loc_hook,
- .store_reg_to_loc = arm_asm_native_store_loc_hook,
.run_template = arm_asm_native_run_template_hook,
};
native_asm_bind_native_operands(t, loc, tmpl, outs, nout, out_locs, ins, nin,
diff --git a/src/arch/mc.c b/src/arch/mc.c
@@ -238,6 +238,11 @@ ObjSymId mc_label_symbol(MCEmitter* m, MCLabel id) {
li->block_sym = obj_symbol(m->obj, name, SB_LOCAL, SK_NOTYPE,
li->placed ? li->sec_id : OBJ_SEC_NONE,
li->placed ? (u64)li->offset : 0u, 0);
+ /* A block-address symbol is a relocation anchor within its function, not a
+ * separately collectible/reorderable content unit. This distinction is
+ * material on Mach-O, where every ordinary symbol under
+ * MH_SUBSECTIONS_VIA_SYMBOLS otherwise starts a new atom. */
+ obj_symbol_set_atom_subordinate(m->obj, li->block_sym, 1);
return li->block_sym;
}
diff --git a/src/arch/native_target.h b/src/arch/native_target.h
@@ -147,19 +147,41 @@ typedef struct NativeAllocClassInfo {
u8 cls; /* NativeAllocClass */
u8 pad[3];
- const Reg* allocable;
- u32 nallocable;
-
+ /* NativeDirectTarget (-O0) value-cache register policy. Optimized
+ * allocation is described independently by NATIVE_REG_ALLOCABLE rows in
+ * `phys`; neither path may infer its value bank from the other. */
+ const Reg* ndt_allocable;
+ u32 ndt_allocable_count;
+
+ /* NativeDirectTarget (-O0) value-cache temporary policy only. Optimized
+ * emission must not infer operand registers from this list; it owns the
+ * independently declared, instruction-scoped `emit_temps` bank below. */
const Reg* scratch;
u32 nscratch;
- /* Optional native-emit scratch for the -O1 optimizer (pass_native_emit) when
- * it needs a different set than the -O0 NDT `scratch`. opt_nscratch == 0 means
- * the optimizer reuses `scratch`. Register-starved backends (arm32) expose a
- * second reserved register here (one the NDT must not take as an operand
- * scratch, e.g. a dedicated backend div/staging temp). */
- const Reg* opt_scratch;
- u32 opt_nscratch;
+ /* Registers owned exclusively by an -O1 native-emission instruction scope.
+ * They are operand-facing temporaries, not MIR values and not backend-private
+ * implementation registers. The set is explicit rather than inherited from
+ * the -O0 cache's `scratch` policy: those two clients have different
+ * lifetimes and may require disjoint banks (notably ARM32). */
+ const Reg* emit_temps;
+ u32 nemit_temps;
+
+ /* Backend-internal registers that may be loaned specifically while binding
+ * a register-only inline-asm block. They are otherwise unavailable to the
+ * optimizer emitter. The asm staging planner uses them only when the block
+ * has no memory constraint (whose address binder may need the same bank). */
+ const Reg* asm_temps;
+ u32 nasm_temps;
+
+ /* Subset of reserved optimizer emitter temps whose contents survive ordinary
+ * backend hooks when they are not passed as operands or named by a clobber
+ * effect. The native emitter may retain clean FS_SPILL values in these and
+ * in dead O1 allocation registers whose ABI preservation is established
+ * (caller-saved, or a callee-saved register present in the known frame)
+ * across adjacent instructions; O1 allocation-register preservation is
+ * governed by machine_op_clobbers below. */
+ u32 emit_cache_mask;
const NativePhysRegInfo* phys;
u32 nphys;
@@ -176,7 +198,7 @@ struct NativeRegInfo {
const NativeAllocClassInfo* classes;
u32 nclasses;
- /* True when this register model's scratch + caller-saved-allocable pool is
+ /* True when this register model's scratch + caller-saved NDT cache pool is
* rich enough that the single-pass (-O0) NativeDirectTarget never needs to
* fall back to a callee-saved register under pressure. When set, NDT is
* enforced caller-saved-only (nd_scratch_acquire skips callee-saved regs;
@@ -199,11 +221,11 @@ struct NativeRegInfo {
/* True when this backend honors MF_SEXT_LOAD by emitting a sign-extending
* load that fills the whole register (aa64 ldrsb/ldrsh -> X), so the shared
- * -O0 NDT may record load_sext on the loaded local and elide the following
+ * Native codegen may record load_sext on the loaded local and elide the following
* CV_SEXT. Clear (the default) on backends whose narrow load zero-extends
* regardless of the flag (x86-64, riscv64 today): there the convert must run,
* so the flag is a no-op and the extend is emitted normally. */
- u8 ndt_load_sext;
+ u8 load_sext;
/* Map a register name to its (Reg, class). `name` is the raw spelling
* ("rax", "x8", "a7"); the caller resolves any Sym to its bytes first so this
@@ -238,13 +260,21 @@ typedef enum NativeLocKind {
NATIVE_LOC_STACK,
NATIVE_LOC_IMM,
NATIVE_LOC_GLOBAL,
+ /* Address value of a concrete frame slot. Distinct from NATIVE_LOC_FRAME,
+ * which is the scalar value stored in the slot, and NATIVE_LOC_ADDR, which
+ * is the scalar value addressed by an effective-address expression. */
+ NATIVE_LOC_FRAME_ADDR,
NATIVE_LOC_ADDR,
} NativeLocKind;
typedef enum NativeAddrBaseKind {
NATIVE_ADDR_BASE_NONE,
+ /* Pointer value already held in a physical register. */
NATIVE_ADDR_BASE_REG,
+ /* Address of the frame slot itself: FP + slot offset. */
NATIVE_ADDR_BASE_FRAME,
+ /* Scalar pointer value stored in a frame slot. base_type is mandatory and
+ * is the exact value type, independent of the colored slot descriptor. */
NATIVE_ADDR_BASE_FRAME_VALUE,
NATIVE_ADDR_BASE_GLOBAL,
} NativeAddrBaseKind;
@@ -252,6 +282,8 @@ typedef enum NativeAddrBaseKind {
typedef enum NativeAddrIndexKind {
NATIVE_ADDR_INDEX_NONE,
NATIVE_ADDR_INDEX_REG,
+ /* Scalar index value stored in a frame slot. index_type is mandatory and is
+ * the exact value type, independent of the colored slot descriptor. */
NATIVE_ADDR_INDEX_FRAME_VALUE,
} NativeAddrIndexKind;
@@ -388,10 +420,13 @@ typedef struct NativeFramePatchState {
/* A semantic machine operation, enough for the target to report the physical
* registers its encoding clobbers as a side effect (e.g. x86 idiv writes
- * rax/rdx, variable shifts use cl). Built by the optimizer from an instruction;
- * the descriptor keeps the backend from depending on the optimizer IR. */
+ * rax/rdx, variable shifts use cl, and a bitfield read-modify-write borrows
+ * fixed temporaries). Built by the optimizer from an instruction; the
+ * descriptor keeps the backend from depending on the optimizer IR. */
typedef enum NativeMachineOpKind {
NATIVE_MOP_BINOP,
+ NATIVE_MOP_BITFIELD_LOAD,
+ NATIVE_MOP_BITFIELD_STORE,
NATIVE_MOP_VA_START,
NATIVE_MOP_VA_ARG,
NATIVE_MOP_ATOMIC_CAS,
@@ -419,6 +454,11 @@ typedef struct NativeMachineOp {
typedef struct NativeCallDesc {
KitCgTypeId fn_type;
NativeLoc callee;
+ /* Each argument is a complete semantic value. REG/IMM carry scalar bits;
+ * FRAME/STACK/ADDR name dereferenceable storage; GLOBAL/FRAME_ADDR are
+ * scalar address values and must classify as one pointer-sized ABI part.
+ * Backends use native_loc_addr_role plus the checked converters below rather
+ * than inferring value-vs-storage semantics from union fields. */
const NativeLoc* args;
const NativeLoc* results;
u32 nargs;
@@ -492,14 +532,18 @@ struct NativeTarget {
* sxtw/uxtw #scale]`), honoring NativeAddr.index_ext. */
int (*can_fold_shift_into_alu)(NativeTarget*);
int (*can_fold_extend_into_addr)(NativeTarget*);
- /* Optional. Report the physical registers the target's encoding of `op`
- * clobbers as a side effect (not its declared operands/results), one bitmask
- * per NativeAllocClass. The optimizer keeps values live ACROSS the
- * instruction out of these registers, so the backend may use them freely (x86
- * idiv writes rax/rdx; a variable shift uses cl; atomics use rax/rcx/rdx).
- * Return non-zero if any register is clobbered, 0 otherwise (the common,
- * unconstrained case). NULL means no instruction clobbers fixed registers
- * (aa64/rv64). */
+ /* Optional, but exhaustive when present. Report every optimizer-visible
+ * physical register the target's encoding of `op` clobbers as a side effect
+ * (not its declared operands/results), one bitmask per NativeAllocClass.
+ * This includes registers that can be MIR hard homes and cache-enabled emit
+ * temps whose retained spill value must be invalidated. Pure backend-private
+ * temporaries are reserved, never cached, and do not appear here. This single
+ * contract lets regalloc keep live values out of fixed clobbers and lets the
+ * emitter retain clean spill values safely. Examples include x86 idiv
+ * (rax/rdx), variable shift (rcx), bitfield RMW and atomics (rax/rcx/rdx),
+ * and format-dependent TLS sequences. Return non-zero if any register is
+ * clobbered, 0 otherwise. NULL promises no optimizer-visible fixed
+ * clobbers. */
int (*machine_op_clobbers)(NativeTarget*, const NativeMachineOp* op,
u32 clobber_mask[NATIVE_CALL_PLAN_CLASSES]);
@@ -586,14 +630,22 @@ struct NativeTarget {
void (*emit)(NativeTarget*, const NativeInst*);
/* All instruction-emission hooks require caller-selected legal physical
- * operands. In particular, dst values are NATIVE_LOC_REG, arithmetic sources
- * are NATIVE_LOC_REG or target-legal immediates, and memory base/index
- * registers in NativeAddr must already be materialized. NativeTarget may
- * validate and assert, but it must not allocate registers. */
+ * value operands. In particular, dst values are NATIVE_LOC_REG and
+ * arithmetic sources are NATIVE_LOC_REG or target-legal immediates.
+ * NativeAddr may retain typed FRAME_VALUE base/index components; the backend
+ * materializes those exact-width values only through its declared internal
+ * registers. It must not allocate or expose a new IR-visible register. */
void (*move)(NativeTarget*, NativeLoc dst_reg, NativeLoc src_reg);
void (*load_imm)(NativeTarget*, NativeLoc dst_reg, i64 imm);
void (*load_const)(NativeTarget*, NativeLoc dst_reg, ConstBytes);
+ /* Address materialization must accept dst_reg aliasing a BASE_REG component
+ * while still consuming any distinct index before the destination write.
+ * The emitter never requests dst_reg aliasing the index itself. */
void (*load_addr)(NativeTarget*, NativeLoc dst_reg, NativeAddr addr);
+ /* A one-register scalar load from a plain BASE_REG address must accept
+ * dst_reg aliasing that base: the effective address is consumed before the
+ * destination is written. This lets the target-neutral emitter reuse an
+ * instruction-leased address register under maximal temporary pressure. */
void (*load)(NativeTarget*, NativeLoc dst_reg, NativeAddr addr, MemAccess);
void (*store)(NativeTarget*, NativeAddr addr, NativeLoc src_reg, MemAccess);
void (*tls_addr_of)(NativeTarget*, NativeLoc dst_reg, ObjSymId sym,
@@ -602,6 +654,8 @@ struct NativeTarget {
AggregateAccess);
void (*set_bytes)(NativeTarget*, NativeAddr dst, NativeLoc byte_value,
AggregateAccess);
+ /* Like load(), a one-register bitfield load must consume a plain BASE_REG
+ * address before writing dst_reg and accept those registers aliasing. */
void (*bitfield_load)(NativeTarget*, NativeLoc dst_reg,
NativeAddr record_addr, BitFieldAccess);
void (*bitfield_store)(NativeTarget*, NativeAddr record_addr,
@@ -632,14 +686,24 @@ struct NativeTarget {
void (*spill)(NativeTarget*, NativeLoc src_reg, NativeFrameSlot, MemAccess);
void (*reload)(NativeTarget*, NativeLoc dst_reg, NativeFrameSlot, MemAccess);
+ /* Mutating ABI-marshalling phase, despite the historical `plan_` name. It
+ * may stage the callee, emit stack arguments and register shuffles, and fill
+ * any remaining generic moves/results in NativeCallPlan. On return, every
+ * populated ABI argument destination stays live until emit_call. */
void (*plan_call)(NativeTarget*, const NativeCallDesc*, NativeCallPlan*);
void (*emit_call)(NativeTarget*, const NativeCallPlan*);
- /* `value` is the single returned local's location, or NULL for a void
- * return. out_rets/out_nrets describe the ABI parts of that one value. */
+ /* The corresponding mutating return-marshalling phase. `value` is the
+ * single returned local's location, or NULL for void; the hook may emit an
+ * indirect-result copy and returns the ordered register moves in
+ * out_rets/out_nrets. Completed destinations remain live through the phase. */
void (*plan_ret)(NativeTarget*, const CGFuncDesc*, const NativeLoc* value,
NativeCallPlanRet** out_rets, u32* out_nrets);
void (*ret)(NativeTarget*);
+ /* Like load(), a one-register atomic load must consume a plain BASE_REG
+ * address before writing dst and accept those registers aliasing. Backends
+ * already materialize any stronger atomic addressing constraint into their
+ * private scratch before the value load. */
void (*atomic_load)(NativeTarget*, NativeLoc dst, NativeAddr addr, MemAccess,
KitCgMemOrder);
void (*atomic_store)(NativeTarget*, NativeAddr addr, NativeLoc src, MemAccess,
@@ -730,6 +794,147 @@ static inline NativeLoc native_loc_stack(KitCgTypeId type, NativeFrameSlot slot,
.v.stack = {.slot = slot, .offset = offset}};
}
+/* A NativeLoc that structurally contains an address has one of two disjoint
+ * semantic roles. Storage locations are dereferenced to obtain a value;
+ * address recipes are themselves scalar pointer values. Keep the role in the
+ * shared contract so ABI marshallers cannot accidentally turn `&slot` into a
+ * load from slot (or turn a stored scalar into its address). */
+typedef enum NativeLocAddrRole {
+ NATIVE_LOC_ADDR_ROLE_NONE,
+ NATIVE_LOC_ADDR_ROLE_STORAGE,
+ NATIVE_LOC_ADDR_ROLE_VALUE,
+} NativeLocAddrRole;
+
+static inline NativeLocAddrRole native_loc_addr_role(NativeLoc loc) {
+ switch ((NativeLocKind)loc.kind) {
+ case NATIVE_LOC_FRAME:
+ case NATIVE_LOC_STACK:
+ case NATIVE_LOC_ADDR:
+ return NATIVE_LOC_ADDR_ROLE_STORAGE;
+ case NATIVE_LOC_GLOBAL:
+ case NATIVE_LOC_FRAME_ADDR:
+ return NATIVE_LOC_ADDR_ROLE_VALUE;
+ default:
+ return NATIVE_LOC_ADDR_ROLE_NONE;
+ }
+}
+
+/* Resolve storage that may be dereferenced. `offset` selects bytes within the
+ * stored object (including an existing STACK/ADDR displacement). */
+static inline int native_loc_storage_addr(NativeLoc loc, i32 offset,
+ NativeAddr* out) {
+ NativeAddr addr = {0};
+ if (!out || native_loc_addr_role(loc) != NATIVE_LOC_ADDR_ROLE_STORAGE)
+ return 0;
+ switch ((NativeLocKind)loc.kind) {
+ case NATIVE_LOC_FRAME:
+ addr.base_kind = NATIVE_ADDR_BASE_FRAME;
+ addr.base.frame = loc.v.frame;
+ addr.base_type = loc.type;
+ break;
+ case NATIVE_LOC_STACK:
+ addr.base_kind = NATIVE_ADDR_BASE_FRAME;
+ addr.base.frame = loc.v.stack.slot;
+ addr.base_type = loc.type;
+ addr.offset = loc.v.stack.offset;
+ break;
+ case NATIVE_LOC_ADDR:
+ addr = loc.v.addr;
+ break;
+ default:
+ return 0;
+ }
+ addr.offset += offset;
+ *out = addr;
+ return 1;
+}
+
+/* Resolve a scalar address recipe. There is intentionally no part offset:
+ * GLOBAL/FRAME_ADDR are one pointer value, not byte-addressable aggregate
+ * storage. A call marshaller that tries to split one must reject the shape. */
+static inline int native_loc_address_value(NativeLoc loc, NativeAddr* out) {
+ NativeAddr addr = {0};
+ if (!out || native_loc_addr_role(loc) != NATIVE_LOC_ADDR_ROLE_VALUE)
+ return 0;
+ switch ((NativeLocKind)loc.kind) {
+ case NATIVE_LOC_FRAME_ADDR:
+ addr.base_kind = NATIVE_ADDR_BASE_FRAME;
+ addr.base.frame = loc.v.frame;
+ addr.base_type = loc.type;
+ break;
+ case NATIVE_LOC_GLOBAL:
+ addr.base_kind = NATIVE_ADDR_BASE_GLOBAL;
+ addr.base.global.sym = loc.v.global.sym;
+ addr.base.global.addend = loc.v.global.addend;
+ addr.base_type = loc.type;
+ break;
+ default:
+ return 0;
+ }
+ *out = addr;
+ return 1;
+}
+
+/* Iterate the exact bytes carried by one ABI part as native load/store widths.
+ * ABIArgPart.size is a semantic byte count, not the rounded size of its
+ * register or stack carrier: a six-byte aggregate part must transfer 4+2
+ * bytes, never one 8-byte access that can cross the source/destination object.
+ *
+ * Chunks are returned from low to high byte offsets, greedily choosing the
+ * largest power-of-two width no greater than max_chunk. All native targets are
+ * little-endian today, so the byte offset is also the register shift in bytes
+ * when packing/unpacking a part. Physical ABI stack-slot sizing remains the
+ * backend's separate responsibility. */
+typedef struct NativePartChunkIter {
+ u32 total;
+ u32 offset;
+ u32 max_chunk;
+} NativePartChunkIter;
+
+static inline NativePartChunkIter native_part_chunks(u32 total,
+ u32 max_chunk) {
+ NativePartChunkIter it;
+ it.total = total;
+ it.offset = 0;
+ it.max_chunk = max_chunk;
+ return it;
+}
+
+static inline int native_part_chunk_next(NativePartChunkIter* it,
+ u32* offset_out, u32* size_out) {
+ u32 chunk = 1u;
+ u32 remaining;
+ if (!it || !offset_out || !size_out || it->offset >= it->total ||
+ it->max_chunk == 0u)
+ return 0;
+ remaining = it->total - it->offset;
+ while (chunk <= remaining / 2u && chunk <= it->max_chunk / 2u) chunk <<= 1;
+ *offset_out = it->offset;
+ *size_out = chunk;
+ it->offset += chunk;
+ return 1;
+}
+
+/* Select the raw byte lane used for one ABI part of an immediate scalar. The
+ * complete value is only i64-wide, so bounds are explicit. Offset zero keeps
+ * the producer's signed numeric value (preserving established narrow-scalar
+ * extension behavior); later parts are bit slices and are zero-extended. */
+static inline int native_loc_imm_part(NativeLoc loc, u32 offset, u32 size,
+ i64* out) {
+ u64 bits, mask;
+ if (!out || loc.kind != NATIVE_LOC_IMM || size == 0u || size > 8u ||
+ offset >= 8u || size > 8u - offset)
+ return 0;
+ if (offset == 0u) {
+ *out = loc.v.imm;
+ return 1;
+ }
+ bits = (u64)loc.v.imm >> (offset * 8u);
+ mask = size == 8u ? ~(u64)0 : (((u64)1u << (size * 8u)) - 1u);
+ *out = (i64)(bits & mask);
+ return 1;
+}
+
static inline int native_loc_is_fp(NativeLoc loc) {
return (NativeAllocClass)loc.cls == NATIVE_REG_FP;
}
diff --git a/src/arch/riscv/native.c b/src/arch/riscv/native.c
@@ -36,12 +36,12 @@
#include "obj/obj.h"
enum {
- RV_TMP0 = 5u, /* t0: emit-internal scratch (reserved, never allocable) */
- RV_TMP1 = 6u, /* t1: emit-internal scratch */
- RV_TMP2 = 7u, /* t2: emit-internal scratch (reserved in phys table) */
- RV_TMP3 = 28u, /* t3: emit-internal scratch (reserved in phys table) */
- RV_FTMP0 = 0u, /* ft0: emit-internal FP scratch */
- RV_FTMP1 = 1u, /* ft1: emit-internal FP scratch */
+ RV_TMP0 = 5u, /* t0: backend-private temp (reserved, never allocable) */
+ RV_TMP1 = 6u, /* t1: backend-private temp */
+ RV_TMP2 = 7u, /* t2: backend/asm temp (reserved in phys table) */
+ RV_TMP3 = 28u, /* t3: backend/asm temp (reserved in phys table) */
+ RV_FTMP0 = 0u, /* ft0: backend/asm FP temp */
+ RV_FTMP1 = 1u, /* ft1: backend/asm FP temp */
RV_FA0 = 10u, /* fa0..fa7 = f10..f17 (FP arg/return registers) */
RV_FA7 = 17u,
/* Single-pass (-O0) tcc-style prologue. The frame-independent entry
@@ -369,6 +369,39 @@ static u32 loc_size32(NativeTarget* t, NativeLoc loc) {
return native_type_size(t, loc.type);
}
+static u32 rv_internal_tmp(RvNativeTarget* a, u32 avoid_mask) {
+ static const u8 temps[] = {RV_TMP1, RV_TMP0, RV_TMP2, RV_TMP3};
+ for (u32 i = 0; i < sizeof temps / sizeof temps[0]; ++i)
+ if (!(avoid_mask & (1u << temps[i]))) return temps[i];
+ rv_panic(a, "no backend temporary for frame component");
+}
+
+/* Load a typed scalar value from a frame slot. Address components retain their
+ * own type because spill-slot coloring may reuse a wider descriptor; loading
+ * the descriptor width would consume neighboring bytes. Far s0 offsets are
+ * materialized through a distinct backend-owned register. */
+static void rv_load_frame_component(RvNativeTarget* a, u32 dst,
+ NativeFrameSlot slot, KitCgTypeId type,
+ u32 avoid_mask) {
+ RvNativeSlot* s = rv_slot_get(a, slot);
+ i32 off = rv_s0_off_slot(s);
+ u32 size;
+ if (!type) rv_panic(a, "frame-value component has no exact type");
+ size = native_type_size(&a->base, type);
+ if (fits_i12(off)) {
+ rv64_emit32(a->base.mc,
+ enc_int_load(a->variant, size, 0, dst, RV_S0, off));
+ return;
+ }
+ {
+ u32 tmp = rv_internal_tmp(a, avoid_mask | (1u << dst) | (1u << RV_S0));
+ rv_emit_load_imm(a->variant, a->base.mc, 1, tmp, (i64)off);
+ rv64_emit32(a->base.mc, rv_add(tmp, RV_S0, tmp));
+ rv64_emit32(a->base.mc,
+ enc_int_load(a->variant, size, 0, dst, tmp, 0));
+ }
+}
+
/* The original rv_is_64 predicate exactly, but sourcing the width from the
* cached descriptor: `size >= 8 || (xlen==64 && is_ptr)`. The is_ptr clause
* keeps the rv32 4-byte-pointer distinction (a stamped 4-byte pointer caches
@@ -413,17 +446,18 @@ static u32 loc_reg(NativeLoc loc) { return loc.v.reg & 0x1fu; }
.spill_cost = 0u, \
.copy_cost = 0u}
-/* t0..t3 (x5,x6,x7,x28) are emit-internal scratch (RV_TMP0..RV_TMP3), reserved
- * and never handed to the allocator or driver. t4/t5/t6 are the native-emit
- * scratch pool (disjoint from the emit temps so a hook can never clobber an
- * operand parked there). Three are needed, not two: a load/store arch must
- * materialize a scaled-index address into a scratch (collapse_addr_to_reg, e.g.
- * `s->arr[i].field`), and the worst case — a 3-operand binop, or such an access
- * — has three native-emit scratches live at once. aarch64 reserves three for
+/* t0..t3 (x5,x6,x7,x28) are backend-private hook/asm temporaries
+ * (RV_TMP0..RV_TMP3), reserved and never exposed as optimizer operand
+ * locations. t4/t5/t6 form the O1 instruction-scoped operand-temp bank,
+ * disjoint from backend temporaries so a hook cannot overwrite a leased
+ * operand. Three are needed, not two: a load/store target can need an address
+ * materialization alongside a 3-operand operation. aarch64 declares three for
* the same reason; x64 needs only two because it folds the index into its
- * addressing mode.
+ * addressing mode. Location MIR retains spills as frame locations until the
+ * consuming instruction opens this lease scope.
*
- * rv_int_allocable is the NDT (-O0) value-cache / scratch pool and is read only
+ * rv_ndt_int_allocable is the NDT (-O0) value-cache / scratch pool and is read
+ * only
* by NativeDirectTarget; the optimizer allocates over the phys[] ALLOCABLE flags
* (s1..s11, callee-saved, saved by its -O1 prologue) instead. The NDT pool is
* the caller-saved argument registers a0..a7: the -O0 cache flushes to frame
@@ -433,9 +467,10 @@ static u32 loc_reg(NativeLoc loc) { return loc.v.reg & 0x1fu; }
* RV_PROLOGUE_WORDS region) from ever overflowing on large/variadic frames the
* way callee-save offsets would. Incoming args are spilled to homes at entry
* before any body op, so caching in a0..a7 cannot clobber a live parameter. */
-static const Reg rv_int_allocable[] = {10u, 11u, 12u, 13u, 14u,
- 15u, 16u, 17u}; /* a0..a7 */
+static const Reg rv_ndt_int_allocable[] = {10u, 11u, 12u, 13u, 14u,
+ 15u, 16u, 17u}; /* a0..a7 */
static const Reg rv_int_scratch[] = {29u, 30u, 31u}; /* t4, t5, t6 */
+static const Reg rv_int_asm_temps[] = {RV_TMP2, RV_TMP3};
static const NativePhysRegInfo rv_int_phys[] = {
RV_PHYS_INT_RESERVED(0u), /* zero */
@@ -445,7 +480,7 @@ static const NativePhysRegInfo rv_int_phys[] = {
RV_PHYS_INT_RESERVED(4u), /* tp */
RV_PHYS_INT_RESERVED(5u), /* t0 = TMP0 */
RV_PHYS_INT_RESERVED(6u), /* t1 = TMP1 */
- RV_PHYS_INT_RESERVED(7u), /* t2 = TMP2 (emit) */
+ RV_PHYS_INT_RESERVED(7u), /* t2 = TMP2 (backend/asm) */
RV_PHYS_INT_RESERVED(8u), /* s0/fp */
RV_PHYS_INT_CALLEE(9u), /* s1 */
RV_PHYS_INT_ARG(10u, 0u), RV_PHYS_INT_ARG(11u, 1u),
@@ -457,10 +492,10 @@ static const NativePhysRegInfo rv_int_phys[] = {
RV_PHYS_INT_CALLEE(22u), RV_PHYS_INT_CALLEE(23u),
RV_PHYS_INT_CALLEE(24u), RV_PHYS_INT_CALLEE(25u),
RV_PHYS_INT_CALLEE(26u), RV_PHYS_INT_CALLEE(27u),
- RV_PHYS_INT_RESERVED(28u), /* t3 = TMP3 (emit) */
- RV_PHYS_INT_RESERVED(29u), /* t4 = native-emit scratch */
- RV_PHYS_INT_RESERVED(30u), /* t5 = native-emit scratch */
- RV_PHYS_INT_RESERVED(31u), /* t6 = native-emit scratch (3rd) */
+ RV_PHYS_INT_RESERVED(28u), /* t3 = TMP3 (backend/asm) */
+ RV_PHYS_INT_RESERVED(29u), /* t4 = O1 operand temp / O0 scratch */
+ RV_PHYS_INT_RESERVED(30u), /* t5 = O1 operand temp / O0 scratch */
+ RV_PHYS_INT_RESERVED(31u), /* t6 = O1 operand temp / O0 scratch */
};
#define RV_PHYS_FP_ARG(r, idx) \
@@ -502,18 +537,19 @@ static const NativePhysRegInfo rv_int_phys[] = {
* overflowing. fa0..fa7 come FIRST (mirroring the int pool's a0..a7):
* nd_cache_alloc scans the pool in order, so fronting the arg registers lets a
* producer materialize a call's fp args directly into their ABI registers
- * (Lever 1, via api_pack_call_args_in_order). ft0/ft1 reserved as emit-internal
- * scratch; ft2/ft3 driver scratch. */
-static const Reg rv_fp_allocable[] = {10u, 11u, 12u, 13u, 14u, 15u,
- 16u, 17u, 4u, 5u, 6u, 7u,
- 28u, 29u, 30u, 31u}; /* fa0-7,ft4-7,ft8-11 */
+ * (Lever 1, via api_pack_call_args_in_order). ft0/ft1 are backend/asm
+ * temporaries; ft2/ft3 form the O1 operand-temp bank and O0 scratch policy. */
+static const Reg rv_ndt_fp_allocable[] = {
+ 10u, 11u, 12u, 13u, 14u, 15u, 16u, 17u, 4u, 5u, 6u, 7u,
+ 28u, 29u, 30u, 31u}; /* fa0-7,ft4-7,ft8-11 */
static const Reg rv_fp_scratch[] = {2u, 3u}; /* ft2, ft3 */
+static const Reg rv_fp_asm_temps[] = {RV_FTMP0, RV_FTMP1};
static const NativePhysRegInfo rv_fp_phys[] = {
RV_PHYS_FP_RESERVED(0u), /* ft0 = FTMP0 */
RV_PHYS_FP_RESERVED(1u), /* ft1 = FTMP1 */
- RV_PHYS_FP_RESERVED(2u), /* ft2 = scratch */
- RV_PHYS_FP_RESERVED(3u), /* ft3 = scratch */
+ RV_PHYS_FP_RESERVED(2u), /* ft2 = O1 operand temp / O0 scratch */
+ RV_PHYS_FP_RESERVED(3u), /* ft3 = O1 operand temp / O0 scratch */
RV_PHYS_FP_CALLER(4u), RV_PHYS_FP_CALLER(5u), RV_PHYS_FP_CALLER(6u),
RV_PHYS_FP_CALLER(7u), RV_PHYS_FP_CALLEE(8u), RV_PHYS_FP_CALLEE(9u),
RV_PHYS_FP_ARG(10u, 0u), RV_PHYS_FP_ARG(11u, 1u), RV_PHYS_FP_ARG(12u, 2u),
@@ -528,10 +564,16 @@ static const NativePhysRegInfo rv_fp_phys[] = {
static const NativeAllocClassInfo rv_classes[] = {
{.cls = NATIVE_REG_INT,
- .allocable = rv_int_allocable,
- .nallocable = sizeof rv_int_allocable / sizeof rv_int_allocable[0],
+ .ndt_allocable = rv_ndt_int_allocable,
+ .ndt_allocable_count =
+ sizeof rv_ndt_int_allocable / sizeof rv_ndt_int_allocable[0],
.scratch = rv_int_scratch,
.nscratch = sizeof rv_int_scratch / sizeof rv_int_scratch[0],
+ .emit_temps = rv_int_scratch,
+ .nemit_temps = sizeof rv_int_scratch / sizeof rv_int_scratch[0],
+ .asm_temps = rv_int_asm_temps,
+ .nasm_temps = sizeof rv_int_asm_temps / sizeof rv_int_asm_temps[0],
+ .emit_cache_mask = (1u << 29u) | (1u << 30u) | (1u << 31u),
.phys = rv_int_phys,
.nphys = sizeof rv_int_phys / sizeof rv_int_phys[0],
/* t0-t6 (5-7,28-31) + a0-a7 (10-17) */
@@ -540,14 +582,20 @@ static const NativeAllocClassInfo rv_classes[] = {
.callee_saved_mask = 0x0ffc0300u,
.arg_mask = 0x0003fc00u,
.ret_mask = 0x00000c00u,
- /* zero,ra,sp,gp,tp,t0,t1,t2,s0 (bits 0-8) + t3 (bit 28). t4/t5 are the
- * driver scratch pool (reserved-from-alloc but listed in scratch[]). */
+ /* zero,ra,sp,gp,tp,t0,t1,t2,s0 (bits 0-8) + t3 (bit 28). The validated
+ * emit_temps bank adds t4/t5/t6 to the optimizer's reserved mask. */
.reserved_mask = 0x000001ffu | (1u << 28)},
{.cls = NATIVE_REG_FP,
- .allocable = rv_fp_allocable,
- .nallocable = sizeof rv_fp_allocable / sizeof rv_fp_allocable[0],
+ .ndt_allocable = rv_ndt_fp_allocable,
+ .ndt_allocable_count =
+ sizeof rv_ndt_fp_allocable / sizeof rv_ndt_fp_allocable[0],
.scratch = rv_fp_scratch,
.nscratch = sizeof rv_fp_scratch / sizeof rv_fp_scratch[0],
+ .emit_temps = rv_fp_scratch,
+ .nemit_temps = sizeof rv_fp_scratch / sizeof rv_fp_scratch[0],
+ .asm_temps = rv_fp_asm_temps,
+ .nasm_temps = sizeof rv_fp_asm_temps / sizeof rv_fp_asm_temps[0],
+ .emit_cache_mask = (1u << 2u) | (1u << 3u),
.phys = rv_fp_phys,
.nphys = sizeof rv_fp_phys / sizeof rv_fp_phys[0],
/* ft0-ft7 (0-7), fa0-fa7 (10-17), ft8-ft11 (28-31) */
@@ -764,7 +812,8 @@ static u32 rv_fold_index(RvNativeTarget* a, u32 base, u32 idx, u8 log2_scale) {
/* Resolve any NativeAddr to a base register + imm12 offset. RISC-V has no
* indexed load/store, so an index is folded into RV_TMP0 via Zba; far offsets
- * and FRAME/FRAME_VALUE/GLOBAL bases are materialized into RV_TMP0/RV_TMP1. */
+ * and GLOBAL or exact-typed FRAME_VALUE components are materialized in the
+ * backend-private RV_TMP0/RV_TMP1 bank. Plain FRAME bases remain s0-relative. */
static void rv_resolve_mem_addr(RvNativeTarget* a, const NativeAddr* addr,
u32* base_out, i32* off_out) {
MCEmitter* mc = a->base.mc;
@@ -782,8 +831,8 @@ static void rv_resolve_mem_addr(RvNativeTarget* a, const NativeAddr* addr,
break;
}
case NATIVE_ADDR_BASE_FRAME_VALUE: {
- RvNativeSlot* s = rv_slot_get(a, addr->base.frame);
- rv64_emit32(mc, rv_ld_ptr(a->variant, RV_TMP0, RV_S0, rv_s0_off_slot(s)));
+ rv_load_frame_component(a, RV_TMP0, addr->base.frame, addr->base_type,
+ 0u);
base = RV_TMP0;
off = addr->offset;
break;
@@ -800,13 +849,15 @@ static void rv_resolve_mem_addr(RvNativeTarget* a, const NativeAddr* addr,
if (addr->index_kind == NATIVE_ADDR_INDEX_REG) {
base = rv_fold_index(a, base, addr->index.reg & 0x1fu, addr->log2_scale);
} else if (addr->index_kind == NATIVE_ADDR_INDEX_FRAME_VALUE) {
- RvNativeSlot* s = rv_slot_get(a, addr->index.frame);
- rv64_emit32(mc, rv_ld_ptr(a->variant, RV_TMP1, RV_S0, rv_s0_off_slot(s)));
- base = rv_fold_index(a, base, RV_TMP1, addr->log2_scale);
+ u32 idx = rv_internal_tmp(a, 1u << base);
+ rv_load_frame_component(a, idx, addr->index.frame, addr->index_type,
+ 1u << base);
+ base = rv_fold_index(a, base, idx, addr->log2_scale);
}
if (!fits_i12(off)) {
- rv_emit_load_imm(a->variant, mc, 1, RV_TMP1, (i64)off);
- rv64_emit32(mc, rv_add(RV_TMP0, base, RV_TMP1));
+ u32 tmp = rv_internal_tmp(a, 1u << base);
+ rv_emit_load_imm(a->variant, mc, 1, tmp, (i64)off);
+ rv64_emit32(mc, rv_add(RV_TMP0, base, tmp));
base = RV_TMP0;
off = 0;
}
@@ -814,6 +865,105 @@ static void rv_resolve_mem_addr(RvNativeTarget* a, const NativeAddr* addr,
*off_out = off;
}
+/* Choose a backend-private integer register for an exact-width ABI transfer.
+ * TMP0/TMP1 are considered last because address resolution normally owns that
+ * pair; callers that choose before resolution therefore exclude them by
+ * passing them as explicit avoids. */
+static u32 rv_part_tmp(RvNativeTarget* a, u32 avoid_a, u32 avoid_b,
+ u32 avoid_c) {
+ static const u8 temps[] = {RV_TMP2, RV_TMP3, RV_TMP1, RV_TMP0};
+ u32 i;
+ for (i = 0; i < (u32)(sizeof temps / sizeof temps[0]); ++i) {
+ u32 r = temps[i];
+ if (r != avoid_a && r != avoid_b && r != avoid_c) return r;
+ }
+ rv_panic(a, "no temporary for exact-width ABI part");
+ return RV_TMP2;
+}
+
+static u32 rv_addr_reg(const NativeAddr* addr, int index) {
+ if (!index && addr->base_kind == NATIVE_ADDR_BASE_REG)
+ return addr->base.reg & 0x1fu;
+ if (index && addr->index_kind == NATIVE_ADDR_INDEX_REG)
+ return addr->index.reg & 0x1fu;
+ return REG_NONE;
+}
+
+/* TMP2/TMP3 are not borrowed by rv_resolve_mem_addr, so a packed store may
+ * retain its source in either one while TMP0/TMP1 materialize the address. */
+static u32 rv_part_preserved_tmp(RvNativeTarget* a, const NativeAddr* addr) {
+ u32 base = rv_addr_reg(addr, 0);
+ u32 index = rv_addr_reg(addr, 1);
+ if (RV_TMP2 != base && RV_TMP2 != index) return RV_TMP2;
+ if (RV_TMP3 != base && RV_TMP3 != index) return RV_TMP3;
+ rv_panic(a, "exact-width ABI address consumes transfer temporaries");
+ return RV_TMP2;
+}
+
+/* RISC-V has no 3/5/6/7-byte scalar load/store. ABI parts of those semantic
+ * sizes still occupy one packed integer register, so move them with bounded
+ * low-to-high 4/2/1 (or RV32 2/1) accesses and explicitly pack/unpack the
+ * register bits. This keeps every memory access within the semantic object;
+ * the separately rounded ABI carrier/stack-slot size is not an access width. */
+static void rv_emit_exact_int_part(RvNativeTarget* a, int is_load, u32 reg,
+ NativeAddr addr, u32 size) {
+ const RiscvVariant* v = a->variant;
+ MCEmitter* mc = a->base.mc;
+ NativePartChunkIter chunks = native_part_chunks(size, v->ptr_bytes);
+ u32 base;
+ i32 off;
+ u32 chunk_off, chunk_size;
+
+ if (!is_load) {
+ /* Preserve the packed source before address resolution, which may borrow
+ * TMP0/TMP1. Avoid any explicit address components as well: a plain
+ * BASE_REG address must remain intact until it has been resolved. */
+ u32 value = rv_part_preserved_tmp(a, &addr);
+ u32 consumed = 0u;
+ if (value != reg) rv64_emit32(mc, rv_addi(value, reg, 0));
+ rv_resolve_mem_addr(a, &addr, &base, &off);
+ if (!fits_i12((i64)off + (i64)size - 1)) {
+ u32 stable = rv_part_tmp(a, value, base, REG_NONE);
+ rv_emit_addr_adjust(v, mc, stable, base, off);
+ base = stable;
+ off = 0;
+ }
+ while (native_part_chunk_next(&chunks, &chunk_off, &chunk_size)) {
+ if (chunk_off != consumed) {
+ rv64_emit32(mc, rv_srli(value, value,
+ (chunk_off - consumed) * 8u));
+ consumed = chunk_off;
+ }
+ rv64_emit32(mc,
+ enc_int_store(v, chunk_size, value, base,
+ off + (i32)chunk_off));
+ }
+ return;
+ }
+
+ rv_resolve_mem_addr(a, &addr, &base, &off);
+ if (base == reg || !fits_i12((i64)off + (i64)size - 1)) {
+ u32 stable = rv_part_tmp(a, reg, base, REG_NONE);
+ rv_emit_addr_adjust(v, mc, stable, base, off);
+ base = stable;
+ off = 0;
+ }
+ {
+ u32 piece = rv_part_tmp(a, reg, base, REG_NONE);
+ int first = 1;
+ while (native_part_chunk_next(&chunks, &chunk_off, &chunk_size)) {
+ u32 dst = first ? reg : piece;
+ rv64_emit32(mc, enc_int_load(v, chunk_size, 0, dst, base,
+ off + (i32)chunk_off));
+ if (!first) {
+ rv64_emit32(mc, rv_slli(piece, piece, chunk_off * 8u));
+ rv64_emit32(mc, rv_or(reg, reg, piece));
+ }
+ first = 0;
+ }
+ }
+}
+
/* Central load/store primitive. is_load: 1 load into reg, 0 store reg to mem.
*/
static void rv_emit_mem(RvNativeTarget* a, int is_load, NativeLoc reg,
@@ -826,6 +976,12 @@ static void rv_emit_mem(RvNativeTarget* a, int is_load, NativeLoc reg,
u32 base;
i32 off;
+ if (!fp && sz != 0u && sz <= a->variant->ptr_bytes &&
+ (sz & (sz - 1u)) != 0u) {
+ rv_emit_exact_int_part(a, is_load, r, addr, sz);
+ return;
+ }
+
rv_resolve_mem_addr(a, &addr, &base, &off);
if (fp) {
rv64_emit32(
@@ -900,8 +1056,7 @@ static void rv_load_addr(NativeTarget* t, NativeLoc dst, NativeAddr addr) {
off = 0;
} else if (addr.base_kind == NATIVE_ADDR_BASE_FRAME_VALUE) {
/* Load the pointer stored in the frame slot, then add the offset. */
- RvNativeSlot* s = rv_slot_get(a, addr.base.frame);
- rv64_emit32(mc, rv_ld_ptr(a->variant, rd, RV_S0, rv_s0_off_slot(s)));
+ rv_load_frame_component(a, rd, addr.base.frame, addr.base_type, 0u);
base = rd;
off = addr.offset;
} else if (addr.base_kind == NATIVE_ADDR_BASE_FRAME) {
@@ -914,11 +1069,21 @@ static void rv_load_addr(NativeTarget* t, NativeLoc dst, NativeAddr addr) {
} else {
rv_panic(a, "unsupported address base in load_addr");
}
- /* Fold any index via Zba sh{1,2,3}add (index << scale) + base. */
- if (addr.index_kind == NATIVE_ADDR_INDEX_REG) {
- u32 idx = addr.index.reg & 0x1fu;
+ /* Fold any index via Zba sh{1,2,3}add (index << scale) + base. Resolve the
+ * byte offset before loading a frame index so the far-offset helper's
+ * scratch cannot overwrite that index. */
+ if (addr.index_kind == NATIVE_ADDR_INDEX_REG ||
+ addr.index_kind == NATIVE_ADDR_INDEX_FRAME_VALUE) {
+ u32 idx;
if (off != 0 || base != rd)
rv_emit_addr_adjust(a->variant, mc, rd, base, off);
+ if (addr.index_kind == NATIVE_ADDR_INDEX_REG) {
+ idx = addr.index.reg & 0x1fu;
+ } else {
+ idx = rv_internal_tmp(a, 1u << rd);
+ rv_load_frame_component(a, idx, addr.index.frame, addr.index_type,
+ 1u << rd);
+ }
switch (addr.log2_scale) {
case 0:
rv64_emit32(mc, rv_add(rd, rd, idx));
@@ -2224,30 +2389,12 @@ static u32 rv_call_stack_bytes(NativeTarget* t, const NativeCallDesc* desc) {
return rv_call_stack_size(t, desc);
}
-/* Resolve a NativeLoc to an addressable NativeAddr (frame/stack/addr). */
-static NativeAddr rv_loc_addr(RvNativeTarget* a, NativeLoc loc, u32 offset) {
+/* Resolve a dereferenceable NativeLoc to its storage address. */
+static NativeAddr rv_storage_addr(RvNativeTarget* a, NativeLoc loc,
+ u32 offset) {
NativeAddr addr;
- memset(&addr, 0, sizeof addr);
- switch ((NativeLocKind)loc.kind) {
- case NATIVE_LOC_FRAME:
- addr.base_kind = NATIVE_ADDR_BASE_FRAME;
- addr.base.frame = loc.v.frame;
- addr.base_type = loc.type;
- addr.offset = (i32)offset;
- return addr;
- case NATIVE_LOC_STACK:
- addr.base_kind = NATIVE_ADDR_BASE_FRAME;
- addr.base.frame = loc.v.stack.slot;
- addr.base_type = loc.type;
- addr.offset = loc.v.stack.offset + (i32)offset;
- return addr;
- case NATIVE_LOC_ADDR:
- addr = loc.v.addr;
- addr.offset += (i32)offset;
- return addr;
- default:
- rv_panic(a, "location is not addressable");
- }
+ if (!native_loc_storage_addr(loc, (i32)offset, &addr))
+ rv_panic(a, "location is not storage");
return addr;
}
@@ -2258,16 +2405,26 @@ static void rv_load_part(NativeTarget* t, NativeLoc dst, NativeLoc src,
rv_move(t, dst, src);
return;
}
- if (src.kind == NATIVE_LOC_FRAME || src.kind == NATIVE_LOC_STACK ||
- src.kind == NATIVE_LOC_ADDR) {
- NativeAddr addr = rv_loc_addr(a, src, offset);
+ if (native_loc_addr_role(src) == NATIVE_LOC_ADDR_ROLE_VALUE) {
+ NativeAddr addr;
+ if (offset != 0u || size != t->c->target.ptr_size ||
+ !native_loc_address_value(src, &addr))
+ rv_panic(a, "split or invalid address-value argument");
+ rv_load_addr(t, dst, addr);
+ return;
+ }
+ if (native_loc_addr_role(src) == NATIVE_LOC_ADDR_ROLE_STORAGE) {
+ NativeAddr addr = rv_storage_addr(a, src, offset);
addr.base_type = dst.type;
rv_emit_mem(a, 1, dst, addr, native_mem_for_type(t, dst.type, size));
return;
}
if (src.kind == NATIVE_LOC_IMM) {
+ i64 part;
+ if (!native_loc_imm_part(src, offset, size, &part))
+ rv_panic(a, "invalid immediate argument part");
rv_emit_load_imm(a->variant, t->mc, loc_is_64(t, dst) ? 1u : 0u,
- loc_reg(dst), src.v.imm);
+ loc_reg(dst), part);
return;
}
rv_panic(a, "unsupported part source");
@@ -2276,9 +2433,8 @@ static void rv_load_part(NativeTarget* t, NativeLoc dst, NativeLoc src,
static void rv_store_part(NativeTarget* t, NativeLoc dst, NativeLoc src,
u32 offset, u32 size) {
RvNativeTarget* a = rv_of(t);
- if (dst.kind == NATIVE_LOC_FRAME || dst.kind == NATIVE_LOC_STACK ||
- dst.kind == NATIVE_LOC_ADDR) {
- NativeAddr addr = rv_loc_addr(a, dst, offset);
+ if (native_loc_addr_role(dst) == NATIVE_LOC_ADDR_ROLE_STORAGE) {
+ NativeAddr addr = rv_storage_addr(a, dst, offset);
addr.base_type = src.type;
rv_emit_mem(a, 0, src, addr, native_mem_for_type(t, src.type, size));
return;
@@ -2291,7 +2447,7 @@ static void rv_store_part(NativeTarget* t, NativeLoc dst, NativeLoc src,
}
static void rv_addr_of_loc(NativeTarget* t, NativeLoc dst, NativeLoc src) {
- NativeAddr addr = rv_loc_addr(rv_of(t), src, 0);
+ NativeAddr addr = rv_storage_addr(rv_of(t), src, 0);
rv_load_addr(t, dst, addr);
}
@@ -2315,6 +2471,50 @@ static void rv_store_outgoing_part(NativeTarget* t, int tail_call,
rv_emit_mem(rv_of(t), 0, src, addr, native_mem_for_type(t, src.type, size));
}
+/* Copy exactly the semantic bytes of a stack-routed value. The caller owns
+ * the separately rounded physical ABI slot and advances its stack cursor by
+ * that carrier size; padding is never treated as readable source storage. */
+static void rv_store_outgoing_value_bytes(NativeTarget* t, int tail_call,
+ u32 stack_off, NativeLoc src,
+ u32 value_size) {
+ RvNativeTarget* a = rv_of(t);
+ NativeLoc tmp = native_loc_reg(src.type, NATIVE_REG_INT, RV_TMP0);
+ u32 off = 0;
+ while (off < value_size) {
+ u32 chunk = value_size - off;
+ if (chunk > a->variant->ptr_bytes) chunk = a->variant->ptr_bytes;
+ rv_load_part(t, tmp, src, off, chunk);
+ rv_store_outgoing_part(t, tail_call, stack_off + off, tmp, chunk);
+ off += chunk;
+ }
+}
+
+/* Give every indirect target one lifetime that is disjoint from argument
+ * transport. `ra` is reserved from allocation and from the t0-t3 backend
+ * temporary bank, while its incoming value is already saved in the function
+ * frame at any real call site. A normal JALR may read and write ra in the same
+ * instruction. A tail site copies this parked value to t1 immediately before
+ * teardown restores the incoming ra.
+ *
+ * Lowering normally presents REG/GLOBAL, but NativeCallDesc permits scalar
+ * address values and dereferenceable storage too. Route every non-global shape
+ * through the same checked part boundary so FRAME/STACK/ADDR/FRAME_ADDR/IMM
+ * remain valid without acquiring an optimizer-owned register. NONE is kept for
+ * planning-only queries that never emit a call. */
+static NativeLoc rv_stage_indirect_callee(NativeTarget* t, NativeLoc callee) {
+ RvNativeTarget* a = rv_of(t);
+ NativeLoc parked;
+ if (callee.kind == NATIVE_LOC_NONE || callee.kind == NATIVE_LOC_GLOBAL)
+ return callee;
+ if (callee.kind == NATIVE_LOC_REG &&
+ (NativeAllocClass)callee.cls != NATIVE_REG_INT)
+ rv_panic(a, "indirect callee is not in an integer register");
+ if (!callee.type) rv_panic(a, "indirect callee has no pointer type");
+ parked = native_loc_reg(callee.type, NATIVE_REG_INT, RV_RA);
+ rv_load_part(t, parked, callee, 0, a->variant->ptr_bytes);
+ return parked;
+}
+
/* NativeTarget bind_param: route incoming param (ABI loc) into dst. */
static void rv_bind_native_param(NativeTarget* t, const CGParamDesc* p,
NativeLoc dst) {
@@ -2413,7 +2613,7 @@ static void rv_emit_one_arg_move(NativeTarget* t, const NativeArgMove* m) {
}
/* Parallel-copy register arg moves via the shared scheduler; cycles break
- * through the int/fp emit scratch (t1 / ft1). */
+ * through backend-private t1/ft1. */
static void rv_emit_reg_arg_moves(NativeTarget* t, NativeArgMove* moves,
u32 n) {
NativeArgShuffle s;
@@ -2449,15 +2649,7 @@ static void rv_plan_call(NativeTarget* t, const NativeCallDesc* desc,
plan->stack_arg_size = rv_call_stack_size(t, desc);
if (plan->stack_arg_size > a->frame.max_outgoing)
a->frame.max_outgoing = plan->stack_arg_size;
- /* Indirect callee in an arg register would be clobbered by arg loads. */
- if (plan->callee.kind == NATIVE_LOC_REG &&
- (NativeAllocClass)plan->callee.cls == NATIVE_REG_INT &&
- plan->callee.v.reg >= RV_A0 && plan->callee.v.reg <= RV_A7) {
- NativeLoc scratch =
- native_loc_reg(plan->callee.type, NATIVE_REG_INT, RV_TMP0);
- rv_move(t, scratch, plan->callee);
- plan->callee = scratch;
- }
+ plan->callee = rv_stage_indirect_callee(t, plan->callee);
{
/* sret returns pass the hidden destination pointer as the implicit first
* integer argument (a0), so the real args start at a1. */
@@ -2472,15 +2664,10 @@ static void rv_plan_call(NativeTarget* t, const NativeCallDesc* desc,
abi && abi->variadic && abi->vararg_on_stack && i >= abi->nparams;
if (ai->kind == ABI_ARG_IGNORE) continue;
if (force_stack) {
- NativeLoc tmpreg =
- native_loc_reg(desc->args[i].type, NATIVE_REG_INT, RV_TMP0);
- u32 slot = a->variant->gp_slot_bytes; /* xlen-word: 8 lp64d / 4 ilp32 */
- u32 n = rv_class_stack_size(a->variant, ai), off = 0;
- while (off < n) {
- rv_load_part(t, tmpreg, desc->args[i], off, slot);
- rv_store_outgoing_part(t, tail, stack + off, tmpreg, slot);
- off += slot;
- }
+ u32 value_size = native_type_size(t, desc->args[i].type);
+ u32 n = rv_class_stack_size(a->variant, ai);
+ rv_store_outgoing_value_bytes(t, tail, stack, desc->args[i],
+ value_size);
stack += n;
continue;
}
@@ -2585,11 +2772,10 @@ static void rv_emit_tail_site(NativeTarget* t, NativeLoc callee) {
u32 n_int = rv_collect_int_saves(a, int_regs);
u32 n_fp = rv_collect_fp_saves(a, fp_regs);
i32 i;
- /* Stage an indirect callee into a reserved scratch (t1) BEFORE the teardown:
- * regalloc parks the function pointer in a callee-saved register so it
- * survives arg marshalling, and the callee-save / s0 / ra restores below
- * would otherwise overwrite it. t1 is reserved (never allocable) and
- * untouched by the restore loop (which only uses t0 for far offsets). */
+ /* Stage the target parked in ra into a reserved scratch BEFORE teardown: the
+ * callee-save / s0 / ra restores below would otherwise overwrite it. t1 is
+ * reserved (never allocable) and untouched by the restore loop (which only
+ * uses t0 for far offsets). */
if (indirect) rv64_emit32(mc, rv_addi(RV_TMP1, loc_reg(callee), 0));
/* Restore callee-saves before tearing the frame down (O1 path; none at -O0).
* Their save offsets are s0-relative via rv_save_off, so the restore is
@@ -2655,7 +2841,7 @@ static void rv_plan_ret(NativeTarget* t, const CGFuncDesc* fd,
dst_addr.base_kind = NATIVE_ADDR_BASE_REG;
dst_addr.base.reg = RV_TMP1;
dst_addr.base_type = value->type;
- src_addr = rv_loc_addr(a, *value, 0);
+ src_addr = rv_storage_addr(a, *value, 0);
src_addr.base_type = value->type;
memset(&access, 0, sizeof access);
access.type = value->type;
@@ -2674,18 +2860,16 @@ static void rv_plan_ret(NativeTarget* t, const CGFuncDesc* fd,
part->cls == ABI_CLASS_FP ? NATIVE_REG_FP : NATIVE_REG_INT;
KitCgTypeId pty = rv_part_scalar_type(part);
Reg rreg = cls == NATIVE_REG_FP ? RV_FA0 + nf++ : RV_A0 + ni++;
- rets[nr].src = *value;
- if (rets[nr].src.kind == NATIVE_LOC_FRAME)
- rets[nr].src =
- native_loc_stack(pty, value->v.frame, (i32)part->src_offset);
- else if (rets[nr].src.kind == NATIVE_LOC_STACK) {
- rets[nr].src.v.stack.offset += (i32)part->src_offset;
- rets[nr].src.type = pty;
- }
- rets[nr].dst = native_loc_reg(pty, cls, rreg);
- rets[nr].mem = native_mem_for_type(t, pty, part->size);
- nr++;
+ NativeLoc dst = native_loc_reg(pty, cls, rreg);
+ /* Returning a memory-homed aggregate through the generic move phase
+ * loses ABIArgPart.size while materializing its scalar carrier type (a
+ * six-byte part becomes an eight-byte load). Marshal here through the
+ * same exact-width boundary as outgoing arguments instead. */
+ rv_load_part(t, dst, *value, part->src_offset, part->size);
}
+ *out_rets = NULL;
+ *out_nrets = 0;
+ return;
} else if (value) {
rets[0].src = *value;
rets[0].dst = native_loc_reg(value->type, NATIVE_REG_INT, RV_A0);
@@ -3093,7 +3277,7 @@ static void rv_va_arg_native(NativeTarget* t, NativeLoc dst, NativeLoc ap_ptr,
KitCgTypeId type) {
RvNativeTarget* a = rv_of(t);
if (rv_va_arg_is_wide(t, type)) {
- rv_va_arg_wide(a, rv_loc_addr(a, dst, 0), rv_va_addr_from_ptr(ap_ptr),
+ rv_va_arg_wide(a, rv_storage_addr(a, dst, 0), rv_va_addr_from_ptr(ap_ptr),
native_type_size(t, type));
return;
}
@@ -3554,6 +3738,27 @@ static void rv_intrinsic(NativeTarget* t, IntrinKind kind,
}
rv_panic(a, "unsupported compiler intrinsic");
}
+
+/* Optimized-hook fixed effects. RISC-V arithmetic, bitfields, aggregate
+ * memory operations, atomics, varargs and Local-Exec TLS use only declared
+ * operands/results plus the reserved t0..t3 / ft0..ft3 backend temporaries.
+ * A syscall is the one exception: argument placement writes the Linux syscall
+ * ABI registers even though the semantic IR operands are ordinary virtual
+ * values. Keep live-across allocations out of those fixed destinations. */
+static int rv_machine_op_clobbers(NativeTarget* t, const NativeMachineOp* op,
+ u32 mask[NATIVE_CALL_PLAN_CLASSES]) {
+ (void)t;
+ mask[0] = mask[1] = mask[2] = 0;
+ if ((NativeMachineOpKind)op->kind != NATIVE_MOP_INTRINSIC ||
+ (IntrinKind)op->intrin != INTRIN_SYSCALL)
+ return 0;
+ /* a7 = syscall number; a0..a5 = up to six arguments / a0 result. */
+ mask[NATIVE_REG_INT] = (1u << RV_A0) | (1u << RV_A1) | (1u << RV_A2) |
+ (1u << RV_A3) | (1u << RV_A4) | (1u << RV_A5) |
+ (1u << RV_A7);
+ return 1;
+}
+
/* ============================ inline asm ============================ */
_Noreturn static void rv_asm_panic_at(Compiler* c, SrcLoc loc,
@@ -3638,7 +3843,7 @@ static void rv_asm_clobber_masks(Compiler* c, SrcLoc loc, const Sym* clobbers,
* asm operand the direct path must self-allocate. */
static Reg rv_asm_alloc_reg(NativeDirectTarget* d, NativeAllocClass cls,
u32 allowed_mask, u32* used_int, u32* used_fp) {
- /* int: a0..a7 (10..17) then t-temps that aren't emit scratch. */
+ /* int: a0..a7 (10..17) then O0-owned t4..t6. */
static const Reg int_pool[] = {10u, 11u, 12u, 13u, 14u, 15u,
16u, 17u, 29u, 30u, 31u};
/* fp: fa0..fa7 (10..17) then ft caller-saved. */
@@ -3835,10 +4040,9 @@ static RvAsmSavedClobber* rv_asm_save_callee_clobbers(RvNativeTarget* a,
}
/* ---- NativeTarget (optimizer) asm hook ----
- * The optimizer pre-allocated every operand register and arranged surrounding
- * data flow, so this binds pre-allocated registers to the template and only
- * materializes memory-operand bases into the reserved scratch + spills the
- * callee-saved registers the asm clobbers. */
+ * The optimized emitter owns register placement, staging, and output
+ * writeback. This hook binds its concrete locations and only materializes
+ * memory-constraint bases in backend-private registers. */
static NativeAddr rv_asm_loc_to_addr(RvNativeTarget* a, SrcLoc loc,
NativeLoc src) {
@@ -3887,61 +4091,6 @@ static Reg rv_asm_native_mem_base(RvNativeTarget* a, SrcLoc loc, NativeLoc src,
return dst;
}
-static u32 rv_asm_reg_mem_size(RvNativeTarget* a, NativeAllocClass cls,
- KitCgTypeId type) {
- u32 sz = native_type_size(&a->base, type);
- if (cls == NATIVE_REG_INT && sz > a->variant->ptr_bytes)
- sz = a->variant->ptr_bytes;
- return sz;
-}
-
-static Reg rv_asm_stage_reg(RvNativeTarget* a, SrcLoc loc, NativeAllocClass cls,
- u32* nint, u32* nfp) {
- static const Reg int_regs[] = {RV_TMP2, RV_TMP3};
- static const Reg fp_regs[] = {RV_FTMP0, RV_FTMP1};
- if (cls == NATIVE_REG_FP) {
- if (*nfp >= (u32)(sizeof fp_regs / sizeof fp_regs[0]))
- rv_asm_panic_at(a->base.c, loc, "too many staged fp asm operands");
- return fp_regs[(*nfp)++];
- }
- if (*nint >= (u32)(sizeof int_regs / sizeof int_regs[0]))
- rv_asm_panic_at(a->base.c, loc, "too many staged integer asm operands");
- return int_regs[(*nint)++];
-}
-
-static void rv_asm_load_loc_to_reg(RvNativeTarget* a, SrcLoc loc, NativeLoc src,
- NativeLoc dst) {
- NativeTarget* t = &a->base;
- NativeAllocClass cls = (NativeAllocClass)dst.cls;
- if (src.kind == NATIVE_LOC_REG) {
- if (src.v.reg != dst.v.reg || src.cls != dst.cls) t->move(t, dst, src);
- return;
- }
- if (src.kind == NATIVE_LOC_IMM) {
- if (cls != NATIVE_REG_INT)
- rv_asm_panic_at(t->c, loc,
- "floating-point immediate asm input is unsupported");
- t->load_imm(t, dst, src.v.imm);
- return;
- }
- rv_emit_mem(
- a, 1, dst, rv_asm_loc_to_addr(a, loc, src),
- native_mem_for_type(t, dst.type, rv_asm_reg_mem_size(a, cls, dst.type)));
-}
-
-static void rv_asm_store_reg_to_loc(RvNativeTarget* a, SrcLoc loc,
- NativeLoc dst, NativeLoc src) {
- NativeTarget* t = &a->base;
- NativeAllocClass cls = (NativeAllocClass)src.cls;
- if (dst.kind == NATIVE_LOC_REG) {
- if (dst.v.reg != src.v.reg || dst.cls != src.cls) t->move(t, dst, src);
- return;
- }
- rv_emit_mem(
- a, 0, src, rv_asm_loc_to_addr(a, loc, dst),
- native_mem_for_type(t, src.type, rv_asm_reg_mem_size(a, cls, src.type)));
-}
-
static void rv_asm_native_panic(NativeTarget* t, SrcLoc loc, const char* msg) {
rv_asm_panic_at(t->c, loc, msg);
}
@@ -3951,23 +4100,6 @@ static Reg rv_asm_native_mem_base_hook(NativeTarget* t, SrcLoc loc,
return rv_asm_native_mem_base(rv_of(t), loc, src, ntmp);
}
-static Reg rv_asm_native_stage_reg_hook(NativeTarget* t, SrcLoc loc,
- NativeAllocClass cls, u32* ntmp,
- u32* nstage_int, u32* nstage_fp) {
- (void)ntmp;
- return rv_asm_stage_reg(rv_of(t), loc, cls, nstage_int, nstage_fp);
-}
-
-static void rv_asm_native_load_loc_hook(NativeTarget* t, SrcLoc loc,
- NativeLoc src, NativeLoc dst) {
- rv_asm_load_loc_to_reg(rv_of(t), loc, src, dst);
-}
-
-static void rv_asm_native_store_loc_hook(NativeTarget* t, SrcLoc loc,
- NativeLoc dst, NativeLoc src) {
- rv_asm_store_reg_to_loc(rv_of(t), loc, dst, src);
-}
-
static void rv_asm_native_run_template_hook(
NativeTarget* t, const char* tmpl, const AsmConstraint* outs, u32 nout,
Operand* bound_outs, const AsmConstraint* ins, u32 nin, Operand* bound_ins,
@@ -3987,15 +4119,10 @@ static void rv_asm_block_native(NativeTarget* t, const char* tmpl,
RvNativeTarget* a = rv_of(t);
SrcLoc loc = a->func ? a->func->loc : (SrcLoc){0, 0, 0};
static const NativeAsmNativeHooks hooks = {
- .opk_reg = RV64_INLINE_OPK_REG,
- .opcls_fp = RV64_INLINE_OPCLS_FP,
.panic = rv_asm_native_panic,
.bound_reg = rv_asm_bound_reg,
.bound_mem = rv_asm_bound_mem,
.mem_base = rv_asm_native_mem_base_hook,
- .stage_reg = rv_asm_native_stage_reg_hook,
- .load_loc_to_reg = rv_asm_native_load_loc_hook,
- .store_reg_to_loc = rv_asm_native_store_loc_hook,
.run_template = rv_asm_native_run_template_hook,
};
native_asm_bind_native_operands(t, loc, tmpl, outs, nout, out_locs, ins, nin,
@@ -4026,6 +4153,7 @@ NativeTarget* rv64_native_target_new(Compiler* c, ObjBuilder* obj,
t->class_for_type = native_class_for_type_fp_le8;
t->imm_legal = rv_imm_legal;
t->addr_legal = rv_addr_legal;
+ t->machine_op_clobbers = rv_machine_op_clobbers;
t->func_begin = rv_func_begin;
t->func_begin_known_frame = rv_func_begin_known_frame;
t->note_frame_state = NULL;
@@ -4248,13 +4376,13 @@ static void rv_direct_asm_block(NativeDirectTarget* d, const char* tmpl,
const Sym* clobbers, u32 nclob,
u32 clobber_abi_sets) {
static const NativeAsmDirectHooks hooks = {
- /* Reserve emit scratch (t0/t1/t2/t3), sp/gp/tp/zero/ra and the frame
- * pointer so the operand allocator never hands them out. */
- .scratch_int = (1u << RV_ZERO) | (1u << RV_RA) | (1u << RV_SP) |
- (1u << RV_GP) | (1u << RV_TP) | (1u << RV_TMP0) |
- (1u << RV_TMP1) | (1u << RV_TMP2) | (1u << RV_TMP3) |
- (1u << RV_S0),
- .scratch_fp =
+ /* Reserve backend temps t0..t3, SP/GP/TP/zero/RA, and the frame pointer so
+ * the direct asm operand allocator never hands them out. */
+ .reserved_int_mask =
+ (1u << RV_ZERO) | (1u << RV_RA) | (1u << RV_SP) | (1u << RV_GP) |
+ (1u << RV_TP) | (1u << RV_TMP0) | (1u << RV_TMP1) |
+ (1u << RV_TMP2) | (1u << RV_TMP3) | (1u << RV_S0),
+ .reserved_fp_mask =
(1u << RV_FTMP0) | (1u << RV_FTMP1) | (1u << 2u) | (1u << 3u),
.opk_reg = RV64_INLINE_OPK_REG,
.opcls_fp = RV64_INLINE_OPCLS_FP,
diff --git a/src/arch/x64/native.c b/src/arch/x64/native.c
@@ -14,14 +14,15 @@
* The single-pass (-O0) prologue reserves a NOP placeholder patched in func_end
* once max_outgoing and callee-saves are known.
*
- * Register model. INT scratch (never allocable, never driver scratch): R10 and
- * R11 — the emit paths' fixed temporaries. FP scratch: XMM14 and XMM15. RSP/RBP
- * are reserved (stack/frame pointers). RAX is reserved too (return value, the
- * div/mul implicit operand), but it is NOT an emit temp, so inline asm may pin
- * an operand to it (the Linux syscall idiom) — see x64_asm_operand_reg_ok.
- * The driver scratch pool is R8/R9 (int) and XMM4/XMM5 (fp), caller-saved on
- * both SysV and Win64 and disjoint from the emit temps so a hook never clobbers
- * an operand parked there. Scratch registers are reserved from allocation.
+ * Register model. R10/R11 and XMM14/XMM15 are backend-private hook/asm
+ * temporaries and are never optimizer operand locations. RSP/RBP are reserved
+ * stack/frame pointers. RAX is reserved too (return value and div/mul implicit
+ * operand), but is not in either temporary bank, so inline asm may pin an
+ * operand to it (the Linux syscall idiom) — see x64_asm_operand_reg_ok. The O1
+ * instruction-scoped operand bank is R8/R9 (int) and XMM4/XMM5 (fp),
+ * caller-saved on both SysV and Win64; the O0 direct target happens to use the
+ * same registers through its independent `scratch` policy. Both banks are
+ * reserved from persistent allocation.
* Callee-saved set is resolved per-OS via x64_abi_for_os at runtime (the
* legality masks below are SysV's, the conservative superset that both ABIs'
* allocators respect — Win64's extra callee-saves RDI/RSI/xmm6-15 only shrink
@@ -49,10 +50,10 @@
#include "obj/obj.h"
enum {
- X64_TMP_INT = X64_R10, /* emit-internal int scratch (reserved) */
- X64_TMP_INT2 = X64_R11, /* emit-internal int scratch (reserved) */
- X64_TMP_FP = X64_XMM0 + 14, /* emit-internal fp scratch (reserved) */
- X64_TMP_FP2 = X64_XMM15, /* emit-internal fp scratch (reserved) */
+ X64_TMP_INT = X64_R10, /* backend-private int temp (reserved) */
+ X64_TMP_INT2 = X64_R11, /* backend-private int temp (reserved) */
+ X64_TMP_FP = X64_XMM0 + 14, /* backend-private fp temp (reserved) */
+ X64_TMP_FP2 = X64_XMM15, /* backend-private fp temp (reserved) */
X64_MAX_REG_ARG_MOVES = 16u,
/* Deferred entry register-binds (-O1): bounded by simultaneously-live
* register-homed param parts, i.e. the allocable register count. */
@@ -93,6 +94,7 @@ typedef struct X64NativeTarget {
u32 next_param_stack;
u8 has_sret;
u8 is_variadic;
+ u8 call_callee_parked;
NativeFrameSlot sret_ptr_slot;
NativeFrameSlot reg_save_slot; /* SysV variadic 176B __va_list_tag area */
@@ -129,6 +131,11 @@ typedef struct X64NativeTarget {
NativeArgMove bind_moves[X64_MAX_BIND_MOVES];
u32 nbind_moves;
+ /* Integer data scratch used while packing/unpacking a non-power-of-two ABI
+ * part. Normally r10, keeping r11 exclusively available to resolve complex
+ * addresses. */
+ Reg part_scratch;
+
const X64ABIRegs* abi;
} X64NativeTarget;
@@ -163,10 +170,12 @@ static u32 loc_size32(NativeTarget* t, NativeLoc loc) {
return native_type_size(t, loc.type);
}
-/* The original x64_is_64 predicate exactly, but sourcing the width from the
- * cached descriptor: `size >= 8 || is_ptr`. Byte-identical. */
+/* Select the physical integer carrier width. Native scalar sizes are
+ * 1/2/4/8, while a small direct aggregate may legitimately remain packed in a
+ * register under the -O0 cache; its 5/6/7-byte representation needs a 64-bit
+ * move even though its semantic size is below eight bytes. */
static int loc_is_64(NativeTarget* t, NativeLoc loc) {
- return loc_size32(t, loc) >= 8u || cg_type_is_ptr(t->c, loc.type);
+ return loc_size32(t, loc) > 4u || cg_type_is_ptr(t->c, loc.type);
}
static u32 loc_reg(NativeLoc loc) { return loc.v.reg & 0xfu; }
@@ -232,11 +241,12 @@ static void emit_jcc_rel32(MCEmitter* mc, u32 cc, MCLabel l);
* (the fixed-size save region covers them). RSI/RDI are safe to cache: they have
* no implicit x86 use — unlike RCX (shift count) and RAX/RDX (div/mul), which
* are therefore excluded — and incoming args are spilled to frame homes at entry
- * before any body op runs. R8/R9 are the driver scratch pool; R10/R11 are emit
- * scratch (reserved); RAX is reserved (return / div-mul, asm-pinnable). */
-static const Reg x64_int_allocable[] = {X64_RSI, X64_RDI, X64_R13, X64_R14,
- X64_R15};
+ * before any body op runs. R8/R9 are also the O1 operand-temp bank; R10/R11 are
+ * backend-private (reserved); RAX is reserved (return/div-mul, asm-pinnable). */
+static const Reg x64_ndt_int_allocable[] = {X64_RSI, X64_RDI, X64_R13,
+ X64_R14, X64_R15};
static const Reg x64_int_scratch[] = {X64_R8, X64_R9};
+static const Reg x64_int_asm_temps[] = {X64_TMP_INT, X64_TMP_INT2};
static const NativePhysRegInfo x64_int_phys[] = {
X64_PHYS_INT_RESERVED(X64_RAX), /* return / div-mul (asm-pinnable) */
@@ -247,10 +257,10 @@ static const NativePhysRegInfo x64_int_phys[] = {
X64_PHYS_INT_RESERVED(X64_RBP), /* frame pointer */
X64_PHYS_INT_ARG(X64_RSI),
X64_PHYS_INT_ARG(X64_RDI),
- X64_PHYS_INT_ARG_RESERVED(X64_R8), /* driver scratch */
- X64_PHYS_INT_ARG_RESERVED(X64_R9), /* driver scratch */
- X64_PHYS_INT_RESERVED(X64_R10), /* emit scratch */
- X64_PHYS_INT_RESERVED(X64_R11), /* emit scratch */
+ X64_PHYS_INT_ARG_RESERVED(X64_R8), /* O1 operand temp / O0 scratch */
+ X64_PHYS_INT_ARG_RESERVED(X64_R9), /* O1 operand temp / O0 scratch */
+ X64_PHYS_INT_RESERVED(X64_R10), /* backend-private temp */
+ X64_PHYS_INT_RESERVED(X64_R11), /* backend-private temp */
X64_PHYS_INT_RESERVED(X64_R12),
X64_PHYS_INT_CALLEE(X64_R13),
X64_PHYS_INT_CALLEE(X64_R14),
@@ -298,12 +308,14 @@ static const NativePhysRegInfo x64_int_phys[] = {
* materialize directly into the ABI arg registers (Lever 1), then xmm6..xmm11.
* xmm0/xmm1 are also the fp ret regs but ndt_result_reg_stable is OFF, so the
* fp result still moves out of xmm0 via the post-call path (often a no-op).
- * xmm4/xmm5 stay driver scratch; xmm14/xmm15 are emit scratch. */
-static const Reg x64_fp_allocable[] = {
+ * xmm4/xmm5 form the O1 operand-temp bank (and O0 scratch policy);
+ * xmm14/xmm15 are backend-private. */
+static const Reg x64_ndt_fp_allocable[] = {
X64_XMM0, X64_XMM1, X64_XMM2, X64_XMM3,
X64_XMM6, X64_XMM7, X64_XMM8, X64_XMM0 + 9,
X64_XMM0 + 10, X64_XMM0 + 11};
static const Reg x64_fp_scratch[] = {X64_XMM4, X64_XMM5};
+static const Reg x64_fp_asm_temps[] = {X64_TMP_FP, X64_TMP_FP2};
static const NativePhysRegInfo x64_fp_phys[] = {
X64_PHYS_FP_ARG_RET(X64_XMM0), X64_PHYS_FP_ARG_RET(X64_XMM1),
@@ -313,16 +325,22 @@ static const NativePhysRegInfo x64_fp_phys[] = {
X64_PHYS_FP_CALLER(X64_XMM8), X64_PHYS_FP_CALLER(X64_XMM0 + 9),
X64_PHYS_FP_CALLER(X64_XMM0 + 10), X64_PHYS_FP_CALLER(X64_XMM0 + 11),
X64_PHYS_FP_RESERVED(X64_XMM0 + 12), X64_PHYS_FP_RESERVED(X64_XMM0 + 13),
- X64_PHYS_FP_RESERVED(X64_XMM0 + 14), /* emit scratch */
- X64_PHYS_FP_RESERVED(X64_XMM15), /* emit scratch */
+ X64_PHYS_FP_RESERVED(X64_XMM0 + 14), /* backend-private temp */
+ X64_PHYS_FP_RESERVED(X64_XMM15), /* backend-private temp */
};
static const NativeAllocClassInfo x64_classes[] = {
{.cls = NATIVE_REG_INT,
- .allocable = x64_int_allocable,
- .nallocable = sizeof x64_int_allocable / sizeof x64_int_allocable[0],
+ .ndt_allocable = x64_ndt_int_allocable,
+ .ndt_allocable_count =
+ sizeof x64_ndt_int_allocable / sizeof x64_ndt_int_allocable[0],
.scratch = x64_int_scratch,
.nscratch = sizeof x64_int_scratch / sizeof x64_int_scratch[0],
+ .emit_temps = x64_int_scratch,
+ .nemit_temps = sizeof x64_int_scratch / sizeof x64_int_scratch[0],
+ .asm_temps = x64_int_asm_temps,
+ .nasm_temps = sizeof x64_int_asm_temps / sizeof x64_int_asm_temps[0],
+ .emit_cache_mask = (1u << X64_R8) | (1u << X64_R9),
.phys = x64_int_phys,
.nphys = sizeof x64_int_phys / sizeof x64_int_phys[0],
/* caller-saved: rax,rcx,rdx,rsi,rdi,r8,r9,r10,r11 (SysV) */
@@ -336,15 +354,21 @@ static const NativeAllocClassInfo x64_classes[] = {
.arg_mask = (1u << X64_RDI) | (1u << X64_RSI) | (1u << X64_RDX) |
(1u << X64_RCX) | (1u << X64_R8) | (1u << X64_R9),
.ret_mask = (1u << X64_RAX) | (1u << X64_RDX),
- /* rax, rsp, rbp reserved; r8/r9 driver scratch; r10/r11 emit scratch */
+ /* rax/rsp/rbp reserved; r8/r9 operand temps; r10/r11 backend-private */
.reserved_mask = (1u << X64_RAX) | (1u << X64_RSP) | (1u << X64_RBP) |
(1u << X64_R8) | (1u << X64_R9) | (1u << X64_R10) |
(1u << X64_R11) | (1u << X64_RBX) | (1u << X64_R12)},
{.cls = NATIVE_REG_FP,
- .allocable = x64_fp_allocable,
- .nallocable = sizeof x64_fp_allocable / sizeof x64_fp_allocable[0],
+ .ndt_allocable = x64_ndt_fp_allocable,
+ .ndt_allocable_count =
+ sizeof x64_ndt_fp_allocable / sizeof x64_ndt_fp_allocable[0],
.scratch = x64_fp_scratch,
.nscratch = sizeof x64_fp_scratch / sizeof x64_fp_scratch[0],
+ .emit_temps = x64_fp_scratch,
+ .nemit_temps = sizeof x64_fp_scratch / sizeof x64_fp_scratch[0],
+ .asm_temps = x64_fp_asm_temps,
+ .nasm_temps = sizeof x64_fp_asm_temps / sizeof x64_fp_asm_temps[0],
+ .emit_cache_mask = (1u << X64_XMM4) | (1u << X64_XMM5),
.phys = x64_fp_phys,
.nphys = sizeof x64_fp_phys / sizeof x64_fp_phys[0],
/* All xmm caller-saved on SysV. */
@@ -352,7 +376,7 @@ static const NativeAllocClassInfo x64_classes[] = {
.callee_saved_mask = 0u,
.arg_mask = 0xffu, /* xmm0..xmm7 */
.ret_mask = (1u << X64_XMM0) | (1u << X64_XMM1),
- /* xmm4/xmm5 driver scratch; xmm14/xmm15 emit scratch. */
+ /* xmm4/xmm5 operand temps; xmm14/xmm15 backend-private. */
.reserved_mask = (1u << X64_XMM4) | (1u << X64_XMM5) |
(1u << (X64_XMM0 + 12)) | (1u << (X64_XMM0 + 13)) |
(1u << (X64_XMM0 + 14)) | (1u << X64_XMM15)},
@@ -389,9 +413,9 @@ static int x64_asm_operand_reg_ok(const NativeRegInfo* ri, NativeAllocClass cls,
(void)ri;
if (cls == NATIVE_REG_INT) {
switch (reg) {
- /* RAX is reserved but not an emit temp, so it is a legal asm pin (the
- * Linux syscall number/return register). R8/R9 are driver scratch and
- * R10/R11 are emit scratch, so those stay excluded. */
+ /* RAX is reserved but not an emitter temp, so it is a legal asm pin (the
+ * Linux syscall number/return register). R8/R9 are optimizer operand
+ * temps and R10/R11 backend/asm temps, so those stay excluded. */
case X64_RAX:
case X64_RBX:
case X64_RCX:
@@ -580,8 +604,9 @@ static void x64_emit_global_lea(NativeTarget* t, u32 dst_reg, ObjSymId sym,
}
}
-/* Resolve a NativeAddr to (base, index, log2_scale, off). Materializes
- * FRAME/FRAME_VALUE/GLOBAL bases into the supplied scratch register. */
+/* Resolve a NativeAddr to (base, index, log2_scale, off). Plain FRAME bases
+ * remain rbp-relative. GLOBAL and FRAME_VALUE components use backend-private
+ * fixed temps; each FRAME_VALUE load uses its exact base_type/index_type. */
static u32 x64_resolve_addr(X64NativeTarget* a, const NativeAddr* addr,
u32 scratch, u32* idx_out, u32* scale_out,
i32* off_out) {
@@ -601,7 +626,10 @@ static u32 x64_resolve_addr(X64NativeTarget* a, const NativeAddr* addr,
}
case NATIVE_ADDR_BASE_FRAME_VALUE: {
X64NativeSlot* s = x64_slot_get(a, addr->base.frame);
- emit_mov_load(t->mc, 8, 0, scratch, X64_RBP, -(i32)s->off);
+ u32 size;
+ if (!addr->base_type) x64_panic(a, "frame-value base has no exact type");
+ size = native_type_size(t, addr->base_type);
+ emit_mov_load(t->mc, size, 0, scratch, X64_RBP, -(i32)s->off);
base = scratch;
off = addr->offset;
break;
@@ -620,8 +648,13 @@ static u32 x64_resolve_addr(X64NativeTarget* a, const NativeAddr* addr,
*scale_out = addr->log2_scale;
} else if (addr->index_kind == NATIVE_ADDR_INDEX_FRAME_VALUE) {
X64NativeSlot* s = x64_slot_get(a, addr->index.frame);
- emit_mov_load(t->mc, 8, 0, X64_TMP_INT2, X64_RBP, -(i32)s->off);
- *idx_out = X64_TMP_INT2;
+ u32 size;
+ u32 index_tmp = base == X64_TMP_INT2 ? X64_TMP_INT : X64_TMP_INT2;
+ if (!addr->index_type)
+ x64_panic(a, "frame-value index has no exact type");
+ size = native_type_size(t, addr->index_type);
+ emit_mov_load(t->mc, size, 0, index_tmp, X64_RBP, -(i32)s->off);
+ *idx_out = index_tmp;
*scale_out = addr->log2_scale;
} else {
*idx_out = REG_NONE;
@@ -633,6 +666,10 @@ static u32 x64_resolve_addr(X64NativeTarget* a, const NativeAddr* addr,
/* ============================ memory ============================ */
+static int x64_native_part_width(u32 size);
+static u32 x64_addr_to_base_reg(X64NativeTarget* a, NativeAddr addr,
+ u32 scratch);
+
/* Central load/store primitive. is_load: 1 load into reg, 0 store reg to mem.
* Materializes the address through X64_TMP_INT2 (r11) for non-reg bases. */
static void x64_emit_mem(X64NativeTarget* a, int is_load, NativeLoc reg,
@@ -645,6 +682,9 @@ static void x64_emit_mem(X64NativeTarget* a, int is_load, NativeLoc reg,
u32 base, idx, scale;
i32 off;
+ if (!fp && !x64_native_part_width(sz))
+ x64_panic(a, "raw integer memory width is not native");
+
/* Global base: fold into a single rip-relative access when local. */
if (addr.base_kind == NATIVE_ADDR_BASE_GLOBAL &&
addr.index_kind == NATIVE_ADDR_INDEX_NONE &&
@@ -725,6 +765,147 @@ static void x64_emit_mem(X64NativeTarget* a, int is_load, NativeLoc reg,
}
}
+static int x64_native_part_width(u32 size) {
+ return size == 1u || size == 2u || size == 4u || size == 8u;
+}
+
+static int x64_addr_explicitly_uses_reg(const NativeAddr* addr, Reg reg) {
+ return (addr->base_kind == NATIVE_ADDR_BASE_REG &&
+ (addr->base.reg & 0xfu) == reg) ||
+ (addr->index_kind == NATIVE_ADDR_INDEX_REG &&
+ (addr->index.reg & 0xfu) == reg);
+}
+
+/* A multi-chunk load cannot overwrite an explicit base/index with its first
+ * chunk and then reuse that destroyed register for the tail. Pin the complete
+ * effective address in r11 first. Generic NativeTarget destinations never use
+ * the backend-private r10/r11 bank; ABI-owned r10 loads use rbp/rsp storage and
+ * therefore do not enter this alias path. */
+static NativeAddr x64_stabilize_part_load_addr(X64NativeTarget* a,
+ NativeAddr addr, Reg dst) {
+ u32 base;
+ KitCgTypeId base_type;
+ if (!x64_addr_explicitly_uses_reg(&addr, dst)) return addr;
+ if (dst == X64_TMP_INT || dst == X64_TMP_INT2)
+ x64_panic(a, "exact-width load aliases private address scratch");
+
+ base_type = addr.base_type;
+ base = x64_addr_to_base_reg(a, addr, X64_TMP_INT2);
+ if (base != X64_TMP_INT2)
+ emit_mov_rr(a->base.mc, 1, X64_TMP_INT2, base);
+ memset(&addr, 0, sizeof addr);
+ addr.base_kind = NATIVE_ADDR_BASE_REG;
+ addr.cls = NATIVE_REG_INT;
+ addr.base.reg = X64_TMP_INT2;
+ addr.base_type = base_type;
+ return addr;
+}
+
+static Reg x64_part_scratch(const X64NativeTarget* a, Reg avoid) {
+ Reg scratch = a->part_scratch;
+ if (scratch != X64_TMP_INT && scratch != X64_TMP_INT2)
+ scratch = X64_TMP_INT2;
+ if (scratch == avoid)
+ scratch = scratch == X64_TMP_INT ? X64_TMP_INT2 : X64_TMP_INT;
+ return scratch;
+}
+
+/* Load an exact-width little-endian integer ABI part. x86 has native memory
+ * operations only for 1/2/4/8 bytes, so a 3/5/6/7-byte tail is assembled from
+ * bounded chunks in the low bits of dst. No chunk crosses the semantic part. */
+static void x64_load_int_part(X64NativeTarget* a, NativeLoc dst,
+ NativeAddr addr, MemAccess mem, u32 size) {
+ NativePartChunkIter chunks;
+ NativeTarget* t = &a->base;
+ Reg rd = (Reg)loc_reg(dst);
+ Reg scratch;
+ NativeLoc tmp;
+ u32 chunk_off, chunk_size;
+ int first = 1;
+
+ if (x64_native_part_width(size)) {
+ mem.size = size;
+ x64_emit_mem(a, 1, dst, addr, mem);
+ return;
+ }
+ if (size == 0u || size > 8u || native_loc_is_fp(dst))
+ x64_panic(a, "invalid exact-width integer part load");
+
+ addr = x64_stabilize_part_load_addr(a, addr, rd);
+ scratch = x64_part_scratch(a, rd);
+ tmp = native_loc_reg(builtin_id(KIT_CG_BUILTIN_I64), NATIVE_REG_INT,
+ scratch);
+ chunks = native_part_chunks(size, 8u);
+ while (native_part_chunk_next(&chunks, &chunk_off, &chunk_size)) {
+ NativeAddr chunk_addr = addr;
+ NativeLoc chunk_dst = first ? dst : tmp;
+ chunk_addr.offset += (i32)chunk_off;
+ mem.size = chunk_size;
+ x64_emit_mem(a, 1, chunk_dst, chunk_addr, mem);
+ if (!first) {
+ emit_shift_imm(t->mc, 1, X64_SHIFT_SUB_SHL, scratch,
+ (u8)(chunk_off * 8u));
+ emit_alu_rr(t->mc, 1, X64_OPC_ALU_OR, rd, scratch);
+ }
+ first = 0;
+ }
+}
+
+/* Store exactly the semantic bytes carried in an integer ABI register. The
+ * first chunk uses the source directly; later chunks use a shifted private
+ * copy, preserving both the source register and adjacent destination bytes. */
+static void x64_store_int_part(X64NativeTarget* a, NativeAddr addr,
+ NativeLoc src, MemAccess mem, u32 size) {
+ NativePartChunkIter chunks;
+ NativeTarget* t = &a->base;
+ Reg rs = (Reg)loc_reg(src);
+ Reg scratch;
+ NativeLoc tmp;
+ u32 chunk_off, chunk_size, shifted = 0;
+ int copied = 0;
+
+ if (x64_native_part_width(size)) {
+ mem.size = size;
+ x64_emit_mem(a, 0, src, addr, mem);
+ return;
+ }
+ if (size == 0u || size > 8u || native_loc_is_fp(src))
+ x64_panic(a, "invalid exact-width integer part store");
+
+ scratch = x64_part_scratch(a, rs);
+ /* x64_resolve_addr owns r11 for FRAME_VALUE/GOT materialization. The only
+ * ABI paths whose packed source itself occupies r10 store to plain rbp/rsp
+ * addresses; reject a future complex-address use instead of replacing the
+ * shifted data with an address component. Generic emitter operands never
+ * occupy either backend-private register and therefore use r10 above. */
+ if (scratch == X64_TMP_INT2 &&
+ (addr.base_kind == NATIVE_ADDR_BASE_FRAME_VALUE ||
+ addr.index_kind == NATIVE_ADDR_INDEX_FRAME_VALUE ||
+ (addr.base_kind == NATIVE_ADDR_BASE_GLOBAL &&
+ x64_use_got_for_sym(t, addr.base.global.sym))))
+ x64_panic(a, "exact-width store aliases address scratch");
+ tmp = native_loc_reg(builtin_id(KIT_CG_BUILTIN_I64), NATIVE_REG_INT,
+ scratch);
+ chunks = native_part_chunks(size, 8u);
+ while (native_part_chunk_next(&chunks, &chunk_off, &chunk_size)) {
+ NativeAddr chunk_addr = addr;
+ NativeLoc chunk_src = src;
+ chunk_addr.offset += (i32)chunk_off;
+ if (chunk_off != 0u) {
+ if (!copied) {
+ emit_mov_rr(t->mc, 1, scratch, rs);
+ copied = 1;
+ }
+ emit_shift_imm(t->mc, 1, X64_SHIFT_SUB_SHR, scratch,
+ (u8)((chunk_off - shifted) * 8u));
+ shifted = chunk_off;
+ chunk_src = tmp;
+ }
+ mem.size = chunk_size;
+ x64_emit_mem(a, 0, chunk_src, chunk_addr, mem);
+ }
+}
+
/* ============================ moves / data ============================ */
static void x64_move(NativeTarget* t, NativeLoc dst, NativeLoc src) {
@@ -751,6 +932,19 @@ static void x64_move(NativeTarget* t, NativeLoc dst, NativeLoc src) {
emit_mov_rr(mc, loc_is_64(t, dst) ? 1 : 0, rd, rs);
}
+/* Preserve an indirect call target below rsp while its arguments are
+ * marshalled. CFA is rbp-based, so this transient balanced pair needs no CFI
+ * adjustment. High registers use the PUSH/POP opcode's REX.B extension. */
+static void x64_push_reg(MCEmitter* mc, Reg reg) {
+ if (reg & 8u) emit1(mc, X64_REX_BASE | X64_REX_B);
+ emit1(mc, X64_OPC_PUSH_R | (reg & 7u));
+}
+
+static void x64_pop_reg(MCEmitter* mc, Reg reg) {
+ if (reg & 8u) emit1(mc, X64_REX_BASE | X64_REX_B);
+ emit1(mc, X64_OPC_POP_R | (reg & 7u));
+}
+
static void x64_load_imm(NativeTarget* t, NativeLoc dst, i64 imm) {
x64_emit_load_imm(t->mc, loc_is_64(t, dst) ? 1 : 0, loc_reg(dst), imm);
}
@@ -800,11 +994,19 @@ static void x64_load_addr(NativeTarget* t, NativeLoc dst, NativeAddr addr) {
static void x64_load(NativeTarget* t, NativeLoc dst, NativeAddr addr,
MemAccess mem) {
- x64_emit_mem(x64_of(t), 1, dst, addr, mem);
+ u32 size = mem.size ? mem.size : loc_size32(t, dst);
+ if (!x64_native_part_width(size) && size <= 8u)
+ x64_load_int_part(x64_of(t), dst, addr, mem, size);
+ else
+ x64_emit_mem(x64_of(t), 1, dst, addr, mem);
}
static void x64_store(NativeTarget* t, NativeAddr addr, NativeLoc src,
MemAccess mem) {
- x64_emit_mem(x64_of(t), 0, src, addr, mem);
+ u32 size = mem.size ? mem.size : loc_size32(t, src);
+ if (!x64_native_part_width(size) && size <= 8u)
+ x64_store_int_part(x64_of(t), addr, src, mem, size);
+ else
+ x64_emit_mem(x64_of(t), 0, src, addr, mem);
}
/* Resolve an addressable NativeAddr to a bare base register (no index, off 0)
@@ -837,52 +1039,41 @@ static u32 x64_addr_to_base_reg(X64NativeTarget* a, NativeAddr addr,
return scratch;
}
-/* copy_bytes: resolve dst into r11 and src into rax (both bare pointers), then
- * unrolled granule copy through rdx. dst is resolved first (its base may live
- * in r11 from a FRAME_VALUE load) and src second so the two never alias. */
+/* Normalize one address at a time, load through R10, preserve the granule in
+ * XMM14 while the destination is normalized, then restore it to R10 for the
+ * store. This keeps both effective addresses and the transfer value entirely
+ * in the backend-private R10/R11/XMM14 bank; no allocator/cache-visible fixed
+ * register is touched. */
static void x64_copy_bytes(NativeTarget* t, NativeAddr dst, NativeAddr src,
AggregateAccess access) {
X64NativeTarget* a = x64_of(t);
- /* Copy chunk by chunk (8/4/2/1) through the value scratch rax, letting
- * x64_emit_mem resolve each address with its own scratch (r11). Uses only the
- * reserved emit scratch (rax/r11) — no ad-hoc allocable temp (previously
- * rdx), which the optimizer may have live across the copy. */
- KitCgTypeId tys[4];
+ MCEmitter* mc = t->mc;
u32 n = access.size, i = 0;
- tys[0] = builtin_id(KIT_CG_BUILTIN_I64);
- tys[1] = builtin_id(KIT_CG_BUILTIN_I32);
- tys[2] = builtin_id(KIT_CG_BUILTIN_I16);
- tys[3] = builtin_id(KIT_CG_BUILTIN_I8);
while (i < n) {
u32 rem = n - i, s;
- KitCgTypeId ty;
NativeAddr sa = src, da = dst;
- NativeLoc val;
- MemAccess mem;
+ u32 sb, db;
if (rem >= 8u) {
s = 8u;
- ty = tys[0];
} else if (rem >= 4u) {
s = 4u;
- ty = tys[1];
} else if (rem >= 2u) {
s = 2u;
- ty = tys[2];
} else {
s = 1u;
- ty = tys[3];
}
sa.offset += (i32)i;
- sa.base_type = ty;
da.offset += (i32)i;
- da.base_type = ty;
- val = native_loc_reg(ty, NATIVE_REG_INT, X64_TMP_INT);
- memset(&mem, 0, sizeof mem);
- mem.type = ty;
- mem.size = s;
- mem.align = s;
- x64_emit_mem(a, 1, val, sa, mem); /* rax = [src + i] */
- x64_emit_mem(a, 0, val, da, mem); /* [dst + i] = rax */
+ sb = x64_addr_to_base_reg(a, sa, X64_TMP_INT2);
+ emit_mov_load(mc, s, 0, X64_TMP_INT, sb, 0);
+ emit_sse_rr_w(mc, 0x66, 0x6E, s == 8u, X64_TMP_FP, X64_TMP_INT);
+ db = x64_addr_to_base_reg(a, da, X64_TMP_INT2);
+ if (db == X64_TMP_INT) {
+ emit_mov_rr(mc, 1, X64_TMP_INT2, db);
+ db = X64_TMP_INT2;
+ }
+ emit_sse_rr_w(mc, 0x66, 0x7E, s == 8u, X64_TMP_FP, X64_TMP_INT);
+ emit_mov_store(mc, s, X64_TMP_INT, db, 0);
i += s;
}
}
@@ -893,34 +1084,43 @@ static void x64_set_bytes(NativeTarget* t, NativeAddr dst, NativeLoc byte_value,
MCEmitter* mc = t->mc;
u32 dr = x64_addr_to_base_reg(a, dst, X64_TMP_INT2);
u32 n = access.size, i = 0;
- /* Broadcast the byte across 8 bytes into rax. */
+ /* R10 holds the broadcast value and R11 the resolved destination. Keep both
+ * entirely backend-private: the variable case broadcasts the low byte in
+ * XMM14 (punpcklbw/punpcklwd/punpckldq), then moves the qword to R10. */
+ if (dr == X64_TMP_INT) {
+ emit_mov_rr(mc, 1, X64_TMP_INT2, dr);
+ dr = X64_TMP_INT2;
+ }
if (byte_value.kind == NATIVE_LOC_IMM) {
u8 b = (u8)(byte_value.v.imm & 0xffu);
u64 b64 = b;
b64 |= b64 << 8;
b64 |= b64 << 16;
b64 |= b64 << 32;
- x64_emit_load_imm(mc, 1, X64_RAX, (i64)b64);
+ x64_emit_load_imm(mc, 1, X64_TMP_INT, (i64)b64);
} else {
- /* Replicate the low byte of a register via multiply by 0x0101..01. */
- x64_emit_load_imm(mc, 1, X64_R11, (i64)0x0101010101010101ll);
- emit_mov_rr(mc, 1, X64_RAX, loc_reg(byte_value));
- emit_imul_rr(mc, 1, X64_RAX, X64_R11);
+ emit_mov_rr(mc, 0, X64_TMP_INT, loc_reg(byte_value));
+ emit_alu_imm32(mc, 0, X64_ALU_SUB_AND, X64_TMP_INT, 0xff);
+ emit_sse_rr_w(mc, 0x66, 0x6E, 0, X64_TMP_FP, X64_TMP_INT);
+ emit_sse_rr(mc, 0x66, 0x60, X64_TMP_FP, X64_TMP_FP);
+ emit_sse_rr(mc, 0x66, 0x61, X64_TMP_FP, X64_TMP_FP);
+ emit_sse_rr(mc, 0x66, 0x62, X64_TMP_FP, X64_TMP_FP);
+ emit_sse_rr_w(mc, 0x66, 0x7E, 1, X64_TMP_FP, X64_TMP_INT);
}
while (i + 8u <= n) {
- emit_mov_store(mc, 8, X64_RAX, dr, (i32)i);
+ emit_mov_store(mc, 8, X64_TMP_INT, dr, (i32)i);
i += 8u;
}
while (i + 4u <= n) {
- emit_mov_store(mc, 4, X64_RAX, dr, (i32)i);
+ emit_mov_store(mc, 4, X64_TMP_INT, dr, (i32)i);
i += 4u;
}
while (i + 2u <= n) {
- emit_mov_store(mc, 2, X64_RAX, dr, (i32)i);
+ emit_mov_store(mc, 2, X64_TMP_INT, dr, (i32)i);
i += 2u;
}
while (i < n) {
- emit_mov_store(mc, 1, X64_RAX, dr, (i32)i);
+ emit_mov_store(mc, 1, X64_TMP_INT, dr, (i32)i);
i += 1u;
}
}
@@ -964,8 +1164,22 @@ static void x64_bitfield_store(NativeTarget* t, NativeAddr ra, NativeLoc src,
u32 src_reg = loc_reg(src);
u32 base;
ra.offset += (i32)bf.storage_offset;
- /* Stabilize the base into r11 before consuming rax/rcx/rdx scratch. */
+ /* Resolve the address before borrowing the fixed value temporaries. A bare
+ * register address may be returned unchanged, so explicitly stabilize it in
+ * r11 when the read-modify-write is about to clobber that register. */
base = x64_addr_to_base_reg(a, ra, X64_TMP_INT2);
+ if (base == X64_RAX || base == X64_RCX || base == X64_RDX) {
+ emit_mov_rr(mc, 1, X64_TMP_INT2, base);
+ base = X64_TMP_INT2;
+ }
+ /* A dying source may legally occupy one of the fixed clobber registers.
+ * Preserve it after address resolution (which may transiently use r10 for a
+ * spilled index) and before loading the storage/mask into rax/rcx. RDX stays
+ * live until it has been copied to RCX below, so it needs no staging. */
+ if (src_reg == X64_RAX || src_reg == X64_RCX) {
+ emit_mov_rr(mc, w, X64_TMP_INT, src_reg);
+ src_reg = X64_TMP_INT;
+ }
/* rax = storage; rax &= ~mask. */
emit_mov_load(mc, storage_bytes, 0, X64_RAX, base, 0);
x64_emit_load_imm(mc, w, X64_RCX, (i64)~mask);
@@ -1031,17 +1245,20 @@ static void x64_binop(NativeTarget* t, BinOp op, NativeLoc dst, NativeLoc aop,
/* Division: rax/rdx implicit; divisor must avoid rax/rdx. */
if (op == BO_SDIV || op == BO_UDIV || op == BO_SREM || op == BO_UREM) {
u32 rb;
- if (ra != X64_RAX) emit_mov_rr(mc, w, X64_RAX, ra);
if (b_imm) {
x64_emit_load_imm(mc, w, X64_R11, imm);
rb = X64_R11;
} else {
rb = loc_reg(bop);
+ /* Preserve the divisor before placing the dividend in RAX. This is a
+ * real parallel-copy edge: a dying divisor may be in RAX while the
+ * dividend is elsewhere, or in RDX before CQO/XOR clears it. */
if (rb == X64_RAX || rb == X64_RDX) {
emit_mov_rr(mc, w, X64_R11, rb);
rb = X64_R11;
}
}
+ if (ra != X64_RAX) emit_mov_rr(mc, w, X64_RAX, ra);
if (op == BO_SDIV || op == BO_SREM) {
emit_cqo_or_cdq(mc, w);
emit_f7_rm(mc, w, X64_F7_SUB_IDIV, rb);
@@ -1069,6 +1286,16 @@ static void x64_binop(NativeTarget* t, BinOp op, NativeLoc dst, NativeLoc aop,
}
{
u32 rb = loc_reg(bop);
+ /* RCX cannot simultaneously hold a distinct result and the shift
+ * count. Compute through backend-private R10, then publish the result
+ * after the instruction has consumed CL. */
+ if (rd == X64_RCX && (ra != X64_RCX || rb != X64_RCX)) {
+ emit_mov_rr(mc, w, X64_TMP_INT, ra);
+ if (rb != X64_RCX) emit_mov_rr(mc, 0, X64_RCX, rb);
+ emit_shift_cl(mc, w, sub, X64_TMP_INT);
+ emit_mov_rr(mc, w, X64_RCX, X64_TMP_INT);
+ return;
+ }
/* Place the count in cl and the value in dst. Stage the count through
* r11 first so neither move clobbers the other when the value already
* sits in rcx or the count sits in dst. (The optimizer additionally
@@ -1398,10 +1625,10 @@ static void x64_convert(NativeTarget* t, ConvKind k, NativeLoc dst,
emit_jmp_rel32(mc, L_done);
mc_label_place(mc, L_high);
emit_mov_rr(mc, 1, X64_R11, rs);
- emit_mov_rr(mc, 1, X64_RAX, rs);
- emit_alu_imm8(mc, 1, X64_ALU_SUB_AND, X64_RAX, 1);
+ emit_mov_rr(mc, 1, X64_TMP_INT, rs);
+ emit_alu_imm8(mc, 1, X64_ALU_SUB_AND, X64_TMP_INT, 1);
emit_shift_imm(mc, 1, X64_SHIFT_SUB_SHR, X64_R11, 1);
- emit_alu_rr(mc, 1, X64_OPC_ALU_OR, X64_R11, X64_RAX);
+ emit_alu_rr(mc, 1, X64_OPC_ALU_OR, X64_R11, X64_TMP_INT);
emit_sse_rr_w(mc, prefix, 0x2A, 1, rd, X64_R11);
emit_sse_rr(mc, prefix, 0x58, rd, rd);
mc_label_place(mc, L_done);
@@ -1543,9 +1770,10 @@ static void x64_cmp_branch(NativeTarget* t, CmpOp op, NativeLoc aop,
if (fp) {
/* Materialize the 0/1 result, then branch on nonzero. */
NativeLoc tmp =
- native_loc_reg(builtin_id(KIT_CG_BUILTIN_I32), NATIVE_REG_INT, X64_RAX);
+ native_loc_reg(builtin_id(KIT_CG_BUILTIN_I32), NATIVE_REG_INT,
+ X64_TMP_INT);
x64_cmp(t, op, tmp, aop, bop);
- emit_test_self(mc, 0, X64_RAX);
+ emit_test_self(mc, 0, X64_TMP_INT);
emit_jcc_rel32(mc, X64_CC_NE, l);
return;
}
@@ -1803,11 +2031,13 @@ static void x64_func_begin_common(NativeTarget* t, const CGFuncDesc* fd) {
a->next_param_stack = 0;
a->has_sret = (abi && abi->has_sret) ? 1u : 0u;
a->is_variadic = (abi && abi->variadic) ? 1u : 0u;
+ a->call_callee_parked = 0;
a->sret_ptr_slot = NATIVE_FRAME_SLOT_NONE;
a->reg_save_slot = NATIVE_FRAME_SLOT_NONE;
a->npatches = 0;
a->nalloca = 0;
a->nbind_moves = 0;
+ a->part_scratch = X64_TMP_INT;
a->slim_frame = 0;
a->redzone_leaf = 0;
/* Single-pass (-O0) reservation: the NDT caches only in its allocable pool's
@@ -2229,50 +2459,71 @@ static u32 x64_signature_stack_bytes(NativeTarget* t, KitCgTypeId fn_type,
return x64_call_stack_size(t, &d);
}
-/* Resolve a NativeLoc to an addressable NativeAddr (frame/stack/addr). */
-static NativeAddr x64_loc_addr(X64NativeTarget* a, NativeLoc loc, u32 offset) {
+/* Resolve a dereferenceable NativeLoc to its storage address. */
+static NativeAddr x64_storage_addr(X64NativeTarget* a, NativeLoc loc,
+ u32 offset) {
NativeAddr addr;
- memset(&addr, 0, sizeof addr);
- switch ((NativeLocKind)loc.kind) {
- case NATIVE_LOC_FRAME:
- addr.base_kind = NATIVE_ADDR_BASE_FRAME;
- addr.base.frame = loc.v.frame;
- addr.base_type = loc.type;
- addr.offset = (i32)offset;
- return addr;
- case NATIVE_LOC_STACK:
- addr.base_kind = NATIVE_ADDR_BASE_FRAME;
- addr.base.frame = loc.v.stack.slot;
- addr.base_type = loc.type;
- addr.offset = loc.v.stack.offset + (i32)offset;
- return addr;
- case NATIVE_LOC_ADDR:
- addr = loc.v.addr;
- addr.offset += (i32)offset;
- return addr;
- default:
- x64_panic(a, "location is not addressable");
- }
+ if (!native_loc_storage_addr(loc, (i32)offset, &addr))
+ x64_panic(a, "location is not storage");
return addr;
}
+static void x64_move_part_reg(NativeTarget* t, NativeLoc dst, NativeLoc src,
+ u32 size) {
+ MCEmitter* mc = t->mc;
+ int dfp = native_loc_is_fp(dst), sfp = native_loc_is_fp(src);
+ u32 rd = loc_reg(dst), rs = loc_reg(src);
+ if (dfp && sfp) {
+ if (size != 4u && size != 8u)
+ x64_panic(x64_of(t), "invalid FP register part width");
+ if (rd != rs) emit_sse_rr(mc, sse_scalar_prefix(size), 0x10, rd, rs);
+ return;
+ }
+ if (dfp && !sfp) {
+ if (size != 4u && size != 8u)
+ x64_panic(x64_of(t), "invalid GPR-to-FP part width");
+ emit_sse_rr_w(mc, 0x66, 0x6E, size == 8u, rd, rs);
+ return;
+ }
+ if (!dfp && sfp) {
+ if (size != 4u && size != 8u)
+ x64_panic(x64_of(t), "invalid FP-to-GPR part width");
+ emit_sse_rr_w(mc, 0x66, 0x7E, size == 8u, rs, rd);
+ return;
+ }
+ if (rd != rs) emit_mov_rr(mc, size > 4u, rd, rs);
+}
+
static void x64_load_part(NativeTarget* t, NativeLoc dst, NativeLoc src,
u32 offset, u32 size) {
X64NativeTarget* a = x64_of(t);
if (src.kind == NATIVE_LOC_REG) {
- x64_move(t, dst, src);
+ x64_move_part_reg(t, dst, src, size);
+ return;
+ }
+ if (native_loc_addr_role(src) == NATIVE_LOC_ADDR_ROLE_VALUE) {
+ NativeAddr addr;
+ if (offset != 0u || size != t->c->target.ptr_size ||
+ !native_loc_address_value(src, &addr))
+ x64_panic(a, "split or invalid address-value argument");
+ x64_load_addr(t, dst, addr);
return;
}
- if (src.kind == NATIVE_LOC_FRAME || src.kind == NATIVE_LOC_STACK ||
- src.kind == NATIVE_LOC_ADDR) {
- NativeAddr addr = x64_loc_addr(a, src, offset);
+ if (native_loc_addr_role(src) == NATIVE_LOC_ADDR_ROLE_STORAGE) {
+ NativeAddr addr = x64_storage_addr(a, src, offset);
+ MemAccess mem = native_mem_for_type(t, dst.type, size);
addr.base_type = dst.type;
- x64_emit_mem(a, 1, dst, addr, native_mem_for_type(t, dst.type, size));
+ if (!x64_native_part_width(size) && size <= 8u)
+ x64_load_int_part(a, dst, addr, mem, size);
+ else
+ x64_emit_mem(a, 1, dst, addr, mem);
return;
}
if (src.kind == NATIVE_LOC_IMM) {
- x64_emit_load_imm(t->mc, loc_is_64(t, dst) ? 1 : 0, loc_reg(dst),
- src.v.imm);
+ i64 part;
+ if (!native_loc_imm_part(src, offset, size, &part))
+ x64_panic(a, "invalid immediate argument part");
+ x64_emit_load_imm(t->mc, size > 4u, loc_reg(dst), part);
return;
}
x64_panic(a, "unsupported part source");
@@ -2281,22 +2532,25 @@ static void x64_load_part(NativeTarget* t, NativeLoc dst, NativeLoc src,
static void x64_store_part(NativeTarget* t, NativeLoc dst, NativeLoc src,
u32 offset, u32 size) {
X64NativeTarget* a = x64_of(t);
- if (dst.kind == NATIVE_LOC_FRAME || dst.kind == NATIVE_LOC_STACK ||
- dst.kind == NATIVE_LOC_ADDR) {
- NativeAddr addr = x64_loc_addr(a, dst, offset);
+ if (native_loc_addr_role(dst) == NATIVE_LOC_ADDR_ROLE_STORAGE) {
+ NativeAddr addr = x64_storage_addr(a, dst, offset);
+ MemAccess mem = native_mem_for_type(t, src.type, size);
addr.base_type = src.type;
- x64_emit_mem(a, 0, src, addr, native_mem_for_type(t, src.type, size));
+ if (!x64_native_part_width(size) && size <= 8u)
+ x64_store_int_part(a, addr, src, mem, size);
+ else
+ x64_emit_mem(a, 0, src, addr, mem);
return;
}
if (dst.kind == NATIVE_LOC_REG) {
- x64_move(t, dst, src);
+ x64_move_part_reg(t, dst, src, size);
return;
}
x64_panic(a, "unsupported part destination");
}
static void x64_addr_of_loc(NativeTarget* t, NativeLoc dst, NativeLoc src) {
- NativeAddr addr = x64_loc_addr(x64_of(t), src, 0);
+ NativeAddr addr = x64_storage_addr(x64_of(t), src, 0);
x64_load_addr(t, dst, addr);
}
@@ -2317,9 +2571,47 @@ static void x64_store_outgoing_part(NativeTarget* t, int tail_call,
addr.offset = (i32)(16u + stack_off);
} else {
addr.base.reg = X64_RSP;
- addr.offset = (i32)stack_off;
+ /* An indirect callee is transiently pushed below rsp for the whole
+ * marshal phase. Keep semantic outgoing offsets anchored at the original
+ * rsp; the balanced pop restores that anchor before call/jmp emission. */
+ addr.offset = (i32)(stack_off + (a->call_callee_parked ? 8u : 0u));
+ }
+ {
+ MemAccess mem = native_mem_for_type(t, src.type, size);
+ if (!x64_native_part_width(size) && size <= 8u)
+ x64_store_int_part(a, addr, src, mem, size);
+ else
+ x64_emit_mem(a, 0, src, addr, mem);
+ }
+}
+
+/* A stack carrier contains the same byte representation as aggregate storage,
+ * so copy a non-power-of-two memory part chunk-by-chunk instead of needlessly
+ * packing it into a GPR and unpacking it again. Besides being smaller, this
+ * needs only one private register while an indirect callee may occupy r11. */
+static void x64_copy_outgoing_part(NativeTarget* t, int tail_call,
+ u32 stack_off, NativeLoc src,
+ const ABIArgPart* part) {
+ NativeAllocClass cls =
+ part->cls == ABI_CLASS_FP ? NATIVE_REG_FP : NATIVE_REG_INT;
+ KitCgTypeId pty = x64_part_scalar_type(part);
+ Reg tmp = cls == NATIVE_REG_FP ? X64_TMP_FP : X64_TMP_INT;
+ NativeLoc tmpreg = native_loc_reg(pty, cls, tmp);
+
+ if (!x64_native_part_width(part->size) && part->size <= 8u &&
+ native_loc_addr_role(src) == NATIVE_LOC_ADDR_ROLE_STORAGE) {
+ NativePartChunkIter chunks = native_part_chunks(part->size, 8u);
+ u32 chunk_off, chunk_size;
+ while (native_part_chunk_next(&chunks, &chunk_off, &chunk_size)) {
+ x64_load_part(t, tmpreg, src, part->src_offset + chunk_off, chunk_size);
+ x64_store_outgoing_part(t, tail_call, stack_off + chunk_off, tmpreg,
+ chunk_size);
+ }
+ return;
}
- x64_emit_mem(a, 0, src, addr, native_mem_for_type(t, src.type, size));
+
+ x64_load_part(t, tmpreg, src, part->src_offset, part->size);
+ x64_store_outgoing_part(t, tail_call, stack_off, tmpreg, part->size);
}
/* NativeTarget bind_param: route incoming param (ABI loc) into dst. */
@@ -2482,7 +2774,7 @@ static void x64_bind_native_param(NativeTarget* t, const CGParamDesc* p,
/* Flush the deferred register-destination param binds as a parallel copy (the
* shared scheduler breaks any cycle the allocator's rotation created through
- * the int/fp emit scratch). Frame-dst and indirect binds were emitted eagerly
+ * backend-private int/fp cycle temporaries). Frame-dst and indirect binds were emitted eagerly
* in bind_param — they only read incoming registers, so they precede this. */
static void x64_bind_params_end(NativeTarget* t) {
X64NativeTarget* a = x64_of(t);
@@ -2586,14 +2878,18 @@ static void x64_plan_call(NativeTarget* t, const NativeCallDesc* desc,
plan->clobber_mask[c] = x64_clobber_mask(aregs, (NativeAllocClass)c);
plan->return_mask[c] = x64_return_mask(abi, (NativeAllocClass)c);
}
- /* Indirect callee in a clobbered/arg register would be lost; stage in r11. */
- if (plan->callee.kind == NATIVE_LOC_REG &&
- (NativeAllocClass)plan->callee.cls == NATIVE_REG_INT &&
- plan->callee.v.reg != X64_R11) {
- NativeLoc scratch =
- native_loc_reg(plan->callee.type, NATIVE_REG_INT, X64_R11);
- x64_move(t, scratch, plan->callee);
- plan->callee = scratch;
+ /* Argument transport owns r10 (part data), r11 (address stabilization), and
+ * rax (parallel-copy cycle breaking). A register-indirect callee therefore
+ * lives in a balanced stack home for the complete marshal phase, independent
+ * of its producer register and every allowed NativeLoc argument shape. */
+ if (plan->callee.kind == NATIVE_LOC_REG) {
+ if ((NativeAllocClass)plan->callee.cls != NATIVE_REG_INT)
+ x64_panic(a, "indirect callee is not in an integer register");
+ if (a->call_callee_parked)
+ x64_panic(a, "nested indirect-callee marshal");
+ x64_push_reg(t->mc, (Reg)loc_reg(plan->callee));
+ a->call_callee_parked = 1;
+ plan->callee = native_loc_reg(plan->callee.type, NATIVE_REG_INT, X64_R11);
}
{
u32 next_int = (abi && abi->has_sret) ? 1u : 0u;
@@ -2629,12 +2925,7 @@ static void x64_plan_call(NativeTarget* t, const NativeCallDesc* desc,
x64_direct_to_stack(aregs, ai, next_int, next_fp)) {
for (p = 0; p < ai->nparts; ++p) {
const ABIArgPart* part = &ai->parts[p];
- NativeAllocClass cls =
- part->cls == ABI_CLASS_FP ? NATIVE_REG_FP : NATIVE_REG_INT;
- Reg tmp = cls == NATIVE_REG_FP ? X64_TMP_FP : X64_TMP_INT;
- NativeLoc tmpreg = native_loc_reg(desc->args[i].type, cls, tmp);
- x64_load_part(t, tmpreg, desc->args[i], part->src_offset, part->size);
- x64_store_outgoing_part(t, tail, stack, tmpreg, part->size);
+ x64_copy_outgoing_part(t, tail, stack, desc->args[i], part);
stack += 8u;
}
continue;
@@ -2668,10 +2959,7 @@ static void x64_plan_call(NativeTarget* t, const NativeCallDesc* desc,
m->size = part->size;
x64_sync_slot(aregs, &next_int, &next_fp);
} else {
- Reg tmp = cls == NATIVE_REG_FP ? X64_TMP_FP : X64_TMP_INT;
- NativeLoc tmpreg = native_loc_reg(desc->args[i].type, cls, tmp);
- x64_load_part(t, tmpreg, desc->args[i], part->src_offset, part->size);
- x64_store_outgoing_part(t, tail, stack, tmpreg, part->size);
+ x64_copy_outgoing_part(t, tail, stack, desc->args[i], part);
stack += 8u;
x64_sync_slot(aregs, &next_int, &next_fp);
}
@@ -2682,7 +2970,7 @@ static void x64_plan_call(NativeTarget* t, const NativeCallDesc* desc,
x64_emit_reg_arg_moves(
t, moves, nmoves,
(plan->callee.kind == NATIVE_LOC_REG && plan->callee.v.reg == X64_R11)
- ? X64_TMP_INT
+ ? X64_RAX
: X64_TMP_INT2);
if (abi && abi->has_sret) {
/* sret pointer in the first int-arg reg. A tail call forwards the
@@ -2698,6 +2986,10 @@ static void x64_plan_call(NativeTarget* t, const NativeCallDesc* desc,
/* Variadic call: AL = number of vector regs used. */
if (abi && abi->variadic)
x64_emit_load_imm(t->mc, 0, X64_RAX, (i64)next_fp);
+ if (a->call_callee_parked) {
+ x64_pop_reg(t->mc, X64_R11);
+ a->call_callee_parked = 0;
+ }
}
/* Return value receipt. */
if (abi && abi->ret.kind == ABI_ARG_DIRECT && desc->nresults) {
@@ -2812,7 +3104,7 @@ static void x64_plan_ret(NativeTarget* t, const CGFuncDesc* fd,
dst_addr.base_kind = NATIVE_ADDR_BASE_REG;
dst_addr.base.reg = X64_R11;
dst_addr.base_type = value->type;
- src_addr = x64_loc_addr(a, *value, 0);
+ src_addr = x64_storage_addr(a, *value, 0);
src_addr.base_type = value->type;
memset(&access, 0, sizeof access);
access.type = value->type;
@@ -2830,6 +3122,33 @@ static void x64_plan_ret(NativeTarget* t, const CGFuncDesc* fd,
u32 ni = 0, nf = 0;
static const u32 ret_int_regs[2] = {X64_RAX, X64_RDX};
u16 p;
+ int exact_parts = 0;
+ for (p = 0; p < abi->ret.nparts; ++p)
+ if (abi->ret.parts[p].cls == ABI_CLASS_INT &&
+ !x64_native_part_width(abi->ret.parts[p].size))
+ exact_parts = 1;
+
+ /* Generic write_phase materializes a register destination from its scalar
+ * type, so an I64 carrier for a 3/5/6/7-byte aggregate part would widen the
+ * source load to eight bytes. Once one exact part is present, marshal every
+ * return part here in ABI order: this preserves already-filled return
+ * registers and keeps each memory read bounded by ABIArgPart.size. */
+ if (exact_parts) {
+ for (p = 0; p < abi->ret.nparts; ++p) {
+ const ABIArgPart* part = &abi->ret.parts[p];
+ NativeAllocClass cls =
+ part->cls == ABI_CLASS_FP ? NATIVE_REG_FP : NATIVE_REG_INT;
+ KitCgTypeId pty = x64_part_scalar_type(part);
+ Reg rreg = cls == NATIVE_REG_FP ? (Reg)(X64_XMM0 + nf++)
+ : (Reg)ret_int_regs[ni++];
+ NativeLoc dst = native_loc_reg(pty, cls, rreg);
+ x64_load_part(t, dst, *value, part->src_offset, part->size);
+ }
+ *out_rets = NULL;
+ *out_nrets = 0;
+ return;
+ }
+
for (p = 0; p < abi->ret.nparts; ++p) {
const ABIArgPart* part = &abi->ret.parts[p];
NativeAllocClass cls =
@@ -2886,14 +3205,15 @@ static void x64_alloca(NativeTarget* t, NativeLoc dst, NativeLoc size,
}
emit_u32le(mc, (u32)aligned);
} else {
- /* rax = (size + 15) & ~15; sub rsp, rax. */
- emit_lea(mc, X64_RAX, rsz, 15);
- emit_rex(mc, 1, 0, 0, X64_RAX);
+ /* r10 = (size + 15) & ~15; sub rsp, r10. */
+ emit_lea(mc, X64_TMP_INT, rsz, 15);
+ emit_rex(mc, 1, 0, 0, X64_TMP_INT);
{
- u8 buf[3] = {X64_OPC_ALU_IMM8, modrm(3u, X64_ALU_SUB_AND, X64_RAX), 0xF0};
+ u8 buf[3] = {X64_OPC_ALU_IMM8,
+ modrm(3u, X64_ALU_SUB_AND, X64_TMP_INT), 0xF0};
mc_emit_bytes(mc, buf, 3);
}
- emit_alu_rr(mc, 1, X64_OPC_ALU_SUB, X64_RSP, X64_RAX);
+ emit_alu_rr(mc, 1, X64_OPC_ALU_SUB, X64_RSP, X64_TMP_INT);
}
a->frame.has_alloca = 1;
/* lea dst, [rsp + max_outgoing] — disp32 patched in func_end. */
@@ -3163,15 +3483,21 @@ static void x64_atomic_store(NativeTarget* t, NativeAddr addr, NativeLoc src,
u32 base = x64_atomic_base(a, addr);
u32 sr = loc_reg(src);
if (mo == KIT_CG_MO_SEQ_CST) {
- /* xchg [mem], r11 implicitly fences. Stage src in rax (r11 holds base). */
- if (sr != X64_RAX) emit_mov_rr(mc, w, X64_RAX, sr);
+ /* XCHG writes the prior memory value into its register operand. Stage the
+ * semantic source in backend-private R10 so that mutation is invisible to
+ * the allocator; R11 holds a materialized address. */
+ if (base == X64_TMP_INT) {
+ emit_mov_rr(mc, 1, X64_TMP_INT2, base);
+ base = X64_TMP_INT2;
+ }
+ if (sr != X64_TMP_INT) emit_mov_rr(mc, w, X64_TMP_INT, sr);
emit_lock_prefix(mc);
- emit_rex(mc, w, X64_RAX, 0, base);
+ emit_rex(mc, w, X64_TMP_INT, 0, base);
{
u8 op = 0x87; /* xchg r/m, r */
mc_emit_bytes(mc, &op, 1);
}
- emit_mem_operand(mc, X64_RAX, base, 0);
+ emit_mem_operand(mc, X64_TMP_INT, base, 0);
return;
}
emit_mov_store(mc, sz, sr, base, 0);
@@ -3189,8 +3515,8 @@ static void x64_atomic_rmw(NativeTarget* t, KitCgAtomicOp op, NativeLoc dst,
u32 vr = loc_reg(val);
(void)mo; /* LOCK ops are full barriers. */
/* The rmw uses fixed rax (prior), rcx (new), rdx (val); the optimizer may
- * have materialized the address into one of them, so keep it out (r11 is the
- * int emit scratch, never an allocated operand). Stage before rdx is loaded.
+ * have materialized the address into one of them, so keep it out (R11 is
+ * backend-private, never an allocated operand). Stage before RDX is loaded.
*/
if (base == X64_RAX || base == X64_RCX || base == X64_RDX) {
emit_mov_rr(mc, 1, X64_TMP_INT2, base);
@@ -3272,8 +3598,8 @@ static void x64_atomic_cas(NativeTarget* t, NativeLoc prior, NativeLoc ok,
(void)success;
(void)failure;
/* cmpxchg uses fixed rax (expected) and rcx (desired). The optimizer may have
- * materialized the address into either; keep it out of both (r11 is the int
- * emit scratch, never an allocated operand). */
+ * materialized the address into either; keep it out of both (R11 is
+ * backend-private, never an allocated operand). */
if (base == X64_RAX || base == X64_RCX) {
emit_mov_rr(mc, 1, X64_TMP_INT2, base);
base = X64_TMP_INT2;
@@ -3321,10 +3647,7 @@ static void x64_fence(NativeTarget* t, KitCgMemOrder mo) {
* object. */
/* Resolve a va_list address into `scratch`, materializing it there if it is not
- * already, so the va field-value scratch registers (rax / r10 / rdx) never
- * alias it. At -O1 the optimizer may place the va_list pointer in any register
- * — including those — and the va code would then clobber the pointer
- * mid-sequence. */
+ * already, so the backend-private field-value register never aliases it. */
static u32 x64_va_base(X64NativeTarget* a, NativeAddr ap, u32 scratch) {
u32 base = x64_addr_to_base_reg(a, ap, scratch);
if (base != scratch) {
@@ -3364,24 +3687,26 @@ static void x64_va_start_core(X64NativeTarget* a, NativeAddr ap) {
if (a->abi->shadow_space) {
/* Win64: *ap = rbp + 16 + named_int*8 + named_stack. */
u32 first = 16u + a->next_param_int * 8u + a->next_param_stack;
- emit_lea(mc, X64_RAX, X64_RBP, (i32)first);
- emit_mov_store(mc, 8, X64_RAX, ap_base, 0);
+ emit_lea(mc, X64_TMP_INT, X64_RBP, (i32)first);
+ emit_mov_store(mc, 8, X64_TMP_INT, ap_base, 0);
return;
}
{
X64NativeSlot* rs = x64_slot_get(a, a->reg_save_slot);
/* gp_offset = next_param_int * 8 */
- x64_emit_load_imm(mc, 0, X64_RAX, (i64)(a->next_param_int * 8u));
- emit_mov_store(mc, 4, X64_RAX, ap_base, 0);
+ x64_emit_load_imm(mc, 0, X64_TMP_INT, (i64)(a->next_param_int * 8u));
+ emit_mov_store(mc, 4, X64_TMP_INT, ap_base, 0);
/* fp_offset = 48 + next_param_fp * 16 */
- x64_emit_load_imm(mc, 0, X64_RAX, (i64)(48u + a->next_param_fp * 16u));
- emit_mov_store(mc, 4, X64_RAX, ap_base, 4);
+ x64_emit_load_imm(mc, 0, X64_TMP_INT,
+ (i64)(48u + a->next_param_fp * 16u));
+ emit_mov_store(mc, 4, X64_TMP_INT, ap_base, 4);
/* overflow_arg_area = rbp + 16 + next_param_stack */
- emit_lea(mc, X64_RAX, X64_RBP, (i32)(16u + a->next_param_stack));
- emit_mov_store(mc, 8, X64_RAX, ap_base, 8);
+ emit_lea(mc, X64_TMP_INT, X64_RBP,
+ (i32)(16u + a->next_param_stack));
+ emit_mov_store(mc, 8, X64_TMP_INT, ap_base, 8);
/* reg_save_area = rbp - reg_save_slot.off */
- emit_lea(mc, X64_RAX, X64_RBP, -(i32)rs->off);
- emit_mov_store(mc, 8, X64_RAX, ap_base, 16);
+ emit_lea(mc, X64_TMP_INT, X64_RBP, -(i32)rs->off);
+ emit_mov_store(mc, 8, X64_TMP_INT, ap_base, 16);
}
}
@@ -3393,15 +3718,15 @@ static void x64_va_arg_core(X64NativeTarget* a, NativeLoc dst, NativeAddr ap,
int is_fp = native_loc_is_fp(dst);
u32 dr = loc_reg(dst);
u32 ap_base = x64_va_base(a, ap, X64_TMP_INT2); /* r11 */
- /* GPR scratch for the offset/address arithmetic. For integer results the
- * destination is itself a (throwaway) scratch GPR — pass_native_emit fetches
- * va_arg into a scratch and copies it to the real destination afterward — so
+ /* GPR temporary for the offset/address arithmetic. For integer results the
+ * destination is itself a throwaway operand temp — pass_native_emit fetches
+ * va_arg into a scoped register and writes it to the real destination — so
* we reuse `dr` and touch no allocable register at all. FP results keep their
- * value in an XMM register, so they borrow the reserved RAX emit scratch.
+ * value in an XMM register, so they borrow backend-private R10.
* Either way only r11 (ap_base) and `gp` are used: the va_list fields are
* advanced in memory (x64_add_mem_imm) and the reg-save base is folded in
* with x64_add_reg_mem, so no third register is needed. */
- u32 gp = is_fp ? X64_RAX : dr;
+ u32 gp = is_fp ? X64_TMP_INT : dr;
if (a->abi->shadow_space) {
/* Win64: gp = *ap; load dr from [gp]; *ap += 8. */
emit_mov_load(mc, 8, 0, gp, ap_base, 0);
@@ -3452,8 +3777,9 @@ static void x64_va_copy_core(X64NativeTarget* a, NativeAddr dst_ap,
NativeAddr src_ap) {
NativeTarget* t = &a->base;
MCEmitter* mc = t->mc;
- /* Resolve dst into r11, src into rax, and copy each qword through the fp emit
- * scratch xmm14. Uses only reserved emit scratch (r11/rax/xmm14), so the
+ /* Resolve dst into R11, src into R10, and copy each qword through
+ * backend-private XMM14. Uses only reserved backend registers
+ * (R11/R10/XMM14), so the
* optimizer's register choice for a va_list pointer can never be clobbered
* and no allocable GPR (previously rdx) is consumed. */
u32 dst_base = x64_va_base(a, dst_ap, X64_TMP_INT2);
@@ -3770,23 +4096,23 @@ static void x64_intrinsic(NativeTarget* t, IntrinKind kind,
i = n; /* copy high-to-low so an overlapping dst > src is safe */
while (i >= 8u) {
i -= 8u;
- emit_mov_load(mc, 8, 0, X64_RAX, sr, (i32)i);
- emit_mov_store(mc, 8, X64_RAX, dr, (i32)i);
+ emit_mov_load(mc, 8, 0, X64_TMP_INT, sr, (i32)i);
+ emit_mov_store(mc, 8, X64_TMP_INT, dr, (i32)i);
}
while (i >= 4u) {
i -= 4u;
- emit_mov_load(mc, 4, 0, X64_RAX, sr, (i32)i);
- emit_mov_store(mc, 4, X64_RAX, dr, (i32)i);
+ emit_mov_load(mc, 4, 0, X64_TMP_INT, sr, (i32)i);
+ emit_mov_store(mc, 4, X64_TMP_INT, dr, (i32)i);
}
while (i >= 2u) {
i -= 2u;
- emit_mov_load(mc, 2, 0, X64_RAX, sr, (i32)i);
- emit_mov_store(mc, 2, X64_RAX, dr, (i32)i);
+ emit_mov_load(mc, 2, 0, X64_TMP_INT, sr, (i32)i);
+ emit_mov_store(mc, 2, X64_TMP_INT, dr, (i32)i);
}
while (i >= 1u) {
i -= 1u;
- emit_mov_load(mc, 1, 0, X64_RAX, sr, (i32)i);
- emit_mov_store(mc, 1, X64_RAX, dr, (i32)i);
+ emit_mov_load(mc, 1, 0, X64_TMP_INT, sr, (i32)i);
+ emit_mov_store(mc, 1, X64_TMP_INT, dr, (i32)i);
}
return;
}
@@ -4110,7 +4436,11 @@ static X64AsmSavedClobber* x64_asm_save_callee_clobbers(X64NativeTarget* a,
return saved;
}
-/* ---- NativeTarget (optimizer) asm hook ---- */
+/* ---- NativeTarget (optimizer) asm hook ----
+ *
+ * Register placement, staging, and output writeback belong to the optimized
+ * emitter. This hook binds those concrete locations and only materializes
+ * memory-constraint bases in backend-private registers. */
static NativeAddr x64_asm_loc_to_addr(X64NativeTarget* a, SrcLoc loc,
NativeLoc src) {
@@ -4157,50 +4487,6 @@ static Reg x64_asm_native_mem_base(X64NativeTarget* a, SrcLoc loc,
return dst;
}
-static Reg x64_asm_stage_reg(X64NativeTarget* a, SrcLoc loc,
- NativeAllocClass cls, u32* nint, u32* nfp) {
- static const Reg int_regs[] = {X64_TMP_INT, X64_TMP_INT2};
- static const Reg fp_regs[] = {X64_TMP_FP, X64_TMP_FP2};
- if (cls == NATIVE_REG_FP) {
- if (*nfp >= (u32)(sizeof fp_regs / sizeof fp_regs[0]))
- x64_asm_panic_at(a->base.c, loc, "too many staged fp asm operands");
- return fp_regs[(*nfp)++];
- }
- if (*nint >= (u32)(sizeof int_regs / sizeof int_regs[0]))
- x64_asm_panic_at(a->base.c, loc, "too many staged integer asm operands");
- return int_regs[(*nint)++];
-}
-
-static void x64_asm_load_loc_to_reg(X64NativeTarget* a, SrcLoc loc,
- NativeLoc src, NativeLoc dst) {
- NativeTarget* t = &a->base;
- NativeAllocClass cls = (NativeAllocClass)dst.cls;
- if (src.kind == NATIVE_LOC_REG) {
- if (src.v.reg != dst.v.reg || src.cls != dst.cls) t->move(t, dst, src);
- return;
- }
- if (src.kind == NATIVE_LOC_IMM) {
- if (cls != NATIVE_REG_INT)
- x64_asm_panic_at(t->c, loc,
- "floating-point immediate asm input is unsupported");
- t->load_imm(t, dst, src.v.imm);
- return;
- }
- x64_emit_mem(a, 1, dst, x64_asm_loc_to_addr(a, loc, src),
- native_mem_for_type(t, dst.type, loc_size32(t, dst)));
-}
-
-static void x64_asm_store_reg_to_loc(X64NativeTarget* a, SrcLoc loc,
- NativeLoc dst, NativeLoc src) {
- NativeTarget* t = &a->base;
- if (dst.kind == NATIVE_LOC_REG) {
- if (dst.v.reg != src.v.reg || dst.cls != src.cls) t->move(t, dst, src);
- return;
- }
- x64_emit_mem(a, 0, src, x64_asm_loc_to_addr(a, loc, dst),
- native_mem_for_type(t, src.type, loc_size32(t, src)));
-}
-
static void x64_asm_native_panic(NativeTarget* t, SrcLoc loc,
const char* msg) {
x64_asm_panic_at(t->c, loc, msg);
@@ -4211,23 +4497,6 @@ static Reg x64_asm_native_mem_base_hook(NativeTarget* t, SrcLoc loc,
return x64_asm_native_mem_base(x64_of(t), loc, src, ntmp);
}
-static Reg x64_asm_native_stage_reg_hook(NativeTarget* t, SrcLoc loc,
- NativeAllocClass cls, u32* ntmp,
- u32* nstage_int, u32* nstage_fp) {
- (void)nstage_int;
- return x64_asm_stage_reg(x64_of(t), loc, cls, ntmp, nstage_fp);
-}
-
-static void x64_asm_native_load_loc_hook(NativeTarget* t, SrcLoc loc,
- NativeLoc src, NativeLoc dst) {
- x64_asm_load_loc_to_reg(x64_of(t), loc, src, dst);
-}
-
-static void x64_asm_native_store_loc_hook(NativeTarget* t, SrcLoc loc,
- NativeLoc dst, NativeLoc src) {
- x64_asm_store_reg_to_loc(x64_of(t), loc, dst, src);
-}
-
static void x64_asm_native_run_template_hook(
NativeTarget* t, const char* tmpl, const AsmConstraint* outs, u32 nout,
Operand* bound_outs, const AsmConstraint* ins, u32 nin, Operand* bound_ins,
@@ -4247,15 +4516,10 @@ static void x64_asm_block_native(NativeTarget* t, const char* tmpl,
X64NativeTarget* a = x64_of(t);
SrcLoc loc = a->func ? a->func->loc : (SrcLoc){0, 0, 0};
static const NativeAsmNativeHooks hooks = {
- .opk_reg = X64_INLINE_OPK_REG,
- .opcls_fp = X64_INLINE_OPCLS_FP,
.panic = x64_asm_native_panic,
.bound_reg = x64_asm_bound_reg,
.bound_mem = x64_asm_bound_mem,
.mem_base = x64_asm_native_mem_base_hook,
- .stage_reg = x64_asm_native_stage_reg_hook,
- .load_loc_to_reg = x64_asm_native_load_loc_hook,
- .store_reg_to_loc = x64_asm_native_store_loc_hook,
.run_template = x64_asm_native_run_template_hook,
};
native_asm_bind_native_operands(t, loc, tmpl, outs, nout, out_locs, ins, nin,
@@ -4274,9 +4538,9 @@ static void x64_set_loc(NativeTarget* t, SrcLoc loc) {
* effect, so the optimizer keeps values live across them out of those registers
* (the backend is then free to use them). idiv/div write rax (quotient) and rdx
* (remainder/sign); a variable shift uses cl; cmpxchg/xadd loops use
- * rax/rcx/rdx; an FP va_arg borrows rax for the gp/fp offset (an integer va_arg
- * does the offset arithmetic in its own destination register, so it clobbers
- * nothing). */
+ * rax/rcx/rdx; bitfield store uses the same trio for its read-modify-write.
+ * All other ordinary helpers use declared results or backend-private R10/R11
+ * and XMM14/XMM15 temporaries. */
static int x64_machine_op_clobbers(NativeTarget* t, const NativeMachineOp* op,
u32 mask[NATIVE_CALL_PLAN_CLASSES]) {
mask[0] = mask[1] = mask[2] = 0;
@@ -4310,22 +4574,21 @@ static int x64_machine_op_clobbers(NativeTarget* t, const NativeMachineOp* op,
default:
return 0;
}
+ case NATIVE_MOP_BITFIELD_LOAD:
+ /* The only non-destination temporary is backend-private R11. */
+ return 0;
+ case NATIVE_MOP_BITFIELD_STORE:
+ mask[NATIVE_REG_INT] =
+ (1u << X64_RAX) | (1u << X64_RCX) | (1u << X64_RDX);
+ return 1;
case NATIVE_MOP_ATOMIC_CAS:
case NATIVE_MOP_ATOMIC_RMW:
mask[NATIVE_REG_INT] =
(1u << X64_RAX) | (1u << X64_RCX) | (1u << X64_RDX);
return 1;
case NATIVE_MOP_VA_START:
- /* x64_va_start_core materializes the va_list field values through RAX
- * (the ap pointer itself lands in the reserved r11 scratch). RAX is the
- * return register, so the allocator may otherwise keep a live value there
- * across the op. */
- mask[NATIVE_REG_INT] = (1u << X64_RAX);
- return 1;
case NATIVE_MOP_VA_ARG:
- if (!op->result_is_fp) return 0;
- mask[NATIVE_REG_INT] = (1u << X64_RAX);
- return 1;
+ return 0;
case NATIVE_MOP_INTRINSIC:
/* The unsigned multiply-overflow intrinsic emits a one-operand MUL, whose
* rdx:rax product clobbers both registers. The signed variant uses a
@@ -4342,10 +4605,13 @@ static int x64_machine_op_clobbers(NativeTarget* t, const NativeMachineOp* op,
return 1;
}
if ((IntrinKind)op->intrin == INTRIN_SYSCALL) {
+ /* R8/R9 are cache-capable emitter temps, so the effect must invalidate
+ * retained spill values there even though they are not MIR homes.
+ * R10/R11 are pure backend-private temps and remain excluded. */
mask[NATIVE_REG_INT] =
(1u << X64_RAX) | (1u << X64_RCX) | (1u << X64_RDX) |
(1u << X64_RSI) | (1u << X64_RDI) | (1u << X64_R8) |
- (1u << X64_R9) | (1u << X64_R10) | (1u << X64_R11);
+ (1u << X64_R9);
return 1;
}
return 0;
@@ -4570,13 +4836,14 @@ static void x64_direct_asm_block(NativeDirectTarget* d, const char* tmpl,
const Sym* clobbers, u32 nclob,
u32 clobber_abi_sets) {
static const NativeAsmDirectHooks hooks = {
- /* Reserve emit scratch (r10,r11), driver scratch (r8,r9), rax (reserved;
- * only self-allocated here when explicitly pinned), and sp/bp. */
- .scratch_int = (1u << X64_RAX) | (1u << X64_R11) | (1u << X64_RSP) |
- (1u << X64_RBP) | (1u << X64_R8) | (1u << X64_R9) |
- (1u << X64_R10),
- .scratch_fp = (1u << X64_XMM4) | (1u << X64_XMM5) |
- (1u << (X64_XMM0 + 14)) | (1u << X64_XMM15),
+ /* Reserve backend temps R10/R11, O0 scratch R8/R9, RAX (reserved; only
+ * self-allocated here when explicitly pinned), and SP/BP. */
+ .reserved_int_mask =
+ (1u << X64_RAX) | (1u << X64_R11) | (1u << X64_RSP) |
+ (1u << X64_RBP) | (1u << X64_R8) | (1u << X64_R9) |
+ (1u << X64_R10),
+ .reserved_fp_mask = (1u << X64_XMM4) | (1u << X64_XMM5) |
+ (1u << (X64_XMM0 + 14)) | (1u << X64_XMM15),
.opk_reg = X64_INLINE_OPK_REG,
.opcls_fp = X64_INLINE_OPCLS_FP,
.panic = x64_asm_panic,
diff --git a/src/build/trace.c b/src/build/trace.c
@@ -333,8 +333,10 @@ static int emit_child_row(KitWriter* out, const uint8_t id[BUILD_HASH_LEN]) {
return emit_bytes(out, "\n");
}
-static int emit_sorted_children(const uint8_t (*rows)[BUILD_HASH_LEN], size_t n,
- KitWriter* out, char* err, size_t errcap) {
+/* Use a flat byte view here: strict C11 cannot add const through a
+ * pointer-to-array conversion (that becomes well-defined only in C23). */
+static int emit_sorted_children(const uint8_t* rows, size_t n, KitWriter* out,
+ char* err, size_t errcap) {
const uint8_t* prev = NULL;
size_t emitted = 0;
if (n && !rows) return set_err(err, errcap, "missing child rows");
@@ -343,7 +345,7 @@ static int emit_sorted_children(const uint8_t (*rows)[BUILD_HASH_LEN], size_t n,
const uint8_t* best = NULL;
size_t i;
for (i = 0; i < n; ++i) {
- const uint8_t* cur = rows[i];
+ const uint8_t* cur = rows + i * BUILD_HASH_LEN;
int after_prev = !prev || id_cmp(prev, cur) < 0;
if (after_prev && (!best || id_cmp(cur, best) < 0)) best = cur;
}
@@ -486,8 +488,8 @@ int build_deepset_emit(const BuildDeepSet* t, KitWriter* out, char* err,
if (emit_sorted_blobs(t->blobs, t->n_blobs, out, err, errcap) !=
BUILD_OK)
return BUILD_ERR;
- if (emit_sorted_children(t->children, t->n_children, out, err, errcap) !=
- BUILD_OK)
+ if (emit_sorted_children(t->children ? t->children[0] : NULL, t->n_children,
+ out, err, errcap) != BUILD_OK)
return BUILD_ERR;
return kit_writer_status(out) == KIT_OK ? BUILD_OK : BUILD_ERR;
}
diff --git a/src/cg/call.c b/src/cg/call.c
@@ -84,9 +84,10 @@ void api_pack_call_arg(KitCg* g, CGLocal* out, KitCgTypeId fty, u32 idx) {
* sub-array so the LIFO pop yields arg0 first, then pop+materialize forward.
*
* Why forward: an argument materialized into a register lands in the first free
- * -O0 cache register, and the cache pool front-loads the ABI argument registers
- * (x0,x1,.. — see aa_int_allocable). Producing arg0 first therefore lands it in
- * x0, arg1 in x1, .. so the per-call parallel-copy collapses to no-ops, instead
+ * -O0 cache register, and the target's `ndt_allocable` order front-loads the
+ * ABI argument registers (x0,x1,.. on aa64). Producing arg0 first therefore
+ * lands it in x0, arg1 in x1, .. so the per-call parallel-copy collapses to
+ * no-ops, instead
* of the reverse order leaving each value in the *wrong* arg register and
* forcing a permutation shuffle. Matches tcc's eager left-to-right argument
* emission.
diff --git a/src/cg/cgir.h b/src/cg/cgir.h
@@ -267,7 +267,7 @@ typedef enum MemFlag {
* the whole register) instead of a zero-extending load plus a CV_SEXT convert.
* Non-bitfield integer loads only. A backend that does not honor it ignores
* the bit and emits a plain load; the shared -O0 NDT only elides the following
- * CV_SEXT when the backend advertises NativeRegInfo.ndt_load_sext, so leaving
+ * CV_SEXT when the backend advertises NativeRegInfo.load_sext, so leaving
* the convert in place stays correct there. */
MF_SEXT_LOAD = 1u << 6,
} MemFlag;
diff --git a/src/cg/native_asm.c b/src/cg/native_asm.c
@@ -158,28 +158,6 @@ NativeAsmRegPinStatus native_asm_resolve_pin(NativeTarget* t, Sym reg,
return NATIVE_ASM_REG_PIN_OK;
}
-NativeAsmPinnedLoc native_asm_prepare_pinned_loc(NativeTarget* t, Sym reg,
- const char* constraint,
- KitCgTypeId type,
- NativeLoc loc) {
- NativeAsmPinnedLoc out;
- NativeAsmRegPin pin;
- memset(&out, 0, sizeof out);
- out.loc = loc;
- out.pin_status = native_asm_resolve_pin(t, reg, constraint, &pin);
- if (out.pin_status == NATIVE_ASM_REG_PIN_ABSENT) return out;
- out.has_pin = 1u;
- if (out.pin_status != NATIVE_ASM_REG_PIN_OK) return out;
- if (loc.kind != NATIVE_LOC_REG) {
- out.loc = native_loc_reg(type, pin.cls, pin.reg);
- out.needs_stage = 1u;
- return out;
- }
- if ((Reg)loc.v.reg != pin.reg || (NativeAllocClass)loc.cls != pin.cls)
- out.wrong_reg = 1u;
- return out;
-}
-
const char* native_asm_pin_status_message(NativeAsmRegPinStatus st) {
switch (st) {
case NATIVE_ASM_REG_PIN_ABSENT:
@@ -234,8 +212,8 @@ void native_asm_bind_direct_operands(NativeDirectTarget* d, const char* tmpl,
native_asm_abi_clobber_masks(d->native, clobber_abi_sets, &abi_int, &abi_fp);
clob_int |= abi_int;
clob_fp |= abi_fp;
- used_int = clob_int | h->scratch_int;
- used_fp = clob_fp | h->scratch_fp;
+ used_int = clob_int | h->reserved_int_mask;
+ used_fp = clob_fp | h->reserved_fp_mask;
for (i = 0; i < nout; ++i) {
const char* body = native_asm_constraint_body(outs[i].str);
@@ -260,14 +238,19 @@ void native_asm_bind_direct_operands(NativeDirectTarget* d, const char* tmpl,
&used_fp);
if (info.cls == NATIVE_REG_FP) {
used_fp |= 1u << reg;
- if (info.fixed_reg != REG_NONE) clob_fp |= 1u << reg;
+ /* Binding itself overwrites the staging register, independently of
+ * whether the template names it as a clobber. Include every bound
+ * register in the preservation set so a target may safely draw a
+ * high-arity direct-asm operand from its callee-saved pool. */
+ clob_fp |= 1u << reg;
} else {
used_int |= 1u << reg;
- if (info.fixed_reg != REG_NONE) clob_int |= 1u << reg;
+ clob_int |= 1u << reg;
}
h->bound_reg(&bound_outs[i], type, info.cls, reg);
} else if (body[0] == 'm') {
Reg reg = h->alloc_reg(d, NATIVE_REG_INT, 0, &used_int, &used_fp);
+ clob_int |= 1u << reg;
h->bound_mem(&bound_outs[i], type, reg);
} else {
h->panic(d, "unsupported output constraint");
@@ -308,10 +291,10 @@ void native_asm_bind_direct_operands(NativeDirectTarget* d, const char* tmpl,
&used_fp);
if (info.cls == NATIVE_REG_FP) {
used_fp |= 1u << reg;
- if (info.fixed_reg != REG_NONE) clob_fp |= 1u << reg;
+ clob_fp |= 1u << reg;
} else {
used_int |= 1u << reg;
- if (info.fixed_reg != REG_NONE) clob_int |= 1u << reg;
+ clob_int |= 1u << reg;
}
h->bound_reg(&bound_ins[i], type, info.cls, reg);
} else if (body[0] == 'i') {
@@ -320,6 +303,7 @@ void native_asm_bind_direct_operands(NativeDirectTarget* d, const char* tmpl,
bound_ins[i] = in_ops[i];
} else if (body[0] == 'm') {
Reg reg = h->alloc_reg(d, NATIVE_REG_INT, 0, &used_int, &used_fp);
+ clob_int |= 1u << reg;
h->bound_mem(&bound_ins[i], type, reg);
} else {
h->panic(d, "unsupported input constraint");
@@ -373,16 +357,32 @@ void native_asm_bind_direct_operands(NativeDirectTarget* d, const char* tmpl,
for (i = nsaved; i > 0; --i) h->restore_one(d, saved, i - 1u);
}
-static void native_asm_bind_native_one(NativeTarget* t, SrcLoc loc,
- Operand* out, const char* constraint,
- KitCgTypeId type, NativeLoc src,
- u32* ntmp,
- const NativeAsmNativeHooks* h) {
- const char* body = native_asm_constraint_body(constraint);
+static void native_asm_validate_native_pin(
+ NativeTarget* t, SrcLoc loc, const AsmConstraint* constraint, NativeLoc src,
+ const NativeAsmNativeHooks* h) {
+ NativeAsmRegPin pin;
+ NativeAsmRegPinStatus status;
+ if (!constraint->reg) return;
+ status = native_asm_resolve_pin(t, constraint->reg, constraint->str, &pin);
+ if (status != NATIVE_ASM_REG_PIN_OK)
+ h->panic(t, loc, native_asm_pin_status_message(status));
+ if (src.kind != NATIVE_LOC_REG || src.cls != (u8)pin.cls ||
+ src.v.reg != pin.reg)
+ h->panic(t, loc, "hard-register asm operand in wrong register");
+}
+
+static void native_asm_bind_native_one(
+ NativeTarget* t, SrcLoc loc, Operand* out,
+ const AsmConstraint* constraint, KitCgTypeId type, NativeLoc src,
+ u32* ntmp, const NativeAsmNativeHooks* h) {
+ const char* body = native_asm_constraint_body(constraint->str);
NativeAsmConstraintInfo info;
- if (native_asm_constraint_reg_info(t, constraint, &info)) {
+ native_asm_validate_native_pin(t, loc, constraint, src, h);
+ if (native_asm_constraint_reg_info(t, constraint->str, &info)) {
if (src.kind != NATIVE_LOC_REG)
h->panic(t, loc, "register asm operand not in a register");
+ if (src.cls != (u8)info.cls)
+ h->panic(t, loc, "register asm operand has wrong register class");
if (info.fixed_reg != REG_NONE && info.fixed_reg != (Reg)src.v.reg)
h->panic(t, loc, "fixed-register asm operand in wrong register");
if (info.allowed_mask &&
@@ -412,37 +412,12 @@ void native_asm_bind_native_operands(
Compiler* c = t->c;
Operand* bound_outs = nout ? arena_zarray(c->tu, Operand, nout) : NULL;
Operand* bound_ins = nin ? arena_zarray(c->tu, Operand, nin) : NULL;
- u8* staged_outs = nout ? arena_zarray(c->tu, u8, nout) : NULL;
- u32 ntmp = 0, nstage_int = 0, nstage_fp = 0;
+ u32 ntmp = 0;
for (u32 i = 0; i < nout; ++i) {
KitCgTypeId type = outs[i].type ? outs[i].type : out_locs[i].type;
- NativeLoc outloc = out_locs[i];
- NativeAsmConstraintInfo info;
- NativeAsmPinnedLoc pinned =
- native_asm_prepare_pinned_loc(t, outs[i].reg, outs[i].str, type, outloc);
- if (pinned.has_pin) {
- if (pinned.pin_status != NATIVE_ASM_REG_PIN_OK)
- h->panic(t, loc, native_asm_pin_status_message(pinned.pin_status));
- if (pinned.wrong_reg)
- h->panic(t, loc, "hard-register asm operand in wrong register");
- outloc = pinned.loc;
- if (pinned.needs_stage) {
- staged_outs[i] = 1u;
- if (outs[i].dir == KIT_CG_ASM_INOUT)
- h->load_loc_to_reg(t, loc, out_locs[i], outloc);
- }
- } else if (native_asm_constraint_reg_info(t, outs[i].str, &info) &&
- info.allowed_mask == 0 && outloc.kind != NATIVE_LOC_REG) {
- Reg r =
- h->stage_reg(t, loc, info.cls, &ntmp, &nstage_int, &nstage_fp);
- outloc = native_loc_reg(type, info.cls, r);
- staged_outs[i] = 1u;
- if (outs[i].dir == KIT_CG_ASM_INOUT)
- h->load_loc_to_reg(t, loc, out_locs[i], outloc);
- }
- native_asm_bind_native_one(t, loc, &bound_outs[i], outs[i].str, type,
- outloc, &ntmp, h);
+ native_asm_bind_native_one(t, loc, &bound_outs[i], &outs[i], type,
+ out_locs[i], &ntmp, h);
}
for (u32 i = 0; i < nin; ++i) {
@@ -455,45 +430,20 @@ void native_asm_bind_native_operands(
h->panic(t, loc, "matching constraint out of range");
if (native_asm_constraint_early(outs[matched].str))
h->panic(t, loc, "matching input names early-clobber output");
+ if (out_locs[matched].kind != NATIVE_LOC_REG ||
+ in_locs[i].kind != NATIVE_LOC_REG ||
+ out_locs[matched].cls != in_locs[i].cls ||
+ out_locs[matched].v.reg != in_locs[i].v.reg)
+ h->panic(t, loc, "matching asm operands are in different registers");
bound_ins[i] = bound_outs[matched];
continue;
}
type = ins[i].type ? ins[i].type : in_locs[i].type;
inloc = in_locs[i];
- {
- NativeAsmConstraintInfo info;
- NativeAsmPinnedLoc pinned =
- native_asm_prepare_pinned_loc(t, ins[i].reg, ins[i].str, type, inloc);
- if (pinned.has_pin) {
- if (pinned.pin_status != NATIVE_ASM_REG_PIN_OK)
- h->panic(t, loc, native_asm_pin_status_message(pinned.pin_status));
- if (pinned.wrong_reg)
- h->panic(t, loc, "hard-register asm operand in wrong register");
- inloc = pinned.loc;
- if (pinned.needs_stage)
- h->load_loc_to_reg(t, loc, in_locs[i], inloc);
- } else if (native_asm_constraint_reg_info(t, ins[i].str, &info) &&
- info.allowed_mask == 0 && inloc.kind != NATIVE_LOC_REG) {
- Reg r =
- h->stage_reg(t, loc, info.cls, &ntmp, &nstage_int, &nstage_fp);
- inloc = native_loc_reg(type, info.cls, r);
- h->load_loc_to_reg(t, loc, in_locs[i], inloc);
- }
- }
- native_asm_bind_native_one(t, loc, &bound_ins[i], ins[i].str, type, inloc,
+ native_asm_bind_native_one(t, loc, &bound_ins[i], &ins[i], type, inloc,
&ntmp, h);
}
h->run_template(t, tmpl, outs, nout, bound_outs, ins, nin, bound_ins, clobbers,
nclob);
-
- for (u32 i = 0; i < nout; ++i) {
- NativeAllocClass cls;
- NativeLoc src;
- if (!staged_outs || !staged_outs[i]) continue;
- if (bound_outs[i].kind != h->opk_reg) continue;
- cls = bound_outs[i].pad[0] == h->opcls_fp ? NATIVE_REG_FP : NATIVE_REG_INT;
- src = native_loc_reg(bound_outs[i].type, cls, (Reg)bound_outs[i].v.local);
- h->store_reg_to_loc(t, loc, out_locs[i], src);
- }
}
diff --git a/src/cg/native_asm.h b/src/cg/native_asm.h
@@ -64,14 +64,6 @@ typedef struct NativeAsmConstraintInfo {
u32 allowed_mask; /* 0 means any valid register in cls. */
} NativeAsmConstraintInfo;
-typedef struct NativeAsmPinnedLoc {
- NativeLoc loc;
- NativeAsmRegPinStatus pin_status;
- u8 has_pin;
- u8 needs_stage;
- u8 wrong_reg;
-} NativeAsmPinnedLoc;
-
int native_asm_constraint_reg_info(NativeTarget* t, const char* constraint,
NativeAsmConstraintInfo* out);
int native_asm_constraint_is_reg(NativeTarget* t, const char* constraint);
@@ -83,10 +75,6 @@ int native_asm_constraint_is_reg(NativeTarget* t, const char* constraint);
NativeAsmRegPinStatus native_asm_resolve_pin(NativeTarget* t, Sym reg,
const char* constraint,
NativeAsmRegPin* out);
-NativeAsmPinnedLoc native_asm_prepare_pinned_loc(NativeTarget* t, Sym reg,
- const char* constraint,
- KitCgTypeId type,
- NativeLoc loc);
const char* native_asm_pin_status_message(NativeAsmRegPinStatus st);
int native_asm_constraint_reg_class(const char* constraint,
NativeAllocClass* cls_out);
@@ -108,11 +96,11 @@ int native_asm_constraint_reg_class(const char* constraint,
typedef struct NativeDirectTarget NativeDirectTarget;
typedef struct NativeAsmDirectHooks {
- /* Registers the operand allocator must never hand out (emit scratch, ABI
- * fixed regs, frame pointer, ...), seeded into the used masks alongside the
- * clobber masks. Per-arch. */
- u32 scratch_int;
- u32 scratch_fp;
+ /* Registers the O0 direct-asm operand allocator must never hand out
+ * (backend-private temporaries, ABI fixed regs, frame pointer, ...), seeded
+ * into the used masks alongside the clobber masks. Per-arch. */
+ u32 reserved_int_mask;
+ u32 reserved_fp_mask;
/* Arch inline-operand discriminators: Operand.kind for a bound register
* pseudo-operand, and the Operand.pad[0] value marking the FP class. */
u8 opk_reg;
@@ -134,9 +122,10 @@ typedef struct NativeAsmDirectHooks {
* naming). */
void (*clobber_masks)(Compiler* c, SrcLoc loc, const Sym* clobbers, u32 nclob,
u32* int_mask, u32* fp_mask);
- /* Spill the callee-saved registers the asm clobbers, returning an opaque,
- * block-lived save record plus its element count. restore_one undoes element
- * idx; the driver replays them in reverse. */
+ /* Spill the callee-saved registers the asm clobbers or the direct binder
+ * uses for operand/address staging, returning an opaque, block-lived save
+ * record plus its element count. restore_one undoes element idx; the driver
+ * replays them in reverse. */
void* (*save_callee_clobbers)(NativeDirectTarget* d, u32 int_mask,
u32 fp_mask, u32* nsaved_out);
void (*restore_one)(NativeDirectTarget* d, void* saved, u32 idx);
@@ -168,28 +157,17 @@ void native_asm_bind_direct_operands(NativeDirectTarget* d, const char* tmpl,
/* ---- Optimized NativeTarget inline-asm binding ----
*
- * The optimizer allocates the operand locations before final native emission.
- * This shared driver validates those locations against the same
- * NativeAsmConstraintInfo used by regalloc, stages frame-resident unrestricted
- * register operands through reserved scratch registers, binds memory operands to
- * concrete base registers, runs the arch assembler, and stores staged outputs
- * back. */
+ * The optimizer's native emitter owns register placement, staging, and output
+ * writeback. This shared driver is deliberately narrower: it validates and
+ * binds the concrete register/immediate locations supplied by the emitter,
+ * materializes memory-operand bases through the backend, and runs the arch
+ * assembler. */
typedef struct NativeAsmNativeHooks {
- u8 opk_reg;
- u8 opcls_fp;
- u8 pad[2];
-
void (*panic)(NativeTarget* t, SrcLoc loc, const char* msg);
void (*bound_reg)(Operand* out, KitCgTypeId type, NativeAllocClass cls,
Reg reg);
void (*bound_mem)(Operand* out, KitCgTypeId type, Reg base);
Reg (*mem_base)(NativeTarget* t, SrcLoc loc, NativeLoc src, u32* ntmp);
- Reg (*stage_reg)(NativeTarget* t, SrcLoc loc, NativeAllocClass cls, u32* ntmp,
- u32* nstage_int, u32* nstage_fp);
- void (*load_loc_to_reg)(NativeTarget* t, SrcLoc loc, NativeLoc src,
- NativeLoc dst);
- void (*store_reg_to_loc)(NativeTarget* t, SrcLoc loc, NativeLoc dst,
- NativeLoc src);
void (*run_template)(NativeTarget* t, const char* tmpl,
const AsmConstraint* outs, u32 nout, Operand* bound_outs,
const AsmConstraint* ins, u32 nin, Operand* bound_ins,
diff --git a/src/cg/native_direct_target.c b/src/cg/native_direct_target.c
@@ -186,8 +186,8 @@ static u32 nd_acquirable_mask(const NativeAllocClassInfo* ci) {
u32 m = 0u, i;
for (i = 0; i < ci->nscratch; ++i)
if (ci->scratch[i] < 32u) m |= 1u << ci->scratch[i];
- for (i = 0; i < ci->nallocable; ++i)
- if (ci->allocable[i] < 32u) m |= 1u << ci->allocable[i];
+ for (i = 0; i < ci->ndt_allocable_count; ++i)
+ if (ci->ndt_allocable[i] < 32u) m |= 1u << ci->ndt_allocable[i];
return m;
}
@@ -248,8 +248,8 @@ static Reg nd_scratch_acquire_span(NativeDirectTarget* d, NativeAllocClass cls,
}
return r;
}
- regs = ci->allocable;
- nregs = ci->nallocable;
+ regs = ci->ndt_allocable;
+ nregs = ci->ndt_allocable_count;
}
/* Under pressure, evict the LRU non-pinned cached local (spilling it to its
* home) and reuse its register as a scratch temporary. A wide run evicts every
@@ -263,8 +263,8 @@ static Reg nd_scratch_acquire_span(NativeDirectTarget* d, NativeAllocClass cls,
return r;
}
} else {
- const Reg* alloc = ci->allocable;
- for (u32 i = 0; i < ci->nallocable; ++i) {
+ const Reg* alloc = ci->ndt_allocable;
+ for (u32 i = 0; i < ci->ndt_allocable_count; ++i) {
Reg r = alloc[i];
u32 k;
int run_ok = 1;
@@ -697,8 +697,8 @@ static Reg nd_pick_cache_victim(NativeDirectTarget* d, NativeAllocClass cls) {
const NativeAllocClassInfo* ci = nd_class_info(d, cls);
Reg best = REG_NONE;
u32 best_use = 0;
- for (u32 i = 0; i < ci->nallocable; ++i) {
- Reg r = ci->allocable[i];
+ for (u32 i = 0; i < ci->ndt_allocable_count; ++i) {
+ Reg r = ci->ndt_allocable[i];
u32 use;
if (r >= 32u) continue;
if (d->reg_owner[cls][r] == CG_LOCAL_NONE) continue;
@@ -1786,7 +1786,7 @@ static void nd_load(CgTarget* t, Operand dst, Operand addr, MemAccess mem) {
d->native->load(d->native, reg, naddr, mem);
nd_dst_writeback(d, dst, reg);
/* A narrow integer load fills the whole register: a plain ldrb/ldrh/ldr-w
- * zero-extends, and (when the backend advertises ndt_load_sext and the access
+ * zero-extends, and (when the backend advertises load_sext and the access
* is MF_SEXT_LOAD) a sign-extending ldrsb/ldrsh sign-extends. Either way a
* subsequent matching widen (CV_ZEXT / CV_SEXT) is a no-op, so record which
* one this load produced — register-resident only — so nd_convert can drop the
@@ -1797,7 +1797,7 @@ static void nd_load(CgTarget* t, Operand dst, Operand addr, MemAccess mem) {
NativeDirectLocal* l = nd_local(d, dst.v.local);
if (l->reg == reg.v.reg) {
if ((mem.flags & MF_SEXT_LOAD) && d->reg_info &&
- d->reg_info->ndt_load_sext)
+ d->reg_info->load_sext)
l->load_sext = 1;
else
l->load_zext = 1;
@@ -2597,8 +2597,8 @@ CgTarget* native_direct_target_new(Compiler* c, ObjBuilder* obj,
u32 mask, n = 0;
if (!ci) continue;
mask = nd_caller_saved_mask(d, (NativeAllocClass)cls);
- for (u32 i = 0; i < ci->nallocable; ++i) {
- Reg r = ci->allocable[i];
+ for (u32 i = 0; i < ci->ndt_allocable_count; ++i) {
+ Reg r = ci->ndt_allocable[i];
if (r >= 32u) continue;
if (caller_only && !(mask & (1u << r))) continue;
d->cache_pool[cls][n++] = r;
diff --git a/src/cg/native_direct_target.h b/src/cg/native_direct_target.h
@@ -50,7 +50,7 @@ typedef struct NativeDirectLocal {
u8 load_zext;
/* The cached register holds this local's value SIGN-extended to the full
* register, from an MF_SEXT_LOAD narrow load that the backend honored with a
- * sign-extending load (aa64 ldrsb/ldrsh, gated on ndt_load_sext). While set, a
+ * sign-extending load (aa64 ldrsb/ldrsh, gated on load_sext). While set, a
* CV_SEXT widening it to any wider integer is a no-op and is elided by
* nd_convert. Set only in nd_load; cleared exactly like load_zext (a spill +
* plain reload would zero-extend, so clearing it forces the real extend to run
diff --git a/src/link/link_relocatable.c b/src/link/link_relocatable.c
@@ -183,6 +183,7 @@ static ObjSymId rel_copy_symbol(Linker* l, ObjBuilder* out,
if (id == OBJ_SYM_NONE)
compiler_panic(l->c, SRCLOC_NONE, "link -r: oom copying symbol");
if (s->flags) obj_symbol_set_flags(out, id, s->flags);
+ if (s->atom_subordinate) obj_symbol_set_atom_subordinate(out, id, 1);
obj_sym_mark_referenced(out, id);
return id;
}
diff --git a/src/obj/macho/emit.c b/src/obj/macho/emit.c
@@ -471,6 +471,8 @@ void emit_macho(Compiler* c, ObjBuilder* ob, Writer* w) {
* field) are already excluded by read_macho before stashing,
* so a plain OR can't double-count. */
n_desc |= s->flags;
+ if (s->atom_subordinate && (type & N_TYPE) == N_SECT)
+ n_desc |= N_ALT_ENTRY;
{
ObjAtomId aid = obj_atom_find_symbol(ob, e.id);
const ObjAtom* atom = obj_atom_get(ob, aid);
diff --git a/src/obj/macho/macho.h b/src/obj/macho/macho.h
@@ -233,6 +233,7 @@ typedef struct MachNlist64 {
#define N_NO_DEAD_STRIP 0x0020u
#define N_WEAK_REF 0x0040u
#define N_WEAK_DEF 0x0080u
+#define N_ALT_ENTRY 0x0200u
#define REFERENCE_FLAG_UNDEFINED_NON_LAZY 0x0u
#define REFERENCE_FLAG_UNDEFINED_LAZY 0x1u
diff --git a/src/obj/macho/read.c b/src/obj/macho/read.c
@@ -635,7 +635,7 @@ ObjBuilder* read_macho(Compiler* c, const char* name, const u8* data,
(SymKind)kind, sec_id, value, size, cmnalign);
obj_sym_mark_referenced(ob, id);
if ((mh_flags & MH_SUBSECTIONS_VIA_SYMBOLS) && type_field == N_SECT &&
- sec_id != OBJ_SEC_NONE) {
+ sec_id != OBJ_SEC_NONE && !(n_desc & N_ALT_ENTRY)) {
MAtomCand* ac = &atom_cands[natom_cands++];
ac->sec = sec_id;
ac->sym = id;
@@ -654,8 +654,11 @@ ObjBuilder* read_macho(Compiler* c, const char* name, const u8* data,
* back in. */
u16 desc_pass = n_desc;
desc_pass &= (u16) ~(N_WEAK_DEF | N_WEAK_REF);
+ if (type_field == N_SECT) desc_pass &= (u16)~N_ALT_ENTRY;
if (kind == SK_COMMON) desc_pass &= 0x00ff; /* drop align field */
if (desc_pass) obj_symbol_set_flags(ob, id, desc_pass);
+ if (type_field == N_SECT && (n_desc & N_ALT_ENTRY))
+ obj_symbol_set_atom_subordinate(ob, id, 1);
sym_macho_to_obj[i] = id;
}
diff --git a/src/obj/obj.c b/src/obj/obj.c
@@ -742,6 +742,7 @@ static ObjSymId obj_symbol_make(ObjBuilder* ob, Sym name, SymBind bind,
s->value = value;
s->size = size;
s->common_align = common_align;
+ s->atom_subordinate = 0;
/* First-wins: record the lowest id for this name so obj_symbol_find returns
* the same symbol the linear scan would. Later same-name symbols (legal for
* STB_LOCAL) do not overwrite. */
@@ -821,6 +822,15 @@ void obj_symbol_set_flags(ObjBuilder* ob, ObjSymId id, u16 flags) {
s->flags = flags;
}
+void obj_symbol_set_atom_subordinate(ObjBuilder* ob, ObjSymId id,
+ int subordinate) {
+ ObjSym* s;
+ if (id == OBJ_SYM_NONE) return;
+ s = Symbols_at(&ob->symbols, id);
+ if (!s) return;
+ s->atom_subordinate = subordinate ? 1u : 0u;
+}
+
void obj_reloc(ObjBuilder* ob, ObjSecId section_id, u32 offset, RelocKind kind,
ObjSymId sym, i64 addend) {
obj_reloc_ex(ob, section_id, offset, kind, sym, addend, 1, 0);
diff --git a/src/obj/obj.h b/src/obj/obj.h
@@ -399,6 +399,10 @@ typedef struct ObjSym {
* is also folded into the sweep, so emit-time symbol loops only need to
* check `removed`. */
u8 removed;
+ /* This symbol names an address subordinate to the preceding content unit
+ * rather than starting a separately collectible/reorderable atom. Mach-O
+ * writes this semantic as N_ALT_ENTRY; other formats may ignore it. */
+ u8 atom_subordinate;
} ObjSym;
typedef struct ObjGroup {
@@ -630,6 +634,7 @@ int obj_weak_alias_at(const ObjBuilder*, u32 i, ObjSymId* sym_out,
* to use the same field for their own pass-through; the contract is
* "bits go in / same bits come out", not a shared semantic. */
void obj_symbol_set_flags(ObjBuilder*, ObjSymId, u16 flags);
+void obj_symbol_set_atom_subordinate(ObjBuilder*, ObjSymId, int subordinate);
/* ---- read side (linker, file emitters, objdump) ---- */
u32 obj_section_count(const ObjBuilder*);
diff --git a/src/opt/cg_ir_lower.c b/src/opt/cg_ir_lower.c
@@ -539,7 +539,11 @@ static void materialize_frame_base(CgIrLower* l, u32 block, CGLocal local,
OptLocalMap* m = local_map(l, local, loc);
if (m->storage.kind == CG_LOCAL_STORAGE_REG) return;
if (materialized_reg_exists(l, local, CG_IR_MAT_BASE)) return;
- PReg r = ir_alloc_preg(l->f, m->type, RC_INT);
+ KitCgTypeId base_type =
+ cg_type_is_ptr(l->c, m->type) ? m->type : cg_type_ptr_to(l->c, m->type);
+ if (!base_type)
+ lower_panic(l, loc, "cannot construct indirect frame-base pointer type");
+ PReg r = ir_alloc_preg(l->f, base_type, RC_INT);
OptOperand ops[2];
ops[1] = opt_frame_operand(m);
if (cg_type_is_ptr(l->c, m->type)) {
@@ -549,7 +553,7 @@ static void materialize_frame_base(CgIrLower* l, u32 block, CGLocal local,
memset(&ops[0], 0, sizeof ops[0]);
ops[0].kind = OPK_REG;
ops[0].cls = RC_INT;
- ops[0].type = m->type;
+ ops[0].type = base_type;
ops[0].v.reg = (Reg)r;
ld->opnds = dup_opt_ops(l, ops, 2);
ld->nopnds = 2;
@@ -566,12 +570,12 @@ static void materialize_frame_base(CgIrLower* l, u32 block, CGLocal local,
memset(&ops[0], 0, sizeof ops[0]);
ops[0].kind = OPK_REG;
ops[0].cls = RC_INT;
- ops[0].type = m->type;
+ ops[0].type = base_type;
ops[0].v.reg = (Reg)r;
ao->opnds = dup_opt_ops(l, ops, 2);
ao->nopnds = 2;
ao->def = (Val)r;
- ao->type = m->type;
+ ao->type = base_type;
}
remember_materialized_reg(l, local, CG_IR_MAT_BASE, (Reg)r, loc);
}
@@ -659,10 +663,13 @@ static OptOperand lower_operand_addr(CgIrLower* l, const Operand* in,
out.cls = RC_INT;
out.v.ind.base =
resolve_materialized_reg(l, in->v.ind.base, CG_IR_MAT_BASE, loc);
+ out.v.ind.base_type = opt_reg_type(l->f, (PReg)out.v.ind.base);
out.v.ind.index = REG_NONE;
- if (in->v.ind.index != CG_LOCAL_NONE)
+ if (in->v.ind.index != CG_LOCAL_NONE) {
out.v.ind.index =
resolve_materialized_reg(l, in->v.ind.index, CG_IR_MAT_INDEX, loc);
+ out.v.ind.index_type = opt_reg_type(l->f, (PReg)out.v.ind.index);
+ }
out.v.ind.log2_scale = in->v.ind.log2_scale;
out.v.ind.ofs = in->v.ind.ofs;
return out;
diff --git a/src/opt/ir.h b/src/opt/ir.h
@@ -63,8 +63,36 @@ typedef enum OptOperandKind {
OPT_OPK_GLOBAL = OPK_GLOBAL,
OPT_OPK_INDIRECT = OPK_INDIRECT,
OPT_OPK_REG = 0xf0u,
+ /* Post-allocation rematerialization recipe: the value is the address of a
+ * frame slot, not the scalar stored in that slot. Semantic HIR never carries
+ * this kind; location MIR and native emission own it. */
+ OPT_OPK_FRAME_ADDR = 0xf1u,
+ /* Post-allocation scalar spill home. Unlike OPK_LOCAL (semantic frame
+ * storage/lvalue), this denotes the value stored in the frame slot. */
+ OPT_OPK_STACK = 0xf2u,
} OptOperandKind;
#define OPK_REG OPT_OPK_REG
+#define OPK_FRAME_ADDR OPT_OPK_FRAME_ADDR
+#define OPK_STACK OPT_OPK_STACK
+
+/* Location kinds for the components of an OPK_INDIRECT effective address.
+ * HIR is zero-initialized and therefore remains register-only. MIR may name a
+ * spilled pointer/index by its stable frame home, or retain a cheap frame
+ * address rematerialization recipe, without inventing a scratch-register
+ * value whose lifetime leaks across instructions. */
+typedef enum OptIndirectLocKind {
+ OPT_INDIRECT_REG = 0,
+ OPT_INDIRECT_FRAME = 1,
+ OPT_INDIRECT_FRAME_ADDR = 2,
+} OptIndirectLocKind;
+
+typedef enum OptOperandFlag {
+ /* Transient walker annotation, never stored in IR. It lets callbacks that
+ * lower or classify recipes distinguish an embedded effective-address
+ * component from a first-class scalar operand without changing the compact
+ * callback ABI. */
+ OPT_OPERAND_WALK_INDIRECT_PART = 1u << 0,
+} OptOperandFlag;
/* Operand riders (O1-PATTERNS L7/L8). Default 0 = no rider, so every existing
* operand and every backend that does not implement the fold keeps behaving as
@@ -86,7 +114,7 @@ typedef struct OptOperand {
* emitted as the shifted-register form (`add xD,xA,xS,lsl #shift`). 0 = the
* register is used as-is. Only stamped on a binop's second source operand. */
u8 shift;
- u8 pad[1];
+ u8 flags; /* OptOperandFlag; stored operands normally have no flags */
KitCgTypeId type;
union {
i64 imm;
@@ -105,7 +133,17 @@ typedef struct OptOperand {
* value that the addressing mode widens to 64 bits with the recorded
* extend (`[Xbase, Wm, sxtw #log2_scale]`). NONE = full-width X index. */
u8 index_ext;
+ /* OptIndirectLocKind for `base` and `index`. */
+ u8 base_kind;
+ u8 index_kind;
i32 ofs;
+ /* The component value types travel with location MIR. Spill-slot
+ * coloring is intentionally allowed to reuse a physical frame slot for
+ * unrelated values, so the slot descriptor cannot be used to recover a
+ * spilled base/index type after allocation. HIR initializes these from
+ * the source PRegs; MIR preserves them while changing only location. */
+ KitCgTypeId base_type;
+ KitCgTypeId index_type;
} ind;
} v;
} OptOperand;
@@ -388,7 +426,7 @@ typedef struct IRBitFieldAux {
/* One per-class clobber bitmask set; see Func.inst_clobbers. */
typedef u32 OptInstClobberMask[OPT_REG_CLASSES];
-#define OPT_MAX_SCRATCH_REGS 4u
+#define OPT_MAX_EMIT_TEMPS 4u
typedef struct IRGepAux {
KitCgTypeId base_type;
@@ -450,6 +488,17 @@ typedef struct IRCasAux {
KitCgMemOrder failure;
} IRCasAux;
+typedef struct IRAsmRegRequirement {
+ /* A register constraint resolved once at the HIR -> native-policy boundary.
+ * It remains an instruction-local placement request; regalloc never turns it
+ * into a persistent MIR location. */
+ u32 allowed_mask; /* 0 means any operand register in `cls` */
+ i32 fixed_reg; /* -1 means no fixed register */
+ u8 cls; /* RegClass */
+ u8 present; /* constraint requires a register */
+ u8 pad[2];
+} IRAsmRegRequirement;
+
typedef struct IRAsmAux {
const char* tmpl;
AsmConstraint* outs;
@@ -464,14 +513,10 @@ typedef struct IRAsmAux {
u32 clobber_abi_sets;
/* Filled by opt_machinize from backend register-name resolution. */
u32 clobber_mask[OPT_REG_CLASSES];
- i32* out_fixed_regs; /* nout, -1 when unconstrained */
- i32* in_fixed_regs; /* nin, -1 when unconstrained */
- u8* out_fixed_cls; /* RegClass, parallel to out_fixed_regs */
- u8* in_fixed_cls; /* RegClass, parallel to in_fixed_regs */
- u32* out_allowed_masks; /* nout, 0 when the whole class is allowed */
- u32* in_allowed_masks; /* nin, 0 when the whole class is allowed */
- u8* out_allowed_cls; /* RegClass, parallel to out_allowed_masks */
- u8* in_allowed_cls; /* RegClass, parallel to in_allowed_masks */
+ IRAsmRegRequirement* out_reg_reqs; /* nout */
+ IRAsmRegRequirement* in_reg_reqs; /* nin */
+ u8 has_memory_constraint;
+ u8 pad[3];
} IRAsmAux;
typedef struct IRIntrinAux {
@@ -584,6 +629,7 @@ typedef struct Block {
typedef struct MFunc {
Block* blocks;
u32 nblocks;
+ u32 blocks_cap;
u32 entry;
u32* emit_order;
u32 emit_order_n;
@@ -619,21 +665,17 @@ typedef struct OptPRegInfo {
u32 live_block_freq;
u32 live_across_call_freq;
u32 spill_cost;
- i32 tied_hard_reg; /* -1 means no fixed/tied physical register need. */
Reg hard_reg;
FrameSlot spill_slot;
u8 alloc_kind; /* OptAllocKind */
u8 cls; /* RegClass */
i8 preferred_hard_reg; /* soft hint for allocator; -1 = no hint */
u8 pad[1];
- u32 forbidden_hard_regs; /* bit r means PReg may not allocate hard reg r. */
- u32 allowed_hard_regs; /* 0 means unrestricted; otherwise positive mask. */
- /* Subset of forbidden_hard_regs that comes from a fixed-register machine
- * clobber (an instruction live across this value destroys reg r — see
- * Func.inst_clobbers). Unlike soft forbids, the return-register hint must not
- * clear these: a value cannot live in a register clobbered within its range.
- */
- u32 clobbered_hard_regs;
+ /* Authoritative correctness constraint. Bit r means this unsplit PReg may
+ * not be allocated to hard register r because an instruction boundary in
+ * its live range reserves or destroys that register. ABI placement
+ * preferences are independent soft hints and never mutate this mask. */
+ u32 forbidden_hard_regs;
} OptPRegInfo;
typedef enum OptUseKind {
@@ -733,8 +775,9 @@ typedef struct Func {
u64 opt_coalesce_merges;
InstId next_inst_id;
/* Per-instruction fixed-register clobbers (one bitmask per reg class),
- * indexed by InstId, sized [next_inst_id]. Built in opt_machinize_native from
- * the target's machine_op_clobbers hook; consulted by the allocator
+ * indexed by InstId, sized [next_inst_id]. Derived from the target's
+ * machine_op_clobbers hook by opt_machinize_native, then refreshed at the
+ * final pre-allocation HIR-shape boundary. Consulted by the allocator
* (pass_lower) to keep values live across an instruction out of the registers
* its encoding destroys (x86 idiv → rax/rdx, etc.). NULL when no instruction
* clobbers. */
@@ -763,8 +806,12 @@ typedef struct Func {
u32 opt_hard_reg_count[OPT_REG_CLASSES];
CGPhysRegInfo opt_phys_regs[OPT_REG_CLASSES][OPT_MAX_HARD_REGS];
u32 opt_phys_reg_count[OPT_REG_CLASSES];
- Reg opt_scratch_regs[OPT_REG_CLASSES][OPT_MAX_SCRATCH_REGS];
- u32 opt_scratch_reg_count[OPT_REG_CLASSES];
+ Reg emit_temp_regs[OPT_REG_CLASSES][OPT_MAX_EMIT_TEMPS];
+ u32 emit_temp_reg_count[OPT_REG_CLASSES];
+ /* Backend-private inline-asm staging registers. Kept as a mask because MIR
+ * verification only needs to enforce that they never become value
+ * locations; final asm emission leases the ordered descriptor list. */
+ u32 asm_temp_mask[OPT_REG_CLASSES];
u32 opt_caller_saved[OPT_REG_CLASSES]; /* bit r set if hard reg r is
caller-saved */
u32 opt_callee_saved[OPT_REG_CLASSES];
diff --git a/src/opt/ir_print.c b/src/opt/ir_print.c
@@ -201,6 +201,16 @@ static void dump_operand(Writer* w, const Operand* op) {
strbuf_put_u64(&sb, (u64)(unsigned)op->v.frame_slot);
dump_sb(w, &sb);
break;
+ case OPK_STACK:
+ strbuf_puts(&sb, "stack#");
+ strbuf_put_u64(&sb, (u64)(unsigned)op->v.frame_slot);
+ dump_sb(w, &sb);
+ break;
+ case OPK_FRAME_ADDR:
+ strbuf_puts(&sb, "&frame#");
+ strbuf_put_u64(&sb, (u64)(unsigned)op->v.frame_slot);
+ dump_sb(w, &sb);
+ break;
case OPK_GLOBAL:
strbuf_puts(&sb, "global#");
strbuf_put_u64(&sb, (u64)(unsigned)op->v.global.sym);
@@ -209,8 +219,28 @@ static void dump_operand(Writer* w, const Operand* op) {
break;
case OPK_INDIRECT:
strbuf_putc(&sb, '[');
- strbuf_putc(&sb, 'v');
+ if (op->v.ind.base_kind == OPT_INDIRECT_FRAME)
+ strbuf_puts(&sb, "frame-value#");
+ else if (op->v.ind.base_kind == OPT_INDIRECT_FRAME_ADDR)
+ strbuf_puts(&sb, "frame#");
+ else
+ strbuf_putc(&sb, 'v');
strbuf_put_u64(&sb, (u64)(unsigned)op->v.ind.base);
+ if (op->v.ind.index_kind != OPT_INDIRECT_REG ||
+ op->v.ind.index != (Reg)REG_NONE) {
+ strbuf_puts(&sb, "+");
+ if (op->v.ind.index_kind == OPT_INDIRECT_FRAME)
+ strbuf_puts(&sb, "frame-value#");
+ else if (op->v.ind.index_kind == OPT_INDIRECT_FRAME_ADDR)
+ strbuf_puts(&sb, "frame#");
+ else
+ strbuf_putc(&sb, 'v');
+ strbuf_put_u64(&sb, (u64)(unsigned)op->v.ind.index);
+ if (op->v.ind.log2_scale) {
+ strbuf_puts(&sb, "<<");
+ strbuf_put_u64(&sb, op->v.ind.log2_scale);
+ }
+ }
strbuf_put_i64_plus(&sb, (i64)(int)op->v.ind.ofs);
strbuf_putc(&sb, ']');
dump_sb(w, &sb);
diff --git a/src/opt/mir_clone.c b/src/opt/mir_clone.c
@@ -0,0 +1,338 @@
+#include <string.h>
+
+#include "opt/opt_internal.h"
+
+/*
+ * MIR-owned containers form an independently mutable graph. The arena owns
+ * both graphs for the lifetime of Func, while the temporary destination Func
+ * view continues to share immutable function-wide context. No container that
+ * an optimizer pass may rewrite is allowed to point into HIR. Leaf descriptors
+ * explicitly left shared below (types, ABI descriptions, symbol/string storage,
+ * and constant byte payloads) are immutable compiler-owned data.
+ */
+
+static u32 clone_cap(u32 count, u32 cap) { return cap < count ? count : cap; }
+
+static void clone_abi_value(Arena* arena, CGABIValue* dst,
+ const CGABIValue* src) {
+ *dst = *src;
+ if (!src->parts || !src->nparts) return;
+ dst->parts = arena_array(arena, CGABIPart, src->nparts);
+ memcpy(dst->parts, src->parts, sizeof(dst->parts[0]) * src->nparts);
+}
+
+static IRCallAux* clone_call_aux(Arena* arena, const IRCallAux* src) {
+ IRCallAux* dst = arena_new(arena, IRCallAux);
+ *dst = *src;
+
+ if (src->desc.args && src->desc.nargs) {
+ dst->desc.args = arena_array(arena, CGABIValue, src->desc.nargs);
+ for (u32 i = 0; i < src->desc.nargs; ++i)
+ clone_abi_value(arena, &dst->desc.args[i], &src->desc.args[i]);
+ }
+ clone_abi_value(arena, &dst->desc.ret, &src->desc.ret);
+
+ if (src->plan.args && src->plan.nargs) {
+ dst->plan.args = arena_array(arena, CGCallPlanMove, src->plan.nargs);
+ memcpy(dst->plan.args, src->plan.args,
+ sizeof(dst->plan.args[0]) * src->plan.nargs);
+ }
+ if (src->plan.rets && src->plan.nrets) {
+ dst->plan.rets = arena_array(arena, CGCallPlanRet, src->plan.nrets);
+ memcpy(dst->plan.rets, src->plan.rets,
+ sizeof(dst->plan.rets[0]) * src->plan.nrets);
+ }
+ if (src->results && src->nresults) {
+ dst->results = arena_array(arena, Val, src->nresults);
+ memcpy(dst->results, src->results,
+ sizeof(dst->results[0]) * src->nresults);
+ }
+ return dst;
+}
+
+static IRRetAux* clone_ret_aux(Arena* arena, const IRRetAux* src) {
+ IRRetAux* dst = arena_new(arena, IRRetAux);
+ *dst = *src;
+ clone_abi_value(arena, &dst->val, &src->val);
+ return dst;
+}
+
+static IRAsmAux* clone_asm_aux(Arena* arena, const IRAsmAux* src) {
+ IRAsmAux* dst = arena_new(arena, IRAsmAux);
+ *dst = *src;
+
+#define CLONE_ASM_ARRAY(field, T, count) \
+ do { \
+ if (src->field && (count)) { \
+ dst->field = arena_array(arena, T, (count)); \
+ memcpy(dst->field, src->field, sizeof(dst->field[0]) * (count)); \
+ } \
+ } while (0)
+
+ CLONE_ASM_ARRAY(outs, AsmConstraint, src->nout);
+ CLONE_ASM_ARRAY(ins, AsmConstraint, src->nin);
+ CLONE_ASM_ARRAY(clobbers, Sym, src->nclob);
+ CLONE_ASM_ARRAY(out_ops, Operand, src->nout);
+ CLONE_ASM_ARRAY(in_ops, Operand, src->nin);
+ CLONE_ASM_ARRAY(out_reg_reqs, IRAsmRegRequirement, src->nout);
+ CLONE_ASM_ARRAY(in_reg_reqs, IRAsmRegRequirement, src->nin);
+
+#undef CLONE_ASM_ARRAY
+ return dst;
+}
+
+static IRIntrinAux* clone_intrin_aux(Arena* arena, const IRIntrinAux* src) {
+ IRIntrinAux* dst = arena_new(arena, IRIntrinAux);
+ *dst = *src;
+ if (src->dsts && src->ndst) {
+ dst->dsts = arena_array(arena, Operand, src->ndst);
+ memcpy(dst->dsts, src->dsts, sizeof(dst->dsts[0]) * src->ndst);
+ }
+ if (src->args && src->narg) {
+ dst->args = arena_array(arena, Operand, src->narg);
+ memcpy(dst->args, src->args, sizeof(dst->args[0]) * src->narg);
+ }
+ if (src->result_vals && src->ndst) {
+ dst->result_vals = arena_array(arena, Val, src->ndst);
+ memcpy(dst->result_vals, src->result_vals,
+ sizeof(dst->result_vals[0]) * src->ndst);
+ }
+ return dst;
+}
+
+static int inst_uses_aux(IROp op) {
+ switch (op) {
+ case IR_PARAM_DECL:
+ case IR_TLS_ADDR_OF:
+ case IR_AGG_COPY:
+ case IR_AGG_SET:
+ case IR_BITFIELD_LOAD:
+ case IR_BITFIELD_STORE:
+ case IR_CALL:
+ case IR_PHI:
+ case IR_SWITCH:
+ case IR_INDIRECT_BRANCH:
+ case IR_LOCAL_STATIC_DATA_BEGIN:
+ case IR_LOCAL_STATIC_DATA_WRITE:
+ case IR_LOCAL_STATIC_DATA_LABEL_ADDR:
+ case IR_RET:
+ case IR_SCOPE_BEGIN:
+ case IR_ATOMIC_LOAD:
+ case IR_ATOMIC_STORE:
+ case IR_ATOMIC_RMW:
+ case IR_ATOMIC_CAS:
+ case IR_ASM_BLOCK:
+ case IR_INTRINSIC:
+ return 1;
+ default:
+ return 0;
+ }
+}
+
+static void* clone_inst_aux(Arena* arena, const Inst* src) {
+ if (!src->extra.aux) return NULL;
+
+#define CLONE_PLAIN_AUX(T) \
+ do { \
+ const T* src_aux = (const T*)src->extra.aux; \
+ T* dst_aux = arena_new(arena, T); \
+ *dst_aux = *src_aux; \
+ return dst_aux; \
+ } while (0)
+
+ switch ((IROp)src->op) {
+ case IR_PARAM_DECL: {
+ const IRParamDeclAux* src_aux =
+ (const IRParamDeclAux*)src->extra.aux;
+ IRParamDeclAux* dst_aux = arena_new(arena, IRParamDeclAux);
+ *dst_aux = *src_aux;
+ if (src_aux->desc.incoming && src_aux->desc.nincoming) {
+ CGABIPart* incoming =
+ arena_array(arena, CGABIPart, src_aux->desc.nincoming);
+ memcpy(incoming, src_aux->desc.incoming,
+ sizeof(incoming[0]) * src_aux->desc.nincoming);
+ dst_aux->desc.incoming = incoming;
+ }
+ return dst_aux;
+ }
+ case IR_TLS_ADDR_OF:
+ CLONE_PLAIN_AUX(IRTlsAux);
+ case IR_AGG_COPY:
+ case IR_AGG_SET:
+ CLONE_PLAIN_AUX(IRAggAux);
+ case IR_BITFIELD_LOAD:
+ case IR_BITFIELD_STORE:
+ CLONE_PLAIN_AUX(IRBitFieldAux);
+ case IR_CALL:
+ return clone_call_aux(arena, (const IRCallAux*)src->extra.aux);
+ case IR_PHI: {
+ const IRPhiAux* src_aux = (const IRPhiAux*)src->extra.aux;
+ IRPhiAux* dst_aux = arena_new(arena, IRPhiAux);
+ *dst_aux = *src_aux;
+ if (src_aux->pred_blocks && src_aux->npreds) {
+ dst_aux->pred_blocks = arena_array(arena, u32, src_aux->npreds);
+ memcpy(dst_aux->pred_blocks, src_aux->pred_blocks,
+ sizeof(dst_aux->pred_blocks[0]) * src_aux->npreds);
+ }
+ if (src_aux->pred_vals && src_aux->npreds) {
+ dst_aux->pred_vals = arena_array(arena, Val, src_aux->npreds);
+ memcpy(dst_aux->pred_vals, src_aux->pred_vals,
+ sizeof(dst_aux->pred_vals[0]) * src_aux->npreds);
+ }
+ return dst_aux;
+ }
+ case IR_SWITCH: {
+ const IRSwitchAux* src_aux = (const IRSwitchAux*)src->extra.aux;
+ IRSwitchAux* dst_aux = arena_new(arena, IRSwitchAux);
+ *dst_aux = *src_aux;
+ if (src_aux->cases && src_aux->ncases) {
+ dst_aux->cases =
+ arena_array(arena, IRSwitchAuxCase, src_aux->ncases);
+ memcpy(dst_aux->cases, src_aux->cases,
+ sizeof(dst_aux->cases[0]) * src_aux->ncases);
+ }
+ return dst_aux;
+ }
+ case IR_INDIRECT_BRANCH: {
+ const IRIndirectAux* src_aux = (const IRIndirectAux*)src->extra.aux;
+ IRIndirectAux* dst_aux = arena_new(arena, IRIndirectAux);
+ *dst_aux = *src_aux;
+ if (src_aux->targets && src_aux->ntargets) {
+ dst_aux->targets = arena_array(arena, u32, src_aux->ntargets);
+ memcpy(dst_aux->targets, src_aux->targets,
+ sizeof(dst_aux->targets[0]) * src_aux->ntargets);
+ }
+ return dst_aux;
+ }
+ case IR_LOCAL_STATIC_DATA_BEGIN:
+ CLONE_PLAIN_AUX(CgIrLocalStaticBeginAux);
+ case IR_LOCAL_STATIC_DATA_WRITE:
+ /* The data payload is an immutable byte string owned by the semantic IR;
+ * only the mutable descriptor object is graph-local. */
+ CLONE_PLAIN_AUX(CgIrLocalStaticWriteAux);
+ case IR_LOCAL_STATIC_DATA_LABEL_ADDR:
+ CLONE_PLAIN_AUX(CgIrLocalStaticLabelAux);
+ case IR_RET:
+ return clone_ret_aux(arena, (const IRRetAux*)src->extra.aux);
+ case IR_SCOPE_BEGIN:
+ CLONE_PLAIN_AUX(IRScopeAux);
+ case IR_ATOMIC_LOAD:
+ case IR_ATOMIC_STORE:
+ case IR_ATOMIC_RMW:
+ CLONE_PLAIN_AUX(IRAtomicAux);
+ case IR_ATOMIC_CAS:
+ CLONE_PLAIN_AUX(IRCasAux);
+ case IR_ASM_BLOCK:
+ return clone_asm_aux(arena, (const IRAsmAux*)src->extra.aux);
+ case IR_INTRINSIC:
+ return clone_intrin_aux(arena, (const IRIntrinAux*)src->extra.aux);
+ default:
+ return NULL;
+ }
+
+#undef CLONE_PLAIN_AUX
+}
+
+static void clone_inst(Arena* arena, Inst* dst, const Inst* src) {
+ *dst = *src;
+ if (src->defs && src->ndefs) {
+ dst->defs = arena_array(arena, Val, src->ndefs);
+ memcpy(dst->defs, src->defs, sizeof(dst->defs[0]) * src->ndefs);
+ }
+ if (src->opnds && src->nopnds) {
+ dst->opnds = arena_array(arena, Operand, src->nopnds);
+ memcpy(dst->opnds, src->opnds, sizeof(dst->opnds[0]) * src->nopnds);
+ }
+ if (inst_uses_aux((IROp)src->op))
+ dst->extra.aux = clone_inst_aux(arena, src);
+}
+
+static Inst* cloned_inst_by_id(Func* f, InstId id) {
+ if (id == INST_ID_NONE) return NULL;
+ for (u32 b = 0; b < f->nblocks; ++b) {
+ Block* block = &f->blocks[b];
+ for (u32 i = 0; i < block->ninsts; ++i)
+ if (block->insts[i].id == id) return &block->insts[i];
+ }
+ return NULL;
+}
+
+void opt_mir_clone_func(Func* dst, const Func* src) {
+ Arena* arena;
+ u32 blocks_cap;
+ u32 frame_slots_cap;
+ u32 emit_order_cap;
+
+ if (!dst || !src) return;
+ *dst = *src;
+ arena = src->arena;
+ dst->mir = NULL;
+ dst->opt_rewritten = 0;
+
+ blocks_cap = clone_cap(src->nblocks, src->blocks_cap);
+ dst->blocks = arena_zarray(arena, Block, blocks_cap ? blocks_cap : 1u);
+ for (u32 b = 0; b < src->nblocks; ++b) {
+ const Block* src_block = &src->blocks[b];
+ Block* dst_block = &dst->blocks[b];
+ u32 inst_cap;
+ *dst_block = *src_block;
+
+ inst_cap = clone_cap(src_block->ninsts, src_block->cap);
+ dst_block->insts =
+ arena_zarray(arena, Inst, inst_cap ? inst_cap : 1u);
+ for (u32 i = 0; i < src_block->ninsts; ++i)
+ clone_inst(arena, &dst_block->insts[i], &src_block->insts[i]);
+
+ if (src_block->preds && src_block->npreds) {
+ dst_block->preds = arena_array(arena, u32, src_block->npreds);
+ memcpy(dst_block->preds, src_block->preds,
+ sizeof(dst_block->preds[0]) * src_block->npreds);
+ }
+ if (src_block->succ && src_block->succ_cap) {
+ dst_block->succ = arena_array(arena, u32, src_block->succ_cap);
+ memcpy(dst_block->succ, src_block->succ,
+ sizeof(dst_block->succ[0]) * src_block->succ_cap);
+ }
+ }
+
+ frame_slots_cap = clone_cap(src->nframe_slots, src->frame_slots_cap);
+ if (src->frame_slots && frame_slots_cap) {
+ dst->frame_slots = arena_zarray(arena, IRFrameSlot, frame_slots_cap);
+ memcpy(dst->frame_slots, src->frame_slots,
+ sizeof(dst->frame_slots[0]) * src->nframe_slots);
+ }
+
+ emit_order_cap = clone_cap(src->emit_order_n, src->emit_order_cap);
+ if (src->emit_order && emit_order_cap) {
+ dst->emit_order = arena_array(arena, u32, emit_order_cap);
+ memcpy(dst->emit_order, src->emit_order,
+ sizeof(dst->emit_order[0]) * src->emit_order_n);
+ }
+
+ /* spill_slot_for may add a call-save home during rewriting. Keep that
+ * allocation bookkeeping private to the lowering view. */
+ if (src->preg_info && opt_reg_count(src)) {
+ u32 nregs = opt_reg_count(src);
+ dst->preg_info = arena_array(arena, OptPRegInfo, nregs);
+ memcpy(dst->preg_info, src->preg_info,
+ sizeof(dst->preg_info[0]) * nregs);
+ }
+
+ if (src->scope_aux_inst && src->scopes_cap) {
+ dst->scope_aux_inst = arena_zarray(arena, Inst*, src->scopes_cap);
+ for (u32 i = 0; i < src->nscopes; ++i) {
+ const Inst* scope = src->scope_aux_inst[i];
+ if (scope) dst->scope_aux_inst[i] = cloned_inst_by_id(dst, scope->id);
+ }
+ }
+
+ /* Def-use entries contain direct Operand pointers into HIR. They are an
+ * analysis cache, not part of the MIR graph, and must never cross this
+ * boundary. */
+ dst->opt_uses = NULL;
+ dst->opt_nuses = 0;
+ dst->opt_uses_cap = 0;
+ dst->opt_first_use_by_val = NULL;
+ dst->opt_first_use_by_val_cap = 0;
+ dst->opt_valid_analyses = 0;
+}
diff --git a/src/opt/opt.c b/src/opt/opt.c
@@ -132,14 +132,17 @@ static void opt_registry_add(OptImpl* o, CgIrFunc* f) {
}
static void opt_dbg_dump(OptImpl* o, Func* f, const char* tag) {
+ Func view;
+ Func* graph = f;
const char* s = kit_debug_getenv("KIT_DUMP");
KitWriter* w = NULL;
size_t len = 0;
const uint8_t* bytes;
if (!s) return;
if (strcmp(s, "1") != 0 && strcmp(s, tag) != 0) return;
+ if (opt_mir_view(f, &view)) graph = &view;
kit_writer_mem(o->c->ctx->heap, &w);
- opt_ir_dump(f, w);
+ opt_ir_dump(graph, w);
bytes = kit_writer_mem_bytes(w, &len);
diag_emit(o->c->ctx->diag, KIT_DIAG_NOTE, f->desc.loc, "DUMP %s:\n%.*s", tag,
(int)len, (const char*)bytes);
@@ -253,6 +256,15 @@ static void opt_o1_native_finish(OptImpl* o, Func* f, int cfg_dirty) {
metrics_scope_end(o->c, "opt.o1.hoist_loop_consts");
opt_verify(f, "o1-hoist-loop-consts");
+ /* Machinization derives fixed-register effects from instruction shape. The
+ * loop-immediate pass above can turn an immediate shift into a register shift
+ * (x64 then uses/clobbers RCX), and the hoister is the last HIR shape mutator
+ * before allocation. Refresh once at this explicit boundary so liveness and
+ * regalloc consume effects for the final HIR, not the pre-transform form. */
+ metrics_scope_begin(o->c, "opt.machine_effects.refresh");
+ opt_refresh_machine_clobbers(f, o->native);
+ metrics_scope_end(o->c, "opt.machine_effects.refresh");
+
metrics_scope_begin(o->c, "opt.live_blocks.pre_dde");
memset(&live, 0, sizeof live);
opt_live_blocks(f, &live);
@@ -301,6 +313,9 @@ static void opt_o1_native_finish(OptImpl* o, Func* f, int cfg_dirty) {
metrics_scope_begin(o->c, "opt.post_ra.jump_cleanup_layout");
opt_mir_jump_cleanup(f, OPT_JUMP_CLEANUP_LAYOUT);
metrics_scope_end(o->c, "opt.post_ra.jump_cleanup_layout");
+ metrics_scope_begin(o->c, "opt.post_ra.layout.verify");
+ opt_mir_verify(f, "post-mir-layout");
+ metrics_scope_end(o->c, "opt.post_ra.layout.verify");
opt_dbg_dump(o, f, "pre-emit");
metrics_scope_begin(o->c, "opt.emit");
diff --git a/src/opt/opt.h b/src/opt/opt.h
@@ -65,6 +65,10 @@ void opt_jump_opt(Func*);
/* ----- lowering / backend prep (per Func, run before NativeTarget emission)
* ----- */
void opt_machinize_native(Func*, NativeTarget* target);
+/* Rebuild target-derived per-instruction fixed-register effects after an HIR
+ * shape-changing pass. The O1 pipeline calls this at its final HIR boundary,
+ * before liveness and allocation consume those effects. */
+void opt_refresh_machine_clobbers(Func*, NativeTarget* target);
void opt_build_loop_tree(Func*);
typedef struct OptBitset {
diff --git a/src/opt/opt_internal.h b/src/opt/opt_internal.h
@@ -9,6 +9,22 @@ typedef struct OptHardRegSet {
u32 cls[OPT_REG_CLASSES];
} OptHardRegSet;
+/* Canonical physical-register effects of one instruction. `uses` and `defs`
+ * are the registers named by the instruction/ABI operands; `clobbers` are
+ * implicit kills (call ABI, inline asm, and target machine constraints).
+ * Emission temporaries are private to native emission and never appear here.
+ * Keeping defs separate from clobbers matters to DCE: an implicit kill is not
+ * a produced value. */
+typedef struct OptRegEffects {
+ OptHardRegSet uses;
+ OptHardRegSet defs;
+ OptHardRegSet clobbers;
+ /* Per-register explicit-use multiplicity, saturated at two. This keeps
+ * combine's single-use decisions in the same canonical operand model (and
+ * preserves the double-reference case `[r + r*scale]`). */
+ u8 use_count[OPT_REG_CLASSES][OPT_MAX_HARD_REGS];
+} OptRegEffects;
+
typedef struct OptHardBlockLive {
OptHardRegSet live_in;
OptHardRegSet live_out;
@@ -148,12 +164,32 @@ int opt_edge_is_fallthrough(Func*, u32 pred, u32 succ);
u32 opt_split_edge(Func*, u32 pred, u32 succ);
int opt_val_in_inst_defs(const Inst*, Val);
+/* One authority for the scalar types which may occupy an indirect index.
+ * Address synthesis and MIR verification must agree on this boundary. */
+int opt_indirect_index_type_valid(Func*, KitCgTypeId);
+/* Canonical role of a direct instruction operand. Unlike legacy def-id
+ * inference, this remains valid after PRegs have been rewritten to physical
+ * registers or frame locations. Aux operands are classified by the central
+ * walker below. */
+int opt_inst_operand_is_def(const Inst*, u32 index);
void opt_walk_operand(Func*, Inst*, Operand*, int is_def, OptOperandWalkFn,
void*);
void opt_walk_abivalue(Func*, Inst*, CGABIValue*, int storage_def,
OptOperandWalkFn, void*);
void opt_walk_inst_operands(Func*, Inst*, OptOperandWalkFn, void*);
+/* Build the independently mutable post-allocation graph used by MIR lowering.
+ * Mutable instruction, operand, aux, and CFG storage never aliases HIR. */
+void opt_mir_clone_func(Func* dst, const Func* src);
+
+/* Present the independently owned MIR graph through the ordinary Func pass
+ * interface. The view deliberately drops HIR pointer-based analyses; callers
+ * must rebuild any analysis they need over MIR. Passes which may replace graph
+ * storage commit those fields, plus any extended instruction-id namespace,
+ * back through opt_mir_commit. */
+int opt_mir_view(Func* f, Func* out);
+void opt_mir_commit(Func* f, const Func* view);
+
/* Make room for k new instructions starting at index `at` in block `bl`.
* Returns a pointer to the first new slot (zero-initialized). Grows the
* block's inst array (doubling) as needed. */
@@ -177,8 +213,8 @@ int opt_hard_empty(const OptHardRegSet*);
int opt_hard_intersects(const OptHardRegSet*, const OptHardRegSet*);
void opt_hard_live_step(OptHardRegSet* live, const OptHardRegSet* use,
const OptHardRegSet* def);
-void opt_hard_inst_use_def(Func*, const Inst*, OptHardRegSet* use,
- OptHardRegSet* def);
+void opt_inst_reg_effects(Func*, const Inst*, OptRegEffects*);
+void opt_reg_effect_kills(const OptRegEffects*, OptHardRegSet*);
OptHardBlockLive* opt_maybe_build_hard_live(Func*);
OptHardRegSet opt_hard_live_out_for_block(const OptHardBlockLive*);
int opt_block_live_out_has_phys_reg(Func*, const OptHardBlockLive*, u32 block,
diff --git a/src/opt/opt_util.c b/src/opt/opt_util.c
@@ -10,40 +10,129 @@ int opt_val_in_inst_defs(const Inst* in, Val v) {
return 0;
}
-static int opt_operand_index_is_def(const Inst* in, u32 i) {
- if (!in || i >= in->nopnds || in->opnds[i].kind != OPK_REG) return 0;
- if (!opt_val_in_inst_defs(in, (Val)in->opnds[i].v.reg)) return 0;
+int opt_indirect_index_type_valid(Func* f, KitCgTypeId type) {
+ u64 size;
+ if (!f || !f->c || type == KIT_CG_TYPE_NONE ||
+ !cg_type_is_int(f->c, type) ||
+ opt_value_reg_class(f->c, type) != RC_INT)
+ return 0;
+ size = cg_type_size(f->c, type);
+ return size != 0u && size <= f->c->target.ptr_size;
+}
+
+int opt_inst_operand_is_def(const Inst* in, u32 i) {
+ if (!in || i >= in->nopnds) return 0;
switch ((IROp)in->op) {
+ case IR_LOAD_IMM:
+ case IR_LOAD_CONST:
+ case IR_TLS_ADDR_OF:
+ case IR_LOAD_LABEL_ADDR:
+ case IR_COPY:
+ case IR_CONVERT:
+ case IR_UNOP:
+ case IR_VA_ARG:
+ case IR_LOAD:
+ case IR_ADDR_OF:
+ case IR_BITFIELD_LOAD:
+ case IR_ATOMIC_LOAD:
+ case IR_BINOP:
+ case IR_CMP:
+ case IR_ALLOCA:
+ case IR_ATOMIC_RMW:
+ return i == 0u;
case IR_ATOMIC_CAS:
- return i == 0 || i == 1;
+ return i == 0u || i == 1u;
+ default:
+ return 0;
+ }
+}
+
+static KitCgTypeId indirect_frame_type(Func* f, FrameSlot slot,
+ KitCgTypeId fallback) {
+ if (f && slot != FRAME_SLOT_NONE && slot <= f->nframe_slots) {
+ KitCgTypeId type = f->frame_slots[slot - 1u].type;
+ if (type) return type;
+ }
+ return fallback;
+}
+
+static void indirect_part_operand(Func* f, const Operand* owner, u32 value,
+ u8 kind, KitCgTypeId component_type,
+ Operand* out) {
+ *out = *owner;
+ out->cls = RC_INT;
+ out->flags |= OPT_OPERAND_WALK_INDIRECT_PART;
+ if (component_type) out->type = component_type;
+ switch ((OptIndirectLocKind)kind) {
+ case OPT_INDIRECT_REG:
+ out->kind = OPK_REG;
+ out->v.reg = (Reg)value;
+ if (!component_type && (PReg)value < opt_reg_count(f) &&
+ opt_reg_type(f, (PReg)value))
+ out->type = opt_reg_type(f, (PReg)value);
+ return;
+ case OPT_INDIRECT_FRAME:
+ out->kind = OPK_STACK;
+ out->v.frame_slot = (FrameSlot)value;
+ if (!component_type)
+ out->type = indirect_frame_type(f, (FrameSlot)value, owner->type);
+ return;
+ case OPT_INDIRECT_FRAME_ADDR:
+ out->kind = OPK_FRAME_ADDR;
+ out->v.frame_slot = (FrameSlot)value;
+ return;
+ }
+}
+
+static void indirect_part_commit(Func* f, Inst* in, Operand* owner,
+ Operand* part, u32* value, u8* kind,
+ KitCgTypeId* component_type) {
+ if (component_type) *component_type = part->type;
+ switch ((OptOperandKind)part->kind) {
+ case OPK_REG:
+ *value = part->v.reg;
+ *kind = OPT_INDIRECT_REG;
+ return;
+ case OPK_STACK:
+ *value = part->v.frame_slot;
+ *kind = OPT_INDIRECT_FRAME;
+ return;
+ case OPK_FRAME_ADDR:
+ *value = part->v.frame_slot;
+ *kind = OPT_INDIRECT_FRAME_ADDR;
+ return;
default:
- return i == 0;
+ compiler_panic(f->c, in ? in->loc : (SrcLoc){0, 0, 0},
+ "opt operand walk: invalid indirect location kind %u",
+ (unsigned)part->kind);
}
+ (void)owner;
}
void opt_walk_operand(Func* f, Inst* in, Operand* op, int is_def,
OptOperandWalkFn fn, void* ctx) {
if (!op || !fn) return;
- if (op->kind == OPK_REG) {
+ if (op->kind != OPK_INDIRECT) {
fn(f, in, op, is_def, ctx);
} else if (op->kind == OPK_INDIRECT) {
- Operand base = *op;
- base.kind = OPK_REG;
- base.cls = RC_INT;
- base.v.reg = op->v.ind.base;
- if ((PReg)base.v.reg < opt_reg_count(f) && opt_reg_type(f, base.v.reg))
- base.type = opt_reg_type(f, base.v.reg);
+ Operand base;
+ /* Expose the complete address owner as well as its scalar components.
+ * Location-aware consumers (notably MIR verification) need to validate
+ * the tuple; register-rewrite callbacks simply ignore non-register kinds. */
+ fn(f, in, op, 0, ctx);
+ indirect_part_operand(f, op, op->v.ind.base, op->v.ind.base_kind,
+ op->v.ind.base_type, &base);
fn(f, in, &base, 0, ctx);
- op->v.ind.base = base.v.reg;
- if (op->v.ind.index != (Reg)REG_NONE) {
- Operand idx = *op;
- idx.kind = OPK_REG;
- idx.cls = RC_INT;
- idx.v.reg = op->v.ind.index;
- if ((PReg)idx.v.reg < opt_reg_count(f) && opt_reg_type(f, idx.v.reg))
- idx.type = opt_reg_type(f, idx.v.reg);
+ indirect_part_commit(f, in, op, &base, &op->v.ind.base,
+ &op->v.ind.base_kind, &op->v.ind.base_type);
+ if (op->v.ind.index_kind != OPT_INDIRECT_REG ||
+ op->v.ind.index != (Reg)REG_NONE) {
+ Operand idx;
+ indirect_part_operand(f, op, op->v.ind.index, op->v.ind.index_kind,
+ op->v.ind.index_type, &idx);
fn(f, in, &idx, 0, ctx);
- op->v.ind.index = idx.v.reg;
+ indirect_part_commit(f, in, op, &idx, &op->v.ind.index,
+ &op->v.ind.index_kind, &op->v.ind.index_type);
}
}
}
@@ -59,11 +148,11 @@ void opt_walk_abivalue(Func* f, Inst* in, CGABIValue* v, int storage_def,
void opt_walk_inst_operands(Func* f, Inst* in, OptOperandWalkFn fn, void* ctx) {
if (!in || !fn) return;
for (u32 i = 0; i < in->nopnds; ++i) {
- int is_def = opt_operand_index_is_def(in, i);
+ int is_def = opt_inst_operand_is_def(in, i);
if (!is_def) opt_walk_operand(f, in, &in->opnds[i], 0, fn, ctx);
}
for (u32 i = 0; i < in->nopnds; ++i) {
- int is_def = opt_operand_index_is_def(in, i);
+ int is_def = opt_inst_operand_is_def(in, i);
if (is_def) opt_walk_operand(f, in, &in->opnds[i], 1, fn, ctx);
}
diff --git a/src/opt/pass_addr_fold.c b/src/opt/pass_addr_fold.c
@@ -560,12 +560,14 @@ static void addr_cse_remap_op(Func* f, Inst* in, Operand* op, int is_def,
void* ctx) {
const PReg* remap = (const PReg*)ctx;
PReg p;
- (void)f;
(void)in;
(void)is_def;
if (!op || op->kind != OPK_REG) return;
p = (PReg)op->v.reg;
- if (p != PREG_NONE && p != 0 && remap[p] != 0) op->v.reg = remap[p];
+ if (p != PREG_NONE && p != 0 && remap[p] != 0) {
+ op->v.reg = remap[p];
+ op->type = opt_reg_type(f, remap[p]);
+ }
}
static void addr_cse_apply_to_inst(Func* f, Inst* in, const PReg* remap) {
diff --git a/src/opt/pass_coalesce.c b/src/opt/pass_coalesce.c
@@ -144,32 +144,12 @@ static int group_conflicts(const CoalesceCtx* c, PReg ra, PReg rb, PReg allow_a,
return 0;
}
-static int hard_reg_possible(Func* f, u8 cls, u32 forbidden, u32 allowed) {
+static int hard_reg_possible(Func* f, u8 cls, u32 forbidden) {
for (u32 i = 0; i < f->opt_hard_reg_count[cls]; ++i) {
Reg r = f->opt_hard_regs[cls][i];
if (r >= 32) continue;
- if (allowed && (allowed & (1u << r)) == 0) continue;
if ((forbidden & (1u << r)) == 0) return 1;
}
- if (allowed) {
- for (Reg r = 0; r < 32; ++r) {
- if ((allowed & (1u << r)) == 0) continue;
- if (forbidden & (1u << r)) continue;
- int in_hard = 0;
- for (u32 i = 0; i < f->opt_hard_reg_count[cls]; ++i) {
- if (f->opt_hard_regs[cls][i] == r) {
- in_hard = 1;
- break;
- }
- }
- if (in_hard) continue;
- for (u32 i = 0; i < f->opt_phys_reg_count[cls]; ++i) {
- const CGPhysRegInfo* pi = &f->opt_phys_regs[cls][i];
- if (pi->reg == r && (pi->flags & CG_REG_RESERVED) == 0) return 1;
- }
- }
- return 0;
- }
return f->opt_hard_reg_count[cls] == 0;
}
@@ -178,32 +158,15 @@ static int group_constraints_compatible(const CoalesceCtx* c, PReg ra,
Func* f = c->f;
u8 cls = opt_reg_cls(f, ra);
KitCgTypeId type = opt_reg_type(f, ra);
- i32 tied = -1;
u32 forbidden = 0;
- u32 allowed = 0;
for (PReg v = 1; v < opt_reg_count(f); ++v) {
PReg r = coalesce_find(f, v);
if (r != ra && r != rb) continue;
if (opt_reg_cls(f, v) != cls || opt_reg_type(f, v) != type) return 0;
const OptPRegInfo* vi = &f->preg_info[v];
forbidden |= vi->forbidden_hard_regs;
- if (vi->allowed_hard_regs) {
- if (allowed) {
- allowed &= vi->allowed_hard_regs;
- if (!allowed) return 0;
- } else {
- allowed = vi->allowed_hard_regs;
- }
- }
- if (vi->tied_hard_reg >= 0) {
- if (tied >= 0 && tied != vi->tied_hard_reg) return 0;
- tied = vi->tied_hard_reg;
- }
}
- if (tied >= 0 && tied < 32 && (forbidden & (1u << (Reg)tied))) return 0;
- if (tied >= 0 && tied < 32 && allowed && (allowed & (1u << (Reg)tied)) == 0)
- return 0;
- return hard_reg_possible(f, cls, forbidden, allowed);
+ return hard_reg_possible(f, cls, forbidden);
}
static void coalesce_union(Func* f, PReg a, PReg b) {
@@ -373,11 +336,9 @@ typedef struct LinCoalState {
PReg* members; /* per root: PReg members of the coalesce group (capped) */
u32 nmembers;
u32 forbidden; /* OR of forbidden_hard_regs over the group */
- u32 allowed; /* intersection of nonzero allowed masks; 0 if none */
- u8 has_allowed; /* a member contributed a positive allowed mask */
u8 init; /* state has been populated for this root */
- i32 tied; /* single tied hard reg, or -1 */
u8 cls;
+ u8 pad[2];
KitCgTypeId type;
} LinCoalState;
@@ -410,29 +371,7 @@ static int lin_constraints_ok(Func* f, const LinCoalState* sa,
const LinCoalState* sb) {
if (sa->cls != sb->cls || sa->type != sb->type) return 0;
u32 forbidden = sa->forbidden | sb->forbidden;
- u32 allowed = 0;
- u8 has_allowed = 0;
- if (sa->has_allowed && sb->has_allowed) {
- allowed = sa->allowed & sb->allowed;
- has_allowed = 1;
- if (!allowed) return 0;
- } else if (sa->has_allowed) {
- allowed = sa->allowed;
- has_allowed = 1;
- } else if (sb->has_allowed) {
- allowed = sb->allowed;
- has_allowed = 1;
- }
- i32 tied = sa->tied;
- if (sb->tied >= 0) {
- if (tied >= 0 && tied != sb->tied) return 0;
- tied = sb->tied;
- }
- if (tied >= 0 && tied < 32 && (forbidden & (1u << (Reg)tied))) return 0;
- if (tied >= 0 && tied < 32 && has_allowed &&
- (allowed & (1u << (Reg)tied)) == 0)
- return 0;
- return hard_reg_possible(f, sa->cls, forbidden, has_allowed ? allowed : 0u);
+ return hard_reg_possible(f, sa->cls, forbidden);
}
void opt_coalesce_linear(Func* f, const OptLiveRangeSet* ranges) {
@@ -482,10 +421,7 @@ void opt_coalesce_linear(Func* f, const OptLiveRangeSet* ranges) {
st[v].members = NULL;
st[v].nmembers = 0;
st[v].forbidden = 0;
- st[v].allowed = 0;
- st[v].has_allowed = 0;
st[v].init = 0;
- st[v].tied = -1;
st[v].cls = 0;
st[v].type = 0;
}
@@ -500,11 +436,6 @@ void opt_coalesce_linear(Func* f, const OptLiveRangeSet* ranges) {
st[v].members[0] = v;
st[v].nmembers = 1;
st[v].forbidden = vi->forbidden_hard_regs;
- if (vi->allowed_hard_regs) {
- st[v].allowed = vi->allowed_hard_regs;
- st[v].has_allowed = 1;
- }
- st[v].tied = vi->tied_hard_reg;
st[v].cls = opt_reg_cls(f, v);
st[v].type = opt_reg_type(f, v);
}
@@ -531,15 +462,6 @@ void opt_coalesce_linear(Func* f, const OptLiveRangeSet* ranges) {
for (u32 m = 0; m < st[orr].nmembers; ++m)
st[nr].members[st[nr].nmembers++] = st[orr].members[m];
st[nr].forbidden |= st[orr].forbidden;
- if (st[orr].has_allowed) {
- if (st[nr].has_allowed)
- st[nr].allowed &= st[orr].allowed;
- else {
- st[nr].allowed = st[orr].allowed;
- st[nr].has_allowed = 1;
- }
- }
- if (st[orr].tied >= 0) st[nr].tied = st[orr].tied;
st[orr].nmembers = 0; /* folded into nr */
++f->opt_coalesce_merges;
}
diff --git a/src/opt/pass_combine.c b/src/opt/pass_combine.c
@@ -37,6 +37,20 @@ static int frame_slot_is_spill(Func* f, FrameSlot fs) {
return f->frame_slots[fs - 1u].kind == FS_SPILL;
}
+typedef enum SpillAccessKind {
+ SPILL_ACCESS_NONE = 0,
+ SPILL_ACCESS_LOAD,
+ SPILL_ACCESS_STORE,
+} SpillAccessKind;
+
+typedef struct SpillAccess {
+ SpillAccessKind kind;
+ FrameSlot slot;
+ u32 size;
+ u16 addr_space;
+ const Operand* value;
+} SpillAccess;
+
/* W6 address-offset profitability gate: is `ofs` an offset the target can keep
* inline in a load/store (so folding it into an indirect whose base producer
* survives does not force an emit-time address build)? Conservatively keyed on
@@ -49,55 +63,51 @@ static int combine_offset_fold_ok(NativeTarget* target, KitCgTypeId type,
return target->imm_legal(target, NATIVE_IMM_ADDR_OFFSET, 0, type, ofs);
}
-static int spill_local_slot(Func* f, const Operand* addr, const MemAccess* mem,
- FrameSlot* out) {
- if (!addr || addr->kind != OPK_LOCAL) return 0;
- if (opt_mem_observable(mem)) return 0;
- if (mem->alias.kind != ALIAS_LOCAL) return 0;
- if (mem->alias.v.local_id != (i32)addr->v.frame_slot) return 0;
- if (!frame_slot_is_spill(f, addr->v.frame_slot)) return 0;
- *out = addr->v.frame_slot;
- return 1;
-}
+/* Normalize location-MIR stack copies to the one spill-access vocabulary
+ * consumed by compaction and stack DSE. Spill traffic exists only after
+ * allocation and deliberately uses IR_COPY + OPK_STACK: recognizing semantic
+ * IR_LOAD/IR_STORE + OPK_LOCAL here would collapse a spill value back into an
+ * addressable source-language local and violate the HIR/MIR boundary. */
+static int direct_spill_access(Func* f, const Inst* in, SpillAccess* out) {
+ SpillAccess access;
+ const Operand* stack = NULL;
+ memset(&access, 0, sizeof access);
+ if (!f || !in || (IROp)in->op != IR_COPY || in->nopnds < 2u ||
+ !in->opnds)
+ return 0;
-static int same_spill_access(Func* f, const Inst* a, const Inst* b,
- FrameSlot* slot_out) {
- FrameSlot as = FRAME_SLOT_NONE;
- FrameSlot bs = FRAME_SLOT_NONE;
- if (!spill_local_slot(f, &a->opnds[0], &a->extra.mem, &as)) return 0;
- if (!spill_local_slot(f, &b->opnds[0], &b->extra.mem, &bs)) return 0;
- if (as != bs) return 0;
- if (a->extra.mem.size != b->extra.mem.size) return 0;
- if (a->extra.mem.addr_space != b->extra.mem.addr_space) return 0;
- if (slot_out) *slot_out = as;
- return 1;
-}
+ if (in->opnds[0].kind == OPK_STACK &&
+ in->opnds[1].kind != OPK_STACK) {
+ access.kind = SPILL_ACCESS_STORE;
+ stack = &in->opnds[0];
+ access.value = &in->opnds[1];
+ } else if (in->opnds[1].kind == OPK_STACK &&
+ in->opnds[0].kind != OPK_STACK) {
+ access.kind = SPILL_ACCESS_LOAD;
+ stack = &in->opnds[1];
+ access.value = &in->opnds[0];
+ } else {
+ return 0;
+ }
-static int load_spill_slot(Func* f, const Inst* in, FrameSlot* slot_out) {
- if ((IROp)in->op != IR_LOAD || in->nopnds < 2) return 0;
- return spill_local_slot(f, &in->opnds[1], &in->extra.mem, slot_out);
-}
+ if (!frame_slot_is_spill(f, stack->v.frame_slot)) return 0;
+ {
+ u64 size = cg_type_size(f->c, stack->type);
+ if (!size || size > UINT32_MAX) return 0;
+ access.slot = stack->v.frame_slot;
+ access.size = (u32)size;
+ }
+ access.addr_space = 0u;
-static int store_spill_slot(Func* f, const Inst* in, FrameSlot* slot_out) {
- if ((IROp)in->op != IR_STORE || in->nopnds < 2) return 0;
- return spill_local_slot(f, &in->opnds[0], &in->extra.mem, slot_out);
+ if (!access.size || access.slot == FRAME_SLOT_NONE || !access.value)
+ return 0;
+ if (out) *out = access;
+ return 1;
}
-static int same_spill_slot_and_size(Func* f, const Inst* a, const Inst* b) {
- FrameSlot as = FRAME_SLOT_NONE;
- FrameSlot bs = FRAME_SLOT_NONE;
- if ((IROp)a->op == IR_LOAD) {
- if (!load_spill_slot(f, a, &as)) return 0;
- } else if (!store_spill_slot(f, a, &as)) {
- return 0;
- }
- if ((IROp)b->op == IR_LOAD) {
- if (!load_spill_slot(f, b, &bs)) return 0;
- } else if (!store_spill_slot(f, b, &bs)) {
- return 0;
- }
- return as == bs && a->extra.mem.size == b->extra.mem.size &&
- a->extra.mem.addr_space == b->extra.mem.addr_space;
+static int same_spill_shape(const SpillAccess* a, const SpillAccess* b) {
+ return a && b && a->slot == b->slot && a->size == b->size &&
+ a->addr_space == b->addr_space;
}
static int same_phys_reg(const Operand* a, const Operand* b) {
@@ -105,208 +115,65 @@ static int same_phys_reg(const Operand* a, const Operand* b) {
a->cls == b->cls && a->v.reg == b->v.reg;
}
-/* Count register references inside a single operand. An OPK_INDIRECT can
- * reference the queried register twice (e.g. `[r4 + r4 * 4]`); we return the
- * true count so single-use accounting is exact. */
-static int count_operand_phys_uses(const Operand* op, const Operand* r) {
+static int operand_uses_phys_reg(const Operand* op, const Operand* r) {
if (!op || !r || r->kind != OPK_REG) return 0;
if (op->kind == OPK_REG)
return op->cls == r->cls && op->v.reg == r->v.reg ? 1 : 0;
if (op->kind == OPK_INDIRECT) {
if (r->cls != RC_INT) return 0;
- int n = 0;
- if (op->v.ind.base == r->v.reg) ++n;
- if (op->v.ind.index != (Reg)REG_NONE && op->v.ind.index == r->v.reg) ++n;
- return n;
+ return (op->v.ind.base_kind == OPT_INDIRECT_REG &&
+ op->v.ind.base == r->v.reg) ||
+ (op->v.ind.index_kind == OPT_INDIRECT_REG &&
+ op->v.ind.index != (Reg)REG_NONE &&
+ op->v.ind.index == r->v.reg);
}
return 0;
}
-static int operand_uses_phys_reg(const Operand* op, const Operand* r) {
- return count_operand_phys_uses(op, r) > 0;
+static int inst_uses_phys_reg(Func* f, const Inst* in, const Operand* reg) {
+ OptRegEffects effects;
+ if (!reg || reg->kind != OPK_REG || reg->cls >= OPT_REG_CLASSES ||
+ reg->v.reg >= OPT_MAX_HARD_REGS)
+ return 0;
+ opt_inst_reg_effects(f, in, &effects);
+ return effects.use_count[reg->cls][reg->v.reg];
}
-static int abi_uses_phys_reg(const CGABIValue* v, const Operand* r) {
- int n = 0;
- if (!v) return 0;
- n += count_operand_phys_uses(&v->storage, r);
- for (u32 i = 0; i < v->nparts; ++i)
- n += count_operand_phys_uses(&v->parts[i].op, r);
- return n;
+static int hard_set_has_reg(const OptHardRegSet* set, const Operand* reg) {
+ if (!set || !reg || reg->kind != OPK_REG ||
+ reg->cls >= OPT_REG_CLASSES || reg->v.reg >= OPT_MAX_HARD_REGS)
+ return 0;
+ return (set->cls[reg->cls] & (1u << reg->v.reg)) != 0;
}
-static int inst_uses_phys_reg(const Inst* in, const Operand* r) {
- int n = 0;
- switch ((IROp)in->op) {
- case IR_COPY:
- case IR_CONVERT:
- case IR_UNOP:
- case IR_VA_ARG:
- if (in->nopnds >= 2) n += count_operand_phys_uses(&in->opnds[1], r);
- break;
- case IR_LOAD:
- case IR_ADDR_OF:
- case IR_BITFIELD_LOAD:
- case IR_ATOMIC_LOAD:
- if (in->nopnds >= 2) n += count_operand_phys_uses(&in->opnds[1], r);
- break;
- case IR_BINOP:
- case IR_CMP:
- if (in->nopnds >= 2) n += count_operand_phys_uses(&in->opnds[1], r);
- if (in->nopnds >= 3) n += count_operand_phys_uses(&in->opnds[2], r);
- break;
- case IR_STORE:
- case IR_AGG_COPY:
- case IR_AGG_SET:
- case IR_BITFIELD_STORE:
- case IR_VA_COPY:
- if (in->nopnds >= 1) n += count_operand_phys_uses(&in->opnds[0], r);
- if (in->nopnds >= 2) n += count_operand_phys_uses(&in->opnds[1], r);
- break;
- case IR_CALL: {
- IRCallAux* aux = (IRCallAux*)in->extra.aux;
- if (!aux) break;
- if (aux->use_plan_replay) {
- n += count_operand_phys_uses(&aux->plan.callee, r);
- for (u32 i = 0; i < aux->plan.nargs; ++i)
- n += count_operand_phys_uses(&aux->plan.args[i].src, r);
- } else {
- n += count_operand_phys_uses(&aux->desc.callee, r);
- for (u32 i = 0; i < aux->desc.nargs; ++i)
- n += abi_uses_phys_reg(&aux->desc.args[i], r);
- }
- break;
- }
- case IR_CMP_BRANCH:
- case IR_CONDBR:
- case IR_SWITCH:
- case IR_INDIRECT_BRANCH:
- for (u32 i = 0; i < in->nopnds; ++i)
- n += count_operand_phys_uses(&in->opnds[i], r);
- break;
- case IR_RET: {
- IRRetAux* aux = (IRRetAux*)in->extra.aux;
- if (aux && aux->present) n += abi_uses_phys_reg(&aux->val, r);
- break;
- }
- case IR_SCOPE_BEGIN:
- break;
- case IR_ALLOCA:
- if (in->nopnds >= 2) n += count_operand_phys_uses(&in->opnds[1], r);
- break;
- case IR_VA_START:
- case IR_VA_END:
- if (in->nopnds >= 1) n += count_operand_phys_uses(&in->opnds[0], r);
- break;
- case IR_ATOMIC_STORE:
- if (in->nopnds >= 1) n += count_operand_phys_uses(&in->opnds[0], r);
- if (in->nopnds >= 2) n += count_operand_phys_uses(&in->opnds[1], r);
- break;
- case IR_ATOMIC_RMW:
- if (in->nopnds >= 2) n += count_operand_phys_uses(&in->opnds[1], r);
- if (in->nopnds >= 3) n += count_operand_phys_uses(&in->opnds[2], r);
- break;
- case IR_ATOMIC_CAS:
- if (in->nopnds >= 3) n += count_operand_phys_uses(&in->opnds[2], r);
- if (in->nopnds >= 4) n += count_operand_phys_uses(&in->opnds[3], r);
- if (in->nopnds >= 5) n += count_operand_phys_uses(&in->opnds[4], r);
- break;
- case IR_ASM_BLOCK: {
- IRAsmAux* aux = (IRAsmAux*)in->extra.aux;
- if (!aux) break;
- for (u32 i = 0; i < aux->nin; ++i)
- n += count_operand_phys_uses(&aux->in_ops[i], r);
- break;
- }
- case IR_INTRINSIC: {
- IRIntrinAux* aux = (IRIntrinAux*)in->extra.aux;
- if (!aux) break;
- for (u32 i = 0; i < aux->narg; ++i)
- n += count_operand_phys_uses(&aux->args[i], r);
- break;
- }
- default:
- break;
- }
- return n;
+static int inst_defines_phys_reg(Func* f, const Inst* in, const Operand* reg) {
+ OptRegEffects effects;
+ opt_inst_reg_effects(f, in, &effects);
+ return hard_set_has_reg(&effects.defs, reg);
}
-static int abi_defines_phys_reg(const CGABIValue* v, const Operand* r) {
- int n = 0;
- if (!v) return 0;
- if (same_phys_reg(&v->storage, r)) ++n;
- for (u32 i = 0; i < v->nparts; ++i)
- if (same_phys_reg(&v->parts[i].op, r)) ++n;
- return n;
+static int inst_clobbers_phys_reg(Func* f, const Inst* in,
+ const Operand* reg) {
+ OptRegEffects effects;
+ opt_inst_reg_effects(f, in, &effects);
+ return hard_set_has_reg(&effects.clobbers, reg);
}
-static int inst_defines_phys_reg(const Inst* in, const Operand* r) {
- if (!r || r->kind != OPK_REG) return 0;
- switch ((IROp)in->op) {
- case IR_LOAD_IMM:
- case IR_LOAD_CONST:
- case IR_LOAD_LABEL_ADDR:
- case IR_COPY:
- case IR_LOAD:
- case IR_ADDR_OF:
- case IR_TLS_ADDR_OF:
- case IR_BITFIELD_LOAD:
- case IR_BINOP:
- case IR_UNOP:
- case IR_CMP:
- case IR_CONVERT:
- case IR_ALLOCA:
- case IR_VA_ARG:
- case IR_ATOMIC_LOAD:
- case IR_ATOMIC_RMW:
- return in->nopnds >= 1 && same_phys_reg(&in->opnds[0], r);
- case IR_CALL: {
- IRCallAux* aux = (IRCallAux*)in->extra.aux;
- if (!aux) return 0;
- if (aux->use_plan_replay) {
- for (u32 i = 0; i < aux->plan.nargs; ++i)
- if (aux->plan.args[i].dst_kind == CG_CALL_PLAN_REG &&
- r->cls == aux->plan.args[i].cls &&
- r->v.reg == aux->plan.args[i].dst_reg)
- return 1;
- for (u32 i = 0; i < aux->plan.nrets; ++i)
- if ((r->cls == aux->plan.rets[i].cls &&
- r->v.reg == aux->plan.rets[i].src_reg) ||
- same_phys_reg(&aux->plan.rets[i].dst, r))
- return 1;
- return 0;
- }
- return abi_defines_phys_reg(&aux->desc.ret, r);
- }
- case IR_ATOMIC_CAS:
- return (in->nopnds >= 1 && same_phys_reg(&in->opnds[0], r)) ||
- (in->nopnds >= 2 && same_phys_reg(&in->opnds[1], r));
- case IR_ASM_BLOCK: {
- IRAsmAux* aux = (IRAsmAux*)in->extra.aux;
- if (!aux) return 0;
- for (u32 i = 0; i < aux->nout; ++i)
- if (same_phys_reg(&aux->out_ops[i], r)) return 1;
- if (r->cls < OPT_REG_CLASSES && r->v.reg < 32 &&
- (aux->clobber_mask[r->cls] & (1u << r->v.reg)))
- return 1;
- return 0;
- }
- case IR_INTRINSIC: {
- IRIntrinAux* aux = (IRIntrinAux*)in->extra.aux;
- if (!aux) return 0;
- for (u32 i = 0; i < aux->ndst; ++i)
- if (same_phys_reg(&aux->dsts[i], r)) return 1;
- return 0;
- }
- default:
- return 0;
- }
+static int inst_kills_phys_reg(Func* f, const Inst* in, const Operand* reg) {
+ OptRegEffects effects;
+ OptHardRegSet kills;
+ opt_inst_reg_effects(f, in, &effects);
+ opt_reg_effect_kills(&effects, &kills);
+ return hard_set_has_reg(&kills, reg);
}
/* True if `in` may write to memory. Used to invalidate memory-reading
* producers (IR_LOAD) when the consumer is past an intervening write. */
static int inst_writes_memory(const Inst* in) {
switch ((IROp)in->op) {
+ case IR_COPY:
+ return in->nopnds >= 1u && in->opnds &&
+ in->opnds[0].kind == OPK_STACK;
case IR_STORE:
case IR_AGG_COPY:
case IR_AGG_SET:
@@ -325,6 +192,9 @@ static int inst_writes_memory(const Inst* in) {
static int inst_reads_memory(const Inst* in) {
switch ((IROp)in->op) {
+ case IR_COPY:
+ return in->nopnds >= 2u && in->opnds &&
+ in->opnds[1].kind == OPK_STACK;
case IR_LOAD:
case IR_BITFIELD_LOAD:
case IR_ATOMIC_LOAD:
@@ -408,9 +278,9 @@ typedef struct AddrCseEntry {
/* W5 (O1.md): local (same-block) redundant-load + pure-compute CSE. Two small
* most-recent-wins rings:
*
- * - LOAD ring: the most-recently-loaded `{address, size, addr_space}` and the
- * register that received the value. A later plain (non-observable,
- * non-bit-field) IR_LOAD of the IDENTICAL address+shape, with the producer
+ * - LOAD ring: the most-recently-loaded `{address, result, MemAccess}` and
+ * the register that received the value. A later plain (non-observable,
+ * non-bit-field) IR_LOAD of the IDENTICAL semantic shape, with the producer
* register still live and NO intervening memory write, is rewritten to an
* IR_COPY off the producer. Aliasing is the whole game: ANY memory write
* (inst_writes_memory: store/aggregate/atomic/call/asm/intrinsic) forgets
@@ -418,12 +288,12 @@ typedef struct AddrCseEntry {
* loads always preserves the reload. Volatile/atomic loads
* (opt_mem_observable) are never recorded or reused.
*
- * - COMPUTE ring: the most-recent pure IR_BINOP/IR_UNOP `{op, operands}` and
- * its destination register. A later identical pure compute whose inputs are
- * unchanged since (and whose producer is still live) is rewritten to an
- * IR_COPY off the producer. Pure computes touch no memory, so only register
- * redefinitions/clobbers invalidate them (tracked via ctx->last_def, which a
- * clobber barrier bumps for every register).
+ * - COMPUTE ring: the most-recent pure IR_BINOP/IR_UNOP `{op, mode, result,
+ * operands}` and its destination register. A later identical pure compute
+ * whose inputs are unchanged since (and whose producer is still live) is
+ * rewritten to an IR_COPY off the producer. Pure computes touch no memory,
+ * so only register redefinitions/clobbers invalidate them (tracked via
+ * ctx->last_def, which a clobber barrier bumps for every register).
*
* Both rings are bounded → O(1) per inst → the pass stays linear. Each rewrite
* is the broader, map-based same-block form distinct from the adjacent
@@ -435,14 +305,16 @@ typedef struct LoadCseEntry {
i32 inst_idx; /* producing IR_LOAD index in this BB, -1 if empty */
Operand dst; /* the register that received the loaded value (OPK_REG) */
Operand addr; /* the load's address operand (OPK_INDIRECT/LOCAL/GLOBAL) */
- u32 size; /* MemAccess size */
- u16 addr_space;
+ KitCgTypeId result_type;
+ MemAccess mem;
} LoadCseEntry;
typedef struct ComputeCseEntry {
i32 inst_idx; /* producing IR_BINOP/IR_UNOP index in this BB, -1 if empty */
Operand dst; /* the result register (OPK_REG) */
u16 op; /* IROp (IR_BINOP / IR_UNOP) */
+ u16 flags; /* per-instruction semantic mode */
+ KitCgTypeId result_type;
i64 sub; /* extra.imm = BinOp/UnOp selector */
Operand a; /* operand 1 */
Operand b; /* operand 2 (kind == 0 sentinel for a unary compute) */
@@ -468,8 +340,8 @@ typedef struct CombineCtx {
/* W6 cmp-immediate tracking: the constant currently held by each integer
* hard register, if known. const_valid is a per-RC_INT-reg bit set when a
* reaching IR_LOAD_IMM defined that register and nothing since has redefined
- * or clobbered it (incl. call/asm/intrinsic clobbers, tracked precisely via
- * opt_hard_inst_use_def). Lets a `cmp wN, wM` whose wM holds a small constant
+ * or clobbered it (incl. call/asm/machine clobbers, tracked precisely via
+ * opt_inst_reg_effects). Lets a `cmp wN, wM` whose wM holds a small constant
* fold to `cmp wN, #k`, even across a call that preserves wM (a callee-saved
* reg). Integer-only: cmp immediates are integer. */
i64 const_val[OPT_MAX_HARD_REGS];
@@ -493,67 +365,25 @@ static void ctx_reset(CombineCtx* ctx) {
ctx->block_change_p = 0;
}
-/* True if `in` is a barrier that conservatively invalidates all prior
- * producers in this BB (modelled after the old find_single_direct_use, which
- * treats every IR_CALL as a clobber regardless of explicit ABI describes). */
-static int inst_is_clobber_barrier(const Inst* in) {
- switch ((IROp)in->op) {
- case IR_CALL:
- case IR_ASM_BLOCK:
- case IR_INTRINSIC:
- return 1;
- default:
- return 0;
+static void ctx_record_kills(CombineCtx* ctx, const OptHardRegSet* kills,
+ i32 i) {
+ for (u32 c = 0; c < OPT_REG_CLASSES; ++c) {
+ u32 mask = kills->cls[c];
+ for (Reg r = 0; r < OPT_MAX_HARD_REGS; ++r)
+ if (mask & (1u << r)) ctx->last_def[c][r] = i;
}
}
-static void ctx_record_reg_def(CombineCtx* ctx, const Operand* op, i32 i) {
- if (op->kind != OPK_REG) return;
- if (op->cls >= OPT_REG_CLASSES || op->v.reg >= OPT_MAX_HARD_REGS) return;
- ctx->last_def[op->cls][op->v.reg] = i;
-}
-
-/* After processing inst at index `i`, mark each register it defines (incl.
- * implicit clobbers from CALL / ASM / INTRINSIC) as defined here. Mark
- * memory if this inst writes.
- *
- * Walks the inst's destination operand(s) directly rather than probing every
- * (cls, reg) pair. The barrier case (CALL/ASM/INTRINSIC) keeps a bulk mark
- * because explicit defs/clobbers aren't always populated in IRCallAux at
- * this point in the pipeline. */
+/* Record the canonical explicit definitions and implicit clobbers after one
+ * instruction. Calls and asm no longer erase every producer: only the ABI or
+ * target-declared registers are invalidated. */
static void ctx_record(CombineCtx* ctx, const Inst* in, i32 i) {
+ OptRegEffects effects;
+ OptHardRegSet kills;
if (inst_writes_memory(in)) ctx->last_mem_def = i;
- if (inst_is_clobber_barrier(in)) {
- for (u32 c = 0; c < OPT_REG_CLASSES; ++c)
- for (u32 r = 0; r < OPT_MAX_HARD_REGS; ++r) ctx->last_def[c][r] = i;
- return;
- }
- switch ((IROp)in->op) {
- case IR_LOAD_IMM:
- case IR_LOAD_CONST:
- case IR_LOAD_LABEL_ADDR:
- case IR_COPY:
- case IR_LOAD:
- case IR_ADDR_OF:
- case IR_TLS_ADDR_OF:
- case IR_BITFIELD_LOAD:
- case IR_BINOP:
- case IR_UNOP:
- case IR_CMP:
- case IR_CONVERT:
- case IR_ALLOCA:
- case IR_VA_ARG:
- case IR_ATOMIC_LOAD:
- case IR_ATOMIC_RMW:
- if (in->nopnds >= 1) ctx_record_reg_def(ctx, &in->opnds[0], i);
- break;
- case IR_ATOMIC_CAS:
- if (in->nopnds >= 1) ctx_record_reg_def(ctx, &in->opnds[0], i);
- if (in->nopnds >= 2) ctx_record_reg_def(ctx, &in->opnds[1], i);
- break;
- default:
- break;
- }
+ opt_inst_reg_effects(ctx->f, in, &effects);
+ opt_reg_effect_kills(&effects, &kills);
+ ctx_record_kills(ctx, &kills, i);
}
/* Lookup the producer of (cls, reg) in this BB, if any. Returns -1 if no
@@ -577,8 +407,7 @@ static i32 ctx_prev_def_before(const CombineCtx* ctx, const Operand* reg,
if (!reg || reg->kind != OPK_REG) return -1;
for (i32 j = before_idx - 1; j >= 0; --j) {
const Inst* prev = &ctx->bl->insts[j];
- if (inst_is_clobber_barrier(prev) || inst_defines_phys_reg(prev, reg))
- return j;
+ if (inst_kills_phys_reg(ctx->f, prev, reg)) return j;
}
return -1;
}
@@ -606,21 +435,6 @@ static void ctx_restore_removed_def(CombineCtx* ctx, const Operand* reg,
* Without this live-range scoping, reuse of a scratch physreg later in the
* block (`sxtw x12, ...; mov x13, x12; mov x12, ...`) makes every fold look
* multi-use and combine rejects almost everything. */
-/* True if `in` redefines or clobbers `def`'s physical register. Consults the
- * instruction's precise def/clobber set (the same one liveness uses) rather
- * than a blanket clobber-barrier test: a call clobbers only its caller-saved
- * set, so a value held in a callee-saved register survives it. Treating every
- * call as killing every register made `killed` spuriously true and let combine
- * skip the cross-block live-out check, deleting a still-live def. */
-static int inst_kills_phys_reg(Func* f, const Inst* in, const Operand* def) {
- OptHardRegSet use, d;
- if (!def || def->kind != OPK_REG || def->cls >= OPT_REG_CLASSES ||
- def->v.reg >= 32)
- return 0;
- opt_hard_inst_use_def(f, in, &use, &d);
- return (d.cls[def->cls] & (1u << def->v.reg)) != 0;
-}
-
static int count_uses_in_live_range(Func* f, const Block* bl, i32 prod_idx,
const Operand* def,
int* killed_in_block_out) {
@@ -628,7 +442,7 @@ static int count_uses_in_live_range(Func* f, const Block* bl, i32 prod_idx,
int killed = 0;
for (i32 i = prod_idx + 1; i < (i32)bl->ninsts; ++i) {
const Inst* in = &bl->insts[i];
- n += inst_uses_phys_reg(in, def);
+ n += inst_uses_phys_reg(f, in, def);
if (inst_kills_phys_reg(f, in, def)) {
killed = 1;
break;
@@ -638,14 +452,14 @@ static int count_uses_in_live_range(Func* f, const Block* bl, i32 prod_idx,
return n;
}
-static int use_after_clobber_before_redef(const Block* bl, i32 prod_idx,
- const Operand* def) {
+static int use_after_clobber_before_redef(Func* f, const Block* bl,
+ i32 prod_idx, const Operand* def) {
int saw_clobber = 0;
for (i32 i = prod_idx + 1; i < (i32)bl->ninsts; ++i) {
const Inst* in = &bl->insts[i];
- if (saw_clobber && inst_uses_phys_reg(in, def)) return 1;
- if (inst_defines_phys_reg(in, def)) return 0;
- if (inst_is_clobber_barrier(in)) saw_clobber = 1;
+ if (saw_clobber && inst_uses_phys_reg(f, in, def)) return 1;
+ if (inst_defines_phys_reg(f, in, def)) return 0;
+ if (inst_clobbers_phys_reg(f, in, def)) saw_clobber = 1;
}
return 0;
}
@@ -800,6 +614,13 @@ static int first_return_reg(Func* f, u8 cls, Reg* out) {
return 0;
}
+static int reg_is_emit_temp(Func* f, u8 cls, Reg reg) {
+ if (!f || cls >= OPT_REG_CLASSES || reg >= OPT_MAX_HARD_REGS) return 0;
+ for (u32 i = 0; i < f->emit_temp_reg_count[cls]; ++i)
+ if (f->emit_temp_regs[cls][i] == reg) return 1;
+ return 0;
+}
+
static int ret_scalar_storage(CGABIValue* v, Operand** out) {
if (!v || v->storage.kind != OPK_REG) return 0;
if (v->nparts > 1) return 0;
@@ -807,21 +628,44 @@ static int ret_scalar_storage(CGABIValue* v, Operand** out) {
return 1;
}
-static int is_target_scratch_reg(Func* f, u8 cls, Reg reg) {
- if (!f || cls >= OPT_REG_CLASSES) return 0;
- for (u32 i = 0; i < f->opt_scratch_reg_count[cls]; ++i) {
- if (f->opt_scratch_regs[cls][i] == reg) return 1;
- }
- return 0;
-}
-
/* ---- Rewrite 1: substitute producer source into uses ---- */
-/* Set the indirect's base or index to `new_reg`. */
-static void set_indirect_field(Operand* ind, Reg old_reg, Reg new_reg) {
- if (ind->v.ind.base == old_reg) ind->v.ind.base = new_reg;
- if (ind->v.ind.index != (Reg)REG_NONE && ind->v.ind.index == old_reg)
- ind->v.ind.index = new_reg;
+/* Retarget a direct register use through a physical copy. The register is a
+ * location; the rest of the operand belongs to the consumer and describes how
+ * that location is interpreted there. In particular, same-register-class
+ * conversions can become IR_COPY, so replacing the whole operand can narrow a
+ * compare or ABI move. It would also discard consumer riders such as `shift`.
+ */
+static int retarget_reg_use(Operand* use, const Operand* def,
+ const Operand* src) {
+ if (!use || !def || !src || use->kind != OPK_REG ||
+ src->kind != OPK_REG || !same_phys_reg(use, def) ||
+ use->cls != src->cls)
+ return 0;
+ use->v.reg = src->v.reg;
+ return 1;
+}
+
+/* Substitute a register-valued address component without confusing a MIR
+ * frame-slot id for a hard-register number. Only physical identity follows a
+ * copy: the component type belongs to the use and records how those bits are
+ * interpreted there. This matters when a same-width conversion became a
+ * copy (pointer source consumed as an integer ptrdiff index, for example). */
+static int set_indirect_reg(Operand* ind, Reg old_reg, const Operand* src) {
+ int n = 0;
+ if (!ind || !src || src->kind != OPK_REG) return 0;
+ if (ind->v.ind.base_kind == OPT_INDIRECT_REG &&
+ ind->v.ind.base == old_reg) {
+ ind->v.ind.base = src->v.reg;
+ ++n;
+ }
+ if (ind->v.ind.index_kind == OPT_INDIRECT_REG &&
+ ind->v.ind.index != (Reg)REG_NONE &&
+ ind->v.ind.index == old_reg) {
+ ind->v.ind.index = src->v.reg;
+ ++n;
+ }
+ return n;
}
/* Substitute `def` -> `src` in a single aux register slot (no slot-index
@@ -829,17 +673,10 @@ static void set_indirect_field(Operand* ind, Reg old_reg, Reg new_reg) {
* register). Returns 1 if rewritten. */
static int subst_one_aux_operand(Operand* op, const Operand* def,
const Operand* src) {
- if (op->kind == OPK_REG && same_phys_reg(op, def)) {
- *op = *src;
- return 1;
- }
+ if (retarget_reg_use(op, def, src)) return 1;
if (op->kind == OPK_INDIRECT && src->kind == OPK_REG && def->cls == RC_INT &&
- src->cls == RC_INT &&
- (op->v.ind.base == def->v.reg ||
- (op->v.ind.index != (Reg)REG_NONE && op->v.ind.index == def->v.reg))) {
- set_indirect_field(op, def->v.reg, src->v.reg);
- return 1;
- }
+ src->cls == RC_INT)
+ return set_indirect_reg(op, def->v.reg, src) != 0;
return 0;
}
@@ -866,21 +703,24 @@ static int subst_consumer_operands(Inst* in, const Operand* def,
/* Direct OPK_REG substitution: requires the slot to be on the whitelist. */
if (op->kind == OPK_REG && same_phys_reg(op, def) &&
combine_subst_slot(in, oi, kind, copy_imm_ok)) {
- *op = *src;
- ++n;
- continue;
+ if (kind == SK_REG) {
+ if (retarget_reg_use(op, def, src)) {
+ ++n;
+ continue;
+ }
+ } else {
+ *op = *src;
+ ++n;
+ continue;
+ }
}
/* Indirect base/index substitution: only OPK_REG src may land here. The
* substitution only changes which register computes the address, not the
* value being stored — safe for IR_STORE / IR_ATOMIC_STORE /
* IR_BITFIELD_STORE / IR_AGG_COPY / IR_AGG_SET as well. */
if (op->kind == OPK_INDIRECT && kind == SK_REG && src->kind == OPK_REG &&
- def->kind == OPK_REG && def->cls == RC_INT && src->cls == RC_INT &&
- (op->v.ind.base == def->v.reg ||
- (op->v.ind.index != (Reg)REG_NONE && op->v.ind.index == def->v.reg))) {
- set_indirect_field(op, def->v.reg, src->v.reg);
- ++n;
- }
+ def->kind == OPK_REG && def->cls == RC_INT && src->cls == RC_INT)
+ n += set_indirect_reg(op, def->v.reg, src);
}
/* Aux register uses: IR_CALL plan/desc args + callee. SK_REG only —
@@ -929,8 +769,6 @@ static int try_substitute_for_reg(CombineCtx* ctx, Inst* in, i32 i, u8 cls,
if (ctx_def_changed_since(ctx, prod->opnds[1].cls, prod->opnds[1].v.reg,
prod_idx))
return 0;
- if (is_target_scratch_reg(ctx->f, prod->opnds[1].cls, prod->opnds[1].v.reg))
- return 0;
kind = SK_REG;
src_op = prod->opnds[1];
} else if (pop == IR_LOAD_IMM) {
@@ -1009,10 +847,12 @@ static void try_substitute_aux_operand(CombineCtx* ctx, Inst* in, i32 i,
if (!seen_mark(seen, op->cls, op->v.reg))
*any |= try_substitute_for_reg(ctx, in, i, op->cls, op->v.reg);
} else if (op->kind == OPK_INDIRECT) {
- if (op->v.ind.base != (Reg)REG_NONE &&
+ if (op->v.ind.base_kind == OPT_INDIRECT_REG &&
+ op->v.ind.base != (Reg)REG_NONE &&
!seen_mark(seen, RC_INT, op->v.ind.base))
*any |= try_substitute_for_reg(ctx, in, i, RC_INT, op->v.ind.base);
- if (op->v.ind.index != (Reg)REG_NONE &&
+ if (op->v.ind.index_kind == OPT_INDIRECT_REG &&
+ op->v.ind.index != (Reg)REG_NONE &&
!seen_mark(seen, RC_INT, op->v.ind.index))
*any |= try_substitute_for_reg(ctx, in, i, RC_INT, op->v.ind.index);
}
@@ -1069,7 +909,8 @@ static int try_addr_synth_one_op(CombineCtx* ctx, Inst* in, i32 i,
/* (a/c) base producer is IR_BINOP IADD. reg+reg synthesizes a base+index
* pair (requires no existing index, sub-rule a); reg+imm folds the immediate
* into ofs (sub-rule c, ok with or without an existing index). */
- if (op->v.ind.base != (Reg)REG_NONE) {
+ if (op->v.ind.base_kind == OPT_INDIRECT_REG &&
+ op->v.ind.base != (Reg)REG_NONE) {
Reg b = op->v.ind.base;
i32 prod_idx = ctx_producer_of(ctx, RC_INT, b);
if (prod_idx >= 0 && prod_idx < i) {
@@ -1089,19 +930,24 @@ static int try_addr_synth_one_op(CombineCtx* ctx, Inst* in, i32 i,
ctx->f, ctx->hard_live, ctx->bl->id, &prod_def));
Operand lhs = prod->opnds[1];
Operand rhs = prod->opnds[2];
- int has_no_index = (op->v.ind.index == (Reg)REG_NONE);
+ int has_no_index =
+ op->v.ind.index_kind == OPT_INDIRECT_REG &&
+ op->v.ind.index == (Reg)REG_NONE;
/* reg + reg: base = lhs, index = rhs, scale=0. Needs an empty index
* slot — cannot stack two indices. This synthesizes a new index, a
* structural change, so keep the single-use guard: with other live
* uses the add must stay, and we'd add an index without retiring it. */
if (single_use && has_no_index && lhs.kind == OPK_REG &&
rhs.kind == OPK_REG && lhs.cls == RC_INT && rhs.cls == RC_INT &&
+ opt_indirect_index_type_valid(ctx->f, rhs.type) &&
!producer_def_aliases_source(&prod_def, &lhs) &&
!producer_def_aliases_source(&prod_def, &rhs) &&
!ctx_def_changed_since(ctx, RC_INT, lhs.v.reg, prod_idx) &&
!ctx_def_changed_since(ctx, RC_INT, rhs.v.reg, prod_idx)) {
op->v.ind.base = lhs.v.reg;
+ op->v.ind.base_type = lhs.type;
op->v.ind.index = rhs.v.reg;
+ op->v.ind.index_type = rhs.type;
op->v.ind.log2_scale = 0;
any = 1;
}
@@ -1127,6 +973,7 @@ static int try_addr_synth_one_op(CombineCtx* ctx, Inst* in, i32 i,
(single_use ||
combine_offset_fold_ok(ctx->target, op->type, sum))) {
op->v.ind.base = lhs.v.reg;
+ op->v.ind.base_type = lhs.type;
op->v.ind.ofs = (i32)sum;
any = 1;
}
@@ -1142,6 +989,7 @@ static int try_addr_synth_one_op(CombineCtx* ctx, Inst* in, i32 i,
(single_use ||
combine_offset_fold_ok(ctx->target, op->type, sum))) {
op->v.ind.base = rhs.v.reg;
+ op->v.ind.base_type = rhs.type;
op->v.ind.ofs = (i32)sum;
any = 1;
}
@@ -1151,7 +999,8 @@ static int try_addr_synth_one_op(CombineCtx* ctx, Inst* in, i32 i,
}
/* (b) index producer is IR_BINOP ISHL reg, imm with scale=0. */
- if (op->v.ind.index != (Reg)REG_NONE && op->v.ind.log2_scale == 0) {
+ if (op->v.ind.index_kind == OPT_INDIRECT_REG &&
+ op->v.ind.index != (Reg)REG_NONE && op->v.ind.log2_scale == 0) {
Reg idx = op->v.ind.index;
i32 prod_idx = ctx_producer_of(ctx, RC_INT, idx);
if (prod_idx >= 0 && prod_idx < i) {
@@ -1178,6 +1027,7 @@ static int try_addr_synth_one_op(CombineCtx* ctx, Inst* in, i32 i,
!ctx_def_changed_since(ctx, RC_INT, prod->opnds[1].v.reg,
prod_idx)) {
op->v.ind.index = prod->opnds[1].v.reg;
+ op->v.ind.index_type = prod->opnds[1].type;
op->v.ind.log2_scale = (u8)sh;
any = 1;
}
@@ -1193,7 +1043,8 @@ static int try_addr_synth_one_op(CombineCtx* ctx, Inst* in, i32 i,
* the shift into log2_scale and then this absorbs the sxtw producing that
* index. Gated behind the target capability: only a backend that emits the
* extended-register addressing form is handed an index_ext rider. */
- if (op->v.ind.index != (Reg)REG_NONE &&
+ if (op->v.ind.index_kind == OPT_INDIRECT_REG &&
+ op->v.ind.index != (Reg)REG_NONE &&
op->v.ind.index_ext == OPT_IDX_EXT_NONE && ctx->target &&
ctx->target->can_fold_extend_into_addr &&
ctx->target->can_fold_extend_into_addr(ctx->target)) {
@@ -1222,6 +1073,7 @@ static int try_addr_synth_one_op(CombineCtx* ctx, Inst* in, i32 i,
!ctx_def_changed_since(ctx, RC_INT, prod->opnds[1].v.reg,
prod_idx)) {
op->v.ind.index = prod->opnds[1].v.reg;
+ op->v.ind.index_type = prod->opnds[1].type;
op->v.ind.index_ext =
sign_p ? (u8)OPT_IDX_EXT_SXTW : (u8)OPT_IDX_EXT_UXTW;
any = 1;
@@ -1346,8 +1198,11 @@ static int try_sink(CombineCtx* ctx, Inst* in, i32 i) {
if (p->kind == OPK_REG) {
if (ctx_def_changed_since(ctx, p->cls, p->v.reg, prod_idx)) return 0;
} else if (p->kind == OPK_INDIRECT) {
- if (ctx_def_changed_since(ctx, RC_INT, p->v.ind.base, prod_idx)) return 0;
- if (p->v.ind.index != (Reg)REG_NONE &&
+ if (p->v.ind.base_kind == OPT_INDIRECT_REG &&
+ ctx_def_changed_since(ctx, RC_INT, p->v.ind.base, prod_idx))
+ return 0;
+ if (p->v.ind.index_kind == OPT_INDIRECT_REG &&
+ p->v.ind.index != (Reg)REG_NONE &&
ctx_def_changed_since(ctx, RC_INT, p->v.ind.index, prod_idx))
return 0;
}
@@ -1359,7 +1214,8 @@ static int try_sink(CombineCtx* ctx, Inst* in, i32 i) {
* copy that originally wrote it. */
for (i32 j = prod_idx + 1; j < i; ++j) {
Inst* mid = &ctx->bl->insts[j];
- if (inst_uses_phys_reg(mid, &dst) || inst_defines_phys_reg(mid, &dst))
+ if (inst_uses_phys_reg(ctx->f, mid, &dst) ||
+ inst_kills_phys_reg(ctx->f, mid, &dst))
return 0;
}
@@ -1370,7 +1226,8 @@ static int try_sink(CombineCtx* ctx, Inst* in, i32 i) {
int uses_total =
count_uses_in_live_range(ctx->f, ctx->bl, prod_idx, &src, &killed);
if (uses_total != 1) return 0;
- if (killed && use_after_clobber_before_redef(ctx->bl, prod_idx, &src))
+ if (killed &&
+ use_after_clobber_before_redef(ctx->f, ctx->bl, prod_idx, &src))
return 0;
if (!killed && opt_block_live_out_has_phys_reg(ctx->f, ctx->hard_live,
ctx->bl->id, &src))
@@ -1385,7 +1242,6 @@ static int try_sink(CombineCtx* ctx, Inst* in, i32 i) {
prod->opnds[2] = tmp;
}
prod->opnds[0] = dst;
- if (prod->def != VAL_NONE) prod->def = (Val)dst.v.reg;
/* Update last-def: producer no longer defines src, now defines dst. If src
* had an earlier reaching definition in the block, keep it visible for
@@ -1438,18 +1294,16 @@ static int try_combine_exts(CombineCtx* ctx, Inst* in, i32 i) {
}
/* L6 (O1-PATTERNS): the SIGNED mirror of the ZEXT-of-load fold above. A
- * narrow integer load fills the ENTIRE destination register on every target
- * (native_direct_target.c:1700): a flagged sign-extending load (MF_SEXT_LOAD,
- * `ldrsb`/`ldrsh`) sign-extends to register width, and a plain load
- * (`ldrb`/`ldrh`) zero-extends to register width. A subsequent SXTB/SXTH/SXTW
- * convert is then a no-op in two shapes — drop it to an IR_COPY (copy-prop +
- * DCE retire it; worst case a same-cost register move):
+ * narrow integer load fills the ENTIRE destination register. A backend that
+ * advertises load_sext honors MF_SEXT_LOAD with a sign-extending load
+ * (`ldrsb`/`ldrsh`); every backend's plain load (`ldrb`/`ldrh`) zero-extends.
+ * A subsequent SXTB/SXTH/SXTW convert is then a no-op in two shapes — drop it
+ * to an IR_COPY (copy-prop + DCE retire it; worst case a same-cost move):
*
- * (a) SEXT of a sign-extending load of equal-or-smaller width
- * (`mem.size <= sb`): the load already replicated the loaded sign bit
- * across the whole register, so re-sign-extending from `sb >= mem.size`
- * bytes re-reads bits that are already that sign extension -> identity.
- * (`ldrsb x; sxtb w` / `ldrsb x; sxtw x` — yyjson's signature.)
+ * (a) SEXT of a sign-extending load of equal-or-smaller width, on a target
+ * that honors MF_SEXT_LOAD (`mem.size <= sb`): the load already
+ * replicated the loaded sign bit across the whole register, so
+ * re-sign-extending from `sb >= mem.size` bytes is an identity.
*
* (b) SEXT of a ZERO-extending load whose value cannot have the sign bit set
* in the widened position (`mem.size < sb`, STRICT): the loaded value
@@ -1463,7 +1317,10 @@ static int try_combine_exts(CombineCtx* ctx, Inst* in, i32 i) {
prod->extra.mem.size && prod->nopnds >= 1 &&
same_reg_operand(&prod->opnds[0], &in->opnds[1])) {
int sext_load = (prod->extra.mem.flags & MF_SEXT_LOAD) != 0;
- if ((sext_load && prod->extra.mem.size <= sb) ||
+ int load_sign_extends =
+ sext_load && ctx->target && ctx->target->regs &&
+ ctx->target->regs->load_sext;
+ if ((load_sign_extends && prod->extra.mem.size <= sb) ||
(!sext_load && prod->extra.mem.size < sb)) {
in->op = IR_COPY;
in->nopnds = 2; /* opnds[0]=dst, opnds[1]=src already in place */
@@ -1793,11 +1650,6 @@ static int try_addr_of_cse(CombineCtx* ctx, Inst* in, i32 i) {
continue;
/* A copy of the producer reg into self is a no-op the dst already holds. */
if (same_phys_reg(&in->opnds[0], &e->dst)) continue;
- /* Native emission may reuse a reserved scratch between MIR instructions
- * while materializing an immediate or frame operand. Such clobbers are
- * deliberately invisible to MIR liveness, so extending an address value
- * held in scratch to this later copy is unsound. */
- if (is_target_scratch_reg(ctx->f, e->dst.cls, e->dst.v.reg)) continue;
/* Rewrite `addr_of rD, <addr>` into `copy rD, rP`. */
in->op = (u16)IR_COPY;
in->opnds[1] = e->dst;
@@ -1810,11 +1662,13 @@ static int try_addr_of_cse(CombineCtx* ctx, Inst* in, i32 i) {
/* ---- Rewrite 8 (W5): local same-block redundant-load elimination ---- */
-/* Two load-address operands name the same address iff identical kind+payload.
- * For an OPK_INDIRECT both the base, index, scale, AND offset must match; the
- * register-unchanged-since check is applied separately by the caller. */
+/* Two load-address operands name the same typed address iff their kind and
+ * payload agree. For an OPK_INDIRECT both the base, index, scale, AND offset
+ * must match; the register-unchanged-since check is applied separately by the
+ * caller. */
static int same_load_addr_operand(const Operand* a, const Operand* b) {
- if (a->kind != b->kind) return 0;
+ if (a->kind != b->kind || a->cls != b->cls || a->type != b->type)
+ return 0;
switch (a->kind) {
case OPK_LOCAL:
return a->v.frame_slot == b->v.frame_slot;
@@ -1824,6 +1678,10 @@ static int same_load_addr_operand(const Operand* a, const Operand* b) {
case OPK_INDIRECT:
return a->v.ind.base == b->v.ind.base &&
a->v.ind.index == b->v.ind.index &&
+ a->v.ind.base_kind == b->v.ind.base_kind &&
+ a->v.ind.index_kind == b->v.ind.index_kind &&
+ a->v.ind.base_type == b->v.ind.base_type &&
+ a->v.ind.index_type == b->v.ind.index_type &&
a->v.ind.log2_scale == b->v.ind.log2_scale &&
a->v.ind.index_ext == b->v.ind.index_ext &&
a->v.ind.ofs == b->v.ind.ofs;
@@ -1832,6 +1690,55 @@ static int same_load_addr_operand(const Operand* a, const Operand* b) {
}
}
+/* Alias roots are optimizer metadata rather than part of the bytes returned by
+ * a load, but keeping them in this deliberately conservative local key avoids
+ * letting CSE erase a distinction made by an upstream alias classifier. */
+static int same_load_alias_root(const AliasRoot* a, const AliasRoot* b) {
+ if (a->kind != b->kind) return 0;
+ switch ((AliasKind)a->kind) {
+ case ALIAS_UNKNOWN:
+ case ALIAS_HEAP:
+ return 1;
+ case ALIAS_LOCAL:
+ return a->v.local_id == b->v.local_id;
+ case ALIAS_GLOBAL:
+ return a->v.global == b->v.global;
+ case ALIAS_PARAM:
+ return a->v.param_idx == b->v.param_idx;
+ case ALIAS_STRING:
+ return a->v.string_id == b->v.string_id;
+ default:
+ return 0;
+ }
+}
+
+/* Compare explicit fields rather than struct bytes: MemAccess contains padding
+ * and a tagged union, so memcmp would make the key depend on inactive storage.
+ * Even fields that usually do not alter the returned bits (alignment and alias
+ * classification) stay in this bounded key. That keeps reuse conservative and
+ * aligned with the richer memory key used by O2 GVN. */
+static int same_load_mem_access(const MemAccess* a, const MemAccess* b) {
+ return a->type == b->type && a->size == b->size && a->align == b->align &&
+ a->flags == b->flags && a->addr_space == b->addr_space &&
+ a->bf_offset == b->bf_offset && a->bf_width == b->bf_width &&
+ a->bf_signed == b->bf_signed &&
+ same_load_alias_root(&a->alias, &b->alias);
+}
+
+/* Physical register identity is intentionally absent: CSE expects distinct
+ * destinations. The value interpretation carried by the instruction and its
+ * destination operand must nevertheless agree before a producer can be copied
+ * into the later destination. */
+static int same_cse_result_shape(const Inst* in, KitCgTypeId result_type,
+ const Operand* dst) {
+ const Operand* current;
+ if (!in || !dst || in->nopnds < 1) return 0;
+ current = &in->opnds[0];
+ return current->kind == OPK_REG && dst->kind == OPK_REG &&
+ in->type == current->type && result_type == dst->type &&
+ in->type == result_type && current->cls == dst->cls;
+}
+
/* A plain, CSE-eligible IR_LOAD: dst is a register, address is a
* direct/indirect memory operand, and the access is neither observable
* (volatile/atomic) nor a bit-field rider. */
@@ -1840,13 +1747,10 @@ static int load_is_cseable(const Inst* in) {
if (in->opnds[0].kind != OPK_REG) return 0;
const Operand* addr = &in->opnds[1];
/* W5 owns pointer-deref (OPK_INDIRECT) and global (OPK_GLOBAL) loads — the
- * `*p` / global-variable same-block reload. Direct frame-slot loads
- * (OPK_LOCAL spill/local reloads) are deliberately EXCLUDED: that traffic is
- * the domain of the adjacent spill compaction in opt_combine_compact_block
- * (store/load, load/load, load/store, store/store pairs) and the linear-scan
- * spill machinery, which the doc says to keep distinct from W5. Reusing a
- * spill reload here both duplicates that machinery and is unsound against the
- * way the spill/coalesce path may re-home a frame slot, so it stays out. */
+ * `*p` / global-variable same-block reload. Semantic OPK_LOCAL traffic stays
+ * out of this narrow CSE, while location-MIR spills are represented as
+ * IR_COPY/OPK_STACK and owned by opt_combine_compact_block. Reusing either
+ * here would duplicate its alias/location authority. */
if (addr->kind != OPK_INDIRECT && addr->kind != OPK_GLOBAL) return 0;
if (opt_mem_observable(&in->extra.mem)) return 0;
if (in->extra.mem.bf_width != 0) return 0; /* bit-field rider */
@@ -1860,8 +1764,12 @@ static int load_is_cseable(const Inst* in) {
* happened AT the recorded load's own index. */
if (in->opnds[0].cls == RC_INT && addr->kind == OPK_INDIRECT) {
Reg d = in->opnds[0].v.reg;
- if (addr->v.ind.base == d) return 0;
- if (addr->v.ind.index != (Reg)REG_NONE && addr->v.ind.index == d) return 0;
+ if (addr->v.ind.base_kind == OPT_INDIRECT_REG &&
+ addr->v.ind.base == d)
+ return 0;
+ if (addr->v.ind.index_kind == OPT_INDIRECT_REG &&
+ addr->v.ind.index != (Reg)REG_NONE && addr->v.ind.index == d)
+ return 0;
}
return 1;
}
@@ -1872,10 +1780,12 @@ static int load_is_cseable(const Inst* in) {
static int load_addr_regs_unchanged(const CombineCtx* ctx, const Operand* addr,
i32 since_idx) {
if (addr->kind != OPK_INDIRECT) return 1;
- if (addr->v.ind.base != (Reg)REG_NONE &&
+ if (addr->v.ind.base_kind == OPT_INDIRECT_REG &&
+ addr->v.ind.base != (Reg)REG_NONE &&
ctx_def_changed_since(ctx, RC_INT, addr->v.ind.base, since_idx))
return 0;
- if (addr->v.ind.index != (Reg)REG_NONE &&
+ if (addr->v.ind.index_kind == OPT_INDIRECT_REG &&
+ addr->v.ind.index != (Reg)REG_NONE &&
ctx_def_changed_since(ctx, RC_INT, addr->v.ind.index, since_idx))
return 0;
return 1;
@@ -1887,20 +1797,19 @@ static void load_cse_record(CombineCtx* ctx, const Inst* in, i32 i) {
e->inst_idx = i;
e->dst = in->opnds[0];
e->addr = in->opnds[1];
- e->size = in->extra.mem.size;
- e->addr_space = in->extra.mem.addr_space;
+ e->result_type = in->type;
+ e->mem = in->extra.mem;
ctx->load_cse_next = (ctx->load_cse_next + 1u) % COMBINE_LOAD_CSE_SLOTS;
}
static int try_local_load_cse(CombineCtx* ctx, Inst* in, i32 i) {
if (!load_is_cseable(in)) return 0;
const Operand* addr = &in->opnds[1];
- u32 size = in->extra.mem.size;
- u16 as = in->extra.mem.addr_space;
for (u32 k = 0; k < COMBINE_LOAD_CSE_SLOTS; ++k) {
const LoadCseEntry* e = &ctx->load_cse[k];
if (e->inst_idx < 0 || e->inst_idx >= i) continue;
- if (e->size != size || e->addr_space != as) continue;
+ if (!same_cse_result_shape(in, e->result_type, &e->dst)) continue;
+ if (!same_load_mem_access(&e->mem, &in->extra.mem)) continue;
if (!same_load_addr_operand(&e->addr, addr)) continue;
/* ALIASING: any memory write between the earlier load and now invalidates
* reuse. ctx->last_mem_def is the index of the most-recent memory writer
@@ -1921,13 +1830,9 @@ static int try_local_load_cse(CombineCtx* ctx, Inst* in, i32 i) {
/* Same-class reuse only (an int load into an FP reg, or vice versa, would
* need a cross-class move, not a same-class copy). */
if (in->opnds[0].cls != e->dst.cls) continue;
- /* Never reuse a value that lives in a target scratch register: native emit
- * repurposes the scratch regs to materialize OPK_LOCAL operands between MIR
- * instructions (invisible to MIR-level liveness), so a new copy off such a
- * register can read a clobbered value. */
- if (is_target_scratch_reg(ctx->f, e->dst.cls, e->dst.v.reg)) continue;
/* Rewrite `load rD, [addr]` into `copy rD, rP`. */
in->op = (u16)IR_COPY;
+ in->flags = 0;
in->opnds[1] = e->dst;
in->nopnds = 2;
ctx->block_change_p = 1;
@@ -1960,9 +1865,11 @@ static int same_compute_operand(const Operand* a, const Operand* b) {
/* The L7 shift rider is part of the operand's value: `x2` and `x2,lsl#2`
* are different inputs, so a ridered and a riderless operand must not be
* treated as the same compute (else CSE would drop the shift). */
- return a->cls == b->cls && a->v.reg == b->v.reg && a->shift == b->shift;
+ return a->cls == b->cls && a->v.reg == b->v.reg &&
+ a->type == b->type && a->shift == b->shift;
case OPK_IMM:
- return a->v.imm == b->v.imm && a->type == b->type;
+ return a->cls == b->cls && a->v.imm == b->v.imm &&
+ a->type == b->type;
default:
return 0;
}
@@ -1982,6 +1889,8 @@ static void compute_cse_record(CombineCtx* ctx, const Inst* in, i32 i) {
e->inst_idx = i;
e->dst = in->opnds[0];
e->op = in->op;
+ e->flags = in->flags;
+ e->result_type = in->type;
e->sub = in->extra.imm;
e->a = in->opnds[1];
if ((IROp)in->op == IR_BINOP)
@@ -2000,7 +1909,10 @@ static int try_local_compute_cse(CombineCtx* ctx, Inst* in, i32 i) {
for (u32 k = 0; k < COMBINE_COMPUTE_CSE_SLOTS; ++k) {
const ComputeCseEntry* e = &ctx->compute_cse[k];
if (e->inst_idx < 0 || e->inst_idx >= i) continue;
- if (e->op != in->op || e->sub != in->extra.imm) continue;
+ if (e->op != in->op || e->flags != in->flags ||
+ e->sub != in->extra.imm)
+ continue;
+ if (!same_cse_result_shape(in, e->result_type, &e->dst)) continue;
if (!same_compute_operand(&e->a, a)) continue;
if (op == IR_BINOP) {
if (!same_compute_operand(&e->b, b)) continue;
@@ -2020,15 +1932,9 @@ static int try_local_compute_cse(CombineCtx* ctx, Inst* in, i32 i) {
/* Self-copy is a no-op; skip. Same class only. */
if (same_phys_reg(&in->opnds[0], &e->dst)) continue;
if (in->opnds[0].cls != e->dst.cls) continue;
- /* Never reuse a value that lives in a target scratch register: the
- * post-combine native emit repurposes the scratch regs to materialize
- * OPK_LOCAL operands BETWEEN MIR instructions (invisible to MIR-level
- * liveness), so extending such a register's live range to a new copy reads
- * a clobbered value. (Same hazard the copy-prop path guards in
- * try_substitute_for_reg.) */
- if (is_target_scratch_reg(ctx->f, e->dst.cls, e->dst.v.reg)) continue;
/* Rewrite the recompute into `copy rD, rP`. */
in->op = (u16)IR_COPY;
+ in->flags = 0;
in->opnds[1] = e->dst;
in->nopnds = 2;
in->extra.imm = 0;
@@ -2049,6 +1955,7 @@ static int try_ret_retarget(Func* f, Block* bl, i32 i) {
Reg ret_reg = REG_NONE;
if (!aux || !aux->present || !ret_scalar_storage(&aux->val, &ret_op) ||
!first_return_reg(f, ret_op->cls, &ret_reg) || ret_reg == (Reg)REG_NONE ||
+ reg_is_emit_temp(f, ret_op->cls, ret_reg) ||
ret_reg == ret_op->v.reg)
return 0;
Inst* producer = &bl->insts[i - 1u];
@@ -2141,7 +2048,7 @@ static int try_fold_const_convert(CombineCtx* ctx, Inst* in, i32 i) {
* We track, per integer hard register, the constant a reaching IR_LOAD_IMM put
* there (const_val / const_valid). On a cmp/cmp_branch we replace an immediate-
* slot register operand holding a target-legal constant with that immediate.
- * Tracking is invalidated precisely (opt_hard_inst_use_def gives each inst's
+ * Tracking is invalidated precisely (opt_inst_reg_effects gives each inst's
* def + clobber set), so a constant in a callee-saved register survives calls
* that preserve it. Linear: O(1) bookkeeping per inst. */
@@ -2149,11 +2056,13 @@ static int try_fold_const_convert(CombineCtx* ctx, Inst* in, i32 i) {
* inst at index `i`. Must run for EVERY inst (called from the forward driver),
* so the tracking stays sound across barriers and redefinitions. */
static void ctx_track_const(CombineCtx* ctx, const Inst* in) {
- OptHardRegSet use, def;
- opt_hard_inst_use_def(ctx->f, in, &use, &def);
+ OptRegEffects effects;
+ OptHardRegSet kills;
+ opt_inst_reg_effects(ctx->f, in, &effects);
+ opt_reg_effect_kills(&effects, &kills);
/* Any register this inst defines or clobbers no longer holds its old known
* constant. (RC_INT only — cmp immediates are integer.) */
- ctx->const_valid &= ~def.cls[RC_INT];
+ ctx->const_valid &= ~kills.cls[RC_INT];
/* A load_imm of an integer hard register records the new constant. */
if ((IROp)in->op == IR_LOAD_IMM && in->nopnds >= 1 &&
in->opnds[0].kind == OPK_REG && in->opnds[0].cls == RC_INT &&
@@ -2222,18 +2131,24 @@ static int opt_combine_fold_block(Func* f, Block* bl,
if (enable_o1_combine_rewrites && try_ret_retarget(f, bl, i)) {
ctx.block_change_p = 1;
- /* The producer's def changed; update ctx for the producer at i-1. */
+ /* The producer's destination changed. Remove every old effect no longer
+ * present, then record its canonical post-rewrite effects. */
Inst* prev = &bl->insts[i - 1];
- Operand probe;
- memset(&probe, 0, sizeof probe);
- probe.kind = OPK_REG;
+ OptRegEffects effects;
+ OptHardRegSet kills;
+ opt_inst_reg_effects(f, prev, &effects);
+ opt_reg_effect_kills(&effects, &kills);
for (u8 c = 0; c < OPT_REG_CLASSES; ++c) {
- probe.cls = c;
for (Reg r = 0; r < OPT_MAX_HARD_REGS; ++r) {
- probe.v.reg = r;
if (ctx.last_def[c][r] == i - 1 &&
- !inst_defines_phys_reg(prev, &probe))
+ !(kills.cls[c] & (1u << r))) {
+ Operand probe;
+ memset(&probe, 0, sizeof probe);
+ probe.kind = OPK_REG;
+ probe.cls = c;
+ probe.v.reg = r;
ctx_restore_removed_def(&ctx, &probe, i - 1);
+ }
}
}
ctx_record(&ctx, prev, i - 1);
@@ -2315,9 +2230,15 @@ static int opt_combine_compact_block(Func* f, Block* bl) {
if (w) {
Inst* prev = &bl->insts[w - 1u];
- if ((IROp)prev->op == IR_STORE && (IROp)in->op == IR_LOAD &&
- same_spill_slot_and_size(f, prev, in) &&
- same_reg_operand(&prev->opnds[1], &in->opnds[0])) {
+ SpillAccess prev_access;
+ SpillAccess in_access;
+ int prev_is_spill = direct_spill_access(f, prev, &prev_access);
+ int in_is_spill = direct_spill_access(f, in, &in_access);
+ if (prev_is_spill && in_is_spill &&
+ prev_access.kind == SPILL_ACCESS_STORE &&
+ in_access.kind == SPILL_ACCESS_LOAD &&
+ same_spill_shape(&prev_access, &in_access) &&
+ same_reg_operand(prev_access.value, in_access.value)) {
changed = 1;
continue;
}
@@ -2333,33 +2254,41 @@ static int opt_combine_compact_block(Func* f, Block* bl) {
* rX is dead with the slot unread, W8 stack-DSE + mir_dce retire the
* store. Guard: only when the stored value is a register of the load
* dst's class (an immediate/odd-class store is left as a real reload). */
- if ((IROp)prev->op == IR_STORE && (IROp)in->op == IR_LOAD &&
- same_spill_slot_and_size(f, prev, in) && prev->nopnds >= 2 &&
- in->nopnds >= 2 && prev->opnds[1].kind == OPK_REG &&
- in->opnds[0].kind == OPK_REG &&
- prev->opnds[1].cls == in->opnds[0].cls) {
+ if (prev_is_spill && in_is_spill &&
+ prev_access.kind == SPILL_ACCESS_STORE &&
+ in_access.kind == SPILL_ACCESS_LOAD &&
+ same_spill_shape(&prev_access, &in_access) &&
+ prev_access.value->kind == OPK_REG &&
+ in_access.value->kind == OPK_REG &&
+ prev_access.value->cls == in_access.value->cls) {
in->op = (u16)IR_COPY;
- in->opnds[1] = prev->opnds[1];
+ in->opnds[1] = *prev_access.value;
in->nopnds = 2;
memset(&in->extra, 0, sizeof in->extra);
changed = 1;
bl->insts[w++] = *in;
continue;
}
- if ((IROp)prev->op == IR_LOAD && (IROp)in->op == IR_STORE &&
- same_spill_slot_and_size(f, prev, in) &&
- same_reg_operand(&prev->opnds[0], &in->opnds[1])) {
+ if (prev_is_spill && in_is_spill &&
+ prev_access.kind == SPILL_ACCESS_LOAD &&
+ in_access.kind == SPILL_ACCESS_STORE &&
+ same_spill_shape(&prev_access, &in_access) &&
+ same_reg_operand(prev_access.value, in_access.value)) {
changed = 1;
continue;
}
- if ((IROp)prev->op == IR_LOAD && (IROp)in->op == IR_LOAD &&
- same_spill_slot_and_size(f, prev, in) &&
- same_reg_operand(&prev->opnds[0], &in->opnds[0])) {
+ if (prev_is_spill && in_is_spill &&
+ prev_access.kind == SPILL_ACCESS_LOAD &&
+ in_access.kind == SPILL_ACCESS_LOAD &&
+ same_spill_shape(&prev_access, &in_access) &&
+ same_reg_operand(prev_access.value, in_access.value)) {
changed = 1;
continue;
}
- if ((IROp)prev->op == IR_STORE && (IROp)in->op == IR_STORE &&
- same_spill_access(f, prev, in, NULL)) {
+ if (prev_is_spill && in_is_spill &&
+ prev_access.kind == SPILL_ACCESS_STORE &&
+ in_access.kind == SPILL_ACCESS_STORE &&
+ same_spill_shape(&prev_access, &in_access)) {
bl->insts[w - 1u] = *in;
changed = 1;
continue;
@@ -2377,8 +2306,8 @@ static int opt_combine_compact_block(Func* f, Block* bl) {
* O1.md W8. A forward MIR scan deletes a spill store when a later store in the
* same block fully overwrites the same stack slot before any possible read.
* Deliberately narrower than the parked O2 DSE: V1 handles exact direct frame
- * spill stores only (IR_STORE of OPK_LOCAL(slot) with ALIAS_LOCAL == slot,
- * !observable, no bit-field rider, exact same {slot,size,addr_space}).
+ * spill stores only (location-MIR IR_COPY/OPK_STACK, exact same
+ * {slot,size,addr_space}).
*
* Fast design (no per-block hash table, no O(nslots) clear): dense per-FrameSlot
* side arrays plus a `gen` counter that is bumped at block start AND on any
@@ -2399,21 +2328,43 @@ typedef struct StackDseState {
u32 gen;
} StackDseState;
-/* A direct, DSE-eligible spill store: IR_STORE of a frame-local spill slot,
- * not observable (volatile/atomic), aliasing exactly that local, no bit-field
- * rider. Fills *slot. */
-static int dse_direct_spill_store(Func* f, const Inst* in, FrameSlot* slot) {
- if (!store_spill_slot(f, in, slot)) return 0;
- if (in->extra.mem.bf_width != 0) return 0; /* bit-field rider: bail */
+/* A direct location-MIR spill store (IR_COPY into OPK_STACK). */
+static int dse_direct_spill_store(Func* f, const Inst* in,
+ SpillAccess* access) {
+ SpillAccess found;
+ if (!direct_spill_access(f, in, &found) ||
+ found.kind != SPILL_ACCESS_STORE)
+ return 0;
+ if (access) *access = found;
return 1;
}
-/* A direct read of a spill slot (the read side that must preserve a pending
- * store). Mirrors load_spill_slot but also covers bit-field/atomic loads of the
- * slot defensively; those route through the barrier path anyway, but a plain
- * IR_LOAD of the slot is the common case to clear. */
-static int dse_direct_spill_load(Func* f, const Inst* in, FrameSlot* slot) {
- return load_spill_slot(f, in, slot);
+typedef struct StackDseOperandCtx {
+ StackDseState* state;
+ int forget_defs;
+} StackDseOperandCtx;
+
+/* Every OPK_STACK use is a real read of its spill home, including locations
+ * nested in an indirect address or an aux descriptor. A non-candidate stack
+ * definition also ends the simple pending-store chain. Use the canonical
+ * operand walker so DSE cannot silently acquire a narrower view of MIR than
+ * liveness/emission. For a candidate IR_COPY store, its one destination is the
+ * new pending store and is therefore retained; any nested source use is still
+ * forgotten before that store is recorded. */
+static void stack_dse_forget_operand(Func* f, Inst* in, Operand* op,
+ int is_def, void* arg) {
+ StackDseOperandCtx* ctx = (StackDseOperandCtx*)arg;
+ StackDseState* st = ctx ? ctx->state : NULL;
+ FrameSlot slot;
+ u32 si;
+ (void)f;
+ (void)in;
+ if (!st || !op || op->kind != OPK_STACK || (is_def && !ctx->forget_defs))
+ return;
+ slot = op->v.frame_slot;
+ if (slot == FRAME_SLOT_NONE || slot > st->nslots) return;
+ si = slot - 1u;
+ if (st->seen_gen[si] == st->gen) st->seen_gen[si] = st->gen - 1u;
}
static void stack_dse_block(Func* f, Block* bl, StackDseState* st,
@@ -2424,15 +2375,19 @@ static void stack_dse_block(Func* f, Block* bl, StackDseState* st,
Inst* in = &bl->insts[i];
if ((IROp)in->op == IR_NOP) continue;
- FrameSlot slot = FRAME_SLOT_NONE;
+ SpillAccess access;
+ int candidate_store = dse_direct_spill_store(f, in, &access);
+ StackDseOperandCtx operand_ctx;
+ operand_ctx.state = st;
+ operand_ctx.forget_defs = !candidate_store;
+ opt_walk_inst_operands(f, in, stack_dse_forget_operand, &operand_ctx);
/* Accepted direct spill store: try to kill a prior pending store of the
* exact same key, then record this one. */
- if (dse_direct_spill_store(f, in, &slot) && slot >= 1u &&
- slot <= st->nslots) {
- u32 sz = in->extra.mem.size;
- u16 as = in->extra.mem.addr_space;
- u32 si = slot - 1u;
+ if (candidate_store && access.slot >= 1u && access.slot <= st->nslots) {
+ u32 sz = access.size;
+ u16 as = access.addr_space;
+ u32 si = access.slot - 1u;
if (st->seen_gen[si] == st->gen) {
if (st->last_size[si] == sz && st->last_aspace[si] == as) {
/* Same {slot,size,addr_space}: the prior store is fully overwritten
@@ -2457,30 +2412,24 @@ static void stack_dse_block(Func* f, Block* bl, StackDseState* st,
continue;
}
- /* Accepted direct spill load: clears the pending store for that slot (it is
- * read, so not dead). */
- if (dse_direct_spill_load(f, in, &slot) && slot >= 1u &&
- slot <= st->nslots) {
- u32 si = slot - 1u;
- if (st->seen_gen[si] == st->gen) st->seen_gen[si] = st->gen - 1u;
- continue;
- }
-
- /* A plain memory read that is NOT a direct spill load cannot alias any
- * pending spill slot: FS_SPILL slots are allocator-private and never
- * address-taken, so no computed pointer can reach them. Such a load (a
- * struct field, a global, a heap deref) is therefore transparent to this
- * spill-only DSE — do not bump gen for it. (A direct spill load was already
- * handled above.) */
- if (inst_reads_memory(in)) continue;
-
+ /* The canonical operand walk above already consumed every explicit spill
+ * read, including stack-valued arithmetic, aux operands, and spilled
+ * indirect components. Any remaining plain memory read cannot alias an
+ * FS_SPILL slot: those homes are allocator-private and never address-taken.
+ * A struct-field/global/heap read is therefore transparent to this
+ * spill-only DSE. */
/* Any other memory writer is a full barrier: a non-spill / non-direct store
* (handled per-slot above only for direct spill stores), an aggregate op, a
* call/asm/intrinsic, or an atomic/volatile access. These could, in
* principle, touch state this narrow pass does not model, so forget every
* pending store in O(1) by bumping gen. (Plain register-only insts —
* binop/copy/convert/load_imm/etc. — are transparent.) */
- if (inst_writes_memory(in)) ++st->gen;
+ if (inst_writes_memory(in)) {
+ ++st->gen;
+ continue;
+ }
+
+ if (inst_reads_memory(in)) continue;
}
}
diff --git a/src/opt/pass_dce.c b/src/opt/pass_dce.c
@@ -20,7 +20,9 @@ static int opt_inst_writes_frame_local(const Inst* in) {
case IR_UNOP:
case IR_CMP:
case IR_CONVERT:
- return in->nopnds > 0 && in->opnds[0].kind == OPK_LOCAL;
+ return in->nopnds > 0 &&
+ (in->opnds[0].kind == OPK_LOCAL ||
+ in->opnds[0].kind == OPK_STACK);
default:
return 0;
}
@@ -184,19 +186,22 @@ void opt_dce(Func* f) {
for (u32 ri = bl->ninsts; ri > 0; --ri) {
u32 i = ri - 1u;
Inst* in = &bl->insts[i];
- OptHardRegSet use, def;
+ OptRegEffects effects;
+ OptHardRegSet kills;
if ((IROp)in->op == IR_NOP) continue;
- opt_hard_inst_use_def(f, in, &use, &def);
- if (!opt_inst_has_side_effect(f, in) && !opt_hard_empty(&def) &&
- !opt_hard_intersects(&def, &live)) {
+ opt_inst_reg_effects(f, in, &effects);
+ opt_reg_effect_kills(&effects, &kills);
+ if (!opt_inst_has_side_effect(f, in) &&
+ !opt_hard_empty(&effects.defs) &&
+ !opt_hard_intersects(&effects.defs, &live)) {
continue;
}
- if (!opt_inst_has_side_effect(f, in) && opt_hard_empty(&def) &&
- in->nopnds == 0) {
+ if (!opt_inst_has_side_effect(f, in) &&
+ opt_hard_empty(&effects.defs) && in->nopnds == 0) {
continue;
}
new_insts[w++] = *in;
- opt_hard_live_step(&live, &use, &def);
+ opt_hard_live_step(&live, &effects.uses, &kills);
}
for (u32 i = 0; i < w / 2; ++i) {
Inst tmp = new_insts[i];
diff --git a/src/opt/pass_hard_live.c b/src/opt/pass_hard_live.c
@@ -3,19 +3,6 @@
#include "core/arena.h"
#include "opt/opt_internal.h"
-u32 opt_call_clobber_mask_for(Func* f, const Inst* in, u8 cls) {
- if (cls >= OPT_REG_CLASSES) return 0;
- if (in && (IROp)in->op == IR_CALL) {
- IRCallAux* aux = (IRCallAux*)in->extra.aux;
- if (aux && aux->plan_valid) return aux->plan.clobber_mask[cls];
- }
- return f->opt_caller_saved[cls];
-}
-static void hard_add(OptHardRegSet* s, u8 cls, Reg r) {
- if (cls >= OPT_REG_CLASSES || r >= 32) return;
- s->cls[cls] |= 1u << r;
-}
-
int opt_hard_empty(const OptHardRegSet* s) {
for (u32 c = 0; c < OPT_REG_CLASSES; ++c)
if (s->cls[c]) return 0;
@@ -51,164 +38,6 @@ void opt_hard_live_step(OptHardRegSet* live, const OptHardRegSet* use,
live->cls[c] = (live->cls[c] & ~def->cls[c]) | use->cls[c];
}
-static void hard_use_operand(OptHardRegSet* s, const Operand* op) {
- if (!op) return;
- if (op->kind == OPK_REG) {
- hard_add(s, op->cls, op->v.reg);
- } else if (op->kind == OPK_INDIRECT) {
- hard_add(s, RC_INT, op->v.ind.base);
- if (op->v.ind.index != (Reg)REG_NONE) hard_add(s, RC_INT, op->v.ind.index);
- }
-}
-
-static void hard_def_operand(OptHardRegSet* s, const Operand* op) {
- if (op && op->kind == OPK_REG) hard_add(s, op->cls, op->v.reg);
-}
-
-static void hard_def_clobber_mask(OptHardRegSet* s, const u32* masks) {
- if (!masks) return;
- for (u32 c = 0; c < OPT_REG_CLASSES; ++c) s->cls[c] |= masks[c];
-}
-
-static void hard_use_abivalue(OptHardRegSet* use, const CGABIValue* v) {
- if (!v) return;
- hard_use_operand(use, &v->storage);
- for (u32 i = 0; i < v->nparts; ++i) hard_use_operand(use, &v->parts[i].op);
-}
-
-static void hard_def_abivalue(OptHardRegSet* def, const CGABIValue* v) {
- if (!v) return;
- hard_def_operand(def, &v->storage);
- for (u32 i = 0; i < v->nparts; ++i) hard_def_operand(def, &v->parts[i].op);
-}
-
-void opt_hard_inst_use_def(Func* f, const Inst* in, OptHardRegSet* use,
- OptHardRegSet* def) {
- memset(use, 0, sizeof *use);
- memset(def, 0, sizeof *def);
- switch ((IROp)in->op) {
- case IR_LOAD_IMM:
- case IR_LOAD_CONST:
- case IR_TLS_ADDR_OF:
- if (in->nopnds >= 1) hard_def_operand(def, &in->opnds[0]);
- break;
- case IR_COPY:
- case IR_CONVERT:
- case IR_UNOP:
- case IR_VA_ARG:
- if (in->nopnds >= 1) hard_def_operand(def, &in->opnds[0]);
- if (in->nopnds >= 2) hard_use_operand(use, &in->opnds[1]);
- break;
- case IR_LOAD:
- case IR_ADDR_OF:
- case IR_BITFIELD_LOAD:
- case IR_ATOMIC_LOAD:
- if (in->nopnds >= 1) hard_def_operand(def, &in->opnds[0]);
- if (in->nopnds >= 2) hard_use_operand(use, &in->opnds[1]);
- break;
- case IR_BINOP:
- case IR_CMP:
- if (in->nopnds >= 1) hard_def_operand(def, &in->opnds[0]);
- if (in->nopnds >= 2) hard_use_operand(use, &in->opnds[1]);
- if (in->nopnds >= 3) hard_use_operand(use, &in->opnds[2]);
- break;
- case IR_STORE:
- case IR_AGG_COPY:
- case IR_AGG_SET:
- case IR_BITFIELD_STORE:
- case IR_VA_COPY:
- if (in->nopnds >= 1) hard_use_operand(use, &in->opnds[0]);
- if (in->nopnds >= 2) hard_use_operand(use, &in->opnds[1]);
- break;
- case IR_CALL: {
- IRCallAux* aux = (IRCallAux*)in->extra.aux;
- if (!aux) break;
- if (aux->use_plan_replay) {
- hard_use_operand(use, &aux->plan.callee);
- for (u32 i = 0; i < aux->plan.nargs; ++i) {
- hard_use_operand(use, &aux->plan.args[i].src);
- if (aux->plan.args[i].dst_kind == CG_CALL_PLAN_REG)
- hard_add(def, aux->plan.args[i].cls, aux->plan.args[i].dst_reg);
- }
- for (u32 i = 0; i < aux->plan.nrets; ++i) {
- hard_add(def, aux->plan.rets[i].cls, aux->plan.rets[i].src_reg);
- hard_def_operand(def, &aux->plan.rets[i].dst);
- }
- } else {
- hard_use_operand(use, &aux->desc.callee);
- for (u32 i = 0; i < aux->desc.nargs; ++i)
- hard_use_abivalue(use, &aux->desc.args[i]);
- hard_def_abivalue(def, &aux->desc.ret);
- }
- for (u32 c = 0; c < OPT_REG_CLASSES; ++c) {
- u32 idx = c;
- u32 mask = opt_call_clobber_mask_for(f, in, (u8)idx);
- u32 old = def->cls[idx];
- def->cls[idx] = old | mask;
- }
- break;
- }
- case IR_CMP_BRANCH:
- case IR_CONDBR:
- case IR_SWITCH:
- case IR_INDIRECT_BRANCH:
- for (u32 i = 0; i < in->nopnds; ++i) hard_use_operand(use, &in->opnds[i]);
- break;
- case IR_LOAD_LABEL_ADDR:
- if (in->nopnds >= 1) hard_def_operand(def, &in->opnds[0]);
- break;
- case IR_RET: {
- IRRetAux* aux = (IRRetAux*)in->extra.aux;
- if (aux && aux->present) hard_use_abivalue(use, &aux->val);
- break;
- }
- case IR_SCOPE_BEGIN:
- break;
- case IR_ALLOCA:
- if (in->nopnds >= 1) hard_def_operand(def, &in->opnds[0]);
- if (in->nopnds >= 2) hard_use_operand(use, &in->opnds[1]);
- break;
- case IR_VA_START:
- case IR_VA_END:
- if (in->nopnds >= 1) hard_use_operand(use, &in->opnds[0]);
- break;
- case IR_ATOMIC_STORE:
- if (in->nopnds >= 1) hard_use_operand(use, &in->opnds[0]);
- if (in->nopnds >= 2) hard_use_operand(use, &in->opnds[1]);
- break;
- case IR_ATOMIC_RMW:
- if (in->nopnds >= 1) hard_def_operand(def, &in->opnds[0]);
- if (in->nopnds >= 2) hard_use_operand(use, &in->opnds[1]);
- if (in->nopnds >= 3) hard_use_operand(use, &in->opnds[2]);
- break;
- case IR_ATOMIC_CAS:
- if (in->nopnds >= 1) hard_def_operand(def, &in->opnds[0]);
- if (in->nopnds >= 2) hard_def_operand(def, &in->opnds[1]);
- if (in->nopnds >= 3) hard_use_operand(use, &in->opnds[2]);
- if (in->nopnds >= 4) hard_use_operand(use, &in->opnds[3]);
- if (in->nopnds >= 5) hard_use_operand(use, &in->opnds[4]);
- break;
- case IR_ASM_BLOCK: {
- IRAsmAux* aux = (IRAsmAux*)in->extra.aux;
- if (!aux) break;
- for (u32 i = 0; i < aux->nin; ++i) hard_use_operand(use, &aux->in_ops[i]);
- for (u32 i = 0; i < aux->nout; ++i)
- hard_def_operand(def, &aux->out_ops[i]);
- hard_def_clobber_mask(def, aux->clobber_mask);
- break;
- }
- case IR_INTRINSIC: {
- IRIntrinAux* aux = (IRIntrinAux*)in->extra.aux;
- if (!aux) break;
- for (u32 i = 0; i < aux->narg; ++i) hard_use_operand(use, &aux->args[i]);
- for (u32 i = 0; i < aux->ndst; ++i) hard_def_operand(def, &aux->dsts[i]);
- break;
- }
- default:
- break;
- }
-}
-
static void hard_live_blocks(Func* f, OptHardBlockLive* live) {
for (u32 b = 0; b < f->nblocks; ++b) {
Block* bl = &f->blocks[b];
@@ -216,12 +45,15 @@ static void hard_live_blocks(Func* f, OptHardBlockLive* live) {
memset(&seen_def, 0, sizeof seen_def);
memset(&live[b], 0, sizeof live[b]);
for (u32 i = 0; i < bl->ninsts; ++i) {
- OptHardRegSet use, def;
- opt_hard_inst_use_def(f, &bl->insts[i], &use, &def);
+ OptRegEffects effects;
+ OptHardRegSet kills;
+ opt_inst_reg_effects(f, &bl->insts[i], &effects);
+ opt_reg_effect_kills(&effects, &kills);
for (u32 c = 0; c < OPT_REG_CLASSES; ++c)
- live[b].live_use.cls[c] |= use.cls[c] & ~seen_def.cls[c];
- hard_or(&seen_def, &def);
- hard_or(&live[b].live_def, &def);
+ live[b].live_use.cls[c] |=
+ effects.uses.cls[c] & ~seen_def.cls[c];
+ hard_or(&seen_def, &kills);
+ hard_or(&live[b].live_def, &kills);
}
}
diff --git a/src/opt/pass_lower.c b/src/opt/pass_lower.c
@@ -161,49 +161,42 @@ static u32 live_update_refs_before_active(u64* live, u32 active_words,
static void forbid_preg_reg(Func* f, PReg v, u8 cls, Reg r) {
if (v == PREG_NONE || v == 0 || v >= opt_reg_count(f) ||
- cls >= OPT_REG_CLASSES || r >= 32)
+ cls >= OPT_REG_CLASSES || r >= OPT_MAX_HARD_REGS)
return;
if (opt_reg_cls(f, v) != cls) return;
f->preg_info[v].forbidden_hard_regs |= 1u << r;
}
-static void apply_fixed_asm_operand(Func* f, Operand* op, i32 fixed,
- u8 fixed_cls) {
- if (!op || op->kind != OPK_REG || fixed < 0) return;
- PReg v = (PReg)op->v.reg;
- if (v == PREG_NONE || v == 0 || v >= opt_reg_count(f)) return;
- if (fixed_cls >= OPT_REG_CLASSES || opt_reg_cls(f, v) != fixed_cls) {
- SrcLoc loc = {0, 0, 0};
- compiler_panic(f->c, loc, "opt asm: fixed register class mismatch");
- }
- f->preg_info[v].tied_hard_reg = fixed;
-}
-
-static void apply_allowed_asm_operand(Func* f, Operand* op, u32 allowed,
- u8 allowed_cls) {
- u32 hard_mask = 0;
- if (!op || op->kind != OPK_REG || !allowed) return;
- PReg v = (PReg)op->v.reg;
- if (v == PREG_NONE || v == 0 || v >= opt_reg_count(f)) return;
- if (allowed_cls >= OPT_REG_CLASSES || opt_reg_cls(f, v) != allowed_cls) {
- SrcLoc loc = {0, 0, 0};
- compiler_panic(f->c, loc, "opt asm: allowed register class mismatch");
- }
- for (u32 i = 0; i < f->opt_hard_reg_count[allowed_cls]; ++i) {
- Reg r = f->opt_hard_regs[allowed_cls][i];
- if (r < 32) hard_mask |= 1u << r;
- }
- if (f->preg_info[v].allowed_hard_regs) {
- u32 both = f->preg_info[v].allowed_hard_regs & allowed;
- if (!both) {
- SrcLoc loc = {0, 0, 0};
- compiler_panic(f->c, loc, "opt asm: conflicting allowed register sets");
- }
- f->preg_info[v].allowed_hard_regs = both;
+static void reserve_asm_requirement(Func* f,
+ const IRAsmRegRequirement* req, u64* use,
+ u64* def, u64* live_after) {
+ u32 mask;
+ if (!f || !f->preg_info || !req || !req->present) return;
+ if (req->cls >= OPT_REG_CLASSES)
+ compiler_panic(f->c, (SrcLoc){0, 0, 0},
+ "opt asm: invalid register requirement class");
+ if (req->fixed_reg >= 0) {
+ if ((u32)req->fixed_reg >= OPT_MAX_HARD_REGS)
+ compiler_panic(f->c, (SrcLoc){0, 0, 0},
+ "opt asm: invalid fixed register requirement");
+ mask = 1u << (u32)req->fixed_reg;
} else {
- f->preg_info[v].allowed_hard_regs = allowed;
+ mask = req->allowed_mask;
+ }
+ /* Generic whole-class constraints need no reservation: ordinary allocated
+ * operands already satisfy them, while spills use the emitter's private temp
+ * bank. Fixed/subset constraints reserve their candidate registers only at
+ * this instruction. With an unsplit allocator the implementation is a
+ * conservative per-value forbid, but placement ownership remains local: the
+ * operand itself is staged rather than pinned for its whole live range. */
+ if (!mask) return;
+ for (PReg v = 1; v < opt_reg_count(f); ++v) {
+ if (opt_reg_cls(f, v) != req->cls ||
+ (!bit_has(use, v) && !bit_has(def, v) &&
+ !(live_after && bit_has(live_after, v))))
+ continue;
+ f->preg_info[v].forbidden_hard_regs |= mask;
}
- f->preg_info[v].forbidden_hard_regs |= hard_mask & ~allowed;
}
static void apply_asm_register_constraints(Func* f, Inst* in, u64* use,
@@ -211,27 +204,17 @@ static void apply_asm_register_constraints(Func* f, Inst* in, u64* use,
IRAsmAux* aux = (IRAsmAux*)in->extra.aux;
if (!aux || !f->preg_info) return;
- for (u32 i = 0; i < aux->nout; ++i) {
- i32 fixed = aux->out_fixed_regs ? aux->out_fixed_regs[i] : -1;
- u8 cls = aux->out_fixed_cls ? aux->out_fixed_cls[i] : 0;
- u32 allowed = aux->out_allowed_masks ? aux->out_allowed_masks[i] : 0;
- u8 allowed_cls = aux->out_allowed_cls ? aux->out_allowed_cls[i] : 0;
- apply_allowed_asm_operand(f, &aux->out_ops[i], allowed, allowed_cls);
- apply_fixed_asm_operand(f, &aux->out_ops[i], fixed, cls);
- }
- for (u32 i = 0; i < aux->nin; ++i) {
- i32 fixed = aux->in_fixed_regs ? aux->in_fixed_regs[i] : -1;
- u8 cls = aux->in_fixed_cls ? aux->in_fixed_cls[i] : 0;
- u32 allowed = aux->in_allowed_masks ? aux->in_allowed_masks[i] : 0;
- u8 allowed_cls = aux->in_allowed_cls ? aux->in_allowed_cls[i] : 0;
- apply_allowed_asm_operand(f, &aux->in_ops[i], allowed, allowed_cls);
- apply_fixed_asm_operand(f, &aux->in_ops[i], fixed, cls);
- }
+ for (u32 i = 0; i < aux->nout; ++i)
+ reserve_asm_requirement(f, aux->out_reg_reqs ? &aux->out_reg_reqs[i] : NULL,
+ use, def, live_after);
+ for (u32 i = 0; i < aux->nin; ++i)
+ reserve_asm_requirement(f, aux->in_reg_reqs ? &aux->in_reg_reqs[i] : NULL,
+ use, def, live_after);
for (u32 cls = 0; cls < OPT_REG_CLASSES; ++cls) {
u32 mask = aux->clobber_mask[cls];
if (!mask) continue;
- for (Reg r = 0; r < 32; ++r) {
+ for (Reg r = 0; r < OPT_MAX_HARD_REGS; ++r) {
if ((mask & (1u << r)) == 0) continue;
for (PReg v = 1; v < opt_reg_count(f); ++v) {
if (!bit_has(use, v) && !bit_has(def, v) &&
@@ -257,17 +240,13 @@ static void apply_machine_reg_clobbers(Func* f, Inst* in, u64* def,
for (u32 cls = 0; cls < OPT_REG_CLASSES; ++cls) {
u32 mask = f->inst_clobbers[in->id][cls];
if (!mask) continue;
- for (Reg r = 0; r < 32; ++r) {
+ for (Reg r = 0; r < OPT_MAX_HARD_REGS; ++r) {
if ((mask & (1u << r)) == 0) continue;
for (PReg v = 1; v < opt_reg_count(f); ++v) {
if (!(live_after && bit_has(live_after, v)) || bit_has(def, v))
continue;
if ((u8)opt_reg_cls(f, v) != (u8)cls) continue;
f->preg_info[v].forbidden_hard_regs |= 1u << r;
- /* Record this as a hard clobber so the return-register hint won't later
- * clear the forbid and place the value in a register the instruction
- * destroys (see set_preg_pref_to_ret_reg). */
- f->preg_info[v].clobbered_hard_regs |= 1u << r;
}
}
}
@@ -285,17 +264,22 @@ static int phys_arg_reg_for_index(Func* f, u8 cls, u32 abi_index, Reg* out) {
return 0;
}
-static int hard_available(Func* f, u8 cls, Reg r);
-
static int is_caller_saved(Func* f, u8 cls, Reg r) {
- if (cls >= OPT_REG_CLASSES || r >= 32) return 0;
+ if (cls >= OPT_REG_CLASSES || r >= OPT_MAX_HARD_REGS) return 0;
return (f->opt_caller_saved[cls] & (1u << r)) != 0;
}
+static int is_emit_temp(Func* f, u8 cls, Reg r) {
+ if (!f || cls >= OPT_REG_CLASSES || r >= OPT_MAX_HARD_REGS) return 0;
+ for (u32 i = 0; i < f->emit_temp_reg_count[cls]; ++i)
+ if (f->emit_temp_regs[cls][i] == r) return 1;
+ return 0;
+}
+
static Reg first_ret_reg(Func* f, u8 cls) {
if (!f || cls >= OPT_REG_CLASSES) return REG_NONE;
u32 mask = f->opt_ret_regs[cls];
- for (Reg r = 0; r < 32; ++r)
+ for (Reg r = 0; r < OPT_MAX_HARD_REGS; ++r)
if (mask & (1u << r)) return r;
return REG_NONE;
}
@@ -307,20 +291,18 @@ static void set_preg_pref_to_ret_reg(Func* f, const Operand* op) {
u8 cls = f->preg_info[v].cls;
if (cls >= OPT_REG_CLASSES) return;
Reg hint = first_ret_reg(f, cls);
- if (hint == REG_NONE || hint >= 32) return;
- /* Don't override a real pin. */
- if (f->preg_info[v].tied_hard_reg >= 0) return;
+ if (hint == REG_NONE || hint >= OPT_MAX_HARD_REGS ||
+ is_emit_temp(f, cls, hint))
+ return;
/* A value live across an instruction that clobbers the ret reg cannot live
* there; skip the hint so the allocator places it elsewhere and the return
* copy moves it into the ret reg (e.g. an accumulator returned past a va_arg,
* which uses rax). */
- if (f->preg_info[v].clobbered_hard_regs & (1u << hint)) return;
- /* The hint reg may not be in opt_hard_regs (e.g. x0 on aa64 is reserved
- * as the ABI ret reg, outside aa_int_allocable); the allocator's
- * preferred-reg branch will still consider it via the unit-overlap
- * precision check. Clear any leftover soft forbid bit so the hint isn't
- * silently blocked. */
- f->preg_info[v].forbidden_hard_regs &= ~(1u << hint);
+ if (f->preg_info[v].forbidden_hard_regs & (1u << hint)) return;
+ /* The hint reg may not be in opt_hard_regs (e.g. x0 on aa64 is reserved as
+ * the ABI return register, outside the NATIVE_REG_ALLOCABLE bank); the
+ * allocator's preferred-reg branch can still consider it. A hard forbid
+ * above always wins. */
f->preg_info[v].preferred_hard_reg = (i8)hint;
}
@@ -336,14 +318,14 @@ static void set_preg_pref_for_abivalue(Func* f, const CGABIValue* v) {
* arg reg for the i-th call argument). */
static void set_preg_pref_to_arg_reg(Func* f, const Operand* op, Reg hint) {
if (!op || op->kind != OPK_REG) return;
- if (hint == REG_NONE || hint >= 32) return;
+ if (hint == REG_NONE || hint >= OPT_MAX_HARD_REGS) return;
PReg v = (PReg)op->v.reg;
if (v == PREG_NONE || v == 0 || v >= opt_reg_count(f)) return;
u8 cls = f->preg_info[v].cls;
if (cls >= OPT_REG_CLASSES) return;
- if (f->preg_info[v].tied_hard_reg >= 0) return;
+ if (is_emit_temp(f, cls, hint)) return;
if (f->preg_info[v].preferred_hard_reg >= 0) return;
- f->preg_info[v].forbidden_hard_regs &= ~(1u << hint);
+ if (f->preg_info[v].forbidden_hard_regs & (1u << hint)) return;
f->preg_info[v].preferred_hard_reg = (i8)hint;
}
@@ -405,10 +387,10 @@ static void set_preg_pref_for_params(Func* f) {
if (*counter < 8u && phys_arg_reg_for_index(f, cls, *counter, &hint)) {
PReg v = (PReg)p->storage.v.reg;
if (v != PREG_NONE && v != 0 && v < opt_reg_count(f) &&
- f->preg_info[v].cls == cls && f->preg_info[v].tied_hard_reg < 0 &&
+ f->preg_info[v].cls == cls &&
f->preg_info[v].preferred_hard_reg < 0 && hint != REG_NONE &&
- hint < 32) {
- f->preg_info[v].forbidden_hard_regs &= ~(1u << hint);
+ hint < OPT_MAX_HARD_REGS && !is_emit_temp(f, cls, hint) &&
+ !(f->preg_info[v].forbidden_hard_regs & (1u << hint))) {
f->preg_info[v].preferred_hard_reg = (i8)hint;
}
}
@@ -473,9 +455,9 @@ static void set_preg_pref_for_call_args(Func* f, const CGCallDesc* desc) {
* copy then emits `mov x0, x8`. Propagating the hint from v_arg to v_value
* lets both land at x0 and turns the copy into an identity move that
* combine elides. Walks insts; for each IR_COPY whose def has a hint and
- * whose single OPK_REG source operand has none, propagate (and clear the
- * source's forbid for the hint reg). Safe because the copy itself dies
- * once both sides share the reg. */
+ * whose single OPK_REG source operand has none, propagate when that placement
+ * remains legal. Safe because the copy itself dies once both sides share the
+ * register. */
static void propagate_hint_through_copies(Func* f) {
if (!f || !f->preg_info) return;
for (u32 b = 0; b < f->nblocks; ++b) {
@@ -492,16 +474,12 @@ static void propagate_hint_through_copies(Func* f) {
if (src == PREG_NONE || src == 0 || src >= opt_reg_count(f)) continue;
i8 dst_pref = f->preg_info[dst].preferred_hard_reg;
if (dst_pref < 0) continue;
- if (f->preg_info[src].tied_hard_reg >= 0) continue;
if (f->preg_info[src].preferred_hard_reg >= 0) continue;
if (f->preg_info[dst].cls != f->preg_info[src].cls) continue;
- /* Don't propagate the hint into a value clobbered out of that register by
- * a machine instruction live across it (e.g. a loop accumulator returned
- * past a va_arg/idiv): it cannot live there. Leave it allocated
- * elsewhere; the copy moves it into the hinted register. */
- if (f->preg_info[src].clobbered_hard_regs & (1u << (Reg)dst_pref))
+ /* A boundary forbid always outranks the copy-placement preference (for
+ * example a loop accumulator returned past va_arg/idiv). */
+ if (f->preg_info[src].forbidden_hard_regs & (1u << (Reg)dst_pref))
continue;
- f->preg_info[src].forbidden_hard_regs &= ~(1u << (Reg)dst_pref);
f->preg_info[src].preferred_hard_reg = dst_pref;
}
}
@@ -561,13 +539,6 @@ static void apply_abi_aliasing_hints(Func* f) {
typedef struct OptAllocator OptAllocator;
-static int hard_available(Func* f, u8 cls, Reg r) {
- if (cls >= OPT_REG_CLASSES) return 0;
- for (u32 i = 0; i < f->opt_hard_reg_count[cls]; ++i)
- if (f->opt_hard_regs[cls][i] == r) return 1;
- return 0;
-}
-
static const CGPhysRegInfo* phys_info_for(Func* f, u8 cls, Reg r) {
if (cls >= OPT_REG_CLASSES) return NULL;
for (u32 i = 0; i < f->opt_phys_reg_count[cls]; ++i)
@@ -603,7 +574,9 @@ static void spill_slot_add_priority(Func* f, FrameSlot slot, u32 cost) {
s->priority = sum < s->priority ? 0xffffffffu : sum; /* saturate on overflow */
}
-static u32 hard_loc_bit(u8 cls, Reg r) { return ((u32)cls * 32u) + (u32)r; }
+static u32 hard_loc_bit(u8 cls, Reg r) {
+ return ((u32)cls * OPT_MAX_HARD_REGS) + (u32)r;
+}
typedef struct OptAllocGroupInfo {
PReg root;
@@ -612,9 +585,7 @@ typedef struct OptAllocGroupInfo {
u32 live_across_call_freq; /* sum over group members (W3-coalesce aware) */
u32 first;
u32 last;
- i32 tied_hard_reg;
u32 forbidden_hard_regs;
- u32 allowed_hard_regs;
u8 cls;
u8 pad[3];
} OptAllocGroupInfo;
@@ -636,7 +607,7 @@ typedef struct OptAllocator {
u64* used_locs;
u32 point_count;
u32 loc_words; /* width of one row, in u64 words */
- u32 hard_loc_bits; /* OPT_REG_CLASSES * 32 */
+ u32 hard_loc_bits; /* OPT_REG_CLASSES * OPT_MAX_HARD_REGS */
/* Stack slot table (parallel arrays). */
FrameSlot* stack_slots;
@@ -741,9 +712,6 @@ static u32 hard_reg_alloc_score(Func* f, const OptAllocator* a,
static int alloc_candidate_higher(const OptAllocCandidate* a,
const OptAllocCandidate* b) {
- int a_tied = a->gi.tied_hard_reg >= 0;
- int b_tied = b->gi.tied_hard_reg >= 0;
- if (a_tied != b_tied) return a_tied > b_tied;
if (a->gi.spill_cost != b->gi.spill_cost)
return a->gi.spill_cost > b->gi.spill_cost;
if (a->gi.live_length != b->gi.live_length)
@@ -811,7 +779,6 @@ static void alloc_group_info(Func* f, const OptAllocator* a,
memset(out, 0, sizeof *out);
out->root = root;
out->first = (u32)~0u;
- out->tied_hard_reg = -1;
out->cls = f->preg_info[root].cls;
for (PReg v = alloc_group_first_member(a, root); v != 0;
v = alloc_group_next_member(a, root, v)) {
@@ -824,46 +791,27 @@ static void alloc_group_info(Func* f, const OptAllocator* a,
if (first < out->first) out->first = first;
if (vi->last_pos > out->last) out->last = vi->last_pos;
out->forbidden_hard_regs |= vi->forbidden_hard_regs;
- if (vi->allowed_hard_regs) {
- if (out->allowed_hard_regs) {
- u32 both = out->allowed_hard_regs & vi->allowed_hard_regs;
- if (!both) {
- SrcLoc loc = {0, 0, 0};
- compiler_panic(f->c, loc,
- "opt asm: conflicting allowed register sets");
- }
- out->allowed_hard_regs = both;
- } else {
- out->allowed_hard_regs = vi->allowed_hard_regs;
- }
- }
- if (vi->tied_hard_reg >= 0) out->tied_hard_reg = vi->tied_hard_reg;
}
if (out->first == (u32)~0u) out->first = 0;
}
-static void opt_init_preg_info_from_ranges(Func* f,
- const OptLiveRangeSet* ranges) {
- OptPRegInfo* old = f->preg_info;
+/* Rebuild allocation metadata exclusively from the current HIR and its live
+ * ranges. OptPRegInfo and OptLoc are products of this pass, never inputs: all
+ * hard constraints and ABI preferences are derived immediately after this
+ * reset. Carrying either table across a rerun makes removed clobbers,
+ * constraints, hints, and spill locations silently authoritative. */
+static void opt_rebuild_preg_info_from_ranges(Func* f,
+ const OptLiveRangeSet* ranges) {
OptPRegInfo* info = arena_zarray(f->arena, OptPRegInfo,
opt_reg_count(f) ? opt_reg_count(f) : 1u);
+ f->preg_locs = NULL;
for (PReg v = 0; v < opt_reg_count(f); ++v) {
- i32 tied = old ? old[v].tied_hard_reg : -1;
- u32 forbidden = old ? old[v].forbidden_hard_regs : 0;
- u32 allowed = old ? old[v].allowed_hard_regs : 0;
- u32 clobbered = old ? old[v].clobbered_hard_regs : 0;
- u32 old_frequency = old ? old[v].frequency : 0;
- i8 pref = old ? old[v].preferred_hard_reg : (i8)-1;
OptPRegInfo* vi = &info[v];
- vi->tied_hard_reg = tied;
- vi->preferred_hard_reg = pref;
+ vi->preferred_hard_reg = (i8)-1;
vi->hard_reg = REG_NONE;
vi->spill_slot = FRAME_SLOT_NONE;
vi->alloc_kind = OPT_ALLOC_NONE;
vi->cls = opt_reg_cls(f, v);
- vi->forbidden_hard_regs = forbidden;
- vi->allowed_hard_regs = allowed;
- vi->clobbered_hard_regs = clobbered;
if (!ranges || v == PREG_NONE || v == 0 ||
ranges->first_range_by_preg[v] == OPT_RANGE_NONE) {
continue;
@@ -885,7 +833,6 @@ static void opt_init_preg_info_from_ranges(Func* f,
vi->live_across_call_freq = ranges->live_across_call_freq_by_preg[v];
vi->spill_cost = ranges->spill_cost_by_preg[v];
vi->frequency = vi->spill_cost;
- if (old_frequency > vi->frequency) vi->frequency = old_frequency;
}
f->preg_info = info;
}
@@ -1106,7 +1053,7 @@ static void opt_assign_ranges(Func* f, const OptLiveRangeSet* ranges,
OptAllocator* a) {
memset(a, 0, sizeof *a);
a->point_count = ranges->point_count ? ranges->point_count : 1u;
- a->hard_loc_bits = OPT_REG_CLASSES * 32u;
+ a->hard_loc_bits = OPT_REG_CLASSES * OPT_MAX_HARD_REGS;
a->loc_words = alloc_loc_words_for_bits(a->hard_loc_bits);
a->used_locs =
arena_zarray(f->arena, u64, (u64)a->point_count * a->loc_words);
@@ -1146,52 +1093,12 @@ static void opt_assign_ranges(Func* f, const OptLiveRangeSet* ranges,
u8 cls = gi.cls;
alloc_compute_group_conflicts(f, a, ranges, v);
- if (gi.tied_hard_reg >= 0) {
- Reg fixed = (Reg)gi.tied_hard_reg;
- /* Machinize has already validated inline-asm hard-register pins against
- * the target's operand-register policy. Some legal pins are ABI registers
- * outside the standard allocable set (aa64 x0, rv64 a7), so the allocator
- * accepts validated physical registers here and relies on the
- * conflict/clobber checks below for placement correctness. */
- if (!hard_available(f, cls, fixed) && !phys_info_for(f, cls, fixed)) {
- SrcLoc loc = {0, 0, 0};
- compiler_panic(
- f->c, loc,
- "opt regalloc: fixed hard reg %u unavailable in class %u",
- (unsigned)fixed, (unsigned)cls);
- }
- if (fixed < 32 && (gi.forbidden_hard_regs & (1u << fixed))) {
- SrcLoc loc = {0, 0, 0};
- compiler_panic(f->c, loc,
- "opt regalloc: fixed hard reg %u is clobbered",
- (unsigned)fixed);
- }
- if (fixed >= 32 || (gi.allowed_hard_regs &&
- (gi.allowed_hard_regs & (1u << fixed)) == 0)) {
- SrcLoc loc = {0, 0, 0};
- compiler_panic(f->c, loc,
- "opt regalloc: fixed hard reg %u violates asm "
- "constraint",
- (unsigned)fixed);
- }
- u32 bit = hard_loc_bit(cls, fixed);
- if (fixed >= 32 || alloc_group_conflicts_bit(a, bit)) {
- SrcLoc loc = {0, 0, 0};
- compiler_panic(f->c, loc, "opt regalloc: conflicting fixed hard reg %u",
- (unsigned)fixed);
- }
- alloc_assign_group_hard(f, a, ranges, v, fixed);
- continue;
- }
-
int found = 0;
Reg best = REG_NONE;
u32 best_score = 0xffffffffu;
for (u32 r = 0; r < f->opt_hard_reg_count[cls]; ++r) {
Reg hr = f->opt_hard_regs[cls][r];
- if (hr >= 32) continue;
- if (gi.allowed_hard_regs && (gi.allowed_hard_regs & (1u << hr)) == 0)
- continue;
+ if (hr >= OPT_MAX_HARD_REGS) continue;
if (gi.forbidden_hard_regs & (1u << hr)) continue;
u32 bit = hard_loc_bit(cls, hr);
if (alloc_group_conflicts_bit(a, bit)) continue;
@@ -1202,28 +1109,10 @@ static void opt_assign_ranges(Func* f, const OptLiveRangeSet* ranges,
best_score = score;
}
}
- if (gi.allowed_hard_regs) {
- for (Reg hr = 0; hr < 32; ++hr) {
- const CGPhysRegInfo* pi;
- if ((gi.allowed_hard_regs & (1u << hr)) == 0) continue;
- if (hard_available(f, cls, hr)) continue;
- if (gi.forbidden_hard_regs & (1u << hr)) continue;
- pi = phys_info_for(f, cls, hr);
- if (!pi || (pi->flags & CG_REG_RESERVED)) continue;
- u32 bit = hard_loc_bit(cls, hr);
- if (alloc_group_conflicts_bit(a, bit)) continue;
- u32 score = hard_reg_alloc_score(f, a, vi, hr);
- if (!found || score < best_score) {
- found = 1;
- best = hr;
- best_score = score;
- }
- }
- }
- /* Also consider the preferred hard reg if it's outside the standard
- * allocable set (e.g. x0 on aa64: reserved as the ABI ret reg, not in
- * aa_int_allocable). Used by apply_abi_aliasing_hints to let an IR_CALL
- * result PReg or IR_RET value PReg land directly in x0, eliding the
+ /* Also consider the preferred hard reg if it's outside the standard O1
+ * allocation bank (e.g. x0 on aa64: reserved as the ABI return register,
+ * without NATIVE_REG_ALLOCABLE). Used by apply_abi_aliasing_hints to let
+ * an IR_CALL result PReg or IR_RET value PReg land directly in x0, eliding the
* materialization move emit_call/emit_ret would otherwise emit. Only
* short-lived PRegs benefit: a value live across a call cannot survive in
* a caller-saved hint reg, and this is the *only* path that can reach an
@@ -1233,12 +1122,12 @@ static void opt_assign_ranges(Func* f, const OptLiveRangeSet* ranges,
* below would otherwise take the hint reg regardless of score, parking a
* cross-call value in x0 where it collides with the next call's result. */
if (vi->preferred_hard_reg >= 0 &&
+ !is_emit_temp(f, cls, (Reg)vi->preferred_hard_reg) &&
!(gi.live_across_call_freq &&
is_caller_saved(f, cls, (Reg)vi->preferred_hard_reg))) {
Reg hint = (Reg)vi->preferred_hard_reg;
int already_tried = 0;
- if (hint < 32 &&
- (!gi.allowed_hard_regs || (gi.allowed_hard_regs & (1u << hint)))) {
+ if (hint < OPT_MAX_HARD_REGS) {
for (u32 r = 0; r < f->opt_hard_reg_count[cls]; ++r) {
if (f->opt_hard_regs[cls][r] == hint) {
already_tried = 1;
@@ -1297,10 +1186,6 @@ static void opt_assign_ranges(Func* f, const OptLiveRangeSet* ranges,
}
if (found) {
alloc_assign_group_hard(f, a, ranges, v, best);
- } else if (gi.allowed_hard_regs) {
- SrcLoc loc = {0, 0, 0};
- compiler_panic(f->c, loc,
- "opt regalloc: no hard register satisfies asm constraint");
} else {
/* gi.spill_cost is the group's aggregated spill-traffic metric
* (alloc_group_info sums each member's frequency/spill_cost). Thread it to
@@ -1395,22 +1280,11 @@ static void out_prepend_inst(Func* f, RewriteOut* out, const Inst* in) {
*dst = *in;
}
-static MemAccess spill_mem(Func* f, PReg v) {
- MemAccess m;
- memset(&m, 0, sizeof m);
- m.type = opt_reg_type(f, v);
- m.size = type_size_fallback(f, opt_reg_type(f, v));
- m.align = m.size >= 8 ? 8 : m.size;
- m.alias.kind = ALIAS_LOCAL;
- m.alias.v.local_id = (i32)spill_slot_for(f, v);
- return m;
-}
-
static Operand spill_addr(Func* f, PReg v) {
Operand o;
memset(&o, 0, sizeof o);
- o.kind = OPK_LOCAL;
- o.cls = RC_INT;
+ o.kind = OPK_STACK;
+ o.cls = opt_preg_loc_cls(f, v);
o.type = opt_reg_type(f, v);
o.v.frame_slot = spill_slot_for(f, v);
return o;
@@ -1427,38 +1301,29 @@ static Operand hard_operand(Func* f, PReg v) {
}
static void append_store_preg(Func* f, RewriteList* out, PReg v) {
- Inst* st = list_push(f, out, IR_STORE);
+ Inst* st = list_push(f, out, IR_COPY);
st->opnds = arena_array(f->arena, Operand, 2);
st->opnds[0] = spill_addr(f, v);
st->opnds[1] = hard_operand(f, v);
st->nopnds = 2;
- st->extra.mem = spill_mem(f, v);
}
static void append_load_preg(Func* f, RewriteList* out, PReg v) {
- Inst* ld = list_push(f, out, IR_LOAD);
+ Inst* ld = list_push(f, out, IR_COPY);
ld->opnds = arena_array(f->arena, Operand, 2);
ld->opnds[0] = hard_operand(f, v);
ld->opnds[1] = spill_addr(f, v);
ld->nopnds = 2;
- ld->extra.mem = spill_mem(f, v);
-}
-
-static Reg scratch_for(Func* f, u8 cls, u32* next) {
- u32 n = f->opt_scratch_reg_count[cls];
- if (!n) return REG_NONE;
- Reg r = f->opt_scratch_regs[cls][*next % n];
- ++*next;
- return r;
}
/* ---------------------------------------------------------------------------
* Rematerialization (doc/plan/O1.md W2).
*
* A spilled value whose single def is *cheaper to recompute than to reload* is
- * recomputed at each use instead of reloaded from its slot; the spill store is
- * dropped entirely (the original def then writes a dead scratch, removed by
- * mir_dce). The v1 set is restricted to input-less producers whose recompute is
+ * carried as a location recipe at each use instead of occupying a spill home;
+ * the original producer is omitted from MIR. Native emission realizes the
+ * recipe inside the consuming instruction's temporary scope. The v1 set is
+ * restricted to input-less producers whose recompute is
* <= reload cost, so the recompute is valid at any use site:
* - IR_LOAD_IMM whose immediate materializes in <= 2 instructions
* (movz, or movz;movk -- i.e. <= 2 nonzero 16-bit halfwords);
@@ -1522,45 +1387,25 @@ static void remat_count_def(Func* f, Inst* in, Operand* op, int is_def,
/* Linear pre-pass: record remat_def[v] iff v has exactly one def and that def
* is in the v1 set. One walk to count defs + snapshot candidate defs, one PReg
* sweep to drop multiply-defined entries. */
-/* Drop `op`'s PReg from the remat set (used to force a spill store + slot reload
- * instead of a recompute-into-scratch). */
+/* Drop `op`'s PReg from the remat set. */
static void remat_exclude_operand(RematInfo* ri, const Operand* op) {
if (!op || op->kind != OPK_REG) return;
PReg v = (PReg)op->v.reg;
if (v != PREG_NONE && v != 0 && v < ri->nregs) ri->remat_def[v] = NULL;
}
-/* Exclude every value used as a call argument from rematerialization. ALL of a
- * call's arguments are live simultaneously at the call, but a spilled remat arg
- * recomputes into the round-robin opt-scratch registers
- * (rewrite_call_arg_operand). With more simultaneously-live spilled args than
- * scratch registers — arm32 has only ip/lr yet up to four register args — the
- * round-robin aliases them and they clobber each other (the many-int variadic
- * O1 miscompile). Keeping these out of the remat set preserves their spill
- * store, so the arg rewrite reads each one from its own slot. Other targets have
- * enough registers that call args rarely spill, so this is a no-op there. */
-static void remat_exclude_call_args(Func* f, RematInfo* ri) {
- for (u32 b = 0; b < f->nblocks; ++b) {
- Block* bl = &f->blocks[b];
- for (u32 i = 0; i < bl->ninsts; ++i) {
- Inst* in = &bl->insts[i];
- IRCallAux* aux;
- if ((IROp)in->op != IR_CALL) continue;
- aux = (IRCallAux*)in->extra.aux;
- if (!aux) continue;
- if (aux->use_plan_replay) {
- for (u32 k = 0; k < aux->plan.nargs; ++k)
- remat_exclude_operand(ri, &aux->plan.args[k].src);
- } else {
- for (u32 k = 0; k < aux->desc.nargs; ++k) {
- CGABIValue* av = (CGABIValue*)&aux->desc.args[k];
- remat_exclude_operand(ri, &av->storage);
- for (u32 p = 0; p < av->nparts; ++p)
- remat_exclude_operand(ri, (Operand*)&av->parts[p].op);
- }
- }
- }
- }
+/* Effective-address components have a deliberately compact reg/frame encoding,
+ * not the full scalar recipe union. Keep an immediate/frame-address producer
+ * in its stable spill home when it is used as an embedded base or index. The
+ * walker marks these transient component operands explicitly. */
+static void remat_exclude_indirect_use(Func* f, Inst* in, Operand* op,
+ int is_def, void* arg) {
+ (void)f;
+ (void)in;
+ if (is_def || !op || op->kind != OPK_REG ||
+ !(op->flags & OPT_OPERAND_WALK_INDIRECT_PART))
+ return;
+ remat_exclude_operand((RematInfo*)arg, op);
}
static void opt_mark_remat(Func* f, RematInfo* ri) {
@@ -1593,7 +1438,11 @@ static void opt_mark_remat(Func* f, RematInfo* ri) {
}
for (u32 v = 0; v < nregs; ++v)
if (def_count[v] != 1u) ri->remat_def[v] = NULL;
- remat_exclude_call_args(f, ri);
+ for (u32 b = 0; b < f->nblocks; ++b) {
+ Block* bl = &f->blocks[b];
+ for (u32 i = 0; i < bl->ninsts; ++i)
+ opt_walk_inst_operands(f, &bl->insts[i], remat_exclude_indirect_use, ri);
+ }
}
static Inst* remat_def_for(const RematInfo* ri, PReg v) {
@@ -1603,43 +1452,41 @@ static Inst* remat_def_for(const RematInfo* ri, PReg v) {
}
typedef struct RewriteCtx {
- RewriteList* before;
- RewriteList* after;
const RematInfo* remat;
- u32 next_scratch[OPT_REG_CLASSES];
} RewriteCtx;
-/* Shared "materialize this spilled PReg at this use" helper. Every spilled-use
- * rewrite path routes through this. When v rematerializes, clone its recorded
- * def into `before` with dst = scratch; otherwise append an IR_LOAD from the
- * slot. The caller has already pointed the operand at `scratch`. */
-static void remat_or_reload_use(Func* f, RewriteCtx* c, Operand* op, PReg v,
- Reg scratch) {
- Inst* def = remat_def_for(c->remat, v);
- if (def) {
- Inst* re = list_push(f, c->before, (IROp)def->op);
- InstId id = re->id;
- *re = *def;
- re->id = id;
- re->opnds = arena_array(f->arena, Operand, def->nopnds ? def->nopnds : 1u);
- if (def->nopnds)
- memcpy(re->opnds, def->opnds, sizeof(Operand) * def->nopnds);
- re->opnds[0] = *op; /* dst already rewritten to scratch */
- re->def = (Val)scratch;
- re->type = op->type;
- return;
+static int remat_operand_for(const RematInfo* ri, PReg v, Operand* out) {
+ const Inst* def = remat_def_for(ri, v);
+ if (!def || !out) return 0;
+ memset(out, 0, sizeof *out);
+ switch ((IROp)def->op) {
+ case IR_LOAD_IMM:
+ out->kind = OPK_IMM;
+ out->cls = def->nopnds ? def->opnds[0].cls : RC_INT;
+ out->type = def->type ? def->type
+ : (def->nopnds ? def->opnds[0].type : 0);
+ out->v.imm = def->extra.imm;
+ return 1;
+ case IR_ADDR_OF:
+ if (def->nopnds < 2 || def->opnds[1].kind != OPK_LOCAL) return 0;
+ out->kind = OPK_FRAME_ADDR;
+ out->cls = RC_INT;
+ out->type = def->type ? def->type : def->opnds[0].type;
+ out->v.frame_slot = def->opnds[1].v.frame_slot;
+ return 1;
+ default:
+ return 0;
}
- Inst* ld = list_push(f, c->before, IR_LOAD);
- ld->opnds = arena_array(f->arena, Operand, 2);
- ld->opnds[0] = *op;
- ld->opnds[1] = spill_addr(f, v);
- ld->nopnds = 2;
- ld->extra.mem = spill_mem(f, v);
}
+/* Translate one virtual value to its stable allocated location. A spill is a
+ * frame operand, not a synthetic reload into an emitter-reserved register.
+ * Cheap input-less values remain recipes and are materialized inside the
+ * consumer's native-emission temp scope. */
static void rewrite_one_operand(Func* f, Inst* owner, Operand* op, int is_def,
void* arg) {
RewriteCtx* c = (RewriteCtx*)arg;
+ (void)owner;
if (op->kind != OPK_REG) return;
PReg v = (PReg)op->v.reg;
if (v == PREG_NONE || v == 0 || v >= opt_reg_count(f)) return;
@@ -1649,107 +1496,12 @@ static void rewrite_one_operand(Func* f, Inst* owner, Operand* op, int is_def,
return;
}
if (alloc_kind != OPT_ALLOC_SPILL) return;
- u8 cls = opt_preg_loc_cls(f, v);
- Reg scratch = scratch_for(f, cls, &c->next_scratch[cls]);
- if (scratch == (Reg)REG_NONE) {
- SrcLoc loc = owner ? owner->loc : (SrcLoc){0, 0, 0};
- compiler_panic(f->c, loc,
- "opt rewrite: no scratch register for spilled class %u",
- (unsigned)cls);
- }
- op->v.reg = scratch;
- if (!is_def) {
- remat_or_reload_use(f, c, op, v, scratch);
- } else if (remat_def_for(c->remat, v)) {
- /* Every use of this PReg rematerializes (the producer is input-less and all
- * three spilled-use paths route through remat_or_reload_use), so no frame
- * use of the slot remains -- drop the spill store. The original def now
- * writes a dead scratch that mir_dce removes. */
- } else {
- Inst* st = list_push(f, c->after, IR_STORE);
- st->opnds = arena_array(f->arena, Operand, 2);
- st->opnds[0] = spill_addr(f, v);
- st->opnds[1] = *op;
- st->nopnds = 2;
- st->extra.mem = spill_mem(f, v);
- }
-}
-
-/* Spilled call-arg use. Without remat the slot becomes a direct OPK_LOCAL frame
- * operand; with remat we recompute into a scratch so the dropped spill store
- * leaves no dangling frame read. */
-static void rewrite_call_arg_operand(Func* f, Operand* op, RewriteCtx* ctx) {
- if (!op || op->kind != OPK_REG) return;
- PReg v = (PReg)op->v.reg;
- if (v == PREG_NONE || v == 0 || v >= opt_reg_count(f)) return;
- u8 alloc_kind = opt_preg_alloc_kind(f, v);
- if (alloc_kind == OPT_ALLOC_HARD) {
- op->v.reg = opt_preg_hard_reg(f, v);
- return;
- }
- if (alloc_kind != OPT_ALLOC_SPILL) return;
- if (remat_def_for(ctx->remat, v)) {
- u8 cls = opt_preg_loc_cls(f, v);
- Reg scratch = scratch_for(f, cls, &ctx->next_scratch[cls]);
- if (scratch == (Reg)REG_NONE)
- compiler_panic(f->c, (SrcLoc){0, 0, 0},
- "opt rewrite: no scratch register for spilled class %u",
- (unsigned)cls);
- op->v.reg = scratch;
- remat_or_reload_use(f, ctx, op, v, scratch);
+ if (!is_def && !(op->flags & OPT_OPERAND_WALK_INDIRECT_PART) &&
+ remat_operand_for(c->remat, v, op))
return;
- }
*op = spill_addr(f, v);
}
-static void rewrite_store_value_operand(Func* f, Inst* owner, Operand* op,
- RewriteCtx* ctx) {
- PReg v;
- u8 alloc_kind;
- if (!op || op->kind != OPK_REG) return;
- v = (PReg)op->v.reg;
- if (v == PREG_NONE || v == 0 || v >= opt_reg_count(f)) return;
- alloc_kind = opt_preg_alloc_kind(f, v);
- if (alloc_kind == OPT_ALLOC_HARD) {
- op->v.reg = opt_preg_hard_reg(f, v);
- return;
- }
- if (alloc_kind == OPT_ALLOC_SPILL) {
- /* A remat candidate must recompute into a scratch (the slot may no longer
- * be written); a non-remat spill may fold into a direct frame operand. */
- if (remat_def_for(ctx->remat, v)) {
- rewrite_one_operand(f, owner, op, 0, ctx);
- return;
- }
- *op = spill_addr(f, v);
- return;
- }
- rewrite_one_operand(f, owner, op, 0, ctx);
-}
-
-static void rewrite_call_arg_indirect_base(Func* f, Inst* owner, Operand* op,
- RewriteCtx* ctx) {
- if (!op || op->kind != OPK_INDIRECT) return;
- Operand base = *op;
- base.kind = OPK_REG;
- base.cls = RC_INT;
- base.v.reg = op->v.ind.base;
- rewrite_one_operand(f, owner, &base, 0, ctx);
- op->v.ind.base = base.v.reg;
-}
-
-static void rewrite_call_arg_value(Func* f, Inst* owner, CGABIValue* v,
- RewriteCtx* ctx) {
- if (!v) return;
- rewrite_call_arg_indirect_base(f, owner, &v->storage, ctx);
- rewrite_call_arg_operand(f, &v->storage, ctx);
- for (u32 i = 0; i < v->nparts; ++i) {
- Operand* op = (Operand*)&v->parts[i].op;
- rewrite_call_arg_indirect_base(f, owner, op, ctx);
- rewrite_call_arg_operand(f, op, ctx);
- }
-}
-
typedef struct RewriteCallSaveCtx {
Func* f;
RewriteList* out;
@@ -1766,7 +1518,7 @@ static void rewrite_call_save_one(PReg v, void* arg) {
if (opt_preg_alloc_kind(f, v) != OPT_ALLOC_HARD) return;
u8 cls = opt_preg_loc_cls(f, v);
Reg hr = opt_preg_hard_reg(f, v);
- if (cls >= OPT_REG_CLASSES || hr >= 32u) return;
+ if (cls >= OPT_REG_CLASSES || hr >= OPT_MAX_HARD_REGS) return;
if ((opt_call_clobber_mask_for(f, c->call, cls) & (1u << hr)) == 0) return;
if (c->emit_restore)
append_load_preg(f, c->out, v);
@@ -1810,6 +1562,16 @@ static PReg* rewrite_collect_call_save_pregs(Func* f, u32* count_out) {
return pregs;
}
+static int rewrite_elides_remat_producer(Func* f, const Inst* in,
+ const RematInfo* remat) {
+ PReg v;
+ if (!in || !remat_inst_is_candidate(f, in)) return 0;
+ v = (PReg)in->def;
+ if (v == PREG_NONE || v == 0 || v >= opt_reg_count(f)) return 0;
+ return opt_preg_alloc_kind(f, v) == OPT_ALLOC_SPILL &&
+ remat_def_for(remat, v) != NULL;
+}
+
static void rewrite_func(Func* f, const OptLiveInfo* live_info) {
u32 words = live_info ? live_info->words : f->opt_live_words;
if (!words) words = bit_words(opt_reg_count(f));
@@ -1829,9 +1591,7 @@ static void rewrite_func(Func* f, const OptLiveInfo* live_info) {
Block* bl = &f->blocks[b];
RewriteOut out;
out_init(f, &out, bl->ninsts + 16u);
- RewriteList before, after, call_saves, call_restores;
- memset(&before, 0, sizeof before);
- memset(&after, 0, sizeof after);
+ RewriteList call_saves, call_restores;
memset(&call_saves, 0, sizeof call_saves);
memset(&call_restores, 0, sizeof call_restores);
live_active_words = live_copy_block_out_active(live_info, b, live, words,
@@ -1847,53 +1607,20 @@ static void rewrite_func(Func* f, const OptLiveInfo* live_info) {
}
refs_reset(&refs);
opt_walk_inst_operands(f, &in, refs_collect, &refs);
- list_reset(&before);
- list_reset(&after);
list_reset(&call_saves);
list_reset(&call_restores);
- RewriteCtx ctx;
- memset(&ctx, 0, sizeof ctx);
- ctx.before = &before;
- ctx.after = &after;
- ctx.remat = &remat;
- if ((IROp)in.op == IR_CALL) {
- IRCallAux* aux = (IRCallAux*)in.extra.aux;
- if (aux) {
- if (aux->use_plan_replay) {
- rewrite_one_operand(f, &in, &aux->plan.callee, 0, &ctx);
- for (u32 k = 0; k < aux->plan.nargs; ++k) {
- rewrite_call_arg_indirect_base(f, &in, &aux->plan.args[k].src,
- &ctx);
- rewrite_call_arg_operand(f, &aux->plan.args[k].src, &ctx);
- }
- for (u32 k = 0; k < aux->plan.nrets; ++k)
- rewrite_one_operand(f, &in, &aux->plan.rets[k].dst, 1, &ctx);
- } else {
- rewrite_one_operand(f, &in, &aux->desc.callee, 0, &ctx);
- for (u32 k = 0; k < aux->desc.nargs; ++k)
- rewrite_call_arg_value(f, &in, (CGABIValue*)&aux->desc.args[k],
- &ctx);
- opt_walk_abivalue(f, &in, &aux->desc.ret, 1, rewrite_one_operand,
- &ctx);
- }
- }
- } else if ((IROp)in.op == IR_STORE && in.nopnds >= 2) {
- opt_walk_operand(f, &in, &in.opnds[0], 0, rewrite_one_operand, &ctx);
- rewrite_store_value_operand(f, &in, &in.opnds[1], &ctx);
- } else {
- opt_walk_inst_operands(f, &in, rewrite_one_operand, &ctx);
- }
+ RewriteCtx ctx = {&remat};
+ int elide = rewrite_elides_remat_producer(f, &in, &remat);
+ opt_walk_inst_operands(f, &in, rewrite_one_operand, &ctx);
if ((IROp)in.op == IR_CALL) {
append_live_call_saves(f, &call_saves, &in, live, live_active_words,
&refs, call_save_pregs, ncall_save_pregs, 0);
append_live_call_saves(f, &call_restores, &in, live, live_active_words,
&refs, call_save_pregs, ncall_save_pregs, 1);
}
- out_prepend_list_reverse(f, &out, &after);
out_prepend_list_reverse(f, &out, &call_restores);
- out_prepend_inst(f, &out, &in);
+ if (!elide) out_prepend_inst(f, &out, &in);
out_prepend_list_reverse(f, &out, &call_saves);
- out_prepend_list_reverse(f, &out, &before);
live_active_words =
live_update_refs_before_active(live, live_active_words, words, &refs);
f->opt_rewrite_live_words_touched += refs.nuses + refs.ndefs;
@@ -1902,33 +1629,41 @@ static void rewrite_func(Func* f, const OptLiveInfo* live_info) {
bl->ninsts = out.cap - out.start;
bl->cap = bl->ninsts;
}
+
+ /* MIR operands are locations, not SSA values. Keep the HIR definition
+ * metadata on the untouched source graph and make the representation
+ * boundary explicit here: post-allocation passes derive register effects
+ * solely from instruction semantics and location operands. */
+ for (u32 b = 0; b < f->nblocks; ++b) {
+ Block* bl = &f->blocks[b];
+ for (u32 i = 0; i < bl->ninsts; ++i) {
+ bl->insts[i].def = VAL_NONE;
+ bl->insts[i].ndefs = 0;
+ bl->insts[i].defs = NULL;
+ }
+ }
f->opt_rewritten = 1;
}
void opt_lower_to_mir(Func* f, const OptLiveInfo* live_info) {
if (!f) return;
- Func phys = *f;
- phys.blocks = f->blocks;
- phys.opt_rewritten = 0;
- phys.mir = NULL;
+ Func phys;
+ opt_mir_clone_func(&phys, f);
rewrite_func(&phys, live_info);
MFunc* m = arena_zarray(f->arena, MFunc, 1);
m->blocks = phys.blocks;
m->nblocks = phys.nblocks;
+ m->blocks_cap = phys.blocks_cap;
m->entry = phys.entry;
m->emit_order_n = phys.emit_order_n;
- m->emit_order_cap = phys.emit_order_n;
- m->emit_order =
- arena_array(f->arena, u32, phys.emit_order_n ? phys.emit_order_n : 1u);
- if (phys.emit_order_n)
- memcpy(m->emit_order, phys.emit_order, sizeof(u32) * phys.emit_order_n);
+ m->emit_order_cap = phys.emit_order_cap;
+ m->emit_order = phys.emit_order;
f->mir = m;
- f->blocks = phys.blocks;
- f->nblocks = phys.nblocks;
- f->blocks_cap = phys.blocks_cap;
+ /* frame slots and instruction ids are function-wide emission metadata. The
+ * HIR graph itself remains untouched and virtual; MIR owns phys.blocks. */
f->frame_slots = phys.frame_slots;
f->nframe_slots = phys.nframe_slots;
f->frame_slots_cap = phys.frame_slots_cap;
@@ -2136,17 +1871,9 @@ static void opt_regalloc_place(Func* f, OptLiveInfo* live_out) {
opt_live_blocks(f, &live);
OptLiveRangeSet ranges;
opt_live_ranges_build(f, &live, &ranges);
- opt_init_preg_info_from_ranges(f, &ranges);
+ opt_rebuild_preg_info_from_ranges(f, &ranges);
opt_apply_asm_constraints_from_live(f, &live);
apply_abi_aliasing_hints(f);
- /* The ABI hint passes clear forbid bits to steer values toward ABI registers
- * (ret reg, arg regs). Restore the hard machine-clobber forbids afterward so
- * no allocation path (normal, preferred-reg, or two-address group) can place
- * a value in a register that an instruction live across it destroys. */
- if (f->preg_info)
- for (PReg v = 1; v < opt_reg_count(f); ++v)
- f->preg_info[v].forbidden_hard_regs |=
- f->preg_info[v].clobbered_hard_regs;
/* Linear move coalescing (O1.md W3): populate the union-find that
* alloc_coalesce_root/alloc_group_member consult so the allocator merges
* copy-related values onto one location. No O(n^2) conflict matrix — see
diff --git a/src/opt/pass_machinize.c b/src/opt/pass_machinize.c
@@ -15,44 +15,53 @@ static int native_resolve_reg(NativeTarget* target, Slice name, Reg* out,
return 0;
}
+static void asm_prepare_reg_requirement(Func* f, NativeTarget* target,
+ const AsmConstraint* constraint,
+ IRAsmRegRequirement* req) {
+ NativeAsmRegPin pin;
+ NativeAsmRegPinStatus status;
+ NativeAsmConstraintInfo info;
+ memset(req, 0, sizeof *req);
+ req->fixed_reg = -1;
+
+ status = native_asm_resolve_pin(target, constraint->reg, constraint->str,
+ &pin);
+ if (status == NATIVE_ASM_REG_PIN_OK) {
+ req->present = 1;
+ req->cls = (u8)pin.cls;
+ req->fixed_reg = (i32)pin.reg;
+ return;
+ }
+ if (status != NATIVE_ASM_REG_PIN_ABSENT)
+ compiler_panic(f->c, (SrcLoc){0, 0, 0}, "opt asm: %s",
+ native_asm_pin_status_message(status));
+
+ memset(&info, 0, sizeof info);
+ if (!native_asm_constraint_reg_info(target, constraint->str, &info)) return;
+ req->present = 1;
+ req->cls = (u8)info.cls;
+ req->allowed_mask = info.allowed_mask;
+ if (info.fixed_reg != REG_NONE) req->fixed_reg = (i32)info.fixed_reg;
+}
+
static void asm_prepare_constraints(Func* f, NativeTarget* target,
IRAsmAux* aux) {
if (!aux) return;
for (u32 c = 0; c < OPT_REG_CLASSES; ++c) aux->clobber_mask[c] = 0;
- if (aux->nout && !aux->out_fixed_regs) {
- aux->out_fixed_regs = arena_array(f->arena, i32, aux->nout);
- aux->out_fixed_cls = arena_zarray(f->arena, u8, aux->nout);
- }
- if (aux->nin && !aux->in_fixed_regs) {
- aux->in_fixed_regs = arena_array(f->arena, i32, aux->nin);
- aux->in_fixed_cls = arena_zarray(f->arena, u8, aux->nin);
- }
- if (aux->nout && !aux->out_allowed_masks) {
- aux->out_allowed_masks = arena_zarray(f->arena, u32, aux->nout);
- aux->out_allowed_cls = arena_zarray(f->arena, u8, aux->nout);
- }
- if (aux->nin && !aux->in_allowed_masks) {
- aux->in_allowed_masks = arena_zarray(f->arena, u32, aux->nin);
- aux->in_allowed_cls = arena_zarray(f->arena, u8, aux->nin);
- }
- for (u32 i = 0; i < aux->nout; ++i) {
- aux->out_fixed_regs[i] = -1;
- aux->out_fixed_cls[i] = 0;
- aux->out_allowed_masks[i] = 0;
- aux->out_allowed_cls[i] = 0;
- }
- for (u32 i = 0; i < aux->nin; ++i) {
- aux->in_fixed_regs[i] = -1;
- aux->in_fixed_cls[i] = 0;
- aux->in_allowed_masks[i] = 0;
- aux->in_allowed_cls[i] = 0;
- }
+ aux->has_memory_constraint = 0;
+ if (aux->nout && !aux->out_reg_reqs)
+ aux->out_reg_reqs =
+ arena_zarray(f->arena, IRAsmRegRequirement, aux->nout);
+ if (aux->nin && !aux->in_reg_reqs)
+ aux->in_reg_reqs =
+ arena_zarray(f->arena, IRAsmRegRequirement, aux->nin);
for (u32 i = 0; i < aux->nclob; ++i) {
Reg r;
RegClass cls;
Slice nm = pool_slice(f->c->global, aux->clobbers[i]);
if (native_resolve_reg(target, nm, &r, &cls) != 0) continue;
- if ((u32)cls < OPT_REG_CLASSES && r < 32) aux->clobber_mask[cls] |= 1u << r;
+ if ((u32)cls < OPT_REG_CLASSES && r < OPT_MAX_HARD_REGS)
+ aux->clobber_mask[cls] |= 1u << r;
}
if (aux->clobber_abi_sets) {
u32 int_mask, fp_mask;
@@ -64,54 +73,16 @@ static void asm_prepare_constraints(Func* f, NativeTarget* target,
aux->clobber_mask[NATIVE_REG_FP] |= fp_mask;
}
for (u32 i = 0; i < aux->nout; ++i) {
- NativeAsmRegPin pin;
- NativeAsmRegPinStatus st = native_asm_resolve_pin(target, aux->outs[i].reg,
- aux->outs[i].str, &pin);
- if (st == NATIVE_ASM_REG_PIN_OK) {
- aux->out_fixed_regs[i] = (i32)pin.reg;
- aux->out_fixed_cls[i] = (u8)pin.cls;
- continue;
- }
- if (st != NATIVE_ASM_REG_PIN_ABSENT) {
- compiler_panic(f->c, (SrcLoc){0, 0, 0}, "opt asm: %s",
- native_asm_pin_status_message(st));
- }
- NativeAsmConstraintInfo info;
- if (native_asm_constraint_reg_info(target, aux->outs[i].str, &info)) {
- if (info.allowed_mask) {
- aux->out_allowed_masks[i] = info.allowed_mask;
- aux->out_allowed_cls[i] = (u8)info.cls;
- }
- if (info.fixed_reg != REG_NONE) {
- aux->out_fixed_regs[i] = (i32)info.fixed_reg;
- aux->out_fixed_cls[i] = (u8)info.cls;
- }
- }
+ asm_prepare_reg_requirement(f, target, &aux->outs[i],
+ &aux->out_reg_reqs[i]);
+ if (native_asm_constraint_body(aux->outs[i].str)[0] == 'm')
+ aux->has_memory_constraint = 1;
}
for (u32 i = 0; i < aux->nin; ++i) {
- NativeAsmRegPin pin;
- NativeAsmRegPinStatus st =
- native_asm_resolve_pin(target, aux->ins[i].reg, aux->ins[i].str, &pin);
- if (st == NATIVE_ASM_REG_PIN_OK) {
- aux->in_fixed_regs[i] = (i32)pin.reg;
- aux->in_fixed_cls[i] = (u8)pin.cls;
- continue;
- }
- if (st != NATIVE_ASM_REG_PIN_ABSENT) {
- compiler_panic(f->c, (SrcLoc){0, 0, 0}, "opt asm: %s",
- native_asm_pin_status_message(st));
- }
- NativeAsmConstraintInfo info;
- if (native_asm_constraint_reg_info(target, aux->ins[i].str, &info)) {
- if (info.allowed_mask) {
- aux->in_allowed_masks[i] = info.allowed_mask;
- aux->in_allowed_cls[i] = (u8)info.cls;
- }
- if (info.fixed_reg != REG_NONE) {
- aux->in_fixed_regs[i] = (i32)info.fixed_reg;
- aux->in_fixed_cls[i] = (u8)info.cls;
- }
- }
+ asm_prepare_reg_requirement(f, target, &aux->ins[i],
+ &aux->in_reg_reqs[i]);
+ if (native_asm_constraint_body(aux->ins[i].str)[0] == 'm')
+ aux->has_memory_constraint = 1;
}
}
@@ -121,7 +92,8 @@ static void machinize_reset(Func* f, NativeTarget* target) {
for (u32 c = 0; c < OPT_REG_CLASSES; ++c) {
f->opt_hard_reg_count[c] = 0;
f->opt_phys_reg_count[c] = 0;
- f->opt_scratch_reg_count[c] = 0;
+ f->emit_temp_reg_count[c] = 0;
+ f->asm_temp_mask[c] = 0;
f->opt_caller_saved[c] = 0;
f->opt_callee_saved[c] = 0;
f->opt_reserved_regs[c] = 0;
@@ -141,19 +113,124 @@ static void machinize_prepare_insts(Func* f, NativeTarget* target) {
}
}
+static const NativePhysRegInfo* class_phys_reg(
+ const NativeAllocClassInfo* ci, Reg reg) {
+ for (u32 i = 0; ci && i < ci->nphys; ++i)
+ if (ci->phys[i].reg == reg) return &ci->phys[i];
+ return NULL;
+}
+
+static u32 validate_reg_list(Func* f, const NativeAllocClassInfo* ci,
+ const Reg* regs, u32 nregs,
+ const char* list_name) {
+ u32 mask = 0;
+ if ((nregs != 0u) != (regs != NULL))
+ compiler_panic(f->c, (SrcLoc){0, 0, 0},
+ "opt_machinize: class %u has inconsistent %s list",
+ (unsigned)ci->cls, list_name);
+ if (nregs > OPT_MAX_HARD_REGS)
+ compiler_panic(f->c, (SrcLoc){0, 0, 0},
+ "opt_machinize: class %u has too many %s registers (%u)",
+ (unsigned)ci->cls, list_name, (unsigned)nregs);
+ for (u32 i = 0; i < nregs; ++i) {
+ Reg reg = regs[i];
+ const NativePhysRegInfo* phys;
+ if (reg >= OPT_MAX_HARD_REGS || (mask & (1u << reg)))
+ compiler_panic(f->c, (SrcLoc){0, 0, 0},
+ "opt_machinize: class %u has invalid/duplicate %s reg %u",
+ (unsigned)ci->cls, list_name, (unsigned)reg);
+ phys = class_phys_reg(ci, reg);
+ if (!phys || phys->cls != ci->cls)
+ compiler_panic(f->c, (SrcLoc){0, 0, 0},
+ "opt_machinize: class %u %s reg %u is not a matching "
+ "physical register",
+ (unsigned)ci->cls, list_name, (unsigned)reg);
+ mask |= 1u << reg;
+ }
+ return mask;
+}
+
static void collect_class(Func* f, NativeTarget* target,
const NativeAllocClassInfo* ci) {
u32 cls = ci->cls;
- if (cls >= OPT_REG_CLASSES) return;
+ u32 known_mask = 0;
+ u32 emit_mask;
+ u32 asm_mask;
+ if (cls >= OPT_REG_CLASSES)
+ compiler_panic(f->c, (SrcLoc){0, 0, 0},
+ "opt_machinize: invalid register class %u", (unsigned)cls);
+ if ((ci->nphys != 0u) != (ci->phys != NULL) ||
+ ci->nphys > OPT_MAX_HARD_REGS)
+ compiler_panic(f->c, (SrcLoc){0, 0, 0},
+ "opt_machinize: class %u has invalid physical table size %u",
+ (unsigned)cls, (unsigned)ci->nphys);
+ for (u32 i = 0; i < ci->nphys; ++i) {
+ const NativePhysRegInfo* phys = &ci->phys[i];
+ if (phys->reg >= OPT_MAX_HARD_REGS || phys->cls != cls ||
+ (known_mask & (1u << phys->reg)))
+ compiler_panic(f->c, (SrcLoc){0, 0, 0},
+ "opt_machinize: class %u has invalid physical reg %u",
+ (unsigned)cls, (unsigned)phys->reg);
+ known_mask |= 1u << phys->reg;
+ }
+ (void)validate_reg_list(f, ci, ci->ndt_allocable, ci->ndt_allocable_count,
+ "O0 allocable");
+ (void)validate_reg_list(f, ci, ci->scratch, ci->nscratch, "O0 scratch");
+ if (ci->nemit_temps > OPT_MAX_EMIT_TEMPS)
+ compiler_panic(f->c, (SrcLoc){0, 0, 0},
+ "opt_machinize: class %u needs %u emit temps (capacity %u)",
+ (unsigned)cls, (unsigned)ci->nemit_temps,
+ (unsigned)OPT_MAX_EMIT_TEMPS);
+ emit_mask = validate_reg_list(f, ci, ci->emit_temps, ci->nemit_temps,
+ "emit temp");
+ asm_mask = validate_reg_list(f, ci, ci->asm_temps, ci->nasm_temps,
+ "asm-internal temp");
+ if (emit_mask & asm_mask)
+ compiler_panic(f->c, (SrcLoc){0, 0, 0},
+ "opt_machinize: class %u emit and asm temp banks overlap",
+ (unsigned)cls);
+ if (asm_mask & ~ci->reserved_mask)
+ compiler_panic(f->c, (SrcLoc){0, 0, 0},
+ "opt_machinize: class %u asm temp bank is not reserved",
+ (unsigned)cls);
+ if (ci->emit_cache_mask & ~emit_mask)
+ compiler_panic(f->c, (SrcLoc){0, 0, 0},
+ "opt_machinize: class %u cache mask escapes emit temp bank",
+ (unsigned)cls);
+ if ((ci->caller_saved_mask | ci->callee_saved_mask | ci->arg_mask |
+ ci->ret_mask | ci->reserved_mask) &
+ ~known_mask)
+ compiler_panic(f->c, (SrcLoc){0, 0, 0},
+ "opt_machinize: class %u register mask names unknown phys",
+ (unsigned)cls);
+ for (u32 i = 0; i < ci->nemit_temps; ++i) {
+ const NativePhysRegInfo* phys = class_phys_reg(ci, ci->emit_temps[i]);
+ if (phys && (phys->flags & NATIVE_REG_ALLOCABLE))
+ compiler_panic(f->c, (SrcLoc){0, 0, 0},
+ "opt_machinize: class %u emit temp %u is allocable",
+ (unsigned)cls, (unsigned)ci->emit_temps[i]);
+ }
+ for (u32 i = 0; i < ci->nasm_temps; ++i) {
+ const NativePhysRegInfo* phys = class_phys_reg(ci, ci->asm_temps[i]);
+ if (phys && (phys->flags & NATIVE_REG_ALLOCABLE))
+ compiler_panic(f->c, (SrcLoc){0, 0, 0},
+ "opt_machinize: class %u asm temp %u is allocable",
+ (unsigned)cls, (unsigned)ci->asm_temps[i]);
+ }
f->opt_caller_saved[cls] =
native_target_caller_saved_mask(target, (NativeAllocClass)cls);
f->opt_callee_saved[cls] =
native_target_callee_saved_mask(target, (NativeAllocClass)cls);
- f->opt_reserved_regs[cls] = ci->reserved_mask;
+ if (emit_mask & ~f->opt_caller_saved[cls])
+ compiler_panic(f->c, (SrcLoc){0, 0, 0},
+ "opt_machinize: class %u emit temp bank is not entirely "
+ "caller-saved",
+ (unsigned)cls);
+ f->opt_reserved_regs[cls] = ci->reserved_mask | emit_mask;
+ f->asm_temp_mask[cls] = asm_mask;
f->opt_arg_regs[cls] = ci->arg_mask;
f->opt_ret_regs[cls] = ci->ret_mask;
- for (u32 i = 0;
- i < ci->nphys && f->opt_phys_reg_count[cls] < OPT_MAX_HARD_REGS; ++i) {
+ for (u32 i = 0; i < ci->nphys; ++i) {
const NativePhysRegInfo* src = &ci->phys[i];
CGPhysRegInfo* dst = &f->opt_phys_regs[cls][f->opt_phys_reg_count[cls]++];
memset(dst, 0, sizeof *dst);
@@ -165,28 +242,37 @@ static void collect_class(Func* f, NativeTarget* target,
f->opt_hard_reg_count[cls] < OPT_MAX_HARD_REGS)
f->opt_hard_regs[cls][f->opt_hard_reg_count[cls]++] = src->reg;
}
- /* The optimizer's native-emit scratch: opt_scratch overrides scratch when a
- * backend needs a different (usually larger) set at -O1 than the -O0 NDT. */
- const Reg* scr = ci->opt_nscratch ? ci->opt_scratch : ci->scratch;
- u32 nscr = ci->opt_nscratch ? ci->opt_nscratch : ci->nscratch;
- for (u32 i = 0; i < nscr && i < OPT_MAX_SCRATCH_REGS; ++i)
- f->opt_scratch_regs[cls][f->opt_scratch_reg_count[cls]++] = scr[i];
+ for (u32 i = 0; i < ci->nemit_temps && i < OPT_MAX_EMIT_TEMPS; ++i)
+ f->emit_temp_regs[cls][f->emit_temp_reg_count[cls]++] =
+ ci->emit_temps[i];
}
static void machinize_collect_regs(Func* f, NativeTarget* target) {
+ u32 seen = 0;
if (!target || !target->regs) return;
- for (u32 i = 0; i < target->regs->nclasses; ++i)
+ if (target->regs->nclasses > OPT_REG_CLASSES ||
+ (target->regs->nclasses != 0u) != (target->regs->classes != NULL))
+ compiler_panic(f->c, (SrcLoc){0, 0, 0},
+ "opt_machinize: invalid register class table");
+ for (u32 i = 0; i < target->regs->nclasses; ++i) {
+ u32 cls = target->regs->classes[i].cls;
+ if (cls >= OPT_REG_CLASSES || (seen & (1u << cls)))
+ compiler_panic(f->c, (SrcLoc){0, 0, 0},
+ "opt_machinize: duplicate/invalid register class %u",
+ (unsigned)cls);
+ seen |= 1u << cls;
collect_class(f, target, &target->regs->classes[i]);
+ }
}
static void machinize_check_overlap(Func* f) {
for (u32 c = 0; c < OPT_REG_CLASSES; ++c) {
for (u32 i = 0; i < f->opt_hard_reg_count[c]; ++i) {
Reg hr = f->opt_hard_regs[c][i];
- for (u32 s = 0; s < f->opt_scratch_reg_count[c]; ++s) {
- if (f->opt_scratch_regs[c][s] == hr) {
+ for (u32 s = 0; s < f->emit_temp_reg_count[c]; ++s) {
+ if (f->emit_temp_regs[c][s] == hr) {
compiler_panic(f->c, (SrcLoc){0, 0, 0},
- "opt_machinize: hard reg %u overlaps scratch reg "
+ "opt_machinize: hard reg %u overlaps emit temp "
"in class %u",
(unsigned)hr, (unsigned)c);
}
@@ -195,13 +281,45 @@ static void machinize_check_overlap(Func* f) {
}
}
-/* Record, per instruction, the physical registers the target's encoding
- * clobbers as a side effect (x86 idiv → rax/rdx, variable shift → cl, atomics,
- * va_arg), so the allocator keeps values live across them out of those
- * registers. The target reports this through machine_op_clobbers; a NULL hook
- * means no instruction has fixed-register clobbers and the side table stays
- * empty. */
-static void machinize_inst_clobbers(Func* f, NativeTarget* target) {
+static u32 machinize_known_phys_mask(const Func* f, u32 cls) {
+ u32 mask = 0;
+ if (!f || cls >= OPT_REG_CLASSES) return 0;
+ for (u32 i = 0; i < f->opt_phys_reg_count[cls]; ++i) {
+ Reg reg = f->opt_phys_regs[cls][i].reg;
+ if (reg < OPT_MAX_HARD_REGS) mask |= 1u << reg;
+ }
+ return mask;
+}
+
+static void machinize_validate_clobber_masks(Func* f, const Inst* in,
+ const u32* masks) {
+ for (u32 cls = 0; cls < OPT_REG_CLASSES; ++cls) {
+ u32 unknown = masks[cls] & ~machinize_known_phys_mask(f, cls);
+ if (unknown)
+ compiler_panic(
+ f->c, in->loc,
+ "opt_machinize: class %u machine clobber mask names unknown "
+ "physical register %#x",
+ (unsigned)cls, (unsigned)unknown);
+ }
+}
+
+/* Record, per instruction, every optimizer-visible register the target's
+ * encoding clobbers as a side effect (division, variable shifts, bitfield RMW,
+ * atomics, intrinsics, and format-dependent TLS among the current cases). The
+ * allocator keeps live-across MIR values out of those registers, and native
+ * emission uses the same exhaustive effect to invalidate clean frame-cache
+ * entries in cache-capable emit temps. Pure backend-private registers stay out
+ * of the table. A NULL hook promises no such clobbers, so the derived side
+ * table stays empty. */
+void opt_refresh_machine_clobbers(Func* f, NativeTarget* target) {
+ if (!f || !target) return;
+ /* Target effects are derived from the current instruction shapes, not
+ * persistent IR state. Replace the table on every refresh so a shape-changing
+ * pass, target change, or instruction-id growth cannot retain stale effects.
+ * The arena owns the superseded allocation. */
+ f->inst_clobbers = NULL;
+ f->inst_clobbers_cap = 0;
if (!target->machine_op_clobbers || !f->next_inst_id) return;
for (u32 b = 0; b < f->nblocks; ++b) {
Block* bl = &f->blocks[b];
@@ -209,6 +327,7 @@ static void machinize_inst_clobbers(Func* f, NativeTarget* target) {
Inst* in = &bl->insts[i];
NativeMachineOp mop;
u32 mask[NATIVE_CALL_PLAN_CLASSES];
+ int has_clobbers;
memset(&mop, 0, sizeof mop);
switch ((IROp)in->op) {
case IR_BINOP:
@@ -217,6 +336,12 @@ static void machinize_inst_clobbers(Func* f, NativeTarget* target) {
mop.second_is_reg =
(u8)(in->nopnds > 2u && in->opnds[2].kind == OPK_REG);
break;
+ case IR_BITFIELD_LOAD:
+ mop.kind = NATIVE_MOP_BITFIELD_LOAD;
+ break;
+ case IR_BITFIELD_STORE:
+ mop.kind = NATIVE_MOP_BITFIELD_STORE;
+ break;
case IR_VA_START:
mop.kind = NATIVE_MOP_VA_START;
break;
@@ -245,7 +370,9 @@ static void machinize_inst_clobbers(Func* f, NativeTarget* target) {
continue;
}
mask[0] = mask[1] = mask[2] = 0;
- if (!target->machine_op_clobbers(target, &mop, mask)) continue;
+ has_clobbers = target->machine_op_clobbers(target, &mop, mask);
+ machinize_validate_clobber_masks(f, in, mask);
+ if (!has_clobbers) continue;
if (in->id == INST_ID_NONE) continue;
if (!f->inst_clobbers) {
f->inst_clobbers_cap = f->next_inst_id;
@@ -264,5 +391,5 @@ void opt_machinize_native(Func* f, NativeTarget* target) {
machinize_prepare_insts(f, target);
machinize_collect_regs(f, target);
machinize_check_overlap(f);
- machinize_inst_clobbers(f, target);
+ opt_refresh_machine_clobbers(f, target);
}
diff --git a/src/opt/pass_mir.c b/src/opt/pass_mir.c
@@ -1,102 +1,575 @@
#include "opt/opt_internal.h"
-static int mir_view(Func* f, Func* out) {
+int opt_mir_view(Func* f, Func* out) {
if (!f || !f->mir) return 0;
*out = *f;
out->blocks = f->mir->blocks;
out->nblocks = f->mir->nblocks;
+ out->blocks_cap = f->mir->blocks_cap;
out->entry = f->mir->entry;
out->emit_order = f->mir->emit_order;
out->emit_order_n = f->mir->emit_order_n;
out->emit_order_cap = f->mir->emit_order_cap;
out->opt_rewritten = 1;
+ /* HIR def-use records contain direct pointers into HIR operands. MIR passes
+ * must rebuild any analysis they need over this physical graph. */
+ out->opt_uses = NULL;
+ out->opt_nuses = 0;
+ out->opt_uses_cap = 0;
+ out->opt_first_use_by_val = NULL;
+ out->opt_first_use_by_val_cap = 0;
+ out->opt_valid_analyses = 0;
+ /* Scope lookup is a recording/semantic cache of direct Inst pointers into
+ * HIR. Structured scopes have already been lowered before MIR exists, and a
+ * physical pass must never inherit pointers back into the semantic graph. */
+ out->scope_aux_inst = NULL;
+ out->nscopes = 0;
+ out->scopes_cap = 0;
out->mir = NULL;
return 1;
}
-static void mir_commit(Func* f, const Func* view) {
+void opt_mir_commit(Func* f, const Func* view) {
if (!f || !f->mir || !view) return;
f->mir->blocks = view->blocks;
f->mir->nblocks = view->nblocks;
+ f->mir->blocks_cap = view->blocks_cap;
f->mir->entry = view->entry;
f->mir->emit_order = view->emit_order;
f->mir->emit_order_n = view->emit_order_n;
f->mir->emit_order_cap = view->emit_order_cap;
+ /* Instruction ids are function-wide side-table keys, but a physical pass
+ * allocates them through its independently mutable Func view. Commit that
+ * namespace with the graph so later MIR analyses can size their tables for
+ * every instruction the pass inserted. */
+ f->next_inst_id = view->next_inst_id;
}
-static void mir_verify_reg(Func* f, const char* stage, Reg r, u8 cls,
- const char* what) {
- if (r == (Reg)REG_NONE || cls >= OPT_REG_CLASSES || r >= OPT_MAX_HARD_REGS) {
- SrcLoc loc = {0, 0, 0};
- compiler_panic(f->c, loc, "opt MIR verify %s: bad %s cls%u reg%u",
- stage ? stage : "?", what, (unsigned)cls, (unsigned)r);
+typedef struct MirVerifyCtx {
+ const char* stage;
+ /* opt_walk_operand presents an indirect owner immediately followed by
+ * transient scalar views of its base and (when present) index. Track that
+ * sequence so a stored OPT_OPERAND_WALK_INDIRECT_PART flag cannot masquerade
+ * as a legitimate walker annotation. */
+ const Operand* indirect;
+ u8 indirect_part; /* 1 = base, 2 = index */
+} MirVerifyCtx;
+
+static void mir_fail(Func* f, const Inst* in, const char* stage,
+ const char* msg, u32 a, u32 b) {
+ compiler_panic(f->c, in ? in->loc : (SrcLoc){0, 0, 0},
+ "opt MIR verify %s: %s [op %u, inst %u] (%u, %u)",
+ stage ? stage : "?", msg, in ? (unsigned)in->op : 0u,
+ in ? (unsigned)in->id : 0u, (unsigned)a, (unsigned)b);
+}
+
+static int mir_pow2(u32 n) { return n && !(n & (n - 1u)); }
+
+static int mir_type_known(Func* f, KitCgTypeId type) {
+ return type != KIT_CG_TYPE_NONE && cg_type_get(f->c, type) != NULL;
+}
+
+static int mir_type_valid(Func* f, KitCgTypeId type) {
+ return mir_type_known(f, type) &&
+ kit_cg_type_is_sized((KitCompiler*)f->c, type) &&
+ cg_type_size(f->c, type) != 0 && cg_type_align(f->c, type) != 0;
+}
+
+static int mir_type_scalar(Func* f, KitCgTypeId type) {
+ const CgType* ty;
+ if (!mir_type_valid(f, type)) return 0;
+ ty = cg_type_get(f->c, type);
+ switch (ty->kind) {
+ case KIT_CG_TYPE_BOOL:
+ case KIT_CG_TYPE_INT:
+ case KIT_CG_TYPE_FLOAT:
+ case KIT_CG_TYPE_PTR:
+ case KIT_CG_TYPE_ENUM:
+ case KIT_CG_TYPE_VARARG_STATE:
+ return 1;
+ default:
+ return 0;
}
}
+static int mir_type_address_component(Func* f, KitCgTypeId type) {
+ return mir_type_scalar(f, type) &&
+ opt_value_reg_class(f->c, type) == RC_INT &&
+ cg_type_size(f->c, type) == f->c->target.ptr_size;
+}
+
+static int mir_reg_known(Func* f, Reg r, u8 cls) {
+ if (cls >= OPT_REG_CLASSES || r == (Reg)REG_NONE ||
+ r >= OPT_MAX_HARD_REGS)
+ return 0;
+ for (u32 i = 0; i < f->opt_phys_reg_count[cls]; ++i) {
+ const CGPhysRegInfo* phys = &f->opt_phys_regs[cls][i];
+ if (phys->reg == r && phys->cls == cls) return 1;
+ }
+ return 0;
+}
+
+static u32 mir_known_reg_mask(Func* f, u8 cls) {
+ u32 mask = 0;
+ if (!f || cls >= OPT_REG_CLASSES) return 0;
+ for (u32 i = 0; i < f->opt_phys_reg_count[cls]; ++i) {
+ Reg r = f->opt_phys_regs[cls][i].reg;
+ if (r < OPT_MAX_HARD_REGS) mask |= 1u << r;
+ }
+ return mask;
+}
+
+static int mir_reg_is_emit_temp(Func* f, Reg r, u8 cls) {
+ if (cls >= OPT_REG_CLASSES) return 0;
+ for (u32 i = 0; i < f->emit_temp_reg_count[cls]; ++i)
+ if (f->emit_temp_regs[cls][i] == r) return 1;
+ return 0;
+}
+
+static int mir_reg_is_asm_temp(Func* f, Reg r, u8 cls) {
+ return cls < OPT_REG_CLASSES && r < OPT_MAX_HARD_REGS &&
+ (f->asm_temp_mask[cls] & (1u << r)) != 0;
+}
+
+static void mir_verify_reg(Func* f, const Inst* in, const char* stage, Reg r,
+ u8 cls, const char* what) {
+ if (!mir_reg_known(f, r, cls))
+ mir_fail(f, in, stage, what, cls, (u32)r);
+ if (mir_reg_is_emit_temp(f, r, cls))
+ mir_fail(f, in, stage, "emitter-owned temp escaped into MIR", cls,
+ (u32)r);
+ if (mir_reg_is_asm_temp(f, r, cls))
+ mir_fail(f, in, stage, "asm-internal temp escaped into MIR", cls,
+ (u32)r);
+}
+
+static IRFrameSlot* mir_frame_slot(Func* f, const Inst* in, const char* stage,
+ FrameSlot id, const char* what) {
+ if (id == FRAME_SLOT_NONE || id > f->nframe_slots)
+ mir_fail(f, in, stage, what, id, f->nframe_slots);
+ return &f->frame_slots[id - 1u];
+}
+
+static void mir_verify_slot_contains(Func* f, const Inst* in,
+ const char* stage,
+ const IRFrameSlot* slot,
+ KitCgTypeId type, const char* what) {
+ u64 size;
+ if (!mir_type_valid(f, type))
+ mir_fail(f, in, stage, what, type, slot ? slot->id : 0u);
+ size = cg_type_size(f->c, type);
+ if (!slot || size > slot->size)
+ mir_fail(f, in, stage, what, slot ? slot->id : 0u, type);
+}
+
+static void mir_verify_slot_fits(Func* f, const Inst* in, const char* stage,
+ const IRFrameSlot* slot, KitCgTypeId type,
+ const char* what) {
+ mir_verify_slot_contains(f, in, stage, slot, type, what);
+ if (cg_type_align(f->c, type) > slot->align)
+ mir_fail(f, in, stage, what, slot->id, type);
+}
+
+static void mir_verify_frame_table(Func* f, const char* stage) {
+ const u16 allowed_flags =
+ (u16)(FSF_ADDR_TAKEN | FSF_MEMORY_REQUIRED | FSF_VOLATILE);
+ for (u32 i = 0; i < f->nframe_slots; ++i) {
+ IRFrameSlot* slot = &f->frame_slots[i];
+ if (slot->id != i + 1u)
+ mir_fail(f, NULL, stage, "non-canonical frame slot id", slot->id,
+ i + 1u);
+ if (slot->kind != FS_LOCAL && slot->kind != FS_PARAM &&
+ slot->kind != FS_SPILL)
+ mir_fail(f, NULL, stage, "invalid optimizer frame slot kind", slot->id,
+ slot->kind);
+ if (!mir_type_valid(f, slot->type))
+ mir_fail(f, NULL, stage, "invalid frame slot type", slot->id,
+ slot->type);
+ if (!slot->size || !mir_pow2(slot->align))
+ mir_fail(f, NULL, stage, "invalid frame slot size/alignment", slot->size,
+ slot->align);
+ if (cg_type_size(f->c, slot->type) > slot->size ||
+ cg_type_align(f->c, slot->type) > slot->align)
+ mir_fail(f, NULL, stage, "frame slot does not fit descriptor type",
+ slot->id, slot->type);
+ if (slot->flags & ~allowed_flags)
+ mir_fail(f, NULL, stage, "invalid frame slot flags", slot->id,
+ slot->flags);
+ if (slot->kind == FS_SPILL) {
+ if (slot->flags)
+ mir_fail(f, NULL, stage, "spill slot carries semantic flags", slot->id,
+ slot->flags);
+ } else if (slot->priority) {
+ mir_fail(f, NULL, stage, "semantic frame slot carries spill priority",
+ slot->id, slot->priority);
+ }
+ }
+}
+
+static void mir_verify_asm_requirement(Func* f, const Inst* in,
+ const char* stage,
+ const IRAsmRegRequirement* req,
+ const Operand* op) {
+ u32 known;
+ if (!req->present) {
+ if (req->allowed_mask || req->fixed_reg != -1 || req->cls)
+ mir_fail(f, in, stage, "non-canonical absent asm requirement",
+ req->allowed_mask, (u32)req->fixed_reg);
+ return;
+ }
+ if (req->cls >= OPT_REG_CLASSES || !op || !mir_type_scalar(f, op->type) ||
+ opt_value_reg_class(f->c, op->type) != req->cls)
+ mir_fail(f, in, stage, "invalid asm requirement class/type", req->cls,
+ op ? op->type : 0u);
+ known = mir_known_reg_mask(f, req->cls);
+ if (req->allowed_mask & ~known)
+ mir_fail(f, in, stage, "asm allowed mask names unknown register",
+ req->cls, req->allowed_mask & ~known);
+ if (req->fixed_reg < -1 || req->fixed_reg >= (i32)OPT_MAX_HARD_REGS ||
+ (req->fixed_reg >= 0 &&
+ !(known & (1u << (u32)req->fixed_reg))))
+ mir_fail(f, in, stage, "asm fixed requirement names unknown register",
+ req->cls, (u32)req->fixed_reg);
+ if (req->fixed_reg >= 0 && req->allowed_mask &&
+ !(req->allowed_mask & (1u << (u32)req->fixed_reg)))
+ mir_fail(f, in, stage, "asm fixed register violates allowed mask",
+ req->cls, (u32)req->fixed_reg);
+}
+
+static void mir_verify_asm_aux(Func* f, const Inst* in, const char* stage) {
+ const IRAsmAux* aux = (const IRAsmAux*)in->extra.aux;
+ if (!aux)
+ mir_fail(f, in, stage, "asm instruction has no descriptor", 0, 0);
+ if ((aux->nout != 0u) != (aux->outs && aux->out_ops && aux->out_reg_reqs) ||
+ (aux->nin != 0u) != (aux->ins && aux->in_ops && aux->in_reg_reqs) ||
+ (aux->nclob != 0u) != (aux->clobbers != NULL) ||
+ aux->has_memory_constraint > 1u)
+ mir_fail(f, in, stage, "inconsistent asm descriptor arrays", aux->nout,
+ aux->nin);
+ for (u32 cls = 0; cls < OPT_REG_CLASSES; ++cls)
+ if (aux->clobber_mask[cls] & ~mir_known_reg_mask(f, (u8)cls))
+ mir_fail(f, in, stage, "asm clobber mask names unknown register", cls,
+ aux->clobber_mask[cls]);
+ for (u32 i = 0; i < aux->nout; ++i)
+ mir_verify_asm_requirement(f, in, stage, &aux->out_reg_reqs[i],
+ &aux->out_ops[i]);
+ for (u32 i = 0; i < aux->nin; ++i)
+ mir_verify_asm_requirement(f, in, stage, &aux->in_reg_reqs[i],
+ &aux->in_ops[i]);
+}
+
+static void mir_verify_scalar_location(Func* f, Inst* in, Operand* op,
+ int is_def, MirVerifyCtx* ctx) {
+ IRFrameSlot* slot;
+ if (op->flags)
+ mir_fail(f, in, ctx->stage, "stored operand has transient flags", op->kind,
+ op->flags);
+ if (op->shift) {
+ if (op->kind != OPK_REG || is_def || op->cls != RC_INT ||
+ op->shift > 4u || (IROp)in->op != IR_BINOP || in->nopnds != 3u ||
+ op != &in->opnds[2])
+ mir_fail(f, in, ctx->stage, "invalid shifted-register rider", op->kind,
+ op->shift);
+ }
+ switch ((OptOperandKind)op->kind) {
+ case OPK_IMM:
+ /* A zero operand is the canonical absent ABI value (notably a void call
+ * result). Every present immediate must still carry a real scalar type. */
+ if (op->type == KIT_CG_TYPE_NONE) {
+ if (op->cls || op->v.imm)
+ mir_fail(f, in, ctx->stage, "malformed absent immediate", op->cls,
+ (u32)op->v.imm);
+ } else if (!mir_type_scalar(f, op->type) || op->cls != RC_INT) {
+ mir_fail(f, in, ctx->stage, "invalid immediate type/class", op->type,
+ op->cls);
+ }
+ return;
+ case OPK_GLOBAL:
+ /* A symbol-address recipe may retain an incomplete array or function
+ * referent type. Native emission needs the symbol identity and address
+ * class, not sized storage; size/alignment checks belong to operations
+ * that actually access the referent. */
+ if (!mir_type_known(f, op->type) || op->cls != RC_INT)
+ mir_fail(f, in, ctx->stage, "invalid global type/class", op->type,
+ op->cls);
+ return;
+ case OPK_LOCAL:
+ if (op->cls != RC_INT)
+ mir_fail(f, in, ctx->stage, "frame local has non-address class",
+ op->cls, op->v.frame_slot);
+ slot = mir_frame_slot(f, in, ctx->stage, op->v.frame_slot,
+ "invalid local frame slot");
+ if (slot->kind != FS_LOCAL && slot->kind != FS_PARAM)
+ mir_fail(f, in, ctx->stage, "local operand names allocator spill",
+ op->v.frame_slot, slot->kind);
+ /* A semantic local may be a typed subobject view into packed storage.
+ * Its access alignment lives in MemAccess and may be lower than the
+ * viewed type's natural alignment; only containment is required here.
+ * Allocator spills below retain the stronger alignment invariant. */
+ mir_verify_slot_contains(f, in, ctx->stage, slot, op->type,
+ "local operand type does not fit frame slot");
+ return;
+ case OPK_STACK:
+ if (!mir_type_scalar(f, op->type) || op->cls >= OPT_REG_CLASSES ||
+ opt_value_reg_class(f->c, op->type) != op->cls)
+ mir_fail(f, in, ctx->stage, "invalid spill value type/class", op->type,
+ op->cls);
+ slot = mir_frame_slot(f, in, ctx->stage, op->v.frame_slot,
+ "invalid spill frame slot");
+ if (slot->kind != FS_SPILL)
+ mir_fail(f, in, ctx->stage, "spill value names semantic frame slot",
+ op->v.frame_slot, slot->kind);
+ mir_verify_slot_fits(f, in, ctx->stage, slot, op->type,
+ "spill value type does not fit frame slot");
+ return;
+ case OPK_FRAME_ADDR:
+ if (is_def || op->cls != RC_INT || !cg_type_is_ptr(f->c, op->type) ||
+ !mir_type_address_component(f, op->type))
+ mir_fail(f, in, ctx->stage, "invalid frame-address recipe", op->type,
+ op->cls);
+ slot = mir_frame_slot(f, in, ctx->stage, op->v.frame_slot,
+ "invalid frame-address slot");
+ if (slot->kind != FS_LOCAL && slot->kind != FS_PARAM)
+ mir_fail(f, in, ctx->stage, "frame-address recipe names spill slot",
+ op->v.frame_slot, slot->kind);
+ return;
+ case OPK_REG:
+ if (!mir_type_scalar(f, op->type) || op->cls >= OPT_REG_CLASSES ||
+ opt_value_reg_class(f->c, op->type) != op->cls)
+ mir_fail(f, in, ctx->stage, "invalid register value type/class",
+ op->type, op->cls);
+ mir_verify_reg(f, in, ctx->stage, op->v.reg, op->cls,
+ "unknown physical register operand");
+ return;
+ case OPK_INDIRECT:
+ mir_fail(f, in, ctx->stage, "nested indirect operand", op->kind, 0);
+ return;
+ default:
+ mir_fail(f, in, ctx->stage, "unknown operand kind", op->kind, 0);
+ return;
+ }
+}
+
+static void mir_verify_indirect_component(Func* f, Inst* in, Operand* part,
+ int is_def, MirVerifyCtx* ctx) {
+ const Operand* owner = ctx->indirect;
+ u32 value;
+ u8 kind;
+ KitCgTypeId type;
+ u8 expect_kind;
+ if (!owner || (ctx->indirect_part != 1u && ctx->indirect_part != 2u))
+ mir_fail(f, in, ctx->stage, "unexpected indirect component", part->kind,
+ part->flags);
+ if (part->flags != OPT_OPERAND_WALK_INDIRECT_PART || is_def || part->shift ||
+ part->cls != RC_INT)
+ mir_fail(f, in, ctx->stage, "malformed indirect component view",
+ part->flags, part->kind);
+ if (ctx->indirect_part == 1u) {
+ value = owner->v.ind.base;
+ kind = owner->v.ind.base_kind;
+ type = owner->v.ind.base_type;
+ } else {
+ value = owner->v.ind.index;
+ kind = owner->v.ind.index_kind;
+ type = owner->v.ind.index_type;
+ }
+ expect_kind = kind == OPT_INDIRECT_REG
+ ? OPK_REG
+ : (kind == OPT_INDIRECT_FRAME ? OPK_STACK
+ : OPK_FRAME_ADDR);
+ if (part->kind != expect_kind || part->type != type ||
+ (part->kind == OPK_REG ? (u32)part->v.reg
+ : (u32)part->v.frame_slot) != value)
+ mir_fail(f, in, ctx->stage, "indirect component view disagrees with owner",
+ ctx->indirect_part, part->kind);
+
+ if (ctx->indirect_part == 1u &&
+ !(owner->v.ind.index_kind == OPT_INDIRECT_REG &&
+ owner->v.ind.index == (Reg)REG_NONE)) {
+ ctx->indirect_part = 2u;
+ } else {
+ ctx->indirect = NULL;
+ ctx->indirect_part = 0;
+ }
+}
+
+static void mir_verify_indirect(Func* f, Inst* in, Operand* op,
+ MirVerifyCtx* ctx) {
+ IRFrameSlot* slot;
+ int has_index;
+ int address_only = (IROp)in->op == IR_ADDR_OF && in->nopnds >= 2u &&
+ op == &in->opnds[1];
+ if (op->flags || op->shift || op->cls != RC_INT)
+ mir_fail(f, in, ctx->stage, "invalid indirect owner shape", op->flags,
+ op->shift);
+ /* IR_ADDR_OF computes the identity of its lvalue and may therefore retain an
+ * incomplete flexible-array referent. Instructions that dereference the
+ * recipe still require sized storage. */
+ if (!(address_only ? mir_type_known(f, op->type)
+ : mir_type_valid(f, op->type)))
+ mir_fail(f, in, ctx->stage, "invalid indirect owner type", op->type,
+ (u32)address_only);
+ if (op->v.ind.base_kind > OPT_INDIRECT_FRAME_ADDR)
+ mir_fail(f, in, ctx->stage, "invalid indirect base location kind",
+ op->v.ind.base_kind, op->v.ind.base);
+ if (!mir_type_address_component(f, op->v.ind.base_type))
+ mir_fail(f, in, ctx->stage, "invalid indirect base type",
+ op->v.ind.base_type, op->v.ind.base_kind);
+ switch ((OptIndirectLocKind)op->v.ind.base_kind) {
+ case OPT_INDIRECT_REG:
+ mir_verify_reg(f, in, ctx->stage, op->v.ind.base, RC_INT,
+ "unknown indirect base register");
+ break;
+ case OPT_INDIRECT_FRAME:
+ slot = mir_frame_slot(f, in, ctx->stage, (FrameSlot)op->v.ind.base,
+ "invalid indirect base spill slot");
+ if (slot->kind != FS_SPILL)
+ mir_fail(f, in, ctx->stage, "indirect base value names semantic slot",
+ op->v.ind.base, slot->kind);
+ mir_verify_slot_fits(f, in, ctx->stage, slot, op->v.ind.base_type,
+ "indirect base type does not fit spill slot");
+ break;
+ case OPT_INDIRECT_FRAME_ADDR:
+ slot = mir_frame_slot(f, in, ctx->stage, (FrameSlot)op->v.ind.base,
+ "invalid indirect frame-address base slot");
+ if (slot->kind != FS_LOCAL && slot->kind != FS_PARAM)
+ mir_fail(f, in, ctx->stage,
+ "indirect frame-address base names spill slot",
+ op->v.ind.base, slot->kind);
+ break;
+ }
+
+ if (op->v.ind.index_kind > OPT_INDIRECT_FRAME_ADDR)
+ mir_fail(f, in, ctx->stage, "invalid indirect index location kind",
+ op->v.ind.index_kind, op->v.ind.index);
+ has_index = !(op->v.ind.index_kind == OPT_INDIRECT_REG &&
+ op->v.ind.index == (Reg)REG_NONE);
+ if (!has_index) {
+ if (op->v.ind.index_type != KIT_CG_TYPE_NONE || op->v.ind.log2_scale ||
+ op->v.ind.index_ext != OPT_IDX_EXT_NONE)
+ mir_fail(f, in, ctx->stage, "non-canonical absent indirect index",
+ op->v.ind.index_type, op->v.ind.log2_scale);
+ } else {
+ if (!opt_indirect_index_type_valid(f, op->v.ind.index_type))
+ mir_fail(f, in, ctx->stage, "invalid indirect index type",
+ op->v.ind.index_type, op->v.ind.index_kind);
+ switch ((OptIndirectLocKind)op->v.ind.index_kind) {
+ case OPT_INDIRECT_REG:
+ mir_verify_reg(f, in, ctx->stage, op->v.ind.index, RC_INT,
+ "unknown indirect index register");
+ break;
+ case OPT_INDIRECT_FRAME:
+ slot = mir_frame_slot(f, in, ctx->stage, (FrameSlot)op->v.ind.index,
+ "invalid indirect index spill slot");
+ if (slot->kind != FS_SPILL)
+ mir_fail(f, in, ctx->stage,
+ "indirect index value names semantic slot",
+ op->v.ind.index, slot->kind);
+ mir_verify_slot_fits(f, in, ctx->stage, slot, op->v.ind.index_type,
+ "indirect index type does not fit spill slot");
+ break;
+ case OPT_INDIRECT_FRAME_ADDR:
+ mir_fail(f, in, ctx->stage,
+ "frame address cannot be an indirect index",
+ op->v.ind.index, op->v.ind.index_type);
+ break;
+ }
+ }
+ if (op->v.ind.log2_scale > 3u)
+ mir_fail(f, in, ctx->stage, "invalid indirect scale",
+ op->v.ind.log2_scale, op->v.ind.index);
+ if (op->v.ind.index_ext > OPT_IDX_EXT_UXTW ||
+ (op->v.ind.index_ext != OPT_IDX_EXT_NONE &&
+ (!has_index || op->v.ind.index_kind == OPT_INDIRECT_FRAME_ADDR ||
+ cg_type_size(f->c, op->v.ind.index_type) != 4u)))
+ mir_fail(f, in, ctx->stage, "invalid indirect index extension",
+ op->v.ind.index_ext, op->v.ind.index_type);
+
+ ctx->indirect = op;
+ ctx->indirect_part = 1u;
+}
+
static void mir_verify_operand(Func* f, Inst* in, Operand* op, int is_def,
void* arg) {
- (void)in;
- (void)is_def;
- const char* stage = (const char*)arg;
+ MirVerifyCtx* ctx = (MirVerifyCtx*)arg;
if (!op) return;
- if (op->kind == OPK_REG) {
- mir_verify_reg(f, stage, op->v.reg, op->cls, "register operand");
- } else if (op->kind == OPK_INDIRECT) {
- mir_verify_reg(f, stage, op->v.ind.base, RC_INT, "indirect base");
- if (op->v.ind.index != (Reg)REG_NONE)
- mir_verify_reg(f, stage, op->v.ind.index, RC_INT, "indirect index");
- } else if (op->kind == OPK_LOCAL) {
- if (op->v.frame_slot == FRAME_SLOT_NONE ||
- op->v.frame_slot > f->nframe_slots) {
- SrcLoc loc = {0, 0, 0};
- compiler_panic(f->c, loc, "opt MIR verify %s: bad frame slot %u",
- stage ? stage : "?", (unsigned)op->v.frame_slot);
- }
+ if (ctx->indirect) {
+ mir_verify_indirect_component(f, in, op, is_def, ctx);
+ return;
+ }
+ if (op->flags)
+ mir_fail(f, in, ctx->stage, "stored operand has transient flags", op->kind,
+ op->flags);
+ if (op->kind == OPK_INDIRECT) {
+ mir_verify_indirect(f, in, op, ctx);
+ return;
}
+ mir_verify_scalar_location(f, in, op, is_def, ctx);
}
void opt_mir_verify(Func* f, const char* stage) {
Func v;
- if (!mir_view(f, &v)) return;
+ if (!opt_mir_view(f, &v)) return;
+ mir_verify_frame_table(&v, stage);
for (u32 b = 0; b < v.nblocks; ++b) {
Block* bl = &v.blocks[b];
+ if (bl->id != b)
+ mir_fail(&v, NULL, stage, "non-canonical MIR block id", bl->id, b);
for (u32 i = 0; i < bl->ninsts; ++i) {
Inst* in = &bl->insts[i];
+ MirVerifyCtx ctx;
+ memset(&ctx, 0, sizeof ctx);
+ ctx.stage = stage;
if ((IROp)in->op == IR_PHI) {
- SrcLoc loc = in->loc;
- compiler_panic(f->c, loc, "opt MIR verify %s: phi survived lowering",
- stage ? stage : "?");
+ mir_fail(&v, in, stage, "phi survived lowering", b, i);
+ }
+ if ((IROp)in->op > IR_INTRINSIC)
+ mir_fail(&v, in, stage, "invalid MIR opcode", in->op, in->id);
+ if ((in->nopnds != 0u) != (in->opnds != NULL))
+ mir_fail(&v, in, stage, "inconsistent MIR operand array", in->nopnds,
+ in->opnds != NULL);
+ if (in->def != VAL_NONE || in->ndefs != 0u || in->defs != NULL)
+ mir_fail(&v, in, stage, "stale semantic definition metadata", in->def,
+ in->ndefs);
+ if ((IROp)in->op == IR_PARAM_DECL) {
+ if (in->nopnds)
+ mir_fail(&v, in, stage, "MIR parameter marker carries operands", b,
+ i);
+ continue;
}
- if ((IROp)in->op == IR_PARAM_DECL) continue;
- opt_walk_inst_operands(&v, in, mir_verify_operand, (void*)stage);
+ if ((IROp)in->op == IR_ASM_BLOCK) mir_verify_asm_aux(&v, in, stage);
+ opt_walk_inst_operands(&v, in, mir_verify_operand, &ctx);
+ if (ctx.indirect)
+ mir_fail(&v, in, stage, "incomplete indirect component walk",
+ ctx.indirect_part, in->id);
}
}
}
void opt_mir_combine(Func* f, NativeTarget* target) {
Func v;
- if (!mir_view(f, &v)) return;
+ if (!opt_mir_view(f, &v)) return;
opt_combine(&v, target);
- mir_commit(f, &v);
+ opt_mir_commit(f, &v);
}
void opt_mir_dce(Func* f) {
Func v;
- if (!mir_view(f, &v)) return;
+ if (!opt_mir_view(f, &v)) return;
opt_dce(&v);
- mir_commit(f, &v);
+ opt_mir_commit(f, &v);
}
void opt_mir_build_cfg(Func* f) {
Func v;
- if (!mir_view(f, &v)) return;
+ if (!opt_mir_view(f, &v)) return;
opt_build_cfg(&v);
- mir_commit(f, &v);
+ opt_mir_commit(f, &v);
}
void opt_mir_jump_cleanup(Func* f, OptJumpCleanupStage stage) {
Func v;
- if (!mir_view(f, &v)) return;
+ if (!opt_mir_view(f, &v)) return;
opt_jump_cleanup(&v, stage);
- mir_commit(f, &v);
+ opt_mir_commit(f, &v);
}
diff --git a/src/opt/pass_native_emit.c b/src/opt/pass_native_emit.c
@@ -1,5 +1,6 @@
#include <string.h>
+#include "cg/native_asm.h"
#include "cg/type.h"
#include "core/metrics.h"
#include "core/pool.h"
@@ -17,6 +18,8 @@
#undef CGCallPlanRet
#undef CGScopeDesc
+typedef struct NativeEmitTempScope NativeEmitTempScope;
+
typedef struct NativeEmitCtx {
Compiler* c;
Func* f;
@@ -34,8 +37,41 @@ typedef struct NativeEmitCtx {
* branch-to-epilogue: func_end places the epilogue label at the very next
* position, so the branch would just jump to the next 4 bytes. */
u8 emitting_terminal_ret;
+ /* Instruction-scoped materialization state. NULL outside emit_inst (notably
+ * during frame planning/parameter binding). */
+ NativeEmitTempScope* temps;
+ /* Physical registers live after each MIR instruction, indexed by InstId.
+ * Used only to scavenge dead caller-saved registers after the dedicated temp
+ * bank is exhausted. */
+ OptHardRegSet* live_after_by_inst;
+ u32 live_after_cap;
+ /* Clean frame-value forwarding cache, indexed by physical register. A
+ * nonzero slot means the register holds the exact value already stored in
+ * that private FS_SPILL slot. Memory is always authoritative. */
+ FrameSlot frame_cache_slot[OPT_REG_CLASSES][OPT_MAX_HARD_REGS];
+ KitCgTypeId frame_cache_type[OPT_REG_CLASSES][OPT_MAX_HARD_REGS];
+ u32 frame_cache_held[OPT_REG_CLASSES];
+ /* Callee-saved registers selected by the pre-frame asm staging plan. The
+ * actual instruction-local allocator is deterministic and chooses the same
+ * registers during emission; these masks make their ABI preservation part of
+ * the known frame instead of allocating save slots mid-body. */
+ u32 asm_stage_callee_used[OPT_REG_CLASSES];
+ /* Callee-saved registers already represented in the known frame, from MIR
+ * allocation or asm staging. A dead register in this set may safely retain a
+ * clean spill value without creating a new ABI preservation obligation. */
+ u32 callee_saved_used[OPT_REG_CLASSES];
} NativeEmitCtx;
+static void frame_cache_clear(NativeEmitCtx* e);
+static void frame_cache_drop_reg(NativeEmitCtx* e, u32 cls, Reg reg);
+static void frame_cache_invalidate_masks(NativeEmitCtx* e,
+ const u32* masks);
+static NativeLoc frame_cache_take_slot(NativeEmitCtx* e, FrameSlot slot,
+ NativeLoc source,
+ NativeAllocClass cls,
+ KitCgTypeId type, Reg avoid_a,
+ Reg avoid_b);
+
static _Noreturn void emit_panic(NativeEmitCtx* e, SrcLoc loc,
const char* msg) {
compiler_panic(e->c, loc, "opt native emit: %s", msg);
@@ -184,6 +220,16 @@ static NativeLoc loc_frame(KitCgTypeId type, NativeAllocClass cls,
return loc;
}
+static NativeLoc loc_frame_addr(KitCgTypeId type, NativeFrameSlot slot) {
+ NativeLoc loc;
+ memset(&loc, 0, sizeof loc);
+ loc.kind = NATIVE_LOC_FRAME_ADDR;
+ loc.cls = NATIVE_REG_INT;
+ loc.type = type;
+ loc.v.frame = slot;
+ return loc;
+}
+
static NativeLoc loc_imm(KitCgTypeId type, i64 imm) {
NativeLoc loc;
memset(&loc, 0, sizeof loc);
@@ -210,45 +256,444 @@ static int loc_same_frame(NativeLoc a, NativeLoc b) {
a.v.frame == b.v.frame;
}
-static Reg scratch_reg(NativeEmitCtx* e, NativeAllocClass cls, Reg a, Reg b,
- SrcLoc loc) {
+#define NATIVE_EMIT_TEMP_CACHE 16u
+
+typedef struct NativeEmitTempCacheEntry {
+ NativeLoc source;
+ NativeLoc result;
+} NativeEmitTempCacheEntry;
+
+struct NativeEmitTempScope {
+ NativeEmitCtx* emit;
+ const Inst* inst;
+ u8 allow_asm_temps;
+ u8 pad[3];
+ u32 leased[OPT_REG_CLASSES];
+ u32 unavailable[OPT_REG_CLASSES];
+ /* Definition registers that currently hold a cached stack source consumed
+ * by this same instruction. They may be used in place as source+destination
+ * even though the normal avoidance set contains the destination register. */
+ u32 cache_def_reuse[OPT_REG_CLASSES];
+ NativeEmitTempCacheEntry cache[NATIVE_EMIT_TEMP_CACHE];
+ u32 ncache;
+};
+
+typedef struct NativeEmitTempMark {
+ u32 leased[OPT_REG_CLASSES];
+ u32 ncache;
+} NativeEmitTempMark;
+
+typedef struct FrameCacheNeedCtx {
+ NativeEmitCtx* emit;
+ u32 regs[OPT_REG_CLASSES];
+} FrameCacheNeedCtx;
+
+static void frame_cache_collect_needed(const Inst* in,
+ FrameCacheNeedCtx* need) {
+ NativeEmitCtx* e = need->emit;
+ for (u32 i = 0; in && i < in->nopnds; ++i) {
+ const OptOperand* op = &in->opnds[i];
+ if (opt_inst_operand_is_def(in, i) || op->kind != OPK_STACK) continue;
+ for (u32 c = 0; c < OPT_REG_CLASSES; ++c)
+ for (Reg r = 0; r < OPT_MAX_HARD_REGS; ++r)
+ if ((e->frame_cache_held[c] & (1u << r)) &&
+ e->frame_cache_slot[c][r] == op->v.frame_slot &&
+ e->frame_cache_type[c][r] == op->type)
+ need->regs[c] |= 1u << r;
+ }
+}
+
+static int temp_loc_equal(NativeLoc a, NativeLoc b) {
+ if (a.kind != b.kind || a.cls != b.cls || a.type != b.type) return 0;
+ switch ((NativeLocKind)a.kind) {
+ case NATIVE_LOC_NONE:
+ return 1;
+ case NATIVE_LOC_REG:
+ return a.v.reg == b.v.reg;
+ case NATIVE_LOC_FRAME:
+ case NATIVE_LOC_FRAME_ADDR:
+ return a.v.frame == b.v.frame;
+ case NATIVE_LOC_STACK:
+ return a.v.stack.slot == b.v.stack.slot &&
+ a.v.stack.offset == b.v.stack.offset;
+ case NATIVE_LOC_IMM:
+ return a.v.imm == b.v.imm;
+ case NATIVE_LOC_GLOBAL:
+ return a.v.global.sym == b.v.global.sym &&
+ a.v.global.addend == b.v.global.addend;
+ case NATIVE_LOC_ADDR:
+ return memcmp(&a.v.addr, &b.v.addr, sizeof a.v.addr) == 0;
+ }
+ return 0;
+}
+
+static void temp_scope_begin(NativeEmitCtx* e, NativeEmitTempScope* scope,
+ const Inst* in) {
+ OptRegEffects effects;
+ FrameCacheNeedCtx needed;
+ u32 live_after[OPT_REG_CLASSES];
+ u32 invalidate[OPT_REG_CLASSES];
+ memset(scope, 0, sizeof *scope);
+ memset(&needed, 0, sizeof needed);
+ memset(live_after, 0, sizeof live_after);
+ scope->emit = e;
+ scope->inst = in;
+ if (in && in->id != INST_ID_NONE && in->id < e->live_after_cap)
+ for (u32 c = 0; c < OPT_REG_CLASSES; ++c) {
+ live_after[c] = e->live_after_by_inst[in->id].cls[c];
+ scope->unavailable[c] = live_after[c];
+ }
+ opt_inst_reg_effects(e->f, in, &effects);
+ needed.emit = e;
+ frame_cache_collect_needed(in, &needed);
+ for (u32 c = 0; c < OPT_REG_CLASSES; ++c) {
+ u32 clobbers = effects.clobbers.cls[c];
+ /* Call clobbers happen after argument/callee staging. A call-local temp is
+ * consumed by the parallel-copy plan before emit_call and may therefore
+ * live in a caller-saved register; forbidding the entire ABI clobber set
+ * makes high-pressure calls impossible precisely when every callee-saved
+ * allocation register is live across them. Calls clear the block cache
+ * below, so this exception cannot preserve a stale value past the call. */
+ if (in && (IROp)in->op == IR_CALL) clobbers = 0u;
+ scope->unavailable[c] |=
+ effects.uses.cls[c] | effects.defs.cls[c] | clobbers;
+ scope->cache_def_reuse[c] =
+ needed.regs[c] & effects.defs.cls[c] &
+ ~(effects.uses.cls[c] | effects.clobbers.cls[c]);
+ invalidate[c] = scope->unavailable[c] & ~scope->cache_def_reuse[c];
+ }
+ if (in && ((IROp)in->op == IR_CALL || (IROp)in->op == IR_ASM_BLOCK))
+ frame_cache_clear(e);
+ else
+ frame_cache_invalidate_masks(e, invalidate);
+ e->temps = scope;
+}
+
+static NativeEmitTempMark temp_scope_mark(NativeEmitCtx* e) {
+ NativeEmitTempMark mark;
+ memset(&mark, 0, sizeof mark);
+ if (!e->temps) return mark;
+ memcpy(mark.leased, e->temps->leased, sizeof mark.leased);
+ mark.ncache = e->temps->ncache;
+ return mark;
+}
+
+static void temp_scope_rewind(NativeEmitCtx* e, NativeEmitTempMark mark) {
+ if (!e->temps) return;
+ memcpy(e->temps->leased, mark.leased, sizeof mark.leased);
+ e->temps->ncache = mark.ncache;
+}
+
+/* Forget every instruction-local materialization fact whose value lived in
+ * `reg`. This does not release the register: a caller that overwrote a cached
+ * value in place still owns the new result through the existing lease. */
+static int temp_scope_drop_cached_reg(NativeEmitCtx* e,
+ NativeAllocClass cls, Reg reg) {
u32 c = (u32)cls;
- if (c < OPT_REG_CLASSES) {
- for (u32 i = 0; i < e->f->opt_scratch_reg_count[c]; ++i) {
- Reg r = e->f->opt_scratch_regs[c][i];
- if (r != a && r != b) return r;
+ u32 out = 0;
+ int owned = 0;
+ if (!e->temps || c >= OPT_REG_CLASSES || reg >= OPT_MAX_HARD_REGS) return 0;
+ for (u32 i = 0; i < e->temps->ncache; ++i) {
+ NativeEmitTempCacheEntry entry = e->temps->cache[i];
+ if (entry.result.kind == NATIVE_LOC_REG && entry.result.cls == (u8)cls &&
+ entry.result.v.reg == reg) {
+ owned = 1;
+ continue;
}
+ if (out != i) e->temps->cache[out] = entry;
+ ++out;
}
- emit_panic(e, loc, "no scratch register for native emission");
+ if (!owned) return 0;
+ e->temps->ncache = out;
+ e->temps->cache_def_reuse[c] &= ~(1u << reg);
+ return 1;
+}
+
+/* Relinquish a register only when this instruction acquired it as a cached
+ * materialization. Earlier ABI-phase destinations are leased without a cache
+ * entry and therefore remain protected. */
+static void temp_scope_release_cached_reg(NativeEmitCtx* e,
+ NativeAllocClass cls, Reg reg) {
+ u32 c = (u32)cls;
+ if (!temp_scope_drop_cached_reg(e, cls, reg)) return;
+ e->temps->leased[c] &= ~(1u << reg);
+}
+
+/* End one ordering phase inside a compound native operation. A call consumes
+ * all argument/callee staging before it clobbers caller-saved registers; none
+ * of those leases or materialization facts may be observed by return
+ * writeback. The instruction scope (and its hard-reg avoidance set) remains
+ * active for the post-call phase. */
+static void temp_scope_phase_barrier(NativeEmitCtx* e) {
+ if (!e->temps) return;
+ memset(e->temps->leased, 0, sizeof e->temps->leased);
+ e->temps->ncache = 0;
+}
+
+/* Keep a completed register destination owned until the current compound
+ * operation reaches its next ordering barrier. write_loc deliberately
+ * rewinds the temporary leases used to perform one move; without this separate
+ * phase lease, a later move could reuse (and overwrite) an earlier ABI
+ * argument or return register while materializing its own source. */
+static void temp_scope_hold_loc(NativeEmitCtx* e, NativeLoc loc) {
+ u32 c;
+ Reg r;
+ if (!e->temps || loc.kind != NATIVE_LOC_REG) return;
+ c = (u32)loc.cls;
+ r = loc.v.reg;
+ if (c >= OPT_REG_CLASSES || r >= OPT_MAX_HARD_REGS)
+ emit_panic(e, e->temps->inst ? e->temps->inst->loc : (SrcLoc){0, 0, 0},
+ "invalid phase-owned register");
+ frame_cache_drop_reg(e, c, r);
+ e->temps->leased[c] |= 1u << r;
+}
+
+static void temp_scope_end(NativeEmitCtx* e, NativeEmitTempScope* scope) {
+ if (e->temps != scope)
+ emit_panic(e, scope->inst ? scope->inst->loc : (SrcLoc){0, 0, 0},
+ "temporary scope mismatch");
+#ifndef NDEBUG
+ for (u32 c = 0; c < OPT_REG_CLASSES; ++c)
+ if (scope->leased[c] & e->frame_cache_held[c])
+ emit_panic(e, scope->inst ? scope->inst->loc : (SrcLoc){0, 0, 0},
+ "temporary lease escaped into frame cache");
+#endif
+ memset(scope->leased, 0, sizeof scope->leased);
+ scope->ncache = 0;
+ e->temps = NULL;
+}
+
+static int temp_reg_available(NativeEmitCtx* e, NativeAllocClass cls, Reg r,
+ Reg a, Reg b) {
+ u32 c = (u32)cls;
+ u32 bit;
+ if (r == a || r == b || r >= 32u || c >= OPT_REG_CLASSES) return 0;
+ bit = 1u << r;
+ if (!e->temps) return 1;
+ return !(e->temps->leased[c] & bit) &&
+ !(e->temps->unavailable[c] & bit) &&
+ !(e->frame_cache_held[c] & bit);
}
-static int scratch_available(NativeEmitCtx* e, NativeAllocClass cls, Reg a,
- Reg b) {
+static Reg temp_acquire(NativeEmitCtx* e, NativeAllocClass cls, Reg a, Reg b,
+ SrcLoc loc) {
u32 c = (u32)cls;
if (c < OPT_REG_CLASSES) {
- for (u32 i = 0; i < e->f->opt_scratch_reg_count[c]; ++i) {
- Reg r = e->f->opt_scratch_regs[c][i];
- if (r != a && r != b) return 1;
+ /* Prefer the target-declared O1 operand-temp bank. These registers are
+ * excluded from allocation, but still respect explicit/implicit instruction
+ * clobbers and other leases in this scope. */
+ for (u32 i = 0; i < e->f->emit_temp_reg_count[c]; ++i) {
+ Reg r = e->f->emit_temp_regs[c][i];
+ if (!temp_reg_available(e, cls, r, a, b)) continue;
+ if (e->temps) e->temps->leased[c] |= 1u << r;
+ return r;
+ }
+ /* A dead caller-saved allocation register is an equally local temporary:
+ * it needs no prologue save and liveness proves its value is dead before
+ * we overwrite it. This is the pressure escape hatch for two-temp targets
+ * and for ARM32, whose backend-internal LR must not be leased. */
+ if (e->temps) {
+ for (u32 i = 0; i < e->f->opt_hard_reg_count[c]; ++i) {
+ Reg r = e->f->opt_hard_regs[c][i];
+ if (r >= 32u || !(e->f->opt_caller_saved[c] & (1u << r))) continue;
+ if (!temp_reg_available(e, cls, r, a, b)) continue;
+ e->temps->leased[c] |= 1u << r;
+ return r;
+ }
+ }
+ /* A clean cache entry is always evictable: its frame store is already
+ * authoritative. Prefer losing the forwarding opportunity over failing a
+ * real instruction's temporary demand. */
+ if (e->temps) {
+ for (u32 i = 0; i < e->f->emit_temp_reg_count[c]; ++i) {
+ Reg r = e->f->emit_temp_regs[c][i];
+ u32 bit = r < 32u ? 1u << r : 0u;
+ if (!bit || !(e->frame_cache_held[c] & bit) || r == a || r == b ||
+ (e->temps->leased[c] & bit) ||
+ (e->temps->unavailable[c] & bit))
+ continue;
+ frame_cache_drop_reg(e, c, r);
+ e->temps->leased[c] |= bit;
+ return r;
+ }
}
}
- return 0;
+ compiler_panic(
+ e->c, loc,
+ "opt native emit: no instruction-local native temporary "
+ "(class=%u op=%u inst=%u leased=0x%08x unavailable=0x%08x "
+ "cached=0x%08x avoid=%u/%u)",
+ (unsigned)c,
+ e->temps && e->temps->inst ? (unsigned)e->temps->inst->op : 0u,
+ e->temps && e->temps->inst ? (unsigned)e->temps->inst->id : 0u,
+ e->temps && c < OPT_REG_CLASSES ? (unsigned)e->temps->leased[c] : 0u,
+ e->temps && c < OPT_REG_CLASSES ? (unsigned)e->temps->unavailable[c]
+ : 0u,
+ c < OPT_REG_CLASSES ? (unsigned)e->frame_cache_held[c] : 0u,
+ (unsigned)a, (unsigned)b);
+}
+
+static int asm_temp_reg_ok(NativeEmitCtx* e, NativeAllocClass cls, Reg r,
+ u32 allowed_mask) {
+ if (r >= 32u || (allowed_mask && !(allowed_mask & (1u << r)))) return 0;
+ return !e->target->regs || !e->target->regs->asm_operand_reg_ok ||
+ e->target->regs->asm_operand_reg_ok(e->target->regs, cls, r);
+}
+
+static const NativeAllocClassInfo* emit_class_info(NativeEmitCtx* e,
+ NativeAllocClass cls) {
+ if (!e->target->regs) return NULL;
+ for (u32 i = 0; i < e->target->regs->nclasses; ++i)
+ if (e->target->regs->classes[i].cls == (u8)cls)
+ return &e->target->regs->classes[i];
+ return NULL;
}
-/* True iff `r` is one of this class's reserved scratch registers. Such a register
- * is never assigned to an allocator-live value: when it appears as an address's
- * base it holds only a transient (a reloaded spilled base or a materialized
- * frame/global address) that is dead after the current access. */
-static int reg_is_scratch(NativeEmitCtx* e, NativeAllocClass cls, Reg r) {
+static void frame_cache_drop_reg(NativeEmitCtx* e, u32 cls, Reg reg) {
+ if (cls >= OPT_REG_CLASSES || reg >= OPT_MAX_HARD_REGS) return;
+ e->frame_cache_slot[cls][reg] = FRAME_SLOT_NONE;
+ e->frame_cache_type[cls][reg] = 0;
+ e->frame_cache_held[cls] &= ~(1u << reg);
+}
+
+static void frame_cache_clear(NativeEmitCtx* e) {
+ memset(e->frame_cache_slot, 0, sizeof e->frame_cache_slot);
+ memset(e->frame_cache_type, 0, sizeof e->frame_cache_type);
+ memset(e->frame_cache_held, 0, sizeof e->frame_cache_held);
+}
+
+static void frame_cache_invalidate_masks(NativeEmitCtx* e,
+ const u32* masks) {
+ if (!masks) return;
+ for (u32 c = 0; c < OPT_REG_CLASSES; ++c) {
+ u32 killed = e->frame_cache_held[c] & masks[c];
+ for (Reg r = 0; killed && r < 32u; ++r)
+ if (killed & (1u << r)) {
+ frame_cache_drop_reg(e, c, r);
+ killed &= ~(1u << r);
+ }
+ }
+}
+
+static u32 frame_cache_retainable_mask(NativeEmitCtx* e,
+ NativeAllocClass cls) {
+ const NativeAllocClassInfo* ci = emit_class_info(e, cls);
u32 c = (u32)cls;
- if (r == (Reg)REG_NONE || c >= OPT_REG_CLASSES) return 0;
- for (u32 i = 0; i < e->f->opt_scratch_reg_count[c]; ++i)
- if (e->f->opt_scratch_regs[c][i] == r) return 1;
+ u32 mask;
+ u32 preserved_allocable;
+ if (!ci || c >= OPT_REG_CLASSES) return 0u;
+ mask = ci->emit_cache_mask;
+ preserved_allocable = native_target_caller_saved_mask(e->target, cls);
+ preserved_allocable |= e->callee_saved_used[c];
+ /* An allocable register is a legal persistent cache only when acquisition
+ * proved it dead and its ABI preservation is already established:
+ * caller-saved until the next call boundary, or a callee-save already present
+ * in the known frame because MIR/asm uses it. The NativeTarget contract
+ * requires ordinary hooks to preserve such registers unless the current
+ * instruction's exhaustive machine_op_clobbers effect says otherwise;
+ * calls/asm clear the cache at their phase boundary. */
+ for (u32 i = 0; i < e->f->opt_hard_reg_count[c]; ++i) {
+ Reg r = e->f->opt_hard_regs[c][i];
+ if (r < 32u && (preserved_allocable & (1u << r))) mask |= 1u << r;
+ }
+ return mask;
+}
+
+/* Acquire a lease satisfying an inline-asm register constraint. Unlike the
+ * general temporary picker this may name a fixed ABI register (for example
+ * x86 "a"). Regalloc has already kept values live across the asm out of that
+ * register; the availability check enforces that contract at the boundary. */
+static Reg temp_acquire_asm(NativeEmitCtx* e, NativeAllocClass cls, Reg fixed,
+ u32 allowed_mask, SrcLoc loc) {
+ u32 c = (u32)cls;
+ if (!e->temps || c >= OPT_REG_CLASSES)
+ emit_panic(e, loc, "asm register staging outside temporary scope");
+ if (fixed != (Reg)REG_NONE) {
+ if (!asm_temp_reg_ok(e, cls, fixed, allowed_mask) ||
+ !temp_reg_available(e, cls, fixed, REG_NONE, REG_NONE))
+ emit_panic(e, loc, "fixed asm staging register is unavailable");
+ e->temps->leased[c] |= 1u << fixed;
+ return fixed;
+ }
+ for (u32 i = 0; i < e->f->emit_temp_reg_count[c]; ++i) {
+ Reg r = e->f->emit_temp_regs[c][i];
+ /* The O1 operand-temp bank is deliberately hidden from user
+ * named-register requirements and allocator homes, but it is valid as an
+ * internally substituted unrestricted asm operand. Restricted masks still
+ * apply. */
+ if (r >= 32u || (allowed_mask && !(allowed_mask & (1u << r))) ||
+ !temp_reg_available(e, cls, r, REG_NONE, REG_NONE))
+ continue;
+ e->temps->leased[c] |= 1u << r;
+ return r;
+ }
+ if (e->temps->allow_asm_temps) {
+ const NativeAllocClassInfo* ci = emit_class_info(e, cls);
+ for (u32 i = 0; ci && i < ci->nasm_temps; ++i) {
+ Reg r = ci->asm_temps[i];
+ if (r >= 32u || (allowed_mask && !(allowed_mask & (1u << r))) ||
+ !temp_reg_available(e, cls, r, REG_NONE, REG_NONE))
+ continue;
+ e->temps->leased[c] |= 1u << r;
+ return r;
+ }
+ }
+ /* An asm boundary may use every target-declared asm register, not just the
+ * allocator's general-purpose subset. This is what makes high-arity blocks
+ * possible without assigning a fixed register to an entire PReg live range.
+ * A setup prepass records any selected callee-saved registers in the known
+ * frame, so choosing one here remains ABI-safe. */
+ for (u32 i = 0; i < e->f->opt_phys_reg_count[c]; ++i) {
+ Reg r = e->f->opt_phys_regs[c][i].reg;
+ if (!asm_temp_reg_ok(e, cls, r, allowed_mask) ||
+ !temp_reg_available(e, cls, r, REG_NONE, REG_NONE))
+ continue;
+ e->temps->leased[c] |= 1u << r;
+ return r;
+ }
+ compiler_panic(e->c, loc,
+ "opt native emit: no register available for asm operand "
+ "staging (class=%u leased=0x%08x unavailable=0x%08x "
+ "allowed=0x%08x)",
+ (unsigned)c, (unsigned)e->temps->leased[c],
+ (unsigned)e->temps->unavailable[c],
+ (unsigned)allowed_mask);
+}
+
+/* A leased register is owned by the current emission scope and may therefore
+ * be overwritten in place. An ordinary OPK_REG may still be live after the
+ * instruction even when the result itself is frame-resident, so reusing it as
+ * the result would silently clobber an allocator-owned value. */
+static int temp_reg_is_leased(NativeEmitCtx* e, NativeAllocClass cls, Reg r) {
+ u32 c = (u32)cls;
+ return e->temps && c < OPT_REG_CLASSES && r < 32u &&
+ (e->temps->leased[c] & (1u << r)) != 0;
+}
+
+static int temp_available(NativeEmitCtx* e, NativeAllocClass cls, Reg a,
+ Reg b) {
+ u32 c = (u32)cls;
+ if (c < OPT_REG_CLASSES) {
+ for (u32 i = 0; i < e->f->emit_temp_reg_count[c]; ++i) {
+ Reg r = e->f->emit_temp_regs[c][i];
+ if (temp_reg_available(e, cls, r, a, b)) return 1;
+ }
+ if (e->temps) {
+ for (u32 i = 0; i < e->f->opt_hard_reg_count[c]; ++i) {
+ Reg r = e->f->opt_hard_regs[c][i];
+ if (r < 32u && (e->f->opt_caller_saved[c] & (1u << r)) &&
+ temp_reg_available(e, cls, r, a, b))
+ return 1;
+ }
+ }
+ }
return 0;
}
-static NativeLoc scratch_loc(NativeEmitCtx* e, KitCgTypeId type,
- NativeAllocClass cls, Reg a, Reg b, SrcLoc loc) {
- return loc_reg(type, cls, scratch_reg(e, cls, a, b, loc));
+static NativeLoc temp_loc_acquire(NativeEmitCtx* e, KitCgTypeId type,
+ NativeAllocClass cls, Reg a, Reg b,
+ SrcLoc loc) {
+ return loc_reg(type, cls, temp_acquire(e, cls, a, b, loc));
}
static NativeFrameSlot map_slot(NativeEmitCtx* e, NativeFrameSlot slot,
@@ -276,6 +721,10 @@ static NativeAddr addr_from_loc(NativeEmitCtx* e, NativeLoc loc,
addr.base_kind = NATIVE_ADDR_BASE_FRAME;
addr.base.frame = loc.v.frame;
return addr;
+ case NATIVE_LOC_FRAME_ADDR:
+ addr.base_kind = NATIVE_ADDR_BASE_FRAME;
+ addr.base.frame = loc.v.frame;
+ return addr;
case NATIVE_LOC_STACK:
addr.base_kind = NATIVE_ADDR_BASE_FRAME;
addr.base.frame = loc.v.stack.slot;
@@ -309,20 +758,90 @@ static NativeAddr addr_from_operand(NativeEmitCtx* e, const OptOperand* op,
addr.base_kind = NATIVE_ADDR_BASE_FRAME;
addr.base.frame = map_slot(e, op->v.frame_slot, loc);
return addr;
+ case OPT_OPK_FRAME_ADDR:
+ addr.base_kind = NATIVE_ADDR_BASE_FRAME;
+ addr.base.frame = map_slot(e, op->v.frame_slot, loc);
+ return addr;
+ case OPT_OPK_STACK:
+ addr.base_type = op->type;
+ {
+ NativeFrameSlot mapped = map_slot(e, op->v.frame_slot, loc);
+ NativeAllocClass cls = class_for_type(e, addr.base_type);
+ NativeLoc held = frame_cache_take_slot(
+ e, op->v.frame_slot, loc_frame(addr.base_type, cls, mapped), cls,
+ addr.base_type, REG_NONE, REG_NONE);
+ if (held.kind == NATIVE_LOC_REG) {
+ addr.base_kind = NATIVE_ADDR_BASE_REG;
+ addr.cls = held.cls;
+ addr.base.reg = held.v.reg;
+ } else {
+ addr.base_kind = NATIVE_ADDR_BASE_FRAME_VALUE;
+ addr.base.frame = mapped;
+ }
+ }
+ return addr;
case OPT_OPK_GLOBAL:
addr.base_kind = NATIVE_ADDR_BASE_GLOBAL;
addr.base.global.sym = op->v.global.sym;
addr.base.global.addend = op->v.global.addend;
return addr;
case OPT_OPK_INDIRECT:
- addr.base_kind = NATIVE_ADDR_BASE_REG;
addr.cls = NATIVE_REG_INT;
- addr.base.reg = op->v.ind.base;
- addr.index_kind = op->v.ind.index == (Reg)REG_NONE
- ? NATIVE_ADDR_INDEX_NONE
- : NATIVE_ADDR_INDEX_REG;
+ addr.base_type = op->v.ind.base_type;
+ if (!addr.base_type)
+ emit_panic(e, loc, "indirect base has no location type");
+ if (op->v.ind.base_kind == OPT_INDIRECT_FRAME) {
+ {
+ NativeFrameSlot mapped = map_slot(e, op->v.ind.base, loc);
+ NativeLoc held = frame_cache_take_slot(
+ e, op->v.ind.base,
+ loc_frame(addr.base_type, NATIVE_REG_INT, mapped),
+ NATIVE_REG_INT, addr.base_type, REG_NONE, REG_NONE);
+ if (held.kind == NATIVE_LOC_REG) {
+ addr.base_kind = NATIVE_ADDR_BASE_REG;
+ addr.base.reg = held.v.reg;
+ } else {
+ addr.base_kind = NATIVE_ADDR_BASE_FRAME_VALUE;
+ addr.base.frame = mapped;
+ }
+ }
+ } else if (op->v.ind.base_kind == OPT_INDIRECT_FRAME_ADDR) {
+ addr.base_kind = NATIVE_ADDR_BASE_FRAME;
+ addr.base.frame = map_slot(e, op->v.ind.base, loc);
+ } else {
+ addr.base_kind = NATIVE_ADDR_BASE_REG;
+ addr.base.reg = op->v.ind.base;
+ }
addr.index_cls = NATIVE_REG_INT;
- addr.index.reg = op->v.ind.index;
+ if (op->v.ind.index_kind == OPT_INDIRECT_FRAME) {
+ addr.index_type = op->v.ind.index_type;
+ if (!addr.index_type)
+ emit_panic(e, loc, "indirect index has no location type");
+ {
+ NativeFrameSlot mapped = map_slot(e, op->v.ind.index, loc);
+ NativeLoc held = frame_cache_take_slot(
+ e, op->v.ind.index,
+ loc_frame(addr.index_type, NATIVE_REG_INT, mapped),
+ NATIVE_REG_INT, addr.index_type, REG_NONE, REG_NONE);
+ if (held.kind == NATIVE_LOC_REG) {
+ addr.index_kind = NATIVE_ADDR_INDEX_REG;
+ addr.index.reg = held.v.reg;
+ } else {
+ addr.index_kind = NATIVE_ADDR_INDEX_FRAME_VALUE;
+ addr.index.frame = mapped;
+ }
+ }
+ } else if (op->v.ind.index_kind == OPT_INDIRECT_FRAME_ADDR) {
+ emit_panic(e, loc, "frame address cannot be an indirect index");
+ } else if (op->v.ind.index == (Reg)REG_NONE) {
+ addr.index_kind = NATIVE_ADDR_INDEX_NONE;
+ } else {
+ addr.index_type = op->v.ind.index_type;
+ if (!addr.index_type)
+ emit_panic(e, loc, "indirect index has no location type");
+ addr.index_kind = NATIVE_ADDR_INDEX_REG;
+ addr.index.reg = op->v.ind.index;
+ }
addr.log2_scale = op->v.ind.log2_scale;
/* L8 extend rider: OPT_IDX_EXT_* and NATIVE_ADDR_IDX_EXT_* share the
* 0=NONE/1=SXTW/2=UXTW encoding. Only set by the addr-fold recognition
@@ -367,7 +886,7 @@ static NativeAddr pointer_addr_from_operand(NativeEmitCtx* e,
return addr;
}
cls = class_for_type(e, op->type);
- r = scratch_reg(e, cls, avoid_a, avoid_b, loc);
+ r = temp_acquire(e, cls, avoid_a, avoid_b, loc);
memset(&frame, 0, sizeof frame);
frame.base_kind = NATIVE_ADDR_BASE_FRAME;
frame.base.frame = map_slot(e, op->v.frame_slot, loc);
@@ -379,6 +898,28 @@ static NativeAddr pointer_addr_from_operand(NativeEmitCtx* e,
addr.base.reg = r;
return addr;
}
+ case OPT_OPK_FRAME_ADDR:
+ addr.base_kind = NATIVE_ADDR_BASE_FRAME;
+ addr.base.frame = map_slot(e, op->v.frame_slot, loc);
+ return addr;
+ case OPT_OPK_STACK:
+ addr.cls = class_for_type(e, op->type);
+ addr.base_type = op->type;
+ {
+ NativeFrameSlot mapped = map_slot(e, op->v.frame_slot, loc);
+ NativeAllocClass cls = (NativeAllocClass)addr.cls;
+ NativeLoc held = frame_cache_take_slot(
+ e, op->v.frame_slot, loc_frame(addr.base_type, cls, mapped), cls,
+ addr.base_type, avoid_a, avoid_b);
+ if (held.kind == NATIVE_LOC_REG) {
+ addr.base_kind = NATIVE_ADDR_BASE_REG;
+ addr.base.reg = held.v.reg;
+ } else {
+ addr.base_kind = NATIVE_ADDR_BASE_FRAME_VALUE;
+ addr.base.frame = mapped;
+ }
+ }
+ return addr;
case OPT_OPK_GLOBAL:
addr.base_kind = NATIVE_ADDR_BASE_GLOBAL;
addr.base.global.sym = op->v.global.sym;
@@ -408,31 +949,37 @@ static Reg addr_index_reg(const NativeAddr* addr) {
static void collapse_addr_to_reg(NativeEmitCtx* e, NativeAddr* addr,
SrcLoc loc) {
- /* Materialize the full address into a reserved scratch register. We normally
- * must not reuse the base register as the destination: the register allocator
+ /* Materialize the full address into an instruction-local temporary. We
+ * normally must not reuse the base register as the destination: physical MIR
* may keep that value live past this memory op (e.g. a pointer stored into
* several of its own fields and then returned), so an in-place `add base,
* base, #off` would corrupt it. Avoid both base and index so load_addr can
* still read them.
*
- * Exception (the only case where no third scratch exists): when both base and
- * index are themselves reserved scratch registers, the address consumed the
- * entire scratch budget and there is nothing left to avoid them with. A
- * scratch base holds only a transient dead after this access, so folding the
- * index into it in place is sound and frees the other scratch for the caller.
- * load_addr reads the base before adding the (scaled) index, so dst==base is
- * correct; dst==index would clobber the index before use — base only. */
+ * If no third temporary is available and the base is itself a scoped
+ * temporary, that base is transient and may be updated in place. load_addr
+ * reads the base before adding the scaled index, so dst==base is sound;
+ * dst==index would clobber the index before use and is never selected. If the
+ * base is caller-owned, temp_acquire reports exhaustion instead of silently
+ * changing its value. */
Reg base = addr_base_reg(addr);
Reg index = addr_index_reg(addr);
Reg r;
- if (scratch_available(e, NATIVE_REG_INT, base, index))
- r = scratch_reg(e, NATIVE_REG_INT, base, index, loc);
- else if (reg_is_scratch(e, NATIVE_REG_INT, base))
+ if (temp_available(e, NATIVE_REG_INT, base, index))
+ r = temp_acquire(e, NATIVE_REG_INT, base, index, loc);
+ else if (base != index && temp_reg_is_leased(e, NATIVE_REG_INT, base))
r = base;
else
- r = scratch_reg(e, NATIVE_REG_INT, base, index, loc); /* diagnoses cleanly */
+ r = temp_acquire(e, NATIVE_REG_INT, base, index, loc); /* diagnoses cleanly */
NativeLoc dst = loc_reg(addr->base_type, NATIVE_REG_INT, r);
e->target->load_addr(e->target, dst, *addr);
+ /* load_addr replaced any prior value fact attached to its destination. The
+ * result remains leased; only the old cache identity is invalid now. */
+ temp_scope_drop_cached_reg(e, NATIVE_REG_INT, r);
+ if (base != r)
+ temp_scope_release_cached_reg(e, NATIVE_REG_INT, base);
+ if (index != r)
+ temp_scope_release_cached_reg(e, NATIVE_REG_INT, index);
memset(addr, 0, sizeof *addr);
addr->base_kind = NATIVE_ADDR_BASE_REG;
addr->cls = NATIVE_REG_INT;
@@ -463,6 +1010,11 @@ static NativeLoc loc_from_operand(NativeEmitCtx* e, const OptOperand* op,
case OPT_OPK_LOCAL:
return loc_frame(op->type, class_for_type(e, op->type),
map_slot(e, op->v.frame_slot, loc));
+ case OPT_OPK_STACK:
+ return loc_frame(op->type, class_for_type(e, op->type),
+ map_slot(e, op->v.frame_slot, loc));
+ case OPT_OPK_FRAME_ADDR:
+ return loc_frame_addr(op->type, map_slot(e, op->v.frame_slot, loc));
case OPT_OPK_INDIRECT: {
NativeLoc out = loc_none();
out.kind = NATIVE_LOC_ADDR;
@@ -482,56 +1034,306 @@ static NativeLoc materialize(NativeEmitCtx* e, NativeLoc src,
NativeAddr addr;
MemAccess mem;
if (src.kind == NATIVE_LOC_REG) return src;
- dst = scratch_loc(e, type ? type : src.type, cls, avoid_a, avoid_b, loc);
+ if (e->temps) {
+ KitCgTypeId want_type = type ? type : src.type;
+ for (u32 i = 0; i < e->temps->ncache; ++i) {
+ NativeEmitTempCacheEntry* entry = &e->temps->cache[i];
+ Reg cached_reg = entry->result.v.reg;
+ u32 cached_bit = cached_reg < 32u ? 1u << cached_reg : 0u;
+ int avoided = cached_reg == avoid_a || cached_reg == avoid_b;
+ if (entry->result.cls != (u8)cls || entry->result.type != want_type ||
+ (avoided &&
+ !(e->temps->cache_def_reuse[(u32)cls] & cached_bit)))
+ continue;
+ if (temp_loc_equal(entry->source, src)) return entry->result;
+ }
+ }
+ dst = temp_loc_acquire(e, type ? type : src.type, cls, avoid_a, avoid_b,
+ loc);
switch ((NativeLocKind)src.kind) {
case NATIVE_LOC_IMM:
e->target->load_imm(e->target, dst, src.v.imm);
- return dst;
+ break;
case NATIVE_LOC_GLOBAL:
addr = addr_from_loc(e, src, loc);
e->target->load_addr(e->target, dst, addr);
- return dst;
+ break;
+ case NATIVE_LOC_FRAME_ADDR:
+ addr = addr_from_loc(e, src, loc);
+ e->target->load_addr(e->target, dst, addr);
+ break;
case NATIVE_LOC_FRAME:
case NATIVE_LOC_STACK:
case NATIVE_LOC_ADDR:
addr = addr_from_loc(e, src, loc);
mem = mem_for_type(e->c, dst.type);
e->target->load(e->target, dst, addr, mem);
- return dst;
+ break;
default:
emit_panic(e, loc, "cannot materialize location");
}
+ if (e->temps && e->temps->ncache < NATIVE_EMIT_TEMP_CACHE) {
+ NativeEmitTempCacheEntry* entry = &e->temps->cache[e->temps->ncache++];
+ entry->source = src;
+ entry->result = dst;
+ }
+ return dst;
+}
+
+static int frame_cache_slot_is_exact_spill(NativeEmitCtx* e,
+ const OptOperand* op,
+ MemAccess mem) {
+ const IRFrameSlot* slot;
+ if (!op || op->kind != OPK_STACK || op->v.frame_slot == FRAME_SLOT_NONE ||
+ op->v.frame_slot > e->f->nframe_slots)
+ return 0;
+ slot = &e->f->frame_slots[op->v.frame_slot - 1u];
+ /* Spill-slot coloring may reuse one slot for different, non-overlapping type
+ * IDs. Exactness is the stored bit width; the cache entry separately records
+ * the current value's type/class and only answers an identical use. */
+ return slot->kind == FS_SPILL && slot->size >= mem.size &&
+ mem.size == type_size_or(e->c, op->type, 0u);
+}
+
+static void frame_cache_invalidate_slot(NativeEmitCtx* e, FrameSlot slot) {
+ if (slot == FRAME_SLOT_NONE) return;
+ for (u32 c = 0; c < OPT_REG_CLASSES; ++c)
+ for (Reg r = 0; r < OPT_MAX_HARD_REGS; ++r)
+ if (e->frame_cache_slot[c][r] == slot)
+ frame_cache_drop_reg(e, c, r);
+}
+
+static int frame_cache_types_compatible(NativeEmitCtx* e,
+ KitCgTypeId cached,
+ KitCgTypeId requested) {
+ (void)e;
+ return cached == requested;
+}
+
+static NativeLoc frame_cache_take_slot(NativeEmitCtx* e, FrameSlot slot,
+ NativeLoc source,
+ NativeAllocClass cls,
+ KitCgTypeId type, Reg avoid_a,
+ Reg avoid_b) {
+ u32 c = (u32)cls;
+ if (!e->temps || slot == FRAME_SLOT_NONE || c >= OPT_REG_CLASSES)
+ return loc_none();
+ for (Reg r = 0; r < OPT_MAX_HARD_REGS; ++r) {
+ u32 bit = 1u << r;
+ NativeLoc result;
+ int avoided = r == avoid_a || r == avoid_b;
+ if (!(e->frame_cache_held[c] & bit) ||
+ (avoided && !(e->temps->cache_def_reuse[c] & bit)) ||
+ e->frame_cache_slot[c][r] != slot ||
+ !frame_cache_types_compatible(e, e->frame_cache_type[c][r], type) ||
+ ((e->temps->unavailable[c] & bit) &&
+ !(e->temps->cache_def_reuse[c] & bit)))
+ continue;
+ frame_cache_drop_reg(e, c, r);
+ e->temps->leased[c] |= bit;
+ result = loc_reg(type, cls, r);
+ if (e->temps->ncache < NATIVE_EMIT_TEMP_CACHE) {
+ NativeEmitTempCacheEntry* entry =
+ &e->temps->cache[e->temps->ncache++];
+ entry->source = source;
+ entry->result = result;
+ }
+ return result;
+ }
+ return loc_none();
+}
+
+static NativeLoc frame_cache_take(NativeEmitCtx* e, const OptOperand* op,
+ NativeLoc source, NativeAllocClass cls,
+ KitCgTypeId type, Reg avoid_a, Reg avoid_b) {
+ if (!op || op->kind != OPK_STACK) return loc_none();
+ return frame_cache_take_slot(e, op->v.frame_slot, source, cls, type,
+ avoid_a, avoid_b);
+}
+
+static void frame_cache_note_store(NativeEmitCtx* e, const OptOperand* op,
+ NativeLoc src, MemAccess mem) {
+ NativeAllocClass cls;
+ u32 c, bit;
+ Reg r;
+ int retainable;
+ if (!op || op->kind != OPK_STACK) return;
+ frame_cache_invalidate_slot(e, op->v.frame_slot);
+ if (!e->temps || src.kind != NATIVE_LOC_REG ||
+ !frame_cache_slot_is_exact_spill(e, op, mem))
+ return;
+ cls = (NativeAllocClass)src.cls;
+ c = (u32)cls;
+ r = src.v.reg;
+ if (c >= OPT_REG_CLASSES || r >= OPT_MAX_HARD_REGS) return;
+ bit = 1u << r;
+ /* Cross-instruction retention follows the explicit NativeTarget preservation
+ * contract: declared cache-enabled emission temps plus liveness-proven dead
+ * O1 allocation registers
+ * whose caller/callee-save preservation is established, with every
+ * exceptional fixed clobber represented by the current instruction's
+ * exhaustive machine effect. */
+ retainable = (frame_cache_retainable_mask(e, cls) & bit) != 0;
+ if (!retainable) return;
+ frame_cache_drop_reg(e, c, r);
+ e->temps->leased[c] &= ~bit; /* transfer scope ownership to block cache */
+ e->frame_cache_slot[c][r] = op->v.frame_slot;
+ e->frame_cache_type[c][r] = op->type;
+ e->frame_cache_held[c] |= bit;
+}
+
+static NativeLoc materialize_operand(NativeEmitCtx* e, const OptOperand* op,
+ NativeAllocClass cls, KitCgTypeId type,
+ Reg avoid_a, Reg avoid_b, SrcLoc loc) {
+ NativeLoc source = loc_from_operand(e, op, loc);
+ NativeLoc cached = frame_cache_take(e, op, source, cls,
+ type ? type : source.type, avoid_a,
+ avoid_b);
+ if (cached.kind != NATIVE_LOC_NONE) return cached;
+ return materialize(e, source, cls, type, avoid_a, avoid_b, loc);
}
static void write_loc(NativeEmitCtx* e, NativeLoc dst, NativeLoc src,
MemAccess mem, SrcLoc loc) {
NativeAddr addr;
NativeLoc tmp;
- if (dst.kind == NATIVE_LOC_NONE) return;
- if (loc_same_frame(dst, src)) return;
+ NativeEmitTempMark mark = temp_scope_mark(e);
+ if (dst.kind == NATIVE_LOC_NONE) goto done;
+ if (loc_same_frame(dst, src)) goto done;
if (dst.kind == NATIVE_LOC_REG) {
if (src.kind == NATIVE_LOC_REG) {
if (dst.v.reg != src.v.reg || dst.cls != src.cls)
e->target->move(e->target, dst, src);
- return;
+ goto done;
}
/* An immediate goes straight into the destination register; routing it
- * through a scratch and then moving would cost an extra instruction. */
+ * through an instruction-local temp and then moving would cost an extra
+ * instruction. */
if (src.kind == NATIVE_LOC_IMM) {
e->target->load_imm(e->target, dst, src.v.imm);
- return;
+ goto done;
}
tmp = materialize(e, src, (NativeAllocClass)dst.cls, dst.type, dst.v.reg,
REG_NONE, loc);
if (tmp.v.reg != dst.v.reg || tmp.cls != dst.cls)
e->target->move(e->target, dst, tmp);
- return;
+ goto done;
}
addr = addr_from_loc(e, dst, loc);
if (src.kind != NATIVE_LOC_REG)
src = materialize(e, src, (NativeAllocClass)dst.cls, dst.type, REG_NONE,
REG_NONE, loc);
e->target->store(e->target, addr, src, mem);
+done:
+ temp_scope_rewind(e, mark);
+}
+
+/* Execute one move in a multi-move ABI phase and retain its register
+ * destination. The backend call/return planners promise that moves are
+ * source-safe in the order returned; this helper supplies the complementary
+ * destination-lifetime guarantee for emitter-side materialization. */
+static void write_phase_loc(NativeEmitCtx* e, NativeLoc dst, NativeLoc src,
+ MemAccess mem, SrcLoc loc) {
+ write_loc(e, dst, src, mem, loc);
+ temp_scope_hold_loc(e, dst);
+}
+
+static void write_operand(NativeEmitCtx* e, const OptOperand* dst,
+ NativeLoc src, MemAccess mem, SrcLoc loc) {
+ NativeLoc real_dst = loc_from_operand(e, dst, loc);
+ /* Keep the register that actually reaches a spill store. Besides avoiding a
+ * second materialization in write_loc, this lets ownership transfer from the
+ * instruction lease to the clean block cache. Constants and copies into a
+ * spill home are common producers and otherwise lose their forwarding fact
+ * at this abstraction boundary. */
+ if (dst && dst->kind == OPK_STACK && src.kind != NATIVE_LOC_REG &&
+ !loc_same_frame(real_dst, src))
+ src = materialize(e, src, (NativeAllocClass)real_dst.cls, real_dst.type,
+ REG_NONE, REG_NONE, loc);
+ write_loc(e, real_dst, src, mem, loc);
+ frame_cache_note_store(e, dst, src, mem);
+}
+
+/* Place a value directly in an already-leased register. This is the boundary
+ * operation used for fixed/constrained asm operands: going through materialize
+ * would acquire an unrelated temporary and then copy, which both wastes a
+ * register and can make a satisfiable high-arity asm look impossible. */
+static void load_loc_into_reg(NativeEmitCtx* e, NativeLoc dst, NativeLoc src,
+ SrcLoc loc) {
+ NativeAddr addr;
+ if (dst.kind != NATIVE_LOC_REG)
+ emit_panic(e, loc, "staging destination is not a register");
+ switch ((NativeLocKind)src.kind) {
+ case NATIVE_LOC_REG:
+ if (src.v.reg != dst.v.reg || src.cls != dst.cls)
+ e->target->move(e->target, dst, src);
+ return;
+ case NATIVE_LOC_IMM:
+ e->target->load_imm(e->target, dst, src.v.imm);
+ return;
+ case NATIVE_LOC_GLOBAL:
+ case NATIVE_LOC_FRAME_ADDR:
+ addr = addr_from_loc(e, src, loc);
+ e->target->load_addr(e->target, dst, addr);
+ return;
+ case NATIVE_LOC_FRAME:
+ case NATIVE_LOC_STACK:
+ case NATIVE_LOC_ADDR:
+ addr = addr_from_loc(e, src, loc);
+ e->target->load(e->target, dst, addr, mem_for_type(e->c, dst.type));
+ return;
+ default:
+ emit_panic(e, loc, "cannot stage asm operand in a register");
+ }
+}
+
+static int asm_reg_loc_satisfies(NativeEmitCtx* e, NativeLoc loc,
+ NativeAllocClass cls, Reg fixed,
+ u32 allowed_mask) {
+ if (loc.kind != NATIVE_LOC_REG || loc.cls != (u8)cls) return 0;
+ if (fixed != (Reg)REG_NONE && loc.v.reg != fixed) return 0;
+ return asm_temp_reg_ok(e, cls, loc.v.reg, allowed_mask);
+}
+
+static int asm_reg_requirement(NativeEmitCtx* e,
+ const IRAsmRegRequirement* req,
+ NativeAllocClass* cls, Reg* fixed,
+ u32* allowed, SrcLoc loc) {
+ if (!req || !req->present) return 0;
+ if (req->cls >= OPT_REG_CLASSES || req->fixed_reg < -1 ||
+ req->fixed_reg >= (i32)OPT_MAX_HARD_REGS)
+ emit_panic(e, loc, "invalid pre-resolved asm register requirement");
+ *cls = (NativeAllocClass)req->cls;
+ *fixed = req->fixed_reg < 0 ? REG_NONE : (Reg)req->fixed_reg;
+ *allowed = req->allowed_mask;
+ return 1;
+}
+
+/* Localize an asm register constraint at the instruction boundary. An
+ * unrestricted allocated register may already satisfy it; fixed/subset
+ * requirements are normally fulfilled by scoped staging. No emitter-owned
+ * register becomes a MIR home. */
+static NativeLoc asm_stage_reg_operand(NativeEmitCtx* e,
+ const AsmConstraint* constraint,
+ const IRAsmRegRequirement* req,
+ NativeLoc real, int load_value,
+ int* staged, SrcLoc loc) {
+ NativeAllocClass cls;
+ KitCgTypeId type;
+ Reg fixed;
+ u32 allowed;
+ Reg r;
+ NativeLoc out;
+ if (staged) *staged = 0;
+ if (!asm_reg_requirement(e, req, &cls, &fixed, &allowed, loc))
+ return real;
+ type = constraint->type ? constraint->type : real.type;
+ if (asm_reg_loc_satisfies(e, real, cls, fixed, allowed)) return real;
+ r = temp_acquire_asm(e, cls, fixed, allowed, loc);
+ out = loc_reg(type, cls, r);
+ if (load_value) load_loc_into_reg(e, out, real, loc);
+ if (staged) *staged = 1;
+ return out;
}
/* For an arithmetic / compare source operand: keep it as an immediate when it
@@ -543,9 +1345,8 @@ static NativeLoc operand_imm_or_reg(NativeEmitCtx* e, const OptOperand* op,
if (op->kind == OPK_IMM && e->target->imm_legal &&
e->target->imm_legal(e->target, use, sub, op->type, op->v.imm))
return loc_imm(op->type, op->v.imm);
- return materialize(e, loc_from_operand(e, op, loc),
- class_for_type(e, op->type), op->type, avoid_a, avoid_b,
- loc);
+ return materialize_operand(e, op, class_for_type(e, op->type), op->type,
+ avoid_a, avoid_b, loc);
}
static Reg loc_avoid_reg(NativeLoc l) {
@@ -562,6 +1363,73 @@ static int type_is_aggregate_or_large(NativeEmitCtx* e, KitCgTypeId type) {
type_size_or(e->c, type, 8u) > e->c->target.ptr_size);
}
+/* Acquire the register that receives a scalar memory load whose MIR result has
+ * a frame home. If the address already consumes every available value temp,
+ * first collapse it to one plain register. A leased integer base is then dead
+ * after address evaluation and may receive the loaded scalar in place: every
+ * NativeTarget load hook consumes a simple effective address before writing
+ * its destination. Never apply this to an allocator-owned base, an FP result,
+ * or a value wider than one machine register. */
+static NativeLoc acquire_frame_load_result(NativeEmitCtx* e,
+ KitCgTypeId type,
+ NativeAddr* addr, SrcLoc loc) {
+ NativeAllocClass cls = class_for_type(e, type);
+ Reg base = addr_base_reg(addr);
+ Reg index = addr_index_reg(addr);
+ if (temp_available(e, cls, base, index))
+ return temp_loc_acquire(e, type, cls, base, index, loc);
+
+ collapse_addr_to_reg(e, addr, loc);
+ base = addr_base_reg(addr);
+ if (cls == NATIVE_REG_INT &&
+ type_size_or(e->c, type, e->c->target.ptr_size) <=
+ e->c->target.ptr_size &&
+ addr->base_kind == NATIVE_ADDR_BASE_REG &&
+ addr->index_kind == NATIVE_ADDR_INDEX_NONE && addr->offset == 0 &&
+ temp_reg_is_leased(e, NATIVE_REG_INT, base))
+ return loc_reg(type, cls, base);
+
+ return temp_loc_acquire(e, type, cls, base, addr_index_reg(addr), loc);
+}
+
+/* Materialize an address whose MIR destination has a frame home. A normal
+ * free temp receives the address directly. Under full pressure, collapse the
+ * expression into its leased base and return that already-materialized value;
+ * collapse_addr_to_reg refuses to overwrite an allocator-owned base. */
+static NativeLoc emit_frame_address_result(NativeEmitCtx* e,
+ KitCgTypeId type,
+ NativeAddr* addr, SrcLoc loc) {
+ NativeAllocClass cls = class_for_type(e, type);
+ Reg base = addr_base_reg(addr);
+ Reg index = addr_index_reg(addr);
+ NativeLoc dst;
+ if (temp_available(e, cls, base, index)) {
+ dst = temp_loc_acquire(e, type, cls, base, index, loc);
+ e->target->load_addr(e->target, dst, *addr);
+ return dst;
+ }
+ collapse_addr_to_reg(e, addr, loc);
+ base = addr_base_reg(addr);
+ if (cls != NATIVE_REG_INT ||
+ !temp_reg_is_leased(e, NATIVE_REG_INT, base))
+ emit_panic(e, loc, "materialized address has no scoped integer owner");
+ return loc_reg(type, cls, base);
+}
+
+/* Materialize a scalar value while an effective address remains live. If
+ * the address consumes the whole value-temp bank, reduce it to one scoped
+ * base first; collapse releases cache-owned components that are dead after
+ * address evaluation. The value then explicitly avoids the surviving base
+ * and index, keeping the backend hook's address and data roles disjoint. */
+static NativeLoc materialize_operand_away_from_addr(
+ NativeEmitCtx* e, const OptOperand* op, NativeAddr* addr, SrcLoc loc) {
+ NativeAllocClass cls = class_for_type(e, op->type);
+ if (!temp_available(e, cls, addr_base_reg(addr), addr_index_reg(addr)))
+ collapse_addr_to_reg(e, addr, loc);
+ return materialize_operand(e, op, cls, op->type, addr_base_reg(addr),
+ addr_index_reg(addr), loc);
+}
+
/* Copy an aggregate / oversized value between two memory locations. dst and
* src must be addressable (frame/global/indirect/reg-as-pointer); used for
* IR_COPY/IR_LOAD/IR_STORE whose value type cannot move through one register.
@@ -730,16 +1598,24 @@ static void emit_call(NativeEmitCtx* e, Inst* in) {
d.flags = aux->desc.flags;
d.tail_policy = aux->desc.tail_policy;
d.inline_policy = aux->desc.inline_policy;
+ /* Materialize a frame-resident indirect callee before plan_call starts its
+ * argument phase. Real backends marshal register arguments inside
+ * plan_call; doing this afterward allowed the emitter to reuse an already
+ * populated ABI argument register for the callee. The acquired lease also
+ * keeps generic plan.args moves from selecting the callee register. */
+ if (d.callee.kind != NATIVE_LOC_REG &&
+ d.callee.kind != NATIVE_LOC_GLOBAL)
+ d.callee = materialize(e, d.callee, NATIVE_REG_INT, d.callee.type,
+ REG_NONE, REG_NONE, in->loc);
e->target->plan_call(e->target, &d, &plan);
for (u32 i = 0; i < plan.nargs; ++i)
- write_loc(e, plan.args[i].dst, plan.args[i].src, plan.args[i].mem, in->loc);
- if (plan.callee.kind != NATIVE_LOC_REG &&
- plan.callee.kind != NATIVE_LOC_GLOBAL)
- plan.callee = materialize(e, plan.callee, NATIVE_REG_INT, plan.callee.type,
- REG_NONE, REG_NONE, in->loc);
+ write_phase_loc(e, plan.args[i].dst, plan.args[i].src, plan.args[i].mem,
+ in->loc);
e->target->emit_call(e->target, &plan);
+ temp_scope_phase_barrier(e);
for (u32 i = 0; i < plan.nrets; ++i)
- write_loc(e, plan.rets[i].dst, plan.rets[i].src, plan.rets[i].mem, in->loc);
+ write_phase_loc(e, plan.rets[i].dst, plan.rets[i].src, plan.rets[i].mem,
+ in->loc);
if (result_slot && final_result.kind != NATIVE_LOC_NONE) {
KitCgTypeId rty = aux->desc.ret.type;
NativeLoc tmp = loc_frame(rty, class_for_type(e, rty), result_slot);
@@ -784,7 +1660,7 @@ static void emit_ret(NativeEmitCtx* e, Inst* in, const CGFuncDesc* fd) {
}
e->target->plan_ret(e->target, fd, values, &rets, &nrets);
for (u32 i = 0; i < nrets; ++i)
- write_loc(e, rets[i].dst, rets[i].src, rets[i].mem, in->loc);
+ write_phase_loc(e, rets[i].dst, rets[i].src, rets[i].mem, in->loc);
/* Skip the trailing branch-to-epilogue when this IR_RET is the very last
* inst emitted: func_end will place the epilogue label at mc_pos right
* after this, so the branch would jump to the next 4 bytes. The actual
@@ -793,8 +1669,8 @@ static void emit_ret(NativeEmitCtx* e, Inst* in, const CGFuncDesc* fd) {
if (!e->emitting_terminal_ret) e->target->ret(e->target);
}
-static void emit_inst(NativeEmitCtx* e, u32 block, u32 order_index, Inst* in,
- const CGFuncDesc* fd) {
+static void emit_inst_body(NativeEmitCtx* e, u32 block, u32 order_index,
+ Inst* in, const CGFuncDesc* fd) {
NativeLoc dst, a, b, src, tmp;
NativeAddr addr, addr2;
Reg dst_reg;
@@ -813,16 +1689,23 @@ static void emit_inst(NativeEmitCtx* e, u32 block, u32 order_index, Inst* in,
return;
case IR_LOAD_IMM:
dst = loc_from_operand(e, &in->opnds[0], in->loc);
- write_loc(e, dst, loc_imm(in->opnds[0].type, in->extra.imm),
- mem_for_type(e->c, in->opnds[0].type), in->loc);
+ write_operand(e, &in->opnds[0],
+ loc_imm(in->opnds[0].type, in->extra.imm),
+ mem_for_type(e->c, in->opnds[0].type), in->loc);
return;
- case IR_LOAD_CONST:
- dst = loc_from_operand(e, &in->opnds[0], in->loc);
+ case IR_LOAD_CONST: {
+ NativeLoc real = loc_from_operand(e, &in->opnds[0], in->loc);
+ dst = real;
if (dst.kind != NATIVE_LOC_REG)
- dst = materialize(e, dst, class_for_type(e, in->opnds[0].type),
- in->opnds[0].type, REG_NONE, REG_NONE, in->loc);
+ dst = temp_loc_acquire(e, in->opnds[0].type,
+ class_for_type(e, in->opnds[0].type), REG_NONE,
+ REG_NONE, in->loc);
e->target->load_const(e->target, dst, in->extra.cbytes);
+ if (real.kind != NATIVE_LOC_REG)
+ write_operand(e, &in->opnds[0], dst,
+ mem_for_type(e->c, in->opnds[0].type), in->loc);
return;
+ }
case IR_COPY:
if (type_is_aggregate_or_large(e, in->opnds[0].type)) {
emit_agg_move(e, addr_from_operand(e, &in->opnds[0], in->loc),
@@ -832,7 +1715,15 @@ static void emit_inst(NativeEmitCtx* e, u32 block, u32 order_index, Inst* in,
}
dst = loc_from_operand(e, &in->opnds[0], in->loc);
src = loc_from_operand(e, &in->opnds[1], in->loc);
- write_loc(e, dst, src, mem_for_type(e->c, in->opnds[0].type), in->loc);
+ if (in->opnds[1].kind == OPK_STACK) {
+ NativeLoc cached = frame_cache_take(
+ e, &in->opnds[1], src,
+ class_for_type(e, in->opnds[1].type), in->opnds[1].type,
+ REG_NONE, REG_NONE);
+ if (cached.kind != NATIVE_LOC_NONE) src = cached;
+ }
+ write_operand(e, &in->opnds[0], src,
+ mem_for_type(e->c, in->opnds[0].type), in->loc);
return;
case IR_LOAD:
if (type_is_aggregate_or_large(e, in->opnds[0].type)) {
@@ -847,14 +1738,9 @@ static void emit_inst(NativeEmitCtx* e, u32 block, u32 order_index, Inst* in,
if (dst.kind == NATIVE_LOC_REG) {
e->target->load(e->target, dst, addr, in->extra.mem);
} else {
- if (!scratch_available(e, class_for_type(e, in->opnds[0].type),
- addr_base_reg(&addr), addr_index_reg(&addr)))
- collapse_addr_to_reg(e, &addr, in->loc);
- tmp = scratch_loc(e, in->opnds[0].type,
- class_for_type(e, in->opnds[0].type),
- addr_base_reg(&addr), addr_index_reg(&addr), in->loc);
+ tmp = acquire_frame_load_result(e, in->opnds[0].type, &addr, in->loc);
e->target->load(e->target, tmp, addr, in->extra.mem);
- write_loc(e, dst, tmp, in->extra.mem, in->loc);
+ write_operand(e, &in->opnds[0], tmp, in->extra.mem, in->loc);
}
return;
case IR_STORE:
@@ -868,29 +1754,26 @@ static void emit_inst(NativeEmitCtx* e, u32 block, u32 order_index, Inst* in,
legalize_addr(e, &addr, in->extra.mem, in->loc);
src = loc_from_operand(e, &in->opnds[1], in->loc);
/* Storing a constant 0 from the hardware zero register avoids
- * materializing 0 into a scratch first (e.g. `strb wzr, [..]` rather than
- * `movz w9,0; strb w9, [..]`). */
+ * materializing 0 into an operand temp first (e.g. `strb wzr, [..]`
+ * rather than `movz w9,0; strb w9, [..]`). */
if (src.kind == NATIVE_LOC_IMM && src.v.imm == 0 &&
e->target->has_store_zero_reg &&
class_for_type(e, in->opnds[1].type) == NATIVE_REG_INT)
src = loc_reg(in->opnds[1].type, NATIVE_REG_INT,
e->target->store_zero_reg);
/* Source register aliases the address base/index (e.g. `*p = (T)p`).
- * Collapse the address into a scratch register: collapse_addr_to_reg
- * selects a scratch distinct from both base and index — hence distinct
- * from `src` — so the store reads `src` and writes through the fresh
- * scratch with no alias. This stays entirely in registers; the frame is
- * fully planned before emission, so emit never allocates a slot here. */
+ * Collapse the address into an instruction-local temp:
+ * collapse_addr_to_reg selects one distinct from both base and index —
+ * hence distinct from `src` — so the store reads `src` and writes through
+ * the fresh temp with no alias. This stays entirely in registers; the
+ * frame is fully planned before emission, so emit never allocates a slot
+ * here. */
if (src.kind == NATIVE_LOC_REG && (src.v.reg == addr_base_reg(&addr) ||
src.v.reg == addr_index_reg(&addr)))
collapse_addr_to_reg(e, &addr, in->loc);
if (src.kind != NATIVE_LOC_REG) {
- if (!scratch_available(e, class_for_type(e, in->opnds[1].type),
- addr_base_reg(&addr), addr_index_reg(&addr)))
- collapse_addr_to_reg(e, &addr, in->loc);
- src = materialize(e, src, class_for_type(e, in->opnds[1].type),
- in->opnds[1].type, addr_base_reg(&addr),
- addr_index_reg(&addr), in->loc);
+ src = materialize_operand_away_from_addr(e, &in->opnds[1], &addr,
+ in->loc);
}
e->target->store(e->target, addr, src, in->extra.mem);
return;
@@ -898,22 +1781,29 @@ static void emit_inst(NativeEmitCtx* e, u32 block, u32 order_index, Inst* in,
NativeLoc real = loc_from_operand(e, &in->opnds[0], in->loc);
addr = addr_from_operand(e, &in->opnds[1], in->loc);
dst = real;
- if (dst.kind != NATIVE_LOC_REG)
- dst = scratch_loc(e, in->opnds[0].type,
- class_for_type(e, in->opnds[0].type), REG_NONE,
- REG_NONE, in->loc);
- e->target->load_addr(e->target, dst, addr);
- if (real.kind != NATIVE_LOC_REG)
- write_loc(e, real, dst, mem_for_type(e->c, in->opnds[0].type), in->loc);
+ if (dst.kind != NATIVE_LOC_REG ||
+ (addr.index_kind == NATIVE_ADDR_INDEX_REG &&
+ dst.v.reg == addr.index.reg)) {
+ dst = emit_frame_address_result(e, in->opnds[0].type, &addr, in->loc);
+ } else {
+ e->target->load_addr(e->target, dst, addr);
+ }
+ if (!temp_loc_equal(real, dst))
+ write_operand(e, &in->opnds[0], dst,
+ mem_for_type(e->c, in->opnds[0].type), in->loc);
return;
}
case IR_TLS_ADDR_OF: {
IRTlsAux* aux = (IRTlsAux*)in->extra.aux;
- dst = loc_from_operand(e, &in->opnds[0], in->loc);
+ NativeLoc real = loc_from_operand(e, &in->opnds[0], in->loc);
+ dst = real;
if (dst.kind != NATIVE_LOC_REG)
- dst = materialize(e, dst, NATIVE_REG_INT, in->opnds[0].type, REG_NONE,
+ dst = temp_loc_acquire(e, in->opnds[0].type, NATIVE_REG_INT, REG_NONE,
REG_NONE, in->loc);
e->target->tls_addr_of(e->target, dst, aux->sym, aux->addend);
+ if (real.kind != NATIVE_LOC_REG)
+ write_operand(e, &in->opnds[0], dst,
+ mem_for_type(e->c, in->opnds[0].type), in->loc);
return;
}
case IR_AGG_COPY: {
@@ -924,6 +1814,8 @@ static void emit_inst(NativeEmitCtx* e, u32 block, u32 order_index, Inst* in,
e, &in->opnds[1], in->loc,
addr.base_kind == NATIVE_ADDR_BASE_REG ? addr.base.reg : REG_NONE,
REG_NONE);
+ legalize_addr(e, &addr, aux->access.mem, in->loc);
+ legalize_addr(e, &addr2, aux->access.mem, in->loc);
e->target->copy_bytes(e->target, addr, addr2, aux->access);
return;
}
@@ -931,46 +1823,46 @@ static void emit_inst(NativeEmitCtx* e, u32 block, u32 order_index, Inst* in,
IRAggAux* aux = (IRAggAux*)in->extra.aux;
addr = pointer_addr_from_operand(e, &in->opnds[0], in->loc, REG_NONE,
REG_NONE);
+ legalize_addr(e, &addr, aux->access.mem, in->loc);
src = loc_from_operand(e, &in->opnds[1], in->loc);
if (src.kind != NATIVE_LOC_REG) {
- if (!scratch_available(e, NATIVE_REG_INT, addr_base_reg(&addr),
- addr_index_reg(&addr)))
- collapse_addr_to_reg(e, &addr, in->loc);
- src = materialize(e, src, NATIVE_REG_INT, in->opnds[1].type,
- addr_base_reg(&addr), addr_index_reg(&addr), in->loc);
+ src = materialize_operand_away_from_addr(e, &in->opnds[1], &addr,
+ in->loc);
}
e->target->set_bytes(e->target, addr, src, aux->access);
return;
}
case IR_BITFIELD_LOAD: {
IRBitFieldAux* aux = (IRBitFieldAux*)in->extra.aux;
- dst = loc_from_operand(e, &in->opnds[0], in->loc);
+ NativeLoc real = loc_from_operand(e, &in->opnds[0], in->loc);
+ dst = real;
addr = addr_from_operand(e, &in->opnds[1], in->loc);
+ legalize_addr(e, &addr, aux->access.storage, in->loc);
if (dst.kind != NATIVE_LOC_REG)
- dst = materialize(e, dst, class_for_type(e, in->opnds[0].type),
- in->opnds[0].type, REG_NONE, REG_NONE, in->loc);
+ dst = acquire_frame_load_result(e, in->opnds[0].type, &addr,
+ in->loc);
e->target->bitfield_load(e->target, dst, addr, aux->access);
+ if (real.kind != NATIVE_LOC_REG)
+ write_operand(e, &in->opnds[0], dst,
+ mem_for_type(e->c, in->opnds[0].type), in->loc);
return;
}
case IR_BITFIELD_STORE: {
IRBitFieldAux* aux = (IRBitFieldAux*)in->extra.aux;
addr = addr_from_operand(e, &in->opnds[0], in->loc);
+ legalize_addr(e, &addr, aux->access.storage, in->loc);
src = loc_from_operand(e, &in->opnds[1], in->loc);
/* The field store is a read-modify-write that keeps the address live
* across the value placement, so the value must not be materialized into
* the address's base/index register. Mirror IR_STORE/IR_AGG_SET: when no
- * other scratch is free, collapse the address to a single scratch first,
+ * other operand temp is free, collapse the address to one temp first,
* then materialize the value avoiding the address regs. Without this, a
- * spilled pointer reloaded into scratch[0] and a value reusing scratch[0]
- * collide — the value `mov`s over the address, nulling it (SIGSEGV in
- * optimized SQLite). */
+ * spilled pointer reloaded into the first temp and a value reusing that
+ * temp collide — the value `mov`s over the address, nulling it (SIGSEGV
+ * in optimized SQLite). */
if (src.kind != NATIVE_LOC_REG) {
- if (!scratch_available(e, class_for_type(e, in->opnds[1].type),
- addr_base_reg(&addr), addr_index_reg(&addr)))
- collapse_addr_to_reg(e, &addr, in->loc);
- src = materialize(e, src, class_for_type(e, in->opnds[1].type),
- in->opnds[1].type, addr_base_reg(&addr),
- addr_index_reg(&addr), in->loc);
+ src = materialize_operand_away_from_addr(e, &in->opnds[1], &addr,
+ in->loc);
}
e->target->bitfield_store(e->target, addr, src, aux->access);
return;
@@ -979,9 +1871,10 @@ static void emit_inst(NativeEmitCtx* e, u32 block, u32 order_index, Inst* in,
dst = loc_from_operand(e, &in->opnds[0], in->loc);
dst_reg = dst.kind == NATIVE_LOC_REG ? dst.v.reg : REG_NONE;
b = loc_from_operand(e, &in->opnds[2], in->loc);
- a = materialize(e, loc_from_operand(e, &in->opnds[1], in->loc),
- class_for_type(e, in->opnds[1].type), in->opnds[1].type,
- dst_reg, loc_avoid_reg(b), in->loc);
+ a = materialize_operand(e, &in->opnds[1],
+ class_for_type(e, in->opnds[1].type),
+ in->opnds[1].type, dst_reg, loc_avoid_reg(b),
+ in->loc);
b = operand_imm_or_reg(e, &in->opnds[2], NATIVE_IMM_BINOP,
(u32)in->extra.imm, a.v.reg, dst_reg, in->loc);
/* L7 shift rider: a single-use SHL folded into this binop's second
@@ -990,62 +1883,85 @@ static void emit_inst(NativeEmitCtx* e, u32 block, u32 order_index, Inst* in,
* pass only stamps it for targets advertising can_fold_shift_into_alu,
* so a backend ignoring NativeLoc.shift never receives a nonzero rider. */
if (b.kind == NATIVE_LOC_REG) b.shift = in->opnds[2].shift;
- if (dst.kind != NATIVE_LOC_REG)
- dst = scratch_loc(e, in->opnds[0].type,
- class_for_type(e, in->opnds[0].type), a.v.reg,
- loc_avoid_reg(b), in->loc);
+ if (dst.kind != NATIVE_LOC_REG) {
+ NativeAllocClass dst_cls = class_for_type(e, in->opnds[0].type);
+ if (a.kind == NATIVE_LOC_REG && a.cls == (u8)dst_cls &&
+ temp_reg_is_leased(e, dst_cls, a.v.reg))
+ dst = loc_reg(in->opnds[0].type, dst_cls, a.v.reg);
+ else
+ dst = temp_loc_acquire(e, in->opnds[0].type, dst_cls, a.v.reg,
+ loc_avoid_reg(b), in->loc);
+ }
e->target->binop(e->target, (BinOp)in->extra.imm, dst, a, b);
if (in->opnds[0].kind != OPK_REG)
- write_loc(e, loc_from_operand(e, &in->opnds[0], in->loc), dst,
- mem_for_type(e->c, in->opnds[0].type), in->loc);
+ write_operand(e, &in->opnds[0], dst,
+ mem_for_type(e->c, in->opnds[0].type), in->loc);
return;
case IR_UNOP:
dst = loc_from_operand(e, &in->opnds[0], in->loc);
dst_reg = dst.kind == NATIVE_LOC_REG ? dst.v.reg : REG_NONE;
- a = materialize(e, loc_from_operand(e, &in->opnds[1], in->loc),
- class_for_type(e, in->opnds[1].type), in->opnds[1].type,
- dst_reg, REG_NONE, in->loc);
- if (dst.kind != NATIVE_LOC_REG)
- dst = scratch_loc(e, in->opnds[0].type,
- class_for_type(e, in->opnds[0].type), a.v.reg,
- REG_NONE, in->loc);
+ a = materialize_operand(e, &in->opnds[1],
+ class_for_type(e, in->opnds[1].type),
+ in->opnds[1].type, dst_reg, REG_NONE, in->loc);
+ if (dst.kind != NATIVE_LOC_REG) {
+ NativeAllocClass dst_cls = class_for_type(e, in->opnds[0].type);
+ if (a.kind == NATIVE_LOC_REG && a.cls == (u8)dst_cls &&
+ temp_reg_is_leased(e, dst_cls, a.v.reg))
+ dst = loc_reg(in->opnds[0].type, dst_cls, a.v.reg);
+ else
+ dst = temp_loc_acquire(e, in->opnds[0].type, dst_cls, a.v.reg,
+ REG_NONE,
+ in->loc);
+ }
e->target->unop(e->target, (UnOp)in->extra.imm, dst, a);
if (in->opnds[0].kind != OPK_REG)
- write_loc(e, loc_from_operand(e, &in->opnds[0], in->loc), dst,
- mem_for_type(e->c, in->opnds[0].type), in->loc);
+ write_operand(e, &in->opnds[0], dst,
+ mem_for_type(e->c, in->opnds[0].type), in->loc);
return;
case IR_CMP:
dst = loc_from_operand(e, &in->opnds[0], in->loc);
dst_reg = dst.kind == NATIVE_LOC_REG ? dst.v.reg : REG_NONE;
b = loc_from_operand(e, &in->opnds[2], in->loc);
- a = materialize(e, loc_from_operand(e, &in->opnds[1], in->loc),
- class_for_type(e, in->opnds[1].type), in->opnds[1].type,
- dst_reg, loc_avoid_reg(b), in->loc);
+ a = materialize_operand(e, &in->opnds[1],
+ class_for_type(e, in->opnds[1].type),
+ in->opnds[1].type, dst_reg, loc_avoid_reg(b),
+ in->loc);
b = operand_imm_or_reg(e, &in->opnds[2], NATIVE_IMM_CMP,
(u32)in->extra.imm, a.v.reg, dst_reg, in->loc);
- if (dst.kind != NATIVE_LOC_REG)
- dst = scratch_loc(e, in->opnds[0].type,
- class_for_type(e, in->opnds[0].type), a.v.reg,
- loc_avoid_reg(b), in->loc);
+ if (dst.kind != NATIVE_LOC_REG) {
+ NativeAllocClass dst_cls = class_for_type(e, in->opnds[0].type);
+ if (a.kind == NATIVE_LOC_REG && a.cls == (u8)dst_cls &&
+ temp_reg_is_leased(e, dst_cls, a.v.reg))
+ dst = loc_reg(in->opnds[0].type, dst_cls, a.v.reg);
+ else
+ dst = temp_loc_acquire(e, in->opnds[0].type, dst_cls, a.v.reg,
+ loc_avoid_reg(b), in->loc);
+ }
e->target->cmp(e->target, (CmpOp)in->extra.imm, dst, a, b);
if (in->opnds[0].kind != OPK_REG)
- write_loc(e, loc_from_operand(e, &in->opnds[0], in->loc), dst,
- mem_for_type(e->c, in->opnds[0].type), in->loc);
+ write_operand(e, &in->opnds[0], dst,
+ mem_for_type(e->c, in->opnds[0].type), in->loc);
return;
case IR_CONVERT:
dst = loc_from_operand(e, &in->opnds[0], in->loc);
dst_reg = dst.kind == NATIVE_LOC_REG ? dst.v.reg : REG_NONE;
- src = materialize(e, loc_from_operand(e, &in->opnds[1], in->loc),
- class_for_type(e, in->opnds[1].type), in->opnds[1].type,
- dst_reg, REG_NONE, in->loc);
- if (dst.kind != NATIVE_LOC_REG)
- dst = scratch_loc(e, in->opnds[0].type,
- class_for_type(e, in->opnds[0].type), src.v.reg,
- REG_NONE, in->loc);
+ src = materialize_operand(e, &in->opnds[1],
+ class_for_type(e, in->opnds[1].type),
+ in->opnds[1].type, dst_reg, REG_NONE, in->loc);
+ if (dst.kind != NATIVE_LOC_REG) {
+ NativeAllocClass dst_cls = class_for_type(e, in->opnds[0].type);
+ if (src.kind == NATIVE_LOC_REG && src.cls == (u8)dst_cls &&
+ temp_reg_is_leased(e, dst_cls, src.v.reg))
+ dst = loc_reg(in->opnds[0].type, dst_cls, src.v.reg);
+ else
+ dst = temp_loc_acquire(e, in->opnds[0].type, dst_cls, src.v.reg,
+ REG_NONE,
+ in->loc);
+ }
e->target->convert(e->target, (ConvKind)in->extra.imm, dst, src);
if (in->opnds[0].kind != OPK_REG)
- write_loc(e, loc_from_operand(e, &in->opnds[0], in->loc), dst,
- mem_for_type(e->c, in->opnds[0].type), in->loc);
+ write_operand(e, &in->opnds[0], dst,
+ mem_for_type(e->c, in->opnds[0].type), in->loc);
return;
case IR_CALL:
emit_call(e, in);
@@ -1059,9 +1975,10 @@ static void emit_inst(NativeEmitCtx* e, u32 block, u32 order_index, Inst* in,
? e->f->emit_order[order_index + 1u]
: UINT32_MAX;
b = loc_from_operand(e, &in->opnds[1], in->loc);
- a = materialize(e, loc_from_operand(e, &in->opnds[0], in->loc),
- class_for_type(e, in->opnds[0].type), in->opnds[0].type,
- REG_NONE, loc_avoid_reg(b), in->loc);
+ a = materialize_operand(e, &in->opnds[0],
+ class_for_type(e, in->opnds[0].type),
+ in->opnds[0].type, REG_NONE, loc_avoid_reg(b),
+ in->loc);
b = operand_imm_or_reg(e, &in->opnds[1], NATIVE_IMM_CMP,
(u32)in->extra.imm, a.v.reg, REG_NONE, in->loc);
e->target->cmp_branch(
@@ -1074,20 +1991,20 @@ static void emit_inst(NativeEmitCtx* e, u32 block, u32 order_index, Inst* in,
}
case IR_SWITCH: {
IRSwitchAux* aux = (IRSwitchAux*)in->extra.aux;
- /* Pin the selector in a register ONCE before the chain. */
- NativeLoc sel =
- materialize(e, loc_from_operand(e, &in->opnds[0], in->loc),
- class_for_type(e, in->opnds[0].type), in->opnds[0].type,
- REG_NONE, REG_NONE, in->loc);
+ /* Lease the selector in one register for the whole chain. */
+ NativeLoc sel = materialize_operand(
+ e, &in->opnds[0], class_for_type(e, in->opnds[0].type),
+ in->opnds[0].type, REG_NONE, REG_NONE, in->loc);
/* Each case value is interpreted at the selector type (matches the CG
* fallback chain in cg/native_direct_target.c). Route each through the
* same NATIVE_IMM_CMP immediate-or-register logic IR_CMP_BRANCH uses, so
* a legal compare immediate becomes `cmp sel, #k` with no per-case
* materialization; an illegal immediate (e.g. too large for the target's
- * compare encoding) falls back to load_imm into a scratch that avoids the
- * pinned selector register. */
+ * compare encoding) falls back to load_imm into a temp that avoids the
+ * leased selector register. */
KitCgTypeId case_type =
aux && aux->selector_type ? aux->selector_type : in->opnds[0].type;
+ NativeEmitTempMark case_mark = temp_scope_mark(e);
for (u32 i = 0; aux && i < aux->ncases; ++i) {
OptOperand case_op;
NativeLoc imm;
@@ -1099,6 +2016,7 @@ static void emit_inst(NativeEmitCtx* e, u32 block, u32 order_index, Inst* in,
sel.v.reg, REG_NONE, in->loc);
e->target->cmp_branch(e->target, CMP_EQ, sel, imm,
ensure_label(e, aux->cases[i].block, in->loc));
+ temp_scope_rewind(e, case_mark);
}
if (aux)
e->target->jump(e->target,
@@ -1112,21 +2030,26 @@ static void emit_inst(NativeEmitCtx* e, u32 block, u32 order_index, Inst* in,
: NULL;
for (u32 i = 0; aux && i < aux->ntargets; ++i)
labels[i] = ensure_label(e, aux->targets[i], in->loc);
- src = materialize(e, loc_from_operand(e, &in->opnds[0], in->loc),
- NATIVE_REG_INT, in->opnds[0].type, REG_NONE, REG_NONE,
- in->loc);
+ src = materialize_operand(e, &in->opnds[0], NATIVE_REG_INT,
+ in->opnds[0].type, REG_NONE, REG_NONE,
+ in->loc);
e->target->indirect_branch(e->target, src, labels,
aux ? aux->ntargets : 0u);
return;
}
- case IR_LOAD_LABEL_ADDR:
- dst = loc_from_operand(e, &in->opnds[0], in->loc);
+ case IR_LOAD_LABEL_ADDR: {
+ NativeLoc real = loc_from_operand(e, &in->opnds[0], in->loc);
+ dst = real;
if (dst.kind != NATIVE_LOC_REG)
- dst = materialize(e, dst, NATIVE_REG_INT, in->opnds[0].type, REG_NONE,
+ dst = temp_loc_acquire(e, in->opnds[0].type, NATIVE_REG_INT, REG_NONE,
REG_NONE, in->loc);
e->target->load_label_addr(e->target, dst,
ensure_label(e, (u32)in->extra.imm, in->loc));
+ if (real.kind != NATIVE_LOC_REG)
+ write_operand(e, &in->opnds[0], dst,
+ mem_for_type(e->c, in->opnds[0].type), in->loc);
return;
+ }
case IR_LOCAL_STATIC_DATA_BEGIN: {
CgIrLocalStaticBeginAux* aux = (CgIrLocalStaticBeginAux*)in->extra.aux;
emit_local_static_begin(e, aux ? &aux->desc : NULL, in->loc);
@@ -1154,38 +2077,44 @@ static void emit_inst(NativeEmitCtx* e, u32 block, u32 order_index, Inst* in,
case IR_RET:
emit_ret(e, in, fd);
return;
- case IR_ALLOCA:
- dst = loc_from_operand(e, &in->opnds[0], in->loc);
- src = materialize(e, loc_from_operand(e, &in->opnds[1], in->loc),
- NATIVE_REG_INT, in->opnds[1].type, REG_NONE, REG_NONE,
- in->loc);
+ case IR_ALLOCA: {
+ NativeLoc real = loc_from_operand(e, &in->opnds[0], in->loc);
+ dst = real;
+ src = materialize_operand(e, &in->opnds[1], NATIVE_REG_INT,
+ in->opnds[1].type, REG_NONE, REG_NONE,
+ in->loc);
if (dst.kind != NATIVE_LOC_REG)
- dst = scratch_loc(e, in->opnds[0].type, NATIVE_REG_INT, src.v.reg,
+ dst = temp_loc_acquire(e, in->opnds[0].type, NATIVE_REG_INT, src.v.reg,
REG_NONE, in->loc);
e->target->alloca_(e->target, dst, src, (u32)in->extra.imm);
+ if (real.kind != NATIVE_LOC_REG)
+ write_operand(e, &in->opnds[0], dst,
+ mem_for_type(e->c, in->opnds[0].type), in->loc);
return;
+ }
case IR_ATOMIC_LOAD: {
IRAtomicAux* aux = (IRAtomicAux*)in->extra.aux;
dst = loc_from_operand(e, &in->opnds[0], in->loc);
addr = pointer_addr_from_operand(e, &in->opnds[1], in->loc, REG_NONE,
REG_NONE);
+ legalize_addr(e, &addr, aux->mem, in->loc);
if (dst.kind != NATIVE_LOC_REG)
- dst = scratch_loc(e, in->opnds[0].type,
- class_for_type(e, in->opnds[0].type), REG_NONE,
- REG_NONE, in->loc);
+ dst = acquire_frame_load_result(e, in->opnds[0].type, &addr,
+ in->loc);
e->target->atomic_load(e->target, dst, addr, aux->mem, aux->mo);
if (in->opnds[0].kind != OPK_REG)
- write_loc(e, loc_from_operand(e, &in->opnds[0], in->loc), dst, aux->mem,
- in->loc);
+ write_operand(e, &in->opnds[0], dst, aux->mem, in->loc);
return;
}
case IR_ATOMIC_STORE: {
IRAtomicAux* aux = (IRAtomicAux*)in->extra.aux;
addr = pointer_addr_from_operand(e, &in->opnds[0], in->loc, REG_NONE,
REG_NONE);
- src = materialize(e, loc_from_operand(e, &in->opnds[1], in->loc),
- class_for_type(e, in->opnds[1].type), in->opnds[1].type,
- REG_NONE, REG_NONE, in->loc);
+ legalize_addr(e, &addr, aux->mem, in->loc);
+ src = loc_from_operand(e, &in->opnds[1], in->loc);
+ if (src.kind != NATIVE_LOC_REG)
+ src = materialize_operand_away_from_addr(e, &in->opnds[1], &addr,
+ in->loc);
e->target->atomic_store(e->target, addr, src, aux->mem, aux->mo);
return;
}
@@ -1194,18 +2123,18 @@ static void emit_inst(NativeEmitCtx* e, u32 block, u32 order_index, Inst* in,
dst = loc_from_operand(e, &in->opnds[0], in->loc);
addr = pointer_addr_from_operand(e, &in->opnds[1], in->loc, REG_NONE,
REG_NONE);
- src = materialize(e, loc_from_operand(e, &in->opnds[2], in->loc),
- class_for_type(e, in->opnds[2].type), in->opnds[2].type,
- REG_NONE, REG_NONE, in->loc);
+ legalize_addr(e, &addr, aux->mem, in->loc);
+ src = materialize_operand(e, &in->opnds[2],
+ class_for_type(e, in->opnds[2].type),
+ in->opnds[2].type, REG_NONE, REG_NONE, in->loc);
if (dst.kind != NATIVE_LOC_REG)
- dst = scratch_loc(e, in->opnds[0].type,
+ dst = temp_loc_acquire(e, in->opnds[0].type,
class_for_type(e, in->opnds[0].type), src.v.reg,
REG_NONE, in->loc);
e->target->atomic_rmw(e->target, (KitCgAtomicOp)aux->op, dst, addr, src,
aux->mem, aux->mo);
if (in->opnds[0].kind != OPK_REG)
- write_loc(e, loc_from_operand(e, &in->opnds[0], in->loc), dst, aux->mem,
- in->loc);
+ write_operand(e, &in->opnds[0], dst, aux->mem, in->loc);
return;
}
case IR_ATOMIC_CAS: {
@@ -1217,77 +2146,76 @@ static void emit_inst(NativeEmitCtx* e, u32 block, u32 order_index, Inst* in,
ok = loc_from_operand(e, &in->opnds[1], in->loc);
addr = pointer_addr_from_operand(e, &in->opnds[2], in->loc, REG_NONE,
REG_NONE);
- expected = materialize(e, loc_from_operand(e, &in->opnds[3], in->loc),
- class_for_type(e, in->opnds[3].type),
- in->opnds[3].type, REG_NONE, REG_NONE, in->loc);
- desired =
- materialize(e, loc_from_operand(e, &in->opnds[4], in->loc),
- class_for_type(e, in->opnds[4].type), in->opnds[4].type,
- expected.v.reg, REG_NONE, in->loc);
+ legalize_addr(e, &addr, aux->mem, in->loc);
+ expected = materialize_operand(
+ e, &in->opnds[3], class_for_type(e, in->opnds[3].type),
+ in->opnds[3].type, REG_NONE, REG_NONE, in->loc);
+ desired = materialize_operand(
+ e, &in->opnds[4], class_for_type(e, in->opnds[4].type),
+ in->opnds[4].type, expected.v.reg, REG_NONE, in->loc);
if (dst.kind != NATIVE_LOC_REG)
- dst = scratch_loc(e, in->opnds[0].type,
+ dst = temp_loc_acquire(e, in->opnds[0].type,
class_for_type(e, in->opnds[0].type), expected.v.reg,
desired.v.reg, in->loc);
if (ok.kind != NATIVE_LOC_REG)
- ok = scratch_loc(e, in->opnds[1].type,
+ ok = temp_loc_acquire(e, in->opnds[1].type,
class_for_type(e, in->opnds[1].type), dst.v.reg,
expected.v.reg, in->loc);
e->target->atomic_cas(e->target, dst, ok, addr, expected, desired,
aux->mem, aux->success, aux->failure);
if (in->opnds[0].kind != OPK_REG)
- write_loc(e, loc_from_operand(e, &in->opnds[0], in->loc), dst, aux->mem,
- in->loc);
+ write_operand(e, &in->opnds[0], dst, aux->mem, in->loc);
if (in->opnds[1].kind != OPK_REG)
- write_loc(e, loc_from_operand(e, &in->opnds[1], in->loc), ok,
- mem_for_type(e->c, in->opnds[1].type), in->loc);
+ write_operand(e, &in->opnds[1], ok,
+ mem_for_type(e->c, in->opnds[1].type), in->loc);
return;
}
case IR_VA_START: {
- NativeLoc ap = materialize(e, loc_from_operand(e, &in->opnds[0], in->loc),
- NATIVE_REG_INT, in->opnds[0].type, REG_NONE,
- REG_NONE, in->loc);
+ NativeLoc ap = materialize_operand(e, &in->opnds[0], NATIVE_REG_INT,
+ in->opnds[0].type, REG_NONE, REG_NONE,
+ in->loc);
e->target->va_start_(e->target, ap);
return;
}
case IR_VA_END: {
- NativeLoc ap = materialize(e, loc_from_operand(e, &in->opnds[0], in->loc),
- NATIVE_REG_INT, in->opnds[0].type, REG_NONE,
- REG_NONE, in->loc);
+ NativeLoc ap = materialize_operand(e, &in->opnds[0], NATIVE_REG_INT,
+ in->opnds[0].type, REG_NONE, REG_NONE,
+ in->loc);
e->target->va_end_(e->target, ap);
return;
}
case IR_VA_COPY: {
- NativeLoc d = materialize(e, loc_from_operand(e, &in->opnds[0], in->loc),
- NATIVE_REG_INT, in->opnds[0].type, REG_NONE,
- REG_NONE, in->loc);
- NativeLoc s = materialize(e, loc_from_operand(e, &in->opnds[1], in->loc),
- NATIVE_REG_INT, in->opnds[1].type, d.v.reg,
- REG_NONE, in->loc);
+ NativeLoc d = materialize_operand(e, &in->opnds[0], NATIVE_REG_INT,
+ in->opnds[0].type, REG_NONE, REG_NONE,
+ in->loc);
+ NativeLoc s = materialize_operand(e, &in->opnds[1], NATIVE_REG_INT,
+ in->opnds[1].type, d.v.reg, REG_NONE,
+ in->loc);
e->target->va_copy_(e->target, d, s);
return;
}
case IR_VA_ARG: {
KitCgTypeId ty = in->opnds[0].type;
- NativeLoc ap = materialize(e, loc_from_operand(e, &in->opnds[1], in->loc),
- NATIVE_REG_INT, in->opnds[1].type, REG_NONE,
- REG_NONE, in->loc);
+ NativeLoc ap = materialize_operand(e, &in->opnds[1], NATIVE_REG_INT,
+ in->opnds[1].type, REG_NONE, REG_NONE,
+ in->loc);
NativeLoc res;
if (type_is_aggregate_or_large(e, ty)) {
/* A value too wide for one register (an 8-byte i64/double on a 32-bit
- * target, or an aggregate) can't pass through a scratch register; hand
- * the target its memory destination so it can copy the value directly.
- */
+ * target, or an aggregate) cannot pass through one instruction-local
+ * register; hand the target its memory destination so it can copy the
+ * value directly. */
e->target->va_arg_(e->target,
loc_from_operand(e, &in->opnds[0], in->loc), ap, ty);
return;
}
/* The result must land in a register distinct from the va_list pointer;
- * fetch into a scratch register, then write to the real destination. */
- res = scratch_loc(e, ty, class_for_type(e, ty), ap.v.reg, REG_NONE,
+ * fetch into an instruction-local temp, then write to the real
+ * destination. */
+ res = temp_loc_acquire(e, ty, class_for_type(e, ty), ap.v.reg, REG_NONE,
in->loc);
e->target->va_arg_(e->target, res, ap, ty);
- write_loc(e, loc_from_operand(e, &in->opnds[0], in->loc), res,
- mem_for_type(e->c, ty), in->loc);
+ write_operand(e, &in->opnds[0], res, mem_for_type(e->c, ty), in->loc);
return;
}
case IR_ASM_BLOCK: {
@@ -1295,23 +2223,61 @@ static void emit_inst(NativeEmitCtx* e, u32 block, u32 order_index, Inst* in,
NativeLoc* out_locs = aux && aux->nout
? arena_array(e->f->arena, NativeLoc, aux->nout)
: NULL;
+ NativeLoc* real_out_locs =
+ aux && aux->nout
+ ? arena_array(e->f->arena, NativeLoc, aux->nout)
+ : NULL;
NativeLoc* in_locs = aux && aux->nin
? arena_array(e->f->arena, NativeLoc, aux->nin)
: NULL;
- /* The optimizer has already allocated registers for the asm operands and
- * placed the input values / consumes the output values through the normal
- * use/def data flow. We only convert each operand to its NativeLoc; the
- * NativeTarget hook binds the pre-allocated registers to the template and
- * saves/restores any callee-saved registers the asm clobbers. */
- for (u32 i = 0; aux && i < aux->nout; ++i)
- out_locs[i] = loc_from_operand(e, &aux->out_ops[i], in->loc);
- for (u32 i = 0; aux && i < aux->nin; ++i)
- in_locs[i] = loc_from_operand(e, &aux->in_ops[i], in->loc);
+ int* staged_outs =
+ aux && aux->nout ? arena_zarray(e->f->arena, int, aux->nout) : NULL;
+ if (e->temps)
+ e->temps->allow_asm_temps =
+ (u8)!(aux && aux->has_memory_constraint);
+ /* Register constraints are an instruction-local placement requirement,
+ * not a property of the value's whole live range. Keep regalloc's chosen
+ * hard register when it satisfies the constraint; otherwise stage the
+ * value through a scoped lease. The backend asm binder validates and
+ * binds concrete register operands; only backend-private materialization
+ * for memory constraints remains on its side of the boundary. */
+ for (u32 i = 0; aux && i < aux->nout; ++i) {
+ real_out_locs[i] =
+ loc_from_operand(e, &aux->out_ops[i], in->loc);
+ out_locs[i] = asm_stage_reg_operand(
+ e, &aux->outs[i],
+ aux->out_reg_reqs ? &aux->out_reg_reqs[i] : NULL,
+ real_out_locs[i],
+ aux->outs[i].dir == KIT_CG_ASM_INOUT, &staged_outs[i], in->loc);
+ }
+ for (u32 i = 0; aux && i < aux->nin; ++i) {
+ NativeLoc real = loc_from_operand(e, &aux->in_ops[i], in->loc);
+ int matched = native_asm_match_index(aux->ins[i].str);
+ if (matched >= 0) {
+ if ((u32)matched >= aux->nout ||
+ out_locs[matched].kind != NATIVE_LOC_REG)
+ emit_panic(e, in->loc, "invalid matching asm operand");
+ if (native_asm_constraint_early(aux->outs[matched].str))
+ emit_panic(e, in->loc,
+ "matching input names early-clobber output");
+ load_loc_into_reg(e, out_locs[matched], real, in->loc);
+ in_locs[i] = out_locs[matched];
+ } else {
+ in_locs[i] = asm_stage_reg_operand(
+ e, &aux->ins[i],
+ aux->in_reg_reqs ? &aux->in_reg_reqs[i] : NULL, real, 1, NULL,
+ in->loc);
+ }
+ }
e->target->asm_block(e->target, aux ? aux->tmpl : "",
aux ? aux->outs : NULL, aux ? aux->nout : 0,
out_locs, aux ? aux->ins : NULL, aux ? aux->nin : 0,
in_locs, aux ? aux->clobbers : NULL,
aux ? aux->nclob : 0);
+ for (u32 i = 0; aux && i < aux->nout; ++i)
+ if (staged_outs[i])
+ write_operand(e, &aux->out_ops[i], out_locs[i],
+ mem_for_type(e->c, out_locs[i].type), in->loc);
return;
}
case IR_BREAK_TO:
@@ -1328,22 +2294,41 @@ static void emit_inst(NativeEmitCtx* e, u32 block, u32 order_index, Inst* in,
NativeLoc* dsts = aux && aux->ndst
? arena_array(e->f->arena, NativeLoc, aux->ndst)
: NULL;
+ NativeLoc* real_dsts =
+ aux && aux->ndst
+ ? arena_array(e->f->arena, NativeLoc, aux->ndst)
+ : NULL;
NativeLoc* args = aux && aux->narg
? arena_array(e->f->arena, NativeLoc, aux->narg)
: NULL;
- for (u32 i = 0; aux && i < aux->ndst; ++i)
- dsts[i] = loc_from_operand(e, &aux->dsts[i], in->loc);
for (u32 i = 0; aux && i < aux->narg; ++i) {
if (aux->args[i].kind == OPK_IMM) {
args[i] = loc_from_operand(e, &aux->args[i], in->loc);
} else {
- args[i] = materialize(e, loc_from_operand(e, &aux->args[i], in->loc),
- class_for_type(e, aux->args[i].type),
- aux->args[i].type, REG_NONE, REG_NONE, in->loc);
+ args[i] = materialize_operand(
+ e, &aux->args[i], class_for_type(e, aux->args[i].type),
+ aux->args[i].type, REG_NONE, REG_NONE, in->loc);
}
}
+ for (u32 i = 0; aux && i < aux->ndst; ++i) {
+ NativeAllocClass cls = class_for_type(e, aux->dsts[i].type);
+ real_dsts[i] = loc_from_operand(e, &aux->dsts[i], in->loc);
+ dsts[i] = real_dsts[i];
+ if (dsts[i].kind != NATIVE_LOC_NONE &&
+ dsts[i].kind != NATIVE_LOC_REG)
+ dsts[i] = temp_loc_acquire(e, aux->dsts[i].type, cls, REG_NONE,
+ REG_NONE,
+ in->loc);
+ }
e->target->intrinsic(e->target, aux->kind, dsts, aux->ndst, args,
aux->narg);
+ for (u32 i = 0; aux && i < aux->ndst; ++i) {
+ if (real_dsts[i].kind == NATIVE_LOC_NONE ||
+ real_dsts[i].kind == NATIVE_LOC_REG)
+ continue;
+ write_operand(e, &aux->dsts[i], dsts[i],
+ mem_for_type(e->c, aux->dsts[i].type), in->loc);
+ }
return;
}
default:
@@ -1351,6 +2336,14 @@ static void emit_inst(NativeEmitCtx* e, u32 block, u32 order_index, Inst* in,
}
}
+static void emit_inst(NativeEmitCtx* e, u32 block, u32 order_index, Inst* in,
+ const CGFuncDesc* fd) {
+ NativeEmitTempScope scope;
+ temp_scope_begin(e, &scope, in);
+ emit_inst_body(e, block, order_index, in, fd);
+ temp_scope_end(e, &scope);
+}
+
static int native_emit_terminates(const Inst* in) {
if (!in) return 0;
switch ((IROp)in->op) {
@@ -1376,6 +2369,10 @@ static int native_emit_terminates(const Inst* in) {
static void emit_block(NativeEmitCtx* e, u32 block, u32 order_index,
const CGFuncDesc* fd) {
if (block >= e->f->nblocks) return;
+ /* No cache fact crosses a control-flow edge. Even a single-predecessor block
+ * may be entered through a layout-independent branch, so block entry is the
+ * simple, explicit ownership boundary. */
+ frame_cache_clear(e);
if (!e->label_placed[block]) {
e->label_placed[block] = 1u;
e->target->label_place(e->target,
@@ -1433,13 +2430,95 @@ static u32 compute_callee_saved_used(NativeEmitCtx* e, u32* used, u32 cap) {
nclasses = ri->nclasses < cap ? ri->nclasses : cap;
for (u32 i = 0; i < ri->nclasses; ++i) {
const NativeAllocClassInfo* ci = &ri->classes[i];
- if (ci->cls < cap)
+ if (ci->cls < cap) {
used[ci->cls] &=
native_target_callee_saved_mask(t, (NativeAllocClass)ci->cls);
+ used[ci->cls] |= e->asm_stage_callee_used[ci->cls];
+ }
}
return nclasses;
}
+static void compute_emit_live_after(NativeEmitCtx* e) {
+ OptHardBlockLive* blocks = opt_maybe_build_hard_live(e->f);
+ e->live_after_cap = e->f->next_inst_id;
+ if (!e->live_after_cap) return;
+ e->live_after_by_inst =
+ arena_zarray(e->f->arena, OptHardRegSet, e->live_after_cap);
+ for (u32 b = 0; b < e->f->nblocks; ++b) {
+ Block* bl = &e->f->blocks[b];
+ OptHardRegSet live =
+ opt_hard_live_out_for_block(blocks ? &blocks[b] : NULL);
+ for (u32 ri = bl->ninsts; ri > 0; --ri) {
+ Inst* in = &bl->insts[ri - 1u];
+ OptRegEffects effects;
+ OptHardRegSet kills;
+ if (in->id != INST_ID_NONE && in->id < e->live_after_cap)
+ e->live_after_by_inst[in->id] = live;
+ opt_inst_reg_effects(e->f, in, &effects);
+ opt_reg_effect_kills(&effects, &kills);
+ opt_hard_live_step(&live, &effects.uses, &kills);
+ }
+ }
+}
+
+static int asm_mir_reg_satisfies(NativeEmitCtx* e, const OptOperand* op,
+ NativeAllocClass cls, Reg fixed,
+ u32 allowed) {
+ NativeLoc loc;
+ if (!op || op->kind != OPK_REG) return 0;
+ loc = loc_reg(op->type, (NativeAllocClass)op->cls, op->v.reg);
+ return asm_reg_loc_satisfies(e, loc, cls, fixed, allowed);
+}
+
+/* Dry-run the instruction-local asm allocator before frame finalization. The
+ * same candidate order and unavailable masks are used by final emission, so
+ * this pass need only retain the ABI-relevant result: which callee-saved
+ * registers the staging leases will overwrite. */
+static void plan_asm_staging(NativeEmitCtx* e) {
+ memset(e->asm_stage_callee_used, 0, sizeof e->asm_stage_callee_used);
+ for (u32 b = 0; b < e->f->nblocks; ++b) {
+ Block* bl = &e->f->blocks[b];
+ for (u32 i = 0; i < bl->ninsts; ++i) {
+ Inst* in = &bl->insts[i];
+ IRAsmAux* aux;
+ NativeEmitTempScope scope;
+ if ((IROp)in->op != IR_ASM_BLOCK) continue;
+ aux = (IRAsmAux*)in->extra.aux;
+ if (!aux) continue;
+ temp_scope_begin(e, &scope, in);
+ scope.allow_asm_temps = (u8)!aux->has_memory_constraint;
+ for (u32 k = 0; k < aux->nout; ++k) {
+ NativeAllocClass cls;
+ Reg fixed;
+ u32 allowed;
+ if (!asm_reg_requirement(
+ e, aux->out_reg_reqs ? &aux->out_reg_reqs[k] : NULL, &cls,
+ &fixed, &allowed, in->loc) ||
+ asm_mir_reg_satisfies(e, &aux->out_ops[k], cls, fixed, allowed))
+ continue;
+ (void)temp_acquire_asm(e, cls, fixed, allowed, in->loc);
+ }
+ for (u32 k = 0; k < aux->nin; ++k) {
+ NativeAllocClass cls;
+ Reg fixed;
+ u32 allowed;
+ if (native_asm_match_index(aux->ins[k].str) >= 0) continue;
+ if (!asm_reg_requirement(
+ e, aux->in_reg_reqs ? &aux->in_reg_reqs[k] : NULL, &cls,
+ &fixed, &allowed, in->loc) ||
+ asm_mir_reg_satisfies(e, &aux->in_ops[k], cls, fixed, allowed))
+ continue;
+ (void)temp_acquire_asm(e, cls, fixed, allowed, in->loc);
+ }
+ for (u32 c = 0; c < OPT_REG_CLASSES; ++c)
+ e->asm_stage_callee_used[c] |=
+ scope.leased[c] & e->f->opt_callee_saved[c];
+ temp_scope_end(e, &scope);
+ }
+ }
+}
+
/* Plan the complete call frame before any code is emitted, then hand it to the
* backend via func_begin_known_frame so the prologue is emitted final. The
* optimizer knows everything the frame needs after register allocation and MIR
@@ -1468,6 +2547,9 @@ static void plan_frame(NativeEmitCtx* e, const CGFuncDesc* fd) {
nclasses = t->reserve_callee_saves
? compute_callee_saved_used(e, used, EMIT_MAX_REG_CLASSES)
: 0u;
+ memset(e->callee_saved_used, 0, sizeof e->callee_saved_used);
+ for (u32 c = 0; c < nclasses && c < OPT_REG_CLASSES; ++c)
+ e->callee_saved_used[c] = used[c];
/* Outgoing-arg area = max stack-arg bytes over all calls; also note alloca.
*/
for (u32 b = 0; b < e->f->nblocks; ++b) {
@@ -1487,10 +2569,10 @@ static void plan_frame(NativeEmitCtx* e, const CGFuncDesc* fd) {
} else if ((IROp)in->op == IR_ASM_BLOCK) {
/* Inline asm may clobber the return-address register or the red zone
* opaquely; disqualifies the frame-eliding tiers (see has_asm). Its
- * callee-saved register clobbers and hard-register operand pins are
- * equally opaque to the operand scan below; count them now so the
- * backend can fold them into the saved set (collected into a single Sym
- * list in a second pass below). */
+ * callee-saved register clobbers and named-register operand
+ * requirements are equally opaque to the operand scan below; count
+ * them now so the backend can fold them into the saved set (collected
+ * into a single Sym list in a second pass below). */
IRAsmAux* aux = (IRAsmAux*)in->extra.aux;
has_asm = 1;
if (aux) {
@@ -1520,10 +2602,10 @@ static void plan_frame(NativeEmitCtx* e, const CGFuncDesc* fd) {
}
}
}
- /* Gather the union of every asm block's clobber names and hard-register
- * operand pins for known-frame callee-save reservation. Keep the raw names:
- * each backend's frame planner uses its clobber parser so diagnostics and
- * ABI exclusions match final emission. */
+ /* Gather the union of every asm block's clobber names and named-register
+ * operand requirements for known-frame callee-save reservation. Keep the raw
+ * names: each backend's frame planner uses its clobber parser so diagnostics
+ * and ABI exclusions match final emission. */
if (nasm_clob) {
u32 n = 0;
asm_clobbers = arena_array(e->f->arena, Sym, nasm_clob);
@@ -1700,20 +2782,7 @@ void opt_emit_native(Compiler* c, Func* f, NativeTarget* target) {
CGFuncDesc fd;
if (!f || !target) return;
memset(&e, 0, sizeof e);
- if (f->mir) {
- view = *f;
- view.blocks = f->mir->blocks;
- view.nblocks = f->mir->nblocks;
- view.entry = f->mir->entry;
- view.emit_order = f->mir->emit_order;
- view.emit_order_n = f->mir->emit_order_n;
- view.emit_order_cap = f->mir->emit_order_cap;
- view.opt_rewritten = 1;
- view.mir = NULL;
- e.f = &view;
- } else {
- e.f = f;
- }
+ e.f = opt_mir_view(f, &view) ? &view : f;
e.c = c;
e.target = target;
metrics_scope_begin(c, "opt.native_emit.setup");
@@ -1722,6 +2791,8 @@ void opt_emit_native(Compiler* c, Func* f, NativeTarget* target) {
arena_zarray(e.f->arena, u8, e.f->nblocks ? e.f->nblocks : 1u);
for (u32 i = 0; i < e.f->nblocks; ++i) e.labels[i] = MC_LABEL_NONE;
fd = semantic_func_desc(&e);
+ compute_emit_live_after(&e);
+ plan_asm_staging(&e);
metrics_scope_end(c, "opt.native_emit.setup");
metrics_scope_begin(c, "opt.native_emit.func_begin");
diff --git a/src/opt/pass_o2.c b/src/opt/pass_o2.c
@@ -781,6 +781,9 @@ static u64 gvn_key_hash(const GvnKey* k) {
h = gvn_mix_u64(h, k->ops[i].v.ind.index);
h = gvn_mix_u64(h, (u64)(i64)k->ops[i].v.ind.ofs);
h = gvn_mix_u64(h, k->ops[i].v.ind.log2_scale);
+ h = gvn_mix_u64(h, k->ops[i].v.ind.index_ext);
+ h = gvn_mix_u64(h, k->ops[i].v.ind.base_type);
+ h = gvn_mix_u64(h, k->ops[i].v.ind.index_type);
} else {
h = gvn_mix_u64(h, k->ops[i].kind == OPK_REG ? k->ops[i].v.reg
: (u64)k->ops[i].v.imm);
@@ -795,7 +798,10 @@ static int gvn_operand_key_equal(const GvnOperandKey* a,
if (a->kind == OPK_INDIRECT)
return a->v.ind.base == b->v.ind.base && a->v.ind.index == b->v.ind.index &&
a->v.ind.ofs == b->v.ind.ofs &&
- a->v.ind.log2_scale == b->v.ind.log2_scale;
+ a->v.ind.log2_scale == b->v.ind.log2_scale &&
+ a->v.ind.index_ext == b->v.ind.index_ext &&
+ a->v.ind.base_type == b->v.ind.base_type &&
+ a->v.ind.index_type == b->v.ind.index_type;
if (a->kind == OPK_REG) return a->v.reg == b->v.reg;
return a->v.imm == b->v.imm;
}
@@ -923,12 +929,15 @@ static int gvn_make_addr_operand_key(GvnCtx* ctx, const Operand* op,
Val base = gvn_find(ctx, (Val)op->v.ind.base);
if (base == VAL_NONE || base >= ctx->f->nvals) return 0;
out->v.ind.base = base;
+ out->v.ind.base_type = ctx->f->val_type[base];
out->v.ind.index = VAL_NONE;
if (op->v.ind.index != REG_NONE) {
Val index = gvn_find(ctx, (Val)op->v.ind.index);
if (index == VAL_NONE || index >= ctx->f->nvals) return 0;
out->v.ind.index = index;
+ out->v.ind.index_type = ctx->f->val_type[index];
out->v.ind.log2_scale = op->v.ind.log2_scale;
+ out->v.ind.index_ext = op->v.ind.index_ext;
}
out->v.ind.ofs = op->v.ind.ofs;
return 1;
@@ -1263,7 +1272,13 @@ static int gvn_operand_key_less(const GvnOperandKey* a,
if (a->v.ind.index != b->v.ind.index)
return a->v.ind.index < b->v.ind.index;
if (a->v.ind.ofs != b->v.ind.ofs) return a->v.ind.ofs < b->v.ind.ofs;
- return a->v.ind.log2_scale < b->v.ind.log2_scale;
+ if (a->v.ind.log2_scale != b->v.ind.log2_scale)
+ return a->v.ind.log2_scale < b->v.ind.log2_scale;
+ if (a->v.ind.index_ext != b->v.ind.index_ext)
+ return a->v.ind.index_ext < b->v.ind.index_ext;
+ if (a->v.ind.base_type != b->v.ind.base_type)
+ return a->v.ind.base_type < b->v.ind.base_type;
+ return a->v.ind.index_type < b->v.ind.index_type;
}
if (a->kind == OPK_REG) return a->v.reg < b->v.reg;
return a->v.imm < b->v.imm;
@@ -1325,9 +1340,11 @@ static void gvn_replace_one_use(Func* f, const OptUse* use, Val repl) {
break;
case OPT_USE_INDIRECT_BASE:
use->operand->v.ind.base = (Reg)repl;
+ use->operand->v.ind.base_type = f->val_type[repl];
break;
case OPT_USE_INDIRECT_INDEX:
use->operand->v.ind.index = (Reg)repl;
+ use->operand->v.ind.index_type = f->val_type[repl];
break;
case OPT_USE_PHI_INPUT: {
IRPhiAux* aux = (IRPhiAux*)in->extra.aux;
diff --git a/src/opt/pass_ssa.c b/src/opt/pass_ssa.c
@@ -283,24 +283,13 @@ static Val reg_stack_top(RegRenameCtx* ctx, Reg r) {
}
static void reg_replace_use(RegRenameCtx* ctx, Operand* op) {
- if (!op) return;
- if (op->kind == OPK_REG) {
- Reg r = op->v.reg;
- Val v = reg_stack_top(ctx, r);
- if (v == VAL_NONE) return;
- op->v.reg = (Reg)v;
- op->type = ctx->f->val_type[v];
- op->cls = ctx->f->val_cls[v];
- } else if (op->kind == OPK_INDIRECT) {
- Reg r = op->v.ind.base;
- Val v = reg_stack_top(ctx, r);
- if (v != VAL_NONE) op->v.ind.base = (Reg)v;
- if (op->v.ind.index != (Reg)REG_NONE) {
- Reg ri = op->v.ind.index;
- Val vi = reg_stack_top(ctx, ri);
- if (vi != VAL_NONE) op->v.ind.index = (Reg)vi;
- }
- }
+ if (!op || op->kind != OPK_REG) return;
+ Reg r = op->v.reg;
+ Val v = reg_stack_top(ctx, r);
+ if (v == VAL_NONE) return;
+ op->v.reg = (Reg)v;
+ op->type = ctx->f->val_type[v];
+ op->cls = ctx->f->val_cls[v];
}
static Val reg_define_operand(RegRenameCtx* ctx, u32 b, u32 i, Inst* in,
diff --git a/src/opt/reg_effects.c b/src/opt/reg_effects.c
@@ -0,0 +1,152 @@
+#include <string.h>
+
+#include "opt/opt_internal.h"
+
+static void effect_add(OptHardRegSet* set, u8 cls, Reg reg) {
+ if (!set || cls >= OPT_REG_CLASSES || reg >= OPT_MAX_HARD_REGS) return;
+ set->cls[cls] |= 1u << reg;
+}
+
+static void effect_add_use(OptRegEffects* effects, u8 cls, Reg reg) {
+ if (!effects || cls >= OPT_REG_CLASSES || reg >= OPT_MAX_HARD_REGS) return;
+ effect_add(&effects->uses, cls, reg);
+ if (effects->use_count[cls][reg] < 2u)
+ ++effects->use_count[cls][reg];
+}
+
+static void effect_use_operand(OptRegEffects* effects, const Operand* op) {
+ if (!effects || !op) return;
+ if (op->kind == OPK_REG) {
+ effect_add_use(effects, op->cls, op->v.reg);
+ } else if (op->kind == OPK_INDIRECT) {
+ if (op->v.ind.base_kind == OPT_INDIRECT_REG)
+ effect_add_use(effects, RC_INT, op->v.ind.base);
+ if (op->v.ind.index_kind == OPT_INDIRECT_REG &&
+ op->v.ind.index != (Reg)REG_NONE)
+ effect_add_use(effects, RC_INT, op->v.ind.index);
+ }
+}
+
+static void effect_def_operand(OptRegEffects* effects, const Operand* op) {
+ if (effects && op && op->kind == OPK_REG)
+ effect_add(&effects->defs, op->cls, op->v.reg);
+}
+
+static void effect_use_abivalue(OptRegEffects* effects, const CGABIValue* v) {
+ if (!v) return;
+ effect_use_operand(effects, &v->storage);
+ for (u32 i = 0; i < v->nparts; ++i)
+ effect_use_operand(effects, &v->parts[i].op);
+}
+
+static void effect_def_abivalue(OptRegEffects* effects, const CGABIValue* v) {
+ if (!v) return;
+ effect_def_operand(effects, &v->storage);
+ for (u32 i = 0; i < v->nparts; ++i)
+ effect_def_operand(effects, &v->parts[i].op);
+}
+
+static void effect_add_masks(OptHardRegSet* set, const u32* masks) {
+ if (!set || !masks) return;
+ for (u32 c = 0; c < OPT_REG_CLASSES; ++c) set->cls[c] |= masks[c];
+}
+
+u32 opt_call_clobber_mask_for(Func* f, const Inst* in, u8 cls) {
+ if (!f || cls >= OPT_REG_CLASSES) return 0;
+ if (in && (IROp)in->op == IR_CALL) {
+ const IRCallAux* aux = (const IRCallAux*)in->extra.aux;
+ if (aux && aux->plan_valid) return aux->plan.clobber_mask[cls];
+ }
+ return f->opt_caller_saved[cls];
+}
+
+static void effect_explicit_operands(Func* f, const Inst* in,
+ OptRegEffects* effects) {
+ for (u32 i = 0; i < in->nopnds; ++i) {
+ if (opt_inst_operand_is_def(in, i))
+ effect_def_operand(effects, &in->opnds[i]);
+ else
+ effect_use_operand(effects, &in->opnds[i]);
+ }
+
+ /* Direct operand roles come from opt_inst_operand_is_def above. Only
+ * register-bearing auxiliary descriptions need instruction-specific work. */
+ switch ((IROp)in->op) {
+ case IR_CALL: {
+ const IRCallAux* aux = (const IRCallAux*)in->extra.aux;
+ if (aux && aux->use_plan_replay) {
+ effect_use_operand(effects, &aux->plan.callee);
+ for (u32 i = 0; i < aux->plan.nargs; ++i) {
+ effect_use_operand(effects, &aux->plan.args[i].src);
+ if (aux->plan.args[i].dst_kind == CG_CALL_PLAN_REG)
+ effect_add(&effects->defs, aux->plan.args[i].cls,
+ aux->plan.args[i].dst_reg);
+ }
+ for (u32 i = 0; i < aux->plan.nrets; ++i) {
+ effect_add(&effects->defs, aux->plan.rets[i].cls,
+ aux->plan.rets[i].src_reg);
+ effect_def_operand(effects, &aux->plan.rets[i].dst);
+ }
+ } else if (aux) {
+ effect_use_operand(effects, &aux->desc.callee);
+ for (u32 i = 0; i < aux->desc.nargs; ++i)
+ effect_use_abivalue(effects, &aux->desc.args[i]);
+ effect_def_abivalue(effects, &aux->desc.ret);
+ }
+ for (u32 c = 0; c < OPT_REG_CLASSES; ++c)
+ effects->clobbers.cls[c] |=
+ opt_call_clobber_mask_for(f, in, (u8)c);
+ break;
+ }
+ case IR_RET: {
+ const IRRetAux* aux = (const IRRetAux*)in->extra.aux;
+ if (aux && aux->present) effect_use_abivalue(effects, &aux->val);
+ break;
+ }
+ case IR_ASM_BLOCK: {
+ const IRAsmAux* aux = (const IRAsmAux*)in->extra.aux;
+ if (!aux) break;
+ for (u32 i = 0; i < aux->nin; ++i)
+ effect_use_operand(effects, &aux->in_ops[i]);
+ for (u32 i = 0; i < aux->nout; ++i)
+ effect_def_operand(effects, &aux->out_ops[i]);
+ effect_add_masks(&effects->clobbers, aux->clobber_mask);
+ break;
+ }
+ case IR_INTRINSIC: {
+ const IRIntrinAux* aux = (const IRIntrinAux*)in->extra.aux;
+ if (!aux) break;
+ for (u32 i = 0; i < aux->narg; ++i)
+ effect_use_operand(effects, &aux->args[i]);
+ for (u32 i = 0; i < aux->ndst; ++i)
+ effect_def_operand(effects, &aux->dsts[i]);
+ break;
+ }
+ default:
+ break;
+ }
+}
+
+static void effect_add_machine_clobbers(Func* f, const Inst* in,
+ OptRegEffects* effects) {
+ if (!f->inst_clobbers || in->id == INST_ID_NONE ||
+ in->id >= f->inst_clobbers_cap)
+ return;
+ effect_add_masks(&effects->clobbers, f->inst_clobbers[in->id]);
+}
+
+void opt_inst_reg_effects(Func* f, const Inst* in, OptRegEffects* effects) {
+ if (!effects) return;
+ memset(effects, 0, sizeof *effects);
+ if (!f || !in) return;
+ effect_explicit_operands(f, in, effects);
+ effect_add_machine_clobbers(f, in, effects);
+}
+
+void opt_reg_effect_kills(const OptRegEffects* effects, OptHardRegSet* kills) {
+ if (!kills) return;
+ memset(kills, 0, sizeof *kills);
+ if (!effects) return;
+ for (u32 c = 0; c < OPT_REG_CLASSES; ++c)
+ kills->cls[c] = effects->defs.cls[c] | effects->clobbers.cls[c];
+}
diff --git a/test/arch/arm32_inline_test.c b/test/arch/arm32_inline_test.c
@@ -3,6 +3,8 @@
#include <stdint.h>
#include <string.h>
+#include <kit/disasm.h>
+
#include "inline_public_test.h"
static void arm32_body(KitCompiler* c, KitCg* cg, KitCgTypeId i64_ty) {
@@ -22,6 +24,80 @@ static void arm32_bad_operand(KitCompiler* c, KitCg* cg, KitCgTypeId i64_ty) {
it_inline_asm(c, cg, "mov %9, r0", NULL, 0, NULL, 0, NULL, 0);
}
+/* r0-r3 are the direct binder's caller-saved operand pool. Five simultaneous
+ * generic inputs therefore put the fifth value in callee-saved r4. */
+static void arm32_generic_callee_saved_operand(KitCompiler* c, KitCg* cg,
+ KitCgTypeId i64_ty) {
+ KitCgTypeId i32_ty = kit_cg_type_builtin(c, KIT_CG_BUILTIN_I32);
+ KitCgAsmOperand ins[5];
+ KitCgLocal locals[5];
+ KitCgLocalAttrs attrs;
+ KitCgMemAccess mem;
+ uint32_t i;
+ (void)i64_ty;
+
+ memset(&attrs, 0, sizeof attrs);
+ memset(&mem, 0, sizeof mem);
+ mem.type = i32_ty;
+ mem.align = kit_cg_type_align(c, i32_ty);
+ for (i = 0; i < 5u; ++i) {
+ locals[i] = kit_cg_local(cg, i32_ty, attrs);
+ kit_cg_push_int(cg, i + 1u, i32_ty);
+ kit_cg_local_write(cg, locals[i], mem);
+ }
+ for (i = 0; i < 5u; ++i) {
+ kit_cg_local_read(cg, locals[i], mem);
+ ins[i] = it_asm_op(c, "r", NULL, i32_ty, KIT_CG_ASM_IN);
+ }
+ it_inline_asm(c, cg, "", NULL, 0, ins, 5, NULL, 0);
+}
+
+static int arm32_is_r4_mem(const KitInsn* insn, const char* mnemonic) {
+ size_t n = strlen(mnemonic);
+ return insn->mnemonic.s && insn->mnemonic.len >= n &&
+ memcmp(insn->mnemonic.s, mnemonic, n) == 0 && insn->operands.s &&
+ insn->operands.len >= 2u && insn->operands.s[0] == 'r' &&
+ insn->operands.s[1] == '4';
+}
+
+static int arm32_has_generic_r4_preservation(InlineTestEnv* env,
+ const InlineText* text) {
+ KitTargetOptions opts;
+ KitTarget* target = NULL;
+ KitDisasmContext dc;
+ KitDisasmIter* it = NULL;
+ KitInsn insn;
+ uint32_t loads_after_save = 0;
+ int saw_save = 0;
+ int ok = 0;
+
+ memset(&opts, 0, sizeof opts);
+ opts.spec = it_target(KIT_ARCH_ARM_32);
+ if (kit_target_new(&env->ctx, &opts, &target) != KIT_OK || !target)
+ goto done;
+ memset(&dc, 0, sizeof dc);
+ dc.target = target;
+ dc.context = env->ctx;
+ if (kit_disasm_iter_new(&dc, text->data, text->len, 0, text->file, &it) !=
+ KIT_OK ||
+ !it)
+ goto done;
+ while (kit_disasm_iter_next(it, &insn) == KIT_ITER_ITEM) {
+ if (!saw_save && arm32_is_r4_mem(&insn, "str")) {
+ saw_save = 1;
+ } else if (saw_save && arm32_is_r4_mem(&insn, "ldr")) {
+ loads_after_save++;
+ }
+ }
+ /* The first load stages operand five; the second restores caller r4. */
+ ok = saw_save && loads_after_save >= 2u;
+
+done:
+ if (it) kit_disasm_iter_free(it);
+ kit_target_free(target);
+ return ok;
+}
+
static int arm32_emit_file_asm(InlineTestEnv* env, const char* src,
InlineText* text) {
KitCompiler* c = NULL;
@@ -199,6 +275,17 @@ int main(void) {
IT_EXPECT(&env, arm32_check_thumb_func(&env),
"arm32 .thumb_func symbol regression failed: %s", env.last_diag);
+ IT_EXPECT(&env,
+ it_emit_text(&env, KIT_ARCH_ARM_32,
+ "arm32_generic_callee_saved_operand",
+ arm32_generic_callee_saved_operand, &text),
+ "failed to emit generic callee-saved asm operand: %s",
+ env.last_diag);
+ if (text.data)
+ IT_EXPECT(&env, arm32_has_generic_r4_preservation(&env, &text),
+ "generic r4 asm operand was not saved and restored");
+ it_text_close(&text);
+
if (env.fails) {
fprintf(stderr, "%d failure(s)\n", env.fails);
return 1;
diff --git a/test/arch/x64_inline_test.c b/test/arch/x64_inline_test.c
@@ -5,6 +5,37 @@
#include "inline_public_test.h"
+/* Collect the architectural XMM registers named by `ucomisd %xmmN,%xmmN`.
+ * The inline case below emits one such non-destructive compare per input. A
+ * full mask therefore proves that all sixteen simultaneous constraints were
+ * bound independently; merely compiling an empty template cannot detect an
+ * accidental many-to-one placement. */
+static uint32_t x64_self_ucomisd_bindings(const uint8_t* p, size_t n,
+ uint32_t* count_out) {
+ uint32_t mask = 0;
+ uint32_t count = 0;
+ for (size_t i = 0; i + 3u < n; ++i) {
+ size_t op = i;
+ uint8_t rex = 0;
+ uint8_t modrm;
+ uint32_t lhs;
+ uint32_t rhs;
+ if (p[op++] != 0x66u) continue;
+ if (op < n && p[op] >= 0x40u && p[op] <= 0x4fu) rex = p[op++];
+ if (op + 2u >= n || p[op] != 0x0fu || p[op + 1u] != 0x2eu)
+ continue;
+ modrm = p[op + 2u];
+ if ((modrm & 0xc0u) != 0xc0u) continue;
+ lhs = ((modrm >> 3u) & 7u) | ((rex & 4u) ? 8u : 0u);
+ rhs = (modrm & 7u) | ((rex & 1u) ? 8u : 0u);
+ if (lhs != rhs) continue;
+ mask |= 1u << lhs;
+ ++count;
+ }
+ if (count_out) *count_out = count;
+ return mask;
+}
+
static void x64_body(KitCompiler* c, KitCg* cg, KitCgTypeId i64_ty) {
KitCgAsmOperand imm;
(void)i64_ty;
@@ -60,6 +91,14 @@ static void x64_push_local_inputs(KitCompiler* c, KitCg* cg, KitCgTypeId ty,
}
static void x64_optimized_stage(KitCompiler* c, KitCg* cg, KitCgTypeId i64_ty) {
+ static const char xmm_use_template[] =
+ "ucomisd %0, %0 ; ucomisd %1, %1 ; ucomisd %2, %2 ; "
+ "ucomisd %3, %3 ; ucomisd %4, %4 ; ucomisd %5, %5 ; "
+ "ucomisd %6, %6 ; ucomisd %7, %7 ; ucomisd %8, %8 ; "
+ "ucomisd %9, %9 ; ucomisd %10, %10 ; ucomisd %11, %11 ; "
+ "ucomisd %12, %12 ; ucomisd %13, %13 ; ucomisd %14, %14 ; "
+ "ucomisd %15, %15";
+ static const char* const cc_clobber[] = {"cc"};
KitCgTypeId f64_ty = kit_cg_type_builtin(c, KIT_CG_BUILTIN_F64);
KitCgAsmOperand ins[18];
KitCgAsmOperand outs[6];
@@ -70,8 +109,12 @@ static void x64_optimized_stage(KitCompiler* c, KitCg* cg, KitCgTypeId i64_ty) {
x64_push_local_inputs(c, cg, i64_ty, "q", ins, 14);
it_inline_asm(c, cg, "", NULL, 0, ins, 14, NULL, 0);
- x64_push_local_inputs(c, cg, f64_ty, "x", ins, 18);
- it_inline_asm(c, cg, "", NULL, 0, ins, 18, NULL, 0);
+ /* Sixteen is the architectural XMM register count. The old spill-to-scratch
+ * rewrite silently aliased simultaneous operands. Reference every input in
+ * the emitted template so the byte-level check can prove that each binding
+ * is distinct; an empty template only proved that planning did not panic. */
+ x64_push_local_inputs(c, cg, f64_ty, "x", ins, 16);
+ it_inline_asm(c, cg, xmm_use_template, NULL, 0, ins, 16, cc_clobber, 1);
for (uint32_t i = 0; i < 6u; ++i)
outs[i] = it_asm_op(c, i & 1u ? "=x" : "=r", NULL,
@@ -127,10 +170,20 @@ int main(void) {
{
InlineText opt;
+ uint32_t xmm_compare_count = 0;
+ uint32_t xmm_bindings = 0;
IT_EXPECT(&env,
it_emit_text_opt(&env, KIT_ARCH_X86_64, "x64_optimized_stage",
x64_optimized_stage, 1, &opt),
"failed to emit optimized x64 inline-asm staging case");
+ if (opt.data)
+ xmm_bindings = x64_self_ucomisd_bindings(
+ opt.data, opt.len, &xmm_compare_count);
+ IT_EXPECT(&env,
+ xmm_compare_count == 16u && xmm_bindings == 0xffffu,
+ "sixteen simultaneous XMM inputs were not independently bound "
+ "(compares=%u mask=%#x)",
+ xmm_compare_count, xmm_bindings);
it_text_close(&opt);
}
diff --git a/test/cg/native_direct_target_test.c b/test/cg/native_direct_target_test.c
@@ -79,21 +79,21 @@ typedef struct MockNative {
static const Reg mock_int_scratch[] = {1u, 2u, 3u};
static const Reg mock_fp_scratch[] = {4u, 5u};
-static const Reg mock_int_allocable[] = {1u, 2u, 3u, 6u};
-static const Reg mock_fp_allocable[] = {4u, 5u, 6u};
+static const Reg mock_ndt_int_allocable[] = {1u, 2u, 3u, 6u};
+static const Reg mock_ndt_fp_allocable[] = {4u, 5u, 6u};
static const NativeAllocClassInfo mock_classes[] = {
{.cls = NATIVE_REG_INT,
- .allocable = mock_int_allocable,
- .nallocable = 4,
+ .ndt_allocable = mock_ndt_int_allocable,
+ .ndt_allocable_count = 4,
.scratch = mock_int_scratch,
.nscratch = 3,
.caller_saved_mask = (1u << 1) | (1u << 2) | (1u << 3),
.arg_mask = 1u << 0,
.ret_mask = 1u << 0},
{.cls = NATIVE_REG_FP,
- .allocable = mock_fp_allocable,
- .nallocable = 3,
+ .ndt_allocable = mock_ndt_fp_allocable,
+ .ndt_allocable_count = 3,
.scratch = mock_fp_scratch,
.nscratch = 2,
.caller_saved_mask = (1u << 4) | (1u << 5),
@@ -419,11 +419,12 @@ static void test_frame_locals_scratch_storeback_and_branches(void) {
EXPECT(count_event(&native, EV_LOAD_IMM) == 3,
"two explicit immediates plus cmp imm materialization expected");
/* With the local register cache, a/b/sum live in registers across the
- * straight-line compute run: the binop reloads neither operand. They are
- * spilled at the branch flush (EV_STORE), and sum is reloaded only after that
- * flush — once for the cmp_branch and once for the return. */
- EXPECT(count_event(&native, EV_LOAD) == 2,
- "sum is reloaded from its home for the cmp_branch and the return");
+ * straight-line compute run. cmp_branch pins sum before flushing, so the
+ * compare needs no reload; only the later return reloads it from its home. */
+ EXPECT(count_event(&native, EV_LOAD) == 1,
+ "sum is reloaded from its home only for the return "
+ "(loads=%d)",
+ count_event(&native, EV_LOAD));
EXPECT(count_event(&native, EV_STORE) >= 3,
"results should store back to frame homes");
EXPECT(count_event(&native, EV_BINOP) == 1, "expected one native binop");
@@ -472,8 +473,9 @@ static void test_call_barrier_storeback_and_max_outgoing(void) {
"call planning should track max outgoing stack size");
EXPECT(count_event(&native, EV_PLAN_CALL) == 1, "expected one call plan");
EXPECT(count_event(&native, EV_EMIT_CALL) == 1, "expected one emitted call");
- EXPECT(count_event(&native, EV_STORE) >= 3,
- "arg setup and result should store through native writes");
+ EXPECT(count_event(&native, EV_STORE) == 2,
+ "selective call setup should emit only its required stores (stores=%d)",
+ count_event(&native, EV_STORE));
t->func_end(t);
tc_fini(&tc);
}
@@ -506,8 +508,10 @@ static void test_b_cached_pointer_base_not_reloaded(void) {
EXPECT(li >= 0 && native.events[li].a == NATIVE_ADDR_BASE_REG,
"dereference addresses the live cache register for p");
}
- EXPECT(count_event(&native, EV_STORE) == 1,
- "p is not spilled; only the load result is written to d's home");
+ EXPECT(count_event(&native, EV_STORE) == 0,
+ "p stays live and the load result remains in the write-back cache "
+ "(stores=%d)",
+ count_event(&native, EV_STORE));
t->func_end(t);
tc_fini(&tc);
}
@@ -549,9 +553,9 @@ static void test_b_cache_survives_store(void) {
tc_fini(&tc);
}
-/* A volatile access must still flush the cache (it may observe memory) and emit
- * the volatile barrier — the escape argument does not apply. */
-static void test_b_volatile_load_flushes(void) {
+/* A volatile access emits the ordering barrier but does not flush unrelated,
+ * non-escaped locals: foreign memory cannot alias their private frame homes. */
+static void test_b_volatile_load_preserves_private_cache(void) {
TestCtx tc;
MockNative native;
CgTarget* t;
@@ -571,8 +575,10 @@ static void test_b_volatile_load_flushes(void) {
EXPECT((native.barrier_flags & NATIVE_DIRECT_BARRIER_VOLATILE) != 0,
"volatile access emits a volatile barrier");
- EXPECT(count_event(&native, EV_STORE) >= 2,
- "volatile flush spills the dirty cached locals (p and a)");
+ EXPECT(count_event(&native, EV_STORE) == 0,
+ "volatile foreign-memory access preserves private cached locals "
+ "(stores=%d)",
+ count_event(&native, EV_STORE));
t->func_end(t);
tc_fini(&tc);
}
@@ -616,7 +622,7 @@ int main(void) {
test_call_barrier_storeback_and_max_outgoing();
test_b_cached_pointer_base_not_reloaded();
test_b_cache_survives_store();
- test_b_volatile_load_flushes();
+ test_b_volatile_load_preserves_private_cache();
test_b_call_still_flushes();
if (g_u.fails) {
fprintf(stderr, "%d/%d checks failed\n", g_u.fails, g_u.checks);
diff --git a/test/lib/exec_bare.sh b/test/lib/exec_bare.sh
@@ -347,7 +347,7 @@ SECTIONS {
}
EOF
"$EXEC_BARE_KIT" as -target riscv64-none-elf \
- -march=rv64imafd_zicsr_zifencei -mabi=lp64 \
+ -march=rv64imafd_zicsr_zifencei -mabi=lp64d \
-o "$dir/start.o" "$dir/start.S" 2>"$dir/as.err"
}
diff --git a/test/macho/kit-roundtrip-macho.c b/test/macho/kit-roundtrip-macho.c
@@ -2,12 +2,13 @@
* reader, then re-emit via the public object builder API. Round-trip oracle
* for the Mach-O writer.
*
- * Usage: kit-roundtrip-macho <in.o> <out.o>
+ * Usage: kit-roundtrip-macho [--require-atom-subordinate] <in.o> <out.o>
*
* Behavior: kit_obj_open detects and parses the input into a KitObjFile;
- * kit_obj_file_builder exposes the already-finalized builder; and
- * kit_obj_builder_emit writes the canonical re-emit to out.o. Diagnostics
- * go to stderr via the libc heap + stderr diag sink. */
+ * kit_obj_file_builder exposes the already-finalized builder; subordinate
+ * symbols are checked against their owning atoms; and kit_obj_builder_emit
+ * writes the canonical re-emit to out.o. Diagnostics go to stderr via the
+ * libc heap + stderr diag sink. */
#include <fcntl.h>
#include <kit/core.h>
@@ -19,6 +20,8 @@
#include <sys/stat.h>
#include <unistd.h>
+#include "obj/obj.h"
+
static void* heap_alloc(KitHeap* h, size_t n, size_t a) {
(void)h;
(void)a;
@@ -98,13 +101,73 @@ static int write_file(const char* path, const uint8_t* data, size_t len) {
return 0;
}
+static int verify_atom_subordinates(KitObjBuilder* ob, int require_one) {
+ ObjSymIter* it = obj_symiter_new(ob);
+ ObjSymEntry e;
+ unsigned count = 0;
+ int failed = 0;
+ if (!it) {
+ fprintf(stderr, "error: cannot inspect Mach-O symbols\n");
+ return -1;
+ }
+ while (obj_symiter_next(it, &e)) {
+ const ObjSym* sym = e.sym;
+ const ObjAtom* atom;
+ ObjAtomId aid;
+ if (!sym || sym->removed || !sym->atom_subordinate) continue;
+ ++count;
+ if (sym->section_id == OBJ_SEC_NONE || sym->kind == SK_UNDEF ||
+ sym->kind == SK_COMMON || sym->kind == SK_ABS) {
+ fprintf(stderr,
+ "error: atom-subordinate symbol %u is not a section "
+ "definition\n",
+ (unsigned)e.id);
+ failed = 1;
+ continue;
+ }
+ aid = obj_atom_find_symbol(ob, e.id);
+ atom = obj_atom_get(ob, aid);
+ if (!atom) {
+ fprintf(stderr,
+ "error: atom-subordinate symbol %u has no owning atom\n",
+ (unsigned)e.id);
+ failed = 1;
+ }
+ for (aid = 1; aid < obj_atom_count(ob); ++aid) {
+ atom = obj_atom_get(ob, aid);
+ if (atom && !atom->removed && atom->signature == e.id) {
+ fprintf(stderr,
+ "error: atom-subordinate symbol %u was reconstructed as an "
+ "independent atom\n",
+ (unsigned)e.id);
+ failed = 1;
+ break;
+ }
+ }
+ }
+ obj_symiter_free(it);
+ if (require_one && count == 0) {
+ fprintf(stderr, "error: no atom-subordinate symbol survived roundtrip\n");
+ failed = 1;
+ }
+ return failed ? -1 : 0;
+}
+
int main(int argc, char** argv) {
- if (argc != 3) {
- fprintf(stderr, "usage: kit-roundtrip-macho <in.o> <out.o>\n");
+ int require_subordinate = 0;
+ int argi = 1;
+ if (argc > 1 && strcmp(argv[1], "--require-atom-subordinate") == 0) {
+ require_subordinate = 1;
+ ++argi;
+ }
+ if (argc - argi != 2) {
+ fprintf(stderr,
+ "usage: kit-roundtrip-macho [--require-atom-subordinate] "
+ "<in.o> <out.o>\n");
return 2;
}
- const char* in_path = argv[1];
- const char* out_path = argv[2];
+ const char* in_path = argv[argi];
+ const char* out_path = argv[argi + 1];
uint8_t* in_data = NULL;
size_t in_len = 0;
@@ -144,6 +207,11 @@ int main(int argc, char** argv) {
free(in_data);
return 1;
}
+ if (verify_atom_subordinates(ob, require_subordinate) != 0) {
+ kit_obj_free(obj);
+ free(in_data);
+ return 1;
+ }
KitWriter* w = NULL;
if (kit_writer_mem(&g_heap, &w) != KIT_OK || !w) {
diff --git a/test/opt/aa64_apple_varargs.sh b/test/opt/aa64_apple_varargs.sh
@@ -0,0 +1,120 @@
+#!/usr/bin/env bash
+# AArch64 call-boundary regression:
+# - Apple unnamed aggregate stack slots keep semantic bytes separate from
+# their rounded physical carriers (Six: 6 in 8, Twelve: 12 in 16).
+# - an indirect target initially in x0 survives exact-width stack-argument
+# marshalling, which owns the backend-private x16/x17 pair.
+set -euo pipefail
+
+ROOT="$(cd "$(dirname "$0")/../.." && pwd)"
+KIT="${KIT:-$ROOT/build/kit}"
+WORK="$ROOT/build/test/opt/aa64_apple_varargs"
+SRC="$WORK/fixture.c"
+mkdir -p "$WORK"
+
+cat > "$SRC" <<'EOF'
+typedef struct Six { unsigned char b[6]; } Six;
+typedef struct Twelve { unsigned char b[12]; } Twelve;
+
+__attribute__((noinline)) void variadic_sink(int tag, ...) {
+ volatile int seen = tag;
+ (void)seen;
+}
+
+__attribute__((noinline)) void pass_apple_varargs(Six six, Twelve twelve) {
+ variadic_sink(7, six, twelve);
+}
+
+typedef int (*Indirect)(int, int, int, int, int, int, int, int, Six);
+
+__attribute__((noinline)) int call_indirect(Indirect fn, Six six) {
+ return fn(1, 2, 3, 4, 5, 6, 7, 8, six);
+}
+
+__attribute__((noinline)) static int check_indirect(
+ int a, int b, int c, int d, int e, int f, int g, int h, Six six) {
+ return a == 1 && b == 2 && c == 3 && d == 4 && e == 5 && f == 6 &&
+ g == 7 && h == 8 && six.b[0] == 9 && six.b[1] == 10 &&
+ six.b[2] == 11 && six.b[3] == 12 && six.b[4] == 13 &&
+ six.b[5] == 14;
+}
+
+int test_main(void) {
+ Six six = {{9, 10, 11, 12, 13, 14}};
+ return call_indirect(check_indirect, six) ? 0 : 1;
+}
+EOF
+
+fail() {
+ printf 'aa64 Apple-varargs regression FAILED: %s\n' "$1" >&2
+ if [ -n "${2:-}" ] && [ -f "$2" ]; then
+ sed 's/^/ | /' "$2" >&2
+ fi
+ exit 1
+}
+
+slice_fn() {
+ local name="$1" input="$2" output="$3"
+ awk -v name="$name" '
+ $0 ~ "<_?" name ">:" { in_fn = 1; next }
+ /^[0-9a-f]+ <[^>]+>:/ { in_fn = 0 }
+ in_fn { print }
+ ' "$input" > "$output"
+}
+
+for opt in O0 O1; do
+ obj="$WORK/apple.$opt.o"
+ dis="$WORK/apple.$opt.dis"
+ body="$WORK/apple.$opt.pass.dis"
+ "$KIT" build-obj -target aarch64-apple-darwin "-$opt" -c "$SRC" -o "$obj" \
+ > "$WORK/apple.$opt.compile.out" 2>&1
+ "$KIT" objdump -d "$obj" > "$dis" 2>&1
+ slice_fn pass_apple_varargs "$dis" "$body"
+ [ -s "$body" ] || fail "$opt pass_apple_varargs missing" "$dis"
+
+ # Six is the <=8 control: it is read at six bytes, zero-extended, and the
+ # complete 8-byte carrier is stored. Twelve occupies the following 16-byte
+ # carrier, but only its live 8+4 bytes may be read or written.
+ grep -Eq 'str[[:space:]]+x[0-9]+, \[sp\]' "$body" ||
+ fail "$opt missing zero-padded Six carrier store" "$body"
+ grep -Eq 'str[[:space:]]+x[0-9]+, \[sp, #8\]' "$body" ||
+ fail "$opt missing first eight bytes of Twelve" "$body"
+ grep -Eq 'str[[:space:]]+w[0-9]+, \[sp, #16\]' "$body" ||
+ fail "$opt missing four-byte Twelve tail" "$body"
+ if grep -Eq 'str[[:space:]]+x[0-9]+, \[sp, #16\]' "$body"; then
+ fail "$opt widened Twelve tail into carrier padding" "$body"
+ fi
+ xloads="$(grep -Ec '(ldr|ldur)[[:space:]]+x[0-9]+, \[x29' "$body" || true)"
+ [ "$xloads" -eq 1 ] ||
+ fail "$opt expected one 8-byte Twelve source load, got $xloads" "$body"
+done
+
+# O1 keeps fn in x0. The target must be saved before the Six stack writes use
+# x16/x17, then moved into x16 only after those writes are complete.
+call_body="$WORK/apple.O1.call.dis"
+slice_fn call_indirect "$WORK/apple.O1.dis" "$call_body"
+[ -s "$call_body" ] || fail 'O1 call_indirect missing' "$WORK/apple.O1.dis"
+stash_line="$(grep -n 'mov[[:space:]]\+x30, x0' "$call_body" | head -n1 | cut -d: -f1 || true)"
+stack_line="$(grep -n 'str[[:space:]]\+w[0-9]\+, \[sp\]' "$call_body" | head -n1 | cut -d: -f1 || true)"
+reload_line="$(grep -n 'mov[[:space:]]\+x16, x30' "$call_body" | head -n1 | cut -d: -f1 || true)"
+call_line="$(grep -n 'blr[[:space:]]\+x16' "$call_body" | head -n1 | cut -d: -f1 || true)"
+if [ -z "$stash_line" ] || [ -z "$stack_line" ] || [ -z "$reload_line" ] ||
+ [ -z "$call_line" ] || [ "$stash_line" -ge "$stack_line" ] ||
+ [ "$stack_line" -ge "$reload_line" ] || [ "$reload_line" -ge "$call_line" ]; then
+ fail 'indirect target lifetime is not stash -> stack args -> reload -> blr' \
+ "$call_body"
+fi
+
+# Execute the indirect-call oracle when this host can run AArch64 natively.
+case "$(uname -m 2>/dev/null)" in
+ arm64|aarch64)
+ if "$KIT" run -O1 -e test_main "$SRC" > "$WORK/native.run.out" 2>&1; then
+ rc=0
+ else
+ rc=$?
+ fi
+ [ "$rc" -eq 0 ] || fail "native indirect call exited $rc" "$WORK/native.run.out"
+ ;;
+esac
+
+printf 'aa64-apple-varargs: ok\n'
diff --git a/test/opt/arm32_emit_temp_pressure.c b/test/opt/arm32_emit_temp_pressure.c
@@ -0,0 +1,15 @@
+/* Keep many scalar values live while emitting three-operand arithmetic. ARM32
+ * allocates the values in r4-r11; the native emitter must satisfy spill-heavy
+ * operands from its explicit r0-r3 temp bank without borrowing IP/LR or
+ * exposing those temporaries in MIR. */
+long arm32_emit_temp_pressure(long s) {
+ long a = s + 1, b = s + 2, c = s + 3, d = s + 4;
+ long e = s + 5, f = s + 6, g = s + 7, h = s + 8;
+ long i = s + 9, j = s + 10, k = s + 11, l = s + 12;
+ long m = s + 13, n = s + 14, o = s + 15, p = s + 16;
+ long products = a * b + c * d + e * f + g * h + i * j + k * l + m * n + o * p;
+ long differences =
+ a - b + c - d + e - f + g - h + i - j + k - l + m - n + o - p;
+ long bits = a ^ b ^ c ^ d ^ e ^ f ^ g ^ h ^ i ^ j ^ k ^ l ^ m ^ n ^ o ^ p;
+ return products + differences + bits + a + p;
+}
diff --git a/test/opt/arm32_emit_temp_pressure.sh b/test/opt/arm32_emit_temp_pressure.sh
@@ -0,0 +1,21 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+ROOT="$(cd "$(dirname "$0")/../.." && pwd)"
+KIT="${KIT:-$ROOT/build/kit}"
+WORK="$ROOT/build/test/opt/arm32_emit_temp_pressure"
+SRC="$ROOT/test/opt/arm32_emit_temp_pressure.c"
+mkdir -p "$WORK"
+
+if ! "$KIT" cc -O1 -target arm-none-eabi -mcpu=cortex-m3 \
+ -mfloat-abi=soft -ffreestanding -std=c11 -c "$SRC" \
+ -o "$WORK/pressure.o" > "$WORK/compile.out" 2>&1; then
+ sed 's/^/ | /' "$WORK/compile.out" >&2
+ exit 1
+fi
+
+if [ ! -s "$WORK/pressure.o" ]; then
+ printf 'arm32_emit_temp_pressure: compiler produced no object\n' >&2
+ exit 1
+fi
+printf 'arm32_emit_temp_pressure: OK\n'
diff --git a/test/opt/combine_cse_test.c b/test/opt/combine_cse_test.c
@@ -0,0 +1,517 @@
+#include <stdio.h>
+#include <string.h>
+
+#include "lib/kit_unit.h"
+#include "opt/opt_internal.h"
+
+typedef struct CseFixture {
+ KitCompiler* kc;
+ Func* f;
+ u32 block;
+ KitCgTypeId i8_type;
+ KitCgTypeId i32_type;
+ KitCgTypeId i64_type;
+} CseFixture;
+
+static Operand reg_op(Reg reg, KitCgTypeId type) {
+ Operand op;
+ memset(&op, 0, sizeof op);
+ op.kind = OPK_REG;
+ op.cls = RC_INT;
+ op.type = type;
+ op.v.reg = reg;
+ return op;
+}
+
+static Operand global_op(ObjSymId sym, KitCgTypeId type) {
+ Operand op;
+ memset(&op, 0, sizeof op);
+ op.kind = OPK_GLOBAL;
+ op.cls = RC_INT;
+ op.type = type;
+ op.v.global.sym = sym;
+ return op;
+}
+
+static Operand stack_op(FrameSlot slot, KitCgTypeId type) {
+ Operand op;
+ memset(&op, 0, sizeof op);
+ op.kind = OPK_STACK;
+ op.cls = RC_INT;
+ op.type = type;
+ op.v.frame_slot = slot;
+ return op;
+}
+
+static int fixture_init(KitUnit* u, CseFixture* fx) {
+ CGFuncDesc desc;
+ memset(fx, 0, sizeof *fx);
+ if (kit_unit_compiler_new(
+ u, kit_unit_target(KIT_ARCH_X86_64, KIT_OS_LINUX, KIT_OBJ_ELF),
+ &fx->kc) != KIT_OK ||
+ !fx->kc)
+ return 0;
+
+ fx->i8_type = kit_cg_type_builtin(fx->kc, KIT_CG_BUILTIN_I8);
+ fx->i32_type = kit_cg_type_builtin(fx->kc, KIT_CG_BUILTIN_I32);
+ fx->i64_type = kit_cg_type_builtin(fx->kc, KIT_CG_BUILTIN_I64);
+ memset(&desc, 0, sizeof desc);
+ desc.fn_type = fx->i64_type;
+ desc.result_type = fx->i64_type;
+ fx->f = ir_func_new((Compiler*)fx->kc, &desc);
+ if (!fx->f) return 0;
+ fx->block = ir_block_new(fx->f);
+ fx->f->entry = fx->block;
+ return 1;
+}
+
+static Inst* emit_load(CseFixture* fx, Reg dst, KitCgTypeId result_type,
+ MemAccess mem) {
+ Inst* in = ir_emit(fx->f, fx->block, IR_LOAD);
+ in->type = result_type;
+ in->nopnds = 2;
+ in->opnds = arena_array(fx->f->arena, Operand, 2);
+ in->opnds[0] = reg_op(dst, result_type);
+ in->opnds[1] = global_op(1u, mem.type);
+ in->extra.mem = mem;
+ return in;
+}
+
+static Inst* emit_add(CseFixture* fx, Reg dst, KitCgTypeId result_type,
+ KitCgTypeId lhs_type, KitCgTypeId rhs_type,
+ u8 rhs_shift) {
+ Inst* in = ir_emit(fx->f, fx->block, IR_BINOP);
+ in->type = result_type;
+ in->nopnds = 3;
+ in->opnds = arena_array(fx->f->arena, Operand, 3);
+ in->opnds[0] = reg_op(dst, result_type);
+ in->opnds[1] = reg_op(4u, lhs_type);
+ in->opnds[2] = reg_op(5u, rhs_type);
+ in->opnds[2].shift = rhs_shift;
+ in->extra.imm = BO_IADD;
+ return in;
+}
+
+static Inst* emit_store(CseFixture* fx, Reg src, KitCgTypeId type, u32 size) {
+ Inst* in = ir_emit(fx->f, fx->block, IR_STORE);
+ in->nopnds = 2;
+ in->opnds = arena_array(fx->f->arena, Operand, 2);
+ in->opnds[0] = global_op(2u, type);
+ in->opnds[1] = reg_op(src, type);
+ in->extra.mem.type = type;
+ in->extra.mem.size = size;
+ in->extra.mem.align = size;
+ in->extra.mem.alias.kind = ALIAS_GLOBAL;
+ in->extra.mem.alias.v.global = 2u;
+ return in;
+}
+
+static FrameSlot add_spill_slot(CseFixture* fx, KitCgTypeId type, u32 size,
+ u32 align) {
+ FrameSlotDesc desc;
+ memset(&desc, 0, sizeof desc);
+ desc.type = type;
+ desc.size = size;
+ desc.align = align;
+ desc.kind = FS_SPILL;
+ return ir_frame_slot_new(fx->f, &desc);
+}
+
+static Inst* emit_stack_copy(CseFixture* fx, Operand dst, Operand src) {
+ Inst* in = ir_emit(fx->f, fx->block, IR_COPY);
+ in->type = dst.type;
+ in->nopnds = 2u;
+ in->opnds = arena_array(fx->f->arena, Operand, 2u);
+ in->opnds[0] = dst;
+ in->opnds[1] = src;
+ return in;
+}
+
+static Inst* find_inst(CseFixture* fx, InstId id) {
+ Block* block = &fx->f->blocks[fx->block];
+ for (u32 i = 0; i < block->ninsts; ++i)
+ if (block->insts[i].id == id) return &block->insts[i];
+ return NULL;
+}
+
+static u32 count_op(const CseFixture* fx, IROp op) {
+ const Block* block = &fx->f->blocks[fx->block];
+ u32 count = 0;
+ for (u32 i = 0; i < block->ninsts; ++i)
+ if ((IROp)block->insts[i].op == op) ++count;
+ return count;
+}
+
+static int is_copy_between(const Inst* in, Reg dst, Reg src,
+ KitCgTypeId type) {
+ return in && (IROp)in->op == IR_COPY && in->flags == 0u &&
+ in->type == type && in->nopnds == 2u && in->opnds != NULL &&
+ in->opnds[0].kind == OPK_REG && in->opnds[0].cls == RC_INT &&
+ in->opnds[0].type == type && in->opnds[0].v.reg == dst &&
+ in->opnds[1].kind == OPK_REG && in->opnds[1].cls == RC_INT &&
+ in->opnds[1].type == type && in->opnds[1].v.reg == src;
+}
+
+static int has_reg_destination(const Inst* in, IROp op, Reg dst,
+ KitCgTypeId type) {
+ return in && (IROp)in->op == op && in->type == type && in->nopnds >= 1u &&
+ in->opnds != NULL && in->opnds[0].kind == OPK_REG &&
+ in->opnds[0].cls == RC_INT && in->opnds[0].type == type &&
+ in->opnds[0].v.reg == dst;
+}
+
+static int is_store_from(const Inst* in, Reg src, KitCgTypeId type) {
+ return in && (IROp)in->op == IR_STORE && in->nopnds == 2u &&
+ in->opnds != NULL && in->opnds[1].kind == OPK_REG &&
+ in->opnds[1].cls == RC_INT && in->opnds[1].type == type &&
+ in->opnds[1].v.reg == src;
+}
+
+static MemAccess byte_mem(const CseFixture* fx) {
+ MemAccess mem;
+ memset(&mem, 0, sizeof mem);
+ mem.type = fx->i8_type;
+ mem.size = 1u;
+ mem.align = 1u;
+ mem.alias.kind = ALIAS_GLOBAL;
+ mem.alias.v.global = 1u;
+ return mem;
+}
+
+static void exact_load_key_is_reused(KitUnit* u) {
+ CseFixture fx;
+ if (!fixture_init(u, &fx)) {
+ CU_EXPECT(u, 0, "compiler allocation failed for exact load CSE test");
+ return;
+ }
+ MemAccess mem = byte_mem(&fx);
+ InstId producer_id = emit_load(&fx, 1u, fx.i32_type, mem)->id;
+ InstId replacement_id = emit_load(&fx, 2u, fx.i32_type, mem)->id;
+ /* Make r2 observable. Combine must forward this memory write to the earlier
+ * loaded value, while any surviving r2 definition remains a valid copy. */
+ InstId store_id = emit_store(&fx, 2u, fx.i32_type, 4u)->id;
+
+ opt_combine(fx.f, NULL);
+ CU_EXPECT(u, count_op(&fx, IR_LOAD) == 1u,
+ "identical loads were not CSE'd");
+ {
+ Inst* producer = find_inst(&fx, producer_id);
+ Inst* replacement = find_inst(&fx, replacement_id);
+ Inst* store = find_inst(&fx, store_id);
+ CU_EXPECT(u,
+ has_reg_destination(producer, IR_LOAD, 1u, fx.i32_type) &&
+ is_store_from(store, 1u, fx.i32_type),
+ "load CSE did not preserve r2 as the destination and reuse "
+ "the value produced in r1 at its consumer");
+ CU_EXPECT(u,
+ !replacement ||
+ is_copy_between(replacement, 2u, 1u, fx.i32_type),
+ "surviving load-CSE replacement does not define r2 from r1");
+ }
+ kit_compiler_free(fx.kc);
+}
+
+static void load_flags_are_part_of_key(KitUnit* u) {
+ CseFixture fx;
+ if (!fixture_init(u, &fx)) {
+ CU_EXPECT(u, 0, "compiler allocation failed for load flag CSE test");
+ return;
+ }
+ MemAccess plain = byte_mem(&fx);
+ MemAccess sext = plain;
+ sext.flags |= MF_SEXT_LOAD;
+ emit_load(&fx, 1u, fx.i32_type, plain);
+ emit_load(&fx, 2u, fx.i32_type, sext);
+
+ opt_combine(fx.f, NULL);
+ CU_EXPECT(u, count_op(&fx, IR_LOAD) == 2u,
+ "load CSE discarded MF_SEXT_LOAD semantics");
+ kit_compiler_free(fx.kc);
+}
+
+static void load_result_type_is_part_of_key(KitUnit* u) {
+ CseFixture fx;
+ if (!fixture_init(u, &fx)) {
+ CU_EXPECT(u, 0, "compiler allocation failed for load result CSE test");
+ return;
+ }
+ MemAccess mem = byte_mem(&fx);
+ emit_load(&fx, 1u, fx.i32_type, mem);
+ emit_load(&fx, 2u, fx.i64_type, mem);
+
+ opt_combine(fx.f, NULL);
+ CU_EXPECT(u, count_op(&fx, IR_LOAD) == 2u,
+ "load CSE reused a differently typed result");
+ kit_compiler_free(fx.kc);
+}
+
+static void load_access_type_is_part_of_key(KitUnit* u) {
+ CseFixture fx;
+ if (!fixture_init(u, &fx)) {
+ CU_EXPECT(u, 0, "compiler allocation failed for access type CSE test");
+ return;
+ }
+ MemAccess byte = byte_mem(&fx);
+ MemAccess other = byte;
+ other.type = fx.i32_type;
+ emit_load(&fx, 1u, fx.i32_type, byte);
+ emit_load(&fx, 2u, fx.i32_type, other);
+
+ opt_combine(fx.f, NULL);
+ CU_EXPECT(u, count_op(&fx, IR_LOAD) == 2u,
+ "load CSE ignored the MemAccess object type");
+ kit_compiler_free(fx.kc);
+}
+
+static void load_alignment_is_part_of_key(KitUnit* u) {
+ CseFixture fx;
+ if (!fixture_init(u, &fx)) {
+ CU_EXPECT(u, 0, "compiler allocation failed for alignment CSE test");
+ return;
+ }
+ MemAccess byte = byte_mem(&fx);
+ MemAccess aligned = byte;
+ aligned.align = 4u;
+ emit_load(&fx, 1u, fx.i32_type, byte);
+ emit_load(&fx, 2u, fx.i32_type, aligned);
+
+ opt_combine(fx.f, NULL);
+ CU_EXPECT(u, count_op(&fx, IR_LOAD) == 2u,
+ "load CSE ignored MemAccess alignment metadata");
+ kit_compiler_free(fx.kc);
+}
+
+static void exact_compute_key_is_reused(KitUnit* u) {
+ CseFixture fx;
+ if (!fixture_init(u, &fx)) {
+ CU_EXPECT(u, 0, "compiler allocation failed for exact compute CSE test");
+ return;
+ }
+ InstId producer_id =
+ emit_add(&fx, 1u, fx.i64_type, fx.i64_type, fx.i64_type, 0u)->id;
+ InstId replacement_id =
+ emit_add(&fx, 2u, fx.i64_type, fx.i64_type, fx.i64_type, 0u)->id;
+ InstId store_id = emit_store(&fx, 2u, fx.i64_type, 8u)->id;
+
+ opt_combine(fx.f, NULL);
+ CU_EXPECT(u, count_op(&fx, IR_BINOP) == 1u,
+ "identical computes were not CSE'd");
+ {
+ Inst* producer = find_inst(&fx, producer_id);
+ Inst* replacement = find_inst(&fx, replacement_id);
+ Inst* store = find_inst(&fx, store_id);
+ CU_EXPECT(u,
+ has_reg_destination(producer, IR_BINOP, 1u, fx.i64_type) &&
+ is_store_from(store, 1u, fx.i64_type),
+ "compute CSE did not preserve r2 as the destination and reuse "
+ "the value produced in r1 at its consumer");
+ CU_EXPECT(u,
+ !replacement ||
+ is_copy_between(replacement, 2u, 1u, fx.i64_type),
+ "surviving compute-CSE replacement does not define r2 from r1");
+ }
+ kit_compiler_free(fx.kc);
+}
+
+static void location_mir_spill_uses_block_dse(KitUnit* u) {
+ CseFixture fx;
+ FrameSlot slot;
+
+ if (!fixture_init(u, &fx)) {
+ CU_EXPECT(u, 0, "compiler allocation failed for spill-use DSE guard");
+ return;
+ }
+ slot = add_spill_slot(&fx, fx.i64_type, 8u, 8u);
+ {
+ InstId first_id =
+ emit_stack_copy(&fx, stack_op(slot, fx.i64_type),
+ reg_op(1u, fx.i64_type))
+ ->id;
+ Inst* use = emit_add(&fx, 2u, fx.i64_type, fx.i64_type, fx.i64_type, 0u);
+ use->opnds[1] = stack_op(slot, fx.i64_type);
+ emit_stack_copy(&fx, stack_op(slot, fx.i64_type),
+ reg_op(3u, fx.i64_type));
+ opt_combine(fx.f, NULL);
+ CU_EXPECT(u, find_inst(&fx, first_id) != NULL,
+ "spill DSE crossed a direct OPK_STACK value use");
+ }
+ kit_compiler_free(fx.kc);
+
+ if (!fixture_init(u, &fx)) {
+ CU_EXPECT(u, 0,
+ "compiler allocation failed for indirect spill-use DSE guard");
+ return;
+ }
+ slot = add_spill_slot(&fx, fx.i64_type, 8u, 8u);
+ {
+ InstId first_id =
+ emit_stack_copy(&fx, stack_op(slot, fx.i64_type),
+ reg_op(1u, fx.i64_type))
+ ->id;
+ MemAccess mem = byte_mem(&fx);
+ Inst* load = emit_load(&fx, 2u, fx.i64_type, mem);
+ memset(&load->opnds[1], 0, sizeof load->opnds[1]);
+ load->opnds[1].kind = OPK_INDIRECT;
+ load->opnds[1].cls = RC_INT;
+ load->opnds[1].type = fx.i64_type;
+ load->opnds[1].v.ind.base_kind = OPT_INDIRECT_FRAME;
+ load->opnds[1].v.ind.base = slot;
+ load->opnds[1].v.ind.base_type = fx.i64_type;
+ load->opnds[1].v.ind.index_kind = OPT_INDIRECT_REG;
+ load->opnds[1].v.ind.index = (Reg)REG_NONE;
+ emit_stack_copy(&fx, stack_op(slot, fx.i64_type),
+ reg_op(3u, fx.i64_type));
+ opt_combine(fx.f, NULL);
+ CU_EXPECT(u, find_inst(&fx, first_id) != NULL,
+ "spill DSE crossed an indirect frame-component use");
+ }
+ kit_compiler_free(fx.kc);
+}
+
+static void compute_result_type_is_part_of_key(KitUnit* u) {
+ CseFixture fx;
+ if (!fixture_init(u, &fx)) {
+ CU_EXPECT(u, 0, "compiler allocation failed for compute result CSE test");
+ return;
+ }
+ emit_add(&fx, 1u, fx.i32_type, fx.i64_type, fx.i64_type, 0u);
+ emit_add(&fx, 2u, fx.i64_type, fx.i64_type, fx.i64_type, 0u);
+
+ opt_combine(fx.f, NULL);
+ CU_EXPECT(u, count_op(&fx, IR_BINOP) == 2u,
+ "compute CSE reused a differently typed destination");
+ kit_compiler_free(fx.kc);
+}
+
+static void compute_reg_input_type_is_part_of_key(KitUnit* u) {
+ CseFixture fx;
+ if (!fixture_init(u, &fx)) {
+ CU_EXPECT(u, 0, "compiler allocation failed for compute input CSE test");
+ return;
+ }
+ emit_add(&fx, 1u, fx.i64_type, fx.i64_type, fx.i32_type, 0u);
+ emit_add(&fx, 2u, fx.i64_type, fx.i64_type, fx.i64_type, 0u);
+
+ opt_combine(fx.f, NULL);
+ CU_EXPECT(u, count_op(&fx, IR_BINOP) == 2u,
+ "compute CSE ignored a register operand's type");
+ kit_compiler_free(fx.kc);
+}
+
+static void compute_shift_rider_is_part_of_key(KitUnit* u) {
+ CseFixture fx;
+ if (!fixture_init(u, &fx)) {
+ CU_EXPECT(u, 0, "compiler allocation failed for compute rider CSE test");
+ return;
+ }
+ emit_add(&fx, 1u, fx.i64_type, fx.i64_type, fx.i64_type, 0u);
+ emit_add(&fx, 2u, fx.i64_type, fx.i64_type, fx.i64_type, 2u);
+
+ opt_combine(fx.f, NULL);
+ CU_EXPECT(u, count_op(&fx, IR_BINOP) == 2u,
+ "compute CSE ignored a shifted-register rider");
+ kit_compiler_free(fx.kc);
+}
+
+static void compute_semantic_flags_are_part_of_key(KitUnit* u) {
+ CseFixture fx;
+ if (!fixture_init(u, &fx)) {
+ CU_EXPECT(u, 0, "compiler allocation failed for compute flag CSE test");
+ return;
+ }
+ Inst* portable = emit_add(&fx, 1u, fx.i64_type, fx.i64_type, fx.i64_type,
+ 0u);
+ Inst* target = emit_add(&fx, 2u, fx.i64_type, fx.i64_type, fx.i64_type, 0u);
+ portable->extra.imm = BO_SHL;
+ target->extra.imm = BO_SHL;
+ target->flags = CG_IR_INST_TARGET_SHIFT_EDGES;
+
+ opt_combine(fx.f, NULL);
+ CU_EXPECT(u, count_op(&fx, IR_BINOP) == 2u,
+ "compute CSE ignored per-instruction semantic flags");
+ kit_compiler_free(fx.kc);
+}
+
+static void location_mir_spill_copies_are_compacted(KitUnit* u) {
+ CseFixture fx;
+ FrameSlot slot;
+ Inst* replacement;
+ InstId replacement_id;
+
+ if (!fixture_init(u, &fx)) {
+ CU_EXPECT(u, 0, "compiler allocation failed for spill-copy forwarding");
+ return;
+ }
+ slot = add_spill_slot(&fx, fx.i64_type, 8u, 8u);
+ emit_stack_copy(&fx, stack_op(slot, fx.i64_type),
+ reg_op(1u, fx.i64_type));
+ replacement_id =
+ emit_stack_copy(&fx, reg_op(2u, fx.i64_type),
+ stack_op(slot, fx.i64_type))
+ ->id;
+ opt_combine(fx.f, NULL);
+ replacement = find_inst(&fx, replacement_id);
+ CU_EXPECT(u,
+ replacement &&
+ is_copy_between(replacement, 2u, 1u, fx.i64_type),
+ "location-MIR spill reload was not forwarded from the stored "
+ "register");
+ kit_compiler_free(fx.kc);
+
+ if (!fixture_init(u, &fx)) {
+ CU_EXPECT(u, 0, "compiler allocation failed for spill-copy elision");
+ return;
+ }
+ slot = add_spill_slot(&fx, fx.i64_type, 8u, 8u);
+ emit_stack_copy(&fx, stack_op(slot, fx.i64_type),
+ reg_op(1u, fx.i64_type));
+ replacement_id =
+ emit_stack_copy(&fx, reg_op(1u, fx.i64_type),
+ stack_op(slot, fx.i64_type))
+ ->id;
+ opt_combine(fx.f, NULL);
+ CU_EXPECT(u, find_inst(&fx, replacement_id) == NULL,
+ "location-MIR same-register spill reload survived compaction");
+ kit_compiler_free(fx.kc);
+
+ if (!fixture_init(u, &fx)) {
+ CU_EXPECT(u, 0, "compiler allocation failed for spill overwrite DSE");
+ return;
+ }
+ slot = add_spill_slot(&fx, fx.i64_type, 8u, 8u);
+ {
+ InstId first_id =
+ emit_stack_copy(&fx, stack_op(slot, fx.i64_type),
+ reg_op(1u, fx.i64_type))
+ ->id;
+ InstId second_id =
+ emit_stack_copy(&fx, stack_op(slot, fx.i64_type),
+ reg_op(2u, fx.i64_type))
+ ->id;
+ opt_combine(fx.f, NULL);
+ CU_EXPECT(u,
+ find_inst(&fx, first_id) == NULL &&
+ find_inst(&fx, second_id) != NULL,
+ "location-MIR spill overwrite did not retire the dead store");
+ }
+ kit_compiler_free(fx.kc);
+}
+
+int main(void) {
+ KitUnit u;
+ kit_unit_init(&u);
+ exact_load_key_is_reused(&u);
+ load_flags_are_part_of_key(&u);
+ load_result_type_is_part_of_key(&u);
+ load_access_type_is_part_of_key(&u);
+ load_alignment_is_part_of_key(&u);
+ exact_compute_key_is_reused(&u);
+ compute_result_type_is_part_of_key(&u);
+ compute_reg_input_type_is_part_of_key(&u);
+ compute_shift_rider_is_part_of_key(&u);
+ compute_semantic_flags_are_part_of_key(&u);
+ location_mir_spill_copies_are_compacted(&u);
+ location_mir_spill_uses_block_dse(&u);
+ kit_unit_summary(&u, "combine-cse");
+ return kit_unit_status(&u);
+}
diff --git a/test/opt/frame_value_backend_test.c b/test/opt/frame_value_backend_test.c
@@ -0,0 +1,766 @@
+/* Real-backend regression for typed FRAME_VALUE address components.
+ *
+ * Spill-slot coloring is allowed to reuse a physical frame slot across values
+ * with different type IDs. NativeAddr therefore carries base_type/index_type
+ * as the value-width authority: a backend must not infer a reload width from
+ * the (possibly wider) slot descriptor. Build two deliberately-wide i64
+ * slots, consume them as a pointer base plus an i32 scaled index, and inspect
+ * only the body bytes emitted by each native backend.
+ *
+ * This also pins the RV64 load_addr path: it used to load the frame base but
+ * silently omit an INDEX_FRAME_VALUE. The final sh2add proves the index
+ * participates in the address.
+ */
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+
+#include "arch/aa64/aa64.h"
+#include "arch/arch.h"
+#include "arch/arm32/arm32.h"
+#include "arch/riscv/rv64.h"
+#include "arch/x64/isa.h"
+#include "arch/x64/x64.h"
+#include "cg/type.h"
+#include "core/pool.h"
+#include "lib/kit_unit.h"
+#include "obj/obj.h"
+
+typedef NativeTarget *(*NativeCtor)(Compiler *, ObjBuilder *, MCEmitter *);
+
+typedef struct DecodedInsn {
+ char mnemonic[20];
+ char operands[96];
+} DecodedInsn;
+
+typedef struct BackendCase {
+ KitArchKind arch;
+ const char *name;
+ NativeCtor ctor;
+ Reg dst_reg;
+ u8 ptr_size;
+} BackendCase;
+
+static void slice_copy(KitSlice src, char *dst, size_t cap) {
+ size_t n = src.len < cap - 1u ? src.len : cap - 1u;
+ if (n && src.s)
+ memcpy(dst, src.s, n);
+ dst[n] = '\0';
+}
+
+static u32 decode_body(KitUnit *unit, Compiler *c, ObjBuilder *obj,
+ ObjSecId text, u32 begin, u32 end, DecodedInsn *out,
+ u32 cap, const char *tag) {
+ const Section *sec = obj_section_get(obj, text);
+ ArchDisasm *dis = NULL;
+ u8 *flat = NULL;
+ u32 size;
+ u32 pos = begin;
+ u32 nout = 0;
+
+ CU_CHECK(unit, sec != NULL, "%s: missing text section", tag);
+ if (!sec)
+ return 0;
+ size = buf_pos(&sec->bytes);
+ CU_CHECK(unit, begin <= end && end <= size,
+ "%s: invalid body range [%u,%u) in %u-byte text", tag, begin, end,
+ size);
+ if (begin > end || end > size)
+ return 0;
+ flat = size ? (u8 *)malloc(size) : NULL;
+ CU_CHECK(unit, size == 0u || flat != NULL,
+ "%s: could not allocate flattened text", tag);
+ if (size && !flat)
+ return 0;
+ if (size)
+ buf_flatten(&sec->bytes, flat);
+ dis = arch_disasm_new(c);
+ CU_CHECK(unit, dis != NULL, "%s: could not create disassembler", tag);
+ if (!dis) {
+ free(flat);
+ return 0;
+ }
+
+ while (pos < end) {
+ KitInsn insn;
+ u32 used;
+ memset(&insn, 0, sizeof insn);
+ used = arch_disasm_decode(dis, flat + pos, end - pos, pos, &insn);
+ CU_CHECK(unit, used != 0u, "%s: undecodable body byte at offset %u", tag,
+ pos);
+ if (!used)
+ break;
+ CU_CHECK(unit, nout < cap, "%s: body exceeds %u decoded instructions", tag,
+ cap);
+ if (nout < cap) {
+ slice_copy(insn.mnemonic, out[nout].mnemonic, sizeof out[nout].mnemonic);
+ slice_copy(insn.operands, out[nout].operands, sizeof out[nout].operands);
+ ++nout;
+ }
+ pos += used;
+ }
+ CU_EXPECT(unit, pos == end, "%s: decoder stopped at %u, body ends at %u", tag,
+ pos, end);
+ arch_disasm_free(dis);
+ free(flat);
+ return nout;
+}
+
+static int has_insn(const DecodedInsn *insns, u32 ninsns, const char *mnemonic,
+ const char *operand) {
+ for (u32 i = 0; i < ninsns; ++i) {
+ if (mnemonic && !strstr(insns[i].mnemonic, mnemonic))
+ continue;
+ if (operand && !strstr(insns[i].operands, operand))
+ continue;
+ return 1;
+ }
+ return 0;
+}
+
+static i32 find_insn(const DecodedInsn *insns, u32 ninsns,
+ const char *mnemonic, const char *operand) {
+ for (u32 i = 0; i < ninsns; ++i) {
+ if (mnemonic && !strstr(insns[i].mnemonic, mnemonic))
+ continue;
+ if (operand && !strstr(insns[i].operands, operand))
+ continue;
+ return (i32)i;
+ }
+ return -1;
+}
+
+static u32 count_insns(const DecodedInsn *insns, u32 ninsns,
+ const char *mnemonic) {
+ u32 count = 0;
+ for (u32 i = 0; i < ninsns; ++i)
+ if (strstr(insns[i].mnemonic, mnemonic))
+ ++count;
+ return count;
+}
+
+static u32 collect_relocs_to(const ObjBuilder *obj, ObjSecId section,
+ u32 begin, u32 end, ObjSymId sym, Reloc *out,
+ u32 cap) {
+ u32 n = 0;
+ for (u32 i = 0; i < obj_reloc_total(obj); ++i) {
+ const Reloc *r = obj_reloc_at(obj, i);
+ if (!r || r->removed || r->section_id != section || r->offset < begin ||
+ r->offset >= end || r->sym != sym)
+ continue;
+ if (n < cap)
+ out[n] = *r;
+ ++n;
+ }
+ return n;
+}
+
+static void check_location_address_roles(KitUnit *unit) {
+ NativeLoc storage = {0};
+ NativeLoc frame_addr = {0};
+ NativeLoc global = {0};
+ NativeLoc imm = {0};
+ NativeAddr addr;
+ i64 part = 0;
+
+ storage.kind = NATIVE_LOC_STACK;
+ storage.type = 11u;
+ storage.v.stack.slot = 7u;
+ storage.v.stack.offset = 5;
+ CU_EXPECT(unit,
+ native_loc_addr_role(storage) == NATIVE_LOC_ADDR_ROLE_STORAGE &&
+ native_loc_storage_addr(storage, 3, &addr) &&
+ addr.base_kind == NATIVE_ADDR_BASE_FRAME &&
+ addr.base.frame == 7u && addr.offset == 8,
+ "STACK must resolve only as offset storage");
+ CU_EXPECT(unit, !native_loc_address_value(storage, &addr),
+ "storage location was accepted as an address value");
+
+ frame_addr.kind = NATIVE_LOC_FRAME_ADDR;
+ frame_addr.type = 12u;
+ frame_addr.v.frame = 9u;
+ CU_EXPECT(unit,
+ native_loc_addr_role(frame_addr) == NATIVE_LOC_ADDR_ROLE_VALUE &&
+ native_loc_address_value(frame_addr, &addr) &&
+ addr.base_kind == NATIVE_ADDR_BASE_FRAME &&
+ addr.base.frame == 9u && addr.offset == 0,
+ "FRAME_ADDR must resolve only as a scalar address recipe");
+ CU_EXPECT(unit, !native_loc_storage_addr(frame_addr, 0, &addr),
+ "frame address value was accepted as storage");
+
+ global.kind = NATIVE_LOC_GLOBAL;
+ global.type = 13u;
+ global.v.global.sym = 17u;
+ global.v.global.addend = 37;
+ CU_EXPECT(unit,
+ native_loc_address_value(global, &addr) &&
+ addr.base_kind == NATIVE_ADDR_BASE_GLOBAL &&
+ addr.base.global.sym == 17u &&
+ addr.base.global.addend == 37,
+ "GLOBAL address recipe lost its symbol/addend");
+ CU_EXPECT(unit, !native_loc_storage_addr(global, 0, &addr),
+ "global address value was accepted as storage");
+
+ imm.kind = NATIVE_LOC_IMM;
+ imm.v.imm = (i64)0x1122334455667788ull;
+ CU_EXPECT(unit,
+ native_loc_imm_part(imm, 4u, 4u, &part) &&
+ (u64)part == 0x11223344u,
+ "immediate ABI part selection returned the wrong high word");
+ CU_EXPECT(unit, !native_loc_imm_part(imm, 7u, 2u, &part),
+ "immediate ABI part selection accepted an out-of-bounds lane");
+}
+
+static void check_typed_frame_address(KitUnit *unit, const BackendCase *tc) {
+ KitTargetSpec spec = kit_unit_target(tc->arch, KIT_OS_LINUX, KIT_OBJ_ELF);
+ KitCompiler *kit_c = NULL;
+ Compiler *c;
+ ObjBuilder *obj = NULL;
+ MCEmitter *mc = NULL;
+ NativeTarget *native;
+ ObjSecId text, data;
+ ObjSymId sym, global_sym;
+ KitCgTypeId void_type, i8_type, i32_type, i64_type, ptr_type, tiny3_type,
+ block8_type, fn_type, call_fn_type, wide_call_fn_type,
+ exact_call_fn_type, forced_vararg_fn_type;
+ KitCgFuncParam call_param, exact_call_param;
+ KitCgFuncResult result;
+ KitCgFuncSig sig, call_sig, forced_vararg_sig;
+ CGFuncDesc func;
+ NativeFrameSlotDesc slot_descs[2];
+ NativeKnownFrameDesc frame;
+ NativeFrameSlot slots[2];
+ NativeAddr addr;
+ NativeLoc dst;
+ NativeLoc call_arg;
+ NativeLoc exact_arg;
+ NativeCallDesc call_desc;
+ NativeCallPlan call_plan;
+ DecodedInsn body[12];
+ DecodedInsn call_body[8];
+ DecodedInsn global_body[12];
+ DecodedInsn imm_body[12];
+ DecodedInsn exact_call_body[24];
+ DecodedInsn forced_vararg_body[32];
+ DecodedInsn copy_frame_value_body[16];
+ DecodedInsn set_frame_value_body[24];
+ DecodedInsn indexed_frame_value_body[12];
+ Reloc global_relocs[4];
+ u32 body_begin, body_end, nbody;
+ u32 call_begin, call_end, ncall;
+ u32 global_begin, global_end, nglobal, nglobal_relocs;
+ u32 imm_begin = 0, imm_end = 0, nimm = 0;
+ u32 exact_call_begin = 0, exact_call_end = 0, nexact_call = 0;
+ u32 forced_vararg_begin = 0, forced_vararg_end = 0, nforced_vararg = 0;
+ u32 copy_frame_value_begin = 0, copy_frame_value_end = 0,
+ ncopy_frame_value = 0;
+ u32 set_frame_value_begin = 0, set_frame_value_end = 0,
+ nset_frame_value = 0;
+ u32 indexed_frame_value_begin = 0, indexed_frame_value_end = 0,
+ nindexed_frame_value = 0;
+
+ spec.ptr_size = tc->ptr_size;
+ spec.ptr_align = tc->ptr_size;
+ CU_CHECK(unit, kit_unit_compiler_new(unit, spec, &kit_c) == KIT_OK && kit_c,
+ "%s: compiler allocation failed", tc->name);
+ if (!kit_c)
+ return;
+ c = (Compiler *)kit_c;
+ obj = obj_new(c);
+ CU_CHECK(unit, obj != NULL, "%s: object allocation failed", tc->name);
+ if (!obj)
+ goto done;
+ mc = mc_new(c, obj);
+ CU_CHECK(unit, mc != NULL, "%s: MC allocation failed", tc->name);
+ if (!mc)
+ goto done;
+ native = tc->ctor(c, obj, mc);
+ CU_CHECK(unit, native != NULL, "%s: native target allocation failed",
+ tc->name);
+ if (!native)
+ goto done;
+
+ text = obj_section(obj, pool_intern_slice(c->global, SLICE_LIT(".text")),
+ SEC_TEXT, SF_EXEC | SF_ALLOC, 16u);
+ data = obj_section(obj, pool_intern_slice(c->global, SLICE_LIT(".data")),
+ SEC_DATA, SF_WRITE | SF_ALLOC, 16u);
+ sym = obj_symbol(obj, pool_intern_slice(c->global, SLICE_LIT("frame_value")),
+ SB_GLOBAL, SK_FUNC, text, 0u, 0u);
+ global_sym = obj_symbol(
+ obj, pool_intern_slice(c->global, SLICE_LIT("address_value_global")),
+ SB_LOCAL, SK_OBJ, data, 0u, 64u);
+ void_type = kit_cg_type_builtin(kit_c, KIT_CG_BUILTIN_VOID);
+ i8_type = kit_cg_type_builtin(kit_c, KIT_CG_BUILTIN_I8);
+ i32_type = kit_cg_type_builtin(kit_c, KIT_CG_BUILTIN_I32);
+ i64_type = kit_cg_type_builtin(kit_c, KIT_CG_BUILTIN_I64);
+ ptr_type = kit_cg_type_ptr(kit_c, void_type, 0u);
+ tiny3_type = kit_cg_type_array(kit_c, i8_type, 3u);
+ block8_type = kit_cg_type_array(kit_c, i8_type, 8u);
+ memset(&result, 0, sizeof result);
+ result.type = void_type;
+ memset(&sig, 0, sizeof sig);
+ sig.result = result;
+ sig.call_conv = KIT_CG_CC_TARGET_C;
+ fn_type = kit_cg_type_func(kit_c, sig);
+ memset(&call_param, 0, sizeof call_param);
+ call_param.type = ptr_type;
+ memset(&call_sig, 0, sizeof call_sig);
+ call_sig.result = result;
+ call_sig.params = &call_param;
+ call_sig.nparams = 1u;
+ call_sig.call_conv = KIT_CG_CC_TARGET_C;
+ call_fn_type = kit_cg_type_func(kit_c, call_sig);
+ call_param.type = i64_type;
+ wide_call_fn_type = kit_cg_type_func(kit_c, call_sig);
+ memset(&exact_call_param, 0, sizeof exact_call_param);
+ exact_call_param.type = tiny3_type;
+ call_sig.params = &exact_call_param;
+ exact_call_fn_type = kit_cg_type_func(kit_c, call_sig);
+ memset(&forced_vararg_sig, 0, sizeof forced_vararg_sig);
+ forced_vararg_sig.result = result;
+ forced_vararg_sig.call_conv = KIT_CG_CC_TARGET_C;
+ forced_vararg_sig.abi_variadic = true;
+ forced_vararg_fn_type = kit_cg_type_func(kit_c, forced_vararg_sig);
+
+ memset(&func, 0, sizeof func);
+ func.sym = sym;
+ func.text_section_id = text;
+ func.fn_type = fn_type;
+ func.result_type = void_type;
+
+ /* Both colored homes advertise i64/8. The address contract below is the
+ * authority: pointer-width base, four-byte index. */
+ memset(slot_descs, 0, sizeof slot_descs);
+ for (u32 i = 0; i < 2u; ++i) {
+ slot_descs[i].type = i64_type;
+ slot_descs[i].size = 8u;
+ slot_descs[i].align = 8u;
+ slot_descs[i].kind = NATIVE_FRAME_SLOT_SPILL;
+ }
+ memset(&frame, 0, sizeof frame);
+ frame.slots = slot_descs;
+ frame.nslots = 2u;
+ /* The RV-only forced-vararg construction below owns one rounded 16-byte
+ * outgoing region. Reserve it before known-frame finalization just as the
+ * optimizer's frame pre-pass does for a real call. */
+ if (tc->arch == KIT_ARCH_RV64) frame.max_outgoing = 16u;
+ /* Keep a real frame record on every target so frame-relative body reloads
+ * have a settled anchor independent of leaf-frame elision. */
+ frame.is_leaf = 0u;
+ native->func_begin_known_frame(native, &func, &frame, slots);
+
+ memset(&addr, 0, sizeof addr);
+ addr.base_kind = NATIVE_ADDR_BASE_FRAME_VALUE;
+ addr.base.frame = slots[0];
+ addr.base_type = ptr_type;
+ addr.cls = NATIVE_REG_INT;
+ addr.index_kind = NATIVE_ADDR_INDEX_FRAME_VALUE;
+ addr.index.frame = slots[1];
+ addr.index_type = i32_type;
+ addr.index_cls = NATIVE_REG_INT;
+ addr.log2_scale = 2u;
+ addr.offset = 12;
+ dst = native_loc_reg(ptr_type, NATIVE_REG_INT, tc->dst_reg);
+ body_begin = mc_pos(mc);
+ native->load_addr(native, dst, addr);
+ body_end = mc_pos(mc);
+
+ /* A post-allocation rematerialization may present `&local` directly to the
+ * ABI marshaller. It is an address value, not the scalar stored in the
+ * frame slot, and must not require an optimizer-owned staging register. */
+ memset(&call_arg, 0, sizeof call_arg);
+ call_arg.kind = NATIVE_LOC_FRAME_ADDR;
+ call_arg.cls = NATIVE_REG_INT;
+ call_arg.type = ptr_type;
+ call_arg.v.frame = slots[0];
+ memset(&call_desc, 0, sizeof call_desc);
+ call_desc.fn_type = call_fn_type;
+ call_desc.args = &call_arg;
+ call_desc.nargs = 1u;
+ memset(&call_plan, 0, sizeof call_plan);
+ call_begin = mc_pos(mc);
+ native->plan_call(native, &call_desc, &call_plan);
+ call_end = mc_pos(mc);
+
+ /* A GLOBAL NativeLoc is the scalar address value `&symbol + addend`, never
+ * the bytes stored at that address. Exercise the ABI boundary directly:
+ * ordinary C currently materializes globals before constructing its call
+ * descriptor, so a source-level call cannot pin this lower-level contract. */
+ memset(&call_arg, 0, sizeof call_arg);
+ call_arg.kind = NATIVE_LOC_GLOBAL;
+ call_arg.cls = NATIVE_REG_INT;
+ call_arg.type = ptr_type;
+ call_arg.v.global.sym = global_sym;
+ call_arg.v.global.addend = 37;
+ memset(&call_desc, 0, sizeof call_desc);
+ call_desc.fn_type = call_fn_type;
+ call_desc.args = &call_arg;
+ call_desc.nargs = 1u;
+ memset(&call_plan, 0, sizeof call_plan);
+ global_begin = mc_pos(mc);
+ native->plan_call(native, &call_desc, &call_plan);
+ global_end = mc_pos(mc);
+
+ /* ARM32 splits an i64 direct argument into low/high 32-bit ABI lanes. A
+ * NativeLoc immediate denotes the complete scalar, so each lane must select
+ * its own source bits instead of rematerializing the low word twice. */
+ if (tc->arch == KIT_ARCH_ARM_32) {
+ memset(&call_arg, 0, sizeof call_arg);
+ call_arg.kind = NATIVE_LOC_IMM;
+ call_arg.cls = NATIVE_REG_INT;
+ call_arg.type = i64_type;
+ call_arg.v.imm = (i64)0x1122334455667788ull;
+ memset(&call_desc, 0, sizeof call_desc);
+ call_desc.fn_type = wide_call_fn_type;
+ call_desc.args = &call_arg;
+ call_desc.nargs = 1u;
+ memset(&call_plan, 0, sizeof call_plan);
+ imm_begin = mc_pos(mc);
+ native->plan_call(native, &call_desc, &call_plan);
+ imm_end = mc_pos(mc);
+ }
+
+ /* x64 lowerers may legally hand the ABI boundary an exact-width aggregate
+ * in NATIVE_LOC_ADDR storage whose explicit base is also its destination ABI
+ * register. Keep a register-indirect callee live across that whole marshal:
+ * the 3-byte argument forces address stabilization through r11, which was
+ * formerly also the parked call target. This backend-level construction pins
+ * the allowed shape even when current source lowering chooses a frame home. */
+ if (tc->arch == KIT_ARCH_X86_64) {
+ memset(&exact_arg, 0, sizeof exact_arg);
+ exact_arg.kind = NATIVE_LOC_ADDR;
+ exact_arg.cls = NATIVE_REG_INT;
+ exact_arg.type = tiny3_type;
+ exact_arg.v.addr.base_kind = NATIVE_ADDR_BASE_REG;
+ exact_arg.v.addr.base.reg = X64_RDI;
+ exact_arg.v.addr.base_type = ptr_type;
+ exact_arg.v.addr.cls = NATIVE_REG_INT;
+ memset(&call_desc, 0, sizeof call_desc);
+ call_desc.fn_type = exact_call_fn_type;
+ call_desc.callee = native_loc_reg(ptr_type, NATIVE_REG_INT, X64_RCX);
+ call_desc.args = &exact_arg;
+ call_desc.nargs = 1u;
+ memset(&call_plan, 0, sizeof call_plan);
+ exact_call_begin = mc_pos(mc);
+ native->plan_call(native, &call_desc, &call_plan);
+ native->emit_call(native, &call_plan);
+ exact_call_end = mc_pos(mc);
+ }
+
+ /* RISC-V does not currently select vararg_on_stack, but it still implements
+ * the shared ABI trait and must not encode a latent rounded-slot overread.
+ * Synthesize that trait for one three-byte unnamed argument: the physical
+ * carrier is eight bytes (and the whole outgoing area rounds to 16), while
+ * source and destination accesses must remain exactly 2+1 bytes. Keep the
+ * indirect target in a STACK home at the same time: current lowering usually
+ * materializes that shape before this boundary, but NativeCallDesc permits it
+ * and the backend must park it outside the exact/address temporary bank. */
+ if (tc->arch == KIT_ARCH_RV64) {
+ const ABIFuncInfo* classified =
+ abi_cg_func_info(c->abi, forced_vararg_fn_type);
+ ABIFuncInfo* synthetic = (ABIFuncInfo*)classified;
+ u8 saved_vararg_on_stack;
+ CU_CHECK(unit, synthetic != NULL,
+ "rv64: variadic function classification failed");
+ if (synthetic) {
+ saved_vararg_on_stack = synthetic->vararg_on_stack;
+ synthetic->vararg_on_stack = 1u;
+ exact_arg = native_loc_stack(tiny3_type, slots[0], 0);
+ memset(&call_desc, 0, sizeof call_desc);
+ call_desc.fn_type = forced_vararg_fn_type;
+ call_desc.callee = native_loc_stack(ptr_type, slots[1], 0);
+ call_desc.args = &exact_arg;
+ call_desc.nargs = 1u;
+ memset(&call_plan, 0, sizeof call_plan);
+ forced_vararg_begin = mc_pos(mc);
+ native->plan_call(native, &call_desc, &call_plan);
+ native->emit_call(native, &call_plan);
+ forced_vararg_end = mc_pos(mc);
+ synthetic->vararg_on_stack = saved_vararg_on_stack;
+ }
+ }
+
+ /* The AArch64 aggregate primitives carry their payload in x16. A
+ * FRAME_VALUE destination must therefore materialize its pointer through a
+ * different address scratch. Otherwise both copy_bytes and widened
+ * set_bytes degenerate to `str x16, [x16]`, storing the pointer value rather
+ * than the aggregate payload. Keep this at the backend boundary so the
+ * contract stays independent of optimizer register-allocation choices. */
+ if (tc->arch == KIT_ARCH_ARM_64) {
+ AggregateAccess access;
+ NativeAddr frame_value_dst;
+ NativeAddr frame_src;
+ NativeLoc fill;
+ MemAccess indexed_mem;
+ memset(&access, 0, sizeof access);
+ access.type = block8_type;
+ access.size = 8u;
+ access.align = 8u;
+ access.mem.type = block8_type;
+ access.mem.size = 8u;
+ access.mem.align = 8u;
+ memset(&frame_value_dst, 0, sizeof frame_value_dst);
+ frame_value_dst.base_kind = NATIVE_ADDR_BASE_FRAME_VALUE;
+ frame_value_dst.base.frame = slots[0];
+ frame_value_dst.base_type = ptr_type;
+ frame_value_dst.cls = NATIVE_REG_INT;
+ memset(&frame_src, 0, sizeof frame_src);
+ frame_src.base_kind = NATIVE_ADDR_BASE_FRAME;
+ frame_src.base.frame = slots[1];
+ frame_src.base_type = ptr_type;
+ frame_src.cls = NATIVE_REG_INT;
+ fill = native_loc_reg(i8_type, NATIVE_REG_INT, 9u);
+
+ copy_frame_value_begin = mc_pos(mc);
+ native->copy_bytes(native, frame_value_dst, frame_src, access);
+ copy_frame_value_end = mc_pos(mc);
+ set_frame_value_begin = mc_pos(mc);
+ native->set_bytes(native, frame_value_dst, fill, access);
+ set_frame_value_end = mc_pos(mc);
+
+ /* A register index remains live until the final addressing-mode
+ * instruction. Pin x16 as that index and require base materialization to
+ * select x17 even when the store payload itself uses neither scratch. */
+ frame_value_dst.index_kind = NATIVE_ADDR_INDEX_REG;
+ frame_value_dst.index.reg = 16u;
+ frame_value_dst.index_type = i64_type;
+ frame_value_dst.index_cls = NATIVE_REG_INT;
+ memset(&indexed_mem, 0, sizeof indexed_mem);
+ indexed_mem.type = i64_type;
+ indexed_mem.size = 8u;
+ indexed_mem.align = 8u;
+ indexed_frame_value_begin = mc_pos(mc);
+ native->store(native, frame_value_dst,
+ native_loc_reg(i64_type, NATIVE_REG_INT, 9u), indexed_mem);
+ indexed_frame_value_end = mc_pos(mc);
+ }
+ native->func_end(native);
+
+ memset(body, 0, sizeof body);
+ nbody = decode_body(unit, c, obj, text, body_begin, body_end, body,
+ (u32)(sizeof body / sizeof body[0]), tc->name);
+ memset(call_body, 0, sizeof call_body);
+ ncall = decode_body(unit, c, obj, text, call_begin, call_end, call_body,
+ (u32)(sizeof call_body / sizeof call_body[0]), tc->name);
+ memset(global_body, 0, sizeof global_body);
+ nglobal = decode_body(unit, c, obj, text, global_begin, global_end,
+ global_body,
+ (u32)(sizeof global_body / sizeof global_body[0]),
+ tc->name);
+ memset(global_relocs, 0, sizeof global_relocs);
+ nglobal_relocs = collect_relocs_to(
+ obj, text, global_begin, global_end, global_sym, global_relocs,
+ (u32)(sizeof global_relocs / sizeof global_relocs[0]));
+ if (tc->arch == KIT_ARCH_ARM_32) {
+ memset(imm_body, 0, sizeof imm_body);
+ nimm = decode_body(unit, c, obj, text, imm_begin, imm_end, imm_body,
+ (u32)(sizeof imm_body / sizeof imm_body[0]), tc->name);
+ }
+ if (tc->arch == KIT_ARCH_X86_64) {
+ memset(exact_call_body, 0, sizeof exact_call_body);
+ nexact_call = decode_body(
+ unit, c, obj, text, exact_call_begin, exact_call_end, exact_call_body,
+ (u32)(sizeof exact_call_body / sizeof exact_call_body[0]), tc->name);
+ }
+ if (tc->arch == KIT_ARCH_RV64) {
+ memset(forced_vararg_body, 0, sizeof forced_vararg_body);
+ nforced_vararg = decode_body(
+ unit, c, obj, text, forced_vararg_begin, forced_vararg_end,
+ forced_vararg_body,
+ (u32)(sizeof forced_vararg_body / sizeof forced_vararg_body[0]),
+ tc->name);
+ }
+ if (tc->arch == KIT_ARCH_ARM_64) {
+ memset(copy_frame_value_body, 0, sizeof copy_frame_value_body);
+ ncopy_frame_value = decode_body(
+ unit, c, obj, text, copy_frame_value_begin, copy_frame_value_end,
+ copy_frame_value_body,
+ (u32)(sizeof copy_frame_value_body / sizeof copy_frame_value_body[0]),
+ "aa64-copy-frame-value");
+ memset(set_frame_value_body, 0, sizeof set_frame_value_body);
+ nset_frame_value = decode_body(
+ unit, c, obj, text, set_frame_value_begin, set_frame_value_end,
+ set_frame_value_body,
+ (u32)(sizeof set_frame_value_body / sizeof set_frame_value_body[0]),
+ "aa64-set-frame-value");
+ memset(indexed_frame_value_body, 0, sizeof indexed_frame_value_body);
+ nindexed_frame_value = decode_body(
+ unit, c, obj, text, indexed_frame_value_begin,
+ indexed_frame_value_end, indexed_frame_value_body,
+ (u32)(sizeof indexed_frame_value_body /
+ sizeof indexed_frame_value_body[0]),
+ "aa64-indexed-frame-value");
+ }
+ CU_EXPECT(unit, nbody >= 3u, "%s: expected typed base/index/add sequence",
+ tc->name);
+ CU_EXPECT(unit, ncall != 0u,
+ "%s: frame-address call argument emitted no materialization",
+ tc->name);
+ CU_EXPECT(unit, nglobal != 0u,
+ "%s: global-address call argument emitted no materialization",
+ tc->name);
+
+ switch (tc->arch) {
+ case KIT_ARCH_X86_64:
+ CU_EXPECT(unit, has_insn(body, nbody, "mov", "r11d"),
+ "x64: i32 frame index was not loaded through r11d");
+ CU_EXPECT(unit, has_insn(body, nbody, "lea", NULL),
+ "x64: scaled frame index was not folded into LEA");
+ CU_EXPECT(unit,
+ nglobal_relocs == 1u && global_relocs[0].kind == R_PC32 &&
+ global_relocs[0].addend == 33,
+ "x64: GLOBAL call arg lost symbol or +37 addend");
+ CU_EXPECT(unit, has_insn(global_body, nglobal, "lea", NULL),
+ "x64: GLOBAL call arg was not materialized with LEA");
+ CU_EXPECT(unit, nglobal == 1u,
+ "x64: GLOBAL call arg unexpectedly dereferenced storage");
+ {
+ i32 push = find_insn(exact_call_body, nexact_call, "push", "rcx");
+ i32 exact_load =
+ find_insn(exact_call_body, nexact_call, "mov", "(%r11)");
+ i32 pop = find_insn(exact_call_body, nexact_call, "pop", "r11");
+ i32 call = find_insn(exact_call_body, nexact_call, "call", "r11");
+ CU_EXPECT(unit, push >= 0,
+ "x64: indirect callee was not parked before marshalling");
+ CU_EXPECT(unit, exact_load > push,
+ "x64: exact address argument did not consume stabilized r11");
+ CU_EXPECT(unit, pop > exact_load,
+ "x64: indirect callee was restored before argument transport");
+ CU_EXPECT(unit, call > pop,
+ "x64: restored indirect callee was not the call target");
+ }
+ break;
+ case KIT_ARCH_ARM_64:
+ CU_EXPECT(unit, has_insn(body, nbody, "ldr", "w17"),
+ "aa64: i32 frame index was not loaded through w17");
+ CU_EXPECT(unit, has_insn(body, nbody, "add", "lsl #2"),
+ "aa64: scaled frame index was not folded into the address");
+ CU_EXPECT(unit,
+ nglobal_relocs == 2u &&
+ global_relocs[0].kind == R_AARCH64_ADR_PREL_PG_HI21 &&
+ global_relocs[1].kind == R_AARCH64_ADD_ABS_LO12_NC &&
+ global_relocs[0].addend == 37 &&
+ global_relocs[1].addend == 37,
+ "aa64: GLOBAL call arg lost symbol or +37 addend");
+ CU_EXPECT(unit, nglobal == 2u,
+ "aa64: GLOBAL call arg unexpectedly dereferenced storage");
+ CU_EXPECT(unit,
+ has_insn(copy_frame_value_body, ncopy_frame_value, "str",
+ "x16, [x17") &&
+ !has_insn(copy_frame_value_body, ncopy_frame_value, "str",
+ "x16, [x16"),
+ "aa64: aggregate copy clobbered x16 while materializing its "
+ "FRAME_VALUE destination");
+ CU_EXPECT(unit,
+ has_insn(set_frame_value_body, nset_frame_value, "str",
+ "x16, [x17") &&
+ !has_insn(set_frame_value_body, nset_frame_value, "str",
+ "x16, [x16"),
+ "aa64: widened aggregate set clobbered x16 while materializing "
+ "its FRAME_VALUE destination");
+ CU_EXPECT(unit,
+ has_insn(indexed_frame_value_body, nindexed_frame_value, "ldr",
+ "x17") &&
+ has_insn(indexed_frame_value_body, nindexed_frame_value,
+ "str", "x9, [x17, x16"),
+ "aa64: FRAME_VALUE base materialization clobbered its x16 "
+ "register index");
+ break;
+ case KIT_ARCH_RV64:
+ CU_EXPECT(unit, has_insn(body, nbody, "lw", "t1"),
+ "rv64: i32 frame index was not loaded with lw");
+ /* The RV decoder does not yet describe Zba, so it intentionally renders
+ * sh2add a0,t1,a0 as its exact raw word. Pinning that word is stronger
+ * than counting instructions: the old dropped-index path had no fourth
+ * instruction at all. */
+ CU_EXPECT(unit, has_insn(body, nbody, ".word", "0x20a34533"),
+ "rv64: frame index was dropped from load_addr");
+ CU_EXPECT(unit,
+ nglobal_relocs == 1u &&
+ global_relocs[0].kind == R_RV_PCREL_HI20,
+ "rv64: GLOBAL call arg did not retain its symbol relocation");
+ CU_EXPECT(unit, has_insn(global_body, nglobal, "addi", "37"),
+ "rv64: GLOBAL call arg lost its +37 addend");
+ CU_EXPECT(unit, nglobal == 3u &&
+ !has_insn(global_body, nglobal, "ld", NULL) &&
+ !has_insn(global_body, nglobal, "lw", NULL),
+ "rv64: GLOBAL call arg unexpectedly dereferenced storage");
+ {
+ i32 park = find_insn(forced_vararg_body, nforced_vararg, "ld", "ra");
+ i32 exact_load =
+ find_insn(forced_vararg_body, nforced_vararg, "lhu", NULL);
+ i32 exact_store =
+ find_insn(forced_vararg_body, nforced_vararg, "sh", NULL);
+ i32 call =
+ find_insn(forced_vararg_body, nforced_vararg, "jalr", "ra, 0(ra)");
+ CU_EXPECT(unit, park >= 0,
+ "rv64: indirect callee was not parked outside transport temps");
+ CU_EXPECT(unit, exact_load > park &&
+ has_insn(forced_vararg_body, nforced_vararg, "lbu",
+ NULL),
+ "rv64: 3-byte forced vararg did not use exact 2+1 loads");
+ CU_EXPECT(unit, exact_store > exact_load &&
+ has_insn(forced_vararg_body, nforced_vararg, "sb",
+ NULL),
+ "rv64: 3-byte forced vararg did not use exact 2+1 stores");
+ CU_EXPECT(unit,
+ count_insns(forced_vararg_body, nforced_vararg, "ld") == 1u &&
+ !has_insn(forced_vararg_body, nforced_vararg, "sd", NULL),
+ "rv64: rounded vararg carrier became a semantic access width");
+ CU_EXPECT(unit, call > exact_store,
+ "rv64: parked indirect target did not survive argument transport");
+ }
+ break;
+ case KIT_ARCH_ARM_32:
+ CU_EXPECT(unit, count_insns(body, nbody, "ldr") == 2u,
+ "arm32: exact-width base/index should require two word loads");
+ CU_EXPECT(unit, has_insn(body, nbody, "add", "lsl #2"),
+ "arm32: scaled frame index was not folded into the address");
+ CU_EXPECT(unit,
+ nglobal_relocs == 2u &&
+ global_relocs[0].kind == R_ARM_THM_MOVW_ABS_NC &&
+ global_relocs[1].kind == R_ARM_THM_MOVT_ABS,
+ "arm32: GLOBAL call arg did not retain MOVW/MOVT relocations");
+ CU_EXPECT(unit, has_insn(global_body, nglobal, "add", "37"),
+ "arm32: GLOBAL call arg lost its +37 addend");
+ CU_EXPECT(unit, nglobal == 3u &&
+ !has_insn(global_body, nglobal, "ldr", NULL),
+ "arm32: GLOBAL call arg unexpectedly dereferenced storage");
+ CU_EXPECT(unit, nimm >= 4u,
+ "arm32: split i64 immediate did not materialize both lanes");
+ CU_EXPECT(unit, has_insn(imm_body, nimm, "movw", "r0, #30600") &&
+ has_insn(imm_body, nimm, "movt", "r0, #21862"),
+ "arm32: i64 immediate low lane is not 0x55667788");
+ CU_EXPECT(unit, has_insn(imm_body, nimm, "movw", "r1, #13124") &&
+ has_insn(imm_body, nimm, "movt", "r1, #4386"),
+ "arm32: i64 immediate high lane is not 0x11223344");
+ break;
+ default:
+ CU_EXPECT(unit, 0, "%s: unhandled architecture", tc->name);
+ break;
+ }
+
+done:
+ if (mc)
+ mc_free(mc);
+ if (obj)
+ obj_free(obj);
+ kit_compiler_free(kit_c);
+}
+
+int main(void) {
+ static const BackendCase cases[] = {
+ {KIT_ARCH_X86_64, "x64", x64_native_target_new, 0u, 8u},
+ {KIT_ARCH_ARM_64, "aa64", aa64_native_target_new, 0u, 8u},
+ {KIT_ARCH_RV64, "rv64", rv64_native_target_new, 10u, 8u},
+ {KIT_ARCH_ARM_32, "arm32", arm32_native_target_new, 0u, 4u},
+ };
+ KitUnit unit;
+ kit_unit_init(&unit);
+ check_location_address_roles(&unit);
+ for (u32 i = 0; i < (u32)(sizeof cases / sizeof cases[0]); ++i)
+ check_typed_frame_address(&unit, &cases[i]);
+ kit_unit_summary(&unit, "frame-value-backends");
+ return kit_unit_status(&unit);
+}
diff --git a/test/opt/location_mir_test.c b/test/opt/location_mir_test.c
@@ -0,0 +1,1162 @@
+#include <stdio.h>
+#include <string.h>
+
+#include <kit/frontend.h>
+
+#include "lib/kit_unit.h"
+#include "opt/opt_internal.h"
+
+typedef struct WalkSummary {
+ FrameSlot stack_uses[4];
+ u32 nstack_uses;
+ u32 reserved_regs_seen;
+ u32 reserved_mask[OPT_REG_CLASSES];
+} WalkSummary;
+
+typedef void (*PanicAction)(void*);
+
+typedef struct PanicInvocation {
+ PanicAction action;
+ void* arg;
+} PanicInvocation;
+
+static KitStatus run_panicking_action(KitCompiler* kc, void* arg) {
+ PanicInvocation* call = (PanicInvocation*)arg;
+ (void)kc;
+ call->action(call->arg);
+ return KIT_OK;
+}
+
+static int expect_compiler_panic(KitUnit* u, Compiler* c, PanicAction action,
+ void* arg, const char* expected) {
+ PanicInvocation call;
+ KitStatus status;
+ call.action = action;
+ call.arg = arg;
+ u->last_diag[0] = '\0';
+ ++u->suppress_fatal;
+ status = kit_frontend_run((KitCompiler*)c, run_panicking_action, &call);
+ --u->suppress_fatal;
+ return status == KIT_ERR && strstr(u->last_diag, expected) != NULL;
+}
+
+static Operand reg_op(PReg reg, KitCgTypeId type) {
+ Operand op;
+ memset(&op, 0, sizeof op);
+ op.kind = OPK_REG;
+ op.cls = RC_INT;
+ op.type = type;
+ op.v.reg = reg;
+ return op;
+}
+
+static Operand indirect_op(PReg base, PReg index, KitCgTypeId base_type,
+ KitCgTypeId index_type) {
+ Operand op;
+ memset(&op, 0, sizeof op);
+ op.kind = OPK_INDIRECT;
+ op.cls = RC_INT;
+ op.type = base_type;
+ op.v.ind.base = base;
+ op.v.ind.index = index;
+ op.v.ind.log2_scale = 3;
+ op.v.ind.base_type = base_type;
+ op.v.ind.index_type = index_type;
+ return op;
+}
+
+static FrameSlot spill_slot(Func* f, KitCgTypeId type) {
+ FrameSlotDesc desc;
+ memset(&desc, 0, sizeof desc);
+ desc.type = type;
+ desc.size = 8;
+ desc.align = 8;
+ desc.kind = FS_SPILL;
+ return ir_frame_slot_new(f, &desc);
+}
+
+static FrameSlot local_slot(Func* f, KitCgTypeId type, u32 size, u32 align) {
+ FrameSlotDesc desc;
+ memset(&desc, 0, sizeof desc);
+ desc.type = type;
+ desc.size = size;
+ desc.align = align;
+ desc.kind = FS_LOCAL;
+ return ir_frame_slot_new(f, &desc);
+}
+
+static void set_stack_loc(Func* f, PReg reg, FrameSlot slot) {
+ f->preg_locs[reg].kind = OPT_LOC_STACK;
+ f->preg_locs[reg].cls = RC_INT;
+ f->preg_locs[reg].spill_slot = slot;
+ f->preg_info[reg].alloc_kind = OPT_ALLOC_SPILL;
+ f->preg_info[reg].cls = RC_INT;
+ f->preg_info[reg].spill_slot = slot;
+}
+
+static void set_hard_loc(Func* f, PReg reg, Reg hard_reg) {
+ f->preg_locs[reg].kind = OPT_LOC_HARD;
+ f->preg_locs[reg].cls = RC_INT;
+ f->preg_locs[reg].hard_reg = hard_reg;
+ f->preg_info[reg].alloc_kind = OPT_ALLOC_HARD;
+ f->preg_info[reg].cls = RC_INT;
+ f->preg_info[reg].hard_reg = hard_reg;
+}
+
+typedef struct MachinizeInvocation {
+ Func* f;
+ NativeTarget* target;
+} MachinizeInvocation;
+
+static void run_machinize(void* arg) {
+ MachinizeInvocation* call = (MachinizeInvocation*)arg;
+ opt_machinize_native(call->f, call->target);
+}
+
+static void run_mir_verify(void* arg) {
+ opt_mir_verify((Func*)arg, "negative-temp-escape");
+}
+
+static void machinize_rejects_cross_owner_registers(KitUnit* u) {
+ KitCompiler* kc = NULL;
+ Compiler* c;
+ CGFuncDesc desc;
+ Func* f;
+ NativePhysRegInfo phys[2];
+ NativeAllocClassInfo cls;
+ NativeRegInfo regs;
+ NativeTarget target;
+ MachinizeInvocation call;
+ Reg emit_temp = 1u;
+ Reg asm_temp = 2u;
+
+ CU_EXPECT(u,
+ kit_unit_compiler_new(
+ u, kit_unit_target(KIT_ARCH_X86_64, KIT_OS_LINUX, KIT_OBJ_ELF),
+ &kc) == KIT_OK &&
+ kc != NULL,
+ "compiler allocation failed for machinize ownership test");
+ if (!kc) return;
+ c = (Compiler*)kc;
+ memset(&desc, 0, sizeof desc);
+ desc.fn_type = kit_cg_type_builtin(kc, KIT_CG_BUILTIN_I64);
+ f = ir_func_new(c, &desc);
+
+ memset(phys, 0, sizeof phys);
+ phys[0].reg = emit_temp;
+ phys[0].cls = NATIVE_REG_INT;
+ phys[0].flags = NATIVE_REG_ALLOCABLE | NATIVE_REG_CALLER_SAVED;
+ phys[1].reg = asm_temp;
+ phys[1].cls = NATIVE_REG_INT;
+ phys[1].flags = NATIVE_REG_CALLER_SAVED | NATIVE_REG_RESERVED;
+ memset(&cls, 0, sizeof cls);
+ cls.cls = NATIVE_REG_INT;
+ cls.emit_temps = &emit_temp;
+ cls.nemit_temps = 1u;
+ cls.phys = phys;
+ cls.nphys = 2u;
+ cls.caller_saved_mask = (1u << emit_temp) | (1u << asm_temp);
+ memset(®s, 0, sizeof regs);
+ regs.classes = &cls;
+ regs.nclasses = 1u;
+ memset(&target, 0, sizeof target);
+ target.c = c;
+ target.regs = ®s;
+ call.f = f;
+ call.target = ⌖
+
+ CU_EXPECT(u,
+ expect_compiler_panic(u, c, run_machinize, &call,
+ "emit temp 1 is allocable"),
+ "machinize accepted a register shared by allocation and the "
+ "emitter-temp owner");
+
+ /* The asm bank has a separate lifetime and must be explicitly reserved.
+ * Exercise that ownership rule independently after making the emit bank
+ * valid, without trying to mirror every table-shape validation branch. */
+ phys[0].flags = NATIVE_REG_CALLER_SAVED | NATIVE_REG_RESERVED;
+ cls.asm_temps = &asm_temp;
+ cls.nasm_temps = 1u;
+ cls.reserved_mask = 1u << emit_temp;
+ CU_EXPECT(u,
+ expect_compiler_panic(u, c, run_machinize, &call,
+ "asm temp bank is not reserved"),
+ "machinize accepted an asm-internal temp outside the reserved "
+ "register owner");
+
+ kit_compiler_free(kc);
+}
+
+static void mir_verify_rejects_private_temp_escape(KitUnit* u) {
+ KitCompiler* kc = NULL;
+ Compiler* c;
+ KitCgTypeId i64_type;
+ CGFuncDesc desc;
+ Func* f;
+ PReg value;
+ Inst* load;
+ u32 entry;
+ Reg temp = 9u;
+
+ CU_EXPECT(u,
+ kit_unit_compiler_new(
+ u, kit_unit_target(KIT_ARCH_X86_64, KIT_OS_LINUX, KIT_OBJ_ELF),
+ &kc) == KIT_OK &&
+ kc != NULL,
+ "compiler allocation failed for MIR temp-escape test");
+ if (!kc) return;
+ c = (Compiler*)kc;
+ i64_type = kit_cg_type_builtin(kc, KIT_CG_BUILTIN_I64);
+ memset(&desc, 0, sizeof desc);
+ desc.fn_type = i64_type;
+ desc.result_type = i64_type;
+ f = ir_func_new(c, &desc);
+ entry = ir_block_new(f);
+ f->entry = entry;
+ ir_note_emit(f, entry);
+ value = ir_alloc_preg(f, i64_type, RC_INT);
+ f->preg_locs = arena_zarray(f->arena, OptLoc, f->npregs);
+ f->preg_info = arena_zarray(f->arena, OptPRegInfo, f->npregs);
+ set_hard_loc(f, value, temp);
+ f->opt_phys_regs[RC_INT][0].reg = temp;
+ f->opt_phys_regs[RC_INT][0].cls = RC_INT;
+ f->opt_phys_reg_count[RC_INT] = 1u;
+ f->emit_temp_regs[RC_INT][0] = temp;
+ f->emit_temp_reg_count[RC_INT] = 1u;
+ f->opt_reserved_regs[RC_INT] = 1u << temp;
+
+ load = ir_emit(f, entry, IR_LOAD_IMM);
+ load->type = i64_type;
+ load->def = value;
+ load->extra.imm = 42;
+ load->nopnds = 1u;
+ load->opnds = arena_array(f->arena, Operand, 1u);
+ load->opnds[0] = reg_op(value, i64_type);
+ opt_lower_to_mir(f, NULL);
+ CU_EXPECT(u, f->mir != NULL,
+ "temp-escape fixture did not lower to location MIR");
+ if (f->mir) {
+ CU_EXPECT(u,
+ expect_compiler_panic(u, c, run_mir_verify, f,
+ "emitter-owned temp escaped into MIR"),
+ "MIR verifier accepted an emitter-owned temporary as a "
+ "persistent value");
+
+ f->emit_temp_reg_count[RC_INT] = 0u;
+ f->asm_temp_mask[RC_INT] = 1u << temp;
+ CU_EXPECT(u,
+ expect_compiler_panic(u, c, run_mir_verify, f,
+ "asm-internal temp escaped into MIR"),
+ "MIR verifier accepted an asm-internal temporary as a "
+ "persistent value");
+ }
+
+ kit_compiler_free(kc);
+}
+
+static void mir_commit_propagates_instruction_namespace(KitUnit* u) {
+ KitCompiler* kc = NULL;
+ Compiler* c;
+ CGFuncDesc desc;
+ Func* f;
+ Func view;
+ InstId committed;
+
+ CU_EXPECT(u,
+ kit_unit_compiler_new(
+ u, kit_unit_target(KIT_ARCH_X86_64, KIT_OS_LINUX, KIT_OBJ_ELF),
+ &kc) == KIT_OK &&
+ kc != NULL,
+ "compiler allocation failed for MIR commit contract test");
+ if (!kc) return;
+ c = (Compiler*)kc;
+ memset(&desc, 0, sizeof desc);
+ desc.fn_type = kit_cg_type_builtin(kc, KIT_CG_BUILTIN_I64);
+ f = ir_func_new(c, &desc);
+ f->mir = arena_zarray(f->arena, MFunc, 1u);
+
+ CU_EXPECT(u, opt_mir_view(f, &view),
+ "MIR commit fixture did not produce a physical view");
+ committed = f->next_inst_id + 7u;
+ view.next_inst_id = committed;
+ opt_mir_commit(f, &view);
+ CU_EXPECT(u, f->next_inst_id == committed,
+ "MIR commit lost the physical graph's instruction-id namespace");
+
+ kit_compiler_free(kc);
+}
+
+static Inst* find_inst(Block* block, InstId id) {
+ for (u32 i = 0; i < block->ninsts; ++i)
+ if (block->insts[i].id == id)
+ return &block->insts[i];
+ return NULL;
+}
+
+static void summarize_location(Func* f, Inst* in, Operand* op, int is_def,
+ void* arg) {
+ WalkSummary* summary = (WalkSummary*)arg;
+ (void)f;
+ (void)in;
+ if (op->kind == OPK_STACK && !is_def &&
+ summary->nstack_uses <
+ (u32)(sizeof summary->stack_uses / sizeof summary->stack_uses[0])) {
+ summary->stack_uses[summary->nstack_uses++] = op->v.frame_slot;
+ }
+ if (op->kind == OPK_REG && op->cls < OPT_REG_CLASSES && op->v.reg < 32u &&
+ (summary->reserved_mask[op->cls] & (1u << op->v.reg))) {
+ ++summary->reserved_regs_seen;
+ }
+}
+
+static void spilled_values_remain_locations(KitUnit* u) {
+ KitCompiler* kc = NULL;
+ Compiler* c;
+ Func* f;
+ KitCgTypeId i64_type, ptr_type;
+ CGFuncDesc desc;
+ PReg bin_dst, bin_lhs, bin_rhs, load_dst, addr_base, addr_index;
+ FrameSlot bin_dst_slot, bin_lhs_slot, bin_rhs_slot;
+ FrameSlot addr_base_slot, addr_index_slot;
+ u32 entry;
+ Inst* bin;
+ Inst* load;
+ InstId bin_id;
+ InstId load_id;
+ Block* hir_blocks;
+ Inst* hir_insts;
+ Operand* hir_bin_opnds;
+ Operand* hir_load_opnds;
+ u32* hir_succ;
+ Block* mir_block;
+ WalkSummary all;
+ WalkSummary address;
+ Func view;
+
+ CU_EXPECT(u,
+ kit_unit_compiler_new(
+ u, kit_unit_target(KIT_ARCH_ARM_64, KIT_OS_LINUX, KIT_OBJ_ELF),
+ &kc) == KIT_OK &&
+ kc != NULL,
+ "compiler allocation failed");
+ if (!kc)
+ return;
+ c = (Compiler*)kc;
+ i64_type = kit_cg_type_builtin(kc, KIT_CG_BUILTIN_I64);
+ ptr_type = kit_cg_type_ptr(
+ kc, kit_cg_type_builtin(kc, KIT_CG_BUILTIN_VOID), 0);
+
+ memset(&desc, 0, sizeof desc);
+ desc.fn_type = i64_type;
+ desc.result_type = i64_type;
+ f = ir_func_new(c, &desc);
+ entry = ir_block_new(f);
+ f->entry = entry;
+ ir_note_emit(f, entry);
+
+ bin_dst = ir_alloc_preg(f, i64_type, RC_INT);
+ bin_lhs = ir_alloc_preg(f, i64_type, RC_INT);
+ bin_rhs = ir_alloc_preg(f, i64_type, RC_INT);
+ load_dst = ir_alloc_preg(f, i64_type, RC_INT);
+ addr_base = ir_alloc_preg(f, ptr_type, RC_INT);
+ addr_index = ir_alloc_preg(f, i64_type, RC_INT);
+
+ bin_dst_slot = spill_slot(f, i64_type);
+ bin_lhs_slot = spill_slot(f, i64_type);
+ bin_rhs_slot = spill_slot(f, i64_type);
+ /* Spill coloring may reuse a physical slot across unrelated type IDs. Keep
+ * intentionally mismatched descriptor types here: location MIR must carry
+ * the base/index value types rather than reconstructing them from the slot. */
+ addr_base_slot = spill_slot(f, i64_type);
+ addr_index_slot = spill_slot(f, ptr_type);
+
+ f->preg_locs = arena_zarray(f->arena, OptLoc, f->npregs);
+ f->preg_info = arena_zarray(f->arena, OptPRegInfo, f->npregs);
+ set_stack_loc(f, bin_dst, bin_dst_slot);
+ set_stack_loc(f, bin_lhs, bin_lhs_slot);
+ set_stack_loc(f, bin_rhs, bin_rhs_slot);
+ set_hard_loc(f, load_dst, 1);
+ set_stack_loc(f, addr_base, addr_base_slot);
+ set_stack_loc(f, addr_index, addr_index_slot);
+
+ /* These are emitter-private temporaries. They may be selected while native
+ * code is emitted, but must never become values in post-regalloc MIR. */
+ f->emit_temp_regs[RC_INT][0] = 9;
+ f->emit_temp_regs[RC_INT][1] = 10;
+ f->emit_temp_regs[RC_INT][2] = 11;
+ f->emit_temp_reg_count[RC_INT] = 3;
+ f->opt_reserved_regs[RC_INT] = (1u << 9) | (1u << 10) | (1u << 11);
+
+ bin = ir_emit(f, entry, IR_BINOP);
+ bin->type = i64_type;
+ bin->def = bin_dst;
+ bin->extra.imm = BO_IADD;
+ bin->nopnds = 3;
+ bin->opnds = arena_array(f->arena, Operand, bin->nopnds);
+ bin->opnds[0] = reg_op(bin_dst, i64_type);
+ bin->opnds[1] = reg_op(bin_lhs, i64_type);
+ bin->opnds[2] = reg_op(bin_rhs, i64_type);
+ bin_id = bin->id;
+
+ load = ir_emit(f, entry, IR_LOAD);
+ load->type = i64_type;
+ load->def = load_dst;
+ load->nopnds = 2;
+ load->opnds = arena_array(f->arena, Operand, load->nopnds);
+ load->opnds[0] = reg_op(load_dst, i64_type);
+ load->opnds[1] = indirect_op(addr_base, addr_index, ptr_type, i64_type);
+ load->extra.mem.type = i64_type;
+ load->extra.mem.size = 8;
+ load->extra.mem.align = 8;
+ load_id = load->id;
+
+ /* Capture the semantic HIR allocation after the last ir_emit (which may
+ * grow the block's instruction array). Lowering must build a physically
+ * separate MIR graph and leave each virtual operand in this graph intact. */
+ hir_blocks = f->blocks;
+ hir_insts = f->blocks[entry].insts;
+ hir_bin_opnds = find_inst(&f->blocks[entry], bin_id)->opnds;
+ hir_load_opnds = find_inst(&f->blocks[entry], load_id)->opnds;
+ hir_succ = f->blocks[entry].succ;
+
+ opt_lower_to_mir(f, NULL);
+ CU_EXPECT(u, f->mir != NULL, "lowering did not produce MIR");
+ if (!f->mir) {
+ kit_compiler_free(kc);
+ return;
+ }
+ mir_block = &f->mir->blocks[entry];
+ bin = find_inst(mir_block, bin_id);
+ load = find_inst(mir_block, load_id);
+
+ CU_EXPECT(u, f->blocks == hir_blocks && f->blocks != f->mir->blocks,
+ "lowering did not preserve a distinct HIR block graph");
+ CU_EXPECT(u,
+ f->mir->blocks_cap == f->blocks_cap &&
+ opt_mir_view(f, &view) && view.blocks == f->mir->blocks &&
+ view.blocks_cap == f->mir->blocks_cap && view.mir == NULL &&
+ view.scope_aux_inst == NULL && view.nscopes == 0,
+ "canonical MIR view retained stale HIR graph metadata");
+ CU_EXPECT(u,
+ f->blocks[entry].insts == hir_insts &&
+ f->mir->blocks[entry].insts != hir_insts,
+ "lowering aliases HIR and MIR instruction arrays");
+ CU_EXPECT(u,
+ f->blocks[entry].succ == hir_succ &&
+ f->mir->blocks[entry].succ != hir_succ,
+ "lowering aliases HIR and MIR CFG arrays");
+ {
+ Inst* hir_bin = find_inst(&f->blocks[entry], bin_id);
+ Inst* hir_load = find_inst(&f->blocks[entry], load_id);
+ CU_EXPECT(u,
+ hir_bin && hir_bin->opnds == hir_bin_opnds &&
+ hir_bin->opnds[0].kind == OPK_REG &&
+ hir_bin->opnds[0].v.reg == bin_dst &&
+ hir_bin->opnds[1].v.reg == bin_lhs &&
+ hir_bin->opnds[2].v.reg == bin_rhs && bin &&
+ bin->opnds != hir_bin_opnds,
+ "location rewrite changed or aliased HIR binop operands");
+ CU_EXPECT(u,
+ hir_load && hir_load->opnds == hir_load_opnds &&
+ hir_load->opnds[0].kind == OPK_REG &&
+ hir_load->opnds[0].v.reg == load_dst &&
+ hir_load->opnds[1].kind == OPK_INDIRECT &&
+ hir_load->opnds[1].v.ind.base == addr_base &&
+ hir_load->opnds[1].v.ind.index == addr_index && load &&
+ load->opnds != hir_load_opnds,
+ "location rewrite changed or aliased HIR address operands");
+ }
+
+ CU_EXPECT(u, f->opt_rewrite_inserted_insts == 0,
+ "location lowering inserted %llu reload/store instructions",
+ (unsigned long long)f->opt_rewrite_inserted_insts);
+ CU_EXPECT(u, mir_block->ninsts == 2,
+ "location MIR should preserve two source instructions, got %u",
+ mir_block->ninsts);
+ CU_EXPECT(u, bin != NULL && bin->op == IR_BINOP,
+ "original binop did not survive location lowering");
+ CU_EXPECT(u, load != NULL && load->op == IR_LOAD,
+ "original indirect load did not survive location lowering");
+ CU_EXPECT(u,
+ load && load->opnds[1].kind == OPK_INDIRECT &&
+ load->opnds[1].v.ind.base_kind == OPT_INDIRECT_FRAME &&
+ load->opnds[1].v.ind.index_kind == OPT_INDIRECT_FRAME &&
+ load->opnds[1].v.ind.base_type == ptr_type &&
+ load->opnds[1].v.ind.index_type == i64_type,
+ "indirect component location/type provenance was not preserved");
+
+ /* OPK_STACK is the post-allocation scalar-value location. OPK_LOCAL remains
+ * a semantic frame lvalue/address, so accepting it here would collapse the
+ * exact boundary this test protects. */
+ if (bin && bin->nopnds == 3) {
+ CU_EXPECT(u,
+ bin->opnds[0].kind == OPK_STACK &&
+ bin->opnds[0].v.frame_slot == bin_dst_slot,
+ "spilled scalar def is not its stable frame location");
+ CU_EXPECT(u,
+ bin->opnds[1].kind == OPK_STACK &&
+ bin->opnds[1].v.frame_slot == bin_lhs_slot,
+ "first spilled scalar use is not its stable frame location");
+ CU_EXPECT(u,
+ bin->opnds[2].kind == OPK_STACK &&
+ bin->opnds[2].v.frame_slot == bin_rhs_slot,
+ "second spilled scalar use is not its stable frame location");
+ } else {
+ CU_EXPECT(u, 0, "lowered binop has the wrong operand shape");
+ }
+
+ memset(&address, 0, sizeof address);
+ address.reserved_mask[RC_INT] = f->opt_reserved_regs[RC_INT];
+ if (load)
+ opt_walk_inst_operands(f, load, summarize_location, &address);
+ CU_EXPECT(u,
+ address.nstack_uses == 2 &&
+ address.stack_uses[0] == addr_base_slot &&
+ address.stack_uses[1] == addr_index_slot,
+ "spilled indirect base/index are not stable frame locations");
+
+ memset(&all, 0, sizeof all);
+ for (u32 cls = 0; cls < OPT_REG_CLASSES; ++cls)
+ all.reserved_mask[cls] = f->opt_reserved_regs[cls];
+ for (u32 i = 0; i < mir_block->ninsts; ++i)
+ opt_walk_inst_operands(f, &mir_block->insts[i], summarize_location, &all);
+ CU_EXPECT(u, all.reserved_regs_seen == 0,
+ "MIR exposes %u emitter-private register operands",
+ all.reserved_regs_seen);
+
+ kit_compiler_free(kc);
+}
+
+static void spilled_frame_address_remains_recipe(KitUnit* u) {
+ KitCompiler* kc = NULL;
+ Compiler* c;
+ Func* f;
+ KitCgTypeId i32_type, ptr_type;
+ CGFuncDesc desc;
+ PReg address, copy;
+ FrameSlot value_slot, address_spill, copy_spill;
+ u32 entry;
+ Inst* addr;
+ Inst* move;
+ InstId addr_id;
+ InstId move_id;
+
+ CU_EXPECT(u,
+ kit_unit_compiler_new(
+ u, kit_unit_target(KIT_ARCH_ARM_64, KIT_OS_LINUX, KIT_OBJ_ELF),
+ &kc) == KIT_OK &&
+ kc != NULL,
+ "compiler allocation failed for frame-address recipe test");
+ if (!kc) return;
+ c = (Compiler*)kc;
+ i32_type = kit_cg_type_builtin(kc, KIT_CG_BUILTIN_I32);
+ ptr_type = kit_cg_type_ptr(kc, i32_type, 0);
+
+ memset(&desc, 0, sizeof desc);
+ desc.fn_type = ptr_type;
+ desc.result_type = ptr_type;
+ f = ir_func_new(c, &desc);
+ entry = ir_block_new(f);
+ f->entry = entry;
+ ir_note_emit(f, entry);
+
+ address = ir_alloc_preg(f, ptr_type, RC_INT);
+ copy = ir_alloc_preg(f, ptr_type, RC_INT);
+ value_slot = local_slot(f, i32_type, 4, 4);
+ address_spill = spill_slot(f, ptr_type);
+ copy_spill = spill_slot(f, ptr_type);
+ f->preg_locs = arena_zarray(f->arena, OptLoc, f->npregs);
+ f->preg_info = arena_zarray(f->arena, OptPRegInfo, f->npregs);
+ set_stack_loc(f, address, address_spill);
+ set_stack_loc(f, copy, copy_spill);
+
+ addr = ir_emit(f, entry, IR_ADDR_OF);
+ addr_id = addr->id;
+ addr->type = ptr_type;
+ addr->def = address;
+ addr->nopnds = 2;
+ addr->opnds = arena_zarray(f->arena, Operand, addr->nopnds);
+ addr->opnds[0] = reg_op(address, ptr_type);
+ addr->opnds[1].kind = OPK_LOCAL;
+ addr->opnds[1].cls = RC_INT;
+ addr->opnds[1].type = i32_type;
+ addr->opnds[1].v.frame_slot = value_slot;
+
+ move = ir_emit(f, entry, IR_COPY);
+ move_id = move->id;
+ move->type = ptr_type;
+ move->def = copy;
+ move->nopnds = 2;
+ move->opnds = arena_zarray(f->arena, Operand, move->nopnds);
+ move->opnds[0] = reg_op(copy, ptr_type);
+ move->opnds[1] = reg_op(address, ptr_type);
+
+ opt_lower_to_mir(f, NULL);
+ opt_mir_combine(f, NULL);
+ opt_mir_dce(f);
+ opt_mir_jump_cleanup(f, OPT_JUMP_CLEANUP_CFG);
+ opt_mir_build_cfg(f);
+ opt_mir_jump_cleanup(f, OPT_JUMP_CLEANUP_LAYOUT);
+ CU_EXPECT(u, f->mir != NULL, "frame-address lowering did not produce MIR");
+ if (f->mir) {
+ Block* block = &f->mir->blocks[entry];
+ Inst* mir_addr = find_inst(block, addr_id);
+ Inst* mir_move = find_inst(block, move_id);
+ CU_EXPECT(u, mir_addr == NULL,
+ "rematerialized frame-address producer survived MIR lowering");
+ CU_EXPECT(u, block->ninsts == 1 && mir_move && mir_move->nopnds == 2,
+ "frame-address recipe lowering has the wrong instruction shape");
+ CU_EXPECT(u,
+ mir_move && mir_move->opnds[0].kind == OPK_STACK &&
+ mir_move->opnds[0].v.frame_slot == copy_spill,
+ "frame-address copy destination is not its spill home");
+ CU_EXPECT(u,
+ mir_move && mir_move->opnds[1].kind == OPK_FRAME_ADDR &&
+ mir_move->opnds[1].v.frame_slot == value_slot,
+ "frame-address recipe was rewritten as an unrelated value");
+ }
+ addr = find_inst(&f->blocks[entry], addr_id);
+ move = find_inst(&f->blocks[entry], move_id);
+ CU_EXPECT(u,
+ addr && addr->opnds[1].kind == OPK_LOCAL &&
+ addr->opnds[1].v.frame_slot == value_slot && move &&
+ move->opnds[1].kind == OPK_REG &&
+ move->opnds[1].v.reg == address,
+ "frame-address location lowering mutated semantic HIR");
+
+ kit_compiler_free(kc);
+}
+
+static void call_aux_is_independent(KitUnit* u) {
+ KitCompiler* kc = NULL;
+ Compiler* c;
+ Func* f;
+ KitCgTypeId i64_type;
+ CGFuncDesc desc;
+ PReg callee_reg, arg_reg, arg_part_reg, ret_reg, ret_part_reg;
+ IRCallAux* hir_aux;
+ IRCallAux* mir_aux;
+ Inst* call;
+ InstId call_id;
+ u32 entry;
+
+ CU_EXPECT(u,
+ kit_unit_compiler_new(
+ u, kit_unit_target(KIT_ARCH_ARM_64, KIT_OS_LINUX, KIT_OBJ_ELF),
+ &kc) == KIT_OK &&
+ kc != NULL,
+ "compiler allocation failed for call-aux clone test");
+ if (!kc) return;
+ c = (Compiler*)kc;
+ i64_type = kit_cg_type_builtin(kc, KIT_CG_BUILTIN_I64);
+
+ memset(&desc, 0, sizeof desc);
+ desc.fn_type = i64_type;
+ f = ir_func_new(c, &desc);
+ entry = ir_block_new(f);
+ f->entry = entry;
+ ir_note_emit(f, entry);
+
+ callee_reg = ir_alloc_preg(f, i64_type, RC_INT);
+ arg_reg = ir_alloc_preg(f, i64_type, RC_INT);
+ arg_part_reg = ir_alloc_preg(f, i64_type, RC_INT);
+ ret_reg = ir_alloc_preg(f, i64_type, RC_INT);
+ ret_part_reg = ir_alloc_preg(f, i64_type, RC_INT);
+ f->preg_locs = arena_zarray(f->arena, OptLoc, f->npregs);
+ f->preg_info = arena_zarray(f->arena, OptPRegInfo, f->npregs);
+ set_hard_loc(f, callee_reg, 1);
+ set_hard_loc(f, arg_reg, 2);
+ set_hard_loc(f, arg_part_reg, 3);
+ set_hard_loc(f, ret_reg, 4);
+ set_hard_loc(f, ret_part_reg, 5);
+
+ call = ir_emit(f, entry, IR_CALL);
+ call_id = call->id;
+ call->def = ret_reg;
+ hir_aux = arena_znew(f->arena, IRCallAux);
+ hir_aux->desc.callee = reg_op(callee_reg, i64_type);
+ hir_aux->desc.nargs = 1;
+ hir_aux->desc.args = arena_zarray(f->arena, CGABIValue, 1);
+ hir_aux->desc.args[0].type = i64_type;
+ hir_aux->desc.args[0].storage = reg_op(arg_reg, i64_type);
+ hir_aux->desc.args[0].nparts = 1;
+ hir_aux->desc.args[0].parts = arena_zarray(f->arena, CGABIPart, 1);
+ hir_aux->desc.args[0].parts[0].op = reg_op(arg_part_reg, i64_type);
+ hir_aux->desc.ret.type = i64_type;
+ hir_aux->desc.ret.storage = reg_op(ret_reg, i64_type);
+ hir_aux->desc.ret.nparts = 1;
+ hir_aux->desc.ret.parts = arena_zarray(f->arena, CGABIPart, 1);
+ hir_aux->desc.ret.parts[0].op = reg_op(ret_part_reg, i64_type);
+ hir_aux->plan.callee = reg_op(callee_reg, i64_type);
+ hir_aux->plan.nargs = 2;
+ hir_aux->plan.args =
+ arena_zarray(f->arena, CGCallPlanMove, hir_aux->plan.nargs);
+ hir_aux->plan.args[0].src = reg_op(arg_reg, i64_type);
+ hir_aux->plan.args[0].dst = reg_op(arg_part_reg, i64_type);
+ hir_aux->plan.args[0].src_kind = CG_CALL_PLAN_SRC_VALUE;
+ hir_aux->plan.args[0].dst_kind = CG_CALL_PLAN_REG;
+ hir_aux->plan.args[0].dst_reg = 3;
+ hir_aux->plan.args[1].src = reg_op(arg_part_reg, i64_type);
+ hir_aux->plan.args[1].dst_kind = CG_CALL_PLAN_STACK;
+ hir_aux->plan.args[1].stack_offset = 24;
+ hir_aux->plan.nrets = 2;
+ hir_aux->plan.rets =
+ arena_zarray(f->arena, CGCallPlanRet, hir_aux->plan.nrets);
+ hir_aux->plan.rets[0].dst = reg_op(ret_reg, i64_type);
+ hir_aux->plan.rets[0].src_reg = 4;
+ hir_aux->plan.rets[1].dst = reg_op(ret_part_reg, i64_type);
+ hir_aux->plan.rets[1].src_reg = 5;
+ hir_aux->plan_valid = 1;
+ hir_aux->nresults = 1;
+ hir_aux->results = arena_array(f->arena, Val, 1);
+ hir_aux->results[0] = ret_reg;
+ call->extra.aux = hir_aux;
+
+ opt_lower_to_mir(f, NULL);
+ call = f->mir ? find_inst(&f->mir->blocks[entry], call_id) : NULL;
+ mir_aux = call ? (IRCallAux*)call->extra.aux : NULL;
+ CU_EXPECT(u, call && mir_aux && mir_aux != hir_aux,
+ "call aux object aliases HIR after MIR lowering");
+ CU_EXPECT(u,
+ mir_aux && mir_aux->desc.args != hir_aux->desc.args &&
+ mir_aux->desc.args[0].parts != hir_aux->desc.args[0].parts &&
+ mir_aux->desc.ret.parts != hir_aux->desc.ret.parts &&
+ mir_aux->plan.args != hir_aux->plan.args &&
+ mir_aux->plan.rets != hir_aux->plan.rets &&
+ mir_aux->results != hir_aux->results,
+ "call aux nested arrays alias HIR after MIR lowering");
+ CU_EXPECT(u,
+ hir_aux->desc.callee.v.reg == callee_reg &&
+ hir_aux->desc.args[0].storage.v.reg == arg_reg &&
+ hir_aux->desc.args[0].parts[0].op.v.reg == arg_part_reg &&
+ hir_aux->desc.ret.storage.v.reg == ret_reg &&
+ hir_aux->desc.ret.parts[0].op.v.reg == ret_part_reg &&
+ mir_aux && mir_aux->desc.callee.v.reg == 1 &&
+ mir_aux->desc.args[0].storage.v.reg == 2 &&
+ mir_aux->desc.args[0].parts[0].op.v.reg == 3 &&
+ mir_aux->desc.ret.storage.v.reg == 4 &&
+ mir_aux->desc.ret.parts[0].op.v.reg == 5,
+ "call operand rewrite crossed the HIR/MIR ownership boundary");
+ CU_EXPECT(u,
+ mir_aux && mir_aux->plan.nargs == 2 && mir_aux->plan.nrets == 2 &&
+ mir_aux->plan.args[0].src.v.reg == arg_reg &&
+ mir_aux->plan.args[1].stack_offset == 24 &&
+ mir_aux->plan.rets[0].dst.v.reg == ret_reg &&
+ mir_aux->plan.rets[1].src_reg == 5,
+ "call plan arrays did not survive MIR cloning");
+
+ if (mir_aux) {
+ mir_aux->plan.args[0].src.kind = OPK_IMM;
+ mir_aux->plan.args[0].src.v.imm = 91;
+ mir_aux->plan.args[1].stack_offset = 88;
+ mir_aux->plan.rets[0].dst.v.reg = 27;
+ mir_aux->plan.rets[1].src_reg = 28;
+ }
+ CU_EXPECT(u,
+ hir_aux->plan.args[0].src.kind == OPK_REG &&
+ hir_aux->plan.args[0].src.v.reg == arg_reg &&
+ hir_aux->plan.args[1].stack_offset == 24 &&
+ hir_aux->plan.rets[0].dst.v.reg == ret_reg &&
+ hir_aux->plan.rets[1].src_reg == 5,
+ "mutating MIR call-plan arrays changed semantic HIR");
+
+ hir_aux->plan.args[1].stack_offset = 104;
+ hir_aux->plan.rets[1].src_reg = 30;
+ CU_EXPECT(u,
+ mir_aux && mir_aux->plan.args[1].stack_offset == 88 &&
+ mir_aux->plan.rets[1].src_reg == 28,
+ "mutating HIR call-plan arrays changed location MIR");
+
+ kit_compiler_free(kc);
+}
+
+static void structured_aux_arrays_are_independent(KitUnit* u) {
+ KitCompiler* kc = NULL;
+ Compiler* c;
+ Func* f;
+ KitCgTypeId i64_type;
+ CGFuncDesc desc;
+ PReg dst0, dst1, arg0, arg1;
+ Inst* phi;
+ Inst* sw;
+ Inst* indirect;
+ Inst* intrin;
+ InstId phi_id, switch_id, indirect_id, intrin_id;
+ IRPhiAux* hir_phi;
+ IRPhiAux* mir_phi;
+ IRSwitchAux* hir_switch;
+ IRSwitchAux* mir_switch;
+ IRIndirectAux* hir_indirect;
+ IRIndirectAux* mir_indirect;
+ IRIntrinAux* hir_intrin;
+ IRIntrinAux* mir_intrin;
+ u32 entry;
+
+ CU_EXPECT(u,
+ kit_unit_compiler_new(
+ u, kit_unit_target(KIT_ARCH_ARM_64, KIT_OS_LINUX, KIT_OBJ_ELF),
+ &kc) == KIT_OK &&
+ kc != NULL,
+ "compiler allocation failed for structured-aux clone test");
+ if (!kc) return;
+ c = (Compiler*)kc;
+ i64_type = kit_cg_type_builtin(kc, KIT_CG_BUILTIN_I64);
+
+ memset(&desc, 0, sizeof desc);
+ desc.fn_type = i64_type;
+ f = ir_func_new(c, &desc);
+ entry = ir_block_new(f);
+ f->entry = entry;
+ ir_note_emit(f, entry);
+
+ dst0 = ir_alloc_preg(f, i64_type, RC_INT);
+ dst1 = ir_alloc_preg(f, i64_type, RC_INT);
+ arg0 = ir_alloc_preg(f, i64_type, RC_INT);
+ arg1 = ir_alloc_preg(f, i64_type, RC_INT);
+ f->preg_locs = arena_zarray(f->arena, OptLoc, f->npregs);
+ f->preg_info = arena_zarray(f->arena, OptPRegInfo, f->npregs);
+ set_hard_loc(f, dst0, 8);
+ set_hard_loc(f, dst1, 9);
+ set_hard_loc(f, arg0, 10);
+ set_hard_loc(f, arg1, 11);
+
+ phi = ir_emit(f, entry, IR_PHI);
+ phi_id = phi->id;
+ hir_phi = arena_znew(f->arena, IRPhiAux);
+ hir_phi->npreds = 2;
+ hir_phi->pred_blocks = arena_array(f->arena, u32, hir_phi->npreds);
+ hir_phi->pred_vals = arena_array(f->arena, Val, hir_phi->npreds);
+ hir_phi->pred_blocks[0] = 3;
+ hir_phi->pred_blocks[1] = 4;
+ hir_phi->pred_vals[0] = 13;
+ hir_phi->pred_vals[1] = 14;
+ phi->extra.aux = hir_phi;
+
+ sw = ir_emit(f, entry, IR_SWITCH);
+ switch_id = sw->id;
+ hir_switch = arena_znew(f->arena, IRSwitchAux);
+ hir_switch->ncases = 2;
+ hir_switch->cases =
+ arena_zarray(f->arena, IRSwitchAuxCase, hir_switch->ncases);
+ hir_switch->cases[0].value = 21;
+ hir_switch->cases[0].block = 5;
+ hir_switch->cases[1].value = 22;
+ hir_switch->cases[1].block = 6;
+ sw->extra.aux = hir_switch;
+
+ indirect = ir_emit(f, entry, IR_INDIRECT_BRANCH);
+ indirect_id = indirect->id;
+ hir_indirect = arena_znew(f->arena, IRIndirectAux);
+ hir_indirect->ntargets = 2;
+ hir_indirect->targets = arena_array(f->arena, u32, hir_indirect->ntargets);
+ hir_indirect->targets[0] = 7;
+ hir_indirect->targets[1] = 8;
+ indirect->extra.aux = hir_indirect;
+
+ intrin = ir_emit(f, entry, IR_INTRINSIC);
+ intrin_id = intrin->id;
+ hir_intrin = arena_znew(f->arena, IRIntrinAux);
+ hir_intrin->kind = INTRIN_SADD_OVERFLOW;
+ hir_intrin->ndst = 2;
+ hir_intrin->narg = 2;
+ hir_intrin->dsts = arena_zarray(f->arena, Operand, hir_intrin->ndst);
+ hir_intrin->args = arena_zarray(f->arena, Operand, hir_intrin->narg);
+ hir_intrin->result_vals = arena_array(f->arena, Val, hir_intrin->ndst);
+ hir_intrin->dsts[0] = reg_op(dst0, i64_type);
+ hir_intrin->dsts[1] = reg_op(dst1, i64_type);
+ hir_intrin->args[0] = reg_op(arg0, i64_type);
+ hir_intrin->args[1] = reg_op(arg1, i64_type);
+ hir_intrin->result_vals[0] = 31;
+ hir_intrin->result_vals[1] = 32;
+ intrin->extra.aux = hir_intrin;
+
+ opt_lower_to_mir(f, NULL);
+ phi = f->mir ? find_inst(&f->mir->blocks[entry], phi_id) : NULL;
+ sw = f->mir ? find_inst(&f->mir->blocks[entry], switch_id) : NULL;
+ indirect =
+ f->mir ? find_inst(&f->mir->blocks[entry], indirect_id) : NULL;
+ intrin = f->mir ? find_inst(&f->mir->blocks[entry], intrin_id) : NULL;
+ mir_phi = phi ? (IRPhiAux*)phi->extra.aux : NULL;
+ mir_switch = sw ? (IRSwitchAux*)sw->extra.aux : NULL;
+ mir_indirect = indirect ? (IRIndirectAux*)indirect->extra.aux : NULL;
+ mir_intrin = intrin ? (IRIntrinAux*)intrin->extra.aux : NULL;
+
+ CU_EXPECT(u,
+ mir_phi && mir_phi != hir_phi &&
+ mir_phi->pred_blocks != hir_phi->pred_blocks &&
+ mir_phi->pred_vals != hir_phi->pred_vals,
+ "phi aux arrays alias semantic HIR");
+ CU_EXPECT(u,
+ mir_switch && mir_switch != hir_switch &&
+ mir_switch->cases != hir_switch->cases,
+ "switch case array aliases semantic HIR");
+ CU_EXPECT(u,
+ mir_indirect && mir_indirect != hir_indirect &&
+ mir_indirect->targets != hir_indirect->targets,
+ "indirect-branch target array aliases semantic HIR");
+ CU_EXPECT(u,
+ mir_intrin && mir_intrin != hir_intrin &&
+ mir_intrin->dsts != hir_intrin->dsts &&
+ mir_intrin->args != hir_intrin->args &&
+ mir_intrin->result_vals != hir_intrin->result_vals,
+ "intrinsic aux arrays alias semantic HIR");
+ CU_EXPECT(u,
+ mir_phi && mir_phi->pred_blocks[0] == 3 &&
+ mir_phi->pred_vals[1] == 14 && mir_switch &&
+ mir_switch->cases[1].value == 22 &&
+ mir_switch->cases[1].block == 6 && mir_indirect &&
+ mir_indirect->targets[1] == 8 && mir_intrin &&
+ mir_intrin->result_vals[0] == 31 &&
+ mir_intrin->dsts[0].v.reg == 8 &&
+ mir_intrin->dsts[1].v.reg == 9 &&
+ mir_intrin->args[0].v.reg == 10 &&
+ mir_intrin->args[1].v.reg == 11,
+ "structured aux payloads did not survive location-MIR cloning");
+
+ if (mir_phi) {
+ mir_phi->pred_blocks[0] = 103;
+ mir_phi->pred_vals[0] = 113;
+ }
+ if (mir_switch) {
+ mir_switch->cases[0].value = 121;
+ mir_switch->cases[0].block = 105;
+ }
+ if (mir_indirect) mir_indirect->targets[0] = 107;
+ if (mir_intrin) {
+ mir_intrin->dsts[0].kind = OPK_IMM;
+ mir_intrin->dsts[0].v.imm = 131;
+ mir_intrin->args[0].kind = OPK_IMM;
+ mir_intrin->args[0].v.imm = 132;
+ mir_intrin->result_vals[0] = 133;
+ }
+ CU_EXPECT(u,
+ hir_phi->pred_blocks[0] == 3 && hir_phi->pred_vals[0] == 13 &&
+ hir_switch->cases[0].value == 21 &&
+ hir_switch->cases[0].block == 5 &&
+ hir_indirect->targets[0] == 7 &&
+ hir_intrin->dsts[0].kind == OPK_REG &&
+ hir_intrin->dsts[0].v.reg == dst0 &&
+ hir_intrin->args[0].kind == OPK_REG &&
+ hir_intrin->args[0].v.reg == arg0 &&
+ hir_intrin->result_vals[0] == 31,
+ "mutating MIR structured aux arrays changed semantic HIR");
+
+ hir_phi->pred_blocks[1] = 204;
+ hir_phi->pred_vals[1] = 214;
+ hir_switch->cases[1].value = 222;
+ hir_switch->cases[1].block = 206;
+ hir_indirect->targets[1] = 208;
+ hir_intrin->dsts[1].v.reg = dst0;
+ hir_intrin->args[1].v.reg = arg0;
+ hir_intrin->result_vals[1] = 232;
+ CU_EXPECT(u,
+ mir_phi && mir_phi->pred_blocks[1] == 4 &&
+ mir_phi->pred_vals[1] == 14 && mir_switch &&
+ mir_switch->cases[1].value == 22 &&
+ mir_switch->cases[1].block == 6 && mir_indirect &&
+ mir_indirect->targets[1] == 8 && mir_intrin &&
+ mir_intrin->dsts[1].v.reg == 9 &&
+ mir_intrin->args[1].v.reg == 11 &&
+ mir_intrin->result_vals[1] == 32,
+ "mutating HIR structured aux arrays changed location MIR");
+
+ kit_compiler_free(kc);
+}
+
+static void asm_aux_resolved_metadata_is_independent(KitUnit* u) {
+ KitCompiler* kc = NULL;
+ Compiler* c;
+ Func* f;
+ KitCgTypeId i64_type;
+ CGFuncDesc desc;
+ PReg out_reg, in_reg;
+ IRAsmAux* hir_aux;
+ IRAsmAux* mir_aux;
+ Inst* asm_in;
+ InstId asm_id;
+ FrameSlot memory_slot;
+ FrameSlotDesc memory_desc;
+ Sym out_name, in_name, clobber_name;
+ u32 entry;
+
+ CU_EXPECT(u,
+ kit_unit_compiler_new(
+ u, kit_unit_target(KIT_ARCH_ARM_64, KIT_OS_LINUX, KIT_OBJ_ELF),
+ &kc) == KIT_OK &&
+ kc != NULL,
+ "compiler allocation failed for asm-aux clone test");
+ if (!kc) return;
+ c = (Compiler*)kc;
+ i64_type = kit_cg_type_builtin(kc, KIT_CG_BUILTIN_I64);
+ out_name = kit_sym_intern(kc, KIT_SLICE_LIT("out"));
+ in_name = kit_sym_intern(kc, KIT_SLICE_LIT("in"));
+ clobber_name = kit_sym_intern(kc, KIT_SLICE_LIT("x12"));
+
+ memset(&desc, 0, sizeof desc);
+ desc.fn_type = i64_type;
+ f = ir_func_new(c, &desc);
+ entry = ir_block_new(f);
+ f->entry = entry;
+ ir_note_emit(f, entry);
+
+ out_reg = ir_alloc_preg(f, i64_type, RC_INT);
+ in_reg = ir_alloc_preg(f, i64_type, RC_INT);
+ f->preg_locs = arena_zarray(f->arena, OptLoc, f->npregs);
+ f->preg_info = arena_zarray(f->arena, OptPRegInfo, f->npregs);
+ set_hard_loc(f, out_reg, 6);
+ set_hard_loc(f, in_reg, 7);
+ memset(&memory_desc, 0, sizeof memory_desc);
+ memory_desc.type = i64_type;
+ memory_desc.size = 8;
+ memory_desc.align = 8;
+ memory_desc.kind = FS_LOCAL;
+ memory_slot = ir_frame_slot_new(f, &memory_desc);
+
+ asm_in = ir_emit(f, entry, IR_ASM_BLOCK);
+ asm_id = asm_in->id;
+ asm_in->def = out_reg;
+ hir_aux = arena_znew(f->arena, IRAsmAux);
+ hir_aux->tmpl = "add %0, %1";
+ hir_aux->nout = 1;
+ hir_aux->nin = 2;
+ hir_aux->nclob = 1;
+ hir_aux->outs = arena_zarray(f->arena, AsmConstraint, hir_aux->nout);
+ hir_aux->ins = arena_zarray(f->arena, AsmConstraint, hir_aux->nin);
+ hir_aux->clobbers = arena_array(f->arena, Sym, hir_aux->nclob);
+ hir_aux->out_ops = arena_zarray(f->arena, Operand, hir_aux->nout);
+ hir_aux->in_ops = arena_zarray(f->arena, Operand, hir_aux->nin);
+ hir_aux->out_reg_reqs =
+ arena_zarray(f->arena, IRAsmRegRequirement, hir_aux->nout);
+ hir_aux->in_reg_reqs =
+ arena_zarray(f->arena, IRAsmRegRequirement, hir_aux->nin);
+ hir_aux->outs[0].str = "=r";
+ hir_aux->outs[0].name = out_name;
+ hir_aux->outs[0].type = i64_type;
+ hir_aux->outs[0].dir = KIT_CG_ASM_OUT;
+ hir_aux->ins[0].str = "r";
+ hir_aux->ins[0].name = in_name;
+ hir_aux->ins[0].type = i64_type;
+ hir_aux->ins[0].dir = KIT_CG_ASM_IN;
+ hir_aux->ins[1].str = "m";
+ hir_aux->ins[1].type = i64_type;
+ hir_aux->ins[1].dir = KIT_CG_ASM_IN;
+ hir_aux->clobbers[0] = clobber_name;
+ hir_aux->out_ops[0] = reg_op(out_reg, i64_type);
+ hir_aux->in_ops[0] = reg_op(in_reg, i64_type);
+ hir_aux->in_ops[1].kind = OPK_LOCAL;
+ hir_aux->in_ops[1].cls = RC_INT;
+ hir_aux->in_ops[1].type = i64_type;
+ hir_aux->in_ops[1].v.frame_slot = memory_slot;
+ hir_aux->out_reg_reqs[0].allowed_mask = 0x000000f0u;
+ hir_aux->out_reg_reqs[0].fixed_reg = 6;
+ hir_aux->out_reg_reqs[0].cls = RC_INT;
+ hir_aux->out_reg_reqs[0].present = 1;
+ hir_aux->in_reg_reqs[0].allowed_mask = 0x0000ff00u;
+ hir_aux->in_reg_reqs[0].fixed_reg = -1;
+ hir_aux->in_reg_reqs[0].cls = RC_INT;
+ hir_aux->in_reg_reqs[0].present = 1;
+ hir_aux->in_reg_reqs[1].fixed_reg = -1;
+ hir_aux->clobber_mask[RC_INT] = (1u << 3) | (1u << 12);
+ hir_aux->clobber_mask[RC_FP] = 1u << 9;
+ hir_aux->clobber_mask[RC_VEC] = 1u << 2;
+ hir_aux->clobber_abi_sets = KIT_CG_ASM_CLOBBER_ABI_CALLER_SAVED;
+ hir_aux->has_memory_constraint = 1;
+ asm_in->extra.aux = hir_aux;
+
+ opt_lower_to_mir(f, NULL);
+ asm_in = f->mir ? find_inst(&f->mir->blocks[entry], asm_id) : NULL;
+ mir_aux = asm_in ? (IRAsmAux*)asm_in->extra.aux : NULL;
+ CU_EXPECT(u, asm_in && mir_aux && mir_aux != hir_aux,
+ "asm aux object aliases HIR after MIR lowering");
+ CU_EXPECT(u,
+ mir_aux && mir_aux->outs != hir_aux->outs &&
+ mir_aux->ins != hir_aux->ins &&
+ mir_aux->clobbers != hir_aux->clobbers &&
+ mir_aux->out_ops != hir_aux->out_ops &&
+ mir_aux->in_ops != hir_aux->in_ops &&
+ mir_aux->out_reg_reqs != hir_aux->out_reg_reqs &&
+ mir_aux->in_reg_reqs != hir_aux->in_reg_reqs,
+ "asm aux nested mutable arrays alias HIR");
+ CU_EXPECT(u,
+ mir_aux &&
+ memcmp(mir_aux->clobber_mask, hir_aux->clobber_mask,
+ sizeof hir_aux->clobber_mask) == 0 &&
+ mir_aux->clobber_abi_sets == hir_aux->clobber_abi_sets &&
+ mir_aux->has_memory_constraint ==
+ hir_aux->has_memory_constraint &&
+ memcmp(mir_aux->out_reg_reqs, hir_aux->out_reg_reqs,
+ sizeof hir_aux->out_reg_reqs[0]) == 0 &&
+ memcmp(mir_aux->in_reg_reqs, hir_aux->in_reg_reqs,
+ sizeof hir_aux->in_reg_reqs[0] * hir_aux->nin) == 0,
+ "resolved asm placement metadata did not survive MIR cloning");
+ CU_EXPECT(u,
+ hir_aux->out_ops[0].kind == OPK_REG &&
+ hir_aux->out_ops[0].v.reg == out_reg &&
+ hir_aux->in_ops[0].kind == OPK_REG &&
+ hir_aux->in_ops[0].v.reg == in_reg && mir_aux &&
+ mir_aux->out_ops[0].kind == OPK_REG &&
+ mir_aux->out_ops[0].v.reg == 6 &&
+ mir_aux->in_ops[0].kind == OPK_REG &&
+ mir_aux->in_ops[0].v.reg == 7 &&
+ hir_aux->in_ops[1].kind == OPK_LOCAL &&
+ hir_aux->in_ops[1].v.frame_slot == memory_slot &&
+ mir_aux->in_ops[1].kind == OPK_LOCAL &&
+ mir_aux->in_ops[1].v.frame_slot == memory_slot,
+ "asm operand rewrite crossed the HIR/MIR ownership boundary");
+
+ if (mir_aux) {
+ mir_aux->out_reg_reqs[0].allowed_mask = 1u << 20;
+ mir_aux->out_reg_reqs[0].fixed_reg = 20;
+ mir_aux->in_reg_reqs[0].present = 0;
+ mir_aux->clobber_mask[RC_INT] = 1u << 22;
+ mir_aux->has_memory_constraint = 0;
+ mir_aux->outs[0].name = in_name;
+ mir_aux->ins[0].str = "m";
+ mir_aux->clobbers[0] = out_name;
+ mir_aux->out_ops[0].v.reg = 20;
+ mir_aux->in_ops[0].kind = OPK_IMM;
+ mir_aux->in_ops[0].v.imm = 42;
+ }
+ CU_EXPECT(u,
+ hir_aux->out_reg_reqs[0].allowed_mask == 0x000000f0u &&
+ hir_aux->out_reg_reqs[0].fixed_reg == 6 &&
+ hir_aux->in_reg_reqs[0].present == 1 &&
+ hir_aux->clobber_mask[RC_INT] ==
+ ((1u << 3) | (1u << 12)) &&
+ hir_aux->has_memory_constraint == 1 &&
+ hir_aux->outs[0].name == out_name &&
+ strcmp(hir_aux->ins[0].str, "r") == 0 &&
+ hir_aux->clobbers[0] == clobber_name &&
+ hir_aux->out_ops[0].kind == OPK_REG &&
+ hir_aux->out_ops[0].v.reg == out_reg &&
+ hir_aux->in_ops[0].kind == OPK_REG &&
+ hir_aux->in_ops[0].v.reg == in_reg,
+ "mutating MIR asm metadata changed the semantic HIR graph");
+
+ /* Prove independence in the other direction as well: the already-mutated
+ * MIR requirements and operands must not observe later HIR changes. */
+ hir_aux->out_reg_reqs[0].allowed_mask = 1u;
+ hir_aux->in_ops[0].v.reg = out_reg;
+ CU_EXPECT(u,
+ mir_aux && mir_aux->out_reg_reqs[0].allowed_mask == (1u << 20) &&
+ mir_aux->in_ops[0].kind == OPK_IMM &&
+ mir_aux->in_ops[0].v.imm == 42,
+ "mutating HIR asm metadata changed the location MIR graph");
+
+ kit_compiler_free(kc);
+}
+
+int main(void) {
+ KitUnit u;
+ kit_unit_init(&u);
+ machinize_rejects_cross_owner_registers(&u);
+ mir_verify_rejects_private_temp_escape(&u);
+ mir_commit_propagates_instruction_namespace(&u);
+ spilled_values_remain_locations(&u);
+ spilled_frame_address_remains_recipe(&u);
+ call_aux_is_independent(&u);
+ structured_aux_arrays_are_independent(&u);
+ asm_aux_resolved_metadata_is_independent(&u);
+ kit_unit_summary(&u, "location-mir");
+ return kit_unit_status(&u);
+}
diff --git a/test/opt/macho_block_alt_entry.sh b/test/opt/macho_block_alt_entry.sh
@@ -0,0 +1,75 @@
+#!/usr/bin/env bash
+# Mach-O MH_SUBSECTIONS_VIA_SYMBOLS regression: relocation-anchor symbols for
+# switch/label blocks must be N_ALT_ENTRY symbols pinned inside their function,
+# not independently collectible atoms. Otherwise --gc-sections can retain the
+# function's entry atom while dropping a directly-branched-to switch dispatch
+# or common return block (intra-function branches are already resolved and do
+# not carry linker relocations).
+set -euo pipefail
+
+ROOT="$(cd "$(dirname "$0")/../.." && pwd)"
+KIT="${KIT:-$ROOT/build/kit}"
+ROUNDTRIP="${ROUNDTRIP:-$ROOT/build/test/kit-roundtrip-macho}"
+WORK="$ROOT/build/test/opt/macho_block_alt_entry"
+mkdir -p "$WORK"
+
+cat > "$WORK/dispatch.c" <<'EOF'
+int dispatch(unsigned x) {
+ if (x < 256) return 7;
+ switch (x) {
+ case 256: return 11;
+ case 257: return 13;
+ case 258: return 17;
+ case 259: return 19;
+ case 260: return 23;
+ case 261: return 29;
+ case 262: return 31;
+ case 263: return 37;
+ case 264: return 41;
+ default: return 43;
+ }
+}
+EOF
+
+cat > "$WORK/main.c" <<'EOF'
+extern int dispatch(unsigned);
+int main(void) { return dispatch(123) == 7 ? 0 : 1; }
+EOF
+
+COMMON=(-target aarch64-apple-darwin -O1 -ffunction-sections
+ -fdata-sections -ffreestanding)
+"$KIT" cc "${COMMON[@]}" -c "$WORK/dispatch.c" -o "$WORK/dispatch.o" \
+ > "$WORK/dispatch.cc.out" 2>&1
+"$KIT" cc "${COMMON[@]}" -c "$WORK/main.c" -o "$WORK/main.o" \
+ > "$WORK/main.cc.out" 2>&1
+
+# Read and re-emit the defining object before linking. This makes the test
+# cover both halves of the format boundary: Mach-O read must collapse pinned
+# block symbols into their owner atom, and emit must preserve the subordinate
+# semantic as N_ALT_ENTRY on the second object too.
+"$ROUNDTRIP" --require-atom-subordinate "$WORK/dispatch.o" \
+ "$WORK/dispatch.roundtrip.o" > "$WORK/dispatch.roundtrip.out" 2>&1
+
+# Force the serialized Mach-O reader/linker path: compiling and linking in one
+# driver invocation can keep the original ObjBuilder and would not exercise
+# MH_SUBSECTIONS_VIA_SYMBOLS reconstruction.
+"$KIT" cc -target aarch64-apple-darwin -nostdlib -Wl,--gc-sections \
+ -Wl,-e,main "$WORK/main.o" "$WORK/dispatch.roundtrip.o" -o "$WORK/linked" \
+ > "$WORK/link.out" 2>&1
+"$KIT" objdump -d "$WORK/linked" > "$WORK/linked.dis" 2>&1
+
+# The case body and indirect table dispatch live after block-label boundaries.
+# Without N_ALT_ENTRY they are collected and the surviving fast-path branches
+# outside __text; retaining both instructions pins the full function content.
+if ! grep -Eq '\bmovz[[:space:]]+w0, 0xb\b' "$WORK/linked.dis"; then
+ printf 'macho_block_alt_entry FAILED: case block was GC-collected\n' >&2
+ sed 's/^/ | /' "$WORK/linked.dis" >&2
+ exit 1
+fi
+if ! grep -Eq '\bbr[[:space:]]+x[0-9]+\b' "$WORK/linked.dis"; then
+ printf 'macho_block_alt_entry FAILED: switch dispatch was GC-collected\n' >&2
+ sed 's/^/ | /' "$WORK/linked.dis" >&2
+ exit 1
+fi
+
+printf 'macho_block_alt_entry: OK (block anchors stay pinned through Mach-O GC)\n'
diff --git a/test/opt/native_emit_frame_dst_test.c b/test/opt/native_emit_frame_dst_test.c
@@ -0,0 +1,1751 @@
+#include <stdio.h>
+#include <string.h>
+
+#include "lib/kit_unit.h"
+#include "opt/opt.h"
+
+/* NativeTarget's vtable was declared against the semantic descriptor before
+ * opt/ir.h introduced its optimizer-private compatibility macro. Keep the
+ * callback signature on that original boundary; use OptCGFuncDesc explicitly
+ * when constructing the optimizer Func below. */
+#undef CGFuncDesc
+
+typedef struct MockNative {
+ NativeTarget base;
+ KitUnit *unit;
+ NativeAllocClassInfo classes[2];
+ NativeRegInfo regs;
+ Reg int_temps[4];
+ NativeCallPlanMove call_arg;
+ NativeCallPlanRet call_ret;
+ NativeCallPlanMove phase_call_args[2];
+ NativeCallPlanRet ret_parts[2];
+ u32 callback_mask;
+ u32 writeback_slots;
+ u32 loads;
+ u32 load_addrs;
+ u32 frame_loads;
+ u32 reg_address_loads;
+ u32 in_place_reg_address_loads;
+ u32 reg_address_stores;
+ u32 in_place_reg_address_bitfield_loads;
+ u32 atomic_loads;
+ u32 atomic_stores;
+ Reg frame_load_dsts[16];
+ u32 nframe_load_dsts;
+ u32 reg_values[32];
+ u32 phase_callee_value;
+ u32 phase_arg_values[2];
+ u32 calls;
+ u8 report_bitfield_store_clobber;
+ u8 plan_two_arg_indirect_call;
+ u8 plan_two_part_ret;
+ u8 reject_indexed_addr;
+ MCLabel next_label;
+} MockNative;
+
+enum {
+ SAW_LOAD_CONST = 1u << 0,
+ SAW_TLS_ADDR = 1u << 1,
+ SAW_BITFIELD_LOAD = 1u << 2,
+ SAW_LABEL_ADDR = 1u << 3,
+ SAW_ALLOCA = 1u << 4,
+ SAW_INTRINSIC = 1u << 5,
+ SAW_BINOP = 1u << 6,
+ SAW_ATOMIC_RMW = 1u << 7,
+ SAW_ATOMIC_CAS = 1u << 8,
+ SAW_BITFIELD_STORE = 1u << 9,
+};
+
+static MockNative *mock_of(NativeTarget *target) {
+ return (MockNative *)target;
+}
+
+static void expect_reg(MockNative *mock, NativeLoc loc, const char *what) {
+ CU_EXPECT(mock->unit, loc.kind == NATIVE_LOC_REG,
+ "%s must receive a physical register, got location kind %u", what,
+ (unsigned)loc.kind);
+}
+
+static NativeAllocClass mock_class_for_type(NativeTarget *target,
+ KitCgTypeId type) {
+ return cg_type_is_float(target->c, type) ? NATIVE_REG_FP : NATIVE_REG_INT;
+}
+
+static int mock_addr_legal(NativeTarget *target, const NativeAddr *addr,
+ MemAccess mem) {
+ (void)mem;
+ return !mock_of(target)->reject_indexed_addr ||
+ addr->index_kind == NATIVE_ADDR_INDEX_NONE;
+}
+
+static void mock_func_begin_known_frame(NativeTarget *target,
+ const CGFuncDesc *desc,
+ const NativeKnownFrameDesc *frame,
+ NativeFrameSlot *out_slots) {
+ (void)target;
+ (void)desc;
+ for (u32 i = 0; i < frame->nslots; ++i)
+ out_slots[i] = i + 1u;
+}
+
+static void mock_func_end(NativeTarget *target) { (void)target; }
+
+static MCLabel mock_label_new(NativeTarget *target) {
+ return ++mock_of(target)->next_label;
+}
+
+static void mock_label_place(NativeTarget *target, MCLabel label) {
+ (void)target;
+ (void)label;
+}
+
+static void mock_load_imm(NativeTarget *target, NativeLoc dst, i64 imm) {
+ MockNative *mock = mock_of(target);
+ expect_reg(mock, dst, "load_imm destination");
+ if (dst.v.reg < 32u) mock->reg_values[dst.v.reg] = (u32)imm;
+}
+
+static void mock_move(NativeTarget *target, NativeLoc dst, NativeLoc src) {
+ MockNative *mock = mock_of(target);
+ expect_reg(mock, dst, "move destination");
+ expect_reg(mock, src, "move source");
+ if (dst.v.reg < 32u && src.v.reg < 32u)
+ mock->reg_values[dst.v.reg] = mock->reg_values[src.v.reg];
+}
+
+static void mock_load_const(NativeTarget *target, NativeLoc dst,
+ ConstBytes bytes) {
+ MockNative *mock = mock_of(target);
+ expect_reg(mock, dst, "load_const destination");
+ CU_EXPECT(mock->unit, bytes.size == 8u,
+ "load_const byte payload should remain intact");
+ mock->callback_mask |= SAW_LOAD_CONST;
+}
+
+static void mock_load(NativeTarget *target, NativeLoc dst, NativeAddr addr,
+ MemAccess mem) {
+ MockNative *mock = mock_of(target);
+ expect_reg(mock, dst, "frame materialization destination");
+ CU_EXPECT(mock->unit, mem.size != 0u,
+ "frame materialization should retain its access width");
+ if (addr.base_kind == NATIVE_ADDR_BASE_FRAME) {
+ if (mock->nframe_load_dsts <
+ sizeof mock->frame_load_dsts / sizeof mock->frame_load_dsts[0])
+ mock->frame_load_dsts[mock->nframe_load_dsts++] = dst.v.reg;
+ ++mock->frame_loads;
+ if (dst.v.reg < 32u)
+ mock->reg_values[dst.v.reg] = addr.base.frame;
+ } else if (addr.base_kind == NATIVE_ADDR_BASE_REG) {
+ ++mock->reg_address_loads;
+ if (dst.v.reg == addr.base.reg) ++mock->in_place_reg_address_loads;
+ } else {
+ CU_EXPECT(mock->unit, 0,
+ "mock load requires a direct frame or register address, got %u",
+ (unsigned)addr.base_kind);
+ }
+ ++mock->loads;
+}
+
+static void mock_load_addr(NativeTarget *target, NativeLoc dst,
+ NativeAddr addr) {
+ MockNative *mock = mock_of(target);
+ u32 value = 0;
+ expect_reg(mock, dst, "address materialization destination");
+ CU_EXPECT(mock->unit,
+ addr.base_kind == NATIVE_ADDR_BASE_REG &&
+ addr.index_kind == NATIVE_ADDR_INDEX_REG,
+ "pressure regression must materialize a register base+index");
+ CU_EXPECT(mock->unit, dst.v.reg == addr.base.reg,
+ "scoped address base should be reusable when no third temp exists");
+ if (addr.base_kind == NATIVE_ADDR_BASE_REG && addr.base.reg < 32u)
+ value = mock->reg_values[addr.base.reg];
+ if (addr.index_kind == NATIVE_ADDR_INDEX_REG && addr.index.reg < 32u)
+ value += mock->reg_values[addr.index.reg] << addr.log2_scale;
+ value += (u32)addr.offset;
+ if (dst.v.reg < 32u) mock->reg_values[dst.v.reg] = value;
+ ++mock->load_addrs;
+}
+
+static void mock_store(NativeTarget *target, NativeAddr addr, NativeLoc src,
+ MemAccess mem) {
+ MockNative *mock = mock_of(target);
+ expect_reg(mock, src, "frame writeback source");
+ CU_EXPECT(mock->unit, mem.size != 0u,
+ "frame writeback should retain its access width");
+ if (addr.base_kind == NATIVE_ADDR_BASE_FRAME) {
+ CU_EXPECT(mock->unit, addr.base.frame > 0u && addr.base.frame < 32u,
+ "frame writeback slot should be mapped and in test range");
+ if (addr.base.frame < 32u)
+ mock->writeback_slots |= 1u << addr.base.frame;
+ } else if (addr.base_kind == NATIVE_ADDR_BASE_REG) {
+ CU_EXPECT(mock->unit, src.v.reg != addr.base.reg,
+ "store value must remain distinct from its address base");
+ ++mock->reg_address_stores;
+ } else {
+ CU_EXPECT(mock->unit, 0,
+ "mock store requires a direct frame or register address, got %u",
+ (unsigned)addr.base_kind);
+ }
+}
+
+static void mock_tls_addr_of(NativeTarget *target, NativeLoc dst, ObjSymId sym,
+ i64 addend) {
+ MockNative *mock = mock_of(target);
+ expect_reg(mock, dst, "tls_addr_of destination");
+ CU_EXPECT(mock->unit, sym == 17u && addend == 9,
+ "tls address metadata should remain intact");
+ mock->callback_mask |= SAW_TLS_ADDR;
+}
+
+static void mock_bitfield_load(NativeTarget *target, NativeLoc dst,
+ NativeAddr addr, BitFieldAccess access) {
+ MockNative *mock = mock_of(target);
+ expect_reg(mock, dst, "bitfield_load destination");
+ CU_EXPECT(mock->unit,
+ addr.base_kind == NATIVE_ADDR_BASE_FRAME ||
+ addr.base_kind == NATIVE_ADDR_BASE_REG,
+ "bitfield record should remain a frame or register address");
+ if (addr.base_kind == NATIVE_ADDR_BASE_REG && dst.v.reg == addr.base.reg)
+ ++mock->in_place_reg_address_bitfield_loads;
+ CU_EXPECT(mock->unit, access.bit_offset == 3u && access.bit_width == 11u,
+ "bitfield geometry should remain intact");
+ mock->callback_mask |= SAW_BITFIELD_LOAD;
+}
+
+static void mock_bitfield_store(NativeTarget *target, NativeAddr addr,
+ NativeLoc src, BitFieldAccess access) {
+ MockNative *mock = mock_of(target);
+ expect_reg(mock, src, "bitfield_store source");
+ CU_EXPECT(mock->unit, addr.base_kind == NATIVE_ADDR_BASE_FRAME,
+ "bitfield store record should remain a frame address");
+ CU_EXPECT(mock->unit, access.bit_offset == 3u && access.bit_width == 11u,
+ "bitfield store geometry should remain intact");
+ mock->callback_mask |= SAW_BITFIELD_STORE;
+}
+
+static int mock_machine_op_clobbers(
+ NativeTarget *target, const NativeMachineOp *op,
+ u32 mask[NATIVE_CALL_PLAN_CLASSES]) {
+ MockNative *mock = mock_of(target);
+ mask[0] = mask[1] = mask[2] = 0u;
+ if (!mock->report_bitfield_store_clobber ||
+ op->kind != NATIVE_MOP_BITFIELD_STORE)
+ return 0;
+ mask[NATIVE_REG_INT] = 1u << 10u;
+ return 1;
+}
+
+static void mock_load_label_addr(NativeTarget *target, NativeLoc dst,
+ MCLabel label) {
+ MockNative *mock = mock_of(target);
+ expect_reg(mock, dst, "load_label_addr destination");
+ CU_EXPECT(mock->unit, label != MC_LABEL_NONE,
+ "label address should reference an allocated native label");
+ mock->callback_mask |= SAW_LABEL_ADDR;
+}
+
+static void mock_alloca(NativeTarget *target, NativeLoc dst, NativeLoc size,
+ u32 align) {
+ MockNative *mock = mock_of(target);
+ expect_reg(mock, dst, "alloca destination");
+ expect_reg(mock, size, "alloca size");
+ CU_EXPECT(mock->unit, dst.v.reg != size.v.reg,
+ "alloca destination must not clobber its size input");
+ CU_EXPECT(mock->unit, align == 32u, "alloca alignment should remain intact");
+ mock->callback_mask |= SAW_ALLOCA;
+}
+
+static void mock_binop(NativeTarget *target, BinOp op, NativeLoc dst,
+ NativeLoc lhs, NativeLoc rhs) {
+ MockNative *mock = mock_of(target);
+ expect_reg(mock, dst, "binop destination");
+ expect_reg(mock, lhs, "binop lhs");
+ expect_reg(mock, rhs, "binop rhs");
+ CU_EXPECT(mock->unit, op == BO_IADD, "binop opcode should remain intact");
+ CU_EXPECT(mock->unit, dst.v.reg != rhs.v.reg && lhs.v.reg != rhs.v.reg,
+ "all-stack binop must keep its rhs distinct while allowing the "
+ "dead lhs register to become the destination");
+ mock->callback_mask |= SAW_BINOP;
+}
+
+static void mock_atomic_rmw(NativeTarget *target, KitCgAtomicOp op,
+ NativeLoc dst, NativeAddr addr, NativeLoc val,
+ MemAccess mem, KitCgMemOrder order) {
+ MockNative *mock = mock_of(target);
+ expect_reg(mock, dst, "atomic_rmw destination");
+ expect_reg(mock, val, "atomic_rmw value");
+ CU_EXPECT(mock->unit, dst.v.reg != val.v.reg,
+ "atomic_rmw destination must not alias its value input");
+ CU_EXPECT(mock->unit, addr.base_kind == NATIVE_ADDR_BASE_FRAME,
+ "atomic_rmw address should retain direct frame storage");
+ CU_EXPECT(mock->unit,
+ op == KIT_CG_ATOMIC_ADD && mem.size == 8u &&
+ order == KIT_CG_MO_SEQ_CST,
+ "atomic_rmw metadata should remain intact");
+ mock->callback_mask |= SAW_ATOMIC_RMW;
+}
+
+static void mock_atomic_load(NativeTarget *target, NativeLoc dst,
+ NativeAddr addr, MemAccess mem,
+ KitCgMemOrder order) {
+ MockNative *mock = mock_of(target);
+ expect_reg(mock, dst, "atomic_load destination");
+ CU_EXPECT(mock->unit,
+ addr.base_kind == NATIVE_ADDR_BASE_REG &&
+ addr.index_kind == NATIVE_ADDR_INDEX_NONE &&
+ dst.v.reg == addr.base.reg,
+ "atomic load should consume and reuse one collapsed scoped base");
+ CU_EXPECT(mock->unit,
+ mem.size == 8u && order == KIT_CG_MO_SEQ_CST,
+ "atomic load metadata should remain intact");
+ ++mock->atomic_loads;
+}
+
+static void mock_atomic_store(NativeTarget *target, NativeAddr addr,
+ NativeLoc val, MemAccess mem,
+ KitCgMemOrder order) {
+ MockNative *mock = mock_of(target);
+ expect_reg(mock, val, "atomic_store value");
+ CU_EXPECT(mock->unit, addr.base_kind == NATIVE_ADDR_BASE_REG,
+ "atomic store pressure address should be materialized");
+ CU_EXPECT(mock->unit,
+ addr.base_kind != NATIVE_ADDR_BASE_REG ||
+ val.v.reg != addr.base.reg,
+ "atomic store reused the overwritten address-base cache fact");
+ CU_EXPECT(mock->unit,
+ mem.size == 8u && order == KIT_CG_MO_SEQ_CST,
+ "atomic store metadata should remain intact");
+ ++mock->atomic_stores;
+}
+
+static void mock_atomic_cas(NativeTarget *target, NativeLoc prior, NativeLoc ok,
+ NativeAddr addr, NativeLoc expected,
+ NativeLoc desired, MemAccess mem,
+ KitCgMemOrder success, KitCgMemOrder failure) {
+ MockNative *mock = mock_of(target);
+ NativeLoc locs[4] = {prior, ok, expected, desired};
+ for (u32 i = 0; i < 4u; ++i)
+ expect_reg(mock, locs[i], "atomic_cas scalar operand");
+ for (u32 i = 0; i < 4u; ++i)
+ for (u32 j = i + 1u; j < 4u; ++j)
+ CU_EXPECT(mock->unit, locs[i].v.reg != locs[j].v.reg,
+ "all-stack atomic_cas must lease four simultaneous registers");
+ CU_EXPECT(mock->unit, addr.base_kind == NATIVE_ADDR_BASE_FRAME,
+ "atomic_cas address should retain direct frame storage");
+ CU_EXPECT(mock->unit,
+ mem.size == 8u && success == KIT_CG_MO_ACQ_REL &&
+ failure == KIT_CG_MO_ACQUIRE,
+ "atomic_cas metadata should remain intact");
+ mock->callback_mask |= SAW_ATOMIC_CAS;
+}
+
+static void mock_intrinsic(NativeTarget *target, IntrinKind kind,
+ const NativeLoc *dsts, u32 ndst,
+ const NativeLoc *args, u32 narg) {
+ MockNative *mock = mock_of(target);
+ CU_EXPECT(mock->unit,
+ kind == INTRIN_UADD_OVERFLOW && ndst == 2u && narg == 2u,
+ "multi-result intrinsic shape should remain intact");
+ for (u32 i = 0; i < ndst; ++i)
+ expect_reg(mock, dsts[i], "intrinsic destination");
+ for (u32 i = 0; i < narg; ++i)
+ expect_reg(mock, args[i], "intrinsic argument");
+ if (ndst == 2u && narg == 2u) {
+ Reg regs[4] = {dsts[0].v.reg, dsts[1].v.reg, args[0].v.reg, args[1].v.reg};
+ for (u32 i = 0; i < 4u; ++i)
+ for (u32 j = i + 1u; j < 4u; ++j)
+ CU_EXPECT(mock->unit, regs[i] != regs[j],
+ "simultaneous intrinsic operands must have distinct temps");
+ }
+ mock->callback_mask |= SAW_INTRINSIC;
+}
+
+static NativeLoc mock_reg_loc(KitCgTypeId type, NativeAllocClass cls, Reg reg) {
+ NativeLoc loc;
+ memset(&loc, 0, sizeof loc);
+ loc.kind = NATIVE_LOC_REG;
+ loc.cls = (u8)cls;
+ loc.type = type;
+ loc.v.reg = reg;
+ return loc;
+}
+
+/* Deliberately source the post-call copy from the same frame value used by
+ * the argument. The call clobbers the sole staging register, so the emitter
+ * must end the pre-call temp/cache phase and reload this source afterwards. */
+static void mock_plan_call(NativeTarget *target, const NativeCallDesc *desc,
+ NativeCallPlan *plan) {
+ MockNative *mock = mock_of(target);
+ if (mock->plan_two_arg_indirect_call) {
+ NativeLoc staged_callee;
+ CU_EXPECT(mock->unit,
+ desc->nargs == 2u && desc->nresults == 0u &&
+ desc->callee.kind == NATIVE_LOC_REG,
+ "indirect callee must be materialized before the argument plan");
+ if (desc->nargs != 2u || desc->nresults != 0u)
+ return;
+ if (desc->callee.kind == NATIVE_LOC_REG) {
+ staged_callee =
+ mock_reg_loc(desc->callee.type, NATIVE_REG_INT, 12u);
+ mock_move(target, staged_callee, desc->callee);
+ plan->callee = staged_callee;
+ mock->phase_callee_value = mock->reg_values[staged_callee.v.reg];
+ } else {
+ /* Keep the mock executable against the pre-fix ordering too: the
+ * emitter used to materialize this plan location after its argument
+ * moves. The expectation above remains the focused red assertion. */
+ plan->callee = desc->callee;
+ if (desc->callee.kind == NATIVE_LOC_FRAME)
+ mock->phase_callee_value = desc->callee.v.frame;
+ }
+ for (u32 i = 0; i < 2u; ++i) {
+ memset(&mock->phase_call_args[i], 0,
+ sizeof mock->phase_call_args[i]);
+ mock->phase_call_args[i].src = desc->args[i];
+ mock->phase_call_args[i].dst =
+ mock_reg_loc(desc->args[i].type, NATIVE_REG_INT, (Reg)(8u + i));
+ mock->phase_call_args[i].mem.type = desc->args[i].type;
+ mock->phase_call_args[i].mem.size = 8u;
+ mock->phase_call_args[i].mem.align = 8u;
+ CU_EXPECT(mock->unit, desc->args[i].kind == NATIVE_LOC_FRAME,
+ "planned call argument %u must retain its frame source", i);
+ if (desc->args[i].kind == NATIVE_LOC_FRAME)
+ mock->phase_arg_values[i] = desc->args[i].v.frame;
+ }
+ plan->args = mock->phase_call_args;
+ plan->nargs = 2u;
+ return;
+ }
+ CU_EXPECT(mock->unit, desc->nargs == 1u && desc->nresults == 1u,
+ "call phase test requires one argument and one result");
+ if (desc->nargs != 1u || desc->nresults != 1u)
+ return;
+ memset(&mock->call_arg, 0, sizeof mock->call_arg);
+ memset(&mock->call_ret, 0, sizeof mock->call_ret);
+ plan->callee = desc->callee;
+ mock->call_arg.src = desc->args[0];
+ mock->call_arg.dst = mock_reg_loc(desc->args[0].type, NATIVE_REG_INT, 8u);
+ mock->call_arg.mem.type = desc->args[0].type;
+ mock->call_arg.mem.size = 8u;
+ mock->call_arg.mem.align = 8u;
+ plan->args = &mock->call_arg;
+ plan->nargs = 1u;
+ mock->call_ret.src = desc->args[0];
+ mock->call_ret.dst = desc->results[0];
+ mock->call_ret.mem.type = desc->results[0].type;
+ mock->call_ret.mem.size = 8u;
+ mock->call_ret.mem.align = 8u;
+ plan->rets = &mock->call_ret;
+ plan->nrets = 1u;
+ plan->clobber_mask[NATIVE_REG_INT] = 1u << 8u;
+}
+
+static void mock_emit_call(NativeTarget *target, const NativeCallPlan *plan) {
+ MockNative *mock = mock_of(target);
+ if (mock->plan_two_arg_indirect_call) {
+ CU_EXPECT(mock->unit,
+ plan->callee.kind == NATIVE_LOC_REG && plan->nargs == 2u &&
+ plan->nrets == 0u,
+ "two-argument indirect call plan changed before emission");
+ if (plan->callee.kind == NATIVE_LOC_REG) {
+ CU_EXPECT(mock->unit,
+ mock->reg_values[plan->callee.v.reg] ==
+ mock->phase_callee_value,
+ "argument staging overwrote the indirect callee");
+ }
+ CU_EXPECT(mock->unit,
+ mock->reg_values[8u] == mock->phase_arg_values[0],
+ "later call setup overwrote completed ABI argument r8");
+ CU_EXPECT(mock->unit,
+ mock->reg_values[9u] == mock->phase_arg_values[1],
+ "second ABI argument did not reach r9");
+ ++mock->calls;
+ return;
+ }
+ CU_EXPECT(mock->unit, plan->nargs == 1u && plan->nrets == 1u,
+ "call plan shape changed before emission");
+ ++mock->calls;
+}
+
+static void mock_plan_ret(NativeTarget *target, const CGFuncDesc *desc,
+ const NativeLoc *value,
+ NativeCallPlanRet **out_rets, u32 *out_nrets) {
+ MockNative *mock = mock_of(target);
+ (void)desc;
+ if (!mock->plan_two_part_ret) {
+ *out_rets = NULL;
+ *out_nrets = 0u;
+ return;
+ }
+ CU_EXPECT(mock->unit, value && value->kind == NATIVE_LOC_FRAME,
+ "two-part return test requires a frame-resident source");
+ if (!value || value->kind != NATIVE_LOC_FRAME) {
+ *out_rets = NULL;
+ *out_nrets = 0u;
+ return;
+ }
+ for (u32 i = 0; i < 2u; ++i) {
+ memset(&mock->ret_parts[i], 0, sizeof mock->ret_parts[i]);
+ mock->ret_parts[i].src =
+ native_loc_stack(value->type, value->v.frame, (i32)(i * 8u));
+ mock->ret_parts[i].dst =
+ mock_reg_loc(value->type, NATIVE_REG_INT, (Reg)(8u + i));
+ mock->ret_parts[i].mem.type = value->type;
+ mock->ret_parts[i].mem.size = 8u;
+ mock->ret_parts[i].mem.align = 8u;
+ }
+ *out_rets = mock->ret_parts;
+ *out_nrets = 2u;
+}
+
+static void mock_ret(NativeTarget *target) { (void)target; }
+
+static void mock_init(MockNative *mock, KitUnit *unit, Compiler *c) {
+ memset(mock, 0, sizeof *mock);
+ mock->unit = unit;
+ mock->base.c = c;
+ mock->int_temps[0] = 8u;
+ mock->int_temps[1] = 9u;
+ mock->classes[NATIVE_REG_INT].cls = NATIVE_REG_INT;
+ mock->classes[NATIVE_REG_INT].scratch = mock->int_temps;
+ mock->classes[NATIVE_REG_INT].nscratch = 2u;
+ mock->classes[NATIVE_REG_INT].emit_temps = mock->int_temps;
+ mock->classes[NATIVE_REG_INT].nemit_temps = 2u;
+ mock->classes[NATIVE_REG_INT].emit_cache_mask =
+ (1u << mock->int_temps[0]) | (1u << mock->int_temps[1]);
+ mock->classes[NATIVE_REG_FP].cls = NATIVE_REG_FP;
+ mock->regs.classes = mock->classes;
+ mock->regs.nclasses = 2u;
+ mock->base.regs = &mock->regs;
+ mock->base.class_for_type = mock_class_for_type;
+ mock->base.addr_legal = mock_addr_legal;
+ mock->base.func_begin_known_frame = mock_func_begin_known_frame;
+ mock->base.func_end = mock_func_end;
+ mock->base.label_new = mock_label_new;
+ mock->base.label_place = mock_label_place;
+ mock->base.load_imm = mock_load_imm;
+ mock->base.move = mock_move;
+ mock->base.load_const = mock_load_const;
+ mock->base.load_addr = mock_load_addr;
+ mock->base.load = mock_load;
+ mock->base.store = mock_store;
+ mock->base.tls_addr_of = mock_tls_addr_of;
+ mock->base.bitfield_load = mock_bitfield_load;
+ mock->base.bitfield_store = mock_bitfield_store;
+ mock->base.load_label_addr = mock_load_label_addr;
+ mock->base.alloca_ = mock_alloca;
+ mock->base.binop = mock_binop;
+ mock->base.atomic_load = mock_atomic_load;
+ mock->base.atomic_store = mock_atomic_store;
+ mock->base.atomic_rmw = mock_atomic_rmw;
+ mock->base.atomic_cas = mock_atomic_cas;
+ mock->base.intrinsic = mock_intrinsic;
+ mock->base.plan_call = mock_plan_call;
+ mock->base.emit_call = mock_emit_call;
+ mock->base.machine_op_clobbers = mock_machine_op_clobbers;
+ mock->base.plan_ret = mock_plan_ret;
+ mock->base.ret = mock_ret;
+}
+
+static Operand stack_op(FrameSlot slot, KitCgTypeId type) {
+ Operand op;
+ memset(&op, 0, sizeof op);
+ op.kind = OPK_STACK;
+ op.cls = RC_INT;
+ op.type = type;
+ op.v.frame_slot = slot;
+ return op;
+}
+
+static Operand frame_addr_op(FrameSlot slot, KitCgTypeId type) {
+ Operand op = stack_op(slot, type);
+ op.kind = OPK_FRAME_ADDR;
+ return op;
+}
+
+static Operand imm_op(i64 value, KitCgTypeId type) {
+ Operand op;
+ memset(&op, 0, sizeof op);
+ op.kind = OPK_IMM;
+ op.cls = RC_INT;
+ op.type = type;
+ op.v.imm = value;
+ return op;
+}
+
+static Operand reg_op(Reg reg, KitCgTypeId type) {
+ Operand op;
+ memset(&op, 0, sizeof op);
+ op.kind = OPK_REG;
+ op.cls = RC_INT;
+ op.type = type;
+ op.v.reg = reg;
+ return op;
+}
+
+static Operand global_op(ObjSymId sym, KitCgTypeId type) {
+ Operand op;
+ memset(&op, 0, sizeof op);
+ op.kind = OPK_GLOBAL;
+ op.cls = RC_INT;
+ op.type = type;
+ op.v.global.sym = sym;
+ return op;
+}
+
+static Operand indirect_frame_op(FrameSlot base, KitCgTypeId ptr_type) {
+ Operand op;
+ memset(&op, 0, sizeof op);
+ op.kind = OPK_INDIRECT;
+ op.cls = RC_INT;
+ op.type = ptr_type;
+ op.v.ind.base = base;
+ op.v.ind.index = REG_NONE;
+ op.v.ind.base_kind = OPT_INDIRECT_FRAME;
+ op.v.ind.index_kind = OPT_INDIRECT_REG;
+ op.v.ind.base_type = ptr_type;
+ return op;
+}
+
+static Operand indirect_frame_index_op(FrameSlot base, FrameSlot index,
+ KitCgTypeId ptr_type,
+ KitCgTypeId index_type) {
+ Operand op = indirect_frame_op(base, ptr_type);
+ op.v.ind.index = index;
+ op.v.ind.index_kind = OPT_INDIRECT_FRAME;
+ op.v.ind.index_type = index_type;
+ op.v.ind.log2_scale = 3u;
+ return op;
+}
+
+static FrameSlot add_slot(Func *f, KitCgTypeId type, u32 size, u32 align) {
+ FrameSlotDesc desc;
+ memset(&desc, 0, sizeof desc);
+ desc.type = type;
+ desc.size = size;
+ desc.align = align;
+ desc.kind = FS_SPILL;
+ return ir_frame_slot_new(f, &desc);
+}
+
+static Inst *emit_with_ops(Func *f, u32 block, IROp op, u32 nopnds) {
+ Inst *in = ir_emit(f, block, op);
+ in->nopnds = nopnds;
+ in->opnds = nopnds ? arena_zarray(f->arena, Operand, nopnds) : NULL;
+ return in;
+}
+
+static void frame_destinations_are_staged(KitUnit *unit) {
+ KitCompiler *kit_c = NULL;
+ Compiler *c;
+ KitCgTypeId i64_type, bool_type, ptr_type;
+ OptCGFuncDesc desc;
+ Func *f;
+ MockNative mock;
+ u32 block;
+ FrameSlot slots[19];
+ u64 constant = 0x1122334455667788ull;
+ Inst *in;
+
+ CU_EXPECT(unit,
+ kit_unit_compiler_new(
+ unit,
+ kit_unit_target(KIT_ARCH_X86_64, KIT_OS_LINUX, KIT_OBJ_ELF),
+ &kit_c) == KIT_OK &&
+ kit_c != NULL,
+ "compiler allocation failed");
+ if (!kit_c)
+ return;
+ c = (Compiler *)kit_c;
+ i64_type = kit_cg_type_builtin(kit_c, KIT_CG_BUILTIN_I64);
+ bool_type = kit_cg_type_builtin(kit_c, KIT_CG_BUILTIN_BOOL);
+ ptr_type = kit_cg_type_ptr(
+ kit_c, kit_cg_type_builtin(kit_c, KIT_CG_BUILTIN_VOID), 0);
+
+ memset(&desc, 0, sizeof desc);
+ desc.result_type = kit_cg_type_builtin(kit_c, KIT_CG_BUILTIN_VOID);
+ f = ir_func_new(c, &desc);
+ block = ir_block_new(f);
+ f->entry = block;
+ ir_note_emit(f, block);
+ f->opt_rewritten = 1;
+ /* Model x64's deliberately narrow driver-temp bank. The third binop temp
+ * and the extra multi-result temps must be scavenged from dead caller-saved
+ * allocation registers, never represented as MIR values. */
+ for (u32 i = 0; i < 2u; ++i) {
+ f->emit_temp_regs[RC_INT][i] = (Reg)(8u + i);
+ f->opt_reserved_regs[RC_INT] |= 1u << (8u + i);
+ }
+ f->emit_temp_reg_count[RC_INT] = 2u;
+ f->opt_hard_regs[RC_INT][0] = 10u;
+ f->opt_hard_regs[RC_INT][1] = 11u;
+ f->opt_hard_reg_count[RC_INT] = 2u;
+ f->opt_caller_saved[RC_INT] = (1u << 10) | (1u << 11);
+
+ slots[0] = add_slot(f, i64_type, 8u, 8u); /* load_const dst */
+ slots[1] = add_slot(f, ptr_type, 8u, 8u); /* tls dst */
+ slots[2] = add_slot(f, i64_type, 8u, 8u); /* bitfield dst */
+ slots[3] = add_slot(f, i64_type, 8u, 8u); /* bitfield record */
+ slots[4] = add_slot(f, ptr_type, 8u, 8u); /* label dst */
+ slots[5] = add_slot(f, ptr_type, 8u, 8u); /* alloca dst */
+ slots[6] = add_slot(f, i64_type, 8u, 8u); /* intrinsic value */
+ slots[7] = add_slot(f, bool_type, 1u, 1u); /* intrinsic overflow */
+ slots[8] = add_slot(f, i64_type, 8u, 8u); /* intrinsic arg 0 */
+ slots[9] = add_slot(f, i64_type, 8u, 8u); /* intrinsic arg 1 */
+ slots[10] = add_slot(f, i64_type, 8u, 8u); /* binop dst */
+ slots[11] = add_slot(f, i64_type, 8u, 8u); /* binop lhs */
+ slots[12] = add_slot(f, i64_type, 8u, 8u); /* binop rhs */
+ slots[13] = add_slot(f, i64_type, 8u, 8u); /* atomic_rmw dst */
+ slots[14] = add_slot(f, i64_type, 8u, 8u); /* atomic storage */
+ slots[15] = add_slot(f, i64_type, 8u, 8u); /* atomic_rmw value */
+ slots[16] = add_slot(f, i64_type, 8u, 8u); /* atomic_cas prior */
+ slots[17] = add_slot(f, bool_type, 1u, 1u); /* atomic_cas ok */
+ slots[18] = add_slot(f, i64_type, 8u, 8u); /* atomic_cas expected */
+ /* Reuse the binop lhs slot as the CAS desired input: the instructions have
+ * disjoint scopes, so this also checks location materialization caching does
+ * not leak between scopes. */
+
+ in = emit_with_ops(f, block, IR_LOAD_CONST, 1u);
+ in->opnds[0] = stack_op(slots[0], i64_type);
+ in->extra.cbytes.type = i64_type;
+ in->extra.cbytes.bytes = (const u8 *)&constant;
+ in->extra.cbytes.size = 8u;
+ in->extra.cbytes.align = 8u;
+
+ in = emit_with_ops(f, block, IR_TLS_ADDR_OF, 1u);
+ in->opnds[0] = stack_op(slots[1], ptr_type);
+ {
+ IRTlsAux *aux = arena_znew(f->arena, IRTlsAux);
+ aux->sym = 17u;
+ aux->addend = 9;
+ in->extra.aux = aux;
+ }
+
+ in = emit_with_ops(f, block, IR_BITFIELD_LOAD, 2u);
+ in->opnds[0] = stack_op(slots[2], i64_type);
+ in->opnds[1] = frame_addr_op(slots[3], ptr_type);
+ {
+ IRBitFieldAux *aux = arena_znew(f->arena, IRBitFieldAux);
+ aux->access.field_type = i64_type;
+ aux->access.storage.type = i64_type;
+ aux->access.storage.size = 8u;
+ aux->access.storage.align = 8u;
+ aux->access.bit_offset = 3u;
+ aux->access.bit_width = 11u;
+ in->extra.aux = aux;
+ }
+
+ in = emit_with_ops(f, block, IR_LOAD_LABEL_ADDR, 1u);
+ in->opnds[0] = stack_op(slots[4], ptr_type);
+ in->extra.imm = block;
+
+ in = emit_with_ops(f, block, IR_ALLOCA, 2u);
+ in->opnds[0] = stack_op(slots[5], ptr_type);
+ in->opnds[1] = imm_op(96, i64_type);
+ in->extra.imm = 32;
+
+ in = emit_with_ops(f, block, IR_INTRINSIC, 0u);
+ {
+ IRIntrinAux *aux = arena_znew(f->arena, IRIntrinAux);
+ aux->kind = INTRIN_UADD_OVERFLOW;
+ aux->ndst = 2u;
+ aux->narg = 2u;
+ aux->dsts = arena_zarray(f->arena, Operand, aux->ndst);
+ aux->args = arena_zarray(f->arena, Operand, aux->narg);
+ aux->dsts[0] = stack_op(slots[6], i64_type);
+ aux->dsts[1] = stack_op(slots[7], bool_type);
+ aux->args[0] = stack_op(slots[8], i64_type);
+ aux->args[1] = stack_op(slots[9], i64_type);
+ in->extra.aux = aux;
+ }
+
+ in = emit_with_ops(f, block, IR_BINOP, 3u);
+ in->opnds[0] = stack_op(slots[10], i64_type);
+ in->opnds[1] = stack_op(slots[11], i64_type);
+ in->opnds[2] = stack_op(slots[12], i64_type);
+ in->extra.imm = BO_IADD;
+
+ in = emit_with_ops(f, block, IR_ATOMIC_RMW, 3u);
+ in->opnds[0] = stack_op(slots[13], i64_type);
+ in->opnds[1] = frame_addr_op(slots[14], ptr_type);
+ in->opnds[2] = stack_op(slots[15], i64_type);
+ {
+ IRAtomicAux *aux = arena_znew(f->arena, IRAtomicAux);
+ aux->op = KIT_CG_ATOMIC_ADD;
+ aux->mo = KIT_CG_MO_SEQ_CST;
+ aux->mem.type = i64_type;
+ aux->mem.size = 8u;
+ aux->mem.align = 8u;
+ in->extra.aux = aux;
+ }
+
+ in = emit_with_ops(f, block, IR_ATOMIC_CAS, 5u);
+ in->opnds[0] = stack_op(slots[16], i64_type);
+ in->opnds[1] = stack_op(slots[17], bool_type);
+ in->opnds[2] = frame_addr_op(slots[14], ptr_type);
+ in->opnds[3] = stack_op(slots[18], i64_type);
+ in->opnds[4] = stack_op(slots[11], i64_type);
+ {
+ IRCasAux *aux = arena_znew(f->arena, IRCasAux);
+ aux->mem.type = i64_type;
+ aux->mem.size = 8u;
+ aux->mem.align = 8u;
+ aux->success = KIT_CG_MO_ACQ_REL;
+ aux->failure = KIT_CG_MO_ACQUIRE;
+ in->extra.aux = aux;
+ }
+
+ mock_init(&mock, unit, c);
+ opt_emit_native(c, f, &mock.base);
+
+ CU_EXPECT(unit,
+ mock.callback_mask ==
+ (SAW_LOAD_CONST | SAW_TLS_ADDR | SAW_BITFIELD_LOAD |
+ SAW_LABEL_ADDR | SAW_ALLOCA | SAW_INTRINSIC | SAW_BINOP |
+ SAW_ATOMIC_RMW | SAW_ATOMIC_CAS),
+ "all frame-destination emitter hooks should be exercised");
+ {
+ u32 expected = (1u << slots[0]) | (1u << slots[1]) | (1u << slots[2]) |
+ (1u << slots[4]) | (1u << slots[5]) | (1u << slots[6]) |
+ (1u << slots[7]) | (1u << slots[10]) | (1u << slots[13]) |
+ (1u << slots[16]) | (1u << slots[17]);
+ CU_EXPECT(unit, mock.writeback_slots == expected,
+ "each staged destination should be written back exactly once "
+ "(got 0x%x expected 0x%x)",
+ mock.writeback_slots, expected);
+ }
+ CU_EXPECT(unit, mock.loads == 7u,
+ "stack intrinsic/binop/atomic inputs should each materialize once");
+
+ kit_compiler_free(kit_c);
+}
+
+static void cached_frame_value_feeds_indirect_address(KitUnit *unit) {
+ KitCompiler *kit_c = NULL;
+ Compiler *c;
+ KitCgTypeId i64_type, ptr_type;
+ OptCGFuncDesc desc;
+ Func *f;
+ MockNative mock;
+ FrameSlot pointer_slot;
+ Inst *in;
+ u32 block;
+
+ CU_EXPECT(unit,
+ kit_unit_compiler_new(
+ unit,
+ kit_unit_target(KIT_ARCH_X86_64, KIT_OS_LINUX, KIT_OBJ_ELF),
+ &kit_c) == KIT_OK &&
+ kit_c != NULL,
+ "compiler allocation failed for indirect frame-cache test");
+ if (!kit_c)
+ return;
+ c = (Compiler *)kit_c;
+ i64_type = kit_cg_type_builtin(kit_c, KIT_CG_BUILTIN_I64);
+ ptr_type = kit_cg_type_ptr(
+ kit_c, kit_cg_type_builtin(kit_c, KIT_CG_BUILTIN_VOID), 0);
+
+ memset(&desc, 0, sizeof desc);
+ desc.result_type = kit_cg_type_builtin(kit_c, KIT_CG_BUILTIN_VOID);
+ f = ir_func_new(c, &desc);
+ block = ir_block_new(f);
+ f->entry = block;
+ ir_note_emit(f, block);
+ f->opt_rewritten = 1;
+ f->emit_temp_regs[RC_INT][0] = 8u;
+ f->emit_temp_reg_count[RC_INT] = 1u;
+ f->opt_reserved_regs[RC_INT] = 1u << 8u;
+ pointer_slot = add_slot(f, ptr_type, 8u, 8u);
+
+ /* The store makes the frame home authoritative while retaining its clean
+ * register copy. The next address consumes that copy as an indirect base. */
+ in = emit_with_ops(f, block, IR_LOAD_IMM, 1u);
+ in->opnds[0] = stack_op(pointer_slot, ptr_type);
+ in->extra.imm = 0x1000;
+
+ in = emit_with_ops(f, block, IR_LOAD, 2u);
+ in->opnds[0] = reg_op(10u, i64_type);
+ in->opnds[1] = indirect_frame_op(pointer_slot, ptr_type);
+ in->extra.mem.type = i64_type;
+ in->extra.mem.size = 8u;
+ in->extra.mem.align = 8u;
+
+ mock_init(&mock, unit, c);
+ opt_emit_native(c, f, &mock.base);
+
+ CU_EXPECT(unit, mock.loads == 1u && mock.reg_address_loads == 1u,
+ "cached spill should feed the indirect base without a frame reload "
+ "(loads=%u register-address-loads=%u)",
+ mock.loads, mock.reg_address_loads);
+ CU_EXPECT(unit, mock.frame_loads == 0u,
+ "cached indirect base unexpectedly reloaded from its frame home");
+
+ kit_compiler_free(kit_c);
+}
+
+static void frame_load_reuses_scoped_address_base_under_pressure(
+ KitUnit *unit) {
+ KitCompiler *kit_c = NULL;
+ Compiler *c;
+ KitCgTypeId i64_type, ptr_type;
+ OptCGFuncDesc desc;
+ Func *f;
+ MockNative mock;
+ FrameSlot base_slot, index_slot, result_slot;
+ Inst *in;
+ u32 block;
+
+ CU_EXPECT(unit,
+ kit_unit_compiler_new(
+ unit,
+ kit_unit_target(KIT_ARCH_X86_64, KIT_OS_LINUX, KIT_OBJ_ELF),
+ &kit_c) == KIT_OK &&
+ kit_c != NULL,
+ "compiler allocation failed for scoped-base pressure test");
+ if (!kit_c) return;
+ c = (Compiler *)kit_c;
+ i64_type = kit_cg_type_builtin(kit_c, KIT_CG_BUILTIN_I64);
+ ptr_type = kit_cg_type_ptr(
+ kit_c, kit_cg_type_builtin(kit_c, KIT_CG_BUILTIN_VOID), 0);
+
+ memset(&desc, 0, sizeof desc);
+ desc.result_type = kit_cg_type_builtin(kit_c, KIT_CG_BUILTIN_VOID);
+ f = ir_func_new(c, &desc);
+ block = ir_block_new(f);
+ f->entry = block;
+ ir_note_emit(f, block);
+ f->opt_rewritten = 1;
+ f->emit_temp_regs[RC_INT][0] = 8u;
+ f->emit_temp_regs[RC_INT][1] = 9u;
+ f->emit_temp_reg_count[RC_INT] = 2u;
+ f->opt_reserved_regs[RC_INT] = (1u << 8u) | (1u << 9u);
+ base_slot = add_slot(f, ptr_type, 8u, 8u);
+ index_slot = add_slot(f, i64_type, 8u, 8u);
+ result_slot = add_slot(f, i64_type, 8u, 8u);
+
+ /* Each frame write leaves a clean value in one emitter temp. The indexed
+ * load consumes both cached values as its address while its result also
+ * needs a register before frame writeback: exactly the x64 SQLite pressure
+ * shape that must reuse the now-transient materialized address base. */
+ in = emit_with_ops(f, block, IR_LOAD_IMM, 1u);
+ in->opnds[0] = stack_op(base_slot, ptr_type);
+ in->extra.imm = 0x1000;
+
+ in = emit_with_ops(f, block, IR_LOAD_IMM, 1u);
+ in->opnds[0] = stack_op(index_slot, i64_type);
+ in->extra.imm = 3;
+
+ in = emit_with_ops(f, block, IR_LOAD, 2u);
+ in->opnds[0] = stack_op(result_slot, i64_type);
+ in->opnds[1] =
+ indirect_frame_index_op(base_slot, index_slot, ptr_type, i64_type);
+ in->extra.mem.type = i64_type;
+ in->extra.mem.size = 8u;
+ in->extra.mem.align = 8u;
+
+ mock_init(&mock, unit, c);
+ opt_emit_native(c, f, &mock.base);
+
+ CU_EXPECT(unit,
+ mock.load_addrs == 1u && mock.reg_address_loads == 1u &&
+ mock.in_place_reg_address_loads == 1u,
+ "frame load should materialize once and consume the scoped base "
+ "in place (addr=%u reg-load=%u in-place=%u)",
+ mock.load_addrs, mock.reg_address_loads,
+ mock.in_place_reg_address_loads);
+ CU_EXPECT(unit, (mock.writeback_slots & (1u << result_slot)) != 0u,
+ "in-place frame load result was not written back");
+
+ kit_compiler_free(kit_c);
+}
+
+static void frame_address_reuses_scoped_base_under_pressure(KitUnit *unit) {
+ KitCompiler *kit_c = NULL;
+ Compiler *c;
+ KitCgTypeId i64_type, ptr_type;
+ OptCGFuncDesc desc;
+ Func *f;
+ MockNative mock;
+ FrameSlot base_slot, index_slot, result_slot;
+ Inst *in;
+ u32 block;
+
+ CU_EXPECT(unit,
+ kit_unit_compiler_new(
+ unit,
+ kit_unit_target(KIT_ARCH_X86_64, KIT_OS_LINUX, KIT_OBJ_ELF),
+ &kit_c) == KIT_OK &&
+ kit_c != NULL,
+ "compiler allocation failed for scoped address-result test");
+ if (!kit_c) return;
+ c = (Compiler *)kit_c;
+ i64_type = kit_cg_type_builtin(kit_c, KIT_CG_BUILTIN_I64);
+ ptr_type = kit_cg_type_ptr(
+ kit_c, kit_cg_type_builtin(kit_c, KIT_CG_BUILTIN_VOID), 0);
+
+ memset(&desc, 0, sizeof desc);
+ desc.result_type = kit_cg_type_builtin(kit_c, KIT_CG_BUILTIN_VOID);
+ f = ir_func_new(c, &desc);
+ block = ir_block_new(f);
+ f->entry = block;
+ ir_note_emit(f, block);
+ f->opt_rewritten = 1;
+ f->emit_temp_regs[RC_INT][0] = 8u;
+ f->emit_temp_regs[RC_INT][1] = 9u;
+ f->emit_temp_reg_count[RC_INT] = 2u;
+ f->opt_reserved_regs[RC_INT] = (1u << 8u) | (1u << 9u);
+ base_slot = add_slot(f, ptr_type, 8u, 8u);
+ index_slot = add_slot(f, i64_type, 8u, 8u);
+ result_slot = add_slot(f, ptr_type, 8u, 8u);
+
+ in = emit_with_ops(f, block, IR_LOAD_IMM, 1u);
+ in->opnds[0] = stack_op(base_slot, ptr_type);
+ in->extra.imm = 0x2000;
+
+ in = emit_with_ops(f, block, IR_LOAD_IMM, 1u);
+ in->opnds[0] = stack_op(index_slot, i64_type);
+ in->extra.imm = 5;
+
+ in = emit_with_ops(f, block, IR_ADDR_OF, 2u);
+ in->opnds[0] = stack_op(result_slot, ptr_type);
+ in->opnds[1] =
+ indirect_frame_index_op(base_slot, index_slot, ptr_type, i64_type);
+
+ mock_init(&mock, unit, c);
+ opt_emit_native(c, f, &mock.base);
+
+ CU_EXPECT(unit, mock.load_addrs == 1u,
+ "frame address should collapse into its scoped base exactly once "
+ "(materializations=%u)",
+ mock.load_addrs);
+ CU_EXPECT(unit, (mock.writeback_slots & (1u << result_slot)) != 0u,
+ "in-place frame address result was not written back");
+
+ kit_compiler_free(kit_c);
+}
+
+static void collapsed_address_releases_index_for_store_value(KitUnit *unit) {
+ KitCompiler *kit_c = NULL;
+ Compiler *c;
+ KitCgTypeId i64_type, ptr_type;
+ OptCGFuncDesc desc;
+ Func *f;
+ MockNative mock;
+ FrameSlot base_slot, index_slot;
+ Inst *in;
+ u32 block;
+
+ CU_EXPECT(unit,
+ kit_unit_compiler_new(
+ unit,
+ kit_unit_target(KIT_ARCH_X86_64, KIT_OS_LINUX, KIT_OBJ_ELF),
+ &kit_c) == KIT_OK &&
+ kit_c != NULL,
+ "compiler allocation failed for collapsed-address store test");
+ if (!kit_c) return;
+ c = (Compiler *)kit_c;
+ i64_type = kit_cg_type_builtin(kit_c, KIT_CG_BUILTIN_I64);
+ ptr_type = kit_cg_type_ptr(
+ kit_c, kit_cg_type_builtin(kit_c, KIT_CG_BUILTIN_VOID), 0);
+
+ memset(&desc, 0, sizeof desc);
+ desc.result_type = kit_cg_type_builtin(kit_c, KIT_CG_BUILTIN_VOID);
+ f = ir_func_new(c, &desc);
+ block = ir_block_new(f);
+ f->entry = block;
+ ir_note_emit(f, block);
+ f->opt_rewritten = 1;
+ f->emit_temp_regs[RC_INT][0] = 8u;
+ f->emit_temp_regs[RC_INT][1] = 9u;
+ f->emit_temp_reg_count[RC_INT] = 2u;
+ f->opt_reserved_regs[RC_INT] = (1u << 8u) | (1u << 9u);
+ base_slot = add_slot(f, ptr_type, 8u, 8u);
+ index_slot = add_slot(f, i64_type, 8u, 8u);
+
+ in = emit_with_ops(f, block, IR_LOAD_IMM, 1u);
+ in->opnds[0] = stack_op(base_slot, ptr_type);
+ in->extra.imm = 0x3000;
+
+ in = emit_with_ops(f, block, IR_LOAD_IMM, 1u);
+ in->opnds[0] = stack_op(index_slot, i64_type);
+ in->extra.imm = 7;
+
+ in = emit_with_ops(f, block, IR_STORE, 2u);
+ in->opnds[0] =
+ indirect_frame_index_op(base_slot, index_slot, ptr_type, i64_type);
+ in->opnds[1] = imm_op(42, i64_type);
+ in->extra.mem.type = i64_type;
+ in->extra.mem.size = 8u;
+ in->extra.mem.align = 8u;
+
+ mock_init(&mock, unit, c);
+ opt_emit_native(c, f, &mock.base);
+
+ CU_EXPECT(unit, mock.load_addrs == 1u && mock.reg_address_stores == 1u,
+ "collapsed store should reuse the dead index lease for its value "
+ "(addr=%u stores=%u)",
+ mock.load_addrs, mock.reg_address_stores);
+
+ kit_compiler_free(kit_c);
+}
+
+static void frame_bitfield_load_reuses_scoped_address_base(KitUnit *unit) {
+ KitCompiler *kit_c = NULL;
+ Compiler *c;
+ KitCgTypeId i64_type, ptr_type;
+ OptCGFuncDesc desc;
+ Func *f;
+ MockNative mock;
+ FrameSlot base_slot, index_slot, result_slot;
+ IRBitFieldAux *aux;
+ Inst *in;
+ u32 block;
+
+ CU_EXPECT(unit,
+ kit_unit_compiler_new(
+ unit,
+ kit_unit_target(KIT_ARCH_X86_64, KIT_OS_LINUX, KIT_OBJ_ELF),
+ &kit_c) == KIT_OK &&
+ kit_c != NULL,
+ "compiler allocation failed for bitfield pressure test");
+ if (!kit_c) return;
+ c = (Compiler *)kit_c;
+ i64_type = kit_cg_type_builtin(kit_c, KIT_CG_BUILTIN_I64);
+ ptr_type = kit_cg_type_ptr(
+ kit_c, kit_cg_type_builtin(kit_c, KIT_CG_BUILTIN_VOID), 0);
+
+ memset(&desc, 0, sizeof desc);
+ desc.result_type = kit_cg_type_builtin(kit_c, KIT_CG_BUILTIN_VOID);
+ f = ir_func_new(c, &desc);
+ block = ir_block_new(f);
+ f->entry = block;
+ ir_note_emit(f, block);
+ f->opt_rewritten = 1;
+ f->emit_temp_regs[RC_INT][0] = 8u;
+ f->emit_temp_regs[RC_INT][1] = 9u;
+ f->emit_temp_reg_count[RC_INT] = 2u;
+ f->opt_reserved_regs[RC_INT] = (1u << 8u) | (1u << 9u);
+ base_slot = add_slot(f, ptr_type, 8u, 8u);
+ index_slot = add_slot(f, i64_type, 8u, 8u);
+ result_slot = add_slot(f, i64_type, 8u, 8u);
+
+ in = emit_with_ops(f, block, IR_LOAD_IMM, 1u);
+ in->opnds[0] = stack_op(base_slot, ptr_type);
+ in->extra.imm = 0x4000;
+
+ in = emit_with_ops(f, block, IR_LOAD_IMM, 1u);
+ in->opnds[0] = stack_op(index_slot, i64_type);
+ in->extra.imm = 9;
+
+ in = emit_with_ops(f, block, IR_BITFIELD_LOAD, 2u);
+ in->opnds[0] = stack_op(result_slot, i64_type);
+ in->opnds[1] =
+ indirect_frame_index_op(base_slot, index_slot, ptr_type, i64_type);
+ aux = arena_znew(f->arena, IRBitFieldAux);
+ aux->access.field_type = i64_type;
+ aux->access.storage.type = i64_type;
+ aux->access.storage.size = 8u;
+ aux->access.storage.align = 8u;
+ aux->access.bit_offset = 3u;
+ aux->access.bit_width = 11u;
+ in->extra.aux = aux;
+
+ mock_init(&mock, unit, c);
+ opt_emit_native(c, f, &mock.base);
+
+ CU_EXPECT(unit,
+ mock.load_addrs == 1u &&
+ mock.in_place_reg_address_bitfield_loads == 1u,
+ "bitfield load should consume its scoped address base in place "
+ "(addr=%u in-place=%u)",
+ mock.load_addrs, mock.in_place_reg_address_bitfield_loads);
+ CU_EXPECT(unit, (mock.writeback_slots & (1u << result_slot)) != 0u,
+ "in-place bitfield result was not written back");
+
+ kit_compiler_free(kit_c);
+}
+
+static void in_place_collapse_drops_overwritten_cache_fact(KitUnit *unit) {
+ KitCompiler *kit_c = NULL;
+ Compiler *c;
+ KitCgTypeId i64_type, ptr_type;
+ OptCGFuncDesc desc;
+ Func *f;
+ MockNative mock;
+ FrameSlot base_slot, index_slot;
+ IRAtomicAux *aux;
+ Inst *in;
+ u32 block;
+
+ CU_EXPECT(unit,
+ kit_unit_compiler_new(
+ unit,
+ kit_unit_target(KIT_ARCH_X86_64, KIT_OS_LINUX, KIT_OBJ_ELF),
+ &kit_c) == KIT_OK &&
+ kit_c != NULL,
+ "compiler allocation failed for collapse cache-fact test");
+ if (!kit_c) return;
+ c = (Compiler *)kit_c;
+ i64_type = kit_cg_type_builtin(kit_c, KIT_CG_BUILTIN_I64);
+ ptr_type = kit_cg_type_ptr(
+ kit_c, kit_cg_type_builtin(kit_c, KIT_CG_BUILTIN_VOID), 0);
+
+ memset(&desc, 0, sizeof desc);
+ desc.result_type = kit_cg_type_builtin(kit_c, KIT_CG_BUILTIN_VOID);
+ f = ir_func_new(c, &desc);
+ block = ir_block_new(f);
+ f->entry = block;
+ ir_note_emit(f, block);
+ f->opt_rewritten = 1;
+ f->emit_temp_regs[RC_INT][0] = 8u;
+ f->emit_temp_regs[RC_INT][1] = 9u;
+ f->emit_temp_reg_count[RC_INT] = 2u;
+ f->opt_reserved_regs[RC_INT] = (1u << 8u) | (1u << 9u);
+ base_slot = add_slot(f, ptr_type, 8u, 8u);
+ index_slot = add_slot(f, i64_type, 8u, 8u);
+
+ in = emit_with_ops(f, block, IR_LOAD_IMM, 1u);
+ in->opnds[0] = stack_op(base_slot, ptr_type);
+ in->extra.imm = 0x5000;
+
+ in = emit_with_ops(f, block, IR_LOAD_IMM, 1u);
+ in->opnds[0] = stack_op(index_slot, i64_type);
+ in->extra.imm = 2;
+
+ /* The address consumes both cached spill values and is collapsed in place
+ * into the cached base register. The atomic value then requests that same
+ * base spill again without an address-avoidance rider. It must reload the
+ * authoritative frame home; the register now contains base+index, not the
+ * cached base value. */
+ in = emit_with_ops(f, block, IR_ATOMIC_STORE, 2u);
+ in->opnds[0] =
+ indirect_frame_index_op(base_slot, index_slot, ptr_type, i64_type);
+ in->opnds[1] = stack_op(base_slot, ptr_type);
+ aux = arena_znew(f->arena, IRAtomicAux);
+ aux->mem.type = ptr_type;
+ aux->mem.size = 8u;
+ aux->mem.align = 8u;
+ aux->mo = KIT_CG_MO_SEQ_CST;
+ in->extra.aux = aux;
+
+ mock_init(&mock, unit, c);
+ mock.reject_indexed_addr = 1u;
+ opt_emit_native(c, f, &mock.base);
+
+ CU_EXPECT(unit,
+ mock.load_addrs == 1u && mock.atomic_stores == 1u &&
+ mock.frame_loads == 1u,
+ "overwritten base source should rematerialize exactly once "
+ "(addr=%u atomic=%u reloads=%u)",
+ mock.load_addrs, mock.atomic_stores, mock.frame_loads);
+
+ kit_compiler_free(kit_c);
+}
+
+static void frame_atomic_load_reuses_scoped_address_base(KitUnit *unit) {
+ KitCompiler *kit_c = NULL;
+ Compiler *c;
+ KitCgTypeId i64_type, ptr_type;
+ OptCGFuncDesc desc;
+ Func *f;
+ MockNative mock;
+ u32 block;
+ FrameSlot base_slot, index_slot, dst_slot;
+ Inst *in;
+ IRAtomicAux *aux;
+
+ CU_EXPECT(unit,
+ kit_unit_compiler_new(
+ unit,
+ kit_unit_target(KIT_ARCH_X86_64, KIT_OS_LINUX, KIT_OBJ_ELF),
+ &kit_c) == KIT_OK &&
+ kit_c != NULL,
+ "compiler allocation failed for atomic-load pressure test");
+ if (!kit_c) return;
+ c = (Compiler *)kit_c;
+ i64_type = kit_cg_type_builtin(kit_c, KIT_CG_BUILTIN_I64);
+ ptr_type = kit_cg_type_ptr(
+ kit_c, kit_cg_type_builtin(kit_c, KIT_CG_BUILTIN_VOID), 0);
+
+ memset(&desc, 0, sizeof desc);
+ desc.result_type = kit_cg_type_builtin(kit_c, KIT_CG_BUILTIN_VOID);
+ f = ir_func_new(c, &desc);
+ block = ir_block_new(f);
+ f->entry = block;
+ ir_note_emit(f, block);
+ f->opt_rewritten = 1;
+ f->emit_temp_regs[RC_INT][0] = 8u;
+ f->emit_temp_regs[RC_INT][1] = 9u;
+ f->emit_temp_reg_count[RC_INT] = 2u;
+ f->opt_reserved_regs[RC_INT] = (1u << 8u) | (1u << 9u);
+ base_slot = add_slot(f, ptr_type, 8u, 8u);
+ index_slot = add_slot(f, i64_type, 8u, 8u);
+ dst_slot = add_slot(f, i64_type, 8u, 8u);
+
+ in = emit_with_ops(f, block, IR_LOAD_IMM, 1u);
+ in->opnds[0] = stack_op(base_slot, ptr_type);
+ in->extra.imm = 0x5000;
+ in = emit_with_ops(f, block, IR_LOAD_IMM, 1u);
+ in->opnds[0] = stack_op(index_slot, i64_type);
+ in->extra.imm = 2;
+
+ in = emit_with_ops(f, block, IR_ATOMIC_LOAD, 2u);
+ in->opnds[0] = stack_op(dst_slot, i64_type);
+ in->opnds[1] =
+ indirect_frame_index_op(base_slot, index_slot, ptr_type, i64_type);
+ aux = arena_znew(f->arena, IRAtomicAux);
+ aux->mem.type = i64_type;
+ aux->mem.size = 8u;
+ aux->mem.align = 8u;
+ aux->mo = KIT_CG_MO_SEQ_CST;
+ in->extra.aux = aux;
+
+ mock_init(&mock, unit, c);
+ opt_emit_native(c, f, &mock.base);
+
+ CU_EXPECT(unit, mock.load_addrs == 1u && mock.atomic_loads == 1u,
+ "atomic load should collapse once and reach its hook once "
+ "(addr=%u atomic=%u)",
+ mock.load_addrs, mock.atomic_loads);
+ kit_compiler_free(kit_c);
+}
+
+static void atomic_store_collapses_address_for_value_temp(KitUnit *unit) {
+ KitCompiler *kit_c = NULL;
+ Compiler *c;
+ KitCgTypeId i64_type, ptr_type;
+ OptCGFuncDesc desc;
+ Func *f;
+ MockNative mock;
+ u32 block;
+ FrameSlot base_slot, index_slot;
+ Inst *in;
+ IRAtomicAux *aux;
+
+ CU_EXPECT(unit,
+ kit_unit_compiler_new(
+ unit,
+ kit_unit_target(KIT_ARCH_X86_64, KIT_OS_LINUX, KIT_OBJ_ELF),
+ &kit_c) == KIT_OK &&
+ kit_c != NULL,
+ "compiler allocation failed for atomic-store pressure test");
+ if (!kit_c) return;
+ c = (Compiler *)kit_c;
+ i64_type = kit_cg_type_builtin(kit_c, KIT_CG_BUILTIN_I64);
+ ptr_type = kit_cg_type_ptr(
+ kit_c, kit_cg_type_builtin(kit_c, KIT_CG_BUILTIN_VOID), 0);
+
+ memset(&desc, 0, sizeof desc);
+ desc.result_type = kit_cg_type_builtin(kit_c, KIT_CG_BUILTIN_VOID);
+ f = ir_func_new(c, &desc);
+ block = ir_block_new(f);
+ f->entry = block;
+ ir_note_emit(f, block);
+ f->opt_rewritten = 1;
+ f->emit_temp_regs[RC_INT][0] = 8u;
+ f->emit_temp_regs[RC_INT][1] = 9u;
+ f->emit_temp_reg_count[RC_INT] = 2u;
+ f->opt_reserved_regs[RC_INT] = (1u << 8u) | (1u << 9u);
+ base_slot = add_slot(f, ptr_type, 8u, 8u);
+ index_slot = add_slot(f, i64_type, 8u, 8u);
+
+ in = emit_with_ops(f, block, IR_LOAD_IMM, 1u);
+ in->opnds[0] = stack_op(base_slot, ptr_type);
+ in->extra.imm = 0x5000;
+ in = emit_with_ops(f, block, IR_LOAD_IMM, 1u);
+ in->opnds[0] = stack_op(index_slot, i64_type);
+ in->extra.imm = 2;
+
+ in = emit_with_ops(f, block, IR_ATOMIC_STORE, 2u);
+ in->opnds[0] =
+ indirect_frame_index_op(base_slot, index_slot, ptr_type, i64_type);
+ in->opnds[1] = imm_op(42, i64_type);
+ aux = arena_znew(f->arena, IRAtomicAux);
+ aux->mem.type = i64_type;
+ aux->mem.size = 8u;
+ aux->mem.align = 8u;
+ aux->mo = KIT_CG_MO_SEQ_CST;
+ in->extra.aux = aux;
+
+ mock_init(&mock, unit, c);
+ opt_emit_native(c, f, &mock.base);
+
+ CU_EXPECT(unit, mock.load_addrs == 1u && mock.atomic_stores == 1u,
+ "atomic store should collapse once and reach its hook once "
+ "(addr=%u atomic=%u)",
+ mock.load_addrs, mock.atomic_stores);
+ kit_compiler_free(kit_c);
+}
+
+static void allocator_cache_obeys_machine_clobbers(KitUnit *unit) {
+ KitCompiler *kit_c = NULL;
+ Compiler *c;
+ KitCgTypeId i64_type, ptr_type;
+ OptCGFuncDesc desc;
+ Func *f;
+ MockNative mock;
+ FrameSlot clobbered_slot, preserved_slot, record_slot;
+ Inst *in, *clobber;
+ NativeMachineOp mop;
+ u32 clobber_mask[NATIVE_CALL_PLAN_CLASSES];
+ u32 block;
+
+ CU_EXPECT(unit,
+ kit_unit_compiler_new(
+ unit,
+ kit_unit_target(KIT_ARCH_X86_64, KIT_OS_LINUX, KIT_OBJ_ELF),
+ &kit_c) == KIT_OK &&
+ kit_c != NULL,
+ "compiler allocation failed for allocator-cache clobber test");
+ if (!kit_c)
+ return;
+ c = (Compiler *)kit_c;
+ i64_type = kit_cg_type_builtin(kit_c, KIT_CG_BUILTIN_I64);
+ ptr_type = kit_cg_type_ptr(
+ kit_c, kit_cg_type_builtin(kit_c, KIT_CG_BUILTIN_VOID), 0);
+
+ memset(&desc, 0, sizeof desc);
+ desc.result_type = kit_cg_type_builtin(kit_c, KIT_CG_BUILTIN_VOID);
+ f = ir_func_new(c, &desc);
+ block = ir_block_new(f);
+ f->entry = block;
+ ir_note_emit(f, block);
+ f->opt_rewritten = 1;
+ f->emit_temp_regs[RC_INT][0] = 8u;
+ f->emit_temp_regs[RC_INT][1] = 9u;
+ f->emit_temp_reg_count[RC_INT] = 2u;
+ f->opt_reserved_regs[RC_INT] = (1u << 8u) | (1u << 9u);
+ f->opt_hard_regs[RC_INT][0] = 10u;
+ f->opt_hard_regs[RC_INT][1] = 11u;
+ f->opt_hard_regs[RC_INT][2] = 12u;
+ f->opt_hard_reg_count[RC_INT] = 3u;
+ f->opt_caller_saved[RC_INT] =
+ (1u << 10u) | (1u << 11u) | (1u << 12u);
+ clobbered_slot = add_slot(f, i64_type, 8u, 8u);
+ preserved_slot = add_slot(f, i64_type, 8u, 8u);
+ record_slot = add_slot(f, i64_type, 8u, 8u);
+
+ /* r10 is dead after this writeback. Its clean copy is therefore eligible
+ * for allocator-register forwarding until a declared machine effect says
+ * the backend encoding destroys r10. */
+ in = emit_with_ops(f, block, IR_COPY, 2u);
+ in->opnds[0] = stack_op(clobbered_slot, i64_type);
+ in->opnds[1] = reg_op(10u, i64_type);
+
+ clobber = emit_with_ops(f, block, IR_BITFIELD_STORE, 2u);
+ clobber->opnds[0] = frame_addr_op(record_slot, ptr_type);
+ clobber->opnds[1] = imm_op(7, i64_type);
+ {
+ IRBitFieldAux *aux = arena_znew(f->arena, IRBitFieldAux);
+ aux->access.field_type = i64_type;
+ aux->access.storage.type = i64_type;
+ aux->access.storage.size = 8u;
+ aux->access.storage.align = 8u;
+ aux->access.bit_offset = 3u;
+ aux->access.bit_width = 11u;
+ clobber->extra.aux = aux;
+ }
+
+ in = emit_with_ops(f, block, IR_COPY, 2u);
+ in->opnds[0] = reg_op(12u, i64_type);
+ in->opnds[1] = stack_op(clobbered_slot, i64_type);
+
+ /* Establish a new, independently dead r10 value for the preservation half
+ * of the contract. The label hook reports no exceptional clobber, so the
+ * following read should forward without touching the frame. */
+ in = emit_with_ops(f, block, IR_LOAD_IMM, 1u);
+ in->opnds[0] = reg_op(10u, i64_type);
+ in->extra.imm = 99;
+
+ in = emit_with_ops(f, block, IR_COPY, 2u);
+ in->opnds[0] = stack_op(preserved_slot, i64_type);
+ in->opnds[1] = reg_op(10u, i64_type);
+
+ in = emit_with_ops(f, block, IR_LOAD_LABEL_ADDR, 1u);
+ in->opnds[0] = reg_op(11u, ptr_type);
+ in->extra.imm = block;
+
+ in = emit_with_ops(f, block, IR_COPY, 2u);
+ in->opnds[0] = reg_op(12u, i64_type);
+ in->opnds[1] = stack_op(preserved_slot, i64_type);
+
+ mock_init(&mock, unit, c);
+ mock.report_bitfield_store_clobber = 1u;
+ /* The O0 direct-target value bank is deliberately empty. Cross-instruction
+ * O1 forwarding must derive allocator-register eligibility from the
+ * machinized `f->opt_hard_regs` table above, never from an unrelated O0
+ * cache policy. */
+ mock.classes[NATIVE_REG_INT].caller_saved_mask = 1u << 10u;
+
+ memset(&mop, 0, sizeof mop);
+ mop.kind = NATIVE_MOP_BITFIELD_STORE;
+ CU_EXPECT(unit,
+ mock.base.machine_op_clobbers(&mock.base, &mop, clobber_mask) &&
+ clobber_mask[NATIVE_REG_INT] == (1u << 10u),
+ "mock bitfield-store effect must report its allocator clobber");
+ f->inst_clobbers_cap = f->next_inst_id;
+ f->inst_clobbers =
+ arena_zarray(f->arena, OptInstClobberMask, f->inst_clobbers_cap);
+ CU_EXPECT(unit,
+ clobber->id != INST_ID_NONE &&
+ clobber->id < f->inst_clobbers_cap,
+ "bitfield store must have a valid clobber-table instruction id");
+ if (clobber->id != INST_ID_NONE && clobber->id < f->inst_clobbers_cap)
+ for (u32 cls = 0; cls < OPT_REG_CLASSES; ++cls)
+ f->inst_clobbers[clobber->id][cls] = clobber_mask[cls];
+
+ opt_emit_native(c, f, &mock.base);
+
+ CU_EXPECT(unit,
+ (mock.callback_mask &
+ (SAW_BITFIELD_STORE | SAW_LABEL_ADDR)) ==
+ (SAW_BITFIELD_STORE | SAW_LABEL_ADDR),
+ "clobbering and ordinary preservation hooks must both execute");
+ CU_EXPECT(unit, mock.frame_loads == 1u && mock.loads == 1u,
+ "declared clobber must force one authoritative reload while the "
+ "ordinary hook preserves forwarding (loads=%u frame-loads=%u)",
+ mock.loads, mock.frame_loads);
+
+ kit_compiler_free(kit_c);
+}
+
+static void call_ends_pre_call_temp_phase(KitUnit *unit) {
+ KitCompiler *kit_c = NULL;
+ Compiler *c;
+ KitCgTypeId i64_type, ptr_type;
+ OptCGFuncDesc desc;
+ Func *f;
+ MockNative mock;
+ FrameSlot arg_slot, result_slot;
+ IRCallAux *aux;
+ Inst *call;
+ u32 block;
+
+ CU_EXPECT(unit,
+ kit_unit_compiler_new(
+ unit,
+ kit_unit_target(KIT_ARCH_X86_64, KIT_OS_LINUX, KIT_OBJ_ELF),
+ &kit_c) == KIT_OK &&
+ kit_c != NULL,
+ "compiler allocation failed for call phase test");
+ if (!kit_c)
+ return;
+ c = (Compiler *)kit_c;
+ i64_type = kit_cg_type_builtin(kit_c, KIT_CG_BUILTIN_I64);
+ ptr_type = kit_cg_type_ptr(
+ kit_c, kit_cg_type_builtin(kit_c, KIT_CG_BUILTIN_VOID), 0);
+
+ memset(&desc, 0, sizeof desc);
+ desc.result_type = kit_cg_type_builtin(kit_c, KIT_CG_BUILTIN_VOID);
+ f = ir_func_new(c, &desc);
+ block = ir_block_new(f);
+ f->entry = block;
+ ir_note_emit(f, block);
+ f->opt_rewritten = 1;
+ f->emit_temp_regs[RC_INT][0] = 8u;
+ f->emit_temp_regs[RC_INT][1] = 9u;
+ f->emit_temp_reg_count[RC_INT] = 2u;
+ f->opt_reserved_regs[RC_INT] = (1u << 8u) | (1u << 9u);
+ f->opt_caller_saved[RC_INT] = (1u << 8u) | (1u << 9u);
+ arg_slot = add_slot(f, i64_type, 8u, 8u);
+ result_slot = add_slot(f, i64_type, 8u, 8u);
+
+ call = emit_with_ops(f, block, IR_CALL, 0u);
+ aux = arena_znew(f->arena, IRCallAux);
+ aux->desc.fn_type = i64_type;
+ aux->desc.callee = global_op(17u, ptr_type);
+ aux->desc.nargs = 1u;
+ aux->desc.args = arena_zarray(f->arena, CGABIValue, 1u);
+ aux->desc.args[0].type = i64_type;
+ aux->desc.args[0].storage = stack_op(arg_slot, i64_type);
+ aux->desc.ret.type = i64_type;
+ aux->desc.ret.storage = stack_op(result_slot, i64_type);
+ call->extra.aux = aux;
+
+ mock_init(&mock, unit, c);
+ /* r9 materializes the frame argument into fixed ABI register r8. Both are
+ * caller-clobbered, so neither pre-call fact may survive into the ret copy.
+ */
+ opt_emit_native(c, f, &mock.base);
+
+ CU_EXPECT(unit, mock.calls == 1u, "mock call was not emitted exactly once");
+ CU_EXPECT(unit, mock.loads == 2u && mock.frame_loads == 2u,
+ "post-call return copy reused a pre-call materialization "
+ "(loads=%u frame-loads=%u)",
+ mock.loads, mock.frame_loads);
+ CU_EXPECT(unit, (mock.writeback_slots & (1u << result_slot)) != 0u,
+ "post-call result was not written to its frame destination");
+
+ kit_compiler_free(kit_c);
+}
+
+static void return_parts_keep_prior_abi_destinations_live(KitUnit *unit) {
+ KitCompiler *kit_c = NULL;
+ Compiler *c;
+ KitCgTypeId i64_type;
+ OptCGFuncDesc desc;
+ Func *f;
+ MockNative mock;
+ FrameSlot value_slot;
+ IRRetAux *aux;
+ Inst *ret;
+ u32 block;
+
+ CU_EXPECT(unit,
+ kit_unit_compiler_new(
+ unit,
+ kit_unit_target(KIT_ARCH_X86_64, KIT_OS_LINUX, KIT_OBJ_ELF),
+ &kit_c) == KIT_OK &&
+ kit_c != NULL,
+ "compiler allocation failed for multi-part return phase test");
+ if (!kit_c)
+ return;
+ c = (Compiler *)kit_c;
+ i64_type = kit_cg_type_builtin(kit_c, KIT_CG_BUILTIN_I64);
+
+ memset(&desc, 0, sizeof desc);
+ desc.result_type = i64_type;
+ f = ir_func_new(c, &desc);
+ block = ir_block_new(f);
+ f->entry = block;
+ ir_note_emit(f, block);
+ f->opt_rewritten = 1;
+ for (u32 i = 0; i < 4u; ++i) {
+ f->emit_temp_regs[RC_INT][i] = (Reg)(8u + i);
+ f->opt_reserved_regs[RC_INT] |= 1u << (8u + i);
+ }
+ f->emit_temp_reg_count[RC_INT] = 4u;
+ value_slot = add_slot(f, i64_type, 16u, 8u);
+
+ ret = emit_with_ops(f, block, IR_RET, 0u);
+ aux = arena_znew(f->arena, IRRetAux);
+ aux->present = 1u;
+ aux->val.type = i64_type;
+ aux->val.storage = stack_op(value_slot, i64_type);
+ ret->extra.aux = aux;
+
+ mock_init(&mock, unit, c);
+ mock.plan_two_part_ret = 1u;
+ mock.int_temps[2] = 10u;
+ mock.int_temps[3] = 11u;
+ mock.classes[NATIVE_REG_INT].nscratch = 4u;
+ mock.classes[NATIVE_REG_INT].nemit_temps = 4u;
+ mock.classes[NATIVE_REG_INT].emit_cache_mask =
+ (1u << 8u) | (1u << 9u) | (1u << 10u) | (1u << 11u);
+ opt_emit_native(c, f, &mock.base);
+
+ CU_EXPECT(unit, mock.nframe_load_dsts == 2u,
+ "two return parts should each load their frame source "
+ "(loads=%u)",
+ mock.nframe_load_dsts);
+ if (mock.nframe_load_dsts == 2u) {
+ CU_EXPECT(unit, mock.frame_load_dsts[0] != 8u,
+ "first return-part load must not overwrite its r8 destination");
+ CU_EXPECT(unit, mock.frame_load_dsts[1] != 8u,
+ "second return-part load reused completed ABI destination r8");
+ CU_EXPECT(unit, mock.frame_load_dsts[1] != 9u,
+ "second return-part load must not overwrite its r9 destination");
+ }
+
+ kit_compiler_free(kit_c);
+}
+
+static void call_setup_owns_callee_and_completed_arguments(KitUnit *unit) {
+ KitCompiler *kit_c = NULL;
+ Compiler *c;
+ KitCgTypeId i64_type, ptr_type;
+ OptCGFuncDesc desc;
+ Func *f;
+ MockNative mock;
+ FrameSlot callee_slot, arg_slots[2];
+ IRCallAux *aux;
+ Inst *call;
+ u32 block;
+
+ CU_EXPECT(unit,
+ kit_unit_compiler_new(
+ unit,
+ kit_unit_target(KIT_ARCH_X86_64, KIT_OS_LINUX, KIT_OBJ_ELF),
+ &kit_c) == KIT_OK &&
+ kit_c != NULL,
+ "compiler allocation failed for call setup ownership test");
+ if (!kit_c)
+ return;
+ c = (Compiler *)kit_c;
+ i64_type = kit_cg_type_builtin(kit_c, KIT_CG_BUILTIN_I64);
+ ptr_type = kit_cg_type_ptr(
+ kit_c, kit_cg_type_builtin(kit_c, KIT_CG_BUILTIN_VOID), 0);
+
+ memset(&desc, 0, sizeof desc);
+ desc.result_type = kit_cg_type_builtin(kit_c, KIT_CG_BUILTIN_VOID);
+ f = ir_func_new(c, &desc);
+ block = ir_block_new(f);
+ f->entry = block;
+ ir_note_emit(f, block);
+ f->opt_rewritten = 1;
+ for (u32 i = 0; i < 4u; ++i) {
+ f->emit_temp_regs[RC_INT][i] = (Reg)(8u + i);
+ f->opt_reserved_regs[RC_INT] |= 1u << (8u + i);
+ }
+ f->emit_temp_reg_count[RC_INT] = 4u;
+ callee_slot = add_slot(f, ptr_type, 8u, 8u);
+ arg_slots[0] = add_slot(f, i64_type, 8u, 8u);
+ arg_slots[1] = add_slot(f, i64_type, 8u, 8u);
+
+ call = emit_with_ops(f, block, IR_CALL, 0u);
+ aux = arena_znew(f->arena, IRCallAux);
+ aux->desc.fn_type = i64_type;
+ aux->desc.callee = stack_op(callee_slot, ptr_type);
+ aux->desc.nargs = 2u;
+ aux->desc.args = arena_zarray(f->arena, CGABIValue, 2u);
+ for (u32 i = 0; i < 2u; ++i) {
+ aux->desc.args[i].type = i64_type;
+ aux->desc.args[i].storage = stack_op(arg_slots[i], i64_type);
+ }
+ call->extra.aux = aux;
+
+ mock_init(&mock, unit, c);
+ mock.plan_two_arg_indirect_call = 1u;
+ mock.int_temps[2] = 10u;
+ mock.int_temps[3] = 11u;
+ mock.classes[NATIVE_REG_INT].nscratch = 4u;
+ mock.classes[NATIVE_REG_INT].nemit_temps = 4u;
+ mock.classes[NATIVE_REG_INT].emit_cache_mask =
+ (1u << 8u) | (1u << 9u) | (1u << 10u) | (1u << 11u);
+ opt_emit_native(c, f, &mock.base);
+
+ CU_EXPECT(unit, mock.calls == 1u,
+ "two-argument indirect call was not emitted exactly once");
+ CU_EXPECT(unit, mock.nframe_load_dsts == 3u && mock.frame_loads == 3u,
+ "callee and two arguments should each load once "
+ "(recorded=%u frame-loads=%u)",
+ mock.nframe_load_dsts, mock.frame_loads);
+
+ kit_compiler_free(kit_c);
+}
+
+int main(void) {
+ KitUnit unit;
+ kit_unit_init(&unit);
+ frame_destinations_are_staged(&unit);
+ cached_frame_value_feeds_indirect_address(&unit);
+ frame_load_reuses_scoped_address_base_under_pressure(&unit);
+ frame_address_reuses_scoped_base_under_pressure(&unit);
+ collapsed_address_releases_index_for_store_value(&unit);
+ frame_bitfield_load_reuses_scoped_address_base(&unit);
+ in_place_collapse_drops_overwritten_cache_fact(&unit);
+ frame_atomic_load_reuses_scoped_address_base(&unit);
+ atomic_store_collapses_address_for_value_temp(&unit);
+ allocator_cache_obeys_machine_clobbers(&unit);
+ call_ends_pre_call_temp_phase(&unit);
+ return_parts_keep_prior_abi_destinations_live(&unit);
+ call_setup_owns_callee_and_completed_arguments(&unit);
+ if (unit.fails) {
+ fprintf(stderr, "native-emit-frame-dst: %d/%d failed\n", unit.fails,
+ unit.checks);
+ return 1;
+ }
+ printf("native-emit-frame-dst: %d checks, 0 failures\n", unit.checks);
+ return 0;
+}
diff --git a/test/opt/native_part_chunk_test.c b/test/opt/native_part_chunk_test.c
@@ -0,0 +1,61 @@
+#include <stdio.h>
+
+#include "arch/native_target.h"
+#include "lib/kit_unit.h"
+
+typedef struct ChunkCase {
+ u32 total;
+ u32 max_chunk;
+ u32 count;
+ u32 offsets[4];
+ u32 sizes[4];
+} ChunkCase;
+
+static void check_case(KitUnit* unit, const ChunkCase* tc) {
+ NativePartChunkIter it = native_part_chunks(tc->total, tc->max_chunk);
+ u32 offset = 0;
+ u32 size = 0;
+ u32 count = 0;
+ u32 covered = 0;
+ while (native_part_chunk_next(&it, &offset, &size)) {
+ CU_EXPECT(unit, count < tc->count,
+ "total=%u max=%u produced an extra chunk", tc->total,
+ tc->max_chunk);
+ if (count >= tc->count) return;
+ CU_EXPECT(unit, offset == tc->offsets[count],
+ "total=%u max=%u chunk %u offset: got %u expected %u", tc->total,
+ tc->max_chunk, count, offset, tc->offsets[count]);
+ CU_EXPECT(unit, size == tc->sizes[count],
+ "total=%u max=%u chunk %u size: got %u expected %u", tc->total,
+ tc->max_chunk, count, size, tc->sizes[count]);
+ CU_EXPECT(unit, size != 0u && (size & (size - 1u)) == 0u &&
+ size <= tc->max_chunk,
+ "total=%u max=%u chunk %u is not a legal native width",
+ tc->total, tc->max_chunk, count);
+ covered += size;
+ count++;
+ }
+ CU_EXPECT(unit, count == tc->count,
+ "total=%u max=%u chunk count: got %u expected %u", tc->total,
+ tc->max_chunk, count, tc->count);
+ CU_EXPECT(unit, covered == tc->total,
+ "total=%u max=%u covered %u bytes", tc->total, tc->max_chunk,
+ covered);
+}
+
+int main(void) {
+ static const ChunkCase cases[] = {
+ {1u, 8u, 1u, {0u}, {1u}},
+ {3u, 8u, 2u, {0u, 2u}, {2u, 1u}},
+ {6u, 8u, 2u, {0u, 4u}, {4u, 2u}},
+ {7u, 8u, 3u, {0u, 4u, 6u}, {4u, 2u, 1u}},
+ {8u, 8u, 1u, {0u}, {8u}},
+ {12u, 4u, 3u, {0u, 4u, 8u}, {4u, 4u, 4u}},
+ };
+ KitUnit unit;
+ kit_unit_init(&unit);
+ for (u32 i = 0; i < (u32)(sizeof cases / sizeof cases[0]); ++i)
+ check_case(&unit, &cases[i]);
+ kit_unit_summary(&unit, "native-part-chunks");
+ return kit_unit_status(&unit);
+}
diff --git a/test/opt/o1_stack_dse.sh b/test/opt/o1_stack_dse.sh
@@ -4,10 +4,12 @@
#
# W8 deletes a spill store when a LATER store to the same spill slot fully
# overwrites it in the same block before any possible read or barrier. It is
-# deliberately exact and spill-only; the correctness boundary is aliasing /
-# partial overlap, so the pass treats every uncertain case (a read of the slot,
-# a different size/addr-space, a call/asm/intrinsic/atomic/volatile, an
-# aggregate or non-direct access) as a barrier that preserves the prior store.
+# deliberately exact and spill-only. The canonical MIR operand walker clears
+# only the pending slot named by any explicit OPK_STACK use, including stack
+# locations nested in indirect or auxiliary operands. Non-candidate memory
+# writes (call/asm/intrinsic/atomic/volatile/aggregate and uncertain stores)
+# remain full barriers; a differently shaped direct spill store replaces the
+# pending shape without treating semantic locals as allocator spill traffic.
#
# Important: in kit's current -O1 pipeline this pass is correct but rarely finds
# work -- upstream rematerialization (W2) + dead-definition-elimination + the
diff --git a/test/opt/o1p_combine.sh b/test/opt/o1p_combine.sh
@@ -337,15 +337,15 @@ printf ' L4: run rc=%s (O0==O1; copy + invert + double-not forms)\n' "$L4_O0"
# implemented by the existing same-block copy propagator try_substitute
# (unconditional SK_REG forwarding into every whitelisted consumer slot,
# pass_combine.c) plus the IR_RET back-propagation try_ret_retarget. The only
-# residual is forwarding an EMIT-SCRATCH-sourced copy, which is unsound at MIR
-# level (native emit repurposes the scratch regs to materialize locals BETWEEN
-# instructions — disabling the scratch-source guard miscompiles sqlite), and the
-# broad reg-to-reg `mov` surplus the catalog counted is ABI-mandated arg-shuffle/
-# save moves + the cross-block register-residency problem (NEEDS-SSA-O2, the
-# catalog's own §3). So this section is a COVERAGE/REGRESSION guard: it asserts
-# the spec's named consumer coalescing holds (no redundant single-use copy
-# survives feeding a convert / store-value / return) rather than introducing a
-# new, unsound fold.
+# former emitter-temp source exception no longer exists: location MIR contains
+# one canonical program home, while emitter temps are instruction-scoped leases
+# that the MIR verifier rejects as persistent operands. The broad reg-to-reg
+# `mov` surplus the catalog counted is ABI-mandated arg-shuffle/save moves plus
+# the cross-block register-residency problem (NEEDS-SSA-O2, the catalog's own
+# §3). So this section is a COVERAGE/REGRESSION guard: it asserts the spec's
+# named consumer coalescing holds (no redundant single-use copy survives
+# feeding a convert / store-value / return) rather than introducing a new,
+# unsound fold.
echo "== L5 single-use copy coalescing (coverage/regression) =="
cat > "$WORK/l5cov.c" <<'EOF'
extern void usel(long);
diff --git a/test/opt/reg_effects_test.c b/test/opt/reg_effects_test.c
@@ -0,0 +1,906 @@
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+
+#include <kit/frontend.h>
+
+#include "arch/aa64/aa64.h"
+#include "arch/riscv/rv64.h"
+#include "arch/x64/x64.h"
+#include "core/pool.h"
+#include "lib/kit_unit.h"
+#include "obj/obj.h"
+#include "opt/opt_internal.h"
+
+static Operand reg_op(u8 cls, Reg reg) {
+ Operand op;
+ memset(&op, 0, sizeof op);
+ op.kind = OPK_REG;
+ op.cls = cls;
+ op.v.reg = reg;
+ return op;
+}
+
+static Operand indirect_op(Reg base, Reg index) {
+ Operand op;
+ memset(&op, 0, sizeof op);
+ op.kind = OPK_INDIRECT;
+ op.cls = RC_INT;
+ op.v.ind.base = base;
+ op.v.ind.index = index;
+ return op;
+}
+
+static u32 reg_bit(Reg reg) { return 1u << reg; }
+
+typedef struct MachinizeRunCtx {
+ Func* f;
+ NativeTarget* target;
+} MachinizeRunCtx;
+
+static int invalid_machine_effect(
+ NativeTarget* target, const NativeMachineOp* op,
+ u32 mask[NATIVE_CALL_PLAN_CLASSES]) {
+ (void)target;
+ (void)op;
+ mask[0] = mask[1] = mask[2] = 0;
+ mask[NATIVE_REG_INT] = reg_bit(31u);
+ return 1;
+}
+
+static KitStatus run_machinize(KitCompiler* compiler, void* arg) {
+ MachinizeRunCtx* run = (MachinizeRunCtx*)arg;
+ (void)compiler;
+ opt_machinize_native(run->f, run->target);
+ return KIT_OK;
+}
+
+static Operand typed_reg_op(u8 cls, Reg reg, KitCgTypeId type) {
+ Operand op = reg_op(cls, reg);
+ op.type = type;
+ return op;
+}
+
+static void expect_body_bytes(KitUnit* u, const Section* sec, u32 begin,
+ const u8* expected, u32 expected_size,
+ const char* what) {
+ u32 size = sec ? buf_pos(&sec->bytes) : 0u;
+ u8* bytes = size ? (u8*)malloc(size) : NULL;
+ CU_EXPECT(u, sec != NULL && begin <= size && size - begin == expected_size,
+ "%s size mismatch: begin=%u size=%u expected=%u", what, begin,
+ size, expected_size);
+ if (!sec || begin > size || size - begin != expected_size ||
+ (size && !bytes)) {
+ free(bytes);
+ return;
+ }
+ if (size) buf_flatten(&sec->bytes, bytes);
+ CU_EXPECT(u, !memcmp(bytes + begin, expected, expected_size),
+ "%s encoding mismatch", what);
+ free(bytes);
+}
+
+static void x64_dying_fixed_sources_are_staged(KitUnit* u) {
+ static const u8 div_expected[] = {
+ 0x49, 0x89, 0xc3, /* mov r11, rax (save divisor) */
+ 0x48, 0x89, 0xd0, /* mov rax, rdx (publish dividend) */
+ 0x48, 0x99, /* cqo */
+ 0x49, 0xf7, 0xfb, /* idiv r11 */
+ 0x49, 0x89, 0xc0, /* mov r8, rax */
+ };
+ static const u8 shift_expected[] = {
+ 0x49, 0x89, 0xf2, /* mov r10, rsi (stage value) */
+ 0x49, 0xd3, 0xe2, /* shl r10, cl */
+ 0x4c, 0x89, 0xd1, /* mov rcx, r10 */
+ };
+ static const u8 copy_to_rax_addr_expected[] = {
+ 0x44, 0x0f, 0xb6, 0x12, /* movzx r10d, byte [rdx] */
+ 0x66, 0x45, 0x0f, 0x6e, 0xf2, /* movd xmm14, r10d */
+ 0x66, 0x45, 0x0f, 0x7e, 0xf2, /* movd r10d, xmm14 */
+ 0x44, 0x88, 0x10, /* mov byte [rax], r10b */
+ };
+ static const u8 copy_from_rax_addr_expected[] = {
+ 0x44, 0x0f, 0xb6, 0x10, /* movzx r10d, byte [rax] */
+ 0x66, 0x45, 0x0f, 0x6e, 0xf2, /* movd xmm14, r10d */
+ 0x66, 0x45, 0x0f, 0x7e, 0xf2, /* movd r10d, xmm14 */
+ 0x44, 0x88, 0x12, /* mov byte [rdx], r10b */
+ };
+ static const u8 set_rax_addr_expected[] = {
+ 0x41, 0x89, 0xca, /* mov r10d, ecx */
+ 0x41, 0x81, 0xe2, 0xff, 0x00, 0x00, 0x00, /* and r10d, 255 */
+ 0x66, 0x45, 0x0f, 0x6e, 0xf2, /* movd xmm14, r10d */
+ 0x66, 0x45, 0x0f, 0x60, 0xf6, /* punpcklbw xmm14 */
+ 0x66, 0x45, 0x0f, 0x61, 0xf6, /* punpcklwd xmm14 */
+ 0x66, 0x45, 0x0f, 0x62, 0xf6, /* punpckldq xmm14 */
+ 0x66, 0x4d, 0x0f, 0x7e, 0xf2, /* movq r10, xmm14 */
+ 0x4c, 0x89, 0x10, /* mov [rax], r10 */
+ };
+ KitCompiler* kc = NULL;
+ Compiler* c;
+ ObjBuilder* obj;
+ MCEmitter* mc;
+ NativeTarget* target;
+ const Section* sec;
+ ObjSecId text;
+ KitCgTypeId i64_type;
+ NativeLoc rax, rcx, rdx, rsi, r8;
+ NativeAddr src_addr, dst_addr;
+ AggregateAccess byte_copy;
+ u32 begin;
+
+ CU_EXPECT(u,
+ kit_unit_compiler_new(
+ u, kit_unit_target(KIT_ARCH_X86_64, KIT_OS_LINUX, KIT_OBJ_ELF),
+ &kc) == KIT_OK &&
+ kc != NULL,
+ "compiler allocation failed for x64 fixed-source staging test");
+ if (!kc) return;
+ c = (Compiler*)kc;
+ obj = obj_new(c);
+ mc = obj ? mc_new(c, obj) : NULL;
+ target = mc ? x64_native_target_new(c, obj, mc) : NULL;
+ CU_EXPECT(u, obj && mc && target,
+ "x64 fixed-source staging backend allocation failed");
+ if (!obj || !mc || !target) {
+ kit_compiler_free(kc);
+ return;
+ }
+ text = obj_section(obj, pool_intern_slice(c->global, SLICE_LIT(".text")),
+ SEC_TEXT, SF_EXEC | SF_ALLOC, 16u);
+ mc_set_section(mc, text);
+ i64_type = kit_cg_type_builtin(kc, KIT_CG_BUILTIN_I64);
+ rax = native_loc_reg(i64_type, NATIVE_REG_INT, 0u);
+ rcx = native_loc_reg(i64_type, NATIVE_REG_INT, 1u);
+ rdx = native_loc_reg(i64_type, NATIVE_REG_INT, 2u);
+ rsi = native_loc_reg(i64_type, NATIVE_REG_INT, 6u);
+ r8 = native_loc_reg(i64_type, NATIVE_REG_INT, 8u);
+
+ begin = mc_pos(mc);
+ /* The divisor dies in RAX while the dividend arrives in RDX. A naive
+ * dividend-first move destroys the divisor before IDIV can consume it. */
+ target->binop(target, BO_SDIV, r8, rdx, rax);
+ sec = obj_section_get(obj, text);
+ expect_body_bytes(u, sec, begin, div_expected, sizeof div_expected,
+ "x64 crossed division operands");
+
+ begin = mc_pos(mc);
+ /* RCX is both the dying count and the requested result. CL must remain the
+ * count until the shift finishes, so the result is computed in private R10
+ * and published afterward. */
+ target->binop(target, BO_SHL, rcx, rsi, rcx);
+ sec = obj_section_get(obj, text);
+ expect_body_bytes(u, sec, begin, shift_expected, sizeof shift_expected,
+ "x64 RCX-result variable shift");
+
+ memset(&src_addr, 0, sizeof src_addr);
+ memset(&dst_addr, 0, sizeof dst_addr);
+ memset(&byte_copy, 0, sizeof byte_copy);
+ src_addr.base_kind = NATIVE_ADDR_BASE_REG;
+ src_addr.base.reg = 2u; /* RDX */
+ src_addr.base_type = i64_type;
+ dst_addr.base_kind = NATIVE_ADDR_BASE_REG;
+ dst_addr.base.reg = 0u; /* RAX */
+ dst_addr.base_type = i64_type;
+ byte_copy.size = 1u;
+
+ begin = mc_pos(mc);
+ target->copy_bytes(target, dst_addr, src_addr, byte_copy);
+ sec = obj_section_get(obj, text);
+ expect_body_bytes(u, sec, begin, copy_to_rax_addr_expected,
+ sizeof copy_to_rax_addr_expected,
+ "x64 copy to RAX-based address");
+
+ begin = mc_pos(mc);
+ target->copy_bytes(target, src_addr, dst_addr, byte_copy);
+ sec = obj_section_get(obj, text);
+ expect_body_bytes(u, sec, begin, copy_from_rax_addr_expected,
+ sizeof copy_from_rax_addr_expected,
+ "x64 copy from RAX-based address");
+
+ byte_copy.size = 8u;
+ begin = mc_pos(mc);
+ target->set_bytes(target, dst_addr, rcx, byte_copy);
+ sec = obj_section_get(obj, text);
+ expect_body_bytes(u, sec, begin, set_rax_addr_expected,
+ sizeof set_rax_addr_expected,
+ "x64 set through RAX-based address");
+ mc_free(mc);
+ kit_compiler_free(kc);
+}
+
+static void explicit_and_machine_effects(KitUnit* u) {
+ Func f;
+ Inst in;
+ Operand opnds[3];
+ OptInstClobberMask machine_clobbers[4];
+ OptRegEffects effects;
+
+ memset(&f, 0, sizeof f);
+ memset(&in, 0, sizeof in);
+ memset(machine_clobbers, 0, sizeof machine_clobbers);
+ f.opt_rewritten = 1;
+ f.emit_temp_regs[RC_INT][0] = 8;
+ f.emit_temp_regs[RC_INT][1] = 9;
+ f.emit_temp_reg_count[RC_INT] = 2;
+ f.inst_clobbers = machine_clobbers;
+ f.inst_clobbers_cap = 4;
+
+ in.op = IR_BINOP;
+ in.id = 2;
+ in.opnds = opnds;
+ in.nopnds = 3;
+ opnds[0] = reg_op(RC_INT, 1);
+ opnds[1] = reg_op(RC_INT, 2);
+ opnds[2] = indirect_op(8, 8);
+ machine_clobbers[in.id][RC_INT] = reg_bit(5);
+
+ opt_inst_reg_effects(&f, &in, &effects);
+ CU_EXPECT(u, effects.uses.cls[RC_INT] ==
+ (reg_bit(2) | reg_bit(8)),
+ "explicit/indirect uses mismatch: %#x", effects.uses.cls[RC_INT]);
+ CU_EXPECT(u, effects.use_count[RC_INT][8] == 2,
+ "indirect base+index use count should saturate at two, got %u",
+ (unsigned)effects.use_count[RC_INT][8]);
+ CU_EXPECT(u, effects.defs.cls[RC_INT] == reg_bit(1),
+ "explicit defs mismatch: %#x", effects.defs.cls[RC_INT]);
+ CU_EXPECT(u, effects.clobbers.cls[RC_INT] == reg_bit(5),
+ "machine clobbers mismatch: %#x",
+ effects.clobbers.cls[RC_INT]);
+
+ opnds[2].v.ind.base_kind = OPT_INDIRECT_FRAME;
+ opnds[2].v.ind.index_kind = OPT_INDIRECT_FRAME_ADDR;
+ opt_inst_reg_effects(&f, &in, &effects);
+ CU_EXPECT(u, effects.uses.cls[RC_INT] == reg_bit(2),
+ "frame-backed indirect components are not hard uses: %#x",
+ effects.uses.cls[RC_INT]);
+ CU_EXPECT(u, effects.clobbers.cls[RC_INT] == reg_bit(5),
+ "frame-backed components do not alter machine clobbers: %#x",
+ effects.clobbers.cls[RC_INT]);
+}
+
+static void emission_temps_are_not_ir_effects(KitUnit* u) {
+ Func f;
+ Inst in;
+ Operand opnds[2];
+ OptRegEffects effects;
+
+ memset(&f, 0, sizeof f);
+ memset(&in, 0, sizeof in);
+ f.emit_temp_regs[RC_INT][0] = 8;
+ f.emit_temp_regs[RC_INT][1] = 9;
+ f.emit_temp_reg_count[RC_INT] = 2;
+ in.op = IR_COPY;
+ in.opnds = opnds;
+ in.nopnds = 2;
+ opnds[0] = reg_op(RC_INT, 8);
+ opnds[1] = reg_op(RC_INT, 2);
+
+ opt_inst_reg_effects(&f, &in, &effects);
+ CU_EXPECT(u, effects.clobbers.cls[RC_INT] == 0,
+ "emission temps must not clobber virtual HIR: %#x",
+ effects.clobbers.cls[RC_INT]);
+
+ f.opt_rewritten = 1;
+ opt_inst_reg_effects(&f, &in, &effects);
+ CU_EXPECT(u, effects.clobbers.cls[RC_INT] == 0,
+ "emission temps must remain private in location MIR: %#x",
+ effects.clobbers.cls[RC_INT]);
+}
+
+static void call_and_asm_effects(KitUnit* u) {
+ Func f;
+ Inst call;
+ IRCallAux call_aux;
+ CGABIValue arg;
+ OptRegEffects effects;
+
+ memset(&f, 0, sizeof f);
+ memset(&call, 0, sizeof call);
+ memset(&call_aux, 0, sizeof call_aux);
+ memset(&arg, 0, sizeof arg);
+ f.opt_caller_saved[RC_INT] = reg_bit(0) | reg_bit(1);
+ call.op = IR_CALL;
+ call.extra.aux = &call_aux;
+ call_aux.desc.callee = reg_op(RC_INT, 10);
+ call_aux.desc.args = &arg;
+ call_aux.desc.nargs = 1;
+ arg.storage = reg_op(RC_INT, 11);
+ call_aux.desc.ret.storage = reg_op(RC_INT, 0);
+
+ opt_inst_reg_effects(&f, &call, &effects);
+ CU_EXPECT(u, effects.uses.cls[RC_INT] ==
+ (reg_bit(10) | reg_bit(11)),
+ "call uses mismatch: %#x", effects.uses.cls[RC_INT]);
+ CU_EXPECT(u, effects.defs.cls[RC_INT] == reg_bit(0),
+ "call explicit results mismatch: %#x", effects.defs.cls[RC_INT]);
+ CU_EXPECT(u, effects.clobbers.cls[RC_INT] ==
+ (reg_bit(0) | reg_bit(1)),
+ "call ABI clobbers mismatch: %#x", effects.clobbers.cls[RC_INT]);
+
+ call_aux.plan_valid = 1;
+ call_aux.plan.clobber_mask[RC_INT] = reg_bit(6);
+ opt_inst_reg_effects(&f, &call, &effects);
+ CU_EXPECT(u, effects.clobbers.cls[RC_INT] == reg_bit(6),
+ "planned call clobbers must override ABI fallback: %#x",
+ effects.clobbers.cls[RC_INT]);
+
+ Inst block;
+ IRAsmAux asm_aux;
+ Operand inputs[1];
+ Operand outputs[1];
+ memset(&block, 0, sizeof block);
+ memset(&asm_aux, 0, sizeof asm_aux);
+ block.op = IR_ASM_BLOCK;
+ block.extra.aux = &asm_aux;
+ inputs[0] = reg_op(RC_INT, 2);
+ outputs[0] = reg_op(RC_INT, 3);
+ asm_aux.in_ops = inputs;
+ asm_aux.nin = 1;
+ asm_aux.out_ops = outputs;
+ asm_aux.nout = 1;
+ asm_aux.clobber_mask[RC_INT] = reg_bit(4);
+
+ opt_inst_reg_effects(&f, &block, &effects);
+ CU_EXPECT(u, effects.uses.cls[RC_INT] == reg_bit(2),
+ "asm uses mismatch: %#x", effects.uses.cls[RC_INT]);
+ CU_EXPECT(u, effects.defs.cls[RC_INT] == reg_bit(3),
+ "asm output defs mismatch: %#x", effects.defs.cls[RC_INT]);
+ CU_EXPECT(u, effects.clobbers.cls[RC_INT] == reg_bit(4),
+ "asm clobbers mismatch: %#x", effects.clobbers.cls[RC_INT]);
+}
+
+static void machinize_rebuilds_derived_clobbers(KitUnit* u) {
+ KitCompiler* kc = NULL;
+ Compiler* c;
+ Func* f;
+ NativeTarget* target;
+ CGFuncDesc desc;
+ Inst* in;
+ InstId id;
+ u32 entry;
+
+ CU_EXPECT(u,
+ kit_unit_compiler_new(
+ u, kit_unit_target(KIT_ARCH_X86_64, KIT_OS_LINUX, KIT_OBJ_ELF),
+ &kc) == KIT_OK &&
+ kc != NULL,
+ "compiler allocation failed for machinize rebuild test");
+ if (!kc) return;
+ c = (Compiler*)kc;
+ memset(&desc, 0, sizeof desc);
+ desc.fn_type = kit_cg_type_builtin(kc, KIT_CG_BUILTIN_I64);
+ f = ir_func_new(c, &desc);
+ entry = ir_block_new(f);
+ f->entry = entry;
+ ir_note_emit(f, entry);
+ in = ir_emit(f, entry, IR_BINOP);
+ in->extra.imm = BO_SDIV;
+ id = in->id;
+ target = x64_native_target_new(c, NULL, NULL);
+
+ opt_machinize_native(f, target);
+ CU_EXPECT(u,
+ f->inst_clobbers && id < f->inst_clobbers_cap &&
+ f->inst_clobbers[id][RC_INT] != 0,
+ "first machinization did not derive the division clobber set");
+
+ /* Change the semantic instruction and rerun the derivation. The old table
+ * must not survive merely because the new instruction reports no fixed
+ * machine effect. */
+ f->blocks[entry].insts[0].op = IR_COPY;
+ opt_machinize_native(f, target);
+ CU_EXPECT(u, f->inst_clobbers == NULL && f->inst_clobbers_cap == 0,
+ "machinization retained a stale per-instruction clobber table");
+
+ f->blocks[entry].insts[0].op = IR_BINOP;
+ f->blocks[entry].insts[0].extra.imm = BO_SDIV;
+ opt_machinize_native(f, target);
+ CU_EXPECT(u,
+ f->inst_clobbers && id < f->inst_clobbers_cap &&
+ f->inst_clobbers[id][RC_INT] != 0,
+ "machinization did not rebuild a cleared clobber table");
+ kit_compiler_free(kc);
+}
+
+static void post_machinize_loop_imm_refreshes_shift_clobber(KitUnit* u) {
+ KitCompiler* kc = NULL;
+ Compiler* c;
+ Func* f;
+ NativeTarget* target;
+ CGFuncDesc desc;
+ KitCgTypeId i64_type;
+ PReg dst;
+ PReg lhs;
+ Inst* shift;
+ Inst* lowered_shift = NULL;
+ InstId shift_id;
+ OptRegEffects effects;
+ u32 entry;
+ u32 initial_mask;
+ u32 stale_mask;
+
+ CU_EXPECT(u,
+ kit_unit_compiler_new(
+ u, kit_unit_target(KIT_ARCH_X86_64, KIT_OS_LINUX, KIT_OBJ_ELF),
+ &kc) == KIT_OK &&
+ kc != NULL,
+ "compiler allocation failed for post-machinize effect refresh");
+ if (!kc) return;
+ c = (Compiler*)kc;
+ i64_type = kit_cg_type_builtin(kc, KIT_CG_BUILTIN_I64);
+ memset(&desc, 0, sizeof desc);
+ desc.fn_type = i64_type;
+ desc.result_type = i64_type;
+ f = ir_func_new(c, &desc);
+ entry = ir_block_new(f);
+ f->entry = entry;
+ ir_note_emit(f, entry);
+ f->blocks[entry].loop_depth = 1;
+
+ dst = ir_alloc_preg(f, i64_type, RC_INT);
+ lhs = ir_alloc_preg(f, i64_type, RC_INT);
+ shift = ir_emit(f, entry, IR_BINOP);
+ shift->type = i64_type;
+ shift->def = dst;
+ shift->extra.imm = BO_SHL;
+ shift->nopnds = 3;
+ shift->opnds = arena_array(f->arena, Operand, shift->nopnds);
+ shift->opnds[0] = typed_reg_op(RC_INT, dst, i64_type);
+ shift->opnds[1] = typed_reg_op(RC_INT, lhs, i64_type);
+ memset(&shift->opnds[2], 0, sizeof shift->opnds[2]);
+ shift->opnds[2].kind = OPK_IMM;
+ shift->opnds[2].cls = RC_INT;
+ shift->opnds[2].type = i64_type;
+ shift->opnds[2].v.imm = 64;
+ shift_id = shift->id;
+
+ target = x64_native_target_new(c, NULL, NULL);
+ opt_machinize_native(f, target);
+ initial_mask = f->inst_clobbers && shift_id < f->inst_clobbers_cap
+ ? f->inst_clobbers[shift_id][RC_INT]
+ : 0u;
+ CU_EXPECT(u, initial_mask == 0,
+ "immediate shift unexpectedly acquired an RCX clobber: %#x",
+ initial_mask);
+
+ /* This is the production pipeline order: machinize first, then lift a
+ * non-foldable loop immediate into a register. On x64 that changes the
+ * encoding to the CL form, so the final HIR boundary must rederive RCX. */
+ opt_lower_loop_imm_operands(f, target);
+ for (u32 i = 0; i < f->blocks[entry].ninsts; ++i) {
+ Inst* in = &f->blocks[entry].insts[i];
+ if (in->id == shift_id) lowered_shift = in;
+ }
+ CU_EXPECT(u,
+ lowered_shift && lowered_shift->nopnds == 3 &&
+ lowered_shift->opnds[2].kind == OPK_REG,
+ "loop immediate was not lowered to a variable shift");
+
+ stale_mask = f->inst_clobbers && shift_id < f->inst_clobbers_cap
+ ? f->inst_clobbers[shift_id][RC_INT]
+ : 0u;
+ CU_EXPECT(u, stale_mask == 0,
+ "modeled pre-refresh table was not the stale immediate-shift form: "
+ "%#x",
+ stale_mask);
+
+ opt_refresh_machine_clobbers(f, target);
+ CU_EXPECT(u,
+ f->inst_clobbers && shift_id < f->inst_clobbers_cap &&
+ f->inst_clobbers[shift_id][RC_INT] == reg_bit(1u),
+ "post-lowering x64 shift clobbers %#x, expected RCX",
+ f->inst_clobbers && shift_id < f->inst_clobbers_cap
+ ? f->inst_clobbers[shift_id][RC_INT]
+ : 0u);
+ if (lowered_shift) {
+ opt_inst_reg_effects(f, lowered_shift, &effects);
+ CU_EXPECT(u, effects.clobbers.cls[RC_INT] == reg_bit(1u),
+ "canonical variable-shift effects omit RCX: %#x",
+ effects.clobbers.cls[RC_INT]);
+ }
+ kit_compiler_free(kc);
+}
+
+static void machinize_rejects_unknown_effect_registers(KitUnit* u) {
+ KitCompiler* kc = NULL;
+ Compiler* c;
+ Func* f;
+ NativeTarget* target;
+ CGFuncDesc desc;
+ MachinizeRunCtx run;
+ KitStatus status;
+ Inst* in;
+ u32 entry;
+
+ CU_EXPECT(u,
+ kit_unit_compiler_new(
+ u, kit_unit_target(KIT_ARCH_X86_64, KIT_OS_LINUX, KIT_OBJ_ELF),
+ &kc) == KIT_OK &&
+ kc != NULL,
+ "compiler allocation failed for invalid machine-effect test");
+ if (!kc) return;
+ c = (Compiler*)kc;
+ memset(&desc, 0, sizeof desc);
+ desc.fn_type = kit_cg_type_builtin(kc, KIT_CG_BUILTIN_I64);
+ f = ir_func_new(c, &desc);
+ entry = ir_block_new(f);
+ f->entry = entry;
+ ir_note_emit(f, entry);
+ in = ir_emit(f, entry, IR_BINOP);
+ in->extra.imm = BO_IADD;
+ target = x64_native_target_new(c, NULL, NULL);
+ target->machine_op_clobbers = invalid_machine_effect;
+ run.f = f;
+ run.target = target;
+
+ u->last_diag[0] = '\0';
+ ++u->suppress_fatal;
+ status = kit_frontend_run(kc, run_machinize, &run);
+ --u->suppress_fatal;
+ CU_EXPECT(u, status == KIT_ERR,
+ "unknown target machine-effect register was accepted");
+ CU_EXPECT(u,
+ strstr(u->last_diag,
+ "machine clobber mask names unknown physical register") !=
+ NULL,
+ "unexpected invalid machine-effect diagnostic: %s", u->last_diag);
+ kit_compiler_free(kc);
+}
+
+static void regalloc_rebuilds_derived_constraints_and_hints(KitUnit* u) {
+ KitCompiler* kc = NULL;
+ Compiler* c;
+ Func* f;
+ NativeTarget* target;
+ CGFuncDesc desc;
+ IRRetAux* ret_aux;
+ Inst* keep_def;
+ Inst* lhs_def;
+ Inst* rhs_def;
+ Inst* div;
+ Inst* keep_use;
+ Inst* ret;
+ PReg keep;
+ PReg lhs;
+ PReg rhs;
+ PReg quotient;
+ PReg keep_sink;
+ KitCgTypeId i64_type;
+ u32 entry;
+ u32 first_frequency;
+ const u32 x64_div_clobbers = reg_bit(0u) | reg_bit(2u); /* rax, rdx */
+
+ CU_EXPECT(u,
+ kit_unit_compiler_new(
+ u, kit_unit_target(KIT_ARCH_X86_64, KIT_OS_LINUX, KIT_OBJ_ELF),
+ &kc) == KIT_OK &&
+ kc != NULL,
+ "compiler allocation failed for regalloc rebuild test");
+ if (!kc) return;
+ c = (Compiler*)kc;
+ i64_type = kit_cg_type_builtin(kc, KIT_CG_BUILTIN_I64);
+ memset(&desc, 0, sizeof desc);
+ desc.fn_type = i64_type;
+ desc.result_type = i64_type;
+ f = ir_func_new(c, &desc);
+ entry = ir_block_new(f);
+ f->entry = entry;
+ ir_note_emit(f, entry);
+
+ keep = ir_alloc_preg(f, i64_type, RC_INT);
+ lhs = ir_alloc_preg(f, i64_type, RC_INT);
+ rhs = ir_alloc_preg(f, i64_type, RC_INT);
+ quotient = ir_alloc_preg(f, i64_type, RC_INT);
+ keep_sink = ir_alloc_preg(f, i64_type, RC_INT);
+
+ keep_def = ir_emit(f, entry, IR_LOAD_IMM);
+ keep_def->type = i64_type;
+ keep_def->def = keep;
+ keep_def->nopnds = 1;
+ keep_def->opnds = arena_array(f->arena, Operand, 1);
+ keep_def->opnds[0] = typed_reg_op(RC_INT, keep, i64_type);
+ keep_def->extra.imm = 7;
+
+ lhs_def = ir_emit(f, entry, IR_LOAD_IMM);
+ lhs_def->type = i64_type;
+ lhs_def->def = lhs;
+ lhs_def->nopnds = 1;
+ lhs_def->opnds = arena_array(f->arena, Operand, 1);
+ lhs_def->opnds[0] = typed_reg_op(RC_INT, lhs, i64_type);
+ lhs_def->extra.imm = 84;
+
+ rhs_def = ir_emit(f, entry, IR_LOAD_IMM);
+ rhs_def->type = i64_type;
+ rhs_def->def = rhs;
+ rhs_def->nopnds = 1;
+ rhs_def->opnds = arena_array(f->arena, Operand, 1);
+ rhs_def->opnds[0] = typed_reg_op(RC_INT, rhs, i64_type);
+ rhs_def->extra.imm = 2;
+
+ div = ir_emit(f, entry, IR_BINOP);
+ div->type = i64_type;
+ div->def = quotient;
+ div->extra.imm = BO_SDIV;
+ div->nopnds = 3;
+ div->opnds = arena_array(f->arena, Operand, 3);
+ div->opnds[0] = typed_reg_op(RC_INT, quotient, i64_type);
+ div->opnds[1] = typed_reg_op(RC_INT, lhs, i64_type);
+ div->opnds[2] = typed_reg_op(RC_INT, rhs, i64_type);
+
+ keep_use = ir_emit(f, entry, IR_COPY);
+ keep_use->type = i64_type;
+ keep_use->def = keep_sink;
+ keep_use->nopnds = 2;
+ keep_use->opnds = arena_array(f->arena, Operand, 2);
+ keep_use->opnds[0] = typed_reg_op(RC_INT, keep_sink, i64_type);
+ keep_use->opnds[1] = typed_reg_op(RC_INT, keep, i64_type);
+
+ ret = ir_emit(f, entry, IR_RET);
+ ret_aux = arena_zarray(f->arena, IRRetAux, 1);
+ ret->extra.aux = ret_aux;
+ ret_aux->present = 1;
+ ret_aux->val.storage = typed_reg_op(RC_INT, quotient, i64_type);
+
+ target = x64_native_target_new(c, NULL, NULL);
+ opt_build_cfg(f);
+ f->blocks[entry].frequency = 100;
+ opt_machinize_native(f, target);
+ opt_regalloc_locations(f, NULL);
+ first_frequency = f->preg_info[keep].frequency;
+ CU_EXPECT(u,
+ (f->preg_info[keep].forbidden_hard_regs & x64_div_clobbers) ==
+ x64_div_clobbers,
+ "value live across div did not acquire rax/rdx forbids: %#x",
+ f->preg_info[keep].forbidden_hard_regs);
+ CU_EXPECT(u, f->preg_info[quotient].preferred_hard_reg == 0,
+ "returned quotient did not acquire the rax ABI hint: %d",
+ (int)f->preg_info[quotient].preferred_hard_reg);
+
+ /* Both fields above are derived from this particular instruction stream.
+ * Remove their sources and rerun the supported allocation entry point. A
+ * previous allocation must not become an implicit input to the next one. */
+ div->extra.imm = BO_IADD;
+ ret_aux->present = 0;
+ f->blocks[entry].frequency = 1;
+ opt_machinize_native(f, target);
+ opt_regalloc_locations(f, NULL);
+ CU_EXPECT(u, f->preg_info[keep].forbidden_hard_regs == 0,
+ "regalloc retained stale fixed-register forbids: %#x",
+ f->preg_info[keep].forbidden_hard_regs);
+ CU_EXPECT(u, f->preg_info[quotient].preferred_hard_reg < 0,
+ "regalloc retained a stale ABI preference: %d",
+ (int)f->preg_info[quotient].preferred_hard_reg);
+ CU_EXPECT(u, f->preg_info[keep].frequency < first_frequency,
+ "regalloc retained stale range frequency %u after recomputing %u",
+ first_frequency, f->preg_info[keep].frequency);
+
+ kit_compiler_free(kc);
+}
+
+static void rv64_syscall_fixed_effects(KitUnit* u) {
+ KitCompiler* kc = NULL;
+ Compiler* c;
+ Func* f;
+ NativeTarget* target;
+ CGFuncDesc desc;
+ IRIntrinAux aux;
+ Inst* in;
+ InstId id;
+ u32 entry;
+ const u32 syscall_regs = (1u << 10u) | (1u << 11u) | (1u << 12u) |
+ (1u << 13u) | (1u << 14u) | (1u << 15u) |
+ (1u << 17u);
+
+ CU_EXPECT(u,
+ kit_unit_compiler_new(
+ u, kit_unit_target(KIT_ARCH_RV64, KIT_OS_LINUX, KIT_OBJ_ELF),
+ &kc) == KIT_OK &&
+ kc != NULL,
+ "compiler allocation failed for rv64 syscall effect test");
+ if (!kc) return;
+ c = (Compiler*)kc;
+ memset(&desc, 0, sizeof desc);
+ memset(&aux, 0, sizeof aux);
+ desc.fn_type = kit_cg_type_builtin(kc, KIT_CG_BUILTIN_I64);
+ f = ir_func_new(c, &desc);
+ entry = ir_block_new(f);
+ f->entry = entry;
+ ir_note_emit(f, entry);
+ in = ir_emit(f, entry, IR_INTRINSIC);
+ in->extra.aux = &aux;
+ aux.kind = INTRIN_SYSCALL;
+ id = in->id;
+ target = rv64_native_target_new(c, NULL, NULL);
+
+ opt_machinize_native(f, target);
+ CU_EXPECT(u,
+ f->inst_clobbers && id < f->inst_clobbers_cap &&
+ f->inst_clobbers[id][RC_INT] == syscall_regs,
+ "rv64 syscall fixed effects mismatch: got %#x, expected %#x",
+ f->inst_clobbers && id < f->inst_clobbers_cap
+ ? f->inst_clobbers[id][RC_INT]
+ : 0u,
+ syscall_regs);
+ kit_compiler_free(kc);
+}
+
+static void aa64_syscall_fixed_effects(KitUnit* u) {
+ KitCompiler* kc = NULL;
+ Compiler* c;
+ Func* f;
+ NativeTarget* target;
+ CGFuncDesc desc;
+ IRIntrinAux aux;
+ Inst* in;
+ InstId id;
+ u32 entry;
+ const u32 syscall_regs = (1u << 0u) | (1u << 1u) | (1u << 2u) |
+ (1u << 3u) | (1u << 4u) | (1u << 5u) |
+ (1u << 8u);
+
+ CU_EXPECT(u,
+ kit_unit_compiler_new(
+ u, kit_unit_target(KIT_ARCH_ARM_64, KIT_OS_LINUX, KIT_OBJ_ELF),
+ &kc) == KIT_OK &&
+ kc != NULL,
+ "compiler allocation failed for aa64 syscall effect test");
+ if (!kc) return;
+ c = (Compiler*)kc;
+ memset(&desc, 0, sizeof desc);
+ memset(&aux, 0, sizeof aux);
+ desc.fn_type = kit_cg_type_builtin(kc, KIT_CG_BUILTIN_I64);
+ f = ir_func_new(c, &desc);
+ entry = ir_block_new(f);
+ f->entry = entry;
+ ir_note_emit(f, entry);
+ in = ir_emit(f, entry, IR_INTRINSIC);
+ in->extra.aux = &aux;
+ aux.kind = INTRIN_SYSCALL;
+ id = in->id;
+ target = aa64_native_target_new(c, NULL, NULL);
+
+ opt_machinize_native(f, target);
+ CU_EXPECT(u,
+ f->inst_clobbers && id < f->inst_clobbers_cap &&
+ f->inst_clobbers[id][RC_INT] == syscall_regs,
+ "aa64 syscall fixed effects mismatch: got %#x, expected %#x",
+ f->inst_clobbers && id < f->inst_clobbers_cap
+ ? f->inst_clobbers[id][RC_INT]
+ : 0u,
+ syscall_regs);
+ kit_compiler_free(kc);
+}
+
+static void x64_tls_model_fixed_effects(KitUnit* u) {
+ static const struct {
+ KitOSKind os;
+ KitObjFmt obj;
+ u32 expected;
+ const char* name;
+ } cases[] = {
+ {KIT_OS_MACOS, KIT_OBJ_MACHO, (1u << 0u) | (1u << 7u),
+ "Mach-O descriptor"},
+ {KIT_OS_LINUX, KIT_OBJ_ELF, 0u, "ELF local-exec"},
+ };
+
+ for (u32 i = 0; i < sizeof cases / sizeof cases[0]; ++i) {
+ KitCompiler* kc = NULL;
+ Compiler* c;
+ Func* f;
+ NativeTarget* target;
+ CGFuncDesc desc;
+ Inst* in;
+ InstId id;
+ u32 entry;
+ u32 got;
+
+ CU_EXPECT(u,
+ kit_unit_compiler_new(
+ u,
+ kit_unit_target(KIT_ARCH_X86_64, cases[i].os, cases[i].obj),
+ &kc) == KIT_OK &&
+ kc != NULL,
+ "compiler allocation failed for x64 %s TLS effect test",
+ cases[i].name);
+ if (!kc) continue;
+ c = (Compiler*)kc;
+ memset(&desc, 0, sizeof desc);
+ desc.fn_type = kit_cg_type_builtin(kc, KIT_CG_BUILTIN_I64);
+ f = ir_func_new(c, &desc);
+ entry = ir_block_new(f);
+ f->entry = entry;
+ ir_note_emit(f, entry);
+ in = ir_emit(f, entry, IR_TLS_ADDR_OF);
+ id = in->id;
+ target = x64_native_target_new(c, NULL, NULL);
+
+ opt_machinize_native(f, target);
+ got = f->inst_clobbers && id < f->inst_clobbers_cap
+ ? f->inst_clobbers[id][RC_INT]
+ : 0u;
+ CU_EXPECT(u, got == cases[i].expected,
+ "x64 %s TLS fixed effects mismatch: got %#x, expected %#x",
+ cases[i].name, got, cases[i].expected);
+ if (!cases[i].expected)
+ CU_EXPECT(u, !f->inst_clobbers && f->inst_clobbers_cap == 0,
+ "x64 %s TLS unexpectedly allocated a clobber table",
+ cases[i].name);
+ kit_compiler_free(kc);
+ }
+}
+
+static void aa64_tls_model_fixed_effects(KitUnit* u) {
+ static const struct {
+ KitOSKind os;
+ KitObjFmt obj;
+ u32 expected;
+ const char* name;
+ } cases[] = {
+ {KIT_OS_MACOS, KIT_OBJ_MACHO, 1u << 0u, "Mach-O descriptor"},
+ {KIT_OS_LINUX, KIT_OBJ_ELF, 0u, "ELF local-exec"},
+ };
+
+ for (u32 i = 0; i < sizeof cases / sizeof cases[0]; ++i) {
+ KitCompiler* kc = NULL;
+ Compiler* c;
+ Func* f;
+ NativeTarget* target;
+ CGFuncDesc desc;
+ Inst* in;
+ InstId id;
+ u32 entry;
+ u32 got;
+
+ CU_EXPECT(u,
+ kit_unit_compiler_new(
+ u,
+ kit_unit_target(KIT_ARCH_ARM_64, cases[i].os, cases[i].obj),
+ &kc) == KIT_OK &&
+ kc != NULL,
+ "compiler allocation failed for aa64 %s TLS effect test",
+ cases[i].name);
+ if (!kc) continue;
+ c = (Compiler*)kc;
+ memset(&desc, 0, sizeof desc);
+ desc.fn_type = kit_cg_type_builtin(kc, KIT_CG_BUILTIN_I64);
+ f = ir_func_new(c, &desc);
+ entry = ir_block_new(f);
+ f->entry = entry;
+ ir_note_emit(f, entry);
+ in = ir_emit(f, entry, IR_TLS_ADDR_OF);
+ id = in->id;
+ target = aa64_native_target_new(c, NULL, NULL);
+
+ opt_machinize_native(f, target);
+ got = f->inst_clobbers && id < f->inst_clobbers_cap
+ ? f->inst_clobbers[id][RC_INT]
+ : 0u;
+ CU_EXPECT(u, got == cases[i].expected,
+ "aa64 %s TLS fixed effects mismatch: got %#x, expected %#x",
+ cases[i].name, got, cases[i].expected);
+ if (!cases[i].expected)
+ CU_EXPECT(u, !f->inst_clobbers && f->inst_clobbers_cap == 0,
+ "aa64 %s TLS unexpectedly allocated a clobber table",
+ cases[i].name);
+ kit_compiler_free(kc);
+ }
+}
+
+int main(void) {
+ KitUnit u;
+ kit_unit_init(&u);
+ explicit_and_machine_effects(&u);
+ emission_temps_are_not_ir_effects(&u);
+ call_and_asm_effects(&u);
+ machinize_rebuilds_derived_clobbers(&u);
+ post_machinize_loop_imm_refreshes_shift_clobber(&u);
+ machinize_rejects_unknown_effect_registers(&u);
+ regalloc_rebuilds_derived_constraints_and_hints(&u);
+ rv64_syscall_fixed_effects(&u);
+ aa64_syscall_fixed_effects(&u);
+ x64_tls_model_fixed_effects(&u);
+ aa64_tls_model_fixed_effects(&u);
+ x64_dying_fixed_sources_are_staged(&u);
+ fprintf(stderr, "reg-effects: %d checks, %d failures\n", u.checks, u.fails);
+ return kit_unit_status(&u);
+}
diff --git a/test/parse/CORPUS.md b/test/parse/CORPUS.md
@@ -11,18 +11,22 @@ isolation; the parser cases prove the C front-end issues *the right cg
sequence* end-to-end. Anything visible at runtime that the parser is
expected to lower must have a parse-level row here.
-Test paths per case (`run.sh`, sourced from `test/cg/run.sh`):
+Test paths per case (`run.sh`, built on `test/lib/kit_corpus.sh`; DREJ run by
+default, while C and W are opt-in):
-- **D** in-process JIT (aarch64 host only) — `parse-runner --jit`.
+- **D** in-process JIT — `parse-runner --jit`; the host architecture must match
+ the selected target.
- **R** ELF roundtrip (host-arch agnostic) — `parse-runner --emit` →
`kit-roundtrip` → `readelf` + `normalize.py` diff.
-- **E** exec via qemu/podman — `parse-runner --emit` + `start.o` →
- `link-exe-runner` → run.
-- **J** jit-via-file (aarch64 host only) — `parse-runner --emit` →
- `jit-runner`.
-
-`W` (DWARF directives) is reserved; once the parser drives `Debug` it
-will read `<name>.dwarf` sidecars and reuse `cg-check-dwarf`.
+- **E** executable run — `parse-runner --emit` + `start.o` →
+ `link-exe-runner` → qemu/podman, or the freestanding bare-metal runner for
+ rv32/arm32.
+- **J** jit-via-file — `parse-runner --emit` → `jit-runner`; the host
+ architecture must match the selected target.
+- **C** portable C backend — `parse-runner --emit-c` → host C compiler → run;
+ host and target must match, and this lane currently runs only at O0.
+- **W** Wasm roundtrip — compile through the Wasm `CgTarget`, then re-lower and
+ run `test_main`; this lane currently runs only at O0.
Sidecars (each missing file uses the documented default):
@@ -30,6 +34,8 @@ Sidecars (each missing file uses the documented default):
`test_main`'s return value.
- `<name>.skip` — single-line reason; the runner SKIPs (treated as
failure unless `KIT_TEST_ALLOW_SKIP=1`).
+- `<name>.<arch>.skip` — whole-case opt-out for one target architecture.
+- `<name>.cbackend.skip` / `<name>.wasm.skip` — lane-specific opt-outs for C/W.
- `cases_err/<name>.errpat` — optional substring assertion against the
parser's stderr. Negative cases without an errpat assert only that
parse-runner exited nonzero.
@@ -43,12 +49,6 @@ Sidecars (each missing file uses the documented default):
accept it yet)
- (deferred) — explicit non-goal for the current pass
-Until the parser lands, every row that exercises a parser feature is
-expected to FAIL — the build's `parse_c` stub panics on entry, so paths
-D and `--emit` both report nonzero. The corpus is fixed in advance so
-each parser-feature landing flips a known set of `·` rows to `★` with no
-case authoring required mid-implementation.
-
## Spine
Initial landing — proves the harness wiring, diagnostic flow, and
@@ -586,6 +586,27 @@ cursor, so mixed and class-only sequences both need direct exercise.
| `variadic_06_mixed` | ★ | interleaved int/double `...`; per-class cursors run independently | 42 |
| `variadic_07_nested_call` | ★ | `va_copy` to a separate cursor that's passed by `va_list*` to a helper, then the original cursor walked again | 42 |
+## Codegen and optimizer regressions
+
+These end-to-end rows preserve frontend types and ownership boundaries through
+CG recording, O1 location MIR, register allocation, and native emission.
+
+| Case | Status | Body | Expected |
+|---|---|---|---|
+| `asm_06_temp_boundary` | ★ | ARM/RV64 fixed and subset asm constraints are satisfied by instruction-local staging without escaping into MIR | 42 |
+| `cg_inline_asm_clobber_return` | ★ | a return-bound value stays live across inline asm that clobbers the ABI return register | 42 |
+| `cg_machine_clobber_return` | ★ | a return-bound value stays live across remainder's implicit fixed-register effects | 42 |
+| `opt_04_global_incomplete_array_decay` | ★ | a completed global array may decay from an address recipe that retains its declaration's incomplete-array type | 42 |
+| `opt_05_frame_addr_call_arg` | ★ | a spilled frame-address recipe is consumed directly as a variadic call argument under register pressure | 38 |
+| `opt_06_flexible_array_element_address` | ★ | taking a flexible-array element address does not treat the incomplete array as a by-value storage operand | 42 |
+| `opt_07_pointer_difference_index` | ★ | copy propagation preserves the integer interpretation of a pointer-difference index | 42 |
+| `opt_08_copy_preserves_compare_type` | ★ | a type-changing physical copy does not replace the compare operand's 64-bit unsigned use-site type | 42 |
+| `opt_09_copy_preserves_call_arg_type` | ★ | a type-changing physical copy does not replace the ABI argument's 64-bit use-site type | 42 |
+| `opt_10_packed_local_member` | ★ | a packed member remains a typed, deliberately unaligned view into semantic frame storage | 0 |
+| `opt_11_large_local_member_loop` | ★ | a lifted large member offset retains pointer arithmetic semantics through MIR address synthesis | 0 |
+| `opt_12_small_aggregate_arg_spill` | ★ | 3-, 5-, 6-, and 7-byte aggregate arguments and returns move exactly their semantic bytes under pressure | 0 |
+| `opt_13_pointer_first_record_copy` | ★ | the first by-value use completes a pointer-introduced record before aggregate-copy IR captures its size | 0 |
+
## Negative cases (cases_err/)
| Case | Status | Surface | Notes |
diff --git a/test/parse/cases/asm_03_register_operand.c b/test/parse/cases/asm_03_register_operand.c
@@ -1,9 +1,10 @@
/* GNU local register variables (`register T x __asm__("reg")`) used as inline-
- * asm operands must occupy the named hard register. Each template below
- * addresses its registers directly (not via %N operand substitution), so the
- * result is correct only when a/b/r really land in the pinned registers — a
- * regression guard for register-variable operand pinning through the native
- * backends (-O0 direct and the optimizer's tied-hard-reg path). 40 + 2 == 42.
+ * asm operands must appear in the named hard register at the asm boundary.
+ * Each template below addresses its registers directly (not via %N operand
+ * substitution), so the result is correct only when a/b/r are staged into
+ * those registers for this instruction — a regression guard for the -O0
+ * binder and the optimizer's instruction-local requirement/staging path.
+ * 40 + 2 == 42.
*
* Arch-guarded; the asm is target-specific. The wasm backend has no native
* hard-register file, so this case opts out of W via sidecar. */
diff --git a/test/parse/cases/asm_06_temp_boundary.c b/test/parse/cases/asm_06_temp_boundary.c
@@ -0,0 +1,19 @@
+/* Register constraints are properties of this asm boundary, not of a value's
+ * entire allocated live range. In particular, ARM r0-r3 and RISC-V t4-t6 are
+ * optimizer-emitter temp banks: explicit pins and restricted constraint sets
+ * must be satisfied by instruction-local staging without escaping into MIR. */
+
+int test_main(void) {
+#if defined(__arm__)
+ register long pinned __asm__("r0") = 40;
+ long low = 2;
+ __asm__ volatile("add r0, r0, %1" : "+r"(pinned) : "l"(low));
+ return (int)pinned;
+#elif defined(__riscv) && __riscv_xlen == 64
+ register long pinned __asm__("t6") = 40;
+ __asm__ volatile("addi t6, t6, 2" : "+r"(pinned));
+ return (int)pinned;
+#else
+ return 42;
+#endif
+}
diff --git a/test/parse/cases/asm_06_temp_boundary.expected b/test/parse/cases/asm_06_temp_boundary.expected
@@ -0,0 +1 @@
+42
diff --git a/test/parse/cases/asm_06_temp_boundary.wasm.skip b/test/parse/cases/asm_06_temp_boundary.wasm.skip
@@ -0,0 +1 @@
+native hard-register constraints are intentionally target-specific
diff --git a/test/parse/cases/cg_inline_asm_clobber_return.c b/test/parse/cases/cg_inline_asm_clobber_return.c
@@ -0,0 +1,20 @@
+/* A return-value preference is only a hint: it must not place a value in the
+ * ABI return register while an intervening inline-asm block clobbers that
+ * register. At -O1 the old allocator cleared the asm-derived forbid while
+ * applying the return hint, so each native case below returned zero instead
+ * of 42. */
+
+long clobber_return_reg(long value) {
+#if defined(__aarch64__)
+ __asm__ volatile("mov x0, #0" : : : "x0");
+#elif defined(__x86_64__)
+ __asm__ volatile("xorq %%rax, %%rax" : : : "rax");
+#elif defined(__riscv) && __riscv_xlen == 64
+ __asm__ volatile("li a0, 0" : : : "a0");
+#elif defined(__arm__)
+ __asm__ volatile("mov r0, #0" : : : "r0");
+#endif
+ return value;
+}
+
+int test_main(void) { return (int)clobber_return_reg(42); }
diff --git a/test/parse/cases/cg_inline_asm_clobber_return.expected b/test/parse/cases/cg_inline_asm_clobber_return.expected
@@ -0,0 +1 @@
+42
diff --git a/test/parse/cases/cg_inline_asm_clobber_return.wasm.skip b/test/parse/cases/cg_inline_asm_clobber_return.wasm.skip
@@ -0,0 +1 @@
+native inline-asm clobber names and ABI return registers do not exist on wasm
diff --git a/test/parse/cases/cg_machine_clobber_return.c b/test/parse/cases/cg_machine_clobber_return.c
@@ -0,0 +1,17 @@
+/* Keep a return-bound value live across an operation with implicit machine
+ * register effects. In particular, x86-64 div writes rax:rdx even though its
+ * MIR destination need not name either register. Return placement must not
+ * erase the live value's fixed-clobber exclusion. */
+
+volatile unsigned long machine_clobber_sink;
+
+unsigned long keep_across_remainder(unsigned long keep,
+ unsigned long dividend,
+ unsigned long divisor) {
+ machine_clobber_sink = dividend % divisor;
+ return keep;
+}
+
+int test_main(void) {
+ return (int)keep_across_remainder(42, 12345, 97);
+}
diff --git a/test/parse/cases/cg_machine_clobber_return.expected b/test/parse/cases/cg_machine_clobber_return.expected
@@ -0,0 +1 @@
+42
diff --git a/test/parse/cases/opt_04_global_incomplete_array_decay.c b/test/parse/cases/opt_04_global_incomplete_array_decay.c
@@ -0,0 +1,13 @@
+/* A global address recipe may retain the declaration's incomplete-array type
+ * even after the initializer completes the object. MIR only needs the symbol
+ * address here; requiring a sized referent rejects ordinary array decay. */
+static const char* const options[] = {"alpha", "beta", "gamma"};
+
+static const char* const* compile_options(void) {
+ return options;
+}
+
+int test_main(void) {
+ const char* const* p = compile_options();
+ return p[0][0] == 'a' && p[2][0] == 'g' ? 42 : 1;
+}
diff --git a/test/parse/cases/opt_04_global_incomplete_array_decay.expected b/test/parse/cases/opt_04_global_incomplete_array_decay.expected
@@ -0,0 +1 @@
+42
diff --git a/test/parse/cases/opt_05_frame_addr_call_arg.c b/test/parse/cases/opt_05_frame_addr_call_arg.c
@@ -0,0 +1,12 @@
+/* Keep enough values live across the call to spill the cheap `&x` producer.
+ * MIR rematerializes it as OPK_FRAME_ADDR; ABI marshalling must consume that
+ * address recipe directly rather than demanding another optimizer temp. */
+__attribute__((noinline)) static void sink(int* p, ...) { *p += 1; }
+
+static int pressure(int a, int b, int c, int d, int e, int f, int g, int h) {
+ int x = a;
+ sink(&x, a + 1, b + 2, c + 3, d + 4, e + 5, f + 6, g + 7, h + 8);
+ return x + a + b + c + d + e + f + g + h;
+}
+
+int test_main(void) { return pressure(1, 2, 3, 4, 5, 6, 7, 8); }
diff --git a/test/parse/cases/opt_05_frame_addr_call_arg.expected b/test/parse/cases/opt_05_frame_addr_call_arg.expected
@@ -0,0 +1 @@
+38
diff --git a/test/parse/cases/opt_06_flexible_array_element_address.c b/test/parse/cases/opt_06_flexible_array_element_address.c
@@ -0,0 +1,37 @@
+/* Taking the address of an element in a flexible array carries the unsized
+ * array lvalue through IR_ADDR_OF. It computes an address and does not access
+ * the incomplete array object as a by-value storage operand. */
+typedef struct Segment {
+ int value;
+} Segment;
+
+typedef struct Iterator {
+ int count;
+ Segment segments[];
+} Iterator;
+
+typedef struct Backing {
+ int count;
+ Segment segments[2];
+} Backing;
+
+/* A union containing the flexible-array structure is allowed to provide the
+ * backing storage. Iterator and Backing have a compatible common initial
+ * sequence, so this exercises the flexible member without casting an
+ * unrelated declared object through Iterator (which would violate C11's
+ * effective-type rules). */
+typedef union Storage {
+ Iterator iterator;
+ Backing backing;
+} Storage;
+
+__attribute__((noinline)) static Segment* segment_at(Iterator* p, int i) {
+ return &p->segments[i];
+}
+
+int test_main(void) {
+ Storage storage = {0};
+ Segment* segment = segment_at(&storage.iterator, 1);
+ segment->value = 42;
+ return storage.backing.segments[1].value;
+}
diff --git a/test/parse/cases/opt_06_flexible_array_element_address.expected b/test/parse/cases/opt_06_flexible_array_element_address.expected
@@ -0,0 +1 @@
+42
diff --git a/test/parse/cases/opt_07_pointer_difference_index.c b/test/parse/cases/opt_07_pointer_difference_index.c
@@ -0,0 +1,14 @@
+/* A same-width pointer-to-ptrdiff conversion may become a physical copy after
+ * allocation. Copy propagation may change the index register, but the
+ * indirect use must retain its integer interpretation. */
+__attribute__((noinline)) static const char* pick(const char* base,
+ const char* tail,
+ char* copy) {
+ return &base[tail - copy];
+}
+
+int test_main(void) {
+ char base[] = "0123456789";
+ char copy[] = "abcdefghij";
+ return *pick(base, copy + 4, copy) == '4' ? 42 : 1;
+}
diff --git a/test/parse/cases/opt_07_pointer_difference_index.expected b/test/parse/cases/opt_07_pointer_difference_index.expected
@@ -0,0 +1 @@
+42
diff --git a/test/parse/cases/opt_08_copy_preserves_compare_type.c b/test/parse/cases/opt_08_copy_preserves_compare_type.c
@@ -0,0 +1,14 @@
+/* A narrow signed load may be folded with its widening conversion, leaving a
+ * physical copy whose source and destination have different semantic types.
+ * Copy propagation may retarget the compare's register, but the compare must
+ * retain its 64-bit unsigned interpretation. On AArch64, losing that use-site
+ * type selected `cmp wN, wM` and discarded the sign-extended high bits. */
+__attribute__((noinline)) static int compare_widened(volatile signed char* p) {
+ long long widened = *p;
+ return (unsigned long long)widened > 0xffffffffULL ? 42 : 1;
+}
+
+int test_main(void) {
+ volatile signed char value = -1;
+ return compare_widened(&value);
+}
diff --git a/test/parse/cases/opt_08_copy_preserves_compare_type.expected b/test/parse/cases/opt_08_copy_preserves_compare_type.expected
@@ -0,0 +1 @@
+42
diff --git a/test/parse/cases/opt_09_copy_preserves_call_arg_type.c b/test/parse/cases/opt_09_copy_preserves_call_arg_type.c
@@ -0,0 +1,17 @@
+/* ABI call plans consume an already-typed value. If a type-changing physical
+ * copy is propagated into a planned argument, only its register identity may
+ * change: replacing the argument operand's 64-bit type with the narrow load's
+ * type makes AArch64 marshal it with `mov wN, wM`, zeroing the high half. */
+__attribute__((noinline)) static int observe(unsigned long long value) {
+ return value > 0xffffffffULL ? 42 : 1;
+}
+
+__attribute__((noinline)) static int pass_widened(volatile signed char* p) {
+ long long widened = *p;
+ return observe((unsigned long long)widened);
+}
+
+int test_main(void) {
+ volatile signed char value = -1;
+ return pass_widened(&value);
+}
diff --git a/test/parse/cases/opt_09_copy_preserves_call_arg_type.expected b/test/parse/cases/opt_09_copy_preserves_call_arg_type.expected
@@ -0,0 +1 @@
+42
diff --git a/test/parse/cases/opt_10_packed_local_member.c b/test/parse/cases/opt_10_packed_local_member.c
@@ -0,0 +1,16 @@
+/* A folded local operand is a typed view into semantic frame storage, not a
+ * scalar spill home. In particular, a member of a packed local can have a
+ * natural type alignment greater than the aggregate frame slot alignment.
+ * The access is valid (and deliberately unaligned); O1 must not reject the
+ * OPK_LOCAL view while lowering physical MIR. */
+typedef struct __attribute__((packed)) Packed {
+ unsigned value;
+} Packed;
+
+__attribute__((noinline)) static unsigned packed_roundtrip(unsigned value) {
+ Packed local;
+ local.value = value;
+ return local.value;
+}
+
+int test_main(void) { return packed_roundtrip(42u) != 42u; }
diff --git a/test/parse/cases/opt_11_large_local_member_loop.c b/test/parse/cases/opt_11_large_local_member_loop.c
@@ -0,0 +1,17 @@
+/* A non-encodable member offset used in a loop is lifted into a register.
+ * Pointer arithmetic keeps that offset register pointer-typed through MIR;
+ * address synthesis must not reinterpret it as an integer scaled index. */
+typedef struct BigLocal {
+ unsigned char padding[5000];
+ int value;
+} BigLocal;
+
+__attribute__((noinline)) static int update_large_member(unsigned count) {
+ BigLocal local;
+ unsigned i;
+ local.value = 0;
+ for (i = 0; i < count; ++i) local.value = 42;
+ return local.value;
+}
+
+int test_main(void) { return update_large_member(3) != 42; }
diff --git a/test/parse/cases/opt_12_small_aggregate_arg_spill.c b/test/parse/cases/opt_12_small_aggregate_arg_spill.c
@@ -0,0 +1,198 @@
+/* Non-power-of-two ABI aggregate parts must move only their semantic bytes.
+ * A widened register spill may overlap an adjacent frame slot; a narrowed
+ * transfer silently drops the tail. Three, five, six, and seven bytes cover
+ * every non-power-of-two integer-carrier width and the 32-bit lane boundary. */
+typedef struct TinyOptions {
+ unsigned char packed;
+ unsigned short max_align;
+ unsigned short align_override;
+} TinyOptions;
+
+typedef struct TinyThree {
+ unsigned char first;
+ unsigned char second;
+ unsigned char third;
+} TinyThree;
+
+typedef struct TinyFive {
+ unsigned char first;
+ unsigned char second;
+ unsigned char third;
+ unsigned char fourth;
+ unsigned char fifth;
+} TinyFive;
+
+typedef struct TinySeven {
+ unsigned char first;
+ unsigned char second;
+ unsigned char third;
+ unsigned char fourth;
+ unsigned char fifth;
+ unsigned char sixth;
+ unsigned char seventh;
+} TinySeven;
+
+typedef struct Builder {
+ void* pool;
+ unsigned kind;
+ unsigned tag;
+ unsigned symbol;
+ TinyOptions opts;
+ TinyThree three;
+ TinyFive five;
+ TinySeven seven;
+} Builder;
+
+__attribute__((noinline)) static void barrier(Builder* b) {
+ volatile unsigned* p = &b->kind;
+ *p = *p;
+}
+
+__attribute__((noinline)) static Builder* begin(Builder* b, unsigned kind,
+ unsigned tag, unsigned symbol,
+ TinyOptions opts) {
+ barrier(b);
+ b->pool = b;
+ b->kind = kind;
+ b->tag = tag;
+ b->symbol = symbol;
+ b->opts = opts;
+ return b;
+}
+
+__attribute__((noinline)) static Builder* begin_three(Builder* b,
+ unsigned kind,
+ unsigned tag,
+ unsigned symbol,
+ TinyThree three) {
+ barrier(b);
+ b->pool = b;
+ b->kind = kind;
+ b->tag = tag;
+ b->symbol = symbol;
+ b->three = three;
+ return b;
+}
+
+__attribute__((noinline)) static Builder* begin_five(Builder* b, unsigned kind,
+ unsigned tag,
+ unsigned symbol,
+ TinyFive five) {
+ barrier(b);
+ b->pool = b;
+ b->kind = kind;
+ b->tag = tag;
+ b->symbol = symbol;
+ b->five = five;
+ return b;
+}
+
+__attribute__((noinline)) static Builder* begin_seven(Builder* b,
+ unsigned kind,
+ unsigned tag,
+ unsigned symbol,
+ TinySeven seven) {
+ barrier(b);
+ b->pool = b;
+ b->kind = kind;
+ b->tag = tag;
+ b->symbol = symbol;
+ b->seven = seven;
+ return b;
+}
+
+/* Returning a short aggregate exercises the call-plan write phase as well as
+ * its incoming argument bind. On 64-bit targets the pointer input and integer
+ * return carrier can alias; on ARM32 a three-byte tail has an I32 carrier. Both
+ * cases must retain the semantic width through the complete return move. */
+__attribute__((noinline)) static TinyOptions load_options(
+ const TinyOptions* opts) {
+ return *opts;
+}
+
+__attribute__((noinline)) static TinyThree load_three(const TinyThree* three) {
+ return *three;
+}
+
+__attribute__((noinline)) static TinyFive load_five(const TinyFive* five) {
+ return *five;
+}
+
+__attribute__((noinline)) static TinySeven load_seven(const TinySeven* seven) {
+ return *seven;
+}
+
+__attribute__((noinline)) static int exercise(Builder* b, unsigned kind,
+ unsigned tag, unsigned symbol,
+ int* sentinel) {
+ TinyOptions opts = {1, 0x2345, 0x6789};
+ TinyOptions returned = load_options(&opts);
+ Builder* out = begin(b, kind, tag, symbol, opts);
+ return out == b && out->kind == kind && out->tag == tag &&
+ out->symbol == symbol && out->opts.packed == opts.packed &&
+ out->opts.max_align == opts.max_align &&
+ out->opts.align_override == opts.align_override &&
+ returned.packed == opts.packed &&
+ returned.max_align == opts.max_align &&
+ returned.align_override == opts.align_override && *sentinel == 42;
+}
+
+__attribute__((noinline)) static int exercise_three(Builder* b, unsigned kind,
+ unsigned tag,
+ unsigned symbol,
+ int* sentinel) {
+ TinyThree three = {0x12, 0x34, 0x56};
+ TinyThree returned = load_three(&three);
+ Builder* out = begin_three(b, kind, tag, symbol, three);
+ return out == b && out->kind == kind && out->tag == tag &&
+ out->symbol == symbol && out->three.first == three.first &&
+ out->three.second == three.second && out->three.third == three.third &&
+ returned.first == three.first && returned.second == three.second &&
+ returned.third == three.third &&
+ *sentinel == 42;
+}
+
+__attribute__((noinline)) static int exercise_five(Builder* b, unsigned kind,
+ unsigned tag,
+ unsigned symbol,
+ int* sentinel) {
+ TinyFive five = {0x10, 0x20, 0x30, 0x40, 0x50};
+ TinyFive returned = load_five(&five);
+ Builder* out = begin_five(b, kind, tag, symbol, five);
+ return out == b && out->kind == kind && out->tag == tag &&
+ out->symbol == symbol && out->five.first == five.first &&
+ out->five.second == five.second && out->five.third == five.third &&
+ out->five.fourth == five.fourth && out->five.fifth == five.fifth &&
+ returned.first == five.first && returned.second == five.second &&
+ returned.third == five.third && returned.fourth == five.fourth &&
+ returned.fifth == five.fifth && *sentinel == 42;
+}
+
+__attribute__((noinline)) static int exercise_seven(Builder* b, unsigned kind,
+ unsigned tag,
+ unsigned symbol,
+ int* sentinel) {
+ TinySeven seven = {0x11, 0x22, 0x33, 0x44, 0x55, 0x66, 0x77};
+ TinySeven returned = load_seven(&seven);
+ Builder* out = begin_seven(b, kind, tag, symbol, seven);
+ return out == b && out->kind == kind && out->tag == tag &&
+ out->symbol == symbol && out->seven.first == seven.first &&
+ out->seven.second == seven.second &&
+ out->seven.third == seven.third &&
+ out->seven.fourth == seven.fourth &&
+ out->seven.fifth == seven.fifth && out->seven.sixth == seven.sixth &&
+ out->seven.seventh == seven.seventh &&
+ returned.first == seven.first && returned.second == seven.second &&
+ returned.third == seven.third && returned.fourth == seven.fourth &&
+ returned.fifth == seven.fifth && returned.sixth == seven.sixth &&
+ returned.seventh == seven.seventh && *sentinel == 42;
+}
+
+int test_main(void) {
+ Builder b = {0};
+ int sentinel = 42;
+ if (!exercise(&b, 11, 22, 33, &sentinel)) return 1;
+ if (!exercise_three(&b, 44, 55, 66, &sentinel)) return 2;
+ if (!exercise_five(&b, 77, 88, 99, &sentinel)) return 3;
+ return exercise_seven(&b, 111, 122, 133, &sentinel) ? 0 : 4;
+}
diff --git a/test/parse/cases/opt_13_pointer_first_record_copy.c b/test/parse/cases/opt_13_pointer_first_record_copy.c
@@ -0,0 +1,23 @@
+/* A record first crossing the CG boundary through a pointer has only a nominal
+ * declaration there. Its first by-value use must complete the record layout
+ * before aggregate-copy IR captures the byte count. Keep clone_value before
+ * test_main so no earlier by-value local happens to complete Value for it. */
+typedef unsigned long long U64;
+
+typedef struct Value {
+ U64 word[8];
+} Value;
+
+__attribute__((noinline)) static void clone_value(Value* dst,
+ const Value* src) {
+ *dst = *src;
+}
+
+int test_main(void) {
+ Value src = {{1, 2, 3, 4, 5, 6, 7, 8}};
+ Value dst = {{0, 0, 0, 0, 0, 0, 0, 0}};
+ clone_value(&dst, &src);
+ for (unsigned i = 0; i < 8; ++i)
+ if (dst.word[i] != src.word[i]) return 1;
+ return 0;
+}
diff --git a/test/toy/cases/155_musttail_indirect_packed_stack.toy b/test/toy/cases/155_musttail_indirect_packed_stack.toy
@@ -0,0 +1,25 @@
+// Keep this call genuinely indirect and force Tiny3 past all four AAPCS32 core
+// argument registers. ARM32 must preserve the target in IP while its exact
+// three-byte stack-argument load uses LR as the carrier and IP to pack byte 2.
+record @[.packed] Tiny3 {
+ x: u8,
+ y: u8,
+ z: u8,
+}
+
+fn @[.noinline] check(a: i32, b: i32, c: i32, d: i32, tiny: Tiny3): i32 {
+ return (a - 11) + (b - 13) + (c - 17) + (d - 19) +
+ (tiny.x as i32 - 23) + (tiny.y as i32 - 29) +
+ (tiny.z as i32 - 31);
+}
+
+fn @[.noinline] forward(fp: *fn(i32, i32, i32, i32, Tiny3): i32,
+ a: i32, b: i32, c: i32, d: i32,
+ tiny: Tiny3): i32 {
+ return musttail fp(a, b, c, d, tiny);
+}
+
+fn main(): i32 {
+ let tiny: Tiny3 = Tiny3 { x: 23, y: 29, z: 31 };
+ return forward(check, 11, 13, 17, 19, tiny);
+}
diff --git a/test/toy/cases/155_musttail_indirect_packed_stack.wasm.skip b/test/toy/cases/155_musttail_indirect_packed_stack.wasm.skip
@@ -0,0 +1 @@
+ARM32-specific packed stack-argument regression; Wasm musttail roundtrip currently produces an invalid local index
diff --git a/test/toy/cases/156_indirect_stack_args.toy b/test/toy/cases/156_indirect_stack_args.toy
@@ -0,0 +1,25 @@
+// An indirect target must survive the complete argument-marshalling phase.
+// Ten integer arguments force two values through the outgoing stack while the
+// target itself arrives as a runtime parameter rather than a direct symbol.
+
+type Sum10 = fn(
+ i64, i64, i64, i64, i64,
+ i64, i64, i64, i64, i64
+): i64;
+
+fn sum10(
+ a0: i64, a1: i64, a2: i64, a3: i64, a4: i64,
+ a5: i64, a6: i64, a7: i64, a8: i64, a9: i64
+): i64 {
+ return a0 + a1 + a2 + a3 + a4 + a5 + a6 + a7 + a8 + a9;
+}
+
+fn apply10(fp: *Sum10, seed: i64): i64 {
+ return fp(seed, 2, 3, 4, 5, 6, 7, 8, 9, 10);
+}
+
+fn __user_main(): i64 {
+ return apply10(sum10, 1) - 55;
+}
+
+fn main(): i32 { return __user_main() as i32; }
