commit 9c17016648794d30aae14b0b408054d45187a567
parent bed853f9f7ddfc1e2dfa97e23198932522268b6c
Author: Ryan Sepassi <rsepassi@gmail.com>
Date: Thu, 11 Jun 2026 16:45:52 -0700
perf(cg): move the delayed cmp/arith payload off the value-stack node
ApiSValue carried a 64-byte delayed cmp/arith union inline, dead for the common
SV_OPERAND value yet zeroed/copied on every push. Move it behind an ApiDelayed*
(node 112 -> 56 bytes):
- Payloads live in a per-function arena pool on KitCg (api_delayed_alloc/_free/
_reset). The arena is reset at func_begin, where the value stack is also
dropped, so a payload never outlives its function and a missed free only
enlarges the reset-per-function arena -- it cannot dangle. A freelist recycles
payloads within a function, so the arena stays at the live-delayed working set.
- fold's chain-fold `*out = *a` becomes a pointer MOVE (out takes a's payload,
a->delayed = NULL) instead of a by-value deep copy; the identity/unary cases
copy the operand into an SV_OPERAND out and leave a's payload for the caller's
api_release to reclaim. No payload is ever shared, so none is double-freed.
- api_release_*/api_materialize_* are the free choke points; api_make_* take g
to allocate.
Byte-identical across the full 60-category gate (cumulative with the Round-7
b+c memo/designated-init change); test-cg-api/opt/toy + test-smoke-x64 green
under the default ASan/UBSan build (validates the pool's manual alloc/free).
Perf on RELEASE: this shrink is NEUTRAL (per-delayed-node alloc/free cancels the
smaller-node savings; the value stack is small and L1-hot). Kept for the
structural cleanup -- the cold payload no longer rides every hot node -- not
speed. The Round-7 win (body-size +9.5% RELEASE) is essentially all the (b)
per-type memo. PERF.md Round-7 corrected to RELEASE numbers (the prior figures
were measured on the ASan build and understated the win) + a methodology note.
Diffstat:
8 files changed, 223 insertions(+), 128 deletions(-)
diff --git a/doc/plan/PERF.md b/doc/plan/PERF.md
@@ -106,24 +106,37 @@ change: only the UUID/build-id bytes move). New tooling:
`scripts/perf_identity_gate.sh` (60-category byte-identical gate) and
`scripts/perf_axis_time.py` (focused A/B axis timer).
-**Round 7 (value-stack frontend, byte-identical).** Two further constant-factor
-wins on the value-stack layer that sits *above* the `CgTarget` seam, gated on the
-same 60-category identity gate: **(b) per-type classification memo** — R6 cached
-the `wide_kind` result on each node but `api_push` still re-derived it (a string of
+**Round 7 (value-stack frontend, byte-identical).** Three structural wins on the
+value-stack layer that sits *above* the `CgTarget` seam, all gated on the same
+60-category identity gate: **(b) per-type classification memo** — R6 cached the
+`wide_kind` result on each node but `api_push` still re-derived it (a string of
`cg_type_get` + alias-chase + ABI round trips) on every operand; the wide/soft
class and the aggregate-place bit are now computed once per type id (builtins
precomputed at `cg_api` init, user types filled lazily in the `CgApiType` slack)
and `api_push` reads both from one packed byte. **(c) designated-init value nodes**
-— the `api_op_*` / `api_make_*` constructors drop their `memset` of the (112-byte)
-`ApiSValue` for a compound literal, so the compiler stores only live fields and
-elides immediately-overwritten ones. Measured (paired vs the Round-6 binary):
-body-size **+3.5%**, ref-density **+2.4%**, locals-per-fn **+2.3%**, fn-count
-**+1.6%**; (b) is the bulk, (c) ~+0.75% (the delayed-union zeroing survives under
-`-ftrivial-auto-var-init=zero`, so (c) only reclaims the memset *call*). The
-node-shrink (move the 64-byte delayed cmp/arith union off the hot node) was
-assessed and **deferred**: the `-O0` fold machinery relies on by-value deep-copy of
-the delayed payload (`*out = *a` then divergent clears in `fold.c`), which a shared
-pointer would break — a larger, riskier change than the ~1–3% it would add.
+— the `api_op_*` / `api_make_*` constructors drop their `memset` for a compound
+literal, so the compiler stores only live fields. **(d) off-node delayed payload**
+— the 64-byte delayed cmp/arith union moves off `ApiSValue` behind a pointer
+(node **112 → 56 bytes**); payloads live in a per-function arena pool on `KitCg`
+(reset at `func_begin`, freelist-recycled within a function), and fold's
+`*out = *a` chain-fold becomes a pointer *move* (`out` takes `a`'s payload,
+`a->delayed = NULL`) so no shared payload is double-freed.
+
+Measured **on a RELEASE build** (paired best-of-9 vs the pre-Round-7 binary;
+`perf_axis_time.py`): body-size **+9.5%**, locals-per-fn **+6.9%**, fn-count
+**+2.8%**, ref-density **+2.5%**. The win is essentially all (b) — the compute
+saved by killing the per-operand type gauntlet. (c) reclaims only the memset
+*call* (the delayed zeroing survives `-ftrivial-auto-var-init=zero`), and (d) is
+**perf-neutral** (the per-delayed-node alloc/free cancels the smaller-node
+savings; the value stack is small and L1-hot regardless) — it is kept for the
+structural cleanup (the cold payload no longer rides every hot node), not speed.
+
+> Methodology note: measure perf on a **RELEASE** build (`make … RELEASE=1
+> BUILD_DIR=build/release`), never the default `make bin` (which is
+> `RELEASE=0` → ASan/UBSan-instrumented). ASan amplifies *memory*-op savings and
+> dilutes *compute* savings, which inverts the apparent ranking: under ASan (d)
+> looked like a big win and (b) small; on RELEASE it is the reverse. The
+> byte-identity gate is build-mode-independent, but the timer is not.
The structural bet is fully in place: a single-pass no-AST C frontend, single-pass
code emission with patch-ups, and a format-neutral linker. The scaling bugs are
diff --git a/src/cg/arith.c b/src/cg/arith.c
@@ -90,7 +90,7 @@ void api_cg_binop(KitCg* g, BinOp iop, u32 flags) {
(rb.kind == OPK_LOCAL || rb.kind == OPK_IMM)) {
int a_owned = api_sv_owns_operand_local(&a, &ra);
int b_owned = api_sv_owns_operand_local(&b, &rb);
- api_push(g, api_make_arith_binop(iop, ra, rb, ty, a_owned, b_owned));
+ api_push(g, api_make_arith_binop(g, iop, ra, rb, ty, a_owned, b_owned));
if (a_owned) a.res = RES_INHERENT;
if (b_owned) b.res = RES_INHERENT;
api_release(g, &a);
@@ -143,7 +143,7 @@ void api_cg_unop(KitCg* g, UnOp iop, u32 flags) {
* api_invert_cmp), so `!(a<b)` becomes UGE (NaN -> true), matching IEEE
* negation. The inverted compare keeps the same i32 result type. */
if (iop == UO_NOT && a.kind == SV_CMP) {
- a.delayed.cmp.op = api_invert_cmp(a.delayed.cmp.op);
+ a.delayed->cmp.op = api_invert_cmp(a.delayed->cmp.op);
api_push(g, a);
return;
}
@@ -164,7 +164,7 @@ void api_cg_unop(KitCg* g, UnOp iop, u32 flags) {
ra = api_force_local_unless_imm(g, &a, ty);
if (can_delay && ra.kind == OPK_LOCAL) {
int a_owned = api_sv_owns_operand_local(&a, &ra);
- api_push(g, api_make_arith_unop(iop, ra, ty, a_owned));
+ api_push(g, api_make_arith_unop(g, iop, ra, ty, a_owned));
if (a_owned) a.res = RES_INHERENT;
api_release(g, &a);
return;
@@ -205,8 +205,10 @@ void api_cg_cmp(KitCg* g, CmpOp cop) {
* If the compare instead escapes into value context it is materialized
* unchanged via api_materialize_cmp_to, which calls T->cmp with the same
* opcode the eager path used to. */
- api_push(g, api_make_cmp(cop, ra, rb, i32, api_sv_owns_operand_local(&a, &ra),
- api_sv_owns_operand_local(&b, &rb)));
+ api_push(g,
+ api_make_cmp(g, cop, ra, rb, i32,
+ api_sv_owns_operand_local(&a, &ra),
+ api_sv_owns_operand_local(&b, &rb)));
}
int api_try_i128_convert(KitCg* g, ConvKind ck, KitCgTypeId sty,
diff --git a/src/cg/control.c b/src/cg/control.c
@@ -31,9 +31,9 @@ void api_branch_if(KitCg* g, ApiSValue* v, int branch_when_true, Label label) {
return;
}
if (v->kind == SV_CMP) {
- CmpOp op = branch_when_true ? v->delayed.cmp.op
- : api_invert_cmp(v->delayed.cmp.op);
- T->cmp_branch(T, op, v->delayed.cmp.a, v->delayed.cmp.b, label);
+ CmpOp op = branch_when_true ? v->delayed->cmp.op
+ : api_invert_cmp(v->delayed->cmp.op);
+ T->cmp_branch(T, op, v->delayed->cmp.a, v->delayed->cmp.b, label);
api_release(g, v);
return;
}
diff --git a/src/cg/fold.c b/src/cg/fold.c
@@ -10,6 +10,44 @@
#include "cg/ir_eval.h"
/* ============================================================
+ * 0. Delayed-payload pool (off-node SV_CMP / SV_ARITH state)
+ *
+ * A delayed value's cmp/arith payload lives here, behind the ApiSValue.delayed
+ * pointer, rather than inline on the value-stack node — which keeps the node
+ * small and its per-push construction free of delayed state. Payloads are
+ * bump-allocated from a per-function arena and recycled through an intrusive
+ * freelist (api_delayed_free), so within a function the arena only grows to the
+ * live-delayed-value working set. api_delayed_reset drops both arena and
+ * freelist at a function boundary (where the value stack is also dropped), so a
+ * payload never outlives its function and a missed free only enlarges the
+ * reset-per-function arena — it cannot dangle or corrupt.
+ * ============================================================ */
+
+ApiDelayed* api_delayed_alloc(KitCg* g) {
+ ApiDelayed* d = g->delayed_free;
+ if (d) {
+ g->delayed_free = d->next_free;
+ return d;
+ }
+ if (!g->delayed_arena_init) {
+ arena_init(&g->delayed_arena, (Heap*)g->c->ctx->heap, 4096);
+ g->delayed_arena_init = 1;
+ }
+ return arena_new(&g->delayed_arena, ApiDelayed);
+}
+
+void api_delayed_free(KitCg* g, ApiDelayed* d) {
+ if (!d) return;
+ d->next_free = g->delayed_free;
+ g->delayed_free = d;
+}
+
+void api_delayed_reset(KitCg* g) {
+ if (g->delayed_arena_init) arena_reset(&g->delayed_arena);
+ g->delayed_free = NULL;
+}
+
+/* ============================================================
* 1. Integer constant folding
* ============================================================ */
@@ -95,17 +133,19 @@ int api_try_fold_int_cmp(KitCg* g, CmpOp op, KitCgTypeId ty, i64 a, i64 b,
* 2a. Delayed compare (SV_CMP) lifecycle
* ============================================================ */
-ApiSValue api_make_cmp(CmpOp op, Operand a, Operand b, KitCgTypeId result_ty,
- int a_owned, int b_owned) {
+ApiSValue api_make_cmp(KitCg* g, CmpOp op, Operand a, Operand b,
+ KitCgTypeId result_ty, int a_owned, int b_owned) {
+ ApiDelayed* d = api_delayed_alloc(g);
+ d->cmp = (ApiDelayedCmp){.op = op,
+ .a = a,
+ .b = b,
+ .a_owned = a_owned ? 1u : 0u,
+ .b_owned = b_owned ? 1u : 0u};
return (ApiSValue){.kind = SV_CMP,
.type = result_ty,
.res = RES_INHERENT,
.source_local = KIT_CG_LOCAL_NONE,
- .delayed.cmp = {.op = op,
- .a = a,
- .b = b,
- .a_owned = a_owned ? 1u : 0u,
- .b_owned = b_owned ? 1u : 0u}};
+ .delayed = d};
}
CmpOp api_invert_cmp(CmpOp op) {
@@ -162,21 +202,16 @@ CmpOp api_invert_cmp(CmpOp op) {
}
void api_release_cmp(KitCg* g, ApiSValue* sv) {
- (void)g;
- memset(&sv->delayed.cmp.a, 0, sizeof sv->delayed.cmp.a);
- memset(&sv->delayed.cmp.b, 0, sizeof sv->delayed.cmp.b);
- sv->delayed.cmp.a_owned = 0;
- sv->delayed.cmp.b_owned = 0;
+ api_delayed_free(g, sv->delayed);
+ sv->delayed = NULL;
sv->kind = SV_OPERAND;
}
void api_materialize_cmp_to(KitCg* g, ApiSValue* sv, Operand dst) {
- g->target->cmp(g->target, sv->delayed.cmp.op, dst, sv->delayed.cmp.a,
- sv->delayed.cmp.b);
- memset(&sv->delayed.cmp.a, 0, sizeof sv->delayed.cmp.a);
- memset(&sv->delayed.cmp.b, 0, sizeof sv->delayed.cmp.b);
- sv->delayed.cmp.a_owned = 0;
- sv->delayed.cmp.b_owned = 0;
+ ApiDelayed* d = sv->delayed;
+ g->target->cmp(g->target, d->cmp.op, dst, d->cmp.a, d->cmp.b);
+ api_delayed_free(g, d);
+ sv->delayed = NULL;
sv->kind = SV_OPERAND;
sv->op = dst;
sv->type = dst.type;
@@ -195,52 +230,51 @@ void api_materialize_cmp_to(KitCg* g, ApiSValue* sv, Operand dst) {
* Track 6.3 flipped the gate back on.
* ============================================================ */
-ApiSValue api_make_arith_unop(UnOp op, Operand a, KitCgTypeId ty, int a_owned) {
+ApiSValue api_make_arith_unop(KitCg* g, UnOp op, Operand a, KitCgTypeId ty,
+ int a_owned) {
+ ApiDelayed* d = api_delayed_alloc(g);
+ d->arith = (ApiDelayedArith){.kind = API_DELAYED_UNOP,
+ .un_op = op,
+ .a = a,
+ .a_owned = a_owned ? 1u : 0u};
return (ApiSValue){.kind = SV_ARITH,
.type = ty,
.res = RES_INHERENT,
.source_local = KIT_CG_LOCAL_NONE,
- .delayed.arith = {.kind = API_DELAYED_UNOP,
- .un_op = op,
- .a = a,
- .a_owned = a_owned ? 1u : 0u}};
-}
-
-ApiSValue api_make_arith_binop(BinOp op, Operand a, Operand b, KitCgTypeId ty,
- int a_owned, int b_owned) {
+ .delayed = d};
+}
+
+ApiSValue api_make_arith_binop(KitCg* g, BinOp op, Operand a, Operand b,
+ KitCgTypeId ty, int a_owned, int b_owned) {
+ ApiDelayed* d = api_delayed_alloc(g);
+ d->arith = (ApiDelayedArith){.kind = API_DELAYED_BINOP,
+ .bin_op = op,
+ .a = a,
+ .b = b,
+ .a_owned = a_owned ? 1u : 0u,
+ .b_owned = b_owned ? 1u : 0u};
return (ApiSValue){.kind = SV_ARITH,
.type = ty,
.res = RES_INHERENT,
.source_local = KIT_CG_LOCAL_NONE,
- .delayed.arith = {.kind = API_DELAYED_BINOP,
- .bin_op = op,
- .a = a,
- .b = b,
- .a_owned = a_owned ? 1u : 0u,
- .b_owned = b_owned ? 1u : 0u}};
+ .delayed = d};
}
void api_release_arith(KitCg* g, ApiSValue* sv) {
- (void)g;
- memset(&sv->delayed.arith.a, 0, sizeof sv->delayed.arith.a);
- memset(&sv->delayed.arith.b, 0, sizeof sv->delayed.arith.b);
- sv->delayed.arith.a_owned = 0;
- sv->delayed.arith.b_owned = 0;
+ api_delayed_free(g, sv->delayed);
+ sv->delayed = NULL;
sv->kind = SV_OPERAND;
}
void api_materialize_arith_to(KitCg* g, ApiSValue* sv, Operand dst) {
- if (sv->delayed.arith.kind == API_DELAYED_UNOP) {
- g->target->unop(g->target, sv->delayed.arith.un_op, dst,
- sv->delayed.arith.a);
+ ApiDelayed* d = sv->delayed;
+ if (d->arith.kind == API_DELAYED_UNOP) {
+ g->target->unop(g->target, d->arith.un_op, dst, d->arith.a);
} else {
- g->target->binop(g->target, sv->delayed.arith.bin_op, dst,
- sv->delayed.arith.a, sv->delayed.arith.b);
+ g->target->binop(g->target, d->arith.bin_op, dst, d->arith.a, d->arith.b);
}
- memset(&sv->delayed.arith.a, 0, sizeof sv->delayed.arith.a);
- memset(&sv->delayed.arith.b, 0, sizeof sv->delayed.arith.b);
- sv->delayed.arith.a_owned = 0;
- sv->delayed.arith.b_owned = 0;
+ api_delayed_free(g, d);
+ sv->delayed = NULL;
sv->kind = SV_OPERAND;
sv->op = dst;
sv->type = dst.type;
@@ -249,8 +283,8 @@ void api_materialize_arith_to(KitCg* g, ApiSValue* sv, Operand dst) {
}
int api_arith_rhs_reusable(const ApiSValue* sv) {
- if (sv->delayed.arith.kind == API_DELAYED_UNOP) return 0;
- switch (sv->delayed.arith.bin_op) {
+ if (sv->delayed->arith.kind == API_DELAYED_UNOP) return 0;
+ switch (sv->delayed->arith.bin_op) {
case BO_IADD:
case BO_IMUL:
case BO_AND:
@@ -440,22 +474,24 @@ int api_try_fold_arith_chain(KitCg* g, BinOp op, KitCgTypeId ty, ApiSValue* a,
ApiSValue* b, ApiSValue* out) {
i64 folded;
BinOp result_op;
- if (a->kind != SV_ARITH || a->delayed.arith.kind != API_DELAYED_BINOP ||
- a->delayed.arith.a.kind != OPK_LOCAL ||
- a->delayed.arith.b.kind != OPK_IMM || b->kind != SV_OPERAND ||
+ ApiDelayed* ad;
+ if (a->kind != SV_ARITH || a->delayed->arith.kind != API_DELAYED_BINOP ||
+ a->delayed->arith.a.kind != OPK_LOCAL ||
+ a->delayed->arith.b.kind != OPK_IMM || b->kind != SV_OPERAND ||
b->op.kind != OPK_IMM) {
return 0;
}
- result_op = a->delayed.arith.bin_op;
- switch (a->delayed.arith.bin_op) {
+ ad = a->delayed;
+ result_op = ad->arith.bin_op;
+ switch (ad->arith.bin_op) {
case BO_IADD:
if (op == BO_IADD) {
- if (!api_try_fold_int_binop(g, BO_IADD, ty, a->delayed.arith.b.v.imm,
+ if (!api_try_fold_int_binop(g, BO_IADD, ty, ad->arith.b.v.imm,
b->op.v.imm, &folded))
return 0;
result_op = BO_IADD;
} else if (op == BO_ISUB) {
- if (!api_try_fold_int_binop(g, BO_ISUB, ty, a->delayed.arith.b.v.imm,
+ if (!api_try_fold_int_binop(g, BO_ISUB, ty, ad->arith.b.v.imm,
b->op.v.imm, &folded))
return 0;
result_op = BO_IADD;
@@ -466,11 +502,11 @@ int api_try_fold_arith_chain(KitCg* g, BinOp op, KitCgTypeId ty, ApiSValue* a,
case BO_ISUB:
if (op == BO_IADD) {
if (!api_try_fold_int_binop(g, BO_ISUB, ty, b->op.v.imm,
- a->delayed.arith.b.v.imm, &folded))
+ ad->arith.b.v.imm, &folded))
return 0;
result_op = BO_IADD;
} else if (op == BO_ISUB) {
- if (!api_try_fold_int_binop(g, BO_IADD, ty, a->delayed.arith.b.v.imm,
+ if (!api_try_fold_int_binop(g, BO_IADD, ty, ad->arith.b.v.imm,
b->op.v.imm, &folded))
return 0;
result_op = BO_ISUB;
@@ -479,23 +515,23 @@ int api_try_fold_arith_chain(KitCg* g, BinOp op, KitCgTypeId ty, ApiSValue* a,
}
break;
case BO_XOR:
- if (op != BO_XOR ||
- !api_try_fold_int_binop(g, BO_XOR, ty, a->delayed.arith.b.v.imm,
- b->op.v.imm, &folded))
+ if (op != BO_XOR || !api_try_fold_int_binop(g, BO_XOR, ty,
+ ad->arith.b.v.imm,
+ b->op.v.imm, &folded))
return 0;
result_op = BO_XOR;
break;
case BO_AND:
- if (op != BO_AND ||
- !api_try_fold_int_binop(g, BO_AND, ty, a->delayed.arith.b.v.imm,
- b->op.v.imm, &folded))
+ if (op != BO_AND || !api_try_fold_int_binop(g, BO_AND, ty,
+ ad->arith.b.v.imm,
+ b->op.v.imm, &folded))
return 0;
result_op = BO_AND;
break;
case BO_OR:
- if (op != BO_OR ||
- !api_try_fold_int_binop(g, BO_OR, ty, a->delayed.arith.b.v.imm,
- b->op.v.imm, &folded))
+ if (op != BO_OR || !api_try_fold_int_binop(g, BO_OR, ty,
+ ad->arith.b.v.imm, b->op.v.imm,
+ &folded))
return 0;
result_op = BO_OR;
break;
@@ -503,34 +539,36 @@ int api_try_fold_arith_chain(KitCg* g, BinOp op, KitCgTypeId ty, ApiSValue* a,
return 0;
}
if (api_op_is_int_identity(g, result_op, ty, folded)) {
- *out = api_make_sv_with_local_ownership(a->delayed.arith.a, ty,
- a->delayed.arith.a_owned);
- a->delayed.arith.a_owned = 0;
- memset(&a->delayed.arith.a, 0, sizeof a->delayed.arith.a);
+ /* Collapses to a's input operand: out is a plain value with no payload, and
+ * a keeps its payload for the caller's api_release to reclaim. The owned
+ * local (if any) is handed to out; a's stale a_owned is never acted on
+ * (api_release_arith only frees the payload). */
+ *out = api_make_sv_with_local_ownership(ad->arith.a, ty, ad->arith.a_owned);
return 1;
}
- a->delayed.arith.bin_op = result_op;
- a->delayed.arith.b.v.imm = folded;
+ /* Chain-folds into an updated delayed binop. Move a's payload to out (a
+ * pointer transfer, not a deep copy) and null a's pointer so the caller's
+ * api_release reclaims nothing — out owns the payload until it materializes. */
+ ad->arith.bin_op = result_op;
+ ad->arith.b.v.imm = folded;
*out = *a;
- a->delayed.arith.a_owned = 0;
- a->delayed.arith.b_owned = 0;
- memset(&a->delayed.arith.a, 0, sizeof a->delayed.arith.a);
- memset(&a->delayed.arith.b, 0, sizeof a->delayed.arith.b);
+ a->delayed = NULL;
return 1;
}
int api_try_fold_unary_chain(ApiSValue* a, UnOp op, KitCgTypeId ty,
ApiSValue* out) {
if (op != UO_BNOT || a->kind != SV_ARITH ||
- a->delayed.arith.kind != API_DELAYED_UNOP ||
- a->delayed.arith.un_op != UO_BNOT ||
- a->delayed.arith.a.kind != OPK_LOCAL) {
+ a->delayed->arith.kind != API_DELAYED_UNOP ||
+ a->delayed->arith.un_op != UO_BNOT ||
+ a->delayed->arith.a.kind != OPK_LOCAL) {
return 0;
}
- *out = api_make_sv_with_local_ownership(a->delayed.arith.a, ty,
- a->delayed.arith.a_owned);
- a->delayed.arith.a_owned = 0;
- memset(&a->delayed.arith.a, 0, sizeof a->delayed.arith.a);
+ /* out borrows a's input operand (a plain value); a keeps its payload for the
+ * caller's api_release to reclaim. */
+ *out =
+ api_make_sv_with_local_ownership(a->delayed->arith.a, ty,
+ a->delayed->arith.a_owned);
return 1;
}
diff --git a/src/cg/fold.h b/src/cg/fold.h
@@ -59,17 +59,18 @@ int api_try_fold_int_cmp(KitCg* g, CmpOp op, KitCgTypeId ty, i64 a, i64 b,
/* ---- 2a. delayed compare (SV_CMP) lifecycle ---- */
-ApiSValue api_make_cmp(CmpOp op, Operand a, Operand b, KitCgTypeId result_ty,
- int a_owned, int b_owned);
+ApiSValue api_make_cmp(KitCg* g, CmpOp op, Operand a, Operand b,
+ KitCgTypeId result_ty, int a_owned, int b_owned);
void api_release_cmp(KitCg* g, ApiSValue* sv);
void api_materialize_cmp_to(KitCg* g, ApiSValue* sv, Operand dst);
CmpOp api_invert_cmp(CmpOp op);
/* ---- 2b. delayed arith (SV_ARITH) lifecycle — live (Track 6.3) ---- */
-ApiSValue api_make_arith_unop(UnOp op, Operand a, KitCgTypeId ty, int a_owned);
-ApiSValue api_make_arith_binop(BinOp op, Operand a, Operand b, KitCgTypeId ty,
- int a_owned, int b_owned);
+ApiSValue api_make_arith_unop(KitCg* g, UnOp op, Operand a, KitCgTypeId ty,
+ int a_owned);
+ApiSValue api_make_arith_binop(KitCg* g, BinOp op, Operand a, Operand b,
+ KitCgTypeId ty, int a_owned, int b_owned);
void api_release_arith(KitCg* g, ApiSValue* sv);
void api_materialize_arith_to(KitCg* g, ApiSValue* sv, Operand dst);
int api_arith_rhs_reusable(const ApiSValue* sv);
diff --git a/src/cg/internal.h b/src/cg/internal.h
@@ -59,6 +59,18 @@ typedef struct ApiDelayedArith {
u8 pad;
} ApiDelayedArith;
+/* The delayed cmp/arith payload of an SV_CMP / SV_ARITH value. It is held off
+ * the hot value-stack node (ApiSValue carries only an ApiDelayed*, NULL for the
+ * common SV_OPERAND case) so the node stays small and its per-push construction
+ * touches no delayed state. Payloads are pooled per function (api_delayed_alloc
+ * / _free, arena-backed with a freelist; see fold.c). `next_free` overlays the
+ * payload while it sits on the freelist. */
+typedef union ApiDelayed {
+ ApiDelayedCmp cmp;
+ ApiDelayedArith arith;
+ union ApiDelayed* next_free;
+} ApiDelayed;
+
/* Scalar wide-class tag cached on every value-stack node (api_push). The wide /
* soft-float dispatch in arith.c re-derived this on every operand with a string
* of type-get + alias-chase round trips; it is a pure function of the value type
@@ -108,10 +120,10 @@ typedef struct ApiBitField {
typedef struct ApiSValue {
Operand op;
- union {
- ApiDelayedCmp cmp;
- ApiDelayedArith arith;
- } delayed;
+ /* Pooled delayed cmp/arith payload, or NULL for the common SV_OPERAND value.
+ * A pointer (not the 64-byte union inline) keeps the node small; only the
+ * SV_CMP / SV_ARITH producers in fold.c allocate one. */
+ ApiDelayed* delayed;
KitCgTypeId type;
u8 kind;
u8 res;
@@ -187,6 +199,15 @@ struct KitCg {
u32 sp;
u32 cap;
+ /* Off-node pool for SV_CMP / SV_ARITH delayed payloads. The arena is reset
+ * per function (api_delayed_reset at func_begin, where the stack is also
+ * dropped); delayed_free is an intrusive freelist that reuses payloads within
+ * a function so the arena only grows to the live delayed-value working set. */
+ Arena delayed_arena;
+ ApiDelayed* delayed_free;
+ u8 delayed_arena_init;
+ u8 delayed_pad[3];
+
ApiSourceLocal* locals;
u32 nlocals;
u32 locals_cap;
@@ -433,6 +454,13 @@ int api_is_wide8_scalar_type(Compiler* c, KitCgTypeId ty);
u8 api_wide_kind_for(Compiler* c, KitCgTypeId ty);
/* Memoized fetch of the packed API_TYPE_CLASS_* byte for `ty` (see above). */
u8 api_type_class(KitCg* g, KitCgTypeId ty);
+
+/* Delayed cmp/arith payload pool (fold.c). alloc returns a payload from the
+ * per-function freelist or the arena; free returns it for reuse; reset drops
+ * the whole pool at a function boundary. */
+ApiDelayed* api_delayed_alloc(KitCg* g);
+void api_delayed_free(KitCg* g, ApiDelayed* d);
+void api_delayed_reset(KitCg* g);
Operand api_op_imm(i64 v, KitCgTypeId ty);
Operand api_op_local(CGLocal r, KitCgTypeId ty);
Operand api_op_global(ObjSymId sym, i64 addend, KitCgTypeId ty);
diff --git a/src/cg/session.c b/src/cg/session.c
@@ -43,6 +43,11 @@ static void cg_free_obj_state(KitCg* g) {
sizeof(*g->data_tls_relocs) * g->data_tls_relocs_cap);
g->data_tls_relocs = NULL;
}
+ if (g->delayed_arena_init) {
+ arena_fini(&g->delayed_arena);
+ g->delayed_arena_init = 0;
+ }
+ g->delayed_free = NULL;
g->sp = 0;
g->cap = 0;
g->nlocals = 0;
@@ -402,6 +407,9 @@ void kit_cg_func_begin_attrs(KitCg* g, KitCgSym cg_sym,
g->nlocals = 0;
g->const_head = KIT_CG_LOCAL_NONE;
g->sp = 0;
+ /* Drop the previous function's delayed cmp/arith payloads along with its
+ * value stack — neither outlives the function. */
+ api_delayed_reset(g);
if (g->debug) {
DebugTypeId dt = api_debug_type(g, fty);
diff --git a/src/cg/value.c b/src/cg/value.c
@@ -9,6 +9,10 @@ _Static_assert(sizeof(ApiBitField) == 12,
_Static_assert(offsetof(ApiSValue, bitfield) + sizeof(ApiBitField) ==
sizeof(ApiSValue),
"ApiSValue size must be unchanged by the cached wide_kind tag");
+/* The 64-byte delayed cmp/arith payload lives off the node behind a pointer
+ * (see ApiDelayed); the value-stack node must stay small. */
+_Static_assert(sizeof(ApiSValue) <= 64,
+ "delayed payload must be off-node; ApiSValue stays small");
int api_type_is_float(Compiler* c, KitCgTypeId ty) {
const CgType* cg;
@@ -428,14 +432,15 @@ void api_ensure_local(KitCg* g, ApiSValue* sv) {
if (sv->kind == SV_CMP) {
KitCgTypeId ty = api_sv_type(sv);
Operand dst;
- if (sv->delayed.cmp.a_owned && sv->delayed.cmp.a.kind == OPK_LOCAL &&
- api_unalias_type(g->c, sv->delayed.cmp.a.type) ==
+ if (sv->delayed->cmp.a_owned && sv->delayed->cmp.a.kind == OPK_LOCAL &&
+ api_unalias_type(g->c, sv->delayed->cmp.a.type) ==
api_unalias_type(g->c, ty)) {
- dst = api_op_local(sv->delayed.cmp.a.v.local, ty);
- } else if (sv->delayed.cmp.b_owned && sv->delayed.cmp.b.kind == OPK_LOCAL &&
- api_unalias_type(g->c, sv->delayed.cmp.b.type) ==
+ dst = api_op_local(sv->delayed->cmp.a.v.local, ty);
+ } else if (sv->delayed->cmp.b_owned &&
+ sv->delayed->cmp.b.kind == OPK_LOCAL &&
+ api_unalias_type(g->c, sv->delayed->cmp.b.type) ==
api_unalias_type(g->c, ty)) {
- dst = api_op_local(sv->delayed.cmp.b.v.local, ty);
+ dst = api_op_local(sv->delayed->cmp.b.v.local, ty);
} else {
CGLocal r =
api_alloc_temp_local(g, ty ? ty : builtin_id(KIT_CG_BUILTIN_I32));
@@ -447,15 +452,15 @@ void api_ensure_local(KitCg* g, ApiSValue* sv) {
if (sv->kind == SV_ARITH) {
KitCgTypeId ty = api_sv_type(sv);
Operand dst;
- if (sv->delayed.arith.a_owned && sv->delayed.arith.a.kind == OPK_LOCAL &&
- api_unalias_type(g->c, sv->delayed.arith.a.type) ==
+ if (sv->delayed->arith.a_owned && sv->delayed->arith.a.kind == OPK_LOCAL &&
+ api_unalias_type(g->c, sv->delayed->arith.a.type) ==
api_unalias_type(g->c, ty)) {
- dst = api_op_local(sv->delayed.arith.a.v.local, ty);
- } else if (api_arith_rhs_reusable(sv) && sv->delayed.arith.b_owned &&
- sv->delayed.arith.b.kind == OPK_LOCAL &&
- api_unalias_type(g->c, sv->delayed.arith.b.type) ==
+ dst = api_op_local(sv->delayed->arith.a.v.local, ty);
+ } else if (api_arith_rhs_reusable(sv) && sv->delayed->arith.b_owned &&
+ sv->delayed->arith.b.kind == OPK_LOCAL &&
+ api_unalias_type(g->c, sv->delayed->arith.b.type) ==
api_unalias_type(g->c, ty)) {
- dst = api_op_local(sv->delayed.arith.b.v.local, ty);
+ dst = api_op_local(sv->delayed->arith.b.v.local, ty);
} else {
CGLocal r =
api_alloc_temp_local(g, ty ? ty : builtin_id(KIT_CG_BUILTIN_I32));
