kit

kit
git clone https://git.ryansepassi.com/git/kit.git
Log | Files | Refs | README

commit 8f565bea093f2924ca3ca86ac810a02501b9e026
parent 32f88989b99113efd573a10ab332c1e7732b54ff
Author: Ryan Sepassi <rsepassi@gmail.com>
Date:   Sun, 10 May 2026 10:34:37 -0700

test: MULTIARCH Phase 1 — per-arch exec helper, x64 smoke

Stand up the test-harness seams doc/MULTIARCH.md §4 phase 1 calls for,
so phase 3's x64 codegen milestones can be gated on real podman/qemu
runs from their first commit. No compiler code moves in this commit.

- test/lib/exec_target.sh replaces test/lib/exec_aarch64.sh. The API
  (exec_target_run/queue/queue_size/flush/supported) takes target arch
  as its first argument; the queue records each entry's arch tag and
  flush groups by arch, running one batched `podman run` per arch so
  the per-launch overhead still amortizes across the whole suite.
  Recognized arches: aarch64, x64. Native, qemu-user, and podman
  (with --platform) are tried in that order.
- test/{cg,parse,link}/run.sh migrated to the new API. The cg
  harness queries cg-runner --arches NAME per case and dispatches
  exec_target_supported / exec_target_queue against the case's arch.
- test/cg/harness: CgCase gains an `arches` mask
  (CG_ARCH_AARCH64 | CG_ARCH_X64); 0 means CG_ARCH_DEFAULT, so the
  existing positional initializers in cases.c stay untouched. New
  cg-runner --arches NAME mode prints one arch token per line.
- test/smoke/x64.sh: builds a freestanding `_start` that direct-syscalls
  exit_group(42) with clang --target=x86_64-linux-gnu -fuse-ld=lld and
  runs it through both exec_target_run and exec_target_queue+flush.
  Wired into test/test.mk as the opt-in `make test-smoke-x64` target
  (mirrors test-musl/test-glibc — needs podman, kept out of default
  `test`).
- doc/MULTIARCH.md: the phase-1/2/3 plan this commit implements
  phase 1 of.

Exit criterion (plan §4 phase 1): test-smoke-x64 reports `2 pass, 0
fail` end-to-end through podman with --platform linux/amd64.

Diffstat:
Adoc/MULTIARCH.md | 286+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Mtest/cg/harness/cg_runner.c | 19+++++++++++++++++++
Mtest/cg/harness/cg_test.h | 16++++++++++++++--
Dtest/lib/exec_aarch64.sh | 132-------------------------------------------------------------------------------
Atest/lib/exec_target.sh | 251+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Mtest/link/run.sh | 21+++++++++++----------
Mtest/parse/run.sh | 20++++++++++----------
Atest/smoke/x64.sh | 136+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Mtest/test.mk | 11++++++++++-
9 files changed, 737 insertions(+), 155 deletions(-)

diff --git a/doc/MULTIARCH.md b/doc/MULTIARCH.md @@ -0,0 +1,286 @@ +# MULTIARCH — plan for adding a second architecture + +Scope: turn cfree from an aarch64-only compiler into one that supports +multiple `(arch, os, objfmt)` triples. The first new arch is x86_64; +the first new platform/objfmt and the asm frontend land later, on the +seams this work establishes. + +Today the codebase has one of each: aarch64 codegen (`src/arch/aarch64.c`), +AAPCS64 ABI (`src/abi/abi.c`), ELF emission with aarch64 relocs +(`src/obj/elf_emit.c` + `src/obj/elf_reloc_aarch64.c`), and an aarch64 +emulator-driven test path (`src/emu/`). `cgtarget_new`, `emit_elf`, +`link_elf`, and the disassembler all panic on non-AArch64 targets. + +The goal of the first phase is to introduce the seams that a second +arch forces — without yet writing x64 codegen. After the seams land, +x64 bring-up is purely additive: new files, no edits to the +arch-aware ones except the dispatch tables. + +--- + +## 1. Target slice for first x64 milestone + +| axis | value | +|-----------|--------------------------------| +| arch | `CFREE_ARCH_X86_64` | +| os | `CFREE_OS_LINUX` | +| objfmt | `CFREE_OBJ_ELF` | +| ABI | SysV AMD64 | +| codemodel | `CFREE_CM_SMALL` (default) | + +Mach-O, PE/COFF, Win64 ABI, and macOS-arm64 all wait until the seams +are validated by a working x86_64-linux-gnu path. This keeps the +arch-seam work decoupled from the objfmt-seam work — only one is on +the critical path at a time. + +--- + +## 2. The seams + +### 2.1 CGTarget construction — dispatch by arch + +`src/arch/aarch64.c:2998` is currently the public `cgtarget_new`. Split: + +- Rename the AArch64 constructor to `aa64_cgtarget_new`, declared in a + new `src/arch/aa64.h`. +- Add `src/arch/x64.h` and `src/arch/x64.c` with `x64_cgtarget_new` and + the equivalent `XImpl` skeleton (vtable wired up to method stubs). +- New `src/arch/cgtarget.c` owns the public `cgtarget_new` and switches + on `c->target.arch`. + +Same dispatch shape for `arch_disasm_new` (already factored via the +`ArchDisasm` hook — just needs a switch). + +`mc_new` does not change yet: a single `MCEmitter` impl serves all +arches. + +### 2.2 MCEmitter fixup encodings — extend the switch + +`src/arch/mc.c:84-134` has aarch64 BL/B.cond/CONDBR19 bit layouts +hardcoded in `apply_fixup`. **Decision:** extend the same switch with +x64 cases (`R_PC32` already works for jumps; add `R_PC8` for short +jumps). No per-arch fixup vtable — the encoding is one-line +little-endian patching, and the abstraction would be premature. + +Action item: update the file's "target-agnostic" header comment to +reflect that `mc.c` is the union of all known fixup encodings, not a +generic library. + +### 2.3 ABI classification — TargetABI vtable + +**Decision:** promote `TargetABI` to carry function pointers for the +parts that vary by `(arch, os)`. `abi_init` selects the right impl set +based on `c->target`. Rationale: even at two ABIs, the SysV AMD64 +classifier is large enough (eight-byte classification, INTEGER/SSE +classes, x87 corners) that an in-line switch in `abi_func_info` would +be ugly; and Win64 + macOS-arm64 are visible on the roadmap, so the +indirection pays off quickly. + +Concrete changes: + +- Add a vtable to `TargetABI` (function pointers for `func_info`, + `record_layout`, `va_list_type`, scalar profiles where they vary). +- Move the AAPCS64 classifier out of `abi.c` into + `src/abi/abi_aapcs64.c`, exposing an `aapcs64_vtable` symbol. +- Add `src/abi/abi_sysv_x64.c` exposing `sysv_x64_vtable`. Initial + classifier returns `ABI_ARG_INDIRECT` for everything (correct, slow, + unblocks bring-up); fill in the eight-byte rules incrementally. +- `abi_init` switches on `(target.arch, target.os)` and copies the + right vtable in. + +The public `abi_func_info` / `abi_record_layout` / `abi_*_type` API in +`src/abi/abi.h` does not change — only the internals dispatch through +the vtable. + +### 2.4 Object format — ELF reloc translator dispatch + +`src/obj/elf_reloc_aarch64.c` already exists as a per-arch reloc +translator; the seam is half there. Finish it: + +- Add `src/obj/elf_reloc_x86_64.c` mirroring it for `R_X86_64_*` codes. +- Extend `RelocKind` in `src/obj/obj.h` with `R_X64_*` entries + (`R_X64_PC32`, `R_X64_PLT32`, `R_X64_GOTPCREL`, ...). +- `src/obj/elf_emit.c:246-249` panics on non-aarch64 and hardcodes + `e_machine`. Replace with a switch that picks `EM_AARCH64` / + `EM_X86_64` and the right reloc-translator function pointer. +- `src/link/link_elf.c:575` — same treatment. + +Mach-O and PE/COFF emitters slot in as peers of `elf_emit.c` later, +each with its own per-arch reloc translator file. The reloc-translator +pattern established here is what makes that cheap. + +### 2.5 Header types per (arch, abi) + +`abi_size_type`, `abi_ptrdiff_type`, `abi_intptr_type`, +`abi_uintptr_type`, `abi_va_list_type` are already abstracted. The +SysV-x64 vtable (§2.3) supplies the right `__va_list_tag` struct +(`gp_offset`, `fp_offset`, `overflow_arg_area`, `reg_save_area`). + +`rt/include/` headers that are arch-conditioned will gate on +`__x86_64__` / `__aarch64__` predefines in the preprocessor (already +the convention used elsewhere in `rt/`). No new mechanism needed. + +--- + +## 3. Test/run path — execute from day 1 + +The harness already uses podman+qemu for aarch64 — see +`test/lib/exec_aarch64.sh`, `test/cg/run.sh:104-118`, and the libc +sysroot extractors in `test/libc/{musl,glibc}/`. The pattern: detect +qemu-user or podman, batch all queued exes through one `podman run` +to amortize launch overhead, fall back to a host serial loop. Test +selection is per-arch with per-target XFAIL. + +x64 inherits this machinery wholesale. There is no new "runner" to +design — only an x64-shaped peer of `exec_aarch64.sh` and a per-arch +dispatch where `cg/run.sh` currently sources the aarch64 helper +unconditionally. The default container image (`alpine:latest`) +already runs linux/amd64 binaries; the `--platform linux/amd64` flag +selection mirrors the existing `is_aarch64` gate in +`exec_aarch64.sh:48`. + +Execution tests are gating from the first hello-world. We do not +build out an x64 lifter inside `src/emu/` — that path is aarch64-only +and stays so. + +--- + +## 4. Phasing + +Three phases. Each is independently mergeable; phase 1 and phase 2 +land with no aarch64 behavior change. + +### Phase 1 — test harness updates + +Stand up the x64 execution path before any compiler code moves, so +phase 3 milestones can be gated on real runs from their first commit. + +1. Generalize `test/lib/exec_aarch64.sh` into a per-arch helper: + either rename to `exec_target.sh` with arch-keyed function names + (`exec_target_run aarch64 ...`), or add a peer + `test/lib/exec_x64.sh` with the same surface. The batched + `podman run` body is identical — only `--platform`, + `RUN_*_IMAGE`, and the `is_<arch>` gate vary. +2. `test/cg/run.sh` (and the other harnesses that source the helper) + dispatch the executor by the case's target arch. The existing + single-helper sourcing (`run.sh:118`) becomes a per-target choice. +3. Sysroot extractors: add `test/libc/{musl,glibc}/` x86_64 variants + if the libc tests claim x64 coverage. For the cg suite (static, + no libc), no sysroot is needed — `alpine:latest` is enough. +4. Smoke test: a hand-rolled or clang-built x86_64 ELF is queued and + flushed through the new executor and exits cleanly. This proves + the harness end-to-end before any cfree-emitted x64 bytes exist. +5. Per-test arch declaration: `test/cg/` cases gain a way to say + which arches they run on (default: all supported). Per-target + XFAIL is keyed off the same. + +Exit criterion: aarch64 suite green; the smoke test runs an external +x86_64 binary through the harness and reports pass on the standard +pass/fail line. + +### Phase 2 — code refactors (multi-arch seams + x64 stubs) + +Pure refactors. No aarch64 output changes; x64 reachable but every +codegen call panics with "x64: not implemented". + +1. **CGTarget dispatch** (§2.1). Rename `aarch64.c::cgtarget_new` to + `aa64_cgtarget_new`, declared in `arch/aa64.h`. New + `arch/cgtarget.c` owns the public `cgtarget_new` and switches on + `c->target.arch`. Same split for `arch_disasm_new`. +2. **x64 skeleton.** Add `arch/x64.{h,c}` with the full vtable wired + up to stub methods that panic on call. `cgtarget_new` dispatches + to it for `CFREE_ARCH_X86_64`. +3. **ABI vtable** (§2.3). Promote `TargetABI` to carry function + pointers for `func_info`, `record_layout`, `va_list_type`, and + the scalar profiles that vary. `abi_init` switches on + `(target.arch, target.os)` and installs the right vtable. +4. **AAPCS64 split.** Move the AAPCS64 classifier from `abi.c` into + `abi/abi_aapcs64.c`, exposing `aapcs64_vtable`. `abi.c` keeps the + generic dispatch and the C-standard-driven scalar/record bits. +5. **SysV-x64 stub.** Add `abi/abi_sysv_x64.c` exposing + `sysv_x64_vtable` with `ABI_ARG_INDIRECT` for everything. Wired + up by `abi_init` for `(X86_64, LINUX)` but unreachable until + phase 3 fills `arch/x64.c`. +6. **ELF reloc dispatch** (§2.4). Add `R_X64_*` to `RelocKind`. New + `obj/elf_reloc_x86_64.c` mirrors `elf_reloc_aarch64.c`. + `obj/elf_emit.c:246-249` and `link/link_elf.c:575` lose their + AArch64-only panics in favor of a switch on `c->target.arch` that + picks `e_machine` (`EM_AARCH64` / `EM_X86_64`) and the reloc + translator. +7. **Allowlist consolidation.** Other `arch != ARM_64` panics + (anywhere they remain after the splits above) move into one + target-validation function, easy to extend. + +Exit criterion: aarch64 suite green and byte-for-byte identical +output objects on a representative `test/cg/` set; an x86_64-linux +target reaches `arch/x64.c`'s stubs (the panic is the proof the +dispatch is wired). + +### Phase 3 — implementations + +Now purely additive. Fill in the stubs from phase 2; each milestone +gates on the podman/qemu execution path from phase 1. + +1. **mc.c x64 fixups.** Add x64 reloc-kind cases to + `arch/mc.c::apply_fixup` (`R_PC32` already works for jumps; + `R_PC8` for short jumps; whatever else x64 codegen actually + emits). +2. **SysV-x64 ABI classifier.** Replace the `ABI_ARG_INDIRECT` stub + in `abi/abi_sysv_x64.c` with the eight-byte INTEGER/SSE + classification. `va_list` returns the SysV `__va_list_tag` + struct. +3. **x64 codegen** in `arch/x64.c`, in the rough order the parser + phases established for aarch64: + 1. Hello-world (`int main(void) { return 0; }`) — exit via + syscall through inline asm, or a libc-less return path. + 2. Integer arithmetic + locals — frame slots, spill/reload on the + x64 register pool. The CG-driven spill/reload from commit + `9724439` is reusable; only the physical-register pool and + load/store encodings change. + 3. Calls — SysV register passing, basic eight-byte classification + paired with the ABI work above. + 4. Loads/stores at every width — `mov` size variants, sign/zero + extension corners. + 5. Compare-and-branch, structured control flow. + 6. Aggregates, bitfields, varargs, atomics, intrinsics. +4. **x64 disassembler** in `arch/` — peer of `aa64_isa.{c,h}`. + Required for the textual disasm path used by some `test/cg/` + gates. +5. **Asm frontend.** Once x64 codegen lands, the asm frontend + (`parse_asm`) is the cheapest way to author instruction-sequence + tests without driving through the C frontend. Lands as a peer of + `src/parse/` consuming `MCEmitter` directly. + +Exit criterion: each x64 milestone owns a `test/cg/` case running +under both the aarch64 emulator and the podman/qemu x64 path; both +report green on the standard pass/fail line. + +--- + +## 5. Naming conventions + +For the new files and exposed symbols: + +- aarch64 → `aa64` prefix in code (`aa64_cgtarget_new`, `aa64_vtable`, + files under `arch/aa64*` and `abi/abi_aapcs64.c`). The existing + `aarch64.c` and `aa64_isa.{c,h}` already mix the two; keep `aa64` as + the going-forward convention. +- x86_64 → `x64` prefix (`x64_cgtarget_new`, files under `arch/x64*` + and `abi/abi_sysv_x64.c`). Avoid `x86_64` in identifiers — too long. +- ELF reloc translators stay arch-suffixed: + `elf_reloc_aarch64.c`, `elf_reloc_x86_64.c` (matches ELF spec + naming). + +--- + +## 6. Validation gates + +A change in this plan is "done" when: + +- Phase A/B/C: aarch64 test suite still green, byte-for-byte identical + output objects on a representative set of `test/cg/` cases. +- Phase D: a hand-rolled x86_64 ELF round-trips through the podman + runner with the right exit code. +- Phase E: each milestone owns a `test/cg/` case running under both + the aarch64 emulator and the podman runner; pass/fail line green + on both. diff --git a/test/cg/harness/cg_runner.c b/test/cg/harness/cg_runner.c @@ -338,6 +338,22 @@ static int mode_expected(const char* name) { return 0; } +/* --arches NAME — print one arch token per line for the named case. + * Used by test/cg/run.sh to decide which exec_target backend to dispatch + * path E through. Empty/zero arches in the registry mean CG_ARCH_DEFAULT + * (aarch64 today). */ +static int mode_arches(const char* name) { + const CgCase* cc = find_case(name); + if (!cc) { + fprintf(stderr, "cg-runner: unknown case '%s'\n", name); + return 2; + } + unsigned arches = cc->arches ? cc->arches : (unsigned)CG_ARCH_DEFAULT; + if (arches & CG_ARCH_AARCH64) fputs("aarch64\n", stdout); + if (arches & CG_ARCH_X64) fputs("x64\n", stdout); + return 0; +} + /* CfreeWriter that wraps stdout; used by --dump-tape. */ typedef struct StdoutWriter { CfreeWriter base; @@ -604,6 +620,7 @@ static int usage(void) { fprintf(stderr, "usage: cg-runner [--opt-level N] --list\n" " cg-runner [--opt-level N] --expected NAME\n" + " cg-runner [--opt-level N] --arches NAME\n" " cg-runner [--opt-level N] --dwarf-checks NAME\n" " cg-runner [--opt-level N] --emit NAME OUT.o\n" " cg-runner [--opt-level N] --jit NAME\n" @@ -627,6 +644,8 @@ int main(int argc, char** argv) { return mode_list(); else if (!strcmp(argv[1], "--expected") && argc == 3) return mode_expected(argv[2]); + else if (!strcmp(argv[1], "--arches") && argc == 3) + return mode_arches(argv[2]); else if (!strcmp(argv[1], "--dwarf-checks") && argc == 3) return mode_dwarf_checks(argv[2]); else if (!strcmp(argv[1], "--emit") && argc == 4) diff --git a/test/cg/harness/cg_test.h b/test/cg/harness/cg_test.h @@ -62,11 +62,23 @@ typedef enum { CG_CASE_MC_ONLY = 1, /* uses MCEmitter only — no CGTarget construction */ } CgCaseKind; +/* Per-case arch mask. Cases tagged with the arches they're known to run + * on; path E (exec) dispatches to the runner for whichever arches the + * case advertises. Today every case is aarch64-only — x86_64 cases + * arrive alongside x64 codegen in MULTIARCH phase 3. CG_ARCH_DEFAULT + * exists so the registry doesn't need a tag on every row. */ +enum { + CG_ARCH_AARCH64 = 1u << 0, + CG_ARCH_X64 = 1u << 1, + CG_ARCH_DEFAULT = CG_ARCH_AARCH64, +}; + typedef struct CgCase { const char* name; CgCaseFn build; - int expected; /* test_main return value (default 0) */ - unsigned kind; /* CgCaseKind */ + int expected; /* test_main return value (default 0) */ + unsigned kind; /* CgCaseKind */ + unsigned arches; /* CG_ARCH_* mask; 0 = CG_ARCH_DEFAULT */ } CgCase; extern const CgCase cg_cases[]; diff --git a/test/lib/exec_aarch64.sh b/test/lib/exec_aarch64.sh @@ -1,132 +0,0 @@ -# test/lib/exec_aarch64.sh — shared aarch64 exec helper for test harnesses. -# -# Sourced by test/{link,cg,parse}/run.sh. Provides two execution modes: -# -# exec_aarch64_run — synchronous one-shot. Used for kernel images and -# negative-test bad/ cases that need an immediate rc. -# exec_aarch64_queue — append to internal queue. -# exec_aarch64_flush — drain the queue. On podman hosts, single -# `podman run` loops over all queued exes inside -# the container, amortizing the ~150 ms per-launch -# podman client round-trip across the whole suite. -# On qemu hosts (or when no runner is available), -# falls back to a serial loop on the host. -# -# Caller contract: -# - Sets the following before sourcing or calling: have_qemu, have_podman, -# is_aarch64, QEMU_BIN. (Already detected by every harness.) -# - For batched podman, sets EXEC_AARCH64_MOUNT_ROOT to a host directory -# that contains *every* exe / out / err / rc path that will be queued. -# The same path is bind-mounted at the same path inside the container, -# so no path translation is needed in the manifest. -# - Optional: RUN_AARCH64_IMAGE overrides the container image -# (default alpine:latest, matching the prior inline implementation). - -# Internal queue arrays. Bash arrays are local to the sourcing shell. -EXEC_AARCH64_NAMES=() -EXEC_AARCH64_EXES=() -EXEC_AARCH64_OUTS=() -EXEC_AARCH64_ERRS=() -EXEC_AARCH64_RCS=() - -exec_aarch64_supported() { - [ "${have_qemu:-0}" -eq 1 ] || [ "${have_podman:-0}" -eq 1 ] -} - -# Synchronous run; sets RUN_RC. Mirrors the prior run_aarch64() in each harness. -exec_aarch64_run() { - local exe="$1" out="$2" err="$3" - if [ "${have_qemu:-0}" -eq 1 ]; then - "$QEMU_BIN" "$exe" >"$out" 2>"$err"; RUN_RC=$?; return - fi - if [ "${have_podman:-0}" -eq 1 ]; then - local dir base platform_flag=() - dir="$(cd "$(dirname "$exe")" && pwd)"; base="$(basename "$exe")" - # `--platform linux/arm64` triggers a registry manifest lookup - # (~30 s) even when the local image is already arm64. Only pass it - # on hosts where the podman machine isn't natively arm64. - [ "${is_aarch64:-0}" -eq 0 ] && platform_flag=(--platform linux/arm64) - podman run --rm "${platform_flag[@]}" --net=none \ - -v "$dir":/work:Z -w /work \ - "${RUN_AARCH64_IMAGE:-alpine:latest}" "./$base" \ - >"$out" 2>"$err" - RUN_RC=$?; return - fi - RUN_RC=127 -} - -# Queue an exe to run later. Stored verbatim; flush writes <rc_file> with -# the integer exit code, and routes stdout/stderr to <out_file>/<err_file>. -exec_aarch64_queue() { - EXEC_AARCH64_NAMES+=("$1") - EXEC_AARCH64_EXES+=("$2") - EXEC_AARCH64_OUTS+=("$3") - EXEC_AARCH64_ERRS+=("$4") - EXEC_AARCH64_RCS+=("$5") -} - -exec_aarch64_queue_size() { echo "${#EXEC_AARCH64_EXES[@]}"; } - -# Drain the queue. Reads back via the .rc files written into the bind-mounted -# tree; callers iterate their own bookkeeping arrays after this returns. -exec_aarch64_flush() { - local n="${#EXEC_AARCH64_EXES[@]}" - [ "$n" -eq 0 ] && return 0 - - if [ "${have_qemu:-0}" -eq 1 ]; then - local i=0 - while [ $i -lt "$n" ]; do - "$QEMU_BIN" "${EXEC_AARCH64_EXES[$i]}" \ - >"${EXEC_AARCH64_OUTS[$i]}" 2>"${EXEC_AARCH64_ERRS[$i]}" - echo $? >"${EXEC_AARCH64_RCS[$i]}" - i=$((i+1)) - done - elif [ "${have_podman:-0}" -eq 1 ]; then - if [ -z "${EXEC_AARCH64_MOUNT_ROOT:-}" ]; then - echo "exec_aarch64_flush: EXEC_AARCH64_MOUNT_ROOT must be set" >&2 - return 2 - fi - local platform_flag=() - [ "${is_aarch64:-0}" -eq 0 ] && platform_flag=(--platform linux/arm64) - local i=0 - # Manifest is fed via stdin; one tab-separated line per case. - # The in-container shell loop runs each exe and writes its rc to - # the bind-mounted .rc file, so the host can poll results after - # `podman run` returns. stdout/stderr from individual exes go to - # their .out/.err files inside the same mount. - { - while [ $i -lt "$n" ]; do - printf '%s\t%s\t%s\t%s\n' \ - "${EXEC_AARCH64_EXES[$i]}" \ - "${EXEC_AARCH64_OUTS[$i]}" \ - "${EXEC_AARCH64_ERRS[$i]}" \ - "${EXEC_AARCH64_RCS[$i]}" - i=$((i+1)) - done - } | podman run -i --rm "${platform_flag[@]}" --net=none \ - -v "$EXEC_AARCH64_MOUNT_ROOT":"$EXEC_AARCH64_MOUNT_ROOT":Z \ - "${RUN_AARCH64_IMAGE:-alpine:latest}" \ - /bin/sh -c ' -set -u -while IFS=" " read -r exe out err rc; do - "$exe" >"$out" 2>"$err" - echo $? >"$rc" -done -' - else - # No runner: mark each as 127, matching the prior fallback. - local i=0 - while [ $i -lt "$n" ]; do - : >"${EXEC_AARCH64_OUTS[$i]}" - : >"${EXEC_AARCH64_ERRS[$i]}" - echo 127 >"${EXEC_AARCH64_RCS[$i]}" - i=$((i+1)) - done - fi - - EXEC_AARCH64_NAMES=() - EXEC_AARCH64_EXES=() - EXEC_AARCH64_OUTS=() - EXEC_AARCH64_ERRS=() - EXEC_AARCH64_RCS=() -} diff --git a/test/lib/exec_target.sh b/test/lib/exec_target.sh @@ -0,0 +1,251 @@ +# test/lib/exec_target.sh — shared per-arch exec helper for test harnesses. +# +# Sourced by test/{link,cg,parse}/run.sh. Provides three execution modes, +# each parameterized by target arch: +# +# exec_target_run <arch> EXE OUT ERR +# Synchronous one-shot. Sets RUN_RC. Used for kernel images and +# negative-test cases that need an immediate rc. +# +# exec_target_queue <arch> NAME EXE OUT ERR RC +# Append a case to the internal queue. The arch tag is stored +# alongside so flush can group cases by arch and run one batched +# podman invocation per arch. +# +# exec_target_queue_size +# Total queue size across all arches. +# +# exec_target_flush +# Drain the queue. Cases are grouped by arch and each group runs +# through one `podman run` (or qemu serial loop). On podman hosts +# this amortizes the ~150 ms per-launch client round-trip across +# the whole suite. +# +# exec_target_supported <arch> +# Returns 0 if some runner is available for arch on this host. +# +# Recognized arches: aarch64, x64. Each maps to a podman --platform +# string and an optional user-mode qemu binary detected on the host. +# +# Caller contract: +# - Sets the following before sourcing or calling: have_qemu (host +# qemu-aarch64), have_podman, is_aarch64, QEMU_BIN (path to +# qemu-aarch64). Already detected by every harness. +# - For batched podman, sets EXEC_TARGET_MOUNT_ROOT to a host +# directory that contains every exe / out / err / rc path that +# will be queued. The same path is bind-mounted at the same path +# inside the container. +# - Optional: RUN_AARCH64_IMAGE / RUN_X64_IMAGE override the +# container image (default alpine:latest, matching the prior +# inline implementation). + +# Internal queue arrays. Each entry's arch is recorded alongside the +# rest so flush can split into per-arch batched runs. +EXEC_TARGET_ARCHES=() +EXEC_TARGET_NAMES=() +EXEC_TARGET_EXES=() +EXEC_TARGET_OUTS=() +EXEC_TARGET_ERRS=() +EXEC_TARGET_RCS=() + +# ---- per-arch capability/dispatch knobs ------------------------------------ +# +# _exec_target_platform <arch> → echoes podman `--platform` value. +# _exec_target_image <arch> → echoes container image to use. +# _exec_target_native <arch> → returns 0 if host can run arch natively +# without any container/qemu indirection. +# _exec_target_qemu <arch> → echoes a host qemu-user binary path if +# one is available, else empty. + +_exec_target_platform() { + case "$1" in + aarch64) echo "linux/arm64" ;; + x64) echo "linux/amd64" ;; + *) echo "" ;; + esac +} + +_exec_target_image() { + case "$1" in + aarch64) echo "${RUN_AARCH64_IMAGE:-alpine:latest}" ;; + x64) echo "${RUN_X64_IMAGE:-alpine:latest}" ;; + *) echo "alpine:latest" ;; + esac +} + +_exec_target_native() { + case "$1" in + aarch64) [ "${is_aarch64:-0}" -eq 1 ] ;; + x64) [ "${is_aarch64:-0}" -eq 0 ] && \ + { [ "$(uname -m 2>/dev/null)" = "x86_64" ] || \ + [ "$(uname -m 2>/dev/null)" = "amd64" ]; } ;; + *) return 1 ;; + esac +} + +_exec_target_qemu() { + case "$1" in + aarch64) [ "${have_qemu:-0}" -eq 1 ] && echo "${QEMU_BIN:-}" ;; + x64) # No qemu-user fallback for x64 in current harnesses. + echo "" ;; + *) echo "" ;; + esac +} + +exec_target_supported() { + local arch="$1" + _exec_target_native "$arch" && return 0 + [ -n "$(_exec_target_qemu "$arch")" ] && return 0 + [ "${have_podman:-0}" -eq 1 ] && return 0 + return 1 +} + +# Synchronous run; sets RUN_RC. +exec_target_run() { + local arch="$1" exe="$2" out="$3" err="$4" + local qemu + if _exec_target_native "$arch"; then + "$exe" >"$out" 2>"$err"; RUN_RC=$?; return + fi + qemu="$(_exec_target_qemu "$arch")" + if [ -n "$qemu" ]; then + "$qemu" "$exe" >"$out" 2>"$err"; RUN_RC=$?; return + fi + if [ "${have_podman:-0}" -eq 1 ]; then + local dir base platform image platform_flag=() + dir="$(cd "$(dirname "$exe")" && pwd)"; base="$(basename "$exe")" + platform="$(_exec_target_platform "$arch")" + image="$(_exec_target_image "$arch")" + # `--platform` triggers a registry manifest lookup (~30 s) even + # when the local image already matches. Only pass it on hosts + # that can't run the arch natively. + if ! _exec_target_native "$arch"; then + platform_flag=(--platform "$platform") + fi + podman run --rm "${platform_flag[@]}" --net=none \ + -v "$dir":/work:Z -w /work \ + "$image" "./$base" \ + >"$out" 2>"$err" + RUN_RC=$?; return + fi + RUN_RC=127 +} + +# Queue an exe to run later. Stored verbatim; flush writes <rc_file> with +# the integer exit code, and routes stdout/stderr to <out_file>/<err_file>. +exec_target_queue() { + EXEC_TARGET_ARCHES+=("$1") + EXEC_TARGET_NAMES+=("$2") + EXEC_TARGET_EXES+=("$3") + EXEC_TARGET_OUTS+=("$4") + EXEC_TARGET_ERRS+=("$5") + EXEC_TARGET_RCS+=("$6") +} + +exec_target_queue_size() { echo "${#EXEC_TARGET_EXES[@]}"; } + +# Internal: drain every entry whose arch matches $1, using qemu (if +# available for that arch), podman batched run, or the no-runner stub. +_exec_target_flush_arch() { + local arch="$1" + local idx=() + local i=0 n="${#EXEC_TARGET_EXES[@]}" + while [ $i -lt "$n" ]; do + [ "${EXEC_TARGET_ARCHES[$i]}" = "$arch" ] && idx+=("$i") + i=$((i+1)) + done + [ "${#idx[@]}" -eq 0 ] && return 0 + + local qemu; qemu="$(_exec_target_qemu "$arch")" + local k + if _exec_target_native "$arch"; then + for k in "${idx[@]}"; do + "${EXEC_TARGET_EXES[$k]}" \ + >"${EXEC_TARGET_OUTS[$k]}" 2>"${EXEC_TARGET_ERRS[$k]}" + echo $? >"${EXEC_TARGET_RCS[$k]}" + done + return 0 + fi + if [ -n "$qemu" ]; then + for k in "${idx[@]}"; do + "$qemu" "${EXEC_TARGET_EXES[$k]}" \ + >"${EXEC_TARGET_OUTS[$k]}" 2>"${EXEC_TARGET_ERRS[$k]}" + echo $? >"${EXEC_TARGET_RCS[$k]}" + done + return 0 + fi + if [ "${have_podman:-0}" -eq 1 ]; then + if [ -z "${EXEC_TARGET_MOUNT_ROOT:-}" ]; then + echo "exec_target_flush: EXEC_TARGET_MOUNT_ROOT must be set" >&2 + return 2 + fi + local platform image platform_flag=() + platform="$(_exec_target_platform "$arch")" + image="$(_exec_target_image "$arch")" + if ! _exec_target_native "$arch"; then + platform_flag=(--platform "$platform") + fi + # Manifest is fed via stdin; one tab-separated line per case. + # The in-container shell loop runs each exe and writes its rc + # to the bind-mounted .rc file, so the host can poll results + # after `podman run` returns. stdout/stderr from individual + # exes go to their .out/.err files inside the same mount. + { + for k in "${idx[@]}"; do + printf '%s\t%s\t%s\t%s\n' \ + "${EXEC_TARGET_EXES[$k]}" \ + "${EXEC_TARGET_OUTS[$k]}" \ + "${EXEC_TARGET_ERRS[$k]}" \ + "${EXEC_TARGET_RCS[$k]}" + done + } | podman run -i --rm "${platform_flag[@]}" --net=none \ + -v "$EXEC_TARGET_MOUNT_ROOT":"$EXEC_TARGET_MOUNT_ROOT":Z \ + "$image" \ + /bin/sh -c ' +set -u +while IFS=" " read -r exe out err rc; do + "$exe" >"$out" 2>"$err" + echo $? >"$rc" +done +' + return 0 + fi + # No runner: mark each as 127, matching the prior fallback. + for k in "${idx[@]}"; do + : >"${EXEC_TARGET_OUTS[$k]}" + : >"${EXEC_TARGET_ERRS[$k]}" + echo 127 >"${EXEC_TARGET_RCS[$k]}" + done +} + +# Drain the queue. Reads back via the .rc files written into the +# bind-mounted tree; callers iterate their own bookkeeping arrays after +# this returns. Each arch present in the queue runs in its own batch. +exec_target_flush() { + [ "${#EXEC_TARGET_EXES[@]}" -eq 0 ] && return 0 + + # Distinct arches in queue order. Bash 3.2 has no associative arrays; + # use a small linear scan. + local seen=() a present k + local i=0 n="${#EXEC_TARGET_ARCHES[@]}" + while [ $i -lt "$n" ]; do + a="${EXEC_TARGET_ARCHES[$i]}" + present=0 + for k in "${seen[@]:-}"; do [ "$k" = "$a" ] && present=1 && break; done + [ "$present" -eq 0 ] && seen+=("$a") + i=$((i+1)) + done + + local rc=0 + for a in "${seen[@]}"; do + _exec_target_flush_arch "$a" || rc=$? + done + + EXEC_TARGET_ARCHES=() + EXEC_TARGET_NAMES=() + EXEC_TARGET_EXES=() + EXEC_TARGET_OUTS=() + EXEC_TARGET_ERRS=() + EXEC_TARGET_RCS=() + return $rc +} diff --git a/test/link/run.sh b/test/link/run.sh @@ -118,12 +118,13 @@ arch_raw="$(uname -m 2>/dev/null || true)" READELF_BIN="$(command -v llvm-readelf 2>/dev/null || command -v readelf 2>/dev/null || true)" -# Shared aarch64 exec helper. Path E queues each linked.exe and we drain -# all cases in a single `podman run` after the main loop — amortizes the -# ~150 ms per-launch podman client overhead across the whole suite. -EXEC_AARCH64_MOUNT_ROOT="$BUILD_DIR" -# shellcheck source=../lib/exec_aarch64.sh -source "$ROOT/test/lib/exec_aarch64.sh" +# Shared per-arch exec helper. Path E queues each linked.exe and we +# drain all cases in a single `podman run` per arch after the main +# loop — amortizes the ~150 ms per-launch podman client overhead +# across the whole suite. +EXEC_TARGET_MOUNT_ROOT="$BUILD_DIR" +# shellcheck source=../lib/exec_target.sh +source "$ROOT/test/lib/exec_target.sh" # ---- locate harness binaries ------------------------------------------------ # The Makefile's `test-link` target builds these as proper Make targets so @@ -402,7 +403,7 @@ for case_dir in "$TEST_DIR/cases"/*/; do for s in "${gc_present_syms[@]:-}"; do [ -n "$s" ] && gcp="${gcp}${s}"$'\n'; done E_GC_ABSENT_LIST+=("$gca") E_GC_PRESENT_LIST+=("$gcp") - exec_aarch64_queue "$name" "$exe" \ + exec_target_queue aarch64 "$name" "$exe" \ "$work/exec.out" "$work/exec.err" "$work/exec.rc" else note_skip "$name/E" "no runner (qemu/podman)" @@ -551,10 +552,10 @@ done # already printed inline above; E results print together here. T_E_BATCH=0 -if [ "$(exec_aarch64_queue_size)" -gt 0 ]; then - printf 'Running path E (%d cases batched)...\n' "$(exec_aarch64_queue_size)" +if [ "$(exec_target_queue_size)" -gt 0 ]; then + printf 'Running path E (%d cases batched)...\n' "$(exec_target_queue_size)" t0=$(now_ms) - exec_aarch64_flush + exec_target_flush T_E_BATCH=$(( $(now_ms) - t0 )); T_E=$(( T_E + T_E_BATCH )) i=0 diff --git a/test/parse/run.sh b/test/parse/run.sh @@ -98,10 +98,10 @@ arch_raw="$(uname -m 2>/dev/null || true)" READELF_BIN="$(command -v llvm-readelf 2>/dev/null || command -v readelf 2>/dev/null || true)" -# Shared aarch64 exec helper — see test/lib/exec_aarch64.sh. -EXEC_AARCH64_MOUNT_ROOT="$BUILD_DIR" -# shellcheck source=../lib/exec_aarch64.sh -source "$ROOT/test/lib/exec_aarch64.sh" +# Shared per-arch exec helper — see test/lib/exec_target.sh. +EXEC_TARGET_MOUNT_ROOT="$BUILD_DIR" +# shellcheck source=../lib/exec_target.sh +source "$ROOT/test/lib/exec_target.sh" # ---- build harness binaries ------------------------------------------------ @@ -271,16 +271,16 @@ for src in "${CASES[@]}"; do >"$work/exec_link.out" 2>"$work/exec_link.err"; then dt=$(( $(now_ms) - t0 )); T_E=$(( T_E + dt )) note_fail "$name/E (link failed, ${dt}ms)" - elif [ $have_runner -eq 1 ]; then + elif exec_target_supported aarch64; then link_dt=$(( $(now_ms) - t0 )); T_E=$(( T_E + link_dt )) E_NAMES+=("$name") E_WORK+=("$work") E_LINK_MS+=("$link_dt") E_EXPECTED+=("$expected_byte") - exec_aarch64_queue "$name" "$exe" \ + exec_target_queue aarch64 "$name" "$exe" \ "$work/exec.out" "$work/exec.err" "$work/exec.rc" else - note_skip "$name/E" "no qemu/podman" + note_skip "$name/E" "no runner for aarch64" fi else note_skip "$name/E" "no link-exe-runner, aarch64 clang, or start.o" @@ -308,10 +308,10 @@ done # ---- batched path-E flush + verification ----------------------------------- T_E_BATCH=0 -if [ "$(exec_aarch64_queue_size)" -gt 0 ]; then - printf 'Running path E (%d cases batched)...\n' "$(exec_aarch64_queue_size)" +if [ "$(exec_target_queue_size)" -gt 0 ]; then + printf 'Running path E (%d cases batched)...\n' "$(exec_target_queue_size)" t0=$(now_ms) - exec_aarch64_flush + exec_target_flush T_E_BATCH=$(( $(now_ms) - t0 )); T_E=$(( T_E + T_E_BATCH )) i=0 diff --git a/test/smoke/x64.sh b/test/smoke/x64.sh @@ -0,0 +1,136 @@ +#!/usr/bin/env bash +# test/smoke/x64.sh — end-to-end smoke test for the x64 podman/qemu path. +# +# Phase-1 of doc/MULTIARCH.md: prove the test/lib/exec_target.sh helper +# can build, queue, and run an x86_64-linux ELF before any cfree-emitted +# x64 bytes exist. Builds a tiny freestanding static executable with +# clang --target=x86_64-linux-gnu and pushes it through +# exec_target_run / exec_target_queue+flush, asserting the expected +# exit code on both paths. +# +# Skipped if clang lacks the x86_64-linux-gnu target or no runner +# (podman or qemu-x86_64) is available. Mirrors test/cg's skip-vs-fail +# convention: skip is treated as failure unless CFREE_TEST_ALLOW_SKIP=1. + +set -u + +ROOT="$(cd "$(dirname "$0")/../.." && pwd)" +BUILD_DIR="$ROOT/build/test/smoke-x64" +mkdir -p "$BUILD_DIR" + +color_red() { printf '\033[31m%s\033[0m' "$1"; } +color_grn() { printf '\033[32m%s\033[0m' "$1"; } +color_yel() { printf '\033[33m%s\033[0m' "$1"; } + +ALLOW_SKIP="${CFREE_TEST_ALLOW_SKIP:-0}" + +# ---- detect prerequisites -------------------------------------------------- + +CLANG_TARGET="--target=x86_64-linux-gnu" +have_clang_x64=0 +if clang $CLANG_TARGET -c -x c - -o /dev/null < /dev/null 2>/dev/null; then + have_clang_x64=1 +fi + +# Cross-link wants an ELF-aware ld. On macOS the host /usr/bin/ld is +# Mach-O only; insist on lld. On a Linux x86_64 host the default host +# linker is fine but lld still works. +have_lld=0 +command -v ld.lld >/dev/null 2>&1 && have_lld=1 + +# Variables expected by exec_target.sh. The aarch64 helper expects these +# names regardless of target arch — they describe the host detection +# rather than the target. For x64-only smoke we don't need QEMU_BIN. +have_qemu=0 +QEMU_BIN="" +have_podman=0 +command -v podman >/dev/null 2>&1 && have_podman=1 +arch_raw="$(uname -m 2>/dev/null || true)" +is_aarch64=0 +{ [ "$arch_raw" = "aarch64" ] || [ "$arch_raw" = "arm64" ]; } && is_aarch64=1 +export have_qemu QEMU_BIN have_podman is_aarch64 + +EXEC_TARGET_MOUNT_ROOT="$BUILD_DIR" +# shellcheck source=../lib/exec_target.sh +source "$ROOT/test/lib/exec_target.sh" + +PASS=0; FAIL=0; SKIP=0 +note_pass() { PASS=$((PASS+1)); printf ' %s %s\n' "$(color_grn PASS)" "$1"; } +note_fail() { FAIL=$((FAIL+1)); printf ' %s %s\n' "$(color_red FAIL)" "$1"; } +note_skip() { SKIP=$((SKIP+1)); printf ' %s %s — %s\n' "$(color_yel SKIP)" "$1" "$2"; } + +if [ $have_clang_x64 -eq 0 ]; then + note_skip "build" "clang --target=x86_64-linux-gnu unavailable" + printf '\nResults: %s pass, %s fail, %s skip\n' "$PASS" "$FAIL" "$SKIP" + if [ "$ALLOW_SKIP" = "1" ]; then exit 0; fi + exit 1 +fi +if [ $have_lld -eq 0 ]; then + note_skip "build" "ld.lld unavailable (needed for ELF cross-link)" + printf '\nResults: %s pass, %s fail, %s skip\n' "$PASS" "$FAIL" "$SKIP" + if [ "$ALLOW_SKIP" = "1" ]; then exit 0; fi + exit 1 +fi +if ! exec_target_supported x64; then + note_skip "exec" "no runner for x64 (podman or qemu-x86_64)" + printf '\nResults: %s pass, %s fail, %s skip\n' "$PASS" "$FAIL" "$SKIP" + if [ "$ALLOW_SKIP" = "1" ]; then exit 0; fi + exit 1 +fi + +# ---- build a tiny freestanding x86_64 ELF ----------------------------------- + +# Direct syscall in _start: SYS_exit_group on x86_64 is 231, exit code +# 42. No libc, no relocations, no PIE. The point is to exercise the +# harness pipeline (clang cross-compile → podman/qemu run → recorded +# rc), not to build a complete program. +SRC="$BUILD_DIR/smoke.c" +cat >"$SRC" <<'EOF' +__attribute__((noreturn)) void _start(void) { + register long rax __asm__("rax") = 231; /* sys_exit_group */ + register long rdi __asm__("rdi") = 42; + __asm__ volatile("syscall" : : "r"(rax), "r"(rdi) : "memory"); + __builtin_unreachable(); +} +EOF + +EXE="$BUILD_DIR/smoke.exe" +if ! clang $CLANG_TARGET -fuse-ld=lld \ + -O1 -ffreestanding -fno-stack-protector \ + -fno-PIC -fno-pie -nostdlib -static \ + -Wl,-e,_start \ + "$SRC" -o "$EXE" 2>"$BUILD_DIR/build.err"; then + note_fail "build (see $BUILD_DIR/build.err)" + printf '\nResults: %s pass, %s fail, %s skip\n' "$PASS" "$FAIL" "$SKIP" + exit 1 +fi + +# ---- exec_target_run --------------------------------------------------------- + +exec_target_run x64 "$EXE" "$BUILD_DIR/run.out" "$BUILD_DIR/run.err" +if [ "$RUN_RC" -eq 42 ]; then + note_pass "exec_target_run x64 (rc=42)" +else + note_fail "exec_target_run x64 (expected 42 got $RUN_RC; see $BUILD_DIR/run.err)" +fi + +# ---- exec_target_queue + flush ---------------------------------------------- + +exec_target_queue x64 smoke "$EXE" \ + "$BUILD_DIR/q.out" "$BUILD_DIR/q.err" "$BUILD_DIR/q.rc" +exec_target_flush +if [ ! -f "$BUILD_DIR/q.rc" ]; then + note_fail "exec_target_flush x64 (no rc file produced)" +else + Q_RC="$(cat "$BUILD_DIR/q.rc")" + if [ "$Q_RC" -eq 42 ]; then + note_pass "exec_target_queue+flush x64 (rc=42)" + else + note_fail "exec_target_queue+flush x64 (expected 42 got $Q_RC; see $BUILD_DIR/q.err)" + fi +fi + +printf '\nResults: %s pass, %s fail, %s skip\n' "$PASS" "$FAIL" "$SKIP" +if [ $FAIL -gt 0 ]; then exit 1; fi +if [ $SKIP -gt 0 ] && [ "$ALLOW_SKIP" != "1" ]; then exit 1; fi +exit 0 diff --git a/test/test.mk b/test/test.mk @@ -24,7 +24,7 @@ # against the public cfree.h surface; reuses cfree-roundtrip, # link-exe-runner, and jit-runner. -.PHONY: test test-lex test-pp test-pp-err test-elf test-ar test-ar-driver test-link test-cg test-dwarf test-debug test-parse test-parse-err test-musl test-glibc test-lib-deps +.PHONY: test test-lex test-pp test-pp-err test-elf test-ar test-ar-driver test-link test-cg test-dwarf test-debug test-parse test-parse-err test-musl test-glibc test-lib-deps test-smoke-x64 test: test-lex test-pp test-pp-err test-elf test-ar test-ar-driver test-link test-cg test-dwarf test-debug test-parse test-parse-err test-lib-deps @@ -130,6 +130,15 @@ test-parse: lib $(PARSE_RUNNER) $(ROUNDTRIP_BIN) $(LINK_EXE_RUNNER) $(JIT_RUNNER test-parse-err: lib $(PARSE_RUNNER) sh test/parse/run_errors.sh +# test-smoke-x64: phase-1 sanity check for doc/MULTIARCH.md. Builds a +# tiny freestanding x86_64 ELF with clang --target=x86_64-linux-gnu and +# runs it through test/lib/exec_target.sh's podman/qemu pipeline, +# proving the harness end-to-end before any cfree-emitted x64 bytes +# exist. Excluded from the default `test` target because it needs +# podman + lld; opt-in via `make test-smoke-x64`. +test-smoke-x64: + bash test/smoke/x64.sh + # test-musl / test-glibc: end-to-end static + dynamic libc link/run on # aarch64. Each variant pulls its own pinned sysroot (podman, ~30s on # first run) and shares the same case files under test/libc/cases/: