commit 246df83a668460fbe7e2b598a23614ebe68c9c11
parent ad732e4b9733ddb81da3f861b595c7e1e6bd1388
Author: Ryan Sepassi <rsepassi@gmail.com>
Date: Mon, 8 Jun 2026 17:18:13 -0700
doc/bootstrap: record aarch64-linux and aarch64-freebsd bootstrap state
- aarch64-linux (musl + glibc) baseline: both -O0/-O1 reach the fixed point,
via the Linux-container path; notes the JIT-TLS / objdump-golden / C-backend
Toy gaps that are not bootstrap blockers.
- aarch64-freebsd: -O0 chain reaches the byte-identical fixed point (Toy
1371/9/39); needed -rdynamic acceptance + ELF Verneed/Versym emission (the
fstat INO64 FBSD_1.0-vs-1.5 mis-bind). The -O1 release chain is blocked by a
pre-existing FreeBSD-target deferred-const-data GLOBAL-binding bug
(.Lkit_ro/.Lkit_jt); gate on bootstrap-debug until fixed.
Diffstat:
| M | doc/plan/BOOTSTRAP.md | | | 127 | ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++------- |
1 file changed, 117 insertions(+), 10 deletions(-)
diff --git a/doc/plan/BOOTSTRAP.md b/doc/plan/BOOTSTRAP.md
@@ -45,24 +45,131 @@ Done (baseline) on aarch64-macos:
8 skip) across the run, link/native, C-backend, and Wasm paths at Toy opt
levels 0 and 1.
-This gives one fully self-hosting configuration. The remaining work is breadth:
-other targets, other host platforms, and guarding the property over time.
+Done on aarch64-linux (ELF), run natively inside an arm64 Linux container from
+the macOS host (see "Bootstrapping a Linux target from a non-Linux host" below):
+
+- **musl** (alpine): both the `-O0` and `-O1` chains reach the fixed point;
+ `cmp stage2/kit stage3/kit` is clean and the per-object check across all 321
+ `*.o`/`*.a` in stage2 vs stage3 reports zero differences in both modes. The
+ bootstrapped stage3 runs the Toy corpus at 1365 pass / 15 fail / 39 skip; the
+ 15 failures are not bootstrap/codegen issues (per-object is byte-identical) —
+ they are Mach-O-tuned `.objdump` golden substrings that differ on ELF, emitted
+ C the container's host clang rejects under `-Werror`, and one Linux JIT-TLS
+ `.tdata`-init discrepancy (see below).
+- **glibc** (debian): reaching the fixed point required a series of kit
+ C-frontend / preprocessor compatibility fixes for the glibc + Linux-UAPI
+ header set (musl's ISO-C headers never exercised them): erase `__extension__`
+ on all targets, map `__signed__`/`__volatile__`/`__const__` to the canonical
+ keywords, and support GNU named variadic macro parameters (`args...`) in the
+ preprocessor.
+
+Done on aarch64-freebsd (ELF), run natively inside the FreeBSD aarch64 VM from
+the macOS host (`scripts/freebsd_bootstrap.sh aarch64`; see "Bootstrapping a
+Linux target from a non-Linux host" — the FreeBSD VM path is the same shape):
+
+- **`-O0` (debug) chain reaches the fixed point**: `cmp stage2/kit stage3/kit`
+ is byte-identical. The bootstrapped stage3 runs the Toy corpus at
+ 1371 pass / 9 fail / 39 skip — the 9 failures are the same non-bootstrap gaps
+ seen on aarch64-linux (Mach-O-tuned `.objdump` golden substrings on ELF, the
+ JIT-TLS `.tdata`-init `R`-lane discrepancy, and one C-backend case the host
+ clang rejects), not codegen issues. Reaching the fixed point required:
+ - kit `cc` accepting `-rdynamic` (FreeBSD's `HOST_ENV_LDFLAGS` passes it; the
+ other ELF hosts do not), and
+ - **ELF symbol-version (Verneed/Versym) emission** in the linker. FreeBSD's
+ INO64 transition left `stat`/`fstat`/... as two incompatible `struct stat`
+ ABIs behind a hidden `FBSD_1.0` (compat) and the default `FBSD_1.5`; kit
+ used to emit unversioned undefined references, so the runtime bound the
+ compat version and read `st_size` at the wrong offset — stage2 then failed
+ to read its own source files. The linker now reads each DSO's
+ `.gnu.version_d`/`.gnu.version` and emits a matching `.gnu.version_r` +
+ `.gnu.version` (gated on the DSO carrying versions, so musl/static links are
+ unchanged; glibc links now also carry correct `GLIBC_*` requirements).
+- **`-O1` (release) chain does NOT yet reach the fixed point.** It is blocked
+ *before* the fixed-point check by a pre-existing, FreeBSD-target-specific
+ codegen bug, unrelated to symbol versioning: the `-O1` *deferred* anonymous
+ const-data path (`api_const_data_can_defer` →
+ `local_static_data_*`) emits the `.Lkit_ro.N` (and sibling `.Lkit_jt.N`)
+ symbols with **GLOBAL** instead of LOCAL binding for the FreeBSD target. The
+ four hosted `driver/env/*.o` adapters then each define a global `.Lkit_ro.0`,
+ and the stage2 `kit` link aborts with `duplicate definition of global symbol
+ '.Lkit_ro.0'`. The identical source at `-O1` emits these symbols LOCAL for
+ aarch64-linux and aarch64-macos, so the bug is in the FreeBSD-target deferred
+ const-data emission, not the link/versioning work. This is the `.Lkit_jt.0`
+ release-bootstrap break tracked elsewhere; gate on `bootstrap-debug` until it
+ is fixed.
+
+This gives three fully self-hosting configurations (aarch64-macos, plus
+aarch64-linux under musl and glibc) and a fourth at `-O0` (aarch64-freebsd).
+The remaining work is breadth: the other native targets, the aarch64-freebsd
+`-O1` const-data binding fix, and guarding the property over time.
## Open problems and next steps
### Widen target and platform coverage
-The fixed point is currently demonstrated only for aarch64-macos. The bootstrap
-should hold for every supported native target and object format. Until each is
-green it is an open question whether its backend + object writer are fully
-deterministic and self-consistent.
+The fixed point holds for aarch64-macos and aarch64-linux (musl + glibc). The
+bootstrap should hold for every supported native target and object format. Until
+each is green it is an open question whether its backend + object writer are
+fully deterministic and self-consistent.
- [ ] Reach the fixed point on x86-64 (ELF and Mach-O) for both `-O0` and `-O1`.
- [ ] Reach the fixed point on rv64 (ELF) for both `-O0` and `-O1`.
-- [ ] Reach the fixed point on aarch64-linux (ELF), distinct from the macOS
- Mach-O path already covered.
-- [ ] For each new configuration, run the per-object diff and the Toy corpus
- through the bootstrapped compiler, not just the final `cmp`.
+- [x] Reach the fixed point on aarch64-linux (ELF), distinct from the macOS
+ Mach-O path already covered. Done for musl and glibc; the aarch64-linux
+ backend + ELF writer are confirmed deterministic and self-consistent.
+- [x] For each new configuration, run the per-object diff and the Toy corpus
+ through the bootstrapped compiler, not just the final `cmp`. Done for
+ aarch64-linux (321/321 objects identical; Toy 1365/15/39).
+
+### Bootstrapping a Linux target from a non-Linux host
+
+`make bootstrap` keys off the build host's own `uname` (`HOST_OS` + machine), so
+it selects the native toolchain and object format with no cross-compilation. To
+bootstrap aarch64-linux from the macOS dev host, run the normal three-stage
+build *inside* an arm64 Linux container, where it is an ordinary native build:
+
+- `scripts/linux_bootstrap.sh [musl|glibc] [both|debug|release]` drives a podman
+ container (alpine for musl, debian for glibc — the same image families the
+ hosted test suite uses), provisions a seed clang + make + libc headers, and
+ runs `make bootstrap` with the stage tree under `build/linux-boot/<libc>/`.
+ `KIT_LINUX_BOOT_TOY=1` additionally runs the Toy corpus through stage3.
+- `make bootstrap-linux` (→ `-musl`) / `make bootstrap-linux-glibc` wrap it.
+
+Three host-environment differences from the macOS reference, all handled by the
+script / `mk/bootstrap.mk`:
+
+- **LeakSanitizer.** The `-O0` chain builds stage1 with ASan+UBSan as on macOS,
+ but LSan (unsupported on Darwin, so never run there) flags kit's arena
+ allocator — which deliberately never frees — and aborts every stage1 `cc`. The
+ container sets `ASAN_OPTIONS=…:detect_leaks=0`, the honest equivalent of
+ Darwin's behavior.
+- **`-lc` for the kit-compiled stages.** On macOS kit gets its system headers via
+ the `-isysroot` in `HOST_SYSROOT_CFLAGS`; on Linux that is empty and kit's
+ hosted profile only wires up the libc include + library dirs once libc is
+ requested, so `mk/bootstrap.mk` passes `HOST_SYSROOT_{C,LD}FLAGS=-lc` to the
+ stage2/3 sub-makes (Linux/FreeBSD only).
+- **glibc header compatibility.** glibc + Linux-UAPI headers use GCC-isms musl's
+ cleaner headers don't; reaching the glibc fixed point needed the C-frontend /
+ pp fixes listed in the baseline above.
+
+Non-bootstrap gaps surfaced by the aarch64-linux Toy run (tracked, not
+fixed-point blockers — the per-object diff is byte-identical):
+
+- **JIT-TLS `.tdata` initializer.** `141_threadlocal_mutate` returns 3 instead
+ of 43 on the R (in-process JIT) lane: the JIT zero-initializes the
+ thread-local block instead of copying its `.tdata` initializer (40). Native
+ link of this case is already `.link.skip`-gated; the JIT path is the gap.
+- **`.objdump` golden substrings** for a few cases (`122_data_entsize`,
+ `127_switch_forced_jump_table`, `62_decl_data_attrs`) encode Mach-O section /
+ entsize spellings and need ELF-flavored sidecars.
+- **C-backend emitted source** for ~7 cases is rejected by the container's host
+ clang under `-Wall -Wextra -Werror` (host-toolchain strictness, varies by
+ clang version).
+
+The native-ELF Toy `L` lane links hosted (`kit cc -lc`, so the crt provides
+`_start`) rather than freestanding `kit ld`, because an ELF executable needs a
+crt entry where Mach-O drives `LC_MAIN` straight to `main`; `test/toy/run.sh`
+selects this automatically on non-Darwin hosts (`KIT_TOY_L_HOSTED`).
These connect to the per-arch backend state tracked in [../CODEGEN.md](../CODEGEN.md)
and [../ARCH.md](../ARCH.md), and to the object/format paths in
