FreeBSD target support

Status and roadmap for compiling, linking, and running FreeBSD binaries with kit, plus the QEMU VM harness used to execute them. Scope is the release support set: arm64 (aarch64), x64 (amd64), rv64 (riscv64) on FreeBSD.

The execution environment (the VMs) is in good shape; the compile/link path is the gating work — FreeBSD cross-compile was previously untested and turns out to need several fixes, most already landed, with one real linker blocker left.

TL;DR

• VM harness: scripts/freebsd_vm.sh. amd64 + aarch64 + rv64 all boot and are SSH-reachable as durable, cached "golden" disks. This is the way to execute FreeBSD binaries on all three arches.
• Compile path: target parsing, runtime variants, COMDAT-group reading, and the FreeBSD-15 libsys split are handled. Remaining blocker: the libc/libsys weak-alias archive cycle (undefined reference to 'openat').

Execution environment — scripts/freebsd_vm.sh

Orchestrated from a macOS/arm64 host with Homebrew QEMU. One command set provides download, provision, run, and SSH for each arch.

Provisioning model (provision once, cache forever)

A first boot does the expensive one-time setup; the result is saved as a compressed golden disk in the download cache ($DL_ROOT, which survives make clean). Later prepare/run restore it in seconds.

• prepare <arch> → ensure_disk: use an existing provisioned disk, else restore the cached golden disk, else expand the pristine image + provision + cache.
• run <arch> boots the golden disk with full networking and host-only SSH forwarding.
• provision <arch> forces a (re)provision.

Two provisioning paths:

• Cloud-init arches (amd64, aarch64): provision_nuageinit boots the BASIC-CLOUDINIT image offline (-netdev user,restrict=on). nuageinit (FreeBSD's native cloud-init) creates the kit user, imports the SSH key, and enables sshd; the firstboot_freebsd_update service fails fast for lack of a route ("mirrors... none found") instead of running a slow networked update and forcing a reboot. The script then SSHes a clean shutdown -p now and caches.
• rv64 (no cloud-init image published): provision_serial drives the serial console with expect: log in as root, pw useradd kit, install the key, sysrc sshd_enable=YES + ifconfig_vtnet0=DHCP, service sshd keygen, power off. Same golden-cache result.

Per-arch boot quirks (all fixed in the script)

• amd64: runs under TCG (no hardware accel on an arm64 host). Two traps, both fixed:
  • The persisted EDK2 NVRAM vars drift so the firmware boots its built-in UEFI Shell instead of the disk — looks exactly like a hang (CPU idle at Shell>). ensure_firmware_vars now resets the vars from the pristine template every run.
  • q35 makes the headless VGA the primary console, so all userland/cloud-init output and the login getty are invisible. amd64 runs -vga none → serial is primary.
• aarch64: -machine virt, HVF-accelerated (fast); serial is already the primary console. The reference path.
• rv64: runs under TCG. Boots via OpenSBI → edk2-riscv (UEFI) → FreeBSD loader. Must pass -machine virt,acpi=off: FreeBSD/riscv64 is FDT-only, and with ACPI advertised the loader reports "no valid device tree blob" and the kernel boots blind. With acpi=off it gets ofwbus0 <Open Firmware Device Tree> and mounts root normally.

Knobs

KIT_FREEBSD_RELEASE (default 15.0-RELEASE), KIT_FREEBSD_MEM, KIT_FREEBSD_CPUS, KIT_FREEBSD_PROVISION_TIMEOUT, KIT_FREEBSD_ACCEL, KIT_FREEBSD_SSH_USER, KIT_FREEBSD_SSH_KEY. See freebsd_vm.sh doctor.

Sysroot extraction

There is no committed FreeBSD sysroot; extract one per arch from a running VM (it is the matching base system). Minimal static-link set:

bash scripts/freebsd_vm.sh ssh <arch> \
  'tar cf - -C / usr/include usr/lib/crt1.o usr/lib/crti.o usr/lib/crtn.o \
      usr/lib/libc.a usr/lib/libsys.a usr/lib/libssp_nonshared.a' \
  | tar xf - -C build/freebsd-sysroot/<arch>

A proper base.txz extraction harness paralleling test/libc/{glibc,musl} is the intended durable mechanism (pin the dist, record the version).

Compile / link path

kit cc --target=<arch>-freebsdN.N --sysroot=<root> [-static] file.c. Driven through the hosted FreeBSD profile in driver/lib/hosted.c (hosted_resolve_freebsd): crt1/Scrt1 + crti + crtn, libc.a (static) or libc.so.7 (dynamic), interp /libexec/ld-elf.so.1, and __FreeBSD__/__ELF__ defines.

Fixed

• Triple parsing (driver/lib/target.c). freebsd was not recognized at all — --target=*-freebsd silently fell through to freestanding. Added prefix matching so versioned tokens (freebsd15.0, freebsd14) parse to KIT_OS_FREEBSD / ELF.
• Runtime variants (driver/lib/runtime.c). Added {x86_64,aarch64,riscv64}-freebsd to kRtVariants (ELF, mirroring the Linux ELF runtime). Without them, cc aborts with "compiler runtime is not available for this target". The driver builds the archive on demand.
• ELF COMDAT groups on read (src/obj/elf/read.c, elf.h). FreeBSD's crt1.o brands the binary by placing its .note.tag ABI note in an ELF COMDAT group whose signature symbol is .freebsd.note*. kit consumes the SHT_GROUP section into an ObjGroup and never keeps it as a real section, which orphaned the signature symbol into a phantom "undefined reference". Fixed: a symbol defined in an SHT_GROUP section is recorded as an absolute defined symbol (it names a group, is never a reloc target).
• FreeBSD 15 libsys split (driver/lib/hosted.c). FreeBSD 15 moved the raw syscall stubs out of libc into libsys. The hosted profile now links libsys.{a,so.7} after libc when the sysroot provides it (pre-15 roots lack it, so it is conditional).
• STB_GNU_UNIQUE binding (src/obj/elf/read.c). Tangential but real: GNU-unique was mapped to SB_LOCAL, hiding such globals from cross-object resolution. Now mapped to global.

Blocker — libc/libsys weak-alias archive cycle

-static links currently fail with undefined reference to 'openat' (and would hit the same class for other syscall wrappers).

Root cause: FreeBSD 15 splits the syscall path across two mutually-recursive archives with weak aliases:

• openat (public) is a weak alias → _openat → __sys_openat.
• libc.a references __sys_openat; libsys.a provides __sys_openat and references back into libc; openat/_openat live in libc.a.

kit's link_ingest_archives (src/link/link_resolve.c) resolves each archive to a fixpoint, but only against the symbols defined before it in link order — it does not re-scan an earlier archive when a later one introduces new undefined references. So a back-reference from libsys.a into libc.a is not satisfied. Re-listing libc.a after libsys.a (the --start-group idiom; currently in the hosted profile) did not resolve it on its own, so weak-definition archive-pull semantics are likely also involved and need confirmation.

Proposed fix (linker work, regression-sensitive — guard with the existing test-link corpus):

• Add archive group semantics: re-scan a set of archives to a fixpoint so cross-archive cycles resolve (--start-group/--end-group, or a global whole-set fixpoint for the hosted-profile archives).
• Confirm/fix weak-definition archive-pull: a strong undefined reference must pull an archive member that defines the symbol weakly (the openat alias case). Verify against test/link archive-demand cases.
• Re-validate musl/glibc static links (hosted_resolve_linux_*) — they share the same ingestion path.

Other known gaps

• __FreeBSD__ version is hardcoded to 14 in hosted_add_freebsd_defines; the VMs are 15.0. Now that the triple carries the version (triple_tok_prefix), thread an OS-version field through KitTargetSpec and derive it.
• Dynamic links are unvalidated: PT_INTERP /libexec/ld-elf.so.1, libc.so.7 direct binding (note /usr/lib/libc.so is a GNU ld linker script GROUP(...) kit cannot parse — the profile already binds libc.so.7 directly), and DT_NEEDED resolution.
• Run validation: once a binary links, confirm the FreeBSD kernel accepts and runs it (ABI brand note, page-zero, stack setup) by executing it in the VM over SSH. Not yet reached.
• kit running on FreeBSD: driver/env/freebsd.c is written from man pages and marked UNTESTED (memfd/execmem dual-map, sysctl, resolver). The VMs now make native verification possible.

Validation matrix (per arch)

For each of aarch64 / x64 / rv64 on FreeBSD:

• kit cc -c produces a valid object with correct predefined macros / data model.
• kit cc -static links a hosted executable against a real base sysroot.
• kit cc (dynamic) links against libc.so.7 with correct PT_INTERP / DT_NEEDED.
• The produced binary runs in the VM and returns the expected output.
• Runtime helpers, atomics, setjmp, coroutines pass targeted tests.

How to reproduce today

# 1. Provision the VMs (one-time, cached afterwards). amd64/rv64 are slow (TCG).
bash scripts/freebsd_vm.sh prepare aarch64
bash scripts/freebsd_vm.sh prepare amd64
bash scripts/freebsd_vm.sh prepare riscv64

# 2. Extract a sysroot from one (see "Sysroot extraction" above).

# 3. Cross-compile (currently fails at the libc/libsys link blocker):
build/kit cc --target=x86_64-freebsd15.0 \
  --sysroot=build/freebsd-sysroot/amd64 -static hello.c -o hello

# 4. Once linking works: run it on the VM.
bash scripts/freebsd_vm.sh run amd64 &        # boot
scp -i build/freebsd-vm/ssh/id_ed25519 -P 2222 hello kit@127.0.0.1:
bash scripts/freebsd_vm.sh ssh amd64 ./hello