commit af5ecc87b77eaf69a9795b2fcc497459f5f024f1
parent e5087df9d440f55ba3e9e8d0b7165593a6c99041
Author: Ryan Sepassi <rsepassi@gmail.com>
Date: Tue, 9 Jun 2026 16:17:04 -0700
plan: SYSROOTS
Diffstat:
| A | doc/plan/SYSROOTS.md | | | 243 | +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ |
1 file changed, 243 insertions(+), 0 deletions(-)
diff --git a/doc/plan/SYSROOTS.md b/doc/plan/SYSROOTS.md
@@ -0,0 +1,243 @@
+# Cross-compilation sysroots
+
+This document is the design and implementation plan for packaging and
+distributing minimal sysroots that let kit users cross-compile for every
+platform in the release support set from any host.
+
+## Goals
+
+- One `kit sysroot install <triple>` installs everything needed to
+ cross-compile for that target.
+- Sysroots are minimal: stubs and headers only, no full framework trees or
+ large dynamic libraries.
+- Every sysroot ships as a signed `.kpkg`, distributed through kit's existing
+ CAS + package infrastructure.
+- Macros and library versions are versioned and reproducible; the kpkg hash is
+ the identity of a sysroot release.
+- macOS SDK headers are legally redistributable only by the user; the macOS
+ kpkg references them by CAS blob hash but does not embed them. Users
+ populate the blobs from a local SDK once; thereafter the kpkg installs
+ normally.
+
+## Support set
+
+| Triple pattern | libc variant | Notes |
+|---|---|---|
+| `{x86_64,aarch64,riscv64}-linux-musl` | musl static + stub `.so` | Compiled from musl source |
+| `{x86_64,aarch64,riscv64}-linux-gnu` | glibc stub `.so` | Stubs generated from reference glibc |
+| `{x86_64,aarch64}-windows-gnu` | mingw-w64 UCRT | Import stubs + headers, open source |
+| `aarch64-apple-macos{14,15}` | libSystem stub | `.tbd` stubs; headers by hash |
+| `{x86_64,aarch64,riscv64}-freebsd` | FreeBSD libc stub | Stubs + headers, BSD license |
+| `{x86_64,aarch64,riscv64,riscv32}-none` | freestanding | kit `rt/` only; no sysroot kpkg |
+
+Freestanding targets require no sysroot kpkg; `libkit_rt.a` and
+`rt/include/` serve that role and are already in-tree.
+
+## Sysroot kpkg layout
+
+Each installed sysroot unpacks into `~/.kit/sysroots/<triple>/`:
+
+```
+<triple>/
+ include/ system headers
+ lib/
+ crt1.o C runtime entry (where applicable)
+ crti.o
+ crtn.o
+ libc.a static libc (musl only; absent on stub-only platforms)
+ libc.so stub or real shared library
+ libm.so
+ libpthread.so
+ ... other platform libs
+ tbd/ macOS .tbd stubs (macOS targets only)
+```
+
+The `cc` driver's `-target` flag resolves the sysroot by triple lookup under
+`~/.kit/sysroots/`. Explicit `--sysroot=<path>` overrides this.
+
+## Library stubs
+
+### ELF stub `.so` (Linux glibc, FreeBSD)
+
+A stub `.so` contains only what the linker needs: SONAME, exported symbol
+table, and GNU version info (`.gnu.version`, `.gnu.version_r`). No PT_LOAD
+segment, no `.text`, no `.data`. The file is ~10–50 KB instead of hundreds of
+KB for the real library.
+
+Kit will include a **stub generator** (`src/obj/elf_stub.c`, exposed as
+`kit sysroot gen-elf-stub`) that:
+
+1. Reads a real `.so` (dynsym + versioning sections).
+2. Emits a minimal ELF with the same SONAME, all exported symbols as
+ `STT_NOTYPE`/`STB_GLOBAL`/`SHN_ABS`, and the original `.gnu.version_r`
+ chain.
+3. Records the source library's build-id in a `.note.kit.stub` section for
+ traceability.
+
+Stubs are generated offline from a reference installation (a glibc or FreeBSD
+system) and committed into the sysroot kpkg. They must be regenerated when the
+upstream ABI version changes.
+
+### macOS `.tbd` stubs
+
+Apple's Text-Based Stub Library (`.tbd`, YAML) is the native macOS stub format
+used inside Xcode SDKs. Kit's Mach-O linker already knows how to read `.tbd`
+files (required for linking against system frameworks). The macOS sysroot kpkg
+contains `.tbd` files for `libSystem`, `libm`, `libpthread`, `libc++`, and the
+core framework stubs.
+
+The `.tbd` files are open/YAML; they can be extracted from any macOS SDK
+installation and are small.
+
+### Windows import stubs (`.dll.a`)
+
+mingw-w64 provides import stub archives for the Windows SDK DLLs (kernel32,
+ucrtbase, etc.) under a MIT-compatible license. The Windows sysroot kpkg
+bundles these directly alongside the mingw-w64 UCRT headers. No generation
+step is needed; the mingw-w64 tree is the upstream source.
+
+## musl: compiled from source
+
+For `*-linux-musl` targets kit compiles musl directly rather than shipping
+pre-built archives. Advantages: reproducible, architecture-correct, no
+redistribution complexity, and a good exercise of kit's self-compilation
+pipeline.
+
+The musl source tree (a pinned release tarball, verified by BLAKE2b hash) is
+bundled in the kpkg for each architecture. `kit sysroot install
+aarch64-linux-musl` fetches the kpkg, extracts the musl source, and compiles
+it with `kit cc -target aarch64-linux-musl` into:
+
+- `libc.a` — static archive
+- `libc.so` (stub only; the real `.so` for dynamic musl is the runtime-provided
+ `ld-musl-<arch>.so.1`)
+- CRT objects: `crt1.o`, `rcrt1.o` (PIE), `Scrt1.o`, `crti.o`, `crtn.o`
+
+The compile is cached by the musl source hash, so reinstalling the same kpkg
+version is a no-op.
+
+## macOS SDK headers: CAS-referenced blobs
+
+Apple's SDK headers may not be redistributed by third parties. The macOS
+sysroot kpkg solves this by **referencing** the header blobs by CAS hash
+without embedding them:
+
+```yaml
+# excerpt from the macOS sysroot kpkg manifest
+headers:
+ required_blobs:
+ - hash: "sha256:abc123..." # usr/include/stdio.h
+ install_path: include/stdio.h
+ - hash: "sha256:def456..." # usr/include/sys/types.h
+ install_path: include/sys/types.h
+ ...
+```
+
+When `kit sysroot install aarch64-apple-macos14` runs:
+
+1. Kit resolves each required blob from the local CAS.
+2. If any blobs are missing, it prints the missing hashes and instructs the
+ user to run `kit sysroot extract-macos-headers`.
+3. Once all blobs are present, the sysroot is materialized normally.
+
+The blobs themselves are ordinary CAS objects — the user may host them on any
+CAS-compatible store (private server, S3, local NAS) and point kit at it via
+`KIT_CAS_REMOTE` or `kit cas remote add`.
+
+### `kit sysroot extract-macos-headers`
+
+```
+kit sysroot extract-macos-headers [--sdk-path=<path>] [--target=<triple>]
+```
+
+Walks the SDK's `usr/include/` (and selected framework headers), ingests each
+file as a CAS blob, and prints a manifest fragment with the resulting hashes.
+This is run once per SDK version by the user on their Mac. It does not upload
+anything; it just populates the local CAS. The user may then push those blobs
+to their preferred remote.
+
+The manifest fragment output is what kit's sysroot release process uses to
+build the macOS kpkg. For users who just want things to work locally, the
+extract step followed by `kit sysroot install` is sufficient.
+
+## `kit sysroot` command surface
+
+```
+kit sysroot list # show installed sysroots
+kit sysroot available # show sysroots available in configured remotes
+kit sysroot install <triple> # fetch + install sysroot kpkg
+kit sysroot install <triple> --from=<path> # install from a local kpkg file
+kit sysroot remove <triple> # uninstall
+kit sysroot update [<triple>] # reinstall latest version
+kit sysroot path <triple> # print install path (for --sysroot= use)
+
+kit sysroot extract-macos-headers [--sdk-path=<path>] [--target=<triple>]
+kit sysroot gen-elf-stub <input.so> -o <stub.so>
+kit sysroot gen-kpkg <triple> --config=<sysroot.toml> -o <output.kpkg>
+```
+
+`gen-kpkg` is the sysroot *builder* used offline by kit's release pipeline. It
+reads a `sysroot.toml` that declares the source for each component (musl source
+hash, upstream `.so` paths to stub-ify, mingw tree path, `.tbd` directory,
+macOS header manifest), runs the appropriate generators, and emits a signed
+kpkg.
+
+## Implementation work items
+
+### Infrastructure
+
+- [ ] `src/obj/elf_stub.c` — ELF stub `.so` generator; reads dynsym +
+ `.gnu.version_r`, emits a no-PT_LOAD ELF. Exposed via `kit sysroot
+ gen-elf-stub`.
+- [ ] `driver/cmd/sysroot.c` — the `kit sysroot` subcommand (list / available /
+ install / remove / update / path / extract-macos-headers / gen-elf-stub /
+ gen-kpkg).
+- [ ] `include/kit/sysroot.h` + `src/api/sysroot.c` — public API for sysroot
+ resolution, used by the `cc` driver's `-target` lookup.
+- [ ] `sysroot.toml` schema — declarative input format for `gen-kpkg`; one file
+ per `(triple, libc-variant, libc-version)`.
+- [ ] macOS blob-reference manifest support in kpkg reader — when a kpkg
+ declares `required_blobs`, validate presence before materializing, and
+ emit a helpful error with the `extract-macos-headers` hint if missing.
+
+### Sysroot sources and generation
+
+- [ ] **musl** — pin a musl release (tarball + BLAKE2b), write a `sysroot.toml`
+ per `{x86_64,aarch64,riscv64}-linux-musl`, implement the in-install
+ compile step.
+- [ ] **Linux glibc** — generate ELF stubs from a reference glibc
+ `{x86_64,aarch64,riscv64}` installation (targeting glibc 2.35 as a
+ reasonable minimum). Package headers + stubs + CRT objects into kpkgs.
+- [ ] **Windows** — package mingw-w64 headers + import stubs for
+ `{x86_64,aarch64}-windows-gnu`. Validate against the VM test suite
+ (`test-hosted`).
+- [ ] **macOS** — generate `.tbd` stubs, write header manifest, validate the
+ blob-reference install flow on a local macOS SDK. Target macOS 14 and 15.
+- [ ] **FreeBSD** — generate ELF stubs + package headers from a FreeBSD 14
+ `{x86_64,aarch64,riscv64}` installation.
+
+### CI and distribution
+
+- [ ] Release pipeline script (`scripts/gen_sysroots.sh`) — runs `gen-kpkg`
+ for each sysroot, signs the resulting kpkgs, and pushes to the release
+ CAS remote.
+- [ ] `kit sysroot available` fetches the index from the configured release
+ remote and shows version + size for each triple.
+- [ ] Add sysroot install smoke tests to `test/hosted` for each supported
+ triple: install sysroot, compile `hello.c`, run on target (or VM).
+
+## Open questions
+
+- **glibc version targeting**: ship stubs for a single baseline version
+ (e.g. 2.35) or per-version kpkgs? Per-version is more correct but multiplies
+ the release surface. Start with a single baseline and version the kpkg
+ (users who need an older ABI can pin the kpkg version).
+- **FreeBSD version**: FreeBSD 14 as the baseline. FreeBSD moves slowly;
+ one kpkg version per major release is likely sufficient.
+- **macOS SDK scraping automation**: the `extract-macos-headers` step is manual.
+ A CI job running on a macOS runner could automate it, but requires a macOS
+ SDK license. For now the manifest is generated manually when an SDK version
+ changes and the blob hashes are committed to the release pipeline config.
+- **CAS remote hosting**: where are the kpkgs and blobs hosted? Options: GitHub
+ Releases (artifacts, size-limited), S3/R2, or a self-hosted kit CAS instance.
+ This is a release infrastructure question, not a kit code question.
