boot2

Playing with the boostrap
git clone https://git.ryansepassi.com/git/boot2.git
Log | Files | Refs | README

commit df64aa7a117864615950a5249af34444eb75b249
parent 76fd67990e47b226037ab1da73887d10684e5aad
Author: Ryan Sepassi <rsepassi@gmail.com>
Date:   Wed, 20 May 2026 10:51:48 -0700

release: per-arch tarball with input + output manifests, validated mint flow

Adds `make release` (and `make package`) which produce a self-contained
per-arch tarball that lets anyone reproduce the boot0..boot6 chain off
the bundled inputs and byte-compare against a hash manifest:

  tools/mkrelease.sh        stages build/<arch>/src/ + boot/ into a per-
                            arch dir, generates INPUT_MANIFEST.txt (sha256
                            of every file under src/ + boot/) and
                            OUTPUT_MANIFEST.txt (sha256 of per-stage
                            artifacts: boot0/{hex2,catm,M0}, boot1/{M1pp,
                            hex2pp}, …, boot6/<kernel>), drops in README +
                            verify.sh, and writes a deterministic tarball
                            (touch mtimes + sorted -T list + ustar format +
                            gzip -n).
  tools/release/verify.sh   ships inside the tarball; stages src/ into
                            build/<arch>/src/, runs boot0..boot6, diffs
                            each artifact's sha256 against the bundled
                            OUTPUT_MANIFEST. Preflight check for the macOS
                            podman-VM /Users/ mount constraint.
  tools/release/README.md   bundled extract + run instructions.
  tools/release.sh          the validated mint flow: clean + package twice
                            from scratch, assert byte-identical tarballs,
                            extract one and run its verify.sh, then
                            promote to dist/. Pass tarballs vault under
                            mktemp so the inter-pass `make clean` can't
                            destroy them.

Make: `make package` is the quick path (build + tar). `make release`
runs tools/release.sh; dist/ is the only directory ever written for
publication. `make clean` only touches build/, so dist/ artifacts
survive iteration.

Also folds in two fixes the release flow surfaced:

  - Makefile prep-musl.sh rule was stale (the script was removed in
    e33c8d1, musl handling is inline in prep-src.sh now). musl/.stamp
    is now a simple touch keyed off src/.stamp. Without this, boot5
    couldn't build from a clean tree.
  - rename src/src/vendor-seed → src/src/stage0-posix to match the
    upstream live-bootstrap directory name. Updates the four bootN.sh
    references and prep-src.sh.

Validated end-to-end on aarch64: pass A and pass B sha256 match
(cfa1531631cd…), bundled verify.sh re-runs boot0..boot6 and diffs all
23 artifacts OK, mint promotes to dist/. ~5 min wall.

Diffstat:
M.gitignore | 1+
MMakefile | 39++++++++++++++++++++++++++++-----------
Mboot/boot0.sh | 4++--
Mboot/boot1.sh | 4++--
Mboot/boot2.sh | 4++--
Mboot/boot3.sh | 4++--
Mbootprep/prep-src.sh | 6+++---
Atools/mkrelease.sh | 210+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Atools/release.sh | 148+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Atools/release/README.md | 104+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Atools/release/verify.sh | 124+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
11 files changed, 626 insertions(+), 22 deletions(-)

diff --git a/.gitignore b/.gitignore @@ -1 +1,2 @@ build/ +dist/ diff --git a/Makefile b/Makefile @@ -60,7 +60,7 @@ OUT_DIR := build/$(ARCH)/$(DRIVER) .SUFFIXES: -.PHONY: all help clean cloc src +.PHONY: all help clean cloc src package release # ── Top-level targets ──────────────────────────────────────────────────── @@ -73,6 +73,8 @@ help: @echo 'Targets (default ARCH=$(ARCH) DRIVER=$(DRIVER)):' @echo ' make all build boot6 kernel' @echo ' make src prep canonical src/ tree (incl. musl)' + @echo ' make package quick: package boot2-<arch>-<rev>.tar.gz from current build' + @echo ' make release validated: clean-rebuild x2 + verify, mint to dist/' @echo ' make build/<arch>/<driver>/boot6/<kn> full chain (kn = Image | kernel.elf)' @echo ' make build/<arch>/<driver>/bootN/<file> any single artifact' @echo ' make test SUITE=<suite> test suite (NAMES=<filter> optional)' @@ -83,6 +85,23 @@ help: clean: rm -rf build/ +# `package`: package a per-arch tarball from the current build tree. +# Depends on the full chain (boot5 musl + boot6 kernel) so +# OUTPUT_MANIFEST.txt has real artifacts to hash. boot5 is not on the +# `all` dep path (boot6 doesn't link musl), so list it explicitly here. +# Lands at build/<arch>/release/boot2-<arch>-<rev>.tar.gz. Fast: no +# reproducibility or verify check. +package: build/$(ARCH)/$(DRIVER)/boot5/libc.a \ + build/$(ARCH)/$(DRIVER)/boot6/$(KERNEL_NAME_$(ARCH)) + DRIVER=$(DRIVER) tools/mkrelease.sh $(ARCH) + +# `release`: the validated path. Cleans, rebuilds + packages twice, +# asserts both tarballs hash-match, extracts one and runs its +# verify.sh, then promotes to dist/. Slow but paranoid; this is the +# only way a tarball lands in dist/. +release: + DRIVER=$(DRIVER) tools/release.sh $(ARCH) + # ── prep-src + boot0..boot6 chain (rules per arch × driver) ────────────── # # The .stamp files are the make-rule pegs. Each rule lists its real @@ -133,21 +152,19 @@ prep_src_arch_srcs = \ # in the podman side automatically. Empty for DRIVER=podman. seed_kernel_dep = $(if $(filter seed,$2),build/$1/podman/boot6/$(KERNEL_NAME_$1)) -# Per-arch prep-src + prep-musl rules. Driver-independent. +# Per-arch prep-src rule. Driver-independent. +# +# prep-src.sh produces the full canonical tree, including the filtered +# musl/ subtree (unpack + overrides + deletes + per-arch skip list, +# plus the generated boot5 enumerate / run.scm). musl/.stamp is a +# secondary peg that boot5 depends on — it's just touched after +# src/.stamp, since the musl tree is populated by the same recipe. define PREP_RULES build/$1/src/.stamp: $$(PREP_SRC_COMMON_SRCS) $$(call prep_src_arch_srcs,$1) bootprep/prep-src.sh $1 @touch $$@ -# Filtered musl tree. Depends on the canonical tree from prep-src and -# (when present) the committed per-arch skip list. If the skip list is -# missing, prep-musl.sh runs bootprep/boot5-calibrate.sh, which drives -# its own boot4 build outside this make graph — keep make's deps simple. -build/$1/src/musl/.stamp: build/$1/src/.stamp \ - bootprep/prep-musl.sh boot/lib-arch.sh \ - bootprep/boot5-enumerate.sh bootprep/boot5-gen-runscm.sh \ - $$(wildcard vendor/musl/skip-$1.txt) - bootprep/prep-musl.sh $1 +build/$1/src/musl/.stamp: build/$1/src/.stamp @mkdir -p $$(@D) && touch $$@ endef diff --git a/boot/boot0.sh b/boot/boot0.sh @@ -7,7 +7,7 @@ ## ## ─── Inputs (sources, from canonical tree) ─────────────────────────── ## build/$ARCH/src/bin/hex0-seed -## build/$ARCH/src/src/vendor-seed/{hex0.hex0, hex1.hex0, hex2.hex1, +## build/$ARCH/src/src/stage0-posix/{hex0.hex0, hex1.hex0, hex2.hex1, ## catm.hex2, M0.hex2, ELF.hex2} ## ## ─── Outputs ────────────────────────────────────────────────────────── @@ -29,7 +29,7 @@ pipeline_init "$STAGE" "$OUT" "$DRIVER" # ─── inputs (from canonical src tree) ───────────────────────────────── pipeline_input hex0-seed "build/$ARCH/src/bin/hex0-seed" for f in hex0.hex0 hex1.hex0 hex2.hex1 catm.hex2 M0.hex2 ELF.hex2; do - pipeline_input_from_src "vendor-seed/$f" + pipeline_input_from_src "stage0-posix/$f" done # ─── pipeline ───────────────────────────────────────────────────────── diff --git a/boot/boot1.sh b/boot/boot1.sh @@ -9,7 +9,7 @@ ## build/$ARCH/src/src/M1pp/M1pp.P1 ## build/$ARCH/src/src/hex2pp/hex2pp.P1 ## build/$ARCH/src/src/P1/P1-$ARCH.M1 -## build/$ARCH/src/src/vendor-seed/ELF.hex2 +## build/$ARCH/src/src/stage0-posix/ELF.hex2 ## ## ─── Inputs (binaries from prior stages) ────────────────────────────── ## build/$ARCH/$DRIVER/boot0/{hex2, M0, catm} @@ -38,7 +38,7 @@ pipeline_input hex2 "$BOOT0/hex2" pipeline_input M0 "$BOOT0/M0" pipeline_input catm "$BOOT0/catm" pipeline_input_from_src "P1/P1-$ARCH.M1" P1.M1 -pipeline_input_from_src vendor-seed/ELF.hex2 +pipeline_input_from_src stage0-posix/ELF.hex2 pipeline_input_from_src M1pp/M1pp.P1 pipeline_input_from_src hex2pp/hex2pp.P1 diff --git a/boot/boot2.sh b/boot/boot2.sh @@ -10,7 +10,7 @@ ## build/$ARCH/src/src/catm/catm.P1pp ## build/$ARCH/src/src/scheme1/scheme1.P1pp ## build/$ARCH/src/src/P1/{P1-$ARCH.M1pp, P1.M1pp, P1pp.P1pp} -## build/$ARCH/src/src/vendor-seed/ELF.hex2 +## build/$ARCH/src/src/stage0-posix/ELF.hex2 ## ## ─── Inputs (binaries from prior stages) ────────────────────────────── ## build/$ARCH/$DRIVER/boot0/catm (only to bootstrap catm.P1pp build) @@ -44,7 +44,7 @@ pipeline_input hex2pp "$BOOT1/hex2pp" pipeline_input_from_src "P1/P1-$ARCH.M1pp" backend.M1pp pipeline_input_from_src P1/P1.M1pp frontend.M1pp pipeline_input_from_src P1/P1pp.P1pp libp1pp.P1pp -pipeline_input_from_src vendor-seed/ELF.hex2 +pipeline_input_from_src stage0-posix/ELF.hex2 pipeline_input_from_src catm/catm.P1pp pipeline_input_from_src scheme1/scheme1.P1pp diff --git a/boot/boot3.sh b/boot/boot3.sh @@ -16,7 +16,7 @@ ## build/$ARCH/src/src/cc/{cc.scm, main.scm} scheme bundle ## build/$ARCH/src/src/P1/{P1-$ARCH.M1pp, P1.M1pp, P1pp.P1pp} M1pp pipeline ## build/$ARCH/src/src/P1/{entry-libc.P1pp, elf-end.P1pp} link framing -## build/$ARCH/src/src/vendor-seed/ELF.hex2 ELF header +## build/$ARCH/src/src/stage0-posix/ELF.hex2 ELF header ## build/$ARCH/src/src/tcc/tcc.flat.c flattened tcc TU ## build/$ARCH/src/src/libc/libc.flat.c flattened mes-libc TU ## @@ -77,7 +77,7 @@ runscm_input_from_src P1/P1.M1pp frontend.M1pp runscm_input_from_src P1/P1pp.P1pp libp1pp.P1pp runscm_input_from_src P1/entry-libc.P1pp runscm_input_from_src P1/elf-end.P1pp -runscm_input_from_src vendor-seed/ELF.hex2 +runscm_input_from_src stage0-posix/ELF.hex2 runscm_input_from_src tcc/tcc.flat.c runscm_input_from_src libc/libc.flat.c diff --git a/bootprep/prep-src.sh b/bootprep/prep-src.sh @@ -11,7 +11,7 @@ ## bin/ binary inputs not built by a stage ## hex0-seed vendored seed only ## src/ everything textual -## vendor-seed/ ELF.hex2 + *.hex0|*.hex1|*.hex2 +## stage0-posix/ ELF.hex2 + *.hex0|*.hex1|*.hex2 ## M1pp/ M1pp.P1 ## hex2pp/ hex2pp.P1 ## P1/ P1*.{M1,M1pp,P1pp}, entry-*.P1pp, @@ -66,10 +66,10 @@ SEED=vendor/seed/$ARCH cp "$SEED/hex0-seed" "$DST_BIN/hex0-seed" -mkdir -p "$DST_SRC/vendor-seed" +mkdir -p "$DST_SRC/stage0-posix" for f in ELF.hex2 hex0.hex0 hex1.hex0 hex2.hex1 catm.hex2 M0.hex2; do [ -e "$SEED/$f" ] || { echo "$TAG missing $SEED/$f" >&2; exit 1; } - cp "$SEED/$f" "$DST_SRC/vendor-seed/$f" + cp "$SEED/$f" "$DST_SRC/stage0-posix/$f" done # ── (2) repo-tree textual sources ───────────────────────────────────── diff --git a/tools/mkrelease.sh b/tools/mkrelease.sh @@ -0,0 +1,210 @@ +#!/bin/sh +## mkrelease.sh — package a per-arch boot2 release tarball. +## +## A release tarball is a self-contained bundle that lets anyone +## reproduce the full boot0..boot6 chain off the bundled inputs and +## byte-compare the outputs against a hash manifest. Layout: +## +## boot2-<arch>[-<rev>].tar.gz +## boot2-<arch>[-<rev>]/ +## README.md extract + run instructions +## verify.sh drives boot0..boot6 + diffs OUTPUT_MANIFEST +## INPUT_MANIFEST.txt sha256 of every input under src/ + boot/ +## OUTPUT_MANIFEST.txt sha256 of expected per-stage artifacts +## (driver-agnostic; the project's seed-accept +## harness verifies podman vs seed equivalence) +## src/ the sealed source tree (from +## build/<arch>/src/, produced by prep-src.sh) +## boot/ boot0..boot6 stage drivers + libs + +## containers/Containerfile.* +## +## The output manifest is generated from the current build outputs in +## build/<arch>/<driver>/boot{0..6}/. mkrelease.sh does NOT rebuild; +## prereqs (`make all ARCH=<arch>`) must already have run. +## +## Usage: tools/mkrelease.sh <arch> +## <arch> ∈ {aarch64, amd64, riscv64} +## Env: +## DRIVER podman (default) | seed — which build tree to hash for the +## output manifest. The manifest is +## claimed driver-agnostic regardless. +## REV short rev string to tag the tarball name; auto-detected from +## `git rev-parse --short HEAD` when unset, "norev" otherwise. + +set -eu + +ARCH=${1:-} +case "$ARCH" in + aarch64|amd64|riscv64) ;; + *) echo "usage: $0 <aarch64|amd64|riscv64>" >&2; exit 2 ;; +esac + +DRIVER=${DRIVER:-podman} +case "$DRIVER" in + podman|seed) ;; + *) echo "[mkrelease] unknown DRIVER=$DRIVER (expected podman|seed)" >&2; exit 2 ;; +esac + +ROOT=$(cd "$(dirname "$0")/.." && pwd) +cd "$ROOT" + +case "$ARCH" in + aarch64) KERNEL_NAME=Image ;; + amd64) KERNEL_NAME=kernel.elf ;; + riscv64) KERNEL_NAME=kernel.elf ;; +esac + +REV=${REV:-} +if [ -z "$REV" ]; then + REV=$(git rev-parse --short HEAD 2>/dev/null || echo norev) +fi +NAME=boot2-$ARCH-$REV + +SRC_TREE=build/$ARCH/src +BUILD_TREE=build/$ARCH/$DRIVER +REL_DIR=build/$ARCH/release +STAGING=$REL_DIR/$NAME +TARBALL=$REL_DIR/$NAME.tar.gz + +[ -d "$SRC_TREE" ] || { echo "[mkrelease] missing $SRC_TREE — run bootprep/prep-src.sh $ARCH" >&2; exit 1; } +[ -f "$BUILD_TREE/boot6/$KERNEL_NAME" ] || { echo "[mkrelease] missing $BUILD_TREE/boot6/$KERNEL_NAME — run 'make all ARCH=$ARCH DRIVER=$DRIVER'" >&2; exit 1; } + +# Portable sha256. Use sha256sum if present; else shasum -a 256. +if command -v sha256sum >/dev/null 2>&1; then + sha256() { sha256sum "$1" | awk '{print $1}'; } +else + sha256() { shasum -a 256 "$1" | awk '{print $1}'; } +fi + +echo "[mkrelease] staging -> $STAGING" +rm -rf "$STAGING" +mkdir -p "$STAGING" + +# ── (1) sealed source tree ──────────────────────────────────────────── +cp -R "$SRC_TREE" "$STAGING/src" + +# ── (2) boot drivers (boot/*.sh, lib-*.sh, containers/Containerfile.*) ─ +cp -R boot "$STAGING/boot" + +# ── (3) README + verify.sh templates ────────────────────────────────── +cp tools/release/README.md "$STAGING/README.md" +cp tools/release/verify.sh "$STAGING/verify.sh" +chmod +x "$STAGING/verify.sh" + +# Substitute @ARCH@ / @REV@ / @KERNEL_NAME@ into shipped docs/scripts. +for f in "$STAGING/README.md" "$STAGING/verify.sh"; do + sed -i.bak \ + -e "s/@ARCH@/$ARCH/g" \ + -e "s/@REV@/$REV/g" \ + -e "s/@KERNEL_NAME@/$KERNEL_NAME/g" \ + "$f" + rm -f "$f.bak" +done + +# ── (4) INPUT_MANIFEST.txt ──────────────────────────────────────────── +echo "[mkrelease] input manifest" +INMAN=$STAGING/INPUT_MANIFEST.txt +( + cd "$STAGING" + find src boot -type f | LC_ALL=C sort | while read -r rel; do + h=$(sha256 "$rel") + printf '%s %s\n' "$h" "$rel" + done +) > "$INMAN" +n_in=$(wc -l < "$INMAN" | tr -d ' ') + +# ── (5) OUTPUT_MANIFEST.txt ─────────────────────────────────────────── +# Per-stage key artifacts (mirrors the stamp-anchored declarations in +# the top-level Makefile). Driver-agnostic. +gen_outputs() { + cat <<EOF +boot0/hex2 +boot0/catm +boot0/M0 +boot1/M1pp +boot1/hex2pp +boot2/catm +boot2/scheme1 +boot3/tcc0 +boot3/libc.P1pp +boot3/tcc.flat.P1pp +boot4/tcc1 +boot4/tcc2 +boot4/tcc3 +boot4/hello +boot4/crt1.o +boot4/libc.a +boot4/libtcc1.a +boot5/libc.a +boot5/crt1.o +boot5/crti.o +boot5/crtn.o +boot5/hello +boot6/$KERNEL_NAME +EOF +} + +echo "[mkrelease] output manifest (from $BUILD_TREE)" +OUTMAN=$STAGING/OUTPUT_MANIFEST.txt +: > "$OUTMAN" +rm -f "$OUTMAN.missing" +gen_outputs | while read -r rel; do + [ -n "$rel" ] || continue + f=$BUILD_TREE/$rel + if [ -e "$f" ]; then + h=$(sha256 "$f") + printf '%s %s\n' "$h" "$rel" >> "$OUTMAN" + else + printf '%s\n' "$rel" >> "$OUTMAN.missing" + fi +done +if [ -s "$OUTMAN.missing" ]; then + echo "[mkrelease] FAIL: missing expected outputs under $BUILD_TREE:" >&2 + sed 's/^/ /' "$OUTMAN.missing" >&2 + echo "[mkrelease] run 'make all ARCH=$ARCH DRIVER=$DRIVER' (and boot5) first" >&2 + rm -f "$OUTMAN.missing" + exit 1 +fi +n_out=$(wc -l < "$OUTMAN" | tr -d ' ') + +# ── (6) tarball — deterministic ────────────────────────────────────── +# Identical inputs must yield byte-identical tarballs. Sources of drift: +# (a) gzip header embeds wall-clock mtime + original filename +# → use `gzip -n` (no name, no timestamp). +# (b) tar entries embed per-file mtime + uid/gid/uname/gname. +# → normalize on-disk mtimes with `touch -t`; override ownership +# in the tar headers via --uid/--gid/--uname/--gname. +# (c) tar entry order = filesystem readdir order. +# → sorted -T file list. +# Note: macOS bsdtar refuses `--mtime` together with `-T -`. We avoid +# that combo by normalizing mtimes on disk first. +# Also: a pipeline like `... | tar | gzip > out` silently produces an +# empty gzip if tar fails. Build the uncompressed tar to a tempfile so +# `set -e` catches tar's exit code, then gzip from disk. +echo "[mkrelease] tar -> $TARBALL" +find "$STAGING" -exec touch -t 200001010000.00 {} + + +TAR_TMP=$PWD/$REL_DIR/$NAME.tar +TARBALL_ABS=$PWD/$TARBALL +rm -f "$TAR_TMP" "$TARBALL" +( + cd "$REL_DIR" + find "$NAME" -print0 | LC_ALL=C sort -z | \ + tar --null -T - \ + --uid 0 --gid 0 --uname '' --gname '' \ + --format ustar \ + -cf "$TAR_TMP" +) +gzip -n -9 < "$TAR_TMP" > "$TARBALL_ABS" +rm -f "$TAR_TMP" + +# Tarball digest (echoed for release notes; not embedded in the tar). +TAR_SHA=$(sha256 "$TARBALL") +printf '%s %s\n' "$TAR_SHA" "$NAME.tar.gz" > "$REL_DIR/$NAME.tar.gz.sha256" + +bytes=$(wc -c < "$TARBALL" | tr -d ' ') +echo "[mkrelease] OK" +echo "[mkrelease] tarball : $TARBALL ($bytes bytes)" +echo "[mkrelease] sha256 : $TAR_SHA" +echo "[mkrelease] inputs : $n_in files" +echo "[mkrelease] outputs : $n_out artifacts" diff --git a/tools/release.sh b/tools/release.sh @@ -0,0 +1,148 @@ +#!/bin/sh +## release.sh — mint a validated boot2 release tarball. +## +## Releases are critical, so the path to minting one is paranoid: +## +## 1. `make clean` — wipe build/ entirely. +## 2. `make package` — full prep-src → boot0..boot6 → mkrelease.sh chain. +## Capture tarball A's sha256. +## 3. `make clean` again — fresh state, no cached intermediates. +## 4. `make package` again — full rebuild + repackage. +## Capture tarball B's sha256. +## 5. Assert A == B. Bit-for-bit reproducibility check; if this drifts, +## something in the input set or build is nondeterministic and the +## release is not safe to publish. +## 6. Extract one tarball into a fresh dir under $HOME (so the macOS +## podman VM can see it) and run the bundled verify.sh. verify.sh +## re-runs boot0..boot6 off the bundled inputs and diffs each +## artifact's sha256 against OUTPUT_MANIFEST.txt. End-to-end proof +## that the shipped tarball reproduces the claimed outputs. +## 7. Promote the validated tarball to dist/<name>.tar.gz with a +## sha256 sidecar. dist/ is the only directory mkrelease.sh / +## release.sh ever writes to that's intended for publication. +## +## On any failure the dist/ output is NOT created; the operator gets a +## clear error pointing at which pass failed. The two intermediate +## tarballs are kept under build/<arch>/release/ for forensic diffing. +## +## Usage: tools/release.sh <arch> +## <arch> ∈ {aarch64, amd64, riscv64} +## Env: +## DRIVER podman (default) | seed — passed through to make. +## REV short rev string for the tarball name; auto-detected from +## git when unset. + +set -eu + +ARCH=${1:-} +case "$ARCH" in + aarch64|amd64|riscv64) ;; + *) echo "usage: $0 <aarch64|amd64|riscv64>" >&2; exit 2 ;; +esac + +DRIVER=${DRIVER:-podman} +case "$DRIVER" in + podman|seed) ;; + *) echo "[release] unknown DRIVER=$DRIVER" >&2; exit 2 ;; +esac + +ROOT=$(cd "$(dirname "$0")/.." && pwd) +cd "$ROOT" + +REV=${REV:-} +if [ -z "$REV" ]; then + REV=$(git rev-parse --short HEAD 2>/dev/null || echo norev) +fi +NAME=boot2-$ARCH-$REV +REL_DIR=build/$ARCH/release +DIST=dist + +# Pass-tarball vault — must live OUTSIDE build/ so it survives the +# `make clean` between passes. mktemp under /tmp; cleaned on exit. +VAULT=$(mktemp -d -t boot2-release-XXXXXX) +trap 'rm -rf "$VAULT"' EXIT + +# Portable sha256. +if command -v sha256sum >/dev/null 2>&1; then + sha256() { sha256sum "$1" | awk '{print $1}'; } +else + sha256() { shasum -a 256 "$1" | awk '{print $1}'; } +fi + +log() { printf '[release] %s\n' "$*"; } +hr() { printf '[release] ===================== %s =====================\n' "$*"; } + +t0=$(date +%s) + +# Two passes from a clean state. Each pass: make clean + make package. +# Save each tarball aside under a pass-specific name so we can compare +# them even if the recipe is rerun by hand later. +do_pass() { + _label=$1; _outvar=$2 + hr "pass $_label" + log "make clean" + make clean >/dev/null + log "make package ARCH=$ARCH DRIVER=$DRIVER REV=$REV" + REV=$REV make package ARCH="$ARCH" DRIVER="$DRIVER" + _src=$REL_DIR/$NAME.tar.gz + _dst=$VAULT/$NAME.pass-$_label.tar.gz + [ -f "$_src" ] || { log "FAIL: pass $_label produced no $_src"; exit 1; } + cp "$_src" "$_dst" + _h=$(sha256 "$_dst") + log "pass $_label sha256: $_h" + eval "$_outvar=\$_h" +} + +do_pass A SHA_A +do_pass B SHA_B + +hr "compare" +if [ "$SHA_A" != "$SHA_B" ]; then + log "FAIL: pass A and pass B produced DIFFERENT tarballs" + log " A: $SHA_A ($VAULT/$NAME.pass-A.tar.gz)" + log " B: $SHA_B ($VAULT/$NAME.pass-B.tar.gz)" + log "" + log "Copying tarballs to dist/.failed-passes/ for inspection (vault is auto-cleaned)." + mkdir -p "$DIST/.failed-passes" + cp "$VAULT/$NAME.pass-A.tar.gz" "$DIST/.failed-passes/" + cp "$VAULT/$NAME.pass-B.tar.gz" "$DIST/.failed-passes/" + log "To investigate, extract both and diff the trees:" + log " mkdir /tmp/A /tmp/B" + log " tar xzf $DIST/.failed-passes/$NAME.pass-A.tar.gz -C /tmp/A" + log " tar xzf $DIST/.failed-passes/$NAME.pass-B.tar.gz -C /tmp/B" + log " diff -r /tmp/A /tmp/B" + log "(check INPUT_MANIFEST.txt and OUTPUT_MANIFEST.txt for hash drift)" + exit 1 +fi +log "OK: both passes produced identical tarballs" +log " sha256 = $SHA_A" + +# End-to-end verify: extract the (still-installed) tarball, run its +# verify.sh, which re-runs boot0..boot6 inside a fresh tree and +# hash-diffs every artifact in OUTPUT_MANIFEST.txt. +hr "verify" +# macOS podman VM only mounts /Users/, so the verify dir must live +# under $HOME. ~/.cache is a stable, throwaway-friendly location. +VERIFY_BASE=$HOME/.cache/boot2-release-verify/$ARCH +log "extract -> $VERIFY_BASE/$NAME" +rm -rf "$VERIFY_BASE" +mkdir -p "$VERIFY_BASE" +tar xzf "$VAULT/$NAME.pass-A.tar.gz" -C "$VERIFY_BASE" + +log "running ./verify.sh (DRIVER=$DRIVER) — this rebuilds the chain" +( cd "$VERIFY_BASE/$NAME" && DRIVER=$DRIVER ./verify.sh ) + +# Promote to dist/ — the only directory we treat as "publishable". +hr "mint" +mkdir -p "$DIST" +cp "$VAULT/$NAME.pass-A.tar.gz" "$DIST/$NAME.tar.gz" +printf '%s %s\n' "$SHA_A" "$NAME.tar.gz" > "$DIST/$NAME.tar.gz.sha256" + +# Clean up the verify dir (it's reproducible from the tarball). +rm -rf "$VERIFY_BASE" + +elapsed=$(( $(date +%s) - t0 )) +bytes=$(wc -c < "$DIST/$NAME.tar.gz" | tr -d ' ') +log "MINTED in ${elapsed}s" +log " tarball : $DIST/$NAME.tar.gz ($bytes bytes)" +log " sha256 : $SHA_A" diff --git a/tools/release/README.md b/tools/release/README.md @@ -0,0 +1,104 @@ +# boot2 release — @ARCH@ @REV@ + +This tarball is a self-contained input bundle for the boot2 bootstrap +chain on `@ARCH@`. Extract it, run `./verify.sh`, and the chain will be +re-run from boot0 through boot6 using only the bundled inputs. The +outputs are sha256-compared against `OUTPUT_MANIFEST.txt`. + +## Contents + +``` +boot2-@ARCH@-@REV@/ +├── README.md this file +├── verify.sh build + diff driver +├── INPUT_MANIFEST.txt sha256 of every file under src/ + boot/ +├── OUTPUT_MANIFEST.txt sha256 of expected per-stage artifacts +├── src/ sealed source tree (canonical inputs) +│ ├── bin/hex0-seed vendored ELF seed (the only opaque input) +│ ├── src/ all textual sources: P1, M1pp, hex2pp, +│ │ scheme1, cc, tcc, mes-libc, musl, kernel +│ └── run/ run.scm files driving boot3..boot6 +└── boot/ boot0..boot6 stage drivers + libs + ├── boot{0..6}.sh + ├── lib-{arch,pipeline,runscm}.sh + └── containers/ Containerfile.{busybox,empty} (DRIVER=podman) +``` + +There is no `bootprep/` and no `vendor/` — every input the chain needs +is already inside `src/`. `bootprep/` exists in the upstream repo to +*populate* `src/`; this tarball ships the populated tree directly. + +## Trust path + +`src/bin/hex0-seed` is the only opaque artifact in the chain (a few +hundred bytes; vendored from live-bootstrap's stage0-posix). Every +other file under `src/` is text and is hashed in `INPUT_MANIFEST.txt`. +You can audit the manifest end-to-end before running anything. + +## Running + +> **macOS note:** with `DRIVER=podman`, the podman VM only sees host +> paths under `/Users/`. Extract the tarball under `$HOME` (not `/tmp` +> or `/private/tmp`), or boot0 will fail with a `statfs ... no such +> file or directory` error. `verify.sh` checks this and aborts early +> with a clearer message. + +```sh +tar xzf boot2-@ARCH@-@REV@.tar.gz +cd boot2-@ARCH@-@REV@ + +# Default: DRIVER=podman. Builds container images on first run. +./verify.sh + +# Or re-run inside the boot6-built kernel under qemu (closes the loop): +DRIVER=seed ./verify.sh +``` + +`verify.sh` does three things: + +1. Stages `src/` into `build/@ARCH@/src/` (the layout the boot stage + scripts expect). +2. Runs `boot/boot0.sh` … `boot/boot6.sh` in order under the selected + driver. +3. Hashes each artifact listed in `OUTPUT_MANIFEST.txt` and prints + `OK` / `DIFFER` / `MISSING` per row. + +Exit status is 0 iff every artifact matches. + +## Drivers + +| DRIVER | runtime | prereqs | +| ---------- | ----------------------------------------------- | ---------------------------------- | +| `podman` | each stage runs in a minimal container | `podman` (rootless ok), `qemu-user-static` for cross-arch | +| `seed` | each stage runs inside `seed-kernel` under qemu | one prior `DRIVER=podman` pass to mint the boot6 kernel; `qemu-system-@ARCH@` | + +The output manifest is **driver-agnostic**: the same artifacts must +hash identically under both drivers. The upstream repo's +`tests/seed-accept.sh` harness verifies this byte-equivalence. + +## Verifying input integrity + +```sh +# macOS: +shasum -a 256 -c INPUT_MANIFEST.txt +# Linux: +sha256sum -c INPUT_MANIFEST.txt +``` + +## Re-running just the diff + +If you've already built and just want to re-check the manifest: + +```sh +./verify.sh --check-only +``` + +## Useful env vars (passthrough to `boot/boot.sh`) + +| var | default | meaning | +| ---------------- | ------- | ---------------------------------- | +| `BOOT3_TIMEOUT` | 1800 | scheme1-driven boot3, seconds | +| `BOOT4_TIMEOUT` | 5400 | tcc1/tcc2/tcc3 self-host chain | +| `BOOT5_TIMEOUT` | 7200 | musl build | +| `BOOT6_TIMEOUT` | 1200 | seed-kernel link | +| `QEMU_MEM` | 3072M | guest RAM for `DRIVER=seed` | diff --git a/tools/release/verify.sh b/tools/release/verify.sh @@ -0,0 +1,124 @@ +#!/bin/sh +## verify.sh — drive boot0..boot6 off the bundled inputs and compare +## the outputs against OUTPUT_MANIFEST.txt. +## +## This script ships inside a boot2-@ARCH@-@REV@ release tarball. It +## stages the sealed src/ tree at build/@ARCH@/src/ (the layout every +## boot/bootN.sh expects), runs the chain end-to-end, then diffs each +## per-stage artifact's sha256 against the bundled manifest. +## +## Usage: +## ./verify.sh # build + verify (DRIVER=podman default) +## DRIVER=seed ./verify.sh # re-run inside the boot6-built kernel +## # (requires one prior DRIVER=podman pass) +## ./verify.sh --check-only # skip build; just diff existing outputs +## +## Env passthrough: BOOT3_TIMEOUT, BOOT4_TIMEOUT, BOOT5_TIMEOUT, +## BOOT6_TIMEOUT, QEMU_MEM — see boot/boot.sh --help. + +set -eu + +ARCH=@ARCH@ +KERNEL_NAME=@KERNEL_NAME@ +DRIVER=${DRIVER:-podman} + +CHECK_ONLY=0 +case "${1:-}" in + --check-only) CHECK_ONLY=1 ;; + -h|--help) + sed -n '2,18p' "$0" | sed 's/^## \{0,1\}//' + exit 0 + ;; + '') ;; + *) echo "verify.sh: unknown argument '$1' (try --help)" >&2; exit 2 ;; +esac + +case "$DRIVER" in + podman|seed) ;; + *) echo "[verify] unknown DRIVER=$DRIVER (expected podman|seed)" >&2; exit 2 ;; +esac + +ROOT=$(cd "$(dirname "$0")" && pwd) +cd "$ROOT" + +# Portable sha256. +if command -v sha256sum >/dev/null 2>&1; then + sha256() { sha256sum "$1" | awk '{print $1}'; } +else + sha256() { shasum -a 256 "$1" | awk '{print $1}'; } +fi + +# ── (0) macOS/podman preflight ──────────────────────────────────────── +# `podman machine` on macOS only shares /Users/ from the host. Extracting +# the tarball under /tmp (= /private/tmp) or anywhere outside /Users/ +# makes the bind mounts invisible to the VM and boot0 fails with a +# cryptic `Error: statfs ... no such file or directory`. Catch it here +# with a clearer message. +if [ "$CHECK_ONLY" = 0 ] && [ "$DRIVER" = podman ] && [ "$(uname -s)" = Darwin ]; then + case "$ROOT" in + /Users/*) ;; + *) cat >&2 <<EOF +[verify] this tarball was extracted to: +[verify] $ROOT +[verify] on macOS, the podman VM only shares /Users/ from the host. +[verify] Bind mounts under any other path (incl. /tmp, /private/tmp) +[verify] will fail at runtime. Re-extract somewhere under \$HOME and +[verify] run ./verify.sh from there. +EOF + exit 2 ;; + esac +fi + +# ── (1) stage sealed src tree into expected layout ──────────────────── +mkdir -p "build/$ARCH" +if [ ! -d "build/$ARCH/src" ]; then + echo "[verify] staging src/ -> build/$ARCH/src/" + cp -R src "build/$ARCH/src" +fi + +# ── (2) drive boot0..boot6 ──────────────────────────────────────────── +if [ "$CHECK_ONLY" = 0 ]; then + export DRIVER + for s in 0 1 2 3 4 5 6; do + echo "[verify] boot$s" + ./boot/boot$s.sh "$ARCH" + done +fi + +# ── (3) diff outputs vs OUTPUT_MANIFEST.txt ─────────────────────────── +BUILD_TREE=build/$ARCH/$DRIVER +echo "[verify] diff $BUILD_TREE vs OUTPUT_MANIFEST.txt" +fail=0 +ok=0 +miss=0 +while IFS= read -r line; do + case "$line" in + ''|\#*) continue ;; + esac + want=$(printf '%s' "$line" | awk '{print $1}') + rel=$(printf '%s' "$line" | awk '{print $2}') + f=$BUILD_TREE/$rel + if [ ! -e "$f" ]; then + printf 'MISSING %s\n' "$rel" + miss=$((miss + 1)) + fail=$((fail + 1)) + continue + fi + got=$(sha256 "$f") + if [ "$got" = "$want" ]; then + printf 'OK %s\n' "$rel" + ok=$((ok + 1)) + else + printf 'DIFFER %s\n' "$rel" + printf ' want %s\n' "$want" + printf ' got %s\n' "$got" + fail=$((fail + 1)) + fi +done < OUTPUT_MANIFEST.txt + +echo "[verify] ok=$ok missing=$miss differ=$((fail - miss)) (driver=$DRIVER)" +if [ "$fail" -gt 0 ]; then + echo "[verify] FAIL" >&2 + exit 1 +fi +echo "[verify] PASS"