Kernel build and image pipeline

This roadmap tracks the work needed for kit to build freestanding kernels from C and assembly sources and emit kernel images that can be passed to QEMU's direct loaders. It deliberately does not cover VM execution, bootloader generation, UEFI application layout, or kit-provided startup code. Kernel startup, privilege-mode entry, page-table setup, stack setup, TLS setup, and boot-protocol compliance are the kernel author's responsibility.

Related: ../DRIVER.md, ../LINK.md, ../OBJ.md, ../RUNTIME.md, LINKER-COMPAT.md, PORT.md.

Scope

The supported path is:

C / asm sources + objects + archives
  -> kit build-obj / build-exe
  -> freestanding static kernel ELF
  -> kit image / objcopy
  -> ELF / flat binary / ROM-style payload / section-concatenated payload

The first targets are freestanding ELF:

• x86_64-none-elf
• aarch64-none-elf
• riscv64-none-elf
• riscv32-none-elf

The output artifacts should be usable with QEMU features such as -kernel, -bios, or -device loader,file=..., depending on the target machine and the kernel's own entry contract. Kit should not decide or implement the boot protocol.

Current baseline

Useful pieces already exist:

• build-obj compiles C / asm / toy / wasm sources into objects, and can combine multiple source objects with ld -r.
• build-exe compiles a source set and links it with object/archive inputs.
• Freestanding triples resolve to non-PIE ELF/WASM targets by default.
• build-* already accepts common freestanding and link flags such as -ffreestanding, -nostdinc, -nostdlib, -nodefaultlibs, -nostartfiles, -static, -pie, -no-pie, -mcmodel=..., -T, -e, -Wl,..., --build-id=..., -ffunction-sections, and -fdata-sections.
• The linker has a structured linker-script subset with section placement, symbol assignment, /DISCARD/, and KEEP roots for --gc-sections.
• The runtime provides freestanding headers and compiler-runtime-style support, but no crt0.
• objcopy can transform object files and rewrite sections/symbols, but does not yet expose raw binary image output.

The desired user-facing build shape stays on the existing tools:

kit build-exe -target x86_64-none-elf \
  -ffreestanding -nostdlib -static -no-pie \
  -mcmodel=kernel -mno-red-zone \
  -ffunction-sections -fdata-sections \
  -T kernel.ld -e _start \
  -Wl,--gc-sections \
  --map kernel.map \
  --symbols kernel.sym \
  -o kernel.elf \
  boot.S kernel.c mm.c

Compile and build-driver support

Do not add a separate kit kernel compile command. Keep build-obj and build-exe as the compile/link front doors, and make their flag surface complete enough for kernel authors.

Required flag support and behavior:

• -ffreestanding / -fhosted: select freestanding vs hosted assumptions, including whether sysroot-hosted profiles may be engaged.
• -nostdinc: suppress all implicit non-resource include paths while still allowing explicit -I / -isystem.
• -nostdlib, -nodefaultlibs, -nostartfiles: precisely control runtime archive and hosted CRT/libc insertion. No startup object is ever invented by kit for a freestanding kernel.
• -static, -no-pie, -fno-pic, -fno-pie: produce static non-PIE kernel code and an ET_EXEC-style image unless the user deliberately opts into PIC/PIE.
• -mcmodel=...: keep existing model selection and add kernel-relevant aliases where the target backend has meaningful behavior.
• -mno-red-zone: disable the x86_64 SysV red zone for kernel code. This must affect backend frame selection, not merely be accepted as syntax.
• -mgeneral-regs-only or equivalent target-feature spelling: give kernels a straightforward way to prevent accidental SIMD/FP codegen where an ABI or privilege-mode context does not save those registers.
• -fno-builtin: accept in build-* and ensure the C frontend does not turn freestanding source into calls or assumptions the kernel did not request.
• -fno-stack-protector / -fstack-protector*: either implement the supported subset or reject unsupported modes explicitly. Silently ignoring stack protector policy is not acceptable for kernels.
• -ffunction-sections and -fdata-sections: compose with linker --gc-sections and script KEEP.
• --group: continue to scope include/define/language/frontend flags to source subsets; do not let link-wide kernel policy drift into per-source groups.

Driver parity to improve:

• Accept common direct linker flags in build-exe where ld already accepts them, instead of requiring every flag to pass through -Wl,.
• Keep cc, ld, and build-exe behavior aligned by routing shared policy through driver/lib/link_flags.* and driver/lib/target.*.
• For freestanding executable links, default diagnostics should be strict: unresolved symbols and dynamic-link artifacts should be errors unless the user explicitly requests an escape hatch.

Linker work

Linker-script subset

Kernel links need a larger structured GNU-ld-compatible subset. The goal is not to interpret arbitrary linker scripts blindly, but to support the constructs needed for deterministic kernel memory layouts with precise diagnostics.

Add support for:

• MEMORY with ORIGIN, LENGTH, attributes, and region-overflow diagnostics.
• Output section placement into memory regions with > REGION.
• Load-memory placement with AT(expr) and AT> REGION, so VMA/LMA split kernels can be represented.
• PHDRS, :phdr, FLAGS(...), and FILEHDR / PHDRS segment attributes, so kernels can control program headers and segment permissions.
• PROVIDE, PROVIDE_HIDDEN, and HIDDEN.
• ASSERT(expr, "message").
• EXTERN(symbol) as a GC root and undefined-symbol declaration.
• Richer input section patterns: *(.text .text.*), file-qualified patterns, and EXCLUDE_FILE.
• Section fills: =0x... and FILL(...).
• Alignment and address helpers: ALIGN, SUBALIGN, BLOCK, ADDR, LOADADDR, SIZEOF, SIZEOF_HEADERS, DEFINED, and ABSOLUTE.
• OUTPUT_ARCH and OUTPUT_FORMAT as validation directives initially. They do not need to drive target selection in the first pass, but mismatches should be reported clearly.

Existing KEEP(...) support must continue to interact correctly with --gc-sections.

Link output side files

Add linker-produced side outputs, available from both ld and build-exe:

• --map FILE: write a deterministic link map.
• --symbols FILE: write post-link absolute symbols.
• --symbols-format=nm|json: start with an nm-like text format; JSON can land once the data model is stable.
• --cref FILE: optional cross-reference table.
• --print-memory-usage: summarize MEMORY regions once linker-script memory regions exist.

The link map should include:

• target triple and output kind,
• entry symbol and address,
• memory regions and usage,
• program headers / segments,
• output sections with VMA, LMA, file offset, size, and alignment,
• input object/archive-member contributions,
• linker-defined symbols,
• discarded sections,
• unresolved symbols,
• section-GC roots and discarded-reason details where practical.

Linker flags and policy

Add or normalize:

• --no-undefined: reject unresolved symbols for executable/freestanding links, not only shared-library output.
• --allow-undefined: explicit escape hatch.
• --defsym name=expr.
• --section-start=.name=addr.
• -Ttext, -Tdata, and -Tbss through build-exe as well as ld.
• --orphan-handling=place|warn|error|discard.
• --fatal-warnings.

Freestanding kernel validation should reject, by default:

• DSO inputs,
• dynamic interpreter paths,
• dynamic sections and PLT/GOT imports,
• unresolved relocations,
• missing entry symbols when -e or ENTRY(...) names one,
• target/object-format mismatches across inputs.

Image command

Add a new kit image command for kernel-image emission. Also add a minimal objcopy -O binary path for compatibility, backed by the same lower-level image emitter.

objcopy should remain an object transformer:

kit objcopy -O binary kernel.elf kernel.bin

kit image should own image-building policy:

kit image --format bin kernel.elf -o kernel.bin

kit image --format bin --from segments --segment PT_LOAD \
  --addr paddr --base 0x80000000 --fill 0x00 --align 4096 \
  --pad-to 2M --max-size 8M \
  --metadata kernel.image.json \
  kernel.elf -o kernel.bin

Image formats

• elf: copy, normalize, strip, or split-debug an existing linked ELF.
• bin: flat binary derived from loadable segments or selected sections.
• rom: flat fixed-size binary with fill, padding, max-size checks, and later optional checksum hooks.
• sections: concatenate explicitly named sections in user-specified order.

Optional later embedded formats, such as Intel HEX, S-record, or UF2, are out of the first kernel-focused pass.

Flat kernel Image (arm64 / riscv64)

QEMU's -kernel path on arm64 and riscv consumes the flat Image format: a raw loadable binary prefixed with a fixed 64-byte header the kernel's loader reads to place and size the image. The first 32 bytes are common to both arches:

offset	field	notes
0	code0 (u32)	first instruction (branch to entry; "MZ" low half if EFI)
4	code1 (u32)	second instruction
8	text_offset	u64 LE — load offset from a 2 MiB-aligned base
16	image_size	u64 LE — effective image size including BSS
24	flags	u64 LE — bit 0 endianness; arm64 bits 1-2 page size, bit 3 placement

The tails differ. arm64: three reserved u64s, magic = "ARM\x64" (0x644d5241) at offset 56, then a u32 PE-offset slot. riscv64: a version u32 (currently 0x2), reserved words, a deprecated "RISCV\0\0\0" magic at 48, and magic2 = "RSC\x05" (0x05435352) at 56.

There are two ways to support this, and they have different costs:

• Author-owned header (pass-through) — already supported. In the Linux-native workflow the kernel's own startup code emits the 64-byte header (head.S places code0/code1 and declares image_size from linker symbols). This matches the project stance that kernel startup is the author's responsibility. kit needs nothing new: kit image --format bin / objcopy -O binary copy PT_LOAD bytes verbatim, so the leading header survives byte-exact and the result is a loadable Image. Only a doc note and a fixture that asserts the header bytes and size are required.

• Kit-synthesized header — small add, the part worth doing. The one field that is awkward for authors is image_size, the in-memory footprint including BSS, which kit already knows: KitObjSegInfo.vsize gives each loadable segment's memory size (the current emitter lays out file_size only and ignores it). So the add is: a ~64-byte header emitter, a new --format arm64-image / --format riscv-image (or an --image-header=arm64|riscv modifier on --format bin), and vsize-based computation of image_size and the memory span.

Because flags encodes boot semantics (endianness, page size, placement), header synthesis is an explicit opt-in, never a default. Those fields are surfaced as explicit options (e.g. --image-endian, --image-page-size, --image-text-offset) rather than invented; kit fills magic/version and image_size deterministically and does not choose a boot policy.

Image flags

Selection:

• --from=segments|sections
• --segment=PT_LOAD (repeatable)
• --only-section NAME (repeatable)
• --remove-section NAME (repeatable)
• --section NAME (repeatable; explicit order for --format sections)

Addressing:

• --addr=vaddr|paddr|lma
• --base ADDR
• --bias N

Holes and layout:

• --fill BYTE
• --fail-on-holes
• --max-hole SIZE
• --align N
• --pad-to SIZE
• --max-size SIZE

ELF/debug:

• --strip-debug
• --split-debug FILE
• --keep-symbols

Validation:

• --require-entry
• --require-symbol NAME (repeatable)
• --require-section NAME (repeatable)
• --no-dynamic

Reporting:

• --metadata FILE: write a deterministic JSON sidecar containing target, object format, entry, build id, selected segments/sections, source ranges, output ranges, base/bias/fill policy, and warnings.

Image semantics

Segment-based flat image emission should:

• read loadable ranges from program headers,
• sort by selected address kind,
• detect overlaps,
• fill or reject holes according to policy,
• derive output offsets from base / lowest selected address,
• preserve bytes exactly as they would be loaded,
• include NOBITS memory ranges in metadata but not necessarily in output bytes unless a selected format requires padding.

Section-based emission should:

• operate on named sections in declared order,
• reject missing sections unless a permissive option is added later,
• concatenate section bytes exactly,
• make address metadata explicit so users do not confuse section concatenation with a loadable memory image.

ROM-style emission should:

• require an explicit size or --pad-to,
• fill unused bytes deterministically,
• fail when selected payload bytes exceed the requested size,
• leave target-specific checksums or reset-vector conventions as future, explicit options.

Initramfs and archive packaging

The kernel-boot pipeline needs one packaging format kit does not yet emit: the SVR4 newc cpio archive the Linux kernel unpacks as its initramfs. An initramfs is a cpio -H newc archive (magic 070701, or 070702 for the CRC variant), optionally compressed, that the kernel's built-in extractor reads at boot; the early-microcode convention is just an uncompressed cpio concatenated ahead of the compressed main archive. This is an archive format, not a boot protocol — the direct analogue of the existing ar and tar paths — so it belongs in kit's byte-utility tool family, not the image emitter. kit packages and inspects the archive; it does not build, mount, or boot it, and does not invent its contents.

Add a kit cpio tool, reusing the src/dist/tar.c patterns and the public kit/compress.h codecs:

• newc (SVR4 "portable") format only — the format the kernel requires. The legacy bin/odc cpio formats are out of scope.
• Create from a directory or explicit file list, list (-t), and extract (-i), with deterministic ordering, normalized mode/uid/gid/mtime, and the closing TRAILER!!! record. Regular files, directories, and symlinks first; special/device nodes (which need explicit major/minor) can come later via a manifest if a use case appears.
• Concatenation: build and accept already-concatenated archives so early-init cpio segments can be assembled and inspected.
• Compression as a flag on kit cpio (e.g. --compress=gzip|lz4, with short -z / --lz4) that build-then-compresses in one step. An initramfs is just a compressed newc archive, so the flag is the whole story — no separate initramfs tool is warranted. gzip and lz4 only, the two initramfs compressors kit already ships; zstd and xz are out of scope, with a clear diagnostic rather than pretend support. On read (-t / -i), -d and auto-detection let a compressed archive round-trip with no separate decompress step.

Gate the tool in driver/main.c (KIT_TOOL_CPIO_ENABLED) alongside the other archive utilities.

Implementation shape

Add a shared image-emission layer rather than burying policy in objcopy:

driver/cmd/image.c        CLI policy for kit image
driver/cmd/objcopy.c      simple -O binary compatibility path
include/kit/image.h       public image-emission API, if we want embedders to use it
src/api/image.c           public wrapper
src/obj/image.c           object/ELF-to-image implementation

The image API should consume already-read object bytes or an opened KitObjFile view, plus explicit options. It should not read the filesystem directly and should not run QEMU or inspect host bootloader installs.

Map/symbol side outputs should be linker-owned rather than image-owned. The image metadata file can reference link-map facts, but it should be a report about the image transform, not a replacement for --map.

Phasing

1. Raw binary baseline

   • Add objcopy -O binary.
   • Add kit image --format bin --from segments.
   • Support --base, --fill, --fail-on-holes, --pad-to, --max-size.
   • Add focused ELF fixtures for x64, aa64, rv64, and rv32.

2. Build/link parity for kernels

   • Add missing build-exe direct flag parity with ld.
   • Add -mno-red-zone, -mgeneral-regs-only, builtin policy, and stack protector policy.
   • Add strict freestanding undefined/dynamic-artifact diagnostics.

3. Link map and symbols

   • Add --map FILE.
   • Add --symbols FILE.
   • Add deterministic tests for section layout, symbols, and discarded sections.

4. Script growth

   • Add MEMORY, region placement, VMA/LMA split, and region overflow checks.
   • Add PHDRS once memory regions are stable.
   • Add ASSERT, PROVIDE, EXTERN, richer input patterns, and orphan handling.

5. Image formats beyond bin

   • Add --format sections.
   • Add --format rom.
   • Add --metadata FILE.
   • Add --format elf normalization/strip/split-debug behavior if it proves cleaner than routing those cases through objcopy and strip.

Acceptance criteria

For each first-pass freestanding target, kit should be able to:

• compile a kernel source set containing C and assembly with build-exe,
• link it with a kernel-owned startup object and linker script,
• emit a static freestanding ELF with a deterministic layout,
• produce a link map and absolute-symbol side file,
• convert the ELF to a flat binary image,
• validate that the ELF/image has no accidental dynamic-loader dependencies,
• reproduce byte-identical outputs from identical inputs and options.

The validation suite should stay targeted:

• one small kernel-link fixture per architecture,
• one script-layout fixture per linker-script feature,
• one image-conversion fixture per image format and hole policy,
• negative tests for unresolved symbols, region overflow, dynamic artifacts, and overlapping image ranges.

Remaining work

Phases 1-5 are landed. Phases 1-4 (flag parity, undefined-symbol policy, --map/--symbols, the linker-script parse surface, kit image --format bin / objcopy -O binary) plus all of the items below shipped; an adversarial review of the integrated diff found and fixed a further set of correctness bugs (coalesced-PT_LOAD bss/perms, --defsym ordering, exact --section-start addressing, NOLOAD relocations, memory-usage accounting, --cref imports, image metadata accuracy, and cc/ld report parity). The checklist below is closed out.

Image formats beyond bin (Phase 5)

• --format rom: fixed-size flat binary; requires --pad-to, deterministic fill, fails when payload exceeds the size.
• --format sections: concatenate --section NAME (repeatable) in declared order, reject missing sections, address metadata made explicit (new section-iteration path in src/obj/image.c).
• --format elf: evaluated and deliberately delegated to objcopy + strip — a standalone implementation would duplicate the object-rewrite / strip / split-debug machinery 1:1. The emitter returns a clear diagnostic pointing to those tools rather than half-building a parallel path.
• --metadata FILE: deterministic JSON sidecar (target, object format, entry, build id, selection, source/output ranges, policy, warnings); the selection field mirrors the emitter's actual segment policy.
• Image selection/validation flags: --only-section / --remove-section / --section; --strip-debug / --split-debug / --keep-symbols (gated to --format elf); --require-entry / --require-symbol / --require-section / --no-dynamic.

Linker-script layout (now honored at layout)

• NOLOAD with PROGBITS content (forces NOBITS / no file bytes; relocations into a NOLOAD section are skipped, not written through a null buffer).
• Inter-section . assignments apply at their textual position (sequence- stamped assignments interleaved with the section walk).
• Multi-byte fills (=0x12345678 / FILL(...)) lay a repeating big-endian pattern.
• Multiple sections sharing one :phdr coalesce into one PT_LOAD; a section listing several phdrs appears under each (with R/W/X perms union).
• Recursion-depth guard in script expression parse and eval.

Linker flags and policy

• --defsym name=expr (can now satisfy an otherwise-undefined reference) and --section-start=.name=addr (lands at the exact requested vaddr).
• -Tdata / -Tbss through build-exe and ld.
• --orphan-handling=place|warn|error|discard and --fatal-warnings.
• --cref FILE (includes imported/undefined symbols) and --print-memory-usage (per-MEMORY-region usage, overflow-safe for high-half regions).
• Map completeness: LMA, discarded sections, and unresolved symbols in --map; input paths normalized to basenames (no absolute-path leak).

Freestanding strict validation (Phase 2 gap)

• Reject dynamic-interpreter paths, dynamic sections, and PLT/GOT imports.
• Reject cross-input target / object-format mismatches.
• Policy surfaced through build-exe, not just ld (both honor the same freestanding_strict trigger).

Tests and fixtures

• Per-arch kernel-link fixtures for x86_64-none-elf, aarch64-none-elf, riscv64-none-elf, and riscv32-none-elf (build → map/symbols → image → validate, reproducible).
• Byte-golden --map / --symbols fixtures (deterministic, committed goldens for a pinned link).
• Negative tests: region overflow, discarded sections, dynamic artifacts, NOLOAD-with-PROGBITS, and cross-arch freestanding rejection.

Newly scoped: initramfs and Image packaging

Added on top of the closed-out Phase 1-5 work; not yet started.

Initramfs / cpio (archive packaging, sibling to ar) — landed as kit cpio. The newc codec lives driver-local in driver/cmd/cpio.c (only the tool consumes it; the driver has no -Isrc, so it mirrors tar.c's stateless append/finish/ iter shape rather than living in the dist subsystem). Create needed two additive host shims: driver_readlink (read symlink targets) and driver_path_lstat (no-follow operand classification + the source executable bit). Bidirectionally interop-verified against host bsdcpio.

• kit cpio (newc/SVR4 only, magic 070701/070702): create (-o) / list (-t) / extract (-i) with deterministic ordering (members sorted by path, a sorted DFS so each directory precedes its children) and normalized metadata (uid/gid 0, mtime 0, sequential inode, mode = type | perms keyed on the source exec bit), closing TRAILER!!! + a 512-byte tail pad. Regular files, directories, and symlinks; ../absolute names refused on both create and extract.
• Archive concatenation for early-init segments: the reader continues past a TRAILER!!!, skips inter-segment zero padding, and resumes on the next 070701/070702 magic (trailing non-cpio data is noted, not fatal). Building a concatenation is shell cat of 512-padded archives.
• Compression as a kit cpio flag (--compress=gzip|lz4, -z/--lz4; -d + always-on auto-detect on read) via the public kit/compress.h codecs — gzip + lz4 only, with a specific diagnostic for a zstd/xz magic or --compress=zstd|xz. An initramfs is just a compressed newc archive, so no separate initramfs tool.
• Tool gating in driver/main.c (KIT_TOOL_CPIO_ENABLED) + the test/cpio/run.sh round-trip fixture (pack → list → extract, byte- deterministic, newc-shape and 512-pad asserts, gzip/lz4/crc/concat coverage, security + usage negatives; optional KIT_CPIO_TEST_HOST=1 cross-check against the host cpio).

Flat kernel Image header (arm64 / riscv64) — landed via the --image-header modifier on --format bin/rom (not separate formats); it overlays the first 64 bytes of the first loadable segment rather than prepending, so the entry branch (code0/code1) is preserved and the output stays the same size as a plain bin.

• Pass-through: an author-emitted 64-byte header survives kit image --format bin byte-exact (magic + image_size preserved); covered by the image-hdr-pt-* fixtures.
• --image-header=arm64|riscv|auto (AUTO infers the arch from the object's machine): synthesizes the 64-byte header; image_size is the in-memory span including BSS, computed from KitObjSegInfo.vsize over all selected loadable segments (BSS-only file_size==0 segments included — the byte ranges skip them, the span must not).
• Explicit boot-semantic options (--image-endian, --image-page-size [arm64-only], --image-text-offset); magic/version filled deterministically; no invented boot policy. The sub-options error without --image-header.
• Per-arch fixtures (aarch64 / riscv64) in test/tools/run.sh: synthesized header is self-consistent (magic at 56, riscv version 2 at 32, image_size counts the BSS), deterministic, and the entry branch is preserved.