commit fcdfd6add1105d14982e257898aae2672d94cfbf
parent b2ebc0984197454989a89dea2ea161ae00896d4f
Author: Ryan Sepassi <rsepassi@gmail.com>
Date: Tue, 16 Jun 2026 11:24:06 -0700
Document kernel image build roadmap
Diffstat:
2 files changed, 374 insertions(+), 0 deletions(-)
diff --git a/doc/plan/KERNEL.md b/doc/plan/KERNEL.md
@@ -0,0 +1,373 @@
+# Kernel build and image pipeline
+
+This roadmap tracks the work needed for kit to build freestanding kernels from
+C and assembly sources and emit kernel images that can be passed to QEMU's
+direct loaders. It deliberately does not cover VM execution, bootloader
+generation, UEFI application layout, or kit-provided startup code. Kernel
+startup, privilege-mode entry, page-table setup, stack setup, TLS setup, and
+boot-protocol compliance are the kernel author's responsibility.
+
+Related: [../DRIVER.md](../DRIVER.md), [../LINK.md](../LINK.md),
+[../OBJ.md](../OBJ.md), [../RUNTIME.md](../RUNTIME.md),
+[LINKER-COMPAT.md](LINKER-COMPAT.md), [PORT.md](PORT.md).
+
+## Scope
+
+The supported path is:
+
+```
+C / asm sources + objects + archives
+ -> kit build-obj / build-exe
+ -> freestanding static kernel ELF
+ -> kit image / objcopy
+ -> ELF / flat binary / ROM-style payload / section-concatenated payload
+```
+
+The first targets are freestanding ELF:
+
+- `x86_64-none-elf`
+- `aarch64-none-elf`
+- `riscv64-none-elf`
+- `riscv32-none-elf`
+
+The output artifacts should be usable with QEMU features such as `-kernel`,
+`-bios`, or `-device loader,file=...`, depending on the target machine and the
+kernel's own entry contract. Kit should not decide or implement the boot
+protocol.
+
+## Current baseline
+
+Useful pieces already exist:
+
+- `build-obj` compiles C / asm / toy / wasm sources into objects, and can combine
+ multiple source objects with `ld -r`.
+- `build-exe` compiles a source set and links it with object/archive inputs.
+- Freestanding triples resolve to non-PIE ELF/WASM targets by default.
+- `build-*` already accepts common freestanding and link flags such as
+ `-ffreestanding`, `-nostdinc`, `-nostdlib`, `-nodefaultlibs`,
+ `-nostartfiles`, `-static`, `-pie`, `-no-pie`, `-mcmodel=...`, `-T`, `-e`,
+ `-Wl,...`, `--build-id=...`, `-ffunction-sections`, and `-fdata-sections`.
+- The linker has a structured linker-script subset with section placement,
+ symbol assignment, `/DISCARD/`, and `KEEP` roots for `--gc-sections`.
+- The runtime provides freestanding headers and compiler-runtime-style support,
+ but no `crt0`.
+- `objcopy` can transform object files and rewrite sections/symbols, but does
+ not yet expose raw binary image output.
+
+The desired user-facing build shape stays on the existing tools:
+
+```
+kit build-exe -target x86_64-none-elf \
+ -ffreestanding -nostdlib -static -no-pie \
+ -mcmodel=kernel -mno-red-zone \
+ -ffunction-sections -fdata-sections \
+ -T kernel.ld -e _start \
+ -Wl,--gc-sections \
+ --map kernel.map \
+ --symbols kernel.sym \
+ -o kernel.elf \
+ boot.S kernel.c mm.c
+```
+
+## Compile and build-driver support
+
+Do not add a separate `kit kernel compile` command. Keep `build-obj` and
+`build-exe` as the compile/link front doors, and make their flag surface complete
+enough for kernel authors.
+
+Required flag support and behavior:
+
+- `-ffreestanding` / `-fhosted`: select freestanding vs hosted assumptions,
+ including whether sysroot-hosted profiles may be engaged.
+- `-nostdinc`: suppress all implicit non-resource include paths while still
+ allowing explicit `-I` / `-isystem`.
+- `-nostdlib`, `-nodefaultlibs`, `-nostartfiles`: precisely control runtime
+ archive and hosted CRT/libc insertion. No startup object is ever invented by
+ kit for a freestanding kernel.
+- `-static`, `-no-pie`, `-fno-pic`, `-fno-pie`: produce static non-PIE kernel
+ code and an ET_EXEC-style image unless the user deliberately opts into PIC/PIE.
+- `-mcmodel=...`: keep existing model selection and add kernel-relevant aliases
+ where the target backend has meaningful behavior.
+- `-mno-red-zone`: disable the x86_64 SysV red zone for kernel code. This must
+ affect backend frame selection, not merely be accepted as syntax.
+- `-mgeneral-regs-only` or equivalent target-feature spelling: give kernels a
+ straightforward way to prevent accidental SIMD/FP codegen where an ABI or
+ privilege-mode context does not save those registers.
+- `-fno-builtin`: accept in `build-*` and ensure the C frontend does not turn
+ freestanding source into calls or assumptions the kernel did not request.
+- `-fno-stack-protector` / `-fstack-protector*`: either implement the supported
+ subset or reject unsupported modes explicitly. Silently ignoring stack
+ protector policy is not acceptable for kernels.
+- `-ffunction-sections` and `-fdata-sections`: compose with linker
+ `--gc-sections` and script `KEEP`.
+- `--group`: continue to scope include/define/language/frontend flags to source
+ subsets; do not let link-wide kernel policy drift into per-source groups.
+
+Driver parity to improve:
+
+- Accept common direct linker flags in `build-exe` where `ld` already accepts
+ them, instead of requiring every flag to pass through `-Wl,`.
+- Keep `cc`, `ld`, and `build-exe` behavior aligned by routing shared policy
+ through `driver/lib/link_flags.*` and `driver/lib/target.*`.
+- For freestanding executable links, default diagnostics should be strict:
+ unresolved symbols and dynamic-link artifacts should be errors unless the user
+ explicitly requests an escape hatch.
+
+## Linker work
+
+### Linker-script subset
+
+Kernel links need a larger structured GNU-ld-compatible subset. The goal is not
+to interpret arbitrary linker scripts blindly, but to support the constructs
+needed for deterministic kernel memory layouts with precise diagnostics.
+
+Add support for:
+
+- `MEMORY` with `ORIGIN`, `LENGTH`, attributes, and region-overflow diagnostics.
+- Output section placement into memory regions with `> REGION`.
+- Load-memory placement with `AT(expr)` and `AT> REGION`, so VMA/LMA split
+ kernels can be represented.
+- `PHDRS`, `:phdr`, `FLAGS(...)`, and `FILEHDR` / `PHDRS` segment attributes,
+ so kernels can control program headers and segment permissions.
+- `PROVIDE`, `PROVIDE_HIDDEN`, and `HIDDEN`.
+- `ASSERT(expr, "message")`.
+- `EXTERN(symbol)` as a GC root and undefined-symbol declaration.
+- Richer input section patterns: `*(.text .text.*)`, file-qualified patterns,
+ and `EXCLUDE_FILE`.
+- Section fills: `=0x...` and `FILL(...)`.
+- Alignment and address helpers: `ALIGN`, `SUBALIGN`, `BLOCK`, `ADDR`,
+ `LOADADDR`, `SIZEOF`, `SIZEOF_HEADERS`, `DEFINED`, and `ABSOLUTE`.
+- `OUTPUT_ARCH` and `OUTPUT_FORMAT` as validation directives initially. They do
+ not need to drive target selection in the first pass, but mismatches should be
+ reported clearly.
+
+Existing `KEEP(...)` support must continue to interact correctly with
+`--gc-sections`.
+
+### Link output side files
+
+Add linker-produced side outputs, available from both `ld` and `build-exe`:
+
+- `--map FILE`: write a deterministic link map.
+- `--symbols FILE`: write post-link absolute symbols.
+- `--symbols-format=nm|json`: start with an nm-like text format; JSON can land
+ once the data model is stable.
+- `--cref FILE`: optional cross-reference table.
+- `--print-memory-usage`: summarize `MEMORY` regions once linker-script memory
+ regions exist.
+
+The link map should include:
+
+- target triple and output kind,
+- entry symbol and address,
+- memory regions and usage,
+- program headers / segments,
+- output sections with VMA, LMA, file offset, size, and alignment,
+- input object/archive-member contributions,
+- linker-defined symbols,
+- discarded sections,
+- unresolved symbols,
+- section-GC roots and discarded-reason details where practical.
+
+### Linker flags and policy
+
+Add or normalize:
+
+- `--no-undefined`: reject unresolved symbols for executable/freestanding links,
+ not only shared-library output.
+- `--allow-undefined`: explicit escape hatch.
+- `--defsym name=expr`.
+- `--section-start=.name=addr`.
+- `-Ttext`, `-Tdata`, and `-Tbss` through `build-exe` as well as `ld`.
+- `--orphan-handling=place|warn|error|discard`.
+- `--fatal-warnings`.
+
+Freestanding kernel validation should reject, by default:
+
+- DSO inputs,
+- dynamic interpreter paths,
+- dynamic sections and PLT/GOT imports,
+- unresolved relocations,
+- missing entry symbols when `-e` or `ENTRY(...)` names one,
+- target/object-format mismatches across inputs.
+
+## Image command
+
+Add a new `kit image` command for kernel-image emission. Also add a minimal
+`objcopy -O binary` path for compatibility, backed by the same lower-level image
+emitter.
+
+`objcopy` should remain an object transformer:
+
+```
+kit objcopy -O binary kernel.elf kernel.bin
+```
+
+`kit image` should own image-building policy:
+
+```
+kit image --format bin kernel.elf -o kernel.bin
+
+kit image --format bin --from segments --segment PT_LOAD \
+ --addr paddr --base 0x80000000 --fill 0x00 --align 4096 \
+ --pad-to 2M --max-size 8M \
+ --metadata kernel.image.json \
+ kernel.elf -o kernel.bin
+```
+
+### Image formats
+
+- `elf`: copy, normalize, strip, or split-debug an existing linked ELF.
+- `bin`: flat binary derived from loadable segments or selected sections.
+- `rom`: flat fixed-size binary with fill, padding, max-size checks, and later
+ optional checksum hooks.
+- `sections`: concatenate explicitly named sections in user-specified order.
+
+Optional later embedded formats, such as Intel HEX, S-record, or UF2, are out of
+the first kernel-focused pass.
+
+### Image flags
+
+Selection:
+
+- `--from=segments|sections`
+- `--segment=PT_LOAD` (repeatable)
+- `--only-section NAME` (repeatable)
+- `--remove-section NAME` (repeatable)
+- `--section NAME` (repeatable; explicit order for `--format sections`)
+
+Addressing:
+
+- `--addr=vaddr|paddr|lma`
+- `--base ADDR`
+- `--bias N`
+
+Holes and layout:
+
+- `--fill BYTE`
+- `--fail-on-holes`
+- `--max-hole SIZE`
+- `--align N`
+- `--pad-to SIZE`
+- `--max-size SIZE`
+
+ELF/debug:
+
+- `--strip-debug`
+- `--split-debug FILE`
+- `--keep-symbols`
+
+Validation:
+
+- `--require-entry`
+- `--require-symbol NAME` (repeatable)
+- `--require-section NAME` (repeatable)
+- `--no-dynamic`
+
+Reporting:
+
+- `--metadata FILE`: write a deterministic JSON sidecar containing target,
+ object format, entry, build id, selected segments/sections, source ranges,
+ output ranges, base/bias/fill policy, and warnings.
+
+### Image semantics
+
+Segment-based flat image emission should:
+
+- read loadable ranges from program headers,
+- sort by selected address kind,
+- detect overlaps,
+- fill or reject holes according to policy,
+- derive output offsets from `base` / lowest selected address,
+- preserve bytes exactly as they would be loaded,
+- include NOBITS memory ranges in metadata but not necessarily in output bytes
+ unless a selected format requires padding.
+
+Section-based emission should:
+
+- operate on named sections in declared order,
+- reject missing sections unless a permissive option is added later,
+- concatenate section bytes exactly,
+- make address metadata explicit so users do not confuse section concatenation
+ with a loadable memory image.
+
+ROM-style emission should:
+
+- require an explicit size or `--pad-to`,
+- fill unused bytes deterministically,
+- fail when selected payload bytes exceed the requested size,
+- leave target-specific checksums or reset-vector conventions as future,
+ explicit options.
+
+## Implementation shape
+
+Add a shared image-emission layer rather than burying policy in `objcopy`:
+
+```
+driver/cmd/image.c CLI policy for kit image
+driver/cmd/objcopy.c simple -O binary compatibility path
+include/kit/image.h public image-emission API, if we want embedders to use it
+src/api/image.c public wrapper
+src/obj/image.c object/ELF-to-image implementation
+```
+
+The image API should consume already-read object bytes or an opened `KitObjFile`
+view, plus explicit options. It should not read the filesystem directly and
+should not run QEMU or inspect host bootloader installs.
+
+Map/symbol side outputs should be linker-owned rather than image-owned. The
+image metadata file can reference link-map facts, but it should be a report about
+the image transform, not a replacement for `--map`.
+
+## Phasing
+
+1. **Raw binary baseline**
+ - Add `objcopy -O binary`.
+ - Add `kit image --format bin --from segments`.
+ - Support `--base`, `--fill`, `--fail-on-holes`, `--pad-to`, `--max-size`.
+ - Add focused ELF fixtures for x64, aa64, rv64, and rv32.
+
+2. **Build/link parity for kernels**
+ - Add missing `build-exe` direct flag parity with `ld`.
+ - Add `-mno-red-zone`, `-mgeneral-regs-only`, builtin policy, and stack
+ protector policy.
+ - Add strict freestanding undefined/dynamic-artifact diagnostics.
+
+3. **Link map and symbols**
+ - Add `--map FILE`.
+ - Add `--symbols FILE`.
+ - Add deterministic tests for section layout, symbols, and discarded
+ sections.
+
+4. **Script growth**
+ - Add `MEMORY`, region placement, VMA/LMA split, and region overflow checks.
+ - Add `PHDRS` once memory regions are stable.
+ - Add `ASSERT`, `PROVIDE`, `EXTERN`, richer input patterns, and orphan
+ handling.
+
+5. **Image formats beyond bin**
+ - Add `--format sections`.
+ - Add `--format rom`.
+ - Add `--metadata FILE`.
+ - Add `--format elf` normalization/strip/split-debug behavior if it proves
+ cleaner than routing those cases through `objcopy` and `strip`.
+
+## Acceptance criteria
+
+For each first-pass freestanding target, kit should be able to:
+
+- compile a kernel source set containing C and assembly with `build-exe`,
+- link it with a kernel-owned startup object and linker script,
+- emit a static freestanding ELF with a deterministic layout,
+- produce a link map and absolute-symbol side file,
+- convert the ELF to a flat binary image,
+- validate that the ELF/image has no accidental dynamic-loader dependencies,
+- reproduce byte-identical outputs from identical inputs and options.
+
+The validation suite should stay targeted:
+
+- one small kernel-link fixture per architecture,
+- one script-layout fixture per linker-script feature,
+- one image-conversion fixture per image format and hole policy,
+- negative tests for unresolved symbols, region overflow, dynamic artifacts, and
+ overlapping image ranges.
diff --git a/doc/plan/README.md b/doc/plan/README.md
@@ -18,6 +18,7 @@ shrinks to whatever remains open.
| [LEX-PP-API.md](LEX-PP-API.md) | Redesigning the lexer -> preprocessor -> parser boundary around lean tokens, lazy spelling/location materialization, and slot-based handoffs for a high-performance lexer/preprocessor rewrite without frontend layer fusion. | [../FRONTENDS.md](../FRONTENDS.md) |
| [LINKER.md](LINKER.md) | Incremental linking: the file-based object-link redesign and remaining non-ELF format coverage. | [../LINK.md](../LINK.md) |
| [LINKER-COMPAT.md](LINKER-COMPAT.md) | Completing system-linker compatibility across the support set: ordered DSO selection, ELF TLS/TLSDESC, shared libraries, relocatable links, runtime/sysroot interoperability, and Rust/toolchain validation. | [../LINK.md](../LINK.md), [../OBJ.md](../OBJ.md), [../DRIVER.md](../DRIVER.md) |
+| [KERNEL.md](KERNEL.md) | Freestanding kernel build and image pipeline: build-obj/build-exe support, linker-script growth, map/symbol side outputs, and image emission via kit image / objcopy -O binary. | [../DRIVER.md](../DRIVER.md), [../LINK.md](../LINK.md), [../OBJ.md](../OBJ.md), [../RUNTIME.md](../RUNTIME.md) |
| [JIT.md](JIT.md) | Function-level hot reload, Go-runtime-style codegen support, and remaining JIT host-portability work. | [../JIT.md](../JIT.md) |
| [DEBUG.md](DEBUG.md) | The Windows debugger host adapter, x64/rv64 displaced single-step, profiling, and DWARF gaps. | [../DBG.md](../DBG.md), [../DWARF.md](../DWARF.md) |
| [INSTRUMENT.md](INSTRUMENT.md) | Debugging/profiling tools in the Valgrind/callgrind/asan/ubsan tradition: the emulator as a dynamic-binary-instrumentation substrate (the `EmuToolHooks` ABI, shadow planes, the memory seam, a guest debugger), plus the cross-frontend compiler-sanitizer story. | [../EMU.md](../EMU.md) |
