commit c5899fea4182a2cc597568712047d0b29cf5002c
parent f5e26d02b0f89dfa410e54faed53cf944961bae3
Author: Ryan Sepassi <rsepassi@gmail.com>
Date: Wed, 17 Jun 2026 07:47:18 -0700
doc: plan ELF shared library support
Diffstat:
2 files changed, 344 insertions(+), 0 deletions(-)
diff --git a/doc/plan/DYNLIB.md b/doc/plan/DYNLIB.md
@@ -0,0 +1,343 @@
+# Dynamic/shared libraries (Linux first)
+
+Goal: make kit produce loadable shared libraries, with Linux/aarch64 as the
+first supported target. The useful first result is:
+
+```
+kit ld -shared -soname libfoo.so.1 -o libfoo.so foo.pic.o
+kit cc -shared -fPIC -Wl,-soname,libfoo.so.1 -o libfoo.so foo.c
+```
+
+where `readelf -h` reports `ET_DYN`, there is no required entry point, no
+`PT_INTERP`, `DT_SONAME` is present when requested, exported symbols are in
+`.dynsym`, unresolved imports become runtime relocations, and a normal Linux
+loader can use the `.so` as a dependency of an executable.
+
+## Current state
+
+kit already has most of the dynamic-executable substrate:
+
+- The public link API has `KIT_LINK_OUTPUT_SHARED`, `soname`, rpath/runpath,
+ exports, and `allow_undefined` fields in `KitLinkSessionOptions`.
+- `driver/cmd/{cc,ld,build}.c` parse `-shared` / `-soname` but reject shared
+ output with "creating dynamic/shared libraries is not yet supported".
+- `src/api/link.c` maps `KIT_LINK_OUTPUT_SHARED` to `link_set_pie(l, 1)`, but
+ warns that `-soname`, rpaths, and explicit exports are ignored.
+- `src/obj/elf/link_dyn.c` synthesizes `.dynsym`, `.dynstr`, `.gnu.hash`,
+ `.rela.dyn`, `.rela.plt`, `.plt`, `.got.plt`, and `.dynamic`, but it is
+ explicitly shaped as ET_DYN executable support. It always chooses an
+ interpreter path when dynamic state exists.
+- `src/obj/elf/link.c` emits `ET_DYN` when `img->pie` is set, but requires
+ `img->entry_sym` and writes `e_entry` from it. It emits dynamic program
+ headers as `PT_PHDR`, `PT_INTERP`, `PT_DYNAMIC`, and `PT_GNU_STACK` whenever
+ `pie && img->dyn`.
+- The linker can already consume DSO inputs: `LINK_INPUT_DSO_BYTES` contributes
+ exported dynsym symbols for resolution and creates `DT_NEEDED` records.
+- AArch64 already has PLT emission in `src/arch/aa64/link.c` and GOT-using
+ relocation rows in `src/arch/aa64/reloc.c`.
+- kit-generated aarch64 code uses `obj_symbol_extern_via_got()` for undefined
+ extern data when the target is ELF PIC/PIE, so some PIC codegen policy is
+ already centralized in `src/obj/obj_secnames.c`.
+
+The missing part is not "all dynamic linking"; it is a distinct DSO output
+mode. PIE executables and DSOs are both `ET_DYN`, but they differ in entry
+requirements, interpreter headers, exported-symbol policy, undefined-symbol
+defaults, dynamic tags, relocation legality, and driver behavior.
+
+## Scope
+
+### Linux/aarch64 MVP
+
+- ELF64 only.
+- Target triples like `aarch64-linux-*`.
+- Work backward from a known-good loader fixture: verify the shared-library
+ test harness with all-clang first, then swap in `kit ld -shared` while
+ keeping clang PIC objects, then swap in `kit cc -fPIC` so the final lane is
+ all kit.
+- Support regular function/data exports, DSO inputs, `DT_NEEDED`,
+ `DT_SONAME`, `DT_RPATH` / `DT_RUNPATH`, `.gnu.hash`, `.rela.dyn`,
+ `.rela.plt`, AArch64 PLT/GOT, `R_AARCH64_RELATIVE`, `GLOB_DAT`, and
+ `JUMP_SLOT`.
+- Default shared-library undefined-symbol policy should match ELF linkers:
+ allow unresolved non-weak references by default in `-shared`, unless a
+ stricter flag is requested.
+
+### Explicitly later
+
+- x86-64 and riscv64 DSOs.
+- TLS models beyond the currently practical local-exec / initial-exec subset.
+- Copy relocations, symbol version definitions, version scripts, GNU
+ `--dynamic-list`, `--exclude-libs`, `-Bsymbolic*`, `--as-needed` parity, and
+ lazy binding.
+- Mach-O dylib and COFF/PE DLL production.
+- Runtime-library policy for building libkit_rt itself as a DSO.
+
+## Implementation checklist
+
+### 1. Split PIE-executable mode from DSO mode
+
+- Add an internal `emit_shared` / `output_shared` bit on `Linker` and mirror it
+ on `LinkImage`, instead of representing shared output only as `emit_pie`.
+- Keep `emit_pie` for dynamic/static PIE executable behavior. Let shared output
+ imply image base 0 and dynamic-section synthesis without implying an
+ executable entry or interpreter.
+- Add `link_set_shared(Linker*, int)` beside `link_set_pie`.
+- In `kit_link_session_new`, for `KIT_LINK_OUTPUT_SHARED`, set shared mode,
+ enable dynamic layout, and do not call the static-exe setup.
+- Update comments in `src/link/link.h`, `src/link/link_internal.h`, and
+ `doc/LINK.md` after the implementation so "PIE/DSO" is not conflated with
+ "PIE executable".
+
+Acceptance:
+
+- A shared link reaches `kit_link_session_resolve` without looking up `_start`.
+- Existing PIE executable tests keep their current behavior.
+
+### 2. Entry and ELF header behavior
+
+- Change `link_resolve_entry` / ELF emit so `KIT_LINK_OUTPUT_SHARED` permits
+ `entry_sym == LINK_SYM_NONE`.
+- In `link_emit_elf`, write `e_type = ET_DYN` for shared output and
+ `e_entry = 0` unless an explicit entry was intentionally accepted.
+- Keep `ET_EXEC` and PIE executable behavior unchanged.
+- Ensure link maps / symbol side-files tolerate no entry.
+
+Acceptance:
+
+- `kit ld -shared ...` emits an ELF header with `Type: DYN` and entry address
+ `0x0`.
+- Missing `_start` is not a diagnostic for shared output.
+
+### 3. Program headers for DSOs
+
+- For shared output, do not synthesize `.interp` and do not emit `PT_INTERP`.
+ DSOs are loaded by another program's interpreter.
+- Decide whether `PT_PHDR` is emitted for DSOs. It is valid and useful when the
+ phdr table is in a loadable segment, but it must not drag in executable-only
+ assumptions.
+- Continue emitting `PT_DYNAMIC` and `PT_GNU_STACK`.
+- Keep `PT_TLS` behavior if TLS sections are present, but gate unsupported TLS
+ reloc models before producing a bogus library.
+- Preserve page/file congruence and the existing headers PT_LOAD behavior; do
+ not disturb scripted/freestanding output.
+
+Acceptance:
+
+- `readelf -l libfoo.so` shows `DYNAMIC` and no `INTERP`.
+- Program headers' `p_vaddr % p_align == p_offset % p_align`.
+
+### 4. Dynamic section tags
+
+- Plumb `KitLinkSessionOptions.soname`, `rpaths`, `runpaths`, and `exports`
+ into `Linker` / `LinkDynState`.
+- Add `DT_SONAME` when `-soname` is set.
+- Add `DT_RPATH` or `DT_RUNPATH` entries according to the existing driver
+ `--enable-new-dtags` / `--disable-new-dtags` policy.
+- Keep `DT_NEEDED` for DSO inputs already used by the link.
+- Do not emit `DT_FLAGS_1 = DF_1_NOW` unconditionally for DSOs unless the link
+ explicitly requests now binding. For the MVP, either omit it or share the
+ current eager-binding behavior intentionally and document it.
+- Rework `.dynstr` construction so soname/path strings are included during
+ layout, not appended at emit time.
+
+Acceptance:
+
+- `readelf -d` shows requested `SONAME`, `NEEDED`, and `RUNPATH`/`RPATH`.
+- No warning from `src/api/link.c` for supported shared-library flags.
+
+### 5. Exported dynamic symbols
+
+- Define the default export set for shared output as all defined non-local,
+ non-hidden, non-section, non-file symbols. Preserve weak and protected
+ bindings correctly.
+- Honor symbol visibility from the object model (`SV_DEFAULT`, `SV_HIDDEN`,
+ `SV_PROTECTED`) in `.dynsym.st_other`.
+- Treat explicit `exports` as a later filtering/augmentation feature unless the
+ first implementation wires it completely; do not silently ignore it.
+- Keep executable `--export-dynamic` separate from DSO exports. A shared library
+ is not an executable with `-E`.
+- Ensure LTO preservation keeps exported DSO symbols alive. The existing
+ `KIT_LINK_OUTPUT_SHARED` preserve-all-nonlocal path in `src/api/link.c` is
+ the right starting point.
+
+Acceptance:
+
+- Hidden symbols are absent from `.dynsym` or marked non-preemptible as
+ appropriate; default-visible globals are exported.
+- A consumer executable can resolve an exported function/data symbol from the
+ produced `.so`.
+
+### 6. Undefined symbols and imports
+
+- For shared output, unresolved global references should become dynsym imports
+ even when they were not satisfied by a link-time DSO input, unless the caller
+ requests `--no-undefined` / `-z defs`.
+- Add a distinct "runtime import with no provider DSO" state or allow
+ `imported` symbols with `dso_input_id == LINK_INPUT_NONE`; make
+ `collect_needed` skip provider-less imports while still emitting dynsym and
+ relocations.
+- Keep executable behavior strict by default: unresolved references in
+ executable links remain errors unless `allow_undefined` or an external
+ resolver applies.
+- Support weak undefined symbols as zero/null without requiring dynamic
+ relocations where the psABI expects that behavior.
+
+Acceptance:
+
+- `kit ld -shared foo.o` can leave `bar` undefined and emits `bar` in
+ `.dynsym` as `UND`.
+- A final executable link fails or succeeds according to its own unresolved
+ policy, independent of the library production step.
+
+### 7. Relocation legality and runtime relocations
+
+- Audit `apply_all_relocs` for shared-output semantics. For DSOs, text
+ relocations should be rejected unless an explicit textrel option exists.
+- AArch64 MVP:
+ - calls/jumps to imported functions route through PLT and `R_AARCH64_JUMP_SLOT`;
+ - absolute data/function pointer slots against imports emit `GLOB_DAT`;
+ - absolute slots against internal non-ABS symbols emit `R_AARCH64_RELATIVE`;
+ - GOT entries for preemptible/default-visible definitions must be handled
+ correctly, not only undefined externals;
+ - direct `ADR_PREL_PG_HI21` / `ADD_ABS_LO12_NC` references to preemptible
+ default-visible DSO definitions must be rejected or emitted through a
+ DSO-safe form.
+- Add a shared-output validation pass that catches reloc kinds the dynamic
+ loader cannot apply for DSOs. This should diagnose the input object and
+ symbol, not panic late in emit.
+- Keep `R_*_COPY` out of DSO output. Copy relocs belong to executable links.
+
+Acceptance:
+
+- Non-PIC absolute/text reloc inputs fail with a clear diagnostic under
+ `-shared`.
+- PIC object fixtures with function calls, data imports, function-pointer
+ initializers, and local address constants load and run through a consumer.
+
+### 8. AArch64 PIC codegen from kit
+
+- Validate that `-fPIC` sets `target.pic = KIT_PIC_PIC` for all compile paths
+ that can feed a shared link.
+- Audit aarch64 backend symbol-address lowering:
+ - undefined extern data currently uses GOT via `obj_symbol_extern_via_got`;
+ - default-visible definitions in the same DSO may still be preemptible under
+ ELF interposition rules, so data accesses may also need GOT-indirect forms
+ unless hidden/protected or `-Bsymbolic` applies;
+ - direct calls can remain direct for local/hidden definitions but external
+ default-visible calls need PLT-compatible relocations when interposable.
+- Make `KIT_CG_OUTPUT_SHARED` and `KIT_CG_INTERPOSITION_*` drive these choices
+ for both `-O0` and `-O1`.
+- Keep local hidden/internal symbols direct so PIC does not pessimize
+ everything.
+
+Acceptance:
+
+- `kit cc -fPIC -c foo.c` produces relocations accepted by `kit ld -shared`.
+- `kit cc -shared -fPIC foo.c` produces the same class of working `.so` as
+ linking equivalent external PIC objects.
+
+### 9. Driver behavior
+
+- Remove the `-shared` rejection in `driver/cmd/ld.c` after the linker support
+ is in place.
+- Remove the `-shared` rejection in `driver/cmd/cc.c` only after kit-generated
+ PIC objects are safe; before that, `kit ld -shared` can be enabled while
+ `kit cc -shared` remains gated.
+- For `kit cc -shared`, default compilation to `-fPIC` if the user did not
+ explicitly choose PIC/PIE/non-PIC. Reject `-shared -fno-pic`.
+- Set default output names consistently (`a.out` is wrong for `-shared`; use
+ user `-o` or a library-shaped fallback only if the driver already has one).
+- `-Wl,-soname` should require `-shared`, as it does now, but become useful.
+- Decide and document `-nostdlib` / runtime archive behavior for shared
+ libraries. Avoid pulling executable startup objects into a DSO.
+- Keep `build-lib -dynamic` gated until the lower-level `ld` and `cc` paths are
+ working; then route it through `KIT_LINK_OUTPUT_SHARED`.
+
+Acceptance:
+
+- The all-clang harness lane is green before either driver gate is removed.
+- `kit ld -shared` is enabled after it passes with clang PIC objects.
+- `kit cc -shared -fPIC` is enabled after kit-generated PIC objects pass the
+ same DSO fixture and no `_start` or crt executable startup is included.
+
+### 10. Test sequencing
+
+Prefer red-green targeted tests. Add small ELF/aarch64 cases before broad
+matrix runs. The sequencing is part of the design: prove the test harness with
+the system toolchain first, then replace one kit component at a time.
+
+- Build one reusable `test/link/elf-dso-aa64/` fixture or equivalent. It should
+ produce a small `libfoo.so` and a consumer executable that calls an exported
+ function, reads exported data, exercises an undefined import supplied by the
+ consumer or another DSO, and checks hidden/default visibility. The fixture has
+ three lanes:
+ - **Harness baseline, all clang:** clang compiles PIC objects, clang links
+ the shared library, and clang links the consumer. This verifies the source
+ fixture, sysroot, runtime loader, qemu/VM lane, `readelf` checks, and run
+ harness before kit is involved.
+ - **Linker swap, kit ld only:** clang still compiles all PIC objects, but
+ `kit ld -shared` links `libfoo.so`; clang or the system linker may still
+ link the consumer. This isolates ELF DSO production, dynamic tags,
+ exports/imports, PLT/GOT, and runtime relocations from kit codegen.
+ - **Compiler swap, all kit:** `kit cc -fPIC` compiles the library objects,
+ `kit ld -shared` links the DSO, and `kit cc`/`kit ld` links the consumer.
+ This is the first lane that validates kit-generated PIC and DSO
+ interposition choices.
+- Each lane should run the same observable checks:
+ - header: ET_DYN, e_entry 0, no PT_INTERP, has PT_DYNAMIC;
+ - dynamic tags: SONAME, NEEDED, RUNPATH/RPATH;
+ - export set: default-visible exported, hidden not exported;
+ - undefined import allowed in DSO and appears in `.dynsym`;
+ - non-PIC relocation rejected with a precise diagnostic;
+ - DSO consumed by a small executable under the Linux/aarch64 lane.
+- Keep object-level tests on the linker-swap lane until they are green; do not
+ debug kit codegen and ELF DSO layout in the same red test.
+- Add kit-codegen tests for `kit cc -fPIC -c` relocation shape only after the
+ clang-object linker-swap lane is stable; then add `kit cc -shared`.
+- Use targeted commands such as `make test-link TARGET=aarch64-linux` or the
+ closest existing selector. Redirect output to a file and inspect/tail that
+ file.
+- Once aarch64 is stable, repeat the same fixtures for x86-64 and riscv64 only
+ after their PLT/GOT relocation coverage is audited.
+
+## Likely milestone order
+
+1. **All-clang harness baseline:** write the aarch64 shared-library fixture and
+ prove it with clang compile, clang `-shared`, clang/system consumer link,
+ `readelf` assertions, and runtime execution. No kit linker/codegen work is
+ considered validated until this lane is green.
+2. **ELF linker skeleton:** internal shared mode, no entry, no interpreter,
+ `ET_DYN` DSO headers.
+3. **Dynamic metadata:** SONAME/RPATH/RUNPATH plumbing and dynstr layout.
+4. **Swap in `kit ld -shared`:** keep clang PIC library objects, but produce
+ the `.so` with kit. Make imports/exports, PLT/GOT, `DT_NEEDED`, and runtime
+ relocations work in this isolated lane.
+5. **Diagnostics:** reject non-PIC/textrel inputs clearly under
+ `kit ld -shared`.
+6. **Enable `kit ld -shared`:** remove the driver gate for object/archive/DSO
+ inputs once the clang-object linker-swap lane is green.
+7. **Swap in kit codegen:** compile the same library sources with
+ `kit cc -fPIC`, audit/fix aarch64 interposition choices, and keep the DSO
+ linked by kit. This creates the all-kit library path.
+8. **Enable `kit cc -shared`:** compile and link source to `.so` through the
+ driver, with no executable startup objects.
+9. **All-kit consumer lane:** link the consumer with kit as well, so the full
+ compile/link/run path is kit-produced except for the host loader/sysroot.
+10. **Broaden:** x86-64/riscv64, build-lib dynamic mode, and additional ELF
+ linker compatibility flags.
+
+## Open design decisions
+
+- Should shared links default to `--allow-shlib-undefined`, with `-z defs` /
+ `--no-undefined` as the strict mode? This matches common ELF behavior and
+ should be the default unless kit intentionally chooses stricter diagnostics.
+- Should `DF_1_NOW` remain the default for DSOs? Current PIE executable output
+ uses eager binding. DSOs can work that way, but it is a policy choice rather
+ than a requirement.
+- How much ELF interposition should codegen honor in v1? A conservative MVP can
+ require `-fvisibility=hidden` for kit-generated DSOs until default-visible
+ definition preemption is fully correct, but `kit ld -shared` for external PIC
+ inputs should not impose that restriction.
+- Do we want a `-Bsymbolic` subset early? It can simplify same-DSO references,
+ but it should be an explicit semantic change, not a hidden implementation
+ shortcut.
diff --git a/doc/plan/README.md b/doc/plan/README.md
@@ -11,6 +11,7 @@ shrinks to whatever remains open (and is deleted once nothing remains open).
| [RELEASE.md](RELEASE.md) | Cross-cutting initial-release punchlist: remaining release blockers and per-subsystem validation gaps (completed items are removed, not checked off). | — |
| [OPTIMIZER.md](OPTIMIZER.md) | Completing the O2 SSA mid-end, live-range splitting/coalescing, the residual O1 generated-code-quality gaps that need SSA, -O0 quality, machine register-constraint improvements, and broader inlining. Carries the known non-deficiencies (what *not* to chase). | [../OPT.md](../OPT.md) |
| [LINKER.md](LINKER.md) | Incremental linking (JIT append + the unbuilt file-based "m2" redesign) **and** system-linker compatibility (ordered DSO/`--as-needed`, ELF TLS planning + TLSDESC, shared/relocatable links, sysroot interop, toolchain validation). | [../LINK.md](../LINK.md), [../OBJ.md](../OBJ.md) |
+| [DYNLIB.md](DYNLIB.md) | Producing ELF shared libraries, Linux/aarch64 first: splitting DSO output from PIE executables, dynamic tags, exports/imports, relocation legality, PIC codegen, driver enablement, and targeted tests. | [../LINK.md](../LINK.md), [../OBJ.md](../OBJ.md) |
| [DEBUG.md](DEBUG.md) | The interactive JIT debugger + DWARF: x64/rv64 session parity, displaced-step instruction coverage, unit/smoke tests, REPL polish, Toy/C REPL frontends, and DWARF producer/consumer gaps (loclists, CFI register recovery, composite locations). | [../DBG.md](../DBG.md), [../DWARF.md](../DWARF.md) |
| [PROF.md](PROF.md) | The not-yet-built host-native sampling profiler (`kit prof`): SIGPROF frame-pointer walk reusing the debugger's signal infrastructure, the `KitProfBuf`/`KitProfWriter` API, and folded/flat output. Complementary to (not rebased on) the emulator callgrind in INSTRUMENT.md. | [../DBG.md](../DBG.md) |
| [INSTRUMENT.md](INSTRUMENT.md) | Debugging/profiling tools in the Valgrind/callgrind/asan/ubsan tradition: the emulator as a dynamic-binary-instrumentation substrate (the `EmuToolHooks` ABI, shadow planes, the memory seam, a guest debugger), plus the cross-frontend compiler-sanitizer story. | [../EMU.md](../EMU.md) |
