Dynamic/shared libraries (Linux first)

Goal: make kit produce loadable shared libraries, with Linux/aarch64 as the first supported target. The useful first result is:

kit ld -shared -soname libfoo.so.1 -o libfoo.so foo.pic.o
kit cc -shared -fPIC -Wl,-soname,libfoo.so.1 -o libfoo.so foo.c

where readelf -h reports ET_DYN, there is no required entry point, no PT_INTERP, DT_SONAME is present when requested, exported symbols are in .dynsym, unresolved imports become runtime relocations, and a normal Linux loader can use the .so as a dependency of an executable.

Current state

kit already has most of the dynamic-executable substrate:

• The public link API has KIT_LINK_OUTPUT_SHARED, soname, rpath/runpath, exports, and allow_undefined fields in KitLinkSessionOptions.
• driver/cmd/{cc,ld,build}.c parse -shared / -soname but reject shared output with "creating dynamic/shared libraries is not yet supported".
• src/api/link.c maps KIT_LINK_OUTPUT_SHARED to link_set_pie(l, 1), but warns that -soname, rpaths, and explicit exports are ignored.
• src/obj/elf/link_dyn.c synthesizes .dynsym, .dynstr, .gnu.hash, .rela.dyn, .rela.plt, .plt, .got.plt, and .dynamic, but it is explicitly shaped as ET_DYN executable support. It always chooses an interpreter path when dynamic state exists.
• src/obj/elf/link.c emits ET_DYN when img->pie is set, but requires img->entry_sym and writes e_entry from it. It emits dynamic program headers as PT_PHDR, PT_INTERP, PT_DYNAMIC, and PT_GNU_STACK whenever pie && img->dyn.
• The linker can already consume DSO inputs: LINK_INPUT_DSO_BYTES contributes exported dynsym symbols for resolution and creates DT_NEEDED records.
• AArch64 already has PLT emission in src/arch/aa64/link.c and GOT-using relocation rows in src/arch/aa64/reloc.c.
• kit-generated aarch64 code uses obj_symbol_extern_via_got() for undefined extern data when the target is ELF PIC/PIE, so some PIC codegen policy is already centralized in src/obj/obj_secnames.c.

The missing part is not "all dynamic linking"; it is a distinct DSO output mode. PIE executables and DSOs are both ET_DYN, but they differ in entry requirements, interpreter headers, exported-symbol policy, undefined-symbol defaults, dynamic tags, relocation legality, and driver behavior.

Scope

Linux/aarch64 MVP

• ELF64 only.
• Target triples like aarch64-linux-*.
• Work backward from a known-good loader fixture: verify the shared-library test harness with all-clang first, then swap in kit ld -shared while keeping clang PIC objects, then swap in kit cc -fPIC so the final lane is all kit.
• Support regular function/data exports, DSO inputs, DT_NEEDED, DT_SONAME, DT_RPATH / DT_RUNPATH, .gnu.hash, .rela.dyn, .rela.plt, AArch64 PLT/GOT, R_AARCH64_RELATIVE, GLOB_DAT, and JUMP_SLOT.
• Default shared-library undefined-symbol policy should match ELF linkers: allow unresolved non-weak references by default in -shared, unless a stricter flag is requested.

Explicitly later

• x86-64 and riscv64 DSOs.
• TLS models beyond the currently practical local-exec / initial-exec subset.
• Copy relocations, symbol version definitions, version scripts, GNU --dynamic-list, --exclude-libs, -Bsymbolic*, --as-needed parity, and lazy binding.
• Mach-O dylib and COFF/PE DLL production.
• Runtime-library policy for building libkit_rt itself as a DSO.

Implementation checklist

1. Split PIE-executable mode from DSO mode

• Add an internal emit_shared / output_shared bit on Linker and mirror it on LinkImage, instead of representing shared output only as emit_pie.
• Keep emit_pie for dynamic/static PIE executable behavior. Let shared output imply image base 0 and dynamic-section synthesis without implying an executable entry or interpreter.
• Add link_set_shared(Linker*, int) beside link_set_pie.
• In kit_link_session_new, for KIT_LINK_OUTPUT_SHARED, set shared mode, enable dynamic layout, and do not call the static-exe setup.
• Update comments in src/link/link.h, src/link/link_internal.h, and doc/LINK.md after the implementation so "PIE/DSO" is not conflated with "PIE executable".

Acceptance:

• A shared link reaches kit_link_session_resolve without looking up _start.
• Existing PIE executable tests keep their current behavior.

2. Entry and ELF header behavior

• Change link_resolve_entry / ELF emit so KIT_LINK_OUTPUT_SHARED permits entry_sym == LINK_SYM_NONE.
• In link_emit_elf, write e_type = ET_DYN for shared output and e_entry = 0 unless an explicit entry was intentionally accepted.
• Keep ET_EXEC and PIE executable behavior unchanged.
• Ensure link maps / symbol side-files tolerate no entry.

Acceptance:

• kit ld -shared ... emits an ELF header with Type: DYN and entry address 0x0.
• Missing _start is not a diagnostic for shared output.

3. Program headers for DSOs

• For shared output, do not synthesize .interp and do not emit PT_INTERP. DSOs are loaded by another program's interpreter.
• Decide whether PT_PHDR is emitted for DSOs. It is valid and useful when the phdr table is in a loadable segment, but it must not drag in executable-only assumptions.
• Continue emitting PT_DYNAMIC and PT_GNU_STACK.
• Keep PT_TLS behavior if TLS sections are present, but gate unsupported TLS reloc models before producing a bogus library.
• Preserve page/file congruence and the existing headers PT_LOAD behavior; do not disturb scripted/freestanding output.

Acceptance:

• readelf -l libfoo.so shows DYNAMIC and no INTERP.
• Program headers' p_vaddr % p_align == p_offset % p_align.

4. Dynamic section tags

• Plumb KitLinkSessionOptions.soname, rpaths, runpaths, and exports into Linker / LinkDynState.
• Add DT_SONAME when -soname is set.
• Add DT_RPATH or DT_RUNPATH entries according to the existing driver --enable-new-dtags / --disable-new-dtags policy.
• Keep DT_NEEDED for DSO inputs already used by the link.
• Do not emit DT_FLAGS_1 = DF_1_NOW unconditionally for DSOs unless the link explicitly requests now binding. For the MVP, either omit it or share the current eager-binding behavior intentionally and document it.
• Rework .dynstr construction so soname/path strings are included during layout, not appended at emit time.

Acceptance:

• readelf -d shows requested SONAME, NEEDED, and RUNPATH/RPATH.
• No warning from src/api/link.c for supported shared-library flags.

5. Exported dynamic symbols

• Define the default export set for shared output as all defined non-local, non-hidden, non-section, non-file symbols. Preserve weak and protected bindings correctly.
• Honor symbol visibility from the object model (SV_DEFAULT, SV_HIDDEN, SV_PROTECTED) in .dynsym.st_other.
• Treat explicit exports as a later filtering/augmentation feature unless the first implementation wires it completely; do not silently ignore it.
• Keep executable --export-dynamic separate from DSO exports. A shared library is not an executable with -E.
• Ensure LTO preservation keeps exported DSO symbols alive. The existing KIT_LINK_OUTPUT_SHARED preserve-all-nonlocal path in src/api/link.c is the right starting point.

Acceptance:

• Hidden symbols are absent from .dynsym or marked non-preemptible as appropriate; default-visible globals are exported.
• A consumer executable can resolve an exported function/data symbol from the produced .so.

6. Undefined symbols and imports

• For shared output, unresolved global references should become dynsym imports even when they were not satisfied by a link-time DSO input, unless the caller requests --no-undefined / -z defs.
• Add a distinct "runtime import with no provider DSO" state or allow imported symbols with dso_input_id == LINK_INPUT_NONE; make collect_needed skip provider-less imports while still emitting dynsym and relocations.
• Keep executable behavior strict by default: unresolved references in executable links remain errors unless allow_undefined or an external resolver applies.
• Support weak undefined symbols as zero/null without requiring dynamic relocations where the psABI expects that behavior.

Acceptance:

• kit ld -shared foo.o can leave bar undefined and emits bar in .dynsym as UND.
• A final executable link fails or succeeds according to its own unresolved policy, independent of the library production step.

7. Relocation legality and runtime relocations

• Audit apply_all_relocs for shared-output semantics. For DSOs, text relocations should be rejected unless an explicit textrel option exists.
• AArch64 MVP:
  • calls/jumps to imported functions route through PLT and R_AARCH64_JUMP_SLOT;
  • absolute data/function pointer slots against imports emit GLOB_DAT;
  • absolute slots against internal non-ABS symbols emit R_AARCH64_RELATIVE;
  • GOT entries for preemptible/default-visible definitions must be handled correctly, not only undefined externals;
  • direct ADR_PREL_PG_HI21 / ADD_ABS_LO12_NC references to preemptible default-visible DSO definitions must be rejected or emitted through a DSO-safe form.
• Add a shared-output validation pass that catches reloc kinds the dynamic loader cannot apply for DSOs. This should diagnose the input object and symbol, not panic late in emit.
• Keep R_*_COPY out of DSO output. Copy relocs belong to executable links.

Acceptance:

• Non-PIC absolute/text reloc inputs fail with a clear diagnostic under -shared.
• PIC object fixtures with function calls, data imports, function-pointer initializers, and local address constants load and run through a consumer.

8. AArch64 PIC codegen from kit

• Validate that -fPIC sets target.pic = KIT_PIC_PIC for all compile paths that can feed a shared link.
• Audit aarch64 backend symbol-address lowering:
  • undefined extern data currently uses GOT via obj_symbol_extern_via_got;
  • default-visible definitions in the same DSO may still be preemptible under ELF interposition rules, so data accesses may also need GOT-indirect forms unless hidden/protected or -Bsymbolic applies;
  • direct calls can remain direct for local/hidden definitions but external default-visible calls need PLT-compatible relocations when interposable.
• Make KIT_CG_OUTPUT_SHARED and KIT_CG_INTERPOSITION_* drive these choices for both -O0 and -O1.
• Keep local hidden/internal symbols direct so PIC does not pessimize everything.

Acceptance:

• kit cc -fPIC -c foo.c produces relocations accepted by kit ld -shared.
• kit cc -shared -fPIC foo.c produces the same class of working .so as linking equivalent external PIC objects.

9. Driver behavior

• Remove the -shared rejection in driver/cmd/ld.c after the linker support is in place.
• Remove the -shared rejection in driver/cmd/cc.c only after kit-generated PIC objects are safe; before that, kit ld -shared can be enabled while kit cc -shared remains gated.
• For kit cc -shared, default compilation to -fPIC if the user did not explicitly choose PIC/PIE/non-PIC. Reject -shared -fno-pic.
• Set default output names consistently (a.out is wrong for -shared; use user -o or a library-shaped fallback only if the driver already has one).
• -Wl,-soname should require -shared, as it does now, but become useful.
• Decide and document -nostdlib / runtime archive behavior for shared libraries. Avoid pulling executable startup objects into a DSO.
• Keep build-lib -dynamic gated until the lower-level ld and cc paths are working; then route it through KIT_LINK_OUTPUT_SHARED.

Acceptance:

• The all-clang harness lane is green before either driver gate is removed.
• kit ld -shared is enabled after it passes with clang PIC objects.
• kit cc -shared -fPIC is enabled after kit-generated PIC objects pass the same DSO fixture and no _start or crt executable startup is included.

10. Test sequencing

Prefer red-green targeted tests. Add small ELF/aarch64 cases before broad matrix runs. The sequencing is part of the design: prove the test harness with the system toolchain first, then replace one kit component at a time.

• Build one reusable test/link/elf-dso-aa64/ fixture or equivalent. It should produce a small libfoo.so and a consumer executable that calls an exported function, reads exported data, exercises an undefined import supplied by the consumer or another DSO, and checks hidden/default visibility. The fixture has three lanes:
  • Harness baseline, all clang: clang compiles PIC objects, clang links the shared library, and clang links the consumer. This verifies the source fixture, sysroot, runtime loader, qemu/VM lane, readelf checks, and run harness before kit is involved.
  • Linker swap, kit ld only: clang still compiles all PIC objects, but kit ld -shared links libfoo.so; clang or the system linker may still link the consumer. This isolates ELF DSO production, dynamic tags, exports/imports, PLT/GOT, and runtime relocations from kit codegen.
  • Compiler swap, all kit: kit cc -fPIC compiles the library objects, kit ld -shared links the DSO, and kit cc/kit ld links the consumer. This is the first lane that validates kit-generated PIC and DSO interposition choices.
• Each lane should run the same observable checks:
  • header: ET_DYN, e_entry 0, no PT_INTERP, has PT_DYNAMIC;
  • dynamic tags: SONAME, NEEDED, RUNPATH/RPATH;
  • export set: default-visible exported, hidden not exported;
  • undefined import allowed in DSO and appears in .dynsym;
  • non-PIC relocation rejected with a precise diagnostic;
  • DSO consumed by a small executable under the Linux/aarch64 lane.
• Keep object-level tests on the linker-swap lane until they are green; do not debug kit codegen and ELF DSO layout in the same red test.
• Add kit-codegen tests for kit cc -fPIC -c relocation shape only after the clang-object linker-swap lane is stable; then add kit cc -shared.
• Use targeted commands such as make test-link TARGET=aarch64-linux or the closest existing selector. Redirect output to a file and inspect/tail that file.
• Once aarch64 is stable, repeat the same fixtures for x86-64 and riscv64 only after their PLT/GOT relocation coverage is audited.

Likely milestone order

1. All-clang harness baseline: write the aarch64 shared-library fixture and prove it with clang compile, clang -shared, clang/system consumer link, readelf assertions, and runtime execution. No kit linker/codegen work is considered validated until this lane is green.
2. ELF linker skeleton: internal shared mode, no entry, no interpreter, ET_DYN DSO headers.
3. Dynamic metadata: SONAME/RPATH/RUNPATH plumbing and dynstr layout.
4. Swap in kit ld -shared: keep clang PIC library objects, but produce the .so with kit. Make imports/exports, PLT/GOT, DT_NEEDED, and runtime relocations work in this isolated lane.
5. Diagnostics: reject non-PIC/textrel inputs clearly under kit ld -shared.
6. Enable kit ld -shared: remove the driver gate for object/archive/DSO inputs once the clang-object linker-swap lane is green.
7. Swap in kit codegen: compile the same library sources with kit cc -fPIC, audit/fix aarch64 interposition choices, and keep the DSO linked by kit. This creates the all-kit library path.
8. Enable kit cc -shared: compile and link source to .so through the driver, with no executable startup objects.
9. All-kit consumer lane: link the consumer with kit as well, so the full compile/link/run path is kit-produced except for the host loader/sysroot.
10. Broaden: x86-64/riscv64, build-lib dynamic mode, and additional ELF linker compatibility flags.

Open design decisions

• Should shared links default to --allow-shlib-undefined, with -z defs / --no-undefined as the strict mode? This matches common ELF behavior and should be the default unless kit intentionally chooses stricter diagnostics.
• Should DF_1_NOW remain the default for DSOs? Current PIE executable output uses eager binding. DSOs can work that way, but it is a policy choice rather than a requirement.
• How much ELF interposition should codegen honor in v1? A conservative MVP can require -fvisibility=hidden for kit-generated DSOs until default-visible definition preemption is fully correct, but kit ld -shared for external PIC inputs should not impose that restriction.
• Do we want a -Bsymbolic subset early? It can simplify same-DSO references, but it should be an explicit semantic change, not a hidden implementation shortcut.