Build coordinator — testing strategy

Historical testing notes. Companion to ../BUILD_COORDINATOR.md (the feature spec) and BUILD_INTERNALS.md (the module breakdown). This doc specifies how the subsystem is tested — the shared harness, the per-module plan, the integration scenarios, and the cross-cutting properties (byte-stability, fail-safe parsing, minimal-rebuild, crash-safety, concurrency, determinism). It mirrors how the layer below is tested: the dist subsystem has no test/dist; it is covered by test/cas + test/pkg (per public surface) plus INTERNAL unit tests. kit/build_coord.h gets test/build.

Why the subsystem is testable by construction

Two properties — both deliberate — carry the plan:

1. L0–L1 are pure (no I/O): build, cfg, defn, trace, protocol, client are byte/value transforms. They get straight INTERNAL unit tests and golden corpus, exactly like the ISA/DWARF units.
2. Every side effect is a host vtable: KitBuildStoreIo, KitBuildExec, KitBuildTransport, KitBuildSched, and KitCasHost. So every L2+ test injects a fake and stays hermetic and deterministic — no real sockets, processes, clock, or network, and no flakiness.

Guiding principle: test what ships. The integration harness drives the real protocol codec, service loop, runner, resolver, and coordinator; only the OS socket and process spawn are stubbed (see Integration).

Repo conventions this plugs into

• C unit tests use test/lib/kit_unit.h (stack-resident KitUnit, CU_CHECK /CU_EXPECT, no globals), registered in test/lib/unit.mk in one of two regimes:
  • INTERNAL (-Isrc, links LIB_OBJS): for modules with no public API — build, cfg, defn, trace, protocol, store, remote, coord, resolve, runner. They call internal symbols (build_shallow_emit, …).
  • PUBLIC (-Iinclude, links LIB_AR): for the public surface — client, bundle, and the kit_build* coordinator API. Proves the public header is self-sufficient.
• Driver / end-to-end is a run.sh driven through the kit binary with test/lib/kit_sh_assert.sh, exactly like test/cas/run.sh / test/pkg/run.sh, wired as a test-build-driver target in test/test.mk.
• Granular targets. Per the project rule (prefer targeted runs), every module gets its own test-build-<module>, aggregated by test-build. A failing resolver test never forces a full-suite rerun.

Shared harness — test/build/lib/

Written once, reused everywhere. The harness is the test investment.

Fake	Backs	Capability
FakeStoreIo	KitBuildStoreIo	rename/remove/temp/sync/lock over a real temp dir, plus fault injection: fail or half-apply the Nth op, to drive crash-safety
FakeExec	KitBuildExec	runs scripted recipes (target → {sources, needs, config-gets, env reads, output files, exit code}) and counts invocations per target; a corrupt-bytes mode backs remote-fetch
FakeWorkspace	KitCasHost.file_io	an in-memory file set a test mutates between builds (edit / delete / create) to drive invalidation; the source of source/glob bytes
ipc_transport	KitBuildTransport	in-memory framed transport over cond-var queues; the recipe callback runs on a KitBuildSched thread (the real protocol path)
thread_sched	KitBuildSched	real pthread-backed threads/mutex/cond for integration + concurrency; a NULL sched exercises sequential degrade
count_source	(a FakeWorkspace view)	counts rehashes/reglobs so refresh tests assert a shared subtree is checked once

Every integration test wraps its kit_build in a watchdog timeout (a deadlock-bug must fail, not wedge CI) — the one cost of the framed-over-thread harness (decision below).

Per-module plan

Layer	Module	Regime	Faked	Key cases
L0	build	INTERNAL	—	build_target_key / build_glob_result_hash id-stability vectors; build_id_eq
L1	cfg	INTERNAL	—	set/overlay/get; canonical sort+dedup; empty-argv fixed id; emit→parse→emit identity; bad-body rejection
L1	trace	INTERNAL	—	shallow/deep/deepset/record emit↔parse; absent leaf (<path> -); record MRU dedup+cap; byte-stable golden + known id
L1	defn	INTERNAL	—	parse + find; sorted-unique enforcement; no whole-file dep (find(T) stable under unrelated edits)
L1	protocol	INTERNAL	—	req/resp encode↔decode; need overlay+argv; multi-frame glob + GLOB_END; frame-bound + malformed rejection
L1	client	PUBLIC	loopback	each kit_build_client_* round-trips against a scripted server
L2	store	INTERNAL	FakeStoreIo + real CAS	put/get idempotence; record RMW dedup+cap; cache materialize verifies; deepset stored as a CAS blob; crash-safety (below)
L2	remote	INTERNAL	FakeExec	verified install; corrupt mirror discarded; remotes tried in order; tree=manifest-then-blobs; deepset via {kind}=blob
L3	coord	INTERNAL	all	memo (file hashed once); futures dedup; sequential degrade (sched==NULL); deepset_load interns+recurses, absent→ERR; pull_once idempotent; stat counters
L4	resolve	INTERNAL/PUBLIC	ipc + FakeExec	the scenario spine + soundness scenarios
L5	runner	INTERNAL	ipc + FakeExec	service loop logs every dep; nonzero exit → no trace; both traces + deepset written; jobs slot released across need (chain deeper than jobs completes)
L5	bundle	PUBLIC	real pkg	export→import round-trip; untrusted signer rejected; tofu pin; deepset + overlay/argv blobs present so an imported trace refreshes/replays
L6	api + driver	PUBLIC + run.sh	ipc / real	kit build end-to-end; kit_build_stats deltas

Integration

The scenario spine

The worked example is the backbone. Each step asserts (output tree-id, KitBuildStats delta) — the stats diff names the path taken:

1. cold → recipes_run += 2;
2. no-op rebuild → deep_hits += 1, recipes_run += 0;
3. comment-edit a dep source → dep recipes_run += 1, parent served by shallow_hits (Phase-2 output-compare), parent recipe not run;
4. flip a consumed config key → only the consuming recipe runs.

Soundness scenarios (the regression suite for the design review)

These are the tests that would have caught the bugs found while pressing on the design — each maps to a finding:

• recipe edit at depth (#8): edit //lib:core's recipe, sources unchanged → //app rebuilds (the original deep trace would have stale-hit).
• repoint (#8): same-content recipe → no rebuild; different-content → rebuild; unrelated stanza edit → nothing busts.
• absent → create (#10): source a missing file, then create it → busts.
• unconsumed config change: subtree that ignores it is not re-run.
• deepset sharing/refresh (#9): a diamond yields one shared child node (assert via count_source: rehashed once); an absent deepset blob → Phase 1 falls through (fail-safe), not a wrong answer.
• materialize miss (#11): evict a hit's output bytes → materialize_misses bumps and the recipe re-runs (no stale tree-id returned).
• verify mode: a recipe that emits a counter → flagged on the re-run compare.
• concurrent fan-out (#15): a recipe need-submits N deps then need-awaits them. Assert each distinct dep built once (futures dedup via KitBuildStats/FakeExec counts), the recorded [dep] set is identical whether the deps complete in order or reverse order (drive completion order via a scripted-recipe delay in FakeExec — determinism must not depend on it), and a submitted-but-never-awaited need contributes no edge and is cancelled. A cycle introduced through need-submit fails fast at submit; jobs = 1 with a wide fan-out still completes (await releases the slot).

Tiers

• Framed-over-thread (primary). The ipc_transport harness runs the real codec + service loop + runner + resolver + coordinator with the recipe on a sched thread. Fast, hermetic, deterministic outcomes (not interleavings). Carries the scenario + soundness suites.
• Subprocess + real socket (smoke). A small run.sh through kit build with real shell recipes and a real unix-socket transport, per platform — proves the real transport/exec/spawn that the framed harness stubs. The fidelity backstop.

Cross-cutting properties

• Byte-stability across platforms. Golden trace/deepset/config/record vectors are checked in; CI on every platform must reproduce identical bytes (a divergence silently breaks trace sharing). Paired with round-trip fuzz (kit_differential.sh style): random valid structures satisfy emit→parse→emit identity; random bytes either reject or re-emit identically (canonical idempotence) — the fail-safe property at scale, and the security guard that a parse error is never a match.
• Minimal-rebuild is the invariant. Every scenario asserts the recipe-run count (via FakeExec / KitBuildStats), not just the output — a cache that's correct but re-runs everything is the failure mode that matters here.
• Crash-safety (decision: fault injection). FakeStoreIo aborts the sequence before/after each step; the test reopens the store and asserts the invariants: content is complete-or-absent under its key (never torn — only orphaned tmp/); the record is prior-or-next, never torn; a missing record entry re-derives. The vtable lets us abort between any two FS effects, which is exactly the granularity the state machine reasons about.
• Concurrency (real threads + ThreadSanitizer). A wide diamond (many parents, one shared dep): assert the dep's recipe runs exactly once (futures dedup) and that jobs = 1 with a deep chain completes (the release-slot-across-need rule; no deadlock). A deterministic cooperative fake-sched for specific interleavings is a stretch goal, not v1.
• Sequential degrade. The same scenario suite runs with sched == NULL: jobs_* no-op, futures complete inline, no thread spawned — identical outputs and stats. (In-process recipes still need a thread, so the sequential suite uses subprocess or pre-seeded traces; see BUILD_COORDINATOR.md §Limits.)

Observability — KitBuildStats

Six cumulative counters on the coordinator (deep_hits, shallow_hits, recipes_run, materialize_misses, object_fetches, trace_pulls), bumped under the coordinator lock and read via kit_build_stats. Tests snapshot before / diff after each kit_build to assert which path served the request and how much work ran — turning "did the recipe run?" and "deep or shallow hit?" into exact assertions. The same surface is user-facing build introspection.

Decisions (this round)

• Integration harness = framed over a thread. The in-process transport runs the recipe on a KitBuildSched thread and exercises the real protocol/service loop — tests what ships, at the cost of per-test watchdog timeouts. Direct- dispatch (synchronous, sequential) is deferred unless in-process recipes become a product feature.
• Observability = KitBuildStats counters (public, snapshot-and-diff) rather than inferring from recipe-run counts alone.
• Crash-safety = fault-injecting FakeStoreIo (deterministic, in-process); a real fork+kill harness is deferred.

Map to the implementation waves

Tests land with their modules (Wave order):

• Wave A (build,cfg,defn,trace,protocol): unit + golden + fuzz; no harness needed. The byte-stability and fail-safe-parse corpus starts here.
• Wave B (store,remote,client): FakeStoreIo (+ fault injection), FakeExec, loopback transport land here.
• Wave C (coord): memo / futures / sequential-degrade / stats tests; the thread_sched and FakeWorkspace fakes land here.
• Wave D (resolve,runner,bundle): the ipc_transport harness, the scenario spine, and the soundness suite — the bulk of the value.
• Wave E (api + driver): the run.sh subprocess smoke + test-build aggregation in test.mk/unit.mk.