Kit black-box release acceptance

This harness audits a shipped Kit distribution or one exact Kit binary. It has no implicit compiler path and never falls back to build/kit, PATH, or a freshly built binary.

Run the full available suite:

test/audit/release/run.sh --dist ./build/dist/kit

Audit one exact binary:

test/audit/release/run.sh --kit ./build/dist/kit/bin/kit

The input must be selected explicitly with exactly one of --dist or --kit. Distribution-only cases become BLOCKED or NOT_APPLICABLE when only a binary is supplied; they never search for neighboring repository artifacts.

Scoped runs

Every case declares module, command, workflow, target, architecture, format, and severity metadata. Filters accept comma-separated shell globs:

test/audit/release/run.sh --dist ./build/dist/kit \
  --module help --command 'cc,build-*' --severity 'p1,p2'

test/audit/release/run.sh --dist ./build/dist/kit \
  --workflow compile --target 'aarch64-*' --arch aarch64 --format elf

Available options are printed by run.sh --help. A filter selects matching cases; generic all/any metadata remains applicable to scoped target, architecture, and format runs. Multi-command journeys declare each command as a comma-separated metadata value, so --command ar selects a workflow tagged cc,ar,ranlib,nm.

Isolation and evidence

The default artifact root is ignored build output: build/audit/release-acceptance/. Each run gets a unique UTC timestamp/PID directory. --out DIR changes the artifact root and --run-id NAME assigns a stable name. An existing run directory is never deleted or overwritten.

Each selected case owns isolated directories for:

• working files;
• HOME, XDG_CACHE_HOME, XDG_CONFIG_HOME, and XDG_DATA_HOME;
• temporary files;
• installation output; and
• retained artifacts.

The case directory records command, stdout, stderr, exit, metadata, assertions.log, state, and reason. Commands are executed once; assertions inspect those saved streams. Modules can retain produced files under the case's artifacts/ directory. A journey with multiple commands additionally records each exact invocation and its streams/status below steps/STEP/; its top-level command and exit direct readers to that directory. The run-level summary.tsv is the complete machine-readable result, while summary.txt is concise.

Result states are:

• PASS: all assertions succeeded;
• FAIL: the product ran but violated an acceptance assertion;
• BLOCKED: a required SDK, runner, distribution component, or other prerequisite was unavailable, with the reason saved;
• NOT_APPLICABLE: the case does not apply to the selected product/target.

The harness exits 1 if any case is FAIL or BLOCKED, 0 otherwise, and 2 for harness usage/setup errors (including filters that match no cases). This prevents missing advertised target prerequisites and misspelled filters from becoming silent skips.

Help and inventory module

modules/help.sh inventories every advertised command and alias through:

• no arguments;
• -h;
• --help; and
• kit help <command>.

It validates stream/status conventions, equivalent help routes, examples, exit documentation, the absence of source-code directions, and the required kit targets surface. The no-argument update probe is run only from an isolated copy of a supplied distribution; with --kit it is explicitly BLOCKED rather than risking an update through an unknown layout.

Run only this module:

test/audit/release/run.sh --dist ./build/dist/kit --module help

The baseline release is expected to report known Phase 1 findings as failures; the retained evidence is intended both for the audit report and for verifying later remediations.

Core workflow module

modules/core.sh exercises four compact compiler/linker journeys:

• an ordinary compile/link/run from an isolated directory, without hidden support or SDK paths;
• explicit SDK compilation, optimized/debug object emission, archive creation and indexing, archive linking, execution, and symbol inspection;
• preprocessing, dependency generation, syntax checking, assembly output, and optimized IR output; and
• optimized portable-C output, including the release-blocking crash case.

These fixtures use -print-sysroot from the exact selected Kit and the supplied distribution's support/; they do not search the repository for a compiler, runtime, or SDK. Run only the positive archive journey with:

test/audit/release/run.sh --dist ./build/dist/kit \
  --module core --workflow archive-link-run

Standalone and relocation module

modules/standalone.sh copies the supplied distribution to a path containing spaces and audits direct, PATH-only, installed multicall, read-only-source, and post-install-move invocation. Each case gets host-development-tool sentinels; any attempted use of clang, cc, gcc, as, ld, archive/binutils tools, or SDK-discovery utilities is retained in artifacts/sentinel.log and fails the acceptance case. The copied distribution is made user-writable only so the post-install relocation step can move it; the preserved input is untouched.

test/audit/release/run.sh --dist ./build/dist/kit --module standalone

Language API module

modules/api.sh treats the installed public headers as the language-developer surface. It compiles every shipped public header as the first include, then builds and runs four out-of-tree programs against only the selected Kit, include/, and lib/libkit.a: context/target/compiler lifecycle, source compilation-session object emission plus public linker composition, direct CG object emission plus public linker composition, and diagnostic-sink behavior. The exact native SDK is obtained from that Kit's -print-sysroot output.

test/audit/release/run.sh --dist ./build/dist/kit --module api

The core, standalone, and api modules require --dist, because their acceptance criteria cover shipped support files, headers, libraries, and relocation. With --kit, selected cases report BLOCKED explicitly.

Adding modules

Add modules/NAME.sh with a POSIX-shell function named audit_module_NAME. Source lib/harness.sh indirectly through run.sh; modules should define cases with audit_case_start, execute the exact selected binary with audit_case_exec_kit (or an explicit program with audit_case_exec), add assertions, and call audit_case_finish. Use audit_case_blocked and audit_case_not_applicable instead of silently skipping a selected case.

Small immutable inputs belong in fixtures/. Do not modify a fixture in place; copy it into the case work directory or artifact directory first.