commit 6a21d4a06c4b95ff4f586305994be0628b49d280
parent b40b8fb7be7f3af9feab1fcdb703b8715f854e37
Author: Ryan Sepassi <rsepassi@gmail.com>
Date: Tue, 16 Jun 2026 21:22:48 -0700
doc(gram): refresh import plan for the gramgen library shift
The standalone parser/lexer-generator being imported moved from
/Users/ryan/code/ll1 to /Users/ryan/code/gram and was renamed+extended
(ll_/llgen_ -> gram_/gramgen_/GRAM_, GRAMAPI marker). Rewrite the plan
against the real current surface and rename LLGEN_IMPORT.md -> GRAM_IMPORT.md.
- Replace the obsolete ll_* rename tables with the actual gram_*/gramgen_*/
gramregex_*/GRAM_* surface and a uniform kit_gram_/KitGram/KIT_GRAM_ scheme
(GRAMAPI -> KIT_API, GRAM_OPAQUE_TYPE -> KIT_GRAM_OPAQUE_TYPE).
- Document the new subsystems: match API, token machines (%machine),
sublexers, standalone re2c-style scanner, parser-codegen, lazy positions
(gram_pos.h), Unicode helpers, regex, sample-token/trace synthesis,
keyword extraction.
- Update file moves (split gen/ lex pipelines), driver flags, and the test map.
- Record settled decisions: full-surface one-pass import; generator imported
verbatim (renames only, no refactor); diagnostics via KitContext.diag;
UCD data vendored in-tree; drop the Python reference generator (one-time fork).
Diffstat:
3 files changed, 585 insertions(+), 391 deletions(-)
diff --git a/doc/plan/GRAM_IMPORT.md b/doc/plan/GRAM_IMPORT.md
@@ -0,0 +1,584 @@
+# Gram Import Plan
+
+This is the import plan for folding `/Users/ryan/code/gram` (the standalone
+`gramgen` project, which builds `libgram.a` + the `gramgen` CLI) into libkit and
+exposing it through the kit driver. The goal is not to make `gram` a language
+frontend. It should land as a reusable parser/lexer-generator subsystem: EBNF
+in, parse / lexer / token-machine tables out, with allocation-free push runtimes
+for generated parsers and lexers.
+
+The standalone project already has the right broad shape for libkit: explicit
+allocation (a caller-supplied `gramgen_allocator`), no hidden CLI dependency in
+the generator API, immutable generated tables, caller-owned runtime state, and a
+`GRAMAPI` visibility marker that already distinguishes the public surface. The
+import work is mostly namespace, host-boundary, build-gating, and file-layout
+discipline.
+
+**This is a one-time fork.** Once the import lands, the kit copy is the source of
+truth and the standalone `/Users/ryan/code/gram` tree is abandoned. We are free
+to rename aggressively, restructure files, and drop the Python reference
+generator entirely; we do not need to stay re-syncable with upstream.
+
+## What Shifted Since the First Draft
+
+This plan originally targeted a project called `ll1` with `ll_*` / `llgen_*`
+symbols. That project was renamed and substantially extended; the import target
+is now `/Users/ryan/code/gram`. Concrete differences that this refresh accounts
+for:
+
+- **Names are already clean.** Public symbols are `gram_*` (runtime),
+ `gramgen_*` (compiler), `gramregex_*` (single-pattern), `GRAM_*`
+ (macros/enums), tagged with the `GRAMAPI` visibility attribute. The old
+ `ll_*` rename table is obsolete; the rename tables below target the real
+ current spellings. The destination spelling is unchanged in spirit:
+ `KitGram` / `kit_gram_` / `KIT_GRAM_`.
+- **New runtime subsystems** beyond the parser/lexer:
+ - **Match API** (`gram_match_*`, `gram_matcher_*`): a re2::Set-style
+ longest-match scanning surface (`find` / `anchored` / `full` + a
+ non-overlapping match iterator) over a compiled lexer grammar.
+ - **Token machines** (`%machine`): a regular language over an *abstract
+ symbol* alphabet, compiled codegen-only to a steppable DFA
+ (`gram_fsm_*`), plus an in-memory AST-directed sampler.
+ - **Sublexers**: a grammar may declare several `%lex` blocks (one main +
+ named blocks); each is its own DFA reachable via `gramgen_lexer_*`.
+ - **Standalone scanner output**: `lexer_standalone` emits a self-contained
+ re2c-style tokenizer with the DFA table dropped (C-only), with selectable
+ `%keywords` strategy (minimal-perfect-hash by default, `fold_keywords` to
+ bake them into the DFA) and an optional `position_lazy` mode.
+ - **Lazy positions** (`grampos.h`): header-only `offset → (line, col)`
+ indexes (byte and UTF-8 flavors) so a lexer can run with positions off and
+ recover them on demand.
+ - **Parser codegen** (`parser_codegen`): an optional pull-based
+ recursive-descent parser emitted alongside the push driver, with optional
+ error recovery (`parser_recover`).
+ - **Token-stream synthesis**: the CLI can emit bounded sample token streams
+ from the parser grammar alone (`--sample-tokens`) and machine traces
+ (`--sample-traces`).
+ - **Keyword extraction**: string-literal tokens shadowed by a general "host"
+ recognizer (e.g. keywords under `IDENT`) are pulled out of the DFA and
+ resolved by a minimal perfect hash (CHD) at scan time.
+- **The split lexer-compiler pipeline**: `gen/` now has a shared NFA→DFA core
+ (`gramgen_lex_dfa.c`), an alphabet-neutral range-set NFA
+ (`gramgen_lex_range.c`), and separate byte / scalar / unicode / token
+ pipelines.
+- **The Python generator is now the *default*** in the standalone Makefile, kept
+ byte-identical to the C generator by parity tests. Because this is a one-time
+ fork, **we drop the Python path entirely** and build only the C generator;
+ generated meta tables are checked in so no generator bootstrap is needed.
+- **`NO_UNICODE` slim build**: upstream can compile out all `%lex :utf8`
+ support (`-DGRAMGEN_NO_UNICODE`), dropping the scalar/unicode pipelines and
+ the ~0.5 MB property tables.
+
+## Decision Summary
+
+- Driver command: `gram`.
+- Public compiler API header: `<kit/gram.h>` (was `gramgen.h`).
+- Public parser runtime header: `<kit/gram_parse.h>` (was `gramparse.h`).
+- Public lexer + match + machine runtime header: `<kit/gram_lex.h>` (was
+ `gramlex.h`).
+- Public lazy-position header (header-only): `<kit/gram_pos.h>` (was
+ `grampos.h`).
+- Public Unicode-helper header: `<kit/gram_unicode.h>` (was `gramunicode.h`).
+- Generated-code support headers: `<kit/support/gram_parse_tables.h>` and
+ `<kit/support/gram_lex_tables.h>`.
+- Visibility marker: fold `GRAMAPI` into kit's existing `KIT_API`
+ (`<kit/core.h>`); the `GRAM_OPAQUE_TYPE` helper macro becomes
+ `KIT_GRAM_OPAQUE_TYPE` and lives in `<kit/gram_parse.h>`. Drop `gramapi.h`.
+- Library subsystem gate: `KIT_GRAM_ENABLED`.
+- Driver tool gate: `KIT_TOOL_GRAM_ENABLED`.
+- Implementation directory: `src/gram/`.
+- Tests: `test/gram/` plus a driver smoke lane.
+- Generated meta tables checked in (`src/gram/meta_tables.c` / `.h`); no Python,
+ no network, no UCD regeneration in the normal build.
+- **First-import scope: full surface, one pass.** Wire up every subsystem
+ (parser, lexer, match API, token machines, sublexers, standalone scanner,
+ parser-codegen, lazy positions, regex) in a single landing — no
+ minimal-core-first staging.
+- **Generator imported verbatim.** `gramgen.c` lands as one `src/gram/`
+ translation unit; apply the rename scheme but do **not** restructure or split
+ it. A clean-factor pass, if ever wanted, is a separate future effort.
+- **Diagnostics route through `KitContext.diag`.** Drop the
+ `gramgen_diagnostic` out-param entirely; there is no bespoke kit error channel.
+- **UCD source data is vendored in-tree** at `data/ucd/17.0.0/` (checksummed) so
+ property-table regeneration stays offline and reproducible.
+
+`gram` / `gramgen` remain source-project names only. They should not appear in
+the libkit API, the installed command name, or user-facing help.
+
+## Boundaries
+
+`gram` is a library subsystem, not a `lang/` frontend. It does not register a
+`KitFrontendVTable`, does not emit `KitCg`, and does not participate in
+source-to-object compilation unless a future frontend chooses to use it
+internally. Its public surface is consumed by embedders, the driver command, and
+generated C files.
+
+The driver remains the only hosted layer. File reads, directory creation,
+stdout, stderr, and CLI allocation policy live in `driver/cmd/gram.c` and the
+hosted driver environment. The library accepts input as `KitSlice`, writes
+output to `KitWriter`, allocates through `KitContext.heap`, and reports
+diagnostics through `KitContext.diag`.
+
+The runtimes are allocation-free and caller-owned by construction, which already
+matches kit discipline:
+
+- Generated grammar tables (`gram_grammar`, `gram_lex_grammar`, the DFA/keyword
+ tables, machine tables) are immutable static data and are fine as globals.
+- Mutable parser, lexer, input, matcher, and match-iterator state lives in
+ caller-owned opaque, stack-allocatable objects (`gram_parser`, `gram_lexer`,
+ `gram_lex_input`, `gram_matcher`, `gram_match_iter`, …), sized by
+ `_Static_assert`-guarded `*_SIZE` constants. These rename mechanically; no
+ context struct is needed because the caller already owns every byte.
+- The match API and token-machine stepping are header-only over those tables.
+- `grampos.h` is entirely header-only `static inline`; it allocates nothing and
+ the caller owns the offset array and the indexed buffer.
+
+The one host-boundary rewrite is the *generator*, which today takes a
+`gramgen_allocator` and reports a single `gramgen_diagnostic` out-param, and
+whose CLI does its own `malloc`/`fopen`/`fprintf`/`exit`. That moves onto
+`KitContext` (heap + diag) and `KitWriter` (see Phase 3).
+
+## File Moves
+
+Import the C generator, runtime, generated meta/property tables, and tests. Do
+**not** import the Python generator.
+
+| Current file | Kit destination | Notes |
+|--------------|-----------------|-------|
+| `include/gramgen.h` | `include/kit/gram.h` | Public compiler API. |
+| `include/gramparse.h` | `include/kit/gram_parse.h` | Public parser runtime API; also hosts `KIT_GRAM_OPAQUE_TYPE`. |
+| `include/gramlex.h` | `include/kit/gram_lex.h` | Public lexer + match + token-machine runtime API. |
+| `include/grampos.h` | `include/kit/gram_pos.h` | Public header-only lazy line/col index. |
+| `include/gramunicode.h` | `include/kit/gram_unicode.h` | Public UTF-8 / Unicode scalar helpers. |
+| `include/gramapi.h` | *(dropped)* | `GRAMAPI`→`KIT_API` (from `<kit/core.h>`); `GRAM_OPAQUE_TYPE`→`KIT_GRAM_OPAQUE_TYPE` in `gram_parse.h`. |
+| `include/gramparse_tables.h` | `include/kit/support/gram_parse_tables.h` | Generated-code support, not ordinary embedder API. |
+| `include/gramlex_tables.h` | `include/kit/support/gram_lex_tables.h` | Generated-code support, not ordinary embedder API. |
+| `include/gramunicode_props.h` | `src/gram/unicode_props.h` | Private generated property-resolver interface (generation-time use). Promote to a public header only if a runtime need appears. |
+| `runtime/gramparse.c` | `src/gram/parse_runtime.c` | Implements `<kit/gram_parse.h>` (push LL(1)/Pratt driver). |
+| `runtime/gramlex.c` | `src/gram/lex_runtime.c` | Implements `<kit/gram_lex.h>` (DFA scanner + match API). |
+| `runtime/gramunicode.c` | `src/gram/unicode.c` | UTF-8 encode/decode + scalar/position helpers. |
+| `runtime/gramunicode_props.c` | `src/gram/unicode_props.c` | Checked-in generated UCD 17.0.0 property tables (~0.5 MB). |
+| `gen/gramgen.c` | `src/gram/generator.c` | Public API impl: front end, parser-table generation, C emitter (~210 KB). Imported as one file, renames only — deliberately not refactored. |
+| `gen/gramgen_ll1.c` | `src/gram/ll1.c` | LL(1), Pratt, FIRST/FOLLOW, validation. |
+| `gen/gramgen_lex_dfa.c` / `.h` | `src/gram/lex_dfa.c` / `lex_dfa.h` | Shared NFA→DFA core (subset construction, minimization, relabel). |
+| `gen/gramgen_lex_range.c` / `.h` | `src/gram/lex_range.c` / `lex_range.h` | Alphabet-neutral range-set NFA (survives `NO_UNICODE`). |
+| `gen/gramgen_lex_byte.c` | `src/gram/lex_byte.c` | Byte lexer pipeline + mode-independent lexer driver. |
+| `gen/gramgen_lex_scalar.c` | `src/gram/lex_scalar.c` | UTF-8 scalar pipeline (excluded under `NO_UNICODE`). |
+| `gen/gramgen_lex_unicode.c` | `src/gram/lex_unicode.c` | Unicode set algebra + property resolver (excluded under `NO_UNICODE`). |
+| `gen/gramgen_lex_tokens.c` | `src/gram/lex_tokens.c` | `%machine` token-alphabet (abstract-symbol) DFA pipeline. |
+| `gen/gramgen_internal.h` | `src/gram/internal.h` | Private to `src/gram/*.c`. |
+| `gen/meta.ebnf` | `src/gram/meta.ebnf` | Source grammar for the generator's own parser; not consumed by normal builds. |
+| generated meta `.c`/`.h` | `src/gram/meta_tables.c`, `src/gram/meta_tables.h` | Check in (generated with the `kit_gram_meta_` prefix) so libkit builds with no generator bootstrap. |
+| `gen/gramgen_cli.c` | `driver/cmd/gram.c` | Rewrite as a Kit driver command over public `<kit/gram.h>`. |
+| `gen/gramgen.py` | *(dropped)* | One-time fork: not imported. |
+| `tools/gen_unicode_props.py` | `scripts/gen_gram_unicode_props.py` | Regeneration helper only; not in the build. |
+| `data/ucd/17.0.0/*` | `data/ucd/17.0.0/*` | Vendored + checksummed; kept in-tree for offline, reproducible `regen-gram-unicode-props`. |
+| `test/*.c`, `test/*.ebnf` | `test/gram/*` | Library tests and fixtures (see Tests). |
+| `test/errors/*` | `test/gram/errors/*` | Parser/lexer error fixtures. |
+| `test/machine_errors/*` | `test/gram/machine_errors/*` | `%machine` error fixtures. |
+| `test/nounicode/*` | `test/gram/nounicode/*` | `NO_UNICODE`-rejection fixtures. |
+| `test/realistic/*` | `test/gram/realistic/*` | JSON / C-like / sharp-edge end-to-end grammars. |
+| `test/standalone/*` | `test/gram/standalone/*` | Standalone-scanner + sublexer fixtures. |
+
+## Public Renames
+
+The standalone `gram_*` / `gramgen_*` / `GRAM_*` names are clean and namespaced,
+but they still violate kit's enforced public-symbol discipline:
+`scripts/lib_reloc_defined_prefixes.py` relocatably links `libkit.a` and fails
+the build on any exported definition whose name is not in
+`ALLOWED_PREFIXES = ("Kit", "kit_", "KIT")`. So every public definition in the
+imported subsystem must be respelled.
+
+### Rename scheme
+
+Apply uniformly across the runtime, the generator's C emitter, and the
+checked-in generated meta tables:
+
+| Current spelling | Kit spelling | Example |
+|------------------|--------------|---------|
+| `GRAMAPI` (visibility attr) | `KIT_API` (from `<kit/core.h>`) | `KIT_API void kit_gram_parser_init(...)` |
+| `GRAM_OPAQUE_TYPE` (macro) | `KIT_GRAM_OPAQUE_TYPE` | declared in `<kit/gram_parse.h>` |
+| `gram_<snake>` / `gramgen_<snake>` *type* | `KitGram<Pascal>` | `gram_parser`→`KitGramParser`, `gramgen_compiled`→`KitGramCompiled` |
+| `gram_<snake>` / `gramgen_<snake>` *function* | `kit_gram_<snake>` | `gram_parser_init`→`kit_gram_parser_init`, `gramgen_compile_text`→`kit_gram_compile_text` |
+| `gramregex_<snake>` | `kit_gram_regex_<snake>` | `gramregex_compile`→`kit_gram_regex_compile` |
+| `GRAM_<UPPER>` (macro / enumerator) | `KIT_GRAM_<UPPER>` | `GRAM_SLOT_SIZE`→`KIT_GRAM_SLOT_SIZE`, `GRAM_PARSE_ACCEPT`→`KIT_GRAM_PARSE_ACCEPT` |
+
+Generated grammar-specific token/rule enums (`TOK_*`, `R_*`, and the
+user-chosen `<prefix>_*`) are *user* artifacts controlled by the generated
+prefix; they are not libkit exports and keep their shape.
+
+### Compiler API (`<kit/gram.h>`)
+
+| Current name | Kit name |
+|--------------|----------|
+| `gramgen_options` | `KitGramOptions` |
+| `gramgen_compiled` | `KitGramCompiled` |
+| `gramgen_diagnostic` | removed — report through `KitContext.diag` |
+| `gramgen_allocator` | removed — allocate through `KitContext.heap` |
+| `gramgen_writer` / `gramgen_write_fn` | removed — stream to `KitWriter` |
+| `gramgen_text` / `gramgen_text_free` | removed — use `kit_writer_mem` for owned in-memory text |
+| `gramgen_compile_text` | `kit_gram_compile_text` |
+| `gramgen_compiled_free` | `kit_gram_free` |
+| `gramgen_dump_sexpr_text` | `kit_gram_dump_sexpr` (writes a `KitWriter`) |
+| `gramgen_generate_c` | `kit_gram_emit_c` (writes header + source `KitWriter`s) |
+| `gramgen_parser_grammar` / `gramgen_lexer_grammar` | `kit_gram_parser_grammar` / `kit_gram_lexer_grammar` |
+| `gramgen_lexer_count` / `_name` / `_grammar_at` / `gramgen_find_lexer` | `kit_gram_lexer_count` / `_name` / `_grammar_at` / `kit_gram_find_lexer` |
+| `gramgen_token_count` / `_name` / `_display` / `gramgen_find_token` | `kit_gram_token_count` / `_name` / `_display` / `kit_gram_find_token` |
+| `gramgen_rule_count` / `_name` / `gramgen_find_rule` | `kit_gram_rule_count` / `_name` / `kit_gram_find_rule` |
+| `gramgen_machine_count` / `_name` / `_rule_count` / `_rule_name` / `_generate` / `gramgen_find_machine` | `kit_gram_machine_*` / `kit_gram_find_machine` |
+| `gramregex_compile` / `gramregex_grammar` / `gramregex_kind` | `kit_gram_regex_compile` / `_grammar` / `_kind` |
+
+The generator API takes a `const KitContext*` and `KitSlice` inputs; the
+allocator becomes `ctx->heap`; C output streams to caller-provided `KitWriter`s.
+
+Proposed core shape:
+
+```c
+typedef struct KitGramCompiled KitGramCompiled;
+
+typedef struct KitGramOptions {
+ KitSlice name; /* optional grammar name override */
+ bool multiline; /* bake newline-aware ^/$ into the lexer */
+ bool parser_codegen;/* also emit a pull-based recursive-descent parser */
+ bool parser_recover;/* error-recovery machinery in the RD parser */
+ bool lexer_standalone; /* self-contained re2c-style scanner (C-only) */
+ bool fold_keywords; /* standalone: fold %keywords into the DFA */
+ bool position_lazy; /* standalone: drop per-token line/col tracking */
+} KitGramOptions;
+
+typedef struct KitGramEmitOptions {
+ KitSlice header_path;
+ KitSlice source_path;
+ KitSlice prefix;
+} KitGramEmitOptions;
+
+KIT_API KitStatus kit_gram_compile_text(const KitContext* ctx,
+ KitSlice text, KitSlice path,
+ const KitGramOptions* opts,
+ KitGramCompiled** out);
+KIT_API void kit_gram_free(KitGramCompiled*);
+
+KIT_API KitStatus kit_gram_emit_c(const KitGramCompiled*,
+ const KitGramEmitOptions* opts,
+ KitWriter* header, KitWriter* source);
+KIT_API KitStatus kit_gram_dump_sexpr(const KitContext* ctx, KitSlice text,
+ KitSlice path,
+ const KitGramOptions* opts,
+ KitWriter* out);
+```
+
+### Parser runtime API (`<kit/gram_parse.h>`)
+
+Types: `gram_tok_kind`→`KitGramTokenKind`, `gram_rule_id`→`KitGramRuleId`,
+`gram_sem`→`KitGramSem`, `gram_token`→`KitGramToken`, `gram_error`→
+`KitGramError`, `gram_err_action`→`KitGramErrorAction`, `gram_actions`→
+`KitGramActions`, `gram_slot`→`KitGramSlot`, `gram_config`→`KitGramConfig`,
+`gram_grammar`→`KitGramGrammar`, `gram_parser`→`KitGramParser`,
+`gram_status`→`KitGramStatus`, `gram_frame`→`KitGramFrame`,
+`gram_gen_status`→`KitGramGenStatus`, `gram_gen_config`→`KitGramGenConfig`.
+
+Constants/enumerators: `GRAM_SLOT_SIZE`→`KIT_GRAM_SLOT_SIZE`,
+`GRAM_PARSER_SIZE`→`KIT_GRAM_PARSER_SIZE`, `GRAM_NEED_MORE`→
+`KIT_GRAM_NEED_MORE`, `GRAM_PARSE_ACCEPT`→`KIT_GRAM_PARSE_ACCEPT`,
+`GRAM_PARSE_ERROR`→`KIT_GRAM_PARSE_ERROR`, `GRAM_ABORT`/`GRAM_SKIP`/`GRAM_RESYNC`
+→`KIT_GRAM_ABORT`/`KIT_GRAM_SKIP`/`KIT_GRAM_RESYNC`,
+`GRAM_GEN_TOKEN`/`_DONE`/`_LIMIT`/`_ERROR`→`KIT_GRAM_GEN_*`.
+
+Functions (representative): `gram_parser_init`/`_push`/`_push_n`/`_finish`/
+`_result`/`_errors`/`_overflowed`/`_depth`/`_context`/`_expects`/`_frames`/
+`_frame`/`_ctl_hwm`/`_val_hwm`/`_rule_name`/`_tok_name`,
+`gram_parser_generate_next`/`gram_parser_generate_scratch_count`,
+`gram_stack_bounds` → all `kit_gram_*` (e.g. `kit_gram_parser_init`,
+`kit_gram_stack_bounds`).
+
+### Lexer + match + machine runtime API (`<kit/gram_lex.h>`)
+
+Types: `gram_lex_grammar`→`KitGramLexGrammar`, `gram_lexer`→`KitGramLexer`,
+`gram_lex_input`→`KitGramLexInput`, `gram_lex_input_span`→`KitGramLexInputSpan`,
+`gram_lex_input_config`→`KitGramLexInputConfig`, `gram_lex_release`→
+`KitGramLexRelease`, `gram_lex_config`→`KitGramLexConfig`, `gram_lex_status`→
+`KitGramLexStatus`, `gram_lex_error`→`KitGramLexError`, `gram_lex_segment`→
+`KitGramLexSegment`, `gram_lex_lexeme`→`KitGramLexLexeme`,
+`gram_lex_hook_result`→`KitGramLexHookResult`, `gram_lex_token_hook`→
+`KitGramLexTokenHook`, `gram_matcher`→`KitGramMatcher`, `gram_match`→
+`KitGramMatch`, `gram_match_opts`→`KitGramMatchOpts`, `gram_match_iter`→
+`KitGramMatchIter`, `gram_fsm_status`→`KitGramFsmStatus`.
+
+Constants/enumerators: `GRAM_LEXER_SIZE`/`GRAM_LEX_INPUT_SIZE`/
+`GRAM_LEX_LEXEME_SIZE`/`GRAM_MATCHER_SIZE`/`GRAM_MATCH_ITER_SIZE`→`KIT_GRAM_*`;
+`GRAM_LEX_TOKEN`/`_NEED_MORE`/`_EOF`/`_ERROR`→`KIT_GRAM_LEX_*`;
+`GRAM_LEX_HOOK_KEEP`/`_SKIP`/`_ERROR`→`KIT_GRAM_LEX_HOOK_*`;
+`GRAM_FSM_OK`/`GRAM_FSM_DEAD`→`KIT_GRAM_FSM_*`.
+
+Functions (representative): `gram_lex_input_init`/`_push`/`_finish`/`_error`,
+`gram_lexer_init`/`_next`/`_lexeme`, `gram_lex_lexeme_next`,
+`gram_lex_keyword_rewrite`, `gram_matcher_bind`, `gram_match_anchored`/`_full`/
+`_find`, `gram_match_iter_init`/`_next` → all `kit_gram_*`.
+
+### Lazy positions (`<kit/gram_pos.h>`)
+
+Header-only `static inline`. Types `gram_line_index`→`KitGramLineIndex`,
+`gram_uline_index`→`KitGramULineIndex`. Functions `gram_line_count`/
+`gram_line_index_build`/`gram_position` and `gram_uline_count`/
+`gram_uline_index_build`/`gram_uposition` → `kit_gram_*`. The internal helpers
+(`gram__cp_advance`, `gram__fold_pos`, `gram__scalar`) become `kit_gram__*`
+(or file-static) — they are implementation detail, not exported, but should be
+respelled for consistency.
+
+### Unicode helpers (`<kit/gram_unicode.h>`)
+
+Types `gram_utf8_status`→`KitGramUtf8Status`, `gram_unicode_pos`→
+`KitGramUnicodePos`. Constant `GRAM_UNICODE_MAX_SCALAR`→
+`KIT_GRAM_UNICODE_MAX_SCALAR`; enumerators `GRAM_UTF8_OK`/`_NEED_MORE`/`_INVALID`
+→`KIT_GRAM_UTF8_*`. Functions `gram_unicode_is_*`, `gram_utf8_encode*`/
+`gram_utf8_decode_one`, `gram_unicode_pos_*`, `gram_unicode_fold_property_key`,
+`gram_utf8_is_cont` → `kit_gram_*`.
+
+### Generated-code support API
+
+The generated-support headers (`<kit/support/gram_parse_tables.h>`,
+`<kit/support/gram_lex_tables.h>`) also rename, since generated C and the
+checked-in meta tables include them:
+
+| Current name | Kit name |
+|--------------|----------|
+| `gram_sym_kind` (`GRAM_S_TERM`/`_RULE`/`_REP`/`_OPT`) | `KitGramSymKind` (`KIT_GRAM_S_*`) |
+| `gram_sym` / `gram_prod` / `gram_rule` | `KitGramSym` / `KitGramProd` / `KitGramRule` |
+| `gram_pratt_op_role` (`GRAM_PO_PREFIX`…`GRAM_PO_CIRCUMFIX`) | `KitGramPrattOpRole` (`KIT_GRAM_PO_*`) |
+| `gram_pratt_op` / `gram_pratt` | `KitGramPrattOp` / `KitGramPratt` |
+| `GRAM_TERM(t)` / `GRAM_RULE(r)` (authoring macros) | `KIT_GRAM_TERM` / `KIT_GRAM_RULE` |
+| `gram_lex_input_mode` (`GRAM_LEX_INPUT_BYTES`/`_UTF8`/`_TOKENS`) | `KitGramLexInputMode` (`KIT_GRAM_LEX_INPUT_*`) |
+| `gram_lex_anchor` (`GRAM_LEX_ANCHOR_NONE`/`_TEXT`/`_LINE`) | `KitGramLexAnchor` (`KIT_GRAM_LEX_ANCHOR_*`) |
+| `gram_lex_accept` / `gram_lex_keyword` / `gram_lex_keyword_table` | `KitGramLexAccept` / `KitGramLexKeyword` / `KitGramLexKeywordTable` |
+| `gram_lex_kw_hash64` (shared MPH hash) | `kit_gram_lex_kw_hash64` |
+| `GRAM_LEX_DEAD` / `GRAM_LEX_ACCEPT_NONE` | `KIT_GRAM_LEX_DEAD` / `KIT_GRAM_LEX_ACCEPT_NONE` |
+
+## Include Rewrites
+
+The generator should emit installed-style includes:
+
+```c
+/* generated header */
+#include <kit/gram_parse.h>
+#include <kit/gram_lex.h> /* only when a generated lexer exists */
+
+/* generated source */
+#include "generated_name.h"
+#include <kit/support/gram_parse_tables.h>
+#include <kit/support/gram_lex_tables.h> /* only when needed */
+```
+
+Two emit modes need care:
+
+- **Standalone scanner** (`lexer_standalone`): the DFA table is dropped and the
+ generated scanner is self-contained C, so it must *not* pull in
+ `<kit/gram_lex.h>`'s runtime-link surface beyond what it actually uses (it may
+ still reference `kit_gram_lex_keyword_rewrite` and the keyword tables).
+- **Parser-only grammars** (no `%lex`): omit the lexer includes entirely.
+
+Private `src/gram/*.c` files include `internal.h` and the public headers they
+implement. They must not include driver headers.
+
+## Driver Command
+
+The user-facing command is:
+
+```text
+kit gram [opts] grammar.ebnf
+ -o FILE output .c (default: <grammar>.c)
+ --header FILE output .h (default: <grammar>.h)
+ --prefix NAME identifier prefix (default: derived from filename)
+ --dump-sexpr print the parsed EBNF syntax tree instead of generating
+ --multiline bake newline-aware ^/$ into the lexer
+ --lexer-standalone emit a self-contained re2c-style scanner (C-only)
+ --fold-keywords standalone: fold %keywords into the DFA (default: MPH)
+ --position-lazy standalone: omit per-token line/col tracking
+ --parser-codegen also emit a pull-based recursive-descent parser
+ --parser-recover emit error-recovery machinery in the RD parser
+ --sample-tokens [--samples N] [--seed N]
+ [--max-depth N] [--max-repeat N] [--max-tokens N]
+ --sample-traces NAME [--samples N] [--seed N] ... (a %machine name)
+```
+
+`-` in place of the grammar path reads the grammar from stdin (default output
+names and prefix then derive from `stdin`).
+
+The first import should preserve existing behavior:
+
+- Default `.c` / `.h` output paths: replace the input suffix.
+- Default prefix: derive from the input basename (sanitized) and append `_`.
+- `--dump-sexpr`: write the meta-grammar dump to stdout.
+- Exit code `0`: success; `1`: compile/diagnostic/I/O failure; `2`: bad usage.
+
+The driver implementation uses `DriverEnv` for `KitContext`, file reads, writer
+opening, diagnostics, and memory. It must not call `malloc`, `free`, `fopen`,
+`fprintf`, or `exit` directly. The sample/trace synthesis paths use the public
+`kit_gram_parser_generate_next` / `kit_gram_machine_generate` entry points.
+
+Driver integration points:
+
+- Add `KIT_TOOL_GRAM_ENABLED` to `include/kit/config.h`.
+- Add `driver/cmd/gram.c`.
+- Add `driver_gram` and `driver_help_gram` to `driver/driver.h`.
+- Add the `gram` row to `driver/main.c`, gated by `KIT_TOOL_GRAM_ENABLED`.
+- Add `$(call tool-cmd,GRAM,gram)` to `mk/driver_srcs.mk`.
+- Keep it in `DRIVER_GROUP_OTHER` for now. It is a developer tool, not a default
+ drop-in binutils/toolchain symlink.
+
+## Build Integration
+
+Library integration points:
+
+- Add `KIT_GRAM_ENABLED` to `include/kit/config.h` as an optional library
+ subsystem.
+- Add `LIB_SRCS_GRAM := $(shell find src/gram -name '*.c' ...)` to
+ `mk/lib_srcs.mk`, included only when `KIT_GRAM_ENABLED` is `1`.
+- Add weak public stubs in `src/api/config_stubs.c` for gated-out public API
+ entry points (`kit_gram_*`). Because the whole subsystem (runtime + generator)
+ is gated together, the public runtime symbols need stubs too when gated out.
+- Keep `src/gram/meta_tables.c` checked in. Normal `make lib` must not require
+ Python, network access, UCD regeneration, or a bootstrap `gram` binary.
+- Add regeneration-only maintenance targets later, e.g. `make regen-gram-meta`
+ (re-emit `meta_tables.{c,h}` from `src/gram/meta.ebnf` with prefix
+ `kit_gram_meta_`, using the just-built `kit gram`) and
+ `make regen-gram-unicode-props` (re-run `scripts/gen_gram_unicode_props.py`
+ against the vendored UCD and diff for drift).
+
+The first import gates runtime and generator together under `KIT_GRAM_ENABLED`.
+If size-sensitive embeddings later need the generated parser runtime without the
+generator compiler, split into `KIT_GRAM_RUNTIME_ENABLED` (runtime + support
+headers) and `KIT_GRAM_ENABLED` (generator, depending on runtime). Do not
+introduce that split until a real embedding benefits; it adds config and stub
+surface.
+
+Upstream's `NO_UNICODE` slim build compiles out `%lex :utf8` (the scalar +
+unicode pipelines and the ~0.5 MB property tables). If kit wants that knob, model
+it as `KIT_GRAM_UNICODE_ENABLED` (default on) that excludes `lex_scalar.c`,
+`lex_unicode.c`, and `unicode_props.c` and makes the generator reject utf8 mode
+with a clear diagnostic. Treat it as a later size-tuning option, not part of the
+first import.
+
+## Implementation Phases
+
+### Phase 0: Freeze Standalone Behavior
+
+- Run the current `gram` test suite (`make test` in `/Users/ryan/code/gram`)
+ with the **C** generator (`GRAMGEN=./build/gramgen_c`) and save the passing
+ command set in the import notes — the C generator is what we keep.
+- Generate and check in `meta_tables.{c,h}` from `meta.ebnf`.
+- Confirm generated output for representative grammars (calc, features, pratt,
+ unicode, keywords, anchors, machine) is stable.
+
+### Phase 1: Mechanical Import, Private Names Still Allowed Internally
+
+- Move files into the destinations above.
+- Keep behavior unchanged while fixing include paths.
+- Add `test/gram` fixtures and a `make test-gram` target, initially allowed to
+ fail until the namespace rewrite lands.
+
+### Phase 2: Public Namespace Rewrite
+
+- Apply the rename scheme to every public `gram_*`, `gramgen_*`, `gramregex_*`,
+ and `GRAM_*` symbol, plus `GRAMAPI`→`KIT_API` and
+ `GRAM_OPAQUE_TYPE`→`KIT_GRAM_OPAQUE_TYPE`.
+- Update the C emitter in `generator.c` so emitted C and the checked-in meta
+ tables use the new names. `generator.c` stays a single translation unit —
+ renames only, no structural split.
+- Run `make test-lib-deps` to catch any leaked public symbol outside `Kit`,
+ `kit_`, or `KIT` — the relocatable-link check is the gate here.
+
+### Phase 3: Kit Context Rewrite
+
+- Replace `gramgen_allocator` with `KitContext.heap`.
+- Replace `gramgen_generate_c` + `gramgen_writer` with `kit_gram_emit_c` over
+ `KitWriter`; replace `gramgen_dump_sexpr_text` + `gramgen_text` with
+ `kit_gram_dump_sexpr` over a `KitWriter` (callers that want owned text use
+ `kit_writer_mem`).
+- Replace the `gramgen_diagnostic` out-param with `KitContext.diag` reporting.
+- Remove hosted libc calls from imported library code (they live only in the
+ driver CLI today, but verify the generator core is clean).
+- The runtimes already avoid `setjmp`/`longjmp` and allocation; confirm the
+ generator's OOM and validation-abort paths unwind via explicit status rather
+ than `exit` once the CLI scaffolding is gone.
+
+### Phase 4: Driver Command
+
+- Port `gen/gramgen_cli.c` to `driver/cmd/gram.c` using the public API only.
+- Add help text consistent with other driver commands.
+- Add a focused driver test: invoke `kit gram` on a small grammar, compile the
+ generated C against libkit, and run the parser and lexer.
+
+### Phase 5: Cleanup and Documentation
+
+- Document the stable runtime API in the public headers.
+- Add a durable design doc under `doc/` only after the subsystem ships (the
+ upstream `doc/DESIGN.md` is a good seed for the algorithms).
+- Add `gram` to `README.md` and `doc/DESIGN.md` capability lists after the
+ command works.
+- Delete `src/gram/meta.ebnf`-only build scaffolding that is not needed at
+ normal build time; keep `meta.ebnf` for the regen target.
+
+## Tests
+
+Targeted tests should land with the import:
+
+- `make test-gram`: direct API compile, table introspection, generated parser,
+ generated lexer, sublexers, Pratt grammar, mixfix Pratt, UTF-8 lexer, match
+ API, token machines, lazy positions, parser-codegen, keyword extraction, and
+ the error fixtures.
+- `make test-driver-gram`: CLI generation and generated-code compile/run smoke.
+- `make test-lib-deps`: symbol discipline (relocatable-link prefix check) and no
+ accidental hosted dependencies.
+
+Map standalone tests as follows:
+
+| Standalone test | Kit test |
+|-----------------|----------|
+| `test/test_gramgen_api.c` | `test/gram/api_test.c` |
+| `test/test_calc.c` | `test/gram/calc_test.c` |
+| `test/test_features.c` | `test/gram/features_test.c` |
+| `test/test_lexer.c` | `test/gram/lexer_test.c` |
+| `test/test_lexer_standalone.c` | `test/gram/lexer_standalone_test.c` |
+| `test/test_pratt_calc.c` | `test/gram/pratt_calc_test.c` |
+| `test/test_pratt_mixfix.c` | `test/gram/pratt_mixfix_test.c` |
+| `test/test_unicode_support.c` | `test/gram/unicode_support_test.c` |
+| `test/test_unicode_lexer.c` | `test/gram/unicode_lexer_test.c` |
+| `test/test_utf8_runtime.c` | `test/gram/utf8_runtime_test.c` |
+| `test/test_match.c` | `test/gram/match_test.c` |
+| `test/test_machine.c` | `test/gram/machine_test.c` |
+| `test/test_position.c` | `test/gram/position_test.c` |
+| `test/test_parser_codegen.c` | `test/gram/parser_codegen_test.c` |
+| `test/test_keywords.c` | `test/gram/keywords_test.c` |
+| `test/test_desugar.c` | `test/gram/desugar_test.c` |
+| `test/test_sublexer_table.c` | `test/gram/sublexer_table_test.c` |
+| `test/realistic/*` | `test/gram/realistic/*` |
+| `test/standalone/*` | `test/gram/standalone/*` |
+| `test/errors/*.ebnf` | `test/gram/errors/*.ebnf` |
+| `test/machine_errors/*.ebnf` | `test/gram/machine_errors/*.ebnf` |
+| `test/nounicode/*.ebnf` | `test/gram/nounicode/*.ebnf` |
+
+Prefer red-green import steps:
+
+1. Add the test target and fixtures first.
+2. Import the runtime until hand-written/generated tables parse again.
+3. Import the generator until direct API tests pass.
+4. Add the driver command and smoke test last.
+
+## Open Questions
+
+These are non-blocking — each has a stated default; revisit only if an embedding
+forces it. (Scope, generator factoring, the diagnostics channel, and in-tree UCD
+data are settled in the Decision Summary.)
+
+- Should generated table-layout headers be documented as stable ABI, or merely
+ stable enough for C emitted by the same libkit version? The first import should
+ promise only same-version compatibility.
+- Should the runtime/generator gate split (and the `NO_UNICODE`-equivalent
+ `KIT_GRAM_UNICODE_ENABLED`) ship in the first import? The plan says no until an
+ embedding needs the size savings.
+- Should `<kit/gram_unicode.h>` be a public header, or private to `src/gram`?
+ Upstream tags it `GRAMAPI`; decide based on whether any generated *runtime*
+ (not just generation-time) code needs it. `gram_pos.h` is clearly worth
+ exposing (header-only, standalone-useful).
+- Should `gram` eventually support non-C output modes? The API should not bake in
+ more than `emit_c` today, but the command can grow `--emit=` later.
diff --git a/doc/plan/LLGEN_IMPORT.md b/doc/plan/LLGEN_IMPORT.md
@@ -1,390 +0,0 @@
-# Gram Import Plan
-
-This is the import plan for folding `/Users/ryan/code/ll1` into libkit and
-exposing it through the kit driver. The goal is not to make `ll1` a language
-frontend. It should land as a reusable parser/lexer-generator subsystem: EBNF in,
-parse and lexer tables out, with allocation-free push runtimes for generated
-parsers and lexers.
-
-The standalone project already has the right broad shape for libkit: explicit
-allocation, no hidden CLI dependency in the generator API, immutable generated
-tables, and caller-owned runtime state. The import work is mostly namespace,
-host-boundary, build-gating, and file-layout discipline.
-
-## Decision Summary
-
-- Driver command: `gram`.
-- Public generator API header: `<kit/gram.h>`.
-- Public parser runtime header: `<kit/gram_parse.h>`.
-- Public lexer runtime header: `<kit/gram_lex.h>`.
-- Generated-code support headers: `<kit/support/gram_parse_tables.h>` and
- `<kit/support/gram_lex_tables.h>`.
-- Library subsystem gate: `KIT_GRAM_ENABLED`.
-- Driver tool gate: `KIT_TOOL_GRAM_ENABLED`.
-- Implementation directory: `src/gram/`.
-- Tests: `test/gram/` plus a driver smoke lane.
-
-`ll1` remains a source repository/project name only. It should not appear in the
-libkit API, installed command name, or user-facing help.
-
-## Boundaries
-
-`gram` is a library subsystem, not a `lang/` frontend. It does not register a
-`KitFrontendVTable`, does not emit `KitCg`, and does not participate in
-source-to-object compilation unless a future frontend chooses to use it
-internally. Its public surface is consumed by embedders, the driver command, and
-generated C files.
-
-The driver remains the only hosted layer. File reads, directory creation, stdout,
-stderr, and CLI allocation policy live in `driver/cmd/gram.c` and the hosted
-driver environment. The library accepts input as `KitSlice`, writes output to
-`KitWriter`, allocates through `KitContext.heap`, and reports diagnostics through
-`KitContext.diag`.
-
-Generated grammar tables are immutable static data and are fine as globals.
-Mutable parser, lexer, and generator state must hang off caller-owned runtime
-objects or explicit libkit handles.
-
-## File Moves
-
-Import the C implementation, generated meta tables, runtime, and tests. Do not
-make the Python generator part of the normal libkit build.
-
-| Current file | Kit destination | Notes |
-|--------------|-----------------|-------|
-| `include/llgen.h` | `include/kit/gram.h` | Public generator API, renamed to Kit types and functions. |
-| `include/llparse.h` | `include/kit/gram_parse.h` | Public parser runtime API. |
-| `include/lllex.h` | `include/kit/gram_lex.h` | Public lexer runtime API. |
-| `include/llparse_tables.h` | `include/kit/support/gram_parse_tables.h` | Public generated-code support, not ordinary embedder API. |
-| `include/lllex_tables.h` | `include/kit/support/gram_lex_tables.h` | Public generated-code support, not ordinary embedder API. |
-| `include/llunicode.h` | `src/gram/unicode.h` | Private helper; expose later as `<kit/unicode.h>` only if there is a broader API need. |
-| `include/llunicode_props.h` | `src/gram/unicode_props.h` | Private generated Unicode property resolver. |
-| `runtime/llparse.c` | `src/gram/parse_runtime.c` | Implements `<kit/gram_parse.h>`. |
-| `runtime/lllex.c` | `src/gram/lex_runtime.c` | Implements `<kit/gram_lex.h>`. |
-| `runtime/llunicode.c` | `src/gram/unicode.c` | Private Unicode helpers for generator and lexer runtime. |
-| `runtime/llunicode_props.c` | `src/gram/unicode_props.c` | Checked-in generated property tables. |
-| `gen/llgen.c` | `src/gram/generator.c` | Public API implementation and C emitter after Kit context rewrite. |
-| `gen/llgen_ll1.c` | `src/gram/ll1.c` | LL(1), Pratt, FIRST/FOLLOW, and validation. |
-| `gen/llgen_lex_byte.c` | `src/gram/lex_byte.c` | Byte-mode lexer compiler. |
-| `gen/llgen_lex_unicode.c` | `src/gram/lex_unicode.c` | Unicode-mode lexer compiler. |
-| `gen/llgen_internal.h` | `src/gram/internal.h` | Private to `src/gram/*.c`. |
-| `gen/meta.ebnf` | `src/gram/meta.ebnf` | Source grammar for the generator's own parser; not consumed by normal builds. |
-| generated `meta` `.c/.h` | `src/gram/meta_tables.c`, `src/gram/meta_tables.h` | Check in generated tables so libkit does not need Python or a previous `gram` to build. |
-| `gen/llgen_cli.c` | `driver/cmd/gram.c` | Rewrite as a Kit driver command using public `<kit/gram.h>`. |
-| `tools/gen_unicode_props.py` | `scripts/gen_gram_unicode_props.py` | Regeneration helper only; not in the build. |
-| `data/ucd/17.0.0/*` | `data/ucd/17.0.0/*` or `test/gram/ucd/17.0.0/*` | Keep if we want reproducible Unicode-table regeneration in-tree. |
-| `test/*.c`, `test/*.ebnf` | `test/gram/*` | Library tests and fixtures. |
-| `test/errors/*` | `test/gram/errors/*` | Error fixtures. |
-
-`gen/llgen.py` should not be imported into libkit. If parity against the Python
-reference is still useful during the transition, keep it temporarily as
-`scripts/gram_ref.py` and exclude it from release/build dependencies. Delete it
-once the imported C generator is trusted.
-
-## Public Renames
-
-The standalone `ll_*` and `llgen_*` names are too short for libkit and would
-violate the public symbol discipline. Public definitions for the imported
-subsystem must use `KitGram`, `kit_gram_`, or `KIT_GRAM_`.
-
-### Generator API
-
-| Standalone name | Kit name |
-|-----------------|----------|
-| `llgen_options` | `KitGramOptions` |
-| `llgen_compiled` | `KitGramCompiled` |
-| `llgen_codegen` | Remove or narrow; prefer `KitWriter` outputs. |
-| `llgen_compile_text` | `kit_gram_compile_text` |
-| `llgen_compiled_free` | `kit_gram_free` |
-| `llgen_dump_sexpr_text` | `kit_gram_dump_sexpr` |
-| `llgen_generate_c` | `kit_gram_emit_c` |
-| `llgen_parser_grammar` | `kit_gram_parser_grammar` |
-| `llgen_lexer_grammar` | `kit_gram_lexer_grammar` |
-| `llgen_token_count` | `kit_gram_token_count` |
-| `llgen_token_name` | `kit_gram_token_name` |
-| `llgen_token_display` | `kit_gram_token_display` |
-| `llgen_find_token` | `kit_gram_find_token` |
-| `llgen_rule_count` | `kit_gram_rule_count` |
-| `llgen_rule_name` | `kit_gram_rule_name` |
-| `llgen_find_rule` | `kit_gram_find_rule` |
-
-The generator API should take a `const KitContext*` and `KitSlice` inputs. It
-should not expose `llgen_allocator`; the allocator becomes `ctx->heap`. C output
-should stream to caller-provided `KitWriter`s. Callers that want owned in-memory
-text can use `kit_writer_mem`.
-
-Proposed core shape:
-
-```c
-typedef struct KitGramCompiled KitGramCompiled;
-
-typedef struct KitGramOptions {
- KitSlice name; /* optional grammar name override */
-} KitGramOptions;
-
-typedef struct KitGramEmitOptions {
- KitSlice header_path;
- KitSlice source_path;
- KitSlice prefix;
-} KitGramEmitOptions;
-
-KIT_API KitStatus kit_gram_compile_text(const KitContext* ctx,
- KitSlice text, KitSlice path,
- const KitGramOptions* opts,
- KitGramCompiled** out);
-KIT_API void kit_gram_free(KitGramCompiled*);
-
-KIT_API KitStatus kit_gram_emit_c(const KitGramCompiled*,
- const KitGramEmitOptions* opts,
- KitWriter* header, KitWriter* source);
-KIT_API KitStatus kit_gram_dump_sexpr(const KitContext* ctx, KitSlice text,
- KitSlice path,
- const KitGramOptions* opts,
- KitWriter* out);
-```
-
-### Parser Runtime API
-
-| Standalone name | Kit name |
-|-----------------|----------|
-| `ll_tok_kind` | `KitGramTokenKind` |
-| `ll_rule_id` | `KitGramRuleId` |
-| `ll_sem` | `KitGramSem` |
-| `ll_token` | `KitGramToken` |
-| `ll_error` | `KitGramParseError` |
-| `ll_err_action` | `KitGramErrorAction` |
-| `LL_ABORT` | `KIT_GRAM_ERROR_ABORT` |
-| `LL_SKIP` | `KIT_GRAM_ERROR_SKIP` |
-| `LL_RESYNC` | `KIT_GRAM_ERROR_RESYNC` |
-| `ll_actions` | `KitGramActions` |
-| `ll_slot` | `KitGramSlot` |
-| `LL_SLOT_SIZE` | `KIT_GRAM_SLOT_SIZE` |
-| `ll_config` | `KitGramParserConfig` |
-| `ll_grammar` | `KitGramGrammar` |
-| `ll_parser` | `KitGramParser` |
-| `LL_PARSER_SIZE` | `KIT_GRAM_PARSER_SIZE` |
-| `ll_status` | `KitGramParseStatus` |
-| `LL_NEED_MORE` | `KIT_GRAM_PARSE_NEED_MORE` |
-| `LL_PARSE_ACCEPT` | `KIT_GRAM_PARSE_ACCEPT` |
-| `LL_PARSE_ERROR` | `KIT_GRAM_PARSE_ERROR` |
-| `ll_parser_init` | `kit_gram_parser_init` |
-| `ll_parser_push` | `kit_gram_parser_push` |
-| `ll_parser_finish` | `kit_gram_parser_finish` |
-| `ll_parser_result` | `kit_gram_parser_result` |
-| `ll_stack_bounds` | `kit_gram_stack_bounds` |
-
-### Lexer Runtime API
-
-| Standalone name | Kit name |
-|-----------------|----------|
-| `ll_lex_grammar` | `KitGramLexGrammar` |
-| `ll_lexer` | `KitGramLexer` |
-| `LL_LEXER_SIZE` | `KIT_GRAM_LEXER_SIZE` |
-| `ll_lex_config` | `KitGramLexConfig` |
-| `ll_lex_status` | `KitGramLexStatus` |
-| `LL_LEX_TOKEN` | `KIT_GRAM_LEX_TOKEN` |
-| `LL_LEX_NEED_MORE` | `KIT_GRAM_LEX_NEED_MORE` |
-| `LL_LEX_EOF` | `KIT_GRAM_LEX_EOF` |
-| `LL_LEX_ERROR` | `KIT_GRAM_LEX_ERROR` |
-| `ll_lex_error` | `KitGramLexError` |
-| `ll_lexer_init` | `kit_gram_lexer_init` |
-| `ll_lexer_push` | `kit_gram_lexer_push` |
-| `ll_lexer_finish` | `kit_gram_lexer_finish` |
-| `ll_lexer_next` | `kit_gram_lexer_next` |
-| `ll_lexer_error` | `kit_gram_lexer_error` |
-
-### Generated-Code Support API
-
-The generated-code support headers should use Kit names too:
-
-| Standalone name | Kit name |
-|-----------------|----------|
-| `ll_sym_kind` | `KitGramSymKind` |
-| `LL_S_TERM` | `KIT_GRAM_SYM_TERM` |
-| `LL_S_RULE` | `KIT_GRAM_SYM_RULE` |
-| `LL_S_REP` | `KIT_GRAM_SYM_REP` |
-| `LL_S_OPT` | `KIT_GRAM_SYM_OPT` |
-| `ll_sym` | `KitGramSym` |
-| `ll_prod` | `KitGramProd` |
-| `ll_pratt_op` | `KitGramPrattOp` |
-| `ll_pratt` | `KitGramPratt` |
-| `ll_rule` | `KitGramRule` |
-| `LL_TERM` | `KIT_GRAM_TERM` |
-| `LL_RULE` | `KIT_GRAM_RULE` |
-| `ll_lex_accept` | `KitGramLexAccept` |
-| `LL_LEX_DEAD` | `KIT_GRAM_LEX_DEAD` |
-| `LL_LEX_ACCEPT_NONE` | `KIT_GRAM_LEX_ACCEPT_NONE` |
-
-Generated grammar-specific token/rule enums (`TOK_*`, `R_*`, and
-`<prefix>_*`) are user artifacts. They may keep their current shape because they
-are controlled by the generated prefix and are not libkit exports.
-
-## Include Rewrites
-
-The generator should emit installed-style includes:
-
-```c
-/* generated header */
-#include <kit/gram_parse.h>
-#include <kit/gram_lex.h> /* only when a generated lexer exists */
-
-/* generated source */
-#include "generated_name.h"
-#include <kit/support/gram_parse_tables.h>
-#include <kit/support/gram_lex_tables.h> /* only when needed */
-```
-
-Private `src/gram/*.c` files include `internal.h` and the public headers they
-implement. They must not include driver headers.
-
-## Driver Command
-
-The user-facing command is:
-
-```text
-kit gram [--dump-sexpr] [--prefix PREFIX] [-o OUT.c] [--header OUT.h] grammar.ebnf
-```
-
-The first import should preserve existing behavior:
-
-- Default `.c` output path: replace the input suffix with `.c`.
-- Default `.h` output path: replace the input suffix with `.h`.
-- Default prefix: derive from the input basename and append `_`.
-- `--dump-sexpr`: write the meta-grammar dump to stdout.
-- Exit code `0`: success.
-- Exit code `1`: compile, diagnostic, or I/O failure.
-- Exit code `2`: bad command-line usage.
-
-The driver implementation should use `DriverEnv` for `KitContext`, file reads,
-writer opening, diagnostics, and memory. The command should not call `malloc`,
-`free`, `fopen`, `fprintf`, or `exit` directly.
-
-Driver integration points:
-
-- Add `KIT_TOOL_GRAM_ENABLED` to `include/kit/config.h`.
-- Add `driver/cmd/gram.c`.
-- Add `driver_gram` and `driver_help_gram` to `driver/driver.h`.
-- Add the `gram` row to `driver/main.c`, gated by `KIT_TOOL_GRAM_ENABLED`.
-- Add `$(call tool-cmd,GRAM,gram)` to `mk/driver_srcs.mk`.
-- Keep it in `DRIVER_GROUP_OTHER` for now. It is a developer tool, not a
- default drop-in binutils/toolchain symlink.
-
-## Build Integration
-
-Library integration points:
-
-- Add `KIT_GRAM_ENABLED` to `include/kit/config.h` as an optional library
- subsystem.
-- Add `LIB_SRCS_GRAM := $(shell find src/gram -name '*.c' ...)` to
- `mk/lib_srcs.mk`, and include it only when `KIT_GRAM_ENABLED` is `1`.
-- Add weak public stubs in `src/api/config_stubs.c` for gated-out generator API
- entry points. Runtime stubs may be omitted if the runtime is considered part
- of the generated-code ABI and the whole subsystem is always enabled in the
- default build; if it is gated, public runtime symbols need stubs too.
-- Keep `src/gram/meta_tables.c` checked in. Normal `make lib` must not require
- Python, network access, UCD regeneration, or a bootstrap `gram` binary.
-- Add a regeneration-only maintenance target later, for example
- `make regen-gram-meta` and `make regen-gram-unicode-props`.
-
-The first import can gate runtime and generator together under
-`KIT_GRAM_ENABLED`. If size-sensitive embeddings need generated parser runtime
-without the generator compiler, split later into:
-
-- `KIT_GRAM_RUNTIME_ENABLED`: parser/lexer runtime plus support headers.
-- `KIT_GRAM_ENABLED`: generator compiler, depending on runtime.
-
-Do not introduce that split until there is a real embedding that benefits from
-it; it adds config and stub surface.
-
-## Implementation Phases
-
-### Phase 0: Freeze Standalone Behavior
-
-- Run the current `ll1` test suite and save the passing command set in the
- import notes.
-- Generate and check in `meta_tables.c` / `meta_tables.h` from `meta.ebnf`.
-- Confirm generated output for representative grammars is stable.
-
-### Phase 1: Mechanical Import, Private Names Still Allowed Internally
-
-- Move files into the destinations above.
-- Keep behavior unchanged while fixing include paths.
-- Add `test/gram` fixtures and a `make test-gram` target, initially allowed to
- fail until the namespace rewrite lands.
-
-### Phase 2: Public Namespace Rewrite
-
-- Rename every public `ll_*`, `llgen_*`, and `LL_*` symbol to `KitGram`,
- `kit_gram_`, or `KIT_GRAM_` spelling.
-- Update emitted C and generated meta tables to use the new names.
-- Run `make test-lib-deps` to catch leaked public symbols outside `Kit`,
- `kit_`, or `KIT`.
-
-### Phase 3: Kit Context Rewrite
-
-- Replace `llgen_allocator` with `KitContext.heap`.
-- Replace generated text return structs with `KitWriter` outputs.
-- Replace direct diagnostics with `KitContext.diag`.
-- Remove hosted libc calls from imported library code.
-- Keep `setjmp`/`longjmp` only if the existing frontend panic pattern accepts
- it for this subsystem; otherwise convert OOM and validation aborts to explicit
- `KitStatus` unwinding.
-
-### Phase 4: Driver Command
-
-- Port `gen/llgen_cli.c` to `driver/cmd/gram.c`.
-- Use the public generator API only.
-- Add help text consistent with other driver commands.
-- Add a focused driver test: invoke `kit gram` on a small grammar, compile the
- generated C against libkit, and run the parser.
-
-### Phase 5: Cleanup and Documentation
-
-- Document the stable runtime API in the public headers.
-- Add a durable design doc under `doc/` only after the subsystem ships.
-- Add `gram` to `README.md` and `doc/DESIGN.md` capability lists after the
- command works.
-- Remove temporary compatibility shims and any retained Python parity path.
-
-## Tests
-
-Targeted tests should land with the import:
-
-- `make test-gram`: direct API compile, table introspection, generated parser,
- generated lexer, Pratt grammar, UTF-8 lexer, and error fixtures.
-- `make test-driver-gram`: CLI generation and generated-code compile/run smoke.
-- `make test-lib-deps`: symbol discipline and no accidental hosted dependencies.
-
-Map standalone tests as follows:
-
-| Standalone test | Kit test |
-|-----------------|----------|
-| `test/test_llgen_api.c` | `test/gram/api_test.c` |
-| `test/test_calc.c` | `test/gram/calc_test.c` |
-| `test/test_features.c` | `test/gram/features_test.c` |
-| `test/test_lexer.c` | `test/gram/lexer_test.c` |
-| `test/test_pratt_calc.c` | `test/gram/pratt_test.c` |
-| `test/test_unicode_support.c` | `test/gram/unicode_test.c` |
-| `test/test_unicode_lexer.c` | `test/gram/unicode_lexer_test.c` |
-| `test/test_utf8_runtime.c` | `test/gram/utf8_runtime_test.c` |
-| `test/errors/*.ebnf` | `test/gram/errors/*.ebnf` |
-
-Prefer red-green import steps:
-
-1. Add the test target and fixtures first.
-2. Import the runtime until hand-written/generated tables parse again.
-3. Import the generator until direct API tests pass.
-4. Add the driver command and smoke test last.
-
-## Open Questions
-
-- Should generated table-layout headers be documented as stable ABI, or merely
- stable enough for C emitted by the same libkit version? The first import should
- promise only same-version compatibility.
-- Should Unicode UCD source data live in-tree permanently, or should only the
- generated property tables be checked in? Keeping the data improves
- reproducible regeneration but increases repository size.
-- Should the runtime/generator gate be split immediately? The plan says no until
- an embedding needs runtime-only size savings.
-- Should `gram` eventually support non-C output modes? The API should not bake
- in more than `emit_c` today, but the command can grow `--emit=` later.
diff --git a/doc/plan/README.md b/doc/plan/README.md
@@ -18,7 +18,7 @@ shrinks to whatever remains open (and is deleted once nothing remains open).
| [ARM32.md](ARM32.md) | 32-bit ARM (`arm-none-eabi`, ARMv7-M/ARMv7E-M Thumb-2, Cortex-M3/M4/M7) freestanding backend: Phase 1 (walking skeleton) is landed; Phase 2 tracks the remaining ops, the -O1 known-frame path, 64-bit, atomics, TLS, and the `qemu-system-arm` cross-test lane. | [../ARCH.md](../ARCH.md), [../PORT.md](../PORT.md) |
| [SYSROOTS.md](SYSROOTS.md) | Cross-compile sysroot packaging: minimal per-target stubs/headers/CRT objects distributed via `.kpkg` for the support set. Design complete, implementation not yet started. | — |
| [BUILD.md](BUILD.md) | A new content-addressed build coordinator (Bazel/Nix-style incremental builds layered on the CAS) — storage state machine, caching algorithm, recipe protocol. Design, not yet built. Distinct from `../BUILD.md` (kit's own Makefile build). | — (new subsystem) |
-| [LLGEN_IMPORT.md](LLGEN_IMPORT.md) | Importing the standalone LL(1)/Pratt parser and lexer generator into libkit, including public API renames, file moves, build gates, and a `kit llgen` command. Not yet started. | — |
+| [GRAM_IMPORT.md](GRAM_IMPORT.md) | One-time import of the standalone `gramgen` EBNF parser/lexer/token-machine generator (`/Users/ryan/code/gram`) into libkit: full `kit_gram_*`/`KitGram`/`KIT_GRAM_` rename, file moves, build gates, and a `kit gram` command. Not yet started. | — |
| [TODO.md](TODO.md) | Open deferred fixes and code smells, plus terse backlog folded from retired plan docs (arch-backend parity, Wasm object backend, Windows x64 self-host, bootstrap breadth). Completed items are removed instead of checked off. A current backlog, not a roadmap. | — |
Speculative, not-committed designs (no code, parked) live in [`../ideas/`](../ideas/)
