gram — parser / lexer generator

gram is a libkit subsystem that turns an EBNF grammar into C: a table-driven LL(1) (+ Pratt) parser, a DFA lexer, regular %machine token automata, and optional standalone (re2c-style) scanners. It is not a lang/ frontend — it registers no KitFrontendVTable and emits no KitCg. Its surface is consumed by embedders, by the kit gram driver command, and by the C it generates.

It was imported as a one-time fork of the standalone gramgen project. The public kit_gram_* names and the interfaces described here are the maintained Kit surface.

Public API

All allocation flows through KitContext.heap; all diagnostics through KitContext.diag. The runtimes are allocation-free and caller-owned.

• <kit/gram.h> — the in-process compiler API. kit_gram_compile_text parses EBNF into a KitGramCompiled; kit_gram_emit_c writes the generated C header+source to two KitWriters; kit_gram_dump_sexpr dumps the parsed syntax tree; introspection (kit_gram_token_*, _rule_*, _lexer_*, _machine_*); kit_gram_regex_compile wraps a single pattern; the kit_gram_*_generate* entry points synthesize sample token streams / machine traces. kit_gram_free releases a compiled grammar.
• <kit/gram_parse.h> — the push, table-driven LL(1)/Pratt parser runtime that generated parsers link against (kit_gram_parser_init/_push/_finish, the KitGramActions listener+value vtable, stack-sizing helpers).
• <kit/gram_lex.h> — the DFA lexer runtime, the re2::Set-style match API (kit_gram_match_find/_anchored/_full, the non-overlapping match iterator), and the %machine stepping types.
• <kit/gram_pos.h> — header-only lazy offset → (line, col) indexes (byte and UTF-8), for lexers run with positions off.
• <kit/gram_unicode.h> — UTF-8 / Unicode scalar helpers.
• <kit/support/gram_parse_tables.h>, <kit/support/gram_lex_tables.h> — the table layouts the generated C fills in (generated-code support, not ordinary embedder API).

Opaque, stack-allocatable runtime objects (KitGramParser, KitGramLexer, KitGramMatcher, …) are sized by _Static_assert-guarded KIT_GRAM_*_SIZE constants, so the caller owns every byte and the driver allocates nothing.

Command

kit gram [options] grammar.ebnf (or - for stdin) generates <grammar>.c / .h (override with -o / --header / --prefix). Other modes: --dump-sexpr, --sample-tokens, --sample-traces NAME. Lexer/parser knobs: --multiline, --lexer-standalone, --fold-keywords, --position-lazy, --parser-codegen, --parser-recover. See kit gram --help.

Implementation layout (src/gram/)

• generator.c — the public compiler API: EBNF front end, parser-table generation, and the C emitter. Imported as one translation unit.
• ll1.c — LL(1) / Pratt construction, FIRST/FOLLOW, validation.
• lex_dfa.c / lex_range.c — shared NFA→DFA core and the alphabet-neutral range-set NFA. lex_byte.c / lex_scalar.c / lex_unicode.c / lex_tokens.c — the byte / UTF-8 scalar / Unicode-property / %machine pipelines.
• parse_runtime.c / lex_runtime.c — the parser and lexer+match runtimes.
• unicode.c — UTF-8 / scalar / position helpers. unicode_props.{c,h} — generated UCD 17.0.0 property tables (a private generated header).
• meta.ebnf + meta_tables.{c,h} — the generator's own EBNF grammar and its checked-in generated parse tables, so a normal build needs no bootstrap.
• internal.h — private to src/gram/*.c.

Build gating

KIT_GRAM_ENABLED (library: runtime + generator, gated together) and KIT_TOOL_GRAM_ENABLED (the gram driver command) in include/kit/config.h. When gated out, src/api/config_stubs.c provides KIT_UNSUPPORTED stubs so the public ABI surface stays complete.

Tests & regeneration

• make test-gram — functional + diagnostic suite (test/gram/): generated parsers/lexers/machines, the in-process API, standalone scanners, lazy positions, table-free link checks, and every errors/ / machine_errors/ fixture.
• make test-driver-gram — kit gram CLI behavior smoke.
• make regen-gram-meta — re-emit meta_tables.{c,h} from meta.ebnf with the just-built kit gram (output checked in; git diff shows drift).
• make regen-gram-unicode-props — re-emit unicode_props.{c,h} from the vendored, checksummed UCD (data/ucd/17.0.0/); offline and reproducible.