kit

kit
git clone https://git.ryansepassi.com/git/kit.git
Log | Files | Refs | README

Frontend shared tagged cell

Goal. Define the end-state shape for the C front half around one shared, compact cell used by the lexer, preprocessor, and parser input ring. This is a design document, not a phasing plan.

The value-stack redesign lives in doc/plan/CG-STACK-API.md: the C parser drives KitCg directly, KitCg owns the single liveness-authoritative value stack, C type/value facts live in the CG language sidecar, and constant values live in the CG constant payload. This document deliberately stops at the front-half cell and the handoff from parser primary to CG stack.

In one line: one compact lexeme cell is shared by lexer, preprocessor, and the parser input ring; parser-only semantic tagging happens only on parser-owned cells; the preprocessor remains independently drainable; the CG stack remains the only live expression stack.

Non-Goals

Ownership Boundaries

  1. Lexer and pp own the lexeme contract. A lexeme cell is complete enough for macro expansion, directives, -E serialization, diagnostics, and replay. It carries lazy LocRef/TextRef, eagerly interned identifiers, punctuator codes in aux, and no C parser state.
  2. The pp is independently drainable. cpp, cc -E, and KIT_PP_DRAIN pull cells into caller-owned storage with no parser attached. The pp reads and writes only the lexeme view.
  3. The parser owns semantic tagging. The parser may tag only cells in its own input ring or short-lived scratch. It must never tag macro-definition storage, directive buffers, or pp-owned source buffers.
  4. Macro replay copies before tagging. Macro bodies are immutable lexeme-cell arrays. When replay emits a token, pp copies one lexeme into the caller's output cell, applies loc/first-token flag overrides, and returns. The parser may then tag that caller-owned copy.
  5. KitCg owns values. Once a primary is accepted, the parser lowers through kit_cg_*, stamps the CG top with kit_cg_retag_*, and advances. The cell can carry debug/lowering facts until the next advance, but expression lifetime is the CG stack slot.

Shared Cell

The durable stream/storage unit stays the current lean-token shape: 32 bytes on LP64, trivially copyable, and without heap ownership. If the implementation renames the type, today's Tok should become an alias of this cell rather than a second representation.

The shared header must not include C parser or CG internals.

typedef enum CFeKind {
  /* 0..0x11ff remain TokKind / pp-internal token kinds. */
  CFE_SEM_FIRST = 0x8000,
  CFE_SEM_VALUE = CFE_SEM_FIRST, /* parser-only: current token lowered as value */
  CFE_SEM_PLACE,                 /* parser-only: current token lowered as place */
} CFeKind;

typedef struct CFeSem {
  const void* lang_type; /* C parser: const Type*; opaque to lex/pp */
  u32 cg_type;          /* KitCgTypeId value, represented without cg includes */
  u32 lang_flags;       /* same C value flags passed to kit_cg_retag_* */
} CFeSem;

typedef struct CFeCell {
  u16 kind;   /* TokKind in lexeme view; CFeKind >= CFE_SEM_FIRST after tagging */
  u16 flags;  /* lexeme: TF_*; semantic: parser-private flags */
  u32 aux;    /* lexeme: Sym/Punct/PP_PARAM; semantic: small discriminator */
  LocRef loc; /* survives both views; line/col materialize lazily through Pp */
  union {
    TextRef text; /* lexeme spelling ref; lexer/pp/drain view */
    CFeSem sem;   /* parser-only semantic seed; pp never reads this */
  } u;
} CFeCell;

_Static_assert(sizeof(CFeSem) <= sizeof(TextRef),
               "semantic overlay must not widen the lexeme cell");
_Static_assert(sizeof(CFeCell) == 32,
               "cell must stay the compact token-stream storage unit");

Cell rules:

Lexer Contract

The lexer writes one caller-owned cell and returns nothing by value:

void lex_next(Lexer*, CFeCell* out);

Lexeme shape:

Materialization remains pp-owned because only Pp retains source buffers, splice tables, and #line overlays:

SrcLoc   pp_materialize_loc(Pp*, LocRef loc);
KitSlice pp_text_slice(Pp*, const CFeCell* cell);
Sym      pp_text_intern(Pp*, const CFeCell* cell);
int      pp_text_eq_cstr(Pp*, const CFeCell* cell, const char* s);

Preprocessor Contract

The pp exposes two drainable pulls, both out-pointer only:

void pp_next_parse(Pp*, CFeCell* out); /* expanded, directives consumed,
                                        * non-directive newlines suppressed */
void pp_next_raw(Pp*, CFeCell* out);   /* expanded, directives consumed,
                                        * TOK_NEWLINE preserved for -E/cpp */
void pp_emit_text(Pp*, Writer* out);   /* drains pp_next_raw */

Internal pp storage is lexeme-only:

typedef struct CFeReplay {
  const CFeCell* cells;
  u32 n;
  Macro* disabled_owner; /* non-NULL while rescanning this macro replacement */
  LocRef loc_override;
  u16 first_flags_or;
} CFeReplay;

Requirements:

Macro Expansion Availability

Use the simpler cpplib-style availability model instead of a per-token hideset table: a macro is disabled while its own replacement-list frame is being rescanned, and an identifier token has one permanent unavailable bit.

The unavailable bit is TF_NO_EXPAND in lexeme view. It means "this identifier token must not be macro-expanded if it is seen again." One bit is sufficient because the token's spelling names at most one macro at the time it is tested; the bit does not need to encode a set of macro names.

Suggested pp-owned state:

typedef struct Macro {
  Sym name;
  u16 disabled_depth;
  /* existing macro definition fields... */
} Macro;

typedef struct TokSrc {
  u8 kind;
  CFeCell* cells;
  u32 i;
  u32 n;
  Macro* disabled_owner; /* non-NULL for macro replacement-list frames */
  /* existing source/replay fields... */
} TokSrc;

Frame rules:

Identifier test in pp_next_*:

if (cell.kind == TOK_IDENT && (cell.flags & TF_NO_EXPAND) == 0) {
  Macro* m = mt_get(pp, tok_ident(&cell));
  if (m && m->disabled_depth != 0) {
    cell.flags |= TF_NO_EXPAND;
    *out = cell;
    return;
  }
  if (m && macro_invocation_is_present(pp, m, &cell)) {
    push_replacement_frame(pp, m, &cell);
    continue;
  }
}

Details that matter:

Parser Input Ring

The parser consumes pp output through a small fixed ring of parser-owned cells. APIs return pointers to parser-owned cells, never cells by value:

typedef struct ParserInput {
  CFeCell cells[3];     /* fixed cur + LL(2) lookahead ring; no VLA */
  u8 cur;               /* index of the current token in cells[] */
  u8 nvalid;            /* number of valid slots from cur, 0..3 */
  CFeCell pending;      /* string-literal fusion / one-token pushback */
  u8 has_pending;
} ParserInput;

void parser_advance(Parser*);
void parser_fill_to(Parser*, u32 depth); /* cold path: fills through depth 0..2 */

static inline u8 parser_ring_idx(const ParserInput* in, u32 depth) {
  return (u8)((in->cur + depth) % 3u); /* depth 0 == current */
}

static inline CFeCell* parser_cur(ParserInput* in) {
  return &in->cells[in->cur];
}

static inline const CFeCell* parser_cur_const(const ParserInput* in) {
  return &in->cells[in->cur];
}

static inline const CFeCell* parser_peek(Parser* p, u32 depth) {
  ParserInput* in = &p->input;
  if (in->nvalid <= depth) parser_fill_to(p, depth);
  return &in->cells[parser_ring_idx(in, depth)]; /* depth 1..2 */
}

Rules:

Parser Tagging

Tagging is local to a consumed primary. It is not a pp feature and it is not an expression stack.

The shared-cell helper has no C or CG header dependency:

static inline void cfe_tag_primary_raw(CFeCell* c, u16 sem_kind,
                                       const void* lang_type,
                                       u32 cg_type,
                                       u32 lang_flags) {
  c->kind = sem_kind;          /* CFE_SEM_VALUE or CFE_SEM_PLACE */
  c->flags = 0;                /* parser-private flags if needed */
  c->u.sem.lang_type = lang_type;
  c->u.sem.cg_type = cg_type;
  c->u.sem.lang_flags = lang_flags;
}

The C parser may wrap it with C/CG-specific types:

static inline void c_tag_primary(CFeCell* c, u16 sem_kind,
                                 const Type* type,
                                 KitCgTypeId cg_type,
                                 u32 c_flags) {
  cfe_tag_primary_raw(c, sem_kind, type, (u32)cg_type, c_flags);
}

Identifier primary:

  1. TOK_IDENT carries Sym in aux.
  2. The parser checks kw_map; otherwise it reads BindingTab once.
  3. The parser emits the corresponding CG seed: local place, global/function address, enum constant, integer constant, or other binding-specific form.
  4. The parser calls kit_cg_retag_top(p->cg, type, flags) with the same semantic facts it wrote into the cell.
  5. The parser advances; the tagged cell is no longer live.

Literal primary:

  1. The parser reads spelling, suffix, and encoding from lexeme view.
  2. The parser decodes only when the literal becomes a value.
  3. The parser pushes through the CG constant/value API. Width-complete integer constants use KitCgConstInt from CG-STACK-API.md.
  4. The parser retags the CG top with C type/value flags.
  5. The parser may tag the current cell as a debug/lowering seed until advance.

Operators never tag tokens. They consume CG stack entries via kit_cg_*, query KitCgSlotInfo, and retag results according to CG-STACK-API.md. The parser input cell can be tagged; the live value is still the CG stack slot.

CG-Facing Dependencies

This design depends on the CG stack contract from CG-STACK-API.md:

void          kit_cg_lang_enable(KitCg*);
KitCgSlotInfo kit_cg_slot_info(KitCg*, u32 depth_from_top);
void          kit_cg_retag_top(KitCg*, const void* lang_type, u32 lang_flags);
void          kit_cg_retag_at(KitCg*, u32 depth_from_top,
                              const void* lang_type, u32 lang_flags);
void          kit_cg_set_top_flags(KitCg*, u32 set, u32 clear);

int  kit_cg_top_const_int_ex(KitCg*, KitCgConstInt* out);
void kit_cg_push_const_int(KitCg*, KitCgTypeId type,
                           const KitCgConstInt* value);

CFeSem deliberately mirrors the non-constant part of KitCgSlotInfo: {lang_type, cg_type, lang_flags}. That keeps parser tagging and CG retagging the same shape without making the pp pay for parser-only facts.

End-State Data Flow

source bytes
  -> lex_next(Lexer*, CFeCell*)              writes lexeme view
  -> pp source/replay stack                  reads/writes lexeme view only
  -> pp_next_parse/raw(Pp*, CFeCell*)        fills caller-owned cell
     -> cpp / cc -E / KIT_PP_DRAIN           drains lexeme cells; no parser
     -> ParserInput ring                     parser-owned cells
        -> primary resolution                optional in-place semantic tag
        -> kit_cg_* push/op + retag          CG stack owns values
        -> CgTarget -> NativeTarget -> MC    backend seam unchanged

The fusion point is the front-half representation and handoff: eliminate by-value token returns and duplicate token shapes, preserve pp drainability, and do not create a second value representation beside KitCg.

Correctness Checklist