commit 80a4f4e82fddce173659bf6d9aec676cb9615f57
parent d59a75252c48f351272b3e61368818584582118b
Author: Ryan Sepassi <rsepassi@gmail.com>
Date: Tue, 9 Jun 2026 12:40:12 -0700
pp: escape backslashes/quotes when building the __FILE__ literal
The preprocessor built the __FILE__ string literal by wrapping the raw
source path in quotes without escaping \ or ". On POSIX paths use '/', so
this was an accidental no-op; on Windows the on-demand rt build passes a
native path (C:\Users\kit\work\...\support\rt\lib\int\int.c), so __FILE__
at int.c:203's compilerrt_abort() (-> __compilerrt_abort_impl(__FILE__,
...)) expanded to "C:\Users\..." where the \U of \Users is an invalid
universal-character-name -> `int.c:203:46: fatal: malformed UCN`. Other
path components silently corrupted the value (e.g. a \r component became a
carriage return). The reported column and the forward-slash spelling in
the message were both red herrings; this was never a lexer/file-read bug
nor a self-compile codegen defect.
Escape \ and " when building the __FILE__ literal (matching
make_stringize, the # operator). do_line now destringizes the #line
filename so file_override stores logical bytes, keeping the now-uniform
escaping from double-escaping a #line "a\\b.c" override.
Regression test test/pp/run_file_escape.sh (wired into test-pp) creates
real backslash-/quote-named source files -- the faithful OS-path repro,
since a #line override cannot reproduce it (its escapes already sit in the
token spelling) -- plus a #line guard for the destringize path.
VM-verified on aarch64-windows: a fixed cross-built kit.exe compiles
hello_printf.c on the VM, building the runtime on-demand (the int.c via a
C:\Users\... path) with no diagnostic, and the resulting exe prints and
returns its exit code.
Diffstat:
5 files changed, 136 insertions(+), 38 deletions(-)
diff --git a/doc/plan/windows.md b/doc/plan/windows.md
@@ -83,26 +83,31 @@ are collected first; the numbered sections are the larger roadmap items.
Concrete defects surfaced during bring-up, each blocking a roadmap item below.
-- **AOT `malformed UCN` building rt on the VM** (blocks native `kit cc`'s
- on-demand rt path, and §5 bootstrap). On the VM, `kit.exe cc <prog>` fails
- while compiling the runtime:
- `support/rt/lib/int/int.c:203:46: fatal: malformed UCN`. That line is
- `if (a == ((di_int)((du_int)1 << (N - 1)))) compilerrt_abort();` — there is no
- `\u`/`\U` anywhere near column 46, so "malformed UCN" is a *misdiagnosis*: the
- Windows-hosted lexer is mis-tokenizing valid source. It is **not** a stale
- deploy — it reproduces with a freshly-shipped rt tree, and with the *baseline*
- `kit.exe` (predating the JIT-gap work), so it is pre-existing. The macOS host
- `build/kit` compiles the *same* `int.c` for `aarch64-windows` cleanly
- (`make rt-aarch64-windows` succeeds), so the bug is specific to the
- **Windows-hosted** kit.exe. Leading hypotheses: (a) a Windows-only file-read /
- line-ending / encoding path feeding the lexer; or (b) a latent
- **self-compile codegen bug** in the cross-built kit.exe that only manifests in
- the lexer's UCN/escape handling — which would also threaten §5 (a stage-2
- kit.exe that miscompiles its own lexer). Worth bisecting against a known-good
- source byte stream on the VM (e.g. `kit cc` a minimal file with the exact
- bytes of line 203) to separate (a) from (b). *Workaround used for verifying
- the JIT fixes:* host-cross-compile the `.exe` on macOS, then run it on the VM,
- which never invokes the VM-side rt build.
+- **AOT `malformed UCN` building rt on the VM — FIXED.** Root cause was neither
+ hypothesis: not a lexer/file-read bug, and (importantly for §5) **not** a
+ self-compile codegen defect. The preprocessor built the `__FILE__` string
+ literal by wrapping the raw source path in quotes **without escaping** `\` or
+ `"` (`pp_expand.c`, the `sym_file__` branch). On POSIX paths use `/`, so this
+ was an accidental no-op; on Windows the on-demand rt build passes a native path
+ (`C:\Users\kit\work\...\support\rt\lib\int\int.c`), so `__FILE__` at line 203's
+ `compilerrt_abort()` (→ `__compilerrt_abort_impl(__FILE__, …)`) expanded to
+ `"C:\Users\…"` — the `\U` of `\Users` is an invalid universal-character-name →
+ "malformed UCN" (and other components silently corrupted the value, e.g. a
+ `\r` component became a carriage return). The reported column (46) and the
+ forward-slash spelling in the message were both red herrings.
+ Fix: the `__FILE__` builder now escapes `\` and `"` (matching `make_stringize`,
+ the `#` operator); and `do_line` destringizes the `#line` filename so
+ `file_override` stores logical bytes — keeping the now-uniform escaping from
+ double-escaping a `#line "a\\b.c"` override. Regression test
+ `test/pp/run_file_escape.sh` (wired into `test-pp`) creates real
+ backslash-/quote-named source files — the faithful OS-path repro, since a
+ `#line` override can't reproduce it (its escapes already sit in the token
+ spelling) — plus a `#line` guard for the destringize path.
+ **VM-verified on aarch64-windows:** a fixed cross-built `kit.exe` compiles
+ `hello_printf.c` on the VM, building the runtime on-demand (the very `int.c`
+ via a `C:\Users\…` path) with no diagnostic, and the resulting `.exe` prints
+ and returns its exit code. The earlier *workaround* (host-cross-compile the
+ `.exe` on macOS to dodge the VM-side rt build) is no longer needed.
- **x64 cross-link fails on `_setjmp`** (blocks all of §2, x86_64-windows
parity). `scripts/windows_cross.sh x64` cannot link an x64 `kit.exe`:
@@ -208,14 +213,15 @@ The self-host milestone: use the cross-built `kit.exe` on the VM to compile
kit's own sources into a stage-2 `kit.exe`, then stage-3, and assert stage-2 ==
stage-3 byte-for-byte (cf. [BOOTSTRAP.md](BOOTSTRAP.md), and the
native-bootstrap analogs `scripts/{linux,freebsd}_bootstrap.sh`). **Unblocked**
-now that the large-frame stack-probe crash is fixed — kit.exe compiles + links +
-runs the full Toy AOT corpus (166/166) on the VM. Next: confirm kit.exe can
-compile the full libkit/driver source set on the VM, then add a
+now that the large-frame stack-probe crash *and* the `__FILE__` `malformed UCN`
+bug are fixed — kit.exe compiles + links + runs the full Toy AOT corpus
+(166/166) on the VM, and now compiles the rt tree on the VM cleanly (the
+on-demand `kit cc` rt build succeeds). The `malformed UCN` bug turned out to be
+a host-path stringization defect, **not** a self-compile codegen defect, so it
+does not threaten stage-2==stage-3 identity. Next: confirm kit.exe can compile
+the full libkit/driver source set on the VM, then add a
`scripts/windows_bootstrap.sh` to drive the VM-side stages and the
-stage-2==stage-3 byte-identity check. **Gated** by the `malformed UCN` bug (see
-*Open bugs / blockers*): a stage-1 kit.exe that can't even compile the rt source
-tree on the VM can't compile libkit there — and if that bug is a self-compile
-codegen defect, it threatens stage-2==stage-3 identity directly.
+stage-2==stage-3 byte-identity check.
## 6. Debugger fault-guard / SEH on Windows
diff --git a/lang/cpp/pp/pp_directive.c b/lang/cpp/pp/pp_directive.c
@@ -3,6 +3,9 @@
#include "pp/pp_priv.h"
+static void destringize(Pp* pp, const Tok* str_tok, char* out, size_t cap,
+ size_t* out_len);
+
/* ============================================================
* If-stack
* ============================================================ */
@@ -866,8 +869,14 @@ static void do_line(Pp* pp, const Tok* line, u32 n, SrcLoc loc) {
{
KitSlice s = kit_sym_str(pp->pool->c, exp.data[1].spelling);
if (s.len >= 2 && s.s[0] == '"' && s.s[s.len - 1] == '"') {
- target_file = kit_sym_intern(
- pp->pool->c, (KitSlice){.s = s.s + 1, .len = s.len - 2});
+ /* Destringize to logical bytes (undo \" and \\): file_override is
+ * stored unescaped, like a real source path, and __FILE__ re-escapes
+ * it uniformly when expanded. */
+ char* fbuf = (char*)arena_alloc(pp->arena, s.len, 1);
+ size_t flen = 0;
+ destringize(pp, &exp.data[1], fbuf, s.len, &flen);
+ target_file =
+ kit_sym_intern(pp->pool->c, (KitSlice){.s = fbuf, .len = flen});
}
}
}
diff --git a/lang/cpp/pp/pp_expand.c b/lang/cpp/pp/pp_expand.c
@@ -1035,14 +1035,28 @@ Tok pp_next_raw(Pp* pp) {
nstr = s.s;
nlen = s.len;
}
- buf = (char*)arena_alloc(pp->arena, nlen + 2, 1);
- buf[0] = '"';
- if (nlen) memcpy(buf + 1, nstr, nlen);
- buf[nlen + 1] = '"';
- t.kind = TOK_STR;
- t.spelling =
- kit_sym_intern(pp->pool->c, (KitSlice){.s = buf, .len = nlen + 2});
- t.v.str = t.spelling;
+ /* The source name is the raw filesystem path (or a #line override,
+ * destringized to logical bytes by do_line). Re-stringize it as a
+ * valid C string literal: escape '\\' and '"'. On POSIX paths use
+ * '/' so this was a no-op; on Windows the path holds backslashes
+ * (e.g. C:\\Users\\...), and emitting them raw turns '\\U'/'\\u'/'\\x'
+ * into bogus escape sequences (the "malformed UCN" on '\\Users'). */
+ {
+ size_t bn = 0;
+ size_t i;
+ buf = (char*)arena_alloc(pp->arena, nlen * 2 + 2, 1);
+ buf[bn++] = '"';
+ for (i = 0; i < nlen; ++i) {
+ char ch = nstr[i];
+ if (ch == '\\' || ch == '"') buf[bn++] = '\\';
+ buf[bn++] = ch;
+ }
+ buf[bn++] = '"';
+ t.kind = TOK_STR;
+ t.spelling =
+ kit_sym_intern(pp->pool->c, (KitSlice){.s = buf, .len = bn});
+ t.v.str = t.spelling;
+ }
return t;
}
if (id == pp->sym_date__) {
diff --git a/mk/test.mk b/mk/test.mk
@@ -107,6 +107,7 @@ TEST_TARGETS = \
test-pp \
test-pp-err \
test-pp-ok \
+ test-pp-file-escape \
test-rt-headers \
test-rt-runtime \
test-rt-backtrace \
@@ -254,7 +255,7 @@ test-wasm-toy: bin
test-wasm-c: bin $(PARSE_RUNNER)
@KIT_TEST_PATHS=W KIT_TEST_ALLOW_SKIP=1 KIT=$(abspath $(BIN)) bash test/parse/run.sh
-test-pp: test-pp-ok test-pp-err
+test-pp: test-pp-ok test-pp-err test-pp-file-escape
test-pp-ok: bin
@KIT=$(abspath $(BIN)) test/pp/run.sh
@@ -262,6 +263,9 @@ test-pp-ok: bin
test-pp-err: bin
@KIT=$(abspath $(BIN)) test/pp/run_errors.sh
+test-pp-file-escape: bin
+ @KIT=$(abspath $(BIN)) bash test/pp/run_file_escape.sh
+
# Best-effort kit binary build: Layer D needs build/kit, but the
# binary may not link until enough libkit symbols exist. The harness
# detects a missing binary and skips that layer; don't break test-elf
diff --git a/test/pp/run_file_escape.sh b/test/pp/run_file_escape.sh
@@ -0,0 +1,65 @@
+#!/usr/bin/env bash
+# test/pp/run_file_escape.sh — regression for __FILE__ string-literal escaping.
+#
+# __FILE__ expands to a string literal of the source path. On POSIX paths use
+# '/', so wrapping the raw name in quotes happens to be a valid string literal;
+# on Windows the path holds backslashes (C:\Users\...), and emitting them raw
+# turns '\U'/'\u'/'\x' into bogus escape sequences — '\Users' was diagnosed as a
+# "malformed UCN" and other components (e.g. '\r') silently corrupted the value.
+# The preprocessor must escape '\' and '"' when building the __FILE__ literal.
+#
+# This reproduces the OS-path case faithfully (a backslash IS a legal filename
+# byte on POSIX): we create files whose *names* contain backslashes/quotes and
+# compile them, rather than a #line override (whose escapes are already in the
+# token spelling, so it cannot reproduce the raw-path bug). A #line case is
+# included too, to guard the symmetric destringize-then-re-escape path.
+#
+# Host-target only: each input is a self-contained TU (no includes), so no
+# sysroot is needed and the test runs anywhere build/kit exists.
+
+set -u
+
+ROOT=${KIT_TEST_ROOT:-$(cd "$(dirname "$0")/../.." && pwd)}
+KIT=${KIT:-"$ROOT/build/kit"}
+export KIT
+
+KIT_KIT_DIR="$ROOT/test/lib"
+. "$ROOT/test/lib/kit_sh_kit.sh"
+kit_require_kit pp-file-escape
+
+kit_report_init
+work=$(mktemp -d "${TMPDIR:-/tmp}/kit-pp-fesc.XXXXXX")
+trap 'rm -rf "$work"' EXIT
+cd "$work" || exit 2
+
+# emit_has NAME SRCFILE NEEDLE : `kit cc -E SRCFILE` must succeed and its output
+# must contain NEEDLE (the correctly-escaped spelling).
+emit_has() {
+ name=$1; src=$2; needle=$3
+ if "$KIT" cc -E "$src" -o "$name.pp" >"$work/$name.out" 2>"$work/$name.err"; then
+ contains "$name" "$name.pp" "$needle"
+ else
+ not_ok "$name" "$work/$name.err"
+ fi
+}
+
+# --- raw OS path: '\U' was the hard "malformed UCN" error -------------------
+printf 'const char *f = __FILE__;\n' > 'seg\Uone.c'
+emit_has emit_escape_U 'seg\Uone.c' 'seg\\Uone.c'
+run_ok compile_backslash_U "$KIT" cc -c 'seg\Uone.c' -o "$work/u.o"
+
+# --- raw OS path: '\r' silently injected a carriage return (no error) -------
+printf 'const char *f = __FILE__;\n' > 'seg\rtwo.c'
+emit_has emit_escape_r 'seg\rtwo.c' 'seg\\rtwo.c'
+
+# --- embedded double-quote in the name must escape too ----------------------
+printf 'const char *f = __FILE__;\n' > 'q"three.c'
+emit_has emit_escape_quote 'q"three.c' 'q\"three.c'
+
+# --- #line override: destringize-then-re-escape must stay correct -----------
+printf '#line 7 "a\\\\Users\\\\z.c"\nconst char *f = __FILE__;\n' > 'lineov.c'
+emit_has emit_line_override 'lineov.c' '"a\\Users\\z.c"'
+run_ok compile_line_override "$KIT" cc -c 'lineov.c' -o "$work/l.o"
+
+kit_summary pp-file-escape
+kit_exit
