Working on Auk

Auk compiler architecture

advanced 35 min read

On this page

The language is docs/spec.md; packages and the core library are in docs/raft.md. This file says how the compiler is built. Every phase is checked against the corpus: examples/*.auk plus packages/*/src/*.auk. "Done" means the corpus compiles and its test blocks pass.

Pipeline#

source ─▶ lex ─▶ parse ─▶ resolve ─▶ check ─▶ lower ─▶ MIR ─▶ LLVM IR ─┬─▶ JIT (ORC): run in this process (run, test, dive, hatch, imprints)
         tokens   AST     modules,   types,    explicit   structured      └─▶ object file + libauk_rt.a ─▶ ld (link) ─▶ native exe (build)
                          names      patterns, RC/control C-like IR
                                     effects,  flow
                                     shapes
  • Written in Auk. The compiler is the Auk project compiler/ (compiler/src/main.auk), built by ./build.sh from a seed auk (the previous build, or the checked-in bitcode in seed/) to the bootstrap fixed point; the runtime is Auk too (runtime/src, the package auk_rt in the raw dialect, docs/lowlevel.md; built with auk build --lib; runtime/c/auk_rt.h documents its C ABI). Dependencies are kept few and deliberate: LLVM 23 through its C API (the llvm package, packages/llvm, linking libLLVM dynamically: Homebrew's llvm@23 on macOS, apt.llvm.org's on Linux; LLVM_CONFIG picks another) and the system linker (no C compiler: building Auk needs LLVM's tools and the platform's linker and C library stubs; only the tests of C interop compile C, with cc); the runtime itself uses only libc, pthreads and system frameworks. The Rust compiler it was ported from is the tag v0-rust-final (design/self-host-plan.md).
  • Backend = LLVM. auk run/test compile the program in memory and call its entry point in the compiler's own process: the runtime (runtime/c) is linked into auk (exported to dlsym), and the JIT defines each runtime symbol the module calls at its address in this process (orcjit.auk). No C compiler is involved.
  • User C functions (FFI). extern fn with a C prototype in a package with the ffi effect (design/ffi-plan.md): compiler/src/cffi.auk parses the prototype and matches each Auk type to its C type (E0040), giving a Conv per parameter and result (FnKind::Ffi, Target::Ffi); mir/lower/ffi.auk marshals the call with helpers from runtime/c/ffi.c. The JIT dlopens the [link] libs of every package and checks each called symbol with dlsym before linking (orcjit.open_libs); auk build passes -l<name>. extern types of user code are plain C pointers: not reference counted, compared and shown by address.
  • Optimization levels (backend.auk, Level): tests, imprints, the debugger, its evaluations and the REPL compile with none (O0); auk run with a few cheap passes (Light: globaldce,function(sroa,early-cse,instcombine,simplifycfg)); auk run -O and auk build with LLVM's default<O2>. Every level first drops functions nothing uses (globaldce), and O0 and Light generate code with the fast instruction selector (FastISel): on AArch64 LLVM's default at O0, GlobalISel, has a pass that is quadratic in a block's size, and SelectionDAG doubles start-up for little gain. AUK_LLVM_PASSES (a pipeline, empty for none) and AUK_LLVM_CODEGEN (0-3) override them, for tuning.
  • auk build (release: O2, no ensures) writes an object file whose main is the entry point and links it with libauk_rt.a (the runtime as a static library: build.sh builds runtime/src with auk build --lib, and the compiler embeds the archive) with the system linker, called directly (no C compiler driver; compiler/src/native.auk, and scripts/link.sh, the same in shell, for build.sh): on macOS Apple's ld with -arch, -platform_version macos <min> <sdk> ($MACOSX_DEPLOYMENT_TARGET or this Mac's version; the SDK's from its SDKSettings.json), -syslibroot <sdk> ($SDKROOT, else the selected developer directory's or the Command Line Tools' SDK, else xcrun's) and -lSystem; on Linux ld.lld (LLVM's bin directory, else the PATH), else GNU ld, making a PIE with glibc's Scrt1.o/crti.o/crtn.o (the first of /usr/lib/<multiarch>, /lib/<multiarch>, /usr/lib64, /lib64, /usr/lib, /lib that has them, all of them as -L), the psABI's dynamic linker (else /bin/sh's PT_INTERP), -lm -lc (GNU ld also GCC's crtbeginS.o/crtendS.o, for __dso_handle, which ld.lld defines itself); both export the symbols (-export_dynamic/--export-dynamic). AUK_LINKER names a C compiler driver (cc, clang) to link with instead, with driver flags. Embedding LLD was the alternative: LLVM's C API has no linker (LLD's entry points are C++, and Homebrew's and apt.llvm.org's libLLVM don't contain it), so it would need a C++ shim, i.e. a C++ compiler.

Source layout#

file contents
compiler/src/main.auk, drive.auk, cli.auk, posix.auk, embedded.auk the auk CLI: command dispatch; drive: check, generate, JIT run, build, traces, project roots, forked children; cli: auk check; embedded: the prelude, docs/spec.md and the error pages compiled in (#embed), so the binary needs no checkout
compiler/src/dump.auk the hidden auk dump tokens|ast|resolve|check|typed: line-oriented dumps with every span, the loaded module tree, the typed program
compiler/src/token.auk, lexer.auk tokens with spans; the newline rule, interpolation parts, raw/triple strings and escapes are all lexer jobs
compiler/src/ast.auk the full AST: items (fn clause, struct, enum, shape, test, top-level let/imprint), patterns, expressions, types; every node has a Span
compiler/src/parser.auk recursive descent; precedence climbing per the spec's table; groups clauses after parsing
compiler/src/resolve.auk, paths.auk the module tree (project src/** + .raft/<dep>/src/** + core), path resolution; paths: file paths, realpath, the OS
compiler/prelude/*.auk the core library as Auk source compiled into the compiler: extern fn bindings to the runtime, @ intrinsics, and generic collection functions written in Auk
compiler/src/types.auk, check.auk, check_expr.auk, check_ops.auk, check_pat.auk, check_shapes.auk, visit.auk types, inference driven by the expected type, local scopes, overloading and UFCS, patterns (typing), shapes (fit, monomorphization, vtables), auto-wrapping to T?/T!; diagnostics with fix-its
compiler/src/units.auk units of measure: parsing <m/s^2> into a canonical form, display, and erase (strips every unit from the typed program, so codegen only sees int/float)
compiler/src/typed.auk the typed program handed to codegen: side tables keyed by node id, resolved call targets, instances to emit
compiler/src/patterns.auk exhaustiveness and reachability (the usefulness algorithm)
compiler/src/check_raw.auk, mir/lower/lowlevel.auk, rawgen.auk, compiler/prelude/raw.auk the runtime's dialect (docs/lowlevel.md, only for the package auk_rt): explicit type arguments, fields through pointers, raw.field_addr/offset_of/fn_of, constant globals and raw assert (checker); its intrinsics, globals and conversions (lowering); memory intrinsics, atomics, _setjmp/_longjmp, raw globals (LLVM); auk build --lib [--bc] writes its export fns as an object or bitcode
compiler/src/effect_check.auk uses checking, the raft.toml effects ceiling, imprint purity
compiler/src/mir.auk MIR: typed expressions (machine types), structured statements (If, Loop with a latch, labels and gotos), functions, generated aggregates and globals; the body builder
compiler/src/mir/lower.auk, mir/lower/ function bodies → MIR, straight from the typed AST (RC retain/release, explicit control flow, pattern tests, FFI calls); emit_eval for the debugger
compiler/src/mir/progen.auk, mir/build.auk program-level generation: refcount helpers, element ops, boxes, derived Show/Eq/Ord/hash/to_value/from_value, literals, constants; build: the whole program (entry points), the debugger tables, imprints
compiler/src/mir/types.auk representations of Auk types: which values hold references, boxed fields, the members of each aggregate
compiler/src/mir/layout.auk sizes, alignments and member offsets of MIR types (C struct rules, matching the runtime's types) and the C ABI classification
compiler/src/backend.auk the LLVM back end's drivers: target machine, levels and pass pipelines, verification, object emission (AUK_EMIT_BC also writes its bitcode: the seed), auk ir --llvm, jit/jit_eval/jit_live
compiler/src/codegen.auk MIR → LLVM IR: variables, aggregates as bytes, the calling conventions, control flow, globals
compiler/src/intrin.auk hot runtime calls compiled inline (checked arithmetic, list access, reference counting, exact float ops)
compiler/src/orcjit.auk ORC LLJIT: links objects compiled by our target machine; each runtime symbol defined at its address in this process (ffi.symbol)
compiler/src/dwarf.auk DWARF for auk build -g through LLVM's DIBuilder
compiler/src/rtlink.auk, native.auk auk build: links the runtime's inlinable functions (its LLVM bitcode, embedded) into the program; native: the object file linked by the system linker (ld, ld.lld) with the embedded libauk_rt.a
compiler/src/mir/printer.auk, mir/reader.auk, driver.auk auk ir: MIR printed as C-like text; the reader of MIR dumps and its dev commands (compiler mir|llvm|build-mir|run-mir)
runtime/c, runtime/src the runtime: C (auk_rt.h is the ABI) and the modules already in Auk (runtime/raft.toml: the package auk_rt, the only one allowed raw; design/runtime-port.md)
build.sh, seed/ builds everything without Rust: the runtime (libauk_rt.a, auk_rt.bc in target/auk-rt/, which the compiler embeds), raft, the compiler from a seed through stage 2; seed/<os>-<arch>.bc.gz: the compiler as bitcode for a fresh clone (scripts/cut_seed.sh)
compiler/src/diag.auk, json.auk diagnostics with stable codes (E0001…), spans, notes and fix-it edits; renders human text or JSON
compiler/src/dbg_eval.auk evaluating Auk expressions at a debugger stop (eval, conditional breakpoints): checks a copy of the program with the expression written in, compiles just it; served to the engine through a mailbox (C can't call Auk)
compiler/src/dap.auk auk dap: the Debug Adapter Protocol on stdin/stdout (VS Code's debugger, the auk-vscode repo), translated to debug engine commands
compiler/src/debug.auk auk debug: starts auk run --debug in a child process on a private socket and drives the debug engine; batch reports or an interactive prompt
compiler/src/holes.auk, hatch.auk auk holes (a brief per ?? hole: expected type, locals in scope, contracts, the tests that call it) and auk hatch (fills holes with the model command, checked and tested)
compiler/src/explain.auk, error_pages.auk auk explain: the pages in docs/errors/*.md, compiled in (scripts/gen_error_pages.sh)
compiler/src/fmt.auk, fmt/ auk fmt: a Wadler-style document printer over the AST
compiler/src/dive.auk, lsp.auk, mcp.auk, live.auk, cffi.auk, stdio.auk auk dive (the REPL), auk lsp (the language server), auk mcp, auk run --live, C prototypes for FFI (cffi)
raft/ the raft package manager, an Auk project (raft/src/main.auk: manifest, semver, TOML, SHA-256)
runtime/c/*.c the runtime (C, below): one file per area (rc, heap, str, arr, map, err, value, copy, catch, stack, task, chan, core, tls, testing, trace, dbg, pixels, gui, gui_host, audio_out, plus compiler: entry points only generated code uses); auk_rt.h declares everything, AUK_API marking what generated code calls; its unit tests in runtime/c/test/

Runtime ABI (runtime/c/auk_rt.h is the source of truth)#

  • Scalars: int → int64_t (all arithmetic is checked; overflow calls auk_panic), float → double, bool → bool, char → uint32_t (code point).
  • Heap objects begin with AukHeader { _Atomic int64_t rc; uint32_t kind; }. auk_retain(p) / auk_release(p) (frees at 0, releasing children). They're atomic because async tasks run on threads.
  • str: an immutable UTF-8 buffer with a byte length, a cached char count and an is_ascii flag. Char indexing is O(1) when ASCII. Interning isn't needed.
  • [T]: a header plus len, cap and elem_size, with elements stored inline (generic code is monomorphized, so element types are known statically). Copy-on-write: mutating with rc > 1 copies first. A slice xs[a..b] / [h, ..t] is a view {base, offset, len} that shares the buffer (O(1)).
  • {K: V}: an insertion-ordered hash map (an index table plus an entries array), copy-on-write.
  • Tuples and structs: C-layout structs (aggregates; generated functions pass them by pointer and return them through a result pointer, runtime calls follow the platform C ABI); RC fields are retained and released by generated code. Enums: a tagged union. A recursive variant field is boxed automatically (a heap cell with a header).
  • T? → {bool some; T val}. T! / T!E → {bool ok; T val; AukErr err}, where AukErr is a heap object (message, code, cause, plus an optional typed payload with a type id, so err(BadUrl(w)) can match after conversion).
  • Closures: {fnptr, env*}, with the env heap-allocated and reference-counted.
  • Value: a C enum mirroring core Value. The compiler generates to_value/from_value per type.
  • Async: async fn compiles to a normal function plus a thunk. Calling it without await creates a AukTask (a thunk run on a pthread of its own, runtime/c/task.c). await t joins. await (a, b) / await [..] starts everything and then joins all of them. A direct await f(x) just calls f (no thread).
  • Core library in the runtime: fs, net (tcp; tls through the system: Network.framework on macOS, OpenSSL (loaded with dlopen) elsewhere, verified against the system trust store), time, rand, env, proc, string and collection primitives, float formatting (shortest exact output), and print.
  • Panics (overflow, index out of range, a failed requires, an unfilled hole, unwrapping none, a stack overflow) print panic: <msg>\n at file:line:col and exit 101, unless the thread has a recovery point (runtime/c/catch.c: a stack of sigsetjmp buffers). Then the panic is recorded and siglongjmps back to it, and nothing in between is cleaned up (owned objects leak).
  • Stacks (runtime/c/stack.c): main and each test run on a thread with a 256 MB stack (AUK_MAIN_STACK), tasks get 32 MB (AUK_TASK_STACK); the memory is only reserved until used. Every Auk thread records its stack bounds and gets an alternate signal stack, and a SIGSEGV/SIGBUS just below the stack (its 256 KB guard, or past it) is the panic stack overflow at the function that overflowed (by its symbol name: auk build only, <unknown> under the JIT): exit 101, or, under a recovery point, the handler returns into a trampoline on the abandoned part of the stack that jumps to it (jumping straight out of the handler would leave macOS thinking the thread is still on its alternate stack). Other faults go on to the previous handler.
  • catch e lifts e into dk_catch{k}(env) returning T!, like a closure but capturing by reference (env holds pointers to the captured variables), plus a C thunk. auk_catch(thunk, env, out) runs the thunk under a recovery point and returns the panic as an Err (panic: <msg> at <loc>), or NULL.
  • Cancellation: t.cancel() (auk_task_cancel) sets the task's flag and bumps auk_cancel_pending. Every user loop starts each iteration with if auk_cancel_pending != 0 { auk_cancel_check(loc) } (one load while nothing is cancelled). Blocking calls (sleep, await, accept, recv, send, a subprocess's output) wait in 50 ms slices on task threads and check the flag too. A cancelled task panics "cancelled", and auk_catch passes that panic on instead of returning it, so no catch keeps a cancelled task alive. await any ts (auk_task_await_any) waits on a condition variable signalled whenever a task finishes.
  • Task links: every task thread runs under a recovery point. A panic ends only the task; await raises it again in the awaiter (where a catch can take it). A task that panicked and is freed without being awaited ends the process with the panic report (unless it was cancelled). Each task keeps the tasks it started; when it ends with a panic, it cancels those still running (Erlang's link).

Contracts, holes, tests, peek, imprint (codegen)#

  • requires → a check at function entry, in every build. ensures → a check at every return in debug and test builds; old(e) values are captured at entry.
  • ?? → auk_hole("file:line", "expected T"). auk build (release) refuses a program that still has holes; run, check and test allow them.
  • auk test [path] builds a test binary with every test block (including the dependencies' tests only when --deps is given), runs each in a forked child process (so a panic fails one test and the rest keep running), and prints ok/FAIL lines plus a summary. Exit code 1 on any failure. A directory means its project's src/lib.auk or src/main.auk; a directory with neither (like examples/) is treated as flat: each top-level .auk file with test blocks is built as its own program (with the directory's raft.toml dependencies), run in sorted order under a == <file> header, followed by a combined == N files: P passed, F failed line.
  • peek e → prints [file:line] <source text of e> = <show(value)> to stderr and returns the value.
  • imprint X = e → the compiler JIT-compiles a helper program that evaluates e (pure functions only) and writes to_value(e). It then embeds the result as a constant. No interpreter is needed.

Back end as built (compiler/src/mir/, backend.auk)#

  • mir/progen.auk: refcount helpers, element ops, derived Show/Eq/Ord/hash/to_value/from_value, literals, constants. mir/lower.auk (+ lower/pat.auk): bodies, straight from the AST + side tables; every value is an owned temporary or a borrowed operand, scopes release on every exit. A list or map element that is only looked at (a field of it, a comparison, a match/is scrutinee, an operand, a call argument) is a borrowed snapshot when nothing evaluated before its use can change a variable (pure_read: no assignment, & argument or catch); a field of an element retains only the field. Variants without fields need no RC (x == Tok.LParen compares tags); xs.push(v) of a fresh v moves it (auk_arr_push_own). A match on str with 6+ literal arms is one auk_str_switch lookup in a constant table sorted by (length, bytes). s += x and s = s + x are auk_str_append: in place when the string is unique, growing geometrically. str(cs[a..b]) is auk_str_from_char_range (no slice). var g = xs[i], changes to g, xs[i] = g in one block moves the element out (move_out: a zero in the slot until the store) when nothing in between mentions the container, changes the index's variables or leaves early, and the container is a plain local of the frame. mir/build.auk and drive.auk: whole-program generation (the entry point: the test runner or the main shim), imprints, and run/build/ir/test.
  • codegen.auk: every MIR variable is an alloca in the entry block (SROA/mem2reg make scalars registers again). Aggregates are untyped memory ([size x i8]): members are byte-offset GEPs from mir/layout.auk, copies are constant-size llvm.memcpy, so LLVM never sees a struct type for an Auk value. Generated functions take aggregates by pointer and return them through a leading noalias result pointer; an aggregate parameter the function never writes or takes the address of is read in place (noalias readonly, the caller passes memory nothing else reaches during the call: an unaddressed local or a fresh copy), any other is copied on entry. Runtime calls (and generated functions the runtime calls back: element ops, drops, thunks, tests, the main shim) use the platform C ABI (mir/layout.auk, c_pass): an aggregate of at most 16 bytes travels as one or two register words (on x86-64 an all-float eightbyte in a float register; the last word loads only the aggregate's own bytes), an AArch64 aggregate of 1-4 doubles in float registers, a bigger result through the sret pointer; small integers carry signext/zeroext. Functions are internal (only the entry point is exported), nounwind (panics siglongjmp, never unwind) and keep frame pointers.
  • Inline runtime calls (intrin.auk, every level but O0): checked + - * (llvm.s*.with.overflow), / % and shifts, auk_arr_len/at, auk_arr_set/push for elements without references (a unique list, rc == 1 and not a slice view, gets a plain store), auk_retain/auk_release (atomic add/sub; handles go to the runtime, the last release frees through auk__free_last), and the exact float ops (sqrt abs floor ceil round). A failing or slow path calls the original runtime function with the same arguments, so panics and edge cases are exactly the runtime's.
  • The runtime's bitcode (rtlink.auk, auk build at O2): build.sh also emits the runtime as LLVM bitcode, linked into the program before it is optimized. Exported functions that touch no private mutable or thread-local runtime state (directly, through internal callees or constant tables) become available_externally (inlinable, never emitted), the rest declarations, as do exported globals and aliases: calls LLVM doesn't inline go to libauk_rt.a, so runtime state stays in one place. If this LLVM can't read the bitcode the build goes on without it. AUK_NO_RT_INLINE=1 turns it off.
  • The JIT (orcjit.auk): the program is compiled to an object file by our own target machine (the same code generation as auk build), and ORC's LLJIT only links it, against the runtime's symbols as absolute addresses plus the process's libraries (libc). Nothing compiles lazily and no LLVM thread runs, so run/test can fork. A debugger evaluation module or a live-reload module gets an LLJIT of its own that is never freed.
  • auk run and auk test run the program in the compiler's process; each test runs in a forked child (the runtime's test runner), imprints are computed in a forked child, auk dive runs each input and auk hatch each candidate's tests in a forked child (hatch's in a process group killed after its timeout). A flat test directory runs each file as auk __jit test <file>.
  • Patterns compile to sequential first-match tests (no decision trees). Self tail calls in tail position become loops (goto dk_tail in auk ir); other recursion stays recursive.
  • A struct or enum field whose type contains the type itself by value is boxed.
  • Top-level lets are lazily initialized globals (initialized at startup, released by the leak-check cleanup).
  • imprints are computed at compile time (drive.auk, eval_imprints): the compiler generates a helper program from the same typed program in which each imprint is computed through its constant getter (so imprints may use other imprints and lets in any order) and its derived to_value tree is written to a records file; the compiler parses it and emits each imprint as a global built from that data at startup out of immortal objects (literal strings, lists, maps, boxes), with no initializer code. A panic while computing one is E0420 at the imprint (with the panic message and location); a cycle through function calls is E0115; a type that isn't Data is E0350 (in the checker). Nothing is cached.
  • A write through a boxed field (a.boss.pay += 1, a.b.c.v = 1) first makes each box on the path unique (copy-on-write, like lists and maps).
  • ensures (debug builds): the function becomes a wrapper around the clause group's body. The wrapper finds the clause that matches (pattern tests and pure guards, in order), binds its names, captures the header's and that clause's old(e) values, calls the body and checks those ensures. Self tail calls stay loops inside the body; the wrapper checks the outer call's result.
  • Execution traces (auk run --trace=json|<file>, auk test --trace=json|<dir>, --trace-limit N, --trace-core; runtime/src/trace.auk). A codegen flag (progen::Trace): without it the generated program is unchanged. With it, every Auk function outside core (with --trace-core, core's too) is emitted as <name>_untraced plus a wrapper <name> that records a call event with the arguments shown (debug Show), calls it, and records ret with the shown value, or err for a T! that failed; every direct call to a traced function first sets the runtime's thread-local trace site (its call site, through auk__trace_site_slot), and main calls auk_trace_init. Self tail calls stay loops inside _untraced, so a tail-recursive loop is one call/ret pair. The runtime adds peek events (auk_peek), panic (auk_panic) and signal (a crash handler on an alternate stack, so a stack overflow still writes its trace), and the test runner a test event per test. Events, one JSON object per line: {"ev":"call","fn":"calc.eval","loc":"calc.auk:66:30","args":["Lit(2.0)"],"depth":3} (loc: the call site, or the definition when called through a function value), {"ev":"ret","fn":"calc.eval","value":"2.0","depth":3}, {"ev":"err","fn":"calc.eval","error":"DivZero","depth":3}, {"ev":"peek","loc":"f.auk:5","expr":"x * 2","value":"84","depth":1}, {"ev":"panic","msg":"...","loc":"f.auk:3:36","depth":4}, {"ev":"signal","signal":11}, {"ev":"test","name":"t","result":"ok"|"FAIL"}, {"ev":"truncated","limit":10000} (the stream stops after the limit; a final panic/signal is still written) or, at the start of a tail, {"ev":"truncated","skipped":N}. Values are cut to 200 chars, and showing one is bounded work (a byte budget in the runtime's string builder stops list/map/str shows early), so a huge argument costs O(1) per call. Nothing is traced while a value is being shown (a user show). Configuration reaches the program through the environment (AUK_TRACE = file, directory or -; AUK_TRACE_LIMIT; AUK_TRACE_TAIL, default 20): auk test --trace=json keeps each test's last events in a ring and the runner prints them under its FAIL line; auk hatch does the same (12 events) so a retry prompt shows what the candidate actually did.
  • Debugger (auk run --debug; runtime/src/dbg.auk). A codegen flag (progen::Debug, set by Trace::debug): without it the generated program is unchanged. With it, every Auk function outside core keeps a frame record, a generated aggregate {fn, parent, site, slot...} on its stack: the body starts with auk_dbg_enter(&frame) (linking it into the thread's debugger stack) and every return is preceded by auk_dbg_leave (dbg_rets; a returned value is computed first). Binding a named local (bind_name) stores its address in its slot (a non-place value is first copied to a variable, so every named local is addressable and lives in memory in debug builds). Every statement (block statements, a block's last expression, an expression body) is a site: it stores the site id in the frame and calls auk_dbg_hook(&frame) while auk_dbg_armed is set (one load otherwise). Each site records the frame slots in scope there: a name goes out of view when its scope is popped, and a let's names last until the end of the enclosing block. catch restores the debugger stack with the trace state. main calls auk_dbg_init first with the debug info as text (dbg_tables in mir/build.auk: files, types, functions with their slots' names and types, sites) and a constant table of show functions, one per local type (AukStr *(void *), debug Show; showing can instantiate more code, so it runs to a fixpoint). Without AUK_DEBUG the program runs as usual. With it, the program connects to that Unix socket, sends {"ev":"ready"} and waits for run. Commands are text lines with an optional @<seq> prefix echoed in the reply: break <file>:<line>|<fn>, delete [id], catch all|uncaught|none (stop on panics; default uncaught), pause, kill, and while stopped continue|step|next|finish, stack [thread], vars [frame] [thread], threads. Events are JSON lines: ready, ok, error, break (id, resolved file/line: the next line with code; one site per function instance), stopped (reason: breakpoint|step|pause|panic, thread, frame, plus bps or the panic's msg/loc/caught), stack, vars (name, type, value, shown with a 1000-char budget; a shadowed name shows its latest binding), threads, exited. All-stop: when one thread stops, the others stop at their next statement. A thread blocked in a call stops when the call returns. Steps lapse at the next stop of any thread. Cost: about 4x on call-heavy code (enter/leave are runtime calls, locals live in memory).
  • auk debug (compiler/src/debug.auk). Batch (--json, --batch, or stdin not a terminal): sets the --break points, runs, and at each stop records {reason, fn, file, line, msg?, loc?, locals: {name: value} (--print filters), stack} then continues; --max-stops N (default 20) and --timeout S (pause, record the stop as timeout, kill) end it early. Text output is one #n <reason> at file:line in fn block per stop, then the program's captured stdout/stderr and exit N; the exit code is the program's (137 when ended). auk mcp's auk_debug tool runs it with --json and its timeout. Interactive (a terminal, or --repl): a gdb-style prompt (help); input typed while the program runs waits for the next stop, an empty line repeats the last command, Ctrl-C sends pause.
  • Parts of values (the variables pane, kids). dbg_tables also emits a children function per local type (Gen::dbg_kids_fn, void f(void *value, void *ctx), NULL for a type without parts) that reports each part with its name, type index and address: auk_dbg_kid for struct fields, tuple elements, the fields of the current variant, an option's some and a result's ok; auk_dbg_kid_arr / auk_dbg_kid_map for a list's elements and a map's values (named by the key shown). The engine command kids <frame> <thread> <local> [i j ...] walks down the part indexes through nested calls (each part's address is a copy on the children function's stack, valid during the call) and lists the parts there (at most 1000, then "more":N); vars and kids items get "kids":true when the value has parts (counted on the value itself, so Empty and [] have none).
  • auk dap (compiler/src/dap.auk). Reads initialize and launch (program, args, cwd, stopOnEntry: a temporary break main) itself, then starts the program through debug::launch with its output streamed as output events (the engine makes stdout line-buffered). Breakpoints per source are replaced as a set (delete then break); panics are exception breakpoints (uncaught, all). Frame ids and variable references are handles into a table cleared on every resume; evaluate resolves name, name.part, name[i] by part names. Threads come from the engine while stopped (task N).
  • Expressions at a stop (eval, break ... if; compiler/src/dbg_eval.auk). auk run --debug installs an eval hook in the runtime (dbg::set_eval_hook) with the debug info and the source of every non-core module as built. To evaluate e at a site, a copy of the program is checked with let __dbg_r = (e) written right before the site's statement (for an expression body, = e / x => e, a block around it: DbgSite::wrap, with the end of its span); if that fails, let __dbg_r = {\n e \n} (a statement such as an assignment). Every module is pinned to its built source (resolve::with_sources), so the names, types and scopes are exactly the stopped statement's. lower::emit_eval compiles only the let's value, in the stopped function's instance (same body and type arguments, so generics and calls resolve as in the program): each local it uses from outside the expression is bound to the frame's slot address for that name (the latest binding in scope), read through void **slots, and the value is returned shown. The function and what it calls go into a fresh JIT module (kept for good; its dk_init runs once), cached by (site, text): about 10 ms to compile, then a call. The engine runs it under a recovery point (a panic is an error), with the debugger busy (no stops inside). Breakpoint conditions are evaluated the same way by the thread that reaches the site, without the engine's lock; a condition that fails to compile or panics stops with cond_error. The expression's functions are fresh copies: they don't stop at breakpoints and have their own constants.
  • Native debug info (auk build -g; compiler/src/dwarf.auk). Lowering emits Stmt::Loc(i) (an index into Program::locs: file, line, column) at every statement site and records each Auk body's name in Program::names. The code generator builds DWARF 4 with LLVM's DIBuilder: one compile unit (named after the program's first source file, language C99 since DWARF has no code for Auk), a subprogram per function and a DILocation from every Loc. An Auk body's subprogram has its Auk name (prog.fact) at its first statement; every generated function (shims, element ops) gets one too, artificial, at line 0, so Auk code LLVM inlines into it keeps its lines. The build stays optimized (O2 with inlining): lldb shows inlined Auk functions as [inlined] prog.add at prog.auk:5 frames, stepping can move between lines out of order, and code the optimizer folds away has no line. Mach-O linkers leave DWARF in the object file, so auk build -g runs LLVM's dsymutil (from the bin directory next to the libLLVM it runs on, else the PATH's) to make <out>.dSYM before the object is deleted. Variables are not described (the instrumented debugger shows Auk values).
  • Task heaps (runtime/c/heap.c, copy.c, chan.c). Each task allocates into a heap of its own. A tracked object has a 32-byte prefix before its header (a doubly linked list, plus the heap), and AukHeader::flags marks it. The main thread, constant initializers and handles (tasks, connections, listeners, channels) are untracked. Spawn arguments are deep-copied into the new task's heap: generated code switches heaps around per-type copy functions, Gen::copy_ex (AukElemOps::copy, AukObj::copy for closure envs and boxes). await deep-copies the result into the awaiter's heap. A channel stores an untracked copy of each message and copies it again into the receiver's heap. Handles are shared, and each heap keeps a ledger of the references its task holds on them. When a task crashes, its arguments are dropped (dk_thunk{i}_drop) and its heap is swept: every object gets an unreachable reference count and is freed by its kind's free routine, then the handle references still on the ledger are released. When the task is freed, its result is released and the rest is swept. So a crash, a cancellation, or a catch inside a task leaves nothing behind. Only a catch on the main thread leaks what the aborted code owned. A spawned async fn call can't take a var argument (E0336); await f(&x) runs in place and can.
  • Live reload (auk run --live, compiler/src/live.auk). The program is compiled with every call to a user function instance going through a process-wide table: load the slot, call through the pointer. Slots are keyed by Program::live_keys (module file, name and signature, so a key survives edits that renumber instances). A parent process compiles and forks a child to run the program. In the child, a watcher thread polls the source files' mtimes every 200 ms. On a change it re-checks and compiles the whole program into a new JIT module, runs that module's dk_init (literals, imprints, constants), and stores the new code addresses into the table. Old modules are never freed. A type error keeps the old code running. If a struct or enum layout changed (file::Name → its fields as text), what happens depends on who can hold the type. live::uses walks the MIR from the entry point (without entering task thunks) and from each thunk. It collects the user types in every reached function's variables, expanding struct fields and enum variants. Indirect calls reach every closure of the same signature; a closure's body isn't reached just by creating it. If main's side can hold a changed type, the child exits 75 and the parent recompiles and starts a fresh child. Otherwise, the new code is installed and every running task whose thunk (by code address, across all modules so far) can hold one is cancelled (task::cancel_running). A task started with exactly one closure, and no other argument that could hold one, records that closure's code address (AukTask::hint; sup children are started this way). Its types are then what its thunk reaches without following indirect calls, plus what that closure reaches (computed on demand from the build's call graph). A closure whose environment holds closures falls back to the wide set. Its sup supervisor restarts it from scratch on the new code, and nothing else stops. Frames that are already running finish their old bodies, and closures, vtables and element ops keep pointing at the code they were made with. Only the explicit table calls switch.
  • AUK_LEAK_CHECK=1 makes auk run / auk test programs fail when runtime objects are still live at exit.

CLI#

auk run|build|check|test|fmt|explain|dive|lsp|ir|spec, plus raft. auk ir [--llvm] [--trace] <file> prints the program's MIR as C-like text (--llvm: the unoptimized LLVM IR). --llvm=o0|light|o2|o3|build[-raw] prints the LLVM IR at a level (build: O2 with the runtime's bitcode linked; -raw: before the pass pipeline), and --release/-g generate the MIR auk build [-g] compiles. auk run -O optimizes fully (see the levels above). check --json emits {file, ok, diagnostics:[{code, severity, line, col, end_line, end_col, message, notes, fixes:[{line, col, end_line, end_col, text}]}]}.

Testing#

Build: ./build.sh [--check] [--test] [--install [<dir>]] [--seed <auk> | --from-seed] builds the runtime (target/auk-rt/libauk_rt.a and auk_rt.bc: runtime/src with the seed's auk build --lib and --lib --bc), raft (by the seed; it installs compiler/.raft) and the compiler from a seed auk (seed → stage 1 → stage 2 = target/bin/auk, then target/bin/raft by stage 2). The seed is --seed, $AUK_SEED, the target/bin/auk of an earlier build, auk on the PATH, or else (or with --from-seed) the checked-in seed/<os>-<arch>.bc.gz: the compiler as LLVM bitcode, compiled with LLVM's llc and linked by scripts/link.sh (a fresh clone needs only LLVM 23 and the system linker, no C compiler). Whatever the seed builds links through scripts/link.sh too (AUK_LINKER, unless set), so a seed from before auk build called the linker itself never runs cc; on Linux it adds compiler-rt's builtins (libgcc's if there is no compiler-rt), which the C code in seeds cut before the runtime was all Auk calls on AArch64 (outline atomics). The runtime's Auk half and the seed: it is built before stage 1 (which embeds the finished runtime), so by the seed; a seed older than the raw dialect (any seed cut before Phase 7a, or the Rust auk) can't, and then builds compiler/ first as stage 0 (embedding an empty runtime, which auk build --lib never links), which builds it (+20 s). --check adds stage 3 and requires it bit-identical to stage 2 (the bootstrap fixed point), and has stage 2 rebuild the runtime's Auk half: if that differs from the seed's (an older compiler compiled it), the whole build runs again with stage 2 as the seed; --test runs the runtime's unit tests (runtime/test/unit/run.sh, built by target/bin/auk or $AUK) and the integration tests against target/bin/; --install copies auk and raft to ~/.auk/bin (or <dir>) with a raft.registry naming this checkout's packages/. About 45 s on an M5 (--test: +30 s). A new seed (scripts/cut_seed.sh, from a clean commit): build.sh --check, then stage 2 builds the compiler once more with AUK_EMIT_BC (the module its object file is made from), linked with the runtime's bitcode, gzipped to seed/<os>-<arch>.bc.gz with a .txt naming the commit; it must then rebuild the same stage 2 from the seed alone. Cut one at every release and whenever the current seed can't build the compiler any more.

Integration tests: tests/runner/, an Auk project of its own (scripts/test_runner.sh [--no-build] [args], or build.sh --test): one module per suite in tests/runner/src/suite/, each test a task run under catch (-j N, filters by substring of suite::name, --list, --ignored for the GUI tests). The tests spawn $AUK_BIN (default target/bin/auk) and $RAFT_BIN (target/bin/raft) with pipes and compare their output; fixtures are in tests/fixtures/, tests/checker/, tests/errors/. The suites:

  • parse, check, check2: the whole corpus parses and checks clean (and its typed program satisfies codegen's invariants, auk typed); small programs get the exact diagnostic codes, messages and fix-its.
  • run: examples run with fixtures from tests/fixtures/, every example and package passes auk test, leak-checked runs (every package's tests), auk ir, codegen edge cases, auk spec, live reload, and every tests/errors/*.auk fails auk check with the // error: E0xxx <message> in its first line.
  • debug: the debugger over its socket, auk debug (batch text and JSON reports, the interactive prompt), auk dap sessions, expressions at a stop, and auk build -g (lldb line and function breakpoints and backtraces; skipped without lldb and dwarfdump/llvm-dwarfdump).
  • trace, units, fmt, lsp, mcp, explain, dive, hatch, holes, ffi, net, gaps, shadow, collections, bitwise, core_io, llvm_package, packages_p0, packages_next: traces, units of measure, the formatter (the corpus is fmt-clean and formats to the same AST), the language and MCP servers, auk explain (every page's examples), the REPL, auk hatch with a scripted model, hole briefs, user FFI, TLS and sockets, and the packages.
  • raft: the package manager against the registry in tests/fixtures/registry/.
  • raw: the runtime's dialect, with the programs in runtime/test/raw/ (in the JIT and built; a --lib library linked with C, lib_main.c) and its errors (E0418, E0419, E0421).
  • sh runtime/test/unit/run.sh [filter]: the runtime's own unit tests (runtime/test/unit/, Auk in the runtime's raw dialect; the port of the C tests, 72, one for one). They are modules of the runtime package itself: run.sh puts runtime/src/*.auk and the tests in one package in a scratch directory, auk build --lib makes one object of runtime and tests, whose export fn main (ut_main.auk) is the runner, and cc links it. rt.auk declares the runtime's API by its C names (extern fns that resolve to the runtime's own definitions), ut.auk is the harness: check/eq/text_eq/has/str_eq, child/expect_panic (fork, both streams into a pipe), scratch memory and directories. Every test runs in a forked child, so a crash or panic ends only that test; the runner prints its output, then ok <name> / FAIL <name>, then N passed, M failed. After each test auk_live_objects() must be back where it was. ASAN=1 instruments everything (the bitcode of runtime and tests through LLVM's asan pass with opt, each function given sanitize_address), with LeakSanitizer checking each test's child at its end (on macOS too); TSAN=1 likewise with ThreadSanitizer; test_conc.auk is the concurrency stress set it is for (TSAN=1 sh runtime/test/unit/run.sh conc: tasks spawned, awaited, cancelled and linked; heaps swept on other threads; channels with several senders and receivers; handle refcounts on many threads; the trace's and the debugger's locks). A sanitizer report in a test's output (or a child's) fails the test. AUK_NET_TESTS=1 makes the tls tests reach real hosts.

Everything at once: scripts/check.sh [step...]: build (./build.sh --check), runtime (the runtime's unit tests), runner (the integration tests), golden (self-consistency: target/bin/auk against itself, 0 differences) and bench (every tests/perf program built and JIT-run on a small input, same output both ways); opt-in seed (./build.sh --from-seed, a fresh clone's build) and vscode (the VS Code extension end to end in a real VS Code, scripts/e2e.sh in an auk-vscode checkout, $AUK_VSCODE, default ../auk-vscode). Lists the steps that failed. Real-host TLS tests are off unless AUK_NET_TESTS=1.

The golden corpus: scripts/golden.sh records what an auk (--ref, default target/bin/auk) makes of every .auk in tests/, examples/, packages/ (not .raft/ copies) and compiler/ (with the prelude, compiler/prelude/; 239 files), in target/golden/ref/<mode>/<path>: tokens and ast (auk dump), check (auk check <f> --json), ir (auk ir, for the programs that check clean), fmt (auk fmt -, the file untouched), and run: stdout, stderr and exit code of the deterministic programs and auk test runs listed in scripts/golden_runs.txt (<name> run|test <path> [args] [< stdin]). Each record is stdout, then === stderr === and stderr if any, then === exit N === if not 0; the repo root, temp dirs and the binary's path are normalized. --compare <auk> records another build with the same CLI into target/golden/compare (re-recording the reference unless --reuse) and reports, per mode, how many files differ and the first lines of each diff; it exits 1 on any difference. --modes tokens,ast and --only <regex> narrow it (a port in progress compares only what it implements), -j N sets the parallelism. A run takes about 25 s; comparing the Rust auk with itself must give 0 differences, which is how nondeterministic output is caught (a "did you mean" chosen in HashMap order was).

The perf baseline: scripts/baseline.sh writes perf-baseline.md: the tests/perf table and compile times (auk check/dump ast/ir over the corpus, auk build and auk run startup), with machine, date and commit. Run on a quiet machine.

Sanitizers. The runtime's unit tests run under ASan with LeakSanitizer (ASAN=1) and TSan (TSAN=1), the runtime instrumented with them, above (no UBSan: its checks come from clang's front end, and the runtime is no longer C). The whole suite with an instrumented runtime linked into auk (the Rust build's AUK_RT_SANITIZE=address) is not in build.sh yet: building stage 2 with an instrumented target/auk-rt/libauk_rt.a and AUK_LINKER=scripts/asan-cc.sh is the way.

Linux. ci/linux/test.sh builds a Debian trixie image (ci/linux/Dockerfile: LLVM 23 from apt.llvm.org, OpenSSL, SQLite, lldb) and runs ./build.sh --check --test in it on a copy of the repository (cloned from /src, so the host's target/ is untouched), starting from the linux-<arch> seed; ci/linux/test.sh shell opens a shell there instead. AUK_LINUX_PLATFORM=linux/amd64 runs x86-64 under emulation (no seed for it yet: --seed an x86-64 auk). On Linux build.sh compiles the runtime with -D_GNU_SOURCE, and programs link -ldl -lpthread. What stays macOS-only: GUI windows (gui.open fails elsewhere; the window tests run only on macOS).

Stuck on a word? Try the glossary. Got a compiler error? Look up its code. Improve this page