Skip to content

← All updates

session

Update 17: AI & LLM serving

The AI story deepened from tensors-in-the-stdlib to language-level ML semantics.

  • layer / model keywords: desugar to plain structs and impls, so a network definition reads like the math: x |> self.l1.forward |> relu |> self.l2.forward.
  • Pipeline operator |>: x |> f |> g is g(f(x)), resolved at parse time.
  • Compile-time shape safety: Tensor<f32, [784, 128]> with symbolic batch dimensions; matmul, conv2d, concat, and friends are shape-checked before the program runs.
  • Tape-based autodiff: grad(w) { ... backward(loss); } records and replays a gradient tape (interpreter runtime; IR support across backends).
  • GPU kernels: @kernel functions with thread_id() / workgroup builtins compile to SPIR-V, with a CPU fallback so kernels are testable in the interpreter.
  • Quantization: the ~> q8 cast operator, symmetric INT8 parameters, qmatmul, and dequantization.
  • LLM serving primitives: KV-cache regions with mandatory memory budgets and sliding-window eviction, gradient checkpointing, differential-privacy budget tracking, simulated tensor parallelism, and LoRA adapter hot-swap that survives hot reload via persist.

Honesty note: checkpointing, DP, and tensor parallelism are partial (interpreter-level or single-rank simulation today); see Feature Status.