Decision log
Newest last. Each entry gives the date, who decided, the decision and the evidence behind it. "Mark" is the project lead; "measured" means the decision follows directly from a measurement on the Baseline or Pilot-1 page.
| ID | Date | By | Decision | Why |
|---|---|---|---|---|
| R1 | 2026-11-05 | analysis | Shrink the teaching prompt, don't remove it | 84 release tags in 90 days. A prompt-free fine-tune freezes one snapshot of the language and writes the old syntax with confidence. Superseded in spirit by D-FT-4: habits plus runtime discovery are the correction mechanism |
| R3 | 2026-11-05 | measured | Drop the "nothing under 13B" rule and measure instead | The rule had no source. The size ladder showed the gap is language knowledge (71–89% of small-model failures are compile errors), which fine-tuning addresses |
| D-FT-0 | 2026-11-05 | Mark + measured | Target Gemma 4 12B dense; 31B as the quality arm; qwen3.8-27b as a teacher only | 12B: 54% one-shot (excluding harness timeouts), and 12 of its 13 agent failures were tool-protocol misses, not wrong code. Gemma 4 is Mark's preferred family; Apache 2.0 |
| D-FT-1 | 2026-11-05 | Mark | Frontier outputs may train the model (Claude and OpenAI code and trajectories) | Our repos hold ~225k lines of frontier-authored AILANG and thousands of verified agent trajectories. Generating data with a local 27B would be slower and worse |
| D-FT-K | 2026-11-05 | analysis | Kill criterion: fine-tune + slim prompt must at least match base + full prompt on the held-out 24 | Keeps the experiment honest; a negative result is recorded, not buried |
| D-FT-H | 2026-11-05 | analysis | Hold out 24 benchmarks before extracting any data | Frontier eval results are solutions to the scored benchmarks. Contamination is checked by exact and normalised-code hashes, plus benchmark-id matching |
| D-FT-3 | 2026-11-06 | Mark | Agent mode is the target; standard one-shot is a prompt problem | The model's job is fleet agent work. One-shot results inform, but do not decide |
| D-FT-4 | 2026-11-06 | Mark | Train habits, memorise only the core | "The whole point of the language has been to help AI self discover." Runtime discovery (check → read error → fix, docs, examples) also counters staleness |
| D-FT-5 | 2026-11-06 | Mark | Expand examples/ into a representative training set | Measured: 25 of ~397 passing examples reached pilot-1's data pool, and 1 reached the mix. 96 of 496 exported stdlib functions appear in no example. The examples are needed for the docs anyway |
| D-FT-2 | 2026-11-06 | Mark | Publish fine-tunes outside the rig, through an artifact bucket linked from this site | Weekend fine-tunes per language release, versioned by AILANG version. Base-model licence (Apache 2.0) to be confirmed against the model card at publish time |
| D-FT-6 | 2026-11-06 | Mark | Agent evals run without the attended-session gate; re-run pilot-1's three agent arms | A wire capture against a scripted model showed the pi session-protocol gate refused every eval write until the model called session_protocol_ack. Every Phase 0 base-12B pass acked first; the fine-tune never did, so its 0/24 was the harness, not the model. Fixed in 5dc3ba42b; agent results before it sit on the other side of a baseline boundary |
| D-FT-7 | 2026-11-06 | Mark | Generate training data with the best models available (frontier: Opus via the Claude CLI subscription, Codex), not the cheapest | "To generate data we can use good models, so we can present best data for fine tuning." Verification keeps generated data correct whichever model writes it, but a weaker generator skews the kept set towards easy cases: gemma-4-31b fixed 49% of stale programs and failed mostly on the harder ones. Cheap models stay for bulk mechanical work only |
| D-FT-8 | 2026-11-06 | Mark + measured | Keep thinking ON: train every step as thought → action, in the format Ollama serves; thoughts are built from facts in each episode, and the agent habits are simulated with real tool output | A live transcript showed pilot-1 emitting correct tool calls inside the thinking channel. It was trained in thinking-off format (an empty thought pre-filled), Ollama serves thinking-on, and the parser intermittently filed the call as reasoning. Switching thinking off at serve time was rejected: it costs reasoning on hard tasks, and the parse failure was random either way. Thoughts state the real error, diff, imports and task; richer reasoning comes from the base model's own verified thinking (self-distillation) |
| D-FT-9 | 2026-11-06 | Mark | The fine-tune lane runs on a lean pi profile (--tool-policy lean): read/bash/edit/write, a short fixed system prompt, no skills/extensions/context discovery. Base 12B is measured on the same profile | pi's default prefix was 7,338 tokens before every turn: 3,625 a skills catalog discovered from the machine, ~2.9k for 16 tool schemas (mcp alone 921). The lean prefix is ~910. A fine-tune must train on the prefix it is served with, so the lean prefix makes training ~2.5–3× cheaper per sample and serving and batching much faster |
| D-FT-10 | 2026-11-09 | Mark ("get ready for another training run") + measured | Pilot-3 trains on the model's own mistakes (on-policy repair data) and a fifth habit: after a failed fix, read the matching section of the reference | Pilot-2's held-out failures were two-thirds parse errors from borrowed syntax, and its runs looped 40–80 times without a lookup. Its repair data had 26 parse-error episodes out of 1,671, and none that looked anything up after a failed fix. The model's broken steps stay in each episode as context, with no loss (pilot-3) |
| D-FT-11 | 2026-11-09 | analysis | Pilot-3 warm-starts from the pilot-2 adapter (1 epoch, half the learning rate, replay sample of pilot-2's data) | Pilot-2's protocol and loop habits measured fine (0 protocol misses); what it lacks is in the new data. This saves a ~21-hour retrain from base. If pilot-3 fails, a from-base run on the combined mix is the control |
| D-FT-12 | 2026-11-09 | measured | Held-out arms run without a wall-clock cap, and base is re-run from scratch | Pilot-2's 5-hour caps cut base and fine-tune-without-prompt to 13 and 19 of 24. Pilot-2's base rows also saw a tool-output notice that pilot-3's runs won't (#2151, fixed), so re-using them would confound the comparison |
| D-FT-13 | 2026-11-10 | Mark + measured | Pilot-4 pools the data and restarts from the pilot-2 adapter: round-1 + round-2 on-policy repairs + a replay sample, not another warm start on pilot-3 | Pilot-3 reached 10/24 (pilot-2: 4/24; 6 newly pass, none newly fail), so the on-policy method stays. Stacking a second warm start would also carry pilot-3's dropped-tool-call habit forward; pooling the rounds (DAgger-style aggregation) keeps everything learned without it (pilot-4) |
| D-FT-14 | 2026-11-10 | Mark | Round-2 collection: pilot-3 on the 57 training benchmarks, one trial, traced | One trial is ~7–8 h of GPU; with round 1 pooled in that is two rounds of the model's own mistakes. It also measures pilot-3 on its own training distribution (pilot-2: 30/57) |
| D-FT-15 | 2026-11-10 | measured | Keep trained writes short, and teach building a file up in steps | Pilot-3's "read then stop" runs were write calls that ran away (one repeated a function 225 times) or ended malformed; ollama dropped them and the agent harness read that as the end of the run. 1.8% of pilot-3's responses vs 0.05% of pilot-2's |
| D-FT-16 | 2026-11-10 | Mark + measured | Every eval is traced, dropped tool calls are counted, and a 30-minute behaviour check runs before the held-out evals. Held-out scope: fine-tune + slim prompt, and base re-run once | The banked eval row could not tell a dropped tool call from a real stop, and the transcripts were not kept; diagnosing it took a re-run. Base re-run once serves later pilots too |
| D-FT-17 | 2026-11-11 | Mark | Held-out swap: run_length_encode becomes run_length_encode_v2 (the same task with a corrected expected output; PR #2229) | The old task's expected output depended on a lossy show that the language no longer prints. The old task stays excluded from training data. Arms run before the swap are compared on the 23 tasks both sets share; the swap landed while pilot-4's two diagnostic arms were running, so those drop the task too |
Open
- Weekly cadence and promotion gate. Proposal: a new fine-tune must match or beat the previous one, and base + full prompt, on the held-out agent set before it is published.
- Staleness measurement. Run the previous fine-tune against the new language each release; slow decay is the evidence that the habits work.