Fine-tuning AILANG models

This section is the open research log for a question we started on 2026-11-05: can a small open-weight model, fine-tuned on AILANG, do useful agent work locally at $0, without the 26k-token teaching prompt? We record measurements, decisions and dead ends as they happen, including the ones that go against us.

Why

• The rig is saturated. A 27B model at a 131K context occupies ~49 GB, and Ollama serves one model at a time. Smaller models free memory for parallel lanes.
• The teaching prompt is the bulk of every agent context. At ~33k tokens, parallel decoding on the rig gains only ~1.26× from batching. At short contexts it gains ~2.8×. A model that does not need the prompt makes parallel agents pay off. See Baseline.
• AILANG was designed for AI self-discovery. Its compiler errors name the fix, and ailang docs, ailang examples and ailang iface expose the language at runtime. A fine-tune that learns to use those tools should stay useful as the language moves.

The approach in one paragraph

Fine-tune for habits, not memorisation. The target model should:

• write its solution through the tool and never answer in chat;
• check every write and apply the fix the compiler names;
• look up a stdlib name before guessing it;
• trust the compiler over its own memory;
• stop once the program is verified.

Only the core syntax is memorised, from canonical examples. Everything else is discovered at run time. That should make a fine-tune decay slowly as AILANG changes, and we plan to measure that decay. See Training data.

Status

Stage	State
Baseline (size ladder, speed, parallelism)	done 2026-11-05; target picked: Gemma 4 12B
Training data (corpus, rules, mix)	pilot-1 mix built; examples expansion under way
Pilot-1 (12B LoRA)	negative, causes found (session gate, thinking format, context)
Pilot-2 (served format, habit episodes)	agent protocol fixed (0 protocol misses), 3× fewer tokens, but 4/24 against base's 6/13
Pilot-3 (its own mistakes, repaired)	10/24 with the slim prompt (pilot-2: 4/24); 6 newly pass, none newly fail. Left behind: dropped runaway writes, long loops
Pilot-4 (second round, short writes, loop-breaking)	in progress: round-2 collection running
Published fine-tunes	per release, from an artifact bucket linked here (D-FT-2)

Pages

• Field notes: what fine-tuning is like day to day, drawn from what happened here.
• Decision log: every decision with its date, owner and reason.
• Baseline: which model size knows enough AILANG to be worth tuning.
• Training data: where the data comes from, the rules, and coverage gaps.
• Pilot-1: the recipe, the training curve, the traps, and the A/B result.
• Pilot-2: training in the served format, and what it fixed and did not.
• Pilot-3: training on the model's own mistakes, and the habit of looking things up.
• Pilot-4: a second round, short writes, and breaking loops.
• Serving costs: rig against Cloud Run, Vertex and per-token hosts.

The working design doc, with the full detail, is design_docs/planned/v1_1_0/m-eval-finetuning-data-pipeline.md in the repository. The tooling is in tools/finetune/.