Benedict Aaron Tjandra

New Google DeepMind Paper. Model weights do not have to remain static artifacts that can only be fine-tuned or averaged. A new capability can now be initialized from both what an agent wrote down and what it previously learned into weights. SkillSmith treats prefix key-value caches as another input modality. For a new task, it interleaves existing prefix weights with task descriptions, examples, and a rationale explaining how the source capabilities relate to the target. A forward pass then produces a fresh prefix cache for a frozen Gemma 3 4B model, so the initial composition happens at inference time rather than through a new target-specific training run. The cleanest evidence is the ablation: on Composite-SNI, Elo is 1,455 with only K-V caches, 1,622 with only text, and 1,714 with both. The gain cannot be explained by richer prompting or a learned weight merger alone. The generated prefixes also give downstream fine-tuning a stronger starting point, especially on difficult, data-sparse MMLU-ProX tasks. The boundary matters: on simpler SNI tasks with roughly 1,000 examples each, fine-tuned methods converge. SkillSmith does not remove training, but it changes where adaptation starts: previous text and previous weights can now be composed into the initialization for a new capability. – arxiv. org/abs/2607.27497 Title: "SkillSmith: Learning to Compose Parametric Skills and Textual Knowledge"