Nature paper converts subword language models into byte-level models for under one per cent of pretraining cost

10 October 2026

A paper published in Nature on 7 October 2026 sets out 'byteification', a two-stage procedure that retrofits existing subword-based large language models into byte-level models. The authors report doing this with 49.1 billion tokens in all, under one per cent of a typical pretraining budget. They build Bolmo 7B and Bolmo 1B from Olmo 3 7B and OLMo 2 1B, Bwen 8B from Qwen3 8B Base, and Blama 8B from Llama 3 8B. Bolmo 7B gains 16.5 percentage points on STEM tasks over BLT 7B, a byte-level model trained from random initialisation. The byteified models substantially outperform their own source models on character-level tasks. Unlike earlier latent-tokeniser designs, the boundary predictor uses one byte of future context, which better matches how subword tokenisers place boundaries.