Changing How AI Reads Text: The Alternative Path Demonstrated by Bolmo

Retrofitting existing language models for byte-level processing improves character understanding while leaving task-dependent trade-offs

A language model that writes fluent paragraphs does not necessarily handle fine-grained text operations equally well. Understanding its capabilities requires examining how it represents text, alongside its size and training data.

Published in its version of record in Nature on October 7, 2026, “Retrofitting language models to operate over bytes” presents a method for changing that representation. Benjamin Minixhofer and colleagues introduce byteification: converting pretrained language models into byte-level models. Bolmo is a model family created through this approach.

From subword fragments to bytes

Many language models divide text into words or word fragments drawn from a fixed vocabulary. This process, known as subword tokenization, has supported effective language modelling but can obscure details needed for character-level tasks.

Byte-level models begin with the bytes of a text’s UTF-8 encoding. A byte should not be confused with a visible character: a character can require multiple bytes. The distinction concerns the underlying representation from which the model processes text.

Bolmo does not simply send every byte independently through an unchanged large network. Lightweight components process byte information and group it into patches. A deep network retained from the source model processes those patches, after which the architecture returns to byte-level outputs. Because grouping still occurs inside the model, describing the method as eliminating every form of tokenization or segmentation would be misleading.

Converting an existing model

The study’s contribution is an alternative to developing byte-level models solely through training from scratch. Its conversion procedure has two stages. First, the existing core model remains frozen while newly added components learn to reproduce the source model’s behaviour. The entire model is then trained to use byte-level information.

The authors apply the approach across Olmo, Qwen3 and Llama 3, demonstrating conversion beyond a single model family.

They describe the additional conversion budget as less than 1% of a typical pretraining budget. This comparison concerns extra training starting from an existing model. It does not include the source model’s original development cost or establish that the entire system costs 1% as much to create.

Character understanding improves, with trade-offs elsewhere

Table 1 provides a concrete comparison between Olmo 3 7B and its converted counterpart, Bolmo 7B.

These are reported scores from the corresponding columns of Table 1. The small difference in the suite average should not be read as statistically established superiority. Pass@1 evaluates obtaining a correct answer from one generation; pass@16 concerns the probability of including a correct answer among 16 candidates.

Bolmo remains close to its source model on the suite average and shows a substantial improvement in character understanding. Coding performance, however, changes direction depending on the metric. Single-generation performance declines, while performance over multiple candidates rises. Other categories, including mathematics, also record lower scores than the source model.

The findings therefore support a more specific conclusion than a universal performance improvement. They show a practical route to competitive byte-level models, with strong character-level results and task-dependent compromises.

Character understanding also benefited from synthetic training examples designed for character-level operations. The authors include a source-model control continued on the same data mix. The result should consequently be read in relation to both architecture and training data, rather than attributed to changing input units alone.

Why representation belongs in the AI debate

Byteification offers researchers a way to test a different model design while reusing substantial prior training. This could make experimentation with text representations more accessible than repeating the full development process for every architectural alternative.

The team also releases code, model checkpoints, training data and evaluation resources. That gives other researchers material with which to inspect the approach and design follow-up experiments.

The character-level results should not be extended into a claim that all languages or real-world workflows will improve. An application that requires a correct first answer may value the coding results differently from one that can evaluate several candidates. Speed and resource requirements likewise need assessment under relevant conditions.

For META-X, the broader question is how much of AI progress can be explained through scale alone. Bolmo highlights representation and grouping as additional design choices that shape capabilities and efficiency. Deciding how a model reads information remains an active part of building better language models.