Cheat sheet

gLM cheat sheet: Nucleotide Transformer

One reference card for Nucleotide Transformer: architecture, tokenization, size, context, training data, what it was shown to do and what it does not do. One card in a set that covers one genomic language model at a time.

GenomicsLLMMachine Learning

Genomic language model · Cheat sheet

Nucleotide Transformer

The systematic study of what pretraining across many genomes buys you, plus the 18-task benchmark to measure it

First public
bioRxiv, January 2023
Published
Nature Methods, February 2025
Lab
InstaDeep, NVIDIA, TU Munich
Architecture
BERT-style Transformer encoder, trained by hiding tokens and predicting them back. Version 2 rebuilt it on newer Transformer components.
Tokenization
k-mer, non-overlapping 6-mers
Parameters
2.5B
Context
6,000 bases, about 1,000 tokens. Version 2 doubled the window to 12,000.
Training data
3,202 human genomes plus 850 genomes from other species, 11 of them model organisms.
Used for
Prediction and classification, by reading out its features or fine-tuning

Headline result

Highest overall performance across its own 18-task benchmark of regulatory and splicing predictions, though Enformer, built specifically for expression, still won some enhancer and chromatin tasks. On splice sites the 2.5B multispecies model matched SpliceAI-10k, a dedicated tool trained on 15,000-base inputs, at PR-AUC 0.98 and 95% top-k accuracy.

What it does not do

Not a generator. It fills in hidden tokens rather than writing new sequence, and each downstream task still needs fine-tuning or a trained readout.

Commonly confused

Bigger was not better here. The 250M version 2 model scored higher on the benchmark than the 2.5B version 1 model, average MCC 0.769 against 0.755.

Download this card PNG, 1080 x 1350, sized for a post or a slide

Every model in the landscape figure is one dot with a tooltip. That is enough to compare them and not enough to use one. These cards are the other half: the same ten fields for every model, in the same order, so you can read one without having read any of the others.

The colour on the tokenization row is the same colour the model carries on the landscape chart, which makes each card a zoom-in on that figure rather than a separate thing to learn.

Two of the fields matter more than the rest. Headline result gives the metric and the condition it was measured under, because the number on its own does not tell you much. What it does not do is the field summaries usually leave out, and it is often the one that tells you whether the model fits your problem.

On the Nucleotide Transformer specifically, the card covers both versions, because the paper does. That is also where the number worth knowing sits. Version 1 scaled to 2.5B parameters, and the card carries that as the size by the same convention the landscape figure uses. But the 250M version 2 model then scored higher on the same 18-task benchmark, average MCC 0.769 against 0.755. Ten times smaller, slightly better. If you are picking a model off this line to actually use, the headline size is the wrong number to sort by.

The headline result carries a hedge worth keeping. The 2.5B multispecies model had the best overall performance across the benchmark, but Enformer still won some enhancer and chromatin tasks. “Best overall” is an average, not a sweep.

The splice site number is the one that travels: PR-AUC 0.98 and 95% top-k accuracy, matching SpliceAI-10k. SpliceAI was trained on 15,000-base inputs for that task specifically, and the Nucleotide Transformer reached it from general pretraining plus fine-tuning.

Numbers are from the Nature Methods paper. Sizes are the largest reported checkpoint, and release is the first public preprint, matching the conventions used on the landscape figure.