Cheat sheet

gLM cheat sheet: Caduceus

One reference card for Caduceus: architecture, tokenization, size, context, training data, what it was shown to do and what it does not do. One card in a set that covers one genomic language model at a time.

GenomicsLLMMachine Learning

Genomic language model · Cheat sheet

Caduceus

The first DNA language model to build strand symmetry and both reading directions into pretraining

First public
arXiv, March 2024
Published
ICML, July 2024
Lab
Cornell, Princeton, Carnegie Mellon
Architecture
MambaDNA. A state-space model, not a Transformer, so no attention and no quadratic cost. Reads both ways at once, either strand giving a match.
Tokenization
Single nucleotide, character-level
Parameters
7.7M
Context
131,072 bases. The longest sequence length it was pretrained at.
Training data
The human reference genome (HG38). One species.
Used for
Prediction and classification, especially over long ranges

Headline result

Outperforms 10x larger Transformer-based models, especially over long ranges. On predicting how a variant shifts gene expression, Caduceus-PS beats Nucleotide Transformer v2 (500M), more so with distance, and past 100,000 bases beats Enformer, built for that task. Best accuracy on all eight Genomics Benchmarks classification tasks.

What it does not do

Human only, and not a generator. Pretrained on one genome, filling in hidden bases rather than writing new sequence.

Commonly confused

"10x larger" is the paper's own phrase, not a measured ratio. The abstract ties it to variant effect prediction against models without bi-directionality or strand symmetry. The 500M baseline is in fact more than 60x this size.

Download this card PNG, 1080 x 1350, sized for a post or a slide

Every model in the landscape figure is one dot with a tooltip. That is enough to compare them and not enough to use one. These cards are the other half: the same ten fields for every model, in the same order, so you can read one without having read any of the others.

The colour on the tokenization row is the same colour the model carries on the landscape chart, which makes each card a zoom-in on that figure rather than a separate thing to learn.

Two of the fields matter more than the rest. Headline result gives the metric and the condition it was measured under, because the number on its own does not tell you much. What it does not do is the field summaries usually leave out, and it is often the one that tells you whether the model fits your problem.

On Caduceus specifically, three things are worth stating.

The size. The paper reports the configuration of its largest pretrained model, 131,072 bases of context at hidden dimension 256 over 16 layers, rather than a parameter count. The 7.7M on the card is the released checkpoint of that configuration, and the same number the landscape figure carries.

The design idea, which is the reason the model is small. Either strand of DNA determines the other exactly: reverse it, swap every base for the one it pairs with. Caduceus encodes that relationship rather than learning it from extra training data, and reads in both directions at once. The paper puts this as the two strands carrying “the same information”, which holds for the sequence and not for the biology. Either string determines the other by a rule with no choices in it. The strands themselves are not interchangeable: only one is the template for any given gene, and genes sit on both. What the model should ignore is which of the two descriptions ended up in the file.

Equivariant, not invariant. An invariant model would return the same output for a sequence and for its reverse complement. Theorem 4.1 says Caduceus does something else: flip the input and the predictions flip to match, reversed along the sequence and complemented. Nothing is discarded, so orientation survives, which matters because promoters and genes have a direction. Invariance is enforced downstream instead, where either strand of an assayed sequence carries the same label. That is an assumption, and the paper’s benchmarks are locus-level, so it fits what it is tested on.

One result did not fit on the card. On the eighteen Nucleotide Transformer tasks, Caduceus beats attention models with orders of magnitude more parameters on eight of them, not on all eighteen.

Numbers are from the ICML paper. Sizes are the largest reported checkpoint, and release is the first public preprint, matching the conventions used on the landscape figure.