Cheat sheet

gLM cheat sheet: GROVER

One reference card for GROVER: architecture, tokenization, size, context, training data, what it was shown to do and what it does not do. One card in a set that covers one genomic language model at a time.

GenomicsLLMMachine Learning

Genomic language model · Cheat sheet

GROVER

Let the genome choose its own vocabulary, then read back what the model learned about sequence context

First public
bioRxiv, July 2023
Published
Nature Machine Intelligence, July 2024
Lab
TU Dresden, CASUS Görlitz
Architecture
BERT. 12 Transformer blocks trained by hiding tokens and predicting them back, kept simple enough to read out what it learned.
Tokenization
BPE, 601-token learned vocabulary
Parameters
~87M
Context
510 tokens. Average token length is 4.07 bases, so roughly 2,000 bases.
Training data
The human reference genome. One species.
Used for
Prediction, and inspecting what the model learned about genome structure

Headline result

On PromScan, finding the 1,000-base window that holds a gene start inside a 10,000-base region, MCC 63% against 52% for Nucleotide Transformer, the next best human model. What it learned also separates promoters, exons, introns, repeat elements and chromatin states, with no labels given.

What it does not do

Human only, and not a generator. The point is the learned vocabulary and what the embeddings encode, not sweeping a benchmark suite.

Commonly confused

The vocabulary was chosen without using any biological task, on purpose. GROVER ranked candidate vocabularies by how well each predicted the next chunk, so no known biology was baked in, and 600 rounds of merging won.

Download this card PNG, 1080 x 1350, sized for a post or a slide

Every model in the landscape figure is one dot with a tooltip. That is enough to compare them and not enough to use one. These cards are the other half: the same ten fields for every model, in the same order, so you can read one without having read any of the others.

The colour on the tokenization row is the same colour the model carries on the landscape chart, which makes each card a zoom-in on that figure rather than a separate thing to learn.

Two of the fields matter more than the rest. Headline result gives the metric and the condition it was measured under, because the number on its own does not tell you much. What it does not do is the field summaries usually leave out, and it is often the one that tells you whether the model fits your problem.

On GROVER specifically, the card’s headline result is not really the point of the paper. The PromScan number is good, MCC 63% against 52% for the next best human model, but GROVER exists to answer a different question: if DNA has no spaces, what are its words?

The answer is a vocabulary the genome chose. Byte-pair encoding starts from the four bases and repeatedly merges the most frequent pair into a longer token, so common patterns become long words and rare ones stay short. Run it for too few cycles and the tokens are uninformative; too many and performance falls off again. GROVER settled on 600 cycles and a 601-token vocabulary, with tokens from one base up to 16 and an average length of 4.07.

How that number was chosen is the part worth copying. Ranking candidate vocabularies on a biological task would have quietly biased the design toward biology already known. So the authors used next-k-mer prediction instead, which asks only whether the model has a sense of sequence context, and picked the vocabulary that did best at it.

What the model then learned is readable back out of the embeddings without supervision: promoters, exons and introns, repeat elements including which strand they sit on, chromatin state, and lexical ambiguity, where the same token means different things in different places.

Numbers are from the Nature Machine Intelligence paper. Sizes are the largest reported checkpoint, and release is the first public preprint, matching the conventions used on the landscape figure.