Joined November 2019
- Tweets425
- Following381
- Followers21.1K
- Likes3.6K
Pinned Tweet
We created SuperBPE🚀, a *superword* tokenizer that includes tokens spanning multiple words.
When pretraining at 8B scale, SuperBPE models consistently outperform the BPE baseline on 30 downstream tasks (+8% MMLU), while also being 27% more efficient at inference time.🧵
ALT Segmentation of the sentence "By the way, I am a fan of the Milky Way" under BPE and SuperBPE.
Alisa Liu retweeted
"In a hole in the ground, there lived a hobbit."
BPE spends 13 tokens on this line, 7 of them on "in", "a", "the" and punctuation. CoBPE attaches those to their words: 6 tokens, same text.
Across English: 30% shorter sequences, better LLMs at the same compute. #COLM2026 🧵
Alisa Liu retweeted
Didn't expect to see LEGATO in the GPT-6 report 🤯🎼
GPT-6 Astra cites our work — and even uses the LEGATO camera subset for its OMR eval.
We’ve since pushed it further with LEGATO 2:
arxiv.org/abs/2607.05769
Alisa Liu retweeted
Motif 3 is using a SuperBPE tokenizer! Very happy to see that superwords are boosting encoding efficiency (=> inference speed) in Korean as well as important domains like math and code!
Motif 3 is officially here.
Today, we’re releasing Motif 3 Base and Motif 3.
Motif 3 Base is our pretrained foundation model, while Motif 3 is built on top of it with post-training using NVIDIA NeMo-RL.
Both models were developed with the support of Korea's Ministry of Science and trained on NVIDIA B200 GPUs.
With Motif 3, we’re taking our first step into the frontier LLM race.
There’s still a long way to go, but we believe we’ve built a model that deserves to be in the race.
We’ve shared the details in our technical report and blog post.
In collaboration with NVIDIA, we’re also releasing an NVFP4 version of Motif 3.
All models will be available on Hugging Face.
Links are in the comments.
Alisa Liu retweeted
super excited to finally get this out. it’s the best model in the world for computer use and any professional workflow, and it’s just generally brilliant, fast, diligent, and it has high taste now! I was shamefully not too agi pilled in that I often did some work myself, but it completely changed since we started dogfooding 5.6 internally. Extremely capable, with little hand holding, and it constantly found issues in its own training/eval.
it’s gonna be even more difficult to find some work that our model can’t help with me i guess :’)
Alisa Liu retweeted
@alisawuffles already fixed tokenizers. i just wish more models would adopt it :( arxiv.org/abs/2503.13423
but totally agree with your post. companies willing to do post-training can see a lot of gains here
I'm joining OpenAI next week!🥹 The job search turned out to be really challenging but also super rewarding, so I wrote a small blog to share what I learned along the way and hopefully make the process a little less mysterious for the next person. alisawuffles.github.io/blog/…
Alisa Liu retweeted
Introducing ><former
Most transformers are rectangles◻️: every layer has the same width
But is that optimal?🤔
We propose variable-width transformers that have different widths across layers, improving loss while cutting compute & KV cache size 🧵
Alisa Liu retweeted
nice pre training work by nous claiming ~2.5x efficiency gains, building on previous research like MTP/SuperBPE. overall intuition is that at each step you want the model to process and predict more tokens
Today we release Token Superposition Training (TST), a modification to the standard LLM pretraining loop that produces a 2-3× wall-clock speedup at matched FLOPs without changing the model architecture, optimizer, tokenizer, or training data.
During the first third of training, the model reads and predicts contiguous bags of tokens, averaging their embeddings on the input side and predicting the next bag with a modified cross-entropy on the output side. For the remainder of the run, it trains normally on next-token prediction. The inference-time model is identical to one produced by conventional pretraining.
Validated at 270M, 600M, and 3B dense scales, and at 10B-A1B MoE.
The work on TST was led by @bloc97_, @gigant_theo, and @theemozilla.
In SuperBPE we found: as tokenizer compression increases, the compute-optimal ratio of train tokens to model params decreases — and remarkably, corresponds to the same underlying ratio of train *bytes* / param! Our new work makes it official: scaling laws depend on compression.
Alisa Liu retweeted
Follow the Flow was accepted to #ACL2026 main! 🎉
T2I models often struggle with text-image alignment issues. These are usually addressed at the diffusion stage. We offer a different POV: analyzing and intervening at the textual encoding.
📄 arxiv.org/abs/2504.01137
Alisa Liu retweeted
It echoes findings in LLMs beyond T2I:
Semantic Leakage in LMs (@hila_gonen)
detokenization and token erasure (From tokens to words + @sheridan_feucht)
model-internal "words" spanning multiple surface words (@alisawuffles, SuperBPE)
Alisa Liu retweeted
Fast Byte Latent Transformer is accepted to ICML 2026! ⚡🥪
Byte-level LMs promise to free us from subword tokenizers, but decoding one byte at a time is super slow.
We make BLT generation more efficient with BLT-D: text diffusion for parallel byte decoding. 1/
Alisa Liu retweeted
We present Compute Optimal Tokenization! 🔡
Common in LLM scaling works stick to one tokenizer, sweeping data/model size.
But what happens when we control the tokenizer’s compression rate (bytes/token)?
Here we sweep tokenizers, params, and data across compute budgets: [1/N]
Alisa Liu retweeted
Anthropic says Opus 4.7 uses “1.0-1.35x more tokens.”
My quick measurements:
Python / TOML / JSONL / structured data: ~1.20-1.30x
English / Dutch prose: ~1.4-1.5x
Korean prose: basically unchanged, though Korean was already poor on 4.6
Alisa Liu retweeted
1/ "New tokenizer" does not imply "new base model," and "new base model" is not the simplest explanation. There are much simpler explanations that fit Anthropic's public description of Opus 4.7 equally well.
Alisa Liu retweeted
4/ Zheng et al. 2025 ("Broken Tokens") showed that non-canonical tokenizations at inference time can improve instruction-tuned model performance. E.g., character-level tokenization improves string manipulation and code understanding tasks. arxiv.org/abs/2506.19004
Alisa Liu retweeted
Wow the first time Olmo is top is for LEAST DEPRESSED that's a huge win. Go team.
Happy models make happy users.
Alisa Liu retweeted
📢 Life update 📢
After a wonderful time at @allen_ai, I've joined @CisLmu at @LMU_Muenchen as a tenure-track assistant professor in NLP. Thrilled to be back in Europe and to start a lab in Munich's flourishing AI ecosystem! 🎉
