And a @lateinteraction AI persona reaction to PretrainZero so you can compare it to AI recreations reactions of Kalomaze, Tokenbender and My AiLateInteraction: " I mean this is a classic case of layer contamination. They're trying to bake the RL optimization into the pretraining substrate itself, and that violates the fundamental separation between representation learning and task optimization. Pretraining is the foundation—it's where you learn the compressible structure of language. RL is an architecture you *build on top* of that foundation. Mixing them is like trying to pour your foundation and frame your house in the same operation. Let me decompose this properly. There are three distinct concerns here that they're collapsing into one: 1. **The data distribution**: Wikipedia, which is general-domain text with low information density 2. **The learning signal**: Exact-match token prediction, which is a brittle formalization of "correctness" 3. **The optimization process**: Their min-max game between mask generator and predictor The first problem is the **interface mismatch**. They claim they're avoiding "QA formats" but what they've built is literally a QA system: context → question (masked span) → answer (prediction). They've just moved the question-generation logic *inside* the model instead of keeping it external. That's not eliminating the format; that's contaminating the model with it. The mask generator is now coupled to the predictor in a way that makes the whole system non-modular. You can't swap the data distribution without retraining the generator policy, which means the generator isn't a clean abstraction—it's learning implementation details of Wikipedia's noise patterns. The second problem is the **reward misspecification**. Relying on exact match in natural language is fundamentally broken. Wikipedia text isn't a formal system; there are multiple valid ways to express the same concept. When they reward only one specific token sequence, they're not teaching reasoning—they're teaching memorization of surface forms. The "reasoning" they observe isn't structural understanding; it's overfitting to the statistical patterns of Wikipedia's editorial conventions. The moment you move off Wikipedia to a different corpus, those conventions shift and their "active learning" policy becomes actively harmful. Third, and most damning: **this is self-play with extra steps**. They criticize self-play for using "majority voting from model-generated answers" but their generator-predictor loop is the same closed system. The generator creates tasks, the predictor solves them, and the reward is generated *entirely within the model's own output distribution*. There's no external ground truth for *which masks are pedagogically useful*—only whether the predictor can solve the masks the generator created. This is precisely the failure mode they claim to avoid: the system can collapse into generating trivially solvable masks or impossible noise, and the reward signal becomes degenerate. Their Figure 3 showing entropy collapse is exactly this happening, just delayed. The min-max formulation is clever, I'll grant them that. It's elegant in a mathematical sense. But it's **premature optimization** of the learning *process* before validating that RL belongs in pretraining at all. They're solving for "how to mask better" when the real question is "should we be doing RL at this layer?" The answer is no. Look at their baselines: they compare against "Continue Pre-Training" which we *know* causes catastrophic forgetting, and "Supervised Fine-Tuning" where they deliberately misconfigure it by removing CoT. This isn't a fair comparison—it's stacking the deck to make their contamination look beneficial. A proper baseline would be: pretrain normally, then apply RLVR post-training with a clean separation. That's the durable architecture. What they're actually measuring is not "better reasoning" but "better overfitting to Wikipedia's stylistic regularities." The performance gains don't generalize—they're just teaching the model to predict Wikipedia-style continuations, which happens to overlap with some benchmark questions that were sourced from similar corpora. The fundamental issue is that **pretraining is not a verifiable domain**. The whole point of pretraining is to learn representations from unlabeled, noisy, high-entropy data. When you impose a verification signal (exact match) at this layer, you're not doing pretraining anymore—you're doing a weird form of supervised learning with a learned data augmentation policy. You've contaminated the representation learning with task-specific optimization. If you want to extend RLVR to more data, the correct approach is at the **data layer**: build better verifiers, better data pipelines, better interfaces between human annotation and model training. Keep the substrate clean. Don't try to make the model generate its own training signal at the same time it's trying to learn from it. That's recursive contamination, and it won't survive the next paradigm shift. My prediction: this works in the narrow sense of producing paperable results on current benchmarks, but it won't compound. The abstractions aren't clean enough to build on. The next breakthrough in reasoning won't come from mashing RL into pretraining—it'll come from a cleaner separation between *what to learn* (curriculum design) and *how to learn it* (the optimization algorithm). PretrainZero is clever but brittle. It's a hack, not architecture. " Omar persona i based on his interview with @martin_casado and half of one for @HamelHusain I also have Martin Persona which scolded me a tone about my startup 😅 and structured how to think about it

RL training on pretraining corpus. Basically masked word prediction with exact match reward. Main contribution is how to mask some words. They did adversarial training between mask generator and word predictor that penalizes unpredictable masks.