taking REINFORCE seriously
Montreal, Qc
Joined October 2011
- Tweets162
- Following4.8K
- Followers1.5K
- Likes15.2K
Pinned Tweet
Very excited to see vLLM supports Pipeline RL’s in-flight weight updates! It allowed our team to quickly and reliably train Qwen base 7B to reason from scratch! Want to hear more? Join us at our Pipeline RL expo talk at CoLM this Thursday 1PM room 524C.
🚀 The RL community keeps pushing boundaries — from better on-policy data and partial rollouts to in-flight weight updates that mix KV caches across models during inference.
Continuing inference while weights change and KV states stay stale sounds wild — but that’s exactly what PipelineRL makes work.
vLLM is proud to power this kind of modular, cutting-edge RL innovation. Give it a try and share your thoughts!
Alexandre retweeted
An exciting update on our progress at Periodic, highlighting some of our ML/infra work to automate challenging X-Ray Diffraction analysis.
We built high-throughput materials labs in Menlo Park to create a loop between experiments and models. The labs generate fresh data, the models learn from it, and then help us decide what to try next.
Using only 1,300 H200s, plus months of our experimental data, we mid-trained and RL’d an open-source model to surpass GPT-6 Astra on our analysis benchmark. We call it Neon.
This is real footage from our lab. We’re focusing first on hard problems in materials science, including superconductors, magnets, and semiconductor materials.
Read our blog posts below.
Alexandre retweeted
At @periodiclabs we are completing the virtuous circle: AI helps to run our labs, AI learns from our labs.
It's been soo much fun working with hardcore scientists and badass LLM infra experts to train a 1T parameter XRD analysis expert on our lab data. Check out our blog posts!
We built high-throughput materials labs in Menlo Park to create a loop between experiments and models. The labs generate fresh data, the models learn from it, and then help us decide what to try next.
Using only 1,300 H200s, plus months of our experimental data, we mid-trained and RL’d an open-source model to surpass GPT-6 Astra on our analysis benchmark. We call it Neon.
This is real footage from our lab. We’re focusing first on hard problems in materials science, including superconductors, magnets, and semiconductor materials.
Read our blog posts below.
Alexandre retweeted
Visited the lab - was struck both by how wide the search space is for materials synthesis experiments, and also how amenable it is to depth first search, where the design and informativeness of your next experiment improves as you pile up more data from previous runs.
We built high-throughput materials labs in Menlo Park to create a loop between experiments and models. The labs generate fresh data, the models learn from it, and then help us decide what to try next.
Using only 1,300 H200s, plus months of our experimental data, we mid-trained and RL’d an open-source model to surpass GPT-6 Astra on our analysis benchmark. We call it Neon.
This is real footage from our lab. We’re focusing first on hard problems in materials science, including superconductors, magnets, and semiconductor materials.
Read our blog posts below.
Alexandre retweeted
We are doing RL to analyze real experimental data from physical labs!
In the process of building RL infra to train trillion parameter models at scale, and we also ended up building our own sandbox solution pbox, and it works great for our scientific workloads!
We built high-throughput materials labs in Menlo Park to create a loop between experiments and models. The labs generate fresh data, the models learn from it, and then help us decide what to try next.
Using only 1,300 H200s, plus months of our experimental data, we mid-trained and RL’d an open-source model to surpass GPT-6 Astra on our analysis benchmark. We call it Neon.
This is real footage from our lab. We’re focusing first on hard problems in materials science, including superconductors, magnets, and semiconductor materials.
Read our blog posts below.
Alexandre retweeted
Genius is 99% perspiration, and 1% inspiration. This is also true for how we trained Periodic Neon to beat frontier models, like Astra and Fable 5.1, on a scientific capability our labs care about.
We trained Neon with reinforcement learning on our lab data using only 1.3K GPUs, orders of magnitude less than Astra. As we scale our labs and compute, we expect our models to tackle much more complex scientific problems.
We built high-throughput materials labs in Menlo Park to create a loop between experiments and models. The labs generate fresh data, the models learn from it, and then help us decide what to try next.
Using only 1,300 H200s, plus months of our experimental data, we mid-trained and RL’d an open-source model to surpass GPT-6 Astra on our analysis benchmark. We call it Neon.
This is real footage from our lab. We’re focusing first on hard problems in materials science, including superconductors, magnets, and semiconductor materials.
Read our blog posts below.
Alexandre retweeted
We’ve spent the past year building high-throughput labs and infrastructure for large-scale physics experiments in our search for better superconductors and magnets.
We ran our first experiments by hand. Now, experiments run continuously in the labs. We’re building AI to make sense of what they produce and to determine which experiments to run next.
We built high-throughput materials labs in Menlo Park to create a loop between experiments and models. The labs generate fresh data, the models learn from it, and then help us decide what to try next.
Using only 1,300 H200s, plus months of our experimental data, we mid-trained and RL’d an open-source model to surpass GPT-6 Astra on our analysis benchmark. We call it Neon.
This is real footage from our lab. We’re focusing first on hard problems in materials science, including superconductors, magnets, and semiconductor materials.
Read our blog posts below.
Alexandre retweeted
We midtrain + RL’d a trillion param LLM to analyze experimental data from our superconductor lab
It’s better at it than Astra & Fable, and our scientists love it
Read our first research report below
We built high-throughput materials labs in Menlo Park to create a loop between experiments and models. The labs generate fresh data, the models learn from it, and then help us decide what to try next.
Using only 1,300 H200s, plus months of our experimental data, we mid-trained and RL’d an open-source model to surpass GPT-6 Astra on our analysis benchmark. We call it Neon.
This is real footage from our lab. We’re focusing first on hard problems in materials science, including superconductors, magnets, and semiconductor materials.
Read our blog posts below.
Alexandre retweeted
We built high-throughput materials labs in Menlo Park to create a loop between experiments and models. The labs generate fresh data, the models learn from it, and then help us decide what to try next.
Using only 1,300 H200s, plus months of our experimental data, we mid-trained and RL’d an open-source model to surpass GPT-6 Astra on our analysis benchmark. We call it Neon.
This is real footage from our lab. We’re focusing first on hard problems in materials science, including superconductors, magnets, and semiconductor materials.
Read our blog posts below.
Alexandre retweeted
At @periodiclabs we are looking for brilliant new colleagues who excel at bridging AI and science. The role's main focus is advanced evals and training data/tasks. But you will also get involved in both internal science and AI work. Apply/spread the word!
jobs.ashbyhq.com/periodic-la…
Alexandre retweeted
damn @periodiclabs just leaked that they're building an AI-accelerated semiconductor research lab
is this what recursive self-improvement looks like?
PipelineRL finally supports vLLM v1!
Our first vLLM V0→V1 run on PipelineRL looked broken.
@ehsk0 and I almost reached for an objective-side correction. That would have been the wrong fix.
The real problem: four mismatches in the rollout backend.
🧵
Alexandre retweeted
Today we’re releasing Laguna XS.2, Poolside’s first open-weight model.
It’s a 33B total / 3B active MoE model built for agentic coding and long-horizon tasks.
Trained fully in-house on our own stack. Runs on a single GPU. Released under Apache 2.0.
Links 👇
Weights: huggingface.co/poolside/Lagu…
API: platform.poolside.ai/
Blog: poolside.ai/blog/laguna-a-de…
Alexandre retweeted
Today we’re releasing Laguna M.1 and Laguna XS.2, our first public models.
Laguna XS.2 is our first open-weight release, with weights available today on Hugging Face:
huggingface.co/poolside/Lagu…
A few details on what went into them: large-scale pre-training, data mixture optimization, synthetic data, optimizer efficiency, and async agent RL.
Alexandre retweeted
New blog on RL scaling laws coming out tomorrow morning. Scaling is one of the most impactful concepts in the history of AI research, but "scaling laws" are an overloaded (sometimes confusing) concept. Scaling laws for pretraining and RL are entirely different concepts. Scaling laws for pretraining are well-defined and have undergone extensive empirical validation, whereas scaling for RL is messy, bespoke, and full of intricate / evolving details. I hope this writeup provides a little clarity to this complex topic.
Alexandre retweeted
If you're at @NeurIPSConf tomorrow, come say hi and learn about our work on training web agents 🤖🌐
I’m at NeurIPS all week and presenting this work tomorrow! Come say hi or DM if you want to talk agent post-training, large-scale RL or Enterprise agents :)
📅 Wednesday, December 3
🕐 11:00 AM
📍Exhibit Hall C,D,E #4109
It’s my last week at @ServiceNowRSRCH after joining ElementAI as an intern in 2018. Grateful for the incredible mentors and collaborators over the years.
Ending things on a high note: PipelineRL won Best Paper at nowAI earlier this month! Hi @lawrennd 👋
Alexandre retweeted
With in-flight updates (PipelineRL, from @alexpiche_, @DBahdanau et. al), we update our actors in the middle of generation. The system is much faster as we don't have to drain the generation queues to update the weights (which is the same problem as static batching).
Alexandre retweeted
to continue the PipelineRL glazing, @finbarrtimbers implemented PipelineRL for open-instruct a little bit ago and it ended up being probably the single biggest speedup to our overall pipeline. We went from 2-week long RL runs to 5-day runs, without sacrificing performance (combined with some other threading etc. updates).
Here's IFEval perf for an internal model (same data, same starting model, same bsz). Same number of training steps, same end perf, but PipelineRL is much faster.
Don't sleep on PipelineRL -- this is one of the biggest jumps in compute efficiency of RL setups that we found in the ScaleRL paper (also validated by Magistral & others before)!
What's the problem PipelineRL solves? In RL for LLMs, we need to send weight updates from trainer to generator (to generate data from our latest policy being trained).
(Conventional PPO-off-policy) A naive approach would be to "start generators on a batch, wait for all sequences to complete, update the model weights for both trainers and generators, and repeat. Unfortunately, this approach leads to idle generators and low pipeline efficiency due to heterogeneous completion times.
(Pipeline-RL) Instead, we simply let the generators continue generating tokens without discarding or finishing ongoing generations in-flight whenever we need to do a weight update -- doing an "in-flight" weight update. As such our KV caches for these generations would be stale, as they would come from LLM with earlier copy(ies) of the weights) but this is ok (see below).
