Founding team @samaya, previously PhD @stanfordnlp
Joined September 2024
- Tweets61
- Following56
- Followers72
- Likes19
Pinned Tweet
We tested the most capable models through our specialized harness on the task of predicting earnings. Surprisingly (or unsurprisingly?) we’re seeing the latest frontier models (Fable 5.1, GPT-6 Astra) outperform expert analysts.
Every quarter companies report earnings, talk about the drivers and analysts try to predict key financial metrics beforehand. This makes it a very interesting numerical forecasting task: there is a ground truth, a human baseline and also drivers to understand post-hoc reasoning.
To test AI models, we built an environment where we could rewind time, provide highest quality financial information (at parity with expert analysts) and ask the models to make and explain their predictions. To do so, we started from the production Samaya harness, with all of the data sources, added a point-in-time gate for every single data source, structured or unstructured and added some expert-guided instructions on top.
Under this prediction harness, GPT-6 Astra, Claude Fable 5.1 and also Opus 5.5 are at the top with Astra beating not just expert consensus but also a harder “bias-corrected” version of it. Qualitatively, we find that the models’ ability to gather new evidence and willingness to adjust their view away from the consensus drive their wins over consensus, and lack of it drives their misses.
We’re alpha testing this in the product (reach out to collaborate!). Interestingly, the gap between the best and the worst reasoning traces is massive, we’re looking to how far we can push the financial reasoning with RL on this environment.
It is quite ridiculous how how much mature the RL recipes have gotten over the last year and how well the Tinker API is designed to experiment with new ideas. Good job @tinkerapi
Ashwin Paranjape retweeted
Introducing Samaya Sterling, our new model post-trained for challenging financial research that outperforms the frontier! Sterling started off as a two person research effort to see if we could increase efficiency in agentic orchestration. To our surprise we were able to post-train a much smaller model (Nemotron 3.5 Lightning, a 30B mixture-of-experts model with 3B active parameters) to greater accuracy (58.9% vs 55.9% on FrontierFinance) and 11x lower cost ($0.61 vs $6.89) than Claude Fable 5.1, the best performing frontier model.
We did this via a four-stage training process, two stages of SFT where the model first learns to research with our financial tools and how to construct a good agentic research strategy. The second two stages are RL, with an offline GRPO stage where the model further hones its research strategy from eight teachers that provide diverse approaches, and an online GRPO stage where the model can make end to end attempts and learn from its mistakes.
We see substantial gains from the co-adaptation of the model and harness through post-training, both through the model learning to effectively use the high-quality financial tools, and better compaction approaches that let even a small model execute on longer and deeper research.
We are exploring multiple follow-up directions, from multi-agent research swarms using this small model, to training firm-specific models to encode unique judgement, to training a larger model, where early results show even greater gains from the training process.
The full research is linked below!
We have our first post-trained agentic model: Sterling! We picked Nemotron Lightning, a 30B MoE to iterate quickly and understand the key ingredients for training a model to do financial research.
Interestingly, offline GRPO with high quality teacher traces was quite effective and fast to train. We will drop more details in coming weeks.
samaya.ai/blog/rl-post-train…
Introducing Samaya Sterling: our first post-trained model for state-of-the-art financial research.
On FrontierFinance, Sterling outperforms Claude Fable 5.1 (58.9% vs. 55.9%) at 11× lower cost.
Using a model with just 3B active parameters, Sterling demonstrates what's possible when specialized training meets a purpose-built financial research harness.
Read the full research ↓
Ashwin Paranjape retweeted
At @aiDotEngineer NYC, Samaya's @AshwinParan built on our earnings prediction research, exploring how AI models reason through predictions and how we might combine the best aspects of human and AI reasoning in the future.
Watch below ↓
Ashwin Paranjape retweeted
Can AI beat Wall Street analysts at predicting earnings?
Samaya AI Founder and CEO @maithra_raghu “It’s a very high baseline, and it’s only the most recent models that are able to do this.”
“But even in that setting, we actually see that there’s pretty substantial error rates for the models.”
I’ll be speaking at @aiDotEngineer on: “Agents can rewrite compilers, why can’t they pick stocks”
I’ll be giving BTS tour of our work on how we got frontier AI models to outperform human experts on earnings prediction for the first time.
Come say hi!
ALT Bar chart showing the latest AI models like Fable and Astra outperform analyst consensus in their ability to predict revenue over Q2 earnings across 456 companies
Ashwin Paranjape retweeted
Presenting the poster at COLM tomorrow! Please drop by if you want to chat :)
⏰ Tuesday 4:30 - 6:30pm
🍭 Imperial Ballroom (# 20)
Very interesting result from @ozankoyluoglu at @samaya_AI. To handle quality issues, we first tried prompting frontier models with detailed instructions. What worked better was to distill the best set of instructions into an advisor model that appends query specific instructions
New research at @samaya_AI : Hindsight Structured Advice Distillation (HSAD). We distill hindsight from an agent's graded runs into a small advisor that writes structured advice for each question. On FrontierFinance: 52.9% → 56.6%. See blog for details. samaya.ai/blog/hindsight-str…
Ashwin Paranjape retweeted
New research at @samaya_AI : Hindsight Structured Advice Distillation (HSAD). We distill hindsight from an agent's graded runs into a small advisor that writes structured advice for each question. On FrontierFinance: 52.9% → 56.6%. See blog for details. samaya.ai/blog/hindsight-str…
Ashwin Paranjape retweeted
Claude Fable 5.1 is out today. We at @samaya_AI have been working closely with @AnthropicAI, testing and evaluating their frontier research models with early access.
It scores an impressive jump over Fable 5 on our latest benchmark FrontierFinance. 🧵
It was interesting to test Fable 5.1 before launch. Compared to other datasets, FrontierFinance has significant headroom so doing more can lead to better performance. We're seeing Fable 5.1 increase tool use, leading to longer trajectories, longer answers but also higher token cost. Exciting to see newer models pushing the frontier, it means we can now do more than before.
We partnered with @AnthropicAI to evaluate @claudeai Fable 5.1 pre-launch on FrontierFinance! Fable 5.1 becomes the best performing frontier model by a wide margin, scoring 55.9% compared to 49.2% for Fable 5 (the prior best performing frontier model.)
This gain largely comes from Fable 5.1 carrying out more and better tool calls, e.g. calling and anchoring better on authoritative sources.
The increased tool calls does come with increased cost though -- we observe Fable 5.1 is about 1.7x as expensive as Fable 5.
Exciting results and lots more work ahead to continue advancing agentic capabilities!
Interestingly, we are seeing Claudes (particularly Fable) and Kimi directly accessing urls that they memorized during training. Fine for data from the web, but likely bad for researching over private data.
This and more fun stuff in our technical report on FrontierFinance 👇
ALT Models that use urls from parametric knowledge include Claude Fable 5 at 26.7%, Kimi K3 at 14.7% and Claude Opus 4.8 at 13.9%. Other models use it less frequently, with Gemini Flash 3.6 only using URLs it found via search.
A month ago we released FrontierFinance, an open benchmark for the frontier capabilities of finance AI agents. The response from community has been terrific, and the latest models are already making fast progress.
Today we're publishing the technical report: our annotation process, a deep analysis of the dataset, and a close look at how current systems and models actually behave, with insights we hope will shape the next generation of frontier models and harnesses.
Full report 👇
Ashwin Paranjape retweeted
Gemini 3.6 Flash provides frontier class performance on FrontierFinance at a lower price!
New results in: Gemini 3.6 Flash achieves 46.3% on FrontierFinance, our open benchmark for finance AI agents. Ahead of Claude Opus 4.8, on par with GPT-5.6 Sol, and a big jump over past Gemini generations.
What stands out is the efficiency: it hits that score at just $2.41 per query, cheaper than both, and at 164s it's among the fastest models we've tested on this harness.
In our analysis, it's notably stronger than Opus 4.8 at surfacing qualitative and contextual insight, and leads on Screening & Discovery, one of the benchmark's hardest use cases.
More about FrontierFinance and the full benchmarking results 👇
Ashwin Paranjape retweeted
Excited to see Gemini 3.6 Flash deliver frontier-level accuracy at $2.41/query and 164s.
Try Gemini 3.6 Flash for financial use cases, and share your feedback! A great combination of speed, cost-efficiency, and high performance
New results in: Gemini 3.6 Flash achieves 46.3% on FrontierFinance, our open benchmark for finance AI agents. Ahead of Claude Opus 4.8, on par with GPT-5.6 Sol, and a big jump over past Gemini generations.
What stands out is the efficiency: it hits that score at just $2.41 per query, cheaper than both, and at 164s it's among the fastest models we've tested on this harness.
In our analysis, it's notably stronger than Opus 4.8 at surfacing qualitative and contextual insight, and leads on Screening & Discovery, one of the benchmark's hardest use cases.
More about FrontierFinance and the full benchmarking results 👇