@avyvar

co-founder @daridotdev // prev @ycombinator, @caltech, basketball data @lakers, software @uber, startups

 Fremont, CA
dari.dev
Joined April 2018
  • Tweets121
  • Following687
  • Followers722
22Media
Today, we're releasing our open-weight, auto-routing model @daridotdev, built for coding agents. We're state-of-the-art on the Pareto Frontier, w/ 70% cost reduction + comparable coding performance to Fable. Bring your own evals, choose your models, or use our defaults.
16
22
7
134
79,380
Are OpenAI models not the same on sub / API? Ran GPT-6 Luna (high) on DeepSWE 1.1 in Codex: > With subscription: 26% solved > With API (via Vercel AI Gateway): 41% solved Thought it might be variance, but the sign-in run also used ~4x fewer reasoning tokens per task
29
3
3
281
25,339
@thsottiaux is this expected?
1
1
36
3,512
DeepSeek-V4.1-Flash-1017 is Pareto-mogging in our initial tests. It's the first open-source model we've tested that's consistently on the Pareto frontier in real-world PRs, running in @pidotdev Some results from Sentry, PostHog, and Next.js PRs:
DeepSeek
@deepseek_aiOct 17
🚀 DeepSeek-V4.1-Flash-1017 is here! Stronger benchmark performance, faster task completion, and shorter outputs than the previous version—a Pareto improvement in performance and cost. Now live on our website, app, and API. 🛠️ Optimized for @DeepSeekHarness, Tencent CodeBuddy, WorkBuddy, and more. Upgrade to DeepSeek Harness v0.2.1-rc.1 or later for the best experience with the new model. 1/4
12
12
3
188
22,941
View tasks and traces here: - selfbench.dev/vercel/next.js - selfbench.dev/PostHog/postho… - selfbench.dev/getsentry/sent…

vercel/next.js: Coding Agent Benchmark · SelfBench

Which coding agent works best on vercel/next.js? Most accurate: GPT-6.1 Sol (High), 84.2% at $0.91 per task. Best for less: GPT-6.1 Sol (Low), 78.9% at $0.56.

selfbench.dev
1
4
1,378
Haiku 5.5 is an incredibly capable model at its size. We ran Haiku on real Pi, Next.js, PostHog, and Sentry tasks, and it was on the Pareto frontier in every test. Slightly higher $/task than Luna for increased performance. Towards intelligence too cheap to meter!
Avyay Varadarajan
@avyvarOct 5
Open-weight models may be costing you more on real work. We built self-bench to turn real GitHub PRs into evals. Testing repos like Next.js, PostHog, and Sentry, open models were usually pricier + less capable. Find the best model for your repo: selfbench.dev
5
1
16
981
Is @MistralAI Large 4 smart or just chonk-y? We ran Le Chonk on real PRs from Next.js, Sentry, Pi, and PostHog. > Mistral Large 4 generally benchmarked worse + similar price as Kimi K3 and GLM 5.3 in the Pi harness. > None of the 3 models were ever on the Pareto frontier.
Avyay Varadarajan
@avyvarOct 5
Open-weight models may be costing you more on real work. We built self-bench to turn real GitHub PRs into evals. Testing repos like Next.js, PostHog, and Sentry, open models were usually pricier + less capable. Find the best model for your repo: selfbench.dev
14
3
1
23
2,897
Avyay Varadarajan retweeted
Techmeme
@TechmemeOct 5
Survey: only 11% of ~400 businesses could forecast AI spending; a study finds lower-priced models cost more than higher-priced models on 32% of 6,800+ tasks (Wall Street Journal) (Visit Techmeme dot com for the link and full context!)
2
1
10
5,515
On our eval of 82 tasks from real @sentry PRs, GLM 5.3 was > 50% more expensive than claude opus 5.5 ($1.83 vs $1.22 per task) > 8.5 points less accurate (74.4% vs 82.9%)
Avyay Varadarajan
@avyvarOct 5
Open-weight models may be costing you more on real work. We built self-bench to turn real GitHub PRs into evals. Testing repos like Next.js, PostHog, and Sentry, open models were usually pricier + less capable. Find the best model for your repo: selfbench.dev
2
16
881
Per-token costs don't mean the model is actually cheaper for your work. In almost all of our tests, "cheaper" per-token models like GLM 5.3 and Kimi K3 were *more* expensive than models like GPT 6.1 Sol per task, due to token efficiency.
1
8
225
All of our work is open-source. We encourage you to try it out on your own repo, and let us know what you find! github.com/mupt-ai/self-benc…

GitHub - mupt-ai/self-bench: Benchmark coding agents and models on your own repository with private...

Benchmark coding agents and models on your own repository with private Harbor-formatted evals built from its merged pull requests. - mupt-ai/self-bench

github.com
7
145
Open-weight models may be costing you more on real work. We built self-bench to turn real GitHub PRs into evals. Testing repos like Next.js, PostHog, and Sentry, open models were usually pricier + less capable. Find the best model for your repo: selfbench.dev
11
10
3
58
11,540
With benchmaxxing, it’s become hard to trust public evals. When a new eval is released, model providers rush to saturate it. This doesn’t mean a model actually performs well in your actual work. Through iterative agent review, self-bench builds quality evals you can review.
2
9
461
Avyay Varadarajan retweeted
Nicolai Ouporov
@nicoupOct 2
One of the most high leverage ways academia can contribute to AI progress is through the development of high quality benchmarks. University labs getting priced out of AI evals is not a future we should be willing to accept as a field.
Anand Kannappan
@anandnk24Oct 2
Prediction: universities will get priced out of AI evals work We just spent $50k yesterday benchmarking 40 long horizon coding tasks As we design harder tasks for agents, trajectories are getting longer and need more tokens. That means substantially more inference and compute. How is this economically feasible for academics working off university grants?
7
6
94
6,298
Avyay Varadarajan retweeted
Paulina Xu
@paulinazhxuOct 1
Today, we're launching Fabriq Developer. Over 500 integrations for your AI app, added in one prompt. Slack, Gmail, HubSpot, Linear, GitHub, Salesforce... your users connect their own accounts, your agent acts through their connection with only the scopes they approved, and we handle the OAuth, tokens, refresh, revocation and audit log. No OAuth code on your side. Free to start.
16
6
28
4,413
it’s all goodhart’s law (when a measure becomes a target it ceases to be a good measure) we’re killing trust in every benchmark like russell westbrook killed trust in the triple double
Deedy
@deedydasSep 30
Replying to @deedydas
Most if not all model companies pay benchmark companies to test their model and multiple versions of their model in advance of launch. This is how benchmarking companies make money.
2
1
15
986
Avyay Varadarajan retweeted
Casey Tewey
@CTeweySep 15
Today we’re launching Keythorn. AI isn’t just thinking anymore. It’s moving money, talking to customers, and taking action. Every new agent brings new risk. The future runs on AI. We’re building the insurance for it. Meet @TryKeythorn ↓
74
30
12
199
31,404
Avyay Varadarajan retweeted
Toby Kim
@_doyeob_Sep 10
Today we're launching > the world's fastest TTS endpoint at 50 ms time-to-first-audio (TTFA) > the world's cheapest modern TTS endpoint at $5 / 1M characters Powered by Qwen3-TTS 1.7B on Nari Labs inference engine. 🚀 5x faster than Cartesia 💸 10x cheaper than ElevenLabs 🎙️ expressive voices, outperforming industry average 🎁 available FREE for a limited time We believe open-source will win: not just in LLMs, but also in multimodal AI. Nari Labs is here to accelerate that future, starting with speech. Try now at narilabs.com
46
48
4
606
43,903
Avyay Varadarajan retweeted
Paulina Xu
@paulinazhxuSep 8
A 14B open model on a single RTX 4090 matched our hosted frontier model on text-to-SQL. That wasn’t supposed to happen. We ran 28 configurations, graded 25,000+ answers, and benchmarked against Snowflake + Databricks. What we learned: model size, model reputation, and vendor benchmark scores are bad shortcuts for predicting performance on your database. So today we’re open-sourcing mnemiq. Test it on your own data ↓
11
17
5
117
15,225
Load more