ML Researcher @YandexResearch | Tabular ML
F#1M/3146 SAE feature
Joined August 2016
- Tweets398
- Following2.1K
- Followers541
- Likes30.5K
Pinned Tweet
I've long argued for more diverse evaluation setups in tabular ML research. This will continue. It was a key motivation behind our new paper
A thread on data scaling in MLPs, GBDTs and tabular foundation models, and what it means for generalization.
arxiv.org/abs/2610.26794
Ivan Rubachev retweeted
it took me a while but I think I am coming around the the believe that finetuning (changing model's weights) is perhaps faster, cheaper and easier then prompt optimization. so many people are willing to run gepa and other prompt optimization, but for a reason I can't quite guess, much fewer are willing to finetune.
you can literally just say to Claude Opus 5-5:
i want a model that does X, find a paper about it, and some dataset. Make a better version using tinker from thinking machine. Start from inkling small model.
All you have to do is put credits on think machines service (if you have not given your credit card and a browser to you coding agent XD).
Ivan Rubachev retweeted
Of all the AI work that deserves to be called out as hype, weight-space programming should be near the bottom of the list. Yes some experiments had problems, but the idea is significant. It’s great that more people are thinking about it. Also what an incredibly clean RL env.
Ivan Rubachev retweeted
Like monkeys to humans: it has become fashionable for top AI thinkers to predict that AI will soon think beyond humanity’s grasp.
I’d like one example. Show me one unthinkable thought.
davidbau.com/archives/2026/1…
Ivan Rubachev retweeted
mini-swe-agent is 130 lines of code and it keeps on winning in harness comparisons.
I burned a shitload of compute so you don't have to. Releasing the Harness Leaderboard.
Which harness should you run your open model in? I ran 28,849 trials to find out.
The leaderboard compares nine coding-agent harnesses on open models. In each comparison only the harness changes: the model, provider and reasoning effort stay fixed, and every task gets three attempts.
The trials ran GLM-5.3-Flash, DeepSeek-V4.1-Flash and Flint on Terminal-Bench 2.1 and DeepSWE, for $4,746 of inference.
Averaged over the three models, mini-swe-agent (@KLieret), a bash-only agent, solves 51.4% of tasks and @openclaw 27.3%. Four others are statistically tied with mini-swe-agent.
One of them is Pi (@pidotdev), at 49.4% for $0.29 per solved task, about 45% less than mini-swe-agent's $0.53.
The best harness depends on the model: mini-swe-agent on GLM-5.3-Flash, Hermes Agent (@NousResearch) on DeepSeek-V4.1-Flash and Kimi Code (@Kimi_Moonshot) on Flint.
@opencode costs $3.35 per solved task, 11.7 times as much as Pi.
See Harness Hub's Harness Lens for what the nine harnesses do differently. It documents the prompts, tools and compaction of 34 coding-agent harnesses and links every claim to its source.
Leaderboard: huggingface.co/spaces/joelni…
Harness Lens: huggingface.co/spaces/harnes…
Ivan Rubachev retweeted
pretty cool that astra is capable of this - feels like the type of hail mary env you can just throw into rl to see how it does
We asked GPT-6 Astra to edit Qwen3.5-0.8B weights to improve math.
It directly changed only 0.016% of weights, fixing 0.8B's reasoning loops & improving math up to 6x and even instruction following 3x, without gradients or training.
We call this Weight-Space Programming: frontier LLMs can now natively read/write neural network weights! 🧵
Blog: rikiyat.github.io/blog/weigh…
Readers added context they thought people might want to know
LLM was given weights explicitly as text in prompt and used code/forward passes (per authors' logs) to ablate Qwen looping direction, a known thinking-mode issue per model card. Primarily fixed repetition in that mode, not general 'natively' improving weights/math.
rikiyat.github.io/blog/weight-sp…
github.com/RikiyaT/lm_und…
huggingface.co/Qwen/Qwen3.5-0…
huggingface.co/RikiyaT/Qwen3.…
Ivan Rubachev retweeted
i trained a 14M-parameter world model on 2D Minecraft in under 10 minutes for $0.1 and turned it into an autoresearch benchmark.
no game engine. the network draws every pixel. play it locally on your phone.
maharishiva.github.io/tinywm…
Ivan Rubachev retweeted
1/ ML is shifting towards “foundation models are all you need.”
These huge networks can do many things, but even fine-tuning now demands unreasonable amounts of GPU memory.
How can parameter-efficient fine-tuning (PEFT) help?
A thread 🪢
TabPack is great! now it has а package
btw, tabpack doesn't just give you good out-of-the-box performance on benchmarks, it also speeds up research. We had a lot of MLP models to run for our paper's scaling study:
teapaw.org/puhsuuu/status/2113696…
We wouldn't even have considered running a grid this big before tabpack
Ivan Rubachev retweeted
I propose a new track:
split data into 3 sets -- train, table and val.
an infinigram index is built on the 100B/1T tokens table set.
we evaluate val loss of the model trained on train set assisted by the infinigram.
I expect ROI of LUT methods will be lower on this track.
Ivan Rubachev retweeted
h-net 2? 👀
I've always believed that intelligence should come from simple, general building blocks that learn their own abstractions directly from raw data; in other words, the bitter lesson. I think we're finally seeing a real step towards AI that can learn from all data.
Language modeling on raw bytes is a microcosm of this ideal, and to me it's a central problem in AI: if we can't build LLMs that learn end-to-end from bytes, how will we handle the rest of the world's data that we understand even less?
Last year we introduced hierarchical networks (H-Nets) as a first step. Our new version finds up to 3-5x efficiency gains in controlled scaling law experiments, and at larger scale matches an open-source model trained on 11T tokens with ~10x less data and compute. The posts below walk through what we found and why we think it works.
Ivan Rubachev retweeted
We have proven a theorem in complexity theory establishing a deterministic space bound strictly lower than the 50-year-old Savitch bound.
Beyond pure theory, we used these proof constructions to build an inference engine that speeds up LLM generation on local models, especially on Apple Silicon.
We show that on any graph, one can validate whether two points are connected in the few-path regime without a prior promise. While a quantum computer can validate this in logarithmic space, we prove that a classical computer can solve it in polynomial time and sub-Savitch deterministic space (O(log^2 M / log log M)).
Ivan Rubachev retweeted
This Monday, Nov 2, Dmitry Baranchuk (@DmitryBaranchuk) will present his paper:
Representation-Space MMD for Diffusion Language Models
Project also co-led by Ilya Drobyshevskiy (@free001style) and Ilia Sudakov (@sudakov_il)!
Ivan Rubachev retweeted
it’s my nature to keep posts rather impersonal, so i have to work extra hard to tweet this sort of personal testimony below from time to time as i think the reminder in it around AI and the “value of human intelligence” is worth it
as of the end of Oct 2026, i have not (yet - you may insist) had the *slightest* moment of questioning the impact of AI on the meaning of any of my work/skills - or even feeling any unusual shift beyond having an amazing (and rather often infuriating) tool
believe it or not, but my ability to produce a completed artifact* (paper, model, open source software, lecture, …) and the effort, expertise, or time one takes to produce has actually barely reduced - despite comprehensive productivity benefits from AI
the reason is that i essentially NEVER choose to work for a pre-determined outcome or some field-wide well-specified problem; i only work on artifacts i aim to perfect according to *my* judgment and where my personalized ambition obviously has no set limit and isn’t somehow pre-loaded onto AI
better tools simply change the costs of my choices - i quite literally work on moving goal-posts if you will! when the fixed constraint are only time and energy, AI simply allows me to re-explore scope and make fewer (or different!!) tradeoffs or compromises, which feels almost no different!
i’m not saying every job or every endeavor - including very advanced and historically prized ones! - looks like this today, but i’m offering:
(1) at least one voice in this moment (of so many annoying loud FUD voices!) that it’s totally possible and i suspect extremely scalable and i predict will remain so for the foreseeable future!
(2) why “someone like me” is indeed choosing happily to be in academia and not elsewhere! (again lots of FUD out here!, but where else do you have the freedom to keep seeking to create value for deciding and solving ~arbitrary ambitious/useful problems, establish credit for it, and train superstars who do the same?)
my fear and worry in this moment is a lot less about Al or "the value of human intelligence" and much more about how concentration of power has a very reliable history of being very dangerous to large numbers of affected people! a VERY different problem imo
*i guess certain of those took way longer than i expected but do stay tuned for a half dozen things on the short horizon…
ICL-based tabular foundation models typically put validation examples in context. Under stronger shifts, this does help TabPFN v3.5, but increasing training data size leads to "inverse scaling".
Surprising to see and I think could help improve future TFMs
8/N
9/N=9
Curious to hear your thoughts and discuss!
cc @DHolzmueller @LeoGrint @LennartPurucker @innixma @nour_shaheen_