deep learning and big computers
local minima (chicago)
Joined July 2023
- Tweets3.4K
- Following2K
- Followers387
- Likes138K
Pinned Tweet
Even the sparsest of attention mechanisms can get good flop efficiency if the MQA query group size is big enough
I'm pretty confident that the scale of compute required to do this could not be constructed using all of the energy and material in the solar system.
If you had an exponential amount of compute you could solve simulate biology exactly and wouldn't need to run any experiments.
Some of you will think "of course, but that's an unrealistic assumption" and others will think "no, you need experiments even with that much compute. That's just How It's Done".
You would of course need evaluations once you have a solution, but you don't in principle need experiments to find it: Experiments are a way to save compute, dating from a time when it was much more expensive.
And I think the presentation is exponential but real systems aren't irreducibly exponential. AlphaFold worked, after all.
biased estimator retweeted
can i say something without everyone getting mad... i often see some mechanistic interpretability experiments here and people getting excited about them without really understanding what's going on and sometimes it feels like mech int is like astrology for ai people lol
biased estimator retweeted
"we did these 23 unnecessary steps on top of the baseline which DON'T improve anything, here's our ICLR submission". genuinely disappointing times to be a reviewer
biased estimator retweeted
my iclr 27 papers to review are some of the strongest evidence against rsi I've seen to date. just the most dreadful claudeslop.
biased estimator retweeted
the Lord came down to see Adam's neural networks. in their greed, they had embedded more features than residual dims.
and the Lord spake, "let us confuse their activations, lest nothing they plan be impossible." thus the Lord scattered the features over the residual stream
Seeing this article while on the red line with a shirtless man punching the window, wishing someone could come make chicago "eerily safe"
biased estimator retweeted
If you see someone on here using a word that ends with -oid, you can safely disregard whatever they're saying. For instance, "A monad is a monoid in the category of endofunctors"
biased estimator retweeted
Bro it’s called a Roth IRA, it’s so sick. You just live in poverty for 50 years, then one day you wake up and your dick doesn’t work, but you get to open your phone and see a big number, and the best part? It’s all tax-free.
biased estimator retweeted
Interpretability blogs are the recipes blogs of ML. Not sure, why I have to read through the story of your grandmother's first house to find out you used gradients.
biased estimator retweeted
Ten years of Sad Affleck memes and it turns out he'd just been trying to install CUDA drivers the whole time.
biased estimator retweeted
The actual cure for student loans is simply to make the schools hold the loan.
If the school is on the hook for the loan, it has a real interest in a) charging a price that reflects the future earnings of students and b) taking reasonable measures to ensure the students succeed
biased estimator retweeted
please stop saying your calculator “adds” numbers, it only simulates addition
biased estimator retweeted
Under a matched compute budget, Looped Transformers consistently underperform !!!
Base > Looped
MoE > Looped-MoE
Valuable compute and research effort are better spent on approaches that deliver clear gains.
Paper: arxiv.org/pdf/2605.09165
Looped Transformerは、FFNがdenseのままだと通常のTransformerより性能が落ちる。しかしFFNをMoEにすると、ループごとに異なるエキスパートを選べるため、弱点のかなりの部分が解消される。
統制実験では、通常のTransformer（Base）、2回ループさせたTransformer（Looped）、通常のMoE（MoE）、2回ループさせたMoE（Looped-MoE）の4つを、計算量を揃えて比較している。その結果、性能は
MoE > Looped-MoE > Base > Looped
の順になった。
同じトークンが同じ物理層を1回目と2回目のループで通過する際、両方で同一のエキスパート集合が選ばれる割合はわずか4〜14%であり、多くの場合、一部または全部が別のエキスパートに切り替わっていた。
つまり、重みは共有していても、ループごとに異なるサブネットワークを使うことで、重み共有による表現力の低下を補っていると考えられる。
ただし、この実験設定で最も性能が高いのは通常のMoEであり、この規模の実験でのLooped-MoEの主な利点は、保存すべきパラメータ数を抑えながら、通常のTransformerを上回る性能を保てる点にある。
また、Looped Transformerでは各ループの終端が自然なEarly Exitの候補になる。途中のループで十分に確信度の高い出力が得られれば、残りのループを省略できる。このEarly Exitは、通常のTransformerやMoEで途中の層から抜ける場合と比較して、同じ計算量削減率でも性能劣化が大幅に小さかった。