hash aggregate retweeted
Let's all agree that this is the correct and final eval for agi
jackhopkins.github.io/factor…
hash aggregate retweeted
2-line proof that LLMs can’t achieve superintelligence:
1. LLMs don’t learn out of distribution.
2. Superintelligence is out of distribution.
hash aggregate retweeted
i'm comically impressed that people are coping on deepseek by spewing bizarre conspiracy theories -- despite deepseek open-sourcing and writing some of the most detail oriented papers ever.
read. replicate. compete.
don't be salty, just makes you look incompetent.
hash aggregate retweeted
If work you're proud of has been rejected from an important conference, just remember that it's a momentary blip and that the true judgement of impact will be whether others benefit from and build on your work.
(In addition to @chelseabfinn's example below, I've had similar experiences: for example, @geoffreyhinton, @OriolVinyalsML and my paper on distillation was rejected from NeurIPS in 2015, so we published it in a workshop and put it on Arxiv, and it is now heavily cited and used quite often in practice).
Disappointed with your ICLR paper being rejected?
Ten years ago today, Sergey and I finished training some of the first end-to-end neutral nets for robot control 🤖
We submitted the paper to RSS on January 23, 2015.
It was rejected for being "incremental" and "unlikely to have much impact"
Our resubmission to NeurIPS was also rejected
It now has >4,000 citations (and more importantly, end-to-end training is widely accepted!)
It's also cool to think about what's changed and what's the same --
- The network was 92k parameters and trained on ~15 minutes of data
- The code was a combination of matlab, caffe, ROS, a custom CUDA kernel for speed, and a low-level 20 Hz controller in C++, all talking to each other. ROS+matlab was as bad as it sounds.
- We pre-trained the encoder and did inference off-board on a workstation with a larger GPU.
- We were paranoid about varying lighting messing up the network, so we did all the experiments after sunset (so long nights running experiments on the robot past 3 am)
Now, we have manipulation policies that are far more dextrous, far more generalizable, and maybe on the cusp of breaking into the real world. :)
(the paper: arxiv.org/abs/1504.00702)
hash aggregate retweeted
Replying to @elonmusk
If you would like people to have more children, you can use your money to reduce child deaths.
This year 5.9 million children died. 16,000 every day.
We know how to save the lives of many of them. What we are lacking is money:
ourworldindata.org/cost-effe…
hash aggregate retweeted
My favorite thing about AI going mainstream is that it sneakily serves as a litmus test for supposed wisdom. Take your favorite philosopher, business expert, sociologist etc who has turned into a “guru” and Google what they say about AI. If you are hands on with AI, you’ll discover that many of these experts are not only wrong about how AI works, but very confidently wrong in a cringe way (I’m not talking about doomer vs e/acc debates, but more mundane things).
It makes you think: maybe you shouldn’t heed their advice in their area of expertise as well? Alas, the best advice is often not to listen to any advice at all.
P.S. the best people who I enjoy listening to have enough humility to say “I don’t know”.
hash aggregate retweeted
Me: coding all the time is dumb, real breakthrough needs deep thinking and abstraction
Also me: we have no clue what's going on, anything fancy we come up with by thinking ends up destroyed by more compute. Coding small changes and testing experimentally is the only way
hash aggregate retweeted
9 years ago I was interning under Ilya at Google Brain. First thing he told me is Solomonoff Induction and AIXI, i.e. why prediction leads to understanding. GPT came from sentiment neurons. The level of clarity he and Geoff had was always inspiring.
hash aggregate retweeted
You don't have to be special to be successful.
Ordinary people can do extraordinary things by choosing to be what most people are unwilling to be: consistent, hardworking, patient, and determined.
Simple, but not easy.
hash aggregate retweeted
Objectively, this would seem an excellent opportunity to release the last of the JFK files from Dallas from over 60 years ago.
Build confidence. Stop speculation.
hash aggregate retweeted
ZAZ the GOAT has dropped yet another banger video.
I'm already 80% through the video and love it. If you're not dropping whatever you're doing to watch this right now, you're falling behind.
(seriously though, love his work, recommend watching)
Just uploaded a 1-hr exclusive video for Part 2.1, with many technical details. youtu.be/bpp6Dz8N2zY. Part 2.2 will be online in about a week.
hash aggregate retweeted
next paper: transformers can solve any problem but on some of them they may compute indefinitely and never provide an answer
What is the performance limit when scaling LLM inference? Sky's the limit.
We have mathematically proven that transformers can solve any problem, provided they are allowed to generate as many intermediate reasoning tokens as needed. Remarkably, constant depth is sufficient.
arxiv.org/abs/2402.12875 (ICLR 2024)
hash aggregate retweeted
The biggest problem with o1-* models is that the chain of thought spam deranges dense context. The context is filled up with noise i.e. about "code coverage" in a query which has nothing to do with coding. In any complex/ambiguous problem, it will misunderstand you on the first shot, you then explain what you actually wanted and by step 2 the CoT noise deranges the model significantly over a standard LLM. Catch-22. They should prune away the noise on subsequent queries. Perhaps as the models learn (from the users), the trajectories will become less noisy. It's also clear that the trajectories are hard-trained for specific purposes which will limit their generalisation.
hash aggregate retweeted
things you definitely do NOT need to understand to be an expert on LLMs:
- linear regression
- bias variance trade off
- most probability distributions (Gaussian Bernoulli poisson etc.)
- RNNs
- LSTMs
- CNNs
- higher-order calculus (beyond first derivatives + chain rule)
- tensorflow
- EM
- kmeans
- boosting & bagging
- decision trees
- random forests
- graph neural networks
- naive bayes
- reinforcement learning (except RLHF, maybe)
- SVM
- Gaussian processes




























