seen/heard >5 reactions to deepseek4 "data is all that matters" claim with "welp research is over" but that is a terrible take during the old days (2016-21ish), if you were hillclimbing - in most cases you had a few knobs and levers for improving the model: (1) some architecture decisions, (2) some loss fn/learning signal differences, (3) some training algorithm differences. you really had to be creative to turn intuition into inductive biases which meaningfully affected the outcome (or come up with posthoc explanations for why your human-guided neural architecture search worked) now, the data is your lever for improving the model and the behavior. each token is a knob, and you can scale to thousands or even trillions. each reward function. if you want to influence the model behavior through inductive biases - there are levers everywhere for those with eyes to see really data has always been the key lever for model improvement and progress in the field (eg imagenet did way more for vision than rcnn) but it was a lever that was impossible to move (pay mturkers to annotate the right data) and took time money and energy now, for the same cost - you can apply an army of agents and spend millions of tokens of compute to calculate the optimal reward signal for each token of training. what is stopping you from improving performance except better research ideas and compute/time (which can be reduced with better research ideas) the surface area for research has only increased, eg the architecture of a data factory is so much more interesting / offers more opportunities for research than a model architecture. the age of research is only beginning

LLMs must be factually accurate. Especially to power autonomous agents for complex, long-horizon tasks. But how do we actually do this in practice? 🧵[1/16] Learn about groundedness, why LLMs hallucinate, and how ContextualAI built the world's most grounded LLM!