ML Engineer
London, United Kingdom
Joined October 2008
- Tweets33.3K
- Following1K
- Followers8.3K
- Likes23.5K
Pinned Tweet
I post the papers I find interesting. There are so many papers published these days, and I frequently miss great papers. I appreciate paper recommendations via DM, but I tend to only post papers I discover on my own to keep my list personally curated.
arxiv.org/abs/2610.32057
Optimization for the indexer in sparse attention. Better sampled top-K selection, and saving only the selected indices (and their corresponding scores).
arxiv.org/abs/2610.31867
Compute optimal proportion of synthetic rewritten data increases with pretraining compute.
arxiv.org/abs/2610.31642
Fixing numerical precision problem in softmax.
arxiv.org/abs/2610.29352
Scaling law for long-context pretraining with respect to token budget for different context lengths. (Also, model merging alternative for LR decay.) I think it would be better to train under long context as much as possible if you have enough long context documents. (arxiv.org/abs/2604.02650)
한국 이민자들을 생각해서 역지사지하라고 하면 누가 가라고 했나? 라는 답이 돌아오겠죠. 이민 이야기가 자주 나오는데 진보에서도 논의의 깊이가 선언적인 권리 혹은 심지어 "경제적" 가치에 머물고 있다는 건 문제.
And the innovation is not incompatible with compute and data. Creativity could come from restriction, but at the same time resources leads to innovation and taste.
arxiv.org/abs/2610.19443
Pretraining with same text with or without images. It is hard to find out text tasks that multimodal training is beneficial, but spatial reasoning could be one of them.
arxiv.org/abs/2610.17590
If we make adequate compromises toward an autoregressive model, then diffusion could become more compute efficient.
arxiv.org/abs/2610.15965
Dealing with data repetition, one of the most interesting issues in pretraining. For this case maybe old deep learning wisdom like small batch sizes could be useful.
arxiv.org/abs/2610.13321
In-house crawling is a must for serious LLM building attempts. How to crawl pages that are potentially useful for pretraining is therefore an important topic.
arxiv.org/abs/2610.13630
Multi-head attention residuals. Uses multi-scale depthwise conv to expand dimensions.
Breaking of shift equivariance due to strided sampling. It would be familiar to many people.
arxiv.org/abs/2610.10478
Measuring agentic performance before post-training. This could be an interesting topic, if we can find out the relationship beyond incorporating more agentic trajectories.
Rosinality retweeted
Completely agree with this perspective. I've been meaning to write a paper about the relationship between training steps and the optimizer.
From the results in arxiv.org/pdf/2505.23725, which were not the main focus of the paper, it became clear to me that the critical batch size corresponds to the minimum number of steps a particular optimizer can tolerate to reach strong performance (e.g., within ~1% of the optimal loss).
For example, arxiv.org/abs/2507.07101 concludes that small batch sizes are needed for training with SGD effectively, but I would posit that this is instead a very large step requirement.
arxiv.org/abs/2610.08975
Practically, how well the optimizers utilize large batches is one of their most important characteristics.
Rosinality retweeted
Why do neural models fail to generalize to longer sequences?
Not just an architectural problem. We find that transformers can lose extrapolation ability as training progresses. Weight decay accelerates this degradation, while properly placed dropout improves extrapolation.
1/
arxiv.org/abs/2610.04906
Tencent is one of few companies that keep publishing papers on environment synthesis.
