ex math+cs @mit_csail | ex-ex ipho gold
🇸🇬
Joined March 2024
- Tweets60
- Following426
- Followers764
- Likes402
Pinned Tweet
Do we really need billion-parameter transformers for time series forecasting?
What if we did it in under 3M parameters?
Introducing Reverso: time series foundation models as small as 200K paramters significantly pushing the Pareto frontier in zero-shot time series forecasting!
Small model, looooong context
Reverso is now improved to handle significantly longer contexts, up to 16k! We pretrain our foundation model to handle inputs at multiple frequency scales, and observe an improvement in forecasting performance while maintaining fewer than 3M params
Reverso also zero-shot transfers to a wide range of tasks. The encoder-decoder architecture of Reverso allows us to adapt its encoder backbone and directly use it on classification, imputation, anomaly detection and probabilistic forecasting, with little to no retraining
Reverso is so small that it even runs locally in your browser: check out this interactive blog for a live demo and read our paper for more details!
demo: shinfxh.github.io/reverso-bl…
arxiv: arxiv.org/abs/2602.17634
github: github.com/shinfxh/reverso
hf: huggingface.co/shinfxh/rever…
Xinghong (Shin) Fu retweeted
New paper 🧵
We show that dynamic short convolutions consistently improve Transformers across scales. We make these gains practical with an efficient parameterization and custom Triton GPU kernels.
The improvements carry over to MoEs and linear attention variants (Mamba-2/GDN).
Xinghong (Shin) Fu retweeted
What if the best visual reasoning steps are ones humans can’t specify? 🤔
Existing VLM reasoning is often constrained by language, pixels, and human-designed intermediates.
We introduce Latent Implicit Visual Reasoning, where we show that VLMs can discover the best visual reasoning steps by themselves — no bboxes, no intermediate images, no extra supervision.
Presenting this week at CVPR!
(1/n)🧵
Xinghong (Shin) Fu retweeted
The biggest reduction in KV cache memory comes not from quantization or MLA, but from latent compaction, along the sequence dimension.
More strong results coming soon with Attention Matching.
Xinghong (Shin) Fu retweeted
Models performed well without, and a little better with, the images. In one case, our no-image model outperformed ALL of the current models on the chest x-ray benchmark—including the private dataset—ranking at the top of the leaderboard. Without looking at a single image.
🤯🤯
Been catching up on some of dwarkesh's podcasts over spring break with the one with terence was quite thought provoking.
When we have millions of AI scientists "trying random things" on the increasingly harder and newer scientific problems, will they converge to the same theory?
Xinghong (Shin) Fu retweeted
Simply adding Gaussian noise to LLMs (one step—no iterations, no learning rate, no gradients) and ensembling them can achieve performance comparable to or even better than standard GRPO/PPO on math reasoning, coding, writing, and chemistry tasks. We call this algorithm RandOpt.
To verify that this is not limited to specific models, we tested it on Qwen, Llama, OLMo3, and VLMs.
What's behind this? We find that in the Gaussian search neighborhood around pretrained LLMs, diverse task experts are densely distributed — a regime we term Neural Thickets.
Paper: arxiv.org/pdf/2603.12228
Code: github.com/sunrainyg/RandOpt
Website: thickets.mit.edu
Learn more about a very efficient and high quality time series model released recently at
xitter.cf/twitter/status/2024856…
@JeffDean
Learn more about a very high quality time series model released by @GoogleResearch a while back at research.google/blog/a-decod…
imagine doing this zero-shot with fewer than 1M params
