Jason Adhinarta retweeted
Does self-distillation work because the teacher knows the answer?
The teacher receives privileged information the student cannot see, such as reference solutions or external feedback. The student learns by matching the teacher’s predictions.
But our comparison of correct, absent, and unrelated reference solutions in OPSD (arxiv.org/abs/2601.18734) shows that the correct reference isn’t necessary for improvement.
If the reference isn’t necessary, what is doing the teaching? 🧵 1/6
Policy gradient RL, self-distillation, and Pedagogical RL all optimize the same objective E[R] − β·KL(π‖π₀) under the hood. They differ only in how privileged info flows and which direction they minimize KL. The punchline: at optimality, a good teacher's use of privileged info vanishes. Good teachers don't cheat. By not training the teacher, most on-policy distillation methods cheat!
See jasonkena.github.io/blog/pos…
grid_sample requires input to be specified in the NCDHW (NCzyx) format but expects grid coordinates to be specified using WHD (xyz) coords
Jason Adhinarta retweeted
Kaiming He, inventor of ResNet, is leaving industry to join MIT faculty in 2024!! He’s one of the most impactful figures in deep learning.
- Residual layer is a fundamental building block of LLMs.
- Faster/Mask R-CNN are industrial standards for image segmentation and robot perception stack.
- Panoptic segmentation redefined a research sub-field in vision.
- Mask AutoEncoder (MAE) is among the best general-purpose self-supervised algorithms for computer vision and beyond.
- Before MAE, Momentum Contrast (MoCo) was a SOTA contrastive learning technique.
- SlowFast network was among the default backbones for video learning until ViTs took over.
- Too many other groundbreaking works to enumerate …
I recently observe an exodus of researchers from big techs to academia. It’s an interesting movement given the current LLM gold rush 🤔
Normally we congratulate someone who joins MIT, but this time I congratulate @MITEECS to have Kaiming! 🎉
issue is being tracked in github.com/numpy/numpy/issue…
