Why is self-supervision in vision still not working? 🤔 When pretraining a transformer on TEXT-only data by predicting the next tokens, we see clear improvement trends as we scale the model, data, and computing. But after trying to pretrain a transformer on IMAGES-only data with contrastive learning (e.g., SimCLR), masked prediction (e.g., MAE), and self-distillation (e.g., Dino), at the end of the day, the most successful vision encoders still have to rely on vision-language paired data (e.g., CLIP. SigLIP). Or have people not yet tried connecting MAE/Dino with LLMs? Why is that? Thoughts?