Mohammad Rastegari (@morastegari) profile picture

Distinguished AI Scientist at Meta. Affiliate Assistant Professor at University of Washington.

Seattle, WA
Joined February 2017
Photo by Mohammad Rastegari (@morastegari)Photo by Mohammad Rastegari (@morastegari)Photo by Mohammad Rastegari (@morastegari)Photo by Mohammad Rastegari (@morastegari)Photo by Mohammad Rastegari (@morastegari)Photo by Mohammad Rastegari (@morastegari)Photo by Mohammad Rastegari (@morastegari)Photo by Mohammad Rastegari (@morastegari)
Mohammad Rastegari retweeted
Like OpenELM, CatLIP is also "Open" github.com/apple/corenet
Apple presents CatLIP CLIP-level Visual Recognition Accuracy with 2.7x Faster Pre-training on Web-scale Image-Text Data Contrastive learning has emerged as a transformative method for learning effective visual representations through the alignment of image and text
6
15
2,620
This work was one of the last works that was done by my team when I was working at Apple. A lot of credit to @sacmehtauw whose dedication was the key to this project. Main point behind here is to show as a contributor to the AI community we play our role to be fully open.
Apple presents OpenELM An Efficient Language Model Family with Open-source Training and Inference Framework The reproducibility and transparency of large language models are crucial for advancing open research, ensuring the trustworthiness of results, and
7
63
25,368
Another great research effort on efficiency of LLMs from our group in #apple
Apple presents Speculative Streaming Fast LLM Inference without Auxiliary Models Speculative decoding is a prominent technique to speed up the inference of a large target language model based on predictions of an auxiliary draft model. While effective, in application-specific settings, it often involves fine-tuning both draft and target models to achieve high acceptance rates. As the number of downstream tasks grows, these draft models add significant complexity to inference systems. We propose Speculative Streaming, a single-model speculative decoding method that fuses drafting into the target model by changing the fine-tuning objective from next token prediction to future n-gram prediction. Speculative Streaming speeds up decoding by 1.8 - 3.1X in a diverse set of tasks, such as Summarization, Structured Queries, and Meaning Representation, without sacrificing generation quality. Additionally, Speculative Streaming is parameter-efficient. It achieves on-par/higher speed-ups than Medusa-style architectures while using ~10000X fewer extra parameters, making it well-suited for resource-constrained devices.
11
1,298
Mohammad Rastegari retweeted
Apple presents Speculative Streaming Fast LLM Inference without Auxiliary Models Speculative decoding is a prominent technique to speed up the inference of a large target language model based on predictions of an auxiliary draft model. While effective, in application-specific settings, it often involves fine-tuning both draft and target models to achieve high acceptance rates. As the number of downstream tasks grows, these draft models add significant complexity to inference systems. We propose Speculative Streaming, a single-model speculative decoding method that fuses drafting into the target model by changing the fine-tuning objective from next token prediction to future n-gram prediction. Speculative Streaming speeds up decoding by 1.8 - 3.1X in a diverse set of tasks, such as Summarization, Structured Queries, and Meaning Representation, without sacrificing generation quality. Additionally, Speculative Streaming is parameter-efficient. It achieves on-par/higher speed-ups than Medusa-style architectures while using ~10000X fewer extra parameters, making it well-suited for resource-constrained devices.
4
68
11
498
72,142
Glad to see a great attention to our research in the Apple MIND team.
LLM in a Flash by Apple is the most upvoted paper on Hugging Face huggingface.co/spaces/hysts/…
9
1,659
This has been one of my favorite directions on enabling #llms to run effectively on device. Thanks to the great team who are pushing state-of-the-art in this direction. In the Apple MIND team, we try to attack research problems that move us to the next level of experiencing AI.
Apple announces LLM in a flash: Efficient Large Language Model Inference with Limited Memory paper page: huggingface.co/papers/2312.1… Large language models (LLMs) are central to modern natural language processing, delivering exceptional performance in various tasks. However, their intensive computational and memory requirements present challenges, especially for devices with limited DRAM capacity. This paper tackles the challenge of efficiently running LLMs that exceed the available DRAM capacity by storing the model parameters on flash memory but bringing them on demand to DRAM. Our method involves constructing an inference cost model that harmonizes with the flash memory behavior, guiding us to optimize in two critical areas: reducing the volume of data transferred from flash and reading data in larger, more contiguous chunks. Within this flash memory-informed framework, we introduce two principal techniques. First, "windowing'" strategically reduces data transfer by reusing previously activated neurons, and second, "row-column bundling", tailored to the sequential data access strengths of flash memory, increases the size of data chunks read from flash memory. These methods collectively enable running models up to twice the size of the available DRAM, with a 4-5x and 20-25x increase in inference speed compared to naive loading approaches in CPU and GPU, respectively. Our integration of sparsity awareness, context-adaptive loading, and a hardware-oriented design paves the way for effective inference of LLMs on devices with limited memory.
14
1,255
Accurate training aware weight quantization was computationally intractable for LLMs. But now in Apple MIND we developed a method to solve the problem very efficiently and it pushes the boundary to 3-bit quantization. eDKM: arxiv.org/abs/2309.00964 #LLM, #LLMoptimizaton
1
8
1
29
4,336
Mohammad Rastegari retweeted
“Reinforce Data, Multiply Impact: Improved Model Accuracy and Robustness with Dataset Reinforcement” is an #iccv2023 paper from #Apple. By just swapping the ImageNet dataset with the “reinforced” ImageNet+ dataset, a model can be trained up to 7x faster to reach the same accuracy
Excited that our "Dataset Reinforcement" paper introducing "ImageNet+ dataset" is accepted to #ICCV2023! Up to ~7x FASTER training! Paper: arxiv.org/abs/2303.08983 ImageNet+/Code: Coming Soon w/ @HPouransari @sacmehtauw @MFarajtabar @morastegari @OncelTuzel Ali Farhadi 1/7
2
14
13,868
Mohammad Rastegari retweeted
Introducing NeuMan, a NeRF representation of human together with the scene. From a single clip (<100 frames), NeuMan can perform view synthesis of the scene without/with the human in novel poses. (1/4) project page: machinelearning.apple.com/re…… code: github.com/apple/ml-neuman
35
314
73
1,684
Mohammad Rastegari retweeted
NeuMan is a new #ECCV2022 paper from our research team @Apple. Using a short (~10s) clip, we reconstruct human and scene radiance fields, and re-render with novel human poses and views. Paper/code/videos: machinelearning.apple.com/re… w. @jwei221, G. Samei, @kwangmoo_yi, @anuragranj
4
26
Mohammad Rastegari retweeted
MobileOne code is released.
Introducing the fastest neural network on mobile so far - MobileOne, an improved one millisecond mobile backbone. Latency range: 0.79 ms - 1.86 ms. ImageNet top1 accuracy range: 71.4 % - 79.4 %. arxiv: arxiv.org/abs/2206.04040 code and models: github.com/apple/ml-mobileon…
1
7
A Transformer that can be as efficient as a CNN but yet maintains the high performance in the large data regime training. #EfficientTransformer, #AppleMI
(1/N) New paper! Separable self-attention for mobile vision transformers. Improves the speed and accuracy of transformer-based mobile vision network, MobileViT, significantly across tasks and input sizes. Paper: arxiv.org/abs/2206.02680
2
19
These CVPR policies are frustrating. Given all the randomness in the review process I feel there is no point submitting paper into conferences anymore. By the law of large data (large number of papers and readers) just submitting to arXiv will be enough for a good paper to shine.
1/2 @CVPR so if somebody posts wrong facts about a paper I co-authored, I may not correct them in an answer in any way? i.e. you prefer the spread of falsehoods? What if someone hates me and creates anon account, acting like me and answering threads about my submission?
3
3
21
Great progress in self-supervised training. Next milestone matching supervised SOAT Imagenet1k with self-supervised without end-to-end fine tuning
1/N The return of patch-based self-supervision! It never worked well and you had to bend over backwards with ResNets (I tried). Now with ViT, very simple patch-based self-supervised pre-training rocks! First BeIT, now Masked AutoEncoders i1k=87.8% arxiv.org/pdf/2111.06377.pdf 🧶
1
5
Deploying ML models on device cannot be static. ML models should adapt themselves to available resource. In our recent research in #Apple, we learn ML models that can dynamically be compressed to any arbitrary sparsity or quantized level at inference time.
Excited to share our new method, "Learning Compressible Subspaces (LCS)", for adaptive model compression in real-time: (arxiv.org/abs/2110.04252).
34
1
173
Here is our work at Apple on building a state-of-the-art parameter efficient visual transformer. @apple_ml_research
New paper! We introduce MobileViT, a light-weight vision transformer. MobileViT delivers better performance across different tasks, shows better generalization capabilities, and are robust to hyper-parameters (e.g., basic augmentation). (1/5) Paper: arxiv.org/abs/2110.02178
10
79
Mohammad Rastegari retweeted
DKM: Differentiable K-Means Clustering Layer for Neural Network Compression deepai.org/publication/dkm-d… by Minsik Cho et al. including @morastegari #KMeans #NaturalLanguageProcessing
6
12