Blending Is All You Need Based on the last month of LLM research papers, it's obvious to me that we are on the verge of seeing some incredible innovation around small language models. Llama 7B and Mistral 7B made it clear to me that we can get more out of these small language models on tasks like coding and common sense reasoning. Phi-2 (2.7B) made it even more clear that you can push these smaller models further with curated high-quality data. What's next? More curated and synthetic data? Innovation around Mixture of Experts and improved architectures? Combining models? Better post-training approaches? Better prompt engineering techniques? Better model augmentation? I mean, there is just a ton to explore here as demonstrated in this new paper that integrates models of moderate size (6B/13B) which can compete or surpass ChatGPT performance. arxiv.org/abs/2401.02994