Given the current foundation model paradigm, I wonder if building/training models will become antiquated. Will future data scientists understand the details of optimization, architectures, etc.? ML may slowly be abstracted in favor of simpler (language model-based) solutions... Some background. When I first got started in machine/deep learning, PyTorch did not yet exist. Implementing back propagation from scratch was relatively common (I was even asked to do this on the spot in an interview!) and doing any deep learning-related research required a significant amount of boilerplate programming. Today, most of the fundamental knowledge required for deep learning (e.g., back propagation, weight initialization, model architectures, etc.) is already abstracted significantly by packages like PyTorch and HuggingFace. Most of the time, we don't think much about these details, choosing instead to use default settings or implementations provided by our framework of choice. Training from scratch is (relatively) uncommon. For the first few years of my career, training machine learning models from scratch was standard. I implemented countless training pipelines for a variety of different models (e.g., regression, random forests, MLPs, RNNs, …). Since the proposal of BERT in 2018, I have used pre-trained transformer models (sometimes with added components based on pre-trained CNNs) for pretty much everything that I do. The variety of different model types that I’ve used throughout my years have been replaced by a few common transformer architectures. Plus, I don’t even implement these architectures myself! I simply download a pre-trained model, fine-tune it on whatever task I’m interested in, and deploy. I have not truly trained/created a model “from scratch” in quite a while. Should you implement training yourself? Even after learning to rely heavily upon pre-trained model architectures and weights, I continued to implement my own fine-tuning code or training loop when using these models. Recently, however, writing your own training loop has become less and less common. In fact, given the widespread availability of optimized training frameworks, implementing your own training code might even be a bad idea! Although PyTorch’s zero_grad(); backward(); step() cadence will be burned into my memory forever, it is now much more efficient to leverage the optimized Trainer classes offered by packages like PyTorch Lightning, MosaicML composer, or other alternatives. Everything keeps getting abstracted! Put simply, the research community has abstracted away a lot of complications associated with deep learning over time. Currently, model architectures, training loops (including all the details of optimization), pre-trained weights (and weight initializations), and more are all handled via a simple setting within a deep learning package. By examining this trend towards abstracting everything in the lens of the growing popularity of LLMs, we have to ask: where/when will this abstraction end? Some considerations. At this point, there are a few common points that most will make to argue against using language models for everything. E.g., inference is expensive, the models are large and difficult to host at scale, fine-tuned models still perform better, etc. Although these points are completely valid, I would also argue the following: - LLM inference speed is improving quickly due to advancements like FlashAttention, parallel transformer blocks, and low precision modules. - Hosting these models has been democratized significantly due to advancements in quantization (and this will probably continue to improve moving forward). - In practice, implementing any machine/deep learning system is a tradeoff between complexity and performance. In many cases, companies will prefer a model that can be implemented in 1 day over a slightly higher-performing model that takes several weeks/months to develop. The simple text-to-text interface of language models is (in my opinion) heavily underestimated. It can be leveraged by someone with minimal (or no) ML-related knowledge to quickly solve tasks that would’ve previously required entire teams of data scientists. Looking forward. With all this being said, I’m not sure how the day-to-day roles of machine learning engineers and data scientists will change moving forward. In my opinion, there is still incredible value in expert knowledge of machine learning fundamentals. However, I believe that the current generations of ML practitioners might find that they use this knowledge less and less over time, just as how my detailed knowledge of back propagation, weight initialization schemes, transformer architecture variants, etc. no longer serves an impactful/practical purpose in my current day-to-day life.