Sam Stevens

An underrated fact about production ML is that LLMs mostly reach the highest-volume decisions indirectly, through offline labels, features, and caches. Most ranking, recommendations currently mostly run on gradient boosting, small classifiers, and rerankers, with LLMs on the side as precomputed features, cached rewrites, or prefill-only scorers. Generating a response while the user waits has simply been too slow for a tens-of-milliseconds budget unless users consume the outputs directly. This will change soon: Most of open inference stacks has been built for a very different regime, and for good reason: for many enterprises, models that can solve open-ended problems agentically have been the fastest route to value. But what about everything else? There, large foundation models often fall short. They’re too slow and too expensive to sit in the critical path. So historically, we’ve often used precomputed features and offline caches, which come with their own problems (Staleness, cache misses, pipeline explosion). Briefly, here’s what’s in this space today (largely ignoring offline approaches): - Small (<10B) prefill-only scoring LLMs - Rerankers - LightGBM / other gradient-boosted tree methods - Small classifiers (e.g., Jev and other decision models, but these have limitations too) These models are all fast, but with limited expressivity, i.e., categorical scores/labels can’t always produce the best answer for every workload. The benefit of high-concurrency, low-latency decode is the additional information full-sequence decode can provide (e.g., a rewritten query or structured filters instead of a single score). This also makes post-training more straightforward, since the output is just text and the standard SFT/RL recipes apply directly easier, and since the model is small, it’s also efficient to post-train. And these models can now be used in more places (e.g., live query rewriting might currently only be used for queries that aren’t precomputed/cached), similar to Amazon search (which mostly takes advantage of DynamoDB, i.e a very offline approach)! We might expect less and less justification to accept lower quality results from not using personalized LLMs; and even though small models have their own deployment challenges (different from agentic ones), you can use them everywhere.