Operational realities and lessons learnt post shipping AI features - The ability to switch models at the function or feature level—without restarting your software—should be baked in from day one. ( Also see openrouter ) - Model providers often tweak internal parameters on an ongoing basis that impact throughput, accuracy, latency, and error rate. - Occasionally, model providers and vendors offer joint discounts that can be leveraged simply by switching models thus saving real costs. - In scenario where your AI feature adoption is growing and your monthly billing quota wouldn't last, incorporate manual progressive degradation of models. ie. Switch to an older model that offers more tokens per dollar. This works as long as the error rate is acceptable to both the business and product teams. Better than switching off the feature. - Integrate evaluation set into automated unit tests, allows assessing new models and verifying new prompt or code changes by QA early in the process. - Teach QA how to build synthetic datasets for input subsets where error rates are high. This takes the load of devs who are doing everything. - Log and more imp incorporate meta tags in logs against all LLM failures and success. You can use this to optimise prompts via DSPy. - When launching brand-new features with no prior customer data / input to evaluate against, do a small % rollout and store the customer inputs, output and then use DSPy to optimise your prompt. This cycle will lead to better results next rollout. - Have tool calling for sensitive or customer profile data ( e.g. PII ). Thus only fetched when explicitly needed and masked. This will also give you an audit trail and you don't do dumb things like injecting it in context window every time. - Use tools like MLflow and educate your SREs or on-call developers on - The non-deterministic nature of LLMs, to prevent false alarms. SREs to distinguish between vendor / LLM side issue vs code. - SREs should know where prompts are stored in the repository. - Uncapped API quota is a disaster waiting to happen, Include a feature flag from day one to toggle entire AI features off asap if need be. - Beyond setting aggressive billing alerts, you’ll need a dashboard that provides a granular understanding of which prompts are called -> how many times -> their average cost. - max_tokens and parallel / batch inference are your friends. Use them.

Notes from completion of a small freelancing project that required near real-time processing of audio files for transcripts. Few hacks that help reduce cost and offer replacement to an expensive SAAS offering ankur.dev/2025/06/27/simple-…