Tweenhaven

As an Inference Infrastructure Engineer, you must build these projects: Systems that prove you can serve, optimize & scale 1.) Self-Hosted Inference Server Build: Serve an open model with vLLM or SGLang behind an API with continuous batching enabled Why: You cannot optimize what you have never served 2.) TTFT/ITL Benchmark Suite Build: Load-testing harness measuring time-to-first-token, inter-token latency, throughput under rising concurrency Why: Latency claims without load curves are marketing not engineering 3.) KV Cache Memory Calculator and Monitor Build: Tool that predicts KV cache VRAM per model and config, then tracks live cache utilization Why: Most production OOMs are KV cache math errors not model size 4.) Prefix Caching Proxy Build: Gateway routing requests with shared system prompts to the same replica to reuse KV blocks Why: Reused cache is free speed: up to 80% TTFT cuts on multi-turn workloads 5.) Quantization Comparison Lab Build: Serve the same model in FP16, FP8, INT8, AWQ benchmark quality vs latency vs VRAM Why: Quantization decisions need measured tradeoffs not vibes 6.) Speculative Decoding Pipeline Build: Small draft model plus large target model with acceptance-rate tracking Why: Faster decode with zero quality loss is the closest free lunch in inference 7.) Triton Kernel from Scratch Build: Write and benchmark a fused kernel (softmax or RMSNorm) against the PyTorch baseline Why: Kernel intuition separates infra engineers from API callers 8.) Chunked Prefill Scheduler Experiment Build: Configure and measure chunked prefill under mixed prefill/decode load Why: Long-context requests starve decode without scheduling control 9.) PagedAttention Deep-Dive Report Build: Reproduce vLLM paging behavior under memory pressure document fragmentation and eviction Why: Understanding the scheduler beats memorizing flags 10.) Disaggregated Prefill/Decode Cluster Build: Split prefill and decode onto separate GPU pools; measure throughput and latency deltas Why: This is the architecture behind every frontier serving stack in 2026 11.) Queue-Based GPU Autoscaler Build: KEDA-style scaling on pending-request queue depth with cold-start mitigation Why: GPUs idling at 30% utilization burn money. Autoscaling is FinOps. 12.) Cost-per-Token Dashboard Build: Per-tenant, per-model token accounting with $/M tokens and MFU utilization tracking Why: Inference is a unit economics game. Whoever measures it wins. 13.) AI Gateway with Fallbacks and Rate Limits Build: Router across self-hosted and API providers with TTFT SLOs, retries, degradation chains Why: Provider outages are guaranteed. User-facing errors are optional. 14.) Chaos Suite for Inference Build: Inject GPU throttling, replica kills, traffic spikes measure SLO burn and recovery Why: Reliability is proven under failure not in demos. 15.) Public Benchmark Teardown Build: Published latency/throughput/cost curves for 3 serving configs with full methodology Why: Public, reproducible benchmarks are the strongest hiring signal in infra. Most people watch tutorials. Builders ship systems. Bookmark & Repost.