Pinned Tweet
Thread of things I've built over the last few years.
Most of them started because I wanted to understand how something actually worked instead of treating it as a black box.
I'll keep adding to this thread whenever I finish something new.
1
2
355
Abhishek retweeted
cold post, but tbh i'd rather say what i want. any backend role works, and llm inference or ai engineer is the best fit. i'm happy to learn new things as long as it isn't frontend, i just don't have the taste for it. dms are open, see bio for my portfolio website link.
cold post, but tbh i'd rather say what i want. any backend role works, and llm inference or ai engineer is the best fit. i'm happy to learn new things as long as it isn't frontend, i just don't have the taste for it. dms are open, see bio for my portfolio website link.
Abhishek retweeted
finally got my first self written harness working. the first tool call works!
spent way too much time debugging, but honestly, writing code from scratch and seeing things work is one of the best parts of learning something new.
finally got my first self written harness working. the first tool call works!
spent way too much time debugging, but honestly, writing code from scratch and seeing things work is one of the best parts of learning something new.
Abhishek retweeted
hehe, need to start writing my own harness. no ai writing the code this time. i want to learn by building it myself.
lfg!!
was learning how to build an agent harness and came across this. fuck, the way they explain it is so simple. it starts with just a loop. must read.
ampcode.com/notes/how-to-bui…
hehe, need to start writing my own harness. no ai writing the code this time. i want to learn by building it myself.
lfg!!
was learning how to build an agent harness and came across this. fuck, the way they explain it is so simple. it starts with just a loop. must read.
ampcode.com/notes/how-to-bui…
was learning how to build an agent harness and came across this. fuck, the way they explain it is so simple. it starts with just a loop. must read.
ampcode.com/notes/how-to-bui…
Abhishek retweeted
continuing from my last update, we've gotten deeper into splitting GPU workloads between prefill and decode.
i'm exploring how KV cache blocks move across nodes using RDMA and load into decode GPUs.
how are people handling this in prod? would love to learn from real setups.
continuing from my last update, we've gotten deeper into splitting GPU workloads between prefill and decode.
i'm exploring how KV cache blocks move across nodes using RDMA and load into decode GPUs.
how are people handling this in prod? would love to learn from real setups.
Abhishek retweeted
continuing my last post on distributed llm inference.
i'm exploring how to split gpu workloads between prefill and decode instead of running both on every gpu.
the tricky part is coordinating the workers and transferring kv cache through nixl.
continuing my last post on distributed llm inference.
i'm exploring how to split gpu workloads between prefill and decode instead of running both on every gpu.
the tricky part is coordinating the workers and transferring kv cache through nixl.
i'm trying to get better at distributed inference hld, and drawing things out in excalidraw has been helping a lot
Abhishek retweeted
nvidia dynamo doesn't run the model. it decides which gpu handles a request, where its kv cache lives, and how to move that cache between workers.
i broke it down here:
therealshek.medium.com/how-n…
nvidia dynamo doesn't run the model. it decides which gpu handles a request, where its kv cache lives, and how to move that cache between workers.
i broke it down here:
therealshek.medium.com/how-n…
was reading about nvidia dynamo, and fuck, the scale of this thing is crazy.
especially loved the idea of disaggregating prefill and decode to improve gpu utilization. there's so much more to inference than just running a model
4
45
trying to build an inference engine from scratch in rust, starting with qwen2.5 1.5b.
even the first steps are making my brain work overtime, ngl. there is so much to understand before things actually start working.
but i'm enjoying it :)
2
17
ngl, it's interesting how an entire llm architecture can be described in a config.json.
spent time understanding what these parameters actually mean. now it's time to turn that understanding into my own inference engine in rust.
2
17
ngl, i underestimated how much i needed to understand candle before building enma.
even basic stuff like creating tensors on cuda, converting to fp16 and reading results back to rust is worth figuring out before i start implementing the actual model
2
10
















