Abhishek (@ImTheRealShek) header
Abhishek (@ImTheRealShek) profile picture

backend engineer → ai infra | rust · go | 23 | koinophobic by design | previously 1x startup

Delhi, India
Joined December 2023
Photo by Abhishek (@ImTheRealShek)Photo by Abhishek (@ImTheRealShek)Photo by Abhishek (@ImTheRealShek)Photo by Abhishek (@ImTheRealShek)Photo by Abhishek (@ImTheRealShek)Photo by Abhishek (@ImTheRealShek)Photo by Abhishek (@ImTheRealShek)Photo by Abhishek (@ImTheRealShek)Photo by Abhishek (@ImTheRealShek)Photo by Abhishek (@ImTheRealShek)
Thread of things I've built over the last few years. Most of them started because I wanted to understand how something actually worked instead of treating it as a black box. I'll keep adding to this thread whenever I finish something new.
1
2
355
Abhishek retweeted
cold post, but tbh i'd rather say what i want. any backend role works, and llm inference or ai engineer is the best fit. i'm happy to learn new things as long as it isn't frontend, i just don't have the taste for it. dms are open, see bio for my portfolio website link.
2
2
17
cold post, but tbh i'd rather say what i want. any backend role works, and llm inference or ai engineer is the best fit. i'm happy to learn new things as long as it isn't frontend, i just don't have the taste for it. dms are open, see bio for my portfolio website link.
2
2
17
Abhishek retweeted
finally got my first self written harness working. the first tool call works! spent way too much time debugging, but honestly, writing code from scratch and seeing things work is one of the best parts of learning something new.
1
1
2
15
finally got my first self written harness working. the first tool call works! spent way too much time debugging, but honestly, writing code from scratch and seeing things work is one of the best parts of learning something new.
1
1
2
15
Abhishek retweeted
hehe, need to start writing my own harness. no ai writing the code this time. i want to learn by building it myself. lfg!!
was learning how to build an agent harness and came across this. fuck, the way they explain it is so simple. it starts with just a loop. must read. ampcode.com/notes/how-to-bui…
1
2
9
hehe, need to start writing my own harness. no ai writing the code this time. i want to learn by building it myself. lfg!!
was learning how to build an agent harness and came across this. fuck, the way they explain it is so simple. it starts with just a loop. must read. ampcode.com/notes/how-to-bui…
1
2
9
was learning how to build an agent harness and came across this. fuck, the way they explain it is so simple. it starts with just a loop. must read. ampcode.com/notes/how-to-bui…
1
2
55
ask for the win and you get shoes. ngl one line like that says more than a long explanation
"If you ask God to help you win a marathon, He will give you shoes, not a medal."
3
14
Abhishek retweeted
continuing from my last update, we've gotten deeper into splitting GPU workloads between prefill and decode. i'm exploring how KV cache blocks move across nodes using RDMA and load into decode GPUs. how are people handling this in prod? would love to learn from real setups.
continuing my last post on distributed llm inference. i'm exploring how to split gpu workloads between prefill and decode instead of running both on every gpu. the tricky part is coordinating the workers and transferring kv cache through nixl.
1
2
19
continuing from my last update, we've gotten deeper into splitting GPU workloads between prefill and decode. i'm exploring how KV cache blocks move across nodes using RDMA and load into decode GPUs. how are people handling this in prod? would love to learn from real setups.
continuing my last post on distributed llm inference. i'm exploring how to split gpu workloads between prefill and decode instead of running both on every gpu. the tricky part is coordinating the workers and transferring kv cache through nixl.
1
2
19
Abhishek retweeted
continuing my last post on distributed llm inference. i'm exploring how to split gpu workloads between prefill and decode instead of running both on every gpu. the tricky part is coordinating the workers and transferring kv cache through nixl.
i'm trying to get better at distributed inference hld, and drawing things out in excalidraw has been helping a lot
1
1
1
1
91
continuing my last post on distributed llm inference. i'm exploring how to split gpu workloads between prefill and decode instead of running both on every gpu. the tricky part is coordinating the workers and transferring kv cache through nixl.
i'm trying to get better at distributed inference hld, and drawing things out in excalidraw has been helping a lot
1
1
1
1
91
i'm trying to get better at distributed inference hld, and drawing things out in excalidraw has been helping a lot
1
2
92
Abhishek retweeted
nvidia dynamo doesn't run the model. it decides which gpu handles a request, where its kv cache lives, and how to move that cache between workers. i broke it down here: therealshek.medium.com/how-n…
1
1
3
84
was reading about nvidia dynamo, and fuck, the scale of this thing is crazy. especially loved the idea of disaggregating prefill and decode to improve gpu utilization. there's so much more to inference than just running a model
4
45
trying to build an inference engine from scratch in rust, starting with qwen2.5 1.5b. even the first steps are making my brain work overtime, ngl. there is so much to understand before things actually start working. but i'm enjoying it :)
2
17
ngl, it's interesting how an entire llm architecture can be described in a config.json. spent time understanding what these parameters actually mean. now it's time to turn that understanding into my own inference engine in rust.
2
17
ngl, i underestimated how much i needed to understand candle before building enma. even basic stuff like creating tensors on cuda, converting to fp16 and reading results back to rust is worth figuring out before i start implementing the actual model
2
10
Abhishek retweeted
built an inference gateway around llama.cpp, but wanted to know how an inference engine actually works. so i started studying qwen2.5 1.5b and made this architecture diagram myself. ngl, connecting all these parts is frying my brain. wish me luck.
1
2
19