Keshi Dai

I probably have some credibility as a person who has worked on @TensorFlow and @PyTorch both (also, caffe / @onnxai / distbelief / a few others that never saw the light), so here are my two cents: (1) Speed doesn't really matter today as long as it is not particularly bad. Under the hood, many are just calling optimized CUDA code anyway. (2) For ultimate speed, especially in the LLM field, many are writing their own runtime engines, and not vanilla framework. For example vLLM, by @zhuohan123 @woosuk_k et al, is a great open source solution. (Lepton builds its own too, and is probably the fastest not-a-framework llm engine right now) (3) TensorFlow and PyTorch are successful in their own terms. For me, TF taught me how to think like a system architect, and PyTorch taught me how to prioritize users' needs over optimization. (4) Simply listing a table showing performance is relatively useless, because every use case come with their own needs. (4.1) For example, if you are running ads and feed recommendation systems, then 1% performance matters, and you pretty much need to write even your own FusedGemmAndReluAndMyWeirdOpsAfterThatOp to make things fast. (4.2) For research though, the flexibility was of ultimate importance, and losing 20-30%, sometimes even 100% speed, is fine - you gain from reducing people hours. (4.3) I want to express thanks to @soumithchintala and the FAIR team, because we had long-lasting arguments back at Meta: I was serving all ads/feed needs so 4.1 was my only priority, and those arguments were a window into the 4.2 land. In retrospect, those was some of the most rewarding experiences in my career. (5) People hours are important for a simple reason: salaries are increasing every year, and NVidia is reducing the dollar cost per TFlops very quickly. (6) A side note... Keras PyTorch wrapper exhibits a universal 2x overhead over native PyTorch. This is... not good. A wrapper abstraction shouldn't bring such a big overhead. (7) Ah the old good war of framework days. I feel nostalgic to have been part of it, and I am excited to move beyond frameworks and to build a truly AI-native cloud at @LeptonAI .