writing our sicp at https://teapaw.org/t.co/mZ01Xi1sCl i like deep learning, gpus, and compilers distributed compilers @coinbase, mathematically trained @UWaterloo
a computer in the cloud
Joined April 2024
- Tweets506
- Following87
- Followers1.3K
- Likes79
jdz retweeted
OpenAI's Whisper model now fully runs on Magnetron and can be used for real time speech to text: github.com/magnetron-researc…
The latest update also introduced an async CPU backend which hides Python latency in heavy workflows by overlapping computation and Python work
there is a very exciting opportunity to re-do the classic introduction to proofs that computer science majors take.
—instead of the substance of number theory motivated by RSA "do you really want to know how that lock symbol in your browser works?"[0], you just go to straight to algebra and analysis like axler, artin, and tao. this would then segue into upper year courses that focus on interpretability and optimization respectively. grad school is then understanding the millenium proofs xD
[0]: cs.uwaterloo.ca/~cbruni/pdfs…
this is insane 🤯 can someone create a matmul-style worklog of all the major optimizations besides "use muon" for us plebs ? maybe the answer is just go take cs312..
our cutedsl lord and saviour @Simon_Vt also recommends "On Teaching Mathematics" maia.ub.es/~vieiro/fitxers/t… which I really like.
It reminds me of Alan Perlis co-authored "Social Processes and Proofs of Theorems and Programs" dl.acm.org/doi/pdf/10.1145/3….
the functional apis that underlie the stateful optimizers are being publicized with torch 2.15 via torch.optim.functional(). use case is optimizing with lower precision optimizers such as bf16 for adamw before the api get's settled. neat 🤓
youtube.com/watch?v=m1CA-saS…
docs.pytorch.org/docs/main/o…
zipping up the back of the book: for automatic differentiation sitp is going to use the same functional notation used in
1. sussman and wisdom's fdg (focusing on the lie and covariant derivative for general relativity)
2. spivak's calculus on manifolds (focusing on stokes' green, and div theorems)
3. frostig's guest chapter in murphy
with derivative as linear operator afforded by borscht.func() api with grad, jvp, and vjp.
—but!, under the hood borscht is going tape-based operator overloading because it is explicitly bridging to torch. not the jax style source transformation. i will have to read the torch source for how torch.func() sits on top of the dispatcher and engine. this is the best i can do to keep the lambda calculus alive given the constraints.
torch.nn's class Module() and torch.optim class Optimizer() are indeed quite hackable :-)
torch.Tensor and torch.Tensor.backward() is natively written in cpp with the dispatcher, a10's native_functions.yaml, and engine's derivative.yaml, so have a little more work to do there :-(
revisiting this tweet storm on "what would a revolutionary rewrite of torch look like" qt as the teaching autograd borscht is being set up properly, particularly the tradeoffs in a mixed source deep learning framework between python and c++, this exchange that spun off from elon's training stack rewrite helps clarify a lot.
—i.e one of the core tensions/pareto curves that exist in the minds of torch's core team is productivity vs performance, which will continue to exist after agents (the segfault remark with $100m training run), and then the original qt lists a possible compromise is keep native extension modules but make them modular via cpython, and have a python implementation as well. maybe if research engineers were all rustaceans we could rewrite torch to be pure rust, which perhaps if you were writing a training stack in-house, is something you might do .. matklad.github.io/2023/03/28…
for sitp's pedagogic goals, it seems nice to follow this. that way borscht can not only use torch as oracle, but also it's own python implementations.
zipping up the back of the book: for automatic differentiation sitp is going to use the same functional notation used in
1. sussman and wisdom's fdg (focusing on the lie and covariant derivative for general relativity)
2. spivak's calculus on manifolds (focusing on stokes' green, and div theorems)
3. frostig's guest chapter in murphy
with derivative as linear operator afforded by borscht.func() api with grad, jvp, and vjp.
—but!, under the hood borscht is going tape-based operator overloading because it is explicitly bridging to torch. not the jax style source transformation. i will have to read the torch source for how torch.func() sits on top of the dispatcher and engine. this is the best i can do to keep the lambda calculus alive given the constraints.
