Jason Rader

Adam depends on the gradient distribution during training, which, as far as I know, we don't understand well? Here, adapted from the Adam paper, v_t is the var estimate, G_t is the gradient r.v. and X_t is an error r.v. for distribution shift.

Using the damped ratio LM variant to optimise the parameters of a simple ODE (docs.kidger.site/optimistix/…) we see no difference between it and the original LM algorithm.

So how does it fair? Unsurprisingly, it behaves basically identically to the original trust-region update. Here is the BFGS algorithm on some select minimisation test problems (averaged over a few different initialisations)

If the ratio is close to 1, the model is good and we lower the regularisation next step. If it's closer to 0, the model is poor and we increase the regularisation. The unregularised model is not the one used by the Tikhnov regularised model. This motives the "damped" ratio.

The typical heuristic to update the amount of regularisation each step is the Levenberg-Marquardt heuristic: Take the ratio between the the actual decrease of a step and the decrease predicted by a local unregularised model.

Tikhnov regularised trust-region methods (*cough* Levenberg-Marquardt) oddly use two different approximations to the objective function at each step. One regularised, one not. What if we just regularised both? 1/

✨Little λ ✨is a tiny utility for function composition, the un-Pythonic way It's a single class which can be used to compose functions with a functional-programming-like syntax. Not intended for serious use, but I think its fun GitHub: github.com/packquickly/littl…

Yesterday I finished the 56 mile Tahoe rim trail endurance run 🏃‍♂️! This was a tough race with temperatures in the 40s (100+f) and an average altitude of 8000ft. But it was truly stunning and personally fulfilling. Now to start training for the next race in September ☺️

Only in PyTorch is Limited-Memory BFGS a very memory intensive optimiser