RL is not the only paradigm for language model alignment or controlled generation! Let’s take a principled probabilistic approach: define a target distribution, perform inference, estimate KLs. Enter our work on twisted SMC for LMs with @brekelmaniac* @AliMakhzani @RogerGrosse

Introducing “Wasserstein Lagrangian Flows”: A novel computational approach for solving Optimal Transport and its variants. Paper: arxiv.org/abs/2310.10649 Led by @k_neklyudov and @brekelmaniac With: @AlexanderTong7 @lazar_atan @lqiang67 The solution of Optimal Transport (OT) and many of its variants, such as Schrödinger bridge, unbalanced OT, or OT with physical constraints, can be formulated as a curve on the density manifold that minimizes a particular Lagrangian action functional objective. We propose a unified computational framework for solving a very general class of action functional minimization problems, which also solves OT and its variants. We derive the dual objective of the Lagrangian on the cotangent space of the density manifold, resulting in a min-max optimization. It turns out the inner optimization, that evaluates the action functional, is a generalized “action-matching” objective, which we have previously shown to be tractable with a neural network parameterization: arxiv.org/abs/2210.06662. The outer optimization modifies the curve to minimize the action functional, subject to endpoint or multi-marginal constraints, using the reparameterization trick. The flexibility of the choice of Lagrangian in our framework gives us a principled way to incorporate different prior information in our problem, which finds many applications in AI4Sicence and generative modeling. We evaluate our method in trajectory inference problems in biological systems; and also show its applications in enabling fast one-step image generation. See Kirill’s thread for more details👇