1/6
I built Ava, a synthesizer that turns text descriptions into editable presets.
Not just audio samples, but actual sounds you can play, tweak, and save.
I wrote about the whole project, with audio examples and an interactive demo:
gisilvs.github.io/blog/ava/
1
4
175
5/6
The results are a bit hit or miss. Some sounds come surprisingly close, while others end up somewhere completely different.
But since they're editable presets, those happy accidents can become something new.
1
20
6/6
I also turned it into a C++ plugin for Mac, so you can actually make music with the generated presets.
Everything is open source:
PyTorch + training:
github.com/gisilvs/ava-torch
C++ plugin:
github.com/gisilvs/ava-cpp
Would love to hear what you think!
19
Gianluigi Silvestri retweeted
Tired to go back to the original papers again and again? Our monograph: a systematic and fundamental recipe you can rely on!
📘 We’re excited to release 《The Principles of Diffusion Models》— with @DrYangSong, @gimdong58085414, @mittu1204, and @StefanoErmon.
It traces the core ideas that shaped diffusion modeling and explains how today’s models work, why they work, and where they’re heading.
🧵You’ll find the link and a few highlights in the thread.
We’d love to hear your thoughts and join some discussions!
⚡ Stay tuned for our markdown version, where you can drop your comments!
Gianluigi Silvestri retweeted
I spent the past month reimplementing DeepMind’s Genie 3 world model from scratch
Ended up making TinyWorlds, a 3M parameter world model capable of generating playable game environments
demo below + everything I learned in thread (full repo at the end)👇🏼
Gianluigi Silvestri retweeted
1/2) Have you have noticed that the forward process in a diffusion model looks a lot like the reparameterization trick in VAEs?
It turns out that there is a deep connection!
Curious? Watch our new vedio in the Generative Memory Lab channels
(link below)
New preprint out! Reach out if you are interested, or let's grab a coffee at ICML this week!
1/3) I am biased, but I think this is going to be big!
CoVAE: Consistency Training of Variational Autoencoders
We unify consistency models with VAEs to obtain a powerful and elegant generative autoencoder!
The brainchild of the brilliant @gisilvs (who is looking for jobs!)
1
11
896
Gianluigi Silvestri retweeted
I am very happy to share our recent work on consistency model led my my brilliant PhD student @gisilvs during his internship at @SonyAI_global in collaboration with @JCJesseLai, @takiko_san and @mittu1204.
"Training Consistency Models with Variational Noise Coupling"
Gianluigi Silvestri retweeted
I am very happy to share:
" Manifolds, Random Matrices and Spectral Gaps: The geometric phases of generative diffusion"
We study the evolving latent geometry of diffusion by analyzing gaps in the Jacobian spectrum with statistical physics methods
Paper: arxiv.org/pdf/2410.05898
Gianluigi Silvestri retweeted
Amazing presentation by Marvin Li on critical windows in diffusion models.
A work very closely related to our research on spontaneous symmetry breaking.
youtu.be/NbnBmi5k1UQ?si=86gH…
Gianluigi Silvestri retweeted
Job alert! I am looking for a talented PhD student to do research on generative diffusion and its connections to statistical physics and memory.
Feel free to send me a PM if you need more info!
You can apply here:
ru.nl/en/working-at/job-oppo…
Otherwise, please retweet!
Gianluigi Silvestri retweeted
An amazing talk by @aaron_lou on diffusion language transformers.
Bracing for the era of large language diffusion!
youtu.be/_1qv_LNjH9U?si=xgNW…
Gianluigi Silvestri retweeted
You should follow our Generative Memory Lab channel!
Plenty of content on generative models, diffusion models, associative memory!
youtube.com/@gemlab3695?si=J…
Gianluigi Silvestri retweeted
A very insightful presentation about probabilistic Hopfield nets and their link with diffusion models!
youtu.be/hK_L9Typ8PM?si=SrH5…
Another fantastic presentation, this time from @k_neklyudov!! You can watch it here: youtube.com/watch?v=LpqrgVDA…
Thanks @k_neklyudov for the great talk and for answering all our questions 👏🙏!
Gianluigi Silvestri retweeted
Modern Hopfield networks are related to transformers, but did you know that they are mathematically equivalent to generative diffusion models?
Happy to share:
"In search of dispersed memories: Generative diffusion models are associative memory networks"
arxiv.org/abs/2309.17290
Fantastic presentation from @dblueeye on diffusion models with discrete diffusion process 👏👏 very clear explanation and promising results for their 'Blackout Diffusion'. Take a look at the presentation here youtu.be/LnCI7I7FmAA?si=9F8n…












