Member of the human race.
San Francisco, CA
Joined April 2018
- Tweets129
- Following737
- Followers434
- Likes2.2K
Pinned Tweet
I’m super excited to announce that my book, Applied Natural Language Processing in the Enterprise (published by @OReillyMedia), is now available! 📕🎉
Learn more at our website 👇appliednlpbook.com
(How) does R1 have an equivalent for the “reasoning_effort” parameter in the o1 API?
Learning to use a chain of thought purely through RL is very cool, but the other important part of o1/o3 was *scaling* inference time compute.
Without this, the length of the chain of thought (and, by extension, the amount of inference compute used) is entirely at the model’s discretion, meaning I can’t get better responses simply by spending more compute at inference time.
i.e. you can't go from an R1 to an "R1 Pro".
Guess I have an answer!
DeepSeek r1 is exciting but misses OpenAI’s test-time scaling plot and needs lots of data.
We introduce s1 reproducing o1-preview scaling & performance with just 1K samples & a simple test-time intervention.
📜arxiv.org/abs/2501.19393
About two years ago I thought the biggest "unsolved" problem with LLMs were:
1. Training and inference are fundamentally different algorithms, which doesn't match my mental model for intelligence (i.e. my brain doesn't flip between discrete "training" and "inference" modes - it feels more like one machine that continually processes new information). I thought this meant that, in practice, LLMs couldn't learn new things (facts, skills, etc.) at inference time. I told myself I was fine with a "bootstrapping phase" (i.e. pretraining) for efficiency reasons.
2. They use a ~constant amount of compute per token with a static state (weights + kv cache). Which seemed to imply that there was an upper bound on their problem solving capacity.
But today, these seem to be largely "solved" with a combination of:
1. In context learning (though maybe we could do a bit better at this with a meta-learning objective).
2. o1-style chain of thought.
3. Context windows that are long enough (and growing) to solve a lot of useful problems .
In hindsight I think they were not very big problems at all.
Ajay Uppili Arasanipalai retweeted
📣 Introducing Llama 3.2: Lightweight models for edge devices, vision models and more!
What’s new?
• Llama 3.2 1B & 3B models deliver state-of-the-art capabilities for their class for several on-device use cases — with support for @Arm, @MediaTek & @Qualcomm on day one.
• Llama 3.2 11B & 90B vision models deliver performance competitive with leading closed models — and can be used as drop-in replacements for Llama 3.1 8B & 70B.
• New Llama Guard models to support multimodal use cases and edge deployments.
• The first official distro of Llama Stack simplifies and supercharges the way developers & enterprises can build around Llama to support agentic applications and more.
Details in the full announcement ➡️ go.fb.me/229ug4
Download Llama 3.2 models ➡️ go.fb.me/w63yfd
These models are available to download now directly from Meta and @HuggingFace — and will be available across offerings from 25+ partners that are rolling out starting today, including @accenture, @awscloud, @AMD, @azure, @Databricks, @Dell, @Deloitte, @FireworksAI_HQ, @GoogleCloud, @GroqInc, @IBMwatsonx, @Infosys, @Intel, @kaggle, @NVIDIA, @OracleCloud, @PwC, @scale_AI, @snowflakeDB, @togethercompute and more.
With Llama 3.2 we’re making it possible to run Llama in even more places, with even more flexible capabilities. We’ve said it before and we’ll say it again: open source AI is how we ensure that these innovations reflect the global community they’re built for and benefit everyone. We’re continuing our drive to make open source the standard with Llama 3.2.
Thank you!
Starting today, open source is leading the way. Introducing Llama 3.1: Our most capable models yet.
Today we’re releasing a collection of new Llama 3.1 models including our long awaited 405B. These models deliver improved reasoning capabilities, a larger 128K token context window and improved support for 8 languages among other improvements. Llama 3.1 405B rivals leading closed source models on state-of-the-art capabilities across a range of tasks in general knowledge, steerability, math, tool use and multilingual translation.
The models are available to download now directly from Meta or @huggingface. With today’s release the ecosystem is also ready to go with 25+ partners rolling out our latest models — including @awscloud, @nvidia, @databricks, @groqinc, @dell, @azure and @googlecloud ready on day one.
More details in the full announcement ➡️ go.fb.me/tpuhb6
Download Llama 3.1 models ➡️ go.fb.me/vq04tr
With these releases we’re setting the stage for unprecedented new opportunities and we can’t wait to see the innovation our newest models will unlock across all levels of the AI community.
Ajay Uppili Arasanipalai retweeted
Starting today, open source is leading the way. Introducing Llama 3.1: Our most capable models yet.
Today we’re releasing a collection of new Llama 3.1 models including our long awaited 405B. These models deliver improved reasoning capabilities, a larger 128K token context window and improved support for 8 languages among other improvements. Llama 3.1 405B rivals leading closed source models on state-of-the-art capabilities across a range of tasks in general knowledge, steerability, math, tool use and multilingual translation.
The models are available to download now directly from Meta or @huggingface. With today’s release the ecosystem is also ready to go with 25+ partners rolling out our latest models — including @awscloud, @nvidia, @databricks, @groqinc, @dell, @azure and @googlecloud ready on day one.
More details in the full announcement ➡️ go.fb.me/tpuhb6
Download Llama 3.1 models ➡️ go.fb.me/vq04tr
With these releases we’re setting the stage for unprecedented new opportunities and we can’t wait to see the innovation our newest models will unlock across all levels of the AI community.
Ajay Uppili Arasanipalai retweeted
Accepted ISMIR paper from @hugggof's internship with us at @descript ! We applied a MaskGIT-style generative model to discrete audio codes (github.com/descriptinc/descr…). We can make music variations, edits, and more! Listen below. Samples here: hugo-does-things.notion.site…
VampNet: Music Generation via Masked Acoustic Token Modeling
paper page: huggingface.co/papers/2307.0…
introduce VampNet, a masked acoustic token modeling approach to music synthesis, compression, inpainting, and variation. We use a variable masking schedule during training which allows us to sample coherent music from the model by applying a variety of masking approaches (called prompts) during inference. VampNet is non-autoregressive, leveraging a bidirectional transformer architecture that attends to all tokens in a forward pass. With just 36 sampling passes, VampNet can generate coherent high-fidelity musical waveforms. We show that by prompting VampNet in various ways, we can apply it to tasks like music compression, inpainting, outpainting, continuation, and looping with variation (vamping). Appropriately prompted, VampNet is capable of maintaining style, genre, instrumentation, and other high-level aspects of the music. This flexible prompting capability makes VampNet a powerful music co-creation tool. Code and audio samples are available online.
Ajay Uppili Arasanipalai retweeted
This was a lot of fun to work on! Excited to get this out there. Listen below! Also, we released our library for handling and augmenting audio signals (with GPU acceleration). Lots of goodies like fast loudness norm, reverb, filters, additive noise, etc. github.com/descriptinc/audio…
✨ Super excited to share our work on neural audio quantizers. It’s especially very timely considering the interest in AudioLMs, MusicLM and MusicGen!
Fully open sourced training + inference code and model weights with MIT license 🎉
arxiv.org/abs/2306.06546
Ajay Uppili Arasanipalai retweeted
✨ Super excited to share our work on neural audio quantizers. It’s especially very timely considering the interest in AudioLMs, MusicLM and MusicGen!
Fully open sourced training + inference code and model weights with MIT license 🎉
arxiv.org/abs/2306.06546
High-Fidelity Audio Compression with Improved RVQGAN
paper page: huggingface.co/papers/2306.0…
Language models have been successfully used to model natural signals, such as images, speech, and music. A key component of these models is a high quality neural compression model that can compress high-dimensional natural signals into lower dimensional discrete tokens. To that end, we introduce a high-fidelity universal neural audio compression algorithm that achieves ~90x compression of 44.1 KHz audio into tokens at just 8kbps bandwidth. We achieve this by combining advances in high-fidelity audio generation with better vector quantization techniques from the image domain, along with improved adversarial and reconstruction losses. We compress all domains (speech, environment, music, etc.) with a single universal model, making it widely applicable to generative modeling of all audio. We compare with competing audio compression algorithms, and find our method outperforms them significantly. We provide thorough ablations for every design choice, as well as open-source code and trained model weights. We hope our work can lay the foundation for the next generation of high-fidelity audio modeling.
This looks amazing!
One Spider-Man wants to change his own destiny. 🕷 Miles Morales returns for the next Spider-Man movie, exclusively in theaters June 2. Watch the new trailer for Across the @SpiderVerse now. #SpiderVerse
Ajay Uppili Arasanipalai retweeted
We hosted #emergencychatgpthackathon this past Sunday for the new ChatGPT and Whisper APIs.
It all came together in just 4 days, but we had 250+ people and 70+ teams demo!
Here’s a recap of our winning demos:
Ajay Uppili Arasanipalai retweeted
When I first saw this movie it was about a week before release. Sony made the Emoji Movie so I didn’t expect much, but then afterwards I had to stay quiet for about a week about what I believed at the time to be the best Spider-man movie of all time. Top 5 theater experience.
Ajay Uppili Arasanipalai retweeted
There's a tradition of film directors and studios congratulating each other for beating their box office records. A THREAD.
1/11: In 1977, when STAR WARS beat Jaws to become the highest-grossing movie ever, Steven Spielberg took out the below ad for George Lucas in @Variety
