Nikos Karampatziakis retweeted
A new version of 🤗 transformers has landed 🛬 v4.44 ended up being a performance-oriented upgrade for LLM users: faster compiled models, lower GPU memory requirements, and even support for mobile devices!
Let's dive 🤿 (resources in a comment)
1️⃣ `torch.compile()` updates
We've been working to expand and improve our compilation support! Recent upgrades include:
- `model.generate()` can be compiled! This is very experimental, and slightly faster than compiling the forward pass alone. More importantly, it opens the door to export the whole generate function into a single graph we can export, to be used in other devices ⚡️
- 3-5x faster compilation time of `model.forward()`, when used in `model.generate()`. tl;dr we now prepate a static-shape attention mask in generate. In essence, the same speed up, but less wait time the first time you run it ⌛️
- support for Whisper! Well, this was actually available since v4.43, released two weeks ago, but we haven't communicated much about it 🤫
2️⃣ CPU-offloaded KV cache
When you're using an LLM with a large context window, you'll notice that your GPU RAM quickly gets devoured by the KV cache. You had to spin up more/larger GPUs just to use the same model, or rely on CPU computations.
Not anymore! v4.44 includes a CPU-offloaded KV cache that only keeps in memory what it needs. In essence, while computing layer N, it moves to the CPU the KV cache for layer N-1, while prefetching to the GPU layer N+1. All computations happen on GPU. This is useful for large context windows or large batch sizes -- it allows you to run larger settings without upgrading your GPU, at a minimal speed penalty. Perfect for the GPU poor team 💛
This was kindly added to our library by @eigenikos 🤗
3️⃣ `torch.export()` support for generation
The PyTorch team is working on `torch.export()` -- in a nutshell, ahead of time compilation to enable multiple downstream uses. You can then use the exported graph with ExecuTorch in e.g. a mobile device!
We can now export LLMs with `torch.export()`. Support was kindly enabled by Guang Yang from the PyTorch team 🤗
And that's it for now! See you next release 🛫
Nikos Karampatziakis retweeted
I would like to invite you to try phi-3-mini: aka.ms/try-phi3-hf-chat. You can also download the weights from HF with more model weights on the way.
Besides what was described in technical report, one specific thing I want to mention is the 128K context support. It takes us a lot of efforts to fit 128K context in such a mini model with little sacrifice on short context quality. Go to play with it and share with us your feedback: aka.ms/try-phi3-hf-chat
Nikos Karampatziakis retweeted
phi-3 is here, and it's ... good :-).
I made a quick short demo to give you a feel of what phi-3-mini (3.8B) can do. Stay tuned for the open weights release and more announcements tomorrow morning!
(And ofc this wouldn't be complete without the usual table of benchmarks!)
Nikos Karampatziakis retweeted
Microsoft announces Phi-3
A Highly Capable Language Model Locally on Your Phone
We introduce phi-3-mini, a 3.8 billion parameter language model trained on 3.3 trillion tokens, whose overall performance, as measured by both academic benchmarks and internal testing,
Nikos Karampatziakis retweeted
➡️Mindblowing pretraining paradigm⬅️
Train the same model to predict the two directions separately🔛
Better results, more parallelization
arxiv.org/abs/2303.07295
@MSFTResearch
@NguynTu24128917 @eigenikos @WeizhuChen
#deepRead
I don't always write deep learning papers, but when I do it's🔥
Really proud of how this work with @NguynTu24128917 and @WeizhuChen came together.
This quoted post is unavailable.
1
3
211
Meet In the Middle (MIM) : A New Pretraining Paradigm.
MIM(2.7B) outperforms CodeGen 16B, Incoder 6.7B, PaLM 540B, LLaMA 65B, FIM 2.7B in Code generation tasks. Read arxiv.org/abs/2303.07295 to know why MIM could be a new pre-training paradigm for left-to-right and infilling LMs.
1
170
Nikos Karampatziakis retweeted
Meet In the Middle (MIM) : A New Pretraining Paradigm.
MIM(2.7B) outperforms CodeGen 16B, Incoder 6.7B, PaLM 540B, LLaMA 65B, FIM 2.7B in Code generation tasks. Read arxiv.org/abs/2303.07295 to know why MIM could be a new pre-training paradigm for left-to-right and infilling LMs.
Nikos Karampatziakis retweeted
New paper with @kwangsungjun
Tight Concentrations and Confidence Sequences
from the Regret of Universal Portfolio
arxiv.org/abs/2110.14099
I particularly like it, let me tell you about it
🧵
Our work on confidence bounds that hold uniformly over time for off-policy evaluation in contextual bandits has been accepted at ICML 🥳 arxiv.org/abs/2102.09540
These confidence sequences enable continuous monitoring of experiments without union bounds or peeling tricks.
This work was inspired by this paper
arxiv.org/abs/2010.09686
which I highly recommend. We had to make some important adjustments for the off-policy case. That is, a straightforward application of these ideas leads to excessively loose bounds when importance weights can be large.
1
6
Too much nagging about this. It was in the CFP from the beginning. You did read the rules before submitting, right? Right?
Now that all papers and supplementary materials are submitted, please remember that ICML strongly discourages advertising preprints on social media or in the press while under submission to ICML (see more details in icml.cc/Conferences/2021/Cal…).
1
Nikos Karampatziakis retweeted
Watch out, econonomics -- A second scientific discpline now officially suffers from a Messiah complex!
#NeurIPS2020 deadline is 22 days away!
Special call out: we welcome papers that use ML to contribute to understanding, treatment, and prevention of COVID-19 (and yes, before you ask, we have recruited ACs and reviewers that are especially qualified to review these submissions).
To derive this start with the block LU
[A B] = [I 0] [A B]
[C D] [L I] [0 S]
so L = C inv(A), S = D - L B
Given vector [p;q] solve the block system via:
[I 0] [w] = [p]
[L I] [z] [q]
and then
[A B] [x] = [w]
[0 S] [y] [z]
Does anyone have a useful mnemonic for blockwise matrix inversion? I have to look it up every time.
en.wikipedia.org/wiki/Block_…
The whole thread on BNNs and blog post by @jacobmbuckman and @carlesgelada reminded me of the "First, you rob a bank..." characterization by Yasser Abu Mostafa youtube.com/watch?v=ihLwJPHk…
Apologies to my Bayesian friends who may find it unfair.
The only relationship between moments of a random variable that I can remember is E[X²] - E[X]² = E[(X-E[X])²] ≥ 0. How to derive relationships between other moments?
For example suppose we know that E[X] = 0 and E[X²] = 1. What can we say about the relationship of E[X³] and E[X⁴]? Letting v = [1 X X²] we have E[vvᵀ] =
1 0 1
0 1 E[X³]
1 E[X³] E[X⁴]
1
6

















