Don Wei (@don_wei) header
Don Wei (@don_wei) profile picture

OpenMotion AI Co-founder, CTO - wearable sensing and multimodal AI. MD-PhD

Joined February 2012
Photo by Don Wei (@don_wei)Photo by Don Wei (@don_wei)Photo by Don Wei (@don_wei)
22
82
16
648
48,998
🎨Spent some time refactoring the 2021 post on diffusion model with new content: lilianweng.github.io/posts/2… ⬇️ ⬇️ ⬇️ 🎬Then another short piece on diffusion video models: lilianweng.github.io/posts/2… (Yes, I had an intensive weekend🥹)
22
162
5
1,038
126,539
New video series: Physics Informed Machine Learning! Physics may be embedded into AI/ML in 5 stages: 1 choose what to Model 2 curate training Data 3 design an Architecture 4 craft a Loss Function, and 5 implement Optimization Algorithm to train the model youtube.com/watch?v=JoFW2uSd…
23
194
13
1,241
78,650
Introducing 💻OpenCI (Open Code Interpreter)! The CodeInterpreter of ChatGPT is what many people used everyday, while open code-LMs are still not good as an interactive coding assistant. We finally created a framework with 68K data such that open models can also learn from human/env feedback (e.g., error msgs 🐞 or new features) to continually refine their code on the fly🪁. The performance of OpenCI is top on many tested benchmarks ↗️! Kudos to @zhengtianyu4 @GeZhang86038849 @xiangyue96 and the team. 🙌
🌟With precise execution & human feedback, a 7B code model hits 90% accuracy on HumanEval! 🚀 Introducing OpenCodeInterpreter: A family of open-source code systems for generating, executing, & refining code.🔄 🤖 Traditional open-source models often fall short in execution capabilities and lack the iterative refinement process seen in cutting-edge systems like the GPT-4 Code Interpreter. ✨ To fill this gap, we construct CodeFeedback, an open-source code instruction dataset encompassing 68K multi-turn interactions between users, python interpreters, and code models. CodeFeedback contains diverse queries from multiple open-source code instruction tuning datasets and Leetcode problems. Check out more in the following threads👇Kudos to the student leads @zhengtianyu4 @GeZhang86038849 and the team @allen_thshen, Xuelin Liu, @billyuchenlin, @bigaidream,@WenhuChen. We will host an online demo on huggingface very soon! Stay tuned!
13
61
6
344
83,726
YOLOv9 is out! It's a real-time object detection model that surpasses all convolution and transformer-based models. How? It introduces programmable gradient information (PGI) and the Generalized Efficient Layer Aggregation Network (GELAN) to improve accuracy. PGI prevents data loss and ensures accurate gradient updates. GELAN optimizes lightweight models with gradient path planning. Models trained from scratch excel on datasets like MS COCO.
12
103
9
570
55,469
Don Wei retweeted
Wow. This takes "chatPDF" to an entirely new level. Don't think this is possible with regular rag either I gave 4 highly tabular pdfs to gemini 1.5 (tesla financials) it understood the entire context, summed all the tabular values and gave me a nearly perfect response.
25
47
12
419
85,611
AI x Robotics is progressing incredibly fast. Significant advancements this week from Stanford/USC, Google DeepMind, Meta, Stability AI, Science Robotics, and more. Here's everything you need to know:
42
319
36
2,669
2,423,920
It was an honor to give a guest lecture yesterday at Stanford’s CS330 class, "Deep Multi-Task and Meta-Learning"! I discussed a few very simple intuitions for how I personally think about large language models. Slides: docs.google.com/presentation… Here are the six intuitions: (1) First, I encouraged viewing next-word prediction as massive multi-task learning. Even though next-word prediction is very simple, because the pre-training data is so large and diverse, LMs learn a lot of tasks from next-word prediction. This can range from simple things like grammar to harder tasks like arithmetic reasoning. Anything that could be found in pre-training data could potentially be learned by a LM. (2) Next, learning from <input, output> pairs (in-context learning) can be cast as next-word prediction. This was popularized by the GPT-3 paper (arxiv.org/abs/2005.14165). It is very convenient to formulate tasks using <input, output> pairs since that is how we have done AI in the past decades. However, I am not sure how long that will prevail. We can do better by adding natural language instructions, showing how the reasoning works, enumerating boundary cases, giving examples of what not to do, etc. (3) A fundamental observation is that tokens have very different information density. Some tokens are easy to predict (e.g., “large language ___” is obviously “model”). Other tokens are very hard to predict (e.g., answer to a math problem), and so LMs should spend more compute before trying to predict them. One way to do this is chain-of-thought prompting (arxiv.org/abs/2201.11903), which encourages LMs to give a reasoning path before giving the final answer, allowing them to do complex reasoning tasks. It is my dream that one day AI will be able to help us with extremely challenging tasks, such as writing a proposal to reduce climate change. Spending more compute on reasoning is a first step in that direction. (4) Increasing compute for pre-training is expected to improve loss (scaling laws, arxiv.org/abs/2001.08361). This seems trivial but the fact that loss hasn’t saturated implies that continued investment in scaling will likely produce more capable models. It is a natural question to ask why scaling improves performance; my two hand-wavy hypotheses are that (1) large LMs can memorize more knowledge about the world and (2) large LMs use more complicated heuristics to get the loss as low as possible. (5) Although overall loss improves smoothly as you scale, individual tasks might improve suddenly (emergent abilities, arxiv.org/abs/2206.07682). Since next-word prediction is massive multi-task learning, you can view the loss as the weighted sum of many individual tasks. When you decrease the loss, it is likely that not all individual tasks improve uniformly. Loss for some tasks might be saturated (larger models no longer improving in grammar since they already have perfect grammar), and other tasks might improve in a more sudden fashion (in order to push loss lower, the larger model has to figure out how to do hard math problems). (6) Finally, I argue that large LMs can actually learn at <input, output> relationships in context. While one paper showed that random labels in in-context examples barely hurts performance (arxiv.org/abs/2202.12837), our recent work found that language models can follow both flipped labels and semantically-unrelated labels (arxiv.org/abs/2303.03846). The catch though, is that this ability only exists in language models that are large enough (e.g., GPT-3.5 and PaLM-1 or larger). Some of these intuitions extendable beyond language: - Intuition 3 (that tokens have different information density), might be generally applicable to most data. For example, in computer vision you may want to spend more compute analyzing the important parts of an image, like someone's facial expression. - Intuition 4 (scaling laws) is applicable not just for compute, but whenever you collect finetuning data. You can plot the scaling curve with # of training examples on the x-axis and performance on the y-axis, and predict how much collecting more data will help. - Intuition 5 (decomposing aggregate metrics into individual tasks) can be applicable whenever you’re using an aggregate metric. With finer-grained categories, you can a much better understanding of what is happening and find out which categories might need the most improvement. View the longer-form summary blog here: jasonwei.net/blog/some-intui… The class is here: cs330.stanford.edu/
19
264
22
1,519
277,786
You know how image generation went from blurry 32x32 texture patches to high-resolution images that are difficult to distinguish from real in roughly a snap of a finger? The same is now happening along the time axis (extending to video) and the repercussions boggle the mind just a bit. Every human becomes a director of multi-modal dreams, like the architect in Inception. Coming back to Earth for a second, image/video generation is a perfect match for data-hungry neural nets because data is plentiful, and the pixels of each image or video are a huge source of bits (soft constraints) on the parameters of the network. When you're training giant neural nets in supervision-rich settings, your train loss = validation loss, and life is so good. My favorite place to keep an eye on the AI video space unfold atm is probably reddit.com/r/aivideo/ , or the individual Discords.
Introducing Pika 1.0, the idea-to-video platform that brings your creativity to life. Create and edit your videos with AI. Rolling out to new users on web and discord, starting today. Sign up at pika.art
185
1,429
117
10,809
2,680,747
New lab paper out today 🥳 Rare amygdala iEEG activity recorded in two patients with treatment-resistant PTSD & used to trigger closed-loop stimulation therapy over 1 yr -> significant patient improvement nature.com/articles/s41467-0…
6
73
10
259
49,249
Adam, a 9-yr old optimizer, is the go-to for training LLMs (eg, GPT-3, OPT, LLAMA). Introducing Sophia, a new optimizer that is 2x faster than Adam on LLMs. Just a few more lines of code could cut your costs from $2M to $1M (if scaling laws hold). arxiv.org/abs/2305.14342 🧵⬇️
87
596
116
3,461
1,049,583
Don Wei retweeted
Drag Your GAN: Interactive Point-based Manipulation on the Generative Image Manifold paper page: huggingface.co/papers/2305.1…
308
5,358
1,319
20,478
3,421,530
Very excited about this! @_EdgeAnalytics announces the launch of our AI Residency Program + our inaugural AI Resident, @don_wei! #AI #datascience #ml #TechNews
We are excited to announce the launch of the @_EdgeAnalytics AI Residency Program! At Edge, we are dedicated to continuous learning. The #AI Residency is our latest investment for ensuring we continue working with incredible people. #datascience #ml medium.com/edge-analytics/ed…
1
4
6
Ok, so I made one for psychiatry!🤓 *Types of Psychiatry Papers*
54
474
119
1,898
This is a day I’ve dreamed of my whole life, this is the reason @DeepMind was founded, to build AI and use it accomplish extraordinary scientific breakthroughs like #AlphaFold 2, to advance science and benefit humanity. I could not be more proud of the incredible team!
Today with @emblebi, we're launching the #AlphaFold Protein Structure Database, which offers the most complete and accurate picture of the human proteome, doubling humanity’s accumulated knowledge of high-accuracy human protein structures - for free: dpmd.ai/alphafolddb 1/
135
1,058
127
5,871
Historic day! Emergency physician Dr. Medell Briggs-Malonson receives the first-ever dose of the Pfizer #COVID19 #vaccine administered by UCLA Health.
3
50
34
297