Ph.D. Student @mldcmu, @cmuneurosci | Previously undergrad @berkeley_ai.
Joined May 2013
- Tweets21
- Following181
- Followers51
- Likes30
Pinned Tweet
📅 Where to catch BrainNRDS at @CVPR
🗣️ Oral — June 13, 13:15-13:30
Session 2B: Human Motion, ExHall A2
🖼️ Poster #220 — June 13, 16:00-18:00
ExHall D
Stop by to chat about decoding motion from fMRI and video generation! 🧠🎥 #CVPR2025
1/6 🚀 Excited to share that BrainNRDS has been accepted as an oral at #CVPR2025!
We decode motion from fMRI activity and use it to generate realistic reconstructions of videos people watched, outperforming strong existing baselines like MindVideo and Stable Video Diffusion.🧠🎥
Jacob Yeung retweeted
At #ECCV2026 presenting DetPO.
TLDR: don't feed a VLM few-shot examples at inference. Use them to find what it gets wrong, then rewrite the class prompt to fix them. No fine-tuning.
Poster #211, Session 5. w/ @NeeharPeri @Matvezy Shruti Jain, John Galeotti, @RamananDeva
Error-driven prompt optimization makes VLMs better at few-shot object detection, outperforming general-purpose prompt optimizers like GEPA. Stop by to see "DetPO: In-Context Learning with Multi-Modal LLMs for Few-Shot Object Detection" at ECCV poster session 5 poster #211!
Jacob Yeung retweeted
World-action models typically imagine the future in RGB – but are pixels really the right representation for robotics?
Our bet is no: RGB spends capacity on fine-grained details and variation that are often irrelevant for robot policies.
DINO features, point tracks, and depth capture more useful features like semantics, motion, and geometry.
But no single modality captures everything — how can we effectively combine them?
We introduce ✨ModAR✨, which predicts the future one modality at a time, with each prediction informing the next. We train from scratch and find that this formulation performs best.
Our 30.1M scratch-trained model even outperforms a 6B video-model-initialized model finetuned on the same data!
🧵 [1/8]
Jacob Yeung retweeted
Before AI can generate professional videos, it needs to see like a professional.
We spent a year with 100+ content creators teaching AI to describe video like a filmmaker would.
Introducing CHAI: Critique-based Human-AI Oversight for Building a Precise Video Language [CVPR'26 Highlight, Top 3%].
Try prompting a video generator for a dolly zoom, dutch angle, point of view, or camera roll. Most fall back to the same bland defaults: a push-in, a level shot, a third-person view. Why? These techniques require a language of cinema that current models rarely speak.
We built that language:
1️⃣ Precise specification: 5-aspect structured captions co-designed with professional cinematographers covering subject, scene, motion, spatial, and camera dynamics
2️⃣ Scalable oversight: LLMs draft captions, humans critique what's wrong and how to fix it
3️⃣ Post-training recipes: Qwen3-VL-8B surpasses Gemini-3.1 and GPT-5
4️⃣ Video generation: fine-tuned Wan follows 400-word cinematic prompts with precise control
Here's how each works 🧵
Work led by CMU and Harvard with @chancharikm, @du_yilun, and @RamananDeva.
📄 Paper: huggingface.co/papers/2604.2…
🌐 Site: linzhiqiu.github.io/papers/c…
Jacob Yeung retweeted
Pretrained ViTs like DINOv2 or CLIP are great, but they produce fixed, generic representations that encode the most salient visual concepts (e.g., "cat").
In human vision, prior priming with language changes how people parse an image. We believe visual encoders should do the same
🚨 Introducing Steerable Visual Representations, a new family of visual features you can steer with text towards specific visual concepts.
Jacob Yeung retweeted
Introducing Vero, the strongest fully open RL recipe for training next-generation visual reasoners.
From charts to spatial to open-ended tasks, Vero sets a new bar.
• sota 8B VLM across 30 benchmarks
• +4.4 avg over four base models (30 evals)
• beats prior RL datasets
🧵👇
Jacob Yeung retweeted
In an exciting collaboration with MSR, we ask a simple question: do LLMs actually behave like rational agents? We test whether an LLM’s stated elicited beliefs can be treated as its true beliefs by checking decision-theoretic coherence. arxiv.org/abs/2602.06286
Jacob Yeung retweeted
Introduce CRISP, a real-to-sim pipeline that recovers human motion and simulatable scene geometry from monocular video!
CRISP builds contact-faithful 3D scene for simulation - 8× fewer sim failures, +43% faster sim, and improves human motion!
Interactive demos👉: crisp-real2sim.github.io/CRI…
Exciting collaboration w/ @JiashunWang @jefftan969 @_Tsukasane @ Jessica Hodgins @shubhtuls @RamananDeva
Jacob Yeung retweeted
Introducing Any4D, a unified transformer for fully feed-forward, dense, metric-scale 4D reconstruction from flexible inputs!
Any4D regresses per-pixel motion + geometry across frames in one pass — 15× faster, 2–3× more accurate reconstructions ⚡📈
Details + code below 👇
Exciting collab with @Nik__V__ @YuchenZhan54250 Tanisha Gupta @akashshrm02 @smash0190 @RamananDeva
Jacob Yeung retweeted
Very excited to share our work on naturalistic audiovisual processing in the human brain 🧠🎬!!
Two cortical mechanisms of audiovisual processing in the human brain biorxiv.org/content/10.1101/… #biorxiv_neursci
Jacob Yeung retweeted
Meet MapAnything – a transformer that directly regresses factored metric 3D scene geometry (from images, calibration, poses, or depth) in an end-to-end way. No pipelines, no extra stages. Just 3D geometry & cameras, straight from any type of input, delivering new state-of-the-art results 🚀
One universal model enables SoTA for:
🔥 Mono Depth Estimation
🔥 Multi-View SfM
🔥 Multi-View Stereo
🔥 Depth Completion
🔥 Registration
… and many more possibilities! – plus everything is metric 🎯
We release code for data processing, training, benchmarking & ablations – everything Apache 2.0!
Details & Links 👇
Jacob Yeung retweeted
Excited to share recent work with @kaihuac5 and @RamananDeva where we learn to do novel view synthesis for dynamic scenes in a self-supervised manner, only from 2D videos!
webpage: cog-nvs.github.io
arxiv: arxiv.org/abs/2507.12646
code (soon): github.com/Kaihua-Chen/cog-n…
Jacob Yeung retweeted
1/6 Retrieval is supposed to improve generation in RAG systems.
But in practice, adding more documents can hurt performance, even when relevant ones are retrieved.
We introduce RAGGED, a framework to measure and diagnose when retrieval helps and when it hurts.
Jacob Yeung retweeted
This was a really fun and exciting workshop #CVPR2025! Huge thanks to all the speakers, organizers and reviewers @CVPR!
We hope to be able to release the video recordings soon!
Join us for the 4D Vision Workshop @CVPR on June 11 starting at 9:20am!
We'll have an incredible lineup of speakers discussing the frontier of 3D computer vision techniques for dynamic world modeling across spatial AI, robotics, astrophysics, and more.
4dvisionworkshop.github.io/
1/6 🚀 Excited to share that BrainNRDS has been accepted as an oral at #CVPR2025!
We decode motion from fMRI activity and use it to generate realistic reconstructions of videos people watched, outperforming strong existing baselines like MindVideo and Stable Video Diffusion.🧠🎥
5/6 We also show that dynamic information isn’t just useful for generation. It is key to understanding brain activity as well.
We observe that video models predict brain responses to dynamic scenes better than image models, especially in visual and somatosensory cortices.
6/6 This was an amazing collaboration with co-authors Andrew Luo, @GabrielSarch, @maggiehende, @RamananDeva, and @TarrLab.
🔗Paper: arxiv.org/abs/2406.02659
🔗Project page: brain-nrds.github.io/