Directing SphereLab @CUHKofficial. Postdoc @MPI_IS. PhD @Cambridge_Uni & @GeorgiaTech. Previous Intern @Google & @nvidia. All opinions are my own.
Joined May 2009
- Tweets1K
- Following799
- Followers2.6K
- Likes3.2K
🚀 Very excited to introduce TTR-Compressor -- A test-time regression framework for sequence compression! I particularly like this work for its fundamentally different perspective on sequence compression.
Unlike popular pooling and projection, TTR-Compressor formulates sequence compression as a test-time regression problem. Instead of directly aggregating tokens, we fit regression keys derived from the input tokens to learnable targets, using the fitted coefficients as the compressed representation.
This leads to a bi-level optimization with an exact closed-form solution to the inner test-time regression. The closed-form solution captures second-order inter-token correlations, allowing TTR-Compressor to reduce redundancy and better preserve distinctive information, going beyond conventional first-order aggregation in pooling and projection.
Strong gains in two applications:
✨ Block-wise sparse attention: +12.24% in RULER retrieval accuracy over attention-based compression.
✨ Visual token merging: Comparable multimodal performance with 55.6% fewer visual tokens than standard MLP merging.
🤔 Takeaway: Sequence compression can be viewed as input-adaptive regression rather than static aggregation. This provides new insights to building more efficient language and multimodal models.
📄 Paper: arxiv.org/abs/2610.31574
🌐 Project: spherelab.ai/ttr-compressor/
💻 Code: github.com/Sphere-AI-Lab/ttr…
Can LLMs manage their own context natively without any external harness? We are teaching LLMs the ability of Native Context Editing. Our method achieves considerable performance gain on popular long-horizon agentic benchmarks (Terminal-Bench, SWE-bench) while reducing token usage significantly.
🤔 This can be viewed as a form of verbalized context management, which leverages LLMs' inherent capabilities to optimize context. Our prior work, Verbalized Machine Learning (arxiv.org/abs/2406.04344), similarly explores context optimization to enhance downstream task performance.
Project: dengyl20.github.io/native-co…
Paper: arxiv.org/pdf/2610.25550
Code: github.com/dengyl20/native-c…
It is always inspiring to see great professors who genuinely love what they do and bring such passion to teaching.
The lectures of my new course "Principles of Deep Representation Learning" are now available at the Youtube site: youtube.com/playlist?list=PL… with new lectures continuing to be added there. I will open-source all the slides later in the textbook website: ma-lab-berkeley.github.io/de…
Weiyang Liu retweeted
🚀 Introducing DeepInfer, our in-house inference framework, now fully open source!
Powering DeepSeek-V4.1-Flash at scale with high throughput and low latency. We’re sharing our production stack to help the community lower compute costs, reduce energy use, and make AI more affordable.
🔗 github.com/deepseek-ai/DeepI…
1/4
🔥 Excited to share our new project on generating functional CAD mechanisms!
Our approach integrates CAD generation and physics-based simulation in a closed loop, allowing designs to be iteratively evaluated and refined for real-world functionality.
The goal is to move beyond CAD models that merely look plausible toward mechanisms that are 3D-printable, physically functional, and ready for practical use.
Check out our project page for more details: thegenlab.ai/funcad
Introducing FunCAD: Designing functional, working CAD mechanical mechanisms from a single line of text ⚙️
Everyone is hyped about GPT-6 Astra’s 3D demos, but building real mechanical assemblies from scratch tells a different story:
❌ Astra’s edge is just an interface — 3D fluency ≠ spatial & physical understanding. Fable beats Astra in physics, and blindfolding Astra barely matters.
❌ Not 3D reconstruction — zero reference images, just a text goal (e.g., "bring a back-row cup to the front") to invent CAD mechanisms that actually work.
To test this, we built FunCADBench (46 mechanism families, from clamps and transmissions to mechanical counters, alarm clocks, and coin sorters) and FunCAD-Harness, which translates abstract requirements into candidate-specific physical tests and uses simulation diagnostics to guide revisions.
What we found:
🙈 Blindfolding Astra barely matters: Agents spend 75–85% of their tool runtime in simulation and rarely inspect renders. Completely blinding Astra to build purely from code + physics logs still solves 20/46 tasks (vs. 22/46 with vision).
🏆 Fable beats Astra at 1/4 the cost: Because visual fluency ≠ physical reasoning, Fable solves 32/46 out of the box (vs. Astra's 22/46), leading on complex mechanisms.
💸 Cheaper models + retries win: Running Grok for a few extra attempts (with retained history) pushes it from 19/46 to 36/46—beating Astra at ~half the cost.
📉 Pass/fail isn't enough: With full diagnostic feedback—catching blocked steering, jammed clock hands, or tipping—Astra jumps from 8/46 to 22/46, but collapses to 0/46 under binary pass/fail.
🖨️ Sim-to-Real: We 3D-printed and assembled the AI-designed locks, counters, timers, and cup-fetching machines—they work in the real world! ✅
For more details, please check below👇
🌐 Demo & Page: thegenlab.ai/funcad/
📄 Paper: arxiv.org/abs/2610.26861
💻 Code: github.com/thegenlab/fun-CAD…
Work with Wenqian Zhang, @yangyi_huang_cn, Zigeng Xu, @Besteuler