[email protected]. Do vision pretaining and architecture design for lfm3 vl. Democratize on-edge intelligence.
Joined July 2022
- Tweets29
- Following28
- Followers49
- Likes4
Glad you like the model. Working on better vision pre-training research for producing even better lfm model.
1
57
Tianshu Yu retweeted
Adding vision to a language model is more than one extra modality. With large images or long video, the vision encoder can use more compute than the language model itself, so efficient models need a different design. Our VLM team, including @rshubert98, @TianshuYu6, @sina_semnani, and @iamsamstevens, explains how they teach LFMs to see, from mid-training to the custom vision encoder built for LFM3.
Tianshu Yu retweeted
The LFM3-VL-2B is the best model for its size for video understanding, quite usable compared to all other models of that size for checking security camera footage for example.
Our omni model was not trained with thinking capability as our vl model. We will pushing for unification in the next onni release.
2
31
No, it’s not public. I wrote it for training so it only work on AMD gpu. Won’t work on any device that you would use. My point is that if you want the true speed of the model you need to profile the inference engine you use. We are going to release a hyper optimized inference engine for lfm3 vl model on only mac book soon.
1
22
If you run on llama.cpp, it will be much faster. A lot of inference engine, doesn’t have a good conv2d varshape kernel, which is critical part of the vision encoder of lfm3. I wrote customized fused kernel for training.
1
38
Tianshu Yu retweeted
LFM3-VL-2B vs Gemma 4 E2B on 50 real scanned receipts.
LFM3 got 41 of 50 receipts fully right. Gemma got 28. 50 receipts from SROIE, same prompt, JSON out, one 5060 Ti 16 GB.
- LFM3-VL-2B: 95% of fields right, 41/50 receipts perfect, 46 s
- Gemma 4 E2B: 87% of fields right, 28/50 receipts perfect, 62 s
Across both tests: Gemma reads messy chat-style text a bit better (95% vs 90%). LFM3 reads receipts better, and was faster both times.
@liquidai @googlegemma
Introducing 𝙻𝙵𝙼𝟹: our new generation of Liquid Foundation Models. With a new tri-hybrid architecture and new multimodal encoders, the LFM3 family is the ideal model suite for specialized intelligence.
> 6 model instances, across text, audio, vision, and omni modalities, generative and decision models.
> 𝙻𝙵𝙼𝟹-𝟷.𝟽𝙱: text, 256k context window.
> 𝙻𝙵𝙼𝟹-𝚅𝙻-𝟸𝙱: vision-language model with video-input support for spatial understanding
> 𝙻𝙵𝙼𝟹-𝙰𝚞𝚍𝚒𝚘-𝟸𝙱: tool calls, voice chat, reasoning in 18 languages.
> 𝙻𝙵𝙼𝟹-𝙾𝚖𝚗𝚒-𝟸𝙱: an omni model combines vision, audio, and language.
> 𝚍𝟷-𝟸𝙱 and 𝚍𝟷-𝙾𝚖𝚗𝚒-𝟸𝙱: decision models with typed outputs and calibrated probabilities in a single pass. Omni adds audio input.
> fine-tune on 1 GPU.
> a new tri-hybrid architecture
> architecture optimized for 14 processor classes across GPUs, CPUs, and NPUs.
> 𝙻𝙵𝙼𝟹-𝚅𝙻-𝟸𝙱 is a beast. It performs competitively with Gemma 4 12B (6x larger), and Gemini 3.5 Flash Lite, and GPT-5.6 Luna, with 1.9B parameters.
> 𝙻𝙵𝙼𝟹-𝟷.𝟽𝙱 is #1 on the Pareto frontier of device intelligence 𝚙𝚒𝚙𝚎𝚝𝚝𝚎.𝚕𝚒𝚚𝚞𝚒𝚍.𝚊𝚒
Every modality for specialized intelligence. Pick the one that fits your use case, customize it, and run it on-device.
1/ 🧵
Tianshu Yu retweeted
pointed lfm3-vl-2b at the whole san fernando valley overnight. 16,692 aerial tiles, one rtx 3060, 5.5 hours.
it found 62,873 pools and drew a box around each one.
not a crazy idea, just “find the pools”. it misses about 1 in 5 and boxes a roof now and then. but it’s a 2B model with zero finetune.
imagine what you can do with this. palantir at home lmao
2/ LFM3-VL-2B
64 slots x 16K. ~143 tok/s on one request, ~583 tok/s total with all 64 describing images at once. 11.5GB VRAM.
config:
llama-server
-m LFM3-VL-2B-Q8_0.gguf
--mmproj mmproj-LFM3-VL-2B-F16.gguf
-ngl 99 -c 1048576 -fa on --jinja
-np 64
--cache-type-k q8_0 --cache-type-v q8_0
-b 2048 -ub 1024
--temp 0.1 --top-p 1.0 --top-k 50 --min-p 0.0 --repeat-penalty 1.1
--reasoning on --no-reasoning-preserve
Write your own on phone serving engine or wait for the pull request for llama.cpp to be merged
1
1
29
Crazy. How did you fit the high res map into the model? I don’t remember putting ton of these satellite data during pre-training. Crazy OOD performance.
1
1
148
The performance gap between audio model and omni model is very small. The gap between vision model and omni, text model and omni are much bigger. We are working unifying those modality into the same model better. It’s our first attempt.
1
2
34