Sudeep Pillai (@spillai) header
Sudeep Pillai (@spillai) profile picture

Sudeep Pillai

@spillai

Buikding visual intelligence @vlmrun, prev @spc, @ToyotaResearch, CS PhD @MIT CSAIL #girldad

 San Francisco, CA
vlm.run
Joined August 2008
  • Tweets2.4K
  • Following2.9K
  • Followers2.9K
128Media
Photo by Sudeep Pillai (@spillai)Photo by Sudeep Pillai (@spillai)Photo by Sudeep Pillai (@spillai)Photo by Sudeep Pillai (@spillai)Photo by Sudeep Pillai (@spillai)Photo by Sudeep Pillai (@spillai)Photo by Sudeep Pillai (@spillai)Photo by Sudeep Pillai (@spillai)Photo by Sudeep Pillai (@spillai)Photo by Sudeep Pillai (@spillai)
Sudeep Pillai retweeted
Jeremy Park, PhD (@jeremyparkphd) profile picture
Jeremy Park, PhD
@jeremyparkphd3h
I made a computer vision tool for basketball shooting practice! I grew up playing basketball and stopped my sophomore year of college when I started dance. I have missed basketball a lot recently, so this past Sunday I bought a ball and shot around on a nice North Carolina fall afternoon. I probably haven’t touched a basketball in months before this, but hey, I can still make 6 shots in a row 😎 For this first demo, I just wanted to show that we can track makes and misses. I also like the hot streak indicator after 3 in a row and the little fire animations. This type of application could help players understand their shooting habits and work to improve their shooting percentage. I’m using ViTPose+ Large for pose estimation to track the shooting motion and SAM 3.1 to segment the ball and the hoop, both models accessed through the @vlmrun Gateway. Would love to hear what you think! The code is open-source on GitHub: github.com/jeremyipark/visio…
5
1
18
761
Sudeep Pillai retweeted
VLM Run (@vlmrun) profile picture
VLM Run
@vlmrunOct 29
New: EmbeddingGemma 2 on the @vlmrun Gateway. One model for text, image, document, video, and audio embeddings. 740M parameters, 768 to 128 dimensions, $0.01 per 1M tokens.
EmbeddingGemma 2, now on VLM Run Gateway. Text, images, documents, video, and audio in one vector space.

ALT EmbeddingGemma 2, now on VLM Run Gateway. Text, images, documents, video, and audio in one vector space.

One embedding space: a map of twelve inputs, each a dot. An invoice image and an invoice PDF sit together with the text an invoice; two different robot videos sit together with the text robot arms; a video about a bakery, a photo of a vintage car and an audio clip each sit next to their text. Images, video, audio, PDFs and text from a live EmbeddingGemma 2 run.

ALT One embedding space: a map of twelve inputs, each a dot. An invoice image and an invoice PDF sit together with the text an invoice; two different robot videos sit together with the text robot arms; a video about a bakery, a photo of a vintage car and an audio clip each sit next to their text. Images, video, audio, PDFs and text from a live EmbeddingGemma 2 run.

Model card: one multimodal embedding model. Text, code, image, document, video, and audio, searched together in one vector space. $0.01 per 1M tokens, 740M total params, 768 to 128 dims picked per request, Apache 2.0 license.

ALT Model card: one multimodal embedding model. Text, code, image, document, video, and audio, searched together in one vector space. $0.01 per 1M tokens, 740M total params, 768 to 128 dims picked per request, Apache 2.0 license.

Get started in a few lines of code: Python calling client.embeddings.create with model google/embeddinggemma-2, an image input, and prompt_name Document, printing 768 dims and 288 tokens; commented lines show the same call with an audio file (speech.wav), a PDF (report.pdf) or a video (clip.mp4). From your terminal: uvx vlmrun gw embed for an image, a video clip, and a text query. Sign up for a free API key at app.vlm.run.

ALT Get started in a few lines of code: Python calling client.embeddings.create with model google/embeddinggemma-2, an image input, and prompt_name Document, printing 768 dims and 288 tokens; commented lines show the same call with an audio file (speech.wav), a PDF (report.pdf) or a video (clip.mp4). From your terminal: uvx vlmrun gw embed for an image, a video clip, and a text query. Sign up for a free API key at app.vlm.run.

1
2
6
197
Sudeep Pillai retweeted
Jeremy Park, PhD (@jeremyparkphd) profile picture
Jeremy Park, PhD
@jeremyparkphdOct 7
I made a computer vision demo for detecting full range of motion in pull-ups! Like my previous chin-ups demo, I'm using ViTPose+ Large through the @vlmrun Gateway for pose estimation. I'm using the vertical displacement of the head as well as the elbow angles to detect full range of motion. In the video, you'll see that the reps where I intentionally didn't go high enough, or started with bent elbows, did not count as full reps. The reps where I started from a dead hang and went over the bar did count as a full rep. Of course, everyone's pull-ups may look different, and we can calibrate the thresholds to fit each person better. In short, this could be used to help people understand and improve their pull-ups. Let me know what you think! The code is open-source on GitHub: github.com/jeremyipark/visio…
20
42
5
551
31,163
Sudeep Pillai retweeted
VLM Run (@vlmrun) profile picture
VLM Run
@vlmrunOct 28
New: Our System One API can now extract JSON fields, fast! Text, numbers, dates, and lists, each read in parallel and returned typed, or null when it's missing. Up to 3.7x faster than constrained JSON schema extraction, and up to 3.3x faster than JSON mode, on the same model. Support for native PDFs (up to 16 pages), images, and text inputs.
Bar chart of p50 latency to extract 10 invoice fields with Gemma 4 26B, /chat/completions vs /systemone. Text: System One 308 ms, JSON Schema 1,132 ms, JSON Mode 1,012 ms. Two-page PDF: 410 ms, 1,260 ms and 1,104 ms.

ALT Bar chart of p50 latency to extract 10 invoice fields with Gemma 4 26B, /chat/completions vs /systemone. Text: System One 308 ms, JSON Schema 1,132 ms, JSON Mode 1,012 ms. Two-page PDF: 410 ms, 1,260 ms and 1,104 ms.

2
2
1
5
486
With our new @typesafeai-compatible SystemOne API, you can now extract details like dates, numbers, and lists from images or PDFs up to 3.5x faster (than json-schema mode), with the same VLM backbone! LLM developers have been working with JSON mode and structured outputs (aka JSON-schema mode) for years, but these can be highly inefficient for simpler vision tasks like invoice extraction. We’ve been working hard to bring fast decision-making and field extraction to our SystemOne API, and this is an early preview of the latencies you can expect from the new interface. Infra is one of those things where principled decisions made early start to compound. Expect to see a lot more on this, especially low-latency image and video streaming support for our SystemOne API, in the coming weeks!
VLM Run (@vlmrun) profile pictureVLM Run
@vlmrunOct 28
New: Our System One API can now extract JSON fields, fast! Text, numbers, dates, and lists, each read in parallel and returned typed, or null when it's missing. Up to 3.7x faster than constrained JSON schema extraction, and up to 3.3x faster than JSON mode, on the same model. Support for native PDFs (up to 16 pages), images, and text inputs.
Bar chart of p50 latency to extract 10 invoice fields with Gemma 4 26B, /chat/completions vs /systemone. Text: System One 308 ms, JSON Schema 1,132 ms, JSON Mode 1,012 ms. Two-page PDF: 410 ms, 1,260 ms and 1,104 ms.

ALT Bar chart of p50 latency to extract 10 invoice fields with Gemma 4 26B, /chat/completions vs /systemone. Text: System One 308 ms, JSON Schema 1,132 ms, JSON Mode 1,012 ms. Two-page PDF: 410 ms, 1,260 ms and 1,104 ms.

1
1
6
242
Sudeep Pillai retweeted
VLM Run (@vlmrun) profile picture
VLM Run
@vlmrunOct 26
Run and compare OCR models side by side on the @vlmrun OCR Playground. For models with grounding, hover any extracted value to see exactly where it came from on the page. Models Supported: PP-OCRv6, LightOnOCR-3-4B, dots.mocr, PaddleOCR-VL-1.6, GLM-OCR, DeepSeek-OCR-2 and more.
1
4
1
7
351
Pretty cool release of strongly-typed object detection with our new SystemOne API on the @vlmrun gateway!
VLM Run (@vlmrun) profile pictureVLM Run
@vlmrunOct 21
We just added @typesafeai's native Image and Box support to our System One API. Pass an Image as the state, ask Box questions, get the region back as four edges. Same TypeSafe SDK: point it at gateway.vlm.run/typesafe.
Image from VLM Run (@vlmrun)
Image from VLM Run (@vlmrun)
1
5
294
Sudeep Pillai retweeted
VLM Run (@vlmrun) profile picture
VLM Run
@vlmrunOct 21
We just added @typesafeai's native Image and Box support to our System One API. Pass an Image as the state, ask Box questions, get the region back as four edges. Same TypeSafe SDK: point it at gateway.vlm.run/typesafe.
Image from VLM Run (@vlmrun)
Image from VLM Run (@vlmrun)
1
2
1
6
474
Sudeep Pillai retweeted
VLM Run (@vlmrun) profile picture
VLM Run
@vlmrunOct 14
Run and compare open-weight OCR models on the new @vlmrun OCR Playground. Auto-eval and compare costs between several VLMs including PP-OCRv6, GLM-OCR, PaddleOCR-VL 1.6, DeepSeek-OCR-2, and the new LightOnOCR-3! Try the OCR Playground on your own data: chat.vlm.run/#ocr
Image from VLM Run (@vlmrun)
2
5
30
2,674,653
We're heading into a new era of sports analytics, where any cricket enthusiast or aspiring athlete can analyze their own personal videos and start building an AI-powered personal coach. I spent about 10 minutes fetching clips, extracting bowling form with the @vlmrun gateway and comparing bowling traces from @Jaspritbumrah93, @malinga_ninety9 and others.
2
6
2
62
3,871
Cricket, golf, or any sport where form is everything - I think this will soon get gamified, and young athletes will hone their skills far faster than we did. Here’s the whole thing in one frame: Run-up on the left, release right of centre, follow-through on the right.
Still of four cricket bowling actions: twelve frozen poses per bowler, running in from the left facing right, through release, to follow-through on the right. Sampled denser towards release.

ALT Still of four cricket bowling actions: twelve frozen poses per bowler, running in from the left facing right, through release, to follow-through on the right. Sampled denser towards release.

1
2
249
Everything on top of this is green-field. Aligning poses so two actions can actually be compared. What buys you control, speed, accuracy; what costs you injuries, and where the leverage is. The whole extraction is < 20 lines. Docs: docs.vlm.run/gateway
A dark code card titled ViTPose, Python. Eighteen lines of Python point the OpenAI client at the VLM Run gateway, send a public video URL with the pose method, and print the frame and detection counts and the first keypoints. Real output is shown as comments underneath: 501 frames, 5609 detections.

ALT A dark code card titled ViTPose, Python. Eighteen lines of Python point the OpenAI client at the VLM Run gateway, send a public video URL with the pose method, and print the frame and detection counts and the first keypoints. Real output is shown as comments underneath: 501 frames, 5609 detections.

2
184
Sudeep Pillai retweeted
VLM Run (@vlmrun) profile picture
VLM Run
@vlmrunOct 7
September changelog. Lots of exciting updates last month, and lots more to come including new guides and cookbooks.
VLM Run September 2026 changelog: gateway beta with 20+ visual models; new System One (s1) API, TypeSafe-compatible, image inputs; TypeSafe SDK works by swapping base_url; s1 engines DiffusionGemma-26B, Gemma4-26B, Qwen3.8-27B; WebSocket for s1 decisions; new gateway docs.

ALT VLM Run September 2026 changelog: gateway beta with 20+ visual models; new System One (s1) API, TypeSafe-compatible, image inputs; TypeSafe SDK works by swapping base_url; s1 engines DiffusionGemma-26B, Gemma4-26B, Qwen3.8-27B; WebSocket for s1 decisions; new gateway docs.

1
1
5
219
VLMs can be surprisingly accurate and token-efficient on the right tasks. In a bake-off between VLM-based document classification vs. OCR-based classification using our new @typesafeai-compatible SystemOne API, we're seeing over ~7x token-efficiency! VLM: 372 tokens/page, fixed. OCR->Text: 2,002 median, 19,159 on the worst page.
6
25
1
205
11,095
more replies
The tail is what breaks at scale. p90 latency, end to end: - VLM: 209ms (+9% over its own p50) - OCR->Text: 1453ms (+168%) You can size a batch for 372 tokens a page. You can't size one for a distribution with a 19k tail.
1
1
405
Try it out today: `uvx vlmrun gw systemone --help` Docs: docs.vlm.run/gateway/system-…
System One - VLM Run

System One - VLM Run

Typed, calibrated decisions over text, JSON, and images in a single read

docs.vlm.run
3
363
Sudeep Pillai retweeted
Jeremy Park, PhD (@jeremyparkphd) profile picture
Jeremy Park, PhD
@jeremyparkphdSep 30
I made a computer vision tool for evaluating deadlift form! I've shared a few deadlift demos in the past. This new demo includes some new models, accessed through the @vlmrun Gateway: - SAM 3.1 to segment the barbell weight plate - ViTPose+ Large to measure the hip hinge angle, which serves as a backup for segmenting the reps. The pose data can definitely be used more later. - Gemma 4 26B through the Gateway's TypeSafe-compatible API to output the probability of the back being rounded or not. The predictions line up well with how I intentionally performed each rep. Of course, there are many caveats to what is good form, as it depends on the individual. All said, this has the pieces in place to quickly make adjustments to fit the individual better. In short, a tool like this gives people data to assess their form and improve over time. Let me know what you think! The code is open-source on GitHub: github.com/jeremyipark/visio…
51
126
11
1,146
92,565
VLMs supported: - @googlegemma diffusiongemma-26B-A4B-it (default) - @googlegemma gemma4-26B-A4B-it - @Alibaba_Qwen Qwen3.8-27B - @Alibaba_Qwen Qwen3.5-0.8B
1
3
694
Docs: docs.vlm.run/gateway/system-… Models: docs.vlm.run/gateway/typesaf… Streaming: docs.vlm.run/gateway/typesaf…
System One - VLM Run

System One - VLM Run

Typed, calibrated decisions over text, JSON, and images in a single read

docs.vlm.run
6
637
Load more