Building the data stack for Physical AI at https://teapaw.org/t.co/vVh4lJnGvd. I admire people who are building things
Stockholm, Sweden
Joined November 2014
- Tweets741
- Following600
- Followers270
- Likes3.3K
Moritz Schiebold retweeted
Should Europe leave frontier AI to the Americans and the Chinese?
The idea has gained ground in Europe. It is a fundamental mistake.
Build or surrender: for now, the choice is still ours.
Full article below.
Moritz Schiebold retweeted
Ultra has been a close partner and customer of ours for a while now and they are just killing it! They are one of the most disciplined deployment focused modern robotics companies out there and an amazing example I hope a lot more companies follow
Robotics has a distribution problem. Today, Ultra is announcing $62M in funding to solve it, including a $50M Series A led by @hiFramework.
We believe that deployment is the bottleneck in robotics.
It’s relatively easy to make a robot look impressive in a controlled environment. It’s much harder to put a robot into the open world, make it useful, keep it running, earn the trust of the people working alongside it, and do that over and over and over again.
So that’s what we’re focusing on.
Moritz Schiebold retweeted
Robotics has a distribution problem. Today, Ultra is announcing $62M in funding to solve it, including a $50M Series A led by @hiFramework.
We believe that deployment is the bottleneck in robotics.
It’s relatively easy to make a robot look impressive in a controlled environment. It’s much harder to put a robot into the open world, make it useful, keep it running, earn the trust of the people working alongside it, and do that over and over and over again.
So that’s what we’re focusing on.
Moritz Schiebold retweeted
I wanted to share something I've been working towards since forever. I've finally built out a full slam + hand reconstruction pipeline that works IN REAL TIME at 30fps on a single rockchip 3588. All streaming directly to @rerundotio from a @BitRobotNetwork robocap. This is similar to what a Quest3 or Project Aria 2 provides, but opensource 🙂
I'm so excited I finally don't have to really on closed source software like the meta quest or others. I finally have a realtime performant pipeline that works on an edge computer, but importantly can also scale to a high performance one when needed.
As I'm working on finding more datasets to test my pipeline, I found a really cool one from Meta. This is Show3D and it does ego + exo data capture on a mobile capture rig, its focused on hand/object interaction in the willd.
I've so far registered it to the @rerundotio OSS catalog, but I think I found a few issues. The tracking confidence is more unstable then I would expect and the object often has low tracking confidence on large swaths of of the segment.
I'll spend more time on it and see if its a me error, or something with the data labels.
Also sadly, they only provide the stereo images, no IMU =/
Moritz Schiebold retweeted
How do you scale 2 million hours of ego data without sacrificing quality?
Earlier this week, we revealed the world’s largest egocentric dataset. Our frontier pipeline has also turned it into the most labeled, annotated, and human QA’ed robotics ego dataset in the world.
We’ve focused on some of the hardest problems in computer vision and ego data to ensure every clip contains SOTA action supervision directly helpful for training dexterous manipulation.
Moritz Schiebold retweeted
It's really cool how easily you can visualize complex datasets nowadays. Thanks to @rerundotio!
Moritz Schiebold retweeted
The LIBERO dataset shows up in many robot-learning benchmarks. Curious what the data behind the benchmark numbers actually look like?
We converted LIBERO to RRD, so you can explore the tele-operated demonstrations directly with Rerun.
Pick a task and watch the motion in 3D alongside camera views and robot states. Explore the data using our query APIs. Train on it using our dataloader.
Data and code 👇
Moritz Schiebold retweeted
@rerundotio'ing fruit flies to poke at object policies
too much fun!
Moritz Schiebold retweeted
Here it is! I rebuilt a visual iniertial multicamera pipeline in rust + python, alll with support for @rerundotio OSS data catalog + visualization.
code here - <github.com/rerun-io/examples…>
The core is rust to make sure it runs fast, but you can also easily call it from python.
I've been unhappy with pretty much all of the other vio pipelines out there, as none of the ones I know about hit the requirements I wanted
As part of my exo ego pipeline, I haven't yet found a VIO/Slam solution that hits all of the features I want. Now that agents make it super easy to write low level systems code, I've almost finished building a portable rust/python slam pipeline that works on both a 5090 AND a Rockchip-class computer.
To really make this work, I needed a good ground truth, and the wonderful work from Collabora and @mateodemayo on the monado slam dataset has been indispensable.
I registered it as a @rerundotio dataset in the OSS catalog to make it very easy to verify that the GT actually looks right and use it as an eval, and it has worked REALLY well.
I'll have more on the VIO pipeline here in the coming days
💯 this! We can only build our way out of this. Complaining or expecting the old guard to change is not a valid strategy at this point. That's why I'm so pumped to see @esra_robotics become a thing.
We can’t complain about Europe lagging in AI while dismissing every effort to catch up.
Surprise: no single initiative will fix everything. Progress has to start somewhere.
I'm increasingly bothered by people dismissing and joking about every effort as “not enough”. If you do that without contributing yourself or suggesting an improvement, then you're part of the problem.
Start supporting the people who are making an effort if you want something to change.
Moritz Schiebold retweeted
We have been working on an experimental rosbag2 storage plugin. Now you can try use rosbag2 to record directly to RRD, Rerun’s native file format.
ros2 bag record -s rrd
This way, your recording is both a rosbag and a Rerun recording. You can stream to the viewer while you record, register to a Rerun server when you’re done, and query and train directly from the bagfile. You can even add more layers later, and the original remains a bagfile.
This sits alongside our existing MCAP support and log SDK as one more way to get robotics data into Rerun.
RRD stores data as indexed, columnar chunks of Apache Arrow record batches. A query reads only the columns and time ranges it needs, and Arrow gives direct paths into tools like DataFusion and out to formats like Parquet. The same columnar access feeds Rerun’s experimental PyTorch dataloader, so you can train a model straight from your registered bags with no export step.
This is still experimental, but we are excited to make it available to test. Issues and PRs are welcome.
@OpenRoboticsOrg
github.com/rerun-io/rosbag2_…
Moritz Schiebold retweeted
Inspired by @bilawalsidhu’s videos, built first prototype combining 3D reconstruction with live camera at Shibuya Crossing.
@mapillary geotagged images + COLMAP + MVSAnywhere → 3D points
YOLO26s-seg + ByteTrack → live tracking of people and vehicles
Visualization @rerundotio
Moritz Schiebold retweeted
48 consistent camera feeds of me all from one casual video capture. This is 4DAnyone. I am testing it out over the holiday week, the goal is 4D Gaussian Splatting from a single camera.
Should out to @pablovelagomez1' for inspiring me to use Rerun to visualize my progress. Right now you see 6 cameras, but I can cycle through all 48.
I hacked together this visualization using ChatGPT's new Astra model in VS Code. Literally took less than 10 minutes.
More coming soon.
Learn more about 4DAnyone Here: 4danyone.github.io/
You can also try Pablo's 6 view 4DAnyone generator on Hugging Face here: huggingface.co/spaces/rerun/…
@jeremyparkphd @Scobleizer
#Computervision #3D #4DGS
Moritz Schiebold retweeted
Alright, I went ahead and built 4DAnyone into a proper @Gradio space. I thought it was cool enough that it deserved the effort and made sure to take advantage of our @rerundotio Gradio integration.
TLDR takes in a single video and is able to output multiview consistent videos. Its kinda incredible how good the quality is. Got it working on my 5090 + DGX Spark, so it works with <32GB cards =]
Sadly, I had to make this version only output 6 videos; otherwise, it would take way too long, but its able to go all the way up to 48 videos locally. Just takes like ~20-30 minutes.
No splatting on this one, as again it would just take too much GPU time that ZeroGPU won't easily support and will eat up all your quota
This also feels like the perfect fit for a new Gradio Workflow, something I'll look into at some point.
Inference time ~5-8 minutes on the space, sped things up in the video to keep it digestible
I got nerdsnipped by @krahets with 4D Anyone. I was working on part 3 of the arkitscenes blog post for @rerundotio, but I just couldn't help but try to bring in 4d splatting from a single video into rerun since we shipped splatting support.
Did a few things here
1. spent a bunch of time optimizing the DiT model so it would run on my 5090. Got memory from ~50gb -> less than 32gb. No need for enterprise cards or super slow dgx spark
2. more importantly, I got the runtime down to ~5 minutes for going from 1 video -> 16 views. This was nearly 40 minutes before on my 5090!
3. I also got a version of 4d splatting training working, its a mess and doesn't look great but hey it works!
Really really cool work, hoping to do more with it in the future when I'm a bit less scattered with all the cool things I want to work on 😅
Moritz Schiebold retweeted
Training for blog part 3 is coming along. Its been a while since I've done model training so this has been fun!
Moritz Schiebold retweeted
The great @patrickc isn’t wrong. But at the same time I feel like no generational company or exceptional founder ever failed because of notary meetings…
Yes, we need to work on bureaucracy. For sure. No, this is not the reason why Germany isn’t producing more world class startups.
From my pov it’s much more nuanced: first and foremost we need to upgrade our mentality: more ambitious input will mathematically produce more ambitious output. Basta.
And only then it’s about more private capital flowing into the ecosystem, talent acquisition, and generally speaking less bureaucratic burdens.
Met a German founder this week and asked him if all the stories one reads about the challenges of startups in Germany are exaggerated. "No, they're understated." Proceeded to describe spending a full day having a 90-page investment contract read to him (mandatory under German law; § 13 BeurkG) by a notary that then charged €30,000. That was for his first company. His second company, needless to say, was not incorporated in Germany.
Moritz Schiebold retweeted
PS: if you like the visuals on this post, I created a data catalog viewer for EgoSuite-Open100k built on @rerundotio. You can search, sort, and filter episodes. Download individual episodes and inspect the data.
Code: github.com/jonstephens85/ego…
