What is 3D Gaussian Splatting? In short, it's a method that can give robots a visual and spatial foundation for their 3D memory. In detail, 3D Gaussian Splatting (3DGS) turns photos or video captured by a robot into a photorealistic 3D reconstruction of its surroundings that it can view from new angles. This helps robots refer back to places and objects that are no longer in view when a task calls for it. Here is how photos become a 3D scene for robots: 1. The system matches details across photos/videos to estimate where the cameras were and build a rough map of 3D points. 2. Those points become tiny, soft blobs called Gaussians. Each has a position, shape, color and opacity. 3. The system renders the room and checks it against the photos. The Gaussians shift, stretch and change color until the views match more closely. Some split to capture finer details, and others are removed. 4. Once the Gaussians have been adjusted to match the photos, the scene can be viewed from new angles. The GPU projects and blends them fast enough for you to look around in real time. The result shows a scene from different viewpoints. However, it doesn’t explain its physics or predict motion. And making it a robot’s memory requires another layer. We've made a comprehensive guide about this hot topic and how Gaussian Splatting becomes Gaussian Memory. A must-read about embodied memory: turingpost.com/p/3d-gaussian…

How much of the past in agent memory should enter the present? Once we decide what to keep, we need to work out how the agent can access it. @jess_weng18, an education engineer at @LangChain, showed several ways to manage this in Deep Agents: 1. A large tool result can be saved to a file. The model receives a short preview and a path to read more as needed. 2. A skill appears as a name and description, with full instructions loaded only when the task calls for them. 3. A subagent can work through a focused problem in its own context. 4. As the conversation grows, older material can be condensed while recent messages stay unchanged. And the most important thing: Continuity depends on the architecture around the model: where information lives, how long it survives, and how a later run finds it. Jess Weng even showed that a persistent memory file could be read by another model. But a condensed memory is easier to use and also easier to misuse because the agent may miss important details. So → Agents need to recognize when the summary is insufficient, and we still need to figure out how much context and experience we can leave out. Here I explain six more perspectives on how to turn an agent’s experience into better work and build agent memory. They are practical and so really worth your attention.: turingpost.com/p/agent-memor…

It's amazing how a few JEPA studies can point the way toward answering a question: What does a machine need to predict well enough to plan what to do next? JEPA's general point is to predict what matters without recreating every pixel. I explored 3 variants - H-JEPA, DeepJEPA and RoboJEPA, and each approaches the question from a different angle: - Planning at distinct levels, from fine local details to big picture - Learning when a prediction needs a second look (more computation) - and Scaling up JEPA’s predictor to make better predictions, and so better actions I’d like to see a robot that brings these ideas together and see how that would change its actions. There’s a lot more to explain about these three JEPAs, so I go through each one in this video: youtube.com/watch?v=YPP_qApE…

I recommend joining an upcoming session from @Rubrik on October 21 → When Agents Go Off-Script It's a technical deep dive for security, IT and AI platform teams that need to know how to get agents back on track when they violate policy. You’ll learn how to: - Detect an agent sharing confidential deal values in a live Salesforce scenario - Trace the violation to the agent, action, and data involved - Restore your environment to a known-good state - Bring Rubrik Agent Cloud’s governance approach to your own deployments Date: October 21, 10 AM PT Don't forget to register here: engage.rubrik.com/register/7…

Text generation now has a third option. Beyond autoregressive and diffusion models. @Stanford and @togethercompute proposed Next-Concept Modeling (NCM) that generates text through a hierarchy of compressed codes, building from a coarse representation to finer detail. These codes are "concepts" in the name - learned representations, not explicit ideas or human-readable outlines. ▪️ NCM works in 2 parts: 1. It starts with a compressed representation. A multiscale residual VQ-VAE (Vector Quantized Variational Autoencoder) learns to turn text into several levels of discrete codes and reconstruct text from them. Level 1 is brief and coarse, and higher levels add what the reconstruction still lacks. 2. Then it adds finer detail. A Transformer generates those codes from coarse to fine, one level at a time, predicting all codes in each level in parallel. An important note: NCM adds detail at successive scales, doing less repeated work than diffusion models and skipping teacher–student distillation. In tests, NCM showed: - 10–31× faster generation than the strongest diffusion baselines, with comparable or better quality. - Higher Sudoku accuracy than the tested autoregressive and diffusion, using fewer model evaluations. ~4–5× fewer training tokens than strong diffusion baselines. That's for small models and specific benchmarks. Compares at large language-model scale are not established yet. Text gen needs this breath of fresh air, and all new ideas are valuable. The NCM approach makes it cheaper, with fewer steps and less work are needed.

Harness is dead, long live the harness? I went to @aiDotEngineer NYC and heard some very strong opinions about harnesses. Some engineers are removing parts of theirs because the models no longer need them. Others are building systems that let an agent work for days. Of course, my immediate instinct was to sort things out and understand what is happening with harness (and if it’s really dead). So many questions: - What is an agent actually doing during those days? - How does it remember what happened without turning an earlier guess into a fact? - And if it can build its own tools, what are we letting it change? This episode starts with a coding agent in about 25 lines of code, and then I drew insights from @_philschmid, @dillonrolnick, @aparnadhinak, @akshat_b, and a lot from @jediahkatz who had an amazing talk. It came out as a full guide with everything you need to know about the current state of harness Watch the full guide on YouTube: youtube.com/watch?v=pEHfZUra…