University of California, SD
Joined May 2014
- Tweets265
- Following426
- Followers4.1K
- Likes642
Pinned Tweet
🎥 New Talk: “Generative, Malleable, and Personal User Interfaces”.
This talk describes our work toward the long-held vision in the field of human-computer interaction.
Talk Link: youtu.be/MbWgRuM-7X8
We are committed to making it as real as we can.
Had a blast discussing Dynamic Task UIs with Simra @its_sshahid from @adaption_ai .
If you missed it, this article I published today
xitter.cf/HaijunXia/status/21133…
summarizes my view of the emerging interface paradigm with AI agents.
We spend a lot of time making models more capable, and far less on how people experience and shape that intelligence.
It’s central to Adaptive Interfaces at @adaption_ai. I’m excited to unpack this with @HaijunXia who has been rethinking the future of interfaces!
How should we approach generative user interface systematically?
Rima @YiningCao3 and I have been following the approach of identifying and designing design patterns. for tasks and interfaces. Chatlet addresses the task pattern of collaborative decision making. A Chatlet in a group chat gathers the context and generates the suitable UIs for the group to make decisions in a more visually informed way. Check it out!
Group decisions happen through conversations, but keeping track of everyone's ideas, preferences, and options isn't always easy.
We've been exploring the idea of Chatlet - an interface takes shape and keeps evolving alongside your conversations. You keep your natural conversations with your friends and colleagues (NOT an AI!), while Chatlet evolves to help everyone think, coordinate, and make decisions together.
We believe generative UIs shouldn't have to start from an explicit prompt. Instead, they could emerge from context: understanding what's happening, recognizing when an interface might be useful, and figuring out what form it should take.
This research project is in collaboration with Nathalie Henry Riche, Siân Lindley, and Hugo Romat from Microsoft Research @MSFTResearch , and @HaijunXia from UC San Diego.
We're now looking for early users to try Chatlet, we'd love your feedback!
🔗 Sign up to try Chatlet: chatlet-signup.fly.dev/
👀 Full demo and learn more: rrrima.me/blog/chatlet
I have always loved well-made explainer videos. They are engaging and educational, but used to take days and weeks to make...
With this #skill, every question can be answered with an interactive video and a scrolly essay. Super cool skill, give it a try!
A fun side project that helps with my hundred thousand whys.
One question → an illustrated scrolly essay + a video you can stop anytime to ask your own question. It answers with a new clip that fits into the video.
This came out of a question I keep returning to: which digital things do we actually want to be malleable? I love scrolly essays and explainer videos, but they never quite answer all my whys. So that's the one I wanted to bend. Had a lot of fun.
more examples: explorables.fly.dev
open source, as a Claude Skill: github.com/Rrrima/explorable…
--
voice-over: @ElevenLabs
music & sound effects: @StabilityAI
everything else: Claude Opus 5.5
--
question for this video: How to measure the IQ and EQ of dogs and cats? And how do the breeds rank?
AI makes us create more and do more, but think less and worse. Almost want to start a company with the name of “Thinking Human Lab”.
I’ve seen at least 6 AI try-on + shopping posts this weekend on this platform.
I wonder if that's a wrong use case of image generation?
An image model is designed to make a generated image look plausible and coherent.
A real try-on is to see what’s wrong with this fit.
Too tight at the shoulders or bad length?
If AI smooths over those imperfections to make the image coherent, it’s deleting the exact signal the shoppers are looking for?
We may look great in the images, but not so great in reality?
i’ve spent a ton of hours and tokens prompting @Muse to make me this custom fashion app but holy crap is it worth it:
- gave it my height, weight, measurements, and photos
- gave it my top 20 clothing stores and budget ranges
- it picks a “hero piece” from the catalog
- then builds a look around it (inventing other common pieces)
- i described everything i like and dislike and i keep iterating each time
- these are my “loves” saved category
Eric's messages may exaggerate a bit and be a bit misleading, but I agree with the direction he is taking: that traditional Graphical UI (GUI) is on the decline.
But UIs are not going away, it how we interact with both the physical and digital worlds. As long as there is information transmission, there is a UI.
If we consider user interface modalities broadly- visual/graphical, language, gestural, etc-we are clearly going through a shift from heavy reliance on GUI to language-driven UIs. No single modality will take over or entirely go away, but the redistribution is real, and GUI is on the decline.
Within GUI, WIMP (btw, P stands for pointer or pointing devices, not pull down as Eric said) UI is being impacted the most.
We need to think about non-traditional, non-WIMP GUIs. Most of the human-agent interfaces, even though generated, are still a continuation of WIMP, IMHO.
User interfaces are not going away.
Yes, humans will use computers alongside agents. When working with agents, we will need the ability to audit what agents do in a human-readable way.
Yes, some interfaces (ex: dashboards, in-chat UI for ephemeral tasks, etc) will be generated. Most important interfaces will be intentionally designed. We will want both deterministic and non-deterministic interfaces in different situations. We're thinking a lot about the best ways to help designers shape non-deterministic systems at Figma.
Yes, new input modalities like voice, VR/AR and BCI will change the way we use computers and introduce lots of new interaction paradigms. This is all very exciting :)
Overall, the surface area of design will continue to grow much broader. Design is the new code and the key differentiator for software moving forward. Expect more and more energy to go into design / craft in the years ahead... it will be how you win or lose.
Eric is right about plenty of things, but the market will prove him wrong on this one...
This is the kind of "AI Magic" I love. Not through delegation and instruction, but through direct interaction with intelligent and dynamic materials.
We have found a local optimum for chat interface; what more do we need than a text box for chat, after all
For many other tasks, shopping, browsing, and complex sensemaking work-rich visual, multimodal interfaces are necessary, and we have not yet found the design pattern for how to integrate them with chat.
Solaris is super cool. It starts to explore what I think of as interface physics.
Working directly with pixels rather than code seems to make it easier to preserve the physics of an interface—how things move, respond, transform, and relate spatially.
Unlike the physical world, where the laws of physics are given, the physics of interfaces can be invented and much more diverse.
Very curious to see how video generation models handle increasingly complex interactions, both within a scene and across scenes.
Today, we're sharing new research on Solaris, our first Interface World Model.
Solaris is a new kind of operating system that generates interactive interfaces frame by frame, in real time, with no code. We find that Solaris outperforms frontier LLMs when generating new interfaces, across structural similarity and information retention. Read more and request early access at the link below.
Haijun Xia retweeted
Interfaces that are generated, not coded.
Excited to finally share Solaris. Video is the universal interface. Eventually, all UI will be chat and gestures in, chat and video out.
Solaris opens new ways to build websites, apps and other interfaces. And most importantly, it gives new ways to train agents in much more dynamic environments.
Interfaces that are generated, not coded.
Solaris is our first step toward our vision of end-to-end neural software, where user interfaces are streamed directly onto your screen by a real-time video model, without any intermediate code or HTML/CSS. Today we're releasing a technical report and opening it up for limited testing by the community.
Amazing work by the team, combining the best of our research in general world models with Runway's HCI spirit.
AI promises great flexibility, yet every single AI workspace app looks the same. AI interfaces deserve more creativity.
A23. This is the current distribution. It's now worth some abstraction. Made a little tool. @Jason
polltheworld.fly.dev
Curious about what hardware changes are being planned here! Feel form factors are fundamentally designed for human I/O, which won't change much.
I've left OpenAI and the Codex team to build Blackstar: A new hardware company building the future of human-computer interaction.
We believe that software is solved. Building apps is now easy, but the next meaningful improvement in human-AI communication requires changing the OS & hardware. That's why we're building a new device entirely.
I'm also excited to announce our $12m seed round led by @AbstractVC, with participation from @naval, @SVAngel, @chapterone, and Timeless, among other amazing angels who've supported us from the old Alex days.
Congratulations to my PhD student, Rima (@YiningCao3), on receiving a Microsoft Research Fellowship!
Rima’s work pushes on a fundamental question: what should native interfaces for human–AI collaboration actually look like? Her research explores generative, malleable, and composable UIs that treat interfaces not as fixed artifacts, but dynamic materials that can adapt to context of human-AI collaboration! Check her work out at rrrima.me/ !!
Excited to see this exploration from @GoogleLabs! My lab has been exploring Generative & Malleable Interfaces for a while. Check out this project we did last year youtu.be/X3cf1U1dFGQ, and its research publication at ACM SIGCHI 2025 dl.acm.org/doi/10.1145/37065…
🚨NEW LABS EXPERIMENT 🚨
Introducing Disco, an experimental way to discover new generative AI features on the web. Our first feature, GenTabs, uses Gemini 3 to remix your open tabs into totally custom apps to help you get more out of the web.
Learn more and join the waitlist → labs.google/disco