Dial roughness down, crank metallic up, stack multiple attributes at once all in a single forward pass!

Need parametric control sliders? A tiny 2-layer MLP trained on synthetic objects can learn CLIP directions for different material properties for real-world images 🖼️

By traversing through CLIP embeddings of two material exemplars, we notice the effect of a smooth blending in the generated image 🖌️

Inspired by InstantStyle, we begin by injecting CLIP embeddings only into the Denoising UNet’s “material block”, which keeps geometry much more intact compared to previous methods.

Imagine a Van Gogh-style teapot turning into glass with one simple slider🎨 Introducing MARBLE, material edits by simply changing CLIP embedding! 🔗 marblecontrol.github.io/ 👏 Internship project with @prafull7, @markb_boss , @jampani_varun at @StabilityAI

I will be presenting today at #CVPR2024 ! Drop by our poster session to learn more about Continuous 3D Words for text-to-image generation: Time & Place: 19 Jun 5pm / Arch 4A-E P190 Project page: ttchengab.github.io/continuo…

Thank you for your time, and feel free to check out more details from our project page, paper, and try out our code! Project Page: ttchengab.github.io/zest Paper: arxiv.org/abs/2404.06425 Code: github.com/ttchengab/zest_co… Video Summary: youtube.com/watch?v=atG1Vvge… 8/8

Finally, ZeST can also be used jointly with other text-driven texturing techniques to perform texturing in 3D from a single image! 7/8

By using ZeST with segmentation models like SAM, we can also apply multiple materials onto multiple objects in a single image: 6/8

ZeST demonstrates tremendous capabilities in modeling not just materials from a flat surface, but also disentangling the reflections from the texture patterns and even modeling transparent materials! Here are some examples: 5/8

ZeST turns to a completely implicit approach. We adapt geometry (ControlNet) and exemplar (IP-Adaptor) guidance to isolate and transfer material from the exemplar to the input image, while applying foreground decolored shading and diffusion inpainting for illumination cues. 4/8

Today, with my collaborators @prafull7 (MIT CSAIL), @jampani_varun (@StabilityAI ), and my supervisors Niki Trigoni and Andrew Markham, we share with you ZeST, a zero-shot, training free method for image-to-image material transfer! Project Page: ttchengab.github.io/zest/ 1/8