Unity Catalog is the industryโ€™s only universal catalog for data and AI.

Joined July 2024
โ€œWhat were our earnings last quarter?โ€ If you get different answers, the KPI is not governed. Metric views define a KPI once in Unity Catalog so humans and agents report the same value. This post shows how to register a metric view as a Unity Catalog securable, add agent metadata, and query it with MEASURE in @ApacheSpark. Check it out ๐Ÿ‘‡ unitycatalog.io/blogs/uc-metโ€ฆ #UnityCatalog #MetricViews #DataGovernance
27
๐Ÿ“ฃ Join us for the next Unity Catalog Community Sync! Weโ€™ll cover what is in flight for the upcoming Unity Catalog v0.8.0 release, including local S3 support, walk through the new UI preview, and open the floor for questions with maintainers and the community. ๐Ÿ—“๏ธ Tuesday, October 20 ๐Ÿ•— 8:00 AM PT ๐Ÿ‘‰ Register: luma.com/gy8naenq #UnityCatalog #OpenLakehouse #OpenSource
41
Native @ApacheIceberg writes are in Unity Catalog 0.7.0! ๐ŸงŠ Until now, Iceberg support in Unity Catalog was read-only and needed Delta UniForm to generate Iceberg metadata. 0.7.0 adds the write side of the Iceberg REST catalog, so you can create, commit to, rename, and drop native Iceberg tables. No UniForm required. This post walks through it: ๐Ÿ”น Spark and PyIceberg reading and writing the same managed tables ๐Ÿ” The same USE CATALOG, USE SCHEMA, and SELECT grants across engines ๐Ÿ’ป A local setup with unitycatalog-playground v0.2.0 Learn more: unitycatalog.io/blogs/unity-โ€ฆ #UnityCatalog #ApacheIceberg
1
1
69
Reminder: The Unity Catalog Community Sync is tomorrow! ๐Ÿšจ Join us for a recap of v0.7, a sneak peek at the v0.8 roadmap, and live Q&A with the maintainers. ๐Ÿ“… Tomorrow, Oct 6 ๐Ÿ• 8:00 AM PT / 17:00 CEST ๐Ÿ“ Virtual RSVP here: luma.com/uc-sync1 #UnityCatalog #AI #DataGovernance
1
49
Join us for next weekโ€™s Unity Catalog Community Sync: v0.7 recap, a first look at v0.8, and Q&A with the maintainers! Weโ€™ll walk through what shipped in v0.7, share the v0.8 roadmap, and open the floor for questions and feedback. ๐Ÿ“… Tues, Oct 6 ๐Ÿ•— 8:00 AM PT / 17:00 CEST ๐Ÿ“ Virtual Register ๐Ÿ‘‰ luma.com/uc-sync1 #UnityCatalog #DataGovernance #AI
58
Control, choice, and cost in an open lakehouse for AI agents. ๐Ÿ‘‡ This conversation explores Unity Catalog as an open metadata center across @ApacheSpark, @DeltaLakeOSS, and @ApacheIceberg. ๐Ÿ” Fine-grained access for humans and agents ๐Ÿ”„ Open standards for swapping engines and formats ๐Ÿ’ฐ Compute, token, and iteration costs with MLflow ๐ŸŽฅ Watch: youtube.com/watch?v=BsxGjbNWโ€ฆ #OpenLakehouse #UnityCatalog #ApacheIceberg
160
New Open Lakehouse + AI episode now available! Stop shipping 3 versions of annual revenue. ๐Ÿ‘‡ Scott Haines and Lisa Cao on metric views in Spark 4.2. ๐Ÿ“„ One YAML KPI. Catalog serves it globally. ๐Ÿค– Same number for dashboards, agents, and AI/BI. ๐Ÿ”— Native interop with Unity Catalog. ๐ŸŽฅ youtu.be/JJkkOBLPge4 #openlakehouse #unitycatalog #apachespark
1
2
228
Ask three teams for last quarterโ€™s earnings. Get three numbers. Metric views in Unity Catalog define the KPI once (YAML: source, joins, filters, dimensions, measures). Humans and agents query the same number. Spark uses MEASURE(). Try it on Unity Catalog 0.6 with Apache Spark 4.3. ๐Ÿ”— Learn more: unitycatalog.io/blogs/uc-metโ€ฆ #UnityCatalog #MetricViews #ApacheSpark #OpenSource
1
95
Introducing the UC Delta API in Unity Catalog 0.5 ๐Ÿ‘‡ ๐Ÿ”น Versioned, discoverable, atomic REST surface. ๐Ÿ”น Native Delta wire format and intent-based commits. ๐Ÿ”น Run it locally with Spark or DuckDB. Read the full post: unitycatalog.io/blogs/unity-โ€ฆ #UnityCatalog #DeltaLake
1
3
297
[Ep 1] Open Lakehouse + AI: The Catalog Layer, Interoperability & AI-Agent Governance Learn how composable lakehouses center the catalog for metadata, versioning, and commit coordination. ๐Ÿ‘‡ ๐Ÿ”ธ Interop: Iceberg REST, Unity Catalog (OSS), Spark, Delta Lake, Delta-RS, Iceberg. ๐Ÿ”ธ Governance: credential vending, row/col filters, column masks, audits, and why โ€œno governance was the easiest governance.โ€ ๐Ÿ”ธ Agents as data customers (Temporal). Fine-grained access beats blanket credentials on 24/7 workloads. ๐ŸŽฅ Full episode: youtube.com/watch?v=dEFAkS7vโ€ฆ #OpenLakehouse #UnityCatalog @ApacheSpark @ApacheIceberg @DeltaLakeOSS
1
2
5
367
Simplified Governance with Catalog-Managed Tables ๐ŸŒ Unity Catalog 0.4.0 introduces support for UC managed tables, enabling data teams to centrally govern, discover, access, and audit their data through Unity Catalog. Instead of relying on scattered storage paths, separate credentials, and manual maintenance, teams can rely on Unity Catalog as the single logical system of record for their data estate. Leverage UC managed tables to strengthen governance, improve performance, and build on the most modern open catalog for the data and AI era. ๐Ÿ“– Check out the announcement and implementation details on the Unity Catalog blog: lnkd.in/eZWkMBaR #unitycatalog #governance #opensource #catalogs #deltalake
1
2
146
With UC managed tables, teams unlock: ๐Ÿ”ธ ๐—จ๐—ป๐—ถ๐—ณ๐—ถ๐—ฒ๐—ฑ ๐—ด๐—ผ๐˜ƒ๐—ฒ๐—ฟ๐—ป๐—ฎ๐—ป๐—ฐ๐—ฒ:ย Unity Catalog centralizes access control, replacing fragmented storage-level policies. This simplifies how teams ensure all engines access data in a governed, consistent manner. ๐Ÿ”ธ ๐—ฆ๐˜๐—ฎ๐—ป๐—ฑ๐—ฎ๐—ฟ๐—ฑ๐—ถ๐˜‡๐—ฒ๐—ฑ ๐—ฑ๐—ถ๐˜€๐—ฐ๐—ผ๐˜ƒ๐—ฒ๐—ฟ๐˜†: Unity Catalog provides stable logical table identifiers, eliminating the need for clients to depend on physical storage paths for discovery. ๐Ÿ”ธ ๐—˜๐—ณ๐—ณ๐—ผ๐—ฟ๐˜๐—น๐—ฒ๐˜€๐˜€ ๐˜๐—ฎ๐—ฏ๐—น๐—ฒ ๐—ผ๐—ฝ๐˜๐—ถ๐—บ๐—ถ๐˜‡๐—ฎ๐˜๐—ถ๐—ผ๐—ป๐˜€: By automating storage tuning and credential management, Unity Catalog removes the burden of manual operational maintenance from data teams. ๐Ÿ”ธ ๐—›๐—ผ๐—น๐—ถ๐˜€๐˜๐—ถ๐—ฐ ๐—ฎ๐˜‚๐—ฑ๐—ถ๐˜๐—ฎ๐—ฏ๐—ถ๐—น๐—ถ๐˜๐˜†:ย Metadata and permissions are centralized in a single interface, allowing for high-level oversight of ownership and access instead of parsing low-level storage logs. ๐Ÿ”ธ ๐—˜๐—ป๐—ณ๐—ผ๐—ฟ๐—ฐ๐—ฒ๐—ฎ๐—ฏ๐—น๐—ฒ ๐—ฐ๐—ผ๐—ป๐˜€๐˜๐—ฟ๐—ฎ๐—ถ๐—ป๐˜๐˜€: Unity Catalog can authoritatively validate or reject schema and constraint changes, preventing incompatible updates that could compromise data integrity or break downstream workloads. ๐Ÿ”ธ ๐—™๐—ฎ๐˜€๐˜๐—ฒ๐—ฟ ๐—พ๐˜‚๐—ฒ๐—ฟ๐˜† ๐—ฝ๐—น๐—ฎ๐—ป๐—ป๐—ถ๐—ป๐—ด ๐—ฎ๐—ป๐—ฑ ๐—ณ๐—ฎ๐˜€๐˜๐—ฒ๐—ฟ ๐˜„๐—ฟ๐—ถ๐˜๐—ฒ๐˜€: Unity Catalog delivers table metadata directly to Delta clients, bypassing cloud storage requests to significantly reduce metadata latency and accelerate query planning and writes.
69
We are excited to announce ๐—จ๐—ป๐—ถ๐˜๐˜† ๐—–๐—ฎ๐˜๐—ฎ๐—น๐—ผ๐—ด ๐Ÿฌ.๐Ÿฐ.๐Ÿฌ which includes exciting new features and many bug-fixes and improvements! ๐ŸŽŠ Check out some of the highlights ๐Ÿ‘‡ ๐—จ๐—– ๐—ฆ๐—ฒ๐—ฟ๐˜ƒ๐—ฒ๐—ฟ โš™๏ธ Storage Credentials for AWS โš™๏ธ External Locations for AWS โš™๏ธ Managed storage location for catalogs and schemas ๐—จ๐—– ๐—ฆ๐—ฝ๐—ฎ๐—ฟ๐—ธ ๐—–๐—ผ๐—ป๐—ป๐—ฒ๐—ฐ๐˜๐—ผ๐—ฟ โšก Credential Renewal Enabled by Default โšก Support for Spark 4.1 and Delta 4.1 โšก Atomic CTAS for Delta Tables in UCSingleCatalog ๐—จ๐—– ๐—”๐—œ ๐Ÿค– DSPY Integration with AI Functions A huge thank you to the awesome community who made this release possible! ๐Ÿ“– Full release notes: github.com/unitycatalog/unitโ€ฆ #unitycatalog #opensource #oss
1
120
Weโ€™re excited to announce the release of Unity Catalog v0.3.1!ย ๐ŸŽ‰ This release includes exciting new features and many bug-fixes and improvements. Version 0.3.1 focuses on three major areas: ๐Ÿ”น ๐—œ๐—บ๐—ฝ๐—ฟ๐—ผ๐˜ƒ๐—ฒ๐—ฑ ๐—๐—ฎ๐˜ƒ๐—ฎ ๐—ฐ๐—น๐—ถ๐—ฒ๐—ป๐˜ ๐—”๐—ฃ๐—œ ๐˜„๐—ถ๐˜๐—ต ๐—ข๐—”๐˜‚๐˜๐—ต ๐˜€๐˜‚๐—ฝ๐—ฝ๐—ผ๐—ฟ๐˜: designed for reliability and extensibility in production environments. ๐Ÿ”น ๐—”๐˜‚๐˜๐—ผ๐—บ๐—ฎ๐˜๐—ถ๐—ฐ ๐—ฐ๐—ฟ๐—ฒ๐—ฑ๐—ฒ๐—ป๐˜๐—ถ๐—ฎ๐—น ๐—ฟ๐—ฒ๐—ป๐—ฒ๐˜„๐—ฎ๐—น: to support long-running workloads cross cloud platforms. ๐Ÿ”น ๐—จ๐—–-๐—บ๐—ฎ๐—ป๐—ฎ๐—ด๐—ฒ๐—ฑ ๐——๐—ฒ๐—น๐˜๐—ฎ ๐˜๐—ฎ๐—ฏ๐—น๐—ฒ๐˜€: this enables Unity Catalog to coordinate table storage and commits centrally. This release is the result of contributions from our growing open-source community. A big thank-you to everyone who reported issues, submitted pull requests, reviewed code, and shared feedback! ๐Ÿ”— Dive into the release notes for the full list of highlights:ย github.com/unitycatalog/unitโ€ฆ #unitycatalog #opensource #oss #catalog
82
Managing data pipelines at scale is complicated, often resulting in the data silo problemโ€”where valuable assets are spread across systems. This makes it difficult to track, secure access, and scale cleanly. The solution is a clear structure paired with centralized governance. โœ… The Medallion Architecture structures your data into three distinct layers: ๐Ÿฅ‰ ๐—•๐—ฟ๐—ผ๐—ป๐˜‡๐—ฒ: Raw, ingested data. ๐Ÿฅˆ ๐—ฆ๐—ถ๐—น๐˜ƒ๐—ฒ๐—ฟ: Cleaned, enriched data. ๐Ÿฅ‡ ๐—š๐—ผ๐—น๐—ฑ: Business-level data, ready for reporting. Pair this framework with Unity Catalog, and you get a unified system to manage, govern, and organize your entire data flow. ๐Ÿ”— Walk through how this works: unitycatalog.io/blogs/buildiโ€ฆ #UnityCatalog #MedallionArchitecture #DataGovernance #OpenSource
1
1
86
โšก๏ธ From vector DB to multimodal lakehouse at petabyte scale. Join us in Mountain View on Nov 13 for โ€œScaling Multimodal AI Lakehouse with Lance & LanceDBโ€ with Chang She (@lancedb) at the Open Lakehouse + AI Mini Summit! Learn: ๐Ÿ”น Lowโ€‘latency random access + search APIs (vectors, text, binaries) ๐Ÿ”น Schema primitives across blobs + metadata for feature engineering ๐Ÿ”น Hybrid search (vector + fullโ€‘text) for training/fineโ€‘tuning at scale Two exciting tracks, one epic afternoon. ๐Ÿ‘ #OpenLakehouse #AI #LanceDB #VectorDB #DataEngineering
2
7
455
Unity Catalog retweeted
Join us at Open Lakehouse + AI Paris on November 24, 6:30โ€“10PM โ€” co-located with the Forward Data Conference! ๐Ÿ‡ซ๐Ÿ‡ท Secure your spot now โฌ‡๏ธ luma.com/OLM-1124 Weโ€™re bringing together data innovators and open source contributors for an evening packed with insight and inspiration. Hear talks from: โœ… Alexandre BERGERE (@DataGalaxy / Datalex) on Building a Scalable Usage Insights Platform with Delta Sharing โœ… Bartosz Konieczny (@waitingforcode) on Design Patterns for the Open Lakehouse โœ… Youssef Mrini & El Ghali Benchekroun (@Databricks) on The Future of Open Table Formats & @unitycatalog_io Food, drinks, networking โ€” and plenty of new ideas (& swag) to take home. ๐ŸŒŸ #opensource #deltalake #oss #apacheiceberg #unitycatalog #lakehouse #openlakehouse #ai
5
8
689
Missed it liveโ“See how an observabilityโ€‘first Telemetry Lake correlates data movement with system behavior to detect issues, diagnose root causes, and adapt in real timeโ€”why correlation > coverage and why the first ~150 characters of signal matter. Watch: youtube.com/watch?v=r04iCvm6โ€ฆ Whatโ€™s inside: @OpenLineage + @opentelemetry + an LLM reasoning layer that turns noisy lineage and traces into prioritized actions to cut TTD/TTR and reduce blast radius. #openlakehouse #datalineage #observability #AI
1
2
104