Databricks Unity Catalog: Complete Guide to Data Governance

Unity Catalog is Databricks' unified governance solution for data and AI assets across the lakehouse. It centralizes access control, audit, lineage, and discovery for tables, views, files, ML models, and AI assets across the workspaces of a Databricks account. Unity Catalog became generally available in August 2022, has been enabled automatically for every Databricks workspace created after November 8, 2023, and was open-sourced in June 2024.

This guide explains what Unity Catalog is, how its model works, and where it fits alongside the broader enterprise governance estate.

What Is Unity Catalog?

Before Unity Catalog, governance on Databricks meant duplicating ACLs across the legacy Hive metastore in every workspace, managing storage credentials manually, and assembling lineage from logs after the fact. Each workspace was its own island.

Unity Catalog replaces that fragmented model with a single account-level service that:

• Holds metadata for every catalog, schema, table, view, function, model, and volume.
• Enforces a unified permissions model with standard SQL grants.
• Captures lineage automatically as queries run.
• Records data access and system activity in an audit log system table.
• Surfaces all governed assets through a searchable Catalog Explorer.

Unity Catalog is built into the platform as its default governance layer. It is also a prerequisite for Lakehouse Federation, for publishing listings in Databricks Marketplace, and for sharing data through the Databricks-hosted Delta Sharing server (now documented as OpenSharing).

Core Concepts

Unity Catalog organizes governance around several primitives:

• Metastore - the top-level container, scoped to one cloud region. Databricks requires one metastore for each region in which an organization operates, and workspaces attach to the metastore in their region.
• Catalog - the first level of the namespace. Catalogs typically map to a domain, environment, or business unit (finance, marketing, prod, dev).
• Schema - the second level (formerly called database in Spark SQL terminology). Schemas group related tables and views.
• Table - managed tables (Delta Lake by default, or Apache Iceberg), whose storage and lifecycle Unity Catalog handles, and external tables (Delta, CSV, JSON, Avro, Parquet, ORC or text), whose files stay in your cloud storage when the table is dropped.
• View - saved SQL queries, including dynamic views with row- and column-level filtering, plus metric views that define business metrics once in YAML.
• Volume - governed storage location for non-tabular files (PDFs, images, audio, ML model artifacts, raw data files).
• Function - registered SQL or Python user-defined functions.
• Model - registered MLflow models, governed alongside the data they consume and produce.
• Storage credential + External location - Unity Catalog-managed authentication to external cloud storage. Storage credentials never need to be embedded in user code.

Three-Level Namespace

Unity Catalog's signature feature is the three-level namespace: every object is addressed as catalog.schema.object. This is a deliberate departure from the legacy Spark two-level (database.table) model.

The third level matters because catalogs are administratively meaningful. They map to teams, environments, or domains, and they are the natural unit for top-level governance decisions: who owns this data, what region is it in, what permission scope applies?

Some example references:

• finance.raw.invoices - raw invoice data in the finance catalog
• marketing.gold.attribution_daily - production attribution table
• main.mlmodels.churn_v3 - registered ML model

Access Control

Unity Catalog uses SQL GRANT and REVOKE statements. The same syntax works across SQL Editor, notebooks, jobs, and the API:

GRANT SELECT ON finance.gold.revenue_daily TO `data-analysts`;
GRANT MODIFY ON marketing.events.campaigns TO `marketing-engineers`;
GRANT USE CATALOG ON finance TO `finance-domain`;

Access can be granted on the metastore, a catalog, a schema, or an individual table, view, volume, function or model; columns are protected with masks rather than grants. Privileges propagate down: a grant on a catalog or schema applies to all its current and future child objects. Common privileges include SELECT, MODIFY, USE CATALOG, USE SCHEMA, EXECUTE (on functions), and READ VOLUME / WRITE VOLUME.

Row- and Column-Level Security

Beyond table-level grants, Unity Catalog enforces row filters and column masks via SQL functions. A row filter is a function that returns a boolean; a column mask is a function that transforms a value. Both attach to a table and are evaluated automatically on every query - including queries from BI tools. The documented syntax binds a mask to one column:

CREATE FUNCTION mask_pii(email STRING)
RETURN CASE WHEN is_account_group_member('analysts') THEN email
            ELSE 'REDACTED' END;

ALTER TABLE finance.raw.customers
ALTER COLUMN email SET MASK mask_pii;

For many tables at once, attribute-based access control (ABAC) policies attach at the catalog, schema or table level and apply row filters and column masks automatically to objects that carry the governed tags a policy targets.

Identity Federation

Users, groups and service principals are managed at the Databricks account level and can be provisioned through SCIM from an identity provider such as Microsoft Entra ID, Okta or OneLogin. Identity federation lets workspaces use those account-level identities, so a grant refers to the same principal in every workspace - no shadow user lists.

Data Lineage

Unity Catalog captures lineage automatically for queries run on Databricks through Databricks SQL or the Spark DataFrame interfaces, down to the column level when source and target are referenced by table name. There is no per-table setup: the table only has to be registered in a Unity Catalog metastore.

Lineage nodes cover tables and views, ML model versions, external assets and file paths, and the jobs and dashboards that read a table appear as its consumers. The Catalog Explorer surfaces lineage as an interactive graph; the same lineage is queryable through the system tables (system.access.table_lineage and system.access.column_lineage), which keep a rolling one-year window. For broader context on how lineage works, see data lineage and column-level lineage.

Audit and Discovery

Data access and system activity, such as reads, schema changes, grants and deletes, are recorded in the audit log system table system.access.audit. Databricks recommends querying that table; on AWS, audit logs can also be delivered as JSON files to an Amazon S3 bucket.

The Catalog Explorer is the discovery UI: a searchable, filterable browser over every governed asset, with descriptions, tags, lineage, owners, and previews. Tags can be applied to catalogs, schemas, tables, columns, views, volumes, functions and models, supporting common governance patterns like pii=true or domain=finance classifications. Catalog and schema tags are inherited by child objects only when ABAC policies are evaluated, and never down to columns.

Volumes for Unstructured Data

Volumes extend Unity Catalog governance to non-tabular files: documents, images, audio, video, ML model artifacts, raw payloads. Volumes are mounted at known paths (/Volumes/<catalog>/<schema>/<volume>/...) and accessed through the same SQL grant system that controls tables.

This matters most for AI workloads. Most enterprise AI requires both structured tables and unstructured documents - and before volumes, governing those two cleanly required two different systems. With volumes, a RAG pipeline can read PDFs, embed them, and write the resulting vectors back to a table, all under one consistent permission model.

Unity Catalog Open Source

On June 13, 2024, Databricks open-sourced Unity Catalog under the Apache 2.0 license, hosted by LF AI & Data, a foundation of the Linux Foundation. The open-source server implements the Iceberg REST Catalog API and is compatible with the Hive metastore API, so engines outside Databricks can read the tables it catalogs; Databricks named DuckDB among the launch supporters. The open-source project and the Databricks-managed service are separate implementations: features covered in this guide, such as automatic lineage capture and the system tables, are documented for the managed service.

How Dawiso Complements Unity Catalog

Unity Catalog governs the lakehouse extremely well. It does not, by itself, govern the rest of the enterprise: the operational systems that feed it, the BI tools that consume it, the business glossary that defines what "revenue" means, and the human stewardship workflows that decide who owns what.

The Dawiso Databricks connector reads Unity Catalog through its system tables and REST API. It ingests catalogs, schemas, tables, views, metric views, columns, volumes, functions and ML models, along with owners, tags, column masks, row filters and data classifications. Column-to-column lineage comes from system.access.column_lineage for activity inside a configurable lineage window (30 days by default), plus deterministic lineage for metric views (from their YAML) and dashboard datasets (from their SQL).

Dawiso then adds what lives outside the platform: business glossary definitions that are linked automatically wherever they appear in object descriptions, stewardship assignments, documented data quality checks together with the external engine that runs them, and lineage beyond Databricks, which can be extended by defining relations manually or importing them. The result is one governance picture: technical depth from Unity Catalog, business semantics and stewardship workflow from Dawiso.

Sources

• Databricks, What is Unity Catalog?
• Databricks, Unity Catalog GA release note, August 25, 2022.
• Databricks Blog, Open Sourcing Unity Catalog, June 13, 2024.
• Databricks, Create a Unity Catalog metastore.
• Databricks, Unity Catalog managed tables for Delta Lake and Apache Iceberg.
• Databricks, Work with external tables.
• Databricks, Unity Catalog metric views.
• Databricks, Unity Catalog privileges reference.
• Databricks, Manually apply row filters and column masks.
• Databricks, Attribute-based access control in Unity Catalog.
• Databricks, Sync users and groups from your identity provider using SCIM.
• Databricks, Lineage in Unity Catalog.
• Databricks, Configure audit log delivery.
• Databricks, Apply tags to Unity Catalog securable objects.
• Databricks, Connect to external databases and catalogs.
• Databricks, What is OpenSharing?
• Databricks, What is Databricks Marketplace?
• Dawiso Help, Databricks.
• Dawiso Help, Business Glossary.
• Dawiso Help, Sources of data lineage.
• Dawiso Help, Data quality and SLAs.