Home/Data Ingestion/Data Connections/Databricks/Unity Catalog

Databricks (Unity Catalog)

Setting up a data connection via Databricks Unity Catalog

This guide explains how Auxia reads your Unity Catalog managed Delta tables directly, using Databricks' credential vending — no tables are copied, exported, or recreated in your environment.

Before proceeding, review the Source Data Requirements to ensure your tables are ready to be connected with Auxia.

1. Overview

Auxia authenticates to your Unity Catalog as a service principal you create, asks Unity Catalog for short-lived credentials scoped to each table, reads that table's underlying data files directly from your cloud storage, and ingests them on a recurring schedule — read-only.

Flow:

Your environment (Databricks + AWS)              Auxia
      │
      │  Unity Catalog managed Delta tables (on Amazon S3)
      │       │
      │       │  grant read access to a service principal
      │       ▼
      │  Unity Catalog ──(vends short-lived, table-scoped credentials)──>
      │                                               │
      │                                               │  Auxia reads the table's
      │                                               │  data files directly from S3
      │                                               │  on your schedule
      │                                               ▼
      │                                           Auxia platform

Time required: ~10 minutes.

Prerequisites:

• Databricks workspace on AWS with Unity Catalog enabled
• Table data backed by Amazon S3
• Source tables are Unity Catalog managed Delta tables
• A metastore/catalog admin who can create a service principal and run GRANT statements

What Auxia provides: Auxia configures and schedules the connection on our side — the initial historical load and recurring incremental reads — and validates row counts against the source. Your Auxia solutions engineer provides the exact grant statements and confirms the connection before go-live.

2. Create a Service Principal

In your Databricks account, create a dedicated service principal and generate an OAuth secret (client ID + secret). Share the following with your Auxia solutions engineer over a secure channel:

• the service principal client ID and secret, and
• your workspace URL — e.g. https://<your-workspace>.cloud.databricks.com (the workspace host, not the Databricks account-console URL).

note

The service principal needs no workspace entitlements (no admin, cluster, or SQL-warehouse access) beyond the table grants in Step 4.

3. Enable External Data Access on the Metastore

A metastore admin enables external data access on the Unity Catalog metastore (the external_access_enabled setting). This one-time setting is what allows credential vending.

4. Grant Read Access to the Service Principal

Run the following for each catalog, schema, and table to be ingested:

GRANT USE CATALOG          ON CATALOG <catalog>                  TO `<service_principal>`;
GRANT USE SCHEMA           ON SCHEMA  <catalog>.<schema>         TO `<service_principal>`;
GRANT SELECT               ON TABLE   <catalog>.<schema>.<table> TO `<service_principal>`;
GRANT EXTERNAL USE SCHEMA  ON SCHEMA  <catalog>.<schema>         TO `<service_principal>`;

EXTERNAL USE SCHEMA is the credential-vending privilege; it can only be granted by the catalog owner.

5. Enable Change Data Feed

Auxia loads new data incrementally using Delta's Change Data Feed (CDF). Enable it on each table before the first load:

ALTER TABLE <catalog>.<schema>.<table>
SET TBLPROPERTIES (delta.enableChangeDataFeed = true);

note

CDF only records changes from the point it is enabled, so turn it on before Auxia's initial load.

6. Confirm the Tables with Auxia

For each table you want to share, provide your Auxia solutions engineer with:

• the fully-qualified name catalog.schema.table (base tables, not views);
• confirmation that the table is append-only, and its event-timestamp / partition column.

Auxia configures and schedules the connection on our side.

7. Security & Governance

• Read-only, scoped to the tables you explicitly grant.
• No standing credentials — each request vends short-lived credentials scoped to a single table's storage location.
• No broad bucket access to your cloud account.
• Auditable — Unity Catalog audits each credential-issuance event; the file-level reads appear in your own cloud storage / CloudTrail logs.
• Revocable — removing a grant (or disabling the service principal) blocks all future access; an already-issued credential remains valid only until it expires (about an hour).
• The only long-lived secret is the service principal's OAuth credential, which you can rotate at any time.

8. Recommended Practices

• Enable Change Data Feed before the first load so incremental reads are available from day one.
• Keep the service principal least-privileged — grant only the catalogs, schemas, and tables to be ingested, and no other workspace entitlements.
• Share schema and a small sample (or a few days of history) first so the connection can be validated before the full historical load.
• Share base tables, not views — Auxia reads the underlying managed Delta tables directly.

9. Summary

Step	Action	Owner
1	Create a dedicated service principal and OAuth secret	You
2	Enable external data access on the metastore	You
3	Grant USE CATALOG, USE SCHEMA, SELECT, EXTERNAL USE SCHEMA	You
4	Enable Change Data Feed on each table	You
5	Share workspace URL, credentials, and the table list	You
6	Configure and schedule the connection; run the initial and incremental loads	Auxia

10. FAQ

Does Auxia create or modify anything in our environment? No. Auxia reads your existing managed tables in place — no external tables, no copies, no schema changes, and no new pipelines.

Can we revoke Auxia's access? Yes, at any time — remove the grant or disable the service principal.

What if a table isn't strictly append-only? Let Auxia know. Change Data Feed captures inserts, updates, and deletes, which Auxia uses for incremental loads.

How do we rotate the service principal secret? Generate a new OAuth secret in Databricks and share it with Auxia. Auxia stores the secret by reference, so rotation does not require reconfiguring the connection.

Need Help?

Contact support@auxia.io or your Auxia solutions engineer.