> ## Documentation Index
> Fetch the complete documentation index at: https://docs.auxia.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Databricks (Unity Catalog)

> Setting up a data connection via Databricks Unity Catalog

This guide explains how Auxia reads your Unity Catalog **managed Delta tables** directly, using Databricks' **credential vending** — no tables are copied, exported, or recreated in your environment.

Before proceeding, review the [Source Data Requirements](/data-ingestion/overview#source-data-requirements) to ensure your tables are ready to be connected with Auxia.

## 1. Overview

Auxia authenticates to your Unity Catalog as a service principal you create, asks Unity Catalog for short-lived credentials scoped to each table, reads that table's underlying data files directly from your cloud storage, and ingests them on a recurring schedule — **read-only**.

**Flow:**

```
Your environment (Databricks + AWS)              Auxia
      │
      │  Unity Catalog managed Delta tables (on Amazon S3)
      │       │
      │       │  grant read access to a service principal
      │       ▼
      │  Unity Catalog ──(vends short-lived, table-scoped credentials)──>
      │                                               │
      │                                               │  Auxia reads the table's
      │                                               │  data files directly from S3
      │                                               │  on your schedule
      │                                               ▼
      │                                           Auxia platform
```

**Time required:** \~10 minutes.

**Prerequisites:**

* Databricks workspace on AWS with **Unity Catalog** enabled
* Table data backed by **Amazon S3**
* Source tables are **Unity Catalog managed Delta** tables
* A metastore/catalog admin who can create a service principal and run `GRANT` statements

**What Auxia provides:** Auxia configures and schedules the connection on our side — the initial historical load and recurring incremental reads — and validates row counts against the source. Your Auxia solutions engineer provides the exact grant statements and confirms the connection before go-live.

## 2. Create a Service Principal

In your Databricks account, create a dedicated **service principal** and generate an **OAuth secret** (client ID + secret). Share the following with your Auxia solutions engineer over a secure channel:

* the service principal **client ID** and **secret**, and
* your **workspace URL** — e.g. `https://<your-workspace>.cloud.databricks.com` (the workspace host, **not** the Databricks account-console URL).

<Note>
  The service principal needs no workspace entitlements (no admin, cluster, or SQL-warehouse access) beyond the table grants in [Step 4](#4-grant-read-access-to-the-service-principal).
</Note>

## 3. Enable External Data Access on the Metastore

A metastore admin enables **external data access** on the Unity Catalog metastore (the `external_access_enabled` setting). This one-time setting is what allows credential vending.

## 4. Grant Read Access to the Service Principal

Run the following for each catalog, schema, and table to be ingested:

```sql theme={null}
GRANT USE CATALOG          ON CATALOG <catalog>                  TO `<service_principal>`;
GRANT USE SCHEMA           ON SCHEMA  <catalog>.<schema>         TO `<service_principal>`;
GRANT SELECT               ON TABLE   <catalog>.<schema>.<table> TO `<service_principal>`;
GRANT EXTERNAL USE SCHEMA  ON SCHEMA  <catalog>.<schema>         TO `<service_principal>`;
```

`EXTERNAL USE SCHEMA` is the credential-vending privilege; it can only be granted by the catalog owner.

## 5. Enable Change Data Feed

Auxia loads new data incrementally using Delta's **Change Data Feed (CDF)**. Enable it on each table **before** the first load:

```sql theme={null}
ALTER TABLE <catalog>.<schema>.<table>
SET TBLPROPERTIES (delta.enableChangeDataFeed = true);
```

<Note>
  CDF only records changes from the point it is enabled, so turn it on before Auxia's initial load.
</Note>

## 6. Confirm the Tables with Auxia

For each table you want to share, provide your Auxia solutions engineer with:

* the fully-qualified name `catalog.schema.table` (base tables, **not** views);
* confirmation that the table is **append-only**, and its **event-timestamp / partition column**.

Auxia configures and schedules the connection on our side.

## 7. Security & Governance

* **Read-only**, scoped to the tables you explicitly grant.
* **No standing credentials** — each request vends short-lived credentials scoped to a single table's storage location.
* **No broad bucket access** to your cloud account.
* **Auditable** — Unity Catalog audits each credential-issuance event; the file-level reads appear in your own cloud storage / CloudTrail logs.
* **Revocable** — removing a grant (or disabling the service principal) blocks all future access; an already-issued credential remains valid only until it expires (about an hour).
* The only long-lived secret is the service principal's OAuth credential, which you can rotate at any time.

## 8. Recommended Practices

* **Enable Change Data Feed before the first load** so incremental reads are available from day one.
* **Keep the service principal least-privileged** — grant only the catalogs, schemas, and tables to be ingested, and no other workspace entitlements.
* **Share schema and a small sample (or a few days of history) first** so the connection can be validated before the full historical load.
* **Share base tables, not views** — Auxia reads the underlying managed Delta tables directly.

## 9. Summary

| Step | Action                                                                       | Owner |
| ---- | ---------------------------------------------------------------------------- | ----- |
| 1    | Create a dedicated service principal and OAuth secret                        | You   |
| 2    | Enable external data access on the metastore                                 | You   |
| 3    | Grant `USE CATALOG`, `USE SCHEMA`, `SELECT`, `EXTERNAL USE SCHEMA`           | You   |
| 4    | Enable Change Data Feed on each table                                        | You   |
| 5    | Share workspace URL, credentials, and the table list                         | You   |
| 6    | Configure and schedule the connection; run the initial and incremental loads | Auxia |

## 10. FAQ

**Does Auxia create or modify anything in our environment?**
No. Auxia reads your existing managed tables in place — no external tables, no copies, no schema changes, and no new pipelines.

**Can we revoke Auxia's access?**
Yes, at any time — remove the grant or disable the service principal.

**What if a table isn't strictly append-only?**
Let Auxia know. Change Data Feed captures inserts, updates, and deletes, which Auxia uses for incremental loads.

**How do we rotate the service principal secret?**
Generate a new OAuth secret in Databricks and share it with Auxia. Auxia stores the secret by reference, so rotation does not require reconfiguring the connection.

## Need Help?

Contact [**support@auxia.io**](mailto:support@auxia.io) or your Auxia solutions engineer.