Databricks - Auxia Docs

Databricks is a data lakehouse platform. Auxia connects to Databricks in one of two ways: by reading your Unity Catalog managed Delta tables directly, with short-lived credentials, or with Delta Transfer, where you export Parquet files to Amazon S3 on a schedule. Both are set up together with the Auxia team.

Setup: Unity Catalog

Recommended · about 10 minutes

Setup: Delta Transfer

About 30 minutes

Choose a method

	Unity Catalog (recommended)	Delta Transfer
How it works	Auxia reads each table’s data files directly, using short-lived credentials Unity Catalog issues per table	A scheduled Databricks notebook writes Parquet files to an S3 bucket you own, and Auxia reads from it
What you set up	A service principal and a few grants	An S3 bucket, a bucket policy, and export notebooks
Ongoing work on your side	None	Keep the export jobs running
Compute cost	No Databricks compute on either side — Auxia reads the files directly	Your cluster or SQL warehouse runs the exports
Freshness	A schedule you choose	Whenever your export jobs run (hourly or daily)
Choose it when	Your tables are Unity Catalog managed Delta tables on Amazon S3	Unity Catalog direct read isn’t an option

Features

Feature	Unity Catalog	Delta Transfer	Notes
Setup	Assisted	Assisted	Coordinated with your Auxia solutions engineer
Nothing copied or exported on your side	✅	❌	Delta Transfer writes Parquet files to your bucket
No credentials exchanged	✅	✅	Unity Catalog issues credentials that expire in about an hour; Delta Transfer grants read access with a bucket policy
Incremental reads	✅	✅	Unity Catalog uses Change Data Feed; Delta Transfer supports watermark and Change Data Feed patterns
Full-table snapshots	—	✅	Delta Transfer pattern for small, slowly changing tables
Audit trail	✅	—	Credential issuance appears in Unity Catalog audit logs, and file reads in your cloud storage logs
Instant revocation	✅	✅	Remove the grant or disable the service principal; or remove the bucket policy
Data Health monitoring	✅	✅

How it works

Unity Catalog. Auxia authenticates as a service principal you create, asks Unity Catalog for short-lived credentials scoped to one table at a time, and reads that table’s underlying files directly from your storage — read-only. Change Data Feed lets Auxia read only what changed since the last run, so enable it on each table before the first load. Auxia runs the initial historical load, then recurring incremental reads, and validates row counts before go-live. Delta Transfer. You export each table to a dedicated S3 bucket as partitioned Parquet with a scheduled notebook, using one of three patterns — append-only, Change Data Feed, or full snapshot — and grant Auxia read access with a bucket policy. Auxia reads from the bucket the same way as the Amazon S3 connector.

Limitations

Both methods are set up with the Auxia team, not in the self-serve wizard.
Unity Catalog direct read requires a Databricks workspace on AWS, with table data on Amazon S3, and Unity Catalog managed Delta tables.
Share base tables rather than views.
See Source data requirements for requirements shared by all connectors.

Setup: Unity Catalog

Setup: Delta Transfer

​Choose a method

​Features

​How it works

​Limitations

Choose a method

Features

How it works

Limitations