> ## Documentation Index
> Fetch the complete documentation index at: https://docs.auxia.io/llms.txt
> Use this file to discover all available pages before exploring further.

# How ingestion works

> The three stages every source goes through, how long each takes for a new customer, and where your data ends up

Every source you connect to Auxia runs through the same three stages, whatever it is and wherever it lives. Knowing the stages makes setup choices and troubleshooting much easier: almost every setup choice belongs to exactly one stage, and when something goes wrong, the first question is always *which stage failed*.

<div style={{margin: '28px 0', padding: '20px', background: 'var(--ifm-color-emphasis-100)', borderRadius: '12px'}}>
  <div style={{display: 'flex', alignItems: 'center', gap: '8px', flexWrap: 'wrap'}}>
    <div style={{flex: '1 1 0', minWidth: '96px', textAlign: 'center', padding: '12px 8px', background: 'var(--ifm-background-color)', border: '1px solid var(--ifm-color-emphasis-300)', borderRadius: '8px'}}>
      <div style={{fontSize: '24px', marginBottom: '6px'}}>🏦</div>
      <div style={{fontWeight: '600', fontSize: '14px'}}>Your data</div>
      <div style={{fontSize: '12px', color: 'var(--ifm-color-emphasis-600)', marginTop: '4px'}}>BigQuery · GCS · S3 · SFTP</div>
    </div>

    <div style={{fontSize: '20px', color: 'var(--ifm-color-emphasis-500)', fontWeight: 'bold'}}>→</div>

    <div style={{flex: '1 1 0', minWidth: '96px', textAlign: 'center', padding: '12px 8px', background: 'var(--ifm-background-color)', border: '1px solid var(--ifm-color-emphasis-300)', borderRadius: '8px'}}>
      <div style={{fontSize: '24px', marginBottom: '6px'}}>📥</div>
      <div style={{fontWeight: '600', fontSize: '14px'}}>1. Transfer</div>
      <div style={{fontSize: '12px', color: 'var(--ifm-color-emphasis-600)', marginTop: '4px'}}>Raw copy, as-is</div>
      <div style={{fontSize: '11px', color: 'var(--ifm-color-emphasis-600)', marginTop: '6px', fontWeight: '600'}}>1–2 days</div>
    </div>

    <div style={{fontSize: '20px', color: 'var(--ifm-color-emphasis-500)', fontWeight: 'bold'}}>→</div>

    <div style={{flex: '1 1 0', minWidth: '96px', textAlign: 'center', padding: '12px 8px', background: '#4F46E5', border: '1px solid #4F46E5', borderRadius: '8px', color: 'white'}}>
      <div style={{fontSize: '24px', marginBottom: '6px'}}>⚡</div>
      <div style={{fontWeight: '600', fontSize: '14px'}}>2. Transform</div>
      <div style={{fontSize: '12px', opacity: 0.85, marginTop: '4px'}}>Cleaned and unified</div>
      <div style={{fontSize: '11px', opacity: 0.85, marginTop: '6px', fontWeight: '600'}}>4–24 hours</div>
    </div>

    <div style={{fontSize: '20px', color: 'var(--ifm-color-emphasis-500)', fontWeight: 'bold'}}>→</div>

    <div style={{flex: '1 1 0', minWidth: '96px', textAlign: 'center', padding: '12px 8px', background: 'var(--ifm-background-color)', border: '1px solid var(--ifm-color-emphasis-300)', borderRadius: '8px'}}>
      <div style={{fontSize: '24px', marginBottom: '6px'}}>🚀</div>
      <div style={{fontWeight: '600', fontSize: '14px'}}>3. Process</div>
      <div style={{fontSize: '12px', color: 'var(--ifm-color-emphasis-600)', marginTop: '4px'}}>Into serving systems</div>
      <div style={{fontSize: '11px', color: 'var(--ifm-color-emphasis-600)', marginTop: '6px', fontWeight: '600'}}>\~1 hour</div>
    </div>
  </div>

  <div style={{display: 'flex', alignItems: 'center', gap: '8px', flexWrap: 'wrap', marginTop: '16px'}}>
    <div style={{fontSize: '13px', color: 'var(--ifm-color-emphasis-600)', marginRight: '4px'}}>Powers →</div>
    <div style={{padding: '6px 10px', background: 'var(--ifm-background-color)', border: '1px solid var(--ifm-color-emphasis-300)', borderRadius: '6px', fontSize: '13px', fontWeight: '500'}}>🎯 Real-time decisioning</div>
    <div style={{padding: '6px 10px', background: 'var(--ifm-background-color)', border: '1px solid var(--ifm-color-emphasis-300)', borderRadius: '6px', fontSize: '13px', fontWeight: '500'}}>🧠 ML features</div>
    <div style={{padding: '6px 10px', background: 'var(--ifm-background-color)', border: '1px solid var(--ifm-color-emphasis-300)', borderRadius: '6px', fontSize: '13px', fontWeight: '500'}}>🤖 Agent Studio</div>
    <div style={{padding: '6px 10px', background: 'var(--ifm-background-color)', border: '1px solid var(--ifm-color-emphasis-300)', borderRadius: '6px', fontSize: '13px', fontWeight: '500'}}>📊 Reporting</div>
  </div>
</div>

<Note>
  The times shown are typical for a **new customer's first connection** — from finishing setup to data being ready in Auxia. Once a connection is live, each source syncs on its own schedule — see [When will my data show up?](#when-will-my-data-show-up)
</Note>

## Stage 1: Transfer

Copies your data **as-is** into Auxia. Nothing is renamed or reshaped.

* For **BigQuery**, Auxia runs a read query against your table.
* For **Google Cloud Storage** and **Amazon S3**, a storage transfer copies your files into an Auxia-side bucket.
* For **SFTP**, you push files to an Auxia-hosted SFTP server.

Choices that belong to this stage: **access grants**, **file format**, and **folder layout**.

**Typical time for a new customer:** 1–2 days.

## Stage 2: Transform

Reads the raw copy and writes Auxia's **unified tables** — the single schema that everything downstream reads. The column mapping you confirm in the setup wizard is exactly what this stage applies; [Map your data](/data-ingestion/concepts/map-your-data) explains how that mapping works.

This is Auxia's proprietary capability: Auxia cleans your data and transforms it into unified data attributes, so your team doesn't have to. There's no code to write and no pipeline to build — onboarding is a guided, no-code setup. The processed data is ready for **day-one modeling** as ML features.

Choices that belong to this stage: **column mapping**, **primary user ID**, **event names**, and **row filters**.

**Typical time for a new customer:** 4–24 hours.

## Stage 3: Process

Pushes the unified data into Auxia's serving systems. From there it powers real-time decisioning, ML feature generation, reporting, and **Auxia Agent Studio** use cases.

**Typical time for a new customer:** about 1 hour.

## When will my data show up?

Three different timings come up when you connect a source. They measure different things:

| Timing | What it measures | Typical |
| - | - | - |
| **Setup** | Your time to complete the connection | About 10–30 minutes, depending on the connector |
| **First load** | For a new customer, from finishing setup to data being ready in Auxia | 1–3 days in total — see the stages above |
| **Ongoing sync** | Once live, how often new data arrives | Depends on the connector — see below |

### Ongoing sync by connector

| Connector | How often new data is read | When a day's data appears |
| - | - | - |
| [Google BigQuery](/data-ingestion/data-connections/google-bigquery) | From about every 10 minutes to daily (hourly by default for tables that support appends) | Within about an hour, or the same morning |
| [Google Cloud Storage](/data-ingestion/data-connections/google-cloud-storage), [Amazon S3](/data-ingestion/data-connections/amazon-s3) | From about hourly to daily (daily by default) | The next morning |
| [Snowflake](/data-ingestion/data-connections/snowflake), [Databricks](/data-ingestion/data-connections/databricks) | From about every 10 minutes to daily, on a schedule you choose (direct connection or Unity Catalog); or whenever your export runs (GCS export or Delta Transfer) | Depends on the schedule |
| [SFTP](/data-ingestion/data-connections/sftp) | An agreed schedule | Depends on the schedule |
| [Amazon Kinesis](/data-ingestion/data-connections/amazon-kinesis), [Google Cloud Pub/Sub](/data-ingestion/data-connections/gcp-pubsub) | Continuously | Near real time |
| [Amplitude](/data-ingestion/data-connections/amplitude) | From real time (Pub/Sub streaming) to every 30 minutes (Export API) | Near real time, or within about 30 minutes |
| [Firebase](/data-ingestion/data-connections/firebase) | As for BigQuery; intraday tables with Firebase streaming export | After Firebase finalizes the daily export, or near real time for intraday tables |
| [Log Events API](/api-reference/log-events) | Per event | Instantly |

<Tip>
  If your data lands late in the day, tell your Auxia contact when it usually arrives. A scheduled run that starts before your data lands finds nothing new — and reports success, not failure.
</Tip>

## How the stages show up in the console

The [Data Health dashboard](/data-ingestion/concepts/monitoring) shows each stage as its own status indicator, so you can see at a glance whether a problem is in reaching your data (Transfer), interpreting it (Transform), or serving it (Process).

<CardGroup cols={2}>
  <Card title="What data is needed" icon="database" href="/data-ingestion/what-data-is-needed">
    The four kinds of data Auxia uses, with examples.
  </Card>

  <Card title="Map your data" icon="table-columns" href="/data-ingestion/concepts/map-your-data">
    How the Transform stage interprets your columns.
  </Card>
</CardGroup>


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.