> ## Documentation Index
> Fetch the complete documentation index at: https://docs.auxia.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Amazon S3

> Setting up a data connection via Amazon S3

## Overview

Auxia's integration uses **Google Cloud Storage Transfer Service (STS)** to securely read data from your AWS S3 bucket via an Auxia-managed AWS IAM role. Data is transferred from your S3 bucket to Auxia's Google Cloud Storage (GCS) infrastructure for model training, feature creation, and performance measurement.

We recommend using **Parquet** format for optimal performance, schema handling, and compression.

Before proceeding, review the [Source Data Requirements](/data-ingestion/overview#source-data-requirements) to ensure your data is ready to be connected with Auxia.

***

## Overview of the Transfer Flow

1. Client's data remains in AWS S3.
2. GCP initiates the transfer via Storage Transfer Service.
3. The transfer job reads from your S3 bucket using an Auxia-managed AWS IAM Role.
4. Permissions are granted for S3 object read operations and, if applicable, decryption.

***

## Setting Up an S3 Connection

### Prerequisites

* An AWS S3 bucket containing the data you want to share with Auxia
* Permission to update your S3 bucket policy
* Access to the [Auxia Console](https://console.auxia.io) with connection editor permissions

### Step 1: Grant Access to an Auxia-Managed IAM Role

The client **does not need to create or manage an AWS IAM role**. Instead, the client grants read-only access to an Auxia-owned IAM role that is used by GCP Storage Transfer Service.

#### Auxia IAM Roles

| Environment     | ARN                                                   | Name                   |
| --------------- | ----------------------------------------------------- | ---------------------- |
| **Production**  | `arn:aws:iam::838011126883:role/AuxiaTransferRole`    | `AuxiaTransferRole`    |
| **Development** | `arn:aws:iam::838011126883:role/AuxiaTransferRoleDev` | `AuxiaTransferRoleDev` |

### Step 2: Update Your S3 Bucket Policy

Update your S3 bucket policy to allow read-only access for the appropriate Auxia role:

```json theme={null}
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::838011126883:role/AuxiaTransferRole"
      },
      "Action": [
        "s3:GetObject",
        "s3:ListBucket",
        "s3:GetBucketLocation"
      ],
      "Resource": [
        "arn:aws:s3:::your-bucket-name",
        "arn:aws:s3:::your-bucket-name/*"
      ]
    }
  ]
}
```

Replace:

* `your-bucket-name` with your actual S3 bucket
* Role ARN with `AuxiaTransferRoleDev` if this is a DEV integration

### Step 3: Configure Storage Transfer (Handled by Auxia)

Auxia will initiate the transfer by configuring the Storage Transfer Service on Google Cloud using the appropriate Auxia-managed IAM role. No action is required from the client for this step.

For reference, see [GCP Docs on AWS S3 Transfer via Web Identity](https://cloud.google.com/storage-transfer/docs/source-amazon-s3).

***

## Expected S3 File Structure

### File Format & Compression Recommendations

| Format                  | Schema Support | Compression Support          | Efficiency | Notes                                  |
| ----------------------- | -------------- | ---------------------------- | ---------- | -------------------------------------- |
| **Parquet (Preferred)** | Embedded       | GZIP, Snappy, ZSTD, LZO, LZ4 | High       | Best for analytical columnar workloads |

### Recommended Formats

* **Parquet** — prefer for performance, schema handling, and compression.
* We cannot accept full file compression (`.parquet.gz`), instead use **block-level compression** supported within the format.
* **CSV/JSON** — only recommended for small, infrequent loads or when simplicity is required.

### File Structure

```
s3://your-bucket-name/data/table_name/date=YYYY-MM-DD/*.parquet
```

If the data is at hourly level:

```
s3://client-bucket/data/{table_name}/dt=YYYY-MM-DD/hr=HH/events_part_1.csv
```

### Immutability

* Once a file is written, it must never be overwritten, modified, or deleted.
* New or updated records must always be written as new files, not by altering existing ones.

***

## Encryption

If your S3 data is encrypted, additional configuration may be needed depending on the encryption type:

### SSE-S3

No extra steps needed.

### SSE-KMS

Ensure the IAM role has `kms:Decrypt` permissions on the key used. Add the following permissions in the key policy:

```json theme={null}
{
  "Sid": "AllowAuxiaTransferRoleDecrypt",
  "Effect": "Allow",
  "Principal": {
    "AWS": "arn:aws:iam::838011126883:role/AuxiaTransferRole"
  },
  "Action": [
    "kms:Decrypt",
    "kms:DescribeKey"
  ],
  "Resource": "arn:aws:kms:REGION:your-account-id:key/your-kms-key-id"
}
```

<Note>
  Use `AuxiaTransferRoleDev` for DEV integrations.
</Note>

### SSE-C

Not supported by GCS STS — please re-encrypt before transfer.

***

## Need Help?

If you need assistance setting up your S3 bucket policy or KMS key permissions, please contact [support@auxia.io](mailto:support@auxia.io) or your Auxia solutions engineer.