Skip to main content
Home/Data Ingestion/Data Connections/Amazon S3

Amazon S3

Setting up a data connection via Amazon S3

Overview​

Auxia's integration uses Google Cloud Storage Transfer Service (STS) to securely read data from your AWS S3 bucket via an Auxia-managed AWS IAM role. Data is transferred from your S3 bucket to Auxia's Google Cloud Storage (GCS) infrastructure for model training, feature creation, and performance measurement.

We recommend using Parquet format for optimal performance, schema handling, and compression.

Before proceeding, review the Source Data Requirements to ensure your data is ready to be connected with Auxia.

Overview of the Transfer Flow​

  1. Client's data remains in AWS S3.
  2. GCP initiates the transfer via Storage Transfer Service.
  3. The transfer job reads from your S3 bucket using an Auxia-managed AWS IAM Role.
  4. Permissions are granted for S3 object read operations and, if applicable, decryption.

Setting Up an S3 Connection​

Prerequisites​

  • An AWS S3 bucket containing the data you want to share with Auxia
  • Permission to update your S3 bucket policy
  • Access to the Auxia Console with connection editor permissions

Step 1: Grant Access to an Auxia-Managed IAM Role​

The client does not need to create or manage an AWS IAM role. Instead, the client grants read-only access to an Auxia-owned IAM role that is used by GCP Storage Transfer Service.

Auxia IAM Roles​

EnvironmentARNName
Productionarn:aws:iam::838011126883:role/AuxiaTransferRoleAuxiaTransferRole
Developmentarn:aws:iam::838011126883:role/AuxiaTransferRoleDevAuxiaTransferRoleDev

Step 2: Update Your S3 Bucket Policy​

Update your S3 bucket policy to allow read-only access for the appropriate Auxia role:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::838011126883:role/AuxiaTransferRole"
      },
      "Action": [
        "s3:GetObject",
        "s3:ListBucket",
        "s3:GetBucketLocation"
      ],
      "Resource": [
        "arn:aws:s3:::your-bucket-name",
        "arn:aws:s3:::your-bucket-name/*"
      ]
    }
  ]
}

Replace:

  • your-bucket-name with your actual S3 bucket
  • Role ARN with AuxiaTransferRoleDev if this is a DEV integration

Step 3: Configure Storage Transfer (Handled by Auxia)​

Auxia will initiate the transfer by configuring the Storage Transfer Service on Google Cloud using the appropriate Auxia-managed IAM role. No action is required from the client for this step.

For reference, see GCP Docs on AWS S3 Transfer via Web Identity.

Expected S3 File Structure​

File Format & Compression Recommendations​

FormatSchema SupportCompression SupportEfficiencyNotes
Parquet (Preferred)EmbeddedGZIP, Snappy, ZSTD, LZO, LZ4HighBest for analytical columnar workloads
  • Parquet — prefer for performance, schema handling, and compression.
  • We cannot accept full file compression (.parquet.gz), instead use block-level compression supported within the format.
  • CSV/JSON — only recommended for small, infrequent loads or when simplicity is required.

File Structure​

s3://your-bucket-name/data/table_name/date=YYYY-MM-DD/*.parquet

If the data is at hourly level:

s3://client-bucket/data/{table_name}/dt=YYYY-MM-DD/hr=HH/events_part_1.csv

Immutability​

  • Once a file is written, it must never be overwritten, modified, or deleted.
  • New or updated records must always be written as new files, not by altering existing ones.

Encryption​

If your S3 data is encrypted, additional configuration may be needed depending on the encryption type:

SSE-S3​

No extra steps needed.

SSE-KMS​

Ensure the IAM role has kms:Decrypt permissions on the key used. Add the following permissions in the key policy:

{
  "Sid": "AllowAuxiaTransferRoleDecrypt",
  "Effect": "Allow",
  "Principal": {
    "AWS": "arn:aws:iam::838011126883:role/AuxiaTransferRole"
  },
  "Action": [
    "kms:Decrypt",
    "kms:DescribeKey"
  ],
  "Resource": "arn:aws:kms:REGION:your-account-id:key/your-kms-key-id"
}
note

Use AuxiaTransferRoleDev for DEV integrations.

SSE-C​

Not supported by GCS STS — please re-encrypt before transfer.

Need Help?​

If you need assistance setting up your S3 bucket policy or KMS key permissions, please contact support@auxia.io or your Auxia solutions engineer.