> ## Documentation Index
> Fetch the complete documentation index at: https://docs.auxia.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Amazon Kinesis

> Streaming event ingestion from an Amazon Kinesis data stream

Auxia can ingest events directly from an **Amazon Kinesis Data Stream** in your AWS account, in near real time. Auxia reads from the stream using an **enhanced fan-out (EFO) consumer** — you keep ownership of the stream, no data is pushed to a third party, and no agent or connector needs to be installed in your environment.

Before proceeding, review the [Source Data Requirements](/data-ingestion/overview#source-data-requirements) to ensure your data is ready to be connected with Auxia.

<Note>
  Kinesis is a **streaming** connection, best suited to event data that you want reflected in Auxia continuously. If your data already lands in Amazon S3 — including via Kinesis Data Firehose — use the [Amazon S3](/data-ingestion/data-connections/amazon-s3) connection instead.
</Note>

## 1. Overview

Auxia consumes the stream from Google Cloud using a Pub/Sub **import topic**, which reads Kinesis shards directly and writes the records into Auxia's warehouse as typed columns. There is no intermediate ETL job for you to operate.

Authentication uses **workload identity federation**: Auxia's Google Cloud service account assumes an Auxia-managed AWS IAM role, and that role reads your stream. No AWS access keys are created or exchanged.

**Flow:**

```
Your environment (AWS)                         Auxia (Google Cloud)
      │
      │  Kinesis Data Stream
      │       │
      │       │  enhanced fan-out consumer registered for Auxia
      │       │
      │       │  read-only access granted to the Auxia IAM role
      │       │  via stream + consumer resource policies
      │       └──────────────────────────────────────────────────────>
      │                                              │
      │                                              │  Pub/Sub import topic
      │                                              │  (reads shards, adapts to resharding)
      │                                              ▼
      │                                         Auxia warehouse
      │                                         (typed, partitioned event table)
```

**Time required:** \~30 minutes on the AWS side.

**Prerequisites:**

* An Amazon Kinesis data stream carrying the events you want to send to Auxia
* Permission to register a stream consumer and to attach **resource policies** to the stream and consumer
* Permission to update the **KMS key policy**, if the stream is encrypted with a customer-managed key (CMK)
* An agreed event schema (see [Record Format and Schema](#4-record-format-and-schema))

<Note>
  Kinesis connections are provisioned by Auxia as part of onboarding rather than self-served in the Console. Your Auxia solutions engineer will coordinate the steps below with you.
</Note>

## 2. What Auxia Needs From You

Share the following with your Auxia solutions engineer to start the connection:

| Detail | Example |
| - | - |
| Stream ARN | `arn:aws:kinesis:us-east-1:111122223333:stream/your-stream-name` |
| AWS region | `us-east-1` |
| Shard count / expected throughput | `8 shards, ~2,000 records/sec peak` |
| KMS key ARN (only if the stream uses a CMK) | `arn:aws:kms:us-east-1:111122223333:key/abcd-1234` |
| Event schema | Field names, types, and the event timestamp field |

## 3. Granting Auxia Access

<Note>
  **Placeholders used in this section**

  Replace these values in every command and policy below with your own:

  | Placeholder | Replace with |
  | - | - |
  | `us-east-1` | The region your stream is in |
  | `111122223333` | Your AWS account ID |
  | `your-stream-name` | Your Kinesis stream name |
  | `auxia:1234567890` | The consumer name and suffix returned in Step 1 |
  | `your-kms-key-id` | Your KMS key ID, if the stream uses a CMK (Step 3) |

  The Auxia role ARN (`arn:aws:iam::838011126883:role/AuxiaTransferRole`) is **not** a placeholder — use it as written, unless your Auxia solutions engineer gives you a different ARN.
</Note>

### Step 1: Register an Enhanced Fan-Out Consumer

Register a stream consumer for Auxia. Enhanced fan-out gives Auxia a dedicated read throughput allocation, so Auxia's reads do not compete with your existing consumers.

```bash theme={null}
aws kinesis register-stream-consumer \
  --stream-arn arn:aws:kinesis:us-east-1:111122223333:stream/your-stream-name \
  --consumer-name auxia
```

Share the resulting **consumer ARN** with Auxia. It looks like:

```
arn:aws:kinesis:us-east-1:111122223333:stream/your-stream-name/consumer/auxia:1234567890
```

### Step 2: Grant the Auxia IAM Role Read Access

You **do not need to create or manage an AWS IAM role**. Instead, grant read access to the Auxia-owned role that Pub/Sub assumes.

#### Auxia IAM Role

| Environment | ARN | Name |
| - | - | - |
| **Production** | `arn:aws:iam::838011126883:role/AuxiaTransferRole` | `AuxiaTransferRole` |

<Note>
  This is the same Auxia-managed role used for [Amazon S3](/data-ingestion/data-connections/amazon-s3) transfers. Your Auxia solutions engineer will confirm the exact Role ARN to use for your integration.
</Note>

Because the stream lives in your account, access is granted with **resource policies** on the stream and on the consumer.

Attach a resource policy to the **stream**, allowing Auxia to enumerate shards:

```json title="stream-policy.json" theme={null}
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowAuxiaListShards",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::838011126883:role/AuxiaTransferRole"
      },
      "Action": [
        "kinesis:ListShards"
      ],
      "Resource": "arn:aws:kinesis:us-east-1:111122223333:stream/your-stream-name"
    }
  ]
}
```

Attach a resource policy to the **consumer**, allowing Auxia to read records:

```json title="consumer-policy.json" theme={null}
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowAuxiaSubscribeToShard",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::838011126883:role/AuxiaTransferRole"
      },
      "Action": [
        "kinesis:SubscribeToShard",
        "kinesis:DescribeStreamConsumer"
      ],
      "Resource": "arn:aws:kinesis:us-east-1:111122223333:stream/your-stream-name/consumer/auxia:1234567890"
    }
  ]
}
```

Apply each policy with `aws kinesis put-resource-policy` — once for the stream, once for the consumer:

```bash theme={null}
aws kinesis put-resource-policy \
  --resource-arn arn:aws:kinesis:us-east-1:111122223333:stream/your-stream-name \
  --policy file://stream-policy.json
```

```bash theme={null}
aws kinesis put-resource-policy \
  --resource-arn arn:aws:kinesis:us-east-1:111122223333:stream/your-stream-name/consumer/auxia:1234567890 \
  --policy file://consumer-policy.json
```

### Step 3: Grant Decrypt Access for Encrypted Streams

If your stream uses **server-side encryption with a customer-managed key (CMK)**, add the Auxia role to the key policy — without this, Auxia can connect to the stream but cannot read records:

```json theme={null}
{
  "Sid": "AllowAuxiaTransferRoleDecrypt",
  "Effect": "Allow",
  "Principal": {
    "AWS": "arn:aws:iam::838011126883:role/AuxiaTransferRole"
  },
  "Action": [
    "kms:Decrypt",
    "kms:DescribeKey"
  ],
  "Resource": "arn:aws:kms:us-east-1:111122223333:key/your-kms-key-id"
}
```

Streams encrypted with the AWS-managed Kinesis key require no additional configuration.

### Step 4: Auxia Configures Ingestion (Handled by Auxia)

Auxia creates the import topic and the destination table, and validates end to end that records land correctly. No action is required from you for this step.

## 4. Record Format and Schema

Records are parsed into typed columns in Auxia's warehouse, so the schema must be agreed **before** you send production traffic.

* **Format:** each Kinesis record must be a single **JSON object**. Field names map to columns by name.
* **Event timestamp:** include a timestamp field (for example `event_timestamp`). The destination table is day-partitioned on it.
* **Extra fields:** fields that are not part of the agreed schema are ignored rather than failing the record, so you can add fields without breaking ingestion.
* **Schema mismatches:** records that cannot be parsed into the schema — for example a string in a numeric column — are routed to a dead-letter queue that Auxia monitors, and are retained for 7 days for inspection and replay.

<Warning>
  Sending traffic before the schema is registered means those records dead-letter instead of landing. Confirm the schema with your Auxia solutions engineer before you cut over production traffic.
</Warning>

## 5. Operational Notes

* **Resharding** is handled automatically — Auxia tracks shard splits and merges without a configuration change on your side.
* **Retention:** keep your stream's retention window long enough to absorb a transient outage. The default 24 hours is usually sufficient; longer retention gives a wider replay window.
* **Immutability:** as with all Auxia sources, records should be append-only. Corrections should be sent as new events rather than by rewriting history.

## Need Help?

If you need assistance registering the consumer, applying resource policies, or granting KMS key permissions, please contact [support@auxia.io](mailto:support@auxia.io) or your Auxia solutions engineer.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.