Amazon Kinesis
Auxia can ingest events directly from an Amazon Kinesis Data Stream in your AWS account, in near real time. Auxia reads from the stream using an enhanced fan-out (EFO) consumer — you keep ownership of the stream, no data is pushed to a third party, and no agent or connector needs to be installed in your environment.
Before proceeding, review the Source Data Requirements to ensure your data is ready to be connected with Auxia.
Kinesis is a streaming connection, best suited to event data that you want reflected in Auxia continuously. If your data already lands in Amazon S3 — including via Kinesis Data Firehose — use the Amazon S3 connection instead.
1. Overview
Auxia consumes the stream from Google Cloud using a Pub/Sub import topic, which reads Kinesis shards directly and writes the records into Auxia's warehouse as typed columns. There is no intermediate ETL job for you to operate.
Authentication uses workload identity federation: Auxia's Google Cloud service account assumes an Auxia-managed AWS IAM role, and that role reads your stream. No AWS access keys are created or exchanged.
Flow:
Your environment (AWS) Auxia (Google Cloud)
│
│ Kinesis Data Stream
│ │
│ │ enhanced fan-out consumer registered for Auxia
│ │
│ │ read-only access granted to the Auxia IAM role
│ │ via stream + consumer resource policies
│ └──────────────────────────────────────────────────────>
│ │
│ │ Pub/Sub import topic
│ │ (reads shards, adapts to resharding)
│ ▼
│ Auxia warehouse
│ (typed, partitioned event table)
Time required: ~30 minutes on the AWS side.
Prerequisites:
- An Amazon Kinesis data stream carrying the events you want to send to Auxia
- Permission to register a stream consumer and to attach resource policies to the stream and consumer
- Permission to update the KMS key policy, if the stream is encrypted with a customer-managed key (CMK)
- An agreed event schema (see Record Format and Schema)
Kinesis connections are provisioned by Auxia as part of onboarding rather than self-served in the Console. Your Auxia solutions engineer will coordinate the steps below with you.
2. What Auxia Needs From You
Share the following with your Auxia solutions engineer to start the connection:
| Detail | Example |
|---|---|
| Stream ARN | arn:aws:kinesis:us-east-1:111122223333:stream/your-stream-name |
| AWS region | us-east-1 |
| Shard count / expected throughput | 8 shards, ~2,000 records/sec peak |
| KMS key ARN (only if the stream uses a CMK) | arn:aws:kms:us-east-1:111122223333:key/abcd-1234 |
| Event schema | Field names, types, and the event timestamp field |
3. Granting Auxia Access
Replace these values in every command and policy below with your own:
| Placeholder | Replace with |
|---|---|
us-east-1 | The region your stream is in |
111122223333 | Your AWS account ID |
your-stream-name | Your Kinesis stream name |
auxia:1234567890 | The consumer name and suffix returned in Step 1 |
your-kms-key-id | Your KMS key ID, if the stream uses a CMK (Step 3) |
The Auxia role ARN (arn:aws:iam::838011126883:role/AuxiaTransferRole) is not a placeholder — use it as written, unless your Auxia solutions engineer gives you a different ARN.
Step 1: Register an Enhanced Fan-Out Consumer
Register a stream consumer for Auxia. Enhanced fan-out gives Auxia a dedicated read throughput allocation, so Auxia's reads do not compete with your existing consumers.
aws kinesis register-stream-consumer \
--stream-arn arn:aws:kinesis:us-east-1:111122223333:stream/your-stream-name \
--consumer-name auxia
Share the resulting consumer ARN with Auxia. It looks like:
arn:aws:kinesis:us-east-1:111122223333:stream/your-stream-name/consumer/auxia:1234567890
Step 2: Grant the Auxia IAM Role Read Access
You do not need to create or manage an AWS IAM role. Instead, grant read access to the Auxia-owned role that Pub/Sub assumes.
Auxia IAM Role
| Environment | ARN | Name |
|---|---|---|
| Production | arn:aws:iam::838011126883:role/AuxiaTransferRole | AuxiaTransferRole |
This is the same Auxia-managed role used for Amazon S3 transfers. Your Auxia solutions engineer will confirm the exact Role ARN to use for your integration.
Because the stream lives in your account, access is granted with resource policies on the stream and on the consumer.
Attach a resource policy to the stream, allowing Auxia to enumerate shards:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "AllowAuxiaListShards",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::838011126883:role/AuxiaTransferRole"
},
"Action": [
"kinesis:ListShards"
],
"Resource": "arn:aws:kinesis:us-east-1:111122223333:stream/your-stream-name"
}
]
}
Attach a resource policy to the consumer, allowing Auxia to read records:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "AllowAuxiaSubscribeToShard",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::838011126883:role/AuxiaTransferRole"
},
"Action": [
"kinesis:SubscribeToShard",
"kinesis:DescribeStreamConsumer"
],
"Resource": "arn:aws:kinesis:us-east-1:111122223333:stream/your-stream-name/consumer/auxia:1234567890"
}
]
}
Apply each policy with aws kinesis put-resource-policy — once for the stream, once for the consumer:
aws kinesis put-resource-policy \
--resource-arn arn:aws:kinesis:us-east-1:111122223333:stream/your-stream-name \
--policy file://stream-policy.json
aws kinesis put-resource-policy \
--resource-arn arn:aws:kinesis:us-east-1:111122223333:stream/your-stream-name/consumer/auxia:1234567890 \
--policy file://consumer-policy.json
Step 3: Grant Decrypt Access for Encrypted Streams
If your stream uses server-side encryption with a customer-managed key (CMK), add the Auxia role to the key policy — without this, Auxia can connect to the stream but cannot read records:
{
"Sid": "AllowAuxiaTransferRoleDecrypt",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::838011126883:role/AuxiaTransferRole"
},
"Action": [
"kms:Decrypt",
"kms:DescribeKey"
],
"Resource": "arn:aws:kms:us-east-1:111122223333:key/your-kms-key-id"
}
Streams encrypted with the AWS-managed Kinesis key require no additional configuration.
Step 4: Auxia Configures Ingestion (Handled by Auxia)
Auxia creates the import topic and the destination table, and validates end to end that records land correctly. No action is required from you for this step.
4. Record Format and Schema
Records are parsed into typed columns in Auxia's warehouse, so the schema must be agreed before you send production traffic.
- Format: each Kinesis record must be a single JSON object. Field names map to columns by name.
- Event timestamp: include a timestamp field (for example
event_timestamp). The destination table is day-partitioned on it. - Extra fields: fields that are not part of the agreed schema are ignored rather than failing the record, so you can add fields without breaking ingestion.
- Schema mismatches: records that cannot be parsed into the schema — for example a string in a numeric column — are routed to a dead-letter queue that Auxia monitors, and are retained for 7 days for inspection and replay.
Sending traffic before the schema is registered means those records dead-letter instead of landing. Confirm the schema with your Auxia solutions engineer before you cut over production traffic.
5. Operational Notes
- Resharding is handled automatically — Auxia tracks shard splits and merges without a configuration change on your side.
- Retention: keep your stream's retention window long enough to absorb a transient outage. The default 24 hours is usually sufficient; longer retention gives a wider replay window.
- Immutability: as with all Auxia sources, records should be append-only. Corrections should be sent as new events rather than by rewriting history.
Need Help?
If you need assistance registering the consumer, applying resource policies, or granting KMS key permissions, please contact support@auxia.io or your Auxia solutions engineer.