Kinesis is a streaming connection, best suited to event data that you want reflected in Auxia continuously. If your data already lands in Amazon S3 — including via Kinesis Data Firehose — use the Amazon S3 connection instead.
1. Overview
Auxia consumes the stream from Google Cloud using a Pub/Sub import topic, which reads Kinesis shards directly and writes the records into Auxia’s warehouse as typed columns. There is no intermediate ETL job for you to operate. Authentication uses workload identity federation: Auxia’s Google Cloud service account assumes an Auxia-managed AWS IAM role, and that role reads your stream. No AWS access keys are created or exchanged. Flow:- An Amazon Kinesis data stream carrying the events you want to send to Auxia
- Permission to register a stream consumer and to attach resource policies to the stream and consumer
- Permission to update the KMS key policy, if the stream is encrypted with a customer-managed key (CMK)
- An agreed event schema (see Record Format and Schema)
Kinesis connections are provisioned by Auxia as part of onboarding rather than self-served in the Console. Your Auxia solutions engineer will coordinate the steps below with you.
2. What Auxia Needs From You
Share the following with your Auxia solutions engineer to start the connection:3. Granting Auxia Access
Placeholders used in this sectionReplace these values in every command and policy below with your own:
The Auxia role ARN (
arn:aws:iam::838011126883:role/AuxiaTransferRole) is not a placeholder — use it as written, unless your Auxia solutions engineer gives you a different ARN.Step 1: Register an Enhanced Fan-Out Consumer
Register a stream consumer for Auxia. Enhanced fan-out gives Auxia a dedicated read throughput allocation, so Auxia’s reads do not compete with your existing consumers.Step 2: Grant the Auxia IAM Role Read Access
You do not need to create or manage an AWS IAM role. Instead, grant read access to the Auxia-owned role that Pub/Sub assumes.Auxia IAM Role
This is the same Auxia-managed role used for Amazon S3 transfers. Your Auxia solutions engineer will confirm the exact Role ARN to use for your integration.
stream-policy.json
consumer-policy.json
aws kinesis put-resource-policy — once for the stream, once for the consumer:
Step 3: Grant Decrypt Access for Encrypted Streams
If your stream uses server-side encryption with a customer-managed key (CMK), add the Auxia role to the key policy — without this, Auxia can connect to the stream but cannot read records:Step 4: Auxia Configures Ingestion (Handled by Auxia)
Auxia creates the import topic and the destination table, and validates end to end that records land correctly. No action is required from you for this step.4. Record Format and Schema
Records are parsed into typed columns in Auxia’s warehouse, so the schema must be agreed before you send production traffic.- Format: each Kinesis record must be a single JSON object. Field names map to columns by name.
- Event timestamp: include a timestamp field (for example
event_timestamp). The destination table is day-partitioned on it. - Extra fields: fields that are not part of the agreed schema are ignored rather than failing the record, so you can add fields without breaking ingestion.
- Schema mismatches: records that cannot be parsed into the schema — for example a string in a numeric column — are routed to a dead-letter queue that Auxia monitors, and are retained for 7 days for inspection and replay.
5. Operational Notes
- Resharding is handled automatically — Auxia tracks shard splits and merges without a configuration change on your side.
- Retention: keep your stream’s retention window long enough to absorb a transient outage. The default 24 hours is usually sufficient; longer retention gives a wider replay window.
- Immutability: as with all Auxia sources, records should be append-only. Corrections should be sent as new events rather than by rewriting history.