Amazon Kinesis - Auxia Docs

Amazon Kinesis Data Streams is AWS’s real-time streaming service. Auxia’s Kinesis connector reads events directly from a stream in your AWS account, in near real time, using a dedicated enhanced fan-out consumer. You keep ownership of the stream, nothing is pushed to a third party, and there’s no agent or connector to install in your environment.

Setup guide

About 30 minutes on the AWS side

All connectors

Compare connection options

Kinesis is a streaming connection, best suited to event data you want reflected in Auxia continuously. If your data already lands in Amazon S3 — including through Kinesis Data Firehose — use the Amazon S3 connector instead.

Features

Feature	Supported	Notes
Setup	Assisted	Provisioned by Auxia as part of onboarding, together with your team
Near-real-time streaming	✅	Records are read continuously as they arrive
Dedicated read throughput	✅	An enhanced fan-out consumer, so Auxia’s reads don’t compete with your other consumers
No access keys	✅	Access is granted with resource policies to an Auxia-managed role
Encrypted streams	✅	AWS-managed key, or a customer-managed KMS key with a key-policy grant
Automatic resharding	✅	Shard splits and merges are handled without configuration changes
Schema evolution	Partial	New fields are ignored rather than failing records; agree schema changes with Auxia to ingest them
Capture deletes	❌	Records must be append-only

How it works

Auxia consumes your stream from Google Cloud with a Pub/Sub import topic, which reads Kinesis shards directly and writes records into Auxia’s warehouse as typed columns. There’s no intermediate ETL job for you to operate. Authentication uses workload identity federation: Auxia’s Google Cloud service account assumes an Auxia-managed AWS IAM role, and that role reads your stream. No AWS access keys are created or exchanged.

Your AWS account                               Auxia (Google Cloud)
  Kinesis Data Stream
    └─ enhanced fan-out consumer ──────────►  Pub/Sub import topic
       (read access for the Auxia role)         (reads shards, follows resharding)
                                                       │
                                                       ▼
                                                Auxia warehouse
                                                (typed, day-partitioned event table)

Record format and schema

Records are parsed into typed columns, so the schema must be agreed before you send production traffic.

Format: each Kinesis record must be a single JSON object with a fixed shape. Field names map to columns by name.
Event timestamp: include a timestamp field (for example event_timestamp). The destination table is partitioned by day on it.
Extra fields that aren’t part of the agreed schema are ignored rather than failing the record, so you can add fields without breaking ingestion.
Schema mismatches — for example a string in a numeric field — send the record to a dead-letter queue that Auxia monitors. Dead-lettered records are kept for 7 days for inspection and replay.

Records sent before the schema is registered are dead-lettered instead of landing. Confirm the schema with your Auxia solutions engineer before you send production traffic.

Operational notes

Retention: keep your stream’s retention window long enough to absorb a transient outage. The default 24 hours is usually enough; longer retention gives a wider replay window.
Append-only: as with every Auxia source, send corrections as new events rather than rewriting history.

Limitations

Each record must be a single JSON object matching the agreed schema.
Kinesis connections are set up with the Auxia team, not in the self-serve wizard.
See Source data requirements for requirements shared by all connectors.

Setup guide

All connectors

​Features

​How it works

​Record format and schema

​Operational notes

​Limitations

Features

How it works

Record format and schema

Operational notes

Limitations