Events and User Attributes ingestion via Pub/Sub

Overview

This document outlines how to publish user events and user attributes using the Auxia system on Pub/Sub. It covers the message format Auxia expects, the schemas, and samples of the resulting messages from Pub/Sub.

Client Requirements

To set up Pub/Sub integration, please provide:

Topic Name (Full Path)
- Example: projects/my-company/topics/user-events
Permission on the Topic
- Grant Auxia’s service account (data-transfer@auxia-gcp.iam.gserviceaccount.com) the pubsub.subscriptions.create permission on the above topic, allowing us to create a subscription on our end.

Message format requirements

Each message published to the topic must follow these requirements so that Auxia can process it:

The message data is a binary-encoded protobuf. Serialize each LogEventsRequest (events) or UserAttributePb (user attributes) with the standard protobuf binary encoding, for example SerializeToString() in Python or toByteArray() in Java and Kotlin, and publish the resulting bytes as the message data. Textproto and JSON payloads are not supported. The textproto samples on this page are shown for readability only.
Each message has a data_type attribute. Set it to EventData for events and UserAttribute for user attributes.

Use the Auxia schemas for events and user attributes exactly as written, including field names, field numbers, and types. Messages that do not follow these requirements cannot be processed.

Events:

Schema for User Events:

Please refer to the schema specified in the documentation for Event Transaction data. Use the exact schema provided at the following link: Auxia User Events Schema.

Textproto Format for Publishing Events:

The sample below shows a LogEventsRequest in textproto for readability. Publish it as a binary-encoded protobuf, as described in Message format requirements. The schema for the LogEventsRequest and ExternalEventPb messages is as per the documentation. Sample textproto:

project_id: "1874"
user_id: "12345"
events {
  event_name: "shop_transaction"
  event_properties {
    key: "spent_current_ammount"
    value {
      long_value: 200
    }
  }
  event_properties {
    key: "shop_id"
    value {
      long_value: 123
    }
  }
  event_properties {
    key: "shop_item_ids"
    value {
      string_value: "abc, def, ghi"
    }
  }
  client_event_timestamp {
    seconds: 1737549801
    nanos: 915585000
  }
  server_received_timestamp {
    seconds: 1737542601
    nanos: 0
  }
}

Sample Message Pulled from Pub/Sub:

{
"ackId": "<>",
"message": {
	"attributes": {
	"data_type": "EventData"
	},
	"data": "CgQxODc0EgUxMjM0NRp3ChBzaG9wX3RyYW5zYWN0aW9uGiAKDXNob3BfaXRlbV9pZHMSDxoNYWJjLCBkZWYsIGdoaRoNCgdzaG9wX2lkEgIIexocChVzcGVudF9jdXJyZW50X2FtbW91bnQSAwjIASoMCOnPw7wGEOjvyrQDMgYIyZfDvAY=",
	"messageId": "<>",
	"publishTime": "2025-01-22T12:43:22.345Z"
	}
}

Decoding the Message from Pub/Sub:

After pulling the message from Pub/Sub, it needs to be decoded into a readable format. The data field in the message contains a base64-encoded string. This string must be decoded into a JSON structure that matches the schema of the LogEventsRequest. Sample decoded message:

{
  "projectId": "1874",
  "userId": "12345",
  "events": [
    {
      "eventName": "shop_transaction",
      "eventProperties": {
        "spent_current_ammount": {
          "longValue": "200"
        },
        "shop_id": {
          "longValue": "123"
        },
        "shop_item_ids": {
          "stringValue": "abc, def, ghi"
        }
      },
      "clientEventTimestamp": "2025-01-22T12:43:21.915585Z",
      "serverReceivedTimestamp": "2025-01-22T10:43:21Z"
    }
  ]
}

User Attributes:

Schema for User Attributes:

Please refer to the documentation of User Attributes. Use the exact schema below:

syntax = "proto3";

import "google/protobuf/timestamp.proto";

message UserAttributePb {
  string user_id = 1;
  map<string, PropertyValuePb> user_properties = 2;
  google.protobuf.Timestamp update_timestamp = 3;
}

message PropertyValuePb {
  oneof property_value {
    int64 long_value = 1;
    double double_value = 2;
    string string_value = 3;
    bool boolean_value = 4;
    google.protobuf.Timestamp timestamp_value = 5;
  }
}

Textproto Format for Publishing User Attributes:

The sample below shows a UserAttributePb in textproto for readability. Publish it as a binary-encoded protobuf, as described in Message format requirements. The schema for the UserAttributePb message is as per the schema mentioned above. Sample textproto:

user_id: "12345"
user_properties {
  key: "language"
  value {
    string_value: "en"
  }
}
user_properties {
  key: "last_access_country"
  value {
    string_value: "us"
  }
}
user_properties {
  key: "country"
  value {
    string_value: "ja"
  }
}
update_timestamp {
  seconds: 1737551302
  nanos: 693271000
}

Sample Message Pulled from Pub/Sub:

{
  "ackId": "<>",
  "message": {
    "attributes": {
      "data_type": "UserAttribute"
    },
    "data": "CgUxMjM0NRIPCgdjb3VudHJ5EgQaAmphEhsKE2xhc3RfYWNjZXNzX2NvdW50cnkSBBoCdXMSEAoIbGFuZ3VhZ2USBBoCZW4aDAjG28O8BhDY88nKAg==",
    "messageId": "<>",
    "publishTime": "2025-01-22T13:08:23.141Z"
  }
}

Decoding the Message from Pub/Sub:

{
  "userId": "12345",
  "userProperties": {
    "language": {
      "stringValue": "en"
    },
    "last_access_country": {
      "stringValue": "us"
    },
    "country": {
      "stringValue": "ja"
    }
  },
  "updateTimestamp": "2025-01-22T13:08:22.693271Z"
}

​Overview

​Client Requirements

​Message format requirements

​Events:

​Schema for User Events:

​Textproto Format for Publishing Events:

​Sample Message Pulled from Pub/Sub:

​Decoding the Message from Pub/Sub:

​User Attributes:

​Schema for User Attributes:

​Textproto Format for Publishing User Attributes:

​Sample Message Pulled from Pub/Sub:

​Decoding the Message from Pub/Sub:

Overview

Client Requirements

Message format requirements

Events:

Schema for User Events:

Textproto Format for Publishing Events:

Sample Message Pulled from Pub/Sub:

Decoding the Message from Pub/Sub:

User Attributes:

Schema for User Attributes:

Textproto Format for Publishing User Attributes:

Sample Message Pulled from Pub/Sub:

Decoding the Message from Pub/Sub: