Atlas Stream Processing Changelog

Adds a new $sessionWindow Atlas Stream Processing pipeline aggregation stage that specifies a session window for aggregation of data. This allow you to run a pipeline on each "session" of activity in an input stream. To learn more, see $sessionWindow.

Supports Azure-hosted Confluent Kafka clusters via Private Link connection.

Adds a new $externalFunction Atlas Stream Processing pipeline stage that triggers processes in a specific AWS Lambda resource. To learn more, see $externalFunction.

Supports using the AWS Transit Gateway to connect to AWS Confluent Cloud clusters. To learn more, see Add a Kafka Transit Gateway Connection.

Supports creating $https connections in the Atlas UI.

Adds the parallelism field to $merge. The field specifies the number of threads to which to distribute write operations, which improves performance.

Allow you to create additional alerts: Output Message Count, DLQ Message Count, Kafka Lag, and Change Stream Delay. To learn more, see Atlas Stream Processing Alerts.

Supports the createUUID expression that takes no arguments and returns UUID BSON type values in Stream Processors.

Adds a new configuration for Window operators to support processingTime in addition to eventTime. To learn more, see Processing Time.

Adds the $meta expression that returns an object containing all streaming metadata for a document.

Adds the parseJsonStrings field to the $https operator, allowing it to parse JSON Strings returned from an API call. To learn more, see $https.

Allows you to add a Kafka Private Link Connection for AWS MSK clusters.

Allows you to deploy Stream Processing Instances on AWS us-east-2. To learn more, see Atlas Stream Processing Regions.

Supports the $currentDate expression that returns the system time of your stream processing instance each time Atlas Stream Processing evaluates it.

Supports reading JSON documents with embedded file signatures (magic bytes).

Fixes an issue that prevented the configuration of hopping windows with hopSize greater than interval.

Changes the executionTimeSecs stat to executionTimeMillis. To view this stat, invoke the sp.processor.stats() command.

Changes the buffering duration for $emit to Apache Kafka sinks from 1000 milliseconds to five milliseconds.

Supports VPC Peering with Confluent on AWS.

Supports using the Atlas Administration API to edit stream processors.

Supports AWS PrivateLink with Confluent.

Supports Azure Private Link with EventHubs.

Increases Apache Kafka Timeout from 10 seconds to 30 seconds.

Always retries processors when the target cluster is in the REPAIRING state.

Fixes the partitionIdleTimeout after the checkpoint restore.

Adds more watermark and window information to verbose stats.

Ensures calls to produce messages to Apache Kafka don't hang indefinitely, even when the producer queue is full.

Supports ingesting multitopic Apache Kafka sources.

Updates Apache Kafka offsets for consumer groups to near-real time.

Supports the compression.type and acks settings in the Apache Kafka $emit configuration.

Supports collectionless $lookup syntax.

Standardizes the way Atlas Stream Processing serializes Apache Kafka errors into a string.

Increases the $emit and $merge dynamic expression limit to 1000.

Handles $lookup with unique connectionName inside window pipeline.

Improves paused cluster errors.

Supports converting from Binary data to various data types using $convert. This allows you to transform Kafka headers into specific data types as needed.

Stream processors that use Apache Kafka as a source expose per-partition watermarks in sp.processor.stats().

Fixes a bug where users were unable to create stream processing instances in certain regions through the API.