Supported Aggregation Pipeline Stages

Atlas Stream Processing extends the aggregation pipeline with stages for processing continuous data streams. These stages combine with existing aggregation stages built in to the default mongod process, enabling you to perform many of the same operations on continuous data as you can perform on data-at-rest. An Atlas Stream Processing pipeline definition cannot exceed 16 MB.

The following table lists the aggregation pipeline stages unique to Atlas Stream Processing and those which exist in modified form in Atlas Stream Processing.

Note

For details on a specific operator, including syntax and examples, click on the link to the operator's reference page.

Aggregation Pipeline Stage	Purpose
`$source`	Specifies a streaming data source to consume messages from.
`$validate`	Validates the documents of a stream against a user-defined schema.
`$https`	Supports `curl`-like calls to arbitrary REST API endpoints within your stream processor.
$lookup	Performs a left outer join to a specified collection to filter in documents from the "joined" collection for processing. This version of the existing `$lookup` stage requires that you specify a Atlas collection in the Connection Registry as the value for the `from` field.
`$hoppingWindow`	Assigns documents from a stream to windows with user-defined durations and intervals between start times.
`$tumblingWindow`	Assigns documents from a stream to non-overlapping, continuous windows with user-defined durations.
$emit	Specifies a stream or time series collection in the connection registry to emit messages to.
$merge	A version of the existing `$merge` stage where the value of the `connectionName` field must always be the name of a remote collection in the Connection Registry.
$sessionWindow	Runs a pipeline on each "session" of activity in an input stream. Each document is assigned to a session window based on its partition and timestamp.

Supported Aggregation Pipeline Stages

Atlas Stream Processing provides a number of extensions to the core MongoDB Aggregation Pipeline syntax. To learn more about these extensions, see Supported Aggregation Pipeline Stages.

Certain core aggregation pipeline stages are only supported when nested within other pipeline stages. The following table shows which aggregation pipeline stages are supported for use within the main pipeline or when nested within the specified aggregation pipeline stages.

Aggregation Stage	Supported in Main Pipeline	Supported in `$tumblingWindow`, `$hoppingWindow`, and `$sessionWindow`	Supported in `$https` and `$externalFunction`
`$addFields`	√	√	√
`$project`	√	√	√
`$replaceRoot`	√	√	√
`$set`	√	√	√
`$redact`	√	√
`$match`	√	√
`$replaceWith`	√	√
`$unset`	√	√
`$unwind`	√	√
`$validate`	√	√
`$https`	√	√
`$externalFunction`	√	√
`$lookup` [1]	√	√
`$merge` [1]	√
`$tumblingWindow`	√
`$hoppingWindow`	√
`$sessionWindow`	√
$emit	√
`$group`		√
`$sort`		√
`$count`		√
`$limit`		√

[1]	(1, 2) Supported with modified syntax.

Back

$createUUID

$source