Skip to main content

AWS S3 with SQS Notifications

A delivery path, not a template

S3 with SQS notification is not an application template; it is the delivery mechanism several templates use. The AWS CloudTrail, CloudWatch LogGroup, and AWS EKS Logs templates each ask for an SQS queue URL and consume the bucket through it, so this page describes the AWS-side wiring those templates depend on. See Integrations for the templates themselves.

Fluency has the ability to directly integrate with Amazon S3 Buckets to read (and write) the data objects within.

A limitation of the AWS S3 API disallows the listing or sorting of objects by time. Instead, only a lexicographical sort of the object name is allowed. As such, directly consuming a data stream from an S3 bucket requires that the objects are created/added in order, both lexicographically and chronologically (i.e. using timestamps as part of the filename).

For buckets where data is added non-lexicographically, AWS S3 buckets have integration with AWS SQS (Simple Queue Service), by way of Event Notifications. This SQS addition allows a consumer to be "notified" when a new object has been added to the S3 bucket, regardless of the sorting of the object name.

By default, Fluency suggests making use of this SQS addition to be notified of new data as they arrive in S3.

Existing S3 Buckets​

For clients with existing S3 Buckets, AWS allows the attachment of an SQS queue. This can be done manually, or by following this CloudFormation script:

https://fluency-cloudformation.s3.us-east-2.amazonaws.com/FluencyPlatformS3AddNotification.yaml

Manual Process​

  1. Create an SQS queue
  2. Create / Attach an IAM policy to allow the S3 Bucket to post Event Notifications to the SQS queue
  3. Configure the S3 Bucket to send Event Notifications
    1. Select the S3 Bucket, Click "Properties" → "Event notifications" → "Create event notification"
    2. Select "All object create events"
    3. Select "SQS queue" as the destination, choosing the SQS queue from step 1

Fluency Configuration​

Login to the Fluency portal.

Select the Integrations option under the Platform menu in the navigation bar. Choose the AWS S3 w/ SQS integration (S3Notification), under the New Integration view, and fill in the required fields.