Skip to main content

Amazon Web Services (AWS)

Amazon Web Services (AWS)​

Fluency integrates with various services provided by AWS, such as CloudTrail and CloudWatch. Additionally, Fluency can read from custom SQS queues, or S3 buckets.

The pages in this section detail the steps to ingress AWS data to Fluency. The integrations can be completed individually, following the steps outlined, or all at once, following a CloudFormation script.

The CloudFormation method greatly simplifies deployment, and is the suggested integration procedure, for most use-cases.

For more detailed information regarding AWS usage, refer to the vendor's documentation.

Official AWS Documentation:

https://aws.amazon.com/documentation/

The following AWS services are supported directly:

  • Amazon Simple Storage Service (S3 Bucket)
  • AWS S3 Bucket w/ SQS Notification (by default)
  • Amazon Kinesis (Stream)
  • Kinesis Data Firehose
  • Amazon GuardDuty
  • AWS Lambda
  • Other API Integrations

The following AWS services are supported indirectly, via applications or use-cases of one or more of the services listed above:

  • AWS CloudTrail
  • Amazon CloudWatch
  • CloudWatch Logs
  • CloudWatch Metrics (Monitoring)
  • Amazon Security Lake

Additionally, the above integrations can be simplified / automated via AWS CloudFormation.

Securely Connecting to AWS​

Fluency supports three (3) methods of securely connecting to your AWS Account(s) and Resources.

AWS IAM User​

This method uses an IAM user's access key and secret to connect. The user is created / set up only once per AWS account. Permissions to access additional resources in the same account are appended to this role after role creation. Fluency provides CloudFormation scripts to simplify this process.

Amazon EC2 Instance Role​

As the Fluency instance is hosted on Amazon EC2, it can make use of the EC2 Instance Role concept to securely connect to other AWS Resources and/or Accounts.

A Fluency instance has an EC2 Instance Role assigned by default. In order to use this role to access your AWS Resources, the AssumeRole API is used to allow this Role to securely connect with your resources without the need to pass around IAM credentials.

An IAM Policy w/ External ID will explicitly define the Fluency Instance Role's scope of access to your account. At any time, you may remove the integration and/or intermediary resources to revoke access.

See: https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html

The instance role is created / set up only once per AWS account. Permissions to access additional resources in the same account are appended to this role after role creation. Fluency provides CloudFormation scripts to simplify this process.

Please note that Instance Role is not available for multi-tenant or shared Fluency deployments. (Please use either the IAM User method, or use Access Key/Secrets directly.)

Access Keys / Secrets​

Note: Recommended for existing integrations already in-place, or for migration from other tools/services.

For this method, each integration and account will have its own set of Keys/Secrets, further complicating access control and management.

Fluency Web Interface​

Sign in to your Fluency portal at https://<companyname>.app.ingext.io — see Fluency Web Interface for the URL formats.

Choose the Integrations option under the Platform menu in the navigation bar.

Navigation bar with the Platform menu open on the Integrations option

In the right of the following page, there are two tabs. The first one shows integrations that are already configured, and the second one presents all available integrations in Fluency.

Existing Integrations tab listing the already-configured integrations

Available Integrations tab listing every integration grouped by category

In the left of the page, there is a menu bar to view the existing integrations. Click the desired title/subtitle to view/edit the configuration. For example:

Left-hand menu bar of the Integrations page, listing existing integrations by title

Left-hand menu bar expanded to a subtitle, opening that integration&#39;s configuration

Switch to the second tab, Available Integrations. AWS integrations are found under the section Cloud Infrastructure (IaaS).

Describes an older version of the product

The Integrations screens above are from an earlier build. In the current interface, application templates are browsed and installed from Platform → Applications, using the Install Application From Template view — see Install From Template. Each guide below carries an Install the Application section that matches the current form; follow that rather than the walkthrough above.

In this section​

Start here — CloudTrail is the page that mints the credentials the rest of the AWS set depends on.

  • AWS CloudTrail — management-event ingestion via S3 and SQS, and the IAM credential every other AWS page reuses.
  • Amazon CloudWatch — CloudWatch Logs, with CloudFormation templates for the Firehose, subscription-filter, Kinesis and Lambda paths.
  • EKS Logs — Kubernetes control-plane and container logs.
  • AWS S3 (with SQS) — collecting from an arbitrary bucket by way of an SQS notification queue.
  • AWS Kinesis — reading from a Kinesis data stream.
  • Kinesis Data Firehose — delivery-stream ingestion.
  • Amazon VPC — the VPC integration overview.
  • AWS GuardDuty — GuardDuty findings.
  • AWS Security Lake — OCSF-normalized data from Security Lake.