Data Sources

Data enters the Platform from a Data Source. The first step in collecting data is getting access to the data: this requires "connecting" the data source to the pipeline.

You define how to connect to a data source by its protocol or type. The Platform supports the following data source types:

v2 interface

1. Plugin Data Source
2. API Pull
3. HTTPs Event Collector (HEC)
4. Prometheus
5. Kinesis Stream
6. AWS S3
7. AWS S3 with SQS
8. Webhook
9. Cloud Syslog
10. Management Queue

Classic v1 interface

1. API Plugin
2. HTTPs Event Collector (HEC)
3. Kinesis Stream
4. AWS S3
5. AWS S3 with SQS
6. Webhook
7. Cloud Syslog
8. Management Queue

Notice that this is not a product list, such as CrowdStrike Falcon or SentinelOne. Connecting to a data source is just a means to move the data from the product. Products often document multiple connection means.

Where is the Add button?

v2 interface

There are three (3) columns on the Platform page (Platform in the navigation rail), where data collection is configured: Sources, Routers, and Sinks. The Sources column is the definition of the data sources. At the bottom of that column is the Add Source + button.

Classic v1 interface

There are three (3) columns on the Platform Configuration page (Platform → Configuration), where data collection is configured: Sources, Routers, and Sinks. The Sources column is the definition of the data sources. At the bottom of that column is the Add Source + button.

Each source card shows a type icon, the source name, and a label for the transport that feeds it, for example Plugin Data Source (data pulled by an installed application), Mgmt Queue (an internal management queue), AWS S3 w/ SQS (an S3 bucket read via SQS notifications), or Cloud Syslog. The outline reports the source's state: green while it is actively moving data, gray while it is idle, and red, with an exclamation mark (!) above the card, when it is in error. A green source can also carry a small green throughput graph above the card's upper-right corner; an idle one has none. A colored line runs from the source's output port to the router it feeds.

Clicking Add Source + begins the process.

Configuration

v2 interface

Add Source + opens the Adding Data Source panel. The first step is to choose the way you want to import that data source, in the Type list; Plugin Data Source is chosen when the panel opens:

Once a type is chosen, the fields needed for that connector appear, on a Basic Info tab and, for most types, an Advanced tab for the source's properties JSON and its FPL receiver. Required fields carry an asterisk (*). Save at the foot creates the source; × closes the panel without saving. Adding a Source lists every type's fields.

Classic v1 interface

Legacy interface

The properties-panel flow and screenshots below show the legacy Configuration page. The current page is shown under Where is the Add button? above.

Properties and status of a pipe appear in a properties panel. Once you click Add Source +, an empty properties window will appear.

The first step is to choose the way you want to import that data source.

Once a type is chosen, the properties needed for that connector will appear.

Naming the Pipe

Regardless of the type of data source you are configuring, every pipe has a name. This is similar to a file name in an operating system. This is the name that will appear in the Configuration panel. It will also be used when reviewing the platform's usage and metrics charts: the Events tab of the source's panel on the Platform page (v1: Platform → Usage).

When naming the pipe, give it a name that lets users know what type of data is coming from this source, like "Fortinet Firewalls."

Reconfiguring a Data Source

v2 interface

Click the source's card to open its panel. The Details tab shows its settings. The Active switch under STATUS turns the source off and on, after a confirmation; Disconnect Source From Router, at the panel's foot, takes an inactive source off its router, and Reboot restarts it. Once the source feeds no router, Delete takes the place of Disconnect Source From Router: a source is deleted only after it has been disconnected. See Source Panel.

Classic v1 interface

There is an edit icon (pencil) next to fields that can be changed. To edit fields:

1. Stop the process.
2. Then make the changes to the fields.
3. Save the changes (disk icon).
4. Restart the process.

Not all fields can be edited. In some cases, it is easier to:

1. Create a new data source connector.
2. Stop the old data source.
3. Delete the previous connection.
4. Connect the new data source to the processor.
5. Start the new source.
6. Validate the new source is operating correctly.
7. Delete the old data source connector.

Demo

An interactive demo of adding a data source is available here: Adding a Data Source (Storylane)