Skip to main content

HTTP Event Collector (HEC)

HTTP Event Collector (HEC) is a method for sending event data over HTTP or HTTPS directly to a receiver, typically in JSON format. It's commonly used for real-time ingestion of data from various sources for indexing and analysis purposes. The use of HTTPS ensures secure transmission of the data.

In the Fluency Platform, HEC can be either a data source or a data sink.

HEC as a Platform Data Source​

You add data sources from the Platform Configuration page.

Menu Choice

Step One: Add the Data Source​

Each column of the Platform Configuration has a button on the bottom. To start, click Add Source + on the bottom of the first (Sources) column.

Platform Configuration page with the Add Source + button at the bottom of the Sources column

This will open up a panel for configuring the properties needed to connect.

Step Two: Select HTTPs Event Collector as the type of Data Source​

When you first click on Add Source +, a properties panel appears. The "API Plugin" appears first. You need to change this to "HTTPs Event Collector."

Source properties panel with the type changed from API Plugin to HTTPs Event Collector

Once you select the HEC, the panel will display the properties form that needs to be completed.

HEC Properties

Step Three: Properties Values​

The form properties are:

  • Name: This is the name of the data source in Fluency.
  • Format: This is the type of data inside the HEC.
    • JSON: JavaScript Object Notation.
    • Text: Raw character stream.
    • Syslog: System Log Protocol.
  • Enable Indexer: Some HEC senders require acknowledgement of the send. Click this option if this is the case.
  • Receiver Name
  • Properties of JSON
  • Tags

Step Four: Save and Connect to Router​

Remember to save the configuration.

Next you will need to create a router. Routers are used to transform, enrich, and route the data to a sink. The choice of the format will determine how the router parses the data.

HEC as a Data Sink​

HEC can also be used as a data sink, to send data from Fluency to a server that uses the HEC protocol.

Splunk-compatible HEC​

The Splunk-compatible HTTP Event Collector lets you send data and application events to a SIEM deployment over the HTTP and Secure HTTP (HTTPS) protocols. HEC uses a token-based authentication model.

You can generate a token and then configure a logging library or HTTP client with that token to send data to an HEC in a specific format. This process eliminates the need for a custom forwarder when you send application events.

Fluency supports Splunk's HEC specifications, allowing applications which already support Splunk data export to effortlessly and securely send data to Fluency.

The HEC Input Passthrough Template​

For a source that can post to a Splunk-compatible HEC endpoint but has no template of its own, the catalog carries a generic receiver. Go to Platform → Applications → Install Application From Template and choose HEC Input Passthrough from the Endpoint Management category. Both parameters are required:

ParameterNotes
datalakePre-filled managed.
datalake index namePre-filled default — change it. Two applications writing to the same index name in the same lake will collide, and the second one aborts.

Setup New HEC Input Passthrough Application panel

Events land in the named index unparsed, which is the trade: no processor to write, but no field extraction either. The passthrough parser cookbook is the starting point if you need to add parsing later.

Several vendors have their own HEC template, which is preferable where one exists because it ships a parser — each is covered on its own page:

Adding a Fluency plug-in for HEC (HTTP Event Collector)​

Describes an older version of the product

The Cloud Integrations page (under the old "Data Ingress" menu) shown in the steps below does not exist in the current interface. In the current interface, integrations are managed on Platform → Integrations, and an HEC feed is added as an HTTPs Event Collector data source on Platform → Configuration (as described earlier on this page). The steps below need to be re-verified.

Sign in to your Fluency portal at https://<companyname>.app.ingext.io — see Fluency Web Interface for the URL formats.

Open the Main Menu from the upper left-hand corner and choose the Cloud Integrations option under the Data Ingress section.

Main Menu with the Cloud Integrations option under the Data Ingress section

On the following page, navigate to the Event Collectors section.

To Add an integration for HEC, choose the HEC icon from the group on the left side of the page to create a new integration endpoint.

Event Collectors section of the integrations page, with the HEC icon

NOTE: If an integration endpoint was setup previously, you can also select and modify it from the
    section on the right side of the page.

In the pop-up window, provide a Customer name for this integration. Normally, this will just be the name of your deployment or organization. This value will be used solely within Fluency.

Provide a short Application (shortname). The value should describe the product/software or data source that is being integrated. This field will be used for ingress, and therfore, will become part of the data JSON record. Normally, it is suggested to use the product name, written using all lowercase letters, and without spaces.

HEC integration dialog with the Customer name and Application shortname fields

Select the HEC integration endpoint from the list on the right side of the page, in the Event Collectors section. Choose the gear icon to view/configure the connector.

Configured HEC endpoint listed under Event Collectors, with its gear configure button

On the following page, Token and Webhook URL are displayed:

HEC connector page displaying the Token and Webhook URL

Keep these values to use for configuring the HEC in your originating application.