Skip to main content

Processors

Once a data record is available, you want the record to stand alone in a JSON format. Processors address transforming and decorating the record.

Once the data source is connected, a second pipe is connected to it that will parse the data. The parsing of data allows for it to be searched and processed. During this parsing process, data can be checked for errors, types changed, look-ups made, and values normalized. Two common parsing adjustments are:

  1. Adjusting the time to correct for timezone or abnormal clock times.
  2. Adding a User-Entity key for later correlation and scoring.

During the parsing phase, some data processing can occur. Most common is the creation of metric data.

To create a route:

  1. Create a Process. (Needs to be done first on the Configuration → Processors page; v1: Platform → Processors)
  2. Add a route.
  3. Assign the process to the route.

In order to create the route, you need to have a process already defined. If you want to create the pipe structure first, you can define a passthrough process that will just forward a raw event. In this scenario you still want to create the passthrough process with the name of the eventual parser that you are going to implement.

The Processors page​

Configuration → Processors (URL path /v2/configuration/processors) lists the FPL processors defined in the account and is where you create, clone, and delete them.

Processors page on the Processors tab, showing the type tabs, search bar, Only deployed filter, sort drop-down, and a grid of processor cards

Under the Processors title and the subtitle FPL processors, receivers, packers, rules, and actions used on the platform graph.:

  • A row of tabs (Processors, Receivers, Packers, Actions, Rules) selects which FPL object type is listed. The header's Add Processor button is relabelled for the selected tab.
  • Beside the tabs, a search field with a Search button, an Only deployed checkbox that keeps only the processors attached to a pipe, and a sort drop-down offering Name: A-Z, Name: Z-A, and Last updated.
  • The main area is a grid of processor cards. Each card shows the processor name, its description (many start with a label such as Standard:, Default:, Template:, or Tool:), the Last Updated date and time, and Pipes: chips naming the pipes the processor is deployed in, or Not deployed. Its footer names the repository the definition comes from, such as Fluency, beside two buttons: Clone, which opens a copy of the processor in the editor, and Delete, which removes the processor.

The processor editor​

Add Processor opens the editor at /v2/configuration/processors/new?type=fpl_processor. Its header carries Back to Processors, the new processor's name as the title, and an Add button. Beneath the Name (pre-filled New_Processor), Type and Description fields, a Code pane holds a starter FPL main function that returns "pass", with a Validate button in its header, beside an Input pane holding the test record and its Run Test button; Output and Console panes show the test results.

Editor for a new processor named New_Processor, with Name, Type, and Description fields above the Code, Input, Output, and Console panels, and an Add button in the header

Click Add to save the new processor and return to the Processors list.

A card's Clone button opens the same editor on a copy of the processor, at /v2/configuration/processors/new?clone=<name>. The copy is named after the source with _Clone appended, such as AWSCloudTrail_Adjustments_Clone, and carries the source's description and code. A processor whose card names a repository such as Fluency is read-only in the editor (Save Changes is disabled), so clone it to change it.

Processes​

Processes are unique. When you edit a process, you are editing the base processor. If you use that process in more than one place, you are editing both locations.

New processes start from Add Processor, in the header of Configuration → Processors.

Defining the name​

The first step is to define the name and provide a description. The processor you are creating is just like a program that can be run from different command lines (pipes). It is for this reason that when you create a temporary stub (passthrough) you give it the name of the eventual process that you want to create.

In the editor, replace the pre-filled New_Processor in Name, and fill in Description. They sit above the code, so the workspace is already open. Choose the name with care: it must differ from every other processor's, and once the processor is saved the editor shows it read-only (clone the processor to save it under another name).

Processor Workspace​

Editor for the repository-managed AWSCloudTrail_Adjustments processor, with read-only fields, the note Repository-managed definition. Clone to save a local version., a disabled Save Changes button, and Input / Events and Console / Logs tabs

Below the fields, the workspace has four sections:

  1. Code. This is the code that will run when a record is received. A processor starts with a code entry point of main(). The pane's footer gives the cursor position and the language, FPL; Validate, in its header, checks the code.
  2. Input. The test record, as JSON, prefilled with an empty record: {"obj": {}, "size": 0}. Run Test runs the code against it. An existing processor also has an Events tab beside Input, with the records it has recently seen.
  3. Output. This shows the outgoing object that is exiting the processor, read-only.
  4. Console. This panel shows the console standard output that a process may write to. This is done by using a printf command. An existing processor also has a Logs tab beside Console.

If you are used to coding, you will have some insight into the use of each section.

Reopening the editor​

Click the processor's card on Configuration → Processors to reopen it in the editor, at /v2/configuration/processors/edit?name=<name>; Save Changes, in the header, saves what you change. It validates the code first and does not save code that fails validation; see Save Changes.

On the Platform page, a pipe's processor opens the processor panel instead, with the processor's code under PROCESSOR CODE and its traffic on the Events tab; see Processor Panel.

Processor Entry Point​

The default passthrough might not be what you want.

function main({obj, size, source}) {
    return "pass"
}

If you are not sure of the shape of the data (how the record is formatted), try viewing the record with a printf statement once you get the data source set up. To do this, change the above code by replacing it with:

function main(envelope) {
    //
    printf("Raw Record: %s", envelope)
    return "pass"
}

Notice that the text editor can highlight with the JavaScript syntax. This will be useful when you code.

The code above is just capturing the incoming object. The name 'envelope' is irrelevant. You are taking the incoming object and sending it to standard output using the printf command.

Using the Record Buffer​

The record buffer is extremely useful. Sometimes you do not know the shape of an incoming record. For example, this is a Linux object sent from a Syslog server:

{
  "obj": {
    "@collector": "local",
    "@event_type": "sshd",
    "@facility": "authpriv",
    "@level": "info",
    "@message": "Disconnected from invalid user gang 23.99.201.14 port 57468 [preauth]",
    "@parser": "fpl-LinuxServerSyslog",
    "@parserVersion": "20231128-3",
    "@sender": "203.0.113.24",
    "@source": "remote",
    "@sshd": {
      "parserError": "skipped"
    },
    "@tags": [
      "sshd"
    ],
    "@timestamp": 1711202399000,
    "@type": "event"
  },
  "props": {},
  "size": 136,
  "source": ""
}

Note: The JSON format is a String format and not raw.

You can copy this and paste it into the Input pane, in place of the empty record it starts with. Make sure the pane is on its Input tab rather than Events.

Then click Run Test. Running the test will take the JSON object in Input and present it as the input to main() in the Code pane, and show the result in Output.

After running the test the object in the output should match the input. This is the effect of a passthrough.

Also, the console (standard output) now contains the print statement to include the "Raw Record" label:

printf 2024-03-23 14:03:51: Raw Record: {"obj":{"@collector":"local","@event_type":"sshd","@facility":"authpriv","@level":"info","@message":"Disconnected from invalid user gang 23.99.201.14 port 57468 [preauth]","@parser":"fpl-LinuxServerSyslog","@parserVersion":"20231128-3","@sender":"203.0.113.24","@source":"remote","@sshd":{"parserError":"skipped"},"@tags":["sshd"],"@timestamp":1711202399000,"@type":"event"},"props":{},"size":136,"source":""}

In the v2 editor, Console follows the printed lines with a blank line and Run Status: with the status main() returned, here pass. The notices Validate and Run Test give are listed under Validate and Run Test.

Matching the incoming record in main()​

For the example above, the JSON has four root properties: obj, props, size, and source. Consider this code snippet in the code editor:

function main({obj, size, source}) {
    // Replacement for Java (Groovy) parser

The main routine is using an enumerated parameter to match the incoming root properties. Another variation would be:

function main(rawObj) {
  printf("Incoming object is %s", rawObj)
    return { status: "abort" }
}

In this version, the code will send the object to standard out (console panel) and pass the object (in this case rawObj) on to the next processor, which is what returning abort does.

Every incoming JSON record is unique. Looking at the processor's recent events, the Events tab beside Input (v1: the event trail), will show the data, but a programmer normally uses printf statements to see the shape and status of the object in the code.

Returning Status​

There are three common statuses that the main function returns:

  1. pass. Sends the record to the defined sink(s).
  2. abort. The JSON record drops to the next processor.
  3. drop. Stop processing the JSON record and discard it.

The returning status directs where the JSON record will go.

Bare string or status object​

The status can be returned either as a bare string (return "abort") or wrapped in an object (return {"status": "abort"}). Both forms appear throughout the shipped examples: Writing RegEx Pattern Parsers uses both in the same file, and the parser cookbooks favour the object form. Follow whichever form the recipe you start from uses.

Leaving the Editor​

Use Back to Processors, above the editor's title, to return to the Processors page. With unsaved changes, the browser first asks you to confirm leaving the page.

Seeing the Created Processor​

Once you save the processor, it has a card of its own on Configuration → Processors.

Notice that because you have not assigned the processor, its Pipes field reads Not deployed. After a pipe is created with this processor (see Data Routers), that will change.

Advanced​

At this point you are ready to finish creating a Pipe. But the power of Platform is in the ability to write complex processors.

There are examples of processes written in the Cookbooks section of this manual.

Last updated on