Skip to main content

Data Storage

Data lakes and indexes​

Storage is administered on the data lakes page. It shows one panel per data lake, with a card for every index in that lake.

The page itself, control by control, is documented in the interface reference at Configuration → Data Lakes (v1: Data Lake → Management). This page covers the storage model behind it: retention, residency, and where the data physically lives.

Open Configuration → Data Lakes (URL path /v2/configuration/data-lakes).

Data Lakes page showing the managed lake panel with its 24-hour totals and a grid of index cards, each with stored count, estimated rate, a sparkline, and queue, merged and imported details, followed by a Create New Index tile

The header carries a Back to Configuration link, the page title Data Lakes, the subtitle Management of data lakes and the indexes associated with them, and an Add Data Lake button on the right. Each lake panel opens with a header row giving the lake name and its storage type (Ingext managed for platform-managed storage), the events its indexes stored in the last 24 hours (Stored (24H)), its estimated rate in events per hour (Est. Rate), and a red Delete. A grid of index cards follows, ending with a Create New Index tile.

Each index card summarizes one index: the index name and the schema it uses (Schema: followed by the schema name), its own Stored (24H) and Est. Rate, an activity sparkline of recent ingest volume, and a footer with Queue (the items waiting to be imported), Merged (when the index was last merged) and Imported (when data was last imported into it); a newly created index shows Status: Not Available in place of that footer. Read the Merged and Imported times together: an index can have a recent Imported time while its data has not yet been merged. Every card has a red Delete, and an index that has virtual tables available also shows a Virtual Tables link that opens the Virtual Tables dialog for that index.

Self-managed or Ingext Managed​

Click Add Data Lake to open the Creating New Data Lake dialog, subtitled Creating a new self/ingext managed data lake:

Creating New Data Lake dialog with the Ingext Managed switch on and Lake Name filled in with managed and read-only

The dialog has two sections:

  • Details: Lake Name and an optional Description. While Ingext Managed is on, Lake Name is filled in with managed and cannot be edited.
  • Storage: an Ingext Managed switch, on by default, that selects platform-managed storage. Turn it off for a self-managed lake: Lake Name becomes an empty, required field, and a required Storage Integration drop-down appears for choosing the storage integration the lake is kept in.

Click Add to create the lake, or Cancel or × to close without saving.

Deleting a lake or an index​

The red Delete in a lake panel's header row asks whether you are sure you want to delete the named data lake, and warns that all indexes in the data lake will also be lost. An index card's red Delete asks whether you are sure you want to delete the named index from the named lake.

Retention​

Data is retained for 365 days and is hot-searchable for the whole of that window. There is no rehydration step and no archive tier to restore from: a query covering the last twelve months reads the same indexes as a query covering the last hour, and the only cost of the longer range is the time it takes to scan it.

That is what makes the task-based search model worth understanding: a year-wide query is a normal thing to run here, not an exceptional one.

Where the data physically lives​

The underlying store is AWS S3. Each enterprise account gets its own segmented bucket, so one account's data is never co-located with another's.

Buckets are placed in the AWS Region nearest to where the data originates, which is how the platform addresses residency requirements such as GDPR. A multi-national account gets one bucket per location rather than one bucket overall: an organization with sites in the USA, Germany and South Africa is given three, and data never leaves the region it was collected in.

What crosses regions is metadata, not events. Behaviors and summaries are fed back to the Overview Summary page (v1: SIEM → Overview Summary) so an analyst sees the whole estate in one queue; drilling into the underlying records reads them in place, within the region that holds them.

Last updated on