The correlation engine runs separately from the integrations and primary data services and can be run on separate hardware to minimize performance impact on using the system.
The engine could be classified as an Extract, Transform, and Load (ETL) processor, which is a standard method of data ingestion.
- Parses the data source into a standard format with a flexible data extractor
- Transforms the data with a standard set of mapping rules
- Correlates the normalized data set with any existing entities in Resolve
- Creates or updates data entities as needed
Resolve's correlation engine uses a flexible data extractor with configurable parsing to extract and convert data into a normalized format. The parsing can be modified via a JSON configuration, minimizing the need for code changes in case of source format modifications.
Be aware that imported source values may not align 1:1 with the fields in Resolve. In some cases, values may be combined or pulled from various places in the data source object. Many data sources spread out vulnerability data in their document structures, which forces Resolve to retrieve these values and combine them to achieve an entity structure matching the Resolve data model.
Defining the mapping
The correlation engine has a default mapping for each source data format. The mapping defines the source values and destination fields for vulnerability data. Any key values that are not in the default mapping are made available for inclusion in the mapping process. Resolve saves any key values that are not in the mapping into a generic data bucket. The values will be discarded prior to import, but if the mapping is modified to include these values, you can include them in the import by adding a destination for them.
Resolve uses Drools, an open source rule engine, for the correlation and loading of entity data. Rules are associated with the data source type. Default rules leverage the common methods for correlation from the defined data source.
Correlation rules are somewhat configurable, especially for assets. This is important because asset uniqueness is a key determination of correlation. See Correlating assets.
The correlation engine has been built with the assumption that data source formats can change at any time. This could include new data objects, changes in key values, or modifications of data types. This means that even if a vendor makes changes to how they are reporting data, the correlation engine can still operate. For the majority of source format modifications, the ability to import and correlate data will remain intact.
However, if a vendor changes the data source format, the correlation engine will most likely need to be updated. For instance, if a key value in a data format is entirely renamed and that value was used for correlation, that could affect how Resolve processes the data.
NetSPI packages updates to handle changes that occur within various data sources. If there is a need to update these on a time-sensitive basis, contact support to get the latest configuration for the supported data sources prior to receiving a broader system update.