Enhance CrowdStrike Integrations with Comprehensive Testing and Coverage Expansion

[jamiehynds]

Current Situation:
We support multiple CrowdStrike data ingestion methods, including the Falcon SIEM Connector, Event Streaming API, Falcon Data Replicator, and REST API. Our integration covers various CrowdStrike modules such as endpoint, vulnerabilities, threat intelligence, host data, and more.

Challenge:
Until recently, testing our CrowdStrike integrations was limited by lack of access to the Falcon platform. Thanks to our new technology partnership with CrowdStrike, we now have direct access to a CrowdStrike instance.

Goals:

Populate the CrowdStrike Instance:

Fully populate our CrowdStrike instance to closely replicate a typical customer deployment.

Align the data in this instance with the types of data supported by our existing integrations.

Continuous Testing and Validation:

Use our integrations to ingest data from the populated CrowdStrike instance into an Elastic instance.

Perform ongoing testing across dashboards, pipelines, performance, and scalability.

Identify and resolve any bugs, performance bottlenecks, or data normalization issues to improve reliability and user experience.

Expand Integration Coverage:

Review and compare our current CrowdStrike integrations against all available CrowdStrike modules.

Prioritize building support for modules we do not currently cover, such as Identity Protection, to broaden our offering and meet evolving customer needs.

[elasticmachine]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

[navnit-elastic]

Here's are some findings from the ongoing testing of CrowdStrike integration. This include findings for alert, falcon, host and vulnerability data streams and not for FDR.

Alert

Findings:

• Alert data stream making multiple requests with the same alert ids. The current implementation fetches a list of alert IDs and then fetches the full details for each alert. However the ids list never emptied, resulting in the same ids being passed in subsequent intervals. This behavior only happens when no new alerts are found in the first API call.
• Instead of querying the "/alerts/queries/alerts/v2” endpoint to get a list of 1000 IDs, formatting them properly, and then sending them to "/alerts/entities/alerts/v2", you can just send a request to the "/alerts/combined/alerts/v1" endpoint and receive up to 1000 full records at a time (supports FQL and pagination).

Falcon

Supported event types:

• DetectionSummaryEvent
• EppDetectionSummaryEvent
• AuthActivityAuditEvent
• UserActivityAuditEvent
• APIActivityAuditEvent
• HashSpreadingEvent
• RemoteResponseSessionStartEvent
• RemoteResponseSessionEndEvent
• FirewallMatchEvent
• CSPMSearchStreamingEvent
• CSPMIOAStreamingEvent
• IncidentSummaryEvent
• CustomerIOCEvent
• IdentityProtectionEvent
• ReconNotificationSummaryEvent
• ScheduledReportNotificationEvent
• MobileDetectionSummaryEvent
• XdrDetectionSummaryEvent
• IdpDetectionSummaryEvent
• DataProtectionDetectionSummaryEvent

Findings:

• README does not mention log file format and default location while fetching logs using the Falcon SIEM Connector, which can lead to passing incorrect files to the input. (CrowdStrike integration not working #11204)
• Revise the documentation to accurately list the supported event types for the integration. The Falcon data stream contains several event types that we currently support and are listed in the README. However, this list is outdated and needs to be updated as per the current supported event types.
• Configuration file name in the README might be wrong. Currently it is cs.falconhoseclient.cf but it should be cs.falconhoseclient.cfg

Host

Findings:

• Host data stream making multiple requests with the same host ids. The current implementation fetches a list of host IDs and then fetches the full details for each host. However the ids list never emptied, resulting in the same ids being passed in subsequent intervals. This behavior only happens when no new hosts are found in the first API call.
• Instead of querying the "/devices/queries/devices/v1" endpoint to get a list of 5000 IDs, formatting them properly, and then sending them to "/devices/entities/devices/v2", you can just send a request to the "/devices/combined/devices/v1" endpoint and receive up to 10000 full records at a time (supports FQL and pagination).

Vulnerability

Findings:

• The integration fails during pagination with the error "Search context expired, 'after' key no longer valid". This happens when the time between paginated requests to the /spotlight/queries/vulnerabilities/v1 endpoint exceeds 120 seconds. ([CrowdStrike]: Vulnerability Data Stream Fails on Pagination Due to Expired 'after' Token #14670)
• Instead of querying the "/spotlight/queries/vulnerabilities/v1" endpoint to get a list of 400 IDs, formatting them properly, and then sending them to "/spotlight/entities/vulnerabilities/v2", you can just send a request to the "/spotlight/combined/vulnerabilities/v1" endpoint and receive up to 5000 full records at a time (supports FQL and pagination). ([CrowdStrike] Migrate to combined API endpoint for vulnerability data stream #14992)
• it would be nice to be able to also be able to choose one or more (or none) facets that you want the data to return (host_info, remediation, , evaluation_logic) as well as bring in the "remediation" and "evaluation_logic" data separately so there is not so much duplication. These details can be gathered using the "/spotlight/entities/remediations/v2" and "/spotlight/combined/evaluation-logic/v1" respectively (We see that the "/spotlight/combined/evaluation-logic/v1" is no longer recommended). This would better align to CrowdStrike's best practice recommendation for extracting vulnerability data. ([CrowdStrike]: Vulnerability Data Stream Fails on Pagination Due to Expired 'after' Token #14670 (comment))

Other Issues

• Current implementation of ingest pipelines does not handle events having CEL failure. This can be handled by adding a terminated processor in the ingest pipelines.
• Update documentation to apply consistency in terminologies and make clear distinctions between the various data sources and destinations. (https://github.com/elastic/security-integrations/issues/562#issuecomment-3101942767)

[navnit-elastic]

I'm working trough issues mentioned above and as well as public issues.

• crowdstrike: update README to clarify the log file format and location for the Falcon SIEM Connector #14789
• crowdstrike: migrate to combined vulnerabilities endpoint #15049
• crowdstrike: migrate to combined alerts endpoint #15291
• crowdstrike: improve windows events mappings in FDR data stream #15342
• crowdstrike: migrate to combined hosts endpoint #15419
• crowdstrike: add support for DataProtectionDetectionSummaryEvent events #15859
• crowdstrike: Add support for more event types in FDR data stream #15846
• crowdstrike: add parsing for CSPM events in FDR data stream #15783

TODO:

• FDR mapping enhancements
• Migrate to combined endpoint
• README/Doc improvements
• Add terminate processor in the ingest pipeline
• Combine system tests configuration files wherever possible.

Add Event Types (FDR & Streaming API)

• DataProtectionDetectionSummaryEvent
• HashSpreadingEvent
• CustomerIOCEvent
• [CrowdStrike]: Expand coverage of FDR events #15341

[andrewkroh]

Instead of querying the "/alerts/queries/alerts/v2” endpoint to get a list of 1000 IDs, formatting them properly, and then sending them to "/alerts/entities/alerts/v2", you can just send a request to the "/alerts/combined/alerts/v1" endpoint and receive up to 1000 full records at a time (supports FQL and pagination).

I'm happy to see this on the list because while reviewing errors from managed Agents processing the crowdstrike.alerts data stream, I noticed HTTP 413 (Request Entity Too Large) responses. I believe these will be resolved by moving to the 'combined' API endpoint since we won't need to post the occasionally large composite_ids list there.

[cpascale43]

+1 for missing DataProtectionDetectionSummaryEvent: https://github.com/elastic/enhancements/issues/25832

@navnit-elastic, this issue mentions that we are missing about 1200 FDR event types - does that match your research?

According to our devs, we only support a list of 238 event types. However, according to Zoox, any single event that CrowdStrike generates can be sent via the FDR stream. Currently, CrowdStrike supports up to 1395 different event types, but Zoox is pre-filtering these down to 879 in order to cut down the massive amount of volume available to them.

[navnit-elastic]

@cpascale43, that's correct. I've opened an issue to expand coverage of FDR event types here (the list is not exhaustive). This will require extensive analysis to identify the most important event types to support. With input from the team, we can prepare a list of events to be added.