Standardize Ingested Data for Response Actions

[raqueltabuyo]

Standardize Ingested Data for Response Actions

Description

To improve bidirectional response actions, we need consistent data ingestion from CrowdStrike, SentinelOne, and Microsoft Defender for Endpoint. This includes host listings, machine action tracking, and essential identifiers.

Requirements

Ensure all three integrations provide:

• A list of hosts (following SentinelOne’s current approach).
• List of action statuses (e.g., "Machine Actions" in Microsoft Defender).
• agent.id - Some systems (e.g,. SentinelOne) have multiple “agent”/“Host” ID, the one needed to be captured is the ID that in turn is used with the external system’s API.
• host.os.type

Impact

• Inconsistent data limits the effectiveness of automated response actions.
• Without standard fields, response workflows may not work uniformly across vendors.

Next Steps

1. Identify discrepancies in host listings, action status tracking, and key identifiers across vendors.
2. Ensure proper ECS mapping and normalization for these fields.
3. Implement required transformations or enrichments to fill data gaps.
4. Prioritize Microsof Defender.

[elasticmachine]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

[raqueltabuyo]

Hey @paul-tavares , could you please provide additional technical information to @kcreddy in this ticket?
Can you provide a deep-dive of the current Sentinel approach for list of hosts as well as more info about what you would like to have for host.os.type, agent.id and list of action statuses?
Thanks in advance.

[kcreddy]

Hey @paul-tavares,

In addition to the above info, I would like to get clarity on the following:

Crowdstrike has host data-stream for fetching host information, similarly SentineOne has agent.
Currently Microsoft Defender for Endpoint integration only has log data-stream, which seem to fetching alerts as per the docs:

Note: When giving the application the API permissions described in the documentation (Windows Defender ATP Alert.Read.All), it will only grant access to read alerts from ATP and nothing else in the Azure Domain

So is the requirement to create another data-stream for Microsoft Defender for Endpoint to fetch host information similar to Crowdstrike and SentineOne and then do the relevant mapping?

[kcreddy]

@paul-tavares, looking for some clarifications:

Ensure all three integrations provide:

• A list of hosts (following SentinelOne’s current approach).
• List of action statuses (e.g., "Machine Actions" in Microsoft Defender).

List of hosts:

• Would like to get clarity on what does "SentinelOne’s current approach" mean, could you please elaborate? This helps in understanding whats the missing piece in Crowdstrike and Microsoft Defender for Endpoint integrations.
  • Crowdstrike already has host data stream which fetches host information. I believe there is no change needed here.
  • Microsoft Defender for Endpoint currently only has 1 data stream which only fetches alerts. This integration needs a new machine data-stream for fetching hosts from List Machines API.

List of action statuses:

• Microsoft Defender for Endpoint has List MachineActions API for fetching all machine actions. Can this be another new data stream, say machine_action?
• Crowdstrike currently doesn't ingest host action statuses. Probably going to need Crowdstrike API documentation to help with this.
• SentinelOne currently also doesn't ingest action statuses. Need official API documentation, as the publicly available one is maintained by community user and I don't see any API that can fetch action statuses for agents.

[paul-tavares]

HI all. Apologies for the delayed response.

Before I answer the questions posted, let me just detail what we need and the pain points we have encountered after having worked with SentinelOne, Microsoft Defender and Crowdstrike integration data (note: I will not be able to comment much on the Crowdstrike data, but @tomsonpl can if needed).

So overall, in order for us to efficiently support the ability for Kibana users to send actions against hosts from these 3rd party EDR system we need the following information on the data that is ingested into Elasticsearch:

Detection Engine (alerts):

1. The ability to identify the type of EDR system - this is already provided today as each integration seems to include a data structure behind a property name that clearly identifies the EDR system - so we're good here
2. The ability to identify the ID of the host for that alert/event. The ID we are looking for should be the same ID that is used by the 3rd party EDR system to pull information about the host/machine from their API or send action requests to that EDR system. This data also is currently available on the alert/events that are ingested
3. The host name
4. The OS type the host is running. This data is not always available.
   1. We currently look at 3 ECS fields to try to figure this out: 'host.os.type', 'host.os.name', 'host.os.platform' and if we find data in these fields, we then try to see if their values matches 'macos', 'windows', 'linux' - which is not always the case.
   2. Standardizing the value populated in these OS type fields would also be much appreciated

It would be great if overall the information I detailed above could be stored in a single, common area in the document that is ingested into elastic, but as long as this information is available somewhere in the document we can still retrieve it.

Supporting Response Actions

In order to better support response actions, it would be ideal (if possible) to also ingest the following type of data if it is available from the 3rd party system:

1. The list of hosts along with their health status, last checking information and isolation state. This might not always be available via one single API from the 3rd party system, but having this information would greatly decrease the amount of API calls we make to the external system.
2. A list of Actions for all the hosts. This is also not always available.
   1. Microsoft Defender does have machine actions as a single API, so having that data would avoid us having to continuously call the 3rd party system's API to determine the status of an action the user took against a host.
   2. I don't know about crowdstrike, but I do know that for SentinelOne, no such thing exists today in their system - they have only an "activity log", which we already pull in and use for this purpose.

To the specific questions asked above

"Microsoft Defender for Endpoint currently only has 1 data stream which only fetches alerts. This integration needs a new machine data-stream for fetching hosts from List Machines API

➡ This is correct. We should create a new datastream and start to pull in the list of Machines from Microsoft and keeping them updated (I'm assuming based on their lastSeen)

"Microsoft Defender for Endpoint has List MachineActions API for fetching all machine actions. Can this be another new data stream, say machine_action?"

➡ Yes please. this would be great. these would need to kept up to date as the records in Microsoft are updated until the action is "complete".

"Crowdstrike currently doesn't ingest host action statuses. Probably going to need Crowdstrike API documentation to help with this.Crowdstrike currently doesn't ingest host action statuses. Probably going to need Crowdstrike API documentation to help with this."

➡ @tomsonpl can probably help here, as he was given access to a customer's environment and may be able to pull in API info. along with some data examples for you.

"SentinelOne currently also doesn't ingest action statuses. Need official API documentation, as the publicly available one is maintained by community user and I don't see any API that can fetch action statuses for agents."

➡ I seem to remember that S1 does not really have a single API that provides a list of action take against a host along with its statuses. I also remember that they don't log the action all to a single location - example: isolation is only logged in the activity log, while running scripts are logged in task list.

BTW: we have access to a SentinelOne env. if you need to login and take a look around. They have API docs there along with an API console.

Let me know if you need access and I'll see if I can set it up for you.