Esql support #233

mashhurs · 2025-04-04T00:14:10Z

Description

ES|QL support:

response_type accepts esql option distinguish from other query types. For the long term this will be deprecated and replaced by query_type if team agrees.
adds ES|QL executor to execute ESQL query and parse/map response to event
validations
- make sure LS (8.17.4+) supports ES|QL (new elasticsearch-ruby client)
- make sure connected ES is greater than 8.11+
- query isn't empty or meaningful that starts with command syntax
informing if query isn't using METADATA which adds _id, _version to the response entries
informing ineffective params such as size, search_api, target if users configure

FYI: failed docs CI isn't related to this change.

Author's check

Common (timeout, internal error, etc..) error cases more tests
Enrichment errors test (in case)
With multiple indices, if field types mismatch, unsupported field type will be filled (reference)
query includes comment
Unit tests to run on >8.17.4
Documentation
Integration tests
Low priority but when using ^...[] special symbols on ES index, query fails but in KB Dev Tools it succeeds

Logs

when credentials wrong

[2025-04-08T14:39:01,060][WARN ][logstash.inputs.elasticsearch.esql][main][6ecdbd14f1bdf461d566eb2807fb23bdf38e032ae8b36a3ff64ee9c4e112ef51] Attempt to ES|QL job but failed. Sleeping for 0.16 {:fail_count=>4, :exception=>"[401] {\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"unable to authenticate user [elastic] for REST request [/_query?format=json]\",\"header\":{\"WWW-Authenticate\":[\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\",\"ApiKey\"]}}],\"type\":\"security_exception\",\"reason\":\"unable to authenticate user [elastic] for REST request [/_query?format=json]\",\"header\":{\"WWW-Authenticate\":[\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\",\"ApiKey\"]}},\"status\":401}"}

when ES is unresponsive at startup

[2025-04-08T14:40:43,942][ERROR][logstash.javapipeline    ][main] Pipeline error {:pipeline_id=>"main", :exception=>#<Elastic::Transport::Transport::Error: Connect to localhost:9200 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused>, :backtrace=>["/logstash/vendor/bundle/jruby/3.1.0/gems/elastic-transport-8.4.0/lib/elastic/transport/transport/base.rb:324:in `perform_request'", "/logstash/vendor/bundle/jruby/3.1.0/gems/elastic-transport-8.4.0/lib/elastic/transport/transport/http/manticore.rb:91:in `perform_request'", "/logstash/vendor/bundle/jruby/3.1.0/gems/elastic-transport-8.4.0/lib/elastic/transport/client.rb:192:in `perform_request'", "/logstash/vendor/bundle/jruby/3.1.0/gems/elasticsearch-8.17.1/lib/elasticsearch.rb:86:in `verify_elasticsearch'", "/logstash/vendor/bundle/jruby/3.1.0/gems/elasticsearch-8.17.1/lib/elasticsearch.rb:69:in `method_missing'", "/logstash/vendor/bundle/jruby/3.1.0/gems/elasticsearch-api-8.17.1/lib/elasticsearch/api/actions/ping.rb:43:in `ping'", "/ls-plugins/logstash-input-elasticsearch/lib/logstash/inputs/elasticsearch.rb:632:in `test_connection!'", "/ls-plugins/logstash-input-elasticsearch/lib/logstash/inputs/elasticsearch.rb:350:in `register'", "/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-mixin-ecs_compatibility_support-1.3.0-java/lib/logstash/plugin_mixins/ecs_compatibility_support/target_check.rb:48:in `register'", "/logstash/logstash-core/lib/logstash/java_pipeline.rb:245:in `block in register_plugins'", "org/jruby/RubyArray.java:1981:in `each'", "/logstash/logstash-core/lib/logstash/java_pipeline.rb:244:in `register_plugins'", "/logstash/logstash-core/lib/logstash/java_pipeline.rb:401:in `start_inputs'", "/logstash/logstash-core/lib/logstash/java_pipeline.rb:325:in `start_workers'", "/logstash/logstash-core/lib/logstash/java_pipeline.rb:198:in `run'", "/logstash/logstash-core/lib/logstash/java_pipeline.rb:150:in `block in start'"], "pipeline.sources"=>["/logstash/config/input-elasticsearch.conf"], :thread=>"#<Thread:0x2a7cfea9 /logstash/logstash-core/lib/logstash/java_pipeline.rb:138 run>"}

when ES is unresponsive with scheduler

[2025-04-08T14:44:00,925][WARN ][logstash.inputs.elasticsearch.esql][main][69ebd8c45e226f6cb40702a83fface05c973db75d4d163d6d546d0a1b38aa425] Attempt to ES|QL job but failed. Sleeping for 0.16 {:fail_count=>4, :exception=>"Connect to localhost:9200 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused"}
[2025-04-08T14:44:01,085][ERROR][logstash.inputs.elasticsearch.esql][main][69ebd8c45e226f6cb40702a83fface05c973db75d4d163d6d546d0a1b38aa425] ES|QL job failed with  {:message=>"Connect to localhost:9200 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused", :cause=>#<Manticore::SocketException: Connect to localhost:9200 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused>}

wrong query

[2025-04-08T14:46:00,908][ERROR][logstash.inputs.elasticsearch.esql][main][2b63d9b563a054f220f92dd2688502c86e75d08da26cc8c71facbd4ed07e8256] ES|QL job failed with  {:message=>"[400] {\"error\":{\"root_cause\":[{\"type\":\"verification_exception\",\"reason\":\"Found 1 problem\\nline 1:1: Unknown index [*datastream-my-index*]\"}],\"type\":\"verification_exception\",\"reason\":\"Found 1 problem\\nline 1:1: Unknown index [*datastream-my-index*]\"},\"status\":400}", :cause=>nil}

no enrichment policy found

[2025-04-08T16:38:54,115][ERROR][logstash.inputs.elasticsearch.esql][main][ef5a0e022f6af6062065dce8128d02a7e5f971da3ff3cd53a9414ff1b90ae97e] ES|QL job failed with  {:message=>"[400] {\"error\":{\"root_cause\":[{\"type\":\"verification_exception\",\"reason\":\"Found 1 problem\\nline 2:28: failed to resolve enrich policy [geo-match-esql-test]; reason [Unknown index [.enrich-geo-match-esql-test]]\"}],\"type\":\"verification_exception\",\"reason\":\"Found 1 problem\\nline 2:28: failed to resolve enrich policy [geo-match-esql-test]; reason [Unknown index [.enrich-geo-match-esql-test]]\"},\"status\":400}", :cause=>nil}

with successful enrichment (data source: https://earthquake.usgs.gov/earthquakes/feed/v1.0/csv.php)

Name	               | Type	| Source indices	| Match field    | Enrich fields
match-test-policy |	match	| significant_month	| depth.            | place
{
      "@version" => "1",
    "@timestamp" => 2025-04-09T00:04:01.033127Z,
         "depth" => 10.0,
         "place" => [
        [0] "181 km ESE of Kimbe, Papua New Guinea",
        [1] "Reykjanes Ridge",
        [2] "Pacific-Antarctic Ridge",
        [3] "Burma (Myanmar)",
        [4] "2025 Mandalay, Burma (Myanmar) Earthquake",
        [5] "120 km SSE of Burica, Panama",
        [6] "34 km NE of Olonkinbyen, Svalbard and Jan Mayen"
    ]
}

when fetching from multiple indices, make sure unsupported types are null filled


[2025-04-08T17:27:54,516][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
Headers: [{"name"=>"@timestamp", "type"=>"date"}, {"name"=>"@version", "type"=>"keyword"}, {"name"=>"agent.build.original", "type"=>"keyword"}, {"name"=>"agent.ephemeral_id", "type"=>"keyword"}, {"name"=>"agent.id", "type"=>"keyword"}, {"name"=>"agent.name", "type"=>"keyword"}, {"name"=>"agent.name.text", "type"=>"text"}, {"name"=>"agent.type", "type"=>"keyword"}, {"name"=>"agent.version", "type"=>"keyword"}, {"name"=>"apache.access.http.request_headers", "type"=>"keyword"}, {"name"=>"apache.access.identity", "type"=>"keyword"}, {"name"=>"apache.access.remote_addresses", "type"=>"keyword"}, {"name"=>"apache.access.response_time", "type"=>"long"}, {"name"=>"apache.access.ssl.cipher", "type"=>"keyword"}, {"name"=>"apache.access.ssl.protocol", "type"=>"keyword"}, {"name"=>"apache.access.tls_handshake.error", "type"=>"keyword"}, {"name"=>"apache.error.module", "type"=>"keyword"}, {"name"=>"client.address", "type"=>"keyword"}, {"name"=>"client.as.number", "type"=>"long"}, {"name"=>"client.as.organization.name", "type"=>"keyword"}, {"name"=>"client.as.organization.name.text", "type"=>"text"}, {"name"=>"client.bytes", "type"=>"long"}, {"name"=>"client.domain", "type"=>"keyword"}, {"name"=>"client.geo.city_name", "type"=>"keyword"}, {"name"=>"client.geo.continent_code", "type"=>"keyword"}, {"name"=>"client.geo.continent_name", "type"=>"keyword"}, {"name"=>"client.geo.country_iso_code", "type"=>"keyword"}, {"name"=>"client.geo.country_name", "type"=>"keyword"}, {"name"=>"client.geo.location", "type"=>"geo_point"}, {"name"=>"client.geo.name", "type"=>"keyword"}, {"name"=>"client.geo.postal_code", "type"=>"keyword"}, {"name"=>"client.geo.region_iso_code", "type"=>"keyword"}, {"name"=>"client.geo.region_name", "type"=>"keyword"}, {"name"=>"client.geo.timezone", "type"=>"keyword"}, {"name"=>"client.ip", "type"=>"ip"}, {"name"=>"client.mac", "type"=>"keyword"}, {"name"=>"client.nat.ip", "type"=>"ip"}, {"name"=>"client.nat.port", "type"=>"long"}, {"name"=>"client.packets", "type"=>"long"}, {"name"=>"client.port", "type"=>"long"}, {"name"=>"client.registered_domain", "type"=>"keyword"}, {"name"=>"client.subdomain", "type"=>"keyword"}, {"name"=>"client.top_level_domain", "type"=>"keyword"}, {"name"=>"client.user.domain", "type"=>"keyword"}, {"name"=>"client.user.email", "type"=>"keyword"}, {"name"=>"client.user.full_name", "type"=>"keyword"}, {"name"=>"client.user.full_name.text", "type"=>"text"}, {"name"=>"client.user.group.domain", "type"=>"keyword"}, {"name"=>"client.user.group.id", "type"=>"keyword"}, {"name"=>"client.user.group.name", "type"=>"keyword"}, {"name"=>"client.user.hash", "type"=>"keyword"}, {"name"=>"client.user.id", "type"=>"keyword"}, {"name"=>"client.user.name", "type"=>"keyword"}, {"name"=>"client.user.name.text", "type"=>"text"}, {"name"=>"client.user.roles", "type"=>"keyword"}, {"name"=>"cloud.account.id", "type"=>"keyword"}, {"name"=>"cloud.account.name", "type"=>"keyword"}, {"name"=>"cloud.availability_zone", "type"=>"keyword"}, {"name"=>"cloud.image.id", "type"=>"keyword"}, {"name"=>"cloud.instance.id", "type"=>"keyword"}, {"name"=>"cloud.instance.name", "type"=>"keyword"}, {"name"=>"cloud.machine.type", "type"=>"keyword"}, {"name"=>"cloud.origin.account.id", "type"=>"keyword"}, {"name"=>"cloud.origin.account.name", "type"=>"keyword"}, {"name"=>"cloud.origin.availability_zone", "type"=>"keyword"}, {"name"=>"cloud.origin.instance.id", "type"=>"keyword"}, {"name"=>"cloud.origin.instance.name", "type"=>"keyword"}, {"name"=>"cloud.origin.machine.type", "type"=>"keyword"}, {"name"=>"cloud.origin.project.id", "type"=>"keyword"}, {"name"=>"cloud.origin.project.name", "type"=>"keyword"}, {"name"=>"cloud.origin.provider", "type"=>"keyword"}, {"name"=>"cloud.origin.region", "type"=>"keyword"}, {"name"=>"cloud.origin.service.name", "type"=>"keyword"}, {"name"=>"cloud.project.id", "type"=>"keyword"}, {"name"=>"cloud.project.name", "type"=>"keyword"}, {"name"=>"cloud.provider", "type"=>"keyword"}, {"name"=>"cloud.region", "type"=>"keyword"}, {"name"=>"cloud.service.name", "type"=>"keyword"}, {"name"=>"cloud.target.account.id", "type"=>"keyword"}, {"name"=>"cloud.target.account.name", "type"=>"keyword"}, {"name"=>"cloud.target.availability_zone", "type"=>"keyword"}, {"name"=>"cloud.target.instance.id", "type"=>"keyword"}, {"name"=>"cloud.target.instance.name", "type"=>"keyword"}, {"name"=>"cloud.target.machine.type", "type"=>"keyword"}, {"name"=>"cloud.target.project.id", "type"=>"keyword"}, {"name"=>"cloud.target.project.name", "type"=>"keyword"}, {"name"=>"cloud.target.provider", "type"=>"keyword"}, {"name"=>"cloud.target.region", "type"=>"keyword"}, {"name"=>"cloud.target.service.name", "type"=>"keyword"}, {"name"=>"container.cpu.usage", "type"=>"double"}, {"name"=>"container.disk.read.bytes", "type"=>"long"}, {"name"=>"container.disk.write.bytes", "type"=>"long"}, {"name"=>"container.id", "type"=>"keyword"}, {"name"=>"container.image.hash.all", "type"=>"keyword"}, {"name"=>"container.image.name", "type"=>"keyword"}, {"name"=>"container.image.tag", "type"=>"keyword"}, {"name"=>"container.memory.usage", "type"=>"double"}, {"name"=>"container.name", "type"=>"keyword"}, {"name"=>"container.network.egress.bytes", "type"=>"long"}, {"name"=>"container.network.ingress.bytes", "type"=>"long"}, {"name"=>"container.runtime", "type"=>"keyword"}, {"name"=>"container.security_context.privileged", "type"=>"boolean"}, {"name"=>"data_stream.dataset", "type"=>"keyword"}, {"name"=>"data_stream.namespace", "type"=>"keyword"}, {"name"=>"data_stream.type", "type"=>"keyword"}, {"name"=>"destination.address", "type"=>"keyword"}, {"name"=>"destination.as.number", "type"=>"long"}, {"name"=>"destination.as.organization.name", "type"=>"keyword"}, {"name"=>"destination.as.organization.name.text", "type"=>"text"}, {"name"=>"destination.bytes", "type"=>"long"}, {"name"=>"destination.domain", "type"=>"keyword"}, {"name"=>"destination.geo.city_name", "type"=>"keyword"}, {"name"=>"destination.geo.continent_code", "type"=>"keyword"}, {"name"=>"destination.geo.continent_name", "type"=>"keyword"}, {"name"=>"destination.geo.country_iso_code", "type"=>"keyword"}, {"name"=>"destination.geo.country_name", "type"=>"keyword"}, {"name"=>"destination.geo.location", "type"=>"geo_point"}, {"name"=>"destination.geo.name", "type"=>"keyword"}, {"name"=>"destination.geo.postal_code", "type"=>"keyword"}, {"name"=>"destination.geo.region_iso_code", "type"=>"keyword"}, {"name"=>"destination.geo.region_name", "type"=>"keyword"}, {"name"=>"destination.geo.timezone", "type"=>"keyword"}, {"name"=>"destination.ip", "type"=>"ip"}, {"name"=>"destination.mac", "type"=>"keyword"}, {"name"=>"destination.nat.ip", "type"=>"ip"}, {"name"=>"destination.nat.port", "type"=>"long"}, {"name"=>"destination.packets", "type"=>"long"}, {"name"=>"destination.port", "type"=>"long"}, {"name"=>"destination.registered_domain", "type"=>"keyword"}, {"name"=>"destination.subdomain", "type"=>"keyword"}, {"name"=>"destination.top_level_domain", "type"=>"keyword"}, {"name"=>"destination.user.domain", "type"=>"keyword"}, {"name"=>"destination.user.email", "type"=>"keyword"}, {"name"=>"destination.user.full_name", "type"=>"keyword"}, {"name"=>"destination.user.full_name.text", "type"=>"text"}, {"name"=>"destination.user.group.domain", "type"=>"keyword"}, {"name"=>"destination.user.group.id", "type"=>"keyword"}, {"name"=>"destination.user.group.name", "type"=>"keyword"}, {"name"=>"destination.user.hash", "type"=>"keyword"}, {"name"=>"destination.user.id", "type"=>"keyword"}, {"name"=>"destination.user.name", "type"=>"keyword"}, {"name"=>"destination.user.name.text", "type"=>"text"}, {"name"=>"destination.user.roles", "type"=>"keyword"}, {"name"=>"device.id", "type"=>"keyword"}, {"name"=>"device.manufacturer", "type"=>"keyword"}, {"name"=>"device.model.identifier", "type"=>"keyword"}, {"name"=>"device.model.name", "type"=>"keyword"}, {"name"=>"dll.code_signature.digest_algorithm", "type"=>"keyword"}, {"name"=>"dll.code_signature.exists", "type"=>"boolean"}, {"name"=>"dll.code_signature.signing_id", "type"=>"keyword"}, {"name"=>"dll.code_signature.status", "type"=>"keyword"}, {"name"=>"dll.code_signature.subject_name", "type"=>"keyword"}, {"name"=>"dll.code_signature.team_id", "type"=>"keyword"}, {"name"=>"dll.code_signature.timestamp", "type"=>"date"}, {"name"=>"dll.code_signature.trusted", "type"=>"boolean"}, {"name"=>"dll.code_signature.valid", "type"=>"boolean"}, {"name"=>"dll.hash.md5", "type"=>"keyword"}, {"name"=>"dll.hash.sha1", "type"=>"keyword"}, {"name"=>"dll.hash.sha256", "type"=>"keyword"}, {"name"=>"dll.hash.sha384", "type"=>"keyword"}, {"name"=>"dll.hash.sha512", "type"=>"keyword"}, {"name"=>"dll.hash.ssdeep", "type"=>"keyword"}, {"name"=>"dll.hash.tlsh", "type"=>"keyword"}, {"name"=>"dll.name", "type"=>"keyword"}, {"name"=>"dll.path", "type"=>"keyword"}, {"name"=>"dll.pe.architecture", "type"=>"keyword"}, {"name"=>"dll.pe.company", "type"=>"keyword"}, {"name"=>"dll.pe.description", "type"=>"keyword"}, {"name"=>"dll.pe.file_version", "type"=>"keyword"}, {"name"=>"dll.pe.go_import_hash", "type"=>"keyword"}, {"name"=>"dll.pe.go_imports", "type"=>"unsupported"}, {"name"=>"dll.pe.go_imports_names_entropy", "type"=>"long"}, {"name"=>"dll.pe.go_imports_names_var_entropy", "type"=>"long"}, {"name"=>"dll.pe.go_stripped", "type"=>"boolean"}, {"name"=>"dll.pe.imphash", "type"=>"keyword"}, {"name"=>"dll.pe.import_hash", "type"=>"keyword"}, {"name"=>"dll.pe.imports", "type"=>"unsupported"}, {"name"=>"dll.pe.imports_names_entropy", "type"=>"long"}, {"name"=>"dll.pe.imports_names_var_entropy", "type"=>"long"}, {"name"=>"dll.pe.original_file_name", "type"=>"keyword"}, {"name"=>"dll.pe.pehash", "type"=>"keyword"}, {"name"=>"dll.pe.product", "type"=>"keyword"}, {"name"=>"dns.answers.class", "type"=>"keyword"}, {"name"=>"dns.answers.data", "type"=>"keyword"}, {"name"=>"dns.answers.name", "type"=>"keyword"}, {"name"=>"dns.answers.ttl", "type"=>"long"}, {"name"=>"dns.answers.type", "type"=>"keyword"}, {"name"=>"dns.header_flags", "type"=>"keyword"}, {"name"=>"dns.id", "type"=>"keyword"}, {"name"=>"dns.op_code", "type"=>"keyword"}, {"name"=>"dns.question.class", "type"=>"keyword"}, {"name"=>"dns.question.name", "type"=>"keyword"}, {"name"=>"dns.question.registered_domain", "type"=>"keyword"}, {"name"=>"dns.question.subdomain", "type"=>"keyword"}, {"name"=>"dns.question.top_level_domain", "type"=>"keyword"}, {"name"=>"dns.question.type", "type"=>"keyword"}, {"name"=>"dns.resolved_ip", "type"=>"ip"}, {"name"=>"dns.response_code", "type"=>"keyword"}, {"name"=>"dns.type", "type"=>"keyword"}, {"name"=>"ecs.version", "type"=>"keyword"}, {"name"=>"elastic_agent.id", "type"=>"keyword"}, {"name"=>"elastic_agent.snapshot", "type"=>"boolean"}, {"name"=>"elastic_agent.version", "type"=>"keyword"}, {"name"=>"email.bcc.address", "type"=>"keyword"}, {"name"=>"email.cc.address", "type"=>"keyword"}, {"name"=>"email.content_type", "type"=>"keyword"}, {"name"=>"email.delivery_timestamp", "type"=>"date"}, {"name"=>"email.direction", "type"=>"keyword"}, {"name"=>"email.from.address", "type"=>"keyword"}, {"name"=>"email.local_id", "type"=>"keyword"}, {"name"=>"email.message_id", "type"=>"keyword"}, {"name"=>"email.origination_timestamp", "type"=>"date"}, {"name"=>"email.reply_to.address", "type"=>"keyword"}, {"name"=>"email.sender.address", "type"=>"keyword"}, {"name"=>"email.subject", "type"=>"keyword"}, {"name"=>"email.subject.text", "type"=>"text"}, {"name"=>"email.to.address", "type"=>"keyword"}, {"name"=>"email.x_mailer", "type"=>"keyword"}, {"name"=>"enterprisesearch.change.access_all_engines", "type"=>"boolean"}, {"name"=>"enterprisesearch.change.api_key", "type"=>"keyword"}, {"name"=>"enterprisesearch.change.authentication_token.ciphertext", "type"=>"keyword"}, {"name"=>"enterprisesearch.change.authentication_token.hash", "type"=>"keyword"}, {"name"=>"enterprisesearch.change.created_at", "type"=>"keyword"}, {"name"=>"enterprisesearch.change.frito_pie_group_ids", "type"=>"keyword"}, {"name"=>"enterprisesearch.change.id", "type"=>"keyword"}, {"name"=>"enterprisesearch.change.loco_moco_account_id", "type"=>"keyword"}, {"name"=>"enterprisesearch.change.name", "type"=>"keyword"}, {"name"=>"enterprisesearch.change.read_access", "type"=>"boolean"}, {"name"=>"enterprisesearch.change.role_type", "type"=>"keyword"}, {"name"=>"enterprisesearch.change.telemetry_last_sent_at", "type"=>"keyword"}, {"name"=>"enterprisesearch.change.token_type", "type"=>"keyword"}, {"name"=>"enterprisesearch.change.updated_at", "type"=>"keyword"}, {"name"=>"enterprisesearch.change.user_oid", "type"=>"keyword"}, {"name"=>"enterprisesearch.change.write_access", "type"=>"boolean"}, {"name"=>"enterprisesearch.entity", "type"=>"keyword"}, {"name"=>"error.code", "type"=>"keyword"}, {"name"=>"error.id", "type"=>"keyword"}, {"name"=>"error.message", "type"=>"text"}, {"name"=>"error.stack_trace", "type"=>"keyword"}, {"name"=>"error.stack_trace.text", "type"=>"text"}, {"name"=>"error.type", "type"=>"keyword"}, {"name"=>"event.action", "type"=>"keyword"}, {"name"=>"event.agent_id_status", "type"=>"keyword"}, {"name"=>"event.category", "type"=>"keyword"}, {"name"=>"event.code", "type"=>"keyword"}, {"name"=>"event.created", "type"=>"date"}, {"name"=>"event.dataset", "type"=>"keyword"}, {"name"=>"event.duration", "type"=>"long"}, {"name"=>"event.end", "type"=>"date"}, {"name"=>"event.hash", "type"=>"keyword"}, {"name"=>"event.id", "type"=>"keyword"}, {"name"=>"event.ingested", "type"=>"date"}, {"name"=>"event.kind", "type"=>"keyword"}, {"name"=>"event.module", "type"=>"keyword"}, {"name"=>"event.original", "type"=>"keyword"}, {"name"=>"event.outcome", "type"=>"keyword"}, {"name"=>"event.provider", "type"=>"keyword"}, {"name"=>"event.reason", "type"=>"keyword"}, {"name"=>"event.reference", "type"=>"keyword"}, {"name"=>"event.risk_score", "type"=>"double"}, {"name"=>"event.risk_score_norm", "type"=>"double"}, {"name"=>"event.sequence", "type"=>"long"}, {"name"=>"event.severity", "type"=>"long"}, {"name"=>"event.start", "type"=>"date"}, {"name"=>"event.timezone", "type"=>"keyword"}, {"name"=>"event.type", "type"=>"keyword"}, {"name"=>"event.url", "type"=>"keyword"}, {"name"=>"faas.coldstart", "type"=>"boolean"}, {"name"=>"faas.execution", "type"=>"keyword"}, {"name"=>"faas.id", "type"=>"keyword"}, {"name"=>"faas.name", "type"=>"keyword"}, {"name"=>"faas.version", "type"=>"keyword"}, {"name"=>"file.accessed", "type"=>"date"}, {"name"=>"file.attributes", "type"=>"keyword"}, {"name"=>"file.code_signature.digest_algorithm", "type"=>"keyword"}, {"name"=>"file.code_signature.exists", "type"=>"boolean"}, {"name"=>"file.code_signature.signing_id", "type"=>"keyword"}, {"name"=>"file.code_signature.status", "type"=>"keyword"}, {"name"=>"file.code_signature.subject_name", "type"=>"keyword"}, {"name"=>"file.code_signature.team_id", "type"=>"keyword"}, {"name"=>"file.code_signature.timestamp", "type"=>"date"}, {"name"=>"file.code_signature.trusted", "type"=>"boolean"}, {"name"=>"file.code_signature.valid", "type"=>"boolean"}, {"name"=>"file.created", "type"=>"date"}, {"name"=>"file.ctime", "type"=>"date"}, {"name"=>"file.device", "type"=>"keyword"}, {"name"=>"file.directory", "type"=>"keyword"}, {"name"=>"file.drive_letter", "type"=>"keyword"}, {"name"=>"file.elf.architecture", "type"=>"keyword"}, {"name"=>"file.elf.byte_order", "type"=>"keyword"}, {"name"=>"file.elf.cpu_type", "type"=>"keyword"}, {"name"=>"file.elf.creation_date", "type"=>"date"}, {"name"=>"file.elf.exports", "type"=>"unsupported"}, {"name"=>"file.elf.go_import_hash", "type"=>"keyword"}, {"name"=>"file.elf.go_imports", "type"=>"unsupported"}, {"name"=>"file.elf.go_imports_names_entropy", "type"=>"long"}, {"name"=>"file.elf.go_imports_names_var_entropy", "type"=>"long"}, {"name"=>"file.elf.go_stripped", "type"=>"boolean"}, {"name"=>"file.elf.header.abi_version", "type"=>"keyword"}, {"name"=>"file.elf.header.class", "type"=>"keyword"}, {"name"=>"file.elf.header.data", "type"=>"keyword"}, {"name"=>"file.elf.header.entrypoint", "type"=>"long"}, {"name"=>"file.elf.header.object_version", "type"=>"keyword"}, {"name"=>"file.elf.header.os_abi", "type"=>"keyword"}, {"name"=>"file.elf.header.type", "type"=>"keyword"}, {"name"=>"file.elf.header.version", "type"=>"keyword"}, {"name"=>"file.elf.import_hash", "type"=>"keyword"}, {"name"=>"file.elf.imports", "type"=>"unsupported"}, {"name"=>"file.elf.imports_names_entropy", "type"=>"long"}, {"name"=>"file.elf.imports_names_var_entropy", "type"=>"long"}, {"name"=>"file.elf.shared_libraries", "type"=>"keyword"}, {"name"=>"file.elf.telfhash", "type"=>"keyword"}, {"name"=>"file.extension", "type"=>"keyword"}, {"name"=>"file.fork_name", "type"=>"keyword"}, {"name"=>"file.gid", "type"=>"keyword"}, {"name"=>"file.group", "type"=>"keyword"}, {"name"=>"file.hash.md5", "type"=>"keyword"}, {"name"=>"file.hash.sha1", "type"=>"keyword"}, {"name"=>"file.hash.sha256", "type"=>"keyword"}, {"name"=>"file.hash.sha384", "type"=>"keyword"}, {"name"=>"file.hash.sha512", "type"=>"keyword"}, {"name"=>"file.hash.ssdeep", "type"=>"keyword"}, {"name"=>"file.hash.tlsh", "type"=>"keyword"}, {"name"=>"file.inode", "type"=>"keyword"}, {"name"=>"file.macho.go_import_hash", "type"=>"keyword"}, {"name"=>"file.macho.go_imports", "type"=>"unsupported"}, {"name"=>"file.macho.go_imports_names_entropy", "type"=>"long"}, {"name"=>"file.macho.go_imports_names_var_entropy", "type"=>"long"}, {"name"=>"file.macho.go_stripped", "type"=>"boolean"}, {"name"=>"file.macho.import_hash", "type"=>"keyword"}, {"name"=>"file.macho.imports", "type"=>"unsupported"}, {"name"=>"file.macho.imports_names_entropy", "type"=>"long"}, {"name"=>"file.macho.imports_names_var_entropy", "type"=>"long"}, {"name"=>"file.macho.symhash", "type"=>"keyword"}, {"name"=>"file.mime_type", "type"=>"keyword"}, {"name"=>"file.mode", "type"=>"keyword"}, {"name"=>"file.mtime", "type"=>"date"}, {"name"=>"file.name", "type"=>"keyword"}, {"name"=>"file.owner", "type"=>"keyword"}, {"name"=>"file.path", "type"=>"keyword"}, {"name"=>"file.path.text", "type"=>"text"}, {"name"=>"file.pe.architecture", "type"=>"keyword"}, {"name"=>"file.pe.company", "type"=>"keyword"}, {"name"=>"file.pe.description", "type"=>"keyword"}, {"name"=>"file.pe.file_version", "type"=>"keyword"}, {"name"=>"file.pe.go_import_hash", "type"=>"keyword"}, {"name"=>"file.pe.go_imports", "type"=>"unsupported"}, {"name"=>"file.pe.go_imports_names_entropy", "type"=>"long"}, {"name"=>"file.pe.go_imports_names_var_entropy", "type"=>"long"}, {"name"=>"file.pe.go_stripped", "type"=>"boolean"}, {"name"=>"file.pe.imphash", "type"=>"keyword"}, {"name"=>"file.pe.import_hash", "type"=>"keyword"}, {"name"=>"file.pe.imports", "type"=>"unsupported"}, {"name"=>"file.pe.imports_names_entropy", "type"=>"long"}, {"name"=>"file.pe.imports_names_var_entropy", "type"=>"long"}, {"name"=>"file.pe.original_file_name", "type"=>"keyword"}, {"name"=>"file.pe.pehash", "type"=>"keyword"}, {"name"=>"file.pe.product", "type"=>"keyword"}, {"name"=>"file.size", "type"=>"long"} ... ]
{
                      "threat.indicator.file.pe.architecture" => nil,
                                          "dns.question.name" => nil,
                                               "event.module" => "apache",
                           "server.as.organization.name.text" => nil,
          "enterprisesearch.change.authentication_token.hash" => nil,
                                             "faas.coldstart" => nil,
                                               "network.name" => nil,
                 "process.group_leader.tty.char_device.major" => nil,
             "threat.indicator.file.code_signature.timestamp" => nil,
                      "process.parent.code_signature.team_id" => nil,
                                    "process.parent.hash.md5" => nil,
                "process.entry_leader.supplemental_groups.id" => nil,
                                               "service.name" => nil,
                              "tls.server.x509.serial_number" => nil,
                                             "client.address" => nil,
                                                   "dll.path" => nil,
                                 "process.parent.hash.sha384" => nil,
                                     "tls.client.hash.sha256" => nil,
                                    "threat.tactic.reference" => nil,
                                                "client.port" => nil,
                                  "source.geo.continent_code" => nil,
                                               "process.args" => nil,
                                                 "event.code" => nil,
                                       "host.geo.region_name" => nil,
                                     "host.user.group.domain" => nil,
                              "process.code_signature.exists" => nil,
                   "threat.indicator.file.elf.header.version" => nil,
                                "destination.geo.postal_code" => nil,
                            "http.response.body.content.text" => nil,
                          "threat.indicator.file.target_path" => nil,
                        "tls.server.x509.public_key_exponent" => nil,
                                                 "file.owner" => nil,
                           "process.previous.executable.text" => nil,
                    "threat.technique.subtechnique.name.text" => nil,
             "threat.indicator.file.x509.issuer.organization" => nil,
                                           "host.user.domain" => nil,
                            "process.entry_leader.group.name" => nil,
                                           "log.syslog.msgid" => nil,
                                             "user.name.text" => nil,
                                               "dll.hash.md5" => nil,
                                       "service.ephemeral_id" => nil,
                                      "client.user.name.text" => nil,
                      "process.session_leader.real_user.name" => nil,
                            "threat.indicator.file.path.text" => nil,
                                            "process.user.id" => nil,
                                        "user.changes.domain" => nil,
                                 "service.origin.environment" => nil,
                "process.entry_leader.working_directory.text" => nil,
                          "vulnerability.score.environmental" => nil,
                                       "host.geo.postal_code" => nil,
                                          "dns.response_code" => nil,
                                          "network.transport" => nil,
                                       "file.elf.import_hash" => nil,
                                        "file.x509.not_after" => nil,
      "threat.indicator.file.x509.subject.distinguished_name" => nil,
                               "process.elf.shared_libraries" => nil,
                          "threat.indicator.file.hash.sha384" => nil,
                               "process.parent.real_group.id" => nil,
                                       "tls.client.hash.sha1" => nil,
                                                  "faas.name" => nil,
                                               "url.original" => nil,
                                             "host.user.name" => nil,
                               "threat.indicator.modified_at" => nil,
                  "threat.indicator.x509.signature_algorithm" => nil,
                               "dll.code_signature.timestamp" => nil,
                    "process.parent.code_signature.timestamp" => nil,
                                               "host.boot.id" => nil,
                                 "log.syslog.structured_data" => nil,
                                   "service.origin.node.name" => nil,
                           "process.parent.group_leader.vpid" => nil,
                                   "orchestrator.resource.id" => nil,
                                      "event.risk_score_norm" => nil,
                                        "file.elf.go_imports" => nil,
                       "process.parent.pe.original_file_name" => nil,
                                       "package.architecture" => nil,
                             "kibana.alert.suppression.start" => nil,
                                          "server.user.email" => nil,
                                    "cloud.target.project.id" => nil,
                                   "log.syslog.facility.code" => nil,
                 "threat.indicator.x509.public_key_algorithm" => nil,
                       "process.group_leader.real_group.name" => nil,
                       "process.entry_leader.executable.text" => nil,
              "process.parent.macho.go_imports_names_entropy" => nil,
                          "process.session_leader.parent.pid" => nil,
                                       "user.effective.email" => nil,
                                             "host.cpu.usage" => nil,
                                         "observer.os.kernel" => nil,
                                         "user.changes.email" => nil,
                                "vulnerability.score.version" => nil,
                                "threat.indicator.file.ctime" => nil,
                                              "error.message" => nil,
                                           "file.target_path" => nil,
                                          "server.user.roles" => nil,
                                                 "file.mtime" => nil,
                                      "file.target_path.text" => nil,
                                  "process.parent.group.name" => nil,
                                      "user_agent.os.version" => nil,
                            "host.risk.calculated_score_norm" => nil,
                                   "process.macho.go_imports" => nil,
                                           "client.user.hash" => nil,
                                                   "trace.id" => nil,
                                      "tls.server.not_before" => nil,
                                        "package.description" => nil,
                              "process.parent.user.name.text" => nil,
                     "process.macho.go_imports_names_entropy" => nil,
                                          "process.user.name" => nil
                 ...
}

…esql option, validations to make sure both LS and ES support the ESQL execution.

… adds by default - might be users are looking for by default.

…ental method formatting.

mashhurs · 2025-04-11T00:15:23Z

docs/index.asciidoc

+|Component |Minimum version
+|{es} |8.11.0 or newer
+|{ls} |8.17.4 or newer
+|This plugin |4.23.0+ (4.x series) or 5.2.0+ (5.x series)


review note: before releasing this change, I will backport and release 4.23.0 first

mashhurs · 2025-04-11T00:16:43Z

lib/logstash/inputs/elasticsearch/esql.rb

+          # retriable already printed error details
+          return if response == false
+
+          if response&.headers&.dig("warning")


review note: interestingly warnings in response headers.

…L feature.

…/info and add docinfo* fields in ineffective fields list.

This was linked to issues Apr 4, 2025

Support ES|QL #221

Open

[ES|QL] Validation of data for the Logstash Elasticsearch Input #228

Open

[ES|QL] Query executor implementation #229

Open

[ES|QL] Unit Tests for ES|QL support in the Logstash Elasticsearch Input #230

Open

mashhurs force-pushed the esql-support branch from de6073b to 52a97c2 Compare April 8, 2025 07:52

mashhurs added 2 commits April 8, 2025 07:36

ES|QL support: ESQL executor implementation, response type to accept …

a307db2

…esql option, validations to make sure both LS and ES support the ESQL execution.

Merge with upstream, warn if query doesn't include METADATA which DSL…

9c35f22

… adds by default - might be users are looking for by default.

mashhurs force-pushed the esql-support branch from 52a97c2 to 9c35f22 Compare April 9, 2025 00:15

mashhurs added 4 commits April 9, 2025 23:57

Run unit tests with the LS version which actually supports the ES|QL.

6f99055

Add query type to the agent. DRY of supported ES/LS versions.

086a592

Remove query type from user-agent since it is useless, put back accid…

e30e0f9

…ental method formatting.

Initial docs added for ES|QL.

7746c14

mashhurs marked this pull request as ready for review April 10, 2025 23:30

Update query to include condition with string.

76303d8

mashhurs commented Apr 11, 2025

View reviewed changes

mashhurs added 4 commits April 11, 2025 22:21

Tested escaped chars cases, uses orignal query.

1fb29f7

Integration tests added.

5d47f2f

Skip the ESQL test if LS with the ES client which doesn't support ESQ…

c291e24

…L feature.

Add comments on response type and query params about ES|QL acceptance…

22e72e9

…/info and add docinfo* fields in ineffective fields list.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Esql support #233

Esql support #233

mashhurs commented Apr 4, 2025 •

edited

Loading

mashhurs Apr 11, 2025

mashhurs Apr 11, 2025 •

edited

Loading

Esql support #233

Are you sure you want to change the base?

Esql support #233

Conversation

mashhurs commented Apr 4, 2025 • edited Loading

Description

Author's check

Logs

mashhurs Apr 11, 2025

Choose a reason for hiding this comment

mashhurs Apr 11, 2025 • edited Loading

Choose a reason for hiding this comment

mashhurs commented Apr 4, 2025 •

edited

Loading

mashhurs Apr 11, 2025 •

edited

Loading