Merge branch 'main' into k1

Signed-off-by: Sandesh Kumar <[email protected]>

CHANGELOG-3.0.md

@@ -14,12 +14,17 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 - [WLM] Add WLM support for search scroll API ([#16981](https://github.com/opensearch-project/OpenSearch/pull/16981))
 - Allow to pass the list settings through environment variables (like [], ["a", "b", "c"], ...) ([#10625](https://github.com/opensearch-project/OpenSearch/pull/10625))
 - Views, simplify data access and manipulation by providing a virtual layer over one or more indices ([#11957](https://github.com/opensearch-project/OpenSearch/pull/11957))
+- Add systemd configurations to strengthen OS core security ([#17107](https://github.com/opensearch-project/OpenSearch/pull/17107))
 - Added pull-based Ingestion (APIs, for ingestion source, a Kafka plugin, and IngestionEngine that pulls data from the ingestion source) ([#16958](https://github.com/opensearch-project/OpenSearch/pull/16958))
 - Added ConfigurationUtils to core for the ease of configuration parsing [#17223](https://github.com/opensearch-project/OpenSearch/pull/17223)
+- Add execution_hint to cardinality aggregator request (#[17312](https://github.com/opensearch-project/OpenSearch/pull/17312))
+- Arrow Flight RPC plugin with Flight server bootstrap logic and client for internode communication ([#16962](https://github.com/opensearch-project/OpenSearch/pull/16962))
+- Added offset management for the pull-based Ingestion ([#17354](https://github.com/opensearch-project/OpenSearch/pull/17354))
 
 ### Dependencies
 - Update Apache Lucene to 10.1.0 ([#16366](https://github.com/opensearch-project/OpenSearch/pull/16366))
 - Bump Apache HttpCore5/HttpClient5 dependencies from 5.2.5/5.3.1 to 5.3.1/5.4.1 to support ExtendedSocketOption in HttpAsyncClient ([#16757](https://github.com/opensearch-project/OpenSearch/pull/16757))
+- Bumps `jetty` version from 9.4.55.v20240627 to 9.4.57.v20241219
 
 ### Changed
 - Changed locale provider from COMPAT to CLDR  ([#14345](https://github.com/opensearch-project/OpenSearch/pull/14345))
@@ -37,6 +42,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 - Stop minimizing automata used for case-insensitive matches ([#17268](https://github.com/opensearch-project/OpenSearch/pull/17268))
 - Refactor the `:server` module `org.opensearch.client` to `org.opensearch.transport.client` to eliminate top level split packages for JPMS support ([#17272](https://github.com/opensearch-project/OpenSearch/pull/17272))
 - Use Lucene `BM25Similarity` as default since the `LegacyBM25Similarity` is marked as deprecated ([#17306](https://github.com/opensearch-project/OpenSearch/pull/17306))
+- Wildcard field index only 3gram of the input data [#17349](https://github.com/opensearch-project/OpenSearch/pull/17349)
 
 ### Deprecated
 
@@ -66,6 +72,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 - Don't over-allocate in HeapBufferedAsyncEntityConsumer in order to consume the response ([#9993](https://github.com/opensearch-project/OpenSearch/pull/9993))
 - Fix swapped field formats in nodes API where `total_indexing_buffer_in_bytes` and `total_indexing_buffer` values were reversed ([#17070](https://github.com/opensearch-project/OpenSearch/pull/17070))
 - Add HTTP/2 protocol support to HttpRequest.HttpVersion ([#17248](https://github.com/opensearch-project/OpenSearch/pull/17248))
+- Fix missing bucket in terms aggregation with missing value ([#17418](https://github.com/opensearch-project/OpenSearch/pull/17418))
 
 ### Security
 

CHANGELOG.md

@@ -11,6 +11,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 - Introduce a setting to disable download of full cluster state from remote on term mismatch([#16798](https://github.com/opensearch-project/OpenSearch/pull/16798/))
 - Added ability to retrieve value from DocValues in a flat_object filed([#16802](https://github.com/opensearch-project/OpenSearch/pull/16802))
 - Improve performace of NumericTermAggregation by avoiding unnecessary sorting([#17252](https://github.com/opensearch-project/OpenSearch/pull/17252))
+- [Rule Based Auto-tagging] Add in-memory attribute value store ([#17342](https://github.com/opensearch-project/OpenSearch/pull/17342))
 - [Star Tree] [Search] Resolving keyword & numeric bucket aggregation with metric aggregation using star-tree ([#17165](https://github.com/opensearch-project/OpenSearch/pull/17165))
 
 ### Dependencies
@@ -21,6 +22,9 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 - Bump `org.apache.ant:ant` from 1.10.14 to 1.10.15 ([#17288](https://github.com/opensearch-project/OpenSearch/pull/17288))
 - Bump netty from 4.1.117.Final to 4.1.118.Final ([#17320](https://github.com/opensearch-project/OpenSearch/pull/17320))
 - Bump `reactor_netty` from 1.1.26 to 1.1.27 ([#17322](https://github.com/opensearch-project/OpenSearch/pull/17322))
+- Bump `me.champeau.gradle.japicmp` from 0.4.5 to 0.4.6 ([#17375](https://github.com/opensearch-project/OpenSearch/pull/17375))
+- Bump `com.google.api.grpc:proto-google-common-protos` from 2.37.1 to 2.52.0 ([#17379](https://github.com/opensearch-project/OpenSearch/pull/17379))
+- Bump `net.minidev:json-smart` from 2.5.1 to 2.5.2 ([#17378](https://github.com/opensearch-project/OpenSearch/pull/17378))
 
 ### Changed
 - Convert transport-reactor-netty4 to use gradle version catalog [#17233](https://github.com/opensearch-project/OpenSearch/pull/17233)
@@ -35,6 +39,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 - Fix exists queries on nested flat_object fields throws exception ([#16803](https://github.com/opensearch-project/OpenSearch/pull/16803))
 - Add highlighting for wildcard search on `match_only_text` field ([#17101](https://github.com/opensearch-project/OpenSearch/pull/17101))
 - Fix illegal argument exception when creating a PIT ([#16781](https://github.com/opensearch-project/OpenSearch/pull/16781))
+- Fix HTTP API calls that hang with 'Accept-Encoding: zstd' ([#17408](https://github.com/opensearch-project/OpenSearch/pull/17408))
 
 ### Security
 

MAINTAINERS.md

@@ -8,7 +8,7 @@ This document contains a list of maintainers in this repo. See [opensearch-proje
 |--------------------------|---------------------------------------------------------|-------------|
 | Anas Alkouz              | [anasalkouz](https://github.com/anasalkouz)             | Amazon      |
 | Andrew Ross              | [andrross](https://github.com/andrross)                 | Amazon      |
-| Andriy Redko             | [reta](https://github.com/reta)                         | Aiven       |
+| Andriy Redko             | [reta](https://github.com/reta)                         | Independent |
 | Ankit Jain               | [jainankitk](https://github.com/jainankitk)             | Amazon      |
 | Ashish Singh             | [ashking94](https://github.com/ashking94)               | Amazon      |
 | Bukhtawar Khan           | [Bukhtawar](https://github.com/Bukhtawar)               | Amazon      |

codecov.yml

@@ -4,6 +4,7 @@ codecov:
 ignore:
   - "test"
   - "benchmarks"
+  - "plugins/arrow-flight-rpc/**/org/apache/arrow/flight/**"
 
 coverage:
   precision: 2

distribution/packages/src/common/env/opensearch

@@ -3,17 +3,17 @@
 ################################
 
 # OpenSearch home directory
-#OPENSEARCH_HOME=/usr/share/opensearch
+OPENSEARCH_HOME=/usr/share/opensearch
 
 # OpenSearch Java path
-#OPENSEARCH_JAVA_HOME=
+#OPENSEARCH_JAVA_HOME=/usr/lib/jvm/java-11-amazon-corretto
 
 # OpenSearch configuration directory
 # Note: this setting will be shared with command-line tools
-OPENSEARCH_PATH_CONF=${path.conf}
+OPENSEARCH_PATH_CONF=/etc/opensearch
 
 # OpenSearch PID directory
-#PID_DIR=/var/run/opensearch
+PID_DIR=/var/run/opensearch
 
 # Additional Java OPTS
 #OPENSEARCH_JAVA_OPTS=
@@ -25,11 +25,12 @@ OPENSEARCH_PATH_CONF=${path.conf}
 # OpenSearch service
 ################################
 
-# SysV init.d
-#
 # The number of seconds to wait before checking if OpenSearch started successfully as a daemon process
 OPENSEARCH_STARTUP_SLEEP_TIME=5
 
+# Notification for systemd
+OPENSEARCH_SD_NOTIFY=true
+
 ################################
 # System properties
 ################################
@@ -49,4 +50,4 @@ OPENSEARCH_STARTUP_SLEEP_TIME=5
 # Maximum number of VMA (Virtual Memory Areas) a process can own
 # When using Systemd, this setting is ignored and the 'vm.max_map_count'
 # property is set at boot time in /usr/lib/sysctl.d/opensearch.conf
-#MAX_MAP_COUNT=262144
+#MAX_MAP_COUNT=262144

distribution/packages/src/common/systemd/opensearch.service

@@ -1,18 +1,25 @@
+# Copyright OpenSearch Contributors
+# SPDX-License-Identifier: Apache-2.0
+#
+# The OpenSearch Contributors require contributions made to
+# this file be licensed under the Apache-2.0 license or a
+# compatible open source license.
+
+# Description:
+# Default opensearch.service file
+
 [Unit]
 Description=OpenSearch
-Documentation=https://www.elastic.co
+Documentation=https://opensearch.org/
 Wants=network-online.target
 After=network-online.target
 
 [Service]
 Type=notify
 RuntimeDirectory=opensearch
 PrivateTmp=true
-Environment=OPENSEARCH_HOME=/usr/share/opensearch
-Environment=OPENSEARCH_PATH_CONF=${path.conf}
-Environment=PID_DIR=/var/run/opensearch
-Environment=OPENSEARCH_SD_NOTIFY=true
-EnvironmentFile=-${path.env}
+EnvironmentFile=-/etc/default/opensearch
+EnvironmentFile=-/etc/sysconfig/opensearch
 
 WorkingDirectory=/usr/share/opensearch
 
@@ -29,6 +36,7 @@ ExecStart=/usr/share/opensearch/bin/systemd-entrypoint -p ${PID_DIR}/opensearch.
 # logging, you can simply remove the "quiet" option from ExecStart.
 StandardOutput=journal
 StandardError=inherit
+SyslogIdentifier=opensearch
 
 # Specifies the maximum file descriptor number that can be opened by this process
 LimitNOFILE=65535
@@ -60,6 +68,97 @@ SuccessExitStatus=143
 # Allow a slow startup before the systemd notifier module kicks in to extend the timeout
 TimeoutStartSec=75
 
+# Prevent modifications to the control group filesystem
+ProtectControlGroups=true
+
+# Prevent loading or reading kernel modules
+ProtectKernelModules=true
+
+# Prevent altering kernel tunables (sysctl parameters)
+ProtectKernelTunables=true
+
+# Set device access policy to 'closed', allowing access only to specific devices
+DevicePolicy=closed
+
+# Make /proc invisible to the service, enhancing isolation
+ProtectProc=invisible
+
+# Make /usr, /boot, and /etc read-only (less restrictive than 'strict')
+ProtectSystem=full
+
+# Prevent changes to control groups (redundant with earlier setting, can be removed)
+ProtectControlGroups=yes
+
+# Prevent changing the execution domain
+LockPersonality=yes
+
+
+# System call filtering
+# System call filterings which restricts which system calls a process can make
+# @ means allowed
+# ~ means not allowed
+SystemCallFilter=@system-service
+SystemCallFilter=~@reboot
+SystemCallFilter=~@swap
+
+SystemCallErrorNumber=EPERM
+
+# Capability restrictions
+# Remove the ability to block system suspends
+CapabilityBoundingSet=~CAP_BLOCK_SUSPEND
+
+# Remove the ability to establish leases on files
+CapabilityBoundingSet=~CAP_LEASE
+
+# Remove the ability to use system resource accounting
+CapabilityBoundingSet=~CAP_SYS_PACCT
+
+# Remove the ability to configure TTY devices
+CapabilityBoundingSet=~CAP_SYS_TTY_CONFIG
+
+# Remov below capabilities:
+# - CAP_SYS_ADMIN: Various system administration operations
+# - CAP_SYS_PTRACE: Ability to trace processes
+# - CAP_NET_ADMIN: Various network-related operations
+CapabilityBoundingSet=~CAP_SYS_ADMIN ~CAP_SYS_PTRACE ~CAP_NET_ADMIN
+
+
+# Address family restrictions
+RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX
+
+# Filesystem Access
+
+ReadWritePaths=/var/log/opensearch
+ReadWritePaths=/var/lib/opensearch
+ReadWritePaths=-/etc/opensearch
+ReadWritePaths=-/mnt/snapshots
+
+## Allow read access to system files
+ReadOnlyPaths=/etc/os-release /usr/lib/os-release /etc/system-release
+
+## Allow read access to Linux IO stats
+ReadOnlyPaths=/proc/self/mountinfo /proc/diskstats
+
+## Allow read access to control group stats
+ReadOnlyPaths=/proc/self/cgroup /sys/fs/cgroup/cpu /sys/fs/cgroup/cpu/-
+ReadOnlyPaths=/sys/fs/cgroup/cpuacct /sys/fs/cgroup/cpuacct/- /sys/fs/cgroup/memory /sys/fs/cgroup/memory/-
+
+
+RestrictNamespaces=true
+
+NoNewPrivileges=true
+
+# Memory and execution protection
+MemoryDenyWriteExecute=true           # Prevent creating writable executable memory mappings
+SystemCallArchitectures=native        # Allow only native system calls
+KeyringMode=private                   # Service does not share key material with other services
+LockPersonality=true                  # Prevent changing ABI personality
+RestrictSUIDSGID=true                 # Prevent creating SUID/SGID files
+RestrictRealtime=true                 # Prevent acquiring realtime scheduling
+ProtectHostname=true                  # Prevent changes to system hostname
+ProtectKernelLogs=true                # Prevent reading/writing kernel logs
+ProtectClock=true                     # Prevent tampering with the system clock
+
 [Install]
 WantedBy=multi-user.target
 

gradle/libs.versions.toml

@@ -82,7 +82,7 @@ opentelemetry         = "1.46.0"
 opentelemetrysemconv  = "1.29.0-alpha"
 
 # arrow dependencies
-arrow                 = "17.0.0"
+arrow                 = "18.1.0"
 flatbuffers           = "2.0.0"
 
 [libraries]

libs/arrow-spi/build.gradle

@@ -10,79 +10,11 @@
  */
 
 testingConventions.enabled = false
+
 dependencies {
   api project(':libs:opensearch-core')
-  api "org.apache.arrow:arrow-vector:${versions.arrow}"
-  api "org.apache.arrow:arrow-format:${versions.arrow}"
-  api "org.apache.arrow:arrow-memory-core:${versions.arrow}"
-  runtimeOnly "org.apache.arrow:arrow-memory-netty-buffer-patch:${versions.arrow}"
-  runtimeOnly "org.apache.arrow:arrow-memory-netty:${versions.arrow}"
-  runtimeOnly "io.netty:netty-buffer:${versions.netty}"
-  runtimeOnly "io.netty:netty-common:${versions.netty}"
-
-  runtimeOnly "com.google.flatbuffers:flatbuffers-java:${versions.flatbuffers}"
-  runtimeOnly "org.slf4j:slf4j-api:${versions.slf4j}"
-  runtimeOnly "com.fasterxml.jackson.core:jackson-databind:${versions.jackson}"
-  api "com.fasterxml.jackson.core:jackson-annotations:${versions.jackson}"
-
-  implementation "commons-codec:commons-codec:${versions.commonscodec}"
 }
 
 tasks.named('forbiddenApisMain').configure {
   replaceSignatureFiles 'jdk-signatures'
 }
-
-tasks.named('thirdPartyAudit').configure {
-  ignoreMissingClasses(
-    // Logging frameworks
-    'org.apache.commons.logging.Log',
-    'org.apache.commons.logging.LogFactory',
-    'org.apache.log4j.Level',
-    'org.apache.log4j.Logger',
-    'org.slf4j.impl.StaticLoggerBinder',
-    'org.slf4j.impl.StaticMDCBinder',
-    'org.slf4j.impl.StaticMarkerBinder',
-
-    // Reactor BlockHound
-    'reactor.blockhound.BlockHound$Builder',
-    'reactor.blockhound.integration.BlockHoundIntegration'
-  )
-
-  ignoreViolations(
-    "io.netty.util.internal.PlatformDependent0",
-    "io.netty.util.internal.PlatformDependent0\$1",
-    "io.netty.util.internal.PlatformDependent0\$2",
-    "io.netty.util.internal.PlatformDependent0\$3",
-    "io.netty.util.internal.PlatformDependent0\$4",
-    "io.netty.util.internal.PlatformDependent0\$6",
-    "io.netty.util.internal.shaded.org.jctools.queues.BaseLinkedQueueConsumerNodeRef",
-    "io.netty.util.internal.shaded.org.jctools.queues.BaseLinkedQueueProducerNodeRef",
-    "io.netty.util.internal.shaded.org.jctools.queues.BaseMpscLinkedArrayQueueColdProducerFields",
-    "io.netty.util.internal.shaded.org.jctools.queues.BaseMpscLinkedArrayQueueConsumerFields",
-    "io.netty.util.internal.shaded.org.jctools.queues.BaseMpscLinkedArrayQueueProducerFields",
-    "io.netty.util.internal.shaded.org.jctools.queues.LinkedQueueNode",
-    "io.netty.util.internal.shaded.org.jctools.queues.MpmcArrayQueueConsumerIndexField",
-    "io.netty.util.internal.shaded.org.jctools.queues.MpmcArrayQueueProducerIndexField",
-    "io.netty.util.internal.shaded.org.jctools.queues.MpscArrayQueueConsumerIndexField",
-    "io.netty.util.internal.shaded.org.jctools.queues.MpscArrayQueueProducerIndexField",
-    "io.netty.util.internal.shaded.org.jctools.queues.MpscArrayQueueProducerLimitField",
-    "io.netty.util.internal.shaded.org.jctools.util.UnsafeAccess",
-    "io.netty.util.internal.shaded.org.jctools.util.UnsafeLongArrayAccess",
-    "io.netty.util.internal.shaded.org.jctools.util.UnsafeRefArrayAccess",
-    "io.netty.util.internal.shaded.org.jctools.queues.unpadded.MpscUnpaddedArrayQueueConsumerIndexField",
-    "io.netty.util.internal.shaded.org.jctools.queues.unpadded.MpscUnpaddedArrayQueueProducerIndexField",
-    "io.netty.util.internal.shaded.org.jctools.queues.unpadded.MpscUnpaddedArrayQueueProducerLimitField",
-    "org.apache.arrow.memory.ArrowBuf",
-    "org.apache.arrow.memory.util.ByteFunctionHelpers",
-    "org.apache.arrow.memory.util.MemoryUtil",
-    "org.apache.arrow.memory.util.MemoryUtil\$1",
-    "org.apache.arrow.memory.util.hash.MurmurHasher",
-    "org.apache.arrow.memory.util.hash.SimpleHasher",
-    "org.apache.arrow.vector.BaseFixedWidthVector",
-    "org.apache.arrow.vector.BitVectorHelper",
-    "org.apache.arrow.vector.Decimal256Vector",
-    "org.apache.arrow.vector.DecimalVector",
-    "org.apache.arrow.vector.util.DecimalUtility",
-    "org.apache.arrow.vector.util.VectorAppender"
-  )
-}