Async write routing table flow #8
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…-project#12989) * Cache the shard routings with no weight for faster access The list of shards to run a query is determined for every request and the weight of the nodes guides the shard selection. Currently, IndexRoutingTable caches the shard routings with weight for faster access. But, during cases where the fail open option is enabled, shards with no weight is also returned lower in the order along with shards with weights. They will be used as fall back if the shards with weights can't be used due to some error. The shard routing with no weight is not cached, hence it does a full loop for every request, this impacts the search latency when the number of shards to query or the number of nodes in the cluster is high. The latency impact is very high when both the number of shards and the number of nodes are high. This change introduces a caching mechanism for shard routing with no weights similar to the existing cache for shard routing with weights. Signed-off-by: Prabhakar Sithanandam <[email protected]> Co-authored-by: Prabhakar Sithanandam <[email protected]>
…2883) * Fix bulk API ignores ingest pipeline for upsert Signed-off-by: Gao Binlong <[email protected]> * Modify change log Signed-off-by: Gao Binlong <[email protected]> * Use existing pipeline instead Signed-off-by: Gao Binlong <[email protected]> --------- Signed-off-by: Gao Binlong <[email protected]>
) * Update links to documentation in rest-api-spec Signed-off-by: Craig Perkins <[email protected]> * Remove changes in Debug_OpenSearch.xml Signed-off-by: Craig Perkins <[email protected]> * Remove changes in Debug_OpenSearch.xml Signed-off-by: Craig Perkins <[email protected]> * Add to CHANGELOG Signed-off-by: Craig Perkins <[email protected]> --------- Signed-off-by: Craig Perkins <[email protected]>
…nsearch-project#13056) Signed-off-by: Andriy Redko <[email protected]>
* Bump up commons-compress to 1.26.0 to fix CVE Signed-off-by: Aman Khare <[email protected]> * Change log entry Signed-off-by: Aman Khare <[email protected]> * Update ignoreMissingClasses Signed-off-by: Aman Khare <[email protected]> * Update commons-codec and commons-lang3 dependencies also Signed-off-by: Aman Khare <[email protected]> * Upgrade commons-codec to 1.16.1 Signed-off-by: Aman Khare <[email protected]> * Add commons-io dependency in plugin-cli build.gradle Signed-off-by: Sandesh Kumar <[email protected]> * Revert "Update ignoreMissingClasses" This reverts commit d92fbda. Signed-off-by: Sandesh Kumar <[email protected]> * Adding SHA for commons-io-2.15.1.jar Signed-off-by: Sandesh Kumar <[email protected]> * adding license, notice files for commons-io Signed-off-by: Sandesh Kumar <[email protected]> * Add missing classes for thirdPartyAudit Signed-off-by: Sandesh Kumar <[email protected]> * Refactor Signed-off-by: Sandesh Kumar <[email protected]> * Test commit - to be reverted Signed-off-by: Sandesh Kumar <[email protected]> * Bump commons-compress to 1.26.1, tika to 2.9.1 Signed-off-by: Sandesh Kumar <[email protected]> * Remove Charsets class from exclusion list - not missing Signed-off-by: Sandesh Kumar <[email protected]> * Update tika to 2.9.2 Signed-off-by: Sandesh Kumar <[email protected]> * commons-io 2.16.0 Signed-off-by: Sandesh Kumar <[email protected]> * Refactor commons-io dependency mentions to avoid manual version setting/update Signed-off-by: Sandesh Kumar <[email protected]> --------- Signed-off-by: Aman Khare <[email protected]> Signed-off-by: Sandesh Kumar <[email protected]> Co-authored-by: Aman Khare <[email protected]>
* Add bwc version 2.13.1 Signed-off-by: GitHub <[email protected]> * Fix version identifier Signed-off-by: Kunal Kotwani <[email protected]> --------- Signed-off-by: GitHub <[email protected]> Signed-off-by: Kunal Kotwani <[email protected]> Co-authored-by: opensearch-ci-bot <[email protected]> Co-authored-by: Kunal Kotwani <[email protected]>
…ieldScripts (opensearch-project#12837) --------- Signed-off-by: Mohammad Qureshi <[email protected]> Signed-off-by: Rishabh Maurya <[email protected]> Co-authored-by: Rishabh Maurya <[email protected]>
…h-project#13044) Signed-off-by: Sachin Kale <[email protected]>
…h-project#12686) Signed-off-by: Shubh Sahu <[email protected]>
…12981) * Finish contributing updates; thanks @acarbonetto Signed-off-by: Stephen Crawford <[email protected]> * Fix indenting Signed-off-by: Stephen Crawford <[email protected]> * Readd build failure notes Signed-off-by: Stephen Crawford <[email protected]> * Update CONTRIBUTING with review tips Signed-off-by: Andrew Carbonetto <[email protected]> * Update CONTRIBUTING with review tips Signed-off-by: Andrew Carbonetto <[email protected]> * Update contributing comment for review comment Signed-off-by: Andrew Carbonetto <[email protected]> * Fix contruibuting doc Signed-off-by: Stephen Crawford <[email protected]> * Update CONTRIBUTING.md Co-authored-by: Owais Kazi <[email protected]> Signed-off-by: Stephen Crawford <[email protected]> * Update CONTRIBUTING.md Co-authored-by: Owais Kazi <[email protected]> Signed-off-by: Stephen Crawford <[email protected]> * Update CONTRIBUTING.md Co-authored-by: Owais Kazi <[email protected]> Signed-off-by: Stephen Crawford <[email protected]> * Update CONTRIBUTING.md Co-authored-by: Owais Kazi <[email protected]> Signed-off-by: Stephen Crawford <[email protected]> * Update CONTRIBUTING.md Co-authored-by: Owais Kazi <[email protected]> Signed-off-by: Stephen Crawford <[email protected]> * Update CONTRIBUTING.md Co-authored-by: Owais Kazi <[email protected]> Signed-off-by: Stephen Crawford <[email protected]> * Update CONTRIBUTING.md Co-authored-by: Owais Kazi <[email protected]> Signed-off-by: Stephen Crawford <[email protected]> * Update CONTRIBUTING.md Co-authored-by: Owais Kazi <[email protected]> Signed-off-by: Stephen Crawford <[email protected]> --------- Signed-off-by: Stephen Crawford <[email protected]> Signed-off-by: Andrew Carbonetto <[email protected]> Signed-off-by: Stephen Crawford <[email protected]> Co-authored-by: Andrew Carbonetto <[email protected]> Co-authored-by: Owais Kazi <[email protected]>
…opensearch-project#12988) Signed-off-by: Ashish Singh <[email protected]>
…rch-project#13067) * Remove reference to INDEX_MAPPER_DYNAMIC_SETTING in AutoCreateIndex Signed-off-by: Craig Perkins <[email protected]> * Add to CHANGELOG Signed-off-by: Craig Perkins <[email protected]> --------- Signed-off-by: Craig Perkins <[email protected]>
…th java.lang.NoSuchMethodError: java.nio.ByteBuffer.flip()Ljava/nio/ByteBuffer (opensearch-project#13100) Signed-off-by: Andriy Redko <[email protected]>
…pensearch-project#13099) WeightedRoutingKey is used as key in the HashMap to retreive the shard routings, for every usage the hash key is computed. WeightedRoutingKey internally uses WeightedRouting. As the number of the shards getting queried increases, the computation cost goes high. Computing the hash key during object creation as the object is immutable and using it through out the object thus saving on the additional computation. Signed-off-by: Prabhakar Sithanandam <[email protected]>
…tadata (opensearch-project#13023) * Introduce interface changes to read/write blob with object metadata --------- Signed-off-by: Sandeep Kumawat <[email protected]> Co-authored-by: Sandeep Kumawat <[email protected]>
…project#12967) * Adding support for more than one protocol for transport Signed-off-by: Vacha Shah <[email protected]> * Adding CHANGELOG entry Signed-off-by: Vacha Shah <[email protected]> * Addressing comments Signed-off-by: Vacha Shah <[email protected]> * Addressing comments Signed-off-by: Vacha Shah <[email protected]> * Removing determineTransportProtocol Signed-off-by: Vacha Shah <[email protected]> * Determine transport protocol only on first byte reference Signed-off-by: Vacha Shah <[email protected]> * Making InboundBytesHandler closeable Signed-off-by: Vacha Shah <[email protected]> * Fixing close() for InboundPipeline Signed-off-by: Vacha Shah <[email protected]> * Adding DeprecatedAPI annotation to japicmp task Signed-off-by: Vacha Shah <[email protected]> * Fixing for detect breaking changes workflow Signed-off-by: Vacha Shah <[email protected]> * Fixing recursion Signed-off-by: Vacha Shah <[email protected]> --------- Signed-off-by: Vacha Shah <[email protected]>
) Signed-off-by: Ashish Singh <[email protected]>
opensearch-project#12960) Signed-off-by: Gaurav Bafna <[email protected]>
…ng. (opensearch-project#12488) * Add support for randomizing remote store enabled testing. Signed-off-by: Rishikesh1159 <[email protected]> * fix spotless check Signed-off-by: Rishikesh1159 <[email protected]> * Updating logic to randomly use remote store once per cluster instead of once per node. Signed-off-by: Rishikesh1159 <[email protected]> * Add Remote Store Randomization within Replication Type Randomization. Signed-off-by: Rishikesh1159 <[email protected]> --------- Signed-off-by: Rishikesh1159 <[email protected]>
) * Bump com.gradle.enterprise from 3.16.2 to 3.17 Bumps com.gradle.enterprise from 3.16.2 to 3.17. --- updated-dependencies: - dependency-name: com.gradle.enterprise dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> * Update changelog Signed-off-by: dependabot[bot] <[email protected]> --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>
opensearch-project#13127) Signed-off-by: Craig Perkins <[email protected]>
…3040) Contributors and maintainers frequently put changelog entries in the wrong section. It's an easy mistake to make! Given that making changes intended for the next minor release is the norm, this change optimizes that process by making `CHANGELOG.md` contain _only_ entries for the next minor. Truly breaking changes intended for the next major are kept in a separate file. Credit to @msfroh for the idea. Signed-off-by: Andrew Ross <[email protected]> Signed-off-by: Kunal Kotwani <[email protected]> Co-authored-by: Kunal Kotwani <[email protected]>
* implement mark() in class FilterStreamInput Signed-off-by: kkewwei <[email protected]> * implement markSupported() in class FilterStreamInput Signed-off-by: kkewwei <[email protected]> * add CHANGELOG Signed-off-by: kkewwei <[email protected]> --------- Signed-off-by: kkewwei <[email protected]>
Signed-off-by: frameflare <[email protected]>
…y-hdfs (opensearch-project#13117) * Bump net.minidev:json-smart in /plugins/repository-hdfs Bumps [net.minidev:json-smart](https://github.com/netplex/json-smart-v2) from 2.5.0 to 2.5.1. - [Release notes](https://github.com/netplex/json-smart-v2/releases) - [Commits](netplex/json-smart-v2@2.5.0...2.5.1) --- updated-dependencies: - dependency-name: net.minidev:json-smart dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> * Updating SHAs Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: Andriy Redko <[email protected]> --------- Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: Andriy Redko <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>
opensearch-project#11868) Signed-off-by: Harish Bhakuni <[email protected]>
…equest (opensearch-project#12850) * Support derived fields definition in search request * adds support for fetch phase on derived fields * adds support for highlighting on derived fields --------- Signed-off-by: Rishabh Maurya <[email protected]>
…tartedShardsBatch (opensearch-project#12504) Signed-off-by: Aman Khare <[email protected]>
…ng flaky test (opensearch-project#13164) Signed-off-by: Andriy Redko <[email protected]>
…oject#13508) Remove check of non zero assertion of time as it is non deterministic Signed-off-by: Gaurav Bafna <[email protected]>
Signed-off-by: Shivansh Arora <[email protected]>
* add use compound file setting Signed-off-by: Samuel Herman <[email protected]> * review feedback Signed-off-by: Samuel Herman <[email protected]> --------- Signed-off-by: Samuel Herman <[email protected]>
…ClusterManagerFailover (opensearch-project#13505) Signed-off-by: Rahul Karajgikar <[email protected]> Co-authored-by: Rahul Karajgikar <[email protected]>
…fault to 'true' (opensearch-project#13532) Signed-off-by: Andriy Redko <[email protected]>
… in /modules/lang-mustache (opensearch-project#13559) * Bump com.github.spullara.mustache.java:compiler Bumps [com.github.spullara.mustache.java:compiler](https://github.com/spullara/mustache.java) from 0.9.11 to 0.9.13. - [Commits](spullara/mustache.java@mustache.java-0.9.11...mustache.java-0.9.13) --- updated-dependencies: - dependency-name: com.github.spullara.mustache.java:compiler dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> * Updating SHAs Signed-off-by: dependabot[bot] <[email protected]> * Update changelog Signed-off-by: dependabot[bot] <[email protected]> --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>
…#13469) * Update version for batch shard allocation to 2.14 Signed-off-by: Shivansh Arora <[email protected]>
…ixtures/hdfs-fixture (opensearch-project#13557) * Bump org.apache.commons:commons-text in /test/fixtures/hdfs-fixture Bumps org.apache.commons:commons-text from 1.11.0 to 1.12.0. --- updated-dependencies: - dependency-name: org.apache.commons:commons-text dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> * Update changelog Signed-off-by: dependabot[bot] <[email protected]> --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>
This solves the bug where RequestStats memory_size metric was going negative in certain scenarios as reported in the issue. It turns out that the issue occurs when an indexShard is deleted and then reallocated on the same node. So whenever stale entries from older shard are deleted, those are accounted for the new shard which has the same shardId. --------- Signed-off-by: Sagar Upadhyaya <[email protected]>
…pensearch-project#13556) * Bump org.hdrhistogram:HdrHistogram from 2.1.12 to 2.2.1 in /server Bumps [org.hdrhistogram:HdrHistogram](https://github.com/HdrHistogram/HdrHistogram) from 2.1.12 to 2.2.1. - [Release notes](https://github.com/HdrHistogram/HdrHistogram/releases) - [Commits](HdrHistogram/HdrHistogram@HdrHistogram-2.1.12...HdrHistogram-2.2.1) --- updated-dependencies: - dependency-name: org.hdrhistogram:HdrHistogram dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> * Updating SHAs Signed-off-by: dependabot[bot] <[email protected]> * Update changelog Signed-off-by: dependabot[bot] <[email protected]> --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>
…an be negative when searching (opensearch-project#13408) Signed-off-by: Gao Binlong <[email protected]>
…eStoreRestoreIT.testRTSRestoreWithNoDataPostRefreshPrimaryReplicaDown (opensearch-project#13489) * add TestIssueLogging on flaky test - org.opensearch.remotestore.RemoteStoreRestoreIT.testRTSRestoreWithNoDataPostRefreshPrimaryReplicaDown Signed-off-by: Varun Bansal <[email protected]>
Signed-off-by: Andrew Ross <[email protected]>
Signed-off-by: Sagar Upadhyaya <[email protected]> Co-authored-by: Sagar Upadhyaya <[email protected]>
Signed-off-by: Shivansh Arora <[email protected]>
…tiple versions (opensearch-project#13624) * Fix get field mapping API returns 404 error in mixed cluster with multiple versions Signed-off-by: Gao Binlong <[email protected]> * modify changelog Signed-off-by: Gao Binlong <[email protected]> * Add comment and remove unit test Signed-off-by: Gao Binlong <[email protected]> --------- Signed-off-by: Gao Binlong <[email protected]>
Signed-off-by: Kunal Khatua <[email protected]> Co-authored-by: Peter Nied <[email protected]> Co-authored-by: Peter Nied <[email protected]>
Signed-off-by: Shivansh Arora <[email protected]>
Signed-off-by: Arpit Bandejiya <[email protected]>
Signed-off-by: Himshikha Gupta <[email protected]>
…' into async-write
…' into async-write
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
[Describe what this change achieves]
Related Issues
Resolves #[Issue number to be closed when this PR is merged]
Check List
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.