Search only replicas (scale to zero) with Reader/Writer Separation #17299
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,87 @@ | ||
| /* | ||
| * SPDX-License-Identifier: Apache-2.0 | ||
| * | ||
| * The OpenSearch Contributors require contributions made to | ||
| * this file be licensed under the Apache-2.0 license or a | ||
| * compatible open source license. | ||
| */ | ||
|
|
||
| package org.opensearch.action.admin.indices.scale.searchonly; | ||
|
|
||
| import org.opensearch.core.common.io.stream.StreamInput; | ||
| import org.opensearch.core.common.io.stream.StreamOutput; | ||
| import org.opensearch.core.index.shard.ShardId; | ||
| import org.opensearch.transport.TransportRequest; | ||
|
|
||
| import java.io.IOException; | ||
| import java.util.List; | ||
|
|
||
| /** | ||
| * A transport request sent to nodes to facilitate shard synchronization during search-only scaling operations. | ||
| * <p> | ||
| * This request is sent from the cluster manager to data nodes that host primary shards for the target index | ||
| * during scale operations. It contains the index name and a list of shard IDs that need to be synchronized | ||
| * before completing a scale-down operation. | ||
| * <p> | ||
| * When a node receives this request, it performs final sync and flush operations on the specified shards, | ||
| * ensuring all operations are committed and the remote store is synced. This is a crucial step in | ||
| * the scale-down process to ensure no data loss occurs when the index transitions to search-only mode. | ||
| */ | ||
| class NodeSearchOnlyRequest extends TransportRequest { | ||
| private final String index; | ||
| private final List<ShardId> shardIds; | ||
|
|
||
| /** | ||
| * Constructs a new NodeSearchOnlyRequest. | ||
| * | ||
| * @param index the name of the index being scaled | ||
| * @param shardIds the list of shard IDs to be synchronized on the target node | ||
| */ | ||
| NodeSearchOnlyRequest(String index, List<ShardId> shardIds) { | ||
| this.index = index; | ||
| this.shardIds = shardIds; | ||
| } | ||
|
|
||
| /** | ||
| * Deserialization constructor. | ||
| * | ||
| * @param in the stream input to read from | ||
| * @throws IOException if there is an I/O error during deserialization | ||
| */ | ||
| NodeSearchOnlyRequest(StreamInput in) throws IOException { | ||
| super(in); | ||
| this.index = in.readString(); | ||
| this.shardIds = in.readList(ShardId::new); | ||
| } | ||
|
|
||
| /** | ||
| * Serializes this request to the given output stream. | ||
| * | ||
| * @param out the output stream to write to | ||
| * @throws IOException if there is an I/O error during serialization | ||
| */ | ||
| @Override | ||
| public void writeTo(StreamOutput out) throws IOException { | ||
| super.writeTo(out); | ||
| out.writeString(index); | ||
| out.writeList(shardIds); | ||
| } | ||
|
|
||
| /** | ||
| * Returns the index name associated with this request. | ||
| * | ||
| * @return the index name | ||
| */ | ||
| String getIndex() { | ||
| return index; | ||
| } | ||
|
|
||
| /** | ||
| * Returns the list of shard IDs to be synchronized. | ||
| * | ||
| * @return the list of shard IDs | ||
| */ | ||
| List<ShardId> getShardIds() { | ||
| return shardIds; | ||
| } | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,89 @@ | ||
| /* | ||
| * SPDX-License-Identifier: Apache-2.0 | ||
| * | ||
| * The OpenSearch Contributors require contributions made to | ||
| * this file be licensed under the Apache-2.0 license or a | ||
| * compatible open source license. | ||
| */ | ||
|
|
||
| package org.opensearch.action.admin.indices.scale.searchonly; | ||
|
|
||
| import org.opensearch.cluster.node.DiscoveryNode; | ||
| import org.opensearch.core.common.io.stream.StreamInput; | ||
| import org.opensearch.core.common.io.stream.StreamOutput; | ||
| import org.opensearch.core.transport.TransportResponse; | ||
|
|
||
| import java.io.IOException; | ||
| import java.util.List; | ||
|
|
||
| /** | ||
| * Response sent from nodes after processing a {@link NodeSearchOnlyRequest} during search-only scaling operations. | ||
| * <p> | ||
| * This response contains information about the node that processed the request and the results of | ||
| * synchronization attempts for each requested shard. The cluster manager uses these responses to | ||
| * determine whether it's safe to proceed with finalizing a scale-down operation. | ||
| * <p> | ||
| * Each response includes details about whether shards have any uncommitted operations or need | ||
| * additional synchronization, which would indicate the scale operation should be delayed until | ||
| * the cluster reaches a stable state. | ||
| */ | ||
| class NodeSearchOnlyResponse extends TransportResponse { | ||
| private final DiscoveryNode node; | ||
| private final List<ShardSearchOnlyResponse> shardResponses; | ||
|
|
||
| /** | ||
| * Constructs a new NodeSearchOnlyResponse. | ||
| * | ||
| * @param node the node that processed the synchronization request | ||
| * @param shardResponses the list of responses from individual shard synchronization attempts | ||
| */ | ||
| NodeSearchOnlyResponse(DiscoveryNode node, List<ShardSearchOnlyResponse> shardResponses) { | ||
| this.node = node; | ||
| this.shardResponses = shardResponses; | ||
| } | ||
|
|
||
| /** | ||
| * Deserialization constructor. | ||
| * | ||
| * @param in the stream input to read from | ||
| * @throws IOException if there is an I/O error during deserialization | ||
| */ | ||
| NodeSearchOnlyResponse(StreamInput in) throws IOException { | ||
| node = new DiscoveryNode(in); | ||
| shardResponses = in.readList(ShardSearchOnlyResponse::new); | ||
| } | ||
|
|
||
| /** | ||
| * Serializes this response to the given output stream. | ||
| * | ||
| * @param out the output stream to write to | ||
| * @throws IOException if there is an I/O error during serialization | ||
| */ | ||
| @Override | ||
| public void writeTo(StreamOutput out) throws IOException { | ||
| node.writeTo(out); | ||
| out.writeList(shardResponses); | ||
| } | ||
|
|
||
| /** | ||
| * Returns the node that processed the synchronization request. | ||
| * | ||
| * @return the discovery node information | ||
| */ | ||
| public DiscoveryNode getNode() { | ||
| return node; | ||
| } | ||
|
|
||
| /** | ||
| * Returns the list of shard-level synchronization responses. | ||
| * <p> | ||
| * These responses contain critical information about the state of each shard, | ||
| * including whether there are uncommitted operations or if additional synchronization | ||
| * is needed before the scale operation can safely proceed. | ||
| * | ||
| * @return the list of shard responses | ||
| */ | ||
| public List<ShardSearchOnlyResponse> getShardResponses() { | ||
| return shardResponses; | ||
| } | ||
| } |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I have used these settings to test the search only replicas with remote store as filesystem, will remove them and change to default once I have the green tests and finalized the PR review.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm wondering if we should add a param to easily set these from command line with ./gradlew run.