Version vector: compute locations only once during commits.

[dlambrig]

During commits with version vector enabled, compute location list only once on the resolver. Recalculating a second time could generate a different random number, hence a different set of locations. The location list is generted with a call to getPushLocations, which chooses random locations if the total number of tLogs mod the replica count results in collisions.

Tested by loading commits from branch version-vector-disk-queue and observing getPushLocations is called in both the resolver and on the CP in assignMutationsToStorageServers. Observed different locations chosen without the PR, resulting in data not sent to tLogs when it should have been. Added an assert if data was generated for a location but was not sent to a tLog to catch any such future cases.

Joshua 20250204-185355-dlambrig-83a293ba8c4a1d87

Code-Reviewer Section

The general pull request guidelines can be found here.

Please check each of the following things and check all boxes before accepting a PR.

• The PR has a description, explaining both the problem and the solution.
• The description mentions which forms of testing were done and the testing seems reasonable.
• Every function/class/actor that was touched is reasonably well documented.

For Release-Branches

If this PR is made against a release-branch, please also check the following:

• This change/bugfix is a cherry-pick from the next younger branch (younger release-branch or main if this is the youngest branch)
• There is a good reason why this PR needs to go into a release branch and this reason is documented (either in the description above or in a linked GitHub issue)

[foundationdb-ci]

Result of foundationdb-pr-clang-ide on Linux CentOS 7

• Commit ID: c54c727
• Duration 0:21:25
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

• Commit ID: c54c727
• Duration 0:39:38
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-clang-arm on Linux CentOS 7

• Commit ID: c54c727
• Duration 0:47:22
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

• Commit ID: c54c727
• Duration 0:54:57
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)
• Cluster Test Logs zip file of the test logs (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-macos on macOS Ventura 13.x

• Commit ID: c54c727
• Duration 1:02:15
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-clang on Linux CentOS 7

• Commit ID: c54c727
• Duration 1:19:12
• Result: ❌ FAILED
• Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr on Linux CentOS 7

• Commit ID: c54c727
• Duration 1:33:00
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[jzhou77]

This is not using tags, which can be a much smaller set than the set storedLocations. So this doesn't seem right.

The idea of logSystem->getPushLocations() is to find the tlogs that should receive the tags. We don't want to send the message to a tlog that doesn't have to receive it.

[dlambrig]

Yes, we may send the message to more tLogs than we need to, which is suboptimal. I will think of a way to optimize that.

[jzhou77]

As I thought more about it, this is called by writeTypedMessage(), which is called almost on every mutation, e.g., in writeMutation(). So I think we need to optimize this to avoid sending every mutation to all tlogs.

[foundationdb-ci]

Result of foundationdb-pr-clang-ide on Linux CentOS 7

• Commit ID: 3c7b5cd
• Duration 0:20:21
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

• Commit ID: 3c7b5cd
• Duration 0:38:18
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-clang-arm on Linux CentOS 7

• Commit ID: 3c7b5cd
• Duration 0:46:22
• Result: ❌ FAILED
• Error: Error while executing command: ctest -j ${NPROC} --no-compress-output -T test --output-on-failure. Reason: exit status 8
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

• Commit ID: 3c7b5cd
• Duration 0:57:35
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)
• Cluster Test Logs zip file of the test logs (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-macos on macOS Ventura 13.x

• Commit ID: 3c7b5cd
• Duration 1:03:41
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-clang on Linux CentOS 7

• Commit ID: 3c7b5cd
• Duration 1:20:40
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr on Linux CentOS 7

• Commit ID: 3c7b5cd
• Duration 1:24:53
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[sbodagala]

What is the difference between "prev_tags" and "tags"? And, why is "tags" being used in place of "prev_tags" in the other/later commit?

[sbodagala]

Also, are we sure that the "prev_tags"/"tags" that is being used here is exactly the same set of tags that the resolver uses too? Or, is it possible that meta mutations (like, move_keys) can cause this set to be different on the resolver and the proxy?

[dlambrig]

What is the difference between "prev_tags" and "tags"? And, why is "tags" being used in place of "prev_tags" in the other/later commit?

They are the same. I can remove tags and just use prev_tags as it is a class member.

[dlambrig]

Also, are we sure that the "prev_tags"/"tags" that is being used here is exactly the same set of tags that the resolver uses too? Or, is it possible that meta mutations (like, move_keys) can cause this set to be different on the resolver and the proxy?

Recall that the reason we moved metadata mutation processing to the Resolver was to handle move_keys and similar cases. We did this in order to calculate the tpcv locations out of the complete set of tags for the batch. Version vector will not work unless that set is complete.