feat(store): Pruning of the transaction and block indexer #24
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…#1439) * ADR 101: Implement pruning mechanism (cometbft#1150) * Added pruning mechanism for blocks and abci block results * Added support for datacompanion and application retain heights --------- Signed-off-by: Thane Thomson <[email protected]> Co-authored-by: Thane Thomson <[email protected]> * Removed extra code added via cherry pick conflicts and regenerated mock * Fixed changelog placement and added entry on pruning mechanism breaking * ADR-101: Metrics to monitor the pruning (cometbft#1234) * Metrics to report on the data companion retain height * Added metric to report the application retain height as well * Added metrics to report the blockstore base height and the abci results base height --------- Co-authored-by: Thane Thomson <[email protected]> * ADR-101: Pruning mechanism minor fixes (cometbft#1246) * Applied @serigo-mena's comments * Update state/pruner.go Co-authored-by: Thane Thomson <[email protected]> * Applied further comments from PR * Fixed batching interval --------- Co-authored-by: Thane Thomson <[email protected]> * Delete bad changelog entry --------- Signed-off-by: Thane Thomson <[email protected]> Co-authored-by: Thane Thomson <[email protected]> Co-authored-by: Daniel Cason <[email protected]>
* state: Invert logic of whether retain heights are set to be more readable Signed-off-by: Thane Thomson <[email protected]> * state: Refactor findMinRetainHeight logic for clarity and to respect config Signed-off-by: Thane Thomson <[email protected]> * state: Rename findMinRetainHeight to findMinBlockRetainHeight for clarity Signed-off-by: Thane Thomson <[email protected]> * state: Rename Pruner.SetApplicationRetainHeight to SetApplicationBlockRetainHeight for clarity Signed-off-by: Thane Thomson <[email protected]> * state: Rename pruner SetCompanionRetainHeight to SetCompanionBlockRetainHeight for clarity Signed-off-by: Thane Thomson <[email protected]> * state: Refactor - shorten local variable name Signed-off-by: Thane Thomson <[email protected]> * state: Refactor - rename local method name for clarity Signed-off-by: Thane Thomson <[email protected]> * state: Refactor - remove redundant condition Signed-off-by: Thane Thomson <[email protected]> * state: Refactor - use helper instead of redefining logic Signed-off-by: Thane Thomson <[email protected]> * state: Refactor - shorten local variable names Signed-off-by: Thane Thomson <[email protected]> * state: Refactor - shorten local variable names Signed-off-by: Thane Thomson <[email protected]> * state: Refactor Pruner.SetApplicationBlockRetainHeight logic The application retain height should be set regardless of what the companion retain height is. Pruning should take place based on whichever of the two values is lower. There is no need to consider the companion retain height in this logic. Signed-off-by: Thane Thomson <[email protected]> * state: Refactor Pruner.SetCompanionBlockRetainHeight logic Similar refactor to that done for Pruner.SetApplicationBlockRetainHeight. There is no need to consider the application retain height during this function call. Signed-off-by: Thane Thomson <[email protected]> * state: Refactor Pruner.SetABCIResRetainHeight logic Aligns the logic of this method similar to that in the other retain height setter methods. Also fixes a logic bug where setting the retain height would skip updating the metrics. Signed-off-by: Thane Thomson <[email protected]> * state: Refactor - shorten local variable names Signed-off-by: Thane Thomson <[email protected]> * state: Simplify pruner logic assuming retain heights are always set We can simplify the pruner logic if we assume that the initial retain heights are always set on node startup. Signed-off-by: Thane Thomson <[email protected]> * node: Tell pruner whether companion is enabled Signed-off-by: Thane Thomson <[email protected]> * state: Expand error message detail Signed-off-by: Thane Thomson <[email protected]> * state+store: Align tests with new logic Signed-off-by: Thane Thomson <[email protected]> * state: Refactor - shorten local variable names Signed-off-by: Thane Thomson <[email protected]> * rpc/grpc: Return trace IDs in error responses from pruning service Signed-off-by: Thane Thomson <[email protected]> * node: Refactor/expand pruning service initialization logic Signed-off-by: Thane Thomson <[email protected]> * test/e2e: Minor cosmetic tweaks to gRPC tests Signed-off-by: Thane Thomson <[email protected]> * test/e2e: Enable data companion pruning Signed-off-by: Thane Thomson <[email protected]> * node: Refactor - extract pruner creation as function Signed-off-by: Thane Thomson <[email protected]> * test/e2e: Only enable companion-based pruning on full nodes and validators Signed-off-by: Thane Thomson <[email protected]> * state: Only start ABCI results pruning routine when data companion is enabled Signed-off-by: Thane Thomson <[email protected]> * state: Fix TestMinRetainHeight logic Signed-off-by: Thane Thomson <[email protected]> * node: Fix companion retain height initialization logic Signed-off-by: Thane Thomson <[email protected]> * test/e2e: Expand testing framework Allows explicit control over whether or not pruning should take a data companion into account for full nodes or validators in our E2E framework. This allows greater flexibility in the E2E framework to test nodes both with and without data companion-related functionality. Expands our standard CI manifest to expect data companions attached to two of the nodes. Signed-off-by: Thane Thomson <[email protected]> * state: Rename background routines Signed-off-by: Thane Thomson <[email protected]> * node: Apply change from code review Signed-off-by: Thane Thomson <[email protected]> --------- Signed-off-by: Thane Thomson <[email protected]>
…cometbft#1476) * indexer: Implement pruning mechanism (cometbft#1176) * done stateDB writes batching * remove forgotten debug print * benchmarking for state db save * state db save benchmarking * batching in store * benchmarking for store batching * Added pruning mechanism -initial version * removed comments * Rewrore pruning mechanism as service. Working code but tests need to change. * Minor edits * Removed comments * Fixed linter * Fixed linter * Fixed state and block store tests Proper ticker used instead of sleep for pruning service job * Fixed failing http test * fixed linter errors and reduced sleep in http test * Fixed e2e test and failing linter * Applied some of @thanethomson's PR comments * add changelog * Separated retain height functions * fixed linter and test * Fixed code in select min height` * Fixed bugs causing failing tests * Added config flag for companion initial height Fixed failing test due to wrong check on blockstore base height * Tested setting the companion initial height via config. Moved pruner initialization into blockExecutor again. * Added config for pruning frequency * minor * Added additional test for companion retain height setting * script for adjasting test loadtime parameeters * Added simple testing tool for goleveldb * measuring tool + pruning setting * draft of tx_index pruning * using bus to prune tx_index * draft of pruning * working prototipe on tx_index txIndexer pruning * consensus: Fix test broken due to missing state store mocks Signed-off-by: Thane Thomson <[email protected]> * config: Refactor to match updated ADR ADR-101 was updated recently to reflect the desired configuration file changes - this updates the code to match the ADR. It also changes the "frequency" variable to more accurately be called an "interval", since it represents a sleep period between pruning operations as opposed to the number of times it should be run per second. The interval is also defined as a time.Duration now to facilitate more ergonomic usage from the configuration file. Signed-off-by: Thane Thomson <[email protected]> * Format Signed-off-by: Thane Thomson <[email protected]> * state: Rename pruner sleep time to "interval" Signed-off-by: Thane Thomson <[email protected]> * state: Refactor pruner config Make the pruner config private such that its configuration is fixed at runtime. With the previous approach, the PrunerInterval method could be abused to change the interval at runtime since it returns a function that can operate on the public type. Signed-off-by: Thane Thomson <[email protected]> * state: Fix minor nits in function docstrings Signed-off-by: Thane Thomson <[email protected]> * state: Remove unused method Signed-off-by: Thane Thomson <[email protected]> * state: Remove unnecessary OnStart/OnStop logic The base service OnStart/OnStop logic is purposefully empty, so there's no need to explicitly call it. And by not defining the OnStop method on Pruner the base service's OnStop (a noop) will automatically be called because of Go's "inheritance" mechanism. Signed-off-by: Thane Thomson <[email protected]> * state: Make pruner belong to Node primarily Move the pruner "ownership" the Node. This doesn't actually have a meaningful effect on the lifetime of the pruner, but it is more of a signal to readers of the code that it should be considered a service in its own right, and the block executor simply has access to it to trigger one method call. Signed-off-by: Thane Thomson <[email protected]> * store: Format comment for readability Signed-off-by: Thane Thomson <[email protected]> * state: Add docstring for PruneABCIResponses Signed-off-by: Thane Thomson <[email protected]> * state: Refactor pruningRoutine to condense logging Signed-off-by: Thane Thomson <[email protected]> * state: Refactor pruningRoutine - extract function for block pruning Signed-off-by: Thane Thomson <[email protected]> * state: Refactor pruningRoutine - extract function for ABCI result pruning Signed-off-by: Thane Thomson <[email protected]> * node: Only save companion retain height on startup if not yet present Expand the check on node creation to only set the companion retain height if 3 conditions are met: 1. The companion retain height has not yet been set previously. 2. The companion is enabled. 3. The configured companion retain height is > 0. Signed-off-by: Thane Thomson <[email protected]> * config: Impose restriction on pruning interval to be > 0 Signed-off-by: Thane Thomson <[email protected]> * test for txindex pruning * node: Fail construction if setting data companion retain height fails Signed-off-by: Thane Thomson <[email protected]> * node: Start pruner on node startup after state sync Signed-off-by: Thane Thomson <[email protected]> * light: Remove sleep from test Signed-off-by: Thane Thomson <[email protected]> * state: Handle error cases where retain height keys are not set Signed-off-by: Thane Thomson <[email protected]> * state: Make Pruner.FindMinRetainHeight private Signed-off-by: Thane Thomson <[email protected]> * state: Improve logging in block executor pruning Signed-off-by: Thane Thomson <[email protected]> * block indexer pruning + test * Format Signed-off-by: Thane Thomson <[email protected]> * state: Export FindMinRetainHeight exclusively for testing Signed-off-by: Thane Thomson <[email protected]> * node: Extract function for initializing companion retain height Signed-off-by: Thane Thomson <[email protected]> * Add changelog entries Signed-off-by: Thane Thomson <[email protected]> * docs: Update configuration-related content Signed-off-by: Thane Thomson <[email protected]> * finilizing * undoing testing setup * Revert "done stateDB writes batching" This reverts commit 78d8d38. * Revert "benchmarking for state db save" This reverts commit f3b757a. * Revert "batching in store" This reverts commit 7ca253f. * Revert "benchmarking for store batching" This reverts commit 2fa2e38. * Revert "add changelog" This reverts commit ec60b4a. * Revert "fixed linter and test" This reverts commit fac927e. * Revert "remove forgotten comments" This reverts commit fae7d59. * fixed go.sum * shut up linter * add changelog * node: Stop pruner when the node stops Signed-off-by: Thane Thomson <[email protected]> * Remove redundunt variable Co-authored-by: Thane Thomson <[email protected]> * remove redundunt search for block heights * rebased to PR 1150 * state: Refactor ABCI response pruning Signed-off-by: Thane Thomson <[email protected]> * state: Add parameter names to Store interface for clarity Signed-off-by: Thane Thomson <[email protected]> * state: Rename helper for clarity Signed-off-by: Thane Thomson <[email protected]> * state: Optimize ABCI responses pruning Avoid iterating through all possible heights when pruning ABCI responses. Signed-off-by: Thane Thomson <[email protected]> * Fixed couting of the last height pruned for abci respnses and blocks * Set application retain height on startup to avoid pruning by the data companion before the application has indicated so * Apply suggestions from code review Co-authored-by: Thane Thomson <[email protected]> * Set app retain height on startup only if it was not set before * Fixed linter * Simplified abci res pruning * fixed tests to support indexerService in pruner * forgotten file * updated test * ADR 101: Recommended updates to pruner service (cometbft#1201) * state: Reference retain heights instead of pruned heights for clarity Signed-off-by: Thane Thomson <[email protected]> * state: Explicitly assign variable values for clarity Signed-off-by: Thane Thomson <[email protected]> * state: Refactor - early returns Signed-off-by: Thane Thomson <[email protected]> * state: Add mutex to Pruner to serialize retain height setting requests Signed-off-by: Thane Thomson <[email protected]> * state: Add convenience getters to Pruner Signed-off-by: Thane Thomson <[email protected]> * state: Export Pruner errors Signed-off-by: Thane Thomson <[email protected]> * state: Rename PrunerInterval option to WithPrunerInterval for clarity Signed-off-by: Thane Thomson <[email protected]> * state: Fix error message references Signed-off-by: Thane Thomson <[email protected]> --------- Signed-off-by: Thane Thomson <[email protected]> Co-authored-by: Jasmina Malicevic <[email protected]> * ADR 101: Add observer for pruner (cometbft#1183) * state: Add observer for Pruner Signed-off-by: Thane Thomson <[email protected]> * state: Wire in pruner observer Signed-off-by: Thane Thomson <[email protected]> * state: Make test more deterministic Signed-off-by: Thane Thomson <[email protected]> * Fix renames Signed-off-by: Thane Thomson <[email protected]> * state: Simplify NoopPrunerObserver usage Signed-off-by: Thane Thomson <[email protected]> * state: Fix order of expected/actual params in assertion Signed-off-by: Thane Thomson <[email protected]> * state: Clarify field meanings Signed-off-by: Thane Thomson <[email protected]> --------- Signed-off-by: Thane Thomson <[email protected]> * persisting last retained height for the block indexer * separate index pruning * adjust go.sum * Separated ABCI result pruning into separate routine Added observer into the pruner tests. * optimizing event search * linter fix * pruning benchmark * fix linter * persisting indexer retain height * indexer last retain height * fix linter * Update state/pruner.go Co-authored-by: Jasmina Malicevic <[email protected]> * remove reduntant dependencies * added separate persisting of block and tx indexer retain heights * ADR-101: implement gRPC `PruningService` (cometbft#1154) * proto: Add proto files for the PruningService * proto: generate *.pb.go files for PruningService * config: add grpc.privileged incl. pruning_service As described in ADR-101, add the node configuration section named grpc.privileged to control the privileged server socket, containing a pruning_service section for the pruning service. * config: add pruning service config to the template * grpc: package for privileged server The privileged server optionally instantiated with the pruning service. * node: add setup for the privileged gRPC server * rpc: enable PruningService in test helper config Also stop the makeAddrs function from growing grotesquely repetitive and replace it with makeAddr returning a single, supposedly random, local address string. * grpc: privileged client with PruningService Add client-side support for the privileged connection that features an optionally enabled pruning service. * e2e tests for PruningService * config: refer to [storage.pruning] section in config.toml Replace potentially confusing text in the comments on the pruning service configuration. Co-authored-by: Thane Thomson <[email protected]> * grpc: tracing with error logs for PruningService --------- Co-authored-by: Andy Nogueira <[email protected]> Co-authored-by: Thane Thomson <[email protected]> * Update state/txindex/indexer_service.go Co-authored-by: Jasmina Malicevic <[email protected]> * Update state/indexer/block/kv/kv.go Co-authored-by: Jasmina Malicevic <[email protected]> * ADR-101: Metrics to monitor the pruning (cometbft#1234) * Metrics to report on the data companion retain height * Added metric to report the application retain height as well * Added metrics to report the blockstore base height and the abci results base height --------- Co-authored-by: Thane Thomson <[email protected]> * using built in bytes function * fix missing import * reducing self-written functions * exporting getKeys for testing * removing indexer service as middle layer between indexers and pruner * separate tx and block indexers * Update state/pruner.go Co-authored-by: Jasmina Malicevic <[email protected]> * Update state/pruner.go Co-authored-by: Jasmina Malicevic <[email protected]> * fix naming * revert go.mod go.sum changes * Update state/pruner.go Co-authored-by: Jasmina Malicevic <[email protected]> * fix ill merge resolution * batch pruning * block indexer: non-nil keyArray * block indexer: remove event keys record optimization * block indexer: remove getEventKeys optimization * periodically flushing pruning batch * revert package renaming, delete unused functions * minor changes * .changelog: add missing method to pruner update * txindexer: change panic to returning an error * remove exposal of GetKeys outside testing --------- Signed-off-by: Thane Thomson <[email protected]> Co-authored-by: Jasmina Malicevic <[email protected]> Co-authored-by: Adi Seredinschi <[email protected]> Co-authored-by: Thane Thomson <[email protected]> Co-authored-by: Mikhail Zabaluev <[email protected]> Co-authored-by: Andy Nogueira <[email protected]> * Fixed bad merge * Fixed changelog structure * Fixed changelog order --------- Signed-off-by: Thane Thomson <[email protected]> Co-authored-by: werty144 <[email protected]> Co-authored-by: Adi Seredinschi <[email protected]> Co-authored-by: Thane Thomson <[email protected]> Co-authored-by: Mikhail Zabaluev <[email protected]> Co-authored-by: Andy Nogueira <[email protected]>
…metbft#1479) * indexer: Implement pruning mechanism (cometbft#1176) * done stateDB writes batching * remove forgotten debug print * benchmarking for state db save * state db save benchmarking * batching in store * benchmarking for store batching * Added pruning mechanism -initial version * removed comments * Rewrore pruning mechanism as service. Working code but tests need to change. * Minor edits * Removed comments * Fixed linter * Fixed linter * Fixed state and block store tests Proper ticker used instead of sleep for pruning service job * Fixed failing http test * fixed linter errors and reduced sleep in http test * Fixed e2e test and failing linter * Applied some of @thanethomson's PR comments * add changelog * Separated retain height functions * fixed linter and test * Fixed code in select min height` * Fixed bugs causing failing tests * Added config flag for companion initial height Fixed failing test due to wrong check on blockstore base height * Tested setting the companion initial height via config. Moved pruner initialization into blockExecutor again. * Added config for pruning frequency * minor * Added additional test for companion retain height setting * script for adjasting test loadtime parameeters * Added simple testing tool for goleveldb * measuring tool + pruning setting * draft of tx_index pruning * using bus to prune tx_index * draft of pruning * working prototipe on tx_index txIndexer pruning * consensus: Fix test broken due to missing state store mocks Signed-off-by: Thane Thomson <[email protected]> * config: Refactor to match updated ADR ADR-101 was updated recently to reflect the desired configuration file changes - this updates the code to match the ADR. It also changes the "frequency" variable to more accurately be called an "interval", since it represents a sleep period between pruning operations as opposed to the number of times it should be run per second. The interval is also defined as a time.Duration now to facilitate more ergonomic usage from the configuration file. Signed-off-by: Thane Thomson <[email protected]> * Format Signed-off-by: Thane Thomson <[email protected]> * state: Rename pruner sleep time to "interval" Signed-off-by: Thane Thomson <[email protected]> * state: Refactor pruner config Make the pruner config private such that its configuration is fixed at runtime. With the previous approach, the PrunerInterval method could be abused to change the interval at runtime since it returns a function that can operate on the public type. Signed-off-by: Thane Thomson <[email protected]> * state: Fix minor nits in function docstrings Signed-off-by: Thane Thomson <[email protected]> * state: Remove unused method Signed-off-by: Thane Thomson <[email protected]> * state: Remove unnecessary OnStart/OnStop logic The base service OnStart/OnStop logic is purposefully empty, so there's no need to explicitly call it. And by not defining the OnStop method on Pruner the base service's OnStop (a noop) will automatically be called because of Go's "inheritance" mechanism. Signed-off-by: Thane Thomson <[email protected]> * state: Make pruner belong to Node primarily Move the pruner "ownership" the Node. This doesn't actually have a meaningful effect on the lifetime of the pruner, but it is more of a signal to readers of the code that it should be considered a service in its own right, and the block executor simply has access to it to trigger one method call. Signed-off-by: Thane Thomson <[email protected]> * store: Format comment for readability Signed-off-by: Thane Thomson <[email protected]> * state: Add docstring for PruneABCIResponses Signed-off-by: Thane Thomson <[email protected]> * state: Refactor pruningRoutine to condense logging Signed-off-by: Thane Thomson <[email protected]> * state: Refactor pruningRoutine - extract function for block pruning Signed-off-by: Thane Thomson <[email protected]> * state: Refactor pruningRoutine - extract function for ABCI result pruning Signed-off-by: Thane Thomson <[email protected]> * node: Only save companion retain height on startup if not yet present Expand the check on node creation to only set the companion retain height if 3 conditions are met: 1. The companion retain height has not yet been set previously. 2. The companion is enabled. 3. The configured companion retain height is > 0. Signed-off-by: Thane Thomson <[email protected]> * config: Impose restriction on pruning interval to be > 0 Signed-off-by: Thane Thomson <[email protected]> * test for txindex pruning * node: Fail construction if setting data companion retain height fails Signed-off-by: Thane Thomson <[email protected]> * node: Start pruner on node startup after state sync Signed-off-by: Thane Thomson <[email protected]> * light: Remove sleep from test Signed-off-by: Thane Thomson <[email protected]> * state: Handle error cases where retain height keys are not set Signed-off-by: Thane Thomson <[email protected]> * state: Make Pruner.FindMinRetainHeight private Signed-off-by: Thane Thomson <[email protected]> * state: Improve logging in block executor pruning Signed-off-by: Thane Thomson <[email protected]> * block indexer pruning + test * Format Signed-off-by: Thane Thomson <[email protected]> * state: Export FindMinRetainHeight exclusively for testing Signed-off-by: Thane Thomson <[email protected]> * node: Extract function for initializing companion retain height Signed-off-by: Thane Thomson <[email protected]> * Add changelog entries Signed-off-by: Thane Thomson <[email protected]> * docs: Update configuration-related content Signed-off-by: Thane Thomson <[email protected]> * finilizing * undoing testing setup * Revert "done stateDB writes batching" This reverts commit 78d8d38. * Revert "benchmarking for state db save" This reverts commit f3b757a. * Revert "batching in store" This reverts commit 7ca253f. * Revert "benchmarking for store batching" This reverts commit 2fa2e38. * Revert "add changelog" This reverts commit ec60b4a. * Revert "fixed linter and test" This reverts commit fac927e. * Revert "remove forgotten comments" This reverts commit fae7d59. * fixed go.sum * shut up linter * add changelog * node: Stop pruner when the node stops Signed-off-by: Thane Thomson <[email protected]> * Remove redundunt variable Co-authored-by: Thane Thomson <[email protected]> * remove redundunt search for block heights * rebased to PR 1150 * state: Refactor ABCI response pruning Signed-off-by: Thane Thomson <[email protected]> * state: Add parameter names to Store interface for clarity Signed-off-by: Thane Thomson <[email protected]> * state: Rename helper for clarity Signed-off-by: Thane Thomson <[email protected]> * state: Optimize ABCI responses pruning Avoid iterating through all possible heights when pruning ABCI responses. Signed-off-by: Thane Thomson <[email protected]> * Fixed couting of the last height pruned for abci respnses and blocks * Set application retain height on startup to avoid pruning by the data companion before the application has indicated so * Apply suggestions from code review Co-authored-by: Thane Thomson <[email protected]> * Set app retain height on startup only if it was not set before * Fixed linter * Simplified abci res pruning * fixed tests to support indexerService in pruner * forgotten file * updated test * ADR 101: Recommended updates to pruner service (cometbft#1201) * state: Reference retain heights instead of pruned heights for clarity Signed-off-by: Thane Thomson <[email protected]> * state: Explicitly assign variable values for clarity Signed-off-by: Thane Thomson <[email protected]> * state: Refactor - early returns Signed-off-by: Thane Thomson <[email protected]> * state: Add mutex to Pruner to serialize retain height setting requests Signed-off-by: Thane Thomson <[email protected]> * state: Add convenience getters to Pruner Signed-off-by: Thane Thomson <[email protected]> * state: Export Pruner errors Signed-off-by: Thane Thomson <[email protected]> * state: Rename PrunerInterval option to WithPrunerInterval for clarity Signed-off-by: Thane Thomson <[email protected]> * state: Fix error message references Signed-off-by: Thane Thomson <[email protected]> --------- Signed-off-by: Thane Thomson <[email protected]> Co-authored-by: Jasmina Malicevic <[email protected]> * ADR 101: Add observer for pruner (cometbft#1183) * state: Add observer for Pruner Signed-off-by: Thane Thomson <[email protected]> * state: Wire in pruner observer Signed-off-by: Thane Thomson <[email protected]> * state: Make test more deterministic Signed-off-by: Thane Thomson <[email protected]> * Fix renames Signed-off-by: Thane Thomson <[email protected]> * state: Simplify NoopPrunerObserver usage Signed-off-by: Thane Thomson <[email protected]> * state: Fix order of expected/actual params in assertion Signed-off-by: Thane Thomson <[email protected]> * state: Clarify field meanings Signed-off-by: Thane Thomson <[email protected]> --------- Signed-off-by: Thane Thomson <[email protected]> * persisting last retained height for the block indexer * separate index pruning * adjust go.sum * Separated ABCI result pruning into separate routine Added observer into the pruner tests. * optimizing event search * linter fix * pruning benchmark * fix linter * persisting indexer retain height * indexer last retain height * fix linter * Update state/pruner.go Co-authored-by: Jasmina Malicevic <[email protected]> * remove reduntant dependencies * added separate persisting of block and tx indexer retain heights * ADR-101: implement gRPC `PruningService` (cometbft#1154) * proto: Add proto files for the PruningService * proto: generate *.pb.go files for PruningService * config: add grpc.privileged incl. pruning_service As described in ADR-101, add the node configuration section named grpc.privileged to control the privileged server socket, containing a pruning_service section for the pruning service. * config: add pruning service config to the template * grpc: package for privileged server The privileged server optionally instantiated with the pruning service. * node: add setup for the privileged gRPC server * rpc: enable PruningService in test helper config Also stop the makeAddrs function from growing grotesquely repetitive and replace it with makeAddr returning a single, supposedly random, local address string. * grpc: privileged client with PruningService Add client-side support for the privileged connection that features an optionally enabled pruning service. * e2e tests for PruningService * config: refer to [storage.pruning] section in config.toml Replace potentially confusing text in the comments on the pruning service configuration. Co-authored-by: Thane Thomson <[email protected]> * grpc: tracing with error logs for PruningService --------- Co-authored-by: Andy Nogueira <[email protected]> Co-authored-by: Thane Thomson <[email protected]> * Update state/txindex/indexer_service.go Co-authored-by: Jasmina Malicevic <[email protected]> * Update state/indexer/block/kv/kv.go Co-authored-by: Jasmina Malicevic <[email protected]> * ADR-101: Metrics to monitor the pruning (cometbft#1234) * Metrics to report on the data companion retain height * Added metric to report the application retain height as well * Added metrics to report the blockstore base height and the abci results base height --------- Co-authored-by: Thane Thomson <[email protected]> * using built in bytes function * fix missing import * reducing self-written functions * exporting getKeys for testing * removing indexer service as middle layer between indexers and pruner * separate tx and block indexers * Update state/pruner.go Co-authored-by: Jasmina Malicevic <[email protected]> * Update state/pruner.go Co-authored-by: Jasmina Malicevic <[email protected]> * fix naming * revert go.mod go.sum changes * Update state/pruner.go Co-authored-by: Jasmina Malicevic <[email protected]> * fix ill merge resolution * batch pruning * block indexer: non-nil keyArray * block indexer: remove event keys record optimization * block indexer: remove getEventKeys optimization * periodically flushing pruning batch * revert package renaming, delete unused functions * minor changes * .changelog: add missing method to pruner update * txindexer: change panic to returning an error * remove exposal of GetKeys outside testing --------- Signed-off-by: Thane Thomson <[email protected]> Co-authored-by: Jasmina Malicevic <[email protected]> Co-authored-by: Adi Seredinschi <[email protected]> Co-authored-by: Thane Thomson <[email protected]> Co-authored-by: Mikhail Zabaluev <[email protected]> Co-authored-by: Andy Nogueira <[email protected]> * Fixed bad merge * Fixed changelog structure * TxIndexer and BlockIndexer pruning metrics (cometbft#1334) * added tx and block indexer pruning metrics * add changelog * added base height metric for tx and block indexer * Fixed changelog order --------- Signed-off-by: Thane Thomson <[email protected]> Co-authored-by: werty144 <[email protected]> Co-authored-by: Adi Seredinschi <[email protected]> Co-authored-by: Thane Thomson <[email protected]> Co-authored-by: Mikhail Zabaluev <[email protected]> Co-authored-by: Andy Nogueira <[email protected]>
Co-authored-by: Sergio Mena <[email protected]>
…tbft into jasmina/pruning0.38
…_0.38 feat: Pruning and compaction port to 0.38
Author
|
Changes contained in #23 |
|
This pull request has been automatically marked as stale because it has not had |
Co-authored-by: Sergio Mena <[email protected]>
…tain height - one can prune the blockstore and not prune the indexer store
…ina/indexer_pruning0.38
|
marcello33
approved these changes
Aug 25, 2025
5 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR is based on #23 and contains the changes from there as well. It should not be merged before #23 is reviewed and merged.
It adds methods to prune the block and tx indexer in comet.