[Backport v4.0.99-ncs1-branch] Settings ZMS backend optimizations #2758
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ction" This reverts commit 14d166f. Signed-off-by: Riadh Ghaddab <[email protected]> (cherry picked from commit 9f25d00)
…emory Storage)" This reverts commit 88e9f91. Signed-off-by: Riadh Ghaddab <[email protected]> (cherry picked from commit 81c27fc)
…orage) This adds the initial backend support for the ZMS storage system. Signed-off-by: Riadh Ghaddab <[email protected]> (cherry picked from commit ef4e8dd) (cherry picked from commit 8220d82)
This adds a on target performance test for Settings SS. Using this test performance of the Setting SS + NVS/ZMS backend can be benchmarked. The test repeatedly write 128 settings entries. Each setting entry has a size of 4 bytes, and path length of 16 bytes (excluding the null-terminator). The test has two variants, with and without Bluetooth scan running. This is useful to benchmark performance along with some component of BT subsystem activated. The test could be enhanced in future to include or create combinations of different functionalities running, when agreesive store operations are happening. Signed-off-by: Omkar Kulkarni <[email protected]> (cherry picked from commit 97166a5) (cherry picked from commit 899c993)
This resolves some addressed comments in this PR zephyrproject-rtos#77930 as well as this PR zephyrproject-rtos#80407 Signed-off-by: Riadh Ghaddab <[email protected]> (cherry picked from commit 5f7cda5) (cherry picked from commit 6981397)
…tive_posix native_posix is deprecated and will be removed in 4.2 Run this sample in native_sim instead. Signed-off-by: Alberto Escolar Piedras <[email protected]> (cherry picked from commit 4a91a55) (cherry picked from commit 6508f27)
This sample could not run on any target as it depedend on zms which is set by no board. But it seems to run just fine in both allowed platforms, so let's add a trivial twister check (so it actually passes instead of waiting for ever), and remove that `depends on`. Signed-off-by: Alberto Escolar Piedras <[email protected]> (cherry picked from commit cc2d31f) (cherry picked from commit fb6eec2)
- adds ability to change the ZMS sample MAX_ITERATIONS and DELETE_ITERATION parameters via Kconfig without manually modifying the source code. - adds ability to reduce flash memory wear on real devices by reducing the number of write iterations. Signed-off-by: Andrej Butok <[email protected]> (cherry picked from commit c887d5e) (cherry picked from commit a09763e)
- fix unwanted "Error: Something went wrong at iteration 0". - the loop_cnt check has no sense for iteration 0, comparing with wrong -1. Signed-off-by: Andrej Butok <[email protected]> (cherry picked from commit 2766c78) (cherry picked from commit 6a46770)
To avoid collisions between IDs used by settings and IDs used directly by subsystems using ZMS API, the MSB is always set to 1 for Setting's name ID written to ZMS backend Add as well a recovery path if the hash linked list is broken. Signed-off-by: Riadh Ghaddab <[email protected]> (cherry picked from commit 333fadd) (cherry picked from commit 924bad3)
Add reference to the new settings backend ZMS and add more details about how it works. Signed-off-by: Riadh Ghaddab <[email protected]> (cherry picked from commit f363015) (cherry picked from commit 399aa6b)
- Allows to build and run ZMS tests on real platforms. - Enables ZMS tests designed for a flash-simulator only when built for qemu_x86 or native_sim targets. Signed-off-by: Andrej Butok <[email protected]> (cherry picked from commit b826d46) (cherry picked from commit 6135a05)
- Sets CONFIG_LOG_BLOCK_IN_THREAD for the zms sample. - Fixes "--- 9999 messages dropped ---". Signed-off-by: Andrej Butok <[email protected]> (cherry picked from commit ac7790e) (cherry picked from commit aa9ccd8)
When deleting a settings entry the linked list is updated by removing the deleted node. This operation is time consuming and add some delays. For applications that use almost the same set of Setting entries, add an option to make this operation faster at a cost of having some empty nodes in the linked list. These empty nodes can be used later when the deleted entry is written again. Each empty node occupies 16B of storage. Upstream PR #: 87792 Signed-off-by: Riadh Ghaddab <[email protected]> (cherry picked from commit 02ab54e077e090e4c9364e4d4954a8c3508e513e) (cherry picked from commit cc0a50a)
When the CONFIG_ZMS_NO_DOUBLE_WRITE is not enabled there is no need to search in the cache for matching ID Upstream PR #: 87792 Signed-off-by: Riadh Ghaddab <[email protected]> (cherry picked from commit 546ed2dbe817794bedb16bae0c81fd4af22675ea) (cherry picked from commit e846a49)
Settings subsystem is storing the name ID and the linked list node ID with only one bit difference at BIT(0). Settings subsystem is also storing the name ID and the data ID in two different ZMS entries at an exact offset of ZMS_DATA_ID_OFFSET. Using the regular lookup function could result in many cache misses. Therefore, to assure the least number of collisions in the lookup cache, the BIT(0) of the hash indicates whether the given ZMS ID represents a linked list entry or not, the BIT(1) indicates whether the ZMS ID is a name or data and the remaining bits of the hash are set to a truncated part of the original hash generated by Settings. Upstream PR #: 87792 Signed-off-by: Riadh Ghaddab <[email protected]> (cherry picked from commit ad2581fede5f6efa570e3e8157ff37290d2dbe21) (cherry picked from commit 4bb33ba)
If the subtree argument is not NULL in the settings_load function, load only the setting entry that corresponds to that subtree. If the subtree argument is NULL or it is not found in the storage, load all the existing entries in the storage. Upstream PR #: 87792 Signed-off-by: Riadh Ghaddab <[email protected]> (cherry picked from commit 86132feb5c59e8a071d9782c5482ab346cfbf6f3) (cherry picked from commit df20b61)
Increase the load performance by adding an optional cache for the linked list hashes. This is used only when the settings_load is called with NULL parameter and we need to load all Settings that exist in the persistent storage. Cache is enabled using SETTINGS_ZMS_LL_CACHE and the size of the cache is set using SETTINGS_ZMS_LL_CACHE_SIZE. Each cache entry will add 8 bytes of RAM usage. Upstream PR #: 87792 Signed-off-by: Riadh Ghaddab <[email protected]> (cherry picked from commit 50c099243557a49b53d793c839299e0a55370f42) (cherry picked from commit 54bbd0a)
…s entry Add a a function settings_load_one that allows to load only one settings entry instead of a complete subtree. Add as well its implementation for ZMS backend. This will allow a faster return if the Settings entry doesn't exist in the persistent storage. Upstream PR #: 87792 Signed-off-by: Riadh Ghaddab <[email protected]> (cherry picked from commit 98e544b4b838cd96012bb4bd5adfb584e67a52f1) (cherry picked from commit 9d10cea)
…ad_one Add a test for the new API settings_load_one that loads only one path from the persistent storage. Upstream PR #: 87792 Signed-off-by: Riadh Ghaddab <[email protected]> (cherry picked from commit 168ae245bdce756fd5db22dd391469ad17eeb9b2) (cherry picked from commit 47ee36a)
Add the functional test for the new backend of ZMS Upstream PR #: 87792 Signed-off-by: Riadh Ghaddab <[email protected]> (cherry picked from commit 18f7736ce3c73f6f10e037aa627293104dc55b9d) (cherry picked from commit 56388fc)
When a power off happens after writing the settings name and before writing the linked list node we cannot write the settings name again in the future. Fix this by writing the linked list node before writing the name. When loading all settings, we already delete linked list node that do not have any name or value written. Adds as well a recover path if a power down happens in the middle of unlinking an LL node after a delete. Upstream PR #: 87792 Signed-off-by: Riadh Ghaddab <[email protected]> (cherry picked from commit bf4d15316b1a3bd9239ddbe179b11006eea1e3b1) (cherry picked from commit d9daa11)
When the linked list is broken, recover it instead of reinitializing it. Upstream PR #: 87792 Signed-off-by: Riadh Ghaddab <[email protected]> (cherry picked from commit 057c1d65fa5822d9145330c87183d0b652707ab2) (cherry picked from commit 4a3c56c)
Add a function to get the value's length of a Key. If it doesn't exist returns 0. Add ZMS implementation for csi_get_val_len() and a default implementation for the other storage systems. Add some functional tests to verify it. Upstream PR #: 87792 Signed-off-by: Riadh Ghaddab <[email protected]> (cherry picked from commit 5a730f71f49e87fb4da35120826e8d45bc62fef6) (cherry picked from commit 2b82fc1)
…f strlen if the provided name in argument is not null this could lead to un undefined behavior. Use strnlen to make this safe Upstream PR #: 87792 Signed-off-by: Riadh Ghaddab <[email protected]> (cherry picked from commit e9ac2ef95bd34a8c4756e46e8f2ddf66e0964dbc) (cherry picked from commit 4efae76)
Add references to the new API functions that were added "csi_load_one" and "csi_get_val_len" Upstream PR #: 87792 Signed-off-by: Riadh Ghaddab <[email protected]> (cherry picked from commit b26ed3a3ec5dae5017ecd4e40164e3b38ca88235) (cherry picked from commit 85ea893)
Clean some parts of the code and refactor it to avoid multiple nested conditions. Upstream PR #: 87792 Signed-off-by: Riadh Ghaddab <[email protected]> (cherry picked from commit 1699785ff3fd7d7e260133898729afff634eea73) (cherry picked from commit 749f6b5)
|
nordicjm
approved these changes
Apr 14, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Backport 749f6b5~27..749f6b5 from #2615.