You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: readme.md
+51-22Lines changed: 51 additions & 22 deletions
Original file line number
Diff line number
Diff line change
@@ -1,6 +1,7 @@
1
1
## Memory-Collection
2
2
3
-
A collection of data structures in motoko that store their data in stable memory. These data structures address the heap storage limitations by allowing developers to leverage the 400GB capacity available in stable memory.
3
+
A collection of data structures in motoko that store their data in stable memory.
4
+
These data structures address the heap storage limitations by allowing developers to leverage the 400GB capacity available in stable memory.
4
5
5
6
### Motivation
6
7
@@ -10,28 +11,51 @@ The heap memory in the Internet Computer is limited to 4GB, which can be a bottl
10
11
11
12
-[MemoryBuffer](./src/MemoryBuffer/readme.md): A persistent buffer with `O(1)` random access.
12
13
-[MemoryBTree](./src/MemoryBTree/readme.md): A persistent B-Tree with `O(log n)` search, insertion, and deletion.
13
-
-[MemoryBTreeSet](./src/MemoryBTreeSet/readme.md): A persistent set with `O(log n)` search, insertion, and deletion.
14
-
> Note: `MemoryBTreeSet` does not have a Stable module version
14
+
- MemoryBTreeSet: A persistent set with `O(log n)` search, insertion, and deletion.
15
15
-[MemoryQueue](./src/MemoryQueue/readme.md): A persistent queue with `O(1)` add and pop operations.
16
16
17
-
Each data structure is implemented using the class+ pattern, which creates a mutable stable store that is wrapped around a class object to provide an easy to use interface. This method allows the data persists accross canister upgrades while also being easy to use.
17
+
Each data structure is implemented using the class+ pattern, which creates a mutable stable store that is wrapped around by a class object for a more familiar object oriented interface. This method allows the data to persists accross canister upgrades while also being simple and easy to use.
18
18
The data structures can be imported using the default path to the library `mo:memory-collection/<data-structure>`.
19
19
Each data structure also has a Stable module that can be used to call function on the stable store directly without wrapping it in a class object. The stable pattern can be accessed using the path `mo:memory-collection/<data-structure>/Stable`.
20
20
21
-
### Utilities
21
+
### TypeUtils
22
22
23
-
The data structures require additional utilities to serialize, deserialize, and compare the data.
24
-
These utilities are needed in order to store, retrieve and search for data in stable memory.
25
-
We have provided these utilities as a set of modules that can be imported and used in your project.
26
-
These utilities are:
23
+
For each data structure we have to define a set of functions during initialization for processing the data structure's given data type so that it can be stored in stable memory.
24
+
To help with this we have provided a set of modules that can be imported and used in your project to help define those functions.
27
25
28
-
-[TypeUtils](./src/TypeUtils/readme.md): A module that provides utilities for working with types.
29
-
-[Blobify](./src/Blobify/readme.md): A module that provides functions for serializing and deserializing primitive motoko types.
30
-
-[MemoryCmp](./src/MemoryCmp/readme.md): A module that provides functions for comparing elements.
26
+
**Main Utility Module**
31
27
32
-
The main module is the `TypeUtils` module, and the other two submodules `Blobify` and `MemoryCmp` are contained within it.
33
-
`TypeUtils` provides utilities for most of motoko's primitive types (e.g. Nat, Int, Text, etc.) and exposes a record containing the submodules when a type is selected.
34
-
This structure allows the user to easily select from one of the preset options or define their own custom utilities for their required use case. More information on how to create custom utilities can be found in the **Create Custom Utilities** section of each data structure's readme.
28
+
-[TypeUtils](./src/TypeUtils/lib.mo)
29
+
30
+
```motoko
31
+
public type TypeUtils<T> = {
32
+
blobify : Blobify<T>;
33
+
cmp: MemoryCmp<T>;
34
+
}
35
+
```
36
+
37
+
**Sub modules within TypeUtils**
38
+
39
+
-[Blobify](./src/TypeUtils/Blobify.mo): A module that provides functions for converting the given data type to a `Blob` and back using the given interface.
40
+
41
+
```motoko
42
+
public type Blobify<T> = {
43
+
to_blob: (T) -> Blob;
44
+
from_blob: (Blob) -> T;
45
+
}
46
+
```
47
+
48
+
-[MemoryCmp](./src/TypeUtils/MemoryCmp.mo): A module that provides functions for comparing two elements of the same type and retrieving their order. The comparison function can either be a `#BlobCmp` which compares the serialized version of the types, or it could be a `#GenCmp` which compares the types in their given data type, which often involves deserializing the stored `Blob` before comparing it.
49
+
50
+
```motoko
51
+
public type MemoryCmp<T> = {
52
+
#GenCmp: (T, T) -> Int8;
53
+
#BlobCmp: (Blob, Blob) -> Int8;
54
+
}
55
+
```
56
+
57
+
`TypeUtils` are provided for most of motoko's primitive types (e.g. Nat, Int, Text, etc.). However, you can define a custom function for a compound datatype using the interface above.
58
+
More information on how to create custom type utilities can be found in the **Create Custom TypeUtils** section of each data structure's readme.
35
59
36
60
## Getting Started
37
61
@@ -50,6 +74,8 @@ This structure allows the user to easily select from one of the preset options o
50
74
51
75
### Usage Examples
52
76
77
+
Usage examples using the preset `TypeUtils`
78
+
53
79
- MemoryBuffer
54
80
55
81
```motoko
@@ -109,16 +135,20 @@ This structure allows the user to easily select from one of the preset options o
109
135
110
136
### Migrating between mops versions
111
137
112
-
The data stored in stable memory is stored in a way that it is unlikely to cause breaking changes between different `mops` versions.
113
-
However, some of the data used by the data structure is stored as heap memory. These include cached fields, like the size, head or tail pointers, and others. The most important of the data stored on the heap is the deallocated memory stored in each [MemoryRegion](https://github.com/NatLabs/memory-region) because it is not backed up in stable memory. For efficient utilization of memory it's important to not lose access to this data. Furthermore, the deallocated memory blocks are stored on the heap for a number of reasons. The first one being accessing data on the heap is faster than stable memory which makes the data structure more efficient since this is a module that is used frequently in all of them.
138
+
The data structures in the memory-collection are designed with backward compatibility in mind, particularly for data stored in stable memory. This approach helps prevent breaking changes between package updates. To support this compatibility, additional checks are implemented, and space is reserved in each data structure to accommodate potential future additions to stable memory.
114
139
115
-
In future, the `MemoryRegion` may have updates improving its performance for reallocating memory. In doing so the structure of the `MemoryRegion` may change. So in order to keep it
140
+
While stable memory ensures data persistence, certain information is stored on the heap for faster access. This includes cached fields from stable memory and memory marked as deallocated in each MemoryRegion. Unlike stable memory where all data is formatted as `Blobs`, heap data utilizes various Motoko data types. This diversity in data types on the heap presents a challenge for compatibility, as adding new data types or modifying existing ones can potentially break compatibility between versions.
116
141
117
-
### Serialization Notes
142
+
Future updates to this library may introduce new fields to cached data or modify the MemoryRegion to enhance performance. These changes could potentially alter the internal structure of the data, particularly on the heap where data types vary. To prevent data loss during such changes, each `StableStore` (the value returned after a call to `.newStableStore()`) is wrapped with the current version identifier. This versioning helps distinguish between different versions and facilitates the migration when the package is updated.
118
143
119
-
> should be moved to the individual data structure readme
144
+
To upgrade your StableStore, use the following code:
145
+
146
+
```motoko
147
+
stable var sstore = MemoryBTree.newStableStore(null);
148
+
sstore := MemoryBTree.upgrade(sstore);
149
+
```
120
150
121
-
- Serialization and deserialization using candid is often much more performant than using Blobify or any other serialization methods. The reason for this is because the to_candid and from_candid functions are system functions in the IC and therefore more efficient than any custom serialization methods. However, the candid contains extra type information included in the serialized data, which can make the serialized data larger than using Blobify. This difference in size can be significant if each of the pieces of data being serialized is small. For example a serialized value of Nat8 value if serialized with Blobify will be 1 bytes, but if serialized into candid will include the magic number (4 bytes), the type (1 byte) and the value (1 byte) for a total of 6 bytes. This difference in size can be significant if each of the pieces of data being serialized is small. However, if the data being serialized is large, the overhead of the extra bytes is negligible. So be mindful of the size of the data being serialized when choosing between Blobify and candid.
151
+
You can call `.upgrade()` either immediately after defining the stable store variable or within the `preupgrade()` system function. It's important to note that when `preupgrade()` is set or modified, the defined logic is executed on the next update to the canister, not the current one.
122
152
123
153
### Notes and Limitations
124
154
@@ -127,4 +157,3 @@ In future, the `MemoryRegion` may have updates improving its performance for rea
127
157
- Each value stored in the data structure is immutable. To update a value, the data is removed and the new value is serialized and stored in its place or in a new location.
128
158
- Currently, stable memory is not garbage collected. Which means that we can't free up a section of memory when it's not longer in use once we have allocated it. We can only mark it as deallocated and re-use it once we need to store more data. So even after one of our data structures is no longer in use and all of its data has been cleared, the `Region` and the memory it allocated will still be reserved.
129
159
- Ideally, serializing compound types can be done easily enough using a predefined serialization function for candid, but since motoko's `to_candid()` and `from_candid()` don't yet support generic types, each type need to have its own serialization utility defined.
130
-
<!-- - We have provided Utilities for comparing and serializing primitive types. However, you will need to define your own custom functions for compound or complex types. This goes for candid as well since it does not yet support generic types you would need to define serialization functions for each of your types. -->
0 commit comments