-
Notifications
You must be signed in to change notification settings - Fork 80
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[v0.4] Move lasso SQL cache in Steve (#473)
* Copy pkg/cache/sql from lasso to pkg/sqlcache * Rename import from github.com/rancher/lasso/pkg/cache/sql to github.com/rancher/steve/pkg/sqlcache * go mod tidy * Fix lint errors * Remove lasso SQL cache mentions * Fix more CI lint errors * fix goimports Signed-off-by: Silvio Moioli <[email protected]> * Fix imports * Fix more linting errors --------- Signed-off-by: Silvio Moioli <[email protected]> Co-authored-by: Silvio Moioli <[email protected]>
- Loading branch information
Showing
53 changed files
with
9,967 additions
and
31 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,157 @@ | ||
| # SQL Cache | ||
|
|
||
| ## Sections | ||
| - [ListOptions Informer](#listoptions-informer) | ||
| - [List Options](#list-options) | ||
| - [ListOption Indexer](#listoptions-indexer) | ||
| - [SQL Store](#sql-store) | ||
| - [Partitions](#partitions) | ||
| - [How to Use](#how-to-use) | ||
| - [Technical Information](#technical-information) | ||
| - [SQL Tables](#sql-tables) | ||
| - [SQLite Driver](#sqlite-driver) | ||
| - [Connection Pooling](#connection-pooling) | ||
| - [Encryption Defaults](#encryption-defaults) | ||
| - [Indexed Fields](#indexed-fields) | ||
| - [ListOptions Behavior](#listoptions-behavior) | ||
| - [Troubleshooting Sqlite](#troubleshooting-sqlite) | ||
|
|
||
|
|
||
|
|
||
| ## ListOptions Informer | ||
| The main usable feature from the SQL cache is the ListOptions Informer. The ListOptionsInformer provides listing functionality, | ||
| like any other informer, but with a wider array of options. The options are configured by informer.ListOptions. | ||
|
|
||
| ### List Options | ||
| ListOptions includes the following: | ||
| * Match filters for indexed fields. Filters are for specifying the value a given field in an object should be in order to | ||
| be included in the list. Filters can be set to equals or not equals. Filters can be set to look for partial matches or | ||
| exact (strict) matches. Filters can be OR'd and AND'd with one another. Filters only work on fields that have been indexed. | ||
| * Primary field and secondary field sorting order. Can choose up to two fields to sort on. Sort order can be ascending | ||
| or descending. Default sorting is to sort on metadata.namespace in ascending first and then sort on metadata.name. | ||
| * Page size to specify how many items to include in a response. | ||
| * Page number to specify offset. For example, a page size of 50 and a page number of 2, will return items starting at | ||
| index 50. Index will be dependent on sort. Page numbers start at 1. | ||
|
|
||
| ### ListOptions Factory | ||
| The ListOptions Factory helps manage multiple ListOption Informers. A user can call Factory.InformerFor(), to create new | ||
| ListOptions informers if they do not exist and retrieve existing ones. | ||
|
|
||
| ### ListOptions Indexer | ||
| Like all other informers, the ListOptions informer uses an indexer to cache objects of the informer's type. A few features | ||
| set the ListOptions Indexer apart from others indexers: | ||
| * an on-disk store instead of an in-memory store. | ||
| * accepts list options backed by SQL queries for extended search/filter/sorting capability. | ||
| * AES GCM encryption using key hierarchy. | ||
|
|
||
| ### SQL Store | ||
| The SQL store is the main interface for interacting with the database. This store backs the indexer, and provides all | ||
| functionality required by the cache.Store interface. | ||
|
|
||
| ### Partitions | ||
| Partitions are constraints for ListOptionsInform ListByOptions() method that are separate from ListOptions. Partitions | ||
| are strict conditions that dictate which namespaces or names can be searched from. These overrule ListOptions and are | ||
| intended to be used as a way of enforcing RBAC. | ||
|
|
||
| ## How to Use | ||
| ```go | ||
| package main | ||
| import( | ||
| "k8s.io/client-go/dynamic" | ||
| "github.com/rancher/steve/pkg/sqlcache/informer" | ||
| "github.com/rancher/steve/pkg/sqlcache/informer/factory" | ||
| ) | ||
|
|
||
| func main() { | ||
| cacheFactory, err := factory.NewCacheFactory() | ||
| if err != nil { | ||
| panic(err) | ||
| } | ||
| // config should be some rest config created from kubeconfig | ||
| // there are other ways to create a config and any client that conforms to k8s.io/client-go/dynamic.ResourceInterface | ||
| // will work. | ||
| client, err := dynamic.NewForConfig(config) | ||
| if err != nil { | ||
| panic(err) | ||
| } | ||
|
|
||
| fields := [][]string{{"metadata", "name"}, {"metadata", "namespace"}} | ||
| opts := &informer.ListOptions{} | ||
| // gvk should be of type k8s.io/apimachinery/pkg/runtime/schema.GroupVersionKind | ||
| c, err := cacheFactory.CacheFor(fields, client, gvk) | ||
| if err != nil { | ||
| panic(err) | ||
| } | ||
|
|
||
| // continueToken will just be an offset that can be used in Resume on a subsequent request to continue | ||
| // to next page | ||
| list, continueToken, err := c.ListByOptions(apiOp.Context(), opts, partitions, namespace) | ||
| if err != nil { | ||
| panic(err) | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| ## Technical Information | ||
|
|
||
| ### SQL Tables | ||
| There are three tables that are created for the ListOption informer: | ||
| * object table - this contains objects, including all their fields, as blobs. These blobs may be encrypted. | ||
| * fields table - this contains specific fields of value for objects. These are specified on informer create and are fields | ||
| that it is desired to filter or order on. | ||
| * indices table - the indices table stores indexes created and objects' values for each index. This backs the generic indexer | ||
| that contains the functionality needed to conform to cache.Indexer. | ||
|
|
||
| ### SQLite Driver | ||
| There are multiple SQLite drivers that this package could have used. One of the most, if not the most, popular SQLite golang | ||
| drivers is [mattn/go-sqlite3](https://github.com/mattn/go-sqlite3). This driver is not being used because it requires enabling | ||
| the cgo option when compiling and at the moment steve's main consumer, rancher, does not compile with cgo. We did not want | ||
| the SQL informer to be the sole driver in switching to using cgo. Instead, modernc's driver which is in pure golang. Side-by-side | ||
| comparisons can be found indicating the cgo version is, as expected, more performant. If in the future it is deemed worthwhile | ||
| then the driver can be easily switched by replacing the empty import in `pkg/cache/sql/store` from `_ "modernc.org/sqlite"` to `_ "github.com/mattn/go-sqlite3"`. | ||
|
|
||
| ### Connection Pooling | ||
| While working with the `database/sql` package for go, it is important to understand how sql.Open() and other methods manage | ||
| connections. Open starts a connection pool; that is to say after calling open once, there may be anywhere from zero to many | ||
| connections attached to a sql.Connection. `database/sql` manages this connection pool under the hood. In most cases, an | ||
| application only need one sql.Connection, although sometimes application use two: one for writes, the other for reads. To | ||
| read more about the `sql` package's connection pooling read [Managing connections](https://go.dev/doc/database/manage-connections). | ||
|
|
||
| The use of connection pooling and the fact that steve potentially has many go routines accessing the same connection pool, | ||
| means we have to be careful with writes. Exclusively using sql transaction to write helps ensure safety. To read more about | ||
| sql transactions read SQLite's [Transaction docs](https://www.sqlite.org/lang_transaction.html). | ||
|
|
||
| ### Encryption Defaults | ||
| By default only specified types are encrypted. These types are hard-coded and defined by defaultEncryptedResourceTypes | ||
| in `pkg/cache/sql/informer/factory/informer_factory.go`. To enabled encryption for all types, set the ENV variable | ||
| `CATTLE_ENCRYPT_CACHE_ALL` to "true". | ||
|
|
||
| The key size used is 256 bits. Data-encryption-keys are stored in the object table and are rotated every 150,000 writes. | ||
|
|
||
| ### Indexed Fields | ||
| Filtering and sorting only work on indexed fields. These fields are defined when using `CacheFor`. Objects will | ||
| have the following indexes by default: | ||
| * Fields in informer.defaultIndexedFields | ||
| * Fields passed to InformerFor() | ||
|
|
||
| ### ListOptions Behavior | ||
| Defaults: | ||
| * Sort.PrimaryField: `metadata.namespace` | ||
| * Sort.SecondaryField: `metadata.name` | ||
| * Sort.PrimaryOrder: `ASC` (ascending) | ||
| * Sort.SecondaryOrder: `ASC` (ascending) | ||
| * All filters have partial matching set to false by default | ||
|
|
||
| There are some uncommon ways someone could use ListOptions where it would be difficult to predict what the result would be. | ||
| Below is a non-exhaustive list of some of these cases and what the behavior is: | ||
| * Setting Pagination.Page but not Pagination.PageSize will cause Page to be ignored | ||
| * Setting Sort.SecondaryField only will sort as though it was Sort.PrimaryField. Sort.SecondaryOrder will still be applied | ||
| and Sort.PrimaryOrder will be ignored | ||
|
|
||
| ### Writing Secure Queries | ||
| Values should be supplied to SQL queries using placeholders, read [Avoiding SQL Injection Risk](https://go.dev/doc/database/sql-injection). Any other portions | ||
| of a query that may be user supplied, such as columns, should be carefully validated against a fixed set of acceptable values. | ||
|
|
||
| ### Troubleshooting SQLite | ||
| A useful tool for troubleshooting the database files is the sqlite command line tool. Another useful tool is the goland | ||
| sqlite plugin. Both of these tools can be used with the database files. |
Oops, something went wrong.