Skip to content

introduce new archive index format based on SQLite #2068

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 11 commits into from
May 14, 2023
Merged

introduce new archive index format based on SQLite #2068

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
e3e7c46
introduce new archive index format based on SQLite
syphar Mar 3, 2023
708c77f
set SQLITE synchronous mode when writing
syphar Mar 3, 2023
71af45f
run `VACUUM` after creating archive index SQLite db
syphar Mar 7, 2023
cec41ce
migrate to a real connection pool for the sqlite connections
syphar Mar 11, 2023
ff78fcb
fix compile
syphar Mar 11, 2023
49bf28a
ignore code block as doctest
syphar Mar 11, 2023
b694df6
add more documentation for limits on sqlite connection pool
syphar Apr 16, 2023
ce868cc
first test for pool
syphar Apr 16, 2023
3b6d40b
don't clone string
syphar May 12, 2023
d57889f
replace panic with error when decoding compression from archive index
syphar May 12, 2023
82de382
replace ```ignore with ```text for to prevent execution as doc-test
syphar May 12, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
168 changes: 168 additions & 0 deletions Cargo.lock
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

3 changes: 3 additions & 0 deletions Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,7 @@ semver = { version = "1.0.4", features = ["serde"] }
slug = "0.1.1"
r2d2 = "0.8"
r2d2_postgres = "0.18"
r2d2_sqlite = "0.21.0"
url = { version = "2.1.1", features = ["serde"] }
docsrs-metadata = { path = "crates/metadata" }
anyhow = { version = "1.0.42", features = ["backtrace"]}
Expand Down Expand Up @@ -69,6 +70,8 @@ bzip2 = "0.4.4"
serde_cbor = "0.11.1"
getrandom = "0.2.1"
itertools = { version = "0.10.5", optional = true}
rusqlite = { version = "0.28.0", features = ["bundled"] }
moka = { version ="0.10.0", default-features = false, features = ["sync"]}

# Async
tokio = { version = "1.0", features = ["rt-multi-thread", "signal", "macros"] }
Expand Down
4 changes: 4 additions & 0 deletions src/config.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,9 @@ pub struct Config {
pub(crate) max_pool_size: u32,
pub(crate) min_pool_idle: u32,

// local pool for sqlite connections
pub(crate) max_sqlite_pool_size: u64,

// Storage params
pub(crate) storage_backend: StorageKind,

Expand Down Expand Up @@ -136,6 +139,7 @@ impl Config {

database_url: require_env("DOCSRS_DATABASE_URL")?,
max_pool_size: env("DOCSRS_MAX_POOL_SIZE", 90)?,
max_sqlite_pool_size: env("DOCSRS_MAX_SQLITE_POOL_SIZE", 500)?,
min_pool_idle: env("DOCSRS_MIN_POOL_IDLE", 10)?,

storage_backend: env("DOCSRS_STORAGE_BACKEND", StorageKind::Database)?,
Expand Down
Loading