Skip to content

readme.md

Latest commit

 

History

History
146 lines (126 loc) · 6.13 KB

readme.md

File metadata and controls

146 lines (126 loc) · 6.13 KB

Crasher

Tool for crashing local Qdrant instances for testing purposes.

Status

Highly experimental, use at your own risk 🔥

Usage

./crasher -h
Tool for crashing Qdrant instances

Usage: crasher [OPTIONS] --working-dir <WORKING_DIR> --exec-path <EXEC_PATH>

Options:
      --working-dir <WORKING_DIR>
          Working directory for Qdrant data
      --storage-backup <STORAGE_BACKUP>
          Backup `storage` directory from `working_dir` between Qdrant restarts (useful to debug storage recovery issues)
      --exec-path <EXEC_PATH>
          Path to executable binary relative to `working_dir`
      --crash-probability <CRASH_PROBABILITY>
          Probability to kill running instance [default: 0.3]
      --sleep-duration-between-crash-sec <SLEEP_DURATION_BETWEEN_CRASH_SEC>
          The time in second to sleep between crashes attempt [default: 5]
      --uris <URIS>
          Qdrant gRPC service URIs (can be used several times to specify several URIs) [default: http://localhost:6334]
      --points-count <POINTS_COUNT>
          Number of points to generate [default: 5000]
      --vector-dimension <VECTOR_DIMENSION>
          Dimension of generated vectors [default: 10]
      --flush-interval-sec <FLUSH_INTERVAL_SEC>
          Configure the flush interval for collections [default: 5]
      --segment-count <SEGMENT_COUNT>
          Configure the number of segment [default: 2]
      --shard-count <SHARD_COUNT>
          Configure the number of shards [default: 3]
      --replication-factor <REPLICATION_FACTOR>
          Replication factor for collections [default: 1]
      --write-consistency-factor <WRITE_CONSISTENCY_FACTOR>
          Writing consistency factor for collections [default: 1]
      --indexing-threshold <INDEXING_THRESHOLD>
          Optimizer indexing threshold
      --memmap-threshold <MEMMAP_THRESHOLD>
          Maximum size (in `KiloBytes`) of vectors to store in-memory per segment
      --prevent-unindexed
          Whether to prevent the creation of large unindexed segments
      --grpc-timeout-ms <GRPC_TIMEOUT_MS>
          Timeout of gRPC client [default: 60000]
      --http-timeout-ms <HTTP_TIMEOUT_MS>
          Timeout of HTTP client [default: 60000]
      --on-disk-payload
          Whether to use on-disk payload storage
      --only-sparse
          Whether to only upsert sparse vectors
      --duplication-factor <DUPLICATION_FACTOR>
          Duplication factor for generating additional named vectors [default: 2]
      --skip-snapshot-restore
          Whether to perform snapshot restore
      --shutdown-on-error
          Whether to shut down qdrant on error (use false to investigate deadlocks)
      --cpu-quota <CPU_QUOTA>
          CPU quotas in percent for the Qdrant binary
      --rng-seed <RNG_SEED>
          Seed for internal RNG
  -h, --help
          Print help
  -V, --version
          Print version

Vector configurations

Each base configuration is multiplied by the --duplication-factor (default: 2).

Dense vectors

Name Distance Storage Datatype Quantization HNSW
dense-vector-mmap Dot mmap - - default
dense-vector-memory Dot memory - - default
dense-vector-uint8 Dot mmap UInt8 - default
dense-vector-float16 Dot mmap Float16 - default
dense-vector-sq Dot mmap - SQ Int8 default
dense-vector-pq Dot mmap - PQ x8 default
dense-vector-bq-1b Dot mmap - BQ 1-bit default
dense-vector-bq-1Hb Dot mmap - BQ 1.5-bit default
dense-vector-bq-2b Dot mmap - BQ 2-bit default
dense-vector-hnsw-inline Dot mmap - BQ 2-bit inline storage
dense-vector-cosine Cosine mmap - - default
dense-vector-euclid Euclid mmap - - default
dense-vector-manhattan Manhattan mmap - - default
dense-vector-memory-sq Dot memory - SQ Int8 default
dense-vector-memory-pq Dot memory - PQ x8 default
dense-vector-memory-bq Dot memory - BQ 1-bit default
dense-vector-memory-bq-1Hb Dot memory - BQ 1.5-bit default
dense-vector-memory-bq-2b Dot memory - BQ 2-bit default
dense-vector-memory-uint8 Dot memory UInt8 - default
dense-vector-memory-float16 Dot memory Float16 - default
dense-vector-memory-cosine Cosine memory - - default
dense-vector-memory-euclid Euclid memory - - default
dense-vector-memory-manhattan Manhattan memory - - default
dense-vector-hnsw-on-disk Dot mmap - - on-disk graph
dense-vector-pq-x16 Dot mmap - PQ x16 default
dense-vector-sq-ram Dot mmap - SQ Int8 (quantile=0.95, always_ram) default

Sparse vectors

Name Storage Datatype Modifier
sparse-vector-index-disk mmap - -
sparse-vector-index-memory memory - -
sparse-vector-uint8 mmap UInt8 -
sparse-vector-float16 mmap Float16 -
sparse-vector-IDF mmap Float32 IDF
sparse-vector-memory-uint8 memory UInt8 -
sparse-vector-memory-float16 memory Float16 -
sparse-vector-memory-IDF memory Float32 IDF

Multi-dense vectors

Name Distance Storage Datatype Quantization
multi-dense-vector-mmap Dot mmap - -
multi-dense-vector-memory Dot memory - -
multi-dense-vector-uint8 Dot mmap UInt8 -
multi-dense-vector-float16 Dot mmap Float16 -
multi-dense-vector-sq Dot mmap - SQ Int8
multi-dense-vector-pq Dot mmap - PQ x8
multi-dense-vector-bq Dot mmap - BQ 2-bit

Examples:

  • using debug binary:
cargo run -r -- --working-dir "/home/agourlay/Workspace/qdrant/" --exec-path "target/debug/qdrant"
  • using release binary:
cargo run -r -- --working-dir "/home/agourlay/Workspace/qdrant/" --exec-path "target/release/qdrant" --indexing-threshold 2000 --memmap-threshold 2000