Skip to content

Commit 340c9e6

Browse files
fcostaoliveirafcostaoliveira
authored
Updated docker org to redis. Included extra docs and visualization features (#35)
* feat: comprehensive Docker improvements and precision summary enhancements - Simplify Docker tagging strategy to only latest and version tags - Change Docker image repository from filipe958/vector-db-benchmark to redis/vector-db-benchmark - Fix Docker volume permission issues with smart entrypoint script - Add precision_summary field with clean QPS, P50, P95 metrics - Update Python version support to include 3.13 - Add comprehensive Redis environment variables (REDIS_AUTH, REDIS_USER, REDIS_CLUSTER) - Create Docker-specific README with Redis 8.2 examples - Update all documentation to use redis/vector-db-benchmark consistently - Enhance GitHub Actions with Redis CLI installation and Poetry support * Including QPS per precision chart on cli.
1 parent 6ebf338 commit 340c9e6

File tree

8 files changed

+369
-52
lines changed

8 files changed

+369
-52
lines changed

‎.github/workflows/docker-publish-master.yml

Lines changed: 3 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
name: Docker Publish - update.redisearch branch
1+
name: Docker Publish - Latest
22

33
on:
44
push:
@@ -11,7 +11,7 @@ on:
1111

1212
env:
1313
REGISTRY: docker.io
14-
IMAGE_NAME: filipe958/vector-db-benchmark
14+
IMAGE_NAME: redis/vector-db-benchmark
1515

1616
jobs:
1717
docker-publish:
@@ -61,8 +61,6 @@ jobs:
6161
images: ${{ env.IMAGE_NAME }}
6262
tags: |
6363
type=raw,value=latest
64-
type=raw,value=update-redisearch-{{sha}}
65-
type=raw,value=update-redisearch-{{date 'YYYYMMDD-HHmmss'}}
6664
6765
- name: Build and push Docker image
6866
uses: docker/build-push-action@v5
@@ -95,4 +93,4 @@ jobs:
9593
echo "docker run --rm ${{ env.IMAGE_NAME }}:latest run.py --help" >> $GITHUB_STEP_SUMMARY
9694
echo "\`\`\`" >> $GITHUB_STEP_SUMMARY
9795
echo "" >> $GITHUB_STEP_SUMMARY
98-
echo "🔗 [View on Docker Hub](https://hub.docker.com/r/filipe958/vector-db-benchmark)" >> $GITHUB_STEP_SUMMARY
96+
echo "🔗 [View on Docker Hub](https://hub.docker.com/r/redis/vector-db-benchmark)" >> $GITHUB_STEP_SUMMARY

‎.github/workflows/docker-publish-release.yml

Lines changed: 2 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@ on:
66

77
env:
88
REGISTRY: docker.io
9-
IMAGE_NAME: filipe958/vector-db-benchmark
9+
IMAGE_NAME: redis/vector-db-benchmark
1010

1111
jobs:
1212
docker-publish:
@@ -57,9 +57,6 @@ jobs:
5757
tags: |
5858
type=ref,event=tag
5959
type=semver,pattern={{version}}
60-
type=semver,pattern={{major}}.{{minor}}
61-
type=semver,pattern={{major}}
62-
type=raw,value=latest,enable={{is_default_branch}}
6360
6461
- name: Build and push Docker image
6562
uses: docker/build-push-action@v5
@@ -110,5 +107,5 @@ jobs:
110107
echo "docker run --rm ${{ env.IMAGE_NAME }}:latest run.py --help" >> $GITHUB_STEP_SUMMARY
111108
echo "\`\`\`" >> $GITHUB_STEP_SUMMARY
112109
echo "" >> $GITHUB_STEP_SUMMARY
113-
echo "🔗 [View on Docker Hub](https://hub.docker.com/r/filipe958/vector-db-benchmark)" >> $GITHUB_STEP_SUMMARY
110+
echo "🔗 [View on Docker Hub](https://hub.docker.com/r/redis/vector-db-benchmark)" >> $GITHUB_STEP_SUMMARY
114111
echo "🔒 [Security Scan Results](https://github.com/${{ github.repository }}/security/code-scanning)" >> $GITHUB_STEP_SUMMARY

‎DOCKER_README.md

Lines changed: 176 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,176 @@
1+
# Redis Vector Database Benchmark
2+
3+
A comprehensive benchmarking tool for vector databases, including Redis (both RediSearch and Vector Sets), Weaviate, Milvus, Qdrant, OpenSearch, Postgres, and others...
4+
5+
In a one-liner cli tool you can get this and much more:
6+
7+
```
8+
docker run --rm --network=host redis/vector-db-benchmark:latest run.py --host localhost --engines vectorsets-fp32-default --datasets glove-100-angular --parallels 100
9+
(...)
10+
================================================================================
11+
BENCHMARK RESULTS SUMMARY
12+
Experiment: vectorsets-fp32-default - glove-100-angular
13+
================================================================================
14+
15+
Precision vs Performance Trade-off:
16+
--------------------------------------------------
17+
Precision QPS P50 (ms) P95 (ms)
18+
--------------------------------------------------
19+
0.86 1408.3 61.877 107.548
20+
0.80 2136.3 38.722 69.102
21+
0.72 2954.3 25.820 48.072
22+
0.68 3566.5 20.229 38.581
23+
24+
QPS vs Precision Trade-off - vectorsets-fp32-default - glove-100-angular (up and to the right is better):
25+
26+
3566 │●
27+
│ ●
28+
│
29+
2594 │
30+
│ ●
31+
│
32+
1621 │ ●
33+
│
34+
│
35+
648 │
36+
│
37+
│
38+
0 │
39+
└────────────────────────────────────────────────────────────
40+
0.680 0.726 0.772 0.817
41+
Precision (0.0 = 0%, 1.0 = 100%)
42+
================================================================================
43+
44+
```
45+
46+
## Quick Start
47+
48+
```bash
49+
# Pull the latest image
50+
docker pull redis/vector-db-benchmark:latest
51+
52+
# Run with help
53+
docker run --rm redis/vector-db-benchmark:latest run.py --help
54+
55+
# Check available datasets
56+
docker run --rm redis/vector-db-benchmark:latest run.py --describe datasets
57+
58+
# Basic Redis benchmark (requires local Redis)
59+
docker run --rm -v $(pwd)/results:/app/results --network=host \
60+
redis/vector-db-benchmark:latest \
61+
run.py --host localhost --engines redis-default-simple --dataset random-100
62+
```
63+
64+
## Features
65+
66+
- **42+ Datasets**: Pre-configured datasets from 25 to 1B+ vectors
67+
- **Multiple Engines**: Redis, Qdrant, Weaviate, Milvus, and more
68+
- **Real-time Monitoring**: Live performance metrics during benchmarks
69+
- **Precision Analysis**: Detailed accuracy vs performance trade-offs
70+
- **Easy Discovery**: `--describe` commands for datasets and engines
71+
72+
## Available Tags
73+
74+
- `latest` - Latest development build from update.redisearch branch
75+
76+
## Redis quick start
77+
78+
### Redis 8.2 with RediSearch
79+
```bash
80+
# Start Redis 8.2 with built-in vector support
81+
docker run -d --name redis-test -p 6379:6379 redis:8.2-rc1-bookworm
82+
83+
# Run benchmark
84+
docker run --rm -v $(pwd)/results:/app/results --network=host \
85+
redis/vector-db-benchmark:latest \
86+
run.py --host localhost --engines redis-default-simple --dataset glove-25-angular
87+
```
88+
89+
90+
## Common Usage Patterns
91+
92+
### Explore Available Options
93+
```bash
94+
# List all datasets
95+
docker run --rm redis/vector-db-benchmark:latest run.py --describe datasets
96+
97+
# List all engines
98+
docker run --rm redis/vector-db-benchmark:latest run.py --describe engines
99+
```
100+
101+
### Run Benchmarks
102+
```bash
103+
# Quick test with small dataset
104+
docker run --rm -v $(pwd)/results:/app/results --network=host \
105+
redis/vector-db-benchmark:latest \
106+
run.py --host localhost --engines redis-default-simple --dataset random-100
107+
108+
# Comprehensive benchmark with multiple configurations
109+
docker run --rm -v $(pwd)/results:/app/results --network=host \
110+
redis/vector-db-benchmark:latest \
111+
run.py --host localhost --engines "*redis*" --dataset glove-25-angular
112+
113+
# With Redis authentication
114+
docker run --rm -v $(pwd)/results:/app/results --network=host \
115+
-e REDIS_AUTH=mypassword -e REDIS_USER=myuser \
116+
redis/vector-db-benchmark:latest \
117+
run.py --host localhost --engines redis-default-simple --dataset random-100
118+
```
119+
120+
### Results Analysis
121+
```bash
122+
# View precision summary
123+
jq '.precision_summary' results/*-summary.json
124+
125+
# View detailed results
126+
jq '.search' results/*-summary.json
127+
```
128+
129+
## Volume Mounts
130+
131+
- `/app/results` - Benchmark results (JSON files)
132+
- `/app/datasets` - Dataset storage (optional, auto-downloaded)
133+
134+
## Environment Variables
135+
136+
- `REDIS_HOST` - Redis server hostname (default: localhost)
137+
- `REDIS_PORT` - Redis server port (default: 6379)
138+
- `REDIS_AUTH` - Redis password (default: None)
139+
- `REDIS_USER` - Redis username (default: None)
140+
- `REDIS_CLUSTER` - Enable Redis cluster mode (default: 0)
141+
142+
## Performance Tips
143+
144+
1. **Use `--network=host`** for best performance with local Redis
145+
2. **Mount results volume** to persist benchmark data
146+
3. **Start with small datasets** (random-100, glove-25-angular) for testing
147+
4. **Use wildcard patterns** to test multiple configurations: `--engines "*-m-16-*"`
148+
149+
## Example Output
150+
151+
```json
152+
{
153+
"precision_summary": {
154+
"0.91": {
155+
"qps": 1924.5,
156+
"p50": 49.828,
157+
"p95": 58.427
158+
},
159+
"0.94": {
160+
"qps": 1819.9,
161+
"p50": 51.68,
162+
"p95": 66.83
163+
}
164+
}
165+
}
166+
```
167+
168+
## Support
169+
170+
- **GitHub**: [redis-performance/vector-db-benchmark](https://github.com/redis-performance/vector-db-benchmark)
171+
- **Issues**: Report bugs and feature requests on GitHub
172+
- **Documentation**: Full documentation available in the repository
173+
174+
## License
175+
176+
This project is licensed under the MIT License - see the repository for details.

‎DOCKER_SETUP.md

Lines changed: 12 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# Docker Setup and Publishing Guide
22

3-
This guide explains how to set up Docker publishing for the `vector-db-benchmark` project to Docker Hub repository `filipe958/vector-db-benchmark`.
3+
This guide explains how to set up Docker publishing for the `vector-db-benchmark` project to Docker Hub repository `redis/vector-db-benchmark`.
44

55
## 🔐 Required GitHub Secrets
66

@@ -57,11 +57,8 @@ Once secrets are configured, Docker images will be automatically published:
5757

5858
### Example Tags for Release v1.2.3
5959
```
60-
filipe958/vector-db-benchmark:v1.2.3
61-
filipe958/vector-db-benchmark:1.2.3
62-
filipe958/vector-db-benchmark:1.2
63-
filipe958/vector-db-benchmark:1
64-
filipe958/vector-db-benchmark:latest
60+
redis/vector-db-benchmark:v1.2.3
61+
redis/vector-db-benchmark:latest
6562
```
6663

6764
## 🛠️ Manual Building and Publishing
@@ -100,32 +97,32 @@ export DOCKER_PASSWORD=your_access_token
10097
### Pull and Run
10198
```bash
10299
# Latest version
103-
docker pull filipe958/vector-db-benchmark:latest
104-
docker run --rm filipe958/vector-db-benchmark:latest run.py --help
100+
docker pull redis/vector-db-benchmark:latest
101+
docker run --rm redis/vector-db-benchmark:latest run.py --help
105102

106103
# Specific version
107-
docker pull filipe958/vector-db-benchmark:v1.2.3
108-
docker run --rm filipe958/vector-db-benchmark:v1.2.3 run.py --help
104+
docker pull redis/vector-db-benchmark:v1.2.3
105+
docker run --rm redis/vector-db-benchmark:v1.2.3 run.py --help
109106
```
110107

111108
### Example Usage
112109
```bash
113110
# Basic Redis benchmark
114-
docker run --rm --network=host filipe958/vector-db-benchmark:latest \
111+
docker run --rm --network=host redis/vector-db-benchmark:latest \
115112
run.py --host localhost --engines redis --dataset random-100 --experiment redis-default-simple
116113

117114
# With custom Redis host
118-
docker run --rm filipe958/vector-db-benchmark:latest \
115+
docker run --rm redis/vector-db-benchmark:latest \
119116
run.py --host redis-server --engines redis --dataset random-100 --experiment redis-default-simple
120117

121118
# With results output (mount current directory)
122119
docker run --rm -v $(pwd)/results:/app/results --network=host \
123-
filipe958/vector-db-benchmark:latest \
120+
redis/vector-db-benchmark:latest \
124121
run.py --host localhost --engines redis --dataset random-100 --experiment redis-default-simple
125122

126123
# Using with Redis container
127124
docker run -d --name redis-test -p 6379:6379 redis:8.2-rc1-bookworm
128-
docker run --rm --network=host filipe958/vector-db-benchmark:latest \
125+
docker run --rm --network=host redis/vector-db-benchmark:latest \
129126
run.py --host localhost --engines redis --experiment redis-default-simple
130127
docker stop redis-test && docker rm redis-test
131128
```
@@ -145,7 +142,7 @@ docker stop redis-test && docker rm redis-test
145142
- Prevents merging PRs with broken Docker builds
146143

147144
### Docker Hub
148-
- View images at: https://hub.docker.com/r/filipe958/vector-db-benchmark
145+
- View images at: https://hub.docker.com/r/redis/vector-db-benchmark
149146
- Check image sizes and platforms
150147
- Review vulnerability scan results
151148

0 commit comments

Comments
 (0)