Skip to content

Commit 5a93889

Browse files
anatolystansleranatolystansler
committed
feat: initial DB Lab
1 parent d3aa08d commit 5a93889

28 files changed

+2796
-1
lines changed

‎.gitignore

+9
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
.DS_Store
2+
3+
/bin/
4+
/db-lab-run/
5+
6+
# Deploy contains Kubernetes configs with secrets, remove from .gitignore when generalized.
7+
/deploy/
8+
9+
/config/config.yaml

‎.gitlab-ci.yml

+21
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
image: golang:1.12.10
2+
3+
cache:
4+
paths:
5+
- /apt-cache
6+
- /go/src/github.com
7+
- /go/src/golang.org
8+
- /go/src/google.golang.org
9+
- /go/src/gopkg.in
10+
11+
before_script:
12+
- make dep
13+
14+
stages:
15+
- test
16+
17+
test:
18+
stage: test
19+
script:
20+
- make vet
21+
- make test

‎.gitlab/merge_request_templates/General.md

+11
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
# Description
2+
3+
[give description of a MR]
4+
5+
# Related issue
6+
7+
[give full link to the related issue]
8+
9+
# Examples
10+
11+
[present examples for reviewers: new CLI output, screenshots, etc]

‎Dockerfile

+1
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
FROM golang:onbuild

‎LICENSE.txt

+34
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
BSD with attribution
2+
3+
Copyright © 2018-2019, Postgres.ai (https://postgres.ai), Nikolay Samokhvalov
4+
5+
Redistribution and use in source and binary forms, with or without
6+
modification, are permitted provided that the following conditions are met:
7+
8+
* Redistributions of source code must retain the above copyright notice, this
9+
list of conditions and the following disclaimer.
10+
11+
* Redistributions in binary form must reproduce the above copyright notice,
12+
this list of conditions and the following disclaimer in the documentation
13+
and/or other materials provided with the distribution.
14+
15+
* Neither the name of the copyright holder nor the names of its
16+
contributors may be used to endorse or promote products derived from
17+
this software without specific prior written permission.
18+
19+
* Redistributions of any form whatsoever and integration into third-party
20+
products (including but not limited to cloud services) must retain the
21+
following acknowledgment in the documentation and copyright notices:
22+
"This product includes software developed by Nikolay Samokhvalov and
23+
the Postgres.ai team (https://postgres.ai/)."
24+
25+
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
26+
AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
27+
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
28+
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
29+
FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
30+
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
31+
SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
32+
CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
33+
OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
34+
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

‎Makefile

+46
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,46 @@
1+
.DEFAULT_GOAL = all
2+
3+
BINARY = dblab
4+
GOARCH = amd64
5+
6+
VERSION?=0.1
7+
BUILD_TIME?=$(shell date -u '+%Y%m%d-%H%M')
8+
COMMIT?=no #$(shell git rev-parse HEAD)
9+
BRANCH?=no #$(shell git rev-parse --abbrev-ref HEAD)
10+
11+
# Symlink into GOPATH
12+
BUILD_DIR=${GOPATH}/${BINARY}
13+
14+
# Setup the -ldflags option for go build here, interpolate the variable values
15+
LDFLAGS = -ldflags "-s -w \
16+
-X main.version=${VERSION} \
17+
-X main.commit=${COMMIT} \
18+
-X main.branch=${BRANCH}\
19+
-X main.buildTime=${BUILD_TIME}"
20+
21+
# Build the project
22+
all: clean dep vet main
23+
24+
dep:
25+
go get -v -d -t ./...
26+
27+
main:
28+
GOARCH=${GOARCH} go build ${LDFLAGS} -o bin/${BINARY} ./src/
29+
30+
test:
31+
go test ./src/
32+
33+
vet:
34+
go vet ./src/...
35+
36+
fmt:
37+
go fmt $$(go list ./... | grep -v /vendor/)
38+
39+
clean:
40+
-rm -f bin/*
41+
42+
run:
43+
go run ${LDFLAGS} ./src/*
44+
45+
.PHONY: all dep main test vet fmt clean run
46+

‎README.md

+164-1
Original file line numberDiff line numberDiff line change
@@ -1,2 +1,165 @@
1-
# db-lab
1+
# Database Lab - Thin clones for faster development
2+
Boost your development process
23

4+
<div align="center">
5+
![Joe Demo](./assets/db-lab.png)
6+
</div>
7+
8+
## Install
9+
10+
*Currently, there are no ready-to-use binaries or a Docker image. The setup
11+
is to be done using the source code.*
12+
13+
1. Get the source code: `git clone https://gitlab.com/postgres-ai/db-lab.git`.
14+
1. Golang is required.
15+
- Ubuntu: In some cases, standard Ubuntu package might not work. See
16+
https://gitlab.com/postgres-ai/db-lab/snippets/1918769.
17+
- On macOS: `brew install golang`
18+
1. Install ZFS (Ubuntu: https://gitlab.com/postgres-ai/db-lab/snippets/1918768).
19+
20+
### ZFS Store
21+
1. Create a ZFS store with a clone of
22+
the production Postgres database (e.g. using WAL-E, WAL-G or Barman archive).
23+
1. Shutdown Postgres, create a new ZFS snapshot
24+
(`sudo zfs snapshot -r zpool@db_state_1`) and remember its name. It will
25+
be needed for further configuration (`initialSnapshot` option in
26+
`config/config.yml`).
27+
1. Start Postgres.
28+
29+
30+
## Run
31+
32+
Deploy DB Lab instance in your infrastructure. You would need to:
33+
1. Create `config/config.yml` (see sample in `config/`).
34+
1. Build `make all` and run DB Lab with some token for REST API authorization
35+
`./bin/dblab -v some-token` (or, with log: `./bin/dblab -v some-token 2>&1 | tee -a dblab.log`).
36+
37+
38+
## REST API
39+
40+
Instance statuses:
41+
- `OK` - instance functions well.
42+
- `WARNING` - still functional, but have some problems, e.g. disk space shortage, insecure connection (upcoming MRs).
43+
44+
Clone statuses:
45+
- `OK` - clone is ready to accept postgres connections.
46+
- `CREATING` - clone is being created.
47+
- `DELETING` - clone is being deleted.
48+
- `FATAL` - fatal error happened (details in status message).
49+
50+
Basic models:
51+
```
52+
Clone
53+
{
54+
id: "xxx",
55+
status: {
56+
code: "OK",
57+
message: "Database is ready"
58+
},
59+
project: "proj1"
60+
snapshot: "" (optional, ID or timestamp)
61+
db: {
62+
username: "postgres"
63+
password: "" (never set on DB Lab side)
64+
host: "xxx",
65+
port: "xxx",
66+
connStr: "xxx"
67+
},
68+
protected: true
69+
deleteAt: "" (timestamp),
70+
name: "",
71+
username: "",
72+
createdAt: ""
73+
}
74+
75+
Error
76+
{
77+
code: "NOT_ENOUGH_DISK_SPACE",
78+
name: "Not enough disk space",
79+
hint: "Stop idle clones, change snapshot policy or increase disk size"
80+
}
81+
```
82+
83+
REST API:
84+
```
85+
Get DB Lab instance status and list clones
86+
GET /status
87+
Response:
88+
{
89+
status: {
90+
code: "OK",
91+
message: "DB Lab is ready"
92+
},
93+
disk: {
94+
size: 1000.0, (bytes)
95+
free: 1200.0
96+
},
97+
expectedCloningTime: 8.0, (secondss)
98+
numClones: 10,
99+
clones: [{
100+
id: "id"
101+
status: {
102+
code: "OK",
103+
message: "Database is ready"
104+
},
105+
...
106+
}, ... ],
107+
snapshots: [{
108+
id: "xxx",
109+
timestamp: "123"
110+
}, ... ]
111+
}
112+
113+
Create a clone
114+
POST /clone/
115+
Request:
116+
{
117+
project: "proj1",
118+
snapshot: (optional): "",
119+
db: {
120+
username: "xxx",
121+
password: "xxx"
122+
}
123+
username: "xxx",
124+
name: "xxx"
125+
}
126+
Response:
127+
{
128+
id: "xxx"
129+
}
130+
131+
Update a clone
132+
PATCH /clone/:id
133+
134+
Reset a clone
135+
POST /clone/:id/reset
136+
137+
Delete a clone
138+
DELETE /clone/:id
139+
140+
Get status of a clone
141+
GET /clone/:id
142+
Response:
143+
{
144+
id: "xxx",
145+
status: {
146+
code: "OK",
147+
message: "Database clone is ready"
148+
},
149+
cloneSize: 1000.0,
150+
cloningTime: 5,
151+
project: "xxx",
152+
snapshot: "xxx",
153+
db: {
154+
username: "xxx",
155+
host: "xxx",
156+
port: "xxx",
157+
connStr: "xxx"
158+
},
159+
protected: true,
160+
deleteAt: "",
161+
name: "xxx",
162+
username: "xxx",
163+
createdAt: "123"
164+
}
165+
```

‎assets/db-lab.png

5.66 KB
Loading

‎config/config.sample.yml

+24
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
# Copy the following to: ./config/config.yaml
2+
3+
server:
4+
verificationToken: "xxx"
5+
port: 3000
6+
7+
provision:
8+
mode: "zfs"
9+
zfs:
10+
pool: "db_lab_pool"
11+
initialSnapshot: "db_state_1"
12+
portPool:
13+
from: 6000
14+
to: 6100
15+
pgVersion: "9.6"
16+
pgBindir: "/usr/lib/postgresql/9.6/bin"
17+
pgDataSubdir: "/pgdata"
18+
19+
cloning:
20+
autoDelete: true
21+
idleTime: 120
22+
accessHost: "localhost"
23+
24+
debug: true

‎config/config.yml

+24
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
# Copy the following to: ./config/config.yaml
2+
3+
server:
4+
verificationToken: "xxx"
5+
port: 3000
6+
7+
provision:
8+
mode: "zfs"
9+
zfs:
10+
pool: "db_lab_pool"
11+
initialSnapshot: "db_state_1"
12+
portPool:
13+
from: 6000
14+
to: 6100
15+
pgVersion: "9.6"
16+
pgBindir: "/usr/lib/postgresql/9.6/bin"
17+
pgDataSubdir: "/pgdata"
18+
19+
cloning:
20+
autoDelete: true
21+
idleTime: 120
22+
accessHost: "35.238.14.209"
23+
24+
debug: true

0 commit comments

Comments
 (0)