Skip to content

YDB database support #1215

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 45 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
45 commits
Select commit Hold shift + click to select a range
32bc791
add skeleton
art22m Oct 19, 2024
d8c5027
implement methods
art22m Oct 22, 2024
8f56312
makefile, cli build
art22m Oct 23, 2024
44abc30
go mod tidy
art22m Oct 23, 2024
4dd2ac5
some fixes
art22m Oct 23, 2024
ee172d8
refactor
art22m Nov 4, 2024
754a74d
fix drop func
art22m Nov 4, 2024
a7e222d
update readme
art22m Nov 4, 2024
532b4ac
add auth methods
art22m Nov 4, 2024
86ae1fa
refactor
art22m Nov 4, 2024
7bfd098
update readme
art22m Nov 4, 2024
bd9bdae
add tests
art22m Nov 4, 2024
0d18ddb
fix test
art22m Nov 9, 2024
2e3253b
rm logs
art22m Nov 9, 2024
5b21bf5
refactor test files
art22m Nov 9, 2024
1aa3455
add tls options
art22m Nov 15, 2024
fecf8d7
change readme
art22m Nov 15, 2024
76093a1
support min tls version
art22m Nov 19, 2024
bfe9fba
fix reviews #1
art22m Dec 8, 2024
8baa73b
fix reviews #2
art22m Dec 8, 2024
81f152f
migrate to database/sql
art22m Dec 15, 2024
063c1d0
Apply suggestions from code review
asmyasnikov Jan 6, 2025
c2ac3bc
fixes
asmyasnikov Jan 6, 2025
1e4515c
reverted dependencies
asmyasnikov Jan 6, 2025
b5aa72f
fixed env
asmyasnikov Jan 6, 2025
3835934
Merge pull request #3 from art22m/support_ydb
asmyasnikov Jan 6, 2025
744dd5e
go mod tidy
asmyasnikov Jan 6, 2025
5444c1e
rename yql to sql
asmyasnikov Jan 6, 2025
8d20ea1
more sql-migrations
asmyasnikov Jan 6, 2025
bcb7e84
add params, table creation
art22m Jan 16, 2025
7edc143
implement locks
art22m Jan 16, 2025
3143ea4
multiple images testing
art22m Jan 16, 2025
970b831
fix tests
art22m Jan 17, 2025
b19af9c
rm query mode selection
art22m Jan 17, 2025
677160f
fix tests
art22m Jan 20, 2025
fa12828
rm force lock
art22m Jan 20, 2025
e2d7653
Merge pull request #4 from art22m/refactor-ydb-lock
asmyasnikov Jan 30, 2025
3b0367b
Merge branch 'golang-migrate:master' into master
asmyasnikov Jan 30, 2025
fd6f3ae
bump ydb-go-sdk
asmyasnikov Jan 30, 2025
63a7f44
exclude ydb 24.2 from specs
asmyasnikov Jan 31, 2025
3a93627
bump ydb-go-sdk
asmyasnikov Feb 7, 2025
8981237
Merge branch 'golang-migrate:master' into master
asmyasnikov Feb 20, 2025
067ca8f
update ydb-go-sdk/v3
asmyasnikov Feb 27, 2025
8d57316
Merge branch 'golang-migrate:master' into master
asmyasnikov Apr 23, 2025
001b2f0
Merge branch 'golang-migrate:master' into master
asmyasnikov Apr 28, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion Makefile
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
SOURCE ?= file go_bindata github github_ee bitbucket aws_s3 google_cloud_storage godoc_vfs gitlab
DATABASE ?= postgres mysql redshift cassandra spanner cockroachdb yugabytedb clickhouse mongodb sqlserver firebird neo4j pgx pgx5 rqlite
DATABASE ?= postgres mysql redshift cassandra spanner cockroachdb yugabytedb clickhouse mongodb sqlserver firebird neo4j pgx pgx5 rqlite ydb
DATABASE_TEST ?= $(DATABASE) sqlite sqlite3 sqlcipher
VERSION ?= $(shell git describe --tags 2>/dev/null | cut -c 2-)
TEST_FLAGS ?=
Expand Down
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,7 @@ Database drivers run migrations. [Add a new database?](database/driver.go)
* [Firebird](database/firebird)
* [MS SQL Server](database/sqlserver)
* [rqlite](database/rqlite)
* [YDB](database/ydb)

### Database URLs

Expand Down
48 changes: 48 additions & 0 deletions database/ydb/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
# [YDB](https://ydb.tech/docs/)

`ydb://[user:password@]host:port/database?QUERY_PARAMS`

| URL Query | Description |
|:----------:|:---------------------------------------:|
| `user` | The user to sign in as. |
| `password` | The user's password. |
| `host` | The host to connect to. |
| `port` | The port to bind to. |
| `database` | The name of the database to connect to. |

| URL Query Params | Description |
|:----------------------------:|:--------------------------------------------------------------------------------------------:|
| `x-auth-token` | Authentication token. |
| `x-migrations-table` | Name of the migrations table (default `schema_migrations`). |
| `x-lock-table` | Name of the table which maintains the migration lock (default `schema_lock`). |
| `x-use-grpcs` | Enables gRPCS protocol for YDB connections (default grpc). |
| `x-tls-ca` | The location of the CA (certificate authority) file. |
| `x-tls-insecure-skip-verify` | Controls whether a client verifies the server's certificate chain and host name. |
| `x-tls-min-version` | Controls the minimum TLS version that is acceptable, use 1.0, 1.1, 1.2 or 1.3 (default 1.2). |

### Secure connection

Query param `x-use-grpcs` enables secure TLS connection that requires certificates.
You can declare root certificate using ENV
variable: `export YDB_SSL_ROOT_CERTIFICATES_FILE=/path/to/ydb/certs/CA.pem` or
by using `x-tls-ca` query param: `?x-tls-ca=/path/to/ydb/certs/CA.pem`.

### Authentication

By default, golang-migrate connects to YDB
using [anonymous credentials](https://ydb.tech/docs/en/recipes/ydb-sdk/auth-anonymous). \
Through the url query, you can change the default behavior:

- To connect to YDB using [static credentials](https://ydb.tech/docs/en/recipes/ydb-sdk/auth-static) you need to specify
username and password:
`ydb://user:password@host:port/database`
- To connect to YDB using [token](https://ydb.tech/docs/en/recipes/ydb-sdk/auth-access-token) you need to specify token
as query parameter:
`ydb://host:port/database?x-auth-token=<YDB_TOKEN>`

### Locks

If golang-migrate fails to acquire the lock and no migrations are currently running.
This may indicate that one of the migrations did not complete successfully.
In this case, you need to analyze the previous migrations, rollback if necessary, and manually remove the lock from the
`x-lock-table`.
1 change: 1 addition & 0 deletions database/ydb/examples/migrations/001_create_users.down.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
DROP TABLE `test/users`;
6 changes: 6 additions & 0 deletions database/ydb/examples/migrations/001_create_users.up.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
CREATE TABLE `test/users` (
id Uint64,
name String,
email String,
PRIMARY KEY (id)
);
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
DROP TABLE `test/cities`;

ALTER TABLE `test/users` DROP COLUMN city;
7 changes: 7 additions & 0 deletions database/ydb/examples/migrations/002_add_city_to_users.up.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
CREATE TABLE `test/cities` (
id Uint64,
name String,
PRIMARY KEY (id)
);

ALTER TABLE `test/users` ADD COLUMN city Uint64;
1 change: 1 addition & 0 deletions database/ydb/examples/migrations/003_create_topic.down.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
DROP TOPIC `test/topic`;
1 change: 1 addition & 0 deletions database/ydb/examples/migrations/003_create_topic.up.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
CREATE TOPIC `test/topic`;
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
ALTER TABLE `test/users` DROP INDEX `users_email_index`;
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
ALTER TABLE `test/users` ADD INDEX `users_email_index` GLOBAL UNIQUE ON (`email`);
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
DROP TABLE `test/books`;
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
CREATE TABLE `test/books` (
user_id Uint64,
name Text,
author Text,
PRIMARY KEY (user_id)
);
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
DROP TABLE `test/movies`;
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
CREATE TABLE `test/movies` (
user_id Uint64,
name Text,
director Text,
PRIMARY KEY (user_id)
);
Loading