Skip to content

Commit 9a9acac

Browse files
pradithyaPradithya Aria
pradithya
and
Pradithya Aria
authored
External Secret Storage Implementation (caraml-dev#85)
* Rename storage to repository * Run gofmt * Run gofmt * Add secret storage repository * Fix mount_type typo * Remove kv_version in vault config * Use ememory storage for keto local testing * Use ememory storage for keto local testing * Fix lint * Fix lint * Update config package and its tests to support default secret storage * Add pathPrefix config * Add basic implementation of vault secret storage * Add authentication to vault * Update secret repository interfaces and remove encryption * Update secret storage repository interfaces * Fix incorrect teardown in SecretStorageTestSuite * Add Service Account Email field to secret storage for iam auth * Update secret storage client registry * Update secret service implementation and tests * Remove encryption key from config * Add default secret storage initialization * Update secrets_api.go * Fix secrets_api_test.go * Add unit test for FindByID * Remove EncryptionKey from tests * Add reference to the corresponding project in secret model * Add end to end test * Update dependency * Update CI * Add map.go * Add secret_storage_repository.go mock * Fix lint * Add mocks * Fix lint * Remove it-test-api-ci * Add SetAll and DeleteAll to secretstorage.Client interface * Remove old vault code * Add secret storage API * Fromat projects_api.go * Update entity model * Properly delete secret key * Allow user to specify secret storage when creating or updating secret * Update local development configuration * Update README.md * Add code comments * Update swagger and client code * Update swagger and client code * Revert "Update swagger and client code" This reverts commit f27ad2c. * Revert "Update swagger and client code" This reverts commit 63beda4. * Update swagger * Update generated client code * Change the ordering in db migration * Change basePath in example * Update MLP_API_BASEPATH * Fix incorrect ordering * Use InternalServerError * Fix comparison * Handle global secret storage migration * Drop types in db migration downgrade * Rename JoinMaps to JoinStringMaps * Use EuiFieldPassword as secret data input * Add godocs * Remove redundant check * Refactor * Refactor secretRepository * Disallow migrating to internal secret storage * Properly handle default secret storage selection * Reformat * Fix lint ui --------- Co-authored-by: Pradithya Aria <[email protected]>
1 parent f6948ba commit 9a9acac

File tree

96 files changed

+7826
-1868
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

96 files changed

+7826
-1868
lines changed

‎.env.development

+2-1
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,5 @@
11
DATABASE_HOST=localhost
22
DATABASE_USER=mlp
33
DATABASE_PASSWORD=mlp
4-
DATABASE_NAME=mlp
4+
DATABASE_NAME=mlp
5+
REACT_APP_OAUTH_CLIENT_ID="[OAUTH_CLIENT_ID]"

‎.github/workflows/ci-pipeline.yml

+2-28
Original file line numberDiff line numberDiff line change
@@ -99,26 +99,6 @@ jobs:
9999
runs-on: ubuntu-latest
100100
env:
101101
GOPATH: ${{ github.workspace }}/.go
102-
services:
103-
postgres:
104-
image: postgres:14.5
105-
env:
106-
POSTGRES_DB: mlp
107-
POSTGRES_USER: mlp
108-
POSTGRES_PASSWORD: mlp
109-
options: >-
110-
--health-cmd pg_isready
111-
--health-interval 10s
112-
--health-timeout 5s
113-
--health-retries 5
114-
ports:
115-
- 5432:5432
116-
keto:
117-
image: oryd/keto:v0.4
118-
env:
119-
DSN: memory
120-
ports:
121-
- 4466:4466
122102
steps:
123103
- uses: actions/checkout@v3
124104
- uses: actions/setup-go@v3
@@ -137,13 +117,7 @@ jobs:
137117
run: make lint-api
138118

139119
- name: Run Integration Test
140-
env:
141-
DATABASE_HOST: localhost
142-
DATABASE_NAME: mlp
143-
DATABASE_USER: mlp
144-
DATABASE_PASSWORD: mlp
145-
KETO_URL: http://localhost:4466
146-
run: make it-test-api-ci
120+
run: make it-test-api
147121

148122
e2e-test:
149123
runs-on: ubuntu-latest
@@ -249,7 +223,7 @@ jobs:
249223
250224
- name: Run E2E tests
251225
env:
252-
MLP_API_BASEPATH: http://mlp.127.0.0.1.nip.io/v1
226+
MLP_API_BASEPATH: http://mlp.127.0.0.1.nip.io
253227
shell: bash
254228
run: |
255229
for example in api/client/examples/*; do

‎Makefile

+20-13
Original file line numberDiff line numberDiff line change
@@ -46,7 +46,8 @@ lint-ui:
4646
.PHONY: fmt
4747
fmt:
4848
@echo "Formatting code..."
49-
gofmt -s -w ${SRC_ROOT}
49+
@goimports -w -local github.com/caraml-dev/mlp $(shell find . -type f -name '*.go' -not -path "**/vendor/*")
50+
@gofmt -s -w .
5051

5152
.PHONY: lint-api
5253
lint-api: setup
@@ -65,19 +66,13 @@ test-api: init-dep-api
6566
@cd ${API_PATH} && go test -v -race -cover -coverprofile cover.out ${API_ALL_PACKAGES}
6667
@cd ${API_PATH} && go tool cover -func cover.out
6768

68-
.PHONY: it-test-api-local
69-
it-test-api-local: local-db start-keto
69+
.PHONY: it-test-api
70+
it-test-api: local-db start-keto start-vault
7071
@echo "> API integration testing locally..."
7172
@cd ${API_PATH} && go test -race -short -cover -coverprofile cover.out -tags integration ${API_ALL_PACKAGES}
7273
@cd ${API_PATH} && go tool cover -func cover.out
7374
@make stop-docker
7475

75-
.PHONY: it-test-api-ci
76-
it-test-api-ci:
77-
@echo "> API integration testing ..."
78-
@cd ${API_PATH} && go test -race -short -cover -coverprofile cover.out -tags integration ${API_ALL_PACKAGES}
79-
@cd ${API_PATH} && go tool cover -func cover.out
80-
8176
# ============================================================
8277
# Building recipes
8378
# ============================================================
@@ -115,9 +110,9 @@ build-image: version
115110
# Run recipes
116111
# ============================================================
117112
.PHONY: run
118-
run: build-api local-db
113+
run: local-env
119114
@echo "> Running application ..."
120-
@./bin/${BIN_NAME} --config config-dev.yaml
115+
@go run api/cmd/main.go --config config-dev.yaml
121116

122117
.PHONY: start-ui
123118
start-ui:
@@ -142,8 +137,15 @@ clean-bin:
142137

143138
generate-client:
144139
@echo "> Generating API client ..."
145-
@swagger-codegen generate -i static/swagger.yaml -l go -o client -DpackageName=client
146-
@goimports -l -w client
140+
@docker run --rm -v $(shell pwd):/local swaggerapi/swagger-codegen-cli:2.4.14 generate \
141+
-i /local/api/static/swagger.yaml \
142+
-l go \
143+
-o /local/api/client \
144+
-DpackageName=client
145+
$(MAKE) fmt
146+
147+
.PHONY: local-env
148+
local-env: local-db start-keto start-vault
147149

148150
.PHONY: local-db
149151
local-db:
@@ -155,6 +157,11 @@ start-keto:
155157
@echo "> Starting up keto server ..."
156158
@docker-compose up -d keto
157159

160+
.PHONY: start-vault
161+
start-vault:
162+
@echo "> Starting up vault server ..."
163+
@docker-compose up -d vault
164+
158165
.PHONY: stop-docker
159166
stop-docker:
160167
@echo "> Stopping Docker compose ..."

‎README.md

+12-7
Original file line numberDiff line numberDiff line change
@@ -22,7 +22,7 @@ The MLP Server provides REST API used across MLP Products. It exposes a shared c
2222

2323
1. [Google Oauth credential](https://developers.google.com/identity/protocols/oauth2/javascript-implicit-flow)
2424

25-
MLP uses Google Sign-in to authenticate the user to access the API and UI. After you get the client ID, specify it into `OAUTH_CLIENT_ID` in `.env.development` file.
25+
MLP uses Google Sign-in to authenticate the user to access the API and UI. After you get the client ID, specify it into `REACT_APP_OAUTH_CLIENT_ID` in `.env.development` file.
2626

2727
### From Docker Compose
2828

@@ -36,14 +36,19 @@ MLP will now be reachable at <http://localhost:8080>.
3636

3737
### From source
3838

39-
To build and run MLP from the source code, you need to have [Go](https://golang.org/doc/install), [Node.js](https://nodejs.org/), and [Yarn](https://yarnpkg.com/) installed. You will also need a running Postgresql database. MLP uses Docker to make the task of setting up databases a little easier. You can run `make local-db` to starting up a Postgres Docker container.
39+
To build and run MLP from the source code, you need to have [Go](https://golang.org/doc/install), [Node.js](https://nodejs.org/), and [Yarn](https://yarnpkg.com/) installed.
40+
You will also need a running Postgresql database, Keto, and Vault servers.
41+
MLP uses Docker to make the task of setting up the dependencies a little easier. You can run `make local-env` to starting up all those dependencies.
4042

4143
```shell script
42-
mkdir -p $GOPATH/src/github.com/caraml-dev
43-
cd $GOPATH/src/github.com/caraml-dev
44-
git clone [email protected]/caraml-dev/mlp.git mlp
45-
cd mlp
46-
make local-db
44+
make local-env
45+
make run
46+
```
47+
48+
OR
49+
50+
```shell script
51+
# `make` will execute `make local-env` and `make run`
4752
make
4853
```
4954

‎api/api/api_test.go

+162
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,162 @@
1+
package api
2+
3+
import (
4+
"fmt"
5+
"net/http"
6+
"strings"
7+
"testing"
8+
"time"
9+
10+
"github.com/gorilla/mux"
11+
"github.com/stretchr/testify/suite"
12+
13+
"github.com/caraml-dev/mlp/api/config"
14+
"github.com/caraml-dev/mlp/api/it/database"
15+
"github.com/caraml-dev/mlp/api/models"
16+
)
17+
18+
type APITestSuite struct {
19+
suite.Suite
20+
21+
// handler under test
22+
route http.Handler
23+
24+
// db cleanup function
25+
cleanupFn func()
26+
27+
// main project to be used for testing
28+
// most test entities (secret and secret_storage) are created under this
29+
mainProject *models.Project
30+
// other project to be used for testing
31+
otherProject *models.Project
32+
33+
// interal secret storage reference
34+
internalSecretStorage *models.SecretStorage
35+
// default secret storage, the test configures it to vault
36+
defaultSecretStorage *models.SecretStorage
37+
// secret storage owned by the project
38+
projectSecretStorage *models.SecretStorage
39+
// existing secrets owned by the project
40+
// in total there are 5 secrets
41+
// the first 2 secrets are stored in internalSecretStorage
42+
// the rest of secrets are stored in defaultSecretStorage
43+
existingSecrets []*models.Secret
44+
}
45+
46+
func (s *APITestSuite) SetupTest() {
47+
db, cleanupFn, err := database.CreateTestDatabase()
48+
s.Require().NoError(err, "Failed to connect to test database")
49+
s.cleanupFn = cleanupFn
50+
51+
// create app context which also specify vault as the default secret storage
52+
appCtx, err := NewAppContext(db, &config.Config{
53+
Port: 0,
54+
Authorization: &config.AuthorizationConfig{
55+
Enabled: false,
56+
},
57+
Mlflow: &config.MlflowConfig{
58+
TrackingURL: "http://mlflow:5000",
59+
},
60+
DefaultSecretStorage: &config.SecretStorage{
61+
Name: "vault",
62+
Type: string(models.VaultSecretStorageType),
63+
Config: models.SecretStorageConfig{
64+
VaultConfig: &models.VaultConfig{
65+
URL: "http://localhost:8200",
66+
Role: "my-role",
67+
MountPath: "secret",
68+
PathPrefix: fmt.Sprintf("api-test/%d/{{ .Project }}", time.Now().Unix()),
69+
AuthMethod: models.TokenAuthMethod,
70+
Token: "root",
71+
},
72+
},
73+
},
74+
})
75+
s.Require().NoError(err, "Failed to create app context")
76+
77+
// create project and otherProject
78+
s.mainProject, err = appCtx.ProjectsService.CreateProject(&models.Project{
79+
Name: "test-project",
80+
})
81+
s.Require().NoError(err, "Failed to create project")
82+
83+
s.otherProject, err = appCtx.ProjectsService.CreateProject(&models.Project{
84+
Name: "other-project",
85+
})
86+
s.Require().NoError(err, "Failed to create other project")
87+
88+
// get reference to internal secret storage
89+
// internal secret storage always have ID 1
90+
s.internalSecretStorage, err = appCtx.SecretStorageService.FindByID(1)
91+
s.Require().NoError(err, "Failed to find internal secret storage")
92+
93+
// get reference to default secret storage
94+
s.defaultSecretStorage = appCtx.DefaultSecretStorage
95+
96+
// create a secret storage owned by the project
97+
s.projectSecretStorage, err = appCtx.SecretStorageService.Create(&models.SecretStorage{
98+
Name: "project-secret-storage",
99+
Type: models.VaultSecretStorageType,
100+
Scope: models.ProjectSecretStorageScope,
101+
ProjectID: &s.mainProject.ID,
102+
Config: models.SecretStorageConfig{
103+
VaultConfig: &models.VaultConfig{
104+
URL: "http://localhost:8200",
105+
Role: "my-role",
106+
MountPath: "secret",
107+
PathPrefix: fmt.Sprintf("project-secret-test/%d", time.Now().Unix()),
108+
AuthMethod: models.TokenAuthMethod,
109+
Token: "root",
110+
},
111+
},
112+
})
113+
s.Require().NoError(err, "Failed to create project secret storage")
114+
115+
//
116+
s.existingSecrets = make([]*models.Secret, 0)
117+
for i := 0; i < 5; i++ {
118+
secretStorageID := s.defaultSecretStorage.ID
119+
// the first 2 secrets will be created in internal secret storage
120+
if i < 2 {
121+
secretStorageID = s.internalSecretStorage.ID
122+
}
123+
124+
secret, err := appCtx.SecretService.Create(&models.Secret{
125+
ProjectID: s.mainProject.ID,
126+
SecretStorageID: &secretStorageID,
127+
Name: fmt.Sprintf("secret-%d", i),
128+
Data: fmt.Sprintf("secret-data-%d", i),
129+
})
130+
s.Require().NoError(err, "Failed to create secret")
131+
s.existingSecrets = append(s.existingSecrets, secret)
132+
}
133+
134+
// initialize controllers and http handler / route
135+
136+
controllers := []Controller{
137+
&ApplicationsController{AppContext: appCtx},
138+
&ProjectsController{AppContext: appCtx},
139+
&SecretsController{AppContext: appCtx},
140+
&SecretStoragesController{AppContext: appCtx},
141+
}
142+
143+
r := NewRouter(appCtx, controllers)
144+
145+
route := mux.NewRouter()
146+
route.PathPrefix(basePath).Handler(
147+
http.StripPrefix(
148+
strings.TrimSuffix(basePath, "/"),
149+
r,
150+
),
151+
)
152+
153+
s.route = route
154+
}
155+
156+
func (s *APITestSuite) TearDownTest() {
157+
s.cleanupFn()
158+
}
159+
160+
func TestAPI(t *testing.T) {
161+
suite.Run(t, new(APITestSuite))
162+
}

0 commit comments

Comments
 (0)