Merge pull request #146 from StatCan/feat/125-go-api-1.6

docs: update backend local debugging and improve experience

Author: cboin1996

.gitignore

@@ -1,2 +1,6 @@
 coverage_unit.txt
 jupyter-apis
+.vscode
+.env
+.task
+__debug_bin

README.md

@@ -9,6 +9,13 @@ A Golang replacement for the **[Kubeflow][kubeflow]** Jupyter Web APIs.
 See **[CONTRIBUTING.md](CONTRIBUTING.md)**
 
 ## Development Environment
+***Note that the frontend will report errors when calling `/api/namespaces` when run locally. This***
+***issue does not arise in production, as the `/api/namespaces` endpoint is unused.***
+
+To initialize the `.env` file for the development environment, use `task env`.
+You will need to fill out your kubeflow cloud account and kubeflow namespace information manually.
+The `thunder-tests` folder contains configuration for testing requests against the backend. Use the `vscode`
+`THUNDER CLIENT` extension to load the tests.
 
 ### Run API Server
 
@@ -19,15 +26,42 @@ context. See _Connecting a Kubeflow cluster_ below for options.
 2. Change directory to project root: `cd jupyter-apis`
 3. Run `go run . -spawner-config samples/spawner_ui_config.yaml`
 
+Alternatively,
+
+1. `task go:dev -w -- -spawner-config samples/spawner_ui_config.yaml` will live-reload the Go server upon changes.
+
+_Recommended_
+
+You can use the vscode debugger to run the backend, just copy the below contents to a file at path `.vscode/launch.json`.
+```json
+{
+    "version": "0.2.0",
+    "configurations": [
+        {
+            "name": "Debug jupyter-api backend",
+            "type": "go",
+            "request": "launch",
+            "mode": "debug",
+            "program": ".",
+            "args": [
+                "-spawner-config",
+                "samples/spawner_ui_config.yaml",
+            ],
+            "envFile": "${workspaceFolder}/.env"
+        }
+    ]
+}
+```
+
 ### Run Front-End
 
 The front-end is configured to proxy requests to the local API server. It
 requires an environment variable (`KF_USER_ID`) to specify the current user –
 this is passed to the API server as an HTTP header.
 
-
 The following can be pasted in a script and executed. This uses the latest node lts version(v16.16.0).
 **NOTE**: `user` is when using vagrant. Use the email adress if it is the dev cluser (please never connect to prod directly)
+
 ```
 cd frontend/common/kubeflow-common-lib
 npm i
@@ -44,6 +78,7 @@ KF_USER_ID=user npm start
 For the kubecost data to be retrievable, the following will need to be executed `kubectl port-forward -n kubecost-system deployment/kubecost-cost-analyzer 9090`
 
 ### Older instructions
+
 1. ~Change directory to front-end folder: `cd frontend`~
 2. ~Install dependencies: `npm install`~
 3. ~Run the front-end `KF_USER_ID=<cloud_email> npm start`~
@@ -103,3 +138,31 @@ and run `vagrant up`.
 
 [go]: https://golang.org/dl/
 [kubeflow]: https://github.com/kubeflow/kubeflow
+
+## Whats Different?
+
+Routes are defined in this repository [here](./main.go).
+
+[Upstream](https://github.com/kubeflow/kubeflow/tree/v1.6.0/components/crud-web-apps/jupyter/backend/apps/common/routes), the endpoints are structures via request type (e.g. `GET`, `PUT`, `DELETE`).
+
+_Note_
+
+- _that not all endpoints are included in the golang implementation_
+- _to find the upstream endpoint, load the [Upstream](https://github.com/kubeflow/kubeflow/tree/v1.6.0/components/crud-web-apps/jupyter/backend/apps/common/routes)
+  and use search with the endpoint text!_
+
+| Request Type | Golang Endpoint                                  | Upstream Python Endpoint                                                                                                                                                      | Purpose                                 |
+| ------------ | ------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------- |
+| GET          | /api/config                                      | [/api/config](https://github.com/kubeflow/kubeflow/blob/v1.6.0/components/crud-web-apps/jupyter/backend/apps/common/routes/get.py#L9)                                         |                                         |
+| GET          | /api/gpus                                        | [/api/gpus](https://github.com/kubeflow/kubeflow/blob/v1.6.0/components/crud-web-apps/jupyter/backend/apps/common/routes/get.py#L52)                                          |                                         |
+| GET          | /api/storageclasses/default                      | [/api/storageclasses/default](https://github.com/kubeflow/kubeflow/blob/v1.6.0/components/crud-web-apps/common/backend/kubeflow/kubeflow/crud_backend/routes/get.py#L26)      |                                         |
+| GET          | /api/namespaces/{namespace}/cost/aggregated      | Not found                                                                                                                                                                     | Get the aggregated kubecost              |
+| GET          | /api/namespaces                                  | [/api/namespaces](https://github.com/kubeflow/kubeflow/blob/v1.6.0/components/crud-web-apps/common/backend/kubeflow/kubeflow/crud_backend/routes/get.py#L10)                  | Get the list of namespaces              |
+| GET          | /api/namespaces/{namespace}                      | Not found                                                                                                                                                                     | Get namespace metadata                  |
+| GET          | /api/namespaces/{namespace}/notebooks            | [/api/namespaces/\<namespace\>/notebooks](https://github.com/kubeflow/kubeflow/blob/v1.6.0/components/crud-web-apps/jupyter/backend/apps/common/routes/get.py#L44)              | Get the list of notebooks               |
+| POST         | /api/namespaces/{namespace}/notebooks            | [/api/namespaces/\<namespace\>/notebooks](https://github.com/kubeflow/kubeflow/blob/v1.6.0/components/crud-web-apps/jupyter/backend/apps/default/routes/post.py#L11)            | Create a notebook                       |
+| DELETE       | /api/namespaces/{namespace}/notebooks/{notebook} | [/api/namespaces/\<namespace\>/notebooks/<notebook>](https://github.com/kubeflow/kubeflow/blob/v1.6.0/components/crud-web-apps/jupyter/backend/apps/common/routes/delete.py#L9) | Update a notebook                       |
+| PATCH        | /api/namespaces/{namespace}/notebooks/{notebook} | [/api/namespaces/\<namespace\>/notebooks/<notebook](https://github.com/kubeflow/kubeflow/blob/v1.6.0/components/crud-web-apps/jupyter/backend/apps/common/routes/patch.py#L19)  | Delete a notebook                       |
+| GET          | /api/namespaces/{namespace}/pvcs                 | [/api/namespaces/\<namespace\>/pvc](https://github.com/kubeflow/kubeflow/blob/v1.6.0/components/crud-web-apps/jupyter/backend/apps/common/routes/get.py#L15)                    | List `PVC`s                             |
+| DELETE       | /api/namespaces/{namespace}/pvcs/{pvc}           | [/api/namespaces/\<namespace\>/pvcs/<pvc>](https://github.com/kubeflow/kubeflow/blob/v1.6.0/components/crud-web-apps/volumes/backend/apps/default/routes/delete.py#L11)         | Delete a `PVC`                          |
+| GET          | /api/namespaces/{namespace}/poddefaults          | [/api/namespaces/\<namespace\>/poddefaults](https://github.com/kubeflow/kubeflow/blob/v1.6.0/components/crud-web-apps/jupyter/backend/apps/common/routes/get.py#L25)            | Get `PodDefault`s for a given namespace |

Taskfile.yml

@@ -75,7 +75,13 @@ tasks:
       desc: Push the built container image
       cmds:
       - docker push {{.CONTAINER_IMAGE}}
-
+  go:dev:
+    desc: run go code locally for development
+    cmds:
+      - go run . {{.CLI_ARGS}}
+      - sleep 1
+    sources:
+      - "*.go"
   # Source: https://github.com/arduino/tooling-project-assets/blob/main/workflow-templates/assets/go-task/Taskfile.yml
   go:build:
     desc: Build the Go code
@@ -163,3 +169,14 @@ tasks:
     desc: stop knative environment
     cmds:
     - "k3d cluster stop {{.CLUSTER_NAME}}"
+
+  env:
+    prefix: ⚙ > env  
+    desc: setup dev environment
+    cmds:
+    - echo "KF_NAMESPACE=">>.env
+    - echo "KF_USER_ID=">>.env
+    - echo "KF_NOTEBOOK_NAME=swag">>.env
+    - echo "KF_PVC_NAME=swag">>.env
+    sources:
+      - .env

access.go

@@ -74,7 +74,7 @@ func (s *server) checkAccess(subjectAccessReviewTemplate authorizationv1.Subject
 
 		// If the user is not permitted, log and return the error and do not process the request
 		if !resp.Status.Allowed {
-			msg := fmt.Sprintf("User %s is not permitted to %s %s.%s.%s for namespace: %s", sar.Spec.User, sar.Spec.ResourceAttributes.Verb, sar.Spec.ResourceAttributes.Group, sar.Spec.ResourceAttributes.Version, sar.Spec.ResourceAttributes.Resource, sar.Spec.ResourceAttributes.Namespace)
+			msg := fmt.Sprintf("User %s is not permitted to %s %s.%s.%s for namespace: '%s'", sar.Spec.User, sar.Spec.ResourceAttributes.Verb, sar.Spec.ResourceAttributes.Group, sar.Spec.ResourceAttributes.Version, sar.Spec.ResourceAttributes.Resource, sar.Spec.ResourceAttributes.Namespace)
 
 			log.Println(msg)
 

main.go

@@ -11,7 +11,6 @@ import (
 	"os/signal"
 	"path"
 	"path/filepath"
-	"sync"
 	"time"
 
 	kubeflowv1 "github.com/StatCan/kubeflow-apis/apis/kubeflow/v1"
@@ -54,8 +53,6 @@ type clientsets struct {
 }
 
 type server struct {
-	mux sync.Mutex
-
 	Config Configuration
 
 	clientsets clientsets
@@ -126,7 +123,7 @@ func main() {
 		log.Fatal(err)
 	}
 
-	err = s.setupListers(gctx)
+	_ = s.setupListers(gctx)
 
 	// Generate the Gorilla Mux router
 	router := mux.NewRouter()
@@ -163,8 +160,8 @@ func main() {
 		Spec: authorizationv1.SubjectAccessReviewSpec{
 			ResourceAttributes: &authorizationv1.ResourceAttributes{
 				Group:    corev1.SchemeGroupVersion.Group,
-				Verb:     "list",
-				Resource: "pods",
+				Verb:     "get",
+				Resource: "namespaces",
 				Version:  corev1.SchemeGroupVersion.Version,
 			},
 		},

thunder-tests/thunderActivity.json

@@ -0,0 +1 @@
+[]

thunder-tests/thunderCollection.json

@@ -0,0 +1,22 @@
+[
+  {
+    "_id": "bf518063-1214-41a0-a602-702e12d98039",
+    "colName": "Golang Kubeflow",
+    "created": "2022-09-23T13:47:44.945Z",
+    "sortNum": 10000,
+    "folders": [],
+    "settings": {
+      "headers": [
+        {
+          "name": "kubeflow-userid",
+          "value": "{{KF_USER_ID}}"
+        }
+      ],
+      "tests": [],
+      "options": {
+        "baseUrl": "http://localhost:5000"
+      },
+      "envId": "0d8a7513-38dd-4e46-93b3-2c0ccb2ee056"
+    }
+  }
+]

thunder-tests/thunderEnvironment.json

@@ -0,0 +1,12 @@
+[
+  {
+    "_id": "0d8a7513-38dd-4e46-93b3-2c0ccb2ee056",
+    "name": "golang-backend",
+    "default": true,
+    "sortNum": 10000,
+    "created": "2022-09-23T13:47:57.807Z",
+    "modified": "2022-09-23T13:48:16.638Z",
+    "data": [],
+    "envFile": "../.env"
+  }
+]