Upgrade to 1.24

Remove rotten protoc-gen-doc
Upgrade tools version to correspond to those in general codebase.
Use new avid-tools

Add .gitattributes
Add dependabot.yml

Install protoc-gen-doc binary in ci workflow

Use new go get -tool command to set tools and remove tools.go
Use 'go install tool' to install all tools.
Move all helper go code into separate directory and copy over
when appropriate otherwise installing tools fails with unknown
import.

AB#10624

Signed-off-by: Paul Hewlett <[email protected]>

Author: eccles

.env.tools

@@ -1 +1 @@
-export TOOLS_BUILDNUMBER=20241108.3
+export TOOLS_BUILDNUMBER=20250403.6

.gitattributes

@@ -0,0 +1,2 @@
+**/go.sum              -diff -merge
+**/go.sum              linguist-generated=true

.github/dependabot.yml

@@ -0,0 +1,22 @@
+# refer: https://github.com/dependabot/dependabot-core/blob/main/.github/dependabot.yml
+version: 2
+updates:
+  # Maintain dependencies for GitHub Actions
+  - package-ecosystem: "github-actions"
+    directory: "/"
+    schedule:
+      # Check for updates managed by Composer once a week
+      interval: "weekly"
+      day: "wednesday"
+      time: "03:00"
+      timezone: "Europe/London"
+
+  # Maintain dependencies for Golang
+  - package-ecosystem: "gomod"
+    directory: "/"
+    schedule:
+      # Check for updates managed by Composer once a week
+      interval: "weekly"
+      day: "wednesday"
+      time: "03:00"
+      timezone: "Europe/London"

.github/workflows/ci.yml

@@ -10,30 +10,40 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v4
+
       - name: Setup go-task
         uses: arduino/setup-task@v1
         with:
           version: 3.x
           repo-token: ${{ secrets.GITHUB_TOKEN }}
+
       - name: Setup Go
         uses: actions/setup-go@v4
         with:
-          go-version: '1.22.x'
+          go-version: '1.24.x'
           token: ${{ secrets.GITHUB_TOKEN }}
           cache-dependency-path: |
             datatrails-common-api/go.sum
+
       - name: Setup protobuf
         uses: arduino/setup-protoc@v2
         with:
           version: "24.3"
           repo-token: ${{ secrets.GITHUB_TOKEN }}
+
+      - name: Setup protoc-gen-doc
+        run: |
+          # download protoc-gen-doc as binary
+          go install github.com/pseudomuto/protoc-gen-doc/cmd/[email protected]
+          which protoc-gen-doc
+
       - name: Generate, Test and Export
         run: |
           task all
 
       - name: Show exports
         working-directory: exported/
-        run: find . -type d
+        run: find . -type f
 
       - name: Publish release artifacts
         if: ${{ github.event_name == 'release' }}

README.md

@@ -2,19 +2,48 @@
 
 Common public api definitions for the DataTrails platform
 
-## Finding and including proto files for depdendecnies
+## Finding and including proto files for dependencies
 
-tools/go.mod is the source of truth for all proto providing dependencies. That file alone specifies both the upstream version we are using *and* is used, via go install, to make the .proto files available locally
+go.mod is the source of truth for all proto providing dependencies. That file alone specifies both the upstream version we are using *and* is used, via go get -tool, to make the .proto files available locally
 
-This corresponds to the practice recommended by grpc-gateway and elsewhere
+This corresponds to the practice recommended by grpc-gateway and elsewhere.
 
-1. *** ALWAYS *** Use tools/go.mod to specify the dependency version.
-2. Add the package to the `go install` command in the apis:preflight task
-3. If necessary, add a var for the new path in any-api **and** then add a reference to that var in the PROTO_INC var.
+1. Execute the `go get -tool` command to add the package to the go.mod file. This is a once-off command that simply
+   sets up go.mod to manages all tools.
+2. If necessary, add a var for the new path in any-api **and** then add a reference to that var in the PROTO_INC var.
+3. Executing 'go install tool' is then sufficient to install required tools and the only command required in workflows.
 
-Following this practice removes the need for dual maintenance of dependency versions in the builder image. It also produces a build cycle that is significantly faster.
+## Using the go tool commands
 
-Cross repository builds in docker while using go.work to refer to locally modified sources don't work. And this setup is essential for an efficient workflow.
+These commands are NOT executed as part of the dev workflow. 
+
+Add tools to the go.mod file:
+
+```
+    go get -tool github.com/grpc-ecosystem/grpc-gateway/v2/[email protected]
+    go get -tool github.com/grpc-ecosystem/grpc-gateway/v2/[email protected]
+    go get -tool google.golang.org/protobuf/cmd/[email protected]
+    go get -tool google.golang.org/grpc/cmd/[email protected]
+    go get -tool github.com/envoyproxy/[email protected]
+```
+
+Inspect the go.mod file and show the 'tool' section:
+
+```
+tool (
+	github.com/envoyproxy/protoc-gen-validate
+	github.com/grpc-ecosystem/grpc-gateway/v2/protoc-gen-grpc-gateway
+	github.com/grpc-ecosystem/grpc-gateway/v2/protoc-gen-openapiv2
+	google.golang.org/grpc/cmd/protoc-gen-go-grpc
+	google.golang.org/protobuf/cmd/protoc-gen-go
+)
+```
+
+Show installed tools
+
+```
+    go tool
+```
 
 ## bootstrap proto files
 
@@ -24,29 +53,13 @@ For this reason we curl the proto's and make them available in our aggregate pro
 
 ## Workflow for updating common apis
 
-### Ensure the go tool chain is setup on your host
+### Ensure tools are setup on your host
 
-It is inordinately cumbersome to set this all up for docker. It can't be
-achieved without imposing significant inneficiencies in the workflow. For any
-engineer working on the api's it is reasonable to expect they can install the
-correct version of go for their host, and likely already have.
+This repo uses the go-task tool and also the go toolchain. The dev workflow is described below.
 
-### Use an avid/src/go.work
-
-First point avid at a clone of this repo so that it's imports will resolve
-directly against the sources in `./api`
-
-### Familiarize with the workflow sub tasks
-
-The general use entry points all use Docker and do not support efficiently and
-iteratively developing avid and other dependents against a local clone of this
-repo.
 
 #### For this repo
 
-You can do everything in the builder image using `task all`. This uses the
-blessed builder image and requires no host setup other than docker.
-
 If you are iterating it is a lot more efficient to use the sub tasks, because they all run on your host.
 
 `task` `apis:bootstrap` -> `task:apis:generate` -> `task:apis:test` -> `task:apis:export`
@@ -68,15 +81,11 @@ If you want to iterate on *just* the helper go code and there tests, do one roun
 
 Then just iterate using `task apis:generate` and `task apis:test`
 
-#### For avid
-
-See the README.md in avid/src/api/README.md
-
 ##### Build one api against locally cloned go-datatrails-common-api
 
 The protos can be included exactly as they are found from a clone of go-datatrails-common-api. *Without* needing to generate, build or export. Eg,
 
     task apis:assetsv2-api \
         DATATRAILS_COMMON_API="../../go-datatrails-common-api"
 
-It is necessary however to run `task apis:bootsrap` after cloning go-datatrails-common
+It is necessary however to run `task apis:bootstrap` after cloning go-datatrails-common-api.

datatrails-common-api/.gitignore

@@ -1 +1,5 @@
+marshalers/
 go.work
+*.go
+*.json
+*.storage

datatrails-common-api/assets/v2/assets/.gitignore

@@ -3,54 +3,49 @@ module github.com/datatrails/go-datatrails-common-api-gen
 // Note: the go code is hosted in github.com/datatrails/go-datatrails-common-api-gen
 // hence the module name
 
-go 1.22
+go 1.24
 
 // This allows this module to operate as tho it were the generated module. This
 // allows us to manage the proto tool dependencies via this go.mod. This go.mod
 // is also used as the go.mod for the generated package.
-// replace github.com/datatrails/go-datatrails-common-api-gen => ./
+replace github.com/datatrails/go-datatrails-common-api-gen => ./
 
 require (
-	github.com/datatrails/go-datatrails-common v0.18.1
-	github.com/envoyproxy/protoc-gen-validate v1.0.4
+	github.com/datatrails/go-datatrails-common v0.26.0
+	github.com/envoyproxy/protoc-gen-validate v1.2.1
 	github.com/google/uuid v1.6.0
-	github.com/grpc-ecosystem/grpc-gateway/v2 v2.20.0
-	github.com/lyft/protoc-gen-star/v2 v2.0.3
-	github.com/pseudomuto/protoc-gen-doc v1.5.1
-	github.com/spf13/afero v1.10.0
-	github.com/stretchr/testify v1.9.0
-	go.mongodb.org/mongo-driver v1.12.1
-	google.golang.org/genproto/googleapis/api v0.0.0-20240701130421-f6361c86f094
-	google.golang.org/grpc v1.65.0
-	google.golang.org/grpc/cmd/protoc-gen-go-grpc v1.3.0
-	google.golang.org/protobuf v1.34.2
-	gopkg.in/yaml.v3 v3.0.1
+	github.com/grpc-ecosystem/grpc-gateway/v2 v2.23.0
+	github.com/stretchr/testify v1.10.0
+	go.mongodb.org/mongo-driver v1.17.3
+	google.golang.org/genproto/googleapis/api v0.0.0-20241202173237-19429a94021a
+	google.golang.org/grpc v1.70.0
+	google.golang.org/protobuf v1.36.6
 )
 
 require (
-	github.com/Masterminds/goutils v1.1.1 // indirect
-	github.com/Masterminds/semver v1.5.0 // indirect
-	github.com/Masterminds/sprig v2.22.0+incompatible // indirect
 	github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc // indirect
-	github.com/gogo/protobuf v1.3.2 // indirect
-	github.com/golang/protobuf v1.5.4 // indirect
-	github.com/huandu/xstrings v1.4.0 // indirect
 	github.com/iancoleman/strcase v0.3.0 // indirect
-	github.com/imdario/mergo v0.3.16 // indirect
-	github.com/mitchellh/copystructure v1.2.0 // indirect
-	github.com/mitchellh/reflectwalk v1.0.2 // indirect
-	github.com/mwitkow/go-proto-validators v0.3.2 // indirect
+	github.com/lyft/protoc-gen-star/v2 v2.0.4-0.20230330145011-496ad1ac90a4 // indirect
 	github.com/opentracing/opentracing-go v1.2.0 // indirect
 	github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 // indirect
-	github.com/pseudomuto/protokit v0.2.1 // indirect
 	github.com/rogpeppe/go-internal v1.6.1 // indirect
+	github.com/spf13/afero v1.10.0 // indirect
 	go.uber.org/multierr v1.11.0 // indirect
 	go.uber.org/zap v1.27.0 // indirect
-	golang.org/x/crypto v0.25.0 // indirect
 	golang.org/x/mod v0.17.0 // indirect
-	golang.org/x/net v0.27.0 // indirect
-	golang.org/x/sys v0.22.0 // indirect
-	golang.org/x/text v0.16.0 // indirect
+	golang.org/x/net v0.37.0 // indirect
+	golang.org/x/sys v0.31.0 // indirect
+	golang.org/x/text v0.23.0 // indirect
 	golang.org/x/tools v0.21.1-0.20240508182429-e35e4ccd0d2d // indirect
-	google.golang.org/genproto/googleapis/rpc v0.0.0-20240701130421-f6361c86f094 // indirect
+	google.golang.org/genproto/googleapis/rpc v0.0.0-20241209162323-e6fa225c2576 // indirect
+	google.golang.org/grpc/cmd/protoc-gen-go-grpc v1.5.1 // indirect
+	gopkg.in/yaml.v3 v3.0.1 // indirect
+)
+
+tool (
+	github.com/envoyproxy/protoc-gen-validate
+	github.com/grpc-ecosystem/grpc-gateway/v2/protoc-gen-grpc-gateway
+	github.com/grpc-ecosystem/grpc-gateway/v2/protoc-gen-openapiv2
+	google.golang.org/grpc/cmd/protoc-gen-go-grpc
+	google.golang.org/protobuf/cmd/protoc-gen-go
 )