Skip to content

Upgrade to 1.24 #45

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 1 commit into from
Closed

Upgrade to 1.24 #45

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .env.tools
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1 +1 @@
export TOOLS_BUILDNUMBER=20241108.3
export TOOLS_BUILDNUMBER=20250415.2
2 changes: 1 addition & 1 deletion .github/dependabot.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,5 +7,5 @@ updates:
schedule:
interval: "weekly"
day: "wednesday"
time: "03:00"
time: "03:15"
timezone: "Europe/London"
8 changes: 6 additions & 2 deletions .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,18 +10,21 @@ jobs:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4

- name: Setup go-task
uses: arduino/setup-task@v2
with:
version: 3.x
repo-token: ${{ secrets.GITHUB_TOKEN }}

- name: Setup Go
uses: actions/setup-go@v5
with:
go-version: '1.23.x'
go-version: '1.24.x'
token: ${{ secrets.GITHUB_TOKEN }}
cache-dependency-path: |
datatrails-common-api/go.sum

- name: Setup protobuf
uses: arduino/setup-protoc@v3
with:
Expand All @@ -30,6 +33,7 @@ jobs:
- name: Setup protoc-gen-doc
run: |
go install github.com/pseudomuto/protoc-gen-doc/cmd/[email protected]
which protoc-gen-doc

- name: Generate, Test and Export
run: |
Expand All @@ -42,7 +46,7 @@ jobs:

- name: Show exports
working-directory: exported/
run: find . -type d
run: find . -type f

- name: Publish release artifacts
if: ${{ github.event_name == 'release' }}
Expand Down
71 changes: 40 additions & 31 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,19 +2,48 @@

Common public api definitions for the DataTrails platform

## Finding and including proto files for depdendecnies
## Finding and including proto files for dependencies

tools/go.mod is the source of truth for all proto providing dependencies. That file alone specifies both the upstream version we are using *and* is used, via go install, to make the .proto files available locally
go.mod is the source of truth for all proto providing dependencies. That file alone specifies both the upstream version we are using *and* is used, via go get -tool, to make the .proto files available locally

This corresponds to the practice recommended by grpc-gateway and elsewhere
This corresponds to the practice recommended by grpc-gateway and elsewhere.

1. *** ALWAYS *** Use tools/go.mod to specify the dependency version.
2. Add the package to the `go install` command in the apis:preflight task
3. If necessary, add a var for the new path in any-api **and** then add a reference to that var in the PROTO_INC var.
1. Execute the `go get -tool` command to add the package to the go.mod file. This is a once-off command that simply
sets up go.mod to manages all tools.
2. If necessary, add a var for the new path in any-api **and** then add a reference to that var in the PROTO_INC var.
3. Executing 'go install tool' is then sufficient to install required tools and the only command required in workflows.

Following this practice removes the need for dual maintenance of dependency versions in the builder image. It also produces a build cycle that is significantly faster.
## Using the go tool commands

Cross repository builds in docker while using go.work to refer to locally modified sources don't work. And this setup is essential for an efficient workflow.
These commands are NOT executed as part of the dev workflow.

Add tools to the go.mod file:

```
go get -tool github.com/grpc-ecosystem/grpc-gateway/v2/[email protected]
go get -tool github.com/grpc-ecosystem/grpc-gateway/v2/[email protected]
go get -tool google.golang.org/protobuf/cmd/[email protected]
go get -tool google.golang.org/grpc/cmd/[email protected]
go get -tool github.com/envoyproxy/[email protected]
```

Inspect the go.mod file and show the 'tool' section:

```
tool (
github.com/envoyproxy/protoc-gen-validate
github.com/grpc-ecosystem/grpc-gateway/v2/protoc-gen-grpc-gateway
github.com/grpc-ecosystem/grpc-gateway/v2/protoc-gen-openapiv2
google.golang.org/grpc/cmd/protoc-gen-go-grpc
google.golang.org/protobuf/cmd/protoc-gen-go
)
```

Show installed tools

```
go tool
```

## bootstrap proto files

Expand All @@ -24,29 +53,13 @@ For this reason we curl the proto's and make them available in our aggregate pro

## Workflow for updating common apis

### Ensure the go tool chain is setup on your host
### Ensure tools are setup on your host

It is inordinately cumbersome to set this all up for docker. It can't be
achieved without imposing significant inneficiencies in the workflow. For any
engineer working on the api's it is reasonable to expect they can install the
correct version of go for their host, and likely already have.
This repo uses the go-task tool and also the go toolchain. The dev workflow is described below.

### Use an avid/src/go.work

First point avid at a clone of this repo so that it's imports will resolve
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you explain how we build against locally changed sources without committing and pushing ?

I don't know if go.work is necessar nor not post this change. But I do want to be able to build forestrie, avid and avid-events against a locally modified common-api.

With this change (and with particular reference to the deleted instructions) how do I do that ?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I dont think anything has changed in this regard

I exercised the workflow commands in this README and all just worked.

The only change is how tools are managed not how they are used

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok, can you please test that and show some evidence of it in the PR. but sounds good thanks

directly against the sources in `./api`

### Familiarize with the workflow sub tasks

The general use entry points all use Docker and do not support efficiently and
iteratively developing avid and other dependents against a local clone of this
repo.

#### For this repo

You can do everything in the builder image using `task all`. This uses the
blessed builder image and requires no host setup other than docker.

If you are iterating it is a lot more efficient to use the sub tasks, because they all run on your host.

`task` `apis:bootstrap` -> `task:apis:generate` -> `task:apis:test` -> `task:apis:export`
Expand All @@ -68,15 +81,11 @@ If you want to iterate on *just* the helper go code and there tests, do one roun

Then just iterate using `task apis:generate` and `task apis:test`

#### For avid
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I assume your not saying "we don't support this for avid" rather you are saying it works for any consuming package ? Do I have that right ?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes - the deleted text was written when we only had avid
its any repo now

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
#### For avid
#### For any repo

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

will do


See the README.md in avid/src/api/README.md

##### Build one api against locally cloned go-datatrails-common-api

The protos can be included exactly as they are found from a clone of go-datatrails-common-api. *Without* needing to generate, build or export. Eg,

task apis:assetsv2-api \
DATATRAILS_COMMON_API="../../go-datatrails-common-api"

It is necessary however to run `task apis:bootsrap` after cloning go-datatrails-common
It is necessary however to run `task apis:bootstrap` after cloning go-datatrails-common-api.
4 changes: 4 additions & 0 deletions datatrails-common-api/.gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1 +1,5 @@
marshalers/
go.work
*.go
*.json
*.storage
5 changes: 0 additions & 5 deletions datatrails-common-api/assets/v2/assets/.gitignore
View file
Open in desktop

This file was deleted.

5 changes: 0 additions & 5 deletions datatrails-common-api/attribute/v2/attribute/.gitignore
View file
Open in desktop

This file was deleted.

5 changes: 0 additions & 5 deletions datatrails-common-api/caps/v1/caps/.gitignore
View file
Open in desktop

This file was deleted.

5 changes: 0 additions & 5 deletions datatrails-common-api/filter/v1/filter/.gitignore
View file
Open in desktop

This file was deleted.

23 changes: 16 additions & 7 deletions datatrails-common-api/go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,19 +3,18 @@ module github.com/datatrails/go-datatrails-common-api-gen
// Note: the go code is hosted in github.com/datatrails/go-datatrails-common-api-gen
// hence the module name

go 1.23.0
go 1.24

// This allows this module to operate as tho it were the generated module. This
// allows us to manage the proto tool dependencies via this go.mod. This go.mod
// is also used as the go.mod for the generated package.
// replace github.com/datatrails/go-datatrails-common-api-gen => ./
replace github.com/datatrails/go-datatrails-common-api-gen => ./

require (
github.com/datatrails/go-datatrails-common v0.28.0
github.com/datatrails/go-datatrails-common v0.29.0
github.com/envoyproxy/protoc-gen-validate v1.2.1
github.com/google/uuid v1.6.0
github.com/grpc-ecosystem/grpc-gateway/v2 v2.23.0
github.com/lyft/protoc-gen-star/v2 v2.0.4-0.20230330145011-496ad1ac90a4
github.com/stretchr/testify v1.10.0
go.mongodb.org/mongo-driver v1.17.3
google.golang.org/genproto/googleapis/api v0.0.0-20250106144421-5f5ef82da422
Expand All @@ -34,9 +33,19 @@ require (
go.uber.org/multierr v1.11.0 // indirect
go.uber.org/zap v1.27.0 // indirect
golang.org/x/mod v0.17.0 // indirect
golang.org/x/net v0.34.0 // indirect
golang.org/x/sys v0.30.0 // indirect
golang.org/x/text v0.22.0 // indirect
golang.org/x/net v0.37.0 // indirect
golang.org/x/sys v0.31.0 // indirect
golang.org/x/text v0.23.0 // indirect
golang.org/x/tools v0.21.1-0.20240508182429-e35e4ccd0d2d // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20250115164207-1a7da9e5054f // indirect
google.golang.org/grpc/cmd/protoc-gen-go-grpc v1.5.1 // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect
)

tool (
github.com/envoyproxy/protoc-gen-validate
github.com/grpc-ecosystem/grpc-gateway/v2/protoc-gen-grpc-gateway
github.com/grpc-ecosystem/grpc-gateway/v2/protoc-gen-openapiv2
google.golang.org/grpc/cmd/protoc-gen-go-grpc
google.golang.org/protobuf/cmd/protoc-gen-go
)
Loading
Loading