Improve test matrix comprehensiveness #54
Conversation
RomainMuller
force-pushed
the
romain.marcadier/add-coverage-report
branch
3 times, most recently
from
8639479 to
8a59852
Compare
RomainMuller
changed the title
RomainMuller
force-pushed
the
romain.marcadier/add-coverage-report
branch
5 times, most recently
from
6dcfdd0 to
0608120
Compare
RomainMuller
changed the title
RomainMuller
force-pushed
the
romain.marcadier/add-coverage-report
branch
from
0608120 to
051966f
Compare
RomainMuller
force-pushed
the
romain.marcadier/add-coverage-report
branch
from
051966f to
7bae7f5
Compare
| docker exec -i gha-${{ github.run_id }} \ | ||
| go run gotest.tools/gotestsum@latest -- \ | ||
| -v -count=10 -shuffle=on -tags='${{ matrix.go-tags }}' \ | ||
| ./... |
There was a problem hiding this comment.
I usually prefer a multi-stage Dockerfile where the final container image only contains the produced binary and its dependencies, without all of the dev tools. This allows us to find setup limitations/issues.
OTOH, it would be valid to say this test's role is just to execute the tests on the given containers, regardless of the setup/onboarding use-cases.
There was a problem hiding this comment.
But what I also think is that setup cases should be tested in the appsec test app https://github.com/DataDog/appsec-go-test-app/blob/main/Dockerfile at the very least
There was a problem hiding this comment.
I'd need to figure out how to pre-build test binaries and correctly execute them then... but I'm not against the idea. We'd then also want bare-metal tests to have separated build & run jobs (where run only has the test binary available).
I largely re-made this from scratch and initially wanted to combine all flavors in a single job (but that is terribly inconvenient to mix docker & bare metal scenarios) |
Run tests with a comprehensive cartesian product of all configurations:
1.19,1.20,1.21datadog.no_waf,go1.222,datadog.no_waf,go1.22This uncovered compatibility gaps in the matrix, which this PR fixes as well:
datadog.no_wafFootnotes
Multiple architectures are only tested for linux platforms due to technical limitations ↩
go1.22is a stand-in for future go releases for whichpuregosupport needs checking ↩