[CI] Use monorepo-diff BK plugin to conditionally run unit tests

[ycombinator]

What does this PR do?

This PR uses the Buildkite monorepo-diff plugin to run unit tests only if relevant files have changed.

Why is it important?

To make CI runs more efficient.

Testing

TODO

[mergify]

This pull request does not have a backport label. Could you fix it @ycombinator? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-./d./d is the label that automatically backports to the 8./d branch. /d is the digit
• backport-active-all is the label that automatically backports to all active branches.
• backport-active-8 is the label that automatically backports to all active minor branches for the 8 major.
• backport-active-9 is the label that automatically backports to all active minor branches for the 9 major.

[ycombinator]

@v1v Could you help me figure out what's going wrong in https://buildkite.com/elastic/elastic-agent/builds/30064? Thanks!

[v1v]

as far as I see, groups are only available from 1.4.0:

• buildkite-plugins/monorepo-diff-buildkite-plugin@a946f0e

at least that' when it's shown in the docs.

---

I normally recommend starting with simple use cases and then iterating.

For instance:

• Create a new monorepo step and start adding paths and steps, so you can test step by step
• then when you get comfortable with the setup, you can replace it.

Otherwise, in my experience, it's hard to debug errors in Buildkite related to the syntax

Another alternative could be to contact BK support; maybe they have some insights I'm not aware of. It definitely isn't easy to find issues in the BK pipeline :/

[elasticmachine]

💛 Build succeeded, but was flaky

• Buildkite Build
• Commit: 8e0ce92

Failed CI Steps

• x86_64:sudo:rpm

History

• 💔 Build #30064 failed b58bc8c
• 💔 Build #30059 failed cc1398f
• 💔 Build #30058 failed 4d6e2cb

cc @ycombinator

[ycombinator]

Thanks for the advice, @v1v. I took your suggestion and simplified the pipeline changes to only have the monorepo plugin apply to a single step: Unit tests - Ubuntu 22.04.

With that, I first made a commit with a change to .go file and, as expected, the Unit tests - Ubuntu 22.04 step ran (see here and here).

Then, I reverted that commit, so the only files in this PR's diff were non-**/*.go files. As such, I was expecting that the Unit tests - Ubuntu 22.04 step would not run but it looks like it did (see here and here).

What am I missing? Thanks.

[v1v]

What am I missing? Thanks.

I'd recommend debugging what the diff command produces first:

    steps:
      - label: "Debug diff"
        command: "git diff --name-only HEAD~1"

      - label: "Unit tests - Ubuntu 22.04"
        key: "unit-tests-2204"
        plugins:
          - monorepo-diff#v1.2.0:
              diff: "git diff --name-only HEAD~1"
              interpolation: false

Or even using https://github.com/buildkite-plugins/monorepo-diff-buildkite-plugin/blob/main/README.md#log_level-optional

Then I'd recommend to see how other projects use it:

For instance, in Beats:

• https://github.com/elastic/beats/blob/f8a4087a3e1635bb65cdb3c885ecbad0d19cfd12/.buildkite/pipeline.yml#L9

Or look for the details that the Platform Productivity team has done with the 'self-service' automation. as far as I see they use the below snippets (pipeline and diff command)

---
steps:
  - label: "Triggering pipelines"
    plugins:
      monorepo-diff#v1.5.1:
        diff: ".buildkite/diff ${BUILDKITE_COMMIT}"
        wait: true
        watch:
          - path: "renovate.json"
            config:
              label: "Verify Renovate configuration"
              command: "renovate-config-validator"
              agents:
                image: "...."
          - path: "catalog-info.yaml"
            config:
              command: "/agent/check-catalog-info.sh"
              agents:
                image: "...."
          - default:
              config:
                label: ":pipeline: Upload default Pipeline"
                command: "buildkite-agent pipeline upload .buildkite/default-pipeline.yml"

#!/bin/bash
# Produces a list of changed files between two commits (works for merges and
# regular commits).
# Used in conjunction with the monorepo-diff-buildkite-plugin to determine
# which pipelines to upload/trigger based on the files changed.

[ $# -lt 1 ] && { echo "argument is missing."; exit 1; }

COMMIT=$1

HEAD_BRANCH=$(git remote show origin | awk '/HEAD branch/ {print $NF}')
BRANCH_POINT_COMMIT=$(git merge-base "$HEAD_BRANCH" "$COMMIT")

if [ "$BUILDKITE_BRANCH" == "$HEAD_BRANCH" ]; then
	echo "diff between $COMMIT and HEAD~1"
        git diff --raw HEAD~1 | awk '{print $6; if($7) {print $7}}'
else
	echo "diff between $COMMIT and $BRANCH_POINT_COMMIT"
        git diff --raw "$COMMIT".."$BRANCH_POINT_COMMIT" | awk '{print $6; if($7) {print $7}}'
fi

---

https://github.com/buildkite/monorepo-example#buildkite-monorepo-example is the official docs from BK about using the plugin..