Skip to content

Commit c7c5f97

Browse files
authored
[CI] Enable Develocity test retry selectively (#21479)
The `sbt-develocity` plugin offers the ability to retry failing tests, and to detect flaky ones. A flaky test is marked as flaky in the Develocity test report, which will help us track them across many CI executions. See for example the flaky test chart in the [Pekko dashboard](https://ge.apache.org/scans/tests?search.relativeStartTime=P90D&search.rootProjectNames=Pekko&search.tags=not:CI&search.timeZoneId=Europe%2FZurich). This PR contains the following configuration: - retry each failing test once (to be adjusted if needed) - don't retry if more than 10 tests fail in the current test run (to be adjusted if needed) - fail the build if a test is flaky - disable test retry in a predefined set of test classes: the compilation test classes. In those classes, we don't have enough granularity for the test retry to be meaningful, because each test is responsible for compiling many independent files. @dotta @c00ler @Duhemm @lrytz
2 parents e9e046d + b4dfab9 commit c7c5f97

File tree

1 file changed

+19
-0
lines changed

1 file changed

+19
-0
lines changed

project/Build.scala

+19
Original file line numberDiff line numberDiff line change
@@ -276,6 +276,18 @@ object Build {
276276
val config = develocityConfiguration.value
277277
val buildScan = config.buildScan
278278
val buildCache = config.buildCache
279+
// disable test retry on compilation test classes
280+
val noRetryTestClasses = Set(
281+
"dotty.tools.dotc.BestEffortOptionsTests",
282+
"dotty.tools.dotc.CompilationTests",
283+
"dotty.tools.dotc.FromTastyTests",
284+
"dotty.tools.dotc.IdempotencyTests",
285+
"dotty.tools.dotc.ScalaJSCompilationTests",
286+
"dotty.tools.dotc.TastyBootstrapTests",
287+
"dotty.tools.dotc.coverage.CoverageTests",
288+
"dotty.tools.dotc.transform.PatmatExhaustivityTest",
289+
"dotty.tools.repl.ScriptedTests"
290+
)
279291
config
280292
.withProjectId(ProjectId("scala3"))
281293
.withServer(config.server.withUrl(Some(url("https://develocity.scala-lang.org"))))
@@ -293,6 +305,13 @@ object Build {
293305
.withLocal(buildCache.local.withEnabled(false))
294306
.withRemote(buildCache.remote.withEnabled(false))
295307
)
308+
.withTestRetryConfiguration(
309+
config.testRetryConfiguration
310+
.withFlakyTestPolicy(FlakyTestPolicy.Fail)
311+
.withMaxRetries(1)
312+
.withMaxFailures(10)
313+
.withClassesFilter((className, _) => !noRetryTestClasses.contains(className))
314+
)
296315
}
297316
)
298317

0 commit comments

Comments
 (0)