Skip to content

Order of test transforms influences flakiness of tests #903

@Gedochao

Description

@Gedochao

This works:

//> using scala 2.13
//> using dep org.scalameta::munit:1.1.0

import munit.Tag

class CustomFlakyTests extends munit.FunSuite {
  override def munitFlakyOK: Boolean = true

  private val CustomFlakyTag = new Tag("CustomFlaky")

  override def munitTestTransforms: List[TestTransform] =
    List(
      new TestTransform(
        "Custom Flaky",
        { test =>
          if (test.tags.contains(CustomFlakyTag)) test.tag(munit.Flaky)
          else test
        }
      )
    ) ++ super.munitTestTransforms

  test("fail".tag(CustomFlakyTag)) {
    assert(false)
  }
}

This doesn't:

//> using scala 2.13
//> using dep org.scalameta::munit:1.1.0

import munit.Tag

class CustomFlakyTests extends munit.FunSuite {
  override def munitFlakyOK: Boolean = true

  private val CustomFlakyTag = new Tag("CustomFlaky")

  override def munitTestTransforms: List[TestTransform] =
    super.munitTestTransforms ++ List(
      new TestTransform(
        "Custom Flaky",
        { test =>
          if (test.tags.contains(CustomFlakyTag)) test.tag(munit.Flaky)
          else test
        }
      )
    )

  test("fail".tag(CustomFlakyTag)) {
    assert(false)
  }
}

Took me a good bit to figure out it's just the matter of order of transforms, and by the time I add the Flaky tag in mine, the tests have already been evaluated.
At least a warning would go a long way here, I think.
Discovered while working on: VirtusLab/scala-packager#213

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions