pytester: Representation of the full collection tree, for testing collection hooks & plugins

[MusicalNinjaDad]

1. Am I reinventing the wheel here? and if not
2. is there an interest in a (more complete) version to be included in pytest?

I'm starting to put a plugin together which will include collection hooks. The How-To on Working with non-python tests suggests "While developing your custom test collection and execution it’s also interesting to just look at the collection tree:"

I'd rather have tests do this both during development and later for regression.

Note: I know this is incomplete, buggy and largely broken - it's not even complete for my own specific purposes...

To that end I put together:

class CollectionTree:
    """
    A (top-down) tree of pytest collection Nodes.

    Designed to enable testing the results of collection plugins via:
    ```
    assert CollectionTree.from_items(pytester.genitems([...])) == CollectionTree.from_dict({...})
    ```

    WARNING: Currently only handles a single tree (one top-level node)
    """

    @classmethod
    def from_items(cls, items: list[pytest.Item]):
        """Create a CollectionTree from a list of collection items, as returned by `pytester.genitems()`."""
        converteditems = [cls(node=item, children=None) if not isinstance(item, cls) else item for item in items]
        parents = {item.node.parent for item in converteditems}
        items_byparent = {parent: [item for item in converteditems if item.node.parent == parent] for parent in parents}
        for parent, children in items_byparent.items():
            if parent.parent is None:  # Top of tree
                return cls(node=parent, children=children)
            return cls(
                node=parent.parent,
                children=[cls(node=parent, children=children)],
            )
        msg = "Items cannot be empty."
        raise ValueError(msg)

    @classmethod
    def from_dict(cls, tree: dict[tuple[str, type], dict | None]):
        """
        Create a dummy CollectionTree from a dict of dicts with following format:

        ```
        {(str: name, type: Nodetype):
            (str: name, type: Nodetype): {
                (str: name, type: Nodetype): None,
                (str: name, type: Nodetype): None
                }
            }
        }
        ```

        For example:
        ```
        tree = {
            (f"<Dir {pytester.path.name}>", pytest.Dir): {
                ("<Module test_module.py>", pytest.Module): {
                    ("<Function test_adder>", pytest.Function): None,
                    ("<Function test_globals>", pytest.Function): None,
                },
            },
        }
        ```
        """  # noqa: D415
        if len(tree) != 1:
            msg = f"Please provide a dict with exactly 1 entry, not {tree}"
            raise ValueError(msg)
        nodedetails, children = next(iter(tree.items()))
        node = cls._DummyNode(*nodedetails)
        if children is None:
            return cls(node=node, children=None)
        return cls(
            node=node,
            children=[cls.from_dict({childnode: grandchildren}) for childnode, grandchildren in children.items()],
        )

    @dataclass
    class _DummyNode:
        """
        A dummy node for a `CollectionTree`, used by `CollectionTree.from_dict()`.

        Compares equal to a genuine `pytest.Node` of the correct type AND where `repr(Node)` == `DummyNode.name`.
        """

        name: str
        nodetype: type
        parent: Self | None = None
        """Currently always None but required to avoid attribute errors if type checking Union[pytest.Node,_DummNode]"""

        def __eq__(self, other: pytest.Item | pytest.Collector | Self):
            if isinstance(other, type(self)):
                return self.name == other.name and self.nodetype == other.nodetype
            return self.name == repr(other) and self.nodetype is type(other)

        def __repr__(self):
            return f"{self.name} ({self.nodetype})"

    def __init__(
        self,
        *_,
        node: pytest.Item | pytest.Collector | _DummyNode,
        children: list[CollectionTree] | None,
    ):
        """Do not directly initiatise a CollectionTree, use the constructors `from_items()` or `from_dict()` instead."""
        self.children = children
        """
        either:
        - if node is `pytest.Collector`: a `list[CollectionTree]` of child nodes
        - if node is `pytest.Item`: `None`
        """
        self.node = node
        """The actual collected node."""

    def __eq__(self, other: Self):
        """CollectionTrees are equal if their children and node attributes are equal."""
        try:
            return self.children == other.children and self.node == other.node
        except AttributeError:
            return False

    def __repr__(self):
        """Indented, multiline representation of the tree to simplify interpreting test failures."""
        if self.children is None:
            children = ""
        else:
            children = indent("\n".join(repr(child).rstrip() for child in self.children), "    ")
        return f"{self.node!r}\n{children}\n"

which allows me to have tests of the form:

@pytest.fixture
def expectedtree(example_dir: pytest.Pytester):
    tree = {
        (f"<Dir {example_dir.path.name}>", pytest.Dir): {
            ("<Module test_module.py>", pytest.Module): {
                ("<Function test_adder>", pytest.Function): None,
                ("<Function test_globals>", pytest.Function): None,
            },
        },
    }
    return CollectionTree.from_dict(tree)

def test_expectedtree_repr(expectedtree: CollectionTree, example_dir: pytest.Pytester):
    assert repr(expectedtree) == dedent(f"""\
        <Dir {example_dir.path.name}> (<class '_pytest.main.Dir'>)
            <Module test_module.py> (<class '_pytest.python.Module'>)
                <Function test_adder> (<class '_pytest.python.Function'>)
                <Function test_globals> (<class '_pytest.python.Function'>)
        """)


def test_collectiontree(expectedtree: CollectionTree, collection_nodes: CollectedDir):
    tree = CollectionTree.from_items(collection_nodes.items)
    assert expectedtree == tree
    assert repr(tree) == dedent(f"""\
        <Dir {collection_nodes.pytester_instance.path.name}>
            <Module test_module.py>
                <Function test_adder>
                <Function test_globals>
        """)

[Note: collection_nodes.items is the output of pytester.genitems([example_dir.path.name])]

Any thoughts? Is this worth pursuing in a more generic context? or even personally within my own project (am I wasting time recreating something that already exists?)

[RonnyPfannschmidt]

Unfortunately its both incorrect and potential breaking on node tree changes

It removes the setup /teardown cost aware test reordering and it's fragile towards changes like the directory node introduction and the instance node removal

The eventually coming Definition node introduction would also break the expected trees there

[MusicalNinjaDad]

Yes, right now it is very much hacky-hack for my specific needs, and doesn't even fulfil those yet.

I just wanted to reach out early to see:

1. if something already existed in pytest which I'd not found and do I was watering my effort
2. If it was worth investing in covering cases I don't specifically need at the moment and upstreaming a correct and complete version