Running feditest integration self-tests.

[steve-bate]

Moved from PR #333 for further discussion.

To run the tests in tests/feditests:

feditest run --testsdir tests/feditest --testplan tests/feditests/mastodon_saas_api.json
or such. Once we have #191, the user should be able >to specify a testplan template that doesn't have a node specification in it, and their particular node >specification as a separate JSON file. Then, that node specification can be defined locally once, and >reused in all of the test runs without having to change the test plan template.

I know there was the "or such" disclaimer, but when I run that command I see:

07:34 $ feditest run --testsdir tests/feditest --testplan tests/feditests/mastodon_saas_api.json
2024-09-17T07:34:36Z [CRITICAL] feditest: <class 'FileNotFoundError'> -- FileNotFoundError [Errno 2] No such file or directory: 'tests/feditests/mastodon_saas_api.json'

The 'tests/feditests/mastodon_saas_api.json' exists relative to the feditest base directory so I don't know what happening yet. Was the verbose option (that included stack traces) removed? That would help in these cases to know which code to explore to determine the cause of the error.

[jernst]

Are you running this from the right directory? File tests/feditests/mastodon_saas_api.json does seem to exist on develop.

-v and -v -v should continue to exist.

[steve-bate]

Are you running this from the right directory?

You'll need to tell me the "right" directory to answer that question. ;-) I'm running from the project root directory (the parent of tests).

I don't see the verbose option for the run command:

10:28 $ feditest run --help
usage: feditest run [-h] [--testsdir [TESTSDIR ...]] [--testplan TESTPLAN] [--nodedriversdir NODEDRIVERSDIR]
                    [--domain DOMAIN] [--interactive] [--who] [--tap [TAP]] [--html HTML] [--template TEMPLATE]
                    [--json [JSON]] [--summary [SUMMARY]]

options:
  -h, --help            show this help message and exit
  --testsdir [TESTSDIR ...]
                        Directory or directories where to find tests
  --testplan TESTPLAN   Name of the file that contains the test plan to run
  --nodedriversdir NODEDRIVERSDIR
                        Directory or directories where to find extra drivers for nodes that can be tested
  --domain DOMAIN       Local-only DNS domain for the DNS hostnames that are auto-generated for nodes
  --interactive         Run the tests interactively
  --who                 Record who ran the test plan on what host.
  --tap [TAP]           Write results in TAP format to stdout, or to the provided file (if given).
  --json [JSON], --testresult [JSON]
                        Write results in JSON format to stdout, or to the provided file (if given).
  --summary [SUMMARY]   Write summary to stdout, or to the provided file (if given). This is the default if no other output
                        option is given

html:
  HTML options

  --html HTML           Write results in HTML format to the provided file.
  --template TEMPLATE   When specifying --html, use this template (defaults to 'default').

However, later I saw it is at the feditest command level. Unfortunately, it didn't have the desired effect and only the terse errors message was shown. Seems like I'll need to dig into the code to diagnose the issue.

[jernst]

It's on feditest itself, not the sub-command run. So feditest -v run ... is the winning ticket ...

The above command should work in the root directory of the feditest repo.

[steve-bate]

Yeah, like I said, the -v on Feditest still doesn't show a stack trace for the error. I'm running from the project root.

I'll debug it when I'm back from my trip.

[jernst]

You need -v -v, not just one, for stack traces. This is on the assumption that a developer using Feditest as a user might want to see more of what's going on (-v) but only somebody very committed, like a FediTest developer, will want to look at the code ("stack trace") and they get to say -v -v.

[jernst]

Nevertheless, let's make the error messages nicer: #350.

[steve-bate]

ok, I finally had enough time to find the issue. The original command line you gave me had an extra "s" in the feditest run --testplan argument (not easy to see for my old eyes).

[jernst]

Ah, seems this is resolved. Sorry for typing rather than copy-pasting.