Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create an end-to-end testing suite for workflows using a bundle server #23

Closed
vdye opened this issue Jan 27, 2023 · 2 comments · Fixed by #33
Closed

Create an end-to-end testing suite for workflows using a bundle server #23

vdye opened this issue Jan 27, 2023 · 2 comments · Fixed by #33
Assignees
Labels
enhancement New feature or request
Milestone

Comments

@vdye
Copy link
Collaborator

vdye commented Jan 27, 2023

  • Probably also in the test/ subdirectory of the repo
  • Should test a full end-to-end workflow: deploy to a separate server, clone/fetch/etc. with Git
  • Tests are designed around scenarios, written from the user's perspective - treat services as a black box. End-to-end tests should look like a PM wrote them, vs. integration tests designed by an engineer.
@vdye vdye transferred this issue from another repository Feb 16, 2023
@vdye vdye added the enhancement New feature or request label Feb 16, 2023
@vdye vdye added this to the v1.0 milestone Feb 16, 2023
@vdye vdye self-assigned this Mar 2, 2023
@vdye
Copy link
Collaborator Author

vdye commented Mar 7, 2023

I think the biggest hurdle here is deciding on a language & framework. Collecting a bunch of rough thoughts on the topic:

What kind of functionality do we want?

  • Since we're testing a bunch of CLIs (git-bundle-server, git-bundle-web-server, git), the language we use here should have relatively simple "call program" syntax; bonus points if it can be configured to run via shell.
  • For the sake of test reporting, whatever framework we use should be able to emit results in some kind of standardized format (e.g. junit).
  • Built-in global "setup" and "teardown" functions would be nice for cleaning up test repositories.
  • Should be configurable to run tests in parallel.

Which language?

Go

I'm actually leaning against Go, since using another language would really reinforce the "black-box" nature of end-to-end tests.

Shell

Shell tests are clearly the best choice for "ease of executing shell commands", but they'd involve a lot of boilerplate and, given our experience with Git's test suite, aren't particularly easy to work with. Leaning no on this one as well.

Typescript (Node.js)

This isn't a bad option, but personally it's not my first choice (npm is not my friend). That said, it's flexible and supports a number of fantastic testing frameworks.

Ruby

My personal top choice, mainly due to its extremely simple shell execution syntax. Like JS, there are a bunch of good test frameworks to choose from.

Test framework?

I really like Cucumber - tests are written in human-readable documents, those documents are portable across languages (should we want to change language), and most language integrations will output results in a standard format. I'm open to other ideas though!

@vdye
Copy link
Collaborator Author

vdye commented Mar 8, 2023

Update: after doing some rough prototyping, I ended up on Typescript after all (mostly because its seems overall better supported in VSCode). It took some finagling to get Cucumber set up the way I wanted it, but at this point I'm able to run tests successfully. Hoping to have one or two scenarios implemented by the end of the week!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants