Cleaning up the structure of the site

Old plan document repurposed as an anachronistic blog post
Blog post with update
New /docs with some ontent
Started gossary
Started FAQ
Possibly more :-)

content/about.md

@@ -37,6 +37,9 @@ Repos:
 * [github.com/fediverse-devnet/feditest-tests-fediverse](https://github.com/fediverse-devnet/feditest-tests-fediverse/):
   the actual tests testing various aspects of the Fediverse.
 
+* [github.com/fediverse-devnet/feditest-tests-sandbox](https://github.com/fediverse-devnet/feditest-tests-sandbox/):
+  tests against a toy sandbox protocol. Helps with understanding FediTest and how to run it.
+
 * [github.com/fediverse-devnet/feditest.org](https://github.com/fediverse-devnet/feditest.org/): this site.
 
 <div style="height: 1em"></div>

content/blog/2024-04-30-update.md

@@ -0,0 +1,116 @@
+---
+title: FediTest implementation update
+date: 2024-04-30
+author: Johannes Ernst
+authorurl: https://j12t.org/
+---
+
+We are making progress! Here is where we are:
+
+* The [FediTest framework](https://github.com/fediverse-devnet/feditest/) now has the
+  right structure. (I think ...) Here are the core abstractions:
+
+  * `TestPlan`, consisting of one or more `TestPlanSessions`, each of which is
+    a sequence of tests that are run against a `Constellation` of `Node`s.
+    `TestPlans` can be generated automatically and edited manually for maximum
+    flexibility.
+
+  * A `Constellation` of `Node`s is essentially a simulated, small Fediverse consisting
+    of two or more `Node`s running software and attempting to communicate with each
+    other. For each test session, a `Constellation` is set up, the tests are run,
+    and the `Constellation` is taken back down.
+
+  * A `Node`, you guessed it, is a website that runs a particular Fediverse application.
+    The Fediverse application running at a `Node` can be unmodified as it would be
+    deployed into production, or instrumented with extra APIs for better test automation.
+
+  * The `Node` is managed (application, DNS etc provisioned, unprovisioned etc) by a
+    `NodeDriver`, whose responsibility it is to know how to do this.
+    We have a few `NodeDriver`s so far,
+    including the `ManualNodeDriver` (the user has to do everything based on prompts),
+    the `SaasNodeDriver` (degenerate version for software that only exists as SaaS)
+    and the `UbosNodeDriver`, which uses [UBOS Gears](https://ubos.net/) for
+    management, as it knows how to do install and restore complex server-side applications
+    and we can reuse this. (Note: UBOS is one of "my" projects.)
+
+  * Subtypes of `Node` let us instrument the running `Node`s, by making them "do things"
+    ("now post a note!") and report on what happened ("has the private message arrived?").
+    We do not have implementations of this functionality yet, except for for certain
+    limited fallbacks. (See next steps below.)
+
+  * `TestRun`, which is the instance of a `TestPlan`, and which produces results
+    either in an ad-hoc format or in the [TAP format](https://node-tap.org/tap-format/).
+
+  * `Test`s can come in two flavors: (1) single functions marked as `@test` run from the
+    beginning to the end and may succeed or fail; (2) classes marked as `@test` run
+    several member functions annotated with `@step` in sequence while maintaining
+    state between the steps. The second one is much more manageable for the complex
+    setups many of the more interesting tests require.
+
+  I believe the FediTest framework now meets all the requirements for being handle the
+  types of tests we want to run that will get us to good test coverage over time. (It was
+  not easy! I will probably need to eat my words :-P )
+
+* We have categorized tests into multiple categories for better management. See
+  [separate directories](https://github.com/fediverse-devnet/feditest-tests-fediverse/tree/develop/tests)
+  in the `tests` directory of the
+  [FediTest-tests-Fediverse repo](https://github.com/fediverse-devnet/feditest-tests-fediverse).
+
+  * Some of the categories match specific standards documents, such as ActivityPub and
+    WebFinger etc. What to test based on the respective standard is annotated with
+    [Annost](https://github.com/jernst/annost) and the annotations are found in the
+    respective `TEST-SOURCES` subdirectory. To look at them, you need to (1) check out
+    the repo, (2) run `git submodule init` (3) run `git submodule update` and
+    (3) open the annotated HTML file in the browser.
+
+  * In lieu of having official standards, we've done the same for non-standard SWICG
+    documents, such as the recent "ActivityPub and HTTP Signatures" community report.
+
+  * The `fediverse` directory tries to define how it is all put together into high-level
+    tests that approximate what users might expect. Weirdly enough, there aren't
+    any documents that describe what those expectations are known to be, or should be.
+    So we collected user scenarios from the community.
+
+* There are some actual tests in those directories, but not many yet. Now that we have
+  the right framework in which to express them, and we have identified what the documents
+  say should be tested, it should be relatively straightforward to write them down in
+  Python.
+
+  Some examples:
+
+  * One test for a WebFinger server:
+    [Do not accept malformed resource parameters](https://github.com/fediverse-devnet/feditest-tests-fediverse/blob/develop/tests/webfinger/server/4_2__4_do_not_accept_malformed_resource_parameters.py).
+    Note the name of the test file is based on the section in the
+    [Annost-annotated WebFinger RFC](https://github.com/fediverse-devnet/feditest-tests-fediverse/blob/develop/tests/webfinger/TEST-SOURCES/index-annost.html)
+    where the test was found.
+
+  * One test for a WebFinger client:
+    [Accept unknown entries](https://github.com/fediverse-devnet/feditest-tests-fediverse/blob/develop/tests/webfinger/client/4_4__2_accept_unknown_entries.py).
+
+  * One test for full-fledged Fediverse interactions:
+    [Follow](https://github.com/fediverse-devnet/feditest-tests-fediverse/blob/develop/tests/fediverse/test_follow.py).
+
+If you want to try and run some of this, go to {{% pageref "/docs/quickstart/" %}}.
+
+## Next steps:
+
+* Code more Fediverse tests similar to
+  [Follow](https://github.com/fediverse-devnet/feditest-tests-fediverse/blob/develop/tests/fediverse/test_follow.py)
+  using high-level APIs, such as the one we've started on `FediverseNode`. Example method:
+  `make_a_follow_b`.
+
+* As we code more high-level tests, expand the high-level APIs.
+
+* Create a `MastodonClientNode` implementation that maps those APIs to the Mastodon
+  client API, which has been implemented not just by Mastodon but by several other
+  Fediverse applications. We expect that we will be able to automate many tests by
+  mapping to this API.
+
+* I'd really like to see a gRPC mapping, which we could use as a language-agnostic way
+  of driving Fediverse applications that aren't written in Python and aren't using
+  the Mastodon API.
+
+* Start producing actual reports from actual testing.
+
+This is all work in progress, of course, so YMMV! Issues? Questions? Chat with us
+on Matrix on [#fediverse-testing:matrix.org](https://matrix.to/#/%23fediverse-testing:matrix.org).

content/docs/_index.md

@@ -0,0 +1,8 @@
+---
+title: Documentation
+---
+
+{{% warning %}}
+Documentation, and FediTest itself, are early work-in-progress. Things are incomplete,
+probably incorrect, and will continue to change as work proceeds.
+{{% /warning %}}

content/docs/faq-feditest-tests-fediverse/_index.md

@@ -0,0 +1,4 @@
+---
+title: FAQ on the Fediverse tests
+weight: 1010
+---

content/docs/faq-feditest-tests-fediverse/test-organization.md

@@ -0,0 +1,9 @@
+---
+title: Why are there separate test sets in directories such as webfinger, activitypub and fediverse?
+---
+
+* Protocols such as WebFinger can be, and are used, in many more situations than just for the Fediverse. So we have a directory `webfinger` that (only) contains tests derived from the IETF standard that defines WebFinger. Similar directories exist for other standards, such as ActivityPub.
+
+* There are other documents that define the particularities how particular standards are used in today's Fediverse, e.g. a Social Web Community Group report on WebFinger. Tests derived from those are defined in corresponding directories, e.g. `swicg-activitypub-webfinger`.
+
+* The directory `fediverse` contains tests that test what the user of today's Fediverse expects, such as that when they follow another account, they will receive their posts. Perhaps surprisingly, the behaviors in this directory are actually not standardized anywhere.

content/docs/faq-feditest-tests-fediverse/why-not-simply-run-application-against-test.md

@@ -0,0 +1,41 @@
+---
+title: Why aren't you simply running tests against a single application-under-test?
+---
+
+That would indeed be much simpler. However, this approach would limit what we could test,
+and we'd like to cover more scenarios than what it can do.
+
+Consider this test: we want to test whether a user `[email protected]` can
+reply to posts made by user `[email protected]`. In this approach, `b.example` might be
+our application-under-test, and `a.example` would have to be provided by the test setup.
+To test this:
+
+1. the test setup needs to simulate hosting a user `[email protected]`. This includes:
+
+   * responding the WebFinger queries (over HTTPS, because the WebFinger standard
+     requires the use of HTTPS)
+   * responding to Actor file queries
+   * having a functioning inbox and an outbox
+   * implementing HTTP signatures so the original post can be delivered to `[email protected]`.
+
+1. the test setup needs to accept and manage a "follow" request.
+
+1. the test setup needs to be able to send a signed "create" activity.
+
+1. the test setup needs to be able to receive the "reply" activity.
+
+To make it realistic, not only does the test setup require HTTPS, but it needs to have
+a valid certificate, and should listen to port 443, not some non-standard port (as those
+are uncommon in production.)
+
+This is doable, but not simple.
+
+Now consider a more complex test in which a third user `[email protected]` replies to
+`[email protected]`'s reply. Or in which `[email protected]` deletes a post some time after it
+has been shared widely. Or where some payload gets garbled in the transition between
+`b.example` and `c.example`. This simple setup would not be able to support these types
+of tests very well, and those scenarios are the ones that are most difficult for
+developers to get under control, so we want to support them.
+
+So we decided that if we want to support larger {{% gls constellation %}} of {{% gls node %}},
+we might as well build things correspondingly from the get-go.

content/docs/faq-feditest/_index.md

@@ -0,0 +1,4 @@
+---
+title: FAQ on the FediTest testing framework
+weight: 1000
+---

content/docs/faq-feditest/is-feditest-just-for-activitypub.md

@@ -0,0 +1,18 @@
+---
+title: Is FediTest just for testing ActivityPub?
+---
+
+Nope, FediTest is a general framework for testing distributed systems, in particularly those that:
+
+* have **heterogeneous {{% gls Node %}}**, i.e. {{% gls Node %}} that run software built
+  by different people, potentially with different feature sets, and using different
+  tech stacks. Like the many applications of the Fediverse that nevertheless need to
+  communicate.
+
+* use **complex protocols**, like the stack that makes today's Fediverse work, which
+  already includes ActivityPub, ActivityStreams, WebFinger, HTTP Signatures and more.
+
+* with non-trivial distributed state, i.e. where just getting the (decentralized) system
+  into the state necessary before a test can be run is not so simple.
+
+This is also a reason why we keep the actual tests in a different Git repository.

content/docs/faq-feditest/is-it-complicated-to-implement-a-node-driver.md

@@ -0,0 +1,19 @@
+---
+Title: Is it complicated to implement a NodeDriver?
+---
+
+You don't actually need to do that. You can use one of the {{% gls NodeDriver %}} that
+works with any application, such as the `ManualFediverseNodeDriver` or the
+`SaasFediverseNodeDriver`. While it is great that they work with any application,
+they don't provide much automation: instead, you will be prompted to perform a bunch
+of activities you'd much rather like to have automated.
+
+Example: a test may require that your application "create" a "Note". Without integration
+of your application, you will need to do that manually (which may be fine, depending
+on what you are trying to do).
+
+If you want (some) automation, you can implement a suitable {{% gl NodeDriver %}}
+gradually and focus just on automating what is most time-consuming for you. That may
+be deploy and undeploy or a {{% gl Node %}}, or it may be performing a certain
+activity (like "create" a "post") or making an observation (like "is account `@@b.example`
+now following `[email protected]`).

content/docs/faq-feditest/role-of-ubos.md

@@ -0,0 +1,32 @@
+---
+title: What's the role of UBOS in Feditest?
+---
+
+[UBOS](https://ubos.net/) has some unique features that are very helpful for testing
+Fediverse applications, such as:
+
+* UBOS knows how to install and configure complex server-side applications with a
+  single command. For example, a single `ubos-admin deploy` can deploy Mastodon,
+  configure the web server, set up TLS, provision a database, start the required
+  background daemons, hook it all up and create an admin user etc. So setting up a
+  test {{% gl constellation %}} becomes  real simple.
+
+* UBOS knows how to restore a previously deployed application to a previously saved
+  state. This is also very useful for testing. A `ubos-admin restore`, applied to
+  a UBOS backup file previously created with `ubos-admin backup`, will restore
+  the application to the exact same state as it had been, including database content,
+  uploaded media etc. This is notoriously difficult to accomplish for server-side
+  applications.
+
+* UBOS knows how to deploy multiple instances of the same application to the same
+  (virtual) host. So you can run two (or five!) Mastodon instances on the same
+  machine. This is also very useful for testing if you want to run several
+  {{% gls Node %}} against each other.
+
+To use features, you need to run [UBOS Linux](https://ubos.net/docs/glossary/linux/),
+typically in a virtual machine or Linux container, and run FediTest in that VM.
+
+However, the use of UBOS is entirely optional. You can alternatively run FediTest on
+your PC or Mac or other Linux distro without touching UBOS. Then you won't get the
+UBOS management features, and you need to either manage your application
+{{% gls Node %}} manually or implement something UBOS-equivalent yourself.

content/docs/faq-feditest/why-not-pytest-or-other-unit-testing-frameworks.md

@@ -0,0 +1,26 @@
+---
+title: Why didn't you build on PyTest or some other unit testing framework?
+---
+
+We started that way, but it didn't turn out to be the right tool for the job.
+
+Consider:
+
+* Setting up {{% gls node %}} is expensive and even when automated can take some time.
+  For example, how quickly can you get a fully-configured instance of Mastodon running
+  on your local test machine?
+  So we want to be able to tightly control which tests run, and how often {{% gls node %}}
+  need to be provisioned or reset in a given {{% gl TestRun %}}. Unit test frameworks
+  don't generally provide much control over this.
+
+* We want to run the same tests against a wide variation of {{% gl Node %}}
+  {{% gls Constellation %}}. For example, we want to test whether Mastodon users can
+  follow users on other Mastodon instances, but also whether they can follow them on Lemmy
+  and WordPress and Threads etc.
+  Just writing a few parameterized tests would quickly explode the length of a
+  {{% gl TestRun %}}. Unit test frameworks don't really tend to make it easy to
+  control exactly which parameterization to run against which other, and which not.
+
+So we wanted to make {{% gls TestPlan %}} explicit and editable, so developers can
+run exactly what they want to run in the exact way they want, instead of the unit
+testing framework picking things for them.

content/docs/glossary/_index.md

@@ -0,0 +1,6 @@
+---
+title: Glossary
+weight: 1020
+---
+
+Definitions for some of the terms we use.

content/docs/glossary/constellation.md

@@ -0,0 +1,8 @@
+---
+title: Constellation
+summary: The set of Nodes in a particular TestPlan
+seealso: [
+    'Node',
+    'TestPlan'
+]
+---

content/docs/glossary/node.md

@@ -0,0 +1,13 @@
+---
+title: Node
+plural: Nodes
+summary: An instance of an application that communicates with other Nodes during testing.
+seealso: [
+    'NodeDriver'
+]
+---
+
+For example, an instance of Mastodon would be a Node.
+
+Also, an API that can be used by tests to control and observe what a Node is going.
+

content/docs/glossary/nodedriver.md

@@ -0,0 +1,8 @@
+---
+title: Node Driver
+plural: Node Drivers
+summary: An API to provision and unprovision Nodes.
+seealso: [
+    'Node'
+]
+---

content/docs/glossary/testplan.md

@@ -0,0 +1,8 @@
+---
+title: TestPlan
+summary: One or more test sessions, each running some tests against a Constellation
+seealso: [
+    'Constellation',
+    'TestRun'
+]
+---

content/docs/glossary/testrun.md

@@ -0,0 +1,7 @@
+---
+title: TestRun
+summary: The execution of a particular TestPlan
+seealso: [
+    'TestPlan'
+]
+---