documentation

Author: eli-bl

.changeset/live_tests.md

@@ -0,0 +1,9 @@
+---
+default: minor
+---
+
+# New category of end-to-end tests
+
+There is a new set of tests that generate client code from an API document and then actually import and execute that code. See [`end_to_end_tests/generated_code_live_tests`](./end_to_end_tests/generated_code_live_tests) for more details.
+
+This does not affect any runtime functionality of openapi-python-client.

CONTRIBUTING.md

@@ -54,17 +54,29 @@ If you think that some of the added code is not testable (or testing it would ad
 2. If you're modifying the way an existing feature works, make sure an existing test generates the _old_ code in `end_to_end_tests/golden-record`. You'll use this to check for the new code once your changes are complete.
 3. If you're improving an error or adding a new error, add a [unit test](#unit-tests)
 
-#### End-to-end tests
+#### End-to-end snapshot tests
 
-This project aims to have all "happy paths" (types of code which _can_ be generated) covered by end to end tests (snapshot tests). In order to check code changes against the previous set of snapshots (called a "golden record" here), you can run `pdm e2e`. To regenerate the snapshots, run `pdm regen`.
+This project aims to have all "happy paths" (types of code which _can_ be generated) covered by end-to-end tests. There are two types of these: snapshot tests, and unit tests of generated code.
 
-There are 4 types of snapshots generated right now, you may have to update only some or all of these depending on the changes you're making. Within the `end_to_end_tets` directory:
+Snapshot tests verify that the generated code is identical to a previously-committed set of snapshots (called a "golden record" here). They are basically regression tests to catch any unintended changes in the generator output.
+
+In order to check code changes against the previous set of snapshots (called a "golden record" here), you can run `pdm e2e`. To regenerate the snapshots, run `pdm regen`.
+
+There are 4 types of snapshots generated right now, you may have to update only some or all of these depending on the changes you're making. Within the `end_to_end_tests` directory:
 
 1. `baseline_openapi_3.0.json` creates `golden-record` for testing OpenAPI 3.0 features
 2. `baseline_openapi_3.1.yaml` is checked against `golden-record` for testing OpenAPI 3.1 features (and ensuring consistency with 3.0)
 3. `test_custom_templates` are used with `baseline_openapi_3.0.json` to generate `custom-templates-golden-record` for testing custom templates
 4. `3.1_specific.openapi.yaml` is used to generate `test-3-1-golden-record` and test 3.1-specific features (things which do not have a 3.0 equivalent)
 
+#### Unit tests of generated code
+
+These verify the runtime behavior of the generated code, without making assertions about the exact implementation of the code. For instance, they can verify that JSON data is correctly decoded into model class attributes.
+
+The tests run the generator against a small API spec (defined inline for each test class), and then import and execute the generated code. This can sometimes identify issues with validation logic, module imports, etc., that might be harder to diagnose via the snapshot tests, especially during development of a new feature.
+
+See [`end_to_end_tests/generated_code_live_tests`](./end_to_end_tests/generated_code_live_tests).
+
 #### Unit tests
 
 > **NOTE**: Several older-style unit tests using mocks exist in this project. These should be phased out rather than updated, as the tests are brittle and difficult to maintain. Only error cases should be tests with unit tests going forward.

end_to_end_tests/generated_code_live_tests/README.md

@@ -0,0 +1,37 @@
+## The `generated_code_live_tests` module
+
+These are end-to-end tests which run the code generator command, but unlike the other tests in `end_to_end_tests`, they are also unit tests _of the behavior of the generated code_.
+
+Each test class follows this pattern:
+
+- Use the decorator `@with_generated_client_fixture`, providing an inline API spec (JSON or YAML) that contains whatever schemas/paths/etc. are relevant to this test class. 
+  - The spec can omit the `openapi:` and `info:` blocks, unless those are relevant to the test.
+  - The decorator creates a temporary file for the inline spec and a temporary directory for the generated code, and runs the client generator.
+  - It creates a `GeneratedClientContext` object (defined in `end_to_end_test_helpers.py`) to keep track of things like the location of the generated code and the output of the generator command.
+  - This object is injected into the test class as a fixture called `generated_client`, although most tests will not need to reference the fixture directly.
+  - `sys.path` is temporarily changed, for the scope of this test class, to allow imports from the generated code.
+- Use the decorator `@with_generated_code_import` to make classes or functions from the generated code available to the tests.
+  - `@with_generated_code_import(".models.MyModel")` would execute `from [client package name].models import MyModel` and inject the imported object into the test class as a fixture called `MyModel`.
+  - `@with_generated_code_import(".models.MyModel", alias="model1")` would do the same thing, but the fixture would be named `model1`.
+  - After the test class finishes, these imports are discarded.
+
+Example:
+
+```python
+
+@with_generated_client_fixture(
+"""
+paths: {}
+components:
+  schemas:
+    MyModel:
+      type: object
+      properties:
+        stringProp: {"type": "string"}
+""")
+@with_generated_code_import(".models.MyModel")
+class TestSimpleJsonObject:
+    def test_encoding(MyModel):
+        instance = MyModel(string_prop="abc")
+        assert instance.to_dict() == {"stringProp": "abc"}
+```