reorganize

Author: eli-bl

.changeset/live_tests.md

@@ -2,13 +2,13 @@
 default: minor
 ---
 
-# New categories of end-to-end tests
+# Functional tests
 
-Automated tests have been extended to include two new types of tests:
+Automated tests have been extended to include a new category of "functional" tests, in [`end_to_end_tests/functional_tests`](./end_to_end_tests/functional_tests). These are of two kinds:
 
-1. Happy-path tests that run the generator from an inline API document and then actually import and execute the generated code. See [`end_to_end_tests/generated_code_live_tests`](./end_to_end_tests/generated_code_live_tests).
+1. Happy-path tests that run the generator from an inline API document and then actually import and execute the generated code.
 2. Warning/error condition tests that run the generator from an inline API document that contains something invalid, and make assertions about the generator's output.
 
-These provide more efficient and granular test coverage than the "golden record"-based end-to-end tests, and also replace some tests that were previously being done against low-level implementation details in `tests/unit`.
+These provide more efficient and granular test coverage than the "golden record"-based end-to-end tests. Also, the low-level unit tests in `tests`, which are dependent on internal implementation details, can now in many cases be replaced by functional tests.
 
 This does not affect any runtime functionality of openapi-python-client.

CONTRIBUTING.md

@@ -69,21 +69,20 @@ There are 4 types of snapshots generated right now, you may have to update only
 3. `test_custom_templates` are used with `baseline_openapi_3.0.json` to generate `custom-templates-golden-record` for testing custom templates
 4. `3.1_specific.openapi.yaml` is used to generate `test-3-1-golden-record` and test 3.1-specific features (things which do not have a 3.0 equivalent)
 
-#### Unit tests of generated code
+#### Functional tests
 
-These verify the runtime behavior of the generated code, without making assertions about the exact implementation of the code. For instance, they can verify that JSON data is correctly decoded into model class attributes.
+These are black-box tests that verify the runtime behavior of generated code, as well as the generator's validation behavior. For instance, they can verify that JSON data is correctly decoded into model class attributes, or that the generator will emit an appropriate warning for an invalid spec.
 
-The tests run the generator against a small API spec (defined inline for each test class), and then import and execute the generated code. This can sometimes identify issues with validation logic, module imports, etc., that might be harder to diagnose via the snapshot tests, especially during development of a new feature.
+This can sometimes identify issues with error handling, validation logic, module imports, etc., that might be harder to diagnose via the snapshot tests, especially during development of a new feature.
 
-See [`end_to_end_tests/generated_code_live_tests`](./end_to_end_tests/generated_code_live_tests).
+See [`end_to_end_tests/functional_tests`](./end_to_end_tests/functional_tests).
 
 #### Other unit tests
 
 These include:
 
 * Regular unit tests of basic pieces of fairly self-contained low-level functionality, such as helper functions. These are implemented in the `tests/unit` directory, using the `pytest` framework.
-* End-to-end tests of invalid spec conditions, where we run the generator against a small spec with some problem, and expect it to print warnings/errors rather than generating code. These are implemented in `end_to_end_tests/generator_errors_and_warnings`.
-* Older-style unit tests of low-level functions like `property_from_data` that have complex behavior. These are brittle and difficult to maintain, and should not be used going forward. Instead, use either unit tests of generated code (to test happy paths), or end-to-end tests of invalid spec conditions (to test for warnings/errors), as described above.
+* Older-style unit tests of low-level functions like `property_from_data` that have complex behavior. These are brittle and difficult to maintain, and should not be used going forward. Instead, they should be migrated to functional tests.
 
 ### Creating a Pull Request
 

end_to_end_tests/generated_code_live_tests/README.md renamed to end_to_end_tests/functional_tests/README.md

@@ -1,6 +1,14 @@
-## The `generated_code_live_tests` module
+## The `functional_tests` module
 
-These are end-to-end tests which run the code generator command, but unlike the other tests in `end_to_end_tests`, they are also unit tests _of the behavior of the generated code_.
+These are end-to-end tests which run the client generator against many small API documents that are specific to various test cases.
+
+Rather than testing low-level implementation details (like the unit tests in `tests`), or making assertions about the exact content of the generated code (like the "golden record"-based end-to-end tests), these treat both the generator and the generated code as black boxes and make assertions about their behavior.
+
+The tests are in two submodules:
+
+# `generated_code_execution`
+
+These tests use valid API specs, and after running the generator, they _import and execute_ pieces of the generated code to verify that it actually works at runtime.
 
 Each test class follows this pattern:
 
@@ -33,3 +41,16 @@ class TestSimpleJsonObject:
         instance = MyModel(string_prop="abc")
         assert instance.to_dict() == {"stringProp": "abc"}
 ```
+
+# `generator_failure_cases`
+
+These run the generator with an invalid API spec and make assertions about the warning/error output. Some of these invalid conditions are expected to only produce warnings about the affected schemas, while others are expected to produce fatal errors that terminate the generator.
+
+For warning conditions, each test class follows this pattern:
+
+- Call `inline_spec_should_cause_warnings`, providing an inline API spec (JSON or YAML). If there are several test methods in the class using the same spec, use a fixture with scope "class" so the generator is only run once.
+- Use `assert_bad_schema_warning` to parse the output and check for a specific warning message for a specific schema name.
+
+Or, for fatal error conditions:
+
+- Call `inline_spec_should_fail`, providing an inline API spec (JSON or YAML).

end_to_end_tests/generated_code_live_tests/test_arrays.py renamed to end_to_end_tests/functional_tests/generated_code_execution/test_arrays.py

@@ -119,7 +119,7 @@ re = {composite = ["regen_e2e", "e2e --snapshot-update"]}
 regen_e2e = "python -m end_to_end_tests.regen_golden_record"
 
 [tool.pdm.scripts.test]
-cmd = "pytest tests end_to_end_tests/test_end_to_end.py end_to_end_tests/generated_code_live_tests end_to_end_tests/generator_errors_and_warnings --basetemp=tests/tmp"
+cmd = "pytest tests end_to_end_tests/test_end_to_end.py end_to_end_tests/functional_tests --basetemp=tests/tmp"
 [tool.pdm.scripts.test.env]
 "TEST_RELATIVE" = "true"