add tests against generated code #217
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1 +1,5 @@ | ||
| """ Generate a complete client and verify that it is correct """ | ||
| import pytest | ||
|
|
||
| pytest.register_assert_rewrite("end_to_end_tests.end_to_end_test_helpers") | ||
| pytest.register_assert_rewrite("end_to_end_tests.functional_tests.helpers") |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,75 @@ | ||
| ## The `functional_tests` module | ||
|
|
||
| These are end-to-end tests which run the client generator against many small API documents that are specific to various test cases. | ||
|
|
||
| Rather than testing low-level implementation details (like the unit tests in `tests`), or making assertions about the exact content of the generated code (like the "golden record"-based end-to-end tests), these treat both the generator and the generated code as black boxes and make assertions about their behavior. | ||
|
|
||
| The tests are in two submodules: | ||
|
|
||
| # `generated_code_execution` | ||
|
|
||
| These tests use valid API specs, and after running the generator, they _import and execute_ pieces of the generated code to verify that it actually works at runtime. | ||
|
|
||
| Each test class follows this pattern: | ||
|
|
||
| - Use the decorator `@with_generated_client_fixture`, providing an inline API spec (JSON or YAML) that contains whatever schemas/paths/etc. are relevant to this test class. | ||
| - The spec can omit the `openapi:`, `info:`, and `paths:`, blocks, unless those are relevant to the test. | ||
| - The decorator creates a temporary file for the inline spec and a temporary directory for the generated code, and runs the client generator. | ||
| - It creates a `GeneratedClientContext` object (defined in `end_to_end_test_helpers.py`) to keep track of things like the location of the generated code and the output of the generator command. | ||
| - This object is injected into the test class as a fixture called `generated_client`, although most tests will not need to reference the fixture directly. | ||
| - `sys.path` is temporarily changed, for the scope of this test class, to allow imports from the generated code. | ||
| - Use the decorator `@with_generated_code_imports` or `@with_generated_code_import` to make classes or functions from the generated code available to the tests. | ||
| - `@with_generated_code_imports(".models.MyModel1", ".models.MyModel2)` would execute `from [package name].models import MyModel1, MyModel2` and inject the imported classes into the test class as fixtures called `MyModel1` and `MyModel2`. | ||
| - `@with_generated_code_import(".api.my_operation.sync", alias="endpoint_method")` would execute `from [package name].api.my_operation import sync`, but the fixture would be named `endpoint_method`. | ||
| - After the test class finishes, these imports are discarded. | ||
|
|
||
| Example: | ||
|
|
||
| ```python | ||
| @with_generated_client_fixture( | ||
| """ | ||
| components: | ||
| schemas: | ||
| MyModel: | ||
| type: object | ||
| properties: | ||
| stringProp: {"type": "string"} | ||
| """) | ||
| @with_generated_code_import(".models.MyModel") | ||
| class TestSimpleJsonObject: | ||
| def test_encoding(self, MyModel): | ||
| instance = MyModel(string_prop="abc") | ||
| assert instance.to_dict() == {"stringProp": "abc"} | ||
| ``` | ||
|
|
||
| # `generator_failure_cases` | ||
|
|
||
| These run the generator with an invalid API spec and make assertions about the warning/error output. Some of these invalid conditions are expected to only produce warnings about the affected schemas, while others are expected to produce fatal errors that terminate the generator. | ||
|
|
||
| For warning conditions, each test class uses `@with_generated_client_fixture` as above, then uses `assert_bad_schema` to parse the output and check for a specific warning message for a specific schema name. | ||
|
|
||
| ```python | ||
| @with_generated_client_fixture( | ||
| """ | ||
| components: | ||
| schemas: | ||
| MyModel: | ||
| # some kind of invalid schema | ||
| """) | ||
| class TestBadSchema: | ||
| def test_encoding(self, generated_client): | ||
| assert_bad_schema(generated_client, "MyModel", "some expected warning text") | ||
| ``` | ||
|
|
||
| Or, for fatal error conditions: | ||
|
|
||
| - Call `inline_spec_should_fail`, providing an inline API spec (JSON or YAML). | ||
|
|
||
| ```python | ||
| class TestBadSpec: | ||
| def test_some_spec_error(self): | ||
| result = inline_spec_should_fail(""" | ||
| # some kind of invalid spec | ||
| """) | ||
| assert "some expected error text" in result.output | ||
| ``` |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We'll be dropping 3.8 anyway, but I removed it from the CI test matrix here because some of the stuff in the new tests (like assertions about type annotations) is a lot harder to do in 3.8.