|
| 1 | +# Using tapir library as a DSL for OpenAPI specification |
| 2 | + |
| 3 | +- Status: accepted |
| 4 | +- Deciders: Yurii Shynbuiev, David Poltorak, Benjamin Voiturier, Ilya Peresadin, Bart Suichies |
| 5 | +- Date: [2022-10-05] |
| 6 | +- Tags: OpenAPI, DSL, Tapir, code-generation, RESTAPI |
| 7 | + |
| 8 | +Related ADR/AIP: [Introduce REST HTTP for existing Node services](https://input-output.atlassian.net/wiki/spaces/AV2/pages/3454500948/AIP+-+001) |
| 9 | + |
| 10 | +## Context and Problem Statement |
| 11 | +Identus Platform will contain the REST API. The decision was made by team consensus during the first AOH meeting to follow "OpenAPI specification first" approach and generate stubs, server side and client side code based on OAS. |
| 12 | +Following this strategy we currently have 4-5 OAS files (Castor, Pollux, Mercury, Configuration). |
| 13 | + |
| 14 | +The following tool was selected for code generation: [OpenAPI Tools](https://github.com/OpenAPITools/openapi-generator) |
| 15 | + |
| 16 | +Instead of using the yaml file as OpenAPI specification and openapi-generator for server and client stub generation - this ADR proposes to use [Tapir](https://tapir.softwaremill.com/en/latest/index.html) Scala library as DSL for OpenAPI specification, `interpret` the endpoint defitions as Scala server and client stub, generate the yaml file, and use openapi-generator for client stubs. |
| 17 | + |
| 18 | +Technology stack that is going to be used in the Identus platform backend: Scala 3 + ZIO ecosystem |
| 19 | + |
| 20 | +Akka framework after version 2.6.x cannot be used because [Lightbend changed the license type to BSL 1.1](https://www.lightbend.com/blog/why-we-are-changing-the-license-for-akka). |
| 21 | + |
| 22 | +Looks like Akka 2.6.x still can be used according to [License FQA](https://www.lightbend.com/akka/license-faq) |
| 23 | + |
| 24 | +Currently, we have a code generation for Akka that is wrapped up into ZIO. Code generation mustache templates for ZIO-http are not available in OpenAPI tools. |
| 25 | + |
| 26 | +Mustache templates and code generation doesn't work out of the box, so the original templates where copied to the project and fixed by @Shota and @Pat. |
| 27 | +Current templates and generator contains constraints that were reported by [@Pat](https://docs.google.com/document/d/1WhUtflM_o-5uSx9LW76lycz2kbk071cVZiv6EtVwhAQ/edit#heading=h.ywcvgffenpz) and [@Shota](https://input-output-rnd.slack.com/archives/G018JE9NHAM/p1664563129397819), this requires engineering time to adopt the OAS for a code generation. @Ben says that we can live with these constraints |
| 28 | + |
| 29 | +Generally, OAS files are written by the engineers with different experience and different view on formatting, schemas, normalization, datatype. For instance, in current templates don't have |
| 30 | +- a consistent way for paginating the entities |
| 31 | +- standard Responses for 4xx and 5xx errors |
| 32 | +- normalized data types (we use ```anyOf```, ```allOf```) |
| 33 | +- query parameters convention for filtering the entities |
| 34 | +- some data types are duplicated in both Castor and Pollux OAS |
| 35 | + |
| 36 | +As OAS specification evolves it's getting harder to manage it because of the size of the file. |
| 37 | +To mitigate this issue @Pat proposed to use well-known tools: |
| 38 | +"Knowing that there are tools like [dhall](https://dhall-lang.org/#) or [CUE](https://cuelang.org/docs/integrations/openapi/) that allow us to write large configuration in yaml (or json) in a typesafe / reuseable way, I'm not hesitant to go contract-first."(c) |
| 39 | + |
| 40 | +Quality and formatting of autogenerated code depend on the template (not all templates are good enough). Making the good code from existing templates require additional time of engineers. |
| 41 | + |
| 42 | +### OpenAPI code generator constraints for Akka server |
| 43 | +#### @Pat |
| 44 | +- oneOf is not supported. It combines everything from the list if it’s an object, discard if it’s a primitive |
| 45 | +- allOf is not supported as stated in the documentation, but testing locally it worked |
| 46 | +- Have to handwrite the serialization layer |
| 47 | +#### @Shota |
| 48 | +- Undefined type ```AnyType```. You can have additionalProperties (```components/schemas/<schema name>/properties/additionalProperties```) in the schema, when you add it, it will generate a type for \<schema name\> that has another type called `AnyType` inside, this type is not defined, it just does not exist in generated code so the compilation will fail, if you get a compilation error in your sources with some `AnyType` that is not defined, look for additionalProperties in your schema |
| 49 | +- Values of type object without properties don’t serialize with spray json. You can have ```componets/schemas/<schema name>/properties/<property name>``` and every property has a type, like string, int, etc.., you can have type as object, but if you do so, you must provide object properties as well like in example below, if you don’t add it, it will generate this object type with Any in scala, and then the Akka marshaller will fail, because we use SprayJson there, and it does not support Reader and Writer for type Any (basically it can’t serialize type Any into json), you could probably define Writer and Reader for type Any to be an empty object, but I personally don’t see a reason to have value of type object and not define what properties it is going to have anyway. |
| 50 | +- ```requestBody``` in every path must be explicitly ```required:true```. It is ```false``` by default, if not marked as ```true``` it will generate a service functions that accepts ```Option[Type]``` instead of ```Type``` but endpoints are always expecting ```Type``` even if required is ```false```, not ```Option[Type]```, then when you try to generate sources you will get compilation error ```expecting Type got Option[Type]``` |
| 51 | + |
| 52 | +## Decision Drivers <!-- optional --> |
| 53 | + |
| 54 | +- enforce type-safety to endpoint definitions using Scala compiler and Tapir DSL, add CI for endpoints definitions |
| 55 | +- make endpoint definitions convenient for engineers by reusing common abstractions and definitions |
| 56 | +- introduce a standard types, schemas and approaches for all endpoint definitions: query, pagination, responses, etc |
| 57 | +- reuse endpoint definitions for creating server and client stubs in Scala |
| 58 | +- align the server side of REST API with the current technology stack (ZIO + ecosystem) |
| 59 | +- have a control over the codebase and data types |
| 60 | +- reduce time-of-maintenance of the code (either OAS should be adapted for generator or mustache templates should be fixed) |
| 61 | +- functional way of implementation of non-functional requirement (metrics, tracing, logging) |
| 62 | +- straight forward generation of Swagger UI, Redoc documentation and Async API documentation based on endpoint definitions |
| 63 | + |
| 64 | +## Considered Options |
| 65 | + |
| 66 | +- use OpenAPI tools (edit OAS manually, generate server stub for Akka and client stubs for any other languages) |
| 67 | +- use OpenAPI tools, but generate code for other server-side library (Play, Finch, Lagom) |
| 68 | +- use Tapir library (edit endpoint definitions as Scala code, reuse endpoint definitions for server stubs, generate OAS based on endpoint definitions, generate client stubs for any other language) |
| 69 | + |
| 70 | +## Decision Outcome |
| 71 | + |
| 72 | +Chosen option:"use Tapir library" till the end of the year, evaluate this solution in 2023 |
| 73 | + |
| 74 | +All endpoint definition are written in Tapir DSL. |
| 75 | + |
| 76 | +OpenAPI specification generated based on endpoint definition and is published as an artefact. (must be a part of CI) |
| 77 | + |
| 78 | +The server side is interpreted using a ZIO-HTTP interpreter to be aligned with the given technology stack. |
| 79 | + |
| 80 | +Client side stubs are generated using OpenAPI tools and OpenAPI specification file. (must be a part of CI) |
| 81 | + |
| 82 | +For server-side code the flow is following: |
| 83 | + |
| 84 | +<pre class="mermaid"> |
| 85 | +graph TD |
| 86 | + ED(Endpoint Definition) --> |Generate| OAS(OpenAPI Specification) |
| 87 | + ED --> |Generate| AAUI(AsyncAPI Specification) |
| 88 | + ED --> |Interpret| SSS(Scala Server Stub) |
| 89 | + ED --> |Interpret| SCS(Scala Client Stub) |
| 90 | + ED --> |Produce| SUI(Swagger UI) |
| 91 | + ED --> |Produce| RUI(Redoc UI) |
| 92 | + OAS --> |Input| OAT(OpenAPI Tools) |
| 93 | + OAT --> |Generate| SS(Server Stub) |
| 94 | + OAT --> |Generate| CS(Client Stub) |
| 95 | +</pre> |
| 96 | + |
| 97 | +### Positive Consequences <!-- optional --> |
| 98 | + |
| 99 | +- Type-safety and OAS configuration as a code will speed up development |
| 100 | +- Generated OpenAPI specification is unified according to the single standard (Tapir generator) |
| 101 | +- Errors in the types and endpoint definitions will be found in compile-time |
| 102 | +- Code generations will be replaced with interpretation with higher guarantees of stability |
| 103 | +- Engineers will save time for feature implementation instead of investigating the issues with AOS files or templates |
| 104 | +- Better management of OAS spec and control over the documentation (Swagger UI, Redoc, Async API for WebSockets) |
| 105 | + |
| 106 | +### Negative Consequences <!-- optional --> |
| 107 | +- Not all engineers will be able to edit the endpoint definitions in Tapir DLS, so either only engineer with Scala knowledge will do this, or knowledge sharing and workshops "How to use Tapir" are required. |
| 108 | +- OAS is going to be generated from the model defined by DLS, so the granular/manual control over the spec will be replaced by Tapir generator |
| 109 | +- There is a risk that Tapir might have some hidden surprises and constraints |
| 110 | + |
| 111 | +### Option 1 & 2: Feature Implementation Workflow |
| 112 | +<pre class="mermaid"> |
| 113 | +graph TD |
| 114 | + U[Start Feature] --> |Edit OAS| A |
| 115 | + A[OAS File] --> |Input| E |
| 116 | + U --> |Edit Template| E |
| 117 | + E[Generator & Templates]-->|Generate Server Code| B(Server Code) |
| 118 | + E -->|Generate Client Code| C(Client Code) |
| 119 | + C -->|Compile| OC(Other Compiler) |
| 120 | + OC -->|Compilation Error| I |
| 121 | + OC -->|Success| T |
| 122 | + E -->|Host file as Swagger UI| D(Swagger) |
| 123 | + B --> |Compile| S(Scala Compiler) |
| 124 | + S --> |Compilation Error| I(Investigate) |
| 125 | + I --> |Try again| U |
| 126 | + S --> |Success| T(Complete Feature) |
| 127 | +</pre> |
| 128 | + |
| 129 | +### Option 3: Feature Implementation Workflow |
| 130 | +<pre class="mermaid"> |
| 131 | +graph TD |
| 132 | + U[Start Feature] --> |Edit Endpoint Specification| ED(Endpoint Definition) |
| 133 | + U --> |Edit Input/Output Types| DM(Domain Model) |
| 134 | + ED --> |Input| TE(Tapir Library) |
| 135 | + DM --> |Input| TE |
| 136 | + TE --> |Generate| A |
| 137 | + TE --> |Interpret| SC(Server Code) |
| 138 | + TE --> |Interpret| CC(Client Code) |
| 139 | + TE --> |Produce| SW(Swagger UI) |
| 140 | + TE --> |Produce| RD(Redoc UI) |
| 141 | + TE --> |Compilation Error| U |
| 142 | + A[OAS File] --> |Input| E |
| 143 | + U --> |Edit Template| E |
| 144 | + E[Generator & Templates]-->|Generate Server Code| B(Server Code) |
| 145 | + E -->|Generate Client Code| C(Client Code) |
| 146 | + C -->|Compile| OC(Other Compiler) |
| 147 | + OC -->|Compilation Error| I |
| 148 | + OC -->|Success| T |
| 149 | + E -->|Host file as Swagger UI| D(Swagger) |
| 150 | + B --> |Compile| S(Scala Compiler) |
| 151 | + S --> |Compilation Error| I(Investigate) |
| 152 | + I --> |Try again| U |
| 153 | + S --> |Success| T(Complete Feature) |
| 154 | +</pre> |
| 155 | + |
| 156 | +## Pros and Cons of the Options <!-- optional --> |
| 157 | + |
| 158 | +### Option 1: use OpenAPI tools and mustache templates for Akka server |
| 159 | + |
| 160 | +- Good, because @Pat and @Shota already stabilized the templates, and we have a working solution |
| 161 | +- Good, because any engineer from CoreDID and Product Foundry team is able to contribute to the documentation |
| 162 | +- Good, because the same source of truth (OAS file) is used to generate Server and Client stub (less integration problems for client stubs) |
| 163 | +- Bad, because there are known constraints in the mustache templates that can slow down engineering |
| 164 | +- Bad, because Akka changed the licence and version 2.6.x will not be supported in 1 year. |
| 165 | +- Bad, because it's hard to keep the same standard for OAS that are written by different engineers |
| 166 | +- Bad, because all OAS files are merged together at infrastructure level which is slightly complex solution for this task. |
| 167 | +- Bad, because Akka Framework is not in ZIO ecosystem (it's not a good practice to use both frameworks) |
| 168 | + |
| 169 | +### Option 2: use OpenAPI tools and mustache templates for alternative Scala server libraries (Finch, Lagom, Play ) |
| 170 | + |
| 171 | +[example | description | pointer to more information | …] <!-- optional --> |
| 172 | +- All ```good``` and ```bad``` are the same as in Option 1 |
| 173 | +- Bad, because we don't know if the mustache templates are good enough for Scala 3 |
| 174 | +- Bad, because we need to evaluate if engineering team have the experience in Finch, Lagom or Play |
| 175 | + |
| 176 | +### Optoin 3: use Tapir as DSL for OpenAPI specification |
| 177 | + |
| 178 | +[example | description | pointer to more information | …] <!-- optional --> |
| 179 | + |
| 180 | +- Good, because type-safety and DLS will save the engineering time by providing a quick feedback loop in compile time |
| 181 | +- Good, because generated OAS will be aligned with the common standards |
| 182 | +- Good, because engineers can define and reuse the abstractions in FP way |
| 183 | +- Good, because entities (inputs/outputs) will be reused by Scala backend library |
| 184 | +- Good, because the endpoint definition will be reused in Scala Server or Client stub |
| 185 | +- Good, because there is no need to generate the code, stubs are interpreted by the library |
| 186 | +- Good, because ZIO-HTTP will be used, which is aligned with the current stack |
| 187 | +- Good, because Open API, Swagger, Redoc, Async API document/site generations are supported |
| 188 | +- Bad, because only Scala engineers will be able to edit the documentation |
| 189 | +- Bad, because the granular control over OAS YAML file will be lost (OAS file is generated automatically) |
| 190 | +- Bad, because we need to spend 3-5 day to transform OAS files into Tapir DSL |
| 191 | + |
| 192 | +## How to migrate from the current state to Tapir? |
| 193 | +### Current state: OpenAPI Tools + mustache templates for Akka server |
| 194 | +### Desired state: Endpoint Definitions in Tapir + ZIO-HTTP |
| 195 | +Estimated migration time is 4-6 days which we don't really want to waste. |
| 196 | + |
| 197 | +So, engineering team can proceed with keeping the existing endpoints in the current state and even work on the new endpoints using generated server stubs for Akka. |
| 198 | + |
| 199 | +At the same time OAS file can be translated to Tapir step-by-step and the endpoint definitions can be [interpreted by Tapir library as Akka routes](https://tapir.softwaremill.com/en/latest/server/akkahttp.html), and attached to the server endpoint in the same way as generated endpoints. |
| 200 | + |
| 201 | +This transitions period might take 2-3 weeks till engineering team get enough knowledge of using Tapir. |
| 202 | + |
| 203 | +Then all the endpoints are translated to Tapir, it will be possible to switch the interpreter from Akka to [ZIO-HTTP library](https://tapir.softwaremill.com/en/latest/server/ziohttp.html). |
| 204 | + |
| 205 | +## Links <!-- optional --> |
| 206 | + |
| 207 | +- [OpenAPI Tools](https://github.com/OpenAPITools/openapi-generator) |
| 208 | +- [Goals of Tapir library](https://tapir.softwaremill.com/en/latest/goals.html) |
| 209 | +- [Tapir](https://tapir.softwaremill.com/en/latest/index.html) |
| 210 | + |
| 211 | +<!-- |
| 212 | +<script src="https://cdnjs.cloudflare.com/ajax/libs/mermaid/9.2.1/mermaid.min.js"/> |
| 213 | +<script> |
| 214 | + mermaid.initialize({ startOnLoad: true }); |
| 215 | +</script> |
| 216 | +--> |
0 commit comments