fmt markdown

Author: pawelchcki

CHANGELOG.md

@@ -34,4 +34,3 @@ More details in [build documentation](https://github.com/DataDog/system-tests/bl
 ![Output on success](./utils/assets/output.png?raw=true)
 
 **[Complete documentation](https://github.com/DataDog/system-tests/blob/main/docs)**
-

README.md

@@ -1 +1 @@
-This folder will contains binaries to be tested
+This folder will contains binaries to be tested

binaries/README.md

@@ -7,12 +7,12 @@ You'll need a CI that with `docker` and `python 3.9` installed, among with very
 A valid `DD_API_KEY` env var for staging must be set.
 
 1. Clone this repo
-2. Copy paste your components' build inside `./binaries` (See [documentation](./binaries.md))
-3. `./build.sh` with relevant `library` (see [documentation](../execute/build.md)). Exemple: `./build.sh java`
-4. `./run.sh`
+1. Copy paste your components' build inside `./binaries` (See [documentation](./binaries.md))
+1. `./build.sh` with relevant `library` (see [documentation](../execute/build.md)). Exemple: `./build.sh java`
+1. `./run.sh`
 
 You will find different template or example:
 
-* [github actions](./github-actions.md)
-* [gitlab CI](./gitlab-ci.md): TODO
-* [azure](https://github.com/DataDog/dd-trace-dotnet/blob/master/.azure-pipelines/ultimate-pipeline.yml) (look for `stage: system_tests`)
+- [github actions](./github-actions.md)
+- [gitlab CI](./gitlab-ci.md): TODO
+- [azure](https://github.com/DataDog/dd-trace-dotnet/blob/master/.azure-pipelines/ultimate-pipeline.yml) (look for `stage: system_tests`)

docs/CI/README.md

@@ -52,4 +52,4 @@ jobs:
           path: |
             logs/
             binaries/
-```
+```

docs/CI/github-actions.md

@@ -3,9 +3,9 @@
 ## Add secrets
 
 1. Install aws-cli
-2. Save a valid github PATH token in a file named GH_TOKEN
-3. Save a valid staging API key in a file named DD_API_KEY
-4. then execute
+1. Save a valid github PATH token in a file named GH_TOKEN
+1. Save a valid staging API key in a file named DD_API_KEY
+1. then execute
 
 ```
 aws-vault exec --debug build-stable-developer  # Enter a token from your MFA device

docs/CI/gitlab-ci.md

@@ -1,11 +1,10 @@
 The RC API is the official way to interact with remote config. It allows to send RC payload to the library durint setup phase, and send request before/after each state change. Here is an example a scenario activating/deactivating ASM:
 
 1. the library starts in an initial state where ASM is disabled. This state is validated with an assertion on a request containing an attack : the request should not been caught by ASM
-2. Then a RC command is sent to activate ASM
-3. another request containing an attack is sent, this one must be reported by ASM
-4. A second command is sent to deactivate ASM
-5. a thirst request containing an attack is sent, this last one should not be seen
-
+1. Then a RC command is sent to activate ASM
+1. another request containing an attack is sent, this one must be reported by ASM
+1. A second command is sent to deactivate ASM
+1. a thirst request containing an attack is sent, this last one should not be seen
 
 Here is the test code performing that test. Please note the magic constants `ACTIVATE_ASM_PAYLOAD` and `DEACTIVATE_ASM_PAYLOAD`: they are encoded RC payload (exemple [here](https://github.com/DataDog/system-tests/blob/7644ceaa3c7ea44ade8bcca8c3bb2a5991d03e34/utils/proxy/rc_mocked_responses_asm_activate_only.json)). We still miss a tool that generate them from human-readable content, it will come in a near future.
 

docs/RC/RC-API.md

@@ -13,8 +13,9 @@ appsec_api_security_rc = EndToEndScenario(
     """,
 )
 ```
+
 In this code example, we can see that we are defining a proxy for remote config, with the name **APPSEC_API_SECURITY_RC**,
-it means that this scenario will mock calls from libraries to `/v7/config` by the content in *utils/proxy/rc_mocked_responses_**appsec_api_security_rc**.json* file.
+it means that this scenario will mock calls from libraries to `/v7/config` by the content in *utils/proxy/rc_mocked_responses\_**appsec_api_security_rc**.json* file.
 
 ### Create mock file
 
@@ -45,4 +46,4 @@ def remote_config_is_applied(data):
     return False
 
 interfaces.library.wait_for(remote_config_is_applied, timeout=30)
-```
+```

docs/RC/README.md

@@ -2,8 +2,8 @@
 
 System tests is a test workbench that allows any kind of functional testing over libraries (AKA tracers) and agents. It's built with several key principles:
 
-* *Black box testing*: only components' interfaces are checked. As those interfaces are very stable, our tests can make assertions without any assumptions regarding underlying implementations. "Check that the car moves, regardless of the engine"
-* *No test isolation*: Yes, it's surprising. But it allows to be very fast. So never hesitate to add a new test. And if you need a very specific test case, we can run it separately.
+- *Black box testing*: only components' interfaces are checked. As those interfaces are very stable, our tests can make assertions without any assumptions regarding underlying implementations. "Check that the car moves, regardless of the engine"
+- *No test isolation*: Yes, it's surprising. But it allows to be very fast. So never hesitate to add a new test. And if you need a very specific test case, we can run it separately.
 
 ## How to run them locally?
 

docs/README.md

@@ -1,8 +1,8 @@
 # RFC status
 
-* Initiated : June 2023
-* Validated : july 2023
-* Implemented : August 2023
+- Initiated : June 2023
+- Validated : july 2023
+- Implemented : August 2023
 
 ### Context
 
@@ -20,7 +20,7 @@ Unfortunatly, it comes with a major drawback: Activating a single test can becom
 To solve this, we'll try to leverage two points :
 
 1. Often, those declarations involve only one component
-2. Often, a component is modified by only one developper
+1. Often, a component is modified by only one developper
 
 The idea is to offer an alternative way to declare those metadata, using one file per component, also known as manifest file. Those files will be inside system tests repo, declaring for a given component some metadata.
 
@@ -54,12 +54,11 @@ Will support:
 Won't support:
 
 - any complex logic
-  _ because there is not limit on the complexity. We need to draw a line based on the ratio format simplicity / number of occurrences. The cutoff point is only test classes, declaring version for weblog variants, or skip reason for the entire class.
+  \_ because there is not limit on the complexity. We need to draw a line based on the ratio format simplicity / number of occurrences. The cutoff point is only test classes, declaring version for weblog variants, or skip reason for the entire class.
 - declaring metadata (bug, flaky, irrelevant) for test methods
   - because their namings are not stable, it would lead to frequent modifications of manifest files, spaming every team
   - because conflict mostly happen at class level
 
-
 ## Example
 
 ```yaml
@@ -79,7 +78,7 @@ tests/:
         "*": missing_feature # All other weblogs: not yet available
 ```
 
-## Format [WIP]
+## Format \[WIP\]
 
 ```json
 {

docs/RFCs/manifest.md

@@ -5,15 +5,16 @@ The components that make up a running test are simple from the outside.
 The idea behind system tests is that we can share the tests for a given feature across implementations.
 
 Enabling a feature within system tests might go like this:
- 1. [Run the system test suite](#running-the-system-tests)
- 1. Inspect `./logs/interfaces` folders to see if the data you want to validate is present
- 1. If the feature you want to validate isn't enabled, enable it.
-    * Probably the correct option: Change the weblog/application image
-    * Enable it through run.sh
-    * Enable it through an environment variable
- 1. [Add a test to verify your data, sending any requests as needed](#how-do-i-add-a-new-test).
- 1. Disable the test for languages which don't yet implement it
- 1. Submit a pull request, ask for review
+
+1. [Run the system test suite](#running-the-system-tests)
+1. Inspect `./logs/interfaces` folders to see if the data you want to validate is present
+1. If the feature you want to validate isn't enabled, enable it.
+   - Probably the correct option: Change the weblog/application image
+   - Enable it through run.sh
+   - Enable it through an environment variable
+1. [Add a test to verify your data, sending any requests as needed](#how-do-i-add-a-new-test).
+1. Disable the test for languages which don't yet implement it
+1. Submit a pull request, ask for review
 
 However, there are many scenarios where a test may not be so simple to implement.
 
@@ -22,16 +23,17 @@ This document aims to give a working understanding of the parts of system-tests,
 ## What are the components of a running test?
 
 When the system tests are executing, there are several main containers of concern.
- - [Tests Container](#tests-container) (aka "runner")
-   - Responsible for running the actual tests, sending traffic, and asserting results
- - [Application Container](#application-container) (aka "weblog")
-   - Swappable webapp language module that must meet an interface
- - [Application Proxy Container](#application-proxy-container)
-   - Mechanism to inspect payloads from the datadog libraries
- - [Agent Container](#agent-container)
-   - Basic Datadog agent image
- - [Agent Proxy Container](#agent-proxy-container)
-   - Mechanism to inspect payloads from the Agent to the Backend
+
+- [Tests Container](#tests-container) (aka "runner")
+  - Responsible for running the actual tests, sending traffic, and asserting results
+- [Application Container](#application-container) (aka "weblog")
+  - Swappable webapp language module that must meet an interface
+- [Application Proxy Container](#application-proxy-container)
+  - Mechanism to inspect payloads from the datadog libraries
+- [Agent Container](#agent-container)
+  - Basic Datadog agent image
+- [Agent Proxy Container](#agent-proxy-container)
+  - Mechanism to inspect payloads from the Agent to the Backend
 
 ```mermaid
 flowchart TD
@@ -51,30 +53,32 @@ The tests then wait on the results, which are available as the logs are collecte
 
 ## What are system-tests bad for?
 
- - Combinatorial-style tests (Permutations of framework runtimes, 3rd libraries versions, operating systems)
- - Cloud deployments, kubernetes, distributed deployments
- - Immediately knowing the reason a feature fails
- - Problems or features which are not shared across tracers
- - Performance or throughput testing
+- Combinatorial-style tests (Permutations of framework runtimes, 3rd libraries versions, operating systems)
+- Cloud deployments, kubernetes, distributed deployments
+- Immediately knowing the reason a feature fails
+- Problems or features which are not shared across tracers
+- Performance or throughput testing
+
+*Examples of bad candidates:*
 
- *Examples of bad candidates:*
-  - The .NET tracer must not write invalid [IL](https://en.wikipedia.org/wiki/Common_Intermediate_Language) for it's earliest supported runtime
-  - The startup overhead of the Java tracer is less than 3s for a given sample application
-  - The python tracer must not fail to retrieve traces for a version range of the mongodb library
+- The .NET tracer must not write invalid [IL](https://en.wikipedia.org/wiki/Common_Intermediate_Language) for it's earliest supported runtime
+- The startup overhead of the Java tracer is less than 3s for a given sample application
+- The python tracer must not fail to retrieve traces for a version range of the mongodb library
 
 ## What are system-tests good for?
 
- - Catching regressions on shared features
- - Wide coverage in a short time frame
- - Shared test coverage across all tracer libraries
- - Ensuring requirements for shared features are met across tracer libraries
- - testing a set of version of any datadog component
+- Catching regressions on shared features
+- Wide coverage in a short time frame
+- Shared test coverage across all tracer libraries
+- Ensuring requirements for shared features are met across tracer libraries
+- testing a set of version of any datadog component
 
 *Examples of good candidates:*
-  - `DD_TAGS` must be parsed correctly and carried as tags on all traces
-  - Tracer libraries must be able to communicate with the agent through Unix Domain Sockets
-  - Sampling rates from the agent are respected when not explicitly configured
-  - All tracer libraries log consistent diagnostic information at startup
+
+- `DD_TAGS` must be parsed correctly and carried as tags on all traces
+- Tracer libraries must be able to communicate with the agent through Unix Domain Sockets
+- Sampling rates from the agent are respected when not explicitly configured
+- All tracer libraries log consistent diagnostic information at startup
 
 ## How do I add a new test?
 
@@ -114,43 +118,49 @@ The `./run.sh` script starts the containers in the background.
 Often, knowing how a container fails to start is as simple as running `docker-compose up {container}` and observing the output.
 
 If there are more in depth problems within a container you may need to adjust the Dockerfile.
- - re-run `./build.sh`
- - start the container via `docker-compose up`
- - `docker exec -it {container-id} bash` to diagnose from within the container
+
+- re-run `./build.sh`
+- start the container via `docker-compose up`
+- `docker exec -it {container-id} bash` to diagnose from within the container
 
 ## What is the structure of the code base?
 
 The entry points of system-tests are observable from `./.github/workflows/ci.yml`.
 
 The `./build.sh` script calls into a nested `./utils/build/build.sh` script.
- - [Click for details about the `./build.sh` script and options available](#building-the-system-tests).
+
+- [Click for details about the `./build.sh` script and options available](#building-the-system-tests).
 
 The first argument to the `./build.sh` script is the language which is built: `./utils/build/docker/{language}`.
- - e.g., `./build.sh dotnet`
+
+- e.g., `./build.sh dotnet`
 
 The `./run.sh` script runs the tests and relies 1-to-1 on what is built in the `./build.sh` step.
- - [Click for details about the `./run.sh` script and options available](#running-the-system-tests).
+
+- [Click for details about the `./run.sh` script and options available](#running-the-system-tests).
 
 The run script ultimately calls the `./docker-compose.yml` file and whichever image is built with the `weblog` tag is tested.
- - [Click for detail about how the images interact with eachother](#what-are-the-components-of-a-running-test)
+
+- [Click for detail about how the images interact with eachother](#what-are-the-components-of-a-running-test)
 
 ## Building the System Tests
 
 The first argument to the `./build.sh` script is the language (`$TEST_LIBRARY`) which is built: `./utils/build/docker/{language}`.
- - `./build.sh cpp`
- - `./build.sh ruby`
- - `./build.sh python`
- - `./build.sh php`
- - `./build.sh nodejs`
- - `./build.sh java`
- - `./build.sh golang`
- - `./build.sh dotnet`
+
+- `./build.sh cpp`
+- `./build.sh ruby`
+- `./build.sh python`
+- `./build.sh php`
+- `./build.sh nodejs`
+- `./build.sh java`
+- `./build.sh golang`
+- `./build.sh dotnet`
 
 There are explicit arguments available for more specific configuration of the build.
- - i.e., `./build.sh {language} --weblog-variant {dockerfile-prefix}`
- - e.g., `./build.sh python --weblog-variant flask-poc`
- - shorter version: ./build.sh python -w flask-poc
 
+- i.e., `./build.sh {language} --weblog-variant {dockerfile-prefix}`
+- e.g., `./build.sh python --weblog-variant flask-poc`
+- shorter version: ./build.sh python -w flask-poc
 
 These arguments determine which Dockerfile is ultimately used in the format of: `./utils/build/docker/{language}/{dockerfile-prefix}.Dockerfile`
 
@@ -159,18 +169,20 @@ These arguments determine which Dockerfile is ultimately used in the format of:
 The build script must be successful before running the tests.
 
 The first argument to the `./run.sh` script is the scenario (`$SCENARIO`) which defaults to `DEFAULT`.
- - `./run.sh`
- - `./run.sh DEFAULT`
- - `./run.sh SAMPLING`
- - `./run.sh PROFILING`
+
+- `./run.sh`
+- `./run.sh DEFAULT`
+- `./run.sh SAMPLING`
+- `./run.sh PROFILING`
 
 You can see all available scenarios within the `./run.sh` script.
 
 The run script sets necessary variables for each scenario, which are then used within the `docker-compose.yml` file.
 
 When debugging tests, it may be useful to only run individual tests, following this example:
- - `./run.sh tests/appsec/test_conf.py::Test_StaticRuleSet::test_basic_hardcoded_ruleset`
- - `./run.sh tests/test_traces.py::Test_Misc::test_main`
+
+- `./run.sh tests/appsec/test_conf.py::Test_StaticRuleSet::test_basic_hardcoded_ruleset`
+- `./run.sh tests/test_traces.py::Test_Misc::test_main`
 
 ## Tests Container
 

docs/architecture/overview.md

@@ -1,5 +1,5 @@
 When a modification is made in system tests, the CI tries to detect which scenario to run :
 
 1. based on modified files in `tests/`, by extracting scenarios targerted by those files
-2. based on any modification in a `tests/**/utils.py`, and applying the logic 1. on any sub file in `tests/**`
-3. based on labels applyied to the PR, if anything outside `tests/` folder is modified.
+1. based on any modification in a `tests/**/utils.py`, and applying the logic 1. on any sub file in `tests/**`
+1. based on labels applyied to the PR, if anything outside `tests/` folder is modified.

docs/edit/CI-and-scenarios.md

@@ -1,6 +1,5 @@
 When you add a test class (see [features](./features.md)), you need to declare what feature it belongs to in the [Feature Parity Dashbaord](https://feature-parity.us1.prod.dog/). To achieve that, use `@features` decorators :
 
-
 ```python
 @features.awesome_tests
 class Test_AwesomeFeature:
@@ -14,9 +13,9 @@ The link to the feature is in the docstring: hover the name, this link will show
 ## Use case 2: the feature does not exists
 
 1. Create it in [Feature Parity Dashbaord](https://feature-parity.us1.prod.dog/)
-2. pick its feature ID (the number in the URL)
-3. copy pasta in `utils/_features.py` (its straightforward)
+1. pick its feature ID (the number in the URL)
+1. copy pasta in `utils/_features.py` (its straightforward)
 
-----
+______________________________________________________________________
 
 If you need any help, please ask on [slack](https://dd.enterprise.slack.com/archives/C025TJ4RZ8X)