Skip to content

Commit 8416c4f

Browse files
russcamrusscam
authored
Use cargo make (#133)
* Use cargo make This commit adds a Makefile.toml including tasks to help with the development of the project. Update CONTRIBUTING.md to detail typical cargo make usage. * Remove rustfmt-nighly crate This commit removes the use of rustfmt-nightly crate and instead invokes the format task with cargo make to format generated code. This change allows all projects to compile and run on rust stable channel
1 parent d903c4c commit 8416c4f

35 files changed

+269
-122
lines changed

‎CONTRIBUTING.md

+89-28
Original file line numberDiff line numberDiff line change
@@ -5,8 +5,8 @@ We ask that _before_ opening a PR however, you check to see if there is an issue
55
wish to make. If there isn't, it's best to open a new issue first to discuss it, to save you time in future
66
and help us further ascertain the crux of the issue. If an issue already exists, please add to the discussion there.
77

8-
Once an issue has been discussed and agreement that it should be acted upon, if you wish to work on a PR
9-
to address it, please assign the issue to yourself, so that others know that it is being worked on.
8+
Once an issue has been discussed and agreement reached that it should be acted upon, if you wish to work on a PR
9+
to address it, please assign the issue to yourself, so that others know that you're working on it.
1010

1111
## Sign the Contributor License Agreement
1212

@@ -15,31 +15,103 @@ before we can accept pull requests from you.
1515

1616
## Development
1717

18-
The following information may help with contributing to this project. The workspace contains two packages:
18+
The following information will help in getting up and running:
1919

20-
### api_generator
20+
### Prerequisites
2121

22-
A small executable to download REST API specs from GitHub and generate much of the client from the specs. Run with
22+
The project makes use of the following, which should be installed
23+
24+
- [**Docker**](https://www.docker.com/)
25+
26+
Docker is used to start instances of Elasticsearch by using
27+
[Elastic's Elasticsearch docker images](https://container-library.elastic.co/).
28+
For Windows, use [Docker with WSL 2 backend](https://docs.docker.com/docker-for-windows/wsl/).
29+
30+
- [**Cargo make**](https://sagiegurari.github.io/cargo-make/)
31+
32+
Cargo make is used to define and configure a set of tasks, and run them as a flow. This helps with performing actions
33+
such as starting an Elasticsearch instance for integration tests
34+
35+
Cargo make can be installed with
36+
37+
```sh
38+
cargo install --force cargo-make
39+
```
40+
41+
### Cargo make
42+
43+
Cargo make is used to define and configure a set of tasks, and run them as a flow. To see all of the tasks defined
2344

2445
```sh
25-
cargo run -p api_generator
46+
cargo make --list-all-steps
2647
```
2748

28-
from the repository root directory, and follow the prompts. The minimum REST API spec version compatible with the
29-
generator is `v7.4.0`.
49+
The `Elasticsearch` category of steps are specifically defined for this project and are defined in
50+
[Makefile.toml](Makefile.toml).
3051

31-
The api_generator makes heavy use of the [`syn`](https://docs.rs/syn/1.0.5/syn/) and [`quote`](https://docs.rs/quote/1.0.2/quote/) crates to generate Rust code from the REST API specs.
32-
The `quote!` macro is particularly useful as it accepts Rust code that can include placeholder tokens (prefixed with `#`)
33-
that will be interpolated during expansion. Unlike procedural macros, the token stream returned by the `quote!` macro
34-
can be `to_string()`'ed and written to disk, and this is used to create much of the client scaffolding.
52+
- Build all packages
53+
54+
```sh
55+
cargo make build
56+
```
57+
58+
- Generate client from REST specs
59+
60+
```sh
61+
cargo make generate-api
62+
```
63+
64+
- Run Elasticsearch package tests
65+
66+
Optionally pass
67+
68+
- `STACK_VERSION`: Elasticsearch version like `7.9.0` or can be
69+
a snapshot release like `7.x-SNAPSHOT`
70+
71+
```sh
72+
cargo make test --env STACK_VERSION=<e.g. 7.9.0>
73+
```
74+
75+
- Run YAML tests
76+
77+
Optionally pass
3578

36-
### elasticsearch
79+
- `STACK_VERSION`: Elasticsearch version like `7.9.0` or can be
80+
a snapshot release like `7.x-SNAPSHOT`
81+
- `TEST_SUITE`: Elasticsearch distribution of `oss` or `xpack`
82+
83+
```sh
84+
cargo make test-yaml --env STACK_VERSION=<e.g. 7.9.0> --env TEST_SUITE=<xpack or oss>
85+
```
3786

38-
The client package crate. The client exposes all Elasticsearch APIs as associated functions, either on
87+
### Packages
88+
89+
The workspace contains the following packages:
90+
91+
- #### `elasticsearch`
92+
93+
The client package crate. The client exposes all Elasticsearch APIs as associated functions, either on
3994
the root client, `Elasticsearch`, or on one of the _namespaced clients_, such as `Cat`, `Indices`, etc. The _namespaced clients_
4095
are based on the grouping of APIs within the [Elasticsearch](https://github.com/elastic/elasticsearch/tree/master/rest-api-spec) and [X-Pack](https://github.com/elastic/elasticsearch/tree/master/x-pack/plugin/src/test/resources/rest-api-spec/api) REST API specs from which much of the client is generated.
4196
All API functions are `async` only, and can be `await`ed.
4297

98+
- #### `api_generator`
99+
100+
A small executable that downloads REST API specs from GitHub and generates much of the client package from the specs.
101+
The minimum REST API spec version compatible with the generator is `v7.4.0`.
102+
103+
The `api_generator` package makes heavy use of the [`syn`](https://docs.rs/syn/1.0.5/syn/) and [`quote`](https://docs.rs/quote/1.0.2/quote/) crates to generate Rust code from the REST API specs.
104+
The `quote!` macro is particularly useful as it accepts Rust code that can include placeholder tokens (prefixed with `#`)
105+
that will be interpolated during expansion. Unlike procedural macros, the token stream returned by the `quote!` macro
106+
can be `to_string()`'ed and written to disk, and this is used to create much of the client scaffolding.
107+
108+
- #### `yaml_test_runner`
109+
110+
A small executable that downloads YAML tests from GitHub and generates client tests from the YAML tests. The
111+
version of YAML tests to download are determined from the commit hash of a running Elasticsearch instance.
112+
113+
The `yaml_test_runner` package can be run with `cargo test` to run the generated client tests.
114+
43115
### Design principles
44116

45117
1. Generate as much of the client as feasible from the REST API specs
@@ -75,18 +147,7 @@ The required toolchain for packages in the workspace are controlled
75147
by a `rust-toolchain` file in the root of each package.
76148
77149
`elasticsearch` package compiles and runs with rust stable.
78-
79-
`elasticsearch` tests incorporate testing the examples
80-
given in the README.md file, using the
81-
[`external_doc`](https://doc.rust-lang.org/unstable-book/language-features/external-doc.html)
82-
experimental feature. To run these tests too, run with cargo nightly,
83-
using the `+<toolchain>` command line override
84-
85-
```sh
86-
cargo +nightly test -p elasticsearch --doc
87-
```
88-
89-
`api_generator` package requires rust nightly.
150+
`api_generator` and `yaml_test_runner` packages require rust nightly.
90151
91152
### Coding style guide
92153
@@ -99,7 +160,7 @@ Rust code can be formatted using [`rustfmt`](https://github.com/rust-lang/rustfm
99160
To format all packages in a workspace, from the workspace root
100161
101162
```sh
102-
cargo fmt
163+
cargo make format
103164
```
104165

105166
It is strongly recommended to run this before opening a PR.
@@ -111,7 +172,7 @@ It is strongly recommended to run this before opening a PR.
111172
Run clippy before opening a PR
112173

113174
```sh
114-
cargo clippy
175+
cargo make clippy
115176
```
116177

117178
### Running MSVC debugger in VS Code

‎Makefile.toml

+149
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,149 @@
1+
[config]
2+
default_to_workspace = false
3+
4+
[env]
5+
# Determines the version of Elasticsearch docker container used
6+
STACK_VERSION = "8.0.0-SNAPSHOT"
7+
# Determines the distribution of docker container used. Either xpack or oss
8+
TEST_SUITE = "xpack"
9+
10+
[tasks.set-oss-env]
11+
category = "Elasticsearch"
12+
description = "Sets ELASTICSEARCH_URL environment variable for later tasks when oss test suite used"
13+
private = true
14+
condition = { env = { "TEST_SUITE" = "oss" } }
15+
env = { "ELASTICSEARCH_URL" = "http://localhost:9200" }
16+
17+
[tasks.set-xpack-env]
18+
category = "Elasticsearch"
19+
description = "Sets ELASTICSEARCH_URL environment variable for later tasks when xpack test suite used"
20+
private = true
21+
condition = { env = { "TEST_SUITE" = "xpack" } }
22+
env = { "ELASTICSEARCH_URL" = "https://elastic:changeme@localhost:9200" }
23+
24+
[tasks.run-yaml-test-runner]
25+
category = "Elasticsearch"
26+
description = '''
27+
Runs yaml_test_runner crate to generate tests from yaml files for a given Elasticsearch commit.
28+
The commit to use is retrieved from the running Elasticsearch instance
29+
'''
30+
private = true
31+
script = ["cargo run -p yaml_test_runner -- -u ${ELASTICSEARCH_URL}"]
32+
dependencies = ["start-elasticsearch"]
33+
34+
[tasks.run-yaml-test-runner.windows]
35+
script = ["cargo run -p yaml_test_runner -- -u %ELASTICSEARCH_URL%"]
36+
37+
[tasks.test-yaml-test-runner]
38+
category = "Elasticsearch"
39+
private = true
40+
condition = { env_set = [ "ELASTICSEARCH_URL" ] }
41+
env = { "ES_TEST_SERVER" = "${ELASTICSEARCH_URL}" }
42+
command = "cargo"
43+
args = ["test", "-p", "yaml_test_runner", "--", "--test-threads=1"]
44+
dependencies = ["generate-yaml-tests"]
45+
46+
[tasks.test-elasticsearch]
47+
category = "Elasticsearch"
48+
private = true
49+
condition = { env_set = [ "ELASTICSEARCH_URL" ], env = { "TEST_SUITE" = "xpack" } }
50+
env = { "ES_TEST_SERVER" = "${ELASTICSEARCH_URL}" }
51+
command = "cargo"
52+
args = ["test", "-p", "elasticsearch"]
53+
dependencies = ["start-elasticsearch"]
54+
55+
[tasks.run-api-generator]
56+
category = "Elasticsearch"
57+
private = true
58+
command = "cargo"
59+
args = ["run", "-p", "api_generator"]
60+
61+
# ============
62+
# Public tasks
63+
# ============
64+
65+
[tasks.start-elasticsearch]
66+
category = "Elasticsearch"
67+
description = "Starts Elasticsearch docker container with the given version and distribution"
68+
condition = { env_set = [ "STACK_VERSION", "TEST_SUITE" ] }
69+
script = ["DETACH=true bash .ci/run-elasticsearch.sh"]
70+
dependencies = ["set-oss-env", "set-xpack-env"]
71+
72+
[tasks.start-elasticsearch.windows]
73+
script = ["bash -c \"STACK_VERSION=%STACK_VERSION% TEST_SUITE=%TEST_SUITE% DETACH=true bash .ci/run-elasticsearch.sh\""]
74+
75+
[tasks.stop-elasticsearch]
76+
category = "Elasticsearch"
77+
description = "Stops Elasticsearch docker container, if running"
78+
condition = { env_set = [ "STACK_VERSION", "TEST_SUITE" ] }
79+
script = ["CLEANUP=true bash .ci/run-elasticsearch.sh"]
80+
dependencies = ["set-oss-env", "set-xpack-env"]
81+
82+
[tasks.stop-elasticsearch.windows]
83+
script = ["bash -c \"STACK_VERSION=%STACK_VERSION% TEST_SUITE=%TEST_SUITE% CLEANUP=true bash .ci/run-elasticsearch.sh\""]
84+
85+
[tasks.test-yaml]
86+
category = "Elasticsearch"
87+
description = "Generates and runs yaml_test_runner crate xpack/oss tests against a given Elasticsearch version"
88+
condition = { env_set = [ "STACK_VERSION", "TEST_SUITE" ] }
89+
dependencies = ["generate-yaml-tests", "test-yaml-test-runner"]
90+
run_task = "stop-elasticsearch"
91+
92+
[tasks.test]
93+
category = "Elasticsearch"
94+
clear = true
95+
description = "Runs Elasticsearch crate tests against a given Elasticsearch version"
96+
env = { "TEST_SUITE" = { value = "xpack", condition = { env_set = ["TEST_SUITE"] } } }
97+
dependencies = ["test-elasticsearch"]
98+
run_task = "stop-elasticsearch"
99+
100+
[tasks.generate-yaml-tests]
101+
category = "Elasticsearch"
102+
description = "Generates Elasticsearch client tests from YAML tests"
103+
dependencies = ["run-yaml-test-runner"]
104+
run_task = "format"
105+
106+
[tasks.generate-api]
107+
category = "Elasticsearch"
108+
description = "Generates Elasticsearch client from REST API specs"
109+
dependencies = ["run-api-generator"]
110+
run_task = "format"
111+
112+
113+
[tasks.default]
114+
clear = true
115+
script = ['''
116+
echo
117+
echo "Main tasks:"
118+
echo "- generate-api: Generates Elasticsearch client from REST API specs"
119+
echo "- start-elasticsearch: Starts Elasticsearch docker container with the given version and distribution"
120+
echo "- stop-elasticsearch: Stops Elasticsearch docker container, if running"
121+
echo "- test-yaml: Generates and runs yaml_test_runner crate xpack/oss tests against a given Elasticsearch version"
122+
echo "- test: Runs Elasticsearch crate tests against a given Elasticsearch version"
123+
echo
124+
echo "Most tasks use these environment variables:"
125+
echo "- STACK_VERSION (default '$STACK_VERSION'): the version of Elasticsearch"
126+
echo "- TEST_SUITE ('oss' or 'xpack', default '$TEST_SUITE'): the distribution of Elasticsearch"
127+
echo
128+
echo "Run 'cargo make --list-all-steps' for a complete list of available tasks."
129+
echo
130+
''']
131+
132+
[tasks.default.windows]
133+
script = ['''
134+
@echo off
135+
echo.
136+
echo Main tasks:
137+
echo - generate-api: Generates Elasticsearch client from REST API specs
138+
echo - start-elasticsearch: Starts Elasticsearch docker container with the given version and distribution
139+
echo - stop-elasticsearch: Stops Elasticsearch docker container, if running
140+
echo - test-yaml: Generates and runs yaml_test_runner crate xpack/oss tests against a given Elasticsearch version
141+
echo - test: Runs Elasticsearch crate tests against a given Elasticsearch version
142+
echo.
143+
echo Most tasks use these environment variables:
144+
echo - STACK_VERSION (default '$STACK_VERSION'): the version of Elasticsearch
145+
echo - TEST_SUITE ('oss' or 'xpack', default '$TEST_SUITE'): the distribution of Elasticsearch
146+
echo.
147+
echo Run 'cargo make --list-all-steps' for a complete list of available tasks.
148+
echo.
149+
''']

‎api_generator/Cargo.toml

-1
Original file line numberDiff line numberDiff line change
@@ -20,7 +20,6 @@ quote = "~0.3"
2020
reduce = "0.1.2"
2121
regex = "1.3.1"
2222
reqwest = "~0.9"
23-
rustfmt-nightly = "1.4.19"
2423
semver = "0.9.0"
2524
serde = "~1"
2625
serde_json = "~1"

‎api_generator/rust-toolchain

-1
This file was deleted.

‎api_generator/src/generator/code_gen/mod.rs

+2-1
Original file line numberDiff line numberDiff line change
@@ -31,8 +31,9 @@ use std::str;
3131
pub fn use_declarations() -> Tokens {
3232
quote!(
3333
#![allow(unused_imports)]
34+
3435
use crate::{
35-
client::{Elasticsearch},
36+
client::Elasticsearch,
3637
params::*,
3738
error::Error,
3839
http::{

‎api_generator/src/generator/code_gen/namespace_clients.rs

+1-1
Original file line numberDiff line numberDiff line change
@@ -103,7 +103,7 @@ pub fn generate(api: &Api, docs_dir: &PathBuf) -> Result<Vec<(String, String)>,
103103
}
104104
));
105105

106-
let generated = rust_fmt(tokens.to_string())?;
106+
let generated = tokens.to_string();
107107
output.push((namespace.to_string(), generated));
108108
}
109109

‎api_generator/src/generator/code_gen/params.rs

+2-1
Original file line numberDiff line numberDiff line change
@@ -29,7 +29,8 @@ pub fn generate(api: &Api) -> Result<String, failure::Error> {
2929
generate_param(&mut tokens, &e);
3030
}
3131

32-
rust_fmt(tokens.to_string())
32+
let generated = tokens.to_string();
33+
Ok(generated)
3334
}
3435

3536
fn generate_param(tokens: &mut Tokens, e: &ApiEnum) {

‎api_generator/src/generator/code_gen/root.rs

+1-1
Original file line numberDiff line numberDiff line change
@@ -56,6 +56,6 @@ pub fn generate(api: &Api, docs_dir: &PathBuf) -> Result<String, failure::Error>
5656
}
5757
));
5858

59-
let generated = rust_fmt(tokens.to_string())?;
59+
let generated = tokens.to_string();
6060
Ok(generated)
6161
}

0 commit comments

Comments
 (0)