Skip to content

Commit 5e1d859

Browse files
authored
Merge pull request #2006 from qdrant/snippet-tools
Automated snippet checking
2 parents e0b9e1a + 135a64f commit 5e1d859

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

76 files changed

+5359
-22
lines changed

.github/workflows/snippets.yml

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
name: Check Snippets
2+
on:
3+
pull_request:
4+
paths:
5+
- 'automation/snippets/**'
6+
- 'qdrant-landing/content/documentation/headless/snippets/**'
7+
- '!qdrant-landing/content/documentation/headless/snippets/**/*.md'
8+
9+
jobs:
10+
convert-snippets:
11+
runs-on: ubuntu-latest
12+
steps:
13+
- uses: actions/checkout@v4
14+
15+
- name: Convert runnable snippets to markdown files
16+
run: automation/snippets/generate-md.py
17+
18+
- name: Check for uncommitted changes
19+
run: |
20+
if [ -n "$(git status --porcelain)" ]; then
21+
git status
22+
echo
23+
echo "Please run automation/snippets/generate-md.py and commit the results."
24+
exit 1
25+
fi
26+
27+
typecheck-snippets:
28+
runs-on: ubuntu-latest
29+
steps:
30+
- uses: actions/checkout@v4
31+
32+
- name: Build snippet-checker Docker image
33+
run: docker build -t snippet-checker automation/snippets/docker
34+
35+
- name: Fetch C# client
36+
run: automation/snippets/docker.sh ./sync-clients.py fetch --csharp
37+
38+
- name: Fetch Java client
39+
run: automation/snippets/docker.sh ./sync-clients.py fetch --java
40+
41+
- name: Fetch TypeScript client
42+
run: automation/snippets/docker.sh ./sync-clients.py fetch --typescript
43+
44+
- name: Typecheck snippets
45+
run: automation/snippets/docker.sh ./check.py build

.gitignore

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,4 @@
11
.idea
2+
__pycache__
23
qdrant-landing/.DS_Store
34
.DS_Store

automation/snippets/.gitignore

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
.gradle
2+
/cache
3+
/clients
4+
/tmp

automation/snippets/README.md

Lines changed: 116 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,116 @@
1+
# Snippet tools
2+
3+
This is a tool to work with code snippets.
4+
5+
1. Automated CI type checking of snippets.
6+
2. Writing new snippets and checking/running/testing them locally.
7+
This tool hides the complexity of setting up every build system/dev env for each client.
8+
Also, it supports testing snippets against unreleased versions of clients (from git branches/PRs).
9+
10+
11+
## Dependencies
12+
13+
To convert runnable snippets into markdown ([`generate-md.py`](./generate-md.py)) you need only python with no extra dependencies.
14+
15+
To typecheck/build snippets, you need a bunch of tools installed.
16+
The all-in-one [`Dockerfile`](./docker/Dockerfile) and [`shell.nix`](./shell.nix) are provided.
17+
Run [`./docker.sh`](./docker.sh) to start a shell in the docker container.
18+
19+
On ARM-based macOS, ensure that the Docker Desktop setting "General" > "Virtual Machine Options" > "Use Rosetta for x86\_64/amd64 emulation on Apple Silicon" is selected.
20+
Note: on macOS, if you see Java compile failures related to `protobuf`, re-running the test will often solve those.
21+
22+
23+
## Workflow: typechecking and running
24+
25+
Clone and build clients (latest git versions).
26+
```bash
27+
./sync-clients.py fetch --csharp --java --typescript
28+
```
29+
30+
(optional) You might specify particular tags/branches/commits instead.
31+
```bash
32+
./sync-clients.py fetch --csharp=1.16.0 --java=b290b14892534fc0d76893d41b8f656855cd589b --typescript=main
33+
```
34+
35+
(optional) Update the lockfiles for templates (see ./sync-clients.py lock --help for syntax).
36+
```bash
37+
./sync-clients.py lock --go=v1.16.0 --rust=branch=master --python=tag=v1.16.0
38+
```
39+
40+
Typecheck/build all snippets or a particular language/snippet.
41+
```bash
42+
./check.py build
43+
./check.py build -l csharp,java
44+
./check.py build ./snippets/create-collection/simple
45+
./check.py build ./snippets/create-collection/simple/go.go
46+
```
47+
48+
Test all testable snippets against localhost Qdrant instance. (you start it yourself).
49+
Testable snippets are the ones that have `test.py` file in the same directory.
50+
```bash
51+
./check.py test
52+
```
53+
54+
Run a particular snippet without running a test.
55+
```bash
56+
./check.py run ./snippets/create-collection/simple/go.go
57+
```
58+
59+
60+
## Workflow: writing new snippets/converting old ones
61+
62+
Convert existing old markdown snippets.
63+
```bash
64+
./migrate-snippet.py ./snippets/create-collection/simple/go.md
65+
```
66+
Or create new snippets from scratch.
67+
68+
To reduce noise, you can hide boilerplate code using special comment:
69+
```go
70+
if err != nil { panic(err) } // @hide
71+
```
72+
The usual wrapping boileplate like `public static void run() throws Exception {` is already hidden by default.
73+
74+
Typecheck/build the snippet.
75+
```bash
76+
./check.py build ./snippets/create-collection/simple
77+
```
78+
79+
(Optional) Start Qdrant and run the snippet.
80+
```bash
81+
./check.py run ./snippets/create-collection/simple/go.go
82+
```
83+
84+
(Optional) Write `test.py` and test it.
85+
```bash
86+
./check.py test ./snippets/create-collection/simple
87+
```
88+
89+
Before committing, regenerate markdown files.
90+
If migrated, also delete old markdown files.
91+
```bash
92+
./generate-md.py
93+
git rm ./snippets/create-collection/simple/go.md # if migrated
94+
git add ./snippets/create-collection/simple/generated/go.md # made by generate-md.py
95+
git add ./snippets/create-collection/simple/go.go
96+
```
97+
98+
99+
## Architecture
100+
101+
For each language, the snippet tester collects all code snippets from the [`snippets/`](../../qdrant-landing/content/documentation/headless/snippets) directory and puts them in a single scratch project in a temporary directory.
102+
Then those projects are compiled/typechecked.
103+
Putting all snippets in a single project rather than compiling them one by one saves time: some build systems like `gradle` take a lot of time to spin up.
104+
105+
Each supported language has:
106+
- A template project in [`templates/`](./templates) directory.
107+
- A Python subclass of `Language` in [`lib/languages/`](./lib/languages). It is responsible for:
108+
- Creating the scratch project from snippets and the template.
109+
- Building/typechecking the project.
110+
- Running a particular snippet for testing.
111+
- A list of build dependencies in [`docker/Dockerfile`](./docker/Dockerfile) and [`shell.nix`](./shell.nix).
112+
- A handler in [`sync-clients.py`](./sync-clients.py) to fetch the particular git revision of the client.
113+
- For Java, C#, Typescript it fetches the client sources to the `clients/` directory, and builds them.
114+
This directory is gitignored.
115+
- For Go, Rust, Python it updates the lockfile in the [`templates/`](./templates) directory.
116+
These lockfiles are checked in this repo.

automation/snippets/check.py

Lines changed: 174 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,174 @@
1+
#!/usr/bin/env python3
2+
3+
import argparse
4+
import importlib.util
5+
import shutil
6+
import subprocess
7+
import sys
8+
import traceback
9+
import types
10+
import typing
11+
from pathlib import Path
12+
13+
from lib import SNIPPETS_DIR, CollectedSnippetsType, collect_snippets, log
14+
from lib.languages import ALL_LANGUAGES, CompileResult, Language, parse_languages
15+
16+
17+
def main() -> None:
18+
parser = argparse.ArgumentParser()
19+
default_langs = ",".join([lang.NAME for lang in ALL_LANGUAGES])
20+
21+
parser.add_argument(
22+
"-l",
23+
"--langs",
24+
help=f"comma-separated list of languages, default is {default_langs}",
25+
default=default_langs,
26+
)
27+
parser.add_argument(
28+
"command",
29+
help="command to execute",
30+
choices=["build", "run", "test"],
31+
)
32+
parser.add_argument(
33+
"snippets",
34+
nargs="*",
35+
help=f"Snippet filenames/directories to process, default: {SNIPPETS_DIR}",
36+
default=[SNIPPETS_DIR],
37+
type=Path,
38+
metavar="SNIPPET_PATH",
39+
)
40+
41+
args = parser.parse_args()
42+
43+
languages = parse_languages(args.langs)
44+
snippets = collect_snippets(args.snippets, languages)
45+
assert args.command in ("build", "run", "test")
46+
47+
# Q: Why not `tempfile.TemporaryDirectory()`?
48+
# A: Harder to debug/inspect generated files.
49+
tmpdir = Path(__file__).parent / "tmp"
50+
shutil.rmtree(tmpdir, ignore_errors=True)
51+
tmpdir.mkdir(parents=True)
52+
build_and_run(tmpdir=tmpdir, snippets=snippets, mode=args.command)
53+
54+
55+
def build_and_run(
56+
tmpdir: Path,
57+
snippets: CollectedSnippetsType,
58+
mode: typing.Literal[
59+
"build", # just build
60+
"run", # build and run each snippet
61+
"test", # build, then run/test all snippets that have a test.py file
62+
],
63+
) -> None:
64+
snippets_by_lang: dict[type[Language], list[Path]] = {}
65+
66+
for snippets2 in snippets.values():
67+
for lang, fname in snippets2.items():
68+
snippets_by_lang.setdefault(lang, []).append(fname)
69+
70+
compile_results: dict[type[Language], CompileResult] = {}
71+
errors: list[str] = []
72+
73+
# Load test modules before compilation.
74+
# We want them to crash early.
75+
test_modules: dict[Path, types.ModuleType] = {}
76+
if mode == "test":
77+
for snippet_dir in snippets:
78+
test_file = snippet_dir / "test.py"
79+
if not test_file.exists():
80+
continue
81+
spec = importlib.util.spec_from_file_location("test_module", test_file)
82+
assert spec is not None
83+
mod = importlib.util.module_from_spec(spec)
84+
assert spec.loader is not None
85+
spec.loader.exec_module(mod)
86+
test_modules[snippet_dir] = mod
87+
88+
log("Compile stage")
89+
for lang, fnames in snippets_by_lang.items():
90+
log(f"· Compiling {lang.NAME} snippets")
91+
try:
92+
res = lang.compile(tmpdir / lang.NAME, fnames)
93+
if res.has_issues:
94+
log(f"· · Compilation had issues for {lang.NAME}")
95+
errors.append(f"Compilation had issues for {lang.NAME}")
96+
compile_results[lang] = res
97+
except Exception as e:
98+
log(f"· · Compilation failed for {lang.NAME}")
99+
errors.append(f"Compilation failed for {lang.NAME}")
100+
if not isinstance(e, subprocess.CalledProcessError):
101+
traceback.print_exc()
102+
103+
if mode in ("run", "test"):
104+
log("Run stage")
105+
for snippet_dir, snippets2 in snippets.items():
106+
if mode == "run":
107+
for lang, snippet_fname in snippets2.items():
108+
if (compile_result := compile_results.get(lang)) is None:
109+
continue
110+
111+
log(f"· Running {snippet_fname}")
112+
p = subprocess.run(
113+
compile_result.run_args[snippet_fname],
114+
text=True,
115+
)
116+
if p.returncode != 0:
117+
log(f"· · Exit code {p.returncode}")
118+
119+
if mode == "test" and snippet_dir in test_modules:
120+
log(f"· Testing snippets in {snippet_dir}")
121+
mod = test_modules[snippet_dir]
122+
123+
for lang, snippet_fname in snippets2.items():
124+
compile_result = compile_results.get(lang)
125+
if compile_result is None:
126+
continue
127+
128+
log(f"· · Testing {snippet_fname}")
129+
130+
output = None
131+
try:
132+
if hasattr(mod, "prepare"):
133+
mod.prepare()
134+
135+
p = subprocess.run(
136+
compile_result.run_args[snippet_fname],
137+
stdout=subprocess.PIPE,
138+
stderr=subprocess.STDOUT,
139+
text=True,
140+
)
141+
142+
output = p.stdout
143+
if p.returncode != 0:
144+
msg = f"Process exited with code {p.returncode}"
145+
raise RuntimeError(msg)
146+
147+
if hasattr(mod, "check"):
148+
mod.check()
149+
except Exception:
150+
log(f"· · · Testing {snippet_fname} failed")
151+
if output:
152+
print(output.rstrip())
153+
traceback.print_exc()
154+
errors.append(f"Testing {snippet_fname} failed")
155+
finally:
156+
try:
157+
if hasattr(mod, "cleanup"):
158+
mod.cleanup()
159+
except Exception:
160+
log(f"· · · Teardown for {snippet_fname} failed")
161+
errors.append(f"Teardown for {snippet_fname} failed")
162+
traceback.print_exc()
163+
164+
if errors:
165+
log("Errors encountered:")
166+
for err in errors:
167+
log(f"· {err}")
168+
sys.exit(1)
169+
elif mode == "test":
170+
log("All tests passed successfully.")
171+
172+
173+
if __name__ == "__main__":
174+
main()

automation/snippets/docker.sh

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
#!/usr/bin/env sh
2+
3+
set -e
4+
5+
IMAGE_NAME="snippet-checker"
6+
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
7+
8+
REPO_ROOT_DIR="$(cd "$SCRIPT_DIR/../.." && pwd)"
9+
10+
# check if on macos
11+
12+
if ! docker image inspect "$IMAGE_NAME" > /dev/null 2>&1; then
13+
# TODO: comment why
14+
[ "$(uname)" != Darwin ] || export DOCKER_DEFAULT_PLATFORM=linux/amd64
15+
docker build -t "$IMAGE_NAME" "$SCRIPT_DIR/docker"
16+
fi
17+
18+
[ $# -ne 0 ] || set -- bash
19+
20+
[ -t 0 ] && TTY_FLAG=--tty || TTY_FLAG=
21+
22+
docker run \
23+
--rm \
24+
--interactive \
25+
$TTY_FLAG \
26+
--network host \
27+
--user "$(id -u):$(id -g)" \
28+
--volume "$REPO_ROOT_DIR:/workspace" \
29+
"$IMAGE_NAME" \
30+
uv run "$@"

0 commit comments

Comments
 (0)