Skip to content

Commit b7e4682

Browse files
mkcorlagruOriolAbril
authored
Add minimal documentation & tweaks (#24)
Adds a user guide, a reference for the typing syntax Adds a user guide, a reference for the typing syntax and the command line as well as tests that verify parts of it. In addition, this ended up incorporating a bigger & smaller tweaks that refactor docstub and change behavior. These came up during documentation when I noticed behavior that was hard to document or didn't seem very intuitive. Co-authored-by: Lars Grüter <lagru@mailbox.org> Co-authored-by: Oriol Abril-Pla <oriol.abril.pla@gmail.com>
1 parent c6f5ee2 commit b7e4682

16 files changed

+522
-118
lines changed

‎README.md‎

Lines changed: 22 additions & 36 deletions
Original file line numberDiff line numberDiff line change
@@ -1,54 +1,40 @@
11
# docstub
22

3-
> [!NOTE]
4-
> In early development!
3+
> [!NOTE] In early development!
4+
> Expect bugs, missing features, and incomplete documentation.
5+
> Docstub is still evaluating which features it needs to support as the community gives feedback.
6+
> Several features are experimental and included to make adoption of docstub easier.
7+
> Long-term, some of these might be discouraged or removed as docstub matures.
58
6-
A command line tool to generate Python stub files (PYI) from type descriptions
7-
in NumPyDoc style docstrings.
9+
docstub is a command-line tool to generate [Python stub files](https://typing.python.org/en/latest/guides/writing_stubs.html) (i.e., PYI files) from type descriptions found in [numpydoc](https://numpydoc.readthedocs.io)-style docstrings.
810

11+
Many packages in the scientific Python ecosystem already describe expected parameter and return types in their docstrings.
12+
Docstub aims to take advantage of these and help with the adoption of type annotations.
13+
It does so by supporting widely used readable conventions such as `array of dtype` or `iterable of int(s)` which it translates into valid type annotations.
914

10-
## Installation
1115

12-
To try out docstub, for now, we recommend installing docstub directly from this
13-
repo:
16+
## Installation & getting started
1417

15-
```shell
16-
pip install 'docstub [optional] @ git+https://github.com/scientific-python/docstub'
17-
```
18+
Please refer to the [user guide](doc/user_guide.md) to get started with docstub.
1819

1920

20-
## Usage & configuration
21-
22-
```shell
23-
cd examples/
24-
docstub example_pkg/
25-
```
26-
will create stub files for `example_pkg/` in `examples/example_pkg-stubs/`.
27-
For now, refer to `docstub --help` for more.
28-
29-
30-
### Declare imports and synonyms
31-
32-
Types in docstrings can and are used without having to import them. However,
33-
when docstub creates stub files from these docstrings it actually needs to
34-
know how to import those unknown types.
35-
36-
> [!TIP]
37-
> docstub already knows about types in Python's `typing` or `collections.abc`
38-
> modules. That means you can just use types like `Literal` or `Sequence`.
39-
40-
While docstub is smart enough to find some types via static analysis of
41-
definitions in the given source directory, it must be told about other types
42-
for now. To do so, refer to the syntax and comments in the
43-
`default_config.toml`.
21+
## Contributing
4422

23+
The best way you can help and contribute right now is by trying docstub out!
24+
Feedback to what features might still be missing or where it breaks for you would be greatly appreciated.
25+
Pointers to where the documentation is confusing and unclear.
4526

46-
## Contributing
27+
Since docstub is still in early development there isn't an official contribution guide yet.
28+
Features and API are still being heavily extended and the internal structure is still somewhat in flux.
29+
That said, if that only entices you, feel free to open a PR.
30+
But please do check in with an issue before you do so.
4731

48-
TBD
32+
Our project follows the [Scientific Python's Code of Conduct](https://scientific-python.org/code_of_conduct/).
4933

5034

5135
## Acknowledgements
5236

5337
Thanks to [docs2stubs](https://github.com/gramster/docs2stubs) by which this
5438
project was heavily inspired and influenced.
39+
40+
And thanks to CZI for supporting this work with an [EOSS grant](https://chanzuckerberg.com/eoss/proposals/from-library-to-protocol-scikit-image-as-an-api-reference/).

‎doc/command_line.md‎

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
# Command line reference
2+
3+
Running
4+
```
5+
docstub --help
6+
```
7+
will print
8+
9+
<!--- The following block is checked by the test suite --->
10+
<!--- begin command-line-help --->
11+
12+
```plain
13+
Usage: docstub [OPTIONS] PACKAGE_PATH
14+
15+
Generate Python stub files with type annotations from docstrings.
16+
17+
Given a path `PACKAGE_PATH` to a Python package, generate stub files for it.
18+
Type descriptions in docstrings will be used to fill in missing inline type
19+
annotations or to override them.
20+
21+
Options:
22+
--version Show the version and exit.
23+
-o, --out-dir PATH Set output directory explicitly. Stubs will be directly
24+
written into that directory while preserving the
25+
directory structure under `PACKAGE_PATH`. Otherwise,
26+
stubs are generated inplace.
27+
--config PATH Set one or more configuration file(s) explicitly.
28+
Otherwise, it will look for a `pyproject.toml` or
29+
`docstub.toml` in the current directory.
30+
--group-errors Group identical errors together and list where they
31+
occurred. Will delay showing errors until all files have
32+
been processed. Otherwise, simply report errors as the
33+
occur.
34+
--allow-errors INT Allow this many or fewer errors. If docstub reports
35+
more, exit with error code '1'. This is useful to adopt
36+
docstub gradually. [default: 0; x>=0]
37+
-v, --verbose Print more details (repeatable).
38+
-h, --help Show this message and exit.
39+
```
40+
41+
<!--- end command-line-help --->

‎doc/typing_syntax.md‎

Lines changed: 124 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,124 @@
1+
# Typing syntax in docstrings
2+
3+
> [!NOTE] In early development!
4+
> Expect bugs, missing features, and incomplete documentation.
5+
> Docstub is still evaluating which features it needs to support as the community gives feedback.
6+
> Several features are experimental and included to make adoption of docstub easier.
7+
> Long-term, some of these might be discouraged or removed as docstub matures.
8+
9+
Docstub defines its own [grammar](../src/docstub/doctype.lark) to parse and transform type information in docstrings into valid type annotations.
10+
This grammar fully supports [Python's conventional typing syntax](https://typing.python.org/en/latest/index.html).
11+
So any type annotation that is valid in Python, can be used in a docstrings as is.
12+
In addition, docstub extends this syntax with several "natural language" expressions that are commonly used in the scientific Python ecosystem.
13+
14+
Docstrings are expected to follow the NumPyDoc style:
15+
```
16+
Section name
17+
------------
18+
name : annotation, optional, extra_info
19+
Description.
20+
```
21+
22+
- `name` might be the name of a parameter or attribute.
23+
Other sections like "Returns" or "Yields" are supported.
24+
- `annotation` the actual type information that will be transformed into the type annotation.
25+
- `optional` and `extra_info` can be appended to provide additional information.
26+
Their presence and content doesn't currently affect the resulting type annotation.
27+
28+
29+
## Unions
30+
31+
In addition to Python's conventional shorthand `|` syntax for [union types](https://typing.python.org/en/latest/spec/concepts.html#union-types), you can use `or` to join types.
32+
33+
| Docstring type | Python type annotation |
34+
|----------------|------------------------|
35+
| `X or Y` | `X \| Y` |
36+
| `int or float` | `int \| float` |
37+
38+
39+
## Containers
40+
41+
The content of containers can be typed using a `CONTAINER of X` like form.
42+
This extends the basic subscription syntax for [generics](https://typing.python.org/en/latest/spec/generics.html#generics).
43+
44+
| Docstring type | Python type annotation |
45+
|-------------------------|------------------------|
46+
| `CONTAINER of X` | `CONTAINER[X]` |
47+
| `CONTAINER of (X or Y)` | `CONTAINER[X \| Y]` |
48+
49+
For the simple case `CONTAINER of X`, where `X` is a name, you can append `(s)` to indicate the plural form.
50+
E.g., `list of float(s)`.
51+
52+
Variants of for [**tuples**](https://typing.python.org/en/latest/spec/tuples.html)
53+
54+
| Docstring type | Python type annotation |
55+
|---------------------|------------------------|
56+
| `tuple of (X, Y)` | `tuple[X, Y]` |
57+
| `tuple of (X, ...)` | `tuple[X, ...]` |
58+
59+
and **mappings** exist.
60+
61+
| Docstring type | Python type annotation |
62+
|----------------------|------------------------|
63+
| `MAPPING of {X: Y}` | `MAPPING[X, Y]` |
64+
| `dict of {str: int}` | `dict[str, int]` |
65+
66+
67+
> [!TIP]
68+
> While it is possible to nest these variants repeatedly, it is discouraged to do so to keep type descriptions readable.
69+
> For complex annotations with nested containers, consider using Python's conventional syntax.
70+
> In the future, docstub may warn against or disallow nesting these natural language variants.
71+
72+
73+
## Shape and dtype syntax for arrays
74+
75+
This expression allows adding shape and datatype information for data structures like [NumPy arrays](https://numpy.org/doc/stable/reference/generated/numpy.ndarray.html).
76+
77+
`array` and `ndarray`, and `array-like` and `array_like` can be used interchange-ably.
78+
79+
| Docstring type | Python type annotation |
80+
|-----------------------------|------------------------|
81+
| `array of DTYPE` | `ndarray[DTYPE]` |
82+
| `ndarray of dtype DTYPE` | `ndarray[DTYPE]` |
83+
| `array-like of DTYPE` | `ArrayLike[DTYPE]` |
84+
| `array_like of dtype DTYPE` | `ArrayLike[DTYPE]` |
85+
86+
> [!NOTE]
87+
> Noting the **shape** of an array in the docstring is supported.
88+
> However, Python's typing system is not yet able to express this information.
89+
> It is therefore not included in the resulting type annotation.
90+
91+
| Docstring type | Python type annotation |
92+
|--------------------------|------------------------|
93+
| `(3,) array of DTYPE` | `ndarray[DTYPE]` |
94+
| `(X, Y) array of DTYPE` | `ndarray[DTYPE]` |
95+
| `([P,] M, N) array-like` | `ArrayLike` |
96+
| `(M, ...) ndarray` | `ArrayLike` |
97+
98+
99+
## Literals
100+
101+
[Literals](https://typing.python.org/en/latest/spec/literal.html#literals) indicate a concrete value instead of type.
102+
Instead of using [`typing.Literal`](https://docs.python.org/3/library/typing.html#typing.Literal), you can enclose literal values in `{...}` in docstrings.
103+
104+
| Docstring type | Python type annotation |
105+
|----------------|------------------------|
106+
| `{1, 2, 3}` | `Literal[1, 2, 3]` |
107+
| `{1, 2, 3}` | `Literal[1, 2, 3]` |
108+
109+
> [!TIP]
110+
> Enclosing a single value `{X}` is currently allowed but discouraged.
111+
> Instead consider the more explicit `Literal[X]`.
112+
113+
114+
## reStructuredText role
115+
116+
Since docstrings are also used to generate documentation with Sphinx, you may want to use [restructuredText roles](https://docutils.sourceforge.io/docs/ref/rst/roles.html) in your type annotations.
117+
Docstub allows for this anywhere where a qualified name can be used.
118+
119+
| Docstring type | Python type annotation |
120+
|----------------------|------------------------|
121+
| `` `X` `` | `X` |
122+
| ``:ref:`X` `` | `X` |
123+
| ``:class:`Y.X` `` | `Y.X` |
124+
| ``:py:class:`Y.X` `` | `Y.X` |

0 commit comments

Comments
 (0)