Add minimal documentation

README.md

@@ -2,53 +2,38 @@
 
 > [!NOTE]
 > In early development!
+> Expect to encounter bugs, missing features, and fatal errors.
 
-A command line tool to generate Python stub files (PYI) from type descriptions
-in NumPyDoc style docstrings.
+docstub is a command-line tool to generate [Python](https://www.python.org) stub files (i.e., PYI files) from type descriptions found in [numpydoc](https://numpydoc.readthedocs.io)-style docstrings.
 
+Many packages in the scientific Python ecosystem already describe expected parameter and return types in their docstrings.
+Docstub aims to take advantage of these and help with the adoption of type annotations.
+It does so by supporting widely used readable conventions such as `array of dtype` or `iterable of int` which it translates into valid type annotations.
 
-## Installation
 
-To try out docstub, for now, we recommend installing docstub directly from this
-repo:
+## Installation & getting started
 
-```shell
-pip install 'docstub [optional] @ git+https://github.com/scientific-python/docstub'
-```
+Please refer to the [user guide](doc/user_guide.md) to get started with docstub.
 
 
-## Usage & configuration
-
-```shell
-cd examples/
-docstub example_pkg/
-```
-will create stub files for `example_pkg/` in `examples/example_pkg-stubs/`.
-For now, refer to `docstub --help` for more.
-
-
-### Declare imports and synonyms
-
-Types in docstrings can and are used without having to import them. However,
-when docstub creates stub files from these docstrings it actually needs to
-know how to import those unknown types.
-
-> [!TIP]
-> docstub already knows about types in Python's `typing` or `collections.abc`
-> modules. That means you can just use types like `Literal` or `Sequence`.
-
-While docstub is smart enough to find some types via static analysis of
-definitions in the given source directory, it must be told about other types
-for now. To do so, refer to the syntax and comments in the
-`default_config.toml`.
+## Contributing
 
+The best way you can help and contribute right now is by trying docstub out!
+Feedback to what features might still be missing or where it breaks for you would be greatly appreciated.
+Pointers to where the documentation is confusing and unclear.
 
-## Contributing
+Since docstub is still in early development there isn't an official contribution guide yet.
+Docstubs features and API is still being heavily extended and the internal structure is still somewhat in flux.
+That said, if that only entices you, feel free to open a PR.
+But please do check in with an issue before you do so.
 
-TBD
+Our project follows the [SciPy code of
+conduct](https://github.com/scipy/scipy/blob/master/doc/source/dev/conduct/code_of_conduct.rst).
 
 
 ## Acknowledgements
 
 Thanks to [docs2stubs](https://github.com/gramster/docs2stubs) by which this
 project was heavily inspired and influenced.
+
+And thanks to CZI for supporting this work with an [EOSS grant](https://chanzuckerberg.com/eoss/proposals/from-library-to-protocol-scikit-image-as-an-api-reference/).

doc/command_line_reference.md

@@ -0,0 +1,27 @@
+# Command line reference
+
+Running
+```
+docstub --help
+```
+will print
+
+<!--- The following block is checked by the test suite --->
+<!--- begin command-line-help --->
+
+```plain
+Usage: docstub [OPTIONS] ROOT_PATH
+
+  Generate Python stub files from docstrings.
+
+Options:
+  --version                Show the version and exit.
+  -o, --out-dir DIRECTORY  Set output directory explicitly.
+  --config FILE            Set configuration file explicitly.
+  --group-errors           Group errors by type and content. Will delay
+                           showing errors until all files have been processed.
+  -v, --verbose            Log more details.
+  -h, --help               Show this message and exit.
+```
+
+<!--- end command-line-help --->

doc/typing_reference.md

@@ -0,0 +1,94 @@
+# Typing with docstub
+
+> [!NOTE]
+> This document is work in progress and might be incomplete.
+
+
+## Docstring annotation syntax
+
+The basic structure of to describe the type of a symbol is as follows:
+```
+name : type_description (, optional) (, extra_info)
+```
+- `name` might be the name of a parameter or attribute.
+  But typing names in other sections like "Returns" or "Yields" is also supported.
+- `type_description`
+- `optional`
+- `extra_info`
+
+
+[Grammar reference](../src/docstub/doctype.lark)
+
+
+### Or
+
+| Docstring type | Python type annotation |
+|----------------|------------------------|
+| `X or Y`       | `X \| Y`               |
+
+
+### Containers
+
+| Docstring type             | Python type annotation |
+|----------------------------|------------------------|
+| `CONTAINER of X`           | `CONTAINER[X]`         |
+| `CONTAINER of (X \| Y)`    | `CONTAINER[X \| Y]`    |
+| `CONTAINER of (X, Y, ...)` | `CONTAINER[X, Y, ...]` |
+
+
+### Shape and dtype syntax for arrays
+
+`array` and `ndarray`, and `array-like` and `array_like` can be used interchange-ably.
+
+| Docstring type              | Python type annotation |
+|-----------------------------|------------------------|
+| `array of DTYPE`            | `ndarray[DTYPE]`       |
+| `ndarray of dtype DTYPE`    | `ndarray[DTYPE]`       |
+| `array-like of DTYPE`       | `ArrayLike[DTYPE]`     |
+| `array_like of dtype DTYPE` | `ArrayLike[DTYPE]`     |
+
+> [!NOTE]
+> Noting the **shape** of an array in the docstring is supported.
+> However, typing is not yet possible and the shape doesn't impact the resulting annotation.
+
+| Docstring type           | Python type annotation |
+|--------------------------|------------------------|
+| `(3,) array of DTYPE`    | `ndarray[DTYPE]`       |
+| `(X, Y) array of DTYPE`  | `ndarray[DTYPE]`       |
+| `([P,] M, N) array-like` | `ArrayLike`            |
+| `(M, ...) ndarray`       | `ArrayLike`            |
+
+
+### Literals
+
+| Docstring type      | Python type annotation     |
+|---------------------|----------------------------|
+| `{1, "string", .2}` | `Literal[1, "string", .2]` |
+| `{X}`               | `Literal[X]`               |
+
+
+### Sphinx-like reference
+
+| Docstring type    | Python type annotation |
+|-------------------|------------------------|
+| ``:ref:`X` ``     | `X`                    |
+| ``:class:`Y.X` `` | `Y.X`                  |
+
+Can be used in any context where a qualified name can be used.
+
+
+## Special cases
+
+### Disable docstub with comment directive
+
+```python
+class Foo:
+    """Docstring."""
+
+    # docstub: off
+    a: int = None
+    b: str = ""
+    c: int = None
+    b: str = ""
+    # docstub: on
+```

doc/user_guide.md

@@ -0,0 +1,104 @@
+# User guide
+
+> [!NOTE]
+> In early development!
+> Expect to encounter bugs, missing features, and fatal errors.
+
+
+## Installation
+
+While a docstub package is already available on PyPI, we recommend trying out docstub by installing directly from GitHub with
+
+```shell
+pip install 'docstub [optional] @ git+https://github.com/scientific-python/docstub'
+```
+
+If you want to pin to a certain commit you can append `@COMMIT_SHA` to the repo URL above.
+
+
+## Getting started
+
+Consider a simple example with the following documented function
+
+<!--- The following block is checked by the test suite --->
+<!--- begin example.py --->
+
+```python
+# example.py
+
+def example_metric(image, *, mask=None, sigma=1.0, method='standard'):
+    """Pretend to calculate a local metric between two images.
+
+    Parameters
+    ----------
+    image : array-like
+        First image.
+    mask : array of dtype uint8, optional
+        Second image.
+    sigma : float or Iterable of float, optional
+        Sigma value for each dimension in `image`. A single value is broadcast
+        to all dimensions.
+    method : {'standard', 'modified'}, optional, default = 'standard'
+        The method to use for calculating the metric.
+
+    Returns
+    -------
+    met : ndarray of dtype float
+    """
+    pass
+```
+
+<!--- end example.py --->
+
+Feeding this input to docstub with
+
+```shell
+docstub simple_script.py
+```
+
+will create `example.pyi` in the same directory
+
+<!--- The following block is checked by the test suite --->
+<!--- begin example.pyi --->
+```python
+# File generated with docstub
+
+from collections.abc import Iterable
+from typing import Literal
+
+import numpy as np
+from numpy.typing import ArrayLike, NDArray
+
+def example_metric(
+    image: ArrayLike,
+    *,
+    mask: NDArray[np.uint8] | None = ...,
+    sigma: float | Iterable[float] = ...,
+    method: Literal["standard", "modified"] = ...
+) -> NDArray[float]: ...
+```
+<!--- end example.pyi --->
+
+There are several interesting things to note here:
+
+- Many existing conventions that the scientific Python ecosystem uses, will work out of the box.
+  In this case, docstub knew how to translate `array-like`, `array of dtype uint8` into a valid type annotation in the stub file.
+
+- In a similar manner, `or` can be used as a "natural language" alternative to `|`.
+
+- Optional arguments that default to `None` are recognized and a `| None` is appended automatically if the type doesn't include it already.
+
+- Common container types such as `Iterable` can be used and a necessary import will be added automatically.
+
+
+## Importing types
+
+
+## Adding your own aliases for docstring descriptions
+
+
+## Adopting docstub gradually
+
+`--group-errors`
+
+`--allow-errors`

tests/test_doc.py

@@ -0,0 +1,71 @@
+"""Test documentation in doc/."""
+
+import re
+from pathlib import Path
+
+import click
+
+from docstub._cli import main as docstub_main
+
+PROJECT_ROOT = Path(__file__).parent.parent
+
+
+def test_getting_started_example(tmp_path):
+    # Load user guide
+    md_file = PROJECT_ROOT / "doc/user_guide.md"
+    with md_file.open("r") as io:
+        md_content = io.read()
+
+    # Extract code block for example.py
+    regex_py = (
+        r"<!--- begin example.py --->"
+        r"\n```python\n(.*)\n```\n"
+        r"<!--- end example.py --->"
+    )
+    matches_py = re.findall(regex_py, md_content, flags=re.DOTALL)
+    assert len(matches_py) == 1
+    py_source = matches_py[0]
+
+    # Create example.py and run docstub on it
+    py_file = tmp_path / "example.py"
+    with py_file.open("x") as io:
+        io.write(py_source)
+    docstub_main([str(py_file)], standalone_mode=False)
+
+    # Load created PYI file, this is what we expect to find in the user guide's
+    # code block for example.pyi
+    pyi_file = py_file.with_suffix(".pyi")
+    assert pyi_file.is_file()
+    with pyi_file.open("r") as io:
+        expected_pyi = io.read().strip()
+
+    # Extract code block for example.pyi from guide
+    regex_pyi = (
+        r"<!--- begin example.pyi --->"
+        r"\n```python\n(.*)\n```\n"
+        r"<!--- end example.pyi --->"
+    )
+    matches_pyi = re.findall(regex_pyi, md_content, flags=re.DOTALL)
+    assert len(matches_pyi) == 1
+    actual_pyi = matches_pyi[0].strip()
+
+    assert expected_pyi == actual_pyi
+
+
+def test_command_line_help():
+    ctx = click.Context(docstub_main, info_name="docstub")
+    expected_help = f"""
+```plain
+{docstub_main.get_help(ctx)}
+```
+""".strip()
+    md_file = PROJECT_ROOT / "doc/command_line_reference.md"
+    with md_file.open("r") as io:
+        md_content = io.read()
+
+    regex = r"<!--- begin command-line-help --->(.*)<!--- end command-line-help --->"
+    matches = re.findall(regex, md_content, flags=re.DOTALL)
+    assert len(matches) == 1
+
+    actual_help = matches[0].strip()
+    assert actual_help == expected_help