Dependencies & Configs: Proposals for improvement and optimisation

[alexted]

It is possible to move most settings from separate files, such as setup.py, setup.cfg, MANIFEST.in, and other configuration files, inside pyproject.toml. Below is an example that combines these files and configures dependencies, package instructions, and other settings to reduce the number of separate files to a minimum:

[build-system]
requires = ["setuptools>=61.0", "wheel"]
build-backend = "setuptools.build_meta"

[project]
name = "flask-jsonrpc"
version = "4.0.0a7"  # Укажите нужную версию
description = "Adds JSONRPC support to Flask."
readme = {file = "README.md", content-type = "text/markdown"}
license = {text = "MIT"}
authors = [
    {name = "Nycholas Oliveira", email = "[email protected]"},
    {name = "Your Name", email = "[email protected]"}
]
maintainers = [{name = "Cenobit Technologies Inc.", email = "[email protected]"}]
keywords = ["flask", "flask-extensions", "jsonrpc", "json-rpc", "rpc"]
classifiers = [
    "Development Status :: 5 - Production/Stable",
    "Environment :: Web Environment",
    "Intended Audience :: Developers",
    "License :: OSI Approved :: MIT License",
    "Operating System :: OS Independent",
    "Programming Language :: Python",
    "Programming Language :: Python :: 3",
    "Programming Language :: Python :: 3 :: Only",
    "Programming Language :: Python :: 3.7",
    "Programming Language :: Python :: 3.8",
    "Programming Language :: Python :: 3.9",
    "Programming Language :: Python :: 3.10",
    "Programming Language :: Python :: 3.11",
    "Programming Language :: Python :: 3.12",
    "Topic :: Internet :: WWW/HTTP :: Dynamic Content",
    "Topic :: Software Development :: Libraries :: Application Frameworks",
]

requires-python = ">=3.7"
dependencies = [
    "Flask>=3.0.0,<4.0",
    "werkzeug>=1.0",
    "typeguard==4.4.0",
    "typing_extensions>=4.3.0",
    "typing_inspect==0.9.0",
    "pydantic!=1.8,!=1.8.1,!=2.0.0,!=2.0.1,!=2.1.0,<3.0.0,>=1.7.4",
    "eval_type_backport==0.2.0"
]

[project.optional-dependencies]
async = ["Flask[async]>=3.0.0,<4.0"]
dotenv = ["Flask[dotenv]>=3.0.0,<4.0"]
dev = [
    "pytest",
    "black",
    "isort",
]
test = [
    "pytest",
    "pytest-cov",
]
lint = [
    "flake8",
    "black",
    "isort",
]
docs = [
    "sphinx",
    "sphinx-rtd-theme",
]

[project.urls]
Donate = "https://github.com/sponsors/nycholas"
Homepage = "https://github.com/cenobites/flask-jsonrpc"
Documentation = "https://flask-jsonrpc.readthedocs.io/"
"Source Code" = "https://github.com/cenobites/flask-jsonrpc"
"Issue Tracker" = "https://github.com/cenobites/flask-jsonrpc/issues/"
Website = "https://flask-jsonrpc.readthedocs.io/"

[tool.setuptools.packages.find]
include = ["flask_jsonrpc"]

# Конфигурация для инструментов
[tool.black]
line-length = 88
target-version = ["py37"]

[tool.isort]
profile = "black"

[tool.pytest.ini_options]
minversion = "6.0"
addopts = "-v --tb=short"
testpaths = ["tests"]

[tool.flake8]
max-line-length = 88
extend-ignore = ["E203", "W503"]

# Конфигурация для tox
[tool.tox]
envlist = ["py37", "py38", "py39", "lint", "test"]

[tool.tox.env.py37]
description = "Run tests on Python 3.7"
package = "wheel"
requires = ["pytest", "pytest-cov"]
deps = ["pytest", "pytest-cov"]
commands = ["pytest"]

[tool.tox.env.py38]
description = "Run tests on Python 3.8"
package = "wheel"
requires = ["pytest", "pytest-cov"]
deps = ["pytest", "pytest-cov"]
commands = ["pytest"]

[tool.tox.env.py39]
description = "Run tests on Python 3.9"
package = "wheel"
requires = ["pytest", "pytest-cov"]
deps = ["pytest", "pytest-cov"]
commands = ["pytest"]

[tool.tox.env.lint]
description = "Run linters (flake8, black, isort)"
deps = ["flake8", "black", "isort"]
commands = [
    "flake8 src tests",
    "black --check src tests",
    "isort --check-only src tests"
]

[tool.tox.env.test]
description = "Run tests with coverage"
deps = ["pytest", "pytest-cov"]
commands = [
    "pytest --cov=flask_jsonrpc --cov-report=term-missing"
]

[tool.cibuildwheel]
before-all = "uname -a"
build-verbosity = 1
skip = [
    "cp3{8,9,10}-*",
    "*-manylinux_i686",
    "*-musllinux_i686",
    "*-win32",
    "pp*",
]
test-requires = [
    "pytest",
    "pytest-cov",
    "pytest-xdist",
    "pytest-sugar",
    "pytest-env",
    "requests"
]
test-command = "pytest -n auto -vv --tb=short --rootdir={project} {project}/tests --cov-fail-under=0"

[tool.cibuildwheel.environment]
MYPYC_ENABLE = "1"

[tool.ruff]
src = ["src"]
fix = true
show-fixes = true
output-format = "concise"
line-length = 120
target-version = "py39"

[tool.ruff.lint]
select = [
    "E",    # pycodestyle error
    "W",    # pycodestyle warning
    "F",    # pyflakes
    "UP",   # pyupgrade
    "ANN",  # flake8-annotations
    "B",    # flake8-bugbear
    "Q",    # flake8-quotes
    "SIM",  # flake8-simplify
    "T",    # flake8-type-checking
    "B",    # flake8-bandit
    "C",    # flake8-copyright
    "I",    # isort
]

[tool.ruff.lint.mccabe]
max-complexity = 10

[tool.ruff.lint.flake8-quotes]
inline-quotes = "single"
docstring-quotes = "double"

[tool.ruff.lint.flake8-type-checking]
exempt-modules = ["typing", "typing_extensions", "annotated_types"]

[tool.ruff.lint.flake8-bandit]
check-typed-exception = true

[tool.ruff.lint.flake8-copyright]
author = "Cenobit Technologies, Inc. http://cenobit.es/"

[tool.ruff.lint.isort]
length-sort = true
combine-as-imports = true
order-by-type = true
force-sort-within-sections = true
split-on-trailing-comma = false
section-order = [
    "future",
    "standard-library",
    "typing-extensions",
    "flask",
    "pydantic",
    "third-party",
    "first-party",
    "local-folder"
]

[tool.ruff.lint.isort.sections]
"flask" = ["flask"]
"pydantic" = ["pydantic"]
"typing-extensions" = ["typing_inspect", "typing_extensions"]

[tool.ruff.lint.pydocstyle]
convention = "google"

[tool.ruff.format]
quote-style = "single"
indent-style = "space"
skip-magic-trailing-comma = true
docstring-code-format = true
docstring-code-line-length = 79

[tool.coverage.run]
branch = true
source = [
    "src/flask_jsonrpc",
    "tests",
]
omit = [
    "*/settings.py",
    "*/fixtures.py",
    "*/tests.py",
    "*/test_*.py",
    "*/*_tests.py",
]

[tool.coverage.paths]
source = [
    "src",
    "*/site-packages",
]

[tool.coverage.report]
fail_under = 100
ignore_errors = true
skip_covered = true
exclude_lines = [
    "pragma: no cover",
    "pragma: no cover ${PRAGMA_VERSION}",
    "def __repr__",
    "if self\\.debug",
    "if settings\\.DEBUG",
    "if current_app\\.config\\['DEBUG'\\]",
    "if app\\.config\\['DEBUG'\\]",
    "raise AssertionError",
    "raise NotImplementedError",
    "if 0:",
    "if __name__ == .__main__.:",
    "if TYPE_CHECKING:",
    "if t\\.TYPE_CHECKING:",
]

[tool.mypy]
plugins = ["pydantic.mypy"]
files = ["src/flask_jsonrpc"]
python_version = "3.12"
pretty = true
strict = true
show_error_codes = true
ignore_errors = false
ignore_missing_imports = false
warn_redundant_casts = true
warn_unused_ignores = true
no_implicit_reexport = true
check_untyped_defs = true
disallow_any_generics = true
disallow_untyped_defs = true

[[tool.mypy.overrides]]
module = [
    "asgiref.*",
    "mypy-werkzeug.datastructures.*",
    "typeguard.*",
    "typing_inspect.*",
    "dotenv.*",
]
ignore_missing_imports = true

[tool.pydantic-mypy]
init_forbid_extra = true
init_typed = true
warn_required_dynamic_aliases = true

[tool.pytype]
inputs = ["src/flask_jsonrpc"]
python_version = "3.11"
disable = ["invalid-annotation"]

[tool.pyright]
pythonVersion = "3.12"
include = ["src/flask_jsonrpc"]
typeCheckingMode = "basic"

Explanation of New Sections

1. [build-system] — Specifies the build system requirements, indicating the need for setuptools and wheel for building the package.

2. [project] — Contains essential project metadata, such as name, version, description, authors, classifiers, and main dependencies. It also includes optional dependencies for different environments like development, testing, linting, and documentation.

3. [project.optional-dependencies] — Defines optional dependencies that can be installed based on the specific needs of the project, such as development tools (dev), testing tools (test), linters (lint), and documentation tools (docs).

4. [project.urls] — Provides URLs related to the project, including the homepage, documentation, and source repository.

5. [tool.setuptools.packages.find] — Configures the packages to be included, similar to find_packages() in setup.py.

6. [tool.black] and [tool.isort] — Configuration settings for code formatting tools like black and isort, defining rules for code style.

7. [tool.pytest.ini_options] — Configuration for pytest, specifying minimum version requirements, command-line options, and test paths.

8. [tool.flake8] — Configuration for the flake8 linter, defining rules such as maximum line length and ignored errors.

9. [tool.tox] — Main section for configuring tox, listing the different environments for testing and their dependencies.

10. [tool.tox.env.] — Configuration for individual tox environments (like py37, py38, lint, and test), specifying dependencies, commands to run, and descriptions for each environment.

This consolidated configuration allows for easier management and eliminates the need for multiple configuration files like setup.py, setup.cfg, MANIFEST.in, .flake8, .isort.cfg, .coveragerc, tox.ini, and requirements/*.txt, streamlining the project structure.

You can then use docker's two-phase image creation in elegant style, where in the first phase you simply convert pyproject.toml to requirements.txt like this:

FROM python:3.12-slim as requirements-stage

WORKDIR /tmp

RUN pip install poetry

COPY pyproject.toml ./poetry.lock /tmp/

RUN poetry export -f requirements.txt --output requirements.txt --without-hashes

# Pull official base image
FROM python:3.12-slim

# Set work directory
WORKDIR /usr/src/app

# Create a non-privileged user that the app will run under.
# See https://docs.docker.com/go/dockerfile-user-best-practices/
ARG UID=10001
RUN adduser \
    --disabled-password \
    --gecos "" \
    --home "/nonexistent" \
    --shell "/sbin/nologin" \
    --no-create-home \
    --uid "${UID}" \
    appuser

# Set environment variables
ENV PYTHONDONTWRITEBYTECODE 1
ENV PYTHONUNBUFFERED 1

# Install dependencies
COPY --from=requirements-stage /tmp/requirements.txt ./requirements.txt
RUN apt update && apt install -y --no-install-recommends gcc python3-dev \
    && pip install --no-cache-dir -U pip setuptools wheel && pip install --no-cache-dir -r requirements.txt \
    && apt purge -y --auto-remove gcc python3-dev

# Copy project files
COPY . .

# Make entrypoint.sh executable
RUN chmod +x ./entrypoint.sh

# Switch to the non-privileged user to run the application.
USER appuser

# Publish network port
EXPOSE 5000

# Execute script to start the application web server
ENTRYPOINT ["./entrypoint.sh"]

[nycholas]

Hi @alexted,

Since the Python packaging tool has changed from setuptools to flit, it’s unfortunate that flit does not support Python C extensions, such as those compiled with Mypyc. For this reason, the project needs to continue using setuptools as its packaging tool. Specifically, the project requires the files setup.py, setup.cfg, and MANIFEST.in.

To fully migrate to flit, it would be necessary to develop a plugin or a custom backend to support Python C extensions. Additionally, there may be consideration of migrating to another packaging tool like Hatch.

The project has transitioned from using black, flake8, and similar tools to ruff, with the entire configuration now located in pyproject.toml.

The tox.ini file is intended to facilitate development and testing, and it is not a project dependency. The readability of the new pyproject.toml file is also improved.

Furthermore, the requirements/* files are maintained to lock the versions of dependencies and to enable GitHub Dependabot to discover new versions and create pull requests for patch management. Currently, this functionality is not feasible within pyproject.toml. Utilizing tools like poetry or pip-compile to generate requirements/* files is a good approach, as it can lock the versions of all dependencies (similar to pip freeze) and help prevent resolution issues with pip.

[nycholas]

Summary it:

1. Fully migrate to flit (or someone with Python C extensions support)
2. Adopt one tool like Poetry, pip-compile, and Pipenv have features which help add and update dependencies on other packages.

Explain that, these two improvements can be made, ;).

[alexted]

Since the Python packaging tool has changed from setuptools to flit, it’s unfortunate that flit does not support Python C extensions, such as those compiled with Mypyc. For this reason, the project needs to continue using setuptools as its packaging tool. Specifically, the project requires the files setup.py, setup.cfg, and MANIFEST.in.

Ok, setuptools is a good choice!

The project has transitioned from using black, flake8, and similar tools to ruff, with the entire configuration now located in pyproject.toml.

Yeah, and it's beautiful!

The tox.ini file is intended to facilitate development and testing, and it is not a project dependency. The readability of the new pyproject.toml file is also improved.

According to PEP 621, the pyproject.toml file has a much larger scope than just accounting for project dependencies.
It can and should be used to centrally manage tools configurations.

Furthermore, the requirements/* files are maintained to lock the versions of dependencies and to enable GitHub Dependabot to discover new versions and create pull requests for patch management. Currently, this functionality is not feasible within pyproject.toml. Utilizing tools like poetry or pip-compile to generate requirements/* files is a good approach, as it can lock the versions of all dependencies (similar to pip freeze) and help prevent resolution issues with pip.

Dependabot has support for pyproject.toml and works fine with it. No problems.
You can block/restrict dependency versions in pyproject.toml - that's not a problem either!

[nycholas]

Dependabot has support for pyproject.toml and works fine with it. No problems.

You are right! Good to know! Also, we can lock the versions with pip-compile, it is a huge update!

pip-compile --generate-hashes pyproject.toml --output-file=requirements/*.txt

Thank you!

[alexted]

Glad to share! I didn't get your comment about using pip-compile - what do you use it for?

[nycholas]

For drops entires setuptools configuration files, I think that to wait for it. By the way, the first intention is to deprecate the setuptools and favor of flit.

It can and should be used to centrally manage tools configurations.

Certainly, but I see tox.ini as a Makefile, a tool to support development with automatizations. I don't see that as a good idea to put all these contents inside the pyproject.toml.

Glad to share! I didn't get your comment about using pip-compile - what do you use it for?

When a dependency gets bumped use the post-action of GitHub Dependabot PRs that update the requirements/* (pip’s lower-level requirements.txt file) to lock the version of the entire environment (pip freeze), not only the direct dependencies. This can help somebody to always test against the same dependency versions. Also, it makes and checks file hashes, to ensure compliance with hash-locked dependency specifiers, and eases uninstallation of packages and dependencies.