Skip to content

deps!: Drop support for Python 3.7 and 3.8 (AI Experiment) #337

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 32 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 16 commits
Commits
Show all changes
32 commits
Select commit Hold shift + click to select a range
9b32164
feat: Drop support for Python 3.7 and 3.8
google-labs-jules[bot] Apr 17, 2025
dccef48
Updates python version in lint.yml
chalmerlowe Apr 18, 2025
f81c738
Updates owlbot, removing reference to 3.8
chalmerlowe Apr 18, 2025
5cb30ca
Updates CONTRIBUTING.rst
chalmerlowe Apr 18, 2025
7f44424
updates pytest warnings
chalmerlowe Apr 18, 2025
3e4e6f6
Removes test_samples-impl ref to older virtualenv package
chalmerlowe Apr 18, 2025
7beea0d
Removes references to pandas older than 1.5.0
chalmerlowe Apr 18, 2025
32d7b8f
Removes pandas older than 1.5 and misc changes
chalmerlowe Apr 18, 2025
e871d73
updates pandas in setup.py
chalmerlowe Apr 18, 2025
2bdf3ef
more updates related to pandas
chalmerlowe Apr 18, 2025
d902e31
still broken
chalmerlowe Apr 18, 2025
a0543b9
Updates FutureWarning tests to account for unittest coverage
chalmerlowe Apr 23, 2025
6a5d943
Updates json array type tests to account for unittest coverage
chalmerlowe Apr 23, 2025
1bbd5f7
updates python version checks to ensure coverage
chalmerlowe Apr 23, 2025
96a5e97
update json test for unittest coverage
chalmerlowe Apr 23, 2025
8af9bee
Update pandas_backports unittests to ensure coverage
chalmerlowe Apr 24, 2025
78574c0
Updates per review comments
chalmerlowe Apr 24, 2025
9ca34b8
Merge branch 'main' into remove-python-37-38
chalmerlowe Apr 24, 2025
de202f4
moves class from version specific compliance file to generic file
chalmerlowe Apr 30, 2025
fd9dd27
Removes weird cut and paste error
chalmerlowe Apr 30, 2025
bfe9115
fix linting errors
chalmerlowe Apr 30, 2025
91e437e
updates import statement to ensure import of JSONArrowType
chalmerlowe Apr 30, 2025
eca1ab7
Revise required github status checks
chalmerlowe Apr 30, 2025
06b4e88
update linting
chalmerlowe Apr 30, 2025
d481be4
temporarily marking a class as no cover
chalmerlowe May 1, 2025
f4a9fc6
more updates
chalmerlowe May 1, 2025
7585a86
marked several snippets as pragma no cover
chalmerlowe May 1, 2025
c777141
updates linting
chalmerlowe May 1, 2025
7d745b0
Updates constraints and setup.py
chalmerlowe May 1, 2025
9511e7a
migrates class from one time compliance file to another
chalmerlowe May 1, 2025
9638e43
updating pyarrow version
chalmerlowe May 1, 2025
d433fae
Updates linting
chalmerlowe May 1, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/lint.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ jobs:
- name: Setup Python
uses: actions/setup-python@v5
with:
python-version: "3.8"
python-version: "3.9"
- name: Install nox
run: |
python -m pip install --upgrade setuptools pip wheel
Expand Down
10 changes: 4 additions & 6 deletions .github/workflows/unittest.yml
Original file line number Diff line number Diff line change
Expand Up @@ -5,13 +5,11 @@ on:
name: unittest
jobs:
unit:
# TODO(https://github.com/googleapis/gapic-generator-python/issues/2303): use `ubuntu-latest` once this bug is fixed.
# Use ubuntu-22.04 until Python 3.7 is removed from the test matrix
# https://docs.github.com/en/actions/using-github-hosted-runners/using-github-hosted-runners/about-github-hosted-runners#standard-github-hosted-runners-for-public-repositories
runs-on: ubuntu-22.04
# Use `ubuntu-latest` runner.
runs-on: ubuntu-latest
strategy:
matrix:
python: ['3.8', '3.9', '3.10', '3.11', '3.12', '3.13']
python: ['3.9', '3.10', '3.11', '3.12', '3.13']
steps:
- name: Checkout
uses: actions/checkout@v4
Expand Down Expand Up @@ -103,7 +101,7 @@ jobs:
- name: Setup Python
uses: actions/setup-python@v5
with:
python-version: "3.8"
python-version: "3.9"
- name: Install coverage
run: |
python -m pip install --upgrade setuptools pip wheel
Expand Down
40 changes: 0 additions & 40 deletions .kokoro/samples/python3.7/common.cfg

This file was deleted.

6 changes: 0 additions & 6 deletions .kokoro/samples/python3.7/continuous.cfg

This file was deleted.

11 changes: 0 additions & 11 deletions .kokoro/samples/python3.7/periodic-head.cfg

This file was deleted.

6 changes: 0 additions & 6 deletions .kokoro/samples/python3.7/periodic.cfg

This file was deleted.

6 changes: 0 additions & 6 deletions .kokoro/samples/python3.7/presubmit.cfg

This file was deleted.

40 changes: 0 additions & 40 deletions .kokoro/samples/python3.8/common.cfg

This file was deleted.

6 changes: 0 additions & 6 deletions .kokoro/samples/python3.8/continuous.cfg

This file was deleted.

11 changes: 0 additions & 11 deletions .kokoro/samples/python3.8/periodic-head.cfg

This file was deleted.

6 changes: 0 additions & 6 deletions .kokoro/samples/python3.8/periodic.cfg

This file was deleted.

6 changes: 0 additions & 6 deletions .kokoro/samples/python3.8/presubmit.cfg

This file was deleted.

3 changes: 1 addition & 2 deletions .kokoro/test-samples-impl.sh
Original file line number Diff line number Diff line change
Expand Up @@ -33,8 +33,7 @@ export PYTHONUNBUFFERED=1
env | grep KOKORO

# Install nox
# `virtualenv==20.26.6` is added for Python 3.7 compatibility
python3.9 -m pip install --upgrade --quiet nox virtualenv==20.26.6
python3.9 -m pip install --upgrade --quiet nox virtualenv

# Use secrets acessor service account to get secrets
if [[ -f "${KOKORO_GFILE_DIR}/secrets_viewer_service_account.json" ]]; then
Expand Down
14 changes: 5 additions & 9 deletions CONTRIBUTING.rst
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ In order to add a feature:
documentation.

- The feature must work fully on the following CPython versions:
3.7, 3.8, 3.9, 3.10, 3.11, 3.12 and 3.13 on both UNIX and Windows.
3.9, 3.10, 3.11, 3.12 and 3.13 on both UNIX and Windows.

- The feature must not add unnecessary dependencies (where
"unnecessary" is of course subjective, but new dependencies should
Expand Down Expand Up @@ -143,12 +143,12 @@ Running System Tests
$ nox -s system

# Run a single system test
$ nox -s system-3.8 -- -k <name of test>
$ nox -s system-3.9 -- -k <name of test>


.. note::

System tests are only configured to run under Python 3.8.
System tests are only configured to run under Python 3.9.
For expediency, we do not run them in older versions of Python 3.

This alone will not run the tests. You'll need to change some local
Expand Down Expand Up @@ -195,11 +195,11 @@ configure them just like the System Tests.

# Run all tests in a folder
$ cd samples/snippets
$ nox -s py-3.8
$ nox -s py-3.9

# Run a single sample test
$ cd samples/snippets
$ nox -s py-3.8 -- -k <name of test>
$ nox -s py-3.9 -- -k <name of test>

********************************************
Note About ``README`` as it pertains to PyPI
Expand All @@ -221,16 +221,12 @@ Supported Python Versions

We support:

- `Python 3.7`_
- `Python 3.8`_
- `Python 3.9`_
- `Python 3.10`_
- `Python 3.11`_
- `Python 3.12`_
- `Python 3.13`_

.. _Python 3.7: https://docs.python.org/3.7/
.. _Python 3.8: https://docs.python.org/3.8/
.. _Python 3.9: https://docs.python.org/3.9/
.. _Python 3.10: https://docs.python.org/3.10/
.. _Python 3.11: https://docs.python.org/3.11/
Expand Down
4 changes: 2 additions & 2 deletions README.rst
Original file line number Diff line number Diff line change
Expand Up @@ -34,11 +34,11 @@ dependencies.

Supported Python Versions
^^^^^^^^^^^^^^^^^^^^^^^^^
Python >= 3.7
Python >= 3.9

Unsupported Python Versions
^^^^^^^^^^^^^^^^^^^^^^^^^^^
Python <= 3.6.
Python <= 3.8.


Mac/Linux
Expand Down
81 changes: 46 additions & 35 deletions db_dtypes/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -21,14 +21,14 @@
import warnings

import numpy
import packaging.version
import pandas
import pandas.api.extensions
from pandas.errors import OutOfBoundsDatetime
import pyarrow
import pyarrow.compute

from db_dtypes import core
from db_dtypes.json import JSONArray, JSONArrowType, JSONDtype
from db_dtypes.version import __version__

from . import _versions_helpers
Expand All @@ -47,15 +47,6 @@
_NP_BOX_DTYPE = "datetime64[us]"


# To use JSONArray and JSONDtype, you'll need Pandas 1.5.0 or later. With the removal
# of Python 3.7 compatibility, the minimum Pandas version will be updated to 1.5.0.
if packaging.version.Version(pandas.__version__) >= packaging.version.Version("1.5.0"):
from db_dtypes.json import JSONArray, JSONArrowType, JSONDtype
else:
JSONArray = None
JSONDtype = None


@pandas.api.extensions.register_extension_dtype
class TimeDtype(core.BaseDatetimeDtype):
"""
Expand Down Expand Up @@ -346,35 +337,55 @@ def __sub__(self, other):

return super().__sub__(other)

# Inside db_dtypes/__init__.py (TEMPORARY DEBUG)
print(f"DEBUG: Inside __init__, JSONArray type: {type(JSONArray)}, value: {repr(JSONArray)}")
print(f"DEBUG: Inside __init__, JSONDtype type: {type(JSONDtype)}, value: {repr(JSONDtype)}")
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We'll want to remove these before merging.

Suggested change
# Inside db_dtypes/__init__.py (TEMPORARY DEBUG)
print(f"DEBUG: Inside __init__, JSONArray type: {type(JSONArray)}, value: {repr(JSONArray)}")
print(f"DEBUG: Inside __init__, JSONDtype type: {type(JSONDtype)}, value: {repr(JSONDtype)}")

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.


sys_major, sys_minor, sys_micro = _versions_helpers.extract_runtime_version()
if sys_major == 3 and sys_minor in (7, 8):
warnings.warn(
"The python-bigquery library will stop supporting Python 3.7 "
"and Python 3.8 in a future major release expected in Q4 2024. "
f"Your Python version is {sys_major}.{sys_minor}.{sys_micro}. We "
"recommend that you update soon to ensure ongoing support. For "
"more details, see: [Google Cloud Client Libraries Supported Python Versions policy](https://cloud.google.com/python/docs/supported-python-versions)",
PendingDeprecationWarning,
)


if not JSONArray or not JSONDtype:
__all__ = [
def _determine_all(json_array_type, json_dtype_type):
"""Determines the list for __all__ based on JSON type availability."""
base_all = [
"__version__",
"DateArray",
"DateDtype",
"TimeArray",
"TimeDtype",
]
else:
__all__ = [
"__version__",
"DateArray",
"DateDtype",
"JSONDtype",
"JSONArray",
"JSONArrowType",
"TimeArray",
"TimeDtype",
]
# Check if both JSON types are available (truthy)
if json_array_type and json_dtype_type:
# print("DEBUG: Condition FALSE, including JSON in __all__") # Keep if needed
return base_all + ["JSONDtype", "JSONArray", "JSONArrowType"]
else:
# print("DEBUG: Condition TRUE, excluding JSON from __all__") # Keep if needed
return base_all

def _check_python_version():
"""Checks the runtime Python version and issues a warning if needed."""
sys_major, sys_minor, sys_micro = _versions_helpers.extract_runtime_version()
if sys_major == 3 and sys_minor in (7, 8):
warnings.warn(
"The python-bigquery library as well as the python-db-dtypes-pandas library no "
"longer supports Python 3.7 and Python 3.8. "
f"Your Python version is {sys_major}.{sys_minor}.{sys_micro}. We "
"recommend that you update soon to ensure ongoing support. For "
"more details, see: [Google Cloud Client Libraries Supported Python Versions policy](https://cloud.google.com/python/docs/supported-python-versions)",
FutureWarning,
stacklevel=2, # Point warning to the caller of __init__
)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you use python_requires in setup.py, I'm having a hard time seeing how someone could end up in this state.

Do we want to issue this in a separate PR and release so folks see it before fully dropping?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can do that.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.



# sys_major, sys_minor, sys_micro = _versions_helpers.extract_runtime_version()
# if sys_major == 3 and sys_minor in (7, 8):
# warnings.warn(
# "The python-bigquery library as well as the python-db-dtypes-pandas library no "
# "longer supports Python 3.7 and Python 3.8. "
# f"Your Python version is {sys_major}.{sys_minor}.{sys_micro}. We "
# "recommend that you update soon to ensure ongoing support. For "
# "more details, see: [Google Cloud Client Libraries Supported Python Versions policy](https://cloud.google.com/python/docs/supported-python-versions)",
# FutureWarning,
# )

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's remove this commented out code.

Suggested change
# sys_major, sys_minor, sys_micro = _versions_helpers.extract_runtime_version()
# if sys_major == 3 and sys_minor in (7, 8):
# warnings.warn(
# "The python-bigquery library as well as the python-db-dtypes-pandas library no "
# "longer supports Python 3.7 and Python 3.8. "
# f"Your Python version is {sys_major}.{sys_minor}.{sys_micro}. We "
# "recommend that you update soon to ensure ongoing support. For "
# "more details, see: [Google Cloud Client Libraries Supported Python Versions policy](https://cloud.google.com/python/docs/supported-python-versions)",
# FutureWarning,
# )

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

# Perform the version check
_check_python_version()

# Assign __all__ by calling the function
__all__ = _determine_all(JSONArray, JSONDtype)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems a bit too "magic" to me, but I suppose it could make testing easier?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This code was originally just in the file with no function def. We added _check_python_version...

To test against supported and unsupported versions we attempted to mock the version but timing became an issue, partly because dependencies also rely on version checks so our mock ended up preventing error-free installs of some dependencies.

We tried half a dozen ways to alter when the mock happened but none hit that sweet spot of reliably allowing the dependencies to do their thing and slipping in to allow our test to run.

Similarly, we ran into a mess of problems with import timing and mocking whether JSONArray JSONDtype did or did not import as expected and so ...

putting them in functions made things much easier in the testing arena.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any hard feelings if we resolve this conversation?

3 changes: 0 additions & 3 deletions db_dtypes/core.py
Original file line number Diff line number Diff line change
Expand Up @@ -186,9 +186,6 @@ def median(
keepdims: bool = False,
skipna: bool = True,
):
if not hasattr(pandas_backports, "numpy_validate_median"):
raise NotImplementedError("Need pandas 1.3 or later to calculate median.")

pandas_backports.numpy_validate_median(
(),
{"out": out, "overwrite_input": overwrite_input, "keepdims": keepdims},
Expand Down
Loading
Loading