chore(ci): drop support for Python 3.7

[leandrodamascena]

Issue number: #3637

PLEASE DO NOT MERGE BEFORE February 8th

Summary

Changes

AWS Lambda runtime will fully deprecate Python 3.7 on February 8th. This means, 3.7 functions will be unable to be updated. Inline with this, Powertools releases will stop supporting it.

User experience

Customers will no longer be able to use Powertools for AWS Lambda (Python) with Python 3.7.

Improvements

In this pull request, we've made some enhancements, taking into account the removal of Python 3.7 support.

1. We've eliminated Python 3.7-specific code from the Logger utility to maintain compatibility.
2. We are importing directly from typing new types added in Python 3.8 and previously only available in the typing_extensions library.
3. Now we are caching properties in the event source data class with the new cached_property decorator.
4. We removed some code from the Streaming utility and added an alias to use the version supported by botocore.

Before

"""
Currently, the same as https://github.com/boto/botocore/blob/b9c540905a6c9/botocore/response.py
We created this because the version of StreamingBody included with the Lambda Runtime is too old, and
doesn't support many of the standard IO methods (like readline).

As soon as the version of botocore included with the Lambda runtime is equal or greater than 1.29.13, we can drop
this file completely. See https://docs.aws.amazon.com/lambda/latest/dg/lambda-python.html.
"""
import logging
from io import IOBase
from typing import Optional

import botocore
from botocore import endpoint
from botocore.compat import set_socket_timeout
from botocore.exceptions import (
    IncompleteReadError,
    ReadTimeoutError,
    ResponseStreamingError,
)
from urllib3.exceptions import ProtocolError as URLLib3ProtocolError
from urllib3.exceptions import ReadTimeoutError as URLLib3ReadTimeoutError

logger = logging.getLogger(__name__)

# Splitting the botocore version string into major, minor, and patch versions,
# and performing a conditional check based on the extracted versions.
major, minor, patch = map(int, botocore.__version__.split("."))

if major == 1 and (minor < 29 or patch < 13):

    class PowertoolsStreamingBody(IOBase):
        """Wrapper class for a HTTP response body.

        This provides a few additional conveniences that do not exist
        in the urllib3 model:
            * Set the timeout on the socket (i.e read() timeouts)
            * Auto validation of content length, if the amount of bytes
              we read does not match the content length, an exception
              is raised.
        """

        _DEFAULT_CHUNK_SIZE = 1024

        def __init__(self, raw_stream, content_length):
            self._raw_stream = raw_stream
            self._content_length = content_length
            self._amount_read = 0

        def __del__(self):
            # Extending destructor in order to preserve the underlying raw_stream.
            # The ability to add custom cleanup logic introduced in Python3.4+.
            # https://www.python.org/dev/peps/pep-0442/
            pass

        def set_socket_timeout(self, timeout):
            """Set the timeout seconds on the socket."""
            # The problem we're trying to solve is to prevent .read() calls from
            # hanging.  This can happen in rare cases.  What we'd like to ideally
            # do is set a timeout on the .read() call so that callers can retry
            # the request.
            # Unfortunately, this isn't currently possible in requests.
            # See: https://github.com/kennethreitz/requests/issues/1803
            # So what we're going to do is reach into the guts of the stream and
            # grab the socket object, which we can set the timeout on.  We're
            # putting in a check here so in case this interface goes away, we'll
            # know.
            try:
                set_socket_timeout(self._raw_stream, timeout)
            except AttributeError:
                logger.error(
                    "Cannot access the socket object of "
                    "a streaming response.  It's possible "
                    "the interface has changed.",
                    exc_info=True,
                )
                raise

        def readable(self):
            try:
                return self._raw_stream.readable()
            except AttributeError:
                return False

        def read(self, amt=None):
            """Read at most amt bytes from the stream.
            If the amt argument is omitted, read all data.
            """
            try:
                chunk = self._raw_stream.read(amt)
            except URLLib3ReadTimeoutError as e:
                raise ReadTimeoutError(endpoint_url=e.url, error=e)
            except URLLib3ProtocolError as e:
                raise ResponseStreamingError(error=e)
            self._amount_read += len(chunk)
            if amt is None or (not chunk and amt > 0):
                # If the server sends empty contents or
                # we ask to read all of the contents, then we know
                # we need to verify the content length.
                self._verify_content_length()
            return chunk

        def readlines(self, hint: Optional[int] = -1):
            return self._raw_stream.readlines(hint)

        def __iter__(self):
            """Return an iterator to yield 1k chunks from the raw stream."""
            return self.iter_chunks(self._DEFAULT_CHUNK_SIZE)

        def __next__(self):
            """Return the next 1k chunk from the raw stream."""
            current_chunk = self.read(self._DEFAULT_CHUNK_SIZE)
            if current_chunk:
                return current_chunk
            raise StopIteration()

        def __enter__(self):
            return self._raw_stream

        def __exit__(self, *args):
            self._raw_stream.close()

        next = __next__  # noqa: A003, VNE003

        def iter_lines(self, chunk_size=_DEFAULT_CHUNK_SIZE, keepends=False):
            """Return an iterator to yield lines from the raw stream.
            This is achieved by reading chunk of bytes (of size chunk_size) at a
            time from the raw stream, and then yielding lines from there.
            """
            pending = b""
            for chunk in self.iter_chunks(chunk_size):
                lines = (pending + chunk).splitlines(True)
                for line in lines[:-1]:
                    yield line.splitlines(keepends)[0]
                pending = lines[-1]
            if pending:
                yield pending.splitlines(keepends)[0]

        def iter_chunks(self, chunk_size=_DEFAULT_CHUNK_SIZE):
            """Return an iterator to yield chunks of chunk_size bytes from the raw
            stream.
            """
            while True:
                current_chunk = self.read(chunk_size)
                if current_chunk == b"":
                    break
                yield current_chunk

        def _verify_content_length(self):
            # See: https://github.com/kennethreitz/requests/issues/1855
            # Basically, our http library doesn't do this for us, so we have
            # to do this ourself.
            if self._content_length is not None and self._amount_read != int(self._content_length):
                raise IncompleteReadError(
                    actual_bytes=self._amount_read,
                    expected_bytes=int(self._content_length),
                )

        def tell(self):
            return self._raw_stream.tell()

        def close(self):
            """Close the underlying http response stream."""
            self._raw_stream.close()

    def convert_to_response_dict(http_response, operation_model):
        """Convert an HTTP response object to a request dict.

        This converts the requests library's HTTP response object to
        a dictionary.

        :type http_response: botocore.vendored.requests.model.Response
        :param http_response: The HTTP response from an AWS service request.

        :rtype: dict
        :return: A response dictionary which will contain the following keys:
            * headers (dict)
            * status_code (int)
            * body (string or file-like object)

        """
        response_dict = {
            "headers": http_response.headers,
            "status_code": http_response.status_code,
            "context": {
                "operation_name": operation_model.name,
            },
        }
        if response_dict["status_code"] >= 300:
            response_dict["body"] = http_response.content
        elif operation_model.has_event_stream_output:
            response_dict["body"] = http_response.raw
        elif operation_model.has_streaming_output:
            length = response_dict["headers"].get("content-length")
            response_dict["body"] = PowertoolsStreamingBody(http_response.raw, length)
        else:
            response_dict["body"] = http_response.content
        return response_dict

    # monkey patch boto3
    endpoint.convert_to_response_dict = convert_to_response_dict
else:
    from botocore.response import StreamingBody

    # Expose PowertoolsStreamingBody as StreamingBody
    vars()["PowertoolsStreamingBody"] = StreamingBody

After

from botocore.response import StreamingBody

PowertoolsStreamingBody = StreamingBody

Checklist

If your change doesn't seem to apply, please leave them unchecked.

• Meet tenets criteria
• I have performed a self-review of this change
• Changes have been tested
• Changes are documented
• PR title follows conventional commit semantics

Is this a breaking change?

RFC issue number:

Checklist:

• Migration process documented
• Implement warnings (if it can live side by side)

Acknowledgment

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Disclaimer: We value your time and bandwidth. As such, any pull requests created on non-triaged issues might not be successful.

[codecov-commenter]

Codecov Report

Attention: 1 lines in your changes are missing coverage. Please review.

Comparison is base (2a33556) 95.50% compared to head (62ea51a) 96.38%.
Report is 2 commits behind head on develop.

Files	Patch %	Lines
...s/utilities/data_classes/kinesis_firehose_event.py	80.00%	1 Missing ⚠️

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #3638      +/-   ##
===========================================
+ Coverage    95.50%   96.38%   +0.87%     
===========================================
  Files          215      214       -1     
  Lines        10140    10030     -110     
  Branches      1867     1846      -21     
===========================================
- Hits          9684     9667      -17     
+ Misses         343      259      -84     
+ Partials       113      104       -9     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[heitorlessa]

On 2, IIRC we can rely on typing_extensions for everything and stop using from typing altogether - it'll handle versions gracefully.

On 4, could you add the botocore version (before/after) for completeness?

Just in case.. are there any # maintenance: notes about 3.7 that we can do as part of this PR too?

---

> grep -i "# maintenance" -R aws_lambda_powertools/ -R tests | grep "3.7"

aws_lambda_powertools//logging/logger.py:        # Maintenance: We can drop this upon Py3.7 EOL. It's a backport for "location" key to work
aws_lambda_powertools//logging/logger.py:        # Maintenance: We can drop this upon Py3.7 EOL. It's a backport for "location" key to work
aws_lambda_powertools//logging/logger.py:        # Maintenance: We can drop this upon Py3.7 EOL. It's a backport for "location" key to work
aws_lambda_powertools//logging/logger.py:        # Maintenance: We can drop this upon Py3.7 EOL. It's a backport for "location" key to work
aws_lambda_powertools//logging/logger.py:        # Maintenance: We can drop this upon Py3.7 EOL. It's a backport for "location" key to work
aws_lambda_powertools//logging/logger.py:        # Maintenance: We can drop this upon Py3.7 EOL. It's a backport for "location" key to work
aws_lambda_powertools//logging/logger.py:        # Maintenance: We can drop this upon Py3.7 EOL. It's a backport for "location" key
tests/e2e/idempotency_redis/conftest.py:    # MAINTENANCE: Add the Stack constructor when Python 3.7 is dropped

[leandrodamascena]

On 2, IIRC we can rely on typing_extensions for everything and stop using from typing altogether - it'll handle versions gracefully.

There are many files where we import from typing and typing_extensions. To keep the proposal of this PR, I will maintain the imports we are using today, but I will open a new PR to resolve this, there are at least 30 files to change and maintain this standard.

On 4, could you add the botocore version (before/after) for completeness?

Adding now.

Just in case.. are there any # maintenance: notes about 3.7 that we can do as part of this PR too?

> grep -i "# maintenance" -R aws_lambda_powertools/ -R tests | grep "3.7"

aws_lambda_powertools//logging/logger.py:        # Maintenance: We can drop this upon Py3.7 EOL. It's a backport for "location" key to work
aws_lambda_powertools//logging/logger.py:        # Maintenance: We can drop this upon Py3.7 EOL. It's a backport for "location" key to work
aws_lambda_powertools//logging/logger.py:        # Maintenance: We can drop this upon Py3.7 EOL. It's a backport for "location" key to work
aws_lambda_powertools//logging/logger.py:        # Maintenance: We can drop this upon Py3.7 EOL. It's a backport for "location" key to work
aws_lambda_powertools//logging/logger.py:        # Maintenance: We can drop this upon Py3.7 EOL. It's a backport for "location" key to work
aws_lambda_powertools//logging/logger.py:        # Maintenance: We can drop this upon Py3.7 EOL. It's a backport for "location" key to work
aws_lambda_powertools//logging/logger.py:        # Maintenance: We can drop this upon Py3.7 EOL. It's a backport for "location" key
tests/e2e/idempotency_redis/conftest.py:    # MAINTENANCE: Add the Stack constructor when Python 3.7 is dropped

I've already removed them all, you're probably running this grep on a branch other than this one, right?

Thanks

[heitorlessa]

Yeah different branch, it was just for illustration ;-)

Will review as soon as I have a break in a customer meeting

[heitorlessa]

one tiny suggestion to add a comment, and one question as I suspect docs code highlighting will be impacted with the change.

[sonarqubecloud]

Quality Gate passed

Kudos, no new issues were introduced!

0 New issues
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarCloud

[heitorlessa]

SHIIIIIPP IT! :) great work as always @leandrodamascena