Cloud composer python311 upgrade

[mjumbewu]

TESTING IN PROGRESS

• Update Dockerfile and Dockerfile.composer to use apache/airflow:2.10.2-python3.11
• Create a requirements-composer-2.11.1-airflow-2.10.2.txt file with the appropriate dependencies
• Create a Python 3.11 environment and install the dependencies

  pip install -r requirements-composer-2.11.1-airflow-2.10.2.txt -r requirements.txt

• Verify that the docker image builds at all. Useful to do this as a separate step for better visibility into errors:

  docker build . -t calitp-airflow-test

• Follow other testing instructions in the airflow/README.md file.

As of 2025 Feb 6, the following packages in requirements.txt of the airflow image need to be updated to work with Python 3.11:

• boto3 -- our Airflow image was pinned to boto3==1.36.15, which requires botocore 1.29.165, requires urllib3<1.27, which is too old for the newer Composer images.
• platformdirs -- our image was pinned to platformdirs<3,>=2.5, whereas the current Composer image requires 4.3.6. Oddly, the previous Composer image we were using required platformdirs==3.2.0, so I'm not actually sure how that was working without conflict (except that our requirements were installed after the Composer requirements).
• pydantic -- We had pinned pydantic==1.9 because of typing extension conflicts, which should no longer be an issue, as we are using Python 3.11 in the new Composer image.

Additionally, the following package version requirements have been loosened in calitp-data-infra:

• pydantic = ">1.9"
• pendulum = ">2.1.2"
• google-cloud-secret-manager = ">2.16.4"

Replaced all imports from pydantic to use pydantic.v1 compatibility layer within Pydantic 2+ (found with pydantic(?!\.v1)).

In order to test the updated calitp-data-infra package, I had to copy the packages folder into the airflow folder, add COPY ./packages/calitp-data-infra/ /tmp/calitp-data-infra/ to the airflow Dockerfile, and then use calitp-data-infra @ file:///tmp/calitp-data-infra in the requirements file.

As a follow-on, we should consider upgrading our use of Pydantic to the latest version. The bump-pydantic tool should help with that.

Description

Describe your changes and why you're making them. Please include the context, motivation, and relevant dependencies.

Resolves #[issue]

Type of change

• Bug fix (non-breaking change which fixes an issue)
• New feature
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Documentation

How has this been tested?

Include commands/logs/screenshots as relevant.

If making changes to dbt models, please run the command poetry run dbt run -s CHANGED_MODEL and poetry run dbt test -s CHANGED_MODEL, then include the output in this section of the PR.

Post-merge follow-ups

Document any actions that must be taken post-merge to deploy or otherwise implement the changes in this PR (for example, running a full refresh of some incremental model in dbt). If these actions will take more than a few hours after the merge or if they will be completed by someone other than the PR author, please create a dedicated follow-up issue and link it here to track resolution.

• No action required
• Actions required (specified below)