feat: add psycopg2 support (Rd 11304) (#513)

* feat: add psycopg2 support (no payload)

* fix: 3.11 tests and compatibility matrix values

* chore: add script to simplify new instrumentation setup

* refactor: span parser for multiple root span handling

Author: therightstuff

.gitignore

@@ -81,9 +81,6 @@ target/
 profile_default/
 ipython_config.py
 
-# pyenv
-.python-version
-
 # pipenv
 #   According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
 #   However, in case of collaboration, if having platform-specific dependencies or dependencies

.python-version

@@ -0,0 +1 @@
+3.9.9

CONTRIBUTING.md

@@ -15,7 +15,17 @@ Getting your machine ready to develop against the package is a straightforward p
 1. Run the setup script: `python setup.py develop`.
 1. Run `pre-commit install` in your repository to install pre-commit hooks
 
-**Note**: If you are using pycharm, ensure that you set it to use the virtualenv virtual environment manager. This is available in the menu under `PyCharm -> Preferences -> Project -> Interpreter`
+### MacOS Users
+
+Psycopg2 cannot be installed without the `pg_config` executable available, for which you'll need to install `postgres`.
+
+You can do this with `brew update && brew install postgresql`.
+
+It might be necessary to throw in a `brew tap homebrew/core` along the way.
+
+### PyCharm Users
+
+If you are using pycharm, ensure that you set it to use the virtualenv virtual environment manager. This is available in the menu under `PyCharm -> Preferences -> Project -> Interpreter`
 
 ## Running the test suite
 
@@ -57,11 +67,8 @@ TEST_ONLY_UNTESTED_NEW_VERSIONS=true python3 -m nox -e integration_tests_flask`
 
 ## Adding support for a new package
 
-The `tested_versions` folder must be present and updated both under `src/lumigo_opentelemetry/instrumentations` and in the tested package's folder under `src/test`. For this to work, create a symlink from the package's `src/test` folder.
+The `tested_versions` folder must be present and updated both under `src/lumigo_opentelemetry/instrumentations` and in the tested package's folder under `src/test`. For this to work, you'll need a symlink from the package's `src/test` folder. It's recommended to use the script below to initialize your `tested_versions` folders:
 
 ```sh
-mkdir -p src/lumigo_opentelemetry/instrumentations/<PACKAGE_NAME>/tested_versions
-touch src/lumigo_opentelemetry/instrumentations/<PACKAGE_NAME>/tested_versions/<PACKAGE_NAME>
-cd src/test/integration/<PACKAGE_NAME>
-ln -s ../../../lumigo_opentelemetry/instrumentations/<PACKAGE_NAME>/tested_versions tested_versions
+./scripts/init_tested_versions.sh
 ```

README.md

@@ -206,26 +206,27 @@ get_current_span().add_event('<error-message>', {'lumigo.type': '<error-type>'})
 | Instrumentation | Package | Supported Versions | | | | |
 | --- | --- | :---: | :---: | :---: | :---: | :---: |
 | | | 3.7 | 3.8 | 3.9 | 3.10 | 3.11 |
-| botocore | [boto3](https://pypi.org/project/boto3) | 1.17.22~1.28.40|1.17.22~1.28.40|1.17.22~1.28.40|1.17.22~1.28.40| |
+| botocore | [boto3](https://pypi.org/project/boto3) | 1.17.22~1.28.40|1.17.22~1.28.40|1.17.22~1.28.40|1.17.22~1.28.40|1.17.22~1.28.40|
 | django | [django](https://pypi.org/project/django) |  |4.2.5|4.2.5|4.2.5|4.2.5|
 | fastapi | [fastapi](https://pypi.org/project/fastapi) | 0.56.1~0.100.0|0.56.1~0.100.0|0.56.1~0.100.0|0.56.1~0.100.0|0.56.1~0.100.0|
 | | | 0.100.0b2~0.103.1| 0.100.0b2~0.103.1| 0.100.0b2~0.103.1| 0.100.0b2~0.103.1| 0.100.0b2~0.103.1|
 |  | [uvicorn](https://pypi.org/project/uvicorn) | 0.11.3~0.22.0|0.11.3~0.22.0|0.11.3~0.22.0|0.11.3~0.22.0|0.12.0~0.22.0|
 | flask | [flask](https://pypi.org/project/flask) | 2.0.0~2.2.5|2.0.0~2.2.5|2.0.0~2.2.5|2.0.0~2.2.5|2.0.0~2.2.5|
 | grpcio | [grpcio](https://pypi.org/project/grpcio) | 1.45.0~1.58.0rc1|1.45.0~1.58.0rc1|1.45.0~1.58.0rc1|1.45.0~1.58.0rc1|1.49.0~1.58.0rc1|
-| kafka_python | [kafka_python](https://pypi.org/project/kafka_python) | 2.0.0~2.0.2|2.0.0~2.0.2|2.0.0~2.0.2|2.0.0~2.0.2| |
-| pika | [pika](https://pypi.org/project/pika) | 1.0.0|1.0.0|1.0.0|1.0.0| |
-| | | 1.0.1~1.3.0| 1.0.1~1.3.0| 1.0.1~1.3.0| 1.0.1~1.3.0| |
-| | | 1.3.0rc5~1.3.2| 1.3.0rc5~1.3.2| 1.3.0rc5~1.3.2| 1.3.0rc5~1.3.2| |
+| kafka_python | [kafka_python](https://pypi.org/project/kafka_python) | 2.0.0~2.0.2|2.0.0~2.0.2|2.0.0~2.0.2|2.0.0~2.0.2|2.0.0~2.0.2|
+| pika | [pika](https://pypi.org/project/pika) | 1.0.0|1.0.0|1.0.0|1.0.0|1.0.0|
+| | | 1.0.1~1.3.0| 1.0.1~1.3.0| 1.0.1~1.3.0| 1.0.1~1.3.0| 1.0.1~1.3.0|
+| | | 1.3.0rc5~1.3.2| 1.3.0rc5~1.3.2| 1.3.0rc5~1.3.2| 1.3.0rc5~1.3.2| 1.3.0rc5~1.3.2|
+| psycopg2 | [psycopg2](https://pypi.org/project/psycopg2) | 2.9|2.9|2.9|2.9.5|2.9.5|
 | pymongo | [pymongo](https://pypi.org/project/pymongo) | 3.1.1~3.3.1|3.1.1~3.3.1|3.1.1~3.3.1|3.1.1~3.3.1|3.1.1~3.3.1|
-| | | 3.5.0~3.13.0| 3.5.0~3.13.0| 3.5.0~3.13.0| 3.5.0~3.13.0| 3.5.0~3.9.0|
-| | | 4.0.1~4.5.0| 4.0.1~4.5.0| 4.0.1~4.5.0| 4.0.1~4.5.0| 3.1|
-| | | 3.1| 3.1| 3.1| 3.1| 3.2|
-| | | 3.2| 3.2| 3.2| 3.2| |
-| | | 4.0| 4.0| 4.0| 4.0| |
-| pymysql | [pymysql](https://pypi.org/project/pymysql) | 0.9.0~0.10.1|0.9.0~0.10.1|0.9.0~0.10.1|0.9.0~0.10.1| |
-| | | 1.0.0~1.0.3| 1.0.0~1.0.3| 1.0.0~1.0.3| 1.0.0~1.0.3| |
-| | | 1.1.0~1.1.0rc2| 1.1.0~1.1.0rc2| 1.1.0~1.1.0rc2| 1.1.0~1.1.0rc2| |
+| | | 3.5.0~3.13.0| 3.5.0~3.13.0| 3.5.0~3.13.0| 3.5.0~3.13.0| 3.5.0~3.13.0|
+| | | 4.0.1~4.5.0| 4.0.1~4.5.0| 4.0.1~4.5.0| 4.0.1~4.5.0| 4.0.1~4.5.0|
+| | | 3.1| 3.1| 3.1| 3.1| 3.1|
+| | | 3.2| 3.2| 3.2| 3.2| 3.2|
+| | | 4.0| 4.0| 4.0| 4.0| 4.0|
+| pymysql | [pymysql](https://pypi.org/project/pymysql) | 0.9.0~0.10.1|0.9.0~0.10.1|0.9.0~0.10.1|0.9.0~0.10.1|0.9.0~0.10.1|
+| | | 1.0.0~1.0.3| 1.0.0~1.0.3| 1.0.0~1.0.3| 1.0.0~1.0.3| 1.0.0~1.0.3|
+| | | 1.1.0~1.1.0rc2| 1.1.0~1.1.0rc2| 1.1.0~1.1.0rc2| 1.1.0~1.1.0rc2| 1.1.0~1.1.0rc2|
 | redis | [redis](https://pypi.org/project/redis) | 4.1.1~4.2.0|4.1.1~4.2.0|4.1.1~4.2.0|4.1.1~4.2.0|4.1.1~4.2.0|
 | | | 4.2.1~4.6.0| 4.2.1~4.6.0| 4.2.1~4.6.0| 4.2.1~4.6.0| 4.2.1~4.6.0|
 | | | 5.0.0~5.0.0rc2| 5.0.0~5.0.0rc2| 5.0.0~5.0.0rc2| 5.0.0~5.0.0rc2| 5.0.0~5.0.0rc2|

noxfile.py

@@ -397,12 +397,12 @@ def component_tests(session):
     component_tests_attr_max_size(
         session=session,
         fastapi_version="0.78.0",  # arbitrary version
-        uvicorn_version="0.16.0",  # arbitrary version
+        uvicorn_version="0.16.0",  # TODO don't update, see https://lumigo.atlassian.net/browse/RD-11466
     )
     component_tests_execution_tags(
         session=session,
         fastapi_version="0.78.0",  # arbitrary version
-        uvicorn_version="0.16.0",  # arbitrary version
+        uvicorn_version="0.16.0",  # TODO don't update, see https://lumigo.atlassian.net/browse/RD-11466
     )
 
 
@@ -797,6 +797,52 @@ def integration_tests_pika(
                 kill_process_and_clean_outputs(temp_file, "uvicorn", session)
 
 
+@nox.session()
+@nox.parametrize(
+    "python,psycopg2_version",
+    [
+        (python, psycopg2_version)
+        for python in python_versions()
+        for psycopg2_version in dependency_versions_to_be_tested(
+            python=python,
+            directory="psycopg2",
+            dependency_name="psycopg2",
+        )
+    ],
+)
+def integration_tests_psycopg2(
+    session,
+    python,
+    psycopg2_version,
+):
+    with TestedVersions.save_tests_result(
+        "psycopg2", python, "psycopg2", psycopg2_version
+    ):
+        install_package("psycopg2", psycopg2_version, session)
+
+        session.install(".")
+
+        temp_file = create_it_tempfile("psycopg2")
+        with session.chdir("src/test/integration/psycopg2"):
+            session.install("-r", OTHER_REQUIREMENTS)
+
+            try:
+                session.run(
+                    "pytest",
+                    "--tb",
+                    "native",
+                    "--log-cli-level=INFO",
+                    "--color=yes",
+                    "-v",
+                    "./tests/test_psycopg2.py",
+                    env={
+                        "LUMIGO_DEBUG_SPANDUMP": temp_file,
+                    },
+                )
+            finally:
+                kill_process_and_clean_outputs(temp_file, "test_psycopg2", session)
+
+
 @nox.session()
 @nox.parametrize(
     "python,pymongo_version",
@@ -1001,6 +1047,7 @@ def integration_tests_redis(
         temp_file = create_it_tempfile("redis")
         with session.chdir("src/test/integration/redis"):
             session.install("-r", OTHER_REQUIREMENTS)
+
             try:
                 session.run(
                     "pytest",

scripts/checks.sh

@@ -5,9 +5,9 @@ set -eo pipefail
 pre-commit run -a
 
 # run u.t
-pushd src/test/
-pytest ci
-pytest unit
+pushd src/test
+    pytest ci
+    pytest unit
 popd
 
 # branch name validation
@@ -16,3 +16,6 @@ then
     # Check if branch contains RD ticket value.
     echo "$CIRCLE_BRANCH" | grep -E "[RDrd]-[0-9]+|master|version-testing-[0-9]{4}[0-9]{2}[0-9]{2}|dependabot/" || { echo "Please create a relevent ticket in Jira and connect it to this branch. Use jiranch." ; exit 1; }
 fi
+
+# update the README's package compatibility matrix
+python3 -m scripts.update_supported_packages_documentation

scripts/init_tested_versions.sh

@@ -0,0 +1,35 @@
+#!/usr/bin/env bash
+
+PACKAGE_NAME="$1"
+SUPPORTED_VERSIONS=(3.7 3.8 3.9 3.10 3.11)
+
+if [ -z "$PACKAGE_NAME" ]; then
+    echo "Usage: $0 <PACKAGE_NAME>"
+    exit 1
+fi
+
+base_folder="src/lumigo_opentelemetry/instrumentations/${PACKAGE_NAME}/tested_versions"
+test_folder="src/test/integration/${PACKAGE_NAME}"
+
+echo "Creating tested versions folders under ${base_folder}..."
+for python_version in "${SUPPORTED_VERSIONS[@]}"; do
+    echo "Creating ${python_version} folder..."
+    version_folder="${base_folder}/${python_version}"
+    mkdir -p "$version_folder"
+    touch "${version_folder}/${PACKAGE_NAME}"
+done
+
+echo ""
+echo "Creating tested versions symlink in ${test_folder}..."
+mkdir -p "$test_folder"
+pushd "$test_folder"
+    if [ -e tested_versions ]; then
+        echo "Removing old tested_versions symlink..."
+        rm tested_versions
+    fi
+    echo "linking from ${test_folder} to ${base_folder}..."
+    ln -s ../../../../${base_folder} tested_versions
+popd
+
+echo ""
+echo "Done"

setup.py

@@ -31,6 +31,7 @@
         "opentelemetry-instrumentation-grpc==0.36b0",
         "opentelemetry-instrumentation-kafka-python==0.36b0",
         "opentelemetry-instrumentation-pika==0.36b0",
+        "opentelemetry-instrumentation-psycopg2==0.36b0",
         "opentelemetry-instrumentation-pymongo==0.36b0",
         "opentelemetry-instrumentation-pymysql==0.36b0",
         "opentelemetry-instrumentation-requests==0.36b0",