Init the new loader extension (#2684)

* Init the dd_library_loader extension for Single Step Instrumentation

* Move load of the extension directly in the api_no_check

* Check the 'core' ext instead of 'json'

* Make the code more robust and PHP 5 compatible

* Rename variable for clarity + suppress compiler warning

* Init CI jobs

* CI try to fix Docke issue with Alpine image

* Register ddtrace during the registration of the loader again.

To workaround the "Module alreay loaded" issue, register ddtrace under the name "ddtrace_injected", then remame it at starup if "ddtrace" is not loaded

* Rename the module key at the bucket level

* Remove PHP5 support + cleanup

* Fix compat for all PHP 7.x

* Cleanup compat code + rework deps handling

* Remove custom zend_hash_find_bucket thanks to Bob's trick

* Use a weak pointer to use the real implementation of zend_hash_set_bucket_key if it exists

* Add defensive barriers

* Improve logs by adding log level and allowing variadic args

* Fix renaming of the module not working properly

* Init tests

* Fix zend_string_init/_release compat with PHP <= 7.2

* Reformat code

* Init README.md

* Add optional dependencies to 'json' and 'standard' modules to make sure ddtrace_injected is loaded after them

* Apply suggestions from code review

Co-authored-by: Bob Weinand <bob.weinand@datadoghq.com>

* Fix some review's comments

* Log error when registering module

* Make the code generic to load every Datadog extensions

* Init telemetry support

* Workaround PHP 7.2 segfault

* Cleanup .gitignore and config.w32

* Improve telemetry

* Retrieve the runtime_version from the engine to work even with future unknown versions

* Compile with -Wextra -Werror

* Add tests

* Merge loader pipeline back into build_packages

* Fix typo

* Fix zend_error_cb handling

* Fix CI

* Set 'datadog.trace.sources_path' INI setting when ddtrace_injected is used

* Fix coding style (using clang-format)

* Don't use symbols that don't exist in PHP5 to avoid a missing symbol error

* Fix telemetry_forwarder arguments

* Fix segfault on PHP 7.2 when calling zend_string_release

* Add test loading using PHP_INI_SCAN_DIR env var

* Address Bob's comments

* Make DD_LOADER_PACKAGE_PATH env var mandatory

---------

Co-authored-by: Bob Weinand <bob.weinand@datadoghq.com>

Author: iamluc

.circleci/continue_config.yml

@@ -534,7 +534,7 @@ commands:
                   -v /tmp/artifacts:/tmp/artifacts \
                   $(if [ -n "${CARGO_TARGET_DIR:-}" ]; then echo -v ${CARGO_TARGET_DIR}:${CARGO_TARGET_DIR} -e CARGO_TARGET_DIR=${CARGO_TARGET_DIR}; fi) \
                   $image \
-                  bash -c 'echo Started; sleep 10000' 2>&1 | tee /tmp/docker.out &
+                  sh -c 'echo Started; sleep 10000' 2>&1 | tee /tmp/docker.out &
               tail -F /tmp/docker.out | grep -Em1 'Started|Error response from daemon'
               if ! grep -q "Error response from daemon" /tmp/docker.out; then
                 container_name=$(docker ps | grep "$image" | awk '{ print $NF }' | head -1)
@@ -549,6 +549,10 @@ commands:
               fi
               sleep 1
             done
+            # On Alpine images, make sure bash is installed
+            if [[ "<< parameters.docker_image >>" == *alpine* ]]; then
+              docker exec -w /home/circleci/datadog -i "${container_name}" sh -c "test -z bash || apk --no-cache add bash" || true
+            fi
             echo 'if [ -f /usr/bin/docker ]; then pid=$$; IFS= readarray -d "" args < <(cat /proc/$pid/cmdline); exec docker exec -w /home/circleci/datadog -i '"${container_name}"' "${args[@]}"; fi' | tee -a $BASH_ENV
             cp $BASH_ENV /tmp/bashenv/bash.sh
             chmod 777 /tmp/bashenv/bash.sh
@@ -2941,6 +2945,129 @@ jobs:
             sed -i -e "s/crate-type.*$/crate-type = [\"rlib\"]/g" Cargo.toml
             cargo clippy --all-targets --all-features -- -D warnings -Aunknown-lints
 
+  "compile_loader":
+    working_directory: ~/datadog
+    parameters:
+      docker_image:
+        type: string
+      os:
+        type: string
+      resource_class:
+        type: string
+        default: medium
+    <<: *BARE_DOCKER_MACHINE
+    steps:
+      - restore_cache:
+          keys:
+            - source-v1-{{ .Branch }}-{{ .Revision }}
+      - <<: *STEP_CHECKOUT
+      - <<: *STEP_ATTACH_WORKSPACE
+      - setup_docker:
+          docker_image: << parameters.docker_image >>
+      - run:
+          name: Compile the loader extension
+          command: |
+            set -xeuo pipefail
+
+            if [[ "<< parameters.os >>" == "linux-musl" ]]; then
+              apk add --no-cache \
+                autoconf \
+                coreutils \
+                g++ \
+                gcc \
+                make
+            fi
+
+            cd loader
+            phpize
+            ./configure
+            make clean
+            make -j all ECHO_ARG="-e" CFLAGS="-std=gnu11 -O2 -g -Wall -Wextra -Werror"
+            cp modules/dd_library_loader.so modules/dd_library_loader-$(uname -m)-<< parameters.os >>.so
+      - persist_to_workspace:
+          root: '.'
+          paths: [ './loader/modules/dd_library_loader-*.so' ]
+      - store_artifacts:
+          path: ./loader/modules/dd_library_loader.so
+
+  "test_loader_linux":
+    working_directory: ~/datadog
+    parameters:
+      docker_image:
+        type: string
+      php_major_minor:
+        # Expected in the format: <major>.<minor>, e.g. 8.2
+        type: string
+      switch_php_version:
+        type: string
+        default: none
+      resource_class:
+        type: string
+        default: medium
+    <<: *BARE_DOCKER_MACHINE
+    steps:
+      - restore_cache:
+          keys:
+            - source-v1-{{ .Branch }}-{{ .Revision }}
+      - <<: *STEP_CHECKOUT
+      - <<: *STEP_ATTACH_WORKSPACE
+      - setup_docker:
+          docker_image: << parameters.docker_image >>
+      - switch_php:
+          php_version: << parameters.switch_php_version >>
+      - run:
+          name: Test the loader
+          command: |
+            set -xeuo pipefail
+
+            arch=$(uname -m)
+            package=$(find ./build/packages -maxdepth 1 -name "dd-library-php-*-${arch}-linux-gnu.tar.gz")
+            rm -rf dd-library-php
+            tar -xzf ${package}
+            export DD_LOADER_PACKAGE_PATH=${PWD}/dd-library-php
+
+            cd loader
+            cp modules/dd_library_loader-$(uname -m)-linux-gnu.so modules/dd_library_loader.so
+            ./bin/test.sh
+
+  "test_loader_alpine":
+    working_directory: ~/datadog
+    parameters:
+      docker_image:
+        type: string
+      alpine_version:
+        type: string
+      alpine_packages:
+        type: string
+      resource_class:
+        type: string
+        default: medium
+    <<: *BARE_DOCKER_MACHINE
+    steps:
+      - restore_cache:
+          keys:
+            - source-v1-{{ .Branch }}-{{ .Revision }}
+      - <<: *STEP_CHECKOUT
+      - <<: *STEP_ATTACH_WORKSPACE
+      - setup_docker:
+          docker_image: << parameters.docker_image >>
+      - run:
+          name: Test the loader
+          command: |
+            set -xeuo pipefail
+
+            apk add --no-cache curl-dev << parameters.alpine_packages >>
+
+            arch=$(uname -m)
+            package=$(find ./build/packages -maxdepth 1 -name "dd-library-php-*-${arch}-linux-musl.tar.gz")
+            rm -rf dd-library-php
+            tar -xzf ${package}
+            export DD_LOADER_PACKAGE_PATH=${PWD}/dd-library-php
+
+            cd loader
+            cp modules/dd_library_loader-$(uname -m)-linux-musl.so modules/dd_library_loader.so
+            ./bin/test.sh
+
   compile_extension_centos:
     working_directory: ~/datadog
     parameters:
@@ -4418,6 +4545,141 @@ workflows:
             export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin
             TEST_LIBRARY=php ./run.sh PARAMETRIC
 
+      - "compile_loader":
+          requires: [ 'Prepare Code' ]
+          name: "Compile Loader Linux x86_64"
+          docker_image: "datadog/dd-trace-ci:php-8.3_buster"
+          resource_class: "medium"
+          os: "linux-gnu"
+
+      - "compile_loader":
+          requires: [ 'Prepare Code' ]
+          name: "Compile Loader Linux aarch64"
+          docker_image: "datadog/dd-trace-ci:php-8.3_buster"
+          resource_class: "arm.medium"
+          os: "linux-gnu"
+
+      - "compile_loader":
+          requires: [ 'Prepare Code' ]
+          name: "Compile Loader Alpine x86_64"
+          docker_image: "datadog/dd-trace-ci:php-compile-extension-alpine-8.0"
+          resource_class: "medium"
+          os: "linux-musl"
+
+      - "compile_loader":
+          requires: [ 'Prepare Code' ]
+          name: "Compile Loader Alpine aarch64"
+          docker_image: "datadog/dd-trace-ci:php-compile-extension-alpine-8.0"
+          resource_class: "arm.medium"
+          os: "linux-musl"
+
+      - "test_loader_linux":
+          requires:
+            - "package extension"
+            - "Compile Loader Linux x86_64"
+          name: "[Linux/x86_64] Loader PHP << matrix.php_major_minor >>-<< matrix.switch_php_version >>"
+          docker_image: "datadog/dd-trace-ci:php-<< matrix.php_major_minor >>_buster"
+          resource_class: "medium"
+          matrix:
+            parameters:
+              php_major_minor:
+                - "5.6"
+                - "7.0"
+                - "7.1"
+                - "7.2"
+                - "7.3"
+              switch_php_version:
+                - debug
+                - nts
+
+      - "test_loader_linux":
+          requires:
+            - "package extension"
+            - "Compile Loader Linux x86_64"
+          name: "[Linux/x86_64] Loader PHP << matrix.php_major_minor >>-<< matrix.switch_php_version >>"
+          docker_image: "datadog/dd-trace-ci:php-<< matrix.php_major_minor >>_buster"
+          resource_class: "medium"
+          matrix:
+            parameters:
+              php_major_minor:
+                - "7.4"
+                - "8.0"
+                - "8.1"
+                - "8.2"
+                - "8.3"
+              switch_php_version:
+                - debug
+                - zts
+                - nts
+
+      - "test_loader_linux":
+          requires:
+            - "package extension"
+            - "Compile Loader Linux aarch64"
+          name: "[Linux/aarch64] Loader PHP << matrix.php_major_minor >>-<< matrix.switch_php_version >>"
+          docker_image: "datadog/dd-trace-ci:php-<< matrix.php_major_minor >>_buster"
+          resource_class: "arm.medium"
+          matrix:
+            parameters:
+              php_major_minor:
+                - "7.0"
+                - "7.1"
+                - "7.2"
+                - "7.3"
+              switch_php_version:
+                - debug
+                - nts
+
+      - "test_loader_linux":
+          requires:
+            - "package extension"
+            - "Compile Loader Linux aarch64"
+          name: "[Linux/aarch64] Loader PHP << matrix.php_major_minor >>-<< matrix.switch_php_version >>"
+          docker_image: "datadog/dd-trace-ci:php-<< matrix.php_major_minor >>_buster"
+          resource_class: "arm.medium"
+          matrix:
+            parameters:
+              php_major_minor:
+                - "7.4"
+                - "8.0"
+                - "8.1"
+                - "8.2"
+                - "8.3"
+              switch_php_version:
+                - debug
+                - zts
+                - nts
+
+      - "test_loader_alpine":
+          requires:
+            - "package extension"
+            - "Compile Loader Alpine x86_64"
+          name: "[Alpine/x86_64] Loader PHP Alpine << matrix.alpine_version >>"
+          docker_image: "alpine:<< matrix.alpine_version >>"
+          resource_class: "medium"
+          alpine_packages: "php83 php83-dev"
+          matrix:
+            parameters:
+              alpine_version:
+                # - "3.19"
+                - "3.20"
+                # - "latest"
+
+      - "test_loader_alpine":
+          requires:
+            - "package extension"
+            - "Compile Loader Alpine aarch64"
+          name: "[Alpine/aarch64] Loader PHP Alpine << matrix.alpine_version >>"
+          docker_image: "alpine:<< matrix.alpine_version >>"
+          resource_class: "arm.medium"
+          alpine_packages: "php83 php83-dev"
+          matrix:
+            parameters:
+              alpine_version:
+                # - "3.19"
+                - "3.20"
+                # - "latest"
+
   build:
     when: << pipeline.parameters.build >>
     jobs:

.gitignore

@@ -13,14 +13,17 @@ composer.lock*
 .tmp/
 **.code-workspace
 /Makefile*
+/loader/Makefile*
 acinclude.m4
 aclocal.m4
 autom4te*
 build
 configure
 coverage/
 /configure.*
+/loader/configure.*
 /config.*
+/loader/config.*
 !config.m4
 !config.w32
 ddtrace.def
@@ -29,6 +32,8 @@ ext/version.h
 *.lo
 *.o
 *.so
+*~
+*.dep
 install-sh
 *.log
 nginx.pid
@@ -49,6 +54,7 @@ tests/ext/*.log
 tests/ext/*.exp
 tests/ext/*.sh
 tests/ext/*.mem
+phpt.*
 tmp-php.ini
 tmp/opcache*
 *.orig

loader/README.md

@@ -0,0 +1,10 @@
+# DD Library Loader
+
+## Getting Started
+
+This Zend extension (**dd_library_loader**) is a universal loader for the **ddtrace** extension.
+
+Once compiled (PHP >= 7.4), the same shared object can be loaded on any PHP versions from 7.0 to 8.3,
+whatever the PHP flavor (NTS/ZTS, debug or not).
+
+When loaded, it will try to find the right **ddtrace.so** file on the filesystem and load it into PHP.

loader/bin/fake_forwarder.sh

@@ -0,0 +1,5 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+# 1st argument is always "library_entrypoint", the 2nd is the json payload
+echo "${@:2}" >> ${FAKE_FORWARDER_LOG_PATH:="/tmp/fake_forwarder.log"}

loader/bin/test.sh

@@ -0,0 +1,43 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+function _exit {
+    if [[ $? -eq 0 ]]; then
+        printf "\nSUCCESS\n"
+        exit 0
+    else
+        printf "\nFAILURE\n"
+        exit 1
+    fi
+}
+trap _exit EXIT
+
+if [[ ! -f run-tests.php ]]; then
+    phpize
+fi
+
+if [[ -z "${DD_LOADER_PACKAGE_PATH:-}" ]]; then
+    DEFAULT="/home/circleci/app/dd-library-php"
+    echo "Note: env var 'DD_LOADER_PACKAGE_PATH' is required but not defined, using the default value (${DEFAULT})"
+    export DD_LOADER_PACKAGE_PATH="${DEFAULT}"
+fi
+
+printf "PHP version\n\n"
+php -n -v
+
+printf "Test load loader\n\n"
+DD_TRACE_DEBUG=1 php -n -d zend_extension=${PWD}/modules/dd_library_loader.so -v
+
+printf "\nRunning PHPT tests\n"
+php -n run-tests.php -q -p $(which php) -n -d zend_extension=${PWD}/modules/dd_library_loader.so --show-diff
+
+printf "\nRunning functional tests\n\n"
+failure=0
+for file in $(find tests/functional/ -name 'test_*.php'); do
+    echo "${file}"
+    php -n ${file} | sed 's/^/     /' || failure=1
+done
+
+if [[ $failure -eq 1 ]]; then
+    exit 1
+fi