Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug]: Laravel's $dontReport and $internalDontReport are ignored #3047

Open
nikalemdzievski opened this issue Jan 20, 2025 · 11 comments · May be fixed by #3074
Open

[Bug]: Laravel's $dontReport and $internalDontReport are ignored #3047

nikalemdzievski opened this issue Jan 20, 2025 · 11 comments · May be fixed by #3074
Assignees
Labels
🐛 bug Something isn't working

Comments

@nikalemdzievski
Copy link

nikalemdzievski commented Jan 20, 2025

Bug report

Laravel's $dontReport and $internalDontReport are ignored.

All of the exceptions listed here are reported on Datadog. Exceptions that are being handled in a specific way (for example return a custom 422 response) are reported.

Weirdly this was not the case a few weeks ago. 3 significant changes that we made were:

  • Upgraded from PHP 8.2 to PHP 8.3
  • Enabled source code integration
  • Set DD_EXCEPTION_REPLAY_ENABLED to true

We checked if disabling DD_EXCEPTION_REPLAY_ENABLED would make any difference but no luck.

Datadog agent version: 7.60.1
DDTrace version 1.5.1

  • Update: we also upgraded to 1.6.3 - Same issue

PHP version

8.3.14

Tracer or profiler version

1.5.1 and 1.6.3

Installed extensions

bcmath
bz2
calendar
Core
ctype
curl
datadog-profiling
date
ddappsec
ddtrace
dom
exif
fileinfo
filter
gd
gettext
hash
iconv
igbinary
intl
json
libxml
mbstring
memcached
msgpack
mysqlnd
openssl
pcntl
pcre
PDO
pdo_pgsql
pdo_sqlite
pgsql
Phar
posix
random
readline
redis
Reflection
session
shmop
SimpleXML
sockets
sodium
SPL
sqlite3
standard
sysvmsg
sysvsem
sysvshm
tokenizer
xml
xmlreader
xmlwriter
xsl
Zend OPcache
zip
zlib

[Zend Modules]
Zend OPcache
datadog-profiling
ddappsec
ddtrace

Output of phpinfo()

dtrace

Datadog PHP tracer extension
For help, check out the documentation at https://docs.datadoghq.com/tracing/languages/php/
(c) Datadog 2020

Datadog tracing support => enabled
Version => 1.5.1
DATADOG TRACER CONFIGURATION => {
"date": "2025-01-20T12:13:59Z",
"os_name": "Linux worker-listener 6.1.119-129.201.amzn2023.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Dec 3 21:07:35 UTC 2024 x86_64",
"os_version": "6.1.119-129.201.amzn2023.x86_64",
"version": "1.5.1",
"lang": "php",
"lang_version": "8.3.13",
"env": "int",
"enabled": true,
"service": "api-v2-worker-listener",
"enabled_cli": true,
"agent_url": "http://localhost:8126",
"debug": false,
"analytics_enabled": false,
"sample_rate": -1,
"sampling_rules": [
{
"service": "api-v2-worker-listener",
"sample_rate": 1
}
],
"tags": [],
"service_mapping": [],
"distributed_tracing_enabled": true,
"dd_version": "2.127.4.76e656b",
"architecture": "x86_64",
"instrumentation_telemetry_enabled": true,
"sapi": "cli",
"datadog.trace.sources_path": "/opt/datadog/dd-library/1.5.1/dd-trace-sources/src",
"open_basedir_configured": false,
"uri_fragment_regex": null,
"uri_mapping_incoming": null,
"uri_mapping_outgoing": null,
"auto_flush_enabled": true,
"generate_root_span": true,
"http_client_split_by_domain": false,
"measure_compile_time": true,
"report_hostname_on_root_span": false,
"traced_internal_functions": null,
"enabled_from_env": true,
"opcache.file_cache": null,
"sidecar_trace_sender": true,
"agent_error": "Failed to connect to localhost port 8126 after 0 ms: Couldn't connect to server"
}

Diagnostics
agent_error => Failed to connect to localhost port 8126 after 0 ms: Couldn't connect to server
Diagnostic checks => failed

Directive => Local Value => Master Value
datadog.agent_host => no value => no value
datadog.amqp_analytics_enabled => Off => Off
datadog.amqp_analytics_sample_rate => 1 => 1
datadog.api_key => no value => no value
datadog.appsec.sca_enabled => Off => Off
datadog.autofinish_spans => Off => Off
datadog.autoload_no_compile => Off => Off
datadog.cakephp_analytics_enabled => Off => Off
datadog.cakephp_analytics_sample_rate => 1 => 1
datadog.codeigniter_analytics_enabled => Off => Off
datadog.codeigniter_analytics_sample_rate => 1 => 1
datadog.crashtracking_enabled => On => On
datadog.curl_analytics_enabled => Off => Off
datadog.curl_analytics_sample_rate => 1 => 1
datadog.dbm_propagation_mode => full => full
datadog.distributed_tracing => On => On
datadog.dogstatsd_port => 8125 => 8125
datadog.dogstatsd_url => no value => no value
datadog.drupal_analytics_enabled => Off => Off
datadog.drupal_analytics_sample_rate => 1 => 1
datadog.dynamic_instrumentation.enabled => Off => Off
datadog.dynamic_instrumentation.redacted_identifiers => no value => no value
datadog.dynamic_instrumentation.redacted_types => no value => no value
datadog.elasticsearch_analytics_enabled => Off => Off
datadog.elasticsearch_analytics_sample_rate => 1 => 1
datadog.eloquent_analytics_enabled => Off => Off
datadog.eloquent_analytics_sample_rate => 1 => 1
datadog.env => int => int
datadog.exception_replay_capture_interval_seconds => 3600 => 3600
datadog.exception_replay_capture_max_frames => -1 => -1
datadog.exception_replay_enabled => Off => Off
datadog.exec_analytics_enabled => Off => Off
datadog.exec_analytics_sample_rate => 1 => 1
datadog.frankenphp_analytics_enabled => Off => Off
datadog.frankenphp_analytics_sample_rate => 1 => 1
datadog.git_commit_sha => [I REMOVED THIS FOR THE PURPOSE OF THIS ISSUE]
datadog.git_repository_url => [I REMOVED THIS FOR THE PURPOSE OF THIS ISSUE]
datadog.googlespanner_analytics_enabled => Off => Off
datadog.googlespanner_analytics_sample_rate => 1 => 1
datadog.guzzle_analytics_enabled => Off => Off
datadog.guzzle_analytics_sample_rate => 1 => 1
datadog.http_server_route_based_naming => On => On
datadog.inject_force => Off => Off
datadog.instrumentation_telemetry_enabled => On => On
datadog.integration_metrics_enabled => On => On
datadog.laminas_analytics_enabled => Off => Off
datadog.laminas_analytics_sample_rate => 1 => 1
datadog.laravel_analytics_enabled => Off => Off
datadog.laravel_analytics_sample_rate => 1 => 1
datadog.laravelqueue_analytics_enabled => Off => Off
datadog.laravelqueue_analytics_sample_rate => 1 => 1
datadog.log_backtrace => Off => Off
datadog.logs_analytics_enabled => Off => Off
datadog.logs_analytics_sample_rate => 1 => 1
datadog.logs_injection => On => On
datadog.lumen_analytics_enabled => Off => Off
datadog.lumen_analytics_sample_rate => 1 => 1
datadog.magento_analytics_enabled => Off => Off
datadog.magento_analytics_sample_rate => 1 => 1
datadog.memcache_analytics_enabled => Off => Off
datadog.memcache_analytics_sample_rate => 1 => 1
datadog.memcached_analytics_enabled => Off => Off
datadog.memcached_analytics_sample_rate => 1 => 1
datadog.mongo_analytics_enabled => Off => Off
datadog.mongo_analytics_sample_rate => 1 => 1
datadog.mongodb_analytics_enabled => Off => Off
datadog.mongodb_analytics_sample_rate => 1 => 1
datadog.mysqli_analytics_enabled => Off => Off
datadog.mysqli_analytics_sample_rate => 1 => 1
datadog.nette_analytics_enabled => Off => Off
datadog.nette_analytics_sample_rate => 1 => 1
datadog.openai_analytics_enabled => Off => Off
datadog.openai_analytics_sample_rate => 1 => 1
datadog.openai_log_prompt_completion_sample_rate => 0.1 => 0.1
datadog.openai_logs_enabled => Off => Off
datadog.openai_metrics_enabled => On => On
datadog.openai_service => no value => no value
datadog.openai_span_char_limit => 128 => 128
datadog.openai_span_prompt_completion_sample_rate => 1.0 => 1.0
datadog.pcntl_analytics_enabled => Off => Off
datadog.pcntl_analytics_sample_rate => 1 => 1
datadog.pdo_analytics_enabled => Off => Off
datadog.pdo_analytics_sample_rate => 1 => 1
datadog.phpredis_analytics_enabled => Off => Off
datadog.phpredis_analytics_sample_rate => 1 => 1
datadog.predis_analytics_enabled => Off => Off
datadog.predis_analytics_sample_rate => 1 => 1
datadog.psr18_analytics_enabled => Off => Off
datadog.psr18_analytics_sample_rate => 1 => 1
datadog.remote_config_enabled => On => On
datadog.remote_config_poll_interval_seconds => 5 => 5
datadog.roadrunner_analytics_enabled => Off => Off
datadog.roadrunner_analytics_sample_rate => 1 => 1
datadog.service => api-v2-worker-listener => api-v2-worker-listener
datadog.service_mapping => no value => no value
datadog.slim_analytics_enabled => Off => Off
datadog.slim_analytics_sample_rate => 1 => 1
datadog.span_sampling_rules => [] => []
datadog.span_sampling_rules_file => no value => no value
datadog.sqlsrv_analytics_enabled => Off => Off
datadog.sqlsrv_analytics_sample_rate => 1 => 1
datadog.swoole_analytics_enabled => Off => Off
datadog.swoole_analytics_sample_rate => 1 => 1
datadog.symfony_analytics_enabled => Off => Off
datadog.symfony_analytics_sample_rate => 1 => 1
datadog.symfonymessenger_analytics_enabled => Off => Off
datadog.symfonymessenger_analytics_sample_rate => 1 => 1
datadog.tags => no value => no value
datadog.telemetry_heartbeat_interval => 60 => 60
datadog.trace.128_bit_traceid_generation_enabled => On => On
datadog.trace.128_bit_traceid_logging_enabled => Off => Off
datadog.trace.agent_connect_timeout => 100 => 100
datadog.trace.agent_debug_verbose_curl => Off => Off
datadog.trace.agent_flush_after_n_requests => 0 => 0
datadog.trace.agent_flush_interval => 1001 => 1001
datadog.trace.agent_max_payload_size => 52428800 => 52428800
datadog.trace.agent_port => 0 => 0
datadog.trace.agent_retries => 0 => 0
datadog.trace.agent_stack_backlog => 12 => 12
datadog.trace.agent_stack_initial_size => 131072 => 131072
datadog.trace.agent_test_session_token => no value => no value
datadog.trace.agent_timeout => 500 => 500
datadog.trace.agent_url => no value => no value
datadog.trace.agentless => Off => Off
datadog.trace.amqp_analytics_enabled => Off => Off
datadog.trace.amqp_analytics_sample_rate => 1 => 1
datadog.trace.amqp_enabled => On => On
datadog.trace.analytics_enabled => Off => Off
datadog.trace.append_trace_ids_to_logs => Off => Off
datadog.trace.auto_flush_enabled => On => On
datadog.trace.beta_high_memory_pressure_percent => 80 => 80
datadog.trace.bgs_connect_timeout => 2000 => 2000
datadog.trace.bgs_timeout => 5000 => 5000
datadog.trace.buffer_size => 2097152 => 2097152
datadog.trace.cakephp_analytics_enabled => Off => Off
datadog.trace.cakephp_analytics_sample_rate => 1 => 1
datadog.trace.cakephp_enabled => On => On
datadog.trace.cli_enabled => On => On
datadog.trace.client_ip_enabled => Off => Off
datadog.trace.client_ip_header => no value => no value
datadog.trace.codeigniter_analytics_enabled => Off => Off
datadog.trace.codeigniter_analytics_sample_rate => 1 => 1
datadog.trace.codeigniter_enabled => On => On
datadog.trace.curl_analytics_enabled => Off => Off
datadog.trace.curl_analytics_sample_rate => 1 => 1
datadog.trace.curl_enabled => On => On
datadog.trace.db_client_split_by_instance => Off => Off
datadog.trace.debug => Off => Off
datadog.trace.debug_curl_output => Off => Off
datadog.trace.debug_prng_seed => -1 => -1
datadog.trace.drupal_analytics_enabled => Off => Off
datadog.trace.drupal_analytics_sample_rate => 1 => 1
datadog.trace.drupal_enabled => On => On
datadog.trace.elasticsearch_analytics_enabled => Off => Off
datadog.trace.elasticsearch_analytics_sample_rate => 1 => 1
datadog.trace.elasticsearch_enabled => On => On
datadog.trace.eloquent_analytics_enabled => Off => Off
datadog.trace.eloquent_analytics_sample_rate => 1 => 1
datadog.trace.eloquent_enabled => On => On
datadog.trace.enabled => On => On
datadog.trace.exec_analytics_enabled => Off => Off
datadog.trace.exec_analytics_sample_rate => 1 => 1
datadog.trace.exec_enabled => On => On
datadog.trace.flush_collect_cycles => Off => Off
datadog.trace.forked_process => On => On
datadog.trace.frankenphp_analytics_enabled => Off => Off
datadog.trace.frankenphp_analytics_sample_rate => 1 => 1
datadog.trace.frankenphp_enabled => On => On
datadog.trace.generate_root_span => On => On
datadog.trace.git_metadata_enabled => On => On
datadog.trace.googlespanner_analytics_enabled => Off => Off
datadog.trace.googlespanner_analytics_sample_rate => 1 => 1
datadog.trace.googlespanner_enabled => On => On
datadog.trace.guzzle_analytics_enabled => Off => Off
datadog.trace.guzzle_analytics_sample_rate => 1 => 1
datadog.trace.guzzle_enabled => On => On
datadog.trace.header_tags => no value => no value
datadog.trace.health_metrics_enabled => Off => Off
datadog.trace.health_metrics_heartbeat_sample_rate => 0.001 => 0.001
datadog.trace.hook_limit => 100 => 100
datadog.trace.http_client_split_by_domain => Off => Off
datadog.trace.http_post_data_param_allowed => no value => no value
datadog.trace.http_url_query_param_allowed => * => *
datadog.trace.laminas_analytics_enabled => Off => Off
datadog.trace.laminas_analytics_sample_rate => 1 => 1
datadog.trace.laminas_enabled => On => On
datadog.trace.laravel_analytics_enabled => Off => Off
datadog.trace.laravel_analytics_sample_rate => 1 => 1
datadog.trace.laravel_enabled => On => On
datadog.trace.laravel_queue_distributed_tracing => On => On
datadog.trace.laravelqueue_analytics_enabled => Off => Off
datadog.trace.laravelqueue_analytics_sample_rate => 1 => 1
datadog.trace.laravelqueue_enabled => On => On
datadog.trace.log_file => no value => no value
datadog.trace.log_level => error => error
datadog.trace.logs_analytics_enabled => Off => Off
datadog.trace.logs_analytics_sample_rate => 1 => 1
datadog.trace.logs_enabled => On => On
datadog.trace.lumen_analytics_enabled => Off => Off
datadog.trace.lumen_analytics_sample_rate => 1 => 1
datadog.trace.lumen_enabled => On => On
datadog.trace.magento_analytics_enabled => Off => Off
datadog.trace.magento_analytics_sample_rate => 1 => 1
datadog.trace.magento_enabled => On => On
datadog.trace.measure_compile_time => On => On
datadog.trace.measure_peak_memory_usage => On => On
datadog.trace.memcache_analytics_enabled => Off => Off
datadog.trace.memcache_analytics_sample_rate => 1 => 1
datadog.trace.memcache_enabled => On => On
datadog.trace.memcached_analytics_enabled => Off => Off
datadog.trace.memcached_analytics_sample_rate => 1 => 1
datadog.trace.memcached_enabled => On => On
datadog.trace.memcached_obfuscation => On => On
datadog.trace.memory_limit => no value => no value
datadog.trace.mongo_analytics_enabled => Off => Off
datadog.trace.mongo_analytics_sample_rate => 1 => 1
datadog.trace.mongo_enabled => On => On
datadog.trace.mongodb_analytics_enabled => Off => Off
datadog.trace.mongodb_analytics_sample_rate => 1 => 1
datadog.trace.mongodb_enabled => On => On
datadog.trace.mysqli_analytics_enabled => Off => Off
datadog.trace.mysqli_analytics_sample_rate => 1 => 1
datadog.trace.mysqli_enabled => On => On
datadog.trace.nette_analytics_enabled => Off => Off
datadog.trace.nette_analytics_sample_rate => 1 => 1
datadog.trace.nette_enabled => On => On
datadog.trace.obfuscation_query_string_regexp => (?i)(?:(?:"|%22)?)(?:(?:old[-]?|new[-]?)?p(?:ass)?w(?:or)?d(?:1|2)?|pass(?:[-]?phrase)?|secret|(?:api[-]?|private[-]?|public[-]?|access[-]?|secret[-]?|app(?:lication)?[-]?)key(?:[-]?id)?|token|consumer[-]?(?:id|key|secret)|sign(?:ed|ature)?|auth(?:entication|orization)?)(?:(?:\s|%20)(?:=|%3D)[^&]+|(?:"|%22)(?:\s|%20)(?::|%3A)(?:\s|%20)*(?:"|%22)(?:%2[^2]|%[^2]|[^"%])+(?:"|%22))|(?:bearer(?:\s|%20)+[a-z0-9.-]+|token(?::|%3A)[a-z0-9]{13}|gh[opsu][0-9a-zA-Z]{36}|eyI-L+.eyI-L+(?:.(?:[\w.+/=-]|%3D|%2F|%2B)+)?|-{5}BEGIN(?:[a-z\s]|%20)+PRIVATE(?:\s|%20)KEY-{5}[^\-]+-{5}END(?:[a-z\s]|%20)+PRIVATE(?:\s|%20)KEY(?:-{5})?(?:\n|%0A)?|(?:ssh-(?:rsa|dss)|ecdsa-[a-z0-9]+-[a-z0-9]+)(?:\s|%20|%09)+(?:[a-z0-9/.+]|%2F|%5C|%2B){100,}(?:=|%3D)*(?:(?:\s|%20|%09)+[a-z0-9.-]+)?) => (?i)(?:(?:"|%22)?)(?:(?:old[-]?|new[-]?)?p(?:ass)?w(?:or)?d(?:1|2)?|pass(?:[-]?phrase)?|secret|(?:api[-]?|private[-]?|public[-]?|access[-]?|secret[-]?|app(?:lication)?[-]?)key(?:[-]?id)?|token|consumer[-]?(?:id|key|secret)|sign(?:ed|ature)?|auth(?:entication|orization)?)(?:(?:\s|%20)(?:=|%3D)[^&]+|(?:"|%22)(?:\s|%20)(?::|%3A)(?:\s|%20)*(?:"|%22)(?:%2[^2]|%[^2]|[^"%])+(?:"|%22))|(?:bearer(?:\s|%20)+[a-z0-9.-]+|token(?::|%3A)[a-z0-9]{13}|gh[opsu][0-9a-zA-Z]{36}|eyI-L+.eyI-L+(?:.(?:[\w.+/=-]|%3D|%2F|%2B)+)?|-{5}BEGIN(?:[a-z\s]|%20)+PRIVATE(?:\s|%20)KEY-{5}[^\-]+-{5}END(?:[a-z\s]|%20)+PRIVATE(?:\s|%20)KEY(?:-{5})?(?:\n|%0A)?|(?:ssh-(?:rsa|dss)|ecdsa-[a-z0-9]+-[a-z0-9]+)(?:\s|%20|%09)+(?:[a-z0-9/.+]|%2F|%5C|%2B){100,}(?:=|%3D)*(?:(?:\s|%20|%09)+[a-z0-9.-]+)?)
datadog.trace.once_logs => On => On
datadog.trace.openai_analytics_enabled => Off => Off
datadog.trace.openai_analytics_sample_rate => 1 => 1
datadog.trace.openai_enabled => On => On
datadog.trace.otel_enabled => Off => Off
datadog.trace.pcntl_analytics_enabled => Off => Off
datadog.trace.pcntl_analytics_sample_rate => 1 => 1
datadog.trace.pcntl_enabled => On => On
datadog.trace.pdo_analytics_enabled => Off => Off
datadog.trace.pdo_analytics_sample_rate => 1 => 1
datadog.trace.pdo_enabled => On => On
datadog.trace.peer_service_defaults_enabled => Off => Off
datadog.trace.peer_service_mapping => no value => no value
datadog.trace.phpredis_analytics_enabled => Off => Off
datadog.trace.phpredis_analytics_sample_rate => 1 => 1
datadog.trace.phpredis_enabled => On => On
datadog.trace.predis_analytics_enabled => Off => Off
datadog.trace.predis_analytics_sample_rate => 1 => 1
datadog.trace.predis_enabled => On => On
datadog.trace.propagate_service => Off => Off
datadog.trace.propagate_user_id_default => Off => Off
datadog.trace.propagation_style => datadog,tracecontext => datadog,tracecontext
datadog.trace.propagation_style_extract => datadog,tracecontext,B3,B3 single header => datadog,tracecontext,B3,B3 single header
datadog.trace.propagation_style_inject => datadog,tracecontext => datadog,tracecontext
datadog.trace.psr18_analytics_enabled => Off => Off
datadog.trace.psr18_analytics_sample_rate => 1 => 1
datadog.trace.psr18_enabled => On => On
datadog.trace.rate_limit => 100 => 100
datadog.trace.redis_client_split_by_host => Off => Off
datadog.trace.remove_autoinstrumentation_orphans => On => On
datadog.trace.remove_integration_service_names_enabled => Off => Off
datadog.trace.remove_root_span_laravel_queue => On => On
datadog.trace.remove_root_span_symfony_messenger => On => On
datadog.trace.report_hostname => Off => Off
datadog.trace.resource_uri_fragment_regex => no value => no value
datadog.trace.resource_uri_mapping_incoming => no value => no value
datadog.trace.resource_uri_mapping_outgoing => no value => no value
datadog.trace.resource_uri_query_param_allowed => no value => no value
datadog.trace.retain_thread_capabilities => Off => Off
datadog.trace.roadrunner_analytics_enabled => Off => Off
datadog.trace.roadrunner_analytics_sample_rate => 1 => 1
datadog.trace.roadrunner_enabled => On => On
datadog.trace.sample_rate => -1 => -1
datadog.trace.sampling_rules => [{"service": "api-v2-worker-listener", "sample_rate": 1.0000}] => [{"service": "api-v2-worker-listener", "sample_rate": 1.0000}]
datadog.trace.sampling_rules_format => glob => glob
datadog.trace.shutdown_timeout => 5000 => 5000
datadog.trace.sidecar_trace_sender => On => On
datadog.trace.slim_analytics_enabled => Off => Off
datadog.trace.slim_analytics_sample_rate => 1 => 1
datadog.trace.slim_enabled => On => On
datadog.trace.sources_path => /opt/datadog/dd-library/1.5.1/dd-trace-sources/src => /opt/datadog/dd-library/1.5.1/dd-trace-sources/src
datadog.trace.spans_limit => 1000 => 1000
datadog.trace.sqlsrv_analytics_enabled => Off => Off
datadog.trace.sqlsrv_analytics_sample_rate => 1 => 1
datadog.trace.sqlsrv_enabled => On => On
datadog.trace.startup_logs => On => On
datadog.trace.swoole_analytics_enabled => Off => Off
datadog.trace.swoole_analytics_sample_rate => 1 => 1
datadog.trace.swoole_enabled => On => On
datadog.trace.symfony_analytics_enabled => Off => Off
datadog.trace.symfony_analytics_sample_rate => 1 => 1
datadog.trace.symfony_enabled => On => On
datadog.trace.symfony_messenger_distributed_tracing => On => On
datadog.trace.symfony_messenger_middlewares => Off => Off
datadog.trace.symfonymessenger_analytics_enabled => Off => Off
datadog.trace.symfonymessenger_analytics_sample_rate => 1 => 1
datadog.trace.symfonymessenger_enabled => On => On
datadog.trace.traced_internal_functions => no value => no value
datadog.trace.url_as_resource_names_enabled => On => On
datadog.trace.warn_legacy_dd_trace => On => On
datadog.trace.web_analytics_enabled => Off => Off
datadog.trace.web_analytics_sample_rate => 1 => 1
datadog.trace.web_enabled => On => On
datadog.trace.wordpress_additional_actions => no value => no value
datadog.trace.wordpress_analytics_enabled => Off => Off
datadog.trace.wordpress_analytics_sample_rate => 1 => 1
datadog.trace.wordpress_callbacks => On => On
datadog.trace.wordpress_enabled => On => On
datadog.trace.x_datadog_tags_max_length => 512 => 512
datadog.trace.yii_analytics_enabled => Off => Off
datadog.trace.yii_analytics_sample_rate => 1 => 1
datadog.trace.yii_enabled => On => On
datadog.trace.zendframework_analytics_enabled => Off => Off
datadog.trace.zendframework_analytics_sample_rate => 1 => 1
datadog.trace.zendframework_enabled => On => On
datadog.version => 2.127.4.76e656b => 2.127.4.76e656b
datadog.web_analytics_enabled => Off => Off
datadog.web_analytics_sample_rate => 1 => 1
datadog.wordpress_analytics_enabled => Off => Off
datadog.wordpress_analytics_sample_rate => 1 => 1
datadog.yii_analytics_enabled => Off => Off
datadog.yii_analytics_sample_rate => 1 => 1
datadog.zendframework_analytics_enabled => Off => Off
datadog.zendframework_analytics_sample_rate => 1 => 1
ddtrace.cgroup_file => /proc/self/cgroup => /proc/self/cgroup
ddtrace.disable => 0 => 0

Upgrading from

Upgraded from PHP 8.2.20 to PHP 8.3.14

@nikalemdzievski nikalemdzievski added the 🐛 bug Something isn't working label Jan 20, 2025
@PROFeNoM
Copy link
Contributor

Hey @nikalemdzievski 👋

Can you please let me know which version of Laravel are you using?

Thanks

@nikalemdzievski
Copy link
Author

@PROFeNoM 10.32. Sorry for missing that!

@PROFeNoM
Copy link
Contributor

Thanks @nikalemdzievski

Additionally, could you please elaborate (perhaps even some screenshots from the Datadog UI) on the erroneous behavior you are experiencing regarding these "reported" errors? In what ways is the behavior that you are experiencing today different from the one you were experiencing three weeks ago?

@nikalemdzievski
Copy link
Author

nikalemdzievski commented Jan 20, 2025

Hey @PROFeNoM ! yes of course.

Here is one example: https://prnt.sc/k5MUi-za_EeM

Here is the code for the exception it self:

<?php

namespace App\Exceptions;

use App\Entities\Gig;
use App\ExternalServices\Segment\Gigs\GigService;
use App\Http\Response\Response;
use App\Traits\Exception\Renderable;
use Auth;
use SegmentAnalytics;
use Symfony\Component\HttpKernel\Exception\HttpException;

class GigIsFullException extends HttpException
{
    use Renderable;

    final const CODE = Response::HTTP_GIG_IS_FULL;
    final const ACTION = Response::HTTP_GIG_IS_FULL_ACTION;
    final const MESSAGE = "You are unable to view this shift because other Upshifters have already filled it. Try again next time!";

    public function __construct(Gig $gig, array $data)
    {
        parent::__construct(self::CODE, self::MESSAGE);

        if (!Auth::check()) {
            return;
        }

        SegmentAnalytics::identifyOrTrack($gig, GigService::GIG_FULL_EXCEPTION, [
            'user'       => Auth::user(),
            'properties' => $data,
        ]);
    }
}

The HttpException class is ignored both in $dontReport and $internalDontReport. On top of that the the exception is caught by the render method in Laravel's Handler class and we return a 422 response. In the end, it looks like a validation popup message to the user.

This code (exception) specifically has been in the repo for a long time and has not been updated. As you can see from the screenshot it was first reported today. There are many other examples like this.

Please let me know if you need more details.

@PROFeNoM
Copy link
Contributor

Thanks for providing additional information, @nikalemdzievski; this is much appreciated!

I believe the behavior you are experiencing comes from changes being rolled out on the Error Tracking product, lifting the limitation of only the service entry span being fingerprinted. Hence, this error on laravel.action now shows up under Error Tracking.

This would explain why this 1. happened all of a sudden and 2. that rollbacks/upgrades of the tracer are ineffective.

If you wish to revert to the previous behavior, you could leverage span kinds to achieve this through query filters or by adding an error tracking data collection rule) to apply it globally.
If they also care about client errors, you can include client spans by using @span.kind:server OR @span.kind:client.

Image

Hope this helps 🙏

@PROFeNoM PROFeNoM self-assigned this Jan 22, 2025
@nikalemdzievski
Copy link
Author

nikalemdzievski commented Jan 22, 2025

@PROFeNoM so far this does the trick. I created a rule where I excluded the operation_name:laravel.action span.
Thank you for that! We had a huge spike of reported errors (which were not a problem) which created friction in our process. This certainly helped.

I’m still uncertain about the functionality of $dontReport and $internalDontReport. Should exceptions listed in these properties be excluded from error tracking? Please let me know if I misunderstand how this works.

@bwoebi
Copy link
Collaborator

bwoebi commented Jan 22, 2025

Not saying yet whether we will change anything while we still look at it, but as an explanation:

Every traced span gets attached exceptions which are in flight when that span terminates (e.g. an exception during laravel.action). This handling is unaware of any Laravel specifics.

Then, specifically, when Laravel catches all exceptions, we look at whether exceptions are to be attached onto the root span or not. At that moment, we check for $dontReport, as we are inside the Laravel exception handler, where we have that information at hand.

I.e. we've been relying on error tracking to not handle anything, but exceptions on root spans as actual reported errors. Now that changed...

@OliviergSingulart
Copy link

Hi there,

I want to throw my 2 cents here that this is not Laravel specific.

We have a Symfony project using datadog for the last few years and starting from when we upgraded the client library, we are seeing pretty much every exception thrown even if they are handled reported in the UI.

This is a big change in functionally and not sure it's one needed, they are not errors if they are handled, only uncaught exceptions should be classified as errors, as it was the case before.

@specialtactics
Copy link

specialtactics commented Jan 24, 2025

Hi

We are also experiencing this issue, can it please be fixed? With sentry for example, we are capturing the exception inside the application's exception handler (Laravel). Whether datadog picks up this by default or not, it needs to be fixed, we shouldn't have to use extra filters everywhere just to not see these exceptions.

Anything in don't report, whether it's by the framework itself, or the application, any exception which is a direct instance of or extends any of those classes, should not appear in datadog. The problem then is that, there could also be multiple exception handlers, so if you try and do this automatically, you need to find these in the application (though in our case it's just the one, but I guess generally I am flagging this as a possible issue).

Just to be clear, as an example, the application's exception handler will have something like;

    protected $dontReport = [
        SomeException::class,
        AnotherException::class,
    ];

And those are in addition to Laravel's protected $internalDontReport in Illuminate\Foundation\Exceptions\Handler

Can we please get some comment from datadog on how to address this, it's very important. Ideally, it would be great if there was a way to specify in our application's exception handler (which already does all this logic) on whether to capture the exception or not. For example;

Image

Until that's possible, or it's automated properly by the agent, this is creating a lot of pollution with exceptions that are logged in datadog, and it's making any feature related to that unreliable due to the massive amounts of noise.


Additionally, to be clear, it's not a matter of Laravel or Symfony, you need to provide a way for the developer to specify to datadog at runtime which exceptions should not be recorded, this is common practise in PHP frameworks and as you can see other bug tracking solutions cater for this accordingly.

@PROFeNoM
Copy link
Contributor

Hi everyone,

Thank you for all the feedback and for sharing your experiences with this issue. Internally, we've discussed this topic alongside the Error Tracking team, particularly after the change from the top-most fingerprinting strategy.

We understand the concerns raised regarding $dontReport and $internalDontReport in Laravel. To address this, we'll work on a fix tracer side to handle this issue comprehensively across Laravel and similar use cases. This is planned for delivery as part of this quarter.

In the meantime, I'd like to point you back to the guidance I shared earlier regarding leveraging query fields and collection rules to filter out unwanted exceptions globally. This should help reduce noise and improve the signal while we work on the fix.

On a related note, @OliviergSingulart, you mentioned encountering a similar issue with a Symfony application. Since, as far as I know, Symfony doesn't have a $dontReport mechanism like Laravel, I'd love to hear more details about your setup and how the exceptions are being handled in your case. This will help us ensure that our fix is robust and works for all frameworks.

Thank you for your patience and for helping us improve Error Tracking! Feel free to reach out with any additional questions, concerns, or use cases.

@OliviergSingulart
Copy link

Hi there,

Indeed Symfony does not have a $dontReport mechanism. My point is that without any change in config on our end, suddenly we got spammed with already handled exceptions being reported as soon as we upgraded the client on the servers.

I don't think this issue is just about handling $dontReport or similar mechanisms, but more about the fact that handled exceptions are reported when they most likely should not (since they are not errors).

I did apply your workaround and it seems to be working, we don't see all those exceptions anymore.

I hope I'm making sense.

Thanks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
🐛 bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants