General Features

APEX has many features available during data collection, listed in the apex_exec --apex:help output:

[khuck@gilgamesh apex-tutorial]$ apex_exec --apex:help

Usage:
apex_exec <APEX options> executable <executable options>

where APEX options are zero or more of:
    --apex:help                   show this usage message
    --apex:debug                  run with APEX in debugger
    --apex:verbose                enable verbose list of APEX environment variables
    --apex:screen                 enable screen text output (on by default)
    --apex:quiet                  disable screen text output
    --apex:csv                    enable csv text output
    --apex:tau                    enable tau profile output
    --apex:taskgraph              enable taskgraph output
                                  (graphviz required for post-processing)
    --apex:tasktree               enable tasktree output
                                  (graphviz required for post-processing)
    --apex:throttle               throttle short-lived timers to reduce overhead (default: off)
    --apex:throttle_calls <value> minimum number of calls before throttling (default: 1000)
    --apex:throttle_per <value>   minimum timer duration in microseconds (default: 10)
    --apex:otf2                   enable OTF2 trace output
    --apex:otf2path <value>       specify location of OTF2 archive
                                  (default: ./OTF2_archive)
    --apex:otf2name <value>       specify name of OTF2 file (default: APEX)
    --apex:gtrace                 enable Google Trace Events output (deprecated)
    --apex:pftrace                enable Perfetto Trace output
    --apex:scatter                enable scatterplot output
                                  (python required for post-processing)
    --apex:openacc                enable OpenACC support
    --apex:kokkos                 enable Kokkos support
    --apex:kokkos_tuning          enable Kokkos runtime autotuning support
    --apex:kokkos_fence           enable Kokkos fences for async kernels
    --apex:raja                   enable RAJA support
    --apex:pthread                enable pthread wrapper support
    --apex:gpu_memory             enable CPU memory wrapper support
    --apex:cpu_memory             enable GPU memory wrapper support
    --apex:untied                 enable tasks to migrate cores/OS threads
                                  during execution (not compatible with trace output)
    --apex:cuda                   enable CUDA/CUPTI measurement (default: off)
    --apex:cuda_counters          enable CUDA/CUPTI counter support (default: off)
    --apex:cuda_driver            enable CUDA driver API callbacks (default: off)
    --apex:cuda_details           enable per-kernel statistics where available (default: off)
    --apex:hip                    enable HIP/ROCTracer measurement (default: off)
    --apex:hip_metrics            enable HIP/ROCProfiler metric support (default: off)
    --apex:hip_counters           enable HIP/ROCTracer counter support (default: off)
    --apex:hip_driver             enable HIP/ROCTracer KSA driver API callbacks (default: off)
    --apex:hip_details            enable per-kernel statistics where available (default: off)
    --apex:monitor_gpu            enable GPU monitoring services (CUDA NVML, ROCm SMI)
    --apex:level0                 enable OneAPI Level0 measurement (default: off)
    --apex:cpuinfo                enable sampling of /proc/cpuinfo (Linux only)
    --apex:meminfo                enable sampling of /proc/meminfo (Linux only)
    --apex:net                    enable sampling of /proc/net/dev (Linux only)
    --apex:status                 enable sampling of /proc/self/status (Linux only)
    --apex:io                     enable sampling of /proc/self/io (Linux only)
    --apex:period <value>         specify frequency of OS/HW sampling
    --apex:ompt                   enable OpenMP profiling (requires runtime support)
    --apex:ompt_simple            only enable OpenMP Tools required events
    --apex:ompt_details           enable all OpenMP Tools events
    --apex:source                 resolve function, file and line info for address lookups with binutils
                                  (default: function only)
    --apex:preload <lib>          extra libraries to load with LD_PRELOAD _before_ APEX libraries
                                  (LD_PRELOAD value is added _after_ APEX libraries)
    --apex:postprocess            run post-process scripts (graphviz, python) on output data after exit

Common Data Collection Features

    --apex:verbose                enable verbose list of APEX environment variables

This option will output all of the APEX environment variables read in at the beginning of execution. For a list of all available variables, run the apex_environment_help program.

    --apex:screen                 enable screen text output (on by default)
    --apex:quiet                  disable screen text output
    --apex:screen_details         enable verbose screen text output

These options will enable/disable the screen output of APEX data, and how much.

    --apex:csv                    enable csv text output
    --apex:tau                    enable tau profile output
    --apex:taskgraph              enable taskgraph output
                                  (graphviz required for post-processing) - deprecated
    --apex:tasktree               enable tasktree output
                                  (python required for post-processing)
    --apex:otf2                   enable OTF2 trace output
    --apex:otf2path <value>       specify location of OTF2 archive
                                  (default: ./OTF2_archive)
    --apex:otf2name <value>       specify name of OTF2 file (default: APEX)
    --apex:gtrace                 enable Google Trace Events output (deprecated)
    --apex:pftrace                enable Perfetto Trace output
    --apex:scatter                enable scatterplot output
                                  (python required for post-processing)

These options control the different data output methods from APEX.

    --apex:throttle               throttle short-lived timers to reduce overhead (default: off)
    --apex:throttle_calls <value> minimum number of calls before throttling (default: 1000)
    --apex:throttle_per <value>   minimum timer duration in microseconds (default: 10)

These options control "throttling" of high-frequency, low-duration events.

    --apex:cpuinfo                enable sampling of /proc/cpuinfo (Linux only)
    --apex:meminfo                enable sampling of /proc/meminfo (Linux only)
    --apex:net                    enable sampling of /proc/net/dev (Linux only)
    --apex:status                 enable sampling of /proc/self/status (Linux only)
    --apex:io                     enable sampling of /proc/self/io (Linux only)
    --apex:period <value>         specify frequency of OS/HW sampling

These options control hardware/OS monitoring support.

    --apex:ompt                   enable OpenMP profiling (requires runtime support)
    --apex:ompt_simple            only enable OpenMP Tools required events
    --apex:ompt_details           enable all OpenMP Tools events

These options control OpenMP support.

    --apex:source                 resolve function, file and line info for address lookups with binutils
                                  (default: function only)
    --apex:preload <lib>          extra libraries to load with LD_PRELOAD _before_ APEX libraries
                                  (LD_PRELOAD value is added _after_ APEX libraries)
    --apex:postprocess            run post-process scripts (graphviz, python) on output data after exit

These are all additional flags for controlling APEX behavior.

Device Features

    --apex:openacc                enable OpenACC support
    --apex:kokkos                 enable Kokkos support
    --apex:kokkos_tuning          enable Kokkos runtime autotuning support
    --apex:kokkos_fence           enable Kokkos fences for async kernels
    --apex:raja                   enable RAJA support
    --apex:pthread                enable pthread wrapper support
    --apex:gpu_memory             enable CPU memory wrapper support
    --apex:cpu_memory             enable GPU memory wrapper support
    --apex:untied                 enable tasks to migrate cores/OS threads
                                  during execution (not compatible with trace output)
    --apex:cuda                   enable CUDA/CUPTI measurement (default: off)
    --apex:cuda_counters          enable CUDA/CUPTI counter support (default: off)
    --apex:cuda_driver            enable CUDA driver API callbacks (default: off)
    --apex:cuda_details           enable per-kernel statistics where available (default: off)
    --apex:hip                    enable HIP/ROCTracer measurement (default: off)
    --apex:hip_metrics            enable HIP/ROCProfiler metric support (default: off)
    --apex:hip_counters           enable HIP/ROCTracer counter support (default: off)
    --apex:hip_driver             enable HIP/ROCTracer KSA driver API callbacks (default: off)
    --apex:hip_details            enable per-kernel statistics where available (default: off)
    --apex:monitor_gpu            enable GPU monitoring services (CUDA NVML, ROCm SMI)
    --apex:level0                 enable OneAPI Level0 measurement (default: off)

These options all control device offload support.

Utilities

• apex_environment_help - utility to show all available APEX environment variable names, types, descriptions and default values.
• apex_exec - utility to preload the APEX support library and set APEX environment variables.

Post-Processing Features

• apex-summary.py - utility to parse APEX CSV files and perform data analysis from multiple ranks.
• apex-treesummary.py - utility to parse APEX task tree CSV files and perform data analysis, data reduction, and generate tree representation of the APEX data from multiple ranks.
• consolidate.py - utility for consolidating concurrency data from multiple ranks.
• counter_scatterplot.py - utility for generating counter scatterplots from APEX scatterplot data.
• gtrace_merger.py - utility for merging Google Trace Events traces from multiple ranks into one file.
• task_scatterplot.py - utility for generating task scatterplots for the top X events in the scatterplot data.
• roofline_stats.py - utility for generating roofline statistics for HIP metric data.