Profiling with perf (HULKs#1928)

* Write documentation on how to use perf with hotspot on the Nao

* rename hulk binary

* Always use sysroot flag when launching hotspot

Author: oleflb

crates/hulk_nao/Cargo.toml

@@ -5,6 +5,10 @@ edition.workspace = true
 license.workspace = true
 homepage.workspace = true
 
+[[bin]]
+name = "hulk"
+path = "src/main.rs"
+
 [package.metadata.pepsi]
 cross-compile = true
 

crates/repository/src/upload.rs

@@ -46,5 +46,5 @@ pub fn get_hulk_binary(profile: &str) -> String {
         other => other,
     };
 
-    format!("target/x86_64-aldebaran-linux-gnu/{profile_directory}/hulk_nao")
+    format!("target/x86_64-aldebaran-linux-gnu/{profile_directory}/hulk")
 }

docs/tooling/overview.md

@@ -9,5 +9,6 @@ Apart from the NAO code our repository contains several tools to aid in the deve
 - [Recording & Replay](./recording_and_replay.md): Post-mortem analysis of game data
 - [Machine Learning](./machine-learning.md): Our tooling to create datasets and neural networks
 - [Behavior Simulator](./behavior_simulator.md): The simulator and viewer to debug and automatically test behavior
-- [Debugging with GDB/LLDB](./debugging.md): How to use a debugger with our software
 - [Parameter Tester](./parameter_tester.md): A tool to test if the parameters can be correctly parsed
+- [Debugging with GDB/LLDB](./debugging.md): How to use a debugger with our software
+- [Profiling with `perf`](./profiling.md): How to profile our software with `perf`

docs/tooling/profiling.md

@@ -0,0 +1,63 @@
+# Profiling on the Nao Robot
+
+We use the perf command to profile applications running on the Nao robot. It comes preinstalled in our robot image.
+
+!!!warning
+    Important: Ensure that your hulk binary includes debug symbols; without them, the profile will be unusable.
+    Enable debug symbols by adding the --profile with-debug option to either the pepsi upload or pepsi build command. This preserves compiler optimizations while retaining symbol information for profiling.
+
+### Step 1: SSH into the Robot
+
+To profile an application running on the Nao, you must SSH into the robot first:
+
+```bash
+pepsi shell <nao-number>
+```
+
+### Step 2: Record a Profile with perf
+
+To profile a running application (e.g., hulk), use:
+
+```bash
+perf record --call-graph dwarf,8192 --aio -z --sample-cpu --mmap-pages 16M --pid $(pidof hulk) sleep 30
+```
+
+This samples the stack traces of the hulk process for 30 seconds.
+
+!!!info
+    It's generally easiest to run perf as root, as otherwise various permissions and kernel knobs must be adjusted.
+
+This command will generate a `perf.data` file containing the recorded samples.
+
+### Step 3: Analyze the Profile with hotspot
+
+We use Hotspot to inspect the `perf.data` file.
+With the binary and `perf.data` in place, launch Hotspot:
+
+```bash
+hotspot \
+  --sysroot ~/.local/share/hulk/sdk/9.0.2/sysroots/corei7-64-aldebaran-linux/ \
+  --appPath ./target/x86_64-aldebaran-linux-gnu/with-debug/ \
+  --kallsyms ./kallsyms
+```
+
+Setting the `sysroot` is not requires when you profile a binary on your system, for example a behavior test case.
+Use the Flame Graph, Top Down, or Bottom Up views to investigate time spent in functions.
+
+### Enable Debug Symbols in the SDK
+
+By default, debug symbols are not enabled in our SDK.
+If you want to profile library code, consider enabling debug symbols in the Yocto distribution config:
+
+```
+diff --git a/meta-hulks/conf/distro/HULKs-OS.conf b/meta-hulks/conf/distro/HULKs-OS.conf
+index 3e23671..982a187 100644
+--- a/meta-hulks/conf/distro/HULKs-OS.conf
++++ b/meta-hulks/conf/distro/HULKs-OS.conf
+@@ -5,4 +5,3 @@ SUMMARY = "HULKs flavoured Nao"
+ DISTRO = "HULKs-OS"
+ DISTRO_NAME = "HULKs-OS"
+ DISTRO_VERSION = "9.0.1"
+-SDKIMAGE_FEATURES:remove = "dbg-pkgs src-pkgs"
+```
+Rebuild the SDK afterward and use in it to build the code and as the `sysroot` argument when launching Hotspot.

mkdocs.yml

@@ -61,8 +61,9 @@ nav:
       - Recording & Replay: tooling/recording_and_replay.md
       - Machine Learning: tooling/machine-learning.md
       - Behavior Simulator: tooling/behavior_simulator.md
-      - Debugging with GDB/LLDB: tooling/debugging.md
       - Parameter Tester: tooling/parameter_tester.md
+      - Debugging with GDB/LLDB: tooling/debugging.md
+      - Profiling with perf: tooling/profiling.md
   - Operating System:
       - Overview: operating_system/overview.md
       - Partitioning: operating_system/partitioning.md