Hyperlight supports gdb debugging of a guest running inside a Hyperlight sandbox on Linux or Windows.
When Hyperlight is compiled with the gdb feature enabled, a Hyperlight sandbox can be configured
to start listening for a gdb connection.
The Hyperlight gdb feature enables guest debugging to:
- stop at an entry point breakpoint which is automatically set by Hyperlight
- add and remove HW breakpoints (maximum 4 set breakpoints at a time)
- add and remove SW breakpoints
- read and write registers
- read and write addresses
- step/continue
- get code offset from target
- stop when a crash occurs and only allow read access to the guest memory and registers
Below is a list describing some cases of expected behavior from a gdb debug session of a guest binary running inside a Hyperlight sandbox.
- when the
gdbfeature is enabled and a SandboxConfiguration is provided a debug port, the created sandbox will wait for a gdb client to connect on the configured port - when the gdb client attaches, the guest vCPU is expected to be stopped at the entry point
- if a gdb client disconnects unexpectedly, the debug session will be closed and the guest will continue executing disregarding any prior breakpoints
- if multiple sandbox instances are created, each instance will have its own gdb thread listening on the configured port
- if two sandbox instances are created with the same debug port, the second instance logs an error and the gdb thread will not be created, but the sandbox will continue to run without gdb debugging
- when a crash happens, the debugger session remains active, and the guest vCPU is stopped, allowing the gdb client to inspect the state of the guest. The debug target will refuse any resume, step actions and write operations to the guest memory and registers until the gdb client disconnects or the sandbox is stopped.
The guest-debugging example in Hyperlight demonstrates how to configure a Hyperlight
sandbox to listen for a gdb client on a specific port.
One can use a gdb config file to provide the symbols and desired configuration.
The below contents of the .gdbinit file can be used to provide a basic configuration
to gdb startup.
# Path to symbols
file path/to/symbols.elf
# The port on which Hyperlight listens for a connection
target remote :8080
set disassembly-flavor intel
set disassemble-next-line on
enable pretty-printer
layout srcOne can find more information about the .gdbinit file at gdbinit(5).
Using the example mentioned at Sandbox configuration one can run the below commands to debug the guest binary:
# Terminal 1
$ cargo run --example guest-debugging --features gdb# Terminal 2
$ cat .gdbinit
file src/tests/rust_guests/bin/debug/simpleguest
target remote :8080
set disassembly-flavor intel
set disassemble-next-line on
enable pretty-printer
layout src
$ gdbTo replicate the above behavior using VSCode follow the below steps:
- To use gdb:
- install the
gdbpackage on the host machine - install the C/C++ Extension Pack extension in VSCode to add debugging capabilities
- install the
- To use lldb:
- install
lldbon the host machine - install the CodeLLDB extension in VSCode to add debugging capabilities
- install
- create a
.vscode/launch.jsonfile in the project directory with the below content:{ "version": "0.2.0", "configurations": [ { "name": "LLDB", "type": "lldb", "request": "launch", "targetCreateCommands": ["target create ${workspaceFolder}/src/tests/rust_guests/bin/debug/simpleguest"], "processCreateCommands": ["gdb-remote localhost:8080"] }, { "name": "GDB", "type": "cppdbg", "request": "launch", "program": "${workspaceFolder}/src/tests/rust_guests/bin/debug/simpleguest", "args": [], "stopAtEntry": true, "hardwareBreakpoints": {"require": false, "limit": 4}, "cwd": "${workspaceFolder}", "environment": [], "externalConsole": false, "MIMode": "gdb", "miDebuggerPath": "/usr/bin/gdb", "miDebuggerServerAddress": "localhost:8080", "setupCommands": [ { "description": "Enable pretty-printing for gdb", "text": "-enable-pretty-printing", "ignoreFailures": true }, { "description": "Set Disassembly Flavor to Intel", "text": "-gdb-set disassembly-flavor intel", "ignoreFailures": true } ] } ] } - in
Run and Debugtab, select eitherGDBorLLDBconfiguration and click on theRunbutton to start the debugging session. The debugger will connect to the Hyperlight sandbox and the guest vCPU will stop at the entry point.
The gdb feature is designed to work like a Request - Response protocol between a thread that accepts commands from a gdb client and main thread of the sandbox.
All the functionality is implemented on the hypervisor side so it has access to the shared memory and the vCPU.
The gdb thread uses the gdbstub crate to handle the communication with the gdb client.
When the gdb client requests one of the supported features mentioned above, a request
is sent over the communication channel to the main thread for the sandbox
to resolve.
Below is a sequence diagram that shows the interaction between the entities involved in the gdb debugging of a Hyperlight guest running inside a KVM or MSHV sandbox.
┌───────────────────────────────────────────────────────────────────────────────────────────────┐
│ Hyperlight Sandbox │
USER │ │
┌────────────┐ │ ┌──────────────┐ ┌───────────────────────────┐ ┌────────┐ │
│ gdb client │ │ │ gdb thread │ │ main sandbox thread │ │ vCPU │ │
└────────────┘ │ └──────────────┘ └───────────────────────────┘ └────────┘ │
| │ | create_gdb_thread | | │
| │ |◄─────────────────────────────────────────┌─┐ vcpu stopped ┌─┐ │
| attach │ ┌─┐ │ │◄──────────────────────────────┴─┘ │
┌─┐───────────────────────┼────────►│ │ │ │ entrypoint breakpoint | │
│ │ attach response │ │ │ │ │ | │
│ │◄──────────────────────┼─────────│ │ │ │ | │
│ │ │ │ │ │ │ | │
│ │ add_breakpoint │ │ │ │ │ | │
│ │───────────────────────┼────────►│ │ add_breakpoint │ │ | │
│ │ │ │ │────────────────────────────────────────►│ │ add_breakpoint | │
│ │ │ │ │ │ │────┐ | │
│ │ │ │ │ │ │ │ | │
│ │ │ │ │ │ │◄───┘ | │
│ │ │ │ │ add_breakpoint response │ │ | │
│ │ add_breakpoint response │ │◄────────────────────────────────────────│ │ | │
│ │◄──────────────────────┬─────────│ │ │ │ | │
│ │ continue │ │ │ │ │ | │
│ │───────────────────────┼────────►│ │ continue │ │ | │
│ │ │ │ │────────────────────────────────────────►│ │ resume vcpu | │
│ │ │ │ │ │ │──────────────────────────────►┌─┐ │
│ │ │ │ │ │ │ │ │ │
│ │ │ │ │ │ │ │ │ │
│ │ │ │ │ │ │ │ │ │
│ │ │ │ │ │ │ │ │ │
│ │ │ │ │ │ │ vcpu stopped │ │ │
│ │ │ │ │ notify vcpu stop reason │ │◄──────────────────────────────┴─┘ │
│ │ notify vcpu stop reason │ │◄────────────────────────────────────────│ │ | │
│ │◄──────────────────────┬─────────│ │ │ │ | │
│ │ continue until end │ │ │ │ │ | │
│ │───────────────────────┼────────►│ │ continue │ │ resume vcpu | │
│ │ │ │ │────────────────────────────────────────►│ │──────────────────────────────►┌─┐ │
│ │ │ │ │ │ │ │ │ │
│ │ │ │ │ comm channel disconnected │ │ vcpu halted │ │ │
│ │ target finished exec│ │ │◄────────────────────────────────────────┤ │◄──────────────────────────────┴─┘ │
│ │◄──────────────────────┼─────────┴─┘ target finished exec └─┘ | │
│ │ │ | | | │
└─┘ │ | | | │
| └───────────────────────────────────────────────────────────────────────────────────────────────┘
When a guest crashes because of an unknown VmExit or unhandled exception, the vCPU state can be optionally dumped to an ELF core dump file.
This can be used to inspect the state of the guest at the time of the crash.
To make Hyperlight dump the state of the vCPU (general purpose registers, registers) to an ELF core dump file, enable the crashdump feature and run.
The feature enables the creation of core dump files for both debug and release builds of Hyperlight hosts.
By default, Hyperlight places the core dumps in the temporary directory (platform specific).
To change this, use the HYPERLIGHT_CORE_DUMP_DIR environment variable to specify a directory.
The name and location of the dump file will be printed to the console and logged as an error message.
NOTE: If the directory provided by HYPERLIGHT_CORE_DUMP_DIR does not exist, Hyperlight places the file in the temporary directory.
NOTE: By enabling the crashdump feature, you instruct Hyperlight to create core dump files for all sandboxes when an unhandled crash occurs.
To selectively disable this feature for a specific sandbox, you can set the guest_core_dump field to false in the SandboxConfiguration.
let mut cfg = SandboxConfiguration::default();
cfg.set_guest_core_dump(false); // Disable core dump for this sandboxYou can also create a core dump of the current state of the guest on demand by calling the generate_crashdump method on the InitializedMultiUseSandbox instance. This can be useful for debugging issues in the guest that do not cause crashes (e.g., a guest function that does not return).
This is only available when the crashdump feature is enabled and then only if the sandbox
is also configured to allow core dumps (which is the default behavior).
Attach to your running process with gdb and call this function:
sudo gdb -p <pid_of_your_process>
(gdb) info threads
# find the thread that is running the guest function you want to debug
(gdb) thread <thread_number>
# switch to the frame where you have access to your MultiUseSandbox instance
(gdb) backtrace
(gdb) frame <frame_number>
# get the pointer to your MultiUseSandbox instance
# Get the sandbox pointer
(gdb) print sandbox
# Call the crashdump function with the pointer
# Call the crashdump function
call sandbox.generate_crashdump()The crashdump should be available /tmp or in the crash dump directory (see HYPERLIGHT_CORE_DUMP_DIR env var). To make this process easier, you can also create a gdb script that automates these steps. You can find an example script here. This script will try and generate a crashdump for every active thread except thread 1 , it assumes that the variable sandbox exists in frame 15 on every thread. You can edit it to fit your needs. Then use it like this:
(gdb) source scripts/dump_all_sandboxes.gdb
(gdb) dump_all_sandboxesAfter the core dump has been created, to inspect the state of the guest, load the core dump file using gdb or lldb.
NOTE: This feature has been tested with version 15.0 of gdb and version 17 of lldb, earlier versions may not work, it is recommended to use these versions or later.
Load the core dump alongside the guest binary that was running when the crash occurred:
gdb <guest-binary> -c <core-dump-file>For example:
gdb src/tests/rust_guests/bin/debug/simpleguest -c /tmp/hl_dumps/hl_core_20260225_T165358.517.elfCommon commands for inspecting the dump:
# View all general-purpose registers (rip, rsp, rflags, etc.)
(gdb) info registers
# Disassemble around the crash site
(gdb) x/10i $rip
# View the stack
(gdb) x/16xg $rsp
# Backtrace (requires debug info in guest binary)
(gdb) bt
# List all memory regions in the dump (snapshot, scratch, mapped regions)
(gdb) info files
# Read memory at a specific address
(gdb) x/s <address> # null-terminated string
(gdb) x/32xb <address> # 32 bytes in hexSee the crashdump example (cargo run --example crashdump --features crashdump)
for a runnable demonstration of both automatic and on-demand crash dumps.
To do this in vscode, the following configuration can be used to add debug configurations:
{
"version": "0.2.0",
"inputs": [
{
"id": "core_dump",
"type": "promptString",
"description": "Path to the core dump file",
},
{
"id": "program",
"type": "promptString",
"description": "Path to the program to debug",
}
],
"configurations": [
{
"name": "[GDB] Load core dump file",
"type": "cppdbg",
"request": "launch",
"program": "${input:program}",
"coreDumpPath": "${input:core_dump}",
"cwd": "${workspaceFolder}",
"MIMode": "gdb",
"externalConsole": false,
"miDebuggerPath": "/usr/bin/gdb",
"setupCommands": [
{
"description": "Enable pretty-printing for gdb",
"text": "-enable-pretty-printing",
"ignoreFailures": true
},
{
"description": "Set Disassembly Flavor to Intel",
"text": "-gdb-set disassembly-flavor intel",
"ignoreFailures": true
}
]
},
{
"name": "[LLDB] Load core dump file",
"type": "lldb",
"request": "launch",
"stopOnEntry": true,
"processCreateCommands": [],
"targetCreateCommands": [
"target create -c ${input:core_dump} ${input:program}",
],
},
]
}
NOTE: The CodeLldb debug session does not stop after launching. To see the code, stack frames and registers you need to
press the pause button. This is a known issue with the CodeLldb extension #1245.
The cppdbg extension works as expected and stops at the entry point of the program.
This section explains how to compile a guest with debugging information but still have optimized code, and how to separate the debug information from the binary.
To create a release build with debug information, you can add a custom profile to your Cargo.toml file:
[profile.release-with-debug]
inherits = "release"
debug = trueThis creates a new profile called release-with-debug that inherits all settings from the release profile but adds debug information.
To reduce the binary size while still having debug information available, you can split the debug information into a separate file. This is useful for production environments where you want smaller binaries but still want to be able to debug crashes.
Here's a step-by-step guide:
-
Build your guest with the release-with-debug profile:
cargo build --profile release-with-debug
-
Locate your binary in the target directory:
TARGET_DIR="target" PROFILE="release-with-debug" ARCH="x86_64-unknown-none" # Your target architecture BUILD_DIR="${TARGET_DIR}/${ARCH}/${PROFILE}" BINARY=$(find "${BUILD_DIR}" -type f -executable -name "guest-binary" | head -1)
-
Extract debug information into a full debug file:
DEBUG_FILE_FULL="${BINARY}.debug.full" objcopy --only-keep-debug "${BINARY}" "${DEBUG_FILE_FULL}"
-
Create a symbols-only debug file (smaller, but still useful for stack traces):
DEBUG_FILE="${BINARY}.debug" objcopy --keep-file-symbols "${DEBUG_FILE_FULL}" "${DEBUG_FILE}"
-
Strip debug information from the original binary but keep function names:
objcopy --strip-debug "${BINARY}" -
Add a debug link to the stripped binary:
objcopy --add-gnu-debuglink="${DEBUG_FILE}" "${BINARY}"
After these steps, you'll have:
- An optimized binary with function names for basic stack traces
- A symbols-only debug file for stack traces
- A full debug file for complete source-level debugging
When you have a core dump from a crashed guest, you can analyze it with different levels of detail using either GDB or LLDB.
-
For basic analysis with function names (stack traces):
gdb ${BINARY} -c /path/to/core.dump -
For full source-level debugging:
gdb -s ${DEBUG_FILE_FULL} ${BINARY} -c /path/to/core.dump
LLDB provides similar capabilities with slightly different commands:
-
For basic analysis with function names (stack traces):
lldb ${BINARY} -c /path/to/core.dump -
For full source-level debugging:
lldb -o "target create -c /path/to/core.dump ${BINARY}" -o "add-dsym ${DEBUG_FILE_FULL}"
-
If your debug symbols are in a separate file:
lldb ${BINARY} -c /path/to/core.dump (lldb) add-dsym ${DEBUG_FILE_FULL}
You can configure VSCode (in .vscode/launch.json) to use these files by modifying the debug configurations:
{
"name": "[GDB] Load core dump with full debug symbols",
"type": "cppdbg",
"request": "launch",
"program": "${input:program}",
"coreDumpPath": "${input:core_dump}",
"cwd": "${workspaceFolder}",
"MIMode": "gdb",
"externalConsole": false,
"miDebuggerPath": "/usr/bin/gdb",
"setupCommands": [
{
"description": "Enable pretty-printing for gdb",
"text": "-enable-pretty-printing",
"ignoreFailures": true
}
]
}{
"name": "[LLDB] Load core dump with full debug symbols",
"type": "lldb",
"request": "launch",
"program": "${input:program}",
"cwd": "${workspaceFolder}",
"processCreateCommands": [],
"targetCreateCommands": [
"target create -c ${input:core_dump} ${input:program}"
],
"postRunCommands": [
// if debug symbols are in a different file
"add-dsym ${input:debug_file_path}"
]
}