Skip to content

Commit 2e7854b

Browse files
matejmatuskamatejmatuska
committed
Add tutorial: troubleshooting and debugging
1 parent 018e97c commit 2e7854b

File tree

2 files changed

+103
-0
lines changed

2 files changed

+103
-0
lines changed

‎docs/source/tutorials/index.rst

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@ Tutorials
88

99
templates/index
1010
howto-first-actor-upgrade
11+
troubleshooting-debugging
1112

1213
.. Indices and tables
1314
.. ==================

‎docs/source/tutorials/troubleshooting-debugging.md

Lines changed: 102 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,102 @@
1+
# Troubleshooting and Debugging
2+
This tutorial covers the different ways of debugging and troubleshooting the in-place upgrade process.
3+
4+
## Debug logs
5+
The leapp framework, as well as actors in the in-place upgrade repositories log important events during the upgrade. Debug logs are written to `/var/log/leapp/leapp-preupgrade.log` or `/var/log/leapp/leapp-upgrade.log`.
6+
7+
8+
```{note}
9+
Debug logs are not enabled by default, run `leapp` with the `--debug` option to enable debug logging.
10+
```
11+
12+
## Troubleshooting with leapp-inspector
13+
In addition to extensive logging, leapp writes a lot of information into it's database at `/var/lib/leapp/leapp.db`, such as messages and their contents, executed actors, logs and messages produced by individual actors and more.
14+
To view data in the database you can use the leapp-inspector tool, see the [docs](https://github.com/oamg/snippets/tree/main/scripts/leappinspector) for installation instructions.
15+
16+
```{note}
17+
`leapp-inspector` does not have to be installed on the system being upgraded. You can copy `leapp.db` to another system and use `leapp-inspector` there. In such case specify path to `leapp.db` using the `--db` option.
18+
```
19+
20+
### Usage
21+
See `leapp-inspector --help` for all options, it's well documented.
22+
23+
#### Inspect Specific Execution
24+
By default, `leapp-inspector` shows information about the most recent leapp execution.
25+
To inspect a specific execution:
26+
27+
1. List the executions and find the ID of the desired execution.
28+
```shell
29+
[root@-20250129142956 ~]$ leapp-inspector executions
30+
######################################################################
31+
Executions of Leapp
32+
######################################################################
33+
Execution | Timestamp
34+
------------------------------------ | ---------------------------
35+
58c2167d-d747-4a98-8660-af457e32711e | 2025-01-29T13:48:12.119698Z
36+
42340943-e06c-4ff7-a2e6-9601eb76bbe0 | 2025-01-29T15:10:10.892776Z
37+
```
38+
2. Add the `--context <execution id>` parameter in subsequent commands.
39+
40+
## Debugging Code
41+
[The Python Debugger](https://docs.python.org/3/library/pdb.html) - `pdb` can be used for standard debugging.
42+
1. Before running leapp insert the following line to set a breakpoint in the code:
43+
```python
44+
import pdb; pdb.set_trace()
45+
```
46+
2. Run leapp and wait for `pdb` prompt to appear.
47+
3. Debug. See the `pdb` docs on how to use it.
48+
4. To continue the upgrade use the `continue` (or `c`) command.
49+
50+
## Inspecting the Target Userspace Container During an Upgrade
51+
Sometimes it's useful to interactively inspect the target userspace container during the upgrade, e.g. to ensure required files are present. The following steps describe how to enter the container:
52+
53+
1. Set a breakpoint at the desired point in the code as described in the previous section
54+
2. Open another terminal session on the machine being upgraded
55+
3. Run the upgrade in the first terminal
56+
4. When the breakpoint is reached (`pdb` prompt is displayed), enter the container in the second terminal. As the container needs a lot of variables passed in, the easiest way to do this is to find the `systemd-nspawn` command that leapp used to enter the container in the debug output in the first terminal. In the following example the first line is the command:
57+
```
58+
2025-01-29 13:49:24.691 DEBUG PID: 40673 leapp.workflow.TargetTransactionFactsCollection.target_userspace_creator: External command has finished: ['systemd-nspawn', '--register=no', '--quiet', '--keep-unit', '--capability=all', '-D', '/var/lib/leapp/scratch/mounts/root_/system_overlay', '--setenv=LEAPP_DEVEL_USE_PERSISTENT_PACKAGE_CACHE=1', '--setenv=LEAPP_NO_RHSM=1', '--setenv=LEAPP_UNSUPPORTED=1', '--setenv=LEAPP_HOSTNAME=leapp-20250129142956', '--setenv=LEAPP_EXPERIMENTAL=0', '--setenv=LEAPP_NO_RHSM_FACTS=1', '--setenv=LEAPP_TARGET_PRODUCT_CHANNEL=ga', '--setenv=LEAPP_UPGRADE_PATH_TARGET_RELEASE=9.6', '--setenv=LEAPP_UPGRADE_PATH_FLAVOUR=default', '--setenv=LEAPP_IPU_IN_PROGRESS=8to9', '--setenv=LEAPP_EXECUTION_ID=58c2167d-d747-4a98-8660-af457e32711e', '--setenv=LEAPP_COMMON_TOOLS=:/etc/leapp/repos.d/system_upgrade/common/tools:/etc/leapp/repos.d/system_upgrade/el8toel9/tools', '--setenv=LEAPP_COMMON_FILES=:/etc/leapp/repos.d/system_upgrade/common/files:/etc/leapp/repos.d/system_upgrade/el8toel9/files', '--setenv=LEAPP_DEVEL_DATABASE_SYNC_OFF=1', 'dnf', 'install', '-y', '--setopt=module_platform_id=platform:el9', '--setopt=keepcache=1', '--releasever', '9.6', '--installroot', '/el9target', '--disablerepo', '*', '--enablerepo', 'BASEOS', '--enablerepo', 'APPSTREAM', 'util-linux', 'kpatch-dnf', 'dnf', 'dnf-command(config-manager)', '-v', '--disableplugin', 'subscription-manager']
59+
2025-01-29 13:49:24.692 DEBUG PID: 40673 leapp.workflow.TargetTransactionFactsCollection.target_userspace_creator: External command has started: ['umount', '-fl', '/var/lib/leapp/scratch/mounts/root_/system_overlay/el9target']
60+
2025-01-29 13:49:24.703 DEBUG PID: 40673 leapp.workflow.TargetTransactionFactsCollection.target_userspace_creator: External command has finished: ['umount', '-fl', '/var/lib/leapp/scratch/mounts/root_/system_overlay/el9target']
61+
2025-01-29 13:49:24.704 DEBUG PID: 40673 leapp.workflow.TargetTransactionFactsCollection.target_userspace_creator: External command has started: ['rm', '-rf', '/var/lib/leapp/scratch/mounts/root_/system_overlay/el9target']
62+
2025-01-29 13:49:24.709 DEBUG PID: 40673 leapp.workflow.TargetTransactionFactsCollection.target_userspace_creator: External command has finished: ['rm', '-rf', '/var/lib/leapp/scratch/mounts/root_/system_overlay/el9target']
63+
> /etc/leapp/repos.d/system_upgrade/common/actors/targetuserspacecreator/libraries/userspacegen.py(569)_copy_certificates()
64+
-> files_owned_by_rpms = _get_files_owned_by_rpms(target_context, '/etc/pki', recursive=True)
65+
(Pdb)
66+
```
67+
Command will contain another shell command at the end (`dnf install ...` in this example) that needs to be stripped. Then run the command in the second terminal to enter and inspect the container:
68+
```bash
69+
# WARNING: DO NOT BLINDLY COPY THIS COMMAND, THE VARIABLE VALUES ARE UNIQUE FOR AN UPGRADE
70+
systemd-nspawn --register=no --quiet --keep-unit --capability=all -D /var/lib/leapp/scratch/mounts/root_/system_overlay --setenv=LEAPP_DEVEL_USE_PERSISTENT_PACKAGE_CACHE=1 --setenv=LEAPP_NO_RHSM=1 --setenv=LEAPP_UNSUPPORTED=1 --setenv=LEAPP_HOSTNAME=leapp-20250129142956 --setenv=LEAPP_EXPERIMENTAL=0 --setenv=LEAPP_NO_RHSM_FACTS=1 --setenv=LEAPP_TARGET_PRODUCT_CHANNEL=ga --setenv=LEAPP_UPGRADE_PATH_TARGET_RELEASE=9.6 --setenv=LEAPP_UPGRADE_PATH_FLAVOUR=default --setenv=LEAPP_IPU_IN_PROGRESS=8to9 --setenv=LEAPP_EXECUTION_ID=58c2167d-d747-4a98-8660-af457e32711e --setenv=LEAPP_COMMON_TOOLS=:/etc/leapp/repos.d/system_upgrade/common/tools:/etc/leapp/repos.d/system_upgrade/el8toel9/tools --setenv=LEAPP_COMMON_FILES=:/etc/leapp/repos.d/system_upgrade/common/files:/etc/leapp/repos.d/system_upgrade/el8toel9/files --setenv=LEAPP_DEVEL_DATABASE_SYNC_OFF=1
71+
```
72+
5. To continue the upgrade, exit the container and resume the debugger (`continue` or `c`).
73+
74+
75+
## Debugging Inside Initramfs
76+
One of the biggest debugging challenges is exploring something in initramfs stage, as currently there is no network connectivity.
77+
78+
```{note}
79+
When working in initramfs stage you will need a serial console. There is not network connectivity, SSH does not work.
80+
```
81+
82+
83+
1. If not already in the dracut emergency console, such as in case of an error, a breakpoint can be set to force leapp to enter the console. The following breakpoints are available:
84+
85+
| Breakpoint | Description |
86+
|----------- |------------ |
87+
| leapp-pre-upgrade | Break right before running leapp in initramfs |
88+
| leapp-upgrade | Break right after LEAPP upgrade, before post-upgrade leapp run |
89+
| leapp-finish | Break after LEAPP save_journal (upgrade initramfs end) |
90+
91+
The breakpoint has to be passed using the `rd.upgrade.break=` argument on the kernel command line. There are (at least) 2 ways to do this:
92+
- Modify the cmdline in GRUB, press Ctrl-X when the Upgrade initramfs is selected and add the argument at the end of the line starting with `LINUX=`
93+
- Modify the [addupgradebootentry actor](https://github.com/oamg/leapp-repository/blob/master/repos/system_upgrade/common/actors/addupgradebootentry/libraries/addupgradebootentry.py#L23) to add the argument
94+
95+
2. Once in the emergency shell, you may want to use some of the utilities provided in `$CWD/leapp_debug_tools.sh` (should be automatically sourced):
96+
- `leapp_dbg_mount` - mounts `/sysroot` as read/write
97+
- `leapp_dbg_source` - the set of commands available in the emergency shell is very limited, this sets PATH and LD_LIBRARY_PATH to be able to run commands from `/sysroot`
98+
- `leapp_dbg_chroot` - change root into `/sysroot`
99+
100+
3. Inspect/modify `/sysroot` as needed
101+
4. To continue the upgrade, exit the shell
102+

0 commit comments

Comments
 (0)