Implement FEATURES=packdebug #1523

thesamesam · 2025-11-29T04:16:17Z

Create a tarball of debug information and source files for use with binary packages and debuginfod.

The tarball is owned by the package, but we add it to UNINSTALL_IGNORE so it is kept around on upgrades etc., and we add it to PKG_INSTALL_MASK so that it is not part of binpkgs themselves (should be fetched via debuginfod).

That is, for FEATURES=packdebug, we:

Create a tarball in /usr/lib/debug/.tarball/${CATEGORY}/${PN}/${PF}-${BUILD_ID}-debug.tar.xz with debug information from /usr/lib/debug/*, /usr/src/debug/*. This is installed but not part of the binpkg.
Set UNINSTALL_IGNORE="/usr/lib/debug/.tarball" to keep old debug information tarballs around to serve.
Set PKG_INSTALL_MASK="/usr/lib/debug/* /usr/src/debug/*" so binpkgs do not contain debug information (they should fetch it via debuginfod instead).

The model is as follows:

binhost

Builds packages with -g* in *FLAGS and has FEATURES=splitdebug
(and possibly FEATURES=installsources) to get debuginfo. Must also
have build IDs enabled in the compiler.

Uses FEATURES=packdebug to create tarballs containing that info.

Runs debuginfod ... -Z tar.xz \ /usr/lib/debug/.tarball \ -Z.gpkg.tar='(bsdtar -xf- -O '*/image.tar.*' | bsdtar --strip-components 1 -cf- @-)<' \ /var/cache/binpkgs
to serve the tarballs generated by packdebug, as well as binpkgs themselves
for clients to request full binaries corresponding to coredumps etc.
binpkg client

Sets the DEBUGINFOD_URLS environment variable to the httpd that
debuginfod runs.

Thanks to Arsen for the original bashrc-based implementation which
the estrip impl. is based upon, motivation for getting this done, and
the idea. See also https://wiki.gentoo.org/wiki/User:Arsen/Deguginfod.

This is also really useful for not just serving binpkgs to clients and
letting them get debuginfo (of course the main usecase), but also getting
debuginfo for local binaries you don't even publish anywhere after an upgrade
when an old process is still running. For this part to work well, you need
the binaries themselves too which the above debuginfod invocation covers.

Note that xz is used for the debug tarball because it supports [0][1] random
access.

[0] https://blog.osandov.com/2024/07/25/making-debuginfod-viable-for-the-linux-kernel.html
[1] https://developers.redhat.com/articles/2025/01/14/debuginfod-project-update-2024#addressing_kernel_vdso_extraction_bottlenecks

Bug: https://sourceware.org/PR33693
Bug: https://bugs.gentoo.org/639376
Bug: https://bugs.gentoo.org/728818
Bug: https://bugs.gentoo.org/953869
Co-authored-by: Arsen Arsenović [email protected]

thesamesam · 2025-11-29T04:45:23Z

Testing very much welcome and encouraged. See https://public-inbox.gentoo.org/gentoo-dev/[email protected]/ for how to get build IDs.

Here's the debuginfod.service I'm using from @ArsenArsen:

# -*- conf-unix -*-

[Unit]
Description=elfutils debuginfod is a server that automatically distributes ELF/DWARF/source code from servers to clients such as debuggers across HTTP.
Documentation=https://sourceware.org/elfutils/Debuginfod.html man:debuginfod(8)
After=network.target

[Service]
DynamicUser=true
ExecStart=/usr/bin/debuginfod --listen-address=127.0.0.1 --database=${CACHE_DIRECTORY}/db.sqlite3 -Z tar.xz /usr/lib/debug/.tarball

ProtectProc=invisible
CapabilityBoundingSet=CAP_DAC_OVERRIDE
AmbientCapabilities=CAP_DAC_OVERRIDE
NoNewPrivileges=true
ProtectSystem=strict
ProtectHome=tmpfs
PrivateTmp=on
PrivateDevices=on
PrivateIPC=on
PrivateUsers=on
ProtectClock=on
ProtectHostname=on
ProtectKernelTunables=on
ProtectKernelModules=on
ProtectKernelLogs=on
ProtectControlGroups=on
LockPersonality=on
RestrictSUIDSGID=on
PrivateMounts=on
SystemCallFilter=@system-service
UnsetEnvironment=DEBUGINFOD_URLS

TemporaryFileSystem=/:ro
BindReadOnlyPaths=/usr/src/debug /usr/bin/ /usr/lib64 /usr/lib
NoExecPaths=/
ExecPaths=/usr/bin/debuginfod

CacheDirectory=debuginfod

[Install]
WantedBy=multi-user.target

bin/misc-functions.sh

lib/portage/dbapi/vartree.py

eli-schwartz

Very exciting. Haven't tested yet, have some quick comments.

eli-schwartz · 2025-11-30T03:33:14Z

bin/misc-functions.sh

+			find \
+				"${D}/${EPREFIX}/usr/lib" \
+				"${D}/${EPREFIX}/usr/src" \
+				-type d -empty -delete 2>/dev/null


So it's ok to have an empty /usr (maybe all the files are in /opt or idk)?

Or, opposite problem: what if the user for whatever reason created /usr/lib/myapp/ but never filled it out? ("But it's undefined behavior" is an acceptable answer, just trying to better understand)

See bfda0d2 and 7b44116.

The /usr/lib/debug and /usr/src/debug directories may be created by the splitdebug and installsources machinery. I think what I'm doing atm indeed assumes there's at least something in /usr which isn't necessarily true.

Any suggestions? For >= EAPI 8 (for Portage behaviour), we can assume that no empty directories should exist in the image at this point, though I think people may create them in pkg_postinst (not sure about pkg_preinst). Sometimes people have to do those hacks because some software wants a genuinely empty directory without the .keepdir indicator we install.

One could say that pedantically what we want is to detect when the mkdir call itself was issued by splitdebug, as that's the only technically correct way to unwind the effects of splitdebug, exactly the effects of splitdebug, and ONLY the effects of splitdebug, but that requires memorizing state.

If leaving the directory around is okay, this code is already okay.

If removing the directory unconditionally is okay, you can use e.g.

(cd "${D}/${EPREFIX}"/usr && find \ ./lib \ ./src \ -type d -empty -exec rmdir -p {} 2>/dev/null)

bin/misc-functions.sh

lib/portage/dbapi/vartree.py

ArsenArsen · 2025-11-30T16:31:31Z

Set PKG_INSTALL_MASK="/usr/lib/debug/* /usr/src/debug/*" so binpkgs do not contain debug information (they should fetch it via debuginfod instead).

it'd be desirable to set INSTALL_MASK also, but with /usr/lib/debug/.tarball that is impossible, right?

thesamesam · 2025-11-30T17:13:27Z

Why would that be desirable? Note that PKG_INSTALL_MASK is for creating the binpkg, not when installing the binpkg.

Do you mean on the producer side, they may want to rely on the tarballs and not have a 'duplicate' copy of the debug info in /usr/lib/debug? I think INSTALL_MASK supports negations, it's just undocumented.

ArsenArsen · 2025-11-30T20:19:14Z

Do you mean on the producer side, they may want to rely on the tarballs and not have a 'duplicate' copy of the debug info in /usr/lib/debug? I think INSTALL_MASK supports negations, it's just undocumented.

yes. if negations are possible, let's document that. I'd like to recommend it. (duplicating the debug info is very undesirable, it can be hundreds of gigabytes)

thesamesam · 2025-11-30T20:27:45Z

Support was introduced by 8f9cff7. Updating docs..

lib/portage/dbapi/vartree.py

thesamesam · 2025-12-04T05:27:58Z

debuginfod also supports clients requesting binaries themselves which can be useful when executables were replaced on-disk. I can see this being handy when debugging a problem reported by a user too (as we can grab the binary they had or have just by build ID).

We need to teach it to read the GPKG format for that, which almost Just Works, except for the image/ prefix: https://sourceware.org/PR33693

thesamesam · 2025-12-05T10:16:15Z

We had discussions on the bug and Arsen came up with a great solution. New debuginfod.service:

# -*- conf-unix -*-

[Unit]
Description=elfutils debuginfod is a server that automatically distributes ELF/DWARF/source code from servers to clients such as debuggers across HTTP.
Documentation=https://sourceware.org/elfutils/Debuginfod.html man:debuginfod(8)
After=network.target

[Service]
DynamicUser=true
# We pass in both the packdebug location and the binpkgs location itself for binaries
ExecStart=/usr/bin/debuginfod --listen-address=127.0.0.1 --database=${CACHE_DIRECTORY}/db.sqlite3 -Z tar.xz /usr/lib/debug/.tarball -Z.gpkg.tar='(bsdtar -xf- -O '*/image.tar.*' | bsdtar --strip-components 1 -cf- @-)<' /var/cache/binpkgs

ProtectProc=invisible
CapabilityBoundingSet=CAP_DAC_OVERRIDE
AmbientCapabilities=CAP_DAC_OVERRIDE
NoNewPrivileges=true
ProtectSystem=strict
ProtectHome=tmpfs
PrivateTmp=on
PrivateDevices=on
PrivateIPC=on
PrivateUsers=on
ProtectClock=on
ProtectHostname=on
ProtectKernelTunables=on
ProtectKernelModules=on
ProtectKernelLogs=on
ProtectControlGroups=on
LockPersonality=on
RestrictSUIDSGID=on
PrivateMounts=on
SystemCallFilter=@system-service
UnsetEnvironment=DEBUGINFOD_URLS

TemporaryFileSystem=/:ro
BindReadOnlyPaths=/usr/src/debug /usr/bin/ /usr/lib64 /usr/lib
NoExecPaths=/
ExecPaths=/usr/bin/debuginfod

CacheDirectory=debuginfod

[Install]
WantedBy=multi-user.target

thesamesam · 2025-12-05T10:26:40Z

Remaining TODOs:

Check if we can exclude easily debug information from installation on the binhost to avoid having duplicate copies (in the tarballs and on disk in the /usr/lib/debug, /usr/src/debug directories). This will be optional and off-by-default. (EDIT: Done via INSTALL_MASK reference in packdebug docs.)
Think about the empty directory issue discussed at Implement FEATURES=packdebug #1523 (comment).
~~Think about the "depclean immediately w/o FEATURES set" issue at Implement FEATURES=packdebug #1523 (comment) (I think this is fine and we just want to document it).~~ (EDIT: Done.)
~~Check this (may need to add to COLLISION_IGNORE or something):~~ (EDIT: Done.)

* Detected file collision(s):
 *
 *      /usr/lib/debug/.tarball/dev-ada/gprbuild/gprbuild-25.0.0-r5-1-debug.tar.xz
 *
 * Searching all installed packages for file collisions...
 *
 * Press Ctrl-C to Stop

Create a tarball of debug information and source files for use with binary packages and debuginfod. The tarball is owned by the package, but we add it to UNINSTALL_IGNORE so it is kept around on upgrades etc., and we add it to PKG_INSTALL_MASK so that it is not part of binpkgs themselves (should be fetched via debuginfod). That is, for FEATURES=packdebug, we: * Create a tarball in /usr/lib/debug/.tarball/${CATEGORY}/${PN}/${PF}-${BUILD_ID}-debug.tar.xz with debug information from /usr/lib/debug/*, /usr/src/debug/*. This is installed but not part of the binpkg. * Set UNINSTALL_IGNORE="/usr/lib/debug/.tarball" to keep old debug information tarballs around to serve. * Set COLLISION_IGNORE="/usr/lib/debug/.tarball" to ignore collisions from such orphaned debug information tarballs left around by UNINSTALL_IGNORE. * Set PKG_INSTALL_MASK="/usr/lib/debug/* /usr/src/debug/*" so binpkgs do not contain debug information (they should fetch it via debuginfod instead). The model is as follows: * binhost Builds packages with -g* in *FLAGS and has FEATURES=splitdebug (and possibly FEATURES=installsources) to get debuginfo. Must also have build IDs enabled in the compiler. Uses FEATURES=packdebug to create tarballs containing that info. Runs `debuginfod ... -Z tar.xz \ /usr/lib/debug/.tarball \ -Z.gpkg.tar='(bsdtar -xf- -O '*/image.tar.*' | bsdtar --strip-components 1 -cf- @-)<' \ /var/cache/binpkgs` to serve the tarballs generated by packdebug, as well as binpkgs themselves for clients to request full binaries corresponding to coredumps etc. Users may want to set INSTALL_MASK in make.conf on the binhost side to avoid duplicate copies of debug information on the live filesystem (/usr/lib/debug, /usr/src/debug) and in the packdebug tarballs: ``` INSTALL_MASK="${INSTALL_MASK} /usr/lib/debug/ /usr/src/debug/ -/usr/lib/debug/.tarball" ``` (That is not done by default because one may wish to have non-debuginfod- aware software work, and it also feels like it's a bit surprising for us to do automatically.) * binpkg client Sets the `DEBUGINFOD_URLS` environment variable to the httpd that `debuginfod` runs. Thanks to Arsen for the original bashrc-based implementation which the estrip impl. is based upon, motivation for getting this done, and the idea. See also https://wiki.gentoo.org/wiki/User:Arsen/Deguginfod. This is also really useful for not just serving binpkgs to clients and letting them get debuginfo (of course the main usecase), but also getting debuginfo for local binaries you don't even publish anywhere after an upgrade when an old process is still running. For this part to work well, you need the binaries themselves too which the above `debuginfod` invocation covers. Note that xz is used for the debug tarball because it supports [0][1] random access. [0] https://blog.osandov.com/2024/07/25/making-debuginfod-viable-for-the-linux-kernel.html [1] https://developers.redhat.com/articles/2025/01/14/debuginfod-project-update-2024#addressing_kernel_vdso_extraction_bottlenecks Bug: https://sourceware.org/PR33693 Bug: https://bugs.gentoo.org/639376 Bug: https://bugs.gentoo.org/728818 Bug: https://bugs.gentoo.org/953869 Co-authored-by: Arsen Arsenović <[email protected]> Signed-off-by: Sam James <[email protected]>

zmedico · 2025-12-26T18:54:26Z

bin/misc-functions.sh

+		"${PORTAGE_TMPDIR}/portage/${CATEGORY}/${PF}/image/${EPREFIX}/usr/lib/debug/". \
+	|| die
+}
+


It would be nicer if we could use ${PORTAGE_BUILDDIR} instead of ${PORTAGE_TMPDIR}/portage/${CATEGORY}/${PF} here.

Ah, thanks!

thesamesam requested review from ArsenArsen, eli-schwartz and zmedico November 29, 2025 04:16

thesamesam force-pushed the sam-packdebug branch from b48a637 to 5abea86 Compare November 29, 2025 04:32

ArsenArsen reviewed Nov 29, 2025

View reviewed changes

bin/misc-functions.sh Outdated Show resolved Hide resolved

lib/portage/dbapi/vartree.py Outdated Show resolved Hide resolved

thesamesam force-pushed the sam-packdebug branch 5 times, most recently from 9257716 to 1f73e84 Compare November 29, 2025 15:15

eli-schwartz reviewed Nov 30, 2025

View reviewed changes

thesamesam force-pushed the sam-packdebug branch from 1f73e84 to 6625596 Compare November 30, 2025 20:41

thesamesam commented Dec 1, 2025

View reviewed changes

lib/portage/dbapi/vartree.py Outdated Show resolved Hide resolved

thesamesam force-pushed the sam-packdebug branch from 6625596 to e969507 Compare December 1, 2025 06:46

thesamesam force-pushed the sam-packdebug branch 2 times, most recently from 3a02ed2 to fe6b096 Compare December 8, 2025 19:16

thesamesam force-pushed the sam-packdebug branch 2 times, most recently from 6946868 to 5632278 Compare December 22, 2025 02:25

thesamesam force-pushed the sam-packdebug branch from 5632278 to 83fced8 Compare December 26, 2025 18:15

zmedico reviewed Dec 26, 2025

View reviewed changes

Implement FEATURES=packdebug #1523

Are you sure you want to change the base?

Implement FEATURES=packdebug #1523

Conversation

thesamesam commented Nov 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

thesamesam commented Nov 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

eli-schwartz left a comment

Choose a reason for hiding this comment

Uh oh!

eli-schwartz Nov 30, 2025

Choose a reason for hiding this comment

Uh oh!

thesamesam Nov 30, 2025

Choose a reason for hiding this comment

Uh oh!

eli-schwartz Dec 1, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

ArsenArsen commented Nov 30, 2025

Uh oh!

thesamesam commented Nov 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ArsenArsen commented Nov 30, 2025

Uh oh!

thesamesam commented Nov 30, 2025

Uh oh!

Uh oh!

thesamesam commented Dec 4, 2025

Uh oh!

thesamesam commented Dec 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

thesamesam commented Dec 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zmedico Dec 26, 2025

Choose a reason for hiding this comment

Uh oh!

thesamesam Dec 27, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

thesamesam commented Nov 29, 2025 •

edited

Loading

thesamesam commented Nov 29, 2025 •

edited

Loading

thesamesam commented Nov 30, 2025 •

edited

Loading

thesamesam commented Dec 5, 2025 •

edited

Loading

thesamesam commented Dec 5, 2025 •

edited

Loading