Implement FEATURES=packdebug

Create a tarball of debug information and source files for use with
binary packages and debuginfod.

The tarball is owned by the package, but we add it to UNINSTALL_IGNORE
so it is kept around on upgrades etc., and we add it to PKG_INSTALL_MASK
so that it is not part of binpkgs themselves (should be fetched via debuginfod).

That is, for FEATURES=packdebug, we:
* Create a tarball in /usr/lib/debug/.tarball/${CATEGORY}/${PN}/${PF}-${BUILD_ID}-debug.tar.xz
  with debug information from /usr/lib/debug/*, /usr/src/debug/*. This
  is installed but not part of the binpkg.
* Set UNINSTALL_IGNORE="/usr/lib/debug/.tarball" to keep old
  debug information tarballs around to serve.
* Set PKG_INSTALL_MASK="/usr/lib/debug/* /usr/src/debug/*" so binpkgs
  do not contain debug information (they should fetch it via debuginfod
  instead).

The model is as follows:
* binhost

  Builds packages with -g* in *FLAGS and has FEATURES=splitdebug
  (and possibly FEATURES=installsources) to get debuginfo. Must also
  have build IDs enabled in the compiler.

  Uses FEATURES=packdebug to create tarballs containing that info.

  Runs `debuginfod ... -Z tar.xz /usr/lib/debug/.tarball` to serve
  the tarballs generated by packdebug.

* binpkg client

  Sets the `DEBUGINFOD_URLS` environment variable to the httpd that
  `debuginfod` runs.

Thanks to Arsen for the original bashrc-based implementation which
the estrip impl. is based upon, motivation for getting this done, and
the idea. See also https://wiki.gentoo.org/wiki/User:Arsen/Deguginfod.

Bug: https://bugs.gentoo.org/639376
Bug: https://bugs.gentoo.org/728818
Bug: https://bugs.gentoo.org/953869
Co-authored-by: Arsen Arsenović <[email protected]>
Signed-off-by: Sam James <[email protected]>

Author: thesamesam

bin/misc-functions.sh

@@ -502,6 +502,35 @@ preinst_selinux_labels() {
 	fi
 }
 
+# Generate a separate tarball with debug information (and sources) for
+# use with debuginfod.
+__generate_packdebug() {
+	local debugpath="${T}"/.tarball
+	if ! [[ -d "${ED}"/usr/src/debug || -d "${ED}"/usr/lib/debug ]]; then
+		return
+	fi
+
+	install -d "${debugpath}"/"${CATEGORY}"{,/"${PN}"} \
+		|| die "Failed to generate target debug directory"
+
+	(
+		local tarfile="${debugpath}/${CATEGORY}/${PN}/${PF}-${BUILD_ID}-debug.tar.xz"
+		cd "${ED}" || die
+		tar -cJf - > "${tarfile}" \
+			"$([[ -d ./usr/src/debug ]] && echo ./usr/src/debug)" \
+			"$([[ -d ./usr/lib/debug ]] && echo ./usr/lib/debug)" \
+		|| die "Failed to pack up debug info for FEATURES=packdebug"
+	)
+
+	# We don't use ${D} here because __generate_packdebug is called from
+	# __dyn_package where ${D} points to a pretend ${D}. We want these files
+	# in the real image but not in the binpkg. Unfortunately, we can't
+	# easily leverage PKG_INSTALL_MASK because of when it runs.
+	mv "${debugpath}" \
+		"${PORTAGE_TMPDIR}"/portage/${CATEGORY}/${PF}/${EPREFIX}/image/usr/lib/debug/ \
+	|| die
+}
+
 __dyn_package() {
 	if ! ___eapi_has_prefix_variables; then
 		local EPREFIX=
@@ -524,6 +553,24 @@ __dyn_package() {
 
 	if [[ ! -z "${BUILD_ID}" ]]; then
 		echo -n "${BUILD_ID}" > "${PORTAGE_BUILDDIR}"/build-info/BUILD_ID
+
+		# We generate the packdebug tarball at this point as we need
+		# the BUILD_ID, but it's not installed as part of the binpkg
+		# by design. We install it later when merging.
+		if contains_word packdebug "${FEATURES}" ; then
+			__generate_packdebug
+
+			# The injected PKG_INSTALL_MASK combined with us
+			# having splitdebug/installsources on may mean we
+			# have an empty /usr/src or /usr/lib left. Prune those.
+			#
+			# XXX: We use ${D}/${EPREFIX} here because we don't set
+			# ${ED} to the fake ${D}.
+			find \
+				"${D}/${EPREFIX}/usr/lib" \
+				"${D}/${EPREFIX}/usr/src" \
+				-type d -empty -delete 2>/dev/null
+		fi
 	fi
 
 	if [[ "${BINPKG_FORMAT}" == "xpak" ]]; then

lib/_emerge/EbuildPhase.py

@@ -128,6 +128,7 @@ class EbuildPhase(CompositeTask):
         "network-sandbox",
         "network-sandbox-proxy",
         "nostrip",
+        "packdebug",
         "preserve-libs",
         "sandbox",
         "selinux",

lib/_emerge/PackagePhase.py

@@ -49,6 +49,14 @@ def _start(self):
                 self._pkg_install_mask = InstallMask(f.read())
         except OSError:
             self._pkg_install_mask = None
+
+        if "packdebug" in self.settings.features:
+            tmp_pkg_install_mask = self._pkg_install_mask or ""
+            # We don't want to include debug information in binpkgs themselves
+            # w/ packdebug as binpkg consumers should fetch them via debuginfod.
+            tmp_pkg_install_mask += " /usr/src/debug/ /usr/lib/debug/"
+            self._pkg_install_mask = InstallMask(tmp_pkg_install_mask)
+
         if self._pkg_install_mask:
             self._proot = os.path.join(self.settings["T"], "packaging")
             self._start_task(

lib/portage/const.py

@@ -212,6 +212,7 @@
         "noman",
         "nostrip",
         "notitles",
+        "packdebug",
         "parallel-fetch",
         "parallel-install",
         "pid-sandbox",

lib/portage/dbapi/vartree.py

@@ -2750,7 +2750,15 @@ def _unmerge_pkgfiles(self, pkgfiles, others_in_slot):
             # process symlinks second-to-last, directories last.
             mydirs = set()
 
-            uninstall_ignore = shlex.split(self.settings.get("UNINSTALL_IGNORE", ""))
+            uninstall_ignore = self.settings.get("UNINSTALL_IGNORE", "")
+            if "packdebug" in self.settings.features:
+                # For packdebug, we don't want to clean up old debuginfo
+                # tarballs that we made, as we want to keep serving them
+                # to binhost clients for some time.
+                uninstall_ignore = self.settings.get("UNINSTALL_IGNORE", "")
+                uninstall_ignore += " /usr/lib/debug/.tarball"
+
+            uninstall_ignore = shlex.split(uninstall_ignore)
 
             def unlink(file_name, lstatobj):
                 if bsd_chflags:

man/make.conf.5

@@ -666,6 +666,16 @@ Prevents the stripping of binaries that are merged to the live filesystem.
 .B notitles
 Disables xterm titlebar updates (which contains status info).
 .TP
+.B packdebug
+Create a tarball of debug information and source files for use with
+debuginfod.  Debug tarballs are placed at
+\fI/usr/lib/debug/.tarball/${CATEGORY}/${PF}-${BUILD_ID}.tar.xz\fR.
+The debug tarball is installed but is not included in binary packages.
+Also, \fIUNINSTALL_IGNORE\fR is automatically modified to leave old debug
+tarballs around to allow debuginfod to serve them after upgrades. See
+\fBsplitdebug\fR for general split debug information (upon which this
+feature depends).
+.TP
 .B parallel\-fetch
 Fetch in the background while compiling. Run
 `tail \-f /var/log/emerge\-fetch.log` in a