publish: linux coredumpts part 1 (#553)

### Summary

 Linux coredumps -- part 1!

 ### Test Plan

 - [x] Check links
 - [x] Check images

Author: gminn

_drafts/linux_coredump.md renamed to _posts/2025-02-14-linux-coredumps-part-1.md

@@ -1,22 +1,23 @@
 ---
-title: Coredumps at Memfault Part 1 - Introduction to Linux Coredumps
+title: Linux Coredumps (Part 1) － Introduction
 description:
   "The basics of Linux coredumps, how they're used at Memfault, and how they're
   captured."
 author: blake
+tags: [linux, coredumps, memfault, debugging]
 ---
 
 One of the core features of the Memfault Linux SDK is the ability to capture and
-analyze crashes. Since the inception of the SDK we've been slowly expanding our
+analyze crashes. Since the inception of the SDK, we've been slowly expanding our
 crash capture and analysis capabilities. Starting from the standard ELF
-coredump, we've added support for capturing only the stack memory, and even
+coredump, we've added support for capturing only the stack memory and even
 capturing just the stack trace with no registers and locals present. This
-article series will give you a high level overview of that journey, and give you
-a deeper understanding of how coredumps work on Linux.\*\*\*\*
+article series will give you a high-level overview of that journey and a deeper
+understanding of how coredumps work on Linux.
 
 <!-- excerpt start -->
 
-In this article we'll start by taking a look at how a Linux coredump is
+In this article, we'll start by taking a look at how a Linux coredump is
 formatted, how you capture them, and how we use them at Memfault.
 
 <!-- excerpt end -->
@@ -27,43 +28,43 @@ formatted, how you capture them, and how we use them at Memfault.
 
 ## What is a Linux Coredump
 
-A linux coredump represents a snapshot of the crashing process' memory. It can
+A Linux coredump represents a snapshot of the crashing process' memory. It can
 be loaded into programs like GDB to inspect the state of the process at the time
 of crash. It is written as an ELF[^elf_format] file. The entirety of the ELF
 format is outside the scope of this article, but we will touch on a few of the
 more important bits when looking at an ELF core file.
 
-## What triggers a cordump
+## What Triggers a Coredump
 
 Coredumps are triggered by certain signals generated by or sent to a program.
 The full list of signals can be found in the signal man page[^man_signal]. Here
 are the signals that cause a coredump:
 
-- SIGABRT: Abnormal termination of the program, such as a call to abort.
-- SIGBUS: Bus error (bad memory access).
-- SIGFPE: Floating-point exception.
-- SIGILL: Illegal instruction.
-- SIGQUIT: Quit from keyboard.
-- SIGSEGV: Invalid memory reference.
-- SIGSYS: Bad system call.
-- SIGTRAP: Trace/breakpoint trap.
-
-Of these the most common culprits you'll likely see are `SIGSEGV`, `SIGBUS`, and
-`SIGABRT`. These are signals that will be generated when a program tries to
-access memory that it doesn't have access to, tries to dereference a null
-pointer, or when the program calls `abort`. These typically indicate a fairly
-serious bug in either your program, or the libraries that it uses.
-
-Coredumps are very useful in these situations, as generally you're going to want
-to inspect the running state of the process a the time of crash. From the
-coredump you can get a backtrace of the crashing thread, the values of the
+- `SIGABRT`: Abnormal termination of the program, such as a call to abort.
+- `SIGBUS`: Bus error (bad memory access).
+- `SIGFPE`: Floating-point exception.
+- `SIGILL`: Illegal instruction.
+- `SIGQUIT`: Quit from keyboard.
+- `SIGSEGV`: Invalid memory reference.
+- `SIGSYS`: Bad system call.
+- `SIGTRAP`: Trace/breakpoint trap.
+
+Of these, the most common culprits you'll likely see are `SIGSEGV`, `SIGBUS`,
+and `SIGABRT`. These signals will be generated when a program tries to access
+memory that it doesn't have access to, tries to dereference a null pointer, or
+when the program calls `abort`. These typically indicate a fairly serious bug in
+either your program or the libraries that it uses.
+
+Coredumps are very useful in these situations, as generally, you're going to
+want to inspect the running state of the process at the time of crash. From the
+coredump, you can get a backtrace of the crashing thread, the values of the
 registers at the time of crash, and the values of the local variables at each
 frame of the backtrace.
 
-## How are coredumps enabled/collected
+## How are Coredumps Enabled/Collected
 
 Enabling coredumps on your Linux device requires a few configuration options. To
-start with you'll need the following options enabled on your kernel at a
+start with, you'll need the following options enabled on your kernel at a
 minimum:
 
 ```c
@@ -74,30 +75,42 @@ CONFIG_CORE_DUMP_DEFAULT_ELF_HEADERS=y
 These settings will enable the kernel to generate coredumps, as well as set the
 default mappings that are present in the coredump. `man core`[^man_core]
 provides a good overview of the options available to you when configuring
-coredumps.
+coredumps. It's worth noting that these options are enabled for most distros by
+default.
 
-### core_pattern
+In addition the kernel configuration, you'll need to set the `ulimit` for the
+process that you want to capture a coredump for. The `ulimit` command is used to
+set the resource limits for a process. The `core` resource limit is the one
+we're interested in. This sets the maximum size of a coredump that can be
+generated by a process. To make things easy, you can set it to unlimited with
+the following command:
+
+```bash
+ulimit -c unlimited
+```
+
+### `core_pattern`
 
 The kernel provides an interface for controlling where and how coredumps are
 written. The `/proc/sys/kernel/core_pattern`[^man_core] file provides two
 methods for capturing coredumps from crashed processes. A coredump can be
-written directly to a file by providing a path directly to it. For example if we
-wanted to write the core file to our `/tmp` directory with both the process name
-and the pid we would write the following to `/proc/sys/kernel/core_pattern`.
+written directly to a file by providing its path. For example, if we wanted to
+write the core file to our `/tmp` directory with both the process name and the
+pid, we would write the following to `/proc/sys/kernel/core_pattern`.
 
 ```bash
 /tmp/core.%e.%p
 ```
 
-In this example `%e` expands to the name of the crashing process, and `%p`
+In this example, `%e` expands to the name of the crashing process, and `%p`
 expands to the PID of the crashing process. More information on the available
 expansions can be found in the `man core`[^man_core] page.
 
-We can also pipe a coredump directly to a program. This is useful when we want
+We can also pipe a coredump directly to a program, which is useful when we want
 to modify the coredump in flight. The coredump is streamed to the provided
-program via `stdin`. The configuration is similar to saving directly to a file
+program via `stdin`. The configuration is similar to saving directly to a file ,
 except the first character must be a `|`. This is how we capture coredumps in
-the Memfault SDK, and will be covered more in depth later in the article.
+the Memfault SDK, and will be covered more in-depth later in this article.
 
 #### `procfs` Shallow Dive
 
@@ -106,32 +119,32 @@ program that is being piped to exits, we have access to the `procfs` of the
 crashing process. But what is `procfs`, and how does it help us with a coredump?
 
 `procfs` gives us direct, usually read-only, access to some of the kernel's data
-structures[^man_proc]. This can be system wide information, or information about
-individual processes. For our purposes we are interested mostly in the
-information about the process that is currently crashing. We can get direct read
-only access to all mapped memory by address through
-`/proc/<pid>/mem`[^man_proc_pid_mem], or look at the command line arguments of
-the process through `/proc/<pid>/cmdline`[^man_proc_pid_cmdline].
+structures[^man_proc]. This can be system-wide information, or information about
+individual processes. We are mostly interested in information about the process
+that is currently crashing. We can get direct, read-only access to all mapped
+memory by address through `/proc/<pid>/mem`[^man_proc_pid_mem], or look at the
+command line arguments of the process through
+`/proc/<pid>/cmdline`[^man_proc_pid_cmdline].
 
 ## Elf Core File Layout
 
 Linux coredumps use a subset of the ELF format. The coredump itself is a
 snapshot of the crashing process' memory, as well as some metadata to help
-debuggers understand the state of the process at the time of crash. We will
+debuggers understand the state of the process at the time of the crash. We will
 touch on the most important aspects of the core file in this article. We will
-not be doing an exhaustive dive into the ELF format, however, if you are
+not be doing an exhaustive dive into the ELF format; however, if you are
 interested in learning more about the ELF format, the ELF File
 Format[^elf_format] is a great resource.
 
 ![]({% img_url linux-coredump/elf-core-layout.png %})
 
 ### ELF Header
 
-The above image gives us a very high level view of the layout of a coredump. To
-start, the ELF header outlines the layout of the file and source of the file. We
-can see if the producing system was 32-bit or 64-bit, little or big endian, and
-the architecture of the system. Additionally it shows the offset to the program
-headers. Here is the layout of the ELF header[^elf_format]:
+The above image gives us a very high-level view of the layout of a coredump. To
+start, the ELF header outlines the layout of the file and the source of the
+file. We can see if the producing system was 32-bit or 64-bit, little or big
+endian, and the architecture of the system. Additionally it shows the offset to
+the program headers. Here is the layout of the ELF header[^elf_format]:
 
 ```c
 typedef struct {
@@ -158,20 +171,20 @@ discussion are broken down below:
 - `e_ident`: This field is an array of bytes that identify the file as an ELF
   file.
 - `e_type`: This field tells us what type of file we are looking at. For our
-  purposes this will always be `ET_CORE`.
+  purposes, this will always be `ET_CORE`.
 - `e_machine`: This field tells us the architecture of the system that produced
   the file. Common values here are
   [`EM_ARM`](https://github.com/torvalds/linux/blob/c45323b7560ec87c37c729b703c86ee65f136d75/include/uapi/linux/elf-em.h#L26)
-  for 32 bit ARM, and
+  for 32-bit ARM, and
   [`EM_AARCH64`](https://github.com/torvalds/linux/blob/c45323b7560ec87c37c729b703c86ee65f136d75/include/uapi/linux/elf-em.h#L46)
   for aarch64.
 - `e_phoff`: This field tells us the offset to the program headers.
 - `e_phentsize`: This field tells us the size of each program header.
 
 ### Program Headers and Segments
 
-The meat of our coredump exists in the program headers. There are a wide variety
-of program header types defined in the Elf File Format[^elf_format]. From the
+The meat of our coredump exists in the program headers. A wide variety of
+program header types are defined in the Elf File Format[^elf_format]. From the
 perspective of the coredump, however, we are primarily interested in the
 `PT_NOTE` and `PT_LOAD` program headers.
 
@@ -192,8 +205,8 @@ typedef struct {
 
 Here is a brief breakdown of the fields we care about in the program header:
 
-- `p_type`: This field tells us what type of segment we are looking at. For our
-  purposes this will be either `PT_NOTE` or `PT_LOAD`.
+- `p_type`: This field tells us what type of segment we are looking at. This
+  will be either `PT_NOTE` or `PT_LOAD` for our purposes.
 - `p_offset`: This field tells us the offset from the beginning of the file
   where the segment starts.
 - `p_vaddr`: This field tells us the virtual address where the segment is
@@ -204,22 +217,22 @@ Here is a brief breakdown of the fields we care about in the program header:
 - `p_memsz`: This field tells us the size of the segment in memory.
 - `p_align`: This field tells us the alignment of the segment.
 
-We'll start by taking a look at the format of the `PT_NOTE` segments. Below is
-the layout of a `PT_NOTE` segment.
+We'll start by looking at the format of the `PT_NOTE` segments. Below is the
+layout of a `PT_NOTE` segment.
 
 ![]({% img_url linux-coredump/elf-note-layout.png %})
 
-The first two fields of the segment are fairly self explanatory, they represent
-the size of both the name and the descriptor. The `name` field is a string that
-represents the type of note. The `desc` field is a structure that contains the
+The first two fields of the segment are fairly self-explanatory, they represent
+the size of both the name and the descriptor. The `name` field is a string
+representing the type of note. The `desc` field is a structure that contains the
 actual data of the note. The `type` field tells us what type of note we are
 looking at. It is an unsigned integer that represents the type of note. It's
 worth noting that the `name` field works as a kind of namespace for the type
 field. Two notes with the same type field can be differentiated by their name
 field.
 
 The `PT_LOAD` segment is a bit more straightforward. This represents a segment
-of memory that was loaded into the process at the time of crash. These can
+of memory that was loaded into the process at the time of the crash. These can
 represent either the stack, heap, or any other segment of memory that was loaded
 into the process.
 
@@ -235,20 +248,20 @@ offering on MCU and Android, we needed a few basic things:
 
 Based on what we've learned about Linux core files so far, they are an obvious
 fit for these requirements. We can use an established system to route
-information about crashed processes, add metadata that helps gives us
-information the device in question, and do all of this without making any source
-modifications to anything running on the system. For this reason our first pass
-at coredumps leave them largely untouched from what the kernel provides. The
-only addition is a note that contains the metadata we use to identify devices
-and the version of software they're running on. This takes advantage of the fact
-that the `PT_NOTE` segment is a free form segment that can be used to add any
-metadata we want to the coredump.
+information about crashed processes, add metadata that helps give us information
+about the device in question, and do all of this without making any source
+modifications to anything running on the system. For this reason, our first pass
+at coredumps leaves them largely untouched compared to what the kernel provides.
+The only addition is a note that contains the metadata we use to identify
+devices and the version of software they're running on. This takes advantage of
+the fact that the `PT_NOTE` segment is a free-form segment that can be used to
+add any metadata we want to the coredump.
 
 This allows us to gather additional information about the process that crashed,
 and more easily stream memory to avoid unnecessary allocations or memory usage.
 
-Now that we've covered all the background information we can start to dive into
-the innards of the `memfault-core-handler`. First we use the pipe operation that
+Now that we've covered all the background information, we can dive into the
+innards of the `memfault-core-handler`. First, we use the pipe operation that
 was outlined earlier.
 [Here](https://github.com/memfault/memfault-linux-sdk/blob/49adfe0ce0cb6082360012b0f0092a31e8030048/meta-memfault/recipes-memfault/memfaultd/files/memfaultd/src/coredump/mod.rs#L14)
 is the pattern we write to `/proc/sys/kernel/core_pattern` to pipe the coredump
@@ -258,27 +271,27 @@ to our handler:
 |/usr/sbin/memfault-core-handler -c /path/to/config %P %e %I %s
 ```
 
-This tells the kernel to pipe the coredump to our handler, and provides the
+This tells the kernel to pipe the coredump to our handler and provides the
 handler with the PID of the crashing process (`%P`), the name of the crashing
-process (%e), the UID of the crashing process (`%I`), and the signal that caused
-the crash (`%s`).
+process (`%e`), the UID of the crashing process (`%I`), and the signal that
+caused the crash (`%s`).
 
-When a crash occurs the kernel will write the coredump to the `stdin` of the
+When a crash occurs, the kernel will write the coredump to the `stdin` of the
 handler. The handler will then read all the program headers into memory. This
-sets us up to do two things. First we'll read all of the `PT_NOTE` segments and
+sets us up to do two things. First, we'll read all of the `PT_NOTE` segments and
 save them in memory. For the first iteration of the handler, we won't do
 anything further with them until we write them to a file. They'll become more
 important in later articles as we get into more of the special sauce of the
 handler.
 
 The next thing the handler does is read all of the memory ranges for each
-`PT_LOAD` segment in the coredump. Instead of storing this in memory we'll
-stream it directly to the output file from `/proc/<pid>/mem`. This is done to
-reduce the memory footprint of the handler, and prevent any issues where we
-would potentially need to seek backwards in the stream. As mentioned before,
-`stdin` is a one way stream, and we can't seek backwards in it.
+`PT_LOAD` segment in the coredump. Instead of storing this in memory, we'll
+stream it directly to the output file from `/proc/<pid>/mem`. We do this to
+reduce the memory footprint of the handler and prevent any issues where we would
+potentially need to seek backwards in the stream. As mentioned before, `stdin`
+is a one way stream, and we can't seek backwards in it.
 
-After we've written all of the `PT_LOAD` segments to the output file we should
+After we've written all of the `PT_LOAD` segments to the output file, we should
 have an ELF coredump that is largely the same as what the kernel would have
 written. The only difference is that we've added a note to the coredump, the
 contents of which we won't cover in this article, as it's not particularly
@@ -290,25 +303,25 @@ our previous ELF layout diagram with the changes we've made.
 ![]({% img_url linux-coredump/elf-core-layout-annotated.png %})
 
 And there we have it! We've copied our coredump over from `stdin` with a few
-minor changes. Now you're probably wondering, why did we go through all of this
+minor changes. Now, you're probably wondering: why did we go through all of this
 trouble to end up with a file that's largely the same as what the kernel would
-have produced? Well for one it allows us to add metadata to the coredump, but it
-also sets the stage for more advanced coredump handling in the future that we'll
-cover in the the next article.
+have produced? Well, for one, it allows us to add metadata to the coredump, but
+it also sets the stage for more advanced coredump handling in the future that
+we'll cover in the next article.
 
 ## Conclusion
 
-We've covered the basics of coredumps on Linux, and how they're used in the
+We've covered the basics of coredumps on Linux and how they're used in the
 Memfault SDK. You should now have a pretty good idea of how things look under
-the hood. While the baseline coredumps are useful, and a known commodity, there
+the hood. While the baseline coredumps are useful and a known commodity, there
 are a few things that aren't great about them. The biggest issue is that they
-can be quite large for processes that have many threads, or do a large amount of
-memory allocation. This can be a large problem for embedded devices that may not
-have a lot of room to store large files. In the next article we'll take a look
-at the steps we've taken to reduce the size of coredumps.
+can be quite large for processes with many threads or do a large amount of
+memory allocation. This can be a significant problem for embedded devices that
+may not have a lot of room to store large files. In the next article, we'll take
+a look at the steps we've taken to reduce the size of coredumps.
 
 In the meantime, if you'd like to poke around the source code for the coredump
-handler you can find it
+handler, you can find it
 [here](https://github.com/memfault/memfaultd/tree/main/memfaultd/src/cli/memfault_core_handler).
 
 <!-- Interrupt Keep START -->