Debug/Troubleshooting Bootstrap containers

[cyrus-mc]

I am trying to use a bootstrap container to setup an ephemeral disk to use for container images (as discussed here: #1991).

Modified the setup script a little as the one shown in the discussion above is based on using an instance store:

#!/usr/bin/env bash
set -ex

ls -al /.bottlerocket/rootfs/dev > /.bottlerocket/rootfs/tmp/bootstrap.output

# the name of the disk we want to manage
DISK=/.bottlerocket/rootfs/dev/nvme2n1

PARTITIONS_CREATED=/.bottlerocket/bootstrap-containers/current/created

# mounts from this mount point will propagate across mount namespaces
BASE_MOUNT_POINT=/.bottlerocket/rootfs/mnt

RUNTIME_ROOT_PATH=/.bottlerocket/rootfs/var/lib/containerd

ls -al $PARTITIONS_CREATED >> /.bottlerocket/rootfs/tmp/bootstrap.output
if [ ! -f $PARTITIONS_CREATED ]; then
  echo "here" >> /.bottlerocket/rootfs/tmp/bootstrap.output
  parted -s $DISK mklabel gpt 1 > /dev/null
  parted -s $DISK mkpart primary ext4 0% 100% 1 > /dev/null
  mkfs.ext4 -F ${DISK}p1

  touch $PARTITIONS_CREATED
  ls -al $PARTITIONS_CREATED >> /.bottlerocket/rootfs/tmp/bootstrap.output
fi

mkdir -p $BASE_MOUNT_POINT/containerd
mount ${DISK}p1 $BASE_MOUNT_POINT/containerd

# If the container's runtime root directory exists, you have to manually delete it
if [ ! -d ${RUNTIME_ROOT_PATH} ]; then
  # Use /mnt since the symlink will be seen from the host's perspective
  ln -snf /mnt/containerd ${RUNTIME_ROOT_PATH}
fi

But the above isn't working. Even the output statements to tmp directory does not work (file bootstrap.output is not created).

Is there a method for troubleshooting bootstrap containers.

[arnaldo2792]

You should be able to see the output of your bootstrap container in the journal. Also, instead of writing to /.bottlerocket/rootfs/tmp try to write to /.bottlerocket/bootstrap-containers/current since tmp might not be accessible by the SELinux label used in bootstrap containers.

[cyrus-mc]

Thanks. That solved it. My issue was a combination of issues with not specifying the correct source URL (runtime doesn't assume default pulls are from docker.io) as well as cross platform issues (running M1 so had to ensure I built my bootstrap container for amd64).

[Riyan-plivo]

when i am running mount command manually on the instance , disk is getting mounted on top of the directory but when i am passing it through a bootstrap container it is not getting mounted in bottlerocket

[arnaldo2792]

Hey @Riyan-plivo, are you having problems with mounting ephemeral storage to be used by the container runtime? Or with mounting any mount from a Bootstrap Container?

For the former, have you considered using Bootstrap Commands? And for the latter, keep in mind that mounts created from bootstrap containers propagate to the host only they were created at /.botterocket/rootfs/mnt from within the bootstrap container