EKS: Baking AMI (pre-pulling container images before instance boot)

[awoimbee]

Hi,
I wanted to share small elements of my journey trying to bake a bottlerocket AMI. I'm posting here because I didn't find any "prior art", if you have a better way, please share !

From my experience, with bottlerocket you're better off using an EBS snapshot of /dev/xvdb rather than using an AMI.

I'm using EKS 1.22, these steps will likely differ in EKS 1.23 (because of dockershim deprecation).
Bottlerocket version: v1.6.X
EDIT: we're now in 2025 (EKS 1.31) and I'm basically still doing the same thing.

The user-data for the ami builder (the instance you'll launch, snapshot the EBS volume, then shutdown):

[settings]
[settings.host-containers.admin]
enabled = true

Launch the instance. Once you're ssh'ed into the admin container, here's how to interact with the local container registry used by kubelet: sudo sheltie ctr -a /run/dockershim.sock -n k8s.io i ls.

Here's how to pull from a private ECR repo:

# install aws cli into the container
sudo yum install unzip -y
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip -q awscliv2.zip
sudo ./aws/install
# get credentials and save them to a config file
printf "[Credentials.\"xxxxxx.dkr.ecr.us-east-1.amazonaws.com\"]\nUsername = 'AWS'\nPassword = '$(aws ecr get-login-password --region us-east-1)'\n" > reg_cfg.toml
# use host-ctr to pull images, it handles private ECR registries
sudo sheltie host-ctr \
        -s /run/dockershim.sock \
        -n k8s.io pull-image \
        --registry-config /proc/${PPID}/root/home/ec2-user/reg_cfg.toml \
        --source MY_IMAGE:MY_TAG

Once you're done, get the volume id of the device /dev/xvdb using aws ec2 describe-instances. Then shutdown the instance and use aws ec2 create-snapshot. You now have an EBS snapshot of the /dev/xvdb device (the one that contains the container images used by bottlerocket) ! Update your EKS worker node lanch template to add something like:

block_device_mappings {
    device_name = "/dev/xvdb"
    ebs {
      snapshot_id = MY_NEWLY_CREATED_SNAPSHOT_ID
    }
  }

[webern]

Thank you for sharing this!

[mamoit]

Dear @awoimbee...
I can't thank you enough!