feat(erofs): initial commit for erofs support

[rchincha]

Fixes opencontainers/image-spec#1190

What type of PR is this?

Which issue does this PR fix:

What does this PR do / Why do we need it:

If an issue # is not available please add repro steps and logs showing the issue:

Testing done on this change:

Automation added to e2e:

Will this break upgrades or downgrades?

Does this PR introduce any user-facing change?:

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[hsiangkao]

Hi @rchincha, I'm currently working on incremental builds and I plan to release erofs-utils v1.8 this month (with multi-threaded mkfs, incremental builds, Intel QPL, etc.), but anyway, I'd suggest use cgo for initial EROFS support (with formal liberofs APIs exported) for now (or even binary integration as the very first step) since this go implemention seems somewhat incomplete (and maybe even broken.. I don't have enough time to look into that since other prioritied stuffs are on hands..)

[rchincha]

@hsiangkao I suspect that cgo path may be what ends up happening. Currently, just prototyping quickly to understand the interfaces and scope of changes/work.

[hsiangkao]

One thing I might need to mention here is that erofs supports external blobs or chunks since Linux 5.16 compared to Squashfs because Dragonfly Nydus once asked this feature to deduplicate data in chunks among different container images.

I'm not sure if it's worthwhile to highlight this, you could just make a tiny metadata with external blobs (which can be used for multiple images) for reference.
Currently only Nydus has userspace tools to generate chunk blobs, if that is interested in other use cases, I could seek more time to implement this in mkfs.erofs too.

[rchincha]

$ cat stacker.yaml

build:
  from:
    type: docker
    url: docker://public.ecr.aws/docker/library/busybox:1.37-glibc

 $ /tmp/stacker  build -layer-type erofs
Creating new OCI Layout at "/data/hdd/rchincha/tmp/lxc/oci"
preparing image build...
imported file hashes (after substitutions):
loading docker://public.ecr.aws/docker/library/busybox:1.37-glibc
Copying blob a46fbb00284b done
Copying config 27a71e19c9 done
Writing manifest to image destination
Storing signatures
mkfs.erofs 1.4                                                                                                                                                                    c_version:           [     1.4]
        c_dbg_lvl:           [       2]
        c_dry_run:           [       0]
filesystem build built successfully

[rchincha]

@hsiangkao an update. Now have a stacker (this PR) that can build OCI images with erofs+dm-verity layers - no change to the overall workflow.

[codecov]

Codecov Report

Attention: Patch coverage is 69.44444% with 11 lines in your changes missing coverage. Please review.

Project coverage is 60.44%. Comparing base (c34ba3a) to head (96e1d27).
Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
pkg/types/layer_type.go	52.63%	9 Missing ⚠️
pkg/overlay/pack.go	66.66%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #626      +/-   ##
==========================================
- Coverage   60.45%   60.44%   -0.02%     
==========================================
  Files          59       59              
  Lines        6446     6457      +11     
==========================================
+ Hits         3897     3903       +6     
- Misses       1903     1908       +5     
  Partials      646      646              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[hsiangkao]

Hi!

@hsiangkao

Is there an equivalent of "unsquashfs" in erofs-utils?

https://manpages.debian.org/testing/squashfs-tools/unsquashfs.1.en.html

SYNOPSIS

unsquashfs [OPTIONS] FILESYSTEM [files to extract or exclude (with -excludes) or cat (with -cat )]

fsck.erofs --extract=<dir> FILESYSTEM to extract the whole filesystem...

Currently, there is no exclude way but it can be added later.

[hsiangkao]

Hi @rchincha, currently azure kernels are somewhat fragmented on kconfig, I've raised issues to ubuntu in order to resolve them, but maybe it still need some time to find proper github runner images in github:
https://bugs.launchpad.net/ubuntu/+source/linux-azure/+bug/2081613
https://bugs.launchpad.net/ubuntu/+source/linux-azure/+bug/2098178

I discussed with ubuntu developpers by email months ago, they suggested: ".. until that lands, you can work around this by installing the linux-modules-extra-azure package in your CI runners. "

It may still cause some inconvenience but will fix with the future release linux-azure release versions.

[raharper]

Approved.

Started a cleanup request issue: #674

[rchincha]

@hsiangkao v1.1.0-rc1 is now released with erofs support

https://github.com/project-stacker/stacker/releases/download/v1.1.0-rc1/stacker

stacker build --layer-type=erofs -f stacker.yaml

[hsiangkao]

@hsiangkao v1.1.0-rc1 is now released with erofs support

https://github.com/project-stacker/stacker/releases/download/v1.1.0-rc1/stacker

stacker build --layer-type=erofs -f stacker.yaml

That is a great news. currently there is an ongoing native erofs library, but it might need more time to get the full functionality.