From 7e33f8885b140da3cfdf315395e6326a89c38d39 Mon Sep 17 00:00:00 2001 From: Terry Howe Date: Wed, 22 Jan 2025 09:13:37 -0700 Subject: [PATCH 1/6] docs: backup and restore specification Signed-off-by: Terry Howe --- docs/proposals/backup-restore.md | 107 +++++++++++++++++++++++++++++++ 1 file changed, 107 insertions(+) create mode 100644 docs/proposals/backup-restore.md diff --git a/docs/proposals/backup-restore.md b/docs/proposals/backup-restore.md new file mode 100644 index 000000000..e5347eeac --- /dev/null +++ b/docs/proposals/backup-restore.md @@ -0,0 +1,107 @@ +# Backup and Restore Commands + +This document describes how the backup and restore commands will work and how the feature will be rolled out. +The proposal is to create the feature as a minimum viable product and follow that up with non breaking features to the commands. +The initial implementation will be the easiest implementation that is useful. + +## Minium Viable Product + +The backup command will initially just support writing to a directory and the restore command will only support reading from a directory. + +### oras backup + +The backup command will initially only support reading a list of files from the command line and writing to a directory: + +```bash +oras backup --output ./mirror registry.k8s.io/kube-apiserver-arm64:v1.31.0 registry.k8s.io/kube-controller-manager-arm64:v1.31.0 +``` + +The generated directory structure is `/`. +The command above puts the OCI layout for the Kubernetes API server in `mirror/kube-api-server-arm64`. +The directory structure with intermediate blobs removed: + +```bash +$ find mirror +mirror +mirror/kube-apiserver-arm64 +mirror/kube-apiserver-arm64 +mirror/kube-apiserver-arm64/ingest +mirror/kube-apiserver-arm64/oci-layout +mirror/kube-apiserver-arm64/blobs +mirror/kube-apiserver-arm64/blobs/sha256 +mirror/kube-apiserver-arm64/blobs/sha256/3f4e2c5863480125882d92060440a5250766bce764fee10acdbac18c872e4dc7 +... +mirror/kube-apiserver-arm64/blobs/sha256/4f80fb2b9442dbecd41e68b598533dcaaf58f9d45cce2e03a715499aa9f6b676 +mirror/kube-apiserver-arm64/index.json +mirror/kube-controller-manager-arm64 +mirror/kube-controller-manager-arm64 +mirror/kube-controller-manager-arm64/ingest +mirror/kube-controller-manager-arm64/oci-layout +mirror/kube-controller-manager-arm64/blobs +mirror/kube-controller-manager-arm64/blobs/sha256 +mirror/kube-controller-manager-arm64/blobs/sha256/3f4e2c5863480125882d92060440a5250766bce764fee10acdbac18c872e4dc7 +... +mirror/kube-controller-manager-arm64/blobs/sha256/4f80fb2b9442dbecd41e68b598533dcaaf58f9d45cce2e03a715499aa9f6b676 +mirror/kube-controller-manager-arm64/index.json +$ +``` + +The directory `mirror` must exist for the initial feature. +Each image will be stored in a subdirectory which matches the repository name. + +### oras restore + +Initially, the restore command will only support reading a directory and writing to a registry: + +```bash +oras restore --input ./mirror localhost:15000/my-mirror +``` + +The above backup example would result in: +```bash +localhost:15000/my-mirror/kube-apiserver-arm64:v1.31.0 +localhost:15000/my-mirror/kube-controller-manager-arm64:v1.31.0 +``` + +A namespace in the registry will be optional. +The registry in the above example could be specified as `localhost:15000`. + + +Any directory in the input that does not contain an `index.json` shall be silently ignored. + +## Compressed Tar File Support + +After the command supports directory write and read, support will be added for write and read to a compressed tar file. +The directory structure in the tar file would be the same as in the directory output. + +```bash +oras backup --output ./mirror.tgz registry.k8s.io/kube-apiserver-arm64:v1.31.0 registry.k8s.io/kube-controller-manager-arm64:v1.31.0 +``` + +There will be no validation on file name. +The file name does not need to end in `tgz`, but the format will be compressed tar file. +If the file exists, it will be overwritten. + +```bash +oras restore --input ./mirror.tgz localhost:15000/my-mirror +``` +If the specified source is a file, the format is assumed to be a compressed tar file. + +## Backup file input from a file + +The backup command will support an `--input filename` argument which will be a file. +The format of the contents of the file is a list of images names separated by newlines. +If the `--input` argument is specified, no images may be specified on the command line. + +## Backup file input from standard input + +The backup command will support the `--input -` argument to read a list of images from standard input. +The format of the input is a list of images names separated by newlines. + +## Restore file input from standard input + +The restore command will support the `--input -` argument to read a compressed tar input from standard input. + +## Optimize blobs + +Add the feature to create blobs that are duplicated between images as hard links in the directory output and compressed tar files. From f4391e4bd0a730b7c44c11156629c3863058b168 Mon Sep 17 00:00:00 2001 From: Terry Howe Date: Thu, 6 Feb 2025 10:37:01 -0700 Subject: [PATCH 2/6] Update docs/proposals/backup-restore.md Co-authored-by: Andrew Block Signed-off-by: Terry Howe --- docs/proposals/backup-restore.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/proposals/backup-restore.md b/docs/proposals/backup-restore.md index e5347eeac..5d7f0031e 100644 --- a/docs/proposals/backup-restore.md +++ b/docs/proposals/backup-restore.md @@ -4,7 +4,7 @@ This document describes how the backup and restore commands will work and how th The proposal is to create the feature as a minimum viable product and follow that up with non breaking features to the commands. The initial implementation will be the easiest implementation that is useful. -## Minium Viable Product +## Minimum Viable Product The backup command will initially just support writing to a directory and the restore command will only support reading from a directory. From aedfdaf1e9010cf8992bb4faf72be679523e34b8 Mon Sep 17 00:00:00 2001 From: Terry Howe Date: Thu, 6 Feb 2025 10:38:33 -0700 Subject: [PATCH 3/6] Update docs/proposals/backup-restore.md Co-authored-by: Andrew Block Signed-off-by: Terry Howe --- docs/proposals/backup-restore.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/proposals/backup-restore.md b/docs/proposals/backup-restore.md index 5d7f0031e..9ee7f6985 100644 --- a/docs/proposals/backup-restore.md +++ b/docs/proposals/backup-restore.md @@ -104,4 +104,4 @@ The restore command will support the `--input -` argument to read a compressed t ## Optimize blobs -Add the feature to create blobs that are duplicated between images as hard links in the directory output and compressed tar files. +A further enhancement is to create blobs that are duplicated between images as hard links in the directory output and compressed tar files. From d22a47bd6007999485693934d71fdf560d8f020d Mon Sep 17 00:00:00 2001 From: Terry Howe Date: Thu, 6 Feb 2025 12:43:42 -0700 Subject: [PATCH 4/6] Update docs/proposals/backup-restore.md Co-authored-by: Andrew Block Signed-off-by: Terry Howe --- docs/proposals/backup-restore.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/proposals/backup-restore.md b/docs/proposals/backup-restore.md index 9ee7f6985..9e6cf0ed2 100644 --- a/docs/proposals/backup-restore.md +++ b/docs/proposals/backup-restore.md @@ -51,7 +51,7 @@ Each image will be stored in a subdirectory which matches the repository name. ### oras restore -Initially, the restore command will only support reading a directory and writing to a registry: +Initially, the restore command will support reading a directory and writing the content to a remote registry: ```bash oras restore --input ./mirror localhost:15000/my-mirror From 69870d228411d20fc971afd648c08f6fe60a6647 Mon Sep 17 00:00:00 2001 From: Terry Howe Date: Thu, 6 Feb 2025 12:50:44 -0700 Subject: [PATCH 5/6] Update docs/proposals/backup-restore.md Co-authored-by: Andrew Block Signed-off-by: Terry Howe --- docs/proposals/backup-restore.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/proposals/backup-restore.md b/docs/proposals/backup-restore.md index 9e6cf0ed2..03433c19b 100644 --- a/docs/proposals/backup-restore.md +++ b/docs/proposals/backup-restore.md @@ -89,7 +89,7 @@ If the specified source is a file, the format is assumed to be a compressed tar ## Backup file input from a file -The backup command will support an `--input filename` argument which will be a file. +The backup command will support an `--input filename` argument which will be a file containing the remove resources to retrieve. The format of the contents of the file is a list of images names separated by newlines. If the `--input` argument is specified, no images may be specified on the command line. From 26f3f1afcb07928a3f519ce6fb9bb5bc8eff646f Mon Sep 17 00:00:00 2001 From: Terry Howe Date: Thu, 6 Feb 2025 12:55:03 -0700 Subject: [PATCH 6/6] address document comments Signed-off-by: Terry Howe --- docs/proposals/backup-restore.md | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/docs/proposals/backup-restore.md b/docs/proposals/backup-restore.md index 03433c19b..63ce1f704 100644 --- a/docs/proposals/backup-restore.md +++ b/docs/proposals/backup-restore.md @@ -1,16 +1,20 @@ # Backup and Restore Commands +The backup and restore commands will add the capability to backup a list of artifacts from a registry and restore them to another registry. +The artifacts that ill be supported are any OCI compatible artifact such as container images, helm charts and other artifacts. +This feature will have many use cases, but the use case that will benefit the most is air gapped registries. +These commands will allow copying artifacts to and from air gapped registries over the sneakernet or other limited communication channel. This document describes how the backup and restore commands will work and how the feature will be rolled out. The proposal is to create the feature as a minimum viable product and follow that up with non breaking features to the commands. -The initial implementation will be the easiest implementation that is useful. ## Minimum Viable Product The backup command will initially just support writing to a directory and the restore command will only support reading from a directory. +As well as the flags described here, the commands will support the normal set of flags to support TLS and authentication. ### oras backup -The backup command will initially only support reading a list of files from the command line and writing to a directory: +The backup command will initially only support reading a list of atifacts from the command line and writing to a directory: ```bash oras backup --output ./mirror registry.k8s.io/kube-apiserver-arm64:v1.31.0 registry.k8s.io/kube-controller-manager-arm64:v1.31.0 @@ -89,7 +93,7 @@ If the specified source is a file, the format is assumed to be a compressed tar ## Backup file input from a file -The backup command will support an `--input filename` argument which will be a file containing the remove resources to retrieve. +The backup command will support an `--input filename` argument which will be a file containing the remote resources to retrieve. The format of the contents of the file is a list of images names separated by newlines. If the `--input` argument is specified, no images may be specified on the command line.