Add/remote registry (#576)

* adding support for upgrade
* simplify dryrun logic
* adding upgrade test for remote registry
* rename to sync  registry and base GitHub provider off Filesystem
* preparing for refactor to add default remote registry
* black and various tweaks to logic
* add support for docgen to a specific branch
* testing support for gitlab
* add github/gitlab name check to classes
* adding support to sync from file
* adding support for registries sync file
* Use shutil.copy2 to preserve file metadata
* ensure we do not add non-container-module directories
* adding separate properties for each of the source url and web url
* add more user/developer docs for using registry

this still needs tests written - I think I will need to make an external
remote registry with some arbitrary change to test, OR just test the
interaction with the remote provider and handle the updates via local
test files.

Signed-off-by: vsoch <[email protected]>

Author: vsoch

CHANGELOG.md

@@ -14,6 +14,8 @@ and **Merged pull requests**. Critical items to know are:
 The versions coincide with releases on pip. Only major versions will be released as tags on Github.
 
 ## [0.0.x](https://github.com/singularityhub/singularity-hpc/tree/main) (0.0.x)
+ - Add support for remote registry and sync commands --all (0.1.0)
+  - support for GitLab and GitHub remotes
  - fixing bug to config set/get nested fields, default for wrapper scripts true (0.0.57)
   - change in behavior, instead of `key:value` for set/add `key value` now also works.
  - adding support for container.yaml overrides (0.0.56)

docs/getting_started/developer-guide.rst

@@ -9,10 +9,40 @@ registry entries and building containers. If you haven't read :ref:`getting_star
 you should do that first.
 
 
-Creating a Registry
-===================
+Developer Commands
+==================
+
+Singularity Registry HPC has a few "developer specific" commands that likely will only be
+used in automation, but are provided here for the interested reader.
+
+Docgen
+------
+
+To generate documentation for a registry (e.g., see `this registry example <https://singularityhub.github.io/shpc-registry>`_ we can use docgen. Docgen, by way of needing to interact with the local filesystem,
+currently only supports generation for a filesystem registry. E.g., here is how to generate a registry module
+(from a local container.yaml) that ultimately will be found in GitHub pages:
+
+.. code-block:: console
+
+    $ shpc docgen --registry . --registry-url https://github.com/singularityhub/shpc-registry python
+    
+And you could easily pipe this to a file. Here is how we generate this programatically in a loop:
+
 
-A registry consists of a database of local containers files, which are added
+.. code-block:: console
+
+    for module in $(shpc show --registry ../shpc-registry); do          
+        flatname=${module#/}
+        name=$(echo ${flatname//\//-})
+        echo "Generating docs for $module, _library/$name.md"
+        shpc docgen --registry ../shpc-registry --registry-url https://github.com/singularityhub/shpc-registry $module > "_library/${name}.md"
+    done
+
+
+Creating a FileSystem Registry
+==============================
+
+A fileystem registry consists of a database of local containers files, which are added
 to the module system as executables for your user base. This typically means that you are a
 linux administrator of your cluster, and shpc should be installed for you to use
 (but your users will not be interacting with it).
@@ -88,9 +118,59 @@ It's reasonable that you can store your recipes alongside these files, in the ``
 folder. If you see a conflict and want to request allowing for a custom install path
 for recipes, please open an issue.
 
+Creating a Remote Registry
+==========================
 
-.. _getting_started-writing-registry-entries:
+If you want to create your own remote registry (currently supported to be on GitHub or GitLab)
+the easiest thing to do is start with one of our shpc provided registries as a template:
+
+ - `**GitHub** <https://github.com/singularityhub/shpc-registry>`_
+ - `**GitLab** <https://gitlab.com/singularityhub/shpc-registry>`_
+
+This means (for either) you'll want to clone the original repository:
+
+.. code-block:: console
+
+    $ git clone https://github.com/singularityhub/shpc-registry my-registry
+    $ cd my-registry
+    
+Ensure you do a fetch to get the github pages branch, which deploys the web interface!
+
+.. code-block:: console
+
+    $ git fetch
+
+At this point, you can create an empty repository to push to. If you don't mind
+it being a fork, you can also just fork the original repository (and then pull
+from it instead). GitLab has a feature to fork and then remove the fork, so that
+is an option too. Ensure that you push the gh-pages branch too (for GitHub only):
+
+.. code-block:: console
 
+    $ git checkout gh-pages
+    $ git push origin gh-pages
+
+Once you have your cloned registry repository, it's up to you for how you want
+to delete / edit / add containers! You'll likely use ``shpc add`` to generate new
+configs, and you might want to delete most of the default containers provided.
+Importantly, you should take note of the workflows in the repository. Generally:
+
+ - We have an update workflow (GitHub) that will check for new versions of containers. This still need to be ported to GitLab.
+ - The docs workflow (on GitHub, this is in the .github-ci.yaml) will deploy docs to GitHub/GitLab pages.
+
+For each of GitLab and GitHub, ensure after you deploy that your pages are enabled.
+It helps to ensure the website (static) URL is in the description to be easily find-able.
+Once it's deployed, ensure you see your containers, and clicking the ``</>`` (code)
+icon shows the library.json that shpc will use. Finally, akin to adding a filesystem registry,
+you can just do the same, but specify your remote URL:
+
+.. code-block:: console
+
+    $ shpc config add registry https://github.com/singularityhub/shpc-registry
+
+And that's it!
+
+.. _getting_started-writing-registry-entries:
 
 Writing Registry Entries
 ========================

docs/getting_started/installation.rst

@@ -69,18 +69,16 @@ Once it's installed, you should be able to inspect the client!
 
 
 You'll next want to configure and create your registry, discussed next in :ref:`getting-started`.
-Generally, remember that your modules will be installed in
-the ``modules`` folder, and container recipes are nested in ``registry``. If you don't
-want your container images (sif files) installed alongside your module recipes,
+
+Generally, remember that your modules will be installed in the ``modules`` folder, and container
+recipes will come from the remote registry `shpc-registry <https://github.com/singularityhub/shpc-registry>`_ by default.  If you don't want your container images (sif files) installed alongside your module recipes,
 then you can define ``container_base`` to be somewhere else. You
 can change these easily with ``shpc config``, as they are defined via these
 variables in the config:
 
 
 .. code-block:: console
  
-
-    $ shpc config add registry /<DIR>
     $ shpc config set module_base /<DIR> 
     $ shpc config set container_base  /<DIR>