new container service is destructive and mostly read-only

Author: lexming

jupyterhub/README.md

@@ -12,7 +12,7 @@ the resource manager [Slurm](https://slurm.schedmd.com/). Users can select the
 resources for their notebooks from the JupyterHub interface thanks to the
 [JupyterHub MOdular Slurm Spawner](https://github.com/silx-kit/jupyterhub_moss),
 which leverages [batchspawner](https://github.com/jupyterhub/batchspawner) to
-submit jobs to Slurm in user's behalf that will launch the single-user server.
+submit jobs to Slurm in user's behalf to launch the single-user server.
 
 The main particularity of our setup is that such jobs are not submitted to
 Slurm from the host running JupyterHub, but from the login nodes of the HPC
@@ -24,21 +24,23 @@ installation capable of submitting jobs to the HPC cluster.
 ## Rootless
 
 JupyterHub is run by a non-root user in a rootless container. Setting up a
-rootless container is well described in the [podman rootless
-tutorial](https://github.com/containers/podman/blob/main/docs/tutorials/rootless_tutorial.md).
-
-We use a [system service](host/etc/systemd/system/jupyterhub.service) to
-execute `podman` by a non-root user `jupyterhub` (*aka* JupyterHub operator).
-This service relies on a [custom shell script](host/usr/local/bin/jupyterhub-init.sh)
-to automatically initialize a new image of the rootless container or start an
-existing one.
-
-The container [binds a few mounts with sensitive configuration
-files](host/usr/local/bin/jupyterhub-init.sh#L59-L66) for JupyterHub, SSL
-certificates for the web server and SSH keys to connect to the login nodes.
-Provisioning these files in the container through bind-mounts allows to have
-secret-free container images and seamlessly deploy updates to the configuration
-of the hub.
+rootless container is well described in the
+[podman rootless tutorial](https://github.com/containers/podman/blob/main/docs/tutorials/rootless_tutorial.md).
+
+We use a [custom system service](container_host/etc/systemd/system/container-jupyterhub.service)
+to start the container with `podman` by the non-root user `jupyterhub` (*aka*
+JupyterHub operator).
+This service regenerates at (re)start any running container with a new one from
+the container image. This approach ensures a clean state of the container and
+allows to easily recover from any runtime issues with it.
+
+The root filesystem in the container is read-only. The only writable space is
+the home directory of the non-root user running the container. We also [bind a
+few read-only mounts](container_host/etc/systemd/system/container-jupyterhub.service#L38)
+with sensitive configuration files for JupyterHub, SSL certificates for the web
+server and SSH keys to connect to the login nodes. Provisioning these files in
+the container through bind-mounts allows to have secret-free container images
+and seamlessly deploy updates to the configuration of the hub.
 
 ## Network
 
@@ -70,34 +72,34 @@ from JupyterHub:
     * [URLs of the VSC OAuth](container/Dockerfile#L72-L76) are defined in the
       environment of the container
 
-    * [OAuth secrets](container/.config/jupyterhub_config.py#L40-L45) are
+    * [OAuth secrets](container/.config/jupyterhub_config.py#L43-L48) are
       defined in JupyterHub's configuration file
 
 * local users beyond the non-root user running JupyterHub are **not needed**
 
 ## Slurm
 
 Integration with Slurm is leveraged through a custom Spawner called
-[VSCSlurmSpawner](container/.config/jupyterhub_config.py#L60) based on
+[VSCSlurmSpawner](container/.config/jupyterhub_config.py#L63) based on
 [MOSlurmSpawner](https://github.com/silx-kit/jupyterhub_moss).
 `VSCSlurmSpawner` allows JupyterHub to generate the user's environment needed
 to spawn its single-user server without any local users. All user settings are
 taken from `vsc-config`.
 
-We modified the [submission command](container/.config/jupyterhub_config.py#L295)
+We modified the [submission command](container/.config/jupyterhub_config.py#L317)
 to execute `sbatch` in the login nodes of the HPC cluster through SSH.
 The login nodes already run Slurm and are the sole systems handling job
 submission in our cluster. Delegating job submission to them avoids having to
 install and configure Slurm in the container running JupyterHub. The hub
 environment is passed over SSH with a strict control over the variables that
-are [sent](container/.ssh/config) and [accepted](slurm_login/etc/ssh/sshd_config)
+are [sent](container/.ssh/config) and [accepted](slurm_host/etc/ssh/sshd_config)
 on both ends.
 
 The SSH connection is established by the non-root user running JupyterHub (the
 hub container does not have other local users). This jupyterhub user has
 special `sudo` permissions on the login nodes to submit jobs to Slurm as other
 users. The specific group of users and list of commands allowed to the
-jupyterhub user are defined in the [sudoers file](slurm_login/etc/sudoers).
+jupyterhub user are defined in the [sudoers file](slurm_host/etc/sudoers).
 
 Single-user server spawn process:
 
@@ -114,7 +116,7 @@ Single-user server spawn process:
    hub environment
 
 5. single-user server job script fully [resets the
-   environment](container/.config/jupyterhub_config.py#L264-L285) before any
+   environment](container/.config/jupyterhub_config.py#L286-L312) before any
    other step is taken to minimize tampering from user's own environment
 
 6. single-user server is launched **without** the mediation of `srun` to be