Skip to content

Commit df7d621

Browse files
authored
Merge pull request #6 from lexming/jh12
update configuration files and documentation for deployment of JupyterHub 3.1
2 parents c6a1463 + ea9e30d commit df7d621

File tree

8 files changed

+158
-221
lines changed

8 files changed

+158
-221
lines changed

jupyterhub/README.md

Lines changed: 24 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,7 @@ the resource manager [Slurm](https://slurm.schedmd.com/). Users can select the
1212
resources for their notebooks from the JupyterHub interface thanks to the
1313
[JupyterHub MOdular Slurm Spawner](https://github.com/silx-kit/jupyterhub_moss),
1414
which leverages [batchspawner](https://github.com/jupyterhub/batchspawner) to
15-
submit jobs to Slurm in user's behalf that will launch the single-user server.
15+
submit jobs to Slurm in user's behalf to launch the single-user server.
1616

1717
The main particularity of our setup is that such jobs are not submitted to
1818
Slurm from the host running JupyterHub, but from the login nodes of the HPC
@@ -24,21 +24,23 @@ installation capable of submitting jobs to the HPC cluster.
2424
## Rootless
2525

2626
JupyterHub is run by a non-root user in a rootless container. Setting up a
27-
rootless container is well described in the [podman rootless
28-
tutorial](https://github.com/containers/podman/blob/main/docs/tutorials/rootless_tutorial.md).
29-
30-
We use a [system service](host/etc/systemd/system/jupyterhub.service) to
31-
execute `podman` by a non-root user `jupyterhub` (*aka* JupyterHub operator).
32-
This service relies on a [custom shell script](host/usr/local/bin/jupyterhub-init.sh)
33-
to automatically initialize a new image of the rootless container or start an
34-
existing one.
35-
36-
The container [binds a few mounts with sensitive configuration
37-
files](host/usr/local/bin/jupyterhub-init.sh#L59-L66) for JupyterHub, SSL
38-
certificates for the web server and SSH keys to connect to the login nodes.
39-
Provisioning these files in the container through bind-mounts allows to have
40-
secret-free container images and seamlessly deploy updates to the configuration
41-
of the hub.
27+
rootless container is well described in the
28+
[podman rootless tutorial](https://github.com/containers/podman/blob/main/docs/tutorials/rootless_tutorial.md).
29+
30+
We use a [custom system service](container_host/etc/systemd/system/container-jupyterhub.service)
31+
to start the container with `podman` by the non-root user `jupyterhub` (*aka*
32+
JupyterHub operator).
33+
This service regenerates at (re)start any running container with a new one from
34+
the container image. This approach ensures a clean state of the container and
35+
allows to easily recover from any runtime issues with it.
36+
37+
The root filesystem in the container is read-only. The only writable space is
38+
the home directory of the non-root user running the container. We also [bind a
39+
few read-only mounts](container_host/etc/systemd/system/container-jupyterhub.service#L38)
40+
with sensitive configuration files for JupyterHub, SSL certificates for the web
41+
server and SSH keys to connect to the login nodes. Provisioning these files in
42+
the container through bind-mounts allows to have secret-free container images
43+
and seamlessly deploy updates to the configuration of the hub.
4244

4345
## Network
4446

@@ -70,34 +72,34 @@ from JupyterHub:
7072
* [URLs of the VSC OAuth](container/Dockerfile#L72-L76) are defined in the
7173
environment of the container
7274

73-
* [OAuth secrets](container/.config/jupyterhub_config.py#L40-L45) are
75+
* [OAuth secrets](container/.config/jupyterhub_config.py#L43-L48) are
7476
defined in JupyterHub's configuration file
7577

7678
* local users beyond the non-root user running JupyterHub are **not needed**
7779

7880
## Slurm
7981

8082
Integration with Slurm is leveraged through a custom Spawner called
81-
[VSCSlurmSpawner](container/.config/jupyterhub_config.py#L60) based on
83+
[VSCSlurmSpawner](container/.config/jupyterhub_config.py#L63) based on
8284
[MOSlurmSpawner](https://github.com/silx-kit/jupyterhub_moss).
8385
`VSCSlurmSpawner` allows JupyterHub to generate the user's environment needed
8486
to spawn its single-user server without any local users. All user settings are
8587
taken from `vsc-config`.
8688

87-
We modified the [submission command](container/.config/jupyterhub_config.py#L295)
89+
We modified the [submission command](container/.config/jupyterhub_config.py#L317)
8890
to execute `sbatch` in the login nodes of the HPC cluster through SSH.
8991
The login nodes already run Slurm and are the sole systems handling job
9092
submission in our cluster. Delegating job submission to them avoids having to
9193
install and configure Slurm in the container running JupyterHub. The hub
9294
environment is passed over SSH with a strict control over the variables that
93-
are [sent](container/.ssh/config) and [accepted](slurm_login/etc/ssh/sshd_config)
95+
are [sent](container/.ssh/config) and [accepted](slurm_host/etc/ssh/sshd_config)
9496
on both ends.
9597

9698
The SSH connection is established by the non-root user running JupyterHub (the
9799
hub container does not have other local users). This jupyterhub user has
98100
special `sudo` permissions on the login nodes to submit jobs to Slurm as other
99101
users. The specific group of users and list of commands allowed to the
100-
jupyterhub user are defined in the [sudoers file](slurm_login/etc/sudoers).
102+
jupyterhub user are defined in the [sudoers file](slurm_host/etc/sudoers).
101103

102104
Single-user server spawn process:
103105

@@ -114,7 +116,7 @@ Single-user server spawn process:
114116
hub environment
115117

116118
5. single-user server job script fully [resets the
117-
environment](container/.config/jupyterhub_config.py#L264-L285) before any
119+
environment](container/.config/jupyterhub_config.py#L286-L312) before any
118120
other step is taken to minimize tampering from user's own environment
119121

120122
6. single-user server is launched **without** the mediation of `srun` to be

jupyterhub/container/.config/jupyterhub_config.py

Lines changed: 73 additions & 45 deletions
Original file line numberDiff line numberDiff line change
@@ -20,12 +20,15 @@
2020
#
2121
#------------------------------------------------------------------------------
2222
# Network configuration
23+
# - listen on all interfaces: proxy is in localhost, users are external and
24+
# spawners are internal
2325
#------------------------------------------------------------------------------
24-
# Listen on all interfaces
25-
# proxy is in localhost, users are external and spawners are internal
26+
# Public facing proxy
2627
c.JupyterHub.bind_url = 'https://0.0.0.0:8000'
28+
c.JupyterHub.port = 8000
29+
# Internal hub
2730
c.JupyterHub.hub_ip = '0.0.0.0'
28-
# IP address or hostname that spawners should use to connect to the Hub API
31+
c.JupyterHub.hub_port = 8081
2932
c.JupyterHub.hub_connect_ip = 'jupyterhub.internal.domain'
3033

3134
#------------------------------------------------------------------------------
@@ -95,6 +98,8 @@ def user_env(self, env):
9598
# - SSH connection stablished as JupyterHub operator
9699
# - define job script parameters and commands launching the notebook
97100
#------------------------------------------------------------------------------
101+
JHUB_VER = "3.1.1"
102+
98103
set_config(c)
99104
c.JupyterHub.spawner_class = VSCSlurmSpawner
100105
c.Spawner.start_timeout = 600 # seconds from job submit to job start
@@ -104,98 +109,119 @@ def user_env(self, env):
104109
vub_lab_environments = {
105110
"2022_default": {
106111
# Text displayed for this environment select option
107-
"description": "2022a: Python v3.10.4 + kernels (default)",
112+
"description": "2022a Default: minimal with all modules available",
108113
# Space separated list of modules to be loaded
109-
"modules": "JupyterHub/2.3.1-GCCcore-11.3.0",
114+
"modules": f"JupyterHub/{JHUB_VER}-GCCcore-11.3.0",
110115
# Path to Python environment bin/ used to start jupyter on the Slurm nodes
111116
"path": "",
112117
# Toggle adding the environment to shell PATH (default: True)
113118
"add_to_path": False,
119+
"group": "Python v3.10.4",
120+
},
121+
"2022_scipy": {
122+
"description": "2022a DataScience: SciPy-bundle + matplotlib + dask",
123+
"modules": (
124+
f"JupyterHub/{JHUB_VER}-GCCcore-11.3.0 "
125+
"SciPy-bundle/2022.05-foss-2022a "
126+
"ipympl/0.9.3-foss-2022a "
127+
"dask-labextension/6.0.0-foss-2022a "
128+
),
129+
"path": "",
130+
"add_to_path": False,
131+
"group": "Python v3.10.4",
132+
},
133+
"2022_nglview": {
134+
"description": "2022a Molecules: DataScience + nglview + 3Dmol",
135+
"modules": (
136+
f"JupyterHub/{JHUB_VER}-GCCcore-11.3.0 "
137+
"SciPy-bundle/2022.05-foss-2022a "
138+
"ipympl/0.9.3-foss-2022a "
139+
"dask-labextension/6.0.0-foss-2022a "
140+
"nglview/3.0.3-foss-2022a "
141+
"py3Dmol/2.0.1.post1-GCCcore-11.3.0 "
142+
),
143+
"path": "",
144+
"add_to_path": False,
145+
"group": "Python v3.10.4",
114146
},
115147
"2022_rstudio": {
116-
"description": "2022a: Python v3.10.4 + RStudio",
148+
"description": "2022a RStudio with R v4.2.1",
117149
"modules": (
118-
"JupyterHub/2.3.1-GCCcore-11.3.0 "
150+
f"JupyterHub/{JHUB_VER}-GCCcore-11.3.0 "
119151
"jupyter-rsession-proxy/2.1.0-GCCcore-11.3.0 "
120152
"RStudio-Server/2022.07.2+576-foss-2022a-Java-11-R-4.2.1 "
121153
"IRkernel/1.3.2-foss-2022a-R-4.2.1 "
122154
),
123155
"path": "",
124156
"add_to_path": False,
157+
"group": "Python v3.10.4",
125158
},
126159
"2022_matlab": {
127-
"description": "2022a: Python v3.10.4 + MATLAB",
160+
"description": "2022a MATLAB",
128161
"modules": (
129162
"MATLAB/2022a-r5 "
130-
"JupyterHub/2.3.1-GCCcore-11.3.0 "
163+
f"JupyterHub/{JHUB_VER}-GCCcore-11.3.0 "
131164
"jupyter-matlab-proxy/0.5.0-GCCcore-11.3.0 "
132165
),
133166
"path": "",
134167
"add_to_path": False,
168+
"group": "Python v3.10.4",
135169
},
136-
"2022_dask": {
137-
"description": "2022a: Python v3.10.4 + dask",
138-
"modules": (
139-
"JupyterHub/2.3.1-GCCcore-11.3.0 "
140-
"dask-labextension/6.0.0-foss-2022a "
141-
),
170+
"2021_default": {
171+
"description": "2021a Default: minimal with all modules available",
172+
"modules": "JupyterHub/2.3.1-GCCcore-10.3.0",
142173
"path": "",
143174
"add_to_path": False,
175+
"group": "Python v3.9.5",
144176
},
145-
"2022_nglview": {
146-
"description": "2022a: Python v3.10.4 + nglview",
177+
"2021_scipy": {
178+
"description": "2021a DataScience: SciPy-bundle + matplotlib + dask",
147179
"modules": (
148-
"JupyterHub/2.3.1-GCCcore-11.3.0 "
149-
"nglview/3.0.3-foss-2022a "
180+
f"JupyterHub/{JHUB_VER}-GCCcore-10.3.0 "
181+
"SciPy-bundle/2021.05-foss-2021a "
182+
"ipympl/0.8.8-foss-2021a "
183+
"dask-labextension/5.3.1-foss-2021a "
150184
),
151185
"path": "",
152186
"add_to_path": False,
187+
"group": "Python v3.9.5",
153188
},
154-
"2021_default": {
155-
"description": "2021a: Python v3.9.5 + kernels (default)",
156-
"modules": "JupyterHub/2.3.1-GCCcore-10.3.0",
189+
"2021_nglview": {
190+
"description": "2021a Molecules: DataScience + nglview",
191+
"modules": (
192+
f"JupyterHub/{JHUB_VER}-GCCcore-10.3.0 "
193+
"SciPy-bundle/2021.05-foss-2021a "
194+
"ipympl/0.8.8-foss-2021a "
195+
"dask-labextension/5.3.1-foss-2021a "
196+
"nglview/3.0.3-foss-2021a "
197+
),
157198
"path": "",
158199
"add_to_path": False,
200+
"group": "Python v3.9.5",
159201
},
160202
"2021_rstudio": {
161-
"description": "2021a: Python v3.9.5 + RStudio",
203+
"description": "2021a RStudio with R v4.1.0",
162204
"modules": (
163-
"JupyterHub/2.3.1-GCCcore-10.3.0 "
205+
f"JupyterHub/{JHUB_VER}-GCCcore-10.3.0 "
164206
"jupyter-rsession-proxy/2.1.0-GCCcore-10.3.0 "
165207
"RStudio-Server/1.4.1717-foss-2021a-Java-11-R-4.1.0 "
166208
"IRkernel/1.2-foss-2021a-R-4.1.0 "
167209
),
168210
"path": "",
169211
"add_to_path": False,
212+
"group": "Python v3.9.5",
170213
},
171214
"2021_matlab": {
172-
"description": "2021a: Python v3.9.5 + MATLAB",
215+
"description": "2021a MATLAB",
173216
"modules": (
174217
"MATLAB/2021a "
175-
"JupyterHub/2.3.1-GCCcore-10.3.0 "
218+
f"JupyterHub/{JHUB_VER}-GCCcore-10.3.0 "
176219
"jupyter-matlab-proxy/0.3.4-GCCcore-10.3.0 "
177220
"MATLAB-Kernel/0.17.1-GCCcore-10.3.0 "
178221
),
179222
"path": "",
180223
"add_to_path": False,
181-
},
182-
"2021_dask": {
183-
"description": "2021a: Python v3.9.5 + dask",
184-
"modules": (
185-
"JupyterHub/2.3.1-GCCcore-10.3.0 "
186-
"dask-labextension/5.3.1-foss-2021a "
187-
),
188-
"path": "",
189-
"add_to_path": False,
190-
},
191-
"2021_nglview": {
192-
"description": "2021a: Python v3.9.5 + nglview",
193-
"modules": (
194-
"JupyterHub/2.3.1-GCCcore-10.3.0 "
195-
"nglview/3.0.3-foss-2021a "
196-
),
197-
"path": "",
198-
"add_to_path": False,
224+
"group": "Python v3.9.5",
199225
},
200226
}
201227

@@ -302,7 +328,9 @@ def user_env(self, env):
302328
c.SlurmSpawner.batch_cancel_cmd = "scancel {{job_id}} "
303329
# protect argument quoting in squeque and sinfo sent through SSH
304330
c.SlurmSpawner.batch_query_cmd = r"squeue -h -j {{job_id}} -o \'%T %B\' "
305-
c.MOSlurmSpawner.slurm_info_cmd = r"sinfo -a --noheader -o \'%R %D %C %G %m\'"
331+
c.MOSlurmSpawner.slurm_info_cmd = (
332+
r"sinfo -N -a --noheader -O \'PartitionName,StateCompact,CPUsState,Gres,GresUsed,Memory,Time\'"
333+
)
306334

307335
# directly launch single-user server (without srun) to avoid issues with MPI software
308336
# job environment is already reset before any step starts

jupyterhub/container/Dockerfile

Lines changed: 9 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -20,11 +20,11 @@
2020
#
2121
###
2222
#
23-
# JupyterHub 2.3 + Oauthenticator + batchspawner
23+
# JupyterHub 3.1 + Oauthenticator + batchspawner + jupyterhub_moss
2424
# based on https://github.com/jupyterhub/oauthenticator/blob/main/examples/full/Dockerfile
2525
# JupyterHub run as non-root user
2626

27-
FROM jupyterhub/jupyterhub:2.3
27+
FROM jupyterhub/jupyterhub:3.1
2828

2929
MAINTAINER VUB-HPC <[email protected]>
3030

@@ -44,16 +44,16 @@ RUN python3 -m pip install --upgrade pip
4444
# install Oauthenticator
4545
RUN python3 -m pip install oauthenticator
4646
# install BatchSpawner and Modular Slurm Spawner (vub-hpc fork)
47-
RUN python3 -m pip install https://github.com/vub-hpc/batchspawner/archive/refs/tags/v1.2.1.tar.gz
48-
RUN python3 -m pip install https://github.com/vub-hpc/jupyterhub_moss/archive/refs/tags/v5.5.2.tar.gz
47+
RUN python3 -m pip install https://github.com/vub-hpc/batchspawner/archive/refs/tags/v1.2.2.tar.gz
48+
RUN python3 -m pip install https://github.com/vub-hpc/jupyterhub_moss/archive/refs/tags/v6.2.1.tar.gz
4949
# install vsc-config
50+
ADD vsc-config /opt/vsc-config
5051
RUN python3 -m pip install vsc-base
51-
COPY vsc-config-master.tar.gz /usr/local/src/
52-
RUN python3 -m pip install /usr/local/src/vsc-config-master.tar.gz
52+
RUN python3 -m pip install /opt/vsc-config
5353
# install static resources for theming
54-
COPY vub-hpc-logo-horiz-color.png /usr/local/share/jupyterhub/static/images/
55-
COPY vub-hpc-logo-square-color.png /usr/local/share/jupyterhub/static/images/
56-
COPY vsc-logo.png /usr/local/share/jupyterhub/static/images/
54+
COPY assets/vub-hpc-logo-horiz-color.png /usr/local/share/jupyterhub/static/images/
55+
COPY assets/vub-hpc-logo-square-color.png /usr/local/share/jupyterhub/static/images/
56+
COPY assets/vsc-logo.png /usr/local/share/jupyterhub/static/images/
5757

5858
# --- JupyterHub operator: non-root user ---
5959
# create user with same UID as outside of container
Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,52 @@
1+
# Copyright 2023 Vrije Universiteit Brussel
2+
#
3+
# This file is part of notebook-platform,
4+
# originally created by the HPC team of Vrij Universiteit Brussel (http://hpc.vub.be),
5+
# with support of Vrije Universiteit Brussel (http://www.vub.be),
6+
# the Flemish Supercomputer Centre (VSC) (https://www.vscentrum.be),
7+
# the Flemish Research Foundation (FWO) (http://www.fwo.be/en)
8+
# and the Department of Economy, Science and Innovation (EWI) (http://www.ewi-vlaanderen.be/en).
9+
#
10+
# https://github.com/vub-hpc/notebook-platform
11+
#
12+
# notebook-platform is free software: you can redistribute it and/or modify
13+
# it under the terms of the GNU General Public License v3 as published by
14+
# the Free Software Foundation.
15+
#
16+
# notebook-platform is distributed in the hope that it will be useful,
17+
# but WITHOUT ANY WARRANTY; without even the implied warranty of
18+
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
19+
# GNU General Public License for more details.
20+
#
21+
###
22+
#
23+
# Unit file for a service running a rootless container in Podman
24+
# generated with `podman generate systemd`
25+
# based on:
26+
# - https://www.redhat.com/sysadmin/podman-shareable-systemd-services
27+
# - https://www.redhat.com/sysadmin/podman-run-pods-systemd-services
28+
#
29+
30+
[Unit]
31+
After=network-online.target
32+
Description=Podman rootless container for JupyterHub
33+
RequiresMountsFor=%t
34+
Wants=network-online.target
35+
36+
[Service]
37+
Environment="PODMAN_SYSTEMD_UNIT=%n"
38+
ExecStart=/usr/bin/podman run --cidfile=%t/%n/%n.ctr-id --cgroups=no-conmon --sdnotify=conmon --rm --replace -d --read-only --mount=type=tmpfs,tmpfs-size=128M,destination=/home/jupyterhub,chown -v /home/jupyterhub/.ssh:/home/jupyterhub/.ssh:ro -v /home/jupyterhub/.config:/home/jupyterhub/.config:ro -v /home/jupyterhub/.ssl:/home/jupyterhub/.ssl:ro --log-driver=journald -v /dev/log:/dev/log -p 8000:8000/tcp -p 8081:8081/tcp --userns=keep-id --name=jupyterhub ghcr.io/vub-hpc/azure-pipelines-jupyterhub:latest jupyterhub -f /home/jupyterhub/.config/jupyterhub_config.py
39+
ExecStartPre=/bin/rm -f %t/%n/%n.ctr-id
40+
ExecStop=/usr/bin/podman stop --ignore --cidfile=%t/%n/%n.ctr-id
41+
ExecStopPost=/usr/bin/podman rm -f --ignore --cidfile=%t/%n/%n.ctr-id
42+
NotifyAccess=all
43+
Restart=on-failure
44+
RuntimeDirectory=%n
45+
TimeoutStopSec=70
46+
Type=notify
47+
User=jupyterhub
48+
Group=jupyterhub
49+
50+
[Install]
51+
WantedBy=multi-user.target
52+

0 commit comments

Comments
 (0)