Migrate start-notebook & start-singleuser to python (#2006)

* Migrate start-notebook.sh to bash

Based on

> Stop using bash, haha 👍

from #1532.

If there's more apetite for this, I'll try to migrate
`start.sh` and `start-singleuser.sh` as well - I think they should
all be merged together. We can remove the `.sh` suffixes for
accuracy, and keep symlinks in so old config still works. Since
the shebang is what is used to launch the correct interpreter,
the `.sh` doesn't matter.

Will help fix #1532,
as I believe all those things are going to be easier to do from
python than bash

* Rename start-notebook.sh to start-notebook

* Cleanup start-notebook a little

* Fix typo

* Migrate start-singleuser as well

* Remove unused import

* Run symlink commands as root

* Combine repetitive RUN commands

* Remove multiple args to env

-u can not be set by shebang, we must set the env var
instead

* Fix conditional inversion

Co-authored-by: Ayaz Salikhov <[email protected]>

* Fix how start-singleuser is exec'd

* Actually call jupyterhub-singleuser in start-singleuser

* Pass through any additional args we get

* Put .py suffix on the start-* scripts

* Add .sh shims for the start-* scripts

* Document start-notebook.sh and start-singleuser.sh

* Partially test start-notebook.sh

* Reflow warning docs

Co-authored-by: Ayaz Salikhov <[email protected]>

---------

Co-authored-by: Ayaz Salikhov <[email protected]>

Author: yuvipanda

README.md

@@ -65,7 +65,7 @@ system when the container exits, but any changes made to the `~/work` directory
 By default, [jupyter's root_dir](https://jupyter-server.readthedocs.io/en/latest/other/full-config.html) is `/home/jovyan`.
 So, new notebooks will be saved there, unless you change the directory in the file browser.
 
-To change the default directory, you will need to specify `ServerApp.root_dir` by adding this line to previous command: `start-notebook.sh --ServerApp.root_dir=/home/jovyan/work`.
+To change the default directory, you will need to specify `ServerApp.root_dir` by adding this line to previous command: `start-notebook.py --ServerApp.root_dir=/home/jovyan/work`.
 ```
 
 ## Contributing

docs/using/common.md

@@ -1,34 +1,34 @@
 # Common Features
 
 Except for `jupyter/docker-stacks-foundation`, a container launched from any Jupyter Docker Stacks image runs a Jupyter Server with the JupyterLab frontend.
-The container does so by executing a `start-notebook.sh` script.
+The container does so by executing a `start-notebook.py` script.
 This script configures the internal container environment and then runs `jupyter lab`, passing any command-line arguments received.
 
 This page describes the options supported by the startup script and how to bypass it to run alternative commands.
 
 ## Jupyter Server Options
 
-You can pass [Jupyter Server options](https://jupyter-server.readthedocs.io/en/latest/operators/public-server.html) to the `start-notebook.sh` script when launching the container.
+You can pass [Jupyter Server options](https://jupyter-server.readthedocs.io/en/latest/operators/public-server.html) to the `start-notebook.py` script when launching the container.
 
 1. For example, to secure the Jupyter Server with a [custom password](https://jupyter-server.readthedocs.io/en/latest/operators/public-server.html#preparing-a-hashed-password)
    hashed using `jupyter_server.auth.passwd()` instead of the default token,
    you can run the following (this hash was generated for the `my-password` password):
 
    ```bash
    docker run -it --rm -p 8888:8888 jupyter/base-notebook \
-       start-notebook.sh --PasswordIdentityProvider.hashed_password='argon2:$argon2id$v=19$m=10240,t=10,p=8$JdAN3fe9J45NvK/EPuGCvA$O/tbxglbwRpOFuBNTYrymAEH6370Q2z+eS1eF4GM6Do'
+       start-notebook.py --PasswordIdentityProvider.hashed_password='argon2:$argon2id$v=19$m=10240,t=10,p=8$JdAN3fe9J45NvK/EPuGCvA$O/tbxglbwRpOFuBNTYrymAEH6370Q2z+eS1eF4GM6Do'
    ```
 
 2. To set the [base URL](https://jupyter-server.readthedocs.io/en/latest/operators/public-server.html#running-the-notebook-with-a-customized-url-prefix) of the Jupyter Server, you can run the following:
 
    ```bash
    docker run  -it --rm -p 8888:8888 jupyter/base-notebook \
-       start-notebook.sh --ServerApp.base_url=/customized/url/prefix/
+       start-notebook.py --ServerApp.base_url=/customized/url/prefix/
    ```
 
 ## Docker Options
 
-You may instruct the `start-notebook.sh` script to customize the container environment before launching the Server.
+You may instruct the `start-notebook.py` script to customize the container environment before launching the Server.
 You do so by passing arguments to the `docker run` command.
 
 ### User-related configurations
@@ -104,7 +104,7 @@ You do so by passing arguments to the `docker run` command.
   You do **not** need this option to allow the user to `conda` or `pip` install additional packages.
   This option is helpful for cases when you wish to give `${NB_USER}` the ability to install OS packages with `apt` or modify other root-owned files in the container.
   You **must** run the container with `--user root` for this option to take effect.
-  (The `start-notebook.sh` script will `su ${NB_USER}` after adding `${NB_USER}` to sudoers.)
+  (The `start-notebook.py` script will `su ${NB_USER}` after adding `${NB_USER}` to sudoers.)
   **You should only enable `sudo` if you trust the user or if the container runs on an isolated host.**
 
 ### Additional runtime configurations
@@ -147,7 +147,7 @@ For example, to mount a host folder containing a `notebook.key` and `notebook.cr
 docker run -it --rm -p 8888:8888 \
     -v /some/host/folder:/etc/ssl/notebook \
     jupyter/base-notebook \
-    start-notebook.sh \
+    start-notebook.py \
     --ServerApp.keyfile=/etc/ssl/notebook/notebook.key \
     --ServerApp.certfile=/etc/ssl/notebook/notebook.crt
 ```
@@ -159,7 +159,7 @@ For example:
 docker run -it --rm -p 8888:8888 \
     -v /some/host/folder/notebook.pem:/etc/ssl/notebook.pem \
     jupyter/base-notebook \
-    start-notebook.sh \
+    start-notebook.py \
     --ServerApp.certfile=/etc/ssl/notebook.pem
 ```
 
@@ -220,7 +220,7 @@ docker run -it --rm \
 
 ### `start.sh`
 
-The `start-notebook.sh` script inherits most of its option handling capability from a more generic `start.sh` script.
+The `start-notebook.py` script inherits most of its option handling capability from a more generic `start.sh` script.
 The `start.sh` script supports all the features described above but allows you to specify an arbitrary command to execute.
 For example, to run the text-based `ipython` console in a container, do the following:
 

docs/using/recipes.md

@@ -375,14 +375,14 @@ Credit: [britishbadger](https://github.com/britishbadger) from [docker-stacks/is
 The default security is very good.
 There are use cases, encouraged by containers, where the jupyter container and the system it runs within lie inside the security boundary.
 It is convenient to launch the server without a password or token in these use cases.
-In this case, you should use the `start-notebook.sh` script to launch the server with no token:
+In this case, you should use the `start-notebook.py` script to launch the server with no token:
 
 For JupyterLab:
 
 ```bash
 docker run -it --rm \
     jupyter/base-notebook \
-    start-notebook.sh --IdentityProvider.token=''
+    start-notebook.py --IdentityProvider.token=''
 ```
 
 For Jupyter Notebook:
@@ -391,7 +391,7 @@ For Jupyter Notebook:
 docker run -it --rm \
     -e DOCKER_STACKS_JUPYTER_CMD=notebook \
     jupyter/base-notebook \
-    start-notebook.sh --IdentityProvider.token=''
+    start-notebook.py --IdentityProvider.token=''
 ```
 
 ## Enable nbclassic-extension spellchecker for markdown (or any other nbclassic-extension)

docs/using/running.md

@@ -69,7 +69,7 @@ Any other changes made in the container will be lost.
 By default, [jupyter's root_dir](https://jupyter-server.readthedocs.io/en/latest/other/full-config.html) is `/home/jovyan`.
 So, new notebooks will be saved there, unless you change the directory in the file browser.
 
-To change the default directory, you will need to specify `ServerApp.root_dir` by adding this line to previous command: `start-notebook.sh --ServerApp.root_dir=/home/jovyan/work`.
+To change the default directory, you will need to specify `ServerApp.root_dir` by adding this line to previous command: `start-notebook.py --ServerApp.root_dir=/home/jovyan/work`.
 ```
 
 ### Example 3

docs/using/selecting.md

@@ -56,10 +56,17 @@ It contains:
 - Everything in `jupyter/docker-stacks-foundation`
 - Minimally functional Server (e.g., no LaTeX support for saving notebooks as PDFs)
 - `notebook`, `jupyterhub` and `jupyterlab` packages
-- A `start-notebook.sh` script as the default command
-- A `start-singleuser.sh` script useful for launching containers in JupyterHub
+- A `start-notebook.py` script as the default command
+- A `start-singleuser.py` script useful for launching containers in JupyterHub
 - Options for a self-signed HTTPS certificate
 
+```{warning}
+`jupyter/base-notebook` also contains `start-notebook.sh` and `start-singleuser.sh` files to maintain backwards compatibility.
+External config that explicitly refers to those files should instead
+update to refer to `start-notebook.py` and `start-singleuser.py`.
+The shim `.sh` files will be removed at some future date.
+```
+
 ### jupyter/minimal-notebook
 
 [Source on GitHub](https://github.com/jupyter/docker-stacks/tree/main/images/minimal-notebook) |

examples/docker-compose/notebook/letsencrypt-notebook.yml

@@ -18,7 +18,7 @@ services:
       USE_HTTPS: "yes"
       PASSWORD: ${PASSWORD}
     command: >
-      start-notebook.sh
+      start-notebook.py
       --ServerApp.certfile=/etc/letsencrypt/fullchain.pem
       --ServerApp.keyfile=/etc/letsencrypt/privkey.pem
 

examples/make-deploy/Makefile

@@ -13,7 +13,7 @@ define RUN_NOTEBOOK
 	--name $(NAME) \
 	-v $(WORK_VOLUME):/home/jovyan/work \
 	$(DOCKER_ARGS) \
-	$(IMAGE) bash -c "$(PRE_CMD) chown jovyan /home/jovyan/work && start-notebook.sh $(ARGS)" > /dev/null
+	$(IMAGE) bash -c "$(PRE_CMD) chown jovyan /home/jovyan/work && start-notebook.py $(ARGS)" > /dev/null
 @echo "DONE: Notebook '$(NAME)' listening on $$(docker-machine ip $$(docker-machine active)):$(PORT)"
 endef
 

examples/openshift/templates.json

@@ -80,7 +80,7 @@
                 "name": "jupyter-notebook",
                 "image": "${NOTEBOOK_IMAGE}",
                 "command": [
-                  "start-notebook.sh",
+                  "start-notebook.py",
                   "--config=/etc/jupyter/openshift/jupyter_server_config.py",
                   "--no-browser",
                   "--ip=0.0.0.0"

examples/source-to-image/README.md

@@ -117,7 +117,7 @@ with the extra system packages, and then use that image with the S2I build to co
 The `run` script in this directory is very simple and just runs the notebook application.
 
 ```bash
-exec start-notebook.sh "$@"
+exec start-notebook.py "$@"
 ```
 
 ## Integration with OpenShift

examples/source-to-image/run

@@ -2,4 +2,4 @@
 
 # Start up the notebook instance.
 
-exec start-notebook.sh "$@"
+exec start-notebook.py "$@"

examples/source-to-image/templates.json

@@ -274,7 +274,7 @@
                     "name": "jupyter-notebook",
                     "image": "${APPLICATION_NAME}:latest",
                     "command": [
-                      "start-notebook.sh",
+                      "start-notebook.py",
                       "--config=/etc/jupyter/openshift/jupyter_server_config.py",
                       "--no-browser",
                       "--ip=0.0.0.0"

images/base-notebook/Dockerfile

@@ -52,10 +52,10 @@ ENV JUPYTER_PORT=8888
 EXPOSE $JUPYTER_PORT
 
 # Configure container startup
-CMD ["start-notebook.sh"]
+CMD ["start-notebook.py"]
 
 # Copy local files as late as possible to avoid cache busting
-COPY start-notebook.sh start-singleuser.sh /usr/local/bin/
+COPY start-notebook.py start-notebook.sh start-singleuser.py start-singleuser.sh /usr/local/bin/
 COPY jupyter_server_config.py docker_healthcheck.py /etc/jupyter/
 
 # Fix permissions on /etc/jupyter as root

images/base-notebook/start-notebook.py

@@ -0,0 +1,41 @@
+#!/usr/bin/env python
+# Copyright (c) Jupyter Development Team.
+# Distributed under the terms of the Modified BSD License.
+import os
+import shlex
+import sys
+
+# If we are in a JupyterHub, we pass on to `start-singleuser.py` instead so it does the right thing
+if "JUPYTERHUB_API_TOKEN" in os.environ:
+    print(
+        "WARNING: using start-singleuser.py instead of start-notebook.py to start a server associated with JupyterHub."
+    )
+    command = ["/usr/local/bin/start-singleuser.py"] + sys.argv[1:]
+    os.execvp(command[0], command)
+
+
+# Wrap everything in start.sh, no matter what
+command = ["/usr/local/bin/start.sh"]
+
+# If we want to survive restarts, tell that to start.sh
+if os.environ.get("RESTARTABLE") == "yes":
+    command.append("run-one-constantly")
+
+# We always launch a jupyter subcommand from this script
+command.append("jupyter")
+
+# Launch the configured subcommand. Note that this should be a single string, so we don't split it
+# We default to lab
+jupyter_command = os.environ.get("DOCKER_STACKS_JUPYTER_CMD", "lab")
+command.append(jupyter_command)
+
+# Append any optional NOTEBOOK_ARGS we were passed in. This is supposed to be multiple args passed
+# on to the notebook command, so we split it correctly with shlex
+if "NOTEBOOK_ARGS" in os.environ:
+    command += shlex.split(os.environ["NOTEBOOK_ARGS"])
+
+# Pass through any other args we were passed on the commandline
+command += sys.argv[1:]
+
+# Execute the command!
+os.execvp(command[0], command)

images/base-notebook/start-notebook.sh

@@ -1,22 +1,5 @@
 #!/bin/bash
-# Copyright (c) Jupyter Development Team.
-# Distributed under the terms of the Modified BSD License.
+# Shim to emit warning and call start-notebook.py
+echo "WARNING: Use start-notebook.py instead"
 
-set -e
-
-# The Jupyter command to launch
-# JupyterLab by default
-DOCKER_STACKS_JUPYTER_CMD="${DOCKER_STACKS_JUPYTER_CMD:=lab}"
-
-if [[ -n "${JUPYTERHUB_API_TOKEN}" ]]; then
-    echo "WARNING: using start-singleuser.sh instead of start-notebook.sh to start a server associated with JupyterHub."
-    exec /usr/local/bin/start-singleuser.sh "$@"
-fi
-
-wrapper=""
-if [[ "${RESTARTABLE}" == "yes" ]]; then
-    wrapper="run-one-constantly"
-fi
-
-# shellcheck disable=SC1091,SC2086
-exec /usr/local/bin/start.sh ${wrapper} jupyter ${DOCKER_STACKS_JUPYTER_CMD} ${NOTEBOOK_ARGS} "$@"
+exec /usr/local/bin/start-notebook.py "$@"

images/base-notebook/start-singleuser.py

@@ -0,0 +1,23 @@
+#!/usr/bin/env python
+# Copyright (c) Jupyter Development Team.
+# Distributed under the terms of the Modified BSD License.
+import os
+import shlex
+import sys
+
+command = ["/usr/local/bin/start.sh", "jupyterhub-singleuser"]
+
+# set default ip to 0.0.0.0
+if "--ip=" not in os.environ.get("NOTEBOOK_ARGS", ""):
+    command.append("--ip=0.0.0.0")
+
+# Append any optional NOTEBOOK_ARGS we were passed in. This is supposed to be multiple args passed
+# on to the notebook command, so we split it correctly with shlex
+if "NOTEBOOK_ARGS" in os.environ:
+    command += shlex.split(os.environ["NOTEBOOK_ARGS"])
+
+# Pass any other args we have been passed through
+command += sys.argv[1:]
+
+# Execute the command!
+os.execvp(command[0], command)

images/base-notebook/start-singleuser.sh

@@ -1,13 +1,5 @@
 #!/bin/bash
-# Copyright (c) Jupyter Development Team.
-# Distributed under the terms of the Modified BSD License.
+# Shim to emit warning and call start-singleuser.py
+echo "WARNING: Use start-singleuser.py instead"
 
-set -e
-
-# set default ip to 0.0.0.0
-if [[ "${NOTEBOOK_ARGS} $*" != *"--ip="* ]]; then
-    NOTEBOOK_ARGS="--ip=0.0.0.0 ${NOTEBOOK_ARGS}"
-fi
-
-# shellcheck disable=SC1091,SC2086
-. /usr/local/bin/start.sh jupyterhub-singleuser ${NOTEBOOK_ARGS} "$@"
+exec /usr/local/bin/start-singleuser.py "$@"

tests/base-notebook/test_container_options.py

@@ -15,7 +15,7 @@ def test_cli_args(container: TrackedContainer, http_client: requests.Session) ->
     """Image should respect command line args (e.g., disabling token security)"""
     host_port = find_free_port()
     running_container = container.run_detached(
-        command=["start-notebook.sh", "--IdentityProvider.token=''"],
+        command=["start-notebook.py", "--IdentityProvider.token=''"],
         ports={"8888/tcp": host_port},
     )
     resp = http_client.get(f"http://localhost:{host_port}")
@@ -102,7 +102,7 @@ def test_custom_internal_port(
     host_port = find_free_port()
     internal_port = env.get("JUPYTER_PORT", 8888)
     running_container = container.run_detached(
-        command=["start-notebook.sh", "--IdentityProvider.token=''"],
+        command=["start-notebook.py", "--IdentityProvider.token=''"],
         environment=env,
         ports={internal_port: host_port},
     )

tests/base-notebook/test_healthcheck.py

@@ -22,23 +22,24 @@
         (["RESTARTABLE=yes"], None, None),
         (["JUPYTER_PORT=8171"], None, None),
         (["JUPYTER_PORT=8117", "DOCKER_STACKS_JUPYTER_CMD=notebook"], None, None),
-        (None, ["start-notebook.sh", "--ServerApp.base_url=/test"], None),
-        (None, ["start-notebook.sh", "--ServerApp.base_url=/test/"], None),
-        (["GEN_CERT=1"], ["start-notebook.sh", "--ServerApp.base_url=/test"], None),
+        (None, ["start-notebook.sh"], None),
+        (None, ["start-notebook.py", "--ServerApp.base_url=/test"], None),
+        (None, ["start-notebook.py", "--ServerApp.base_url=/test/"], None),
+        (["GEN_CERT=1"], ["start-notebook.py", "--ServerApp.base_url=/test"], None),
         (
             ["GEN_CERT=1", "JUPYTER_PORT=7891"],
-            ["start-notebook.sh", "--ServerApp.base_url=/test"],
+            ["start-notebook.py", "--ServerApp.base_url=/test"],
             None,
         ),
         (["NB_USER=testuser", "CHOWN_HOME=1"], None, "root"),
         (
             ["NB_USER=testuser", "CHOWN_HOME=1"],
-            ["start-notebook.sh", "--ServerApp.base_url=/test"],
+            ["start-notebook.py", "--ServerApp.base_url=/test"],
             "root",
         ),
         (
             ["NB_USER=testuser", "CHOWN_HOME=1", "JUPYTER_PORT=8123"],
-            ["start-notebook.sh", "--ServerApp.base_url=/test"],
+            ["start-notebook.py", "--ServerApp.base_url=/test"],
             "root",
         ),
     ],
@@ -85,7 +86,7 @@ def test_health(
                 "HTTPS_PROXY=host.docker.internal",
                 "HTTP_PROXY=host.docker.internal",
             ],
-            ["start-notebook.sh", "--ServerApp.base_url=/test"],
+            ["start-notebook.py", "--ServerApp.base_url=/test"],
             "root",
         ),
     ],
@@ -122,12 +123,12 @@ def test_health_proxy(
         (["NB_USER=testuser", "CHOWN_HOME=1"], None, None),
         (
             ["NB_USER=testuser", "CHOWN_HOME=1"],
-            ["start-notebook.sh", "--ServerApp.base_url=/test"],
+            ["start-notebook.py", "--ServerApp.base_url=/test"],
             None,
         ),
         (
             ["NB_USER=testuser", "CHOWN_HOME=1", "JUPYTER_PORT=8123"],
-            ["start-notebook.sh", "--ServerApp.base_url=/test"],
+            ["start-notebook.py", "--ServerApp.base_url=/test"],
             None,
         ),
     ],