conda execute a Jupyter notebook

[pelson]

One potential approach:

• - create a temporary environment by interpreting a part of the nbformat metadata JSON payload
• - start a kernel using the temporary environment (ipython or otherwise)
• - join an existing Jupyter notebookapp (or start a new one) which uses the kernel and the notebook that was passed to us

Ping @minrk / @takluyver for your wisdom. Are there any patterns I can follow? This is essentially an extension of https://github.com/takluyver/nbopen but with the addition of ensuring there is an appropriate kernel running.

[pelson]

Original thread came from https://twitter.com/mrocklin/status/655100657016242176

@Carreau mentioned the nbconvert --execute option.

It seems nbconvert is already in this space, the only way I would want conda-execute to add anything is if we can make use of the fact that we can define a full environment. It seems that conda execute my_analysis.ipynb could do a number of things:

• launch the Jupyter notebook with a suitable kernel without running any of the actual code
• run the notebook and convert to html (as nbconvert is currently doing)
• run the notebook and save on top of the existing notebook

Rather than bake any of this in, conda-execute would essentially call out to the appropriate executable:

$ cat my_analysis.ipynb
{
  "metadata" : {
    "conda-execute": {
        "env": ['python >2', 'numpy==1.10', 'nbconvert'],
        "run_with": ["nbconvert",  "--execute"]
    },
  },
  "nbformat": 4,
  "nbformat_minor": 0,
  "cells" : [
  ...
   ]
}

Of course, to do this, conda-execute does need to read the metadata from the notebook format.

[takluyver]

I think that it would be valuable to have conda setting up environments based on notebook metadata. One way to do this might be a custom kernelspec manager which could look at metadata when a notebook is opened and construct a suitable environment at that time. I think we've talked to @ahmadia about a similar idea w/r/t hashdist.

[ahmadia]

Hi folks. We've been trying to figure out how to do this from the conda side as well. As @takluyver mentioned, one option is to attach the conda environment directly to the notebook metadata. See the discussion here: conda-archive/conda-env#111 and the feature here: http://conda.pydata.org/docs/commands/env/conda-env-attach.html

From there, you need a corresponding launch action. One option is conda launch from @ijstokes - https://github.com/conda/conda-launch

cc: @chdoig @damianavila @pwang @malev

I'm happy to discuss here a bit more about what the various options are and what we like/dislike about each.

[ahmadia]

Of course, anything we do for conda should be similarly applicable to HashDist. Thanks for the mention @takluyver.

[Carreau]

For nbconvert --execute you need to replace the execute preprocessor by one that start kernels in an env. Kernelspec already can have an "env" metadata so overwriting PATH, should be sufficient.

Running the notebook app in the env seem unnecessary as IIUC, you want to run the notebook kernel in the env,

run the notebook and convert to html (as nbconvert is currently doing)
run the notebook and save on top of the existing notebook

This will already be taken care of if you use --to ipynb or --to html with nbconvert.
I guess you might want to use nbconvert as a library instead of shelling out, we can figure that out.

[ahmadia]

Running the notebook app in the env seem unnecessary as IIUC, you want to run the notebook kernel in the env,

This made sense when the kernel spec was changing rapidly and you wanted the app version to match the kernel version. I agree that this is not a hard requirement for using environments.