WIP: Add basic tests for examples notebooks

[ghost]

In #2236 there is a discussion about the status of examples notebooks. I checked some of them, and it turned some of the notebooks are broken because of API changes in a sense that they throw exceptions on execution, mostly of API changes as far as I can see.

So I tried to find a way to test them all at least in some basic way. Although it is hard to test code logic in the notebooks, it is at least easy to just make sure that the code in them can be run without errors.

This PR adds a test that runs all the notebooks using nbconvert executor. It turns out that the following notebooks should be fixed:

• Diagnosing_biased_Inference_with_Divergences.ipynb

  KeyError: 'tau_log_'
  

• Euler-Maruyama and SDEs.ipynb

  MissingInputError: Input 0 of the graph (indices start from 0), used to compute Shape(xh),
  was not provided and not given a value.
  

• LKJ.ipynb

  IndexError: index out of bounds
  

• Model Comparison.ipynb

  AttributeError: module 'pymc3' has no attribute 'compare_plot'
  

• PyMC3_tips_and_heuristic.ipynb

  TimeoutError: Cell execution timed out
  

  Timeout here is not because of heavy calculations, but because the notebook creates QtConsole at some point and waits for it.
• SMC2_gaussians.ipynb

  TypeError: unsupported operand type(s) for -: 'str' and 'int'
  

• bayesian_neural_network_advi.ipynb

  MissingInputError: Input 0 of the graph (indices start from 0), used to compute Shape(w_in_1),
  was not provided and not given a value.
  

• bayesian_neural_network_opvi-advi.ipynb

  AttributeError: 'MeanField' object has no attribute 'sample_vp'
  

• dependent_density_regression.ipynb

  KeyError: 'avconv'
  

• gaussian_process.ipynb

  AttributeError: module 'pymc3.variational' has no attribute 'svgd'
  

• getting_started.ipynb

  ValueError: Cannot compute test value: input 1 (Elemwise{add,no_inplace}.0) of Op
  Elemwise{sub,no_inplace}(Elemwise{mul,no_inplace}.0, Elemwise{add,no_inplace}.0) missing default value.
  

• lda-advi-aevb.ipynb

  ValueError: Cannot compute test value: input 0 (Subtensor{::, :int64:}.0) of Op
  Reshape{1}(Subtensor{::, :int64:}.0, TensorConstant{(1,) of -1}) missing default value.
  

• multilevel_modeling.ipynb

  AttributeError: 'Series' object has no attribute 'order'
  

• posterior_predictive.ipynb

  MissingInputError: Input 0 of the graph (indices start from 0), used to compute
  Shape(sd_log__), was not provided and not given a value.
  

• probabilistic_matrix_factorization.ipynb

  IndexError: index out of bounds
  

• rugby_analytics.ipynb

  jupyter_client.kernelspec.NoSuchKernel: No such kernel named conda-env-pymc3_dev-py
  

These notebooks run on CPU for more than 15 minutes, so the tests for them are timeouted:

• GLM-logistic.ipynb
• GP-introduction.ipynb
• GP-slice-sampling.ipynb
• convolutional_vae_keras_advi.ipynb
• dawid-skene.ipynb
• survival_analysis.ipynb

I'm not sure what would be an elegant way for dealing with these long-running notebooks that time out.

[junpenglao]

I am not sure this is the best way to do it, as testing it in Travis will take forever. I think we had this discussion at some point before @ColCarroll ?
Maybe it is time to move the notebooks into a model zoo and maintain it separately? It might be easier for people to contribute examples, demos, and analysis etc.

[fonnesbeck]

We can't test them all, unfortunately.

[ColCarroll]

yeah, it will almost surely time out.

I wrote https://github.com/ColCarroll/carpo (pip install carpo) to regenerate all the notebooks. At a glance, it is super similar to what you wrote, @a-rodin, though more dangerous in that it overwrites notebooks. I like the idea of an easy way to check all the notebooks for errors.

@junpenglao I'm not sure what a model zoo would look like for pymc3! One thought would be that it is pre-computed traces for large models. Another thought is that it is pre-written functions for common models (https://github.com/ColCarroll/sampled could help with that)

[junpenglao]

Yeah model zoo is probably not the right term, i meant something like an example showcase in a separate repository.

[kyleabeauchamp]

We already are having trouble maintaining the examples we have, so I'm not sure moving them outside of the main repo will help with that. IMHO, maintained examples can live here and unmaintained examples can live in people's personal githubs or gists.

[aseyboldt]

Wow, this is worse than I expected. I'll fix getting-started (that's probably the theano flag change), LKJ and rugby_analytics.

[fonnesbeck]

If we are diligent about going in and manually checking that they run, say, before each release the I think that's fine. They don't all have to be tested every time we check in new code. Regular review will be important however.

I've added a task to the release checklist on the wiki to check the notebooks.

[aseyboldt]

getting_started and rugby should be fixed now. #2255 might also have fixed some of the other notebooks.

[ericmjl]

@fonnesbeck @ColCarroll I'm happy to be responsible for checking the notebooks pre-releases, if it takes a load off your plate! I have access to a fairly fast GPU tower at home, and can probably use the method @aseyboldt proposed up here (execute nbconvert) to handle the changes. FWIW I helped with matplotlib's docs (in a similar fashion) as well, and it'll give me another way to learn more about the code base too.

[ColCarroll]

That sounds great to me -- if you put up a similar post to this one, you can farm out help for updating APIs. Also see @kyleabeauchamp 's work on making the test suite run on the GPU (#2264)

[fonnesbeck]

@a-rodin are you going to continue this work, or can it be closed?

[twiecki]

Closing because of age. Feel free to reopen if you want to fix this up.