Rerun notebooks #3397

canyon289 · 2019-03-09T05:25:10Z

Trying to rerun all notebooks but been running into tons of errors because basically api changes in multiple libraries are breaking a bunch of the notebooks.

I'll push changes here and check off the ones that I have running. If anyone wants to help out thatd be great!

My current env

alabaster 0.7.12 pypi_0 pypi
algopy 0.5.7 pypi_0 pypi
argh 0.26.2 pypi_0 pypi
arviz 0.3.3 pypi_0 pypi
astroid 2.2.5 pypi_0 pypi
atomicwrites 1.3.0 pypi_0 pypi
attrs 19.1.0 pypi_0 pypi
babel 2.6.0 pypi_0 pypi
backcall 0.1.0 pypi_0 pypi
blas 1.0 mkl
bleach 3.1.0 pypi_0 pypi
bokeh 1.0.4 pypi_0 pypi
ca-certificates 2019.1.23 0
certifi 2018.11.29 py37_0
cftime 1.0.3.4 pypi_0 pypi
chardet 3.0.4 pypi_0 pypi
commonmark 0.5.4 pypi_0 pypi
coverage 4.5.2 pypi_0 pypi
cycler 0.10.0 pypi_0 pypi
decorator 4.3.2 pypi_0 pypi
defusedxml 0.5.0 pypi_0 pypi
docutils 0.14 pypi_0 pypi
entrypoints 0.3 pypi_0 pypi
h5py 2.9.0 pypi_0 pypi
idna 2.8 pypi_0 pypi
imagesize 1.1.0 pypi_0 pypi
intel-openmp 2019.1 144
ipykernel 5.1.0 pypi_0 pypi
ipython 7.3.0 pypi_0 pypi
ipython-genutils 0.2.0 pypi_0 pypi
isort 4.3.13 pypi_0 pypi
jedi 0.13.3 pypi_0 pypi
jinja2 2.10 pypi_0 pypi
jsonschema 3.0.1 pypi_0 pypi
jupyter-client 5.2.4 pypi_0 pypi
jupyter-core 4.4.0 pypi_0 pypi
jupyterlab 0.35.4 pypi_0 pypi
jupyterlab-server 0.2.0 pypi_0 pypi
keras 2.2.4 pypi_0 pypi
keras-applications 1.0.7 pypi_0 pypi
keras-preprocessing 1.0.9 pypi_0 pypi
kiwisolver 1.0.1 pypi_0 pypi
lazy-object-proxy 1.3.1 pypi_0 pypi
libedit 3.1.20181209 hc058e9b_0
libffi 3.2.1 hd88cf55_4
libgcc-ng 8.2.0 hdf63c60_1
libgfortran-ng 7.3.0 hdf63c60_0
libstdcxx-ng 8.2.0 hdf63c60_1
livereload 2.6.0 pypi_0 pypi
markupsafe 1.1.1 pypi_0 pypi
matplotlib 3.0.3 pypi_0 pypi
mccabe 0.6.1 pypi_0 pypi
mistune 0.8.4 pypi_0 pypi
mkl 2019.1 144
mkl_fft 1.0.10 py37ha843d7b_0
mkl_random 1.0.2 py37hd81dba3_0
more-itertools 6.0.0 pypi_0 pypi
nbconvert 5.4.1 pypi_0 pypi
nbformat 4.4.0 pypi_0 pypi
nbsphinx 0.4.2 pypi_0 pypi
ncurses 6.1 he6710b0_1
netcdf4 1.4.3.2 pypi_0 pypi
nose 1.3.7 pypi_0 pypi
nose-parameterized 0.6.0 pypi_0 pypi
notebook 5.7.5 pypi_0 pypi
numdifftools 0.9.20 pypi_0 pypi
numpy 1.16.2 py37h7e9f1db_0
numpy-base 1.16.2 py37hde5b4d6_0
numpydoc 0.7.0 pypi_0 pypi
openssl 1.1.1b h7b6447c_1
packaging 19.0 pypi_0 pypi
pandas 0.24.1 py37he6710b0_0
pandocfilters 1.4.2 pypi_0 pypi
parso 0.3.4 pypi_0 pypi
pathtools 0.1.2 pypi_0 pypi
patsy 0.5.1 pypi_0 pypi
pexpect 4.6.0 pypi_0 pypi
pickleshare 0.7.5 pypi_0 pypi
pillow 5.4.1 pypi_0 pypi
pip 19.0.3 py37_0
pluggy 0.9.0 pypi_0 pypi
port-for 0.3.1 pypi_0 pypi
prometheus-client 0.6.0 pypi_0 pypi
prompt-toolkit 2.0.9 pypi_0 pypi
ptyprocess 0.6.0 pypi_0 pypi
py 1.8.0 pypi_0 pypi
pycodestyle 2.5.0 pypi_0 pypi
pyflakes 2.1.1 pypi_0 pypi
pygments 2.3.1 pypi_0 pypi
pylint 2.3.1 pypi_0 pypi
pymc3 3.6 dev_0
pyparsing 2.3.1 pypi_0 pypi
pyrsistent 0.14.11 pypi_0 pypi
pytest 4.3.0 pypi_0 pypi
pytest-cov 2.6.1 pypi_0 pypi
python 3.7.2 h0371630_0
python-dateutil 2.8.0 py37_0
python-graphviz 0.10.1 pypi_0 pypi
pytz 2018.9 py37_0
pyyaml 3.13 pypi_0 pypi
pyzmq 18.0.1 pypi_0 pypi
readline 7.0 h7b6447c_5
recommonmark 0.5.0 pypi_0 pypi
requests 2.21.0 pypi_0 pypi
scipy 1.2.1 pypi_0 pypi
seaborn 0.9.0 pypi_0 pypi
send2trash 1.5.0 pypi_0 pypi
setuptools 40.8.0 py37_0
six 1.12.0 pypi_0 pypi
snowballstemmer 1.2.1 pypi_0 pypi
sphinx 1.8.4 pypi_0 pypi
sphinx-autobuild 0.7.1 pypi_0 pypi
sphinxcontrib-websupport 1.1.0 pypi_0 pypi
sqlite 3.26.0 h7b6447c_0
statsmodels 0.9.0 py37h035aef0_0
terminado 0.8.1 pypi_0 pypi
testpath 0.4.2 pypi_0 pypi
theano 1.0.4 pypi_0 pypi
tk 8.6.8 hbc83047_0
tornado 6.0.1 pypi_0 pypi
tqdm 4.31.1 pypi_0 pypi
traitlets 4.3.2 pypi_0 pypi
typed-ast 1.3.1 pypi_0 pypi
urllib3 1.24.1 pypi_0 pypi
watchdog 0.9.0 pypi_0 pypi
wcwidth 0.1.7 pypi_0 pypi
webencodings 0.5.1 pypi_0 pypi
wheel 0.33.1 py37_0
wrapt 1.11.1 pypi_0 pypi
xarray 0.11.3 pypi_0 pypi
xz 5.2.4 h14c3975_4
zlib 1.2.11 h7b6447c_3

junpenglao · 2019-03-09T06:59:42Z

I will help.
Running the notebooks I wrote and the GLM notebooks.

junpenglao · 2019-03-09T07:06:49Z

Also, we might want to prioritize those notebook have not been run for a while.

ColCarroll · 2019-03-09T15:03:37Z

I will run the GP- notebooks

canyon289 · 2019-03-09T15:16:19Z

Also, we might want to prioritize those notebook have not been run for a while.

How can we tell which one hasn't been run in a while?

ColCarroll · 2019-03-09T15:37:52Z

Via https://serverfault.com/questions/401437/how-to-retrieve-the-last-modification-date-of-all-files-in-a-git-repository and some messing around:

git ls-tree -r --name-only HEAD | grep notebooks | while read filename; do   echo "$(git log -1 --format="%ai" -- $filename) $filename"; done | sort

ColCarroll · 2019-03-09T15:38:33Z

2017-03-27 11:36:12 +0200 docs/source/notebooks/live_sample_plots.ipynb
2017-04-26 13:42:44 +0000 docs/source/notebooks/dawid-skene.ipynb
2017-06-23 16:10:24 -0400 docs/source/notebooks/GP-slice-sampling.ipynb
2017-08-17 13:52:51 +0300 docs/source/notebooks/gaussian-mixture-model-advi.ipynb
2018-01-26 06:20:54 -0300 docs/source/notebooks/smc.png
2018-02-03 18:07:03 -0600 docs/source/notebooks/normalizing_flows_overview.ipynb
2018-03-07 12:05:34 -0300 docs/source/notebooks/dp_mix.ipynb
2018-06-22 18:27:08 -0300 docs/source/notebooks/SMC2_gaussians.ipynb
2018-06-24 17:12:12 -0700 docs/source/notebooks/GP-MeansAndCovs.ipynb
2018-07-12 04:00:25 +0000 docs/source/notebooks/GLM-negative-binomial-regression.ipynb
2018-07-30 02:03:19 -0400 docs/source/notebooks/sampling_compound_step.ipynb
2018-07-31 13:32:52 -0400 docs/source/notebooks/dependent_density_regression.ipynb
2018-07-31 13:32:52 -0400 docs/source/notebooks/GP-SparseApprox.ipynb
2018-07-31 13:32:52 -0400 docs/source/notebooks/GP-TProcess.ipynb
2018-07-31 13:32:52 -0400 docs/source/notebooks/marginalized_gaussian_mixture_model.ipynb
2018-10-07 09:36:42 +0000 docs/source/notebooks/LKJ.ipynb
2018-11-15 01:14:31 +0900 docs/source/notebooks/lda-advi-aevb.ipynb
2018-12-19 13:15:50 +0100 docs/source/notebooks/cox_model.ipynb
2018-12-19 13:15:50 +0100 docs/source/notebooks/gaussian_process.ipynb
2018-12-19 18:27:06 +0100 docs/source/notebooks/table_of_contents_tutorials.js
2018-12-22 12:14:33 +0100 docs/source/notebooks/bayesian_neural_network_advi.ipynb
2018-12-22 12:14:33 +0100 docs/source/notebooks/blackbox_external_likelihood.ipynb
2018-12-22 12:14:33 +0100 docs/source/notebooks/censored_data.ipynb
2018-12-22 12:14:33 +0100 docs/source/notebooks/convolutional_vae_keras_advi.ipynb
2018-12-22 12:14:33 +0100 docs/source/notebooks/empirical-approx-overview.ipynb
2018-12-22 12:14:33 +0100 docs/source/notebooks/Euler-Maruyama_and_SDEs.ipynb
2018-12-22 12:14:33 +0100 docs/source/notebooks/gaussian_mixture_model.ipynb
2018-12-22 12:14:33 +0100 docs/source/notebooks/getting_started.ipynb
2018-12-22 12:14:33 +0100 docs/source/notebooks/GLM-hierarchical-advi-minibatch.ipynb
2018-12-22 12:14:33 +0100 docs/source/notebooks/GLM-hierarchical.ipynb
2018-12-22 12:14:33 +0100 docs/source/notebooks/GLM.ipynb
2018-12-22 12:14:33 +0100 docs/source/notebooks/GLM-linear.ipynb
2018-12-22 12:14:33 +0100 docs/source/notebooks/GLM-model-selection.ipynb
2018-12-22 12:14:33 +0100 docs/source/notebooks/GLM-poisson-regression.ipynb
2018-12-22 12:14:33 +0100 docs/source/notebooks/GLM-robust.ipynb
2018-12-22 12:14:33 +0100 docs/source/notebooks/GLM-robust-with-outlier-detection.ipynb
2018-12-22 12:14:33 +0100 docs/source/notebooks/GP-Latent.ipynb
2018-12-22 12:14:33 +0100 docs/source/notebooks/GP-MaunaLoa.ipynb
2018-12-22 12:14:33 +0100 docs/source/notebooks/GP-smoothing.ipynb
2018-12-22 12:14:33 +0100 docs/source/notebooks/howto_debugging.ipynb
2018-12-22 12:14:33 +0100 docs/source/notebooks/lasso_block_update.ipynb
2018-12-22 12:14:33 +0100 docs/source/notebooks/multilevel_modeling.ipynb
2018-12-22 12:14:33 +0100 docs/source/notebooks/MvGaussianRandomWalk_demo.ipynb
2018-12-22 12:14:33 +0100 docs/source/notebooks/posterior_predictive.ipynb
2018-12-22 12:14:33 +0100 docs/source/notebooks/rugby_analytics.ipynb
2018-12-22 12:14:33 +0100 docs/source/notebooks/sampler-stats.ipynb
2018-12-22 12:14:33 +0100 docs/source/notebooks/survival_analysis.ipynb
2018-12-22 12:14:33 +0100 docs/source/notebooks/updating_priors.ipynb
2018-12-22 12:14:33 +0100 docs/source/notebooks/variational_api_quickstart.ipynb
2018-12-26 20:41:20 +0000 docs/source/notebooks/GP-Kron.ipynb
2018-12-27 20:26:54 +0000 docs/source/notebooks/Diagnosing_biased_Inference_with_Divergences.ipynb
2018-12-27 20:26:54 +0000 docs/source/notebooks/GLM-hierarchical-binominal-model.ipynb
2018-12-27 20:26:54 +0000 docs/source/notebooks/GLM-rolling-regression.ipynb
2018-12-27 20:26:54 +0000 docs/source/notebooks/GP-MaunaLoa2.ipynb
2018-12-27 20:26:54 +0000 docs/source/notebooks/hierarchical_partial_pooling.ipynb
2018-12-27 20:26:54 +0000 docs/source/notebooks/PyMC3_tips_and_heuristic.ipynb
2018-12-27 20:26:54 +0000 docs/source/notebooks/stochastic_volatility.ipynb
2018-12-27 20:26:54 +0000 docs/source/notebooks/weibull_aft.ipynb
2019-01-04 01:12:42 -0500 docs/source/notebooks/GLM-logistic.ipynb
2019-01-15 08:37:30 -0500 docs/source/notebooks/BEST.ipynb
2019-01-25 17:42:26 -0300 docs/source/notebooks/model_averaging.ipynb
2019-01-28 19:06:17 +0100 docs/source/notebooks/model_comparison.ipynb
2019-03-07 17:33:21 +0000 docs/source/notebooks/table_of_contents_examples.js
2019-03-07 20:18:23 +0000 docs/source/notebooks/ODE_parameter_estimation.ipynb
2019-03-08 20:41:43 -0800 docs/source/notebooks/Bayes_factor.ipynb
2019-03-08 21:10:16 -0800 docs/source/notebooks/api_quickstart.ipynb
2019-03-08 21:11:09 -0800 docs/source/notebooks/AR.ipynb
2019-03-08 21:25:58 -0800 docs/source/notebooks/bayes_param_survival_pymc3.ipynb
2019-03-09 08:03:41 +0100 docs/source/notebooks/profiling.ipynb
2019-03-09 10:31:01 -0500 docs/source/notebooks/GP-Marginal.ipynb

fonnesbeck · 2019-03-09T18:53:08Z

This might be an opportunity to retire notebooks that are either redundant or are not well-annotated. I think the suite is currently too big. Might be worth considering moving most of them to the pymc resources repo.

twiecki · 2019-03-19T12:30:03Z

empirical-approx-overview.ipynb has many problems, not sure if it's worth keeping around. @ferrine?

ferrine · 2019-03-26T21:02:08Z

@twiecki, sorry for late update, let me have a look this night

ferrine · 2019-03-26T21:04:22Z

Do you mean bad documented?

twiecki · 2019-03-26T21:06:22Z

Yeah exactly. I think it's quite difficult to follow.

…

On Tue, Mar 26, 2019, 22:04 Maxim Kochurov ***@***.***> wrote: Do you mean bad documented? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3397 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AApJmHrRMLuhrGv-nh4wIxD2mrM-8QFpks5vaotXgaJpZM4bmfnE> .

junpenglao · 2019-03-26T21:41:41Z

I did a rebase, hopefully did not mess up anything...

junpenglao · 2019-04-06T09:48:12Z

This is blocking our release - should we split the effort and run the essential notebooks?

junpenglao · 2019-04-06T09:49:13Z

I would say definitively the GP ones should be rerun

ferrine · 2019-04-06T10:42:30Z

@junpenglao if Empirical notebook is a concern, I can take it later and move them to inline docs. For now it is better option to leave it as is and open an issue.

PS I'm on vacations now

junpenglao · 2019-04-24T11:09:43Z

The Dawid-Skene notebook is fairly OK I remember - any reason why removing it?

twiecki · 2019-04-24T11:50:28Z

I couldn't run it and gave up, if we figure it out should be fine.

…

On Wed, Apr 24, 2019, 13:09 Junpeng Lao ***@***.***> wrote: The Dawid-Skene notebook is fairly OK I remember - any reason why removing it? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3397 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAFETGDPWOBXGIRQGNDPACDPSA5XTANCNFSM4G4Z7HCA> .

…into rerun_notebooks

junpenglao · 2019-05-01T18:17:21Z

Oh nice @lucianopaz ! I think we are finally ready to merge and do a release?

lucianopaz · 2019-05-01T18:19:28Z

Only the big manualoa2 is left, which takes like a full day to run

lucianopaz · 2019-05-01T18:24:39Z

Releasewise, I would really like #3456 to make it into the release

canyon289 · 2019-05-02T05:56:46Z

Is anyone planning on running that one? I can run it on my desktop and leave it on all day

canyon289 · 2019-05-07T05:28:21Z

Will push manualoa2 if it finishes running on my desktop and merge in a separate branch

canyon289 added the WIP label Mar 9, 2019

twiecki force-pushed the rerun_notebooks branch from 98e5746 to 6ea7444 Compare April 16, 2019 19:50

canyon289 and others added 13 commits April 16, 2019 21:51

Fix Bayes facotr plot

9bf2052

Fix api_quickstart

6ffa27f

Fix AR

b10440b

Fix bayes_param_survival

6910491

rerun notebook: profiling

41f6b69

Rerun gp-marginal

8de50e7

Rerun gp-kron

00c280c

Rerun BEST notebook

f41c83f

Rerun censored data

1ba1a98

Rerun gp-latent

4a90891

Rerun gp-maunaloa

50a8e2b

rerun pymc3_tips_and_heuristic

4933d71

rerun + fix (most) GLM related docs

6ec7ba1

lucianopaz added 16 commits April 30, 2019 17:46

Reran gaussian_process

2a6feea

Reran GP-MeansAndCovs

82de3bd

Merge branch 'rerun_notebooks' of https://github.com/canyon289/pymc3 …

de4b033

…into rerun_notebooks

Removing slice sampling. This step method was move to pymc-experiments

add16c4

Reran dp_mix

5362f50

Reran GP-smoothing

c7444dc

Reran lda-advi

b71c93a

Reran GP-Tprocess

fc66cb2

Fixed dp_mix

8f6eb68

Fix for GP-MeansAndCovs

fd47636

Fix for GP-SparseApprox

df6262b

Fix for GP-TProcess

7d2fa50

Fix for GP-smoothing

b411452

Fix for dp_mix

096b235

Fix for gaussian_process

3673291

Fix for lda-advi-aevb

3cdde2a

Update Euler-Maruyama.

b253803

canyon289 merged commit 55ccacf into pymc-devs:master May 7, 2019

junpenglao removed the WIP label May 12, 2019

This was referenced May 12, 2019

Rerun NBs now that plotting backend is switched #3393

Closed

Fixes to DP mixture notebook #3459

Merged

farhanreynaldo mentioned this pull request Nov 13, 2021

Update Hierarchical binomial notebook pymc-devs/pymc-examples#248

Merged

canyon289 deleted the rerun_notebooks branch January 30, 2022 18:36

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Rerun notebooks #3397

Rerun notebooks #3397

canyon289 commented Mar 9, 2019 •

edited by lucianopaz

Loading

junpenglao commented Mar 9, 2019 •

edited

Loading

junpenglao commented Mar 9, 2019

ColCarroll commented Mar 9, 2019

canyon289 commented Mar 9, 2019

ColCarroll commented Mar 9, 2019

ColCarroll commented Mar 9, 2019

fonnesbeck commented Mar 9, 2019

twiecki commented Mar 19, 2019

ferrine commented Mar 26, 2019

ferrine commented Mar 26, 2019

twiecki commented Mar 26, 2019 via email

junpenglao commented Mar 26, 2019

junpenglao commented Apr 6, 2019

junpenglao commented Apr 6, 2019

ferrine commented Apr 6, 2019

junpenglao commented Apr 24, 2019

twiecki commented Apr 24, 2019 via email

junpenglao commented May 1, 2019

lucianopaz commented May 1, 2019

lucianopaz commented May 1, 2019

canyon289 commented May 2, 2019

canyon289 commented May 7, 2019

Rerun notebooks #3397

Rerun notebooks #3397

Conversation

canyon289 commented Mar 9, 2019 • edited by lucianopaz Loading

junpenglao commented Mar 9, 2019 • edited Loading

junpenglao commented Mar 9, 2019

ColCarroll commented Mar 9, 2019

canyon289 commented Mar 9, 2019

ColCarroll commented Mar 9, 2019

ColCarroll commented Mar 9, 2019

fonnesbeck commented Mar 9, 2019

twiecki commented Mar 19, 2019

ferrine commented Mar 26, 2019

ferrine commented Mar 26, 2019

twiecki commented Mar 26, 2019 via email

junpenglao commented Mar 26, 2019

junpenglao commented Apr 6, 2019

junpenglao commented Apr 6, 2019

ferrine commented Apr 6, 2019

junpenglao commented Apr 24, 2019

twiecki commented Apr 24, 2019 via email

junpenglao commented May 1, 2019

lucianopaz commented May 1, 2019

lucianopaz commented May 1, 2019

canyon289 commented May 2, 2019

canyon289 commented May 7, 2019

canyon289 commented Mar 9, 2019 •

edited by lucianopaz

Loading

junpenglao commented Mar 9, 2019 •

edited

Loading