comment: code complexity somewhat overwhelming

[jstrong-tios]

this is not a bug, and feel free to immediately close this issue. I cloned pymc3 in an effort to understand how HMC/NUTS sampling works. What I found was in order to understand how the NUTS class works, I would need to understand:

• an involved class hierarchy (BlockedStep > GradientSharedStep > BaseHMC > NUTS)
• the Model class that inherits from three other classes in addition to a custom metaclass (class Model(Context, Factor, WithMemoization, metaclass=InitContextMeta):)
• Point - an object-creating function with capitalized name (expected class Point, searching initially failed)
• QuadPotential > QuadPotentialDiagAdapt
• CpuLeapFrogIntegrator
• ArrayOrdering, DictToArrayBijection
• IntEnum > Competence (also unique class decorator)
• bonus: rare try/except/else usage

Needless to say, it was a bit overwhelming. I'm not complaining about it, or urging you to "fix" it. This is just an anecdotal aside about what it's like for someone coming at pymc3 fresh to try to dive into the code.

[kyleabeauchamp]

There are a number of "teaching" HMC implementations out there (e.g., https://github.com/rmcgibbo/pyhmc). I think it's unavoidable to have pretty ugly type-checking and class hierarchies in production/general-purpose HMC codebases.

[fonnesbeck]

The design goals of PyMC3 center around making models easy to implement and be highly performant, and neither of these goals are necessarily consistent with being able to easily digest the implementation. We are mainly concerned with steering our users clear of having to write Theano code, while still enjoying all the benefits of having their model run on Theano as a computational backend.

That said, PyMC4 will involve a rewrite of the PyMC codebase (this time, around TF and TF Probability), so it offers an opportunity to go back and change things that perhaps ought to have been done differently in the first place (though the design goals are largely the same). We would greatly welcome your feedback on our proposed approach(es) as that work starts to come together!

[jstrong-tios]

out of curiosity, what do you see as the primary driver of complexity in this codebase?

• handling/computing an arbitrary computational graph
• making things play nicely with theano
• performance and/or thread/process synchronization
• providing flexible and resilient api
• related to flexible api: use of Model context manager
• inheritance four layers deep is just how I roll

asking in order to learn from your experience as have often faced similar issues with python projects as they grow larger. thanks!

[rpgoldman]

One of the things that drives complexity in debugging is that what look like the classes (e.g., pm.Normal) are not the real classes (e.g., FreeRV, ObservedRV). I remember this being quite confusing when I was just starting out.

Another issue is that the class hierarchy doesn't line up nicely with Theano. So if you look at the code for draw_values for example, there's a lot of odd branches to try to cover the space of possible values, and there's no good way for the PyMC3 programmer to be sure they have actually exhausted all the possible alternatives. It would be nice to have a mutually exclusive and exhaustive set of classes to work with.

One thing I was wondering was why we don't extend the Theano class hierarchy so that, for example, pm.FreeRV is a subclass of theano.TensorVariable.

[junpenglao]

There are indeed a lot of complexity, personally I think historically it is designed to have the (conditional) log_prob function with its gradient easily generated, then we found edge cases and need to add more complexity to fix bugs and maintain backward compatibility. Some of which are because design "mistake" (shape handling, mostly).

In case you missed it, https://docs.pymc.io/developer_guide.html might help understanding the design.

[junpenglao]

A few more thoughts:

inheritance four layers deep is just how I roll

Wait until you check out the variational inference part. 😬

Part of the complexity I would blame it on premature optimization. The class hierarchy in NUTS and VI is because we want to have more abstraction so that we can modify and add sub method easier. By submethod I mean things like adaptation in NUTS/HMC, loss function and approximation in VI. In retrospect it might clearer and more standalone if we implement each method as its own class, the con is that we will have much more code redundancy and addition of new method much more difficult (well not that it is easy now, but if you know the system well modification became much easier like #3605 #3656).

[junpenglao]

One thing I was wondering was why we don't extend the Theano class hierarchy so that, for example, pm.FreeRV is a subclass of theano.TensorVariable.

@rpgoldman It is: https://github.com/pymc-devs/pymc3/blob/cc5527988f72bbd2d62c7248226f62dcfbd7403d/pymc3/model.py#L34
It's clearer before #3619, which makes me think maybe we should push this change to upstream theano.

[brandonwillard]

One thing I was wondering was why we don't extend the Theano class hierarchy so that, for example, pm.FreeRV is a subclass of theano.TensorVariable.

@rpgoldman It is:

https://github.com/pymc-devs/pymc3/blob/cc5527988f72bbd2d62c7248226f62dcfbd7403d/pymc3/model.py#L34

It's clearer before #3619, which makes me think maybe we should push this change to upstream theano.

As soon as you start going in that direction, you'll end up here, for better or worse.

[eigenfoo]

@jstrong-tios for my two cents, and having just gone through the process of learning HMC, I found the following resources to be very helpful in understanding HMC and NUTS at a higher level.

• Colin Carroll's minimc library: https://github.com/colcarroll/minimc
• Michael Betancourt's Conceptual Introduction to HMC: https://arxiv.org/abs/1701.02434
• The original NUTS paper by Hoffman and Gelman: http://www.stat.columbia.edu/~gelman/research/published/nuts.pdf
• Chapter 5 of the MCMC Handbook by Radford Neal: http://mcmchandbook.net/HandbookChapter5.pdf

Hope this helps! Since this isn't a development issue, I'll close this issue, but please feel free to continue the discussion.

[jstrong-tios]

@eigenfoo thank you! I had already come across minimc and it is outstanding! That was the one that got everything to click for me - Colin Carroll is awesome. I have not come across the Neal handbook - so that may be a big assist. Was just starting to dig into the NUTS paper earlier today and felt I would be able to make some sense of it this time. Thanks again for your help!