Add [Nonlinear.ReverseAD] submodule

[odow]

Testing this as a submodule of MOI to see what the impact would be on JuMP

Docs: https://jump.dev/MathOptInterface.jl/previews/PR1803/submodules/Nonlinear/overview/

Before merging:

• Rename to _Nonlinear, or otherwise indicate that the API is experimental and subject to change. Added a warning to the docs.

The majority of this development was carried out in the JuMP PRs:

• DNMY: exploration of a refactored Nonlinear API design JuMP.jl#2939
• DNMY: refactored _Derivatives on new .Nonlinear submodule JuMP.jl#2942
• DNMY: stitch JuMP back together using Nonlinear and ReverseAD JuMP.jl#2943

Nonlinear.ReverseAD is a minor refactoring of code that previously
existed in JuMP under the _Derivatives submodule, and prior to that
the ReverseDiffSparse.jl package.

Once merged open issues for:

• Changing the parsing of constraints from lhs - rhs <= 0 Nonlinear: change how constraints are parsed #1817
• Adding support for hessians of user-defined functions [ReverseAD] Add support for user-defined hessians #1819

[mlubin]

This is too big to review. One possible split is 1) introduce the nonlinear data structures into MOI 2) add the ReverseAD backend

[matbesancon]

Does this only introduce symbolic nonlinear and not oracle-based nonlinear?

[odow]

This is too big to review.

Yeah. Now that it's in MOI, I'll break it back up again.

Does this only introduce symbolic nonlinear and not oracle-based nonlinear?

It is (just) a refactoring of JuMP's current nonlinear interface. But it's also designed to be extensible, so that SymbolicAD is now an add-on: lanl-ansi/MathOptSymbolicAD.jl#17

optimize!(model; differentiation_backend = SymbolicAD.DefaultBackend())
optimize!(model; differentiation_backend = MOI.Nonlinear.SparseReverseMode())

[matbesancon]

having to interpolate can make some friction, we should offer a substitute function and/or macro

[odow]

having to interpolate can make some friction, we should offer a substitute function and/or macro

That's JuMP's job. It doesn't matter if this interface has some friction.

[mlubin]

I reject the premise of the question because this code was one of the first implementations of reverse-mode AD in Julia. This discussion as worded doesn't really fit as package documentation. What would fit is a discussion of the design goals of ReverseAD. It's fair to mention the history of the code as well.

[odow]

How's this now? I only made minor changes to split the history and identify design goals. But can make larger changes if you want?

[mlubin]

Oops, I left a comment pending.