🔀 MERGE: Improve Math Parsing (#217)

Author: chrisjsewell

codecov.yml

@@ -0,0 +1,10 @@
+coverage:
+  status:
+    project:
+      default:
+        target: 90%
+        threshold: 0.5%
+    patch:
+      default:
+        target: 85%
+        threshold: 0.5%

docs/conf.py

@@ -67,7 +67,8 @@
 
 myst_amsmath_enable = True
 myst_admonition_enable = True
-myst_html_img = True
+myst_html_img_enable = True
+myst_dmath_enable = True
 
 
 def run_apidoc(app):

docs/using/intro.md

@@ -224,20 +224,46 @@ To do so, use the keywords beginning `myst_`.
   - `None`
   - [URI schemes](https://en.wikipedia.org/wiki/List_of_URI_schemes) that will be recognised as external URLs in `[](scheme:loc)` syntax, or set `None` to recognise all.
     Other links will be resolved as internal cross-references.
-* - `myst_html_img`
+* - `myst_html_img_enable`
   - `False`
   - Convert HTML `<img>` elements to sphinx image nodes, see the [image syntax](syntax/images) for details
-* - `myst_math_delimiters`
-  - "dollars"
-  - Delimiters for parsing math, see the [Math syntax](syntax/math) for details
-* - `myst_amsmath_enable`
-  - `False`
-  - Enable direct parsing of [amsmath LaTeX environments](https://ctan.org/pkg/amsmath), [see here](syntax/amsmath)  for details.
 * - `myst_admonition_enable`
   - `False`
   - Enable admonition style directives, [see here](syntax/admonitions) for details.
 `````
 
+Math specific, see the [Math syntax](syntax/math) for more details:
+
+`````{list-table}
+:header-rows: 1
+
+* - Option
+  - Default
+  - Description
+* - `myst_dmath_enable`
+  - `True`
+  - Enable parsing of dollar `$` and `$$` encapsulated math
+* - `myst_dmath_allow_labels`
+  - `True`
+  - Parse `$$...$$ (label)` syntax (if dmath enabled)
+* - `myst_dmath_allow_space`
+  - `True`
+  - If False then inline math will only be parsed if there are no initial/final spaces,
+    e.g. `$a$` but not `$ a$` or `$a $`
+* - `myst_dmath_allow_digits`
+  - `True`
+  - If False then inline math will only be parsed if there are no initial/final digits,
+    e.g. `$a$` but not `1$a$` or `$a$2` (this is useful for using `$` as currency)
+* - `myst_amsmath_enable`
+  - `False`
+  - Enable direct parsing of [amsmath LaTeX environments](https://ctan.org/pkg/amsmath)
+* - `myst_override_mathjax`
+  - `True`
+  - If using [sphinx.ext.mathjax](https://www.sphinx-doc.org/en/master/usage/extensions/math.html#module-sphinx.ext.mathjax) (the default) then `mathjax_config` will be overridden,
+  to ignore `$` delimiters and LaTeX environments, which should instead be handled by
+  `myst_dmath_enable` and `myst_amsmath_enable` respectively.
+`````
+
 ### Disable markdown syntax for the parser
 
 If you'd like to either enable or disable custom markdown syntax, use `myst_disable_syntax`.

docs/using/syntax.md

@@ -31,8 +31,8 @@ described in the [CommonMark Spec](https://spec.commonmark.org/0.29/), which the
 
 Block tokens span multiple lines of content. They are broken down into two sections:
 
-* {ref}`extended-block-tokens` contains *extra* tokens that are not in CommonMark.
-* {ref}`commonmark-block-tokens` contains CommonMark tokens that also work, for reference.
+- {ref}`extended-block-tokens` contains *extra* tokens that are not in CommonMark.
+- {ref}`commonmark-block-tokens` contains CommonMark tokens that also work, for reference.
 
 In addition to these summaries of block-level syntax, see {ref}`extra-markdown-syntax`.
 
@@ -181,8 +181,8 @@ we have shown equivalent rST syntax for many MyST markdown features below.
 Span (or inline) tokens are defined on a single line of content. They are broken down into two
 sections below:
 
-* {ref}`extended-span-tokens` contains *extra* tokens that are not in CommonMark.
-* {ref}`commonmark-span-tokens` contains CommonMark tokens that also work, for reference.
+- {ref}`extended-span-tokens` contains *extra* tokens that are not in CommonMark.
+- {ref}`commonmark-span-tokens` contains CommonMark tokens that also work, for reference.
 
 In addition to these summaries of inline syntax, see {ref}`extra-markdown-syntax`.
 
@@ -648,19 +648,25 @@ header-rows: 1
 
 (syntax/math)=
 
-### Math shortcuts
+## Math shortcuts
 
-The style of math parsing is governed by the `myst_math_delimiters` option set in the sphinx `conf.py` [configuration file](https://www.sphinx-doc.org/en/master/usage/configuration.html).
-The two common settings are:
+Math is parsed by setting, in the sphinx `conf.py` [configuration file](https://www.sphinx-doc.org/en/master/usage/configuration.html) one or both of:
 
-- `myst_math_delimiters = "dollars"` (default)
-  - inline: `$...$` or `$$...$$`
-  - display: `$$...$$`
-  - display + equation label: `$$...$$ (1)`
-- `myst_math_delimiters = "brackets"`
-  - inline: `\(...\)`
-  - display: `\[...\]`
-  - display + equation label: `\[...\] (1)`
+- `myst_dmath_enable=True` (the default) for parsing of dollar `$` and `$$` encapsulated math.
+- `myst_amsmath_enable=True` (off by default) for direct parsing of [amsmath LaTeX environments](https://ctan.org/pkg/amsmath).
+
+These options enable their respective Markdown parser plugins, as detailed in the [markdown-it plugin guide](markdown_it:md/plugins).
+
+### Dollar delimited math
+
+Enabling dollar math will parse the following syntax:
+
+- Inline math: `$...$`
+- Display (block) math: `$$...$$`
+
+Additionally if `myst_dmath_allow_labels=True` is set (the default):
+
+- Display (block) math with equation label: `$$...$$ (1)`
 
 For example, `$x_{hey}=it+is^{math}$` renders as $x_{hey}=it+is^{math}$.
 This is equivalent to writing:
@@ -669,11 +675,14 @@ This is equivalent to writing:
 {math}`x_{hey}=it+is^{math}`
 ```
 
-```{tip}
-Math can be escaped (negated) by adding a `\` before the first symbol, e.g. `\$a$` renders as \$a$.
-```
+:::{admonition,tip} Escaping Dollars
+Math can be escaped (negated) by adding a `\` before the first symbol, e.g. `\$a$` renders as \$a\$.
+Escaping can also be used inside math, e.g. `$a=\$3$` renders as $a=\$3$.
 
-Block-level math can then be provided with `$$` signs that wrap the math block you'd like to parse.
+Conversely `\\` will negate the escaping, so `\\$a$` renders as \\$a$.
+:::
+
+Block-level math can be specified with `$$` signs that wrap the math block you'd like to parse.
 For example:
 
 ```latex
@@ -721,9 +730,17 @@ $$ (eqn:best)
 
 This is the best equation {eq}`eqn:best`
 
+There are a few other options available to control dollar math parsing:
+
+`myst_dmath_allow_space=False`, will cause inline math to only be parsed if there are no initial / final spaces, e.g. `$a$` but not `$ a$` or `$a $`.
+
+`myst_dmath_allow_digits=False`, will cause inline math to only be parsed if there are no initial / final digits, e.g. `$a$` but not `1$a$` or `$a$2`.
+
+These options can both be useful if you also wish to use `$` as a unit of currency.
+
 (syntax/amsmath)=
 
-### Direct LaTeX Math (optional)
+### Direct LaTeX Math
 
 You can enable direct parsing of [amsmath](https://ctan.org/pkg/amsmath) LaTeX equations by setting `myst_amsmath_enable = True` in your sphinx `conf.py`.
 These top-level math environments will then be directly parsed:
@@ -763,9 +780,56 @@ a_{21}& =b_{21}&
 We hope to implement this in a future update (see [executablebooks/MyST-Parser#202](https://github.com/executablebooks/MyST-Parser/issues/202))!
 :::
 
+### Math in other block elements
+
+Math will also work when nested in other block elements, like lists or quotes:
+
+```md
+- $$ a = 1 $$
+- \begin{gather*}
+  a_1=b_1+c_1\\a_2=b_2+c_2-d_2+e_2
+  \end{gather*}
+
+> $$ a = 1 $$
+> \begin{gather*}
+  a_1=b_1+c_1\\a_2=b_2+c_2-d_2+e_2
+  \end{gather*}
+```
+
+- $$ a = 1 $$
+- \begin{gather*}
+  a_1=b_1+c_1\\a_2=b_2+c_2-d_2+e_2
+  \end{gather*}
+
+> $$ a = 1 $$
+> \begin{gather*}
+  a_1=b_1+c_1\\a_2=b_2+c_2-d_2+e_2
+  \end{gather*}
+
+### Mathjax and math parsing
+
+When building HTML using the [sphinx.ext.mathjax](https://www.sphinx-doc.org/en/master/usage/extensions/math.html#module-sphinx.ext.mathjax) extension (enabled by default), its default configuration is to also search for `$` delimiters and LaTeX environments (see [the tex2jax preprocessor](https://docs.mathjax.org/en/v2.7-latest/options/preprocessors/tex2jax.html#configure-tex2jax)).
+
+Since such parsing is already covered by the plugins above, MyST-Parser disables this behaviour by overriding the `mathjax_config` option with:
+
+```python
+mathjax_config = {
+  "tex2jax": {
+  "inlineMath": [["\\(", "\\)"]],
+  "displayMath": [["\\[", "\\]"]],
+  "processRefs": False,
+  "processEnvironments": False,
+  }
+}
+```
+
+Since these delimiters are how `sphinx.ext.mathjax` wraps the math content in the built HTML documents.
+
+To inhibit this override, set `myst_override_mathjax=False`.
+
 (syntax/frontmatter)=
 
-### Front Matter
+## Front Matter
 
 This is a YAML block at the start of the document, as used for example in
 [jekyll](https://jekyllrb.com/docs/front-matter/). Sphinx intercepts these data and
@@ -786,7 +850,7 @@ This is an orphan document, not specified in any toctrees.
 
 (syntax/comments)=
 
-### Comments
+## Comments
 
 You may add comments by putting the `%` character at the beginning of a line. This will
 prevent the line from being parsed into the output document.
@@ -819,7 +883,7 @@ another line
 
 (syntax/blockbreaks)=
 
-### Block Breaks
+## Block Breaks
 
 You may add a block break by putting `+++` at the beginning of a line.
 This constuct's intended use case is for mapping to cell based document formats,
@@ -839,7 +903,7 @@ Is below, but it won't be parsed into the document.
 
 (syntax/targets)=
 
-### Targets and Cross-Referencing
+## Targets and Cross-Referencing
 
 Targets are used to define custom anchors that you can refer to elsewhere in your
 documentation. They generally go before section titles so that you can easily refer
@@ -897,7 +961,7 @@ markdown: `[](syntax.md)` will result in: [](syntax.md).
 
 (syntax/images)=
 
-### Images
+## Images
 
 MyST provides a few different syntaxes for including images in your documentation, as explained below.
 
@@ -935,7 +999,7 @@ The final option is directly using HTML, which is also parsed by MyST.
 This is usually a bad option, because the HTML is treated as raw text during the build process and so sphinx will not recognise that the image file is to be copied, and will not output the HTML into non-HTML output formats.
 
 HTML parsing to the rescue!
-By setting `myst_html_img = True` in the sphinx `conf.py` configuration file, MySt-Parser will attempt to convert any isolated `img` tags (i.e. not wrapped in any other HTML) to the internal representation used in sphinx.
+By setting `myst_html_img_enable = True` in the sphinx `conf.py` configuration file, MySt-Parser will attempt to convert any isolated `img` tags (i.e. not wrapped in any other HTML) to the internal representation used in sphinx.
 
 ```md
 <img src="img/fun-fish.png" alt="fishy" class="bg-primary" width="200px">
@@ -947,7 +1011,7 @@ Allowed attributes are equivalent to the `image` directive: src, alt, class, wid
 Any other attributes will be dropped.
 
 (syntax/footnotes)=
-### Footnotes
+## Footnotes
 
 Footnote labels **start with `^`** and can then be any alpha-numeric string (no spaces),
 which is case-insensitive.

myst_parser/__init__.py

@@ -19,42 +19,45 @@ def setup_sphinx(app):
     # so that it can be called by external packages like myst_nb
     from myst_parser.myst_refs import MystReferenceResolver
     from myst_parser.myst_amsmath import MystAmsMathTransform
+    from myst_parser.main import MdParserConfig
 
     app.add_post_transform(MystReferenceResolver)
     app.add_post_transform(MystAmsMathTransform)
 
-    app.add_config_value("myst_disable_syntax", (), "env")
-    # see https://en.wikipedia.org/wiki/List_of_URI_schemes
-    app.add_config_value("myst_url_schemes", None, "env")
-    app.add_config_value("myst_math_delimiters", "dollars", "env")
-    app.add_config_value("myst_amsmath_enable", False, "env")
-    app.add_config_value("myst_admonition_enable", False, "env")
-    app.add_config_value("myst_html_img", False, "env")
+    for name, default in MdParserConfig().as_dict().items():
+        if not name == "renderer":
+            app.add_config_value(f"myst_{name}", default, "env")
 
-    app.connect("config-inited", validate_config)
+    app.connect("builder-inited", create_myst_config)
 
 
-def validate_config(app, config):
+def create_myst_config(app):
     from sphinx.util import logging
+    from sphinx.util.console import bold
+    from myst_parser.main import MdParserConfig
 
     logger = logging.getLogger(__name__)
 
-    # TODO raise errors or log error with sphinx?
+    values = {
+        name: app.config[f"myst_{name}"]
+        for name in MdParserConfig().as_dict().keys()
+        if name != "renderer"
+    }
+
     try:
-        for s in config.myst_disable_syntax:
-            assert isinstance(s, str)
-    except (AssertionError, TypeError):
-        logger.error("myst_disable_syntax config option not of type List[str]")
-
-    allowed_delimiters = ["brackets", "kramdown", "dollars", "julia"]
-    if config.myst_math_delimiters not in allowed_delimiters:
-        logger.error(
-            "myst_math_delimiters config option not an allowed name: "
-            + f"{allowed_delimiters}"
-        )
-
-    if not isinstance(config.myst_amsmath_enable, bool):
-        logger.error("myst_amsmath_enable config option not of type boolean")
-
-    if not isinstance(config.myst_admonition_enable, bool):
-        logger.error("myst_admonition_enable config option not of type boolean")
+        app.env.myst_config = MdParserConfig(**values)
+        logger.info(bold("myst v%s:") + " %s", __version__, app.env.myst_config)
+    except (TypeError, ValueError) as error:
+        logger.error("myst configuration invalid: %s", error.args[0])
+        app.env.myst_config = MdParserConfig()
+
+    # https://docs.mathjax.org/en/v2.7-latest/options/preprocessors/tex2jax.html#configure-tex2jax
+    if app.env.myst_config.override_mathjax:
+        app.config.mathjax_config = {
+            "tex2jax": {
+                "inlineMath": [["\\(", "\\)"]],
+                "displayMath": [["\\[", "\\]"]],
+                "processRefs": False,
+                "processEnvironments": False,
+            }
+        }

myst_parser/cli/benchmark.py

@@ -77,17 +77,16 @@ def run_myst_parser_html(package, text):
 @benchmark("myst_parser.main")
 def run_myst_parser_docutils(package, text):
     package.to_docutils(
-        text, renderer="docutils", options={"ignore_missing_refs": True}
+        text,
+        package.MdParserConfig(renderer="docutils"),
+        options={"ignore_missing_refs": True},
     )
 
 
 @benchmark("myst_parser.main")
 def run_myst_parser_sphinx(package, text):
     package.to_docutils(
-        text,
-        renderer="sphinx",
-        options={"ignore_missing_refs": True},
-        in_sphinx_env=True,
+        text, options={"ignore_missing_refs": True}, in_sphinx_env=True,
     )
 
 

myst_parser/docutils_renderer.py

@@ -470,7 +470,7 @@ def render_html_inline(self, token):
 
     def render_html_block(self, token):
         node = None
-        if self.config.get("myst_html_img", False):
+        if self.config.get("enable_html_img", False):
             node = HTMLImgParser().parse(token.content, self.document, token.map[0])
         if node is None:
             node = nodes.raw("", token.content, format="html")