fix: decode HTML entities before slugifying header IDs

When a markdown header contains HTML entities (e.g. `# <othertext`),
`header_id_from_text` passed the raw entity string to `_slugify` before
HTML-decoding it.  The `&` and `;` were stripped as non-word characters
but the entity name letters (e.g. `lt`) were kept, silently corrupting
the generated ID (`ltothertext` instead of `othertext`).

Fix: call `html.unescape()` on the header text before slugifying so that
entity characters are resolved to their actual Unicode code points first,
then stripped (or kept) by the slug logic as any other character would be.

Closes #649

Author: gaoflow

lib/markdown2.py

@@ -116,6 +116,7 @@
 __author__ = "Trent Mick"
 
 import argparse
+import html
 import logging
 import re
 import sys
@@ -1578,7 +1579,7 @@ def header_id_from_text(self,
             None to not have an id attribute and to exclude this header from
             the TOC (if the "toc" extra is specified).
         """
-        header_id = _slugify(text)
+        header_id = _slugify(html.unescape(text))
         if prefix and isinstance(prefix, str):
             header_id = prefix + '-' + header_id
 

test/tm-cases/header_ids_entity.html

@@ -0,0 +1,5 @@
+<h1 id="escaped-lt">&lt;escaped-lt</h1>
+
+<h1 id="rd-notes">R&amp;D Notes</h1>
+
+<h1 id="normal-header">Normal Header</h1>

test/tm-cases/header_ids_entity.opts

@@ -0,0 +1 @@
+{"extras": ["header-ids"]}

test/tm-cases/header_ids_entity.text

@@ -0,0 +1,5 @@
+# <escaped-lt
+
+# R&D Notes
+
+# Normal Header