README-Unicode.md: Fix Markdown format for linting and update it

Signed-off-by: Bernhard Kaindl <[email protected]>

Author: bernhardkaindl

README-Unicode.md

@@ -1,40 +1,64 @@
-Python3 Unicode migration in the XCP package
-============================================
+# Python3 Unicode migration in the XCP package
+
+## Problem
+
+Python3.6 on XS8 does not have an all-encompassing default UTF-8 mode for I/O.
+
+Newer Python versions have an UTF-8 mode that they even enable by default.
+Python3.6 only enabled UTF-8 for I/O when an UTF-8 locale is used.
+See below for more background info on the UTF-8 mode.
+
+For situations where UTF-8 enabled, we have to specify UTF-8 explicitly.
+
+Such sitation happens when LANG or LC_* variables are not set for UTF-8.
+XAPI plugins like auto-cert-kit find themself in this situation.
+
+Example:
+For reading UTF-8 files like the `pciids` file, add `encoding="utf-8"`.
+This applies especailly to `open()` and `Popen()` when files my contain UTF-8.
+
+This also applies when en/decoding to/form `urllib` which uses bytes.
+`urllib` has to use bytes as HTTP data can of course also be binary, e.g. compressed.
+
+## Migrating `subprocess.Popen()`
 
-Migrating `subprocess.Popen()`
-------------------------------
 With Python3, the `stdin`, `stdout` and `stderr` pipes for `Popen()` default to `bytes`(binary mode). Binary mode is much safer because it foregoes the encode/decode.
 
 The for working with strings, existing users need to either enable text mode (when safe, it will attempt to decode and encode!) or be able to use bytes instead.
 
 For cases where the data is guaranteed to be pure ASCII, such as when resting the `proc.stdout` of `lspci -nm`, it is sufficient to use:
+
 ```py
 open(["lspci, "-nm"], stdout=subprocess.PIPE, universal_newlines=True)
 ```
+
 This is possible because `universal_newlines=True` is accepted by Python2 and Python3.
 On Python3, it also enables text mode for `subprocess.PIPE` streams (newline conversion
 not needed, but text mode is needed)
 
-Migrating `builtins.open()`
----------------------------
+## Migrating `builtins.open()`
+
 On Python3, `builtins.open()` can be used in a number of modes:
+
 - Binary mode (when `"b"` is in `mode`): `read()` and `write()` use `bytes`.
 - Text mode (Python3 default up to Python 3.6), when UTF-8 character encoding is not set by the locale
-- UTF-8 mode (default since Python 3.7): https://peps.python.org/pep-0540/
+- UTF-8 mode (default since Python 3.7): <https://peps.python.org/pep-0540/>
 
 When no Unicode locale in force, like in XAPI plugins, Python3 will be in text mode or UTF-8 (since Python 3.7, but existing XS is on 3.6):
 
-* By default, `read()` on files `open()`ed without selecting binary mode attempts
+- By default, `read()` on files `open()`ed without selecting binary mode attempts
   to decode the data into the Python3 Unicode string type.
   This fails for binary data.
   Any `ord() >= 128`, when no UTF-8 locale is active With Python 3.6, triggers `UnicodeDecodeError`.
 
-* Thus, all `open()` calls which might open binary files have to be converted to binary
+- Thus, all `open()` calls which might open binary files have to be converted to binary
   or UTF-8 mode unless the caller is sure he is opening an ASCII file.
   But even then, enabling an error handler to handle decoding errors is recommended:
+
   ```py
   open(filename, errors="replace")
   ```
+
   But neither `errors=` nor `encoding=` is accepted by Python2, so a wrapper is likely best.
 
 ### Binary mode
@@ -43,21 +67,18 @@ When decoding bytes to strings is not needed, binary mode can be great because i
 
 However, when strings need to returned from the library, something like `bytes.decode(errors="ignore")` to get strings is needed.
 
-### Text mode
-
-Text mode using the `ascii` codec should be only used when it is ensured that the data can only consist of ASCII characters (0-127). Sadly, it is the default in Python 3.6 when the Python interpreter was not started using an UTF-8 locale for the LC_CTYPE locale category (set by LC_ALL, LC_CTYPE, LANG environment variables, overriding each other in that order) 
-
 ### UTF-8 mode
 
 Most if the time, the `UTF-8` codec should be used since even simple text files which are even documented to contain only ASCII characters like `"/usr/share/hwdata/pci.ids"` in fact __do__ contain UTF-8 characters.
 
 Some files or some output data from commands even contains legacy `ISO-8859-1` chars, and even the `UTF-8` codec would raise `UnicodeDecodeError` for these.
 When this is known to be the case, `encoding="iso-8859-1` could be tried (not tested yet).
 
-###  Problems
+### Problems
 
 With the locale set to C (XAPI plugins have that), Python's default mode changes
 between 3.6 and 3.7:
+
 ```py
 for i in 3.{6,7,10,11};do echo -n "3.$i: ";
    LC_ALL=C python3.$i -c 'import locale,sys;print(locale.getpreferredencoding())';done
@@ -66,7 +87,9 @@ for i in 3.{6,7,10,11};do echo -n "3.$i: ";
 3.10: UTF-8
 3.11: utf-8
 ```
+
 This has the effect that in Python 3.6, the default codec for XAPI plugins is `ascii`:
+
 ```py
 for i in 2.7 3.{6,7};do echo "$i:";
   LC_ALL=C python$i -c 'open("/usr/share/hwdata/pci.ids").read()';done
@@ -79,13 +102,15 @@ Traceback (most recent call last):
 UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 97850: ordinal not in range(128)
 3.7:
 ```
+
 This error means that the `'ascii' codec` cannot handle input ord() >= 128, and as some Video cards use `²` to reference their power, the `ascii` codec chokes on them.
 
 It means `xcp.pci.PCIIds()` cannot use `open("/usr/share/hwdata/pci.ids").read()`.
 
 While Python 3.7 and newer use UTF-8 mode by default, it does not set up an error handler for `UnicodeDecodeError`.
 
 As it happens, some older tools output ISO-8859-1 characters hard-coded and these aren't valid UTF-8 sequences, and even newer Python versions need error handlers to not fail:
+
 ```py
 echo -e "\0262"  # ISO-8859-1 for: "²"
 python3 -c 'open(".text").read()'
@@ -133,6 +158,7 @@ tests/test_bootloader.py line 38 in TestLinuxBootloader.setUp()
 tests/test_pci.py line 96 in TestPCIIds.test_videoclass_by_mock_calls()
 tests/test_pci.py line 110 in TestPCIIds.mock_lspci_using_open_testfile()
 ```
+
 Of course, `xcp/net/ifrename` won't be affected but it would be good to fix the
 warning for them as well in an intelligent way. See the proposal for that below.
 
@@ -141,6 +167,7 @@ arguments we need to pass to ensure that all users of open() will work, we need
 to make passing the arguments conditional on Python >= 3.
 
 1. Overriding `open()`, while technically working would not only affect xcp.python but the entire program:
+
     ```py
     if sys.version_info >= (3, 0):
         original_open = __builtins__["open"]
@@ -152,7 +179,9 @@ to make passing the arguments conditional on Python >= 3.
                 return original_open(*args, **kwargs)
             __builtins__["open"] = uopen
     ```
+
 2. This is sufficient but is not very nice:
+
     ```py
     # xcp/utf8mode.py
     if sys.version_info >= (3, 0):
@@ -165,9 +194,11 @@ to make passing the arguments conditional on Python >= 3.
     - open(filename)
     + open(filename, **open_utf8args)
     ```
+
    But, `pylint` will still warn about these lines, so I propose:
 
 3. Instead, use a wrapper function, which will also silence the `pylint` warnings at the locations which have been changed to use it:
+
     ```py
     # xcp/utf8mode.py
     if sys.version_info >= (3, 0):
@@ -189,4 +220,4 @@ Using the 3rd option, the `pylint` warnings for the changed locations
 explicitly disabling them.
 
 PS: Since utf8open() still returns a context-manager, `with open(...) as f:`
-would still work.
+would still work.