Platinum BLUE converter (#122)

* Non-SigMF Converters
    * Initial code by KelseyCreekSoftware
    * fromfile() now autodetects SigMF, BLUE, & WAV formats automatically
    * allow converters to --archive (.sigmf) or create SigMF pairs (.sigmf-data & .sigmf-meta)
    * Converters now support conversion to non-conforming dataset without
      writing datafiles back to disk
    * BLUE
        * Validated implementation against lots of files beyond nonsigmf-examples repO
        * parse into (fixed, adjunct, extended) keys
        * add CRC validation for bluefile metadata
        * add support for metadata-only (0 sample) BLUE files
    * WAV
        * Drop support for float WAV files; tricky to support NCD
    * Single entry point (sigmf_convert)
    * conversion returns metadata for newly created file
    * enforce checksum calculation
    * account for metadata-only files with heading & trailing bytes
    * ensure for NCD conversion that both input & output are in same folder
    * homologate API for wav & blue converter
* integrate feedback from KelseyCreekSoftware
    * add conversion checks
    * add UX improvements
    * windows path issues
    * require input and output files for converters
* NCD Improvements
    * fixed some usage edge cases with all NCD files
    * Fix bug in sigmffile._count_samples for NCD files
    * Fig bug in read_samples when using some NCD files with header & trailing bytes
* Hashing
    * simplify implementation
    * add related tests
    * move from sigmf_hash to hashing -> API change
* Docs
    * new docs for converters
    * simplify doc build procedure
    * rewrite general use patterns
    * added support table for BLUE format codes
* Tests
    * add tests for blue (.cdif) and wav (.wav) converters
    * added tests for all supported BLUE types
    * split converter tests into separate files
    * allow epsilon differences due to float conversion patterns
    * simplify wav tests
* Project
    * Increment to v1.6.0
    * Simplified install: drop scipy optional dependency and [apps] entirely
    * Add utils.get_magic_bytes() for autodetection purposes

Author: Teque5

.github/workflows/main.yml

@@ -22,7 +22,7 @@ jobs:
       - name: Install dependencies
         run: |
           python -m pip install --upgrade pip
-          pip install .[test,apps]
+          pip install .[test]
       - name: Test with pytest
         run: |
           coverage run

.gitignore

@@ -3,6 +3,7 @@ __pycache__/
 *.swp
 *.py[cod]
 .cache
+.vscode
 
 # packaging related
 dist/

.readthedocs.yaml

@@ -18,7 +18,6 @@ python:
       path: .
       extra_requirements:
         - test
-        - apps
     - requirements: docs/requirements.txt
 
 # Build documentation in the "docs/" directory with Sphinx

README.md

@@ -12,10 +12,28 @@ freely under the terms GNU Lesser GPL v3 License.
 
 This module follows the SigMF specification [html](https://sigmf.org/)/[pdf](https://sigmf.github.io/SigMF/sigmf-spec.pdf) from the [spec repository](https://github.com/sigmf/SigMF).
 
-To install the latest PyPI release, install from pip:
+### Install
 
 ```bash
 pip install sigmf
 ```
 
-**[Please visit the documentation for examples & more info.](https://sigmf.readthedocs.io/en/latest/)**
+### Read SigMF
+
+```python
+import sigmf
+
+# read SigMF recording
+meta = sigmf.fromfile("recording.sigmf-meta")
+samples = meta[0:1024]  # get first 1024 samples
+sample_rate = meta.sample_rate  # get sample rate
+
+
+# read other formats containing RF time series as SigMF
+meta = sigmf.fromfile("recording.wav")   # WAV
+meta = sigmf.fromfile("recording.cdif")  # BLUE / Platinum
+```
+
+### Docs
+
+**[Please visit our documentation for full API reference and more info.](https://sigmf.readthedocs.io/en/latest/)**

docs/Makefile

@@ -12,7 +12,11 @@ BUILDDIR      = build
 help:
 	@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
 
-.PHONY: help Makefile
+clean:
+	rm -rf "$(BUILDDIR)"
+	rm -rf "$(SOURCEDIR)/_autosummary"
+
+.PHONY: help clean Makefile
 
 # Catch-all target: route all unknown targets to Sphinx using the new
 # "make mode" option.  $(O) is meant as a shortcut for $(SPHINXOPTS).

docs/source/api.rst

@@ -7,12 +7,13 @@ SigMF API
    :template: custom-module-template.rst
    :recursive:
 
-   sigmf.apps.convert_wav
    sigmf.archive
    sigmf.archivereader
+   sigmf.convert.blue
+   sigmf.convert.wav
    sigmf.error
    sigmf.schema
-   sigmf.sigmf_hash
+   sigmf.hashing
    sigmf.sigmffile
    sigmf.utils
    sigmf.validate

docs/source/converters.rst

@@ -0,0 +1,171 @@
+==========
+Converters
+==========
+
+The SigMF Python library includes converters to import data from various RF recording formats into SigMF.
+Converters can create standard SigMF file pairs or Non-Conforming Datasets (NCDs) that reference the original files.
+
+Overview
+--------
+
+Conversion is available for:
+
+* **BLUE files** - MIDAS Blue and Platinum BLUE RF recordings (usually ``.cdif``)
+* **WAV files** - Audio recordings (``.wav``)
+
+All converters return a :class:`~sigmf.SigMFFile` object with converted metadata.
+
+
+Fromfile Auto-Detection
+~~~~~~~~~~~~~~~~~~~~~~~
+
+The :func:`~sigmf.sigmffile.fromfile` function automatically detects input file
+formats and reads without writing any output files:
+
+.. code-block:: python
+
+    import sigmf
+
+    # auto-detect and create NCD for any supported format
+    meta = sigmf.fromfile("recording.cdif")  # BLUE file
+    meta = sigmf.fromfile("recording.wav")   # WAV file
+    meta = sigmf.fromfile("recording.sigmf")  # SigMF archive
+
+    all_samples = meta.read_samples()
+    sample_rate = meta.sample_rate
+
+
+Python API
+~~~~~~~~~~~
+
+For programmatic access, use the individual converter functions directly:
+
+.. code-block:: python
+
+    from sigmf.convert.wav import wav_to_sigmf
+    from sigmf.convert.blue import blue_to_sigmf
+
+    # convert WAV to SigMF archive
+    _ = wav_to_sigmf(wav_path="recording.wav", out_path="recording", create_archive=True)
+
+    # convert BLUE to SigMF pair and return metadata for new files
+    meta = blue_to_sigmf(blue_path="recording.cdif", out_path="recording")
+
+
+Command Line Usage
+~~~~~~~~~~~~~~~~~~
+
+Converters are accessed through a unified command-line interface that automatically detects file formats:
+
+.. code-block:: bash
+
+    # unified converter
+    sigmf_convert input_file output_file
+
+    # examples
+    sigmf_convert recording.cdif recording.sigmf
+    sigmf_convert recording.wav recording.sigmf
+
+The converter uses magic byte detection to automatically identify BLUE and WAV file formats.
+No need to remember format-specific commands!
+
+
+Output Options
+~~~~~~~~~~~~~~
+
+The unified converter supports multiple output modes:
+
+.. code-block:: bash
+
+    # standard conversion (creates out.sigmf-data and out.sigmf-meta files)
+    sigmf_convert in.wav out
+
+    # archive mode (creates single out.sigmf archive)
+    sigmf_convert in.wav out --archive
+
+    # non-conforming dataset (creates out.sigmf-meta only, references original file)
+    sigmf_convert in.wav out --ncd
+
+    # extra verbose output
+    sigmf_convert in.wav out -vv
+
+**Important**: When using ``--ncd``, the input and output files must be in the same directory.
+This ensures proper relative path references in the metadata.
+
+
+BLUE Converter
+--------------
+
+The BLUE converter handles CDIF (.cdif) recordings while placing BLUE header information into the following global fields:
+
+* ``blue:fixed`` - Fixed header information (at start of file).
+* ``blue:adjunct`` - Adjunct header information (after fixed header).
+* ``blue:extended`` - Extended header information (at end of file). Note any duplicate fields will have a suffix like ``_1``, ``_2``, etc appended.
+
+.. autofunction:: sigmf.convert.blue.blue_to_sigmf
+
+Examples
+~~~~~~~~
+
+.. code-block:: python
+
+    from sigmf.convert.blue import blue_to_sigmf
+
+    # standard conversion
+    meta = blue_to_sigmf(blue_path="recording.cdif", out_path="recording")
+
+    # create NCD automatically (metadata-only, references original file) but don't save any output file
+    meta = blue_to_sigmf(blue_path="recording.cdif")
+
+    # access standard SigMF data & metadata
+    all_samples = meta.read_samples()
+    sample_rate = meta.sample_rate
+
+    # access BLUE-specific metadata
+    blue_type = meta.get_global_field("blue:fixed")["type"]  # e.g., 1000
+    blue_version = meta.get_global_field("blue:fixed")["keywords"]["IO"]  # e.g., "X-Midas"
+
+Tested Formats
+~~~~~~~~~~~~~~
+
+BLUE files use a 2-digit format code where the first is the sample type (row) and the second is the sample size (column).
+For example ``SB`` contains real values with 8 bits per sample and ``CF`` contains complex values with 32 bits per component (64 bits per sample).
+
+The following table summarizes tested BLUE formats and their compatibility with the converter:
+
+.. csv-table::
+    :header-rows: 1
+    :stub-columns: 1
+
+    "Code", ":abbr:`B (int8)`", ":abbr:`I (int16)`", ":abbr:`L (int32)`", ":abbr:`X (int64)`", ":abbr:`F (float32)`", ":abbr:`D (float64)`", ":abbr:`P (packed)`", ":abbr:`N (int4)`"
+    ":abbr:`S (scalar)`", "✅", "✅", "✅", "✅", "✅", "✅", "❌", "❌"
+    ":abbr:`C (complex)`", "✅", "✅", "✅", "✅", "✅", ""✅", "❌", "❌"
+
+**Legend:**
+    * ✅ = Tested and known working
+    * ❌ = Unsupported
+
+
+WAV Converter
+-------------
+
+Converts WAV audio recordings to SigMF format.
+
+.. autofunction:: sigmf.convert.wav.wav_to_sigmf
+
+Examples
+~~~~~~~~
+
+.. code-block:: python
+
+    from sigmf.convert.wav import wav_to_sigmf
+
+    # standard conversion
+    meta = wav_to_sigmf(wav_path="recording.wav", out_path="recording")
+
+    # create NCD automatically (metadata-only, references original file)
+    meta = wav_to_sigmf(wav_path="recording.wav")
+
+    # access standard SigMF data & metadata
+    all_samples = meta.read_samples()
+    sample_rate_hz = meta.sample_rate

docs/source/developers.rst

@@ -60,9 +60,9 @@ To build the docs and host locally:
 .. code-block:: console
 
    $ cd docs
+   $ make clean
    $ make html
-   $ cd build/html/
-   $ python3 -m http.server
+   $ python3 -m http.server --directory build/html/
 
 --------------
 Find an Issue?

docs/source/index.rst

@@ -8,11 +8,12 @@ It offers a *simple* and *intuitive* API for Python developers.
 
 ..
    Note: The toolversion & specversion below are replaced dynamically during build.
+   The root __init__.py file is used as the sole source of truth for these values.
 
 This documentation is for version |toolversion| of the library, which is
 compatible with version |specversion| of the SigMF specification.
 
-To get started, see the :doc:`quickstart` section or learn how to :ref:`install` the library.
+To get started, see `quickstart`.
 
 -----
 
@@ -23,6 +24,7 @@ To get started, see the :doc:`quickstart` section or learn how to :ref:`install`
 
    quickstart
    advanced
+   converters
    developers
 
 .. toctree::

pyproject.toml

@@ -33,17 +33,14 @@ dependencies = [
 
     [project.scripts]
     sigmf_validate = "sigmf.validate:main"
-    sigmf_convert_wav = "sigmf.apps.convert_wav:main [apps]"
+    sigmf_convert = "sigmf.convert.__main__:main"
     [project.optional-dependencies]
     test = [
         "pylint",
         "pytest",
         "pytest-cov",
         "hypothesis",   # next-gen testing framework
     ]
-    apps = [
-        "scipy",        # for wav i/o
-    ]
 
 [tool.setuptools]
 packages = ["sigmf"]
@@ -106,6 +103,6 @@ legacy_tox_ini = '''
 
     [testenv]
     usedevelop = True
-    deps = .[test,apps]
+    deps = .[test]
     commands = coverage run
 '''