Last changes before 0.6.0

	modified:   README.rst
	modified:   doc/conf.py
	new file:   doc/figures/pypeit_spec2d_noise_postscrub.png
	new file:   doc/figures/pypeit_spec2d_noise_prescrub.png
	new file:   doc/figures/scrubber_line_fitting.png
	modified:   doc/include/dependencies_table.rst
	modified:   doc/scrub_deveny_pickup.rst
	modified:   obstools/scrub_deveny_pickup.py
	modified:   setup.cfg

Author: tbowers7

README.rst

@@ -110,7 +110,7 @@ action is to setup a clean python environment into which the installation will
 occur.  This mitigates any possible dependency conflicts with other packages
 you use.
 
-The recommended method of setting up a new envrionment is with ``conda``:
+The recommended method of setting up a new environment is with ``conda``:
 
 .. code-block:: console
 

doc/conf.py

@@ -47,6 +47,7 @@
     "sphinx.ext.mathjax",
     "sphinx.ext.ifconfig",
     "sphinx.ext.viewcode",
+    "sphinx_subfigure",
     # "sphinx.ext.autosectionlabel",
 ]
 

doc/figures/pypeit_spec2d_noise_postscrub.png

@@ -1,6 +1,6 @@
-================================  ===========================================================================================================================================================================
-Python Version                    ``>=3.10,<3.12``                                                                                                                                                           
-Required for users                ``astropy>=5.1``, ``ccdproc``, ``darkdetect``, ``matplotlib``, ``numpy>=1.22``, ``pysimplegui``, ``requests``, ``scipy>=1.9``, ``setuptools``, ``setuptools_scm``, ``tqdm``
-Optional ``pypeit`` requirements  ``pypeit[specutils]>=1.14.0``                                                                                                                                              
-Required for developers           ``black``, ``pylint``, ``pypeit[specutils]>=1.14.0``, ``pyyaml``, ``sphinx``, ``sphinx-automodapi``, ``sphinx_rtd_theme==1.2.2``, ``stomp.py``, ``xmltodict``              
-================================  ===========================================================================================================================================================================
+================================  ===================================================================================================================================================================================
+Python Version                    ``>=3.10,<3.12``                                                                                                                                                                   
+Required for users                ``astropy>=5.1``, ``ccdproc``, ``darkdetect``, ``matplotlib``, ``numpy>=1.22``, ``pysimplegui``, ``requests``, ``scipy>=1.9``, ``setuptools``, ``setuptools_scm``, ``tqdm``        
+Optional ``pypeit`` requirements  ``pypeit[specutils]>=1.14.0``                                                                                                                                                      
+Required for developers           ``black``, ``pylint``, ``pypeit[specutils]>=1.14.0``, ``pyyaml``, ``sphinx``, ``sphinx-automodapi``, ``sphinx-subfigure``, ``sphinx_rtd_theme==1.2.2``, ``stomp.py``, ``xmltodict``
+================================  ===================================================================================================================================================================================

doc/figures/pypeit_spec2d_noise_prescrub.png

@@ -60,6 +60,9 @@ signal.  This patten is then subtracted from the original image and saved to
 a FITS file for usual data reduction processing (preferably with PypeIt) to
 yield an extracted 1D spectrum for analysis.
 
+Example Pre- and Post-Scrubbed Data Products
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
 To illustrate the need for and utility of this tool, :numref:`spec1d_comps`
 shows a comparison of the extracted 1D spectra for two different object types
 from two different programs (and nights) from both the original and scrubbed
@@ -81,6 +84,36 @@ from the spectra.
    estimates a mean :math:`SNR = 4.8` for the scrubbed galaxy spectrum and
    a mean :math:`SNR = 7.2` for the scrubbed white dwarf spectrum.
 
+Of equivalent interest to the quality of the extracted spectra is the noise
+remaining after extraction of sky and objects.  Shown in :numref:`prepost_noise`
+are the noise analysis plots from PyepIt for the pre- and post-scrubbed
+versions of the 2D spectral image shown in :numref:`raw_frame`.  The images and
+pixel histograms are of the residual noise image, which is the science image
+minus the object and sky models, and then divided by the uncertainty image.
+The ideal pixel histogram would be a gaussian with width :math:`\sigma=1`.
+
+.. _prepost_noise:
+.. subfigure:: A|B
+   :gap: 2px
+   :class-grid: outline
+
+   .. image:: figures/pypeit_spec2d_noise_prescrub.png
+      :alt: Pre-scrubbed noise analysis plot
+
+   .. image:: figures/pypeit_spec2d_noise_postscrub.png
+      :alt: Post-scrubbed noise analysis plot
+
+   -- Noise analysis of the pre- (top) and post-scrubbed (bottom) versions of
+   the frame shown in :numref:`raw_frame`.  Image values are residual image
+   divided by uncertainty, so the ideal pixel histogram would be a gaussian
+   with width :math:`\sigma=1`.  Note the improvement in both the visual
+   quality of the scrubbed frame and the width and shape of the pixel histogram
+   compared to the pre-scrubbed frame.
+
+
+Outline
+^^^^^^^
+
 This document begins with a description of how to use this tool to clean the
 EMI pickup noise from your data, and moves on to lay out the details of what
 the tool does to your data and other points to consider.
@@ -493,11 +526,11 @@ than that predicted from the FFT (green dashed line in the second panel of
 Pickup Noise Pattern Construction
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-Finally, we use the sinusoid fits to produce a pattern image.  This is the
+The final result of sinusoid fits is a constructed pattern image.  This is the
 zero-mean sinusoid (sans quadratic polynomial) that *should* represent only the
-EMI pickup noise (as an additive AC-only signal).  Subtract that and make
-additional QA plots to show how totally awesome this is!!!
-
+EMI pickup noise (as an additive AC-only signal).  :numref:`image_comparisons`
+shows the process of pattern construction and its effect on the processed
+science image.
 
 .. _image_comparisons:
 .. figure:: figures/scrubber_image_comparisons.*
@@ -629,3 +662,14 @@ panel in :numref:`image_comparisons`).  The result is shown in
    discussed above) from artifacts and ringing in the FFT.  The peak at 169.1
    pixels, indicating the frequency with the most power in the flattened array:
    most likely the period of the AC EMI pickup noise.
+
+Also, say something about the actual line-by-line fits in terms of how good a
+sinusoid fits each one.  Show a figure like :numref:`line_by_line`.
+
+.. _line_by_line:
+.. figure:: figures/scrubber_line_fitting.png
+   :class: with-shadow
+   :alt: Line-by-line fitting examples
+
+   -- Examples of individual line fits for 4 randomly selected lines from the
+   image shown in :numref:`raw_frame`.

doc/figures/scrubber_line_fitting.png

@@ -131,7 +131,7 @@ def iterative_pypeit_clean(
     try:
         # Look for the spec2d file
         spec2d_file = [
-            next(d.joinpath("Science").glob(f"spec2d_{filename.stem}-*"))
+            next(d.joinpath("Science").glob(f"spec2d_{filename.stem}-*.fits"))
             for d in pyp_dir
         ][0]
     except (StopIteration, IndexError):

doc/include/dependencies_table.rst

@@ -48,6 +48,7 @@ install_requires =
 docs =
     sphinx
     sphinx-automodapi
+    sphinx-subfigure
     sphinx_rtd_theme==1.2.2
 
 broker =
@@ -61,6 +62,7 @@ pypeit =
 dev =
     sphinx
     sphinx-automodapi
+    sphinx-subfigure
     sphinx_rtd_theme==1.2.2
     pylint
     black