Updating existing site detail

Makefile

@@ -7,6 +7,9 @@ SPHINXBUILD   = sphinx-build
 PAPER         =
 BUILDDIR      = _build
 
+CURRENT_DOC_DIR = ./static_root/doc
+TEMP_TEST_DOC_DIR = ./doc
+
 # User-friendly check for sphinx-build
 ifeq ($(shell which $(SPHINXBUILD) >/dev/null 2>&1; echo $$?), 1)
 $(error The '$(SPHINXBUILD)' command was not found. Make sure you have Sphinx installed, then set the SPHINXBUILD environment variable to point to the full path of the '$(SPHINXBUILD)' executable. Alternatively you can add the directory with the executable to your PATH. If you don't have Sphinx installed, grab it from http://sphinx-doc.org/)
@@ -166,7 +169,46 @@ changes:
 	@echo "The overview file is in $(BUILDDIR)/changes."
 
 linkcheck:
-	$(SPHINXBUILD) -b linkcheck $(ALLSPHINXOPTS) $(BUILDDIR)/linkcheck
+	# For some reason, the doc directory for our sources is set to
+	# ./static_root/doc but Sphinx expects it to be at ./doc and this has yet to
+	# be addressed. The effect of this discontinuity is that when Sphinx is
+	# running, it looks in the incorrect location with relative paths for static
+	# files we are serving up. This ultimately is not an issue for the
+	# functionality of the website as the links are indeed consistent in the
+	# static files generated, but it causes tests such as `linkcheck` to fail.
+	# For this reason, this make command has been modified to accoun for this
+	# discontinuity to preserve the validity of the test, but this change may
+	# have impact on future modifications to the website's internal structure
+	# and/or a fix for this issue, hence the in-line documentation of the problem
+	# (which gets printed to the console upon `linkcheck`'s failure).
+	#
+	# Here's what the make command was prior to this modification (and likely
+	# should be converted back to after addressing the issue):
+	#
+	# 	$(SPHINXBUILD) -b linkcheck $(ALLSPHINXOPTS) $(BUILDDIR)/linkcheck
+	# 	@echo
+	# 	@echo "Link check complete; look for any errors in the above output " \
+	# 	      "or in $(BUILDDIR)/linkcheck/output.txt."
+	#
+	# Here's an overview of the logic for circumventing the issue:
+	#
+	# (1) Attempt to create the symlink to the actual doc directory in the
+	#     location that Sphinx expects it to be.
+	#     => Success
+	#         (a) Run linkcheck as we've properly configured our local structure
+	#             to be what Sphinx expects.
+	#         (b) Delete the symlink hack/patch, regardless of `linkcheck`'s
+	#             success as we know we create ./doc as a symlink in (1) (as
+	#             opposed to it being a directory containing its own data
+	#             possibly created at a later time without knowledge of this
+	#             fix).
+	#     => Failure
+	#         (a) Do no proceed and let this message be printed for future
+	#             debugging/maintainance effors.
+	test -h  $(TEMP_TEST_DOC_DIR) \
+	|| ln -sT $(CURRENT_DOC_DIR) $(TEMP_TEST_DOC_DIR) \
+	&& $(SPHINXBUILD) -b linkcheck $(ALLSPHINXOPTS) $(BUILDDIR)/linkcheck \
+	;  rm $(TEMP_TEST_DOC_DIR)
 	@echo
 	@echo "Link check complete; look for any errors in the above output " \
 	      "or in $(BUILDDIR)/linkcheck/output.txt."

README.md

@@ -1,11 +1,160 @@
 diffpy.github.io
 ================
 
-Sphinx sources for the diffpy.org web page.  This repository contains 3 branches:
+Sphinx sources for the [diffpy.org][site] web page.
 
-1. `source` sphinx sources for the web page
-2. `master` sources compiled to HTML format that are published via GitHub pages
-3. `archive` converted subversion repository for the old diffpy.org sources
+### Quick Jump:
+
+- [GitHub Structure](#github-structure)
+- [Where to Make Changes](#where-to-make-changes)
+  - [Adding Citations](#adding-citations)
+    - [Adding Publications that Describe a DiffPy Project (the "Reference" Section)](#reference-section-1)
+    - [Adding Other Publications (the "Publication Using DiffPy-CMI" Section)](#reference-section-2)
+- [Publishing New Version of Existing Project](#new-version)
+- [Publishing New Project](#publishing-new-project)
+- [Publishing Changes](#publishing-changes)
+  - [Testing Changes](#testing-changes)
+  - [Contributors (Submitting Changes)](#submitting-changes)
+  - [Maintainers (Publishing Changes)](#publishing-changes)
+
+
+# GitHub Structure
+
+This repository contains 3 branches:
+
+1. [`source`][source] sphinx sources for the web page
+2. [`master`][master] sources compiled to HTML format that are published via GitHub pages
+3. [`archive`][archive] converted subversion repository for the old diffpy.org sources
 
 Please, see the [Wiki](https://github.com/diffpy/diffpy.github.io/wiki) for more
 information and tips about this project.
+
+
+# Where to Make Changes
+
+## Adding Citations
+
+<a name="reference-section-1"></a>
+### Adding Publications that Describe a DiffPy Project (the "Reference" Section)
+
+When adding a new publication to the list of [references used within the website](https://www.diffpy.org/publications.html), add the citation text of the publication to the documentation *only once* as a named snippet in [abbreviations.txt](https://github.com/diffpy/diffpy.github.io/blob/source/abbreviations.txt#L286). For example:
+```
+.. |citeJuhasJac18| replace::
+   P. Juhás, J. N. Louwen, L. van Eijck, E. T. C. Vogt, S. J. L. Billinge,
+   `PDFgetN3: atomic pair distribution functions from
+   neutron powder diffraction data using ad hoc corrections
+   <https://doi.org/10.1107/S1600576718010002>`__,
+   *J. Appl. Crystallogr.*, **51**, 1492--1497 (2018)
+   |downloadJuhasJac18|
+```
+Here, `|citeJuhasJac18|` is the name that can be used elsewhere in the documentation and the Sphinx documentation generator will replace all instances of this tag with the indented text following the `replace::` directive.
+
+**Important:** After defining the publication's tag as described above, make sure to add the publication to the list of publications maintained in [publications.rst](https://github.com/diffpy/diffpy.github.io/blob/source/publications.rst). Make sure that you add the reference to the proper section and do so in descending reverse chronological order (i.e., the newest citations should appear at the top of their respective sections).
+
+<a name="reference-section-note">*Note:*</a> In this example, the citation is for a publication which describes a product of the DiffPy-CMI project (namely, PDFgetN3). For publications which describe a component of DiffPy-CMI, we provide a link to download the publication directly from the [diffpy.org][site] website. Here, the link is provided via the `|downloadJuhasJac18|` tag which is the identifier for another snippet within [abbreviations.txt](https://github.com/diffpy/diffpy.github.io/blob/source/abbreviations.txt#L294) following the definition of `|citeJuhasJac18|`, seen here as:
+```
+.. |downloadJuhasJac18| image:: /images/pdficon_small.png
+   :target: /doc/pdfgetx/Juhas-jac-2018.pdf
+```
+Furthermore, note that since the initial use of `|downloadJuhasJac18|` occurs within the indented text of the definition of `|citeJuhasJac18|`, the link to `|downloadJuhasJac18|` will appear everywhere `|citeJuhasJac18|` does.
+
+Lastly, make sure that the linked publication has been included within this project's files, placed in an appropriate directory (typically, the parent directory of the project that the paper relates to). In the above example, the publication covers the PDFgetN3 feature of the `pdfgetx` package, thus it is placed within `pdfgetx`'s documentation directory and referenced accordingly with the line
+```
+   :target: /doc/pdfgetx/Juhas-jac-2018.pdf
+```
+
+
+<a name="reference-section-2"></a>
+### Adding Other Publications (the "Publication Using DiffPy-CMI" Section)
+
+Adding references to publications that do not describe the release/use of a product within the DiffPy-CMI project (e.g., papers which use some component of DiffPy-CMI), we simply provide the usual citation text (with appropriate DOI link). To add a citation of this type, refer to the information in [Reference Section](#reference-section-1), but disregard everything starting at, and following, the [Note](#reference-section-note).
+
+
+<a name="new-version"></a>
+# Publishing New Version of Existing Project
+
+For releasing an updated version of a project, first follow the release procedure outlined in the [group wiki](https://gitlab.thebillingegroup.com/resources/group-wiki/-/wikis/Finalizing-a-Project's-(Re)-Release) for packaging and deploying your project.
+
+In what follows, [pdfgetx v2.1.1](https://github.com/diffpy/diffpy.pdfgetx/releases/tag/v2.1.1) was chosen to show example commands for the steps outlined.
+
+After following the steps necessary for releasing your project, grab the set of documentation that will be provided with the deliverable upon user's request (typically, this would be something like the files hosted by GitHub on the GitHub releases page). Create a new directory for the updated version's documentation to live in
+```
+$ mkdir diffpy.github.io/doc/pdfgetx/2.1.1
+```
+Copy the updated documentation to this directory
+```
+$ tar --directory=static_root/doc/pdfgetx/2.1.1/ \
+  --strip-components=4 \
+  -vxzf ~/Downloads/diffpy.pdfgetx-2.1.1.tar.gz \
+  diffpy.pdfgetx-2.1.1/doc/manual/_build/PDFgetXNS3_manual.pdf
+
+$ tar --directory=static_root/doc/pdfgetx/2.1.1/ \
+  --strip-components=5 \
+  --exclude=objects.inv \
+  --exclude=.buildinfo \
+  -vxzf ~/Downloads/diffpy.pdfgetx-2.1.1.tar.gz \
+  diffpy.pdfgetx-2.1.1/doc/manual/_build/html/
+
+$ cp ~/Downloads/pdfgetxn3-examples.zip static_root/doc/pdfgetx/2.1.1/
+```
+Make sure to include all relevant files (e.g., `PDFgetXNS3_manual.pdf` and `pdfgetxn3-examples.zip`), exclude any files we don't want to provide to the user (e.g., `objects.inv` and `.buildinfo` from `diffpy.pdfgetx-2.1.1.tar.gz`)
+
+Finally, edit the [landing page of your project](https://github.com/diffpy/diffpy.github.io/blob/source/products/pdfgetx.rst) (in the [`source` branch][source]) to properly document and provide the updated version of your project. Once the preceding steps are complete, see the instructions of the [Publishing Changes](#submitting-changes) section for publishing these changes.
+
+
+# Publishing New Project
+
+For adding a new project to the website, see one of the existing projects (e.g., [pdfgetx](https://www.diffpy.org/products/pdfgetx.html)) as reference.
+
+In what follows in this section, you will be working within the [`source`][source] branch, unless otherwise specified.
+
+You will need to create a directory for the project to live in within (e.g., [/static_root/doc/pdfgetx](https://github.com/diffpy/diffpy.github.io/tree/source/static_root/doc/pdfgetx)), then write a landing page for the project (e.g., [/products/pdfgetx.rst](https://github.com/diffpy/diffpy.github.io/blob/source/products/pdfgetx.rst)) which will provide any necessary information or files needed for a user to use the project. Once this is complete, see [Publishing New Version of Existing Project](#new-version) for steps on publishing the project.
+
+
+# Publishing Changes
+
+In order to test and/or publish changes, activate a conda environment that has a working installation of Sphinx (or make sure you have the necessary packages installed at a system level). If unsure how to achieve this, see the [Install Sphinx](https://gitlab.thebillingegroup.com/resources/group-wiki/-/wikis/Finalizing-a-Project's-(Re)-Release#install-sphinx) section of the projecct release wiki.
+
+## Testing Changes
+
+Make sure you have an active installation of Sphinx as per [Publishing Changes](#publishing-changes), run the set of Sphinx validation command(s) to check that the static files that Sphinx will create to be served on the website are written properly:
+```
+$ make Makefile linkcheck SPHINXOPTS="-W"
+```
+*NOTE:* the `"-W"` flag forces warnings to be treated as errors. If you believe there is a falsely reported warning that should be ignored that is preventing the test from passing, rerun the above command with the `SPHINXOPTS="-W"` portion omitted.
+
+Additionally, one can visually/user-experience validate that the changes are what is expected by running:
+```
+$ make Makefile html
+```
+Then opening `/_build/html/index.html` which will open a local instance of the website with the proposed changes in place.
+
+<a name="submitting-changes"></a>
+## Contributors (Submitting Changes)
+
+In order to update the contents of [diffpy.org][site], create a new branch off of [`source`][source] in your local fork of this repo and make whatever changes you need to. Once the desired edits are complete, follow the steps outlined in [Testing Changes](#testing-changes) to verify the site won't break as a result of these changes. Finally, after verifying that the changes won't break things, push your changes to your fork and open a pull-request to submit the changes back into this repo.
+
+<a name="publishing-changes"></a>
+## Maintainers (Publishing Changes)
+
+First, follow the steps outlines in [Testing Changes](#testing-changes) to verify that the changes won't mess things up (too bad). Note that these tests, as they currently exist, are not extensive and it's quite possible that something indeed may appear incorrectly on the website (hence the manual review portion). After reviewing the changes, have Sphinx generate the static files to be hosted on the website:
+```
+$ make Makefile html
+```
+
+Prepare the changes to be published:
+```
+$ make Makefile publish-prepare
+```
+
+Finally, publish the changes:
+```
+$ make Makefile publish-push
+```
+
+
+
+[site]: <https://www.diffpy.org/> "diffpy.org"
+[source]: <https://github.com/diffpy/diffpy.github.io/tree/source> "source"
+[master]: <https://github.com/diffpy/diffpy.github.io/tree/master> "master"
+[archive]: <https://github.com/diffpy/diffpy.github.io/tree/archive> "archive"

abbreviations.txt

@@ -293,3 +293,15 @@
 
 .. |downloadJuhasJac18| image:: /images/pdficon_small.png
    :target: /doc/pdfgetx/Juhas-jac-2018.pdf
+
+
+.. |citeLiuJac20| replace::
+   CH Liu, E. Janke, R. Li, P. Juhás, O. Gang, D. V. Talapin,
+   S. J. L. Billinge,
+   `sasPDF: pair distribution function analysis of nanoparticle assemblies
+   from small-angle-scattering data
+   <https://arxiv.org/abs/1910.08186>`__.
+   |downloadLiuJac20|
+
+.. |downloadLiuJac20| image:: /images/pdficon_small.png
+   :target: /doc/pdfgetx/Liu-jac-2020.pdf

products/pdfgetx.rst

@@ -67,7 +67,17 @@ Sorry for the complicated process.  We hope you enjoy the software!
 Documentation
 =============
 
-Version 2.1.0 - latest
+Version 2.1.1 - latest
+----------------------
+
+* `installation instructions <../doc/pdfgetx/2.1.1/install.html>`__
+* `user manual <../doc/pdfgetx/2.1.1/index.html>`__,
+  `printable manual <../doc/pdfgetx/2.1.1/PDFgetXNS3_manual.pdf>`__
+* `tutorial files <../doc/pdfgetx/2.1.1/pdfgetxn3-examples.zip>`__
+* `release notes <../doc/pdfgetx/2.1.1/release.html>`__
+
+
+Version 2.1.0
 ----------------------
 
 * `installation instructions <../doc/pdfgetx/2.1.0/install.html>`__
@@ -76,6 +86,7 @@ Version 2.1.0 - latest
 * `tutorial files <../doc/pdfgetx/2.1.0/pdfgetxn3-examples.zip>`__
 * `release notes <../doc/pdfgetx/2.1.0/release.html>`__
 
+
 Version 2.0.0
 -------------
 

publications.rst

@@ -6,8 +6,12 @@ References
 The following papers in the literature describe the DiffPy products.
 Please cite us if our software has been used in your research.
 
+* |citeLiuJac20|
+
 * |citeJuhasJac18|
 
+* |citeShiJac18|
+
 * |citeJuhasAca15|
 
 * |citeGranlundAca15|

static_root/doc/pdfgetx/2.1.1/_sources/files.rst.txt

@@ -0,0 +1,102 @@
+Files used in PDF extraction
+========================================================================
+
+.. include:: abbreviations.txt
+.. _my-configfile:
+.. program:: pdfgetx3
+
+Configuration file
+------------------------------------------------------------------------
+
+Configuration files may define the PDF processing parameters.
+By default, the :program:`pdfgetx3` program attempts to read
+:file:`.pdfgetx3.cfg` file from the user HOME directory,
+then :file:`.pdfgetx3.cfg` and :file:`pdfgetx3.cfg` files
+from the current working directory.
+If configuration file has a different name, it needs to be specified
+with the :option:`-c, --config <-c>` option.
+The :program:`pdfgetn3` program works in the same way,
+except it checks for configuration files
+:file:`~/.pdfgetn3.cfg`, :file:`.pdfgetn3.cfg` and :file:`pdfgetn3.cfg`.
+
+The easiest way of creating a configuration file is to generate
+a template content using the :option:`--createconfig` option as ::
+
+   pdfgetx3 --createconfig=test.cfg
+
+and then change the generated test.cfg file in your favorite text
+editor.  The configuration file follows a simple "varname=value" syntax,
+any lines starting with "#" are ignored as comments.
+
+The configuration file has several sections marked as ``[SECTIONNAME]``.
+The ``[DEFAULT]`` section is mandatory and it contains the default
+global settings.  Any other sections are optional and they are applied
+only when selected with the :option:`-s, --section <-s>` option on the
+command line.  Thus ::
+
+   pdfgetx3 --config=test.cfg --section=nacl
+
+would read the parameters from the ``[nacl]`` section after reading the
+defaults.  Having several sections in the configuration
+file is useful when there are multiple measurements that share most of
+the parameters, but differ in a few of them, for example in chemical
+composition.  The configuration file can then contain sections per each
+sample that define only the composition, while all other parameters are
+specified just once in the global DEFAULT section.
+
+
+Input files
+------------------------------------------------------------------------
+
+PDFgetX3 and PDFgetN3 accept input powder diffraction data
+in the form of two-column text file,
+where the first column x is either the scattering angle
+|twotheta| in degrees, momentum transfer *Q* in inverse nanometers or *Q*
+in inverse ångströms.  The second column y contains the corresponding
+scattered intensities normalized per unit solid angle.  The actual type
+of the x-values is identified by the :confval:`dataformat` parameter.
+The input files may contain header with comments or metadata, and the
+actual data are read from the first long section of numerical values.
+
+The input files are usually passed as command-line arguments to
+:program:`pdfgetx3` or :program:`pdfgetn3` programs and must be paths
+accessible from the current working directory.
+
+Input files can be also defined by setting the :confval:`inputfile`
+value in the configuration file.  The :option:`-d, --datapath <-d>`
+option can be then used to provide additional data directories to
+search for these inputs and for the :confval:`backgroundfile`.
+This is to support configuration files located in a different
+directory than the data.
+
+When the :option:`--find <-f>` option is active, the pdfgetx3 arguments
+are understood as filename patterns and the input files are found
+in the current or specified directory.
+
+
+Output files
+------------------------------------------------------------------------
+
+PDFgetX3 and PDFgetN3 can produce up to four different output data files:
+
+* .iq -- |IQ|, the background-corrected
+  intensities sampled on a regular *Q*-space grid in inverse ångströms,
+
+* .sq -- |SQ|, the total scattering structure function,
+  with intensities normalized by average scattering factors and
+  corrected by a polynomial fit,
+
+* .fq -- |FQ|, the reduced structure function equal to *Q*\ (|SQ| - 1),
+
+* .gr -- |Gr|, the resultant PDF, where the first column is the
+  separation *r* in ångströms and the second is the function *G* in
+  Å\ :sup:`-2`.
+
+You can specify what output files should be produced by setting the
+:confval:`outputtypes` parameter in the configuration file or by
+passing the :option:`-t, --outputtypes <-t>` on the command line.
+
+The header of all output files contains the parameter values that were
+used in the calculation and thus it is by itself a valid configuration
+file.  When passed as an argument to the :option:`--config <-c>` option,
+the PDFgetX3 will reproduce the previous calculation.