Merge branch 'release/0.15.3'

Author: Roy Keyes

Diff for: CHANGELOG

@@ -1,3 +1,41 @@
+2016-05-19 v0.15.3
+
+* modify TestSortElementTree to test for new commandline switch for keeping all results that bypasses the compress events functions (Christopher Barnes)
+*   Updating GetTheDocs URL in README.md (Nicholas Rejack)
+*   Allow setuptools_scm to set the version when it's importing redi.py (Nicholas Rejack)
+*   docs/conf.py needs to get the version from one directory up. (Nicholas Rejack)
+*   Adding 2 necessary elements to validate formEvents with HCV RNA forms. Fixing whitespace. (Nicholas Rejack)
+*   Adding setuptools_scm to docs/conf.py. Now using dynamically-set version number (based on Git tag) (Nicholas Rejack)
+*   Adding setuptools_scm to redi.py. Now using dynamically-set version number (based on Git tag) (Nicholas Rejack)
+* Adding setuptools_scm to setup.py. Now using dynamically-set version number (based on Git tag) (Nicholas Rejack)
+*   Fixing broken test by reordering XML statements so they validate. (Nicholas Rejack)
+*   Modifying redi/redi.py to add validation for XSD for formEvents.xml (Nicholas Rejack)
+*   Added redi/utils/formEvents.xsd to validate formEvents.xml (Nicholas Rejack)
+* add info on tests needed before I can make a pull request, to TODO (Christopher Barnes)
+* comment out test for --keep-all, add TODO to fix this later (Christopher Barnes)
+* Add 2 TODOs that cover issues with the timestamp and how it affects conmeds. (Buck72)
+* take the -keep-all logic out of get_key_date (Christopher Barnes)
+* take the -keep-all logic out of get_key_date (Christopher Barnes)
+* add keep-all-results logic to get_key_dates (Christopher Barnes)
+* add keep-all-results logic to get_key_dates (Christopher Barnes)
+* mod TestSortElementTree.py to have test keep-all results inputs (Christopher Barnes)
+* mod TestSortElementTree.py to have correct number of args(3) to handle new input keep-all (Christopher Barnes)
+* fixed incorrect ref to input_file to input_file_path on line 400 in _run (Christopher Barnes)
+* set flase correctly in sort elements to turn OFF compression (Christopher Barnes)
+* add keep all to create person xml func (Christopher Barnes)
+* add keep all to create person xml func (Christopher Barnes)
+* add keep all to create person xml func (Christopher Barnes)
+* change all refs to --kepp all to use true and false (Christopher Barnes)
+* fix inputs to sort_element_tree, pass in keep all results (Christopher Barnes)
+* fix inputs to _run, change inputfilename to input_file_path (Christopher Barnes)
+* fix inputs to _run, added -K and -f args and set defaults to none (Christopher Barnes)
+* process raw file correctly as part of keep all results (Christopher Barnes)
+* process raw file correctly as part of keep all results (Christopher Barnes)
+* cleanup -K keep all switch (Christopher Barnes)
+* cleanup -K keep all switch (Christopher Barnes)
+* cleanup -K keep all switch (Christopher Barnes)
+* add switch to keep all results (Christopher Barnes)
+
 2016-03-17 v0.15.2
 
  * Summary: Hotfix to fix logging behavior when input file is not present.

Diff for: README.md

@@ -28,5 +28,5 @@ You can view a presentation of the RED-I tool in action on
 [youtube](https://www.youtube.com/watch?v=0x04y5SNPL8&feature=youtu.be)
 
 For the official documentation please refer to
-[http://redi.readthedocs.org](http://redi.readthedocs.org) or
+[http://redi.readthedocs.io](http://redi.readthedocs.io) or
 [RED-I Project](https://github.com/ctsit/redi/blob/master/docs/about.rst).

Diff for: docs/conf.py

@@ -14,6 +14,7 @@
 
 import sys
 import os
+from setuptools_scm import get_version
 
 # If extensions (or modules to document with autodoc) are in another directory,
 # add these directories to sys.path here. If the directory is relative to the
@@ -61,9 +62,9 @@
 # built documents.
 #
 # The short X.Y version.
-version = '0.15.2'
+version = get_version(root='..', relative_to=__file__)
 # The full version, including alpha/beta/rc tags.
-release = '0.15.2'
+release = version
 
 # The language for content autogenerated by Sphinx. Refer to documentation
 # for a list of supported languages.

Diff for: redi/TODO.md

@@ -24,3 +24,16 @@
 TODO: make tests requirement- msg=sudo easy_install sftpserver
 TODO: make all settings files pythonic aka config.yml .. etc using YAML standard
 TODO: take NOT_DONE examples for values without units data points out of vagrant/redi_out_reference.csv for hcv examples
+<<<<<<< HEAD
+# TODO: -K 'keep all results' fail if in the settings.ini doesn't have Y-M-D H:M:S
+
+
+
+# TODO: Modify dates on ConMeds forms to validate for H:M:S.
+=======
+
+** TODO: To be able to make pull request for keep-all switch . Write unit test(s) for all the functions touched by the keep-all commandline switch.
+** TODO: TO be able to make a pull request for keep-all switch. Write end-to-end tests to take in fake conmeds data, call the sithc, pass in the file, use redcap     api to query out the data and compare to input csv. Use N Rejack scripts to create test data.
+** TODO: change install redcap setting to enable table based authentication.    
+** TODO: find out why sometimes redcap can't talk to mysql. restart mysql fixes. may need to add this to the bootstrap to restart apache and mysql right at the e    nd.
+>>>>>>> afd8c4190fafc4092152fcfb7c5dcee57cd5eb47

Diff for: redi/redi.py

@@ -22,7 +22,7 @@
 
 Usage:
     redi.py -h | --help
-    redi.py [-v] [-V] [-k] [-e] [-d] [-f=<path>] [-r] [-c=<path>] [-D=<datadir>] [-s] [-b]
+    redi.py [-v] [-V] [-k] [-e] [-d] [-f=<path>] [-r] [-c=<path>] [-D=<datadir>] [-s] [-b] [-K]
 
 Options:
     -h --help                   Show this help message and exit
@@ -52,10 +52,9 @@
                                 [default:False]
     -b --bulk-send-blanks       Send blank events in bulk instead of
                                 individually [default:False]
+    -K --keep-all               Keep all results, do not compress by date
 """
-
 __author__ = "University of Florida CTS-IT Team"
-__version__ = "0.15.0"
 __email__ = "[email protected]"
 __status__ = "Development"
 
@@ -90,12 +89,19 @@
 import utils.SimpleConfigParser as SimpleConfigParser
 import utils.GetEmrData as GetEmrData
 from utils.GetEmrData import EmrFileAccessDetails
+from setuptools_scm import get_version
 
 #from memory_profiler import profile
 
+# set the version using setuptools_scm
+# allow for setting the version when redi is imported
+try:
+    __version__ = get_version()
+except:
+    __version__ = get_version(root='..', relative_to=__file__)
+
 # Command line default argument values
 _person_form_events_service = None
-
 translational_table_tree = None
 
 DEFAULT_DATA_DIRECTORY = os.getcwd()
@@ -136,15 +142,16 @@ def main():
     - write the Final ElementTree to EAV
     """
 
-
     # TODO: UPDATE COMMENT HERE
     global _person_form_events_service
 
     # obtaining command line arguments for path to configuration directory
     args = docopt(__doc__, help=True)
 
-
+    # capture any cli args passed in that are needed to pass into other funcs.
     data_directory = args['--datadir']
+    keep_all_results = args['--keep-all']
+
     if data_directory is None:
         data_directory = DEFAULT_DATA_DIRECTORY
 
@@ -262,7 +269,7 @@ def main():
     _run(config_file, configuration_directory, do_keep_gen_files, dry_run,
          get_emr_data, settings, output_files, db_path, raw_txt_file, redcap_client,
          report_courier, report_creator, args['--resume'],
-         args['--skip-blanks'], args['--bulk-send-blanks'])
+         args['--skip-blanks'], args['--bulk-send-blanks'], keep_all_results, input_file_path)
 
     # TODO: post processing will go here
 
@@ -351,7 +358,7 @@ def connect_to_redcap(email_settings, redcap_settings, dry_run=False):
 def _run(config_file, configuration_directory, do_keep_gen_files, dry_run,
          get_emr_data, settings, data_folder, database_path, raw_txt_file, redcap_client,
          report_courier, report_creator, resume=False, skip_blanks=False,
-         bulk_send_blanks=False):
+         bulk_send_blanks=False, keep_all_results=False, input_file_path=None):
     global translational_table_tree
 
     assert _person_form_events_service is not None
@@ -391,6 +398,15 @@ def _run(config_file, configuration_directory, do_keep_gen_files, dry_run,
         # delete rawEscaped.txt
         GetEmrData.cleanup(escaped_file)
 
+    # TODO: clean this up as well was the get_emr_ stuff above
+
+    # if either -K or -f are specifed run the steps to make raw.xml
+    if (keep_all_results != False or input_file_path != None):
+        GetEmrData.data_preprocessing(raw_txt_file, escaped_file)
+        GetEmrData.generate_xml(escaped_file, raw_xml_file)
+        GetEmrData.cleanup(escaped_file)
+
+
 
     raw_xml_file = os.path.join(configuration_directory, settings.raw_xml_file)
     email_settings = get_email_settings(settings)
@@ -416,7 +432,7 @@ def _run(config_file, configuration_directory, do_keep_gen_files, dry_run,
         _create_person_form_event_tree_with_data(
             config_file, configuration_directory, redcap_client,
             form_events_file, raw_xml_file, rules, settings, data_folder,
-            translation_table_file)
+            translation_table_file, keep_all_results)
 
         _store_run_data(data_folder, alert_summary,
                         person_form_event_tree_with_data, rule_errors,
@@ -495,7 +511,7 @@ def _run(config_file, configuration_directory, do_keep_gen_files, dry_run,
 
 def _create_person_form_event_tree_with_data(
         config_file, configuration_directory, redcap_client, form_events_file,
-        raw_xml_file, rules, settings, data_folder, translation_table_file):
+        raw_xml_file, rules, settings, data_folder, translation_table_file, keep_all_results):
 
     global translational_table_tree
     # parse the raw.xml file and fill the etree rawElementTree
@@ -602,7 +618,7 @@ def _create_person_form_event_tree_with_data(
         data,
         os.path.join(data_folder, 'rawDataWithDatumAndUnitsFieldNames.xml'))
     # sort the data tree and compress
-    sort_element_tree(data, data_folder)
+    sort_element_tree(data, data_folder, keep_all_results)
     write_element_tree_to_file(data, os.path.join(data_folder, \
         'rawDataSortedAfterCompression.xml'))
     # update eventName element
@@ -702,6 +718,8 @@ def parse_form_events(form_events_file):
         logger.info("Form events file contains {} lines." \
                 .format(str(sum(1 for line in raw))))
 
+    validate_xml_file_and_extract_data(form_events_file, pkg_resources.resource_filename(
+        'redi', 'utils/formEvents.xsd'))
     data = etree.parse(form_events_file)
     event_sum = len(data.findall(".//event"))
     logger.debug(str(event_sum) + " total events read into tree.")
@@ -856,7 +874,7 @@ def update_redcap_form(data, lookup_data, undefined):
         undefined)
 
 
-def sort_element_tree(data, data_folder):
+def sort_element_tree(data, data_folder, keep_all_results):
     """
     Sort element tree based on three given indices.
     @see #update_time_stamp()
@@ -876,7 +894,15 @@ def sort_element_tree(data, data_folder):
     write_element_tree_to_file(data, os.path.join(data_folder,
         "rawDataSortedBeforeCompression.xml"))
 
-    compress_data_using_study_form_date(data)
+    # TODO: look at adding a switch to RED-I, that will need to be caught here, that
+    #       will allow another behavioe here that will let us keep all results vs
+    #       the current behavior of sorting the events by timestamp and keeping only
+    #       the first one to occur on a given day. Example: whne this feature is
+    #       implemented red-i will be able to keep only 1 data point for each day
+    #       for 50 days or keep 50 data points that may occur on the same day and
+    #       map the 50 into 50 event slots in redcap.
+    if (keep_all_results == False):
+        compress_data_using_study_form_date(data)
 
     #batch.printxml(container)
 
@@ -949,13 +975,7 @@ def compress_data_using_study_form_date(data):
             logger.debug("Remove duplicate result using key: {}".format(key_debug))
             subj.getparent().remove(subj)
 
-# TODO: look at adding a switch to RED-I, that will need to be caught here, that
-#       will allow another behavioe here that will let us keep all results vs
-#       the current behavior of sorting the events by timestamp and keeping only
-#       the first one to occur on a given day. Example: whne this feature is
-#       implemented red-i will be able to keep only 1 data point for each day
-#       for 50 days or keep 50 data points that may occur on the same day and
-#       map the 50 into 50 event slots in redcap.
+
 
     filt = dict()
 
@@ -1003,7 +1023,7 @@ def get_key_timestamp(ele):
     return (study_id, form_name, timestamp)
 
 
-def get_key_date(ele):
+def get_key_date(ele,keep_all_results=False):
     """
     Helper function for #compress_data_using_study_form_date()
 

Diff for: redi/utils/formEvents.xsd

@@ -0,0 +1,33 @@
+<?xml version ="1.0"?>
+<xs:schema attributeFormDefault="unqualified" elementFormDefault="unqualified" xmlns:xs="http://www.w3.org/2001/XMLSchema">
+    <xs:element name="redcapProject">
+	    <xs:complexType>
+		     <xs:sequence>
+			    <xs:element name="name" type="xs:string" />
+				<xs:element name="form" maxOccurs="unbounded">
+				    <xs:complexType>
+					    <xs:sequence>
+                            <xs:element name="name" type="xs:string"/>
+                            <xs:element name="formDateField" type="xs:string"/>
+    					    <xs:element name="formFieldDetected" type="xs:string" minOccurs="0"/>
+						    <xs:element name="formQuantifiedField" type="xs:string" minOccurs="0"/>
+					        <xs:element name="formCompletedFieldName" type="xs:string"/>
+					        <xs:element name="formCompletedFieldValue" type="xs:int"/>
+					        <xs:element name="formImportedFieldName"  type="xs:string" minOccurs="0"/>
+						    <xs:element name="formImportedFieldValue"  type="xs:string" minOccurs="0"/>
+						    <xs:element name="formFieldNotes" type="xs:string" minOccurs="0"/>
+						    <xs:element name="formFieldVerbatim" type="xs:string" minOccurs="0"/>
+						    <xs:element name="event" maxOccurs="unbounded">
+						        <xs:complexType>
+			    			        <xs:sequence>
+				    			        <xs:element name="name" type="xs:string"/>
+					        	    </xs:sequence>
+					            </xs:complexType>
+		    	            </xs:element>
+					    </xs:sequence>
+				    </xs:complexType>
+				</xs:element>
+			</xs:sequence>
+		</xs:complexType>
+	</xs:element>
+</xs:schema>

Diff for: setup.py

@@ -18,7 +18,7 @@
 
 setup(
     name='redi',
-    version='0.15.2',
+    use_scm_version=True,
     author='https://www.ctsi.ufl.edu/research/study-development/informatics-consulting/',
     author_email='[email protected]',
     packages=find_packages(exclude=['test']),
@@ -27,7 +27,7 @@
         'redi': ['utils/*.xsl', 'utils/*.xsd']
     },
     url='https://github.com/ctsit/redi',
-    download_url = 'https://github.com/ctsit/redi/releases/tag/0.15.1',
+    download_url = 'https://github.com/ctsit/redi/releases/tag/0.15.3',
     keywords = ['EMR', 'EHR', 'REDCap', 'Clinical Data'],
     license='BSD 3-Clause',
     description='REDCap Electronic Data Importer',
@@ -54,5 +54,6 @@
     setup_requires=[
         "nose >= 1.0",
         "nosexcover >= 1.0.10",
+        "setuptools_scm >= 1.10.1"
     ],
 )

Diff for: test/TestParseAll.py

@@ -45,8 +45,8 @@ def test_parse_form_events(self):
 		<name>cbc</name>
 		<formDateField>cbc_lbdtc</formDateField>
 		<formCompletedFieldName>cbc_complete</formCompletedFieldName>
-		<formImportedFieldName>cbc_nximport</formImportedFieldName>
         <formCompletedFieldValue>2</formCompletedFieldValue>
+		<formImportedFieldName>cbc_nximport</formImportedFieldName>
         <formImportedFieldValue>Y</formImportedFieldValue>
 		<event>
     		<name>1_arm_1</name>
@@ -62,8 +62,8 @@ def test_parse_form_events(self):
 		<name>chemistry</name>
 		<formDateField>chem_lbdtc</formDateField>
 		<formCompletedFieldName>chemistry_complete</formCompletedFieldName>
-		<formImportedFieldName>chem_nximport</formImportedFieldName>
         <formCompletedFieldValue>2</formCompletedFieldValue>
+		<formImportedFieldName>chem_nximport</formImportedFieldName>
         <formImportedFieldValue>Y</formImportedFieldValue>
 		<event>
 		    <name>1_arm_1</name>