Merge pull request #101 from NCAR/devel

Updates

Author: sherimickelson

CODE_OF_CONDUCT.md

@@ -0,0 +1,84 @@
+# Contributor Code of Conduct
+_The Contributor Code of Conduct is for participants in our software projects and community._
+
+## Our Pledge
+We, as contributors, creators, stewards, and maintainers (participants), of PyConform pledge to make participation in our software, system or hardware project and community a safe, productive, welcoming and inclusive experience for everyone.
+All participants are required to abide by this Code of Conduct.
+This includes respectful treatment of everyone regardless of age, body size, disability, ethnicity, gender identity or expression, level of experience, nationality, political affiliation, veteran status, pregnancy, genetic information, physical appearance, race, religion, or sexual orientation, as well as any other characteristic protected under applicable US federal or state law.
+
+## Our Standards
+Examples of behaviors that contribute to a positive environment include:
+
+* All participants are treated with respect and consideration, valuing a diversity of views and opinions
+* Be considerate, respectful, and collaborative
+* Communicate openly with respect for others, critiquing ideas rather than individuals and gracefully accepting criticism
+* Acknowledging the contributions of others
+* Avoid personal attacks directed toward other participants
+* Be mindful of your surroundings and of your fellow participants 
+* Alert UCAR staff and suppliers/vendors if you notice a dangerous situation or someone in distress
+* Respect the rules and policies of the project and venue
+
+Examples of unacceptable behavior include, but are not limited to:
+
+* Harassment, intimidation, or discrimination in any form
+* Physical, verbal, or written abuse by anyone to anyone, including repeated use of pronouns other than those requested
+* Unwelcome sexual attention or advances
+* Personal attacks directed at other guests, members, participants, etc.
+* Publishing others' private information, such as a physical or electronic address, without explicit permission
+* Alarming, intimidating, threatening, or hostile comments or conduct
+* Inappropriate use of nudity and/or sexual images 
+* Threatening or stalking anyone, including a participant
+* Other conduct which could reasonably be considered inappropriate in a professional setting
+
+## Scope
+This Code of Conduct applies to all spaces managed by the Project whether they be physical, online or face-to-face.
+This includes project code, code repository, associated web pages, documentation, mailing lists, project websites and wiki pages, issue tracker, meetings, telecons, events, project social media accounts, and any other forums created by the project team which the community uses for communication.
+In addition, violations of this Code of Conduct outside these spaces may affect a person's ability to participate within them.
+Representation of a project may be further defined and clarified by project maintainers.
+
+## Community Responsibilities
+Everyone in the community is empowered to respond to people who are showing unacceptable behavior.
+They can talk to them privately or publicly.
+Anyone requested to stop unacceptable behavior is expected to comply immediately.
+If the behavior continues concerns may be brought to the project administrators or to any other party listed in the [Reporting](#reporting) section below.
+
+## Project Administrator Responsibilities
+Project administrators are responsible for clarifying the standards of acceptable behavior and are encouraged to model appropriate behavior and provide support when people in the community point out inappropriate behavior.
+Project administrator(s) are normally the ones that would be tasked to carry out the actions in the [Consequences](#consequences) section below.
+
+Project administrators are also expected to keep this Code of Conduct updated with the main one housed at UCAR, as listed below in the [Attribution](#attribution) section.
+
+## Reporting
+Instances of unacceptable behavior can be brought to the attention of the project administrator(s) who may take any action as outlined in the [Consequences](#consequences) section below.
+However, making a report to a project administrator is not considered an 'official report' to UCAR. 
+
+Instances of unacceptable behavior may also be reported directly to UCAR pursuant to [UCAR's Harassment Reporting and Complaint Procedure](https://www2.fin.ucar.edu/procedures/hr/harassment-reporting-and-complaint-procedure), or anonymously through [UCAR's EthicsPoint Hotline](https://www2.fin.ucar.edu/ethics/anonymous-reporting).
+
+Complaints received by UCAR will be handled pursuant to the procedures outlined in UCAR's Harassment Reporting and Complaint Procedure.
+Complaints to UCAR will be held as confidential as practicable under the circumstances, and retaliation against a person who initiates a complaint or an inquiry about inappropriate behavior will not be tolerated.
+
+Any Contributor can use these reporting methods even if they are not directly affiliated with UCAR.
+The Frequently Asked Questions (FAQ) page for reporting is [here](https://www2.fin.ucar.edu/procedures/hr/reporting-faqs).
+
+## Consequences
+Upon receipt of a complaint, the project administrator(s) may take any action deemed necessary and appropriate under the circumstances.
+Such action can include things such as: removing, editing, or rejecting comments, commits, code, wiki edits, email, issues, and other contributions that are not aligned to this Code of Conduct, or banning temporarily or permanently any contributor for other behaviors that are deemed inappropriate, threatening, offensive, or harmful.
+Project administrators also have the right to report violations to UCAR HR and/or UCAR's Office of Diversity, Equity and Inclusion (ODEI), as well as a participant's home institution and/or law enforcement.
+In the event an incident is reported to UCAR, UCAR will follow its Harassment Reporting and Complaint Procedure.
+
+## Process for Changes
+All UCAR managed projects are required to adopt this Contributor Code of Conduct.
+Adoption is assumed even if not expressly stated in the repository.
+Projects should fill in sections where prompted with project-specific information, including, project name and adoption date.
+
+Projects that adopt this Code of Conduct need to stay up to date with UCAR's Contributor Code of Conduct, linked with a DOI in the [Attribution](#attribution) section below.
+Projects can make limited substantive changes to the Code of Conduct, however, the changes must be limited in scope and may not contradict the UCAR Contributor Code of Conduct.
+
+## Attribution
+This Code of Conduct was originally adapted from the [Contributor Covenant](http://contributor-covenant.org/version/1/4), version 1.4.
+We then aligned it with the UCAR Participant Code of Conduct, which also borrows from the American Geophysical Union (AGU) Code of Conduct.
+The UCAR Participant Code of Conduct applies to both UCAR employees as well as participants in activities run by UCAR.
+The original version of this for all software projects that have strong management from UCAR or UCAR staff is available on the UCAR website at https://doi.org/10.5065/6w2c-a132.
+The date that it was adopted by this project was Apr/7th/2020
+When responding to complaints, UCAR HR and ODEI will do so based on the latest published version.
+Therefore, any project-specific changes should follow the [Process for Changes](#process-for-changes) section above.

README.rst

@@ -1,3 +1,6 @@
+.. image:: https://zenodo.org/badge/DOI/10.5281/zenodo.3895009.svg
+   :target: https://doi.org/10.5281/zenodo.3895009
+
 PyConform
 =========
 

scripts/iconform

@@ -141,13 +141,13 @@ def fill_missing_glob_attributes(attr, table, v, grids):
     for a,d in attr.iteritems():
       if d is not None:
         if "__FILL__" in d:
-            if "activity_id" in a:
+            if "activity_id" in a and "activity_id" in table:
                 attr["activity_id"] = table["activity_id"] 
             elif "creation_date" in a:
                 attr["creation_date"]=""
-            elif "data_specs_version" in a:
+            elif "data_specs_version" in a and "data_specs_version" in table:
                 attr["data_specs_version"] = table["data_specs_version"]
-            elif "experiment" in a:
+            elif "experiment" in a and "experiment" in table:
                 attr["experiment"] = table["experiment"]
             elif "external_variables" in a:
                 if "cell_measures" in v.keys():
@@ -252,23 +252,28 @@ def fill_missing_glob_attributes(attr, table, v, grids):
         attr["forcing_index"] = int(pre)
 
     if "further_info_url" in attr.keys():
+        url_ok = True
         if "__FILL__" in attr["further_info_url"]:       
             if 'mip_era' in attr.keys():
                 mip_era = attr['mip_era']
             else:
                 mip_era = ''
+                url_ok = False
             if 'institution_id' in attr.keys():
                 institution_id = attr['institution_id']
             else:
                 institution_id = ''
+                url_ok = False
             if 'source_id' in attr.keys():
                 source_id = attr['source_id']
             else:
                 source_id = ''
+                url_ok = False
             if 'experiment_id' in attr.keys():
                 experiment_id = attr['experiment_id']
             else:
                 experiment_id = ''
+                url_ok = False
             if 'sub_experiment_id' in attr.keys():
                 sub_experiment_id = attr['sub_experiment_id']
             else:
@@ -277,8 +282,10 @@ def fill_missing_glob_attributes(attr, table, v, grids):
                 ripf = attr["variant_label"]
             else:
                 ripf = ''
-            info_url = "{0}.{1}.{2}.{3}.{4}.{5}".format(mip_era, institution_id, source_id, experiment_id, sub_experiment_id, ripf)
-            attr['further_info_url'] = "https://furtherinfo.es-doc.org/" + info_url
+                url_ok = False
+            if url_ok:
+                info_url = "{0}.{1}.{2}.{3}.{4}.{5}".format(mip_era, institution_id, source_id, experiment_id, sub_experiment_id, ripf)
+                attr['further_info_url'] = "https://furtherinfo.es-doc.org/" + info_url
     if "grid" in attr.keys():
         if len(attr["realm"])>0:
             attr["grid"] = grids[attr["realm"].split()[0]]
@@ -319,7 +326,8 @@ def defineVar(v, varName, attr, table_info, definition, ig, experiment, out_dir)
                                                      str(attributes['forcing_index'])))
     else:
         ripf = ''
-
+    if len(ripf) < 1:
+        ripf = "None"
     if 'mip_era' in attributes.keys():
         mip_era = attributes['mip_era']
     else:
@@ -328,6 +336,8 @@ def defineVar(v, varName, attr, table_info, definition, ig, experiment, out_dir)
         activity_id = attributes['activity_id']
     else:
         activity_id = ''
+    if '__FILL__' in activity_id:
+        activity_id = 'None' 
     if 'institution_id' in attributes.keys():
         institution_id = attributes['institution_id']
     else:
@@ -344,6 +354,8 @@ def defineVar(v, varName, attr, table_info, definition, ig, experiment, out_dir)
         sub_experiment_id = attributes['sub_experiment_id']
     else:
         sub_experiment_id = ''
+    if '--ALL--' in experiment:
+        experiment = 'None'
 
     f_format = attributes["netcdf_type"]
     valid_formats = ['NETCDF4','NETCDF4_CLASSIC','NETCDF3_CLASSIC','NETCDF3_64BIT_OFFSET','NETCDF3_64BIT_DATA']
@@ -382,6 +394,12 @@ def defineVar(v, varName, attr, table_info, definition, ig, experiment, out_dir)
                vid, mipTable, source_id, experiment, ripf, grid, dst))
     var = {}
 
+    # Remove any __FILL__ values in attributes
+    for a in attributes.keys():
+        if isinstance(attributes[a],str):
+            if '__FILL__' in attributes[a]:
+                attributes[a] = '' 
+
     # put together the dictionary entry for this variable    
     var["attributes"] = v2
     var["definition"] = definition
@@ -705,8 +723,8 @@ def main(argv=None):
         tables = ['--ALL--']
 
     dq = dreq.loadDreq()
-    if '--ALL--' in exps:
-        exps = dq.inx.experiment.label.keys()
+#S    if '--ALL--' in exps:
+#S        exps = dq.inx.experiment.label.keys()
 
     # Go through a user supplied list if specified
     variables = []

source/pyconform/miptableparser.py

@@ -371,16 +371,23 @@ def parse_table(self,exp,mips,tables,v_list,table_var_fields,table_axes_fields,t
         # Get a list of mips for the experiment
 #        print sorted(dq.inx.experiment.label.keys()),len(dq.inx.experiment.label.keys())
         e_mip = []
-        e_id = dq.inx.experiment.label[exp]
+        if '--ALL--' in exp:
+            e_id = dq.inx.experiment.label['historical']
+        else:
+            e_id = dq.inx.experiment.label[exp]
         if len(e_id)==0:
-            print '\033[91m','Invalid experiment name.  Please choose from the folowing options:','\033[0m' 
+#            print '\033[91m','Invalid experiment name.  Please choose from the folowing options:','\033[0m' 
+#            print sorted(dq.inx.experiment.label.keys()),len(dq.inx.experiment.label.keys())
+#            return {} 
+            print '\033[91m','This experiment name is not registered within the data request.','\033[0m'
+            print '\033[91m','Using a generic exp name instead.  If you think this name should be registered, verify the name is correct from the list below.','\033[0m'
             print sorted(dq.inx.experiment.label.keys()),len(dq.inx.experiment.label.keys())
-            return {} 
+            e_id = dq.inx.experiment.label['historical']
         activity_id = dq.inx.uid[e_id[0]].mip
 
         e_vars =[]
+        activity_id = dq.inx.uid[e_id[0]].mip
         e_vars.append(dq.inx.iref_by_sect[e_id[0]].a)
-        #if len(e_vars['requestItem']) == 0:
         e_vars.append(dq.inx.iref_by_sect[dq.inx.uid[e_id[0]].egid].a)
         e_vars.append(dq.inx.iref_by_sect[activity_id].a)
         total_request = {}
@@ -389,18 +396,20 @@ def parse_table(self,exp,mips,tables,v_list,table_var_fields,table_axes_fields,t
 
             table_info = {}
             dr = dq.inx.uid[ri]
-            if dr.mip in mips or '--ALL--' in mips:
+
+            if dr.mip in mips or '--ALL--' in mips or '--ALL--' in exp:
 
                 table_dict = {}
                 variables = {}
                 axes = {}
                 table_info = {}
                 data = {}
-
-                table_info['experiment'] = dq.inx.uid[e_id[0]].title
-                table_info['experiment_id'] = exp
+                
                 table_info['data_specs_version'] = dreq.version
-                table_info['activity_id'] = activity_id
+                if '--ALL--' not in exp:
+                    table_info['experiment'] = dq.inx.uid[e_id[0]].title
+                    table_info['experiment_id'] = exp
+                    table_info['activity_id'] = activity_id
 
 
                 rl = dq.inx.requestLink.uid[dr.rlid]
@@ -409,7 +418,7 @@ def parse_table(self,exp,mips,tables,v_list,table_var_fields,table_axes_fields,t
                 axes_list = []
                 var_list = []
                 if len(v_list) == 0:
-                    var_list = vars['requestVar'] 
+                    var_list = vars['requestVar']
                 else:
                     # Make a list of all cmip vars and their id's
                     all_vars = {}
@@ -633,6 +642,5 @@ def parse_table(self,exp,mips,tables,v_list,table_var_fields,table_axes_fields,t
         #for k in sorted(total_request.keys()):
         #    v = total_request[k]
         #    print k, len(v['variables'])
-
         return total_request