implement wildcard pattern for input

[yucongalicechen]

• Initial commit
• Related to Issue Implement input function to handle skipped files and check ouputs and header #52, examples are here user input files UCs #48
• I skipped file_list files as usual. Shall we allow the user to specify something like "file*"?

[sbillinge]

very nicely done. Just one small typo and one suggestion in the input.

I also see some copy-pasted code which makes me wonder if it might be better to simply look for the wildcard and expand it before passing into this code block. Kind of like a prefilter that does something like

for input_name in args.input:
     if '*" in name 
             do the globbing and find the list of files
             add the glob list to args.inputs and remove the "*" name

then again,

for input_name in args.inputs:
    code unchanged from before

which would be easier to maintain.

[sbillinge]

maybe change to, ./*.chi to be closer to that description.

What happens if the wild-card expands to files and directories? We should tell the user what happens then.

[yucongalicechen]

• I added a function expand_wildcard_file to take care of wildcard patterns and return the expanded list of inputs. Shall we put the two expand functions into set_input_lists for convenience? So that in the labpdfproc.py and tests, we only need to call for set_input_lists.
• I wrote an error message for wildcard patterns that cannot match any files. But if the user specifies an invalid wildcard pattern like "**", this error message might not be that helpful as the original error message from Python (as it would tell user that it is an invalid wildcard pattern). I currently cannot think of a way to distinguish between these two situations, unless checking that if wildcard pattern is one of the invalid cases, but there are many invalid cases to check..

[sbillinge]

please see inline comments.

[sbillinge]

re your comment, I agree that it might be good to put the two pre-filters into the expand_user_input function. Perhaps we shoud also make them private functions (add an underscore at the beginning) which makes the code more readable. It is nice that we have independent tests for them, no need to undo that.

[sbillinge]

we may need a test for a user putting a wildcard in the file-list file. I think your code may currently fail this test, but let's make the test and see.

Almost there, see a few last cleaning edits, and also I think my request to not support wildcards on directories may have gotten lost in those long comments on the last review, bu tplease remove support for wildcards on directories.

[sbillinge]

nice work. I could merge it like this, but there are a couple of final comments.

[sbillinge]

I think we may be able to remove if file_list not in file.name because file_list files have been removed already.

[yucongalicechen]

I think this is for files in the glob directory (not in args.input), so if we have a file list in the same directory as the wildcard, then it'll be loaded if we don't skip it.

[sbillinge]

I see, yes better to have it to be on the safe side. It may be better to make it stricter so if file.name == file_list.txt but it is probably ok as it is.