You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
When running batch files, DCP passes in pipeline and -i, but not --data-file (because it shouldn't need it because path info is in the batch file). HOWEVER, apparently in this case, the -i setting does do the "let's look at every file in the path", which in the default of empty string, means it tries to list every file in the entire dang bucket. Not the desired behavior! Not sure we can fix at the code level because sometimes a batched pipeline might need -i, but should investigate more, and document what we find.
The text was updated successfully, but these errors were encountered:
When running batch files, DCP passes in
pipeline
and-i
, but not--data-file
(because it shouldn't need it because path info is in the batch file). HOWEVER, apparently in this case, the -i setting does do the "let's look at every file in the path", which in the default of empty string, means it tries to list every file in the entire dang bucket. Not the desired behavior! Not sure we can fix at the code level because sometimes a batched pipeline might need -i, but should investigate more, and document what we find.The text was updated successfully, but these errors were encountered: