Skip to content

Conversation

ashwinpankaj
Copy link

@ashwinpankaj ashwinpankaj commented Jan 8, 2024

#213

Found two misleading logs while testing SpoolDirLineDelimitedSourceConnector

  1. Finished processing 0 record(s) in 0 second(s) is logged even though connector is processing records
  2. Failed to delete input.path sub-directory: /tmp/data/input/fix.json is logged even though that file does not exist in the path as it has been moved after processing

Fixes

  1. Ensure SpoolDirLineDelimitedSourceTask updates recordCount in AbstractSourceTask and calculate processing speed in milliseconds (sometimes files are processed in less than a second, which leaves secondsElapsed set to 0)
  2. getInputPathSubDirsToCleanup() should only return dirs which exist

@ashwinpankaj ashwinpankaj changed the title Fix misleading messages logged by SpoolDirLineDelimitedSourceConnector Fix misleading messages logged by SpoolDirLineDelimitedSourceConnector #213 Jan 8, 2024
@ashwinpankaj ashwinpankaj changed the title Fix misleading messages logged by SpoolDirLineDelimitedSourceConnector #213 Fix misleading messages logged by SpoolDirLineDelimitedSourceConnector Jan 8, 2024
@ashwinpankaj
Copy link
Author

@jcustenborder could you please review this PR ?

1 similar comment
@ashwinpankaj
Copy link
Author

@jcustenborder could you please review this PR ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant