Fails to delete subdirectory - but there is none?

[talljhawkins]

Confused here - can you help to see if this is an error or just me badly configuring:
I have the following:
"input.file.pattern": "^.*\.fuu$",
"input.path": "c:\B2BiInput\LWFW\input",
"finished.path": "c:\B2BiInput\LWFW\finished",
"error.path": "c:\B2BiInput\LWFW\errors",

The processing works just fine but it always logs an error for every file it processes:

Failed to delete input.path sub-directory: c:\B2BiInput\LWFW\input<I've redacted the filename.fuu> (com.github.jcustenborder.kafka.connect.spooldir.InputFile:242)

Is it trying to delete the subdirectory? c:\B2BiInput\LWFW or perhaps c:\B2BiInput\LWFW\input - and why? The file gets deleted/moved OK so that's not a problem.

I've tried this on a few systems and always the same error.

[jcustenborder]

hmmm. Of hand I have not seen anything like this. Can you enable trace logging?

[abrahamgreyson]

Same here, the file has been deleted after message commited to topic, but log shows the same:

[2024-09-13 18:32:58,655] INFO WorkerSourceTask{id=data-0} Committing offsets for 317995 acknowledged messages (org.apache.kafka.connect.runtime.WorkerSourceTask)
[2024-09-13 18:33:02,788] ERROR Failed to delete input.path sub-directory: /usr/share/upstream/data/data-20240911-1.csv (com.github.jcustenborder.kafka.connect.spooldir.InputFile)

Here is the conf:

{
  "tasks.max": "1",
  "connector.class": "com.github.jcustenborder.kafka.connect.spooldir.SpoolDirCsvSourceConnector",
  "input.path": "/usr/share/upstream/data",
  "error.path": "/usr/share/upstream/error",
  "finished.path": "/usr/share/upstream/finished",
  "input.file.pattern": "data.*\\.csv",
  "halt.on.error": "true",
  "topic": "data",
  "csv.first.row.as.header": "true",
  "schema.generation.enabled": "true",
  "cleanup.policy": "DELETE",
  "empty.poll.wait.ms": "180000",
  "csv.rfc.4180.parser.enabled": "true"
}

It's diffcult to me to change log level as I'm new to Java, but the connect container is based on Confluent/Connect

[aheiska]

We see the same thing. For every file imported there is:

"log.level":"ERROR", "message":"Failed to delete input.path sub-directory: /mnt/..../20250203_131546.csv"

Files seems to be deleted successfully, so this this is just effect in log. Though monitoring that checks errors from logs goes wild. Our cleanup.policy is DELETE. Directory where file is read on CIFS, which very well might matter. Spooldir is 2.0.65 and connect is run on Linux and filesystem is mounted from Windows host.

Looking at code: File.delete() seems to return false. Currently quite busy, but I will later test local version that uses java.nio.file.Files.delete() and has more logging to see better what is happening here (assuming windows + File oddities mentioned at least at https://mail.openjdk.org/pipermail/core-libs-dev/2012-August/011314.html)

[jcustenborder]

hmmmm I wonder if there is a locking issue or something.

[aheiska]

After a more detailed investigation, I believe I have identified the root cause of this. It occurs during the subdirectory cleanup procedure, where original file is found for deletion, causing it to be deleted twice, naturally failing on the second attempt.

When the cleanup policy is set to delete, the following occurs:

1. The file is deleted at InputFile:280
2. The subdirectory cleanup process is then triggered at InputFile:284.
3. The same file appears in getInputPathSubDirsToCleanup (InputFile:219).
   • This might be related to Relative file path is not retrieved properly #206, which was addressed in closes issue #206 #209.
4. Second delete fails at cleanupInputDirSubDirs (InputFile:239)
5. The error is logged at InputFile:240.

This behavior occurs regardless of whether the input directory contains subdirectories.

Workaround for current version: setting "cleanup.policy.maintain.relative.path": "true" solves this, because condition at InputFile:221 causes subdirectory list be be null, so cleanup code is never run.

---

Also while at it, idea static analysis finds that at this.inputPathSubDir.split(File.separator) InputFile:224 "File. separator is used as a regex; will not work on Windows".

I quickly tested this with java 8 & 21. Split fails on java 21 (java.util.regex.PatternSyntaxException) but succeeds on java 8 (returned array has 1 item, the input string).