hanging at java.net.SocketInputStream.socketRead0

[menzowindhouwer]

java.net.HttpURLConnection may hang is special cases (see https://stackoverflow.com/questions/15874834/httpclient-hangs-on-socketread0-with-successfully-executed-method/19189036#19189036). No timeouts are triggered, so the only thing left is killing the harvest, but this skips post processing.

• Option 1:
  Enable a way in the python control script to indicate the need to kill the harvest, but do the post processing. Can be done by monitoring for a STOP file, if seen kill the harvest and proceed with the post processing.

• Option 2:
  The harvester catches SIGUSR1, stops, and just exists with 0, so post processing continuous.

• Option 3:
  An overall timeout - if a harvesting thread takes more than it just gets terminated (e.g. monitoring thread interrupts it). Maybe JMX or another mechanism can be used to tweak these timeouts so that a running harvest can be meddled with.