image download and video creation was stuck for ~5days

[dneise]

I have no explanation... from the screen log, I was able to gather only this:

[...]
2017-08-22 12:45:51|INFO|la_palma_overview|Video already created, skipping
2017-08-22 12:46:51|INFO|la_palma_overview|Video already created, skipping
2017-08-22 12:47:51|INFO|la_palma_overview|Video already created, skipping
2017-08-27 09:02:38|INFO|la_palma_overview|Video already created, skipping
2017-08-27 09:03:39|INFO|la_palma_overview|Video already created, skipping
2017-08-27 09:04:39|INFO|la_palma_overview|Video already created, skipping
[...]

I have no idea, what might be the reason for this behavior.

All I could imagine is, that the same thing happened to overview_video is what also happened a few times to the FACT++ programs. When somebody is in the screen and stays in this "scroll mode", then the stdout buffer of the programs blocks the entire program at some point.

[dneise]

Yesterday in the morning ... could have been around that time, I connected to the screen ... and saw this "Video already created, skipping". I assumed it was stuck in an endless loop and decided that I had not the resources to fix this on the weekend.

But apparently just this connecting to the screen .. kind of fixed the issue. since later that day, this turns up:

[...]
2017-08-27 12:58:53|INFO|la_palma_overview|Video already created, skipping
2017-08-27 12:59:53|INFO|la_palma_overview|Video already created, skipping
2017-08-27 13:00:53|INFO|la_palma_overview|Waiting for next night
2017-08-27 13:01:53|INFO|la_palma_overview|Waiting for next night
[...]

Which, I think, is normal (I hardly ever looked at the video-process during daytime).

And then later again this showed up:

2017-08-27 17:58:10|INFO|la_palma_overview|Waiting for next night
2017-08-27 17:59:10|INFO|la_palma_overview|Waiting for next night
2017-08-27 18:00:10|INFO|la_palma_overview|Getting image
2017-08-27 18:00:52|INFO|la_palma_overview|done

So I would say ... it is running again just fine.

I took great care to go out of "scroll mode" before disconnecting... so it should run fine now.

[dneise]

I played around with this a bit...
so I let run this code:

import time
import datetime

while True:
    print(datetime.datetime.now())
    time.sleep(1e-3)

in a screen on newdaq ... and immediately entered screens "scroll mode"... I went into scroll mode at 2017-08-28 08:06:56.480907.... waited until roughly 08:10 and exited scroll mode by hitting Esc.

I found this:

2017-08-28 08:06:57.327541
2017-08-28 08:10:31.499726

So after going into scroll mode my little lest program kept running for another second or so. Or I should rather say, it kept writing to stdout for another second or so... until the stdout buffer was full, then it was stuck. until I went out of scroll mode.

So this shows again the "dangerousness" of "scroll mode".

[dneise]

I think the easiest remedy for this is to use logging to file instead of logging to stdout.

Maybe the most easy way to do this is to "monkey-patch" print... like this?

import logging

logging.basicConfig(
    level=logging.DEBUG,
    format='%(asctime)s %(message)s',
    handlers=[
        logging.FileHandler('log.log'),
    ])  

This is for setting up the logger.

And this next line would monkey-patch print

print = logging.info

Better is of course to explicitly call logging.whatever() where ever one wants to log something.