Docker auto restart

[naanlizard]

Today our OME crashed (see #1639) - we have OME running in docker with docker compose, here's the relevant bit of the compose file

services:
  ovenmediaengine:
      image: privateregistry/hostname/ovenmediaengine:production
      hostname: hostname-ome-production
      container_name: hostname-ome-production
      networks:
          servername-docker-network:
              ipv4_address: 172.20.0.18
      ports:
          - "1935:1935/tcp" # RTMP ingest
          - "3333:3333"     # WebRTC Signaling - I believe unused?
          - "3334:3334"     # WebRTC TLS
          - "3478:3478/tcp" # WebRTC TURN
          - "10000-10005:10000-10005/udp" # WebRTC Candidate
#          - "799:799"       # HTTP LLHLS
#          - "800:800"       # HTTPS LLHLS
      env_file: .env
      volumes:
          - $CERT_FOLDER_PATH:/etc/letsencrypt:ro
          - $DVR_PATH:/tmp/ome_dvr
          - $RECORDINGS_ROOT_DIR:$RECORDINGS_ROOT_DIR
      restart: always
      deploy:
        resources:
          limits:
            memory: 50G
      logging:
        driver: "loki"
        options:
          max-size: "10g"
          max-file: "3"
          loki-url: http://hostname-loki-production:3100/loki/api/v1/push
          loki-max-backoff: 800ms
          loki-retries: '2'

Now, the container didn't actually restart despite the complete crash of OME, which I would think it should. Docker showed exited iirc in the docker status, though I foolishly didn't double check this as I was in a rush to get things working again.

Anyone understand what might make it such that OME doesn't restart automatically like it should? In the linked issue, I include our dockerfile, perhaps it's the entrypoint bit screwing things up?

[bchah]

All I can confirm is that I've seen the same behavior, where OME in docker does not restart after a crash even with restart: always enabled.

[bchah]

Yeah you're probably right there - my entry point script starts up OME plus a million other things. Come to think of it I haven't tried crashing the vanilla OME docker image so this problem might be unique to us! 😅

[naanlizard]

If you can crash ome on demand right now with that bug, could I ask you to try it with the vanilla image?

[bchah]

Untitled.mp4

Yep, it's our entry point scripts. This makes sense since Docker won't terminate the container until the entry point command exits. So if our scripts merely launch OME (among other things) then the container hasn't really crashed.

Depending on how your script works you could get the PID of OME immediately after it is launched, like:

# ome starts
PID=$!
# do your stuff
wait $PID
exit 0

That should cause the container to restart if OME exits for any reason.

[naanlizard]

I don't think we actually do anything in our entrypoint, can you give me a sanity check here?

Dockerfile

entrypoint.sh

While the default CMD in the OME dockerfile is just

CMD ["/opt/ovenmediaengine/bin/OvenMediaEngine", "-c", "origin_conf"]

So in our case I think we can just remove the CMD line in our custom Dockerfile, right? Or would it need to be the default CMD from the Dockerfile 🤔

[bchah]

I think your primary process is the bash shell itself rather than OME because of how your CMD is constructed in your Dockerfile:

CMD ["bash", "entrypoint.sh"]

Try removing that and running the default CMD instead?