'zulip_events_email_senders' spawned repeatedly after exit status 1 and unable to add new users

[aksagg]

I'm running the Zulip docker stack I shamelessly copied from @so-rose in response to an issue.
The services starts up and i'm able to access it but, logs show that 'zulip_events_email_senders' spawned repeatedly after exit status 1. Here is the snippet from the log:

Operations to perform:
  Apply all migrations: analytics, auth, confirmation, contenttypes, otp_static, otp_totp, sessions, social_django, two_factor, zerver
Running migrations:
  No migrations to apply.
Zulip migration succeeded.
Post setup scripts execution ...
No post-setup.d folder found. Continuing.
=== End Bootstrap Phase ===
=== Begin Run Phase ===
Certbot is not scheduled to run.
Starting Zulip using supervisor with "/etc/supervisor/supervisord.conf" config ...
2023-07-11 23:56:44,363 CRIT Supervisor is running as root.  Privileges were not dropped because no user is specified in the config file.  If you intend to run as root, you can set user=root in the config file to avoid this message.
2023-07-11 23:56:44,363 WARN No file matches via include "/etc/supervisor/conf.d/*.conf"
2023-07-11 23:56:44,363 INFO Included extra file "/etc/supervisor/conf.d/zulip/cron.conf" during parsing
2023-07-11 23:56:44,364 INFO Included extra file "/etc/supervisor/conf.d/zulip/go-camo.conf" during parsing
2023-07-11 23:56:44,364 INFO Included extra file "/etc/supervisor/conf.d/zulip/nginx.conf" during parsing
2023-07-11 23:56:44,364 INFO Included extra file "/etc/supervisor/conf.d/zulip/smokescreen.conf" during parsing
2023-07-11 23:56:44,364 INFO Included extra file "/etc/supervisor/conf.d/zulip/zulip-once.conf" during parsing
2023-07-11 23:56:44,364 INFO Included extra file "/etc/supervisor/conf.d/zulip/zulip.conf" during parsing
2023-07-11 23:56:44,364 INFO Included extra file "/etc/supervisor/conf.d/zulip/zulip_db.conf" during parsing
2023-07-11 23:56:44,369 INFO RPC interface 'supervisor' initialized
2023-07-11 23:56:44,370 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2023-07-11 23:56:44,370 INFO supervisord started with pid 13
...
2023-07-11 23:56:47,632 INFO success: zulip_events_user_presence entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-07-11 23:57:07,051 INFO exited: zulip_events_email_senders (exit status 1; not expected)
2023-07-11 23:57:08,055 INFO spawned: 'zulip_events_email_senders' with pid 258
2023-07-11 23:57:09,057 INFO success: zulip_events_email_senders entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-07-11 23:57:14,772 INFO exited: zulip_events_email_senders (exit status 1; not expected)
2023-07-11 23:57:15,776 INFO spawned: 'zulip_events_email_senders' with pid 262
2023-07-11 23:57:16,778 INFO success: zulip_events_email_senders entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-07-11 23:57:23,965 INFO exited: zulip_events_email_senders (exit status 1; not expected)

I thought this was an SMTP settings issue so I replaced the settings with :

SETTING_EMAIL_BACKEND : "django.core.mail.backends.filebased.EmailBackend"
SETTING_EMAIL_FILE_PATH: "/var/log/zulip/emails"

as explained in the docs.

I was able to generate the new org creation link by running su zulip -c /home/zulip/deployments/current/manage.py generate_realm_creation_link'` and create the organization. But I'm not able to add new users. What am I doing wrong?

Here is my stack file for Zulip

version: "3.8"

services:
  zulip-db:
    image: "zulip/zulip-postgresql:14" 
    restart: unless-stopped
    secrets:
      - chat__postgres_pass
    environment:
      POSTGRES_DB: "zulip"
      POSTGRES_USER: "zulip"
      POSTGRES_PASSWORD_FILE: /run/secrets/chat__postgres_pass
    volumes:
      - $DOCKERDIR/appdata/chat/db:/var/lib/postgresql/data:rw
    networks:
      - private
  
  zulip-memcached:
    image: memcached:alpine
    restart: unless-stopped
    command:
      - "sh"
      - "-euc"
      - |
        echo 'mech_list: plain' > "$$SASL_CONF_PATH"
        echo "zulip@$$HOSTNAME:$$(cat $$MEMCACHED_PASSWORD_FILE)" > "$$MEMCACHED_SASL_PWDB"
        echo "zulip@localhost:$$(cat $$MEMCACHED_PASSWORD_FILE)" >> "$$MEMCACHED_SASL_PWDB"
        exec memcached -S
    secrets:
      - chat__memcached_pass
    environment:
      SASL_CONF_PATH: "/home/memcache/memcached.conf"
      MEMCACHED_SASL_PWDB: "/home/memcache/memcached-sasl-db"
      MEMCACHED_PASSWORD_FILE: /run/secrets/chat__memcached_pass
    networks:
      - private
  
  zulip-rabbitmq:
    image: rabbitmq:3.7.7
    restart: unless-stopped
    secrets:
      - chat__rabbitmq_pass
    environment:
      RABBITMQ_DEFAULT_USER: "zulip"
      RABBITMQ_DEFAULT_PASS_FILE: /run/secrets/chat__rabbitmq_pass
    volumes:
      - $DOCKERDIR/appdata/chat/rabbitmq:/var/lib/rabbitmq:rw
    networks:
      - private
  
  zulip-redis:
    image: redis:alpine
    secrets:
      - chat__redis_pass
    command:
      - "sh"
      - "-euc"
      - |
        echo "requirepass '$$(cat $$REDIS_PASS_FILE)'" > /etc/redis.conf
        exec redis-server /etc/redis.conf
    environment:
      REDIS_PASS_FILE: /run/secrets/chat__redis_pass
    volumes:
      - $DOCKERDIR/appdata/chat/redis:/data:rw
    networks:
      - private
  
  zulip:
    image: zulip/docker-zulip:5.7-0
    entrypoint: []
    command: [
      "/bin/sh", "-c",
      '
      export SECRETS_rabbitmq_password="$$(cat $$SECRETS_rabbitmq_password_FILE)" &&
      export SECRETS_postgres_password="$$(cat $$SECRETS_postgres_password_FILE)" &&
      export SECRETS_memcached_password="$$(cat $$SECRETS_memcached_password_FILE)" &&
      export SECRETS_redis_password="$$(cat $$SECRETS_redis_password_FILE)" &&
      export SECRETS_secret_key="$$(cat $$SECRETS_secret_key_FILE)" &&
      export SECRETS_email_password="$$(cat $$SECRETS_email_password_FILE)" &&
      /sbin/entrypoint.sh app:run
      '
    ]
    ulimits:
      nofile:
        soft: 1000000
        hard: 1048576
    secrets:
      - chat__postgres_pass
      - chat__memcached_pass
      - chat__rabbitmq_pass
      - chat__redis_pass
      - chat__secret_key
      - chat__email_pass
    environment:
      DB_HOST: "zulip-db"
      DB_HOST_PORT: "5432"
      DB_USER: "zulip"
      
      SETTING_MEMCACHED_LOCATION: "zulip-memcached:11211"
      SETTING_RABBITMQ_HOST: "zulip-rabbitmq"
      SETTING_REDIS_HOST: "zulip-redis"
      
      SECRETS_rabbitmq_password_FILE: /run/secrets/chat__rabbitmq_pass
      SECRETS_postgres_password_FILE: /run/secrets/chat__postgres_pass
      SECRETS_memcached_password_FILE: /run/secrets/chat__memcached_pass
      SECRETS_redis_password_FILE: /run/secrets/chat__redis_pass
      
      SECRETS_secret_key_FILE: /run/secrets/chat__secret_key
      SECRETS_email_password_FILE: /run/secrets/chat__email_pass
      
      DISABLE_HTTPS: "True"
      SSL_CERTIFICATE_GENERATION: "self-signed"
      SETTING_EXTERNAL_HOST: "chat.$DOMAINNAME_CLOUD_SERVER"
      
      SETTING_ZULIP_ADMINISTRATOR: $ADMIN_EMAIL__USERNAME
      SETTING_NOREPLY_EMAIL_ADDRESS: $NOREPLY_EMAIL__USERNAME
      SETTING_ADD_TOKENS_TO_NOREPLY_ADDRESS: "False"
      # SMTP
      # SETTING_EMAIL_HOST: $EMAIL__HOST
      # SETTING_EMAIL_PORT: $EMAIL__PORT
      # SETTING_EMAIL_USE_SSL: "True"
      # SETTING_EMAIL_HOST_USER: $EMAIL__USERNAME
      # https://zulip.readthedocs.io/en/latest/production/email.html#logging-outgoing-email-to-a-file-for-prototyping
      SETTING_EMAIL_BACKEND : "django.core.mail.backends.filebased.EmailBackend"
      SETTING_EMAIL_FILE_PATH: "/var/log/zulip/emails"
      #AUTH
      ZULIP_AUTH_BACKENDS: "EmailAuthBackend"
      # Uncomment this when configuring the mobile push notifications service
      # SETTING_PUSH_NOTIFICATION_BOUNCER_URL: 'https://push.zulipchat.com'
    volumes:
      - $DOCKERDIR/appdata/chat/zulip:/data:rw
      - $DOCKERDIR/appdata/chat/zulip/emails:/var/log/zulip/emails:rw
    labels:
      - "traefik.enable=true"
      ## HTTP Routers
      - "traefik.http.routers.zulip-rtr.entrypoints=https"
      - "traefik.http.routers.zulip-rtr.rule=Host(`chat.$DOMAINNAME_CLOUD_SERVER`)"
      - "traefik.http.routers.zulip-rtr.tls=true"
      ## Middlewares
      - "traefik.http.routers.zulip-rtr.middlewares=chain-no-auth@file"
      ## HTTP Services
      - "traefik.http.routers.zulip-rtr.service=zulip-svc"
      - "traefik.http.services.zulip-svc.loadbalancer.server.port=80"
    networks:
      - t2_proxy
      - private
    
<boiler plate>

[alexmv]

Check the logs on the container, in /var/log/zulip/events_email_senders.log?

You may have better success getting suppoer for this sort of production issue on our development community, in #production help.

[alexmv]

See also #300.

[alexmv]

I'm going to resolve this issue; we can use #300 if there are general improvements to make, and we can help debug your specific issue in #production help if need be.