Comments included as part of minfds value #1230

mnaberez · 2019-04-17T15:07:10Z

@BillMedernach wrote in #1229 (comment):

I'm running into the same error on the minfds config param where the comment is getting parsed as part of the configuration value. If I downgrade from 4.0.0 or 4.0.1 to 3.3.5 the issue goes away.

BillMedernach · 2019-04-17T15:19:35Z

Error: Error: invalid literal for long() with base 10: '1024 ; (min. avail startup file descriptors;default 1024)'

Configuration excerpt:

[supervisord]
logfile=/mnt/md0/server/logs/supervisord.log ; (main log file;default $CWD/supervisord.log)
logfile_maxbytes=50MB                              ; (max main logfile bytes b4 rotation;default 50MB)
logfile_backups=10                                 ; (num of main logfile rotation backups;default 10)
loglevel=info                                      ; (log level;default info; others: debug,warn,trace)
pidfile=/mnt/md0/server/logs/supervisord.pid ; (supervisord pidfile;default supervisord.pid)
nodaemon=false                                     ; (start in foreground if true;default false)
minfds=1024                                        ; (min. avail startup file descriptors;default 1024)
minprocs=200                                       ; (min. avail process descriptors;default 200)

Removing the comment or downgrading to 3.3.5 fixes the problem.

mnaberez · 2019-04-17T15:25:02Z

@BillMedernach What Python version are you using (python -V)?

References #1229 References #1230

BillMedernach · 2019-04-17T15:45:31Z

@mnaberez We're running 2.7.15

mnaberez · 2019-04-17T16:39:03Z

I added tests for this case (minfds) and it passes on CI. The CI environment is using 2.7.14. I was not able to reproduce this locally yet.

Are you able to consistently reproduce it with the config above?

BillMedernach · 2019-04-17T16:50:03Z

Are you able to consistently reproduce it with the config above?

Yes, that config always produces that error on v4.0.0+.

mnaberez · 2019-04-17T17:01:22Z

Could you please attach the actual supervisord.conf file where you're seeing this happen? You can drag it into the GitHub comment box, or alternatively paste the output of either hexdump supervisord.conf or base64 supervisord.conf. The tests that I added cover both spaces and tabs before the semicolon, so I don't know what is wrong. I will try to reproduce it using your byte-for-byte exact config file. Hopefully that reproduces it and then we look into what's wrong.

BillMedernach · 2019-04-17T17:06:47Z

[unix_http_server]
file=/tmp/supervisor.sock   ; (the path to the socket file)
chmod=0700                 ; sockef file mode (default 0700)
;chown=nobody:nogroup       ; socket file uid:gid owner
;username=REDACTED              ; (default is no username (open server))
;password=REDACTED               ; (default is no password (open server))

[supervisord]
logfile=/mnt/md0/server/logs/supervisord.log ; (main log file;default $CWD/supervisord.log)
logfile_maxbytes=50MB                              ; (max main logfile bytes b4 rotation;default 50MB)
logfile_backups=10                                 ; (num of main logfile rotation backups;default 10)
loglevel=info                                      ; (log level;default info; others: debug,warn,trace)
pidfile=/mnt/md0/server/logs/supervisord.pid ; (supervisord pidfile;default supervisord.pid)
nodaemon=false                                     ; (start in foreground if true;default false)
minfds=1024                                        ; (min. avail startup file descriptors;default 1024)
minprocs=200                                       ; (min. avail process descriptors;default 200)

[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface

[supervisorctl]
serverurl=unix:///tmp/supervisor.sock ; use a unix:// URL  for a unix socket
history_file=~/.sc_history            ; use readline history if available

[program:dyno]
command=/srv/server/bin/dyno foreground
directory=/srv/server
user=root
startsecs=10
autorestart=true
stopsignal=TERM
stopwaitsecs=3600
redirect_stderr=true
stdout_logfile=/mnt/md0/server/logs/subprocess.log
environment=PATH=/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin,HOME=/root

[program:check_nginx]
command=/srv/server/bin/check_nginx_keepalive.py
directory=/srv/server
user=root
startsecs=10
autorestart=true
stopsignal=TERM
stopwaitsecs=3600
redirect_stderr=true
stdout_logfile=/mnt/md0/server/logs/subprocess-check-nginx.log
environment=PATH=/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin,HOME=/root

[program:check_server]
command=/srv/server/bin/check_server.py
directory=/srv/server
user=root
startsecs=10
autorestart=true
stopsignal=TERM
stopwaitsecs=3600
redirect_stderr=true
stdout_logfile=/mnt/md0/server/logs/subprocess-check-server.log
environment=PATH=/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin,HOME=/root

mnaberez · 2019-04-17T17:17:08Z

Here's what I get with that config. This is Supervisor 4.0.1 running on Python 2.7.15:

2019-04-17 10:12:48,177 CRIT Supervisor is running as root.  Privileges were not dropped because no user is specified in the config file.  If you intend to run as root, you can set user=root in the config file to avoid this message.
2019-04-17 10:12:48,177 INFO Increased RLIMIT_NOFILE limit to 1024
2019-04-17 10:12:48,190 INFO RPC interface 'supervisor' initialized
2019-04-17 10:12:48,190 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2019-04-17 10:12:48,190 INFO supervisord started with pid 3011
2019-04-17 10:12:49,194 INFO spawnerr: can't find command '/srv/server/bin/dyno'
2019-04-17 10:12:49,195 INFO spawnerr: can't find command '/srv/server/bin/check_server.py'
2019-04-17 10:12:49,195 INFO spawnerr: can't find command '/srv/server/bin/check_nginx_keepalive.py'
2019-04-17 10:12:50,196 INFO spawnerr: can't find command '/srv/server/bin/dyno'
2019-04-17 10:12:50,197 INFO spawnerr: can't find command '/srv/server/bin/check_server.py'
2019-04-17 10:12:50,197 INFO spawnerr: can't find command '/srv/server/bin/check_nginx_keepalive.py'
2019-04-17 10:12:52,200 INFO spawnerr: can't find command '/srv/server/bin/dyno'
2019-04-17 10:12:52,201 INFO spawnerr: can't find command '/srv/server/bin/check_server.py'
2019-04-17 10:12:52,201 INFO spawnerr: can't find command '/srv/server/bin/check_nginx_keepalive.py'
2019-04-17 10:12:55,208 INFO spawnerr: can't find command '/srv/server/bin/dyno'
2019-04-17 10:12:55,209 INFO gave up: dyno entered FATAL state, too many start retries too quickly
2019-04-17 10:12:55,209 INFO spawnerr: can't find command '/srv/server/bin/check_server.py'
2019-04-17 10:12:55,209 INFO gave up: check_server entered FATAL state, too many start retries too quickly
2019-04-17 10:12:55,209 INFO spawnerr: can't find command '/srv/server/bin/check_nginx_keepalive.py'
2019-04-17 10:12:55,209 INFO gave up: check_nginx entered FATAL state, too many start retries too quickly

It is not halting with Error: invalid literal for long() with base 10: '1024 ; (min. avail startup file descriptors;default 1024)'. There are errors from programs not starting but the midfds error would have occurred before it tried to spawn them.

Maybe something is getting lost with copy/paste. Can you please hexdump supervisord.conf or base64 supervisord.conf? I will try it with your byte-for-byte exact config file. Please be certain this is exact config file where you are consistently seeing the error.

BillMedernach · 2019-04-17T17:21:01Z

Here's the base64. I did redact the username and password we use though.

W3VuaXhfaHR0cF9zZXJ2ZXJdCmZpbGU9L3RtcC9zdXBlcnZpc29yLnNvY2sgICA7ICh0aGUgcGF0aCB0byB0aGUgc29ja2V0IGZpbGUpCmNobW9kPTA3MDAgICAgICAgICAgICAgICAgIDsgc29ja2VmIGZpbGUgbW9kZSAoZGVmYXVsdCAwNzAwKQo7Y2hvd249bm9ib2R5Om5vZ3JvdXAgICAgICAgOyBzb2NrZXQgZmlsZSB1aWQ6Z2lkIG93bmVyCjt1c2VybmFtZT1SRURBQ1RFRCAgICAgICAgICAgICAgOyAoZGVmYXVsdCBpcyBubyB1c2VybmFtZSAob3BlbiBzZXJ2ZXIpKQo7cGFzc3dvcmQ9UkVEQUNURUQgICAgICAgICAgICAgICA7IChkZWZhdWx0IGlzIG5vIHBhc3N3b3JkIChvcGVuIHNlcnZlcikpCgpbc3VwZXJ2aXNvcmRdCmxvZ2ZpbGU9L21udC9tZDAvc2VydmVyL2xvZ3Mvc3VwZXJ2aXNvcmQubG9nIDsgKG1haW4gbG9nIGZpbGU7ZGVmYXVsdCAkQ1dEL3N1cGVydmlzb3JkLmxvZykKbG9nZmlsZV9tYXhieXRlcz01ME1CICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgOyAobWF4IG1haW4gbG9nZmlsZSBieXRlcyBiNCByb3RhdGlvbjtkZWZhdWx0IDUwTUIpCmxvZ2ZpbGVfYmFja3Vwcz0xMCAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIDsgKG51bSBvZiBtYWluIGxvZ2ZpbGUgcm90YXRpb24gYmFja3VwcztkZWZhdWx0IDEwKQpsb2dsZXZlbD1pbmZvICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICA7IChsb2cgbGV2ZWw7ZGVmYXVsdCBpbmZvOyBvdGhlcnM6IGRlYnVnLHdhcm4sdHJhY2UpCnBpZGZpbGU9L21udC9tZDAvc2VydmVyL2xvZ3Mvc3VwZXJ2aXNvcmQucGlkIDsgKHN1cGVydmlzb3JkIHBpZGZpbGU7ZGVmYXVsdCBzdXBlcnZpc29yZC5waWQpCm5vZGFlbW9uPWZhbHNlICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIDsgKHN0YXJ0IGluIGZvcmVncm91bmQgaWYgdHJ1ZTtkZWZhdWx0IGZhbHNlKQptaW5mZHM9MTAyNCAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICA7IChtaW4uIGF2YWlsIHN0YXJ0dXAgZmlsZSBkZXNjcmlwdG9ycztkZWZhdWx0IDEwMjQpCm1pbnByb2NzPTIwMCAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIDsgKG1pbi4gYXZhaWwgcHJvY2VzcyBkZXNjcmlwdG9ycztkZWZhdWx0IDIwMCkKCltycGNpbnRlcmZhY2U6c3VwZXJ2aXNvcl0Kc3VwZXJ2aXNvci5ycGNpbnRlcmZhY2VfZmFjdG9yeSA9IHN1cGVydmlzb3IucnBjaW50ZXJmYWNlOm1ha2VfbWFpbl9ycGNpbnRlcmZhY2UKCltzdXBlcnZpc29yY3RsXQpzZXJ2ZXJ1cmw9dW5peDovLy90bXAvc3VwZXJ2aXNvci5zb2NrIDsgdXNlIGEgdW5peDovLyBVUkwgIGZvciBhIHVuaXggc29ja2V0Cmhpc3RvcnlfZmlsZT1+Ly5zY19oaXN0b3J5ICAgICAgICAgICAgOyB1c2UgcmVhZGxpbmUgaGlzdG9yeSBpZiBhdmFpbGFibGUKCltwcm9ncmFtOmR5bm9dCmNvbW1hbmQ9L3Nydi9zZXJ2ZXIvYmluL2R5bm8gZm9yZWdyb3VuZApkaXJlY3Rvcnk9L3Nydi9zZXJ2ZXIKdXNlcj1yb290CnN0YXJ0c2Vjcz0xMAphdXRvcmVzdGFydD10cnVlCnN0b3BzaWduYWw9VEVSTQpzdG9wd2FpdHNlY3M9MzYwMApyZWRpcmVjdF9zdGRlcnI9dHJ1ZQpzdGRvdXRfbG9nZmlsZT0vbW50L21kMC9zZXJ2ZXIvbG9ncy9zdWJwcm9jZXNzLmxvZwplbnZpcm9ubWVudD1QQVRIPS91c3IvbG9jYWwvYmluOi9zYmluOi9iaW46L3Vzci9zYmluOi91c3IvYmluLEhPTUU9L3Jvb3QKCltwcm9ncmFtOmNoZWNrX25naW54XQpjb21tYW5kPS9zcnYvc2VydmVyL2Jpbi9jaGVja19uZ2lueF9rZWVwYWxpdmUucHkKZGlyZWN0b3J5PS9zcnYvc2VydmVyCnVzZXI9cm9vdApzdGFydHNlY3M9MTAKYXV0b3Jlc3RhcnQ9dHJ1ZQpzdG9wc2lnbmFsPVRFUk0Kc3RvcHdhaXRzZWNzPTM2MDAKcmVkaXJlY3Rfc3RkZXJyPXRydWUKc3Rkb3V0X2xvZ2ZpbGU9L21udC9tZDAvc2VydmVyL2xvZ3Mvc3VicHJvY2Vzcy1jaGVjay1uZ2lueC5sb2cKZW52aXJvbm1lbnQ9UEFUSD0vdXNyL2xvY2FsL2Jpbjovc2JpbjovYmluOi91c3Ivc2JpbjovdXNyL2JpbixIT01FPS9yb290CgpbcHJvZ3JhbTpjaGVja19zZXJ2ZXJdCmNvbW1hbmQ9L3Nydi9zZXJ2ZXIvYmluL2NoZWNrX3NlcnZlci5weQpkaXJlY3Rvcnk9L3Nydi9zZXJ2ZXIKdXNlcj1yb290CnN0YXJ0c2Vjcz0xMAphdXRvcmVzdGFydD10cnVlCnN0b3BzaWduYWw9VEVSTQpzdG9wd2FpdHNlY3M9MzYwMApyZWRpcmVjdF9zdGRlcnI9dHJ1ZQpzdGRvdXRfbG9nZmlsZT0vbW50L21kMC9zZXJ2ZXIvbG9ncy9zdWJwcm9jZXNzLWNoZWNrLXNlcnZlci5sb2cKZW52aXJvbm1lbnQ9UEFUSD0vdXNyL2xvY2FsL2Jpbjovc2JpbjovYmluOi91c3Ivc2JpbjovdXNyL2JpbixIT01FPS9yb290Cg==

mnaberez · 2019-04-17T17:32:14Z

$ base64 --decode supervisord.b64 > supervisord.conf

$ openssl sha1 supervisord.conf 
SHA1(supervisord.conf)= 2976fd7b3661f0f49c636a311cd899f2ff18882b

$ python -V
Python 2.7.15

$ supervisord --version 
4.0.1

$ sudo supervisord -n -c supervisord.conf 
2019-04-17 10:23:48,381 CRIT Supervisor is running as root.  Privileges were not dropped because no user is specified in the config file.  If you intend to run as root, you can set user=root in the config file to avoid this message.
2019-04-17 10:23:48,381 INFO Increased RLIMIT_NOFILE limit to 1024
2019-04-17 10:23:48,391 INFO RPC interface 'supervisor' initialized
2019-04-17 10:23:48,391 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2019-04-17 10:23:48,392 INFO supervisord started with pid 3205
2019-04-17 10:23:49,397 INFO spawnerr: can't find command '/srv/server/bin/dyno'
2019-04-17 10:23:49,397 INFO spawnerr: can't find command '/srv/server/bin/check_server.py'
2019-04-17 10:23:49,397 INFO spawnerr: can't find command '/srv/server/bin/check_nginx_keepalive.py'
2019-04-17 10:23:50,402 INFO spawnerr: can't find command '/srv/server/bin/dyno'

I believe I'm using the same Supervisor version (4.0.1), same Python version (2.7.15), and a byte-for-byte identical config file as you. I can't get the error to happen. I'd appreciate if you could please confirm all this on your setup again. After that, I'm not sure, any ideas would be appreciated.

BillMedernach · 2019-04-17T17:47:24Z

@mnaberez My apologies. I didn't have a way to get the actual host this was running on before but now I have an exact replica of it. The issue is still occurring but we're actually running Python 2.7.16.

Small difference, but perhaps the issue lies there.

I can also confirm this does not happen with v3.4.0 by the way.

jenyak · 2019-04-17T18:17:43Z

I have same error
supervisord -n -c supervisord.conf
Error: invalid literal for long() with base 10: '1024 ; min. avail startup file descriptors; default 1024'
Python 2.7.16 supervisord 4.0.1

I remove everything, my supervisord.conf only has

[supervisord]
minfds=1024                  ; min. avail startup file descriptors; default 1024

mnaberez · 2019-04-17T19:47:12Z

I was able to reproduce it. Here's what I tried:

Fresh VM running Ubuntu 18.04.2 Bionic, with Python 2.7.15rc1 installed via apt install python-minimal python-pip and Supervisor 4.0.1 installed via pip install supervisor. The error does not occur.
Fresh VM running Ubuntu 19.04 Disco (development branch), with Python 2.7.16 installed via apt install python-minimal python-pip and Supervisor 4.0.1 installed via pip install supervisor. The error occurs.

This is the exact config file that I used in both cases:

$ cat supervisord.conf
[supervisord]
minfds=1024			; min. avail startup file descriptors; default 1024

$ hexdump -C supervisord.conf
00000000  5b 73 75 70 65 72 76 69  73 6f 72 64 5d 0a 6d 69  |[supervisord].mi|
00000010  6e 66 64 73 3d 31 30 32  34 09 09 09 3b 20 6d 69  |nfds=1024...; mi|
00000020  6e 2e 20 61 76 61 69 6c  20 73 74 61 72 74 75 70  |n. avail startup|
00000030  20 66 69 6c 65 20 64 65  73 63 72 69 70 74 6f 72  | file descriptor|
00000040  73 3b 20 64 65 66 61 75  6c 74 20 31 30 32 34 0a  |s; default 1024.|
00000050

$ base64 supervisord.conf
W3N1cGVydmlzb3JkXQptaW5mZHM9MTAyNAkJCTsgbWluLiBhdmFpbCBzdGFydHVwIGZpbGUgZGVz
Y3JpcHRvcnM7IGRlZmF1bHQgMTAyNAo=

$ sha1sum supervisord.conf
SHA1(supervisord.conf)= 021443d5a887781b2cdc3e73f4f5bd1de0f59ce4

mnaberez · 2019-04-17T20:19:58Z

This error occurs when a configparser package is available on a Python 2 installation. On the fresh Ubuntu 19.04 install noted above:

Python 2.7.16 (default, Apr  6 2019, 01:42:57)
[GCC 8.3.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import configparser
>>> configparser.__doc__
u'Convenience module importing everything from backports.configparser.'

In Python 3, the ConfigParser module was renamed to configparser. Starting with Supervisor 4.0.0, we try to import one or the other for 2/3 compatibility. Our code is not expecting to find a configparser module on Python 2. On the Ubuntu 19.04 system that I tried, and apparently your systems, it is importing the wrong module. If you run the Supervisor test suite (python setup.py test) on such a system, there are many test failures from config parsing.

It can be fixed with this patch against master (d3f48a8):

diff --git a/supervisor/compat.py b/supervisor/compat.py
index 25b5e15..ef78033 100644
--- a/supervisor/compat.py
+++ b/supervisor/compat.py
@@ -62,9 +62,9 @@ except ImportError: # pragma: no cover
     syslog = None
 
 try: # pragma: no cover
-    import configparser as ConfigParser
+    import ConfigParser
 except ImportError: # pragma: no cover
-   import ConfigParser
+    import configparser as ConfigParser

I will test it further and then make a point release with this fix.

BillMedernach · 2019-04-17T20:36:06Z

Fantastic. Thanks for all your help!

mnaberez · 2019-04-17T21:11:56Z

Supervisor 4.0.2 (PyPI package; includes this fix)

@BillMedernach @jenyak @sdanbury Thank you!

sdanbury · 2019-05-03T08:46:03Z

Great find! Good work.

References Supervisor#1229 References Supervisor#1230 (cherry picked from commit a1643db)

(cherry picked from commit 2c2cd35)

mnaberez mentioned this issue Apr 17, 2019

Comments included as part of logfile path #1229

Closed

mnaberez added a commit that referenced this issue Apr 17, 2019

Add more tests for inline comments in config

a1643db

References #1229 References #1230

This comment has been minimized.

Sign in to view

mnaberez closed this as completed in 2c2cd35 Apr 17, 2019

mnaberez added a commit that referenced this issue Apr 17, 2019

Add changelog entry for 2c2cd35 (#1230)

bf7b4b2

mnaberez added a commit that referenced this issue Apr 17, 2019

Add test env for Python 2.7 with configparser (#1230)

6bd04b6

alexsilva pushed a commit to alexsilva/supervisor that referenced this issue Aug 22, 2019

Add more tests for inline comments in config

0c9348e

References Supervisor#1229 References Supervisor#1230 (cherry picked from commit a1643db)

alexsilva pushed a commit to alexsilva/supervisor that referenced this issue Aug 22, 2019

Fix importing stdlib ConfigParser on Python 2. Closes Supervisor#1230

385d09c

(cherry picked from commit 2c2cd35)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Comments included as part of minfds value #1230

Comments included as part of minfds value #1230

mnaberez commented Apr 17, 2019

BillMedernach commented Apr 17, 2019 •

edited

Loading

mnaberez commented Apr 17, 2019

BillMedernach commented Apr 17, 2019

mnaberez commented Apr 17, 2019

BillMedernach commented Apr 17, 2019

mnaberez commented Apr 17, 2019

BillMedernach commented Apr 17, 2019

mnaberez commented Apr 17, 2019 •

edited

Loading

BillMedernach commented Apr 17, 2019

mnaberez commented Apr 17, 2019

BillMedernach commented Apr 17, 2019

This comment has been minimized.

jenyak commented Apr 17, 2019 •

edited

Loading

mnaberez commented Apr 17, 2019

This comment has been minimized.

mnaberez commented Apr 17, 2019

BillMedernach commented Apr 17, 2019

mnaberez commented Apr 17, 2019

sdanbury commented May 3, 2019

Comments included as part of minfds value #1230

Comments included as part of minfds value #1230

Comments

mnaberez commented Apr 17, 2019

BillMedernach commented Apr 17, 2019 • edited Loading

mnaberez commented Apr 17, 2019

BillMedernach commented Apr 17, 2019

mnaberez commented Apr 17, 2019

BillMedernach commented Apr 17, 2019

mnaberez commented Apr 17, 2019

BillMedernach commented Apr 17, 2019

mnaberez commented Apr 17, 2019 • edited Loading

BillMedernach commented Apr 17, 2019

mnaberez commented Apr 17, 2019

BillMedernach commented Apr 17, 2019

This comment has been minimized.

jenyak commented Apr 17, 2019 • edited Loading

mnaberez commented Apr 17, 2019

This comment has been minimized.

mnaberez commented Apr 17, 2019

BillMedernach commented Apr 17, 2019

mnaberez commented Apr 17, 2019

sdanbury commented May 3, 2019

BillMedernach commented Apr 17, 2019 •

edited

Loading

mnaberez commented Apr 17, 2019 •

edited

Loading

jenyak commented Apr 17, 2019 •

edited

Loading