Data export failing on self-hosted instance

[edenist]

This question is a follow up to an issue I opened a couple of weeks ago in regards to data export on our local deployment of polis.

#1438

That ended up being due to an edge-case of running polis without any AWS infrastructure and was quickly fixed [Thank you @metasoarous!]
I am still having issues getting data export to work, however I do not think it is currently due to a bug, but more likely due to a missing configuration within our deployment.

This is the command I am running for data export:

docker compose -f docker-compose.yml -f docker-compose.dev.yml run math clojure -M:run export -Z 5umbm9ayfx -f 5umbm9ayfx-test1.zip

And the console output and error which results:

WARN[0000] The "GIT_HASH" variable is not set. Defaulting to a blank string. 
[+] Running 1/0
 ⠿ Container polis-postgres  Running                                                                                                                                                                                                         0.0s
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
WARNING: min already refers to: #'clojure.core/min in namespace: polismath.math.clusters, being replaced by: #'clojure.core.matrix.operators/min
WARNING: max already refers to: #'clojure.core/max in namespace: polismath.math.clusters, being replaced by: #'clojure.core.matrix.operators/max
WARNING: min already refers to: #'clojure.core/min in namespace: polismath.math.conversation, being replaced by: #'clojure.core.matrix.operators/min
WARNING: max already refers to: #'clojure.core/max in namespace: polismath.math.conversation, being replaced by: #'clojure.core.matrix.operators/max
WARNING: min already refers to: #'clojure.core/min in namespace: polismath.math.corr, being replaced by: #'clojure.core.matrix.operators/min
WARNING: max already refers to: #'clojure.core/max in namespace: polismath.math.corr, being replaced by: #'clojure.core.matrix.operators/max
22-07-15 07:03:10 6e66949c0b63 INFO [polismath.runner:200] - Runner main function executed
22-07-15 07:03:10 6e66949c0b63 INFO [polismath.system:47] - export system overrides: {:math-env :prod, :export {:temp-dir "."}}
22-07-15 07:03:10 6e66949c0b63 INFO [polismath.components.config:161] - >> Starting config component
22-07-15 07:03:10 6e66949c0b63 INFO [polismath.components.core-matrix-boot:39] - >> Starting CoreMatrixBooter with implementation: :vectorz
22-07-15 07:03:11 6e66949c0b63 INFO [polismath.components.postgres:58] - >> Starting Postgres component
22-07-15 07:03:11 6e66949c0b63 INFO [polismath.conv-man:425] - >> Starting ConversationManager
22-07-15 07:03:11 6e66949c0b63 INFO [polismath.darwin.core:236] - Skipping AWS connection
22-07-15 07:03:11 6e66949c0b63 INFO [polismath.runner:163] - config :prod
22-07-15 07:03:11 6e66949c0b63 INFO [polismath.runner:164] - config {:expiry-days 6, :temp-dir ".", :private-url-base "http://localhost:8080"}
22-07-15 07:03:11 6e66949c0b63 INFO [polismath.darwin.export:604] - Exporting data for zid = nil , zinvite = 5umbm9ayfx
22-07-15 07:03:11 6e66949c0b63 DEBUG [polismath.components.postgres:106] - get-zid-from-zinvite for zinvite 5umbm9ayfx
Exception in thread "main" java.lang.IllegalArgumentException: db-spec null is missing a required parameter
	at clojure.java.jdbc$get_connection.invokeStatic(jdbc.clj:309)
	at clojure.java.jdbc$get_connection.invoke(jdbc.clj:193)
	at clojure.java.jdbc$db_query_with_resultset.invokeStatic(jdbc.clj:848)
	at clojure.java.jdbc$db_query_with_resultset.invoke(jdbc.clj:818)
	at clojure.java.jdbc$query.invokeStatic(jdbc.clj:874)
	at clojure.java.jdbc$query.invoke(jdbc.clj:854)
	at clojure.java.jdbc$query.invokeStatic(jdbc.clj:867)
	at clojure.java.jdbc$query.invoke(jdbc.clj:854)
	at polismath.components.postgres$query.invokeStatic(postgres.clj:53)
	at polismath.components.postgres$query.invoke(postgres.clj:47)
	at polismath.components.postgres$query.invokeStatic(postgres.clj:52)
	at polismath.components.postgres$query.invoke(postgres.clj:47)
	at polismath.components.postgres$get_zid_from_zinvite.invokeStatic(postgres.clj:108)
	at polismath.components.postgres$get_zid_from_zinvite.invoke(postgres.clj:104)
	at polismath.darwin.export$get_export_data.invokeStatic(export.clj:567)
	at polismath.darwin.export$get_export_data.invoke(export.clj:565)
	at polismath.darwin.export$export_conversation.invokeStatic(export.clj:607)
	at polismath.darwin.export$export_conversation.invoke(export.clj:597)
	at polismath.runner$run_export.invokeStatic(runner.clj:195)
	at polismath.runner$run_export.invoke(runner.clj:158)
	at polismath.runner$_main.invokeStatic(runner.clj:224)
	at polismath.runner$_main.doInvoke(runner.clj:199)
	at clojure.lang.RestFn.applyTo(RestFn.java:137)
	at clojure.lang.Var.applyTo(Var.java:705)
	at clojure.core$apply.invokeStatic(core.clj:665)
	at clojure.main$main_opt.invokeStatic(main.clj:491)
	at clojure.main$main_opt.invoke(main.clj:487)
	at clojure.main$main.invokeStatic(main.clj:598)
	at clojure.main$main.doInvoke(main.clj:561)
	at clojure.lang.RestFn.applyTo(RestFn.java:137)
	at clojure.lang.Var.applyTo(Var.java:705)
	at clojure.main.main(main.java:37)

I've been trying to debug the backtrace [clojure is not my forte, heh], and I suspect the object 'component' being passed to db-spec jdbc/query is empty when it should contain the database parameters.

Given I have tried re-deploying a near stock instance fresh from git with only changes to my polis.config.js files in client-{admin, participation, report} and the fs-config.json in file server to reflect the values for our local server, I suspect there are some additional settings I must be missing if this error is still occurring.

I have tried running with MATH_ENV=prod and dev in math/docker-dev.env and server/docker-dev.env and neither value makes a difference to this.
Likewise, I have tried running docker compose with -f docker-compose.dev.yml and without, which also makes no difference. Is the docker-compose.dev.yml required when running polis and/or the export command?

Sorry for the mouthful of questions. I've been pushing through this deployment and I believe this is the last piece of the puzzle for us.
If anyone has any light to shed on this matter it would be incredibly appreciated.

[metasoarous]

Hi @edenist.

Thanks for continuing to push through this issue; It'll be great to have this smoothed out for others.

I think you are correct about the problem being that the database config/credentials are not making it through to the math worker.

Given that, here are some things I would check:

• You should have a DATABASE_URL entry in your math/docker-dev.env, matching the configuration in server.
• Does the math worker pick up on the database when running the entire system? If you are seeing the visualization update in the participation view when you submit votes, then things should be configured properly.
• Is the database container running when you execute the command you specified? I think the error might look different if that was the problem, but if the above doesn't get you through the issue, I'd try making sure that the database container is actually running first by executing docker compose -f docker-compose.yml -f docker-compose.dev.yml run postgres in a separate shell.

Please let me know if that gets you through the problem, and feel free to submit a PR if you think the documentation could be improved anywhere.

Thanks again!

[edenist]

Hi @metasoarous,

I have double-checked all of the points you mentioned.

• DATABASE_URL is defined and matches in both math/docker-dev.env and server/docker-dev.env
• The visualization does appear for participants [after 7 people have submitted a response]. I can assume that this means the math worker does detect the database correctly. I certainly can't see any error messages to suggest otherwise.
• The database container is running prior to executing that command, for clarity here is the output from docker ps before and after running the command [I have had cases of the math worker crashing under certain configurations].

polis@polis-dev:~/polis$ docker ps 
CONTAINER ID   IMAGE                           COMMAND                  CREATED      STATUS         PORTS                                                                      NAMES
9dc8d42f9404   compdem/polis-nginx-proxy:dev   "/docker-entrypoint.…"   5 days ago   Up 4 minutes   0.0.0.0:80->80/tcp, :::80->80/tcp, 0.0.0.0:443->443/tcp, :::443->443/tcp   polis-nginx-proxy
9aebbef0d5ac   maildev/maildev:1.1.0           "bin/maildev --web 8…"   5 days ago   Up 4 minutes   0.0.0.0:25->25/tcp, :::25->25/tcp, 0.0.0.0:1080->80/tcp, :::1080->80/tcp   polis-maildev-1
108285dace54   compdem/polis-server:dev        "docker-entrypoint.s…"   5 days ago   Up 4 minutes   5000/tcp, 0.0.0.0:9229->9229/tcp, :::9229->9229/tcp                        polis-server-1
a5e81073c7fa   compdem/polis-postgres:dev      "docker-entrypoint.s…"   5 days ago   Up 4 minutes   5432/tcp                                                                   polis-postgres
659b7f8ec991   compdem/polis-file-server:dev   "docker-entrypoint.s…"   5 days ago   Up 4 minutes   8080/tcp                                                                   polis-file-server

Suffice to say none of those issues appear to me to be the cause of this error.

I'm not sure if is relevant, but in the error I reported previously, when the export failed due to an assertion on aws details, I can see the following db-spec object in the error log:

:db-spec {:classname "org.postgresql.Driver", :subprotocol "postgresql", :subname "//postgres:5432/polis-dev", :make-pool? true, :user "postgres", :password "REDACTED", :ssl false, :sslfactory "org.postgresql.ssl.NonValidatingFactory"}

Could it thus be something falling over expecting AWS details elsewhere in the code, which was missed when you made the AWS details optional?

Thanks once again for your support with this. This is the last hurdle to get the site production-ready for us, and yes I've got some PRs we plan on submitting based on our experience getting everything running which will hopefully help others :-)

[metasoarous]

Thanks for your persistence here.

I just pushed an update to the dev branch which should fix this issue. When I added a conditional check to see if there are AWS credentials to connect with, I failed to ensure that we were continuing to return the rest of the configuration data to the system component handling the export, which explains the issue here.

Please let me know if this works for you now.

[edenist]

This commit has now fixed the issue. Our export function is now working for us.
Thank you so much for your work in getting this working, it is immensely appreciated!

[ebarry]

Thanks everyone for carefully and generously working through this issue 🌳