Getting errors on dashboard

[qq20230530]

Not all the time, but sometimes get such error:

When executing the same queries alone, everything is fine, no errors.
But on dashboard, sometimes get errors.
Does anyone know what the reason might be?

[justinclift]

Hopefully you have command line access to the server this is happening on?

If you do, then (in theory) you should be able to extract the error message info from the backend logs.

The exact process for doing that though can vary. Any idea how your Redash was installed?

eg Docker containers vs something else (etc) 😄

[qq20230530]

Thank you for looking at my question.
I have command line access.
Installed Redash using Amazon AMI.
ap-northeast-1(Tokyo region)

[justinclift]

Ahhh. I'm not personally familiar with the Amazon AMI version of Redash, so might not be super helpful here.

If it's something that uses Docker for running the Redash pieces though, then we might be ok.

With a Docker based approach, most of the time Redash gets installed into the /opt/redash directory.

So, you could change into that directory then run docker-compose logs > somefile.txt to capture the logs to a file you can do stuff with.

For example, you can then search that file for the string "error" with a timestamp of about when it happened. The file should contain the error messages for whatever happened on the back end. 😄

If the Amazon AMI doesn't use a docker based approach though... then I'm not sure. 😬

[qq20230530]

Thank you for giving me such a detailed answer.
Will check tomorrow.
On my server, Redash is also installed in /opt/Redash.
Hopefully I can find something useful😊

[qq20230530]

Followed your instruction and extraced logs successfully.
Thanks a lot!!!

[justinclift]

Awesome. 😁

[gaecoli]

This kind of error is generally caused by a timeout or an error encountered during SQL operation that has not been handled friendly. You can post your log, and the community will help solve it

[qq20230530]

Interestingly, yesterday, both my colleague and I got this intermittent error on dashboard. But since this morning, I no longer encounter this error on dashboard, but my colleague still does.
Does the internet connection also matter??
We are both working from home, so our internet environment is differrent.
These are the errors I found in the logs:
Have been staring at them all the morning, and am about to faint......Please help me out! Thank you!!!

�[36mnginx_1 |�[0m 2023/06/28 07:19:43 [warn] 6#6: *448 an upstream response is buffered to a temporary file /var/cache/nginx/proxy_temp/3/00/0000000003 while reading upstream, client: xxx.xx.xxx.xx, server: , request: "GET /api/queries/10/results/155.json HTTP/1.1", upstream: "http://172.18.0.8:5000/api/queries/10/results/155.json", host: "xx.xxx.xxx.xxx", referrer: "http://xx.xxx.xxx.xxx/queries/10"

[36;1mpostgres_1 |�[0m LOG: unexpected EOF on client connection with an open transaction

�[33mserver_1 |�[0m [2023-06-28 07:36:48,261][PID:50][ERROR][sqlalchemy.pool.impl.QueuePool] Exception during reset or similar
�[33mserver_1 |�[0m Traceback (most recent call last):
�[33mserver_1 |�[0m File "/usr/local/lib/python3.7/site-packages/sqlalchemy/pool/base.py", line 680, in _finalize_fairy
�[33mserver_1 |�[0m fairy._reset(pool)
�[33mserver_1 |�[0m File "/usr/local/lib/python3.7/site-packages/sqlalchemy/pool/base.py", line 865, in _reset
�[33mserver_1 |�[0m self._reset_agent.rollback()
�[33mserver_1 |�[0m File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1713, in rollback
�[33mserver_1 |�[0m self._do_rollback()
�[33mserver_1 |�[0m File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1751, in _do_rollback
�[33mserver_1 |�[0m self.connection._rollback_impl()

�[36mnginx_1 |�[0m 2023/06/28 07:39:34 [error] 6#6: *1143 upstream timed out (110: Connection timed out) while reading response header from upstream, client: xxx.xx.xxx.xx, server: , request: "GET /api/jobs/437f0d9b-938c-4cfb-9387-fcb89668b2a9 HTTP/1.1", upstream: "http://172.18.0.8:5000/api/jobs/437f0d9b-938c-4cfb-9387-fcb89668b2a9", host: "xx.xxx.xxx.xxx", referrer: "http://xxxxxxxx"

�[35madhoc_worker_1 |�[0m [2023-06-28 05:03:45,076][PID:658][INFO][rq.job.redash.tasks.queries.execution] job.func_name=redash.tasks.queries.execution.execute_query job.id=xxxxxxxxx job=execute_query query_hash=xxxxxxx ds_id=3 data_length=None error=[Unknown column 'xxxx' in 'xxxxxxx']

[36mnginx_1 |�[0m 2023/06/28 08:16:15 [error] 6#6: *2113 upstream prematurely closed connection while reading response header from upstream, client: xxxxxx, server: , request: "GET /api/queries/11/results/249.json HTTP/1.1", upstream: "http://172.18.0.8:5000/api/queries/11/results/249.json", host: "xxxxxxx", referrer: "http://xxxxxxxxxx/queries/11"

[35madhoc_worker_1 |�[0m [2023-06-28 07:42:10,100][PID:1019][ERROR][rq.worker] rq.exceptions.ShutDownImminentException: shut down imminent (signal: SIGALRM)
�[35madhoc_worker_1 |�[0m Traceback (most recent call last):
�[35madhoc_worker_1 |�[0m File "/usr/local/lib/python3.7/site-packages/rq/worker.py", line 934, in perform_job
�[35madhoc_worker_1 |�[0m rv = job.perform()
�[35madhoc_worker_1 |�[0m File "/usr/local/lib/python3.7/site-packages/rq/job.py", line 686, in perform
�[35madhoc_worker_1 |�[0m self._result = self._execute()
�[35madhoc_worker_1 |�[0m File "/usr/local/lib/python3.7/site-packages/rq/job.py", line 692, in _execute
�[35madhoc_worker_1 |�[0m return self.func(*self.args, **self.kwargs)
�[35madhoc_worker_1 |�[0m File "/app/redash/tasks/queries/execution.py", line 288, in execute_query
�[35madhoc_worker_1 |�[0m scheduled_query_id is not None,
�[35madhoc_worker_1 |�[0m File "/app/redash/tasks/queries/execution.py", line 233, in run
�[35madhoc_worker_1 |�[0m updated_query_ids = models.Query.update_latest_result(query_result)
�[35madhoc_worker_1 |�[0m File "/app/redash/models/init.py", line 783, in update_latest_result
�[35madhoc_worker_1 |�[0m query_ids = [q.id for q in queries]
�[35madhoc_worker_1 |�[0m File "/usr/local/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3341, in iter
�[35madhoc_worker_1 |�[0m self.session._autoflush()
�[35madhoc_worker_1 |�[0m File "/usr/local/lib/python3.7/site-packages/sqlalchemy/orm/session.py", line 1577, in _autoflush
�[35madhoc_worker_1 |�[0m self.flush()
�[35madhoc_worker_1 |�[0m File "/usr/local/lib/python3.7/site-packages/sqlalchemy/orm/session.py", line 2470, in flush
�[35madhoc_worker_1 |�[0m self._flush(objects)
�[35madhoc_worker_1 |�[0m File "/usr/local/lib/python3.7/site-packages/sqlalchemy/orm/session.py", line 2608, in _flush
�[35madhoc_worker_1 |�[0m transaction.rollback(_capture_exception=True)
�[35madhoc_worker_1 |�[0m File "/usr/local/lib/python3.7/site-packages/sqlalchemy/util/langhelpers.py", line 68, in exit
�[35madhoc_worker_1 |�[0m compat.reraise(exc_type, exc_value, exc_tb)
�[35madhoc_worker_1 |�[0m File "/usr/local/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 153, in reraise
�[35madhoc_worker_1 |�[0m raise value
�[35madhoc_worker_1 |�[0m File "/usr/local/lib/python3.7/site-packages/sqlalchemy/orm/session.py", line 2568, in _flush
�[35madhoc_worker_1 |�[0m flush_context.execute()
�[35madhoc_worker_1 |�[0m File "/usr/local/lib/python3.7/site-packages/sqlalchemy/orm/unitofwork.py", line 422, in execute
�[35madhoc_worker_1 |�[0m rec.execute(self)
�[35madhoc_worker_1 |�[0m File "/usr/local/lib/python3.7/site-packages/sqlalchemy/orm/unitofwork.py", line 589, in execute
�[35madhoc_worker_1 |�[0m uow,
�[35madhoc_worker_1 |�[0m File "/usr/local/lib/python3.7/site-packages/sqlalchemy/orm/persistence.py", line 213, in save_obj
�[35madhoc_worker_1 |�[0m ) in _organize_states_for_save(base_mapper, states, uowtransaction):
�[35madhoc_worker_1 |�[0m File "/usr/local/lib/python3.7/site-packages/sqlalchemy/orm/persistence.py", line 387, in _organize_states_for_save
�[35madhoc_worker_1 |�[0m mapper.dispatch.before_update(mapper, connection, state)
�[35madhoc_worker_1 |�[0m File "/usr/local/lib/python3.7/site-packages/sqlalchemy/event/attr.py", line 322, in call
�[35madhoc_worker_1 |�[0m fn(*args, **kw)
�[35madhoc_worker_1 |�[0m File "/usr/local/lib/python3.7/site-packages/sqlalchemy/orm/events.py", line 627, in wrap
�[35madhoc_worker_1 |�[0m fn(*arg, **kw)
�[35madhoc_worker_1 |�[0m File "/app/redash/models/init.py", line 877, in receive_before_insert_update
�[35madhoc_worker_1 |�[0m target.update_query_hash()
�[35madhoc_worker_1 |�[0m File "/app/redash/models/init.py", line 871, in update_query_hash
�[35madhoc_worker_1 |�[0m self.query_hash = query_runner.gen_query_hash(self.query_text, should_apply_auto_limit)
�[35madhoc_worker_1 |�[0m File "/app/redash/query_runner/init.py", line 201, in gen_query_hash
�[35madhoc_worker_1 |�[0m query_text = self.apply_auto_limit(query_text, set_auto_limit)
�[35madhoc_worker_1 |�[0m File "/app/redash/query_runner/init.py", line 232, in apply_auto_limit
�[35madhoc_worker_1 |�[0m if query_is_select_no_limit(last_query):
�[35madhoc_worker_1 |�[0m File "/app/redash/utils/init.py", line 218, in query_is_select_no_limit
�[35madhoc_worker_1 |�[0m parsed_query = sqlparse.parse(query)[0]
�[35madhoc_worker_1 |�[0m File "/usr/local/lib/python3.7/site-packages/sqlparse/init.py", line 32, in parse
�[35madhoc_worker_1 |�[0m return tuple(parsestream(sql, encoding))
�[35madhoc_worker_1 |�[0m File "/usr/local/lib/python3.7/site-packages/sqlparse/engine/filter_stack.py", line 37, in run
�[35madhoc_worker_1 |�[0m stmt = grouping.group(stmt)
�[35madhoc_worker_1 |�[0m File "/usr/local/lib/python3.7/site-packages/sqlparse/engine/grouping.py", line 371, in group
�[35madhoc_worker_1 |�[0m func(stmt)
�[35madhoc_worker_1 |�[0m File "/usr/local/lib/python3.7/site-packages/sqlparse/engine/grouping.py", line 171, in group_comparison
�[35madhoc_worker_1 |�[0m valid_prev, valid_next, post, extend=False)
�[35madhoc_worker_1 |�[0m File "/usr/local/lib/python3.7/site-packages/sqlparse/engine/grouping.py", line 393, in _group
�[35madhoc_worker_1 |�[0m _group(token, cls, match, valid_prev, valid_next, post, extend)
�[35madhoc_worker_1 |�[0m File "/usr/local/lib/python3.7/site-packages/sqlparse/engine/grouping.py", line 393, in _group
�[35madhoc_worker_1 |�[0m _group(token, cls, match, valid_prev, valid_next, post, extend)
�[35madhoc_worker_1 |�[0m File "/usr/local/lib/python3.7/site-packages/sqlparse/engine/grouping.py", line 393, in _group
�[35madhoc_worker_1 |�[0m _group(token, cls, match, valid_prev, valid_next, post, extend)
�[35madhoc_worker_1 |�[0m [Previous line repeated 1 more time]
�[35madhoc_worker_1 |�[0m File "/usr/local/lib/python3.7/site-packages/sqlparse/engine/grouping.py", line 399, in _group
�[35madhoc_worker_1 |�[0m grp = tlist.group_tokens(cls, from_idx, to_idx, extend=extend)
�[35madhoc_worker_1 |�[0m File "/usr/local/lib/python3.7/site-packages/sqlparse/sql.py", line 311, in group_tokens
�[35madhoc_worker_1 |�[0m grp = grp_cls(subtokens)
�[35madhoc_worker_1 |�[0m File "/usr/local/lib/python3.7/site-packages/sqlparse/sql.py", line 143, in init
�[35madhoc_worker_1 |�[0m super(TokenList, self).init(None, text_type(self))
�[35madhoc_worker_1 |�[0m File "/usr/local/lib/python3.7/site-packages/sqlparse/sql.py", line 33, in init
�[35madhoc_worker_1 |�[0m self.value = value
�[35madhoc_worker_1 |�[0m File "/usr/local/lib/python3.7/site-packages/rq/worker.py", line 1108, in request_force_stop_sigrtmin
�[35madhoc_worker_1 |�[0m raise ShutDownImminentException('shut down imminent (signal: %s)' % signal_name(signum), info)
�[35madhoc_worker_1 |�[0m rq.exceptions.ShutDownImminentException: shut down imminent (signal: SIGALRM)
�[35madhoc_worker_1 |�[0m Traceback (most recent call last):
�[35madhoc_worker_1 |�[0m File "/usr/local/lib/python3.7/site-packages/rq/worker.py", line 934, in perform_job
�[35madhoc_worker_1 |�[0m rv = job.perform()
�[35madhoc_worker_1 |�[0m File "/usr/local/lib/python3.7/site-packages/rq/job.py", line 686, in perform
�[35madhoc_worker_1 |�[0m self._result = self._execute()
�[35madhoc_worker_1 |�[0m File "/usr/local/lib/python3.7/site-packages/rq/job.py", line 692, in _execute
�[35madhoc_worker_1 |�[0m return self.func(*self.args, **self.kwargs)
�[35madhoc_worker_1 |�[0m File "/app/redash/tasks/queries/execution.py", line 288, in execute_query
�[35madhoc_worker_1 |�[0m scheduled_query_id is not None,
�[35madhoc_worker_1 |�[0m File "/app/redash/tasks/queries/execution.py", line 233, in run
�[35madhoc_worker_1 |�[0m updated_query_ids = models.Query.update_latest_result(query_result)
�[35madhoc_worker_1 |�[0m File "/app/redash/models/init.py", line 783, in update_latest_result
�[35madhoc_worker_1 |�[0m query_ids = [q.id for q in queries]

[gaecoli]

It seems that redash did not display the sql error return in the error message due to nginx timeout, and uniformly returned a service error problem.

[justinclift]

Yeah, it sort of, possibly looks like an interrupted network connection. Maybe.