Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Redis Running out of hand (Production) #1922

Open
mhmdksh opened this issue Jan 29, 2025 · 9 comments
Open

Redis Running out of hand (Production) #1922

mhmdksh opened this issue Jan 29, 2025 · 9 comments
Assignees
@CarlosQ96
Labels
bug Something isn't working performance

Comments

@mhmdksh
Copy link
Collaborator

mhmdksh commented Jan 29, 2025 •
edited
Loading

The latest pushes have cause our redis instance on production to go wild and consume so much memory. (> 66% of System Memory)

Check the MEM % Tab:
Image

This has been going on for more than two weeks as you see below:

Image

Note that this is on Production Only and not on staging
@mhmdksh mhmdksh added bug Something isn't working performance labels Jan 29, 2025
@mhmdksh mhmdksh changed the title Redis Running out of hand Redis Running out of hand (Production) Jan 29, 2025
@mhmdksh
Copy link
Collaborator Author

mhmdksh commented Feb 3, 2025

CC: @divine-comedian @CarlosQ96

@github-project-automation github-project-automation bot moved this to New Issues in All-Devs Feb 3, 2025
@divine-comedian divine-comedian moved this from New Issues to Product Backlog in All-Devs Feb 3, 2025
@mhmdksh
Copy link
Collaborator Author

mhmdksh commented Feb 3, 2025

@CarlosQ96 I have issued some troubleshooting commands on our redis host, to help check if we can have more visibility on this:

Some Redis-CLI Troubleshooting Outputs:

127.0.0.1:6379> SLOWLOG GET
 1) 1) (integer) 310
    2) (integer) 1738586949
    3) (integer) 19062
    4) 1) "get"
       2) "fa9fb7363303289fcd54aab3702c9953"
    5) "172.20.0.8:38174"
    6) ""
 2) 1) (integer) 309
    2) (integer) 1738584365
    3) (integer) 15352
    4) 1) "get"
       2) "f4630de723ef0c19181920ed5c0bf3a5"
    5) "172.20.0.8:38174"
    6) ""
 3) 1) (integer) 308
    2) (integer) 1738584322
    3) (integer) 11855
    4) 1) "get"
       2) "f4630de723ef0c19181920ed5c0bf3a5"
    5) "172.20.0.8:38174"
    6) ""
 4) 1) (integer) 307
    2) (integer) 1738581743
    3) (integer) 10168
    4) 1) "get"
       2) "fa9fb7363303289fcd54aab3702c9953"
    5) "172.20.0.8:38174"
    6) ""
 5) 1) (integer) 306
    2) (integer) 1738581374
    3) (integer) 10169
    4) 1) "get"
       2) "80265e0eb695926adc9a169e34957e87"
    5) "172.20.0.8:38174"
    6) ""
 6) 1) (integer) 305
    2) (integer) 1738575907
    3) (integer) 34310
    4) 1) "get"
       2) "9e9ea25b5ecc4a2fc054a0f7a800155c"
    5) "172.20.0.8:38174"
    6) ""
 7) 1) (integer) 304
    2) (integer) 1738570413
    3) (integer) 12059
    4) 1) "set"
       2) "user-likedProjectsCount-recurring-8002"
       3) "{\"identifier\":\"user-likedProjectsCount-recurring-8002\",\"query\":\"SELECT COUNT(DISTINCT(\\\"reaction\\\".\\\"id\\\")) AS \\\"cnt\\\" FROM \\\"pu... (335 more bytes)"
       4) "PX"
       5) "60000"
    5) "172.20.0.8:38174"
    6) ""
 8) 1) (integer) 303
    2) (integer) 1738562377
    3) (integer) 13792
    4) 1) "get"
       2) "80265e0eb695926adc9a169e34957e87"
    5) "172.20.0.8:38174"
    6) ""
 9) 1) (integer) 302
    2) (integer) 1738548980
    3) (integer) 10855
    4) 1) "get"
       2) "80265e0eb695926adc9a169e34957e87"
    5) "172.20.0.8:38174"
    6) ""
10) 1) (integer) 301
    2) (integer) 1738530909
    3) (integer) 10031
    4) 1) "get"
       2) "ce1e209d7ac5cf73422d2245744aa8da"
    5) "172.20.0.8:38174"
    6) ""

General Monitoring Logs:

127.0.0.1:6379> MONITOR
OK
1738588806.182022 [0 172.20.0.8:38192] "brpoplpush" "bull:verify-donations-queue:wait" "bull:verify-donations-queue:active" "5"
1738588806.328042 [0 172.20.0.8:54758] "evalsha" "f7a4e2e3f5835e307a3b720b30234203c9ed8d84" "6" "bull:send-broadcast-notifications:delayed" "bull:send-broadcast-notifications:active" "bull:send-broadcast-notifications:wait" "bull:send-broadcast-notifications:priority" "bull:send-broadcast-notifications:paused" "bull:send-broadcast-notifications:meta-paused" "bull:send-broadcast-notifications:" "1738588806327" "1eddd732-1345-4a6b-9e07-daa9d4b90341"
1738588806.328109 [0 lua] "ZRANGEBYSCORE" "bull:send-broadcast-notifications:delayed" "0" "7121259750715392" "LIMIT" "0" "1000"
1738588806.328118 [0 lua] "ZRANGE" "bull:send-broadcast-notifications:delayed" "0" "0" "WITHSCORES"
1738588806.329028 [0 172.20.0.8:54766] "evalsha" "f7a4e2e3f5835e307a3b720b30234203c9ed8d84" "6" "bull:send-project-related-notifications:delayed" "bull:send-project-related-notifications:active" "bull:send-project-related-notifications:wait" "bull:send-project-related-notifications:priority" "bull:send-project-related-notifications:paused" "bull:send-project-related-notifications:meta-paused" "bull:send-project-related-notifications:" "1738588806328" "60916e7a-5b50-48af-84ef-a8062d274246"
1738588806.329104 [0 lua] "ZRANGEBYSCORE" "bull:send-project-related-notifications:delayed" "0" "7121259750719488" "LIMIT" "0" "1000"
1738588806.329115 [0 lua] "ZRANGE" "bull:send-project-related-notifications:delayed" "0" "0" "WITHSCORES"
1738588806.889135 [0 172.20.0.8:38108] "evalsha" "f7a4e2e3f5835e307a3b720b30234203c9ed8d84" "6" "bull:fill-snapshot-balance-aggregator:delayed" "bull:fill-snapshot-balance-aggregator:active" "bull:fill-snapshot-balance-aggregator:wait" "bull:fill-snapshot-balance-aggregator:priority" "bull:fill-snapshot-balance-aggregator:paused" "bull:fill-snapshot-balance-aggregator:meta-paused" "bull:fill-snapshot-balance-aggregator:" "1738588806888" "e9af072f-456c-4b3e-bcc6-9d7566f1f905"
1738588806.889204 [0 lua] "ZRANGEBYSCORE" "bull:fill-snapshot-balance-aggregator:delayed" "0" "7121259753013248" "LIMIT" "0" "1000"
1738588806.889213 [0 lua] "ZRANGE" "bull:fill-snapshot-balance-aggregator:delayed" "0" "0" "WITHSCORES"
1738588806.889411 [0 172.20.0.8:38090] "evalsha" "f7a4e2e3f5835e307a3b720b30234203c9ed8d84" "6" "bull:send-project-related-notifications:delayed" "bull:send-project-related-notifications:active" "bull:send-project-related-notifications:wait" "bull:send-project-related-notifications:priority" "bull:send-project-related-notifications:paused" "bull:send-project-related-notifications:meta-paused" "bull:send-project-related-notifications:" "1738588806889" "2e0ec1e9-7f0f-4364-ab3d-535030dd1702"
1738588806.889442 [0 lua] "ZRANGEBYSCORE" "bull:send-project-related-notifications:delayed" "0" "7121259753017344" "LIMIT" "0" "1000"
1738588806.889447 [0 lua] "ZRANGE" "bull:send-project-related-notifications:delayed" "0" "0" "WITHSCORES"
1738588806.889457 [0 172.20.0.8:38120] "evalsha" "f7a4e2e3f5835e307a3b720b30234203c9ed8d84" "6" "bull:update-recurring-donations-stream-queue:delayed" "bull:update-recurring-donations-stream-queue:active" "bull:update-recurring-donations-stream-queue:wait" "bull:update-recurring-donations-stream-queue:priority" "bull:update-recurring-donations-stream-queue:paused" "bull:update-recurring-donations-stream-queue:meta-paused" "bull:update-recurring-donations-stream-queue:" "1738588806889" "09e8a64e-92a7-4991-b74f-251b27694394"
1738588806.889478 [0 lua] "ZRANGEBYSCORE" "bull:update-recurring-donations-stream-queue:delayed" "0" "7121259753017344" "LIMIT" "0" "1000"
1738588806.889483 [0 lua] "ZRANGE" "bull:update-recurring-donations-stream-queue:delayed" "0" "0" "WITHSCORES"
1738588806.889490 [0 172.20.0.8:38078] "evalsha" "f7a4e2e3f5835e307a3b720b30234203c9ed8d84" "6" "bull:send-broadcast-notifications:delayed" "bull:send-broadcast-notifications:active" "bull:send-broadcast-notifications:wait" "bull:send-broadcast-notifications:priority" "bull:send-broadcast-notifications:paused" "bull:send-broadcast-notifications:meta-paused" "bull:send-broadcast-notifications:" "1738588806889" "5544ce95-1fe7-4b67-8274-1f45552a753e"
1738588806.889505 [0 lua] "ZRANGEBYSCORE" "bull:send-broadcast-notifications:delayed" "0" "7121259753017344" "LIMIT" "0" "1000"
1738588806.889509 [0 lua] "ZRANGE" "bull:send-broadcast-notifications:delayed" "0" "0" "WITHSCORES"
1738588806.889517 [0 172.20.0.8:38126] "evalsha" "f7a4e2e3f5835e307a3b720b30234203c9ed8d84" "6" "bull:verify-recurring-donations-queue:delayed" "bull:verify-recurring-donations-queue:active" "bull:verify-recurring-donations-queue:wait" "bull:verify-recurring-donations-queue:priority" "bull:verify-recurring-donations-queue:paused" "bull:verify-recurring-donations-queue:meta-paused" "bull:verify-recurring-donations-queue:" "1738588806889" "9f557cb5-2d87-446e-9440-39bd844b2dc9"
1738588806.889533 [0 lua] "ZRANGEBYSCORE" "bull:verify-recurring-donations-queue:delayed" "0" "7121259753017344" "LIMIT" "0" "1000"
1738588806.889536 [0 lua] "ZRANGE" "bull:verify-recurring-donations-queue:delayed" "0" "0" "WITHSCORES"
1738588806.889543 [0 172.20.0.8:38098] "evalsha" "f7a4e2e3f5835e307a3b720b30234203c9ed8d84" "6" "bull:verify-donations-queue:delayed" "bull:verify-donations-queue:active" "bull:verify-donations-queue:wait" "bull:verify-donations-queue:priority" "bull:verify-donations-queue:paused" "bull:verify-donations-queue:meta-paused" "bull:verify-donations-queue:" "1738588806889" "9a81a945-3c04-4905-a930-df9731cfe701"
1738588806.889557 [0 lua] "ZRANGEBYSCORE" "bull:verify-donations-queue:delayed" "0" "7121259753017344" "LIMIT" "0" "1000"
1738588806.889561 [0 lua] "ZRANGE" "bull:verify-donations-queue:delayed" "0" "0" "WITHSCORES"
1738588806.889831 [0 172.20.0.8:38122] "evalsha" "f7a4e2e3f5835e307a3b720b30234203c9ed8d84" "6" "bull:user-token-balances-stream-queue:delayed" "bull:user-token-balances-stream-queue:active" "bull:user-token-balances-stream-queue:wait" "bull:user-token-balances-stream-queue:priority" "bull:user-token-balances-stream-queue:paused" "bull:user-token-balances-stream-queue:meta-paused" "bull:user-token-balances-stream-queue:" "1738588806889" "a742a40b-60df-4ca6-bcad-1a51d00bd1f2"
1738588806.889859 [0 lua] "ZRANGEBYSCORE" "bull:user-token-balances-stream-queue:delayed" "0" "7121259753017344" "LIMIT" "0" "1000"
1738588806.889863 [0 lua] "ZRANGE" "bull:user-token-balances-stream-queue:delayed" "0" "0" "WITHSCORES"
1738588809.348830 [0 172.20.0.8:38062] "get" "projectCampaignCache-for-projectBySlug"
1738588809.403832 [1 172.20.0.8:38174] "get" "findActiveQfRound"
1738588809.405836 [1 172.20.0.8:38174] "get" "projectDonationsSqrtRootSum_10798_13"
1738588809.407700 [1 172.20.0.8:38174] "set" "projectDonationsSqrtRootSum_10798_13" "{\"identifier\":\"projectDonationsSqrtRootSum_10798_13\",\"query\":\"SELECT \\\"sqrtRootSum\\\" FROM \\\"public\\\".\\\"project_estimated_matching_view\\\" \\\"ProjectEstimatedMatchingView\\\" WHERE \\\"projectId\\\" = $1 AND \\\"qfRoundId\\\" = $2 -- PARAMETERS: [10798,13]\",\"time\":1738588809407,\"duration\":120000,\"result\":\"[]\"}" "PX" "120000"
1738588809.409233 [1 172.20.0.8:38174] "get" "getDonationsTotalSqrtRootSumSquared_13"
1738588809.411532 [1 172.20.0.8:38174] "set" "getDonationsTotalSqrtRootSumSquared_13" "{\"identifier\":\"getDonationsTotalSqrtRootSumSquared_13\",\"query\":\"SELECT SUM(\\\"sqrtRootSumSquared\\\") AS \\\"totalSqrtRootSumSquared\\\" FROM \\\"public\\\".\\\"project_estimated_matching_view\\\" \\\"ProjectEstimatedMatchingView\\\" WHERE \\\"qfRoundId\\\" = $1 -- PARAMETERS: [13]\",\"time\":1738588809411,\"duration\":120000,\"result\":\"[{\\\"totalSqrtRootSumSquared\\\":118726.22624024193}]\"}" "PX" "120000"
1738588810.197876 [0 172.20.0.8:54782] "brpoplpush" "bull:send-broadcast-notifications:wait" "bull:send-broadcast-notifications:active" "5"
1738588810.299045 [0 172.20.0.8:38206] "brpoplpush" "bull:fill-snapshot-balance-aggregator:wait" "bull:fill-snapshot-balance-aggregator:active" "5"
1738588810.299222 [0 172.20.0.8:38154] "brpoplpush" "bull:send-broadcast-notifications:wait" "bull:send-broadcast-notifications:active" "5"
1738588810.299833 [0 172.20.0.8:38218] "brpoplpush" "bull:user-token-balances-stream-queue:wait" "bull:user-token-balances-stream-queue:active" "5"
1738588810.299929 [0 172.20.0.8:38196] "brpoplpush" "bull:update-recurring-donations-stream-queue:wait" "bull:update-recurring-donations-stream-queue:active" "5"
1738588810.300346 [0 172.20.0.8:38178] "brpoplpush" "bull:verify-recurring-donations-queue:wait" "bull:verify-recurring-donations-queue:active" "5"
1738588810.300470 [0 172.20.0.8:38166] "brpoplpush" "bull:send-project-related-notifications:wait" "bull:send-project-related-notifications:active" "5"
1738588810.801503 [0 172.20.0.8:54788] "brpoplpush" "bull:send-project-related-notifications:wait" "bull:send-project-related-notifications:active" "5"
1738588811.202953 [0 172.20.0.8:38192] "brpoplpush" "bull:verify-donations-queue:wait" "bull:verify-donations-queue:active" "5"
1738588811.329950 [0 172.20.0.8:54758] "evalsha" "f7a4e2e3f5835e307a3b720b30234203c9ed8d84" "6" "bull:send-broadcast-notifications:delayed" "bull:send-broadcast-notifications:active" "bull:send-broadcast-notifications:wait" "bull:send-broadcast-notifications:priority" "bull:send-broadcast-notifications:paused" "bull:send-broadcast-notifications:meta-paused" "bull:send-broadcast-notifications:" "1738588811329" "1eddd732-1345-4a6b-9e07-daa9d4b90341"
1738588811.330013 [0 lua] "ZRANGEBYSCORE" "bull:send-broadcast-notifications:delayed" "0" "7121259771203584" "LIMIT" "0" "1000"
1738588811.330022 [0 lua] "ZRANGE" "bull:send-broadcast-notifications:delayed" "0" "0" "WITHSCORES"
1738588811.330666 [0 172.20.0.8:54766] "evalsha" "f7a4e2e3f5835e307a3b720b30234203c9ed8d84" "6" "bull:send-project-related-notifications:delayed" "bull:send-project-related-notifications:active" "bull:send-project-related-notifications:wait" "bull:send-project-related-notifications:priority" "bull:send-project-related-notifications:paused" "bull:send-project-related-notifications:meta-paused" "bull:send-project-related-notifications:" "1738588811329" "60916e7a-5b50-48af-84ef-a8062d274246"
1738588811.330761 [0 lua] "ZRANGEBYSCORE" "bull:send-project-related-notifications:delayed" "0" "7121259771203584" "LIMIT" "0" "1000"
1738588811.330768 [0 lua] "ZRANGE" "bull:send-project-related-notifications:delayed" "0" "0" "WITHSCORES"
1738588811.890224 [0 172.20.0.8:38108] "evalsha" "f7a4e2e3f5835e307a3b720b30234203c9ed8d84" "6" "bull:fill-snapshot-balance-aggregator:delayed" "bull:fill-snapshot-balance-aggregator:active" "bull:fill-snapshot-balance-aggregator:wait" "bull:fill-snapshot-balance-aggregator:priority" "bull:fill-snapshot-balance-aggregator:paused" "bull:fill-snapshot-balance-aggregator:meta-paused" "bull:fill-snapshot-balance-aggregator:" "1738588811889" "e9af072f-456c-4b3e-bcc6-9d7566f1f905"
1738588811.890288 [0 lua] "ZRANGEBYSCORE" "bull:fill-snapshot-balance-aggregator:delayed" "0" "7121259773497344" "LIMIT" "0" "1000"
1738588811.890297 [0 lua] "ZRANGE" "bull:fill-snapshot-balance-aggregator:delayed" "0" "0" "WITHSCORES"
1738588811.890987 [0 172.20.0.8:38098] "evalsha" "f7a4e2e3f5835e307a3b720b30234203c9ed8d84" "6" "bull:verify-donations-queue:delayed" "bull:verify-donations-queue:active" "bull:verify-donations-queue:wait" "bull:verify-donations-queue:priority" "bull:verify-donations-queue:paused" "bull:verify-donations-queue:meta-paused" "bull:verify-donations-queue:" "1738588811890" "9a81a945-3c04-4905-a930-df9731cfe701"
1738588811.891014 [0 lua] "ZRANGEBYSCORE" "bull:verify-donations-queue:delayed" "0" "7121259773501440" "LIMIT" "0" "1000"
1738588811.891020 [0 lua] "ZRANGE" "bull:verify-donations-queue:delayed" "0" "0" "WITHSCORES"
1738588811.891031 [0 172.20.0.8:38126] "evalsha" "f7a4e2e3f5835e307a3b720b30234203c9ed8d84" "6" "bull:verify-recurring-donations-queue:delayed" "bull:verify-recurring-donations-queue:active" "bull:verify-recurring-donations-queue:wait" "bull:verify-recurring-donations-queue:priority" "bull:verify-recurring-donations-queue:paused" "bull:verify-recurring-donations-queue:meta-paused" "bull:verify-recurring-donations-queue:" "1738588811890" "9f557cb5-2d87-446e-9440-39bd844b2dc9"
1738588811.891112 [0 lua] "ZRANGEBYSCORE" "bull:verify-recurring-donations-queue:delayed" "0" "7121259773501440" "LIMIT" "0" "1000"
1738588811.891120 [0 lua] "ZRANGE" "bull:verify-recurring-donations-queue:delayed" "0" "0" "WITHSCORES"
1738588811.891137 [0 172.20.0.8:38078] "evalsha" "f7a4e2e3f5835e307a3b720b30234203c9ed8d84" "6" "bull:send-broadcast-notifications:delayed" "bull:send-broadcast-notifications:active" "bull:send-broadcast-notifications:wait" "bull:send-broadcast-notifications:priority" "bull:send-broadcast-notifications:paused" "bull:send-broadcast-notifications:meta-paused" "bull:send-broadcast-notifications:" "1738588811890" "5544ce95-1fe7-4b67-8274-1f45552a753e"
1738588811.891154 [0 lua] "ZRANGEBYSCORE" "bull:send-broadcast-notifications:delayed" "0" "7121259773501440" "LIMIT" "0" "1000"
1738588811.891158 [0 lua] "ZRANGE" "bull:send-broadcast-notifications:delayed" "0" "0" "WITHSCORES"
1738588811.891222 [0 172.20.0.8:38120] "evalsha" "f7a4e2e3f5835e307a3b720b30234203c9ed8d84" "6" "bull:update-recurring-donations-stream-queue:delayed" "bull:update-recurring-donations-stream-queue:active" "bull:update-recurring-donations-stream-queue:wait" "bull:update-recurring-donations-stream-queue:priority" "bull:update-recurring-donations-stream-queue:paused" "bull:update-recurring-donations-stream-queue:meta-paused" "bull:update-recurring-donations-stream-queue:" "1738588811890" "09e8a64e-92a7-4991-b74f-251b27694394"
1738588811.891258 [0 lua] "ZRANGEBYSCORE" "bull:update-recurring-donations-stream-queue:delayed" "0" "7121259773501440" "LIMIT" "0" "1000"
1738588811.891276 [0 lua] "ZRANGE" "bull:update-recurring-donations-stream-queue:delayed" "0" "0" "WITHSCORES"
1738588811.891286 [0 172.20.0.8:38090] "evalsha" "f7a4e2e3f5835e307a3b720b30234203c9ed8d84" "6" "bull:send-project-related-notifications:delayed" "bull:send-project-related-notifications:active" "bull:send-project-related-notifications:wait" "bull:send-project-related-notifications:priority" "bull:send-project-related-notifications:paused" "bull:send-project-related-notifications:meta-paused" "bull:send-project-related-notifications:" "1738588811890" "2e0ec1e9-7f0f-4364-ab3d-535030dd1702"
1738588811.891307 [0 lua] "ZRANGEBYSCORE" "bull:send-project-related-notifications:delayed" "0" "7121259773501440" "LIMIT" "0" "1000"
1738588811.891312 [0 lua] "ZRANGE" "bull:send-project-related-notifications:delayed" "0" "0" "WITHSCORES"
1738588811.891321 [0 172.20.0.8:38122] "evalsha" "f7a4e2e3f5835e307a3b720b30234203c9ed8d84" "6" "bull:user-token-balances-stream-queue:delayed" "bull:user-token-balances-stream-queue:active" "bull:user-token-balances-stream-queue:wait" "bull:user-token-balances-stream-queue:priority" "bull:user-token-balances-stream-queue:paused" "bull:user-token-balances-stream-queue:meta-paused" "bull:user-token-balances-stream-queue:" "1738588811890" "a742a40b-60df-4ca6-bcad-1a51d00bd1f2"
1738588811.891342 [0 lua] "ZRANGEBYSCORE" "bull:user-token-balances-stream-queue:delayed" "0" "7121259773501440" "LIMIT" "0" "1000"
1738588811.891346 [0 lua] "ZRANGE" "bull:user-token-balances-stream-queue:delayed" "0" "0" "WITHSCORES"
1738588812.029560 [1 172.20.0.8:38174] "get" "findActiveQfRound"
1738588812.031983 [1 172.20.0.8:38174] "get" "user-likedProjectsCount-recurring-5363"
1738588812.032519 [1 172.20.0.8:38174] "get" "user-donationsCount-normal-5363"
1738588812.033197 [1 172.20.0.8:38174] "get" "findPowerBoostingsCountByUserId-recurring-5363"
1738588812.036454 [1 172.20.0.8:38174] "set" "user-likedProjectsCount-recurring-5363" "{\"identifier\":\"user-likedProjectsCount-recurring-5363\",\"query\":\"SELECT COUNT(DISTINCT(\\\"reaction\\\".\\\"id\\\")) AS \\\"cnt\\\" FROM \\\"public\\\".\\\"reaction\\\" \\\"reaction\\\" INNER JOIN \\\"public\\\".\\\"project\\\" \\\"project\\\" ON \\\"project\\\".\\\"id\\\"=\\\"reaction\\\".\\\"projectId\\\" WHERE \\\"reaction\\\".\\\"userId\\\" = $1 AND \\\"project\\\".\\\"statusId\\\" = 5 AND \\\"project\\\".\\\"reviewStatus\\\" = $2 -- PARAMETERS: [5363,\\\"Listed\\\"]\",\"time\":1738588812036,\"duration\":60000,\"result\":\"[{\\\"cnt\\\":\\\"0\\\"}]\"}" "PX" "60000"
1738588812.045970 [1 172.20.0.8:38174] "set" "user-donationsCount-normal-5363" "{\"identifier\":\"user-donationsCount-normal-5363\",\"query\":\"SELECT COUNT(1) AS \\\"cnt\\\" FROM \\\"public\\\".\\\"donation\\\" \\\"donation\\\" WHERE donation.\\\"userId\\\" = $1 AND \\\"donation\\\".\\\"status\\\" = $2 AND donation.\\\"recurringDonationId\\\" IS NULL -- PARAMETERS: [5363,\\\"verified\\\"]\",\"time\":1738588812045,\"duration\":60000,\"result\":\"[{\\\"cnt\\\":\\\"1\\\"}]\"}" "PX" "60000"
1738588812.046898 [1 172.20.0.8:38174] "get" "user-donationsCount-recurring-5363"
1738588812.047170 [1 172.20.0.8:38174] "set" "findPowerBoostingsCountByUserId-recurring-5363" "{\"identifier\":\"findPowerBoostingsCountByUserId-recurring-5363\",\"query\":\"SELECT COUNT(DISTINCT(\\\"powerBoosting\\\".\\\"id\\\")) AS \\\"cnt\\\" FROM \\\"public\\\".\\\"power_boosting\\\" \\\"powerBoosting\\\" LEFT JOIN \\\"public\\\".\\\"project\\\" \\\"project\\\" ON \\\"project\\\".\\\"id\\\"=\\\"powerBoosting\\\".\\\"projectId\\\"  LEFT JOIN \\\"public\\\".\\\"user\\\" \\\"user\\\" ON \\\"user\\\".\\\"id\\\"=\\\"powerBoosting\\\".\\\"userId\\\" WHERE percentage > 0 AND \\\"userId\\\" =5363 -- PARAMETERS: []\",\"time\":1738588812047,\"duration\":60000,\"result\":\"[{\\\"cnt\\\":\\\"0\\\"}]\"}" "PX" "60000"
1738588812.048236 [1 172.20.0.8:38174] "set" "user-donationsCount-recurring-5363" "{\"identifier\":\"user-donationsCount-recurring-5363\",\"query\":\"SELECT COUNT(1) AS \\\"cnt\\\" FROM \\\"public\\\".\\\"recurring_donation\\\" \\\"recurring_donation\\\" WHERE recurring_donation.\\\"donorId\\\" = $1 AND \\\"recurring_donation\\\".\\\"totalUsdStreamed\\\" > 0 -- PARAMETERS: [5363]\",\"time\":1738588812048,\"duration\":60000,\"result\":\"[{\\\"cnt\\\":\\\"0\\\"}]\"}" "PX" "60000"
1738588812.397541 [0 172.20.0.8:54758] "evalsha" "8f164ba1ad6fb7bf4e96a2785f4a877eca39e44f" "7" "bull:send-broadcast-notifications:stalled" "bull:send-broadcast-notifications:wait" "bull:send-broadcast-notifications:active" "bull:send-broadcast-notifications:failed" "bull:send-broadcast-notifications:stalled-check" "bull:send-broadcast-notifications:meta-paused" "bull:send-broadcast-notifications:paused" "1" "bull:send-broadcast-notifications:" "1738588812397" "30000"
1738588812.398139 [0 lua] "EXISTS" "bull:send-broadcast-notifications:stalled-check"
1738588812.398158 [0 lua] "SET" "bull:send-broadcast-notifications:stalled-check" "1738588812397" "PX" "30000"
1738588812.398169 [0 lua] "SMEMBERS" "bull:send-broadcast-notifications:stalled"
1738588812.398181 [0 lua] "LRANGE" "bull:send-broadcast-notifications:active" "0" "-1"
1738588812.398353 [0 172.20.0.8:54766] "evalsha" "8f164ba1ad6fb7bf4e96a2785f4a877eca39e44f" "7" "bull:send-project-related-notifications:stalled" "bull:send-project-related-notifications:wait" "bull:send-project-related-notifications:active" "bull:send-project-related-notifications:failed" "bull:send-project-related-notifications:stalled-check" "bull:send-project-related-notifications:meta-paused" "bull:send-project-related-notifications:paused" "1" "bull:send-project-related-notifications:" "1738588812397" "30000"
1738588812.398390 [0 lua] "EXISTS" "bull:send-project-related-notifications:stalled-check"
1738588812.398399 [0 lua] "SET" "bull:send-project-related-notifications:stalled-check" "1738588812397" "PX" "30000"
1738588812.398411 [0 lua] "SMEMBERS" "bull:send-project-related-notifications:stalled"
1738588812.398420 [0 lua] "LRANGE" "bull:send-project-related-notifications:active" "0" "-1"
1738588812.420998 [1 172.20.0.8:38174] "get" "findActiveQfRound"
1738588812.423680 [1 172.20.0.8:38174] "get" "user-likedProjectsCount-recurring-23729"
1738588812.424253 [1 172.20.0.8:38174] "get" "user-donationsCount-normal-23729"
1738588812.425225 [1 172.20.0.8:38174] "get" "findPowerBoostingsCountByUserId-recurring-23729"
1738588812.427446 [1 172.20.0.8:38174] "set" "user-likedProjectsCount-recurring-23729" "{\"identifier\":\"user-likedProjectsCount-recurring-23729\",\"query\":\"SELECT COUNT(DISTINCT(\\\"reaction\\\".\\\"id\\\")) AS \\\"cnt\\\" FROM \\\"public\\\".\\\"reaction\\\" \\\"reaction\\\" INNER JOIN \\\"public\\\".\\\"project\\\" \\\"project\\\" ON \\\"project\\\".\\\"id\\\"=\\\"reaction\\\".\\\"projectId\\\" WHERE \\\"reaction\\\".\\\"userId\\\" = $1 AND \\\"project\\\".\\\"statusId\\\" = 5 AND \\\"project\\\".\\\"reviewStatus\\\" = $2 -- PARAMETERS: [23729,\\\"Listed\\\"]\",\"time\":1738588812427,\"duration\":60000,\"result\":\"[{\\\"cnt\\\":\\\"0\\\"}]\"}" "PX" "60000"
1738588812.427709 [1 172.20.0.8:38174] "set" "user-donationsCount-normal-23729" "{\"identifier\":\"user-donationsCount-normal-23729\",\"query\":\"SELECT COUNT(1) AS \\\"cnt\\\" FROM \\\"public\\\".\\\"donation\\\" \\\"donation\\\" WHERE donation.\\\"userId\\\" = $1 AND \\\"donation\\\".\\\"status\\\" = $2 AND donation.\\\"recurringDonationId\\\" IS NULL -- PARAMETERS: [23729,\\\"verified\\\"]\",\"time\":1738588812427,\"duration\":60000,\"result\":\"[{\\\"cnt\\\":\\\"2\\\"}]\"}" "PX" "60000"
1738588812.427987 [1 172.20.0.8:38174] "set" "findPowerBoostingsCountByUserId-recurring-23729" "{\"identifier\":\"findPowerBoostingsCountByUserId-recurring-23729\",\"query\":\"SELECT COUNT(DISTINCT(\\\"powerBoosting\\\".\\\"id\\\")) AS \\\"cnt\\\" FROM \\\"public\\\".\\\"power_boosting\\\" \\\"powerBoosting\\\" LEFT JOIN \\\"public\\\".\\\"project\\\" \\\"project\\\" ON \\\"project\\\".\\\"id\\\"=\\\"powerBoosting\\\".\\\"projectId\\\"  LEFT JOIN \\\"public\\\".\\\"user\\\" \\\"user\\\" ON \\\"user\\\".\\\"id\\\"=\\\"powerBoosting\\\".\\\"userId\\\" WHERE percentage > 0 AND \\\"userId\\\" =23729 -- PARAMETERS: []\",\"time\":1738588812427,\"duration\":60000,\"result\":\"[{\\\"cnt\\\":\\\"0\\\"}]\"}" "PX" "60000"
1738588812.428949 [1 172.20.0.8:38174] "get" "user-donationsCount-recurring-23729"
1738588812.429988 [1 172.20.0.8:38174] "set" "user-donationsCount-recurring-23729" "{\"identifier\":\"user-donationsCount-recurring-23729\",\"query\":\"SELECT COUNT(1) AS \\\"cnt\\\" FROM \\\"public\\\".\\\"recurring_donation\\\" \\\"recurring_donation\\\" WHERE recurring_donation.\\\"donorId\\\" = $1 AND \\\"recurring_donation\\\".\\\"totalUsdStreamed\\\" > 0 -- PARAMETERS: [23729]\",\"time\":1738588812429,\"duration\":60000,\"result\":\"[{\\\"cnt\\\":\\\"0\\\"}]\"}" "PX" "60000"
1738588815.219339 [0 172.20.0.8:54782] "brpoplpush" "bull:send-broadcast-notifications:wait" "bull:send-broadcast-notifications:active" "5"
1738588815.320578 [0 172.20.0.8:38178] "brpoplpush" "bull:verify-recurring-donations-queue:wait" "bull:verify-recurring-donations-queue:active" "5"
1738588815.321256 [0 172.20.0.8:38166] "brpoplpush" "bull:send-project-related-notifications:wait" "bull:send-project-related-notifications:active" "5"
1738588815.321273 [0 172.20.0.8:38218] "brpoplpush" "bull:user-token-balances-stream-queue:wait" "bull:user-token-balances-stream-queue:active" "5"
1738588815.321296 [0 172.20.0.8:38196] "brpoplpush" "bull:update-recurring-donations-stream-queue:wait" "bull:update-recurring-donations-stream-queue:active" "5"
1738588815.321441 [0 172.20.0.8:38206] "brpoplpush" "bull:fill-snapshot-balance-aggregator:wait" "bull:fill-snapshot-balance-aggregator:active" "5"
1738588815.321773 [0 172.20.0.8:38154] "brpoplpush" "bull:send-broadcast-notifications:wait" "bull:send-broadcast-notifications:active" "5"
1738588815.822627 [0 172.20.0.8:54788] "brpoplpush" "bull:send-project-related-notifications:wait" "bull:send-project-related-notifications:active" "5"
1738588816.223880 [0 172.20.0.8:38192] "brpoplpush" "bull:verify-donations-queue:wait" "bull:verify-donations-queue:active" "5"
1738588816.330947 [0 172.20.0.8:54758] "evalsha" "f7a4e2e3f5835e307a3b720b30234203c9ed8d84" "6" "bull:send-broadcast-notifications:delayed" "bull:send-broadcast-notifications:active" "bull:send-broadcast-notifications:wait" "bull:send-broadcast-notifications:priority" "bull:send-broadcast-notifications:paused" "bull:send-broadcast-notifications:meta-paused" "bull:send-broadcast-notifications:" "1738588816330" "1eddd732-1345-4a6b-9e07-daa9d4b90341"
1738588816.331033 [0 lua] "ZRANGEBYSCORE" "bull:send-broadcast-notifications:delayed" "0" "7121259791687680" "LIMIT" "0" "1000"
1738588816.331042 [0 lua] "ZRANGE" "bull:send-broadcast-notifications:delayed" "0" "0" "WITHSCORES"
1738588816.331953 [0 172.20.0.8:54766] "evalsha" "f7a4e2e3f5835e307a3b720b30234203c9ed8d84" "6" "bull:send-project-related-notifications:delayed" "bull:send-project-related-notifications:active" "bull:send-project-related-notifications:wait" "bull:send-project-related-notifications:priority" "bull:send-project-related-notifications:paused" "bull:send-project-related-notifications:meta-paused" "bull:send-project-related-notifications:" "1738588816330" "60916e7a-5b50-48af-84ef-a8062d274246"
1738588816.331987 [0 lua] "ZRANGEBYSCORE" "bull:send-project-related-notifications:delayed" "0" "7121259791687680" "LIMIT" "0" "1000"
1738588816.331993 [0 lua] "ZRANGE" "bull:send-project-related-notifications:delayed" "0" "0" "WITHSCORES"
1738588816.892220 [0 172.20.0.8:38108] "evalsha" "f7a4e2e3f5835e307a3b720b30234203c9ed8d84" "6" "bull:fill-snapshot-balance-aggregator:delayed" "bull:fill-snapshot-balance-aggregator:active" "bull:fill-snapshot-balance-aggregator:wait" "bull:fill-snapshot-balance-aggregator:priority" "bull:fill-snapshot-balance-aggregator:paused" "bull:fill-snapshot-balance-aggregator:meta-paused" "bull:fill-snapshot-balance-aggregator:" "1738588816891" "e9af072f-456c-4b3e-bcc6-9d7566f1f905"
1738588816.892305 [0 lua] "ZRANGEBYSCORE" "bull:fill-snapshot-balance-aggregator:delayed" "0" "7121259793985536" "LIMIT" "0" "1000"
1738588816.892319 [0 lua] "ZRANGE" "bull:fill-snapshot-balance-aggregator:delayed" "0" "0" "WITHSCORES"
1738588816.892595 [0 172.20.0.8:38122] "evalsha" "f7a4e2e3f5835e307a3b720b30234203c9ed8d84" "6" "bull:user-token-balances-stream-queue:delayed" "bull:user-token-balances-stream-queue:active" "bull:user-token-balances-stream-queue:wait" "bull:user-token-balances-stream-queue:priority" "bull:user-token-balances-stream-queue:paused" "bull:user-token-balances-stream-queue:meta-paused" "bull:user-token-balances-stream-queue:" "1738588816892" "a742a40b-60df-4ca6-bcad-1a51d00bd1f2"
1738588816.892639 [0 lua] "ZRANGEBYSCORE" "bull:user-token-balances-stream-queue:delayed" "0" "7121259793989632" "LIMIT" "0" "1000"
1738588816.892646 [0 lua] "ZRANGE" "bull:user-token-balances-stream-queue:delayed" "0" "0" "WITHSCORES"
1738588816.892660 [0 172.20.0.8:38090] "evalsha" "f7a4e2e3f5835e307a3b720b30234203c9ed8d84" "6" "bull:send-project-related-notifications:delayed" "bull:send-project-related-notifications:active" "bull:send-project-related-notifications:wait" "bull:send-project-related-notifications:priority" "bull:send-project-related-notifications:paused" "bull:send-project-related-notifications:meta-paused" "bull:send-project-related-notifications:" "1738588816892" "2e0ec1e9-7f0f-4364-ab3d-535030dd1702"
1738588816.892717 [0 lua] "ZRANGEBYSCORE" "bull:send-project-related-notifications:delayed" "0" "7121259793989632" "LIMIT" "0" "1000"
1738588816.892725 [0 lua] "ZRANGE" "bull:send-project-related-notifications:delayed" "0" "0" "WITHSCORES"
1738588816.892738 [0 172.20.0.8:38120] "evalsha" "f7a4e2e3f5835e307a3b720b30234203c9ed8d84" "6" "bull:update-recurring-donations-stream-queue:delayed" "bull:update-recurring-donations-stream-queue:active" "bull:update-recurring-donations-stream-queue:wait" "bull:update-recurring-donations-stream-queue:priority" "bull:update-recurring-donations-stream-queue:paused" "bull:update-recurring-donations-stream-queue:meta-paused" "bull:update-recurring-donations-stream-queue:" "1738588816892" "09e8a64e-92a7-4991-b74f-251b27694394"
1738588816.892768 [0 lua] "ZRANGEBYSCORE" "bull:update-recurring-donations-stream-queue:delayed" "0" "7121259793989632" "LIMIT" "0" "1000"
1738588816.892775 [0 lua] "ZRANGE" "bull:update-recurring-donations-stream-queue:delayed" "0" "0" "WITHSCORES"
1738588816.892788 [0 172.20.0.8:38078] "evalsha" "f7a4e2e3f5835e307a3b720b30234203c9ed8d84" "6" "bull:send-broadcast-notifications:delayed" "bull:send-broadcast-notifications:active" "bull:send-broadcast-notifications:wait" "bull:send-broadcast-notifications:priority" "bull:send-broadcast-notifications:paused" "bull:send-broadcast-notifications:meta-paused" "bull:send-broadcast-notifications:" "1738588816892" "5544ce95-1fe7-4b67-8274-1f45552a753e"
1738588816.892814 [0 lua] "ZRANGEBYSCORE" "bull:send-broadcast-notifications:delayed" "0" "7121259793989632" "LIMIT" "0" "1000"
1738588816.892821 [0 lua] "ZRANGE" "bull:send-broadcast-notifications:delayed" "0" "0" "WITHSCORES"
1738588816.894665 [0 172.20.0.8:38126] "evalsha" "f7a4e2e3f5835e307a3b720b30234203c9ed8d84" "6" "bull:verify-recurring-donations-queue:delayed" "bull:verify-recurring-donations-queue:active" "bull:verify-recurring-donations-queue:wait" "bull:verify-recurring-donations-queue:priority" "bull:verify-recurring-donations-queue:paused" "bull:verify-recurring-donations-queue:meta-paused" "bull:verify-recurring-donations-queue:" "1738588816892" "9f557cb5-2d87-446e-9440-39bd844b2dc9"
1738588816.894699 [0 lua] "ZRANGEBYSCORE" "bull:verify-recurring-donations-queue:delayed" "0" "7121259793989632" "LIMIT" "0" "1000"
1738588816.894706 [0 lua] "ZRANGE" "bull:verify-recurring-donations-queue:delayed" "0" "0" "WITHSCORES"
1738588816.894718 [0 172.20.0.8:38098] "evalsha" "f7a4e2e3f5835e307a3b720b30234203c9ed8d84" "6" "bull:verify-donations-queue:delayed" "bull:verify-donations-queue:active" "bull:verify-donations-queue:wait" "bull:verify-donations-queue:priority" "bull:verify-donations-queue:paused" "bull:verify-donations-queue:meta-paused" "bull:verify-donations-queue:" "1738588816892" "9a81a945-3c04-4905-a930-df9731cfe701"
1738588816.894740 [0 lua] "ZRANGEBYSCORE" "bull:verify-donations-queue:delayed" "0" "7121259793989632" "LIMIT" "0" "1000"
1738588816.895019 [0 lua] "ZRANGE" "bull:verify-donations-queue:delayed" "0" "0" "WITHSCORES"
1738588816.994811 [0 172.20.0.8:38062] "get" "projectCampaignCache-for-projectBySlug"
1738588817.028234 [1 172.20.0.8:38174] "get" "findActiveQfRound"
1738588817.029239 [1 172.20.0.8:38174] "get" "projectDonationsSqrtRootSum_10852_13"
1738588817.030233 [1 172.20.0.8:38174] "set" "projectDonationsSqrtRootSum_10852_13" "{\"identifier\":\"projectDonationsSqrtRootSum_10852_13\",\"query\":\"SELECT \\\"sqrtRootSum\\\" FROM \\\"public\\\".\\\"project_estimated_matching_view\\\" \\\"ProjectEstimatedMatchingView\\\" WHERE \\\"projectId\\\" = $1 AND \\\"qfRoundId\\\" = $2 -- PARAMETERS: [10852,13]\",\"time\":1738588817030,\"duration\":120000,\"result\":\"[]\"}" "PX" "120000"
1738588817.030651 [1 172.20.0.8:38174] "get" "getDonationsTotalSqrtRootSumSquared_13"
1738588818.005048 [0 172.20.0.8:38078] "evalsha" "8f164ba1ad6fb7bf4e96a2785f4a877eca39e44f" "7" "bull:send-broadcast-notifications:stalled" "bull:send-broadcast-notifications:wait" "bull:send-broadcast-notifications:active" "bull:send-broadcast-notifications:failed" "bull:send-broadcast-notifications:stalled-check" "bull:send-broadcast-notifications:meta-paused" "bull:send-broadcast-notifications:paused" "1" "bull:send-broadcast-notifications:" "1738588818004" "30000"
1738588818.005112 [0 lua] "EXISTS" "bull:send-broadcast-notifications:stalled-check"
1738588818.005487 [0 172.20.0.8:38090] "evalsha" "8f164ba1ad6fb7bf4e96a2785f4a877eca39e44f" "7" "bull:send-project-related-notifications:stalled" "bull:send-project-related-notifications:wait" "bull:send-project-related-notifications:active" "bull:send-project-related-notifications:failed" "bull:send-project-related-notifications:stalled-check" "bull:send-project-related-notifications:meta-paused" "bull:send-project-related-notifications:paused" "1" "bull:send-project-related-notifications:" "1738588818004" "30000"
1738588818.005522 [0 lua] "EXISTS" "bull:send-project-related-notifications:stalled-check"
1738588819.500718 [0 172.20.0.8:38062] "get" "projectCampaignCache-for-projectBySlug"
1738588819.581541 [1 172.20.0.8:38174] "get" "findActiveQfRound"
1738588819.582526 [1 172.20.0.8:38174] "get" "projectDonationsSqrtRootSum_3519_13"
1738588819.583679 [1 172.20.0.8:38174] "set" "projectDonationsSqrtRootSum_3519_13" "{\"identifier\":\"projectDonationsSqrtRootSum_3519_13\",\"query\":\"SELECT \\\"sqrtRootSum\\\" FROM \\\"public\\\".\\\"project_estimated_matching_view\\\" \\\"ProjectEstimatedMatchingView\\\" WHERE \\\"projectId\\\" = $1 AND \\\"qfRoundId\\\" = $2 -- PARAMETERS: [3519,13]\",\"time\":1738588819583,\"duration\":120000,\"result\":\"[]\"}" "PX" "120000"
1738588819.584211 [1 172.20.0.8:38174] "get" "getDonationsTotalSqrtRootSumSquared_13"
1738588820.241258 [0 172.20.0.8:54782] "brpoplpush" "bull:send-broadcast-notifications:wait" "bull:send-broadcast-notifications:active" "5"
1738588820.342420 [0 172.20.0.8:38218] "brpoplpush" "bull:user-token-balances-stream-queue:wait" "bull:user-token-balances-stream-queue:active" "5"
1738588820.342819 [0 172.20.0.8:38196] "brpoplpush" "bull:update-recurring-donations-stream-queue:wait" "bull:update-recurring-donations-stream-queue:active" "5"
1738588820.342832 [0 172.20.0.8:38166] "brpoplpush" "bull:send-project-related-notifications:wait" "bull:send-project-related-notifications:active" "5"
1738588820.343117 [0 172.20.0.8:38206] "brpoplpush" "bull:fill-snapshot-balance-aggregator:wait" "bull:fill-snapshot-balance-aggregator:active" "5"
1738588820.343344 [0 172.20.0.8:38154] "brpoplpush" "bull:send-broadcast-notifications:wait" "bull:send-broadcast-notifications:active" "5"
1738588820.343572 [0 172.20.0.8:38178] "brpoplpush" "bull:verify-recurring-donations-queue:wait" "bull:verify-recurring-donations-queue:active" "5"
1738588820.844416 [0 172.20.0.8:54788] "brpoplpush" "bull:send-project-related-notifications:wait" "bull:send-project-related-notifications:active" "5"
1738588821.245441 [0 172.20.0.8:38192] "brpoplpush" "bull:verify-donations-queue:wait" "bull:verify-donations-queue:active" "5"
1738588821.332781 [0 172.20.0.8:54758] "evalsha" "f7a4e2e3f5835e307a3b720b30234203c9ed8d84" "6" "bull:send-broadcast-notifications:delayed" "bull:send-broadcast-notifications:active" "bull:send-broadcast-notifications:wait" "bull:send-broadcast-notifications:priority" "bull:send-broadcast-notifications:paused" "bull:send-broadcast-notifications:meta-paused" "bull:send-broadcast-notifications:" "1738588821331" "1eddd732-1345-4a6b-9e07-daa9d4b90341"
1738588821.332871 [0 lua] "ZRANGEBYSCORE" "bull:send-broadcast-notifications:delayed" "0" "7121259812171776" "LIMIT" "0" "1000"
1738588821.332884 [0 lua] "ZRANGE" "bull:send-broadcast-notifications:delayed" "0" "0" "WITHSCORES"
1738588821.333512 [0 172.20.0.8:54766] "evalsha" "f7a4e2e3f5835e307a3b720b30234203c9ed8d84" "6" "bull:send-project-related-notifications:delayed" "bull:send-project-related-notifications:active" "bull:send-project-related-notifications:wait" "bull:send-project-related-notifications:priority" "bull:send-project-related-notifications:paused" "bull:send-project-related-notifications:meta-paused" "bull:send-project-related-notifications:" "1738588821333" "60916e7a-5b50-48af-84ef-a8062d274246"
1738588821.333589 [0 lua] "ZRANGEBYSCORE" "bull:send-project-related-notifications:delayed" "0" "7121259812179968" "LIMIT" "0" "1000"
1738588821.333598 [0 lua] "ZRANGE" "bull:send-project-related-notifications:delayed" "0" "0" "WITHSCORES"
1738588821.348251 [0 172.20.0.8:38062] "get" "projectCampaignCache-for-projectBySlug"
1738588821.378403 [1 172.20.0.8:38174] "get" "findActiveQfRound"
1738588821.379725 [1 172.20.0.8:38174] "get" "projectDonationsSqrtRootSum_2827_13"
1738588821.381291 [1 172.20.0.8:38174] "set" "projectDonationsSqrtRootSum_2827_13" "{\"identifier\":\"projectDonationsSqrtRootSum_2827_13\",\"query\":\"SELECT \\\"sqrtRootSum\\\" FROM \\\"public\\\".\\\"project_estimated_matching_view\\\" \\\"ProjectEstimatedMatchingView\\\" WHERE \\\"projectId\\\" = $1 AND \\\"qfRoundId\\\" = $2 -- PARAMETERS: [2827,13]\",\"time\":1738588821381,\"duration\":120000,\"result\":\"[]\"}" "PX" "120000"
1738588821.381900 [1 172.20.0.8:38174] "get" "getDonationsTotalSqrtRootSumSquared_13"
1738588821.893893 [0 172.20.0.8:38108] "evalsha" "f7a4e2e3f5835e307a3b720b30234203c9ed8d84" "6" "bull:fill-snapshot-balance-aggregator:delayed" "bull:fill-snapshot-balance-aggregator:active" "bull:fill-snapshot-balance-aggregator:wait" "bull:fill-snapshot-balance-aggregator:priority" "bull:fill-snapshot-balance-aggregator:paused" "bull:fill-snapshot-balance-aggregator:meta-paused" "bull:fill-snapshot-balance-aggregator:" "1738588821893" "e9af072f-456c-4b3e-bcc6-9d7566f1f905"
1738588821.893958 [0 lua] "ZRANGEBYSCORE" "bull:fill-snapshot-balance-aggregator:delayed" "0" "7121259814473728" "LIMIT" "0" "1000"
1738588821.893966 [0 lua] "ZRANGE" "bull:fill-snapshot-balance-aggregator:delayed" "0" "0" "WITHSCORES"
1738588821.894057 [0 172.20.0.8:38078] "evalsha" "f7a4e2e3f5835e307a3b720b30234203c9ed8d84" "6" "bull:send-broadcast-notifications:delayed" "bull:send-broadcast-notifications:active" "bull:send-broadcast-notifications:wait" "bull:send-broadcast-notifications:priority" "bull:send-broadcast-notifications:paused" "bull:send-broadcast-notifications:meta-paused" "bull:send-broadcast-notifications:" "1738588821893" "5544ce95-1fe7-4b67-8274-1f45552a753e"
1738588821.894088 [0 lua] "ZRANGEBYSCORE" "bull:send-broadcast-notifications:delayed" "0" "7121259814473728" "LIMIT" "0" "1000"
1738588821.894541 [0 lua] "ZRANGE" "bull:send-broadcast-notifications:delayed" "0" "0" "WITHSCORES"
1738588821.894557 [0 172.20.0.8:38120] "evalsha" "f7a4e2e3f5835e307a3b720b30234203c9ed8d84" "6" "bull:update-recurring-donations-stream-queue:delayed" "bull:update-recurring-donations-stream-queue:active" "bull:update-recurring-donations-stream-queue:wait" "bull:update-recurring-donations-stream-queue:priority" "bull:update-recurring-donations-stream-queue:paused" "bull:update-recurring-donations-stream-queue:meta-paused" "bull:update-recurring-donations-stream-queue:" "1738588821893" "09e8a64e-92a7-4991-b74f-251b27694394"
1738588821.894594 [0 lua] "ZRANGEBYSCORE" "bull:update-recurring-donations-stream-queue:delayed" "0" "7121259814473728" "LIMIT" "0" "1000"
1738588821.894599 [0 lua] "ZRANGE" "bull:update-recurring-donations-stream-queue:delayed" "0" "0" "WITHSCORES"
1738588821.895999 [0 172.20.0.8:38090] "evalsha" "f7a4e2e3f5835e307a3b720b30234203c9ed8d84" "6" "bull:send-project-related-notifications:delayed" "bull:send-project-related-notifications:active" "bull:send-project-related-notifications:wait" "bull:send-project-related-notifications:priority" "bull:send-project-related-notifications:paused" "bull:send-project-related-notifications:meta-paused" "bull:send-project-related-notifications:" "1738588821894" "2e0ec1e9-7f0f-4364-ab3d-535030dd1702"
1738588821.896096 [0 lua] "ZRANGEBYSCORE" "bull:send-project-related-notifications:delayed" "0" "7121259814477824" "LIMIT" "0" "1000"
1738588821.896103 [0 lua] "ZRANGE" "bull:send-project-related-notifications:delayed" "0" "0" "WITHSCORES"
1738588821.896154 [0 172.20.0.8:38122] "evalsha" "f7a4e2e3f5835e307a3b720b30234203c9ed8d84" "6" "bull:user-token-balances-stream-queue:delayed" "bull:user-token-balances-stream-queue:active" "bull:user-token-balances-stream-queue:wait" "bull:user-token-balances-stream-queue:priority" "bull:user-token-balances-stream-queue:paused" "bull:user-token-balances-stream-queue:meta-paused" "bull:user-token-balances-stream-queue:" "1738588821895" "a742a40b-60df-4ca6-bcad-1a51d00bd1f2"
1738588821.896179 [0 lua] "ZRANGEBYSCORE" "bull:user-token-balances-stream-queue:delayed" "0" "7121259814481920" "LIMIT" "0" "1000"
1738588821.896183 [0 lua] "ZRANGE" "bull:user-token-balances-stream-queue:delayed" "0" "0" "WITHSCORES"
1738588821.896190 [0 172.20.0.8:38098] "evalsha" "f7a4e2e3f5835e307a3b720b30234203c9ed8d84" "6" "bull:verify-donations-queue:delayed" "bull:verify-donations-queue:active" "bull:verify-donations-queue:wait" "bull:verify-donations-queue:priority" "bull:verify-donations-queue:paused" "bull:verify-donations-queue:meta-paused" "bull:verify-donations-queue:" "1738588821895" "9a81a945-3c04-4905-a930-df9731cfe701"
1738588821.896205 [0 lua] "ZRANGEBYSCORE" "bull:verify-donations-queue:delayed" "0" "7121259814481920" "LIMIT" "0" "1000"
1738588821.896209 [0 lua] "ZRANGE" "bull:verify-donations-queue:delayed" "0" "0" "WITHSCORES"
1738588821.896216 [0 172.20.0.8:38126] "evalsha" "f7a4e2e3f5835e307a3b720b30234203c9ed8d84" "6" "bull:verify-recurring-donations-queue:delayed" "bull:verify-recurring-donations-queue:active" "bull:verify-recurring-donations-queue:wait" "bull:verify-recurring-donations-queue:priority" "bull:verify-recurring-donations-queue:paused" "bull:verify-recurring-donations-queue:meta-paused" "bull:verify-recurring-donations-queue:" "1738588821895" "9f557cb5-2d87-446e-9440-39bd844b2dc9"
1738588821.896263 [0 lua] "ZRANGEBYSCORE" "bull:verify-recurring-donations-queue:delayed" "0" "7121259814481920" "LIMIT" "0" "1000"
1738588821.896269 [0 lua] "ZRANGE" "bull:verify-recurring-donations-queue:delayed" "0" "0" "WITHSCORES"
1738588822.087062 [0 172.20.0.8:38098] "evalsha" "8f164ba1ad6fb7bf4e96a2785f4a877eca39e44f" "7" "bull:verify-donations-queue:stalled" "bull:verify-donations-queue:wait" "bull:verify-donations-queue:active" "bull:verify-donations-queue:failed" "bull:verify-donations-queue:stalled-check" "bull:verify-donations-queue:meta-paused" "bull:verify-donations-queue:paused" "1" "bull:verify-donations-queue:" "1738588822086" "30000"
1738588822.087121 [0 lua] "EXISTS" "bull:verify-donations-queue:stalled-check"
1738588822.087136 [0 lua] "SET" "bull:verify-donations-queue:stalled-check" "1738588822086" "PX" "30000"
1738588822.087147 [0 lua] "SMEMBERS" "bull:verify-donations-queue:stalled"
1738588822.087157 [0 lua] "LRANGE" "bull:verify-donations-queue:active" "0" "-1"
1738588822.091421 [0 172.20.0.8:38126] "evalsha" "8f164ba1ad6fb7bf4e96a2785f4a877eca39e44f" "7" "bull:verify-recurring-donations-queue:stalled" "bull:verify-recurring-donations-queue:wait" "bull:verify-recurring-donations-queue:active" "bull:verify-recurring-donations-queue:failed" "bull:verify-recurring-donations-queue:stalled-check" "bull:verify-recurring-donations-queue:meta-paused" "bull:verify-recurring-donations-queue:paused" "1" "bull:verify-recurring-donations-queue:" "1738588822091" "30000"
1738588822.091481 [0 lua] "EXISTS" "bull:verify-recurring-donations-queue:stalled-check"
1738588822.091496 [0 lua] "SET" "bull:verify-recurring-donations-queue:stalled-check" "1738588822091" "PX" "30000"
1738588822.091506 [0 lua] "SMEMBERS" "bull:verify-recurring-donations-queue:stalled"
1738588822.091515 [0 lua] "LRANGE" "bull:verify-recurring-donations-queue:active" "0" "-1"
1738588822.095406 [0 172.20.0.8:38108] "evalsha" "8f164ba1ad6fb7bf4e96a2785f4a877eca39e44f" "7" "bull:fill-snapshot-balance-aggregator:stalled" "bull:fill-snapshot-balance-aggregator:wait" "bull:fill-snapshot-balance-aggregator:active" "bull:fill-snapshot-balance-aggregator:failed" "bull:fill-snapshot-balance-aggregator:stalled-check" "bull:fill-snapshot-balance-aggregator:meta-paused" "bull:fill-snapshot-balance-aggregator:paused" "1" "bull:fill-snapshot-balance-aggregator:" "1738588822095" "30000"
1738588822.095442 [0 lua] "EXISTS" "bull:fill-snapshot-balance-aggregator:stalled-check"
1738588822.095454 [0 lua] "SET" "bull:fill-snapshot-balance-aggregator:stalled-check" "1738588822095" "PX" "30000"
1738588822.095462 [0 lua] "SMEMBERS" "bull:fill-snapshot-balance-aggregator:stalled"
1738588822.095470 [0 lua] "LRANGE" "bull:fill-snapshot-balance-aggregator:active" "0" "-1"
1738588822.095539 [0 172.20.0.8:38120] "evalsha" "8f164ba1ad6fb7bf4e96a2785f4a877eca39e44f" "7" "bull:update-recurring-donations-stream-queue:stalled" "bull:update-recurring-donations-stream-queue:wait" "bull:update-recurring-donations-stream-queue:active" "bull:update-recurring-donations-stream-queue:failed" "bull:update-recurring-donations-stream-queue:stalled-check" "bull:update-recurring-donations-stream-queue:meta-paused" "bull:update-recurring-donations-stream-queue:paused" "1" "bull:update-recurring-donations-stream-queue:" "1738588822095" "30000"
1738588822.095560 [0 lua] "EXISTS" "bull:update-recurring-donations-stream-queue:stalled-check"
1738588822.095566 [0 lua] "SET" "bull:update-recurring-donations-stream-queue:stalled-check" "1738588822095" "PX" "30000"
1738588822.095572 [0 lua] "SMEMBERS" "bull:update-recurring-donations-stream-queue:stalled"
1738588822.095577 [0 lua] "LRANGE" "bull:update-recurring-donations-stream-queue:active" "0" "-1"
1738588822.095919 [0 172.20.0.8:38122] "evalsha" "8f164ba1ad6fb7bf4e96a2785f4a877eca39e44f" "7" "bull:user-token-balances-stream-queue:stalled" "bull:user-token-balances-stream-queue:wait" "bull:user-token-balances-stream-queue:active" "bull:user-token-balances-stream-queue:failed" "bull:user-token-balances-stream-queue:stalled-check" "bull:user-token-balances-stream-queue:meta-paused" "bull:user-token-balances-stream-queue:paused" "1" "bull:user-token-balances-stream-queue:" "1738588822095" "30000"
1738588822.095945 [0 lua] "EXISTS" "bull:user-token-balances-stream-queue:stalled-check"
1738588823.349413 [1 172.20.0.8:38174] "get" "findActiveQfRound"
1738588823.351752 [1 172.20.0.8:38174] "get" "user-likedProjectsCount-recurring-6744"
1738588823.353022 [1 172.20.0.8:38174] "get" "user-donationsCount-normal-6744"
1738588823.353704 [1 172.20.0.8:38174] "get" "findPowerBoostingsCountByUserId-recurring-6744"
1738588823.356534 [1 172.20.0.8:38174] "set" "user-likedProjectsCount-recurring-6744" "{\"identifier\":\"user-likedProjectsCount-recurring-6744\",\"query\":\"SELECT COUNT(DISTINCT(\\\"reaction\\\".\\\"id\\\")) AS \\\"cnt\\\" FROM \\\"public\\\".\\\"reaction\\\" \\\"reaction\\\" INNER JOIN \\\"public\\\".\\\"project\\\" \\\"project\\\" ON \\\"project\\\".\\\"id\\\"=\\\"reaction\\\".\\\"projectId\\\" WHERE \\\"reaction\\\".\\\"userId\\\" = $1 AND \\\"project\\\".\\\"statusId\\\" = 5 AND \\\"project\\\".\\\"reviewStatus\\\" = $2 -- PARAMETERS: [6744,\\\"Listed\\\"]\",\"time\":1738588823356,\"duration\":60000,\"result\":\"[{\\\"cnt\\\":\\\"0\\\"}]\"}" "PX" "60000"
1738588823.369759 [1 172.20.0.8:38174] "set" "findPowerBoostingsCountByUserId-recurring-6744" "{\"identifier\":\"findPowerBoostingsCountByUserId-recurring-6744\",\"query\":\"SELECT COUNT(DISTINCT(\\\"powerBoosting\\\".\\\"id\\\")) AS \\\"cnt\\\" FROM \\\"public\\\".\\\"power_boosting\\\" \\\"powerBoosting\\\" LEFT JOIN \\\"public\\\".\\\"project\\\" \\\"project\\\" ON \\\"project\\\".\\\"id\\\"=\\\"powerBoosting\\\".\\\"projectId\\\"  LEFT JOIN \\\"public\\\".\\\"user\\\" \\\"user\\\" ON \\\"user\\\".\\\"id\\\"=\\\"powerBoosting\\\".\\\"userId\\\" WHERE percentage > 0 AND \\\"userId\\\" =6744 -- PARAMETERS: []\",\"time\":1738588823369,\"duration\":60000,\"result\":\"[{\\\"cnt\\\":\\\"0\\\"}]\"}" "PX" "60000"
1738588823.371882 [1 172.20.0.8:38174] "set" "user-donationsCount-normal-6744" "{\"identifier\":\"user-donationsCount-normal-6744\",\"query\":\"SELECT COUNT(1) AS \\\"cnt\\\" FROM \\\"public\\\".\\\"donation\\\" \\\"donation\\\" WHERE donation.\\\"userId\\\" = $1 AND \\\"donation\\\".\\\"status\\\" = $2 AND donation.\\\"recurringDonationId\\\" IS NULL -- PARAMETERS: [6744,\\\"verified\\\"]\",\"time\":1738588823371,\"duration\":60000,\"result\":\"[{\\\"cnt\\\":\\\"0\\\"}]\"}" "PX" "60000"
1738588823.372891 [1 172.20.0.8:38174] "get" "user-donationsCount-recurring-6744"
1738588823.374094 [1 172.20.0.8:38174] "set" "user-donationsCount-recurring-6744" "{\"identifier\":\"user-donationsCount-recurring-6744\",\"query\":\"SELECT COUNT(1) AS \\\"cnt\\\" FROM \\\"public\\\".\\\"recurring_donation\\\" \\\"recurring_donation\\\" WHERE recurring_donation.\\\"donorId\\\" = $1 AND \\\"recurring_donation\\\".\\\"totalUsdStreamed\\\" > 0 -- PARAMETERS: [6744]\",\"time\":1738588823373,\"duration\":60000,\"result\":\"[{\\\"cnt\\\":\\\"0\\\"}]\"}" "PX" "60000"
1738588823.559472 [1 172.20.0.8:38174] "get" "findActiveQfRound"
1738588823.561729 [1 172.20.0.8:38174] "get" "user-likedProjectsCount-recurring-3029"
1738588823.562129 [1 172.20.0.8:38174] "get" "user-donationsCount-normal-3029"
1738588823.562622 [1 172.20.0.8:38174] "get" "findPowerBoostingsCountByUserId-recurring-3029"
1738588823.564302 [1 172.20.0.8:38174] "set" "user-donationsCount-normal-3029" "{\"identifier\":\"user-donationsCount-normal-3029\",\"query\":\"SELECT COUNT(1) AS \\\"cnt\\\" FROM \\\"public\\\".\\\"donation\\\" \\\"donation\\\" WHERE donation.\\\"userId\\\" = $1 AND \\\"donation\\\".\\\"status\\\" = $2 AND donation.\\\"recurringDonationId\\\" IS NULL -- PARAMETERS: [3029,\\\"verified\\\"]\",\"time\":1738588823564,\"duration\":60000,\"result\":\"[{\\\"cnt\\\":\\\"2\\\"}]\"}" "PX" "60000"
1738588823.564579 [1 172.20.0.8:38174] "set" "user-likedProjectsCount-recurring-3029" "{\"identifier\":\"user-likedProjectsCount-recurring-3029\",\"query\":\"SELECT COUNT(DISTINCT(\\\"reaction\\\".\\\"id\\\")) AS \\\"cnt\\\" FROM \\\"public\\\".\\\"reaction\\\" \\\"reaction\\\" INNER JOIN \\\"public\\\".\\\"project\\\" \\\"project\\\" ON \\\"project\\\".\\\"id\\\"=\\\"reaction\\\".\\\"projectId\\\" WHERE \\\"reaction\\\".\\\"userId\\\" = $1 AND \\\"project\\\".\\\"statusId\\\" = 5 AND \\\"project\\\".\\\"reviewStatus\\\" = $2 -- PARAMETERS: [3029,\\\"Listed\\\"]\",\"time\":1738588823564,\"duration\":60000,\"result\":\"[{\\\"cnt\\\":\\\"0\\\"}]\"}" "PX" "60000"
1738588823.564936 [1 172.20.0.8:38174] "set" "findPowerBoostingsCountByUserId-recurring-3029" "{\"identifier\":\"findPowerBoostingsCountByUserId-recurring-3029\",\"query\":\"SELECT COUNT(DISTINCT(\\\"powerBoosting\\\".\\\"id\\\")) AS \\\"cnt\\\" FROM \\\"public\\\".\\\"power_boosting\\\" \\\"powerBoosting\\\" LEFT JOIN \\\"public\\\".\\\"project\\\" \\\"project\\\" ON \\\"project\\\".\\\"id\\\"=\\\"powerBoosting\\\".\\\"projectId\\\"  LEFT JOIN \\\"public\\\".\\\"user\\\" \\\"user\\\" ON \\\"user\\\".\\\"id\\\"=\\\"powerBoosting\\\".\\\"userId\\\" WHERE percentage > 0 AND \\\"userId\\\" =3029 -- PARAMETERS: []\",\"time\":1738588823564,\"duration\":60000,\"result\":\"[{\\\"cnt\\\":\\\"0\\\"}]\"}" "PX" "60000"
1738588823.565386 [1 172.20.0.8:38174] "get" "user-donationsCount-recurring-3029"
1738588823.566304 [1 172.20.0.8:38174] "set" "user-donationsCount-recurring-3029" "{\"identifier\":\"user-donationsCount-recurring-3029\",\"query\":\"SELECT COUNT(1) AS \\\"cnt\\\" FROM \\\"public\\\".\\\"recurring_donation\\\" \\\"recurring_donation\\\" WHERE recurring_donation.\\\"donorId\\\" = $1 AND \\\"recurring_donation\\\".\\\"totalUsdStreamed\\\" > 0 -- PARAMETERS: [3029]\",\"time\":1738588823566,\"duration\":60000,\"result\":\"[{\\\"cnt\\\":\\\"0\\\"}]\"}" "PX" "60000"
1738588825.263438 [0 172.20.0.8:54782] "brpoplpush" "bull:send-broadcast-notifications:wait" "bull:send-broadcast-notifications:active" "5"
1738588825.364337 [0 172.20.0.8:38178] "brpoplpush" "bull:verify-recurring-donations-queue:wait" "bull:verify-recurring-donations-queue:active" "5"
1738588825.364462 [0 172.20.0.8:38206] "brpoplpush" "bull:fill-snapshot-balance-aggregator:wait" "bull:fill-snapshot-balance-aggregator:active" "5"
1738588825.364586 [0 172.20.0.8:38154] "brpoplpush" "bull:send-broadcast-notifications:wait" "bull:send-broadcast-notifications:active" "5"
1738588825.364709 [0 172.20.0.8:38218] "brpoplpush" "bull:user-token-balances-stream-queue:wait" "bull:user-token-balances-stream-queue:active" "5"
1738588825.364820 [0 172.20.0.8:38196] "brpoplpush" "bull:update-recurring-donations-stream-queue:wait" "bull:update-recurring-donations-stream-queue:active" "5"
1738588825.365020 [0 172.20.0.8:38166] "brpoplpush" "bull:send-project-related-notifications:wait" "bull:send-project-related-notifications:active" "5"
1738588825.866266 [0 172.20.0.8:54788] "brpoplpush" "bull:send-project-related-notifications:wait" "bull:send-project-related-notifications:active" "5"
1738588826.268341 [0 172.20.0.8:38192] "brpoplpush" "bull:verify-donations-queue:wait" "bull:verify-donations-queue:active" "5"
1738588826.334730 [0 172.20.0.8:54758] "evalsha" "f7a4e2e3f5835e307a3b720b30234203c9ed8d84" "6" "bull:send-broadcast-notifications:delayed" "bull:send-broadcast-notifications:active" "bull:send-broadcast-notifications:wait" "bull:send-broadcast-notifications:priority" "bull:send-broadcast-notifications:paused" "bull:send-broadcast-notifications:meta-paused" "bull:send-broadcast-notifications:" "1738588826334" "1eddd732-1345-4a6b-9e07-daa9d4b90341"
1738588826.334809 [0 lua] "ZRANGEBYSCORE" "bull:send-broadcast-notifications:delayed" "0" "7121259832664064" "LIMIT" "0" "1000"
1738588826.334819 [0 lua] "ZRANGE" "bull:send-broadcast-notifications:delayed" "0" "0" "WITHSCORES"
1738588826.335403 [0 172.20.0.8:54766] "evalsha" "f7a4e2e3f5835e307a3b720b30234203c9ed8d84" "6" "bull:send-project-related-notifications:delayed" "bull:send-project-related-notifications:active" "bull:send-project-related-notifications:wait" "bull:send-project-related-notifications:priority" "bull:send-project-related-notifications:paused" "bull:send-project-related-notifications:meta-paused" "bull:send-project-related-notifications:" "1738588826334" "60916e7a-5b50-48af-84ef-a8062d274246"
1738588826.335501 [0 lua] "ZRANGEBYSCORE" "bull:send-project-related-notifications:delayed" "0" "7121259832664064" "LIMIT" "0" "1000"
1738588826.335516 [0 lua] "ZRANGE" "bull:send-project-related-notifications:delayed" "0" "0" "WITHSCORES"
1738588826.894175 [0 172.20.0.8:38108] "evalsha" "f7a4e2e3f5835e307a3b720b30234203c9ed8d84" "6" "bull:fill-snapshot-balance-aggregator:delayed" "bull:fill-snapshot-balance-aggregator:active" "bull:fill-snapshot-balance-aggregator:wait" "bull:fill-snapshot-balance-aggregator:priority" "bull:fill-snapshot-balance-aggregator:paused" "bull:fill-snapshot-balance-aggregator:meta-paused" "bull:fill-snapshot-balance-aggregator:" "1738588826893" "e9af072f-456c-4b3e-bcc6-9d7566f1f905"
1738588826.894237 [0 lua] "ZRANGEBYSCORE" "bull:fill-snapshot-balance-aggregator:delayed" "0" "7121259834953728" "LIMIT" "0" "1000"
1738588826.894246 [0 lua] "ZRANGE" "bull:fill-snapshot-balance-aggregator:delayed" "0" "0" "WITHSCORES"
1738588826.894931 [0 172.20.0.8:38120] "evalsha" "f7a4e2e3f5835e307a3b720b30234203c9ed8d84" "6" "bull:update-recurring-donations-stream-queue:delayed" "bull:update-recurring-donations-stream-queue:active" "bull:update-recurring-donations-stream-queue:wait" "bull:update-recurring-donations-stream-queue:priority" "bull:update-recurring-donations-stream-queue:paused" "bull:update-recurring-donations-stream-queue:meta-paused" "bull:update-recurring-donations-stream-queue:" "1738588826894" "09e8a64e-92a7-4991-b74f-251b27694394"
1738588826.894966 [0 lua] "ZRANGEBYSCORE" "bull:update-recurring-donations-stream-queue:delayed" "0" "7121259834957824" "LIMIT" "0" "1000"
1738588826.894972 [0 lua] "ZRANGE" "bull:update-recurring-donations-stream-queue:delayed" "0" "0" "WITHSCORES"
1738588826.896570 [0 172.20.0.8:38078] "evalsha" "f7a4e2e3f5835e307a3b720b30234203c9ed8d84" "6" "bull:send-broadcast-notifications:delayed" "bull:send-broadcast-notifications:active" "bull:send-broadcast-notifications:wait" "bull:send-broadcast-notifications:priority" "bull:send-broadcast-notifications:paused" "bull:send-broadcast-notifications:meta-paused" "bull:send-broadcast-notifications:" "1738588826896" "5544ce95-1fe7-4b67-8274-1f45552a753e"
1738588826.896639 [0 lua] "ZRANGEBYSCORE" "bull:send-broadcast-notifications:delayed" "0" "7121259834966016" "LIMIT" "0" "1000"
1738588826.896650 [0 lua] "ZRANGE" "bull:send-broadcast-notifications:delayed" "0" "0" "WITHSCORES"
1738588826.896780 [0 172.20.0.8:38126] "evalsha" "f7a4e2e3f5835e307a3b720b30234203c9ed8d84" "6" "bull:verify-recurring-donations-queue:delayed" "bull:verify-recurring-donations-queue:active" "bull:verify-recurring-donations-queue:wait" "bull:verify-recurring-donations-queue:priority" "bull:verify-recurring-donations-queue:paused" "bull:verify-recurring-donations-queue:meta-paused" "bull:verify-recurring-donations-queue:" "1738588826896" "9f557cb5-2d87-446e-9440-39bd844b2dc9"
1738588826.896804 [0 lua] "ZRANGEBYSCORE" "bull:verify-recurring-donations-queue:delayed" "0" "7121259834966016" "LIMIT" "0" "1000"
1738588826.896808 [0 lua] "ZRANGE" "bull:verify-recurring-donations-queue:delayed" "0" "0" "WITHSCORES"
1738588826.896819 [0 172.20.0.8:38098] "evalsha" "f7a4e2e3f5835e307a3b720b30234203c9ed8d84" "6" "bull:verify-donations-queue:delayed" "bull:verify-donations-queue:active" "bull:verify-donations-queue:wait" "bull:verify-donations-queue:priority" "bull:verify-donations-queue:paused" "bull:verify-donations-queue:meta-paused" "bull:verify-donations-queue:" "1738588826896" "9a81a945-3c04-4905-a930-df9731cfe701"
1738588826.896845 [0 lua] "ZRANGEBYSCORE" "bull:verify-donations-queue:delayed" "0" "7121259834966016" "LIMIT" "0" "1000"
1738588826.896849 [0 lua] "ZRANGE" "bull:verify-donations-queue:delayed" "0" "0" "WITHSCORES"
1738588826.896857 [0 172.20.0.8:38122] "evalsha" "f7a4e2e3f5835e307a3b720b30234203c9ed8d84" "6" "bull:user-token-balances-stream-queue:delayed" "bull:user-token-balances-stream-queue:active" "bull:user-token-balances-stream-queue:wait" "bull:user-token-balances-stream-queue:priority" "bull:user-token-balances-stream-queue:paused" "bull:user-token-balances-stream-queue:meta-paused" "bull:user-token-balances-stream-queue:" "1738588826896" "a742a40b-60df-4ca6-bcad-1a51d00bd1f2"
1738588826.896877 [0 lua] "ZRANGEBYSCORE" "bull:user-token-balances-stream-queue:delayed" "0" "7121259834966016" "LIMIT" "0" "1000"
1738588826.896881 [0 lua] "ZRANGE" "bull:user-token-balances-stream-queue:delayed" "0" "0" "WITHSCORES"
1738588826.896969 [0 172.20.0.8:38090] "evalsha" "f7a4e2e3f5835e307a3b720b30234203c9ed8d84" "6" "bull:send-project-related-notifications:delayed" "bull:send-project-related-notifications:active" "bull:send-project-related-notifications:wait" "bull:send-project-related-notifications:priority" "bull:send-project-related-notifications:paused" "bull:send-project-related-notifications:meta-paused" "bull:send-project-related-notifications:" "1738588826896" "2e0ec1e9-7f0f-4364-ab3d-535030dd1702"
1738588826.896992 [0 lua] "ZRANGEBYSCORE" "bull:send-project-related-notifications:delayed" "0" "7121259834966016" "LIMIT" "0" "1000"
1738588826.896996 [0 lua] "ZRANGE" "bull:send-project-related-notifications:delayed" "0" "0" "WITHSCORES"
1738588829.343396 [0 172.20.0.8:38062] "get" "projectCampaignCache-for-projectBySlug"
1738588829.375211 [1 172.20.0.8:38174] "get" "findActiveQfRound"
1738588829.376109 [1 172.20.0.8:38174] "get" "projectDonationsSqrtRootSum_2053_13"
1738588829.377466 [1 172.20.0.8:38174] "set" "projectDonationsSqrtRootSum_2053_13" "{\"identifier\":\"projectDonationsSqrtRootSum_2053_13\",\"query\":\"SELECT \\\"sqrtRootSum\\\" FROM \\\"public\\\".\\\"project_estimated_matching_view\\\" \\\"ProjectEstimatedMatchingView\\\" WHERE \\\"projectId\\\" = $1 AND \\\"qfRoundId\\\" = $2 -- PARAMETERS: [2053,13]\",\"time\":1738588829377,\"duration\":120000,\"result\":\"[]\"}" "PX" "120000"
1738588829.377948 [1 172.20.0.8:38174] "get" "getDonationsTotalSqrtRootSumSquared_13"
1738588829.505414 [0 172.20.0.8:38062] "get" "projectCampaignCache-for-projectBySlug"
1738588829.525635 [1 172.20.0.8:38174] "get" "findActiveQfRound"
1738588829.527125 [1 172.20.0.8:38174] "get" "projectDonationsSqrtRootSum_14645_13"
1738588829.527626 [1 172.20.0.8:38174] "get" "getDonationsTotalSqrtRootSumSquared_13"
1738588830.283114 [0 172.20.0.8:54782] "brpoplpush" "bull:send-broadcast-notifications:wait" "bull:send-broadcast-notifications:active" "5"
1738588830.383227 [0 172.20.0.8:38166] "brpoplpush" "bull:send-project-related-notifications:wait" "bull:send-project-related-notifications:active" "5"
1738588830.383726 [0 172.20.0.8:38218] "brpoplpush" "bull:user-token-balances-stream-queue:wait" "bull:user-token-balances-stream-queue:active" "5"
1738588830.384078 [0 172.20.0.8:38196] "brpoplpush" "bull:update-recurring-donations-stream-queue:wait" "bull:update-recurring-donations-stream-queue:active" "5"
1738588830.384398 [0 172.20.0.8:38178] "brpoplpush" "bull:verify-recurring-donations-queue:wait" "bull:verify-recurring-donations-queue:active" "5"
1738588830.384533 [0 172.20.0.8:38206] "brpoplpush" "bull:fill-snapshot-balance-aggregator:wait" "bull:fill-snapshot-balance-aggregator:active" "5"
1738588830.384961 [0 172.20.0.8:38154] "brpoplpush" "bull:send-broadcast-notifications:wait" "bull:send-broadcast-notifications:active" "5"
1738588830.885741 [0 172.20.0.8:54788] "brpoplpush" "bull:send-project-related-notifications:wait" "bull:send-project-related-notifications:active" "5"
1738588831.286749 [0 172.20.0.8:38192] "brpoplpush" "bull:verify-donations-queue:wait" "bull:verify-donations-queue:active" "5"
1738588831.337052 [0 172.20.0.8:54758] "evalsha" "f7a4e2e3f5835e307a3b720b30234203c9ed8d84" "6" "bull:send-broadcast-notifications:delayed" "bull:send-broadcast-notifications:active" "bull:send-broadcast-notifications:wait" "bull:send-broadcast-notifications:priority" "bull:send-broadcast-notifications:paused" "bull:send-broadcast-notifications:meta-paused" "bull:send-broadcast-notifications:" "1738588831336" "1eddd732-1345-4a6b-9e07-daa9d4b90341"
1738588831.337127 [0 lua] "ZRANGEBYSCORE" "bull:send-broadcast-notifications:delayed" "0" "7121259853152256" "LIMIT" "0" "1000"
1738588831.337137 [0 lua] "ZRANGE" "bull:send-broadcast-notifications:delayed" "0" "0" "WITHSCORES"
1738588831.337750 [0 172.20.0.8:54766] "evalsha" "f7a4e2e3f5835e307a3b720b30234203c9ed8d84" "6" "bull:send-project-related-notifications:delayed" "bull:send-project-related-notifications:active" "bull:send-project-related-notifications:wait" "bull:send-project-related-notifications:priority" "bull:send-project-related-notifications:paused" "bull:send-project-related-notifications:meta-paused" "bull:send-project-related-notifications:" "1738588831336" "60916e7a-5b50-48af-84ef-a8062d274246"
1738588831.337785 [0 lua] "ZRANGEBYSCORE" "bull:send-project-related-notifications:delayed" "0" "7121259853152256" "LIMIT" "0" "1000"
1738588831.337792 [0 lua] "ZRANGE" "bull:send-project-related-notifications:delayed" "0" "0" "WITHSCORES"
1738588831.895471 [0 172.20.0.8:38108] "evalsha" "f7a4e2e3f5835e307a3b720b30234203c9ed8d84" "6" "bull:fill-snapshot-balance-aggregator:delayed" "bull:fill-snapshot-balance-aggregator:active" "bull:fill-snapshot-balance-aggregator:wait" "bull:fill-snapshot-balance-aggregator:priority" "bull:fill-snapshot-balance-aggregator:paused" "bull:fill-snapshot-balance-aggregator:meta-paused" "bull:fill-snapshot-balance-aggregator:" "1738588831895" "e9af072f-456c-4b3e-bcc6-9d7566f1f905"

@divine-comedian divine-comedian moved this from Product Backlog to Sprint Backlog in All-Devs Feb 3, 2025
@CarlosQ96 CarlosQ96 moved this from Sprint Backlog to In Progress in All-Devs Feb 5, 2025
@divine-comedian divine-comedian mentioned this issue Feb 5, 2025
Open
@mhmdksh
Copy link
Collaborator Author

mhmdksh commented Feb 7, 2025 •
edited
Loading

Adding these finds in the issue after scanning redis for big keys:

Command:

docker exec -it redis-all redis-cli --bigkeys

Output:

# Scanning the entire keyspace to find biggest keys as well as
# average sizes per key type.  You can use -i 0.1 to sleep 0.1 sec
# per 100 SCAN commands (not usually needed).

[00.00%] Biggest hash   found so far '"bull:send-project-related-notifications:92102"' with 9 fields
[02.19%] Biggest hash   found so far '"bull:identify:6207"' with 13 fields
[07.14%] Biggest zset   found so far '"bull:identify:completed"' with 7099 members
[08.47%] Biggest string found so far '"cache-price-cryptocompare-glo-dollar"' with 28 bytes
[36.08%] Biggest string found so far '"cache-price-cryptocompare-giveth"' with 30 bytes
[46.53%] Biggest string found so far '"projectCampaignCache-for-projectBySlug"' with 712 bytes
[49.80%] Biggest string found so far '"sess:O685gBhGP-ZoFC273LZUlFSomjY0936n"' with 787 bytes
[52.37%] Biggest zset   found so far '"bull:verify-donations-queue:completed"' with 16327 members
[56.41%] Biggest string found so far '"sess:Dahg75Zf5y7wsUXIOPZ2Emp1Z_lxx01z"' with 1008 bytes
[69.54%] Biggest zset   found so far '"bull:send-project-related-notifications:completed"' with 229925 members

-------- summary -------

Sampled 262589 keys in the keyspace!
Total key length in bytes is 11414962 (avg len 43.47)

Biggest   hash found '"bull:identify:6207"' has 13 fields
Biggest string found '"sess:Dahg75Zf5y7wsUXIOPZ2Emp1Z_lxx01z"' has 1008 bytes
Biggest   zset found '"bull:send-project-related-notifications:completed"' has 229925 members

0 lists with 0 items (00.00% of keys, avg size 0.00)
262561 hashs with 2363148 fields (99.99% of keys, avg size 9.00)
22 strings with 2827 bytes (00.01% of keys, avg size 128.50)
0 streams with 0 entries (00.00% of keys, avg size 0.00)
0 sets with 0 members (00.00% of keys, avg size 0.00)
6 zsets with 262561 members (00.00% of keys, avg size 43760.17)

CC: @CarlosQ96

@divine-comedian
Copy link
Collaborator

what is the update on the final solution for this issue?

@kkatusic
Copy link
Collaborator

what is the update on the final solution for this issue?

@divine-comedian, final solution would be to remove bull jobs from Redis memory, @CarlosQ96 will say more about that.
In short, when some job (bull job, not cron job) is added to the memory after its execution it should be removed from memory, only blocker now does that job need to exist in memory, that's what @CarlosQ96 should analyse.

@divine-comedian
Copy link
Collaborator

Still not sure how to get the memory down

notification jobs are executed in the future is eating up the memory,
not sure what else can be done to bring it down
could be the cache as well - givpower cache can save some space

many unknowns

@CarlosQ96
Copy link
Collaborator

What is filling the memory are the bull queust (notification center mostly) that have no expiration

Image

I removed non expirable keys in staging with

docker exec -it redis-all sh -c '
  redis-cli -n 0 --scan | while read key; do 
    ttl=$(redis-cli -n 0 ttl "$key"); 
    if [ "$ttl" -eq -1 ]; then 
      redis-cli -n 0 del "$key"; 
    fi; 
  done
'

memory went down 370mbs. In prod which has over 4-5x more stuck keys, Will. decrease by 1gb++.

@kkatusic and me working on adding expiration to all enqueus using the bull system.
@mhmdksh @geleeroyale

@kkatusic kkatusic mentioned this issue Feb 11, 2025
Merged
@CarlosQ96 CarlosQ96 moved this from In Progress to Code Review/PR in All-Devs Feb 11, 2025
@divine-comedian
Copy link
Collaborator

excellent! So next steps is prepare for release at the end of next sprint and monitor to ensure memory usage drops.

Anything else we need to do for this issue? @CarlosQ96

@CarlosQ96 CarlosQ96 moved this from Code Review/PR to QA in All-Devs Feb 13, 2025
@divine-comedian
Copy link
Collaborator

should not see any unexpired keys inside of bull queues - will check this before we merge to prod.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@CarlosQ96 CarlosQ96

Labels
bug Something isn't working performance
Projects
All-Devs
Status: QA
Milestone
No milestone
Development

No branches or pull requests
4 participants
@kkatusic @mhmdksh @divine-comedian @CarlosQ96