Stacks API looks like it is up and responds truthy when blockchain config is missing events_observer block.

[muthusleuths]

Describe the bug
While running a Stacks 2.1 Radon blockchain node alongside Stacks API v7 it throws the below error when /extended/v1/block endpoint is called during genesis sync

To Reproduce
Steps to reproduce the behavior:

1. Go to 'https://github.com/stacks-network/stacks-blockchain-docker'
2. Update config for 'Stacks 2.1 Radon or rc4(mainnet) and API v7.0.0'
3. Begin a genesis sync
4. See below error

stacks-blockchain-api  | {"level":"error","message":"HTTP GET /extended/v1/block","req":{"headers":{"accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9","accept-encoding":"gzip, deflate","accept-language":"en-US,en;q=0.9","cache-control":"max-age=0","connection":"keep-alive","host":"3.228.177.236:3999","upgrade-insecure-requests":"1","user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36"},"httpVersion":"1.1","method":"GET","originalUrl":"/extended/v1/block","query":{},"url":"/extended/v1/block"},"res":{"statusCode":500},"responseTime":139,"timestamp":"2023-02-13T21:51:08.464Z"}
stacks-blockchain-api  | {"date":"Mon Feb 13 2023 21:51:08 GMT+0000 (Coordinated Universal Time)","error":{"errorTag":"bbae3ad3-7f7d-479f-8b53-bb866a1bcaca"},"exception":true,"level":"error","message":"middlewareError","req":{"headers":{"accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9","accept-encoding":"gzip, deflate","accept-language":"en-US,en;q=0.9","cache-control":"max-age=0","connection":"keep-alive","host":"3.228.177.236:3999","upgrade-insecure-requests":"1","user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36"},"httpVersion":"1.1","method":"GET","originalUrl":"/extended/v1/block","query":{},"url":"/extended/v1/block"},"stack":"TypeError: Cannot read properties of undefined (reading 'count')\n    at /app/src/datastore/pg-store.ts:488:41\n    at processTicksAndRejections (node:internal/process/task_queues:96:5)\n    at scope (/app/node_modules/postgres/cjs/src/index.js:222:18)\n    at Function.begin (/app/node_modules/postgres/cjs/src/index.js:209:14)\n    at PgStore.getBlocksWithMetadata (/app/src/datastore/pg-store.ts:483:12)\n    at getBlocksWithMetadata (/app/src/api/controllers/db-controller.ts:559:18)\n    at /app/src/api/routes/block.ts:22:34","timestamp":"2023-02-13T21:51:08.466Z"}
stacks-blockchain-api  | TypeError: Cannot read properties of undefined (reading 'count')
stacks-blockchain-api  |     at /app/src/datastore/pg-store.ts:488:41
stacks-blockchain-api  |     at processTicksAndRejections (node:internal/process/task_queues:96:5)
stacks-blockchain-api  |     at scope (/app/node_modules/postgres/cjs/src/index.js:222:18)
stacks-blockchain-api  |     at Function.begin (/app/node_modules/postgres/cjs/src/index.js:209:14)
stacks-blockchain-api  |     at PgStore.getBlocksWithMetadata (/app/src/datastore/pg-store.ts:483:12)
stacks-blockchain-api  |     at getBlocksWithMetadata (/app/src/api/controllers/db-controller.ts:559:18)
stacks-blockchain-api  |     at /app/src/api/routes/block.ts:22:34

Expected behavior
It returns the recent blocks around what block height /v2/info endpoint returns

Screenshots

@rafaelcr I think another issue here is also the fact that the extended/v1/status returns a status of ready which is misleading given the underlying issue seen. Please refer to the screenshot above. Is the status endpoint the right way to get a full healthcheck for the API ? If so, perhaps this needs to be addressed as well.

[muthusleuths]

One more data point here. If I run the exact same images (Stacks Blockchain RC4 and API v7) in testnet it works as expected. See screenshots below:

[muthusleuths]

The root cause was due to blockchain node config missing event_observer block. A better error message on the /extended/v1/status to indicate no events are being received would help identify the root cause better.

[saralab]

Thanks for flagging this @muthudotbtc , we are looking into this and will have an update this week.

[rafaelcr]

The cause for the TypeError: Cannot read properties of undefined (reading 'count') error is that when no blocks are received, the materialized view that calculates this property does not get refreshed (thus producing this error). The API should catch this situation and return an alternative count if that is the case.

As far as the /extended/v1/status improvements that were suggested, we would need to establish a maximum wait time in between node messages so we can alter the status from ready to something else if this time has passed. We already do something like this in our metrics and alerts but we could reflect this on the API.