Respond with `503 - Service Unavailable` while stacks-node is syncing

[zone117x]

When the API and stacks-node are both started, the API immediately boots the event observer http server in order to prevent an inter-process deadlock. However, the API waits until the stacks-node's http server is responsive to /v2/info before initializing it's own public http server. It does this because many endpoints will not function correctly until the stacks-node RPC server is responsive. The code responsible for this behavior is:
https://github.com/blockstack/stacks-blockchain-api/blob/538d3b1fbd9f06edd407552a6c61671de1b45923/src/index.ts#L115-L121

While the stacks-node is initializing, it can be unresponsive for quite some time (hours in some cases), and the API can appear to be in a confusing state.

The API should init its public http server and respond with 503 - Service Unavailable status during this period. The body should include the API's currently known block height in the response payload. Once stacks-network/stacks-core#2768 is ready, the API can use it to return a more useful syncing progress indicator.

[wileyj]

While 503 is technically correct for the status during startup, i'd argue to use a custom 5XX status code here since health checks would see the 503 and assume the service is dead (restarting the service in some cases).

We had a similar discussion for stacks-blockchain: stacks-network/stacks-core#2768 (comment)

If the proposed 503 is used, it would become more difficult to know if there was an actual 503 or if the service was simply init'ing without manually checking the logs of the service.

[MHassanTariq]

I'm working on this issue

[zone117x]

@MHassanTariq might be best to hold off until stacks-network/stacks-core#2768 is completed or further along, since we'll likely copy their status code and the sync progress if possible.

[agraebe]

removing priority because this is on hold until 2768 is further along

[sabbyanandan]

Hi, @zone117x. Given various improvements in upstream and Stacks Blockchain API, this is an old issue and likely not an issue anymore. When you get a chance, review this once more, and decide whether we can close this, please?

[zone117x]

It would probably still be helpful if the API responded with something to indicated sync status. However, any changes for that are still blocked on upstream, e.g. stacks-network/stacks-core#3729 (and potentially another change for that sync/health info to be pushed through the event emitter).