API Test Coverage

[Eshwari007]

The objective below was provided by @andresgalante to outline the start of API automation for the below.

The goal is to bring test coverage up to 80% in these scenarios:

• Testing the ?unanchored param is working correctly during the window of time where a tx is only in a microblock
• Similar with mempool txs
• Handle block and microblock reorgs for /extended/v1/:address/transactions path

Use Case #1:
Testing the ?unanchored param is working correctly during the window of time where a tx is only in a microblock

Scenarios:

https://stacks-node-api.mainnet.stacks.co//extended/v1/microblock/unanchored/txs

1- Possible scenarios based on the block information in the explorer, there could be a particular window of time where there is no microblock transaction but all transactions confirmed in the anchor block. Thus querying the API above will return API results =0 yet it's a valid test as it's an accurate result.

2. Possible scenario when the parameter complies to the below:

"anchor_mode": "any",
"is_unanchored": true,
"microblock_canonical": true,

This will return all results of tx which are unanchored given a particular time.

3. Possible scenario with particular time as mentioned above
   burn_block_time":
   "parent_burn_block_time":

There are two time parameters which I assume would be the potential window of time, would like to identify the valid one for the objective.

Use Case #2: Similar to Mempool tx

https://stacks-node-api.mainnet.stacks.co/extended/v1/tx/mempool

1. Possible scenario with any type of anchor mode and post condition which has the tx in mempool tx

"anchor_mode": "any","tx_status": "pending",

2. Possible scenario to verify all transactions to have a status of _pending_are in the MEMPOOL

3. Possible test if the total tx returned is growing than the initial round of API call
   "total": 108,
   "results": [}

In this case ,for an example, total was 100 on the first API call but grows as time of the day extends to 108

4. Possible test if all the tx in mempool is indicating pending n its tx _status

5. Possible test what is the acceptable limit of tx which can be in pending status ( to monitor if there is >400 tx for instance we can raise the alert to deep dive the blockage)

Use Case #3: Handle block and microblock reorgs for /extended/v1/:address/transactions path

1. This use case is a little vague for me as am not quite sure if the query is for blocks based on block height for particular addresses in an account

2. Will this use case be utilizing the following API,e.g:`https://stacks-node-api.mainnet.stacks.co/extended/v1/address/SP17YZQB1228EK9MPHQXA8GC4G3HVWZ66X7VRPMAX/mempool?

3. Will these parameters be vital for use case 3?
   "block_height":
   "block_hash":
   "microblock_hash":
   "microblock_sequence

I would like to discuss the details with @rafaelcr and @zone117x and get the API team's priority on what they need to test around as well as their order for the 80% use case coverage. Let's set up a call to document the expected results and the definition of 'Done' for this API testing goal to guarantee we cover all potential scenarios.

Last but not least, is there any API documentation that can be of good use to me?

Note:

I have worked with the first two use case(s) when they were introduced along with web wallet testing but for the third use case would need some extra inputs in order to deep dive into the needs of the testing.

Other questions:
Environment specific
-Stacks node version?
-Stacks API client version?

cc:@hirosystems/ux-team I will be working on the above for API automation.

[andresgalante]

API docs ares here: https://docs.hiro.so/api

[Eshwari007]

Use Case(s) redefined:

API testing on Mainnet
STX Only

*Mempool has been moved as Use Case # 1 (based on @rafaelcr 's input)

Use Case # 1 : Mempool transaction

https://stacks-node-api.mainnet.stacks.co/extended/v1/tx/mempool

Scenario 1: Possible scenario with any type of anchor mode and post condition which has the tx in mempool tx
"anchor_mode": "any","tx_status": "pending",

Scenario 2: Possible scenario to verify all transactions to have a status of _pending_are in the MEMPOOL

Scenario 3: Possible test if the total tx returned is growing than the initial round of API call
"total": 108,
"results": [}

In this case ,for an example, total was 100 on the first API call but grows as time of the day extends to 108

Scenario 4: Tx number in mempool also drops as tx gets confirmed in anchor block/ microblock

Scenario 5: Possible test if all the tx in mempool is indicating pending n its tx _status

Scenario 6: Possible test what is the acceptable limit of tx which can be in pending status ( to monitor if there is >400 tx for instance we can raise the alert to deep dive the blockage)

Use Case 2: Unanchored/Microblock transaction

https://stacks-node-api.mainnet.stacks.co//extended/v1/microblock/unanchored/txs

Testing the ?unanchored param is working correctly during the window of time where a tx is only in a microblock

Scenario 1: Possible scenarios based on the block information in the explorer, there could be a particular window of time where there is no microblock transaction but all transactions confirmed in the anchor block. Thus querying the API above will return API results =0 yet it's a valid test as it's an accurate result.

Scenario 2 : Possible scenario when the parameter complies to the below especially with anchor_mode is only microblock or only anchor block (need input on ='only microblock"??)
"anchor_mode": "any",
"is_unanchored": true,
"microblock_canonical": true,

This will return all results of tx which are unanchored given a particular time.

Scenario 3: Possible scenario with particular parameters as below:
block_height": ,
"block_hash":
"index_block_hash":
"microblock_hash":
"microblock_sequence":

Additional test WIP
1.contract call
2.contract deploy
3.call endpoint to query the contract and see if the return source code

[Eshwari007]

Possible edge case(s) which needs to be considered when building the API test:

• A tx at times may not end up in mempool but is transferred directly to anchorblock
• Nodes need to be hydrated in order to move the transaction therefore delay can be expected between a few seconds to a few minutes when a tx is submitted
• API call may return with no data when the transaction is delayed hence the best way is to subscribe to a web socket in order to monitor the tx
• Tx may not be picked up in microblocks at times with the similar edge case mentioned above for mempool
• Tx end to end use case also depends on miners' activity, as tx with low fees may/may not be picked up
• Network latency is also something that needs to be looked out for as if many tx are queued up, the delay could occur hence causing the API endpoint to return 0 results.

[Eshwari007]

The scenarios above are divided into smaller sections for ease of API individual test which will cover all the necessary coverage
https://docs.google.com/spreadsheets/d/11wIG70qqZjvbZ08vlTmVKBUmI794xMz-RvK2XJhyePg/edit#gid=0
(the doc is WIP)

[Eshwari007]

Tech Stack:Visual Code,Thunder Client and Postman

Test Status

• Successfully constructed transaction and broadcast tx with tx id via Stacks.js
• Tx id/contract id is able to be retrieved in the Explorer
• Tx id is monitored via WebSocket(modified WebSocket to listen to transaction based on senders address)
• First transaction which was tested was in anchorMode: AnchorMode.Any.
• Second and third tx will be tested in OffChainOnly and OnChainOnly.

Test Findings
Some of the edge cases which I found during API testing

• At times ,Stacks node issue occurs where that tx is not being listed correctly in the mempool for some reason
• Two different transactions were able to be broadcasted when their nonce are similar #.(one will eventually fail)
• Transactions were able to broadcast when the balance is low
• If the '"transaction not found' due to some reason, the API test will throw an error when its tracking the tx id from the senders' address.
• Transaction which is generated may not be picked up in mempool and is transferred directly to anchorblock
• Transaction which is generated is not included in microblock and gets included in anchorblock

The conditioned use case above is able to be monitored manually but when it's automated, the results can be returned as "false positive".
The event that triggers the API use case is negative hence when the automation fails does not mean the API query has failed but simply that the transaction has gotten its way to one of the above edge use case(s).

Test Next Step

• Concurrently run the API Calls when the tx are being listened in WebSocket subscription in mempool, microblock, and anchor block locally
• Moving this up to automation by identifying a seamless architecture( will be discussing it with @CharlieC3 on it)

Note: Am running couple test as we speak and will update the ticket along with it
cc @rafaelcr @zone117x @andresgalante

[Eshwari007]

Manual POX was approved and presented.
API Collection automated is in Thunder Client
Trying to convert the automation into java scripts that will call the API request in the terminal.
(WIP)

[andresgalante]

@Eshwari007 would you mind publishing the code on Github. The most effective way to review code and give feedback is by sharing it. You can open a PR, even if it's a WIP PR, for review.

What do you mean by "Manual POX was approved"?

[Eshwari007]

@Eshwari007 would you mind publishing the code on Github. The most effective way to review code and give feedback is by sharing it. You can open a PR, even if it's a WIP PR, for review.

What do you mean by "Manual POX was approved"?

My bad-meant 'POC' ,accidentally typed POX.

GitHub repo:
Just created the WIP PR on Github
https://github.com/hirosystems/API-Automation

Next Steps:

• Automate the API in Github actions suite of tests where we can see results in the console (Github actions output)
• Test the buildtime in GitHub runner as these tests are long-running (waiting for changes) ,will likely consume a lot of build time in GitHub actions .

Do ping if any name tweaks are needed, currently have the permissions set as Public but can limit it to team setting as well.

[Eshwari007]

Scheduled a call tomorrow with @CharlieC3 on ways to implement the next steps esp, importing all tests on GitHub actions.