Skip to content

Bug in micro-fork handling #670

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
zone117x opened this issue Jul 19, 2021 · 1 comment · Fixed by #686
Closed

Bug in micro-fork handling #670

zone117x opened this issue Jul 19, 2021 · 1 comment · Fixed by #686
Assignees
@zone117x
Labels
bug Something isn't working P1 Priority 1 Critical functionality not working Impacts all/most customers

Comments

@zone117x
Copy link
Member

Transaction 0xc03f1204350a2ef9b367252ef92aa46a95091b946b7622185b09d1b3cfa5097e appears to have been part of a re-org involving an orphaned microblock, and ended up without any canonical tx rows in the db.

https://stacks-node-api-microblocks.testnet.stacks.co/extended/v1/tx/0xc03f1204350a2ef9b367252ef92aa46a95091b946b7622185b09d1b3cfa5097e

{
  "tx_id": "0xc03f1204350a2ef9b367252ef92aa46a95091b946b7622185b09d1b3cfa5097e",
  "nonce": 5,
  "fee_rate": "180",
  "sender_address": "STM6S3AESTK9NAYE3Z7RS00T11ER8JJCDNTKG711",
  "sponsored": false,
  "post_condition_mode": "deny",
  "post_conditions": [],
  "anchor_mode": "any",
  "tx_status": "pending",
  "receipt_time": 1626683691,
  "receipt_time_iso": "2021-07-19T08:34:51.000Z",
  "tx_type": "token_transfer",
  "token_transfer": {
    "recipient_address": "ST2CEP848SACBBX7KHVC4TBZXBV0JH6SC0WF439NF",
    "amount": "4000000",
    "memo": "0x00000000000000000000000000000000000000000000000000000000000000000000"
  }
}

The pending tx data is being pulled from the mempool db. The actual status of the transaction in the db for the 2 transaction rows are marked canonical=false for both forks. Also, the tx's canonical anchor block appears to have confirmed a microblock stream, but does not list any microblocks in the microblocks_accepted response.
https://stacks-node-api-microblocks.testnet.stacks.co/extended/v1/block/0x70b1ce7c8ac98bbc6bbe2c40688ce22c2047d56764ddb34f1fa611adc6c4ff34

{
  "canonical": true,
  "height": 1305,
  "hash": "0x70b1ce7c8ac98bbc6bbe2c40688ce22c2047d56764ddb34f1fa611adc6c4ff34",
  "parent_block_hash": "0x2cd4089bb3d7b8093aff328b3a19c7531122cadc8dc54194345a6e051f17bfd1",
  "burn_block_time": 1626683913,
  "burn_block_time_iso": "2021-07-19T08:38:33.000Z",
  "burn_block_hash": "0x000000000000002b55a494891599e51d5e012bb4e16a167bc4e1dc61fbbe4db8",
  "burn_block_height": 2035439,
  "miner_txid": "0x939ed90b8fce2badf9d22094496cde775cbbeda82ea6a7bec677fcdcb2bfb8ab",
  "parent_microblock_hash": "0x5bf01fc04270bfaaae77d5ec416dbaabdd003f6bd645c7882e0e870805b56309",
  "parent_microblock_sequence": 2,
  "txs": [
    "0x15216ef9d08c32d057dfdb489ac27ccfa63278bf9f15ffd714d01761226abab5",
    "0x5def7bddbfbaa6f861a481c170738bb94e10711d4427a070aca4f613f4c1301c",
    "0x35286f91cb9256c2e8fb40a812a779d8fbc1065fa5e16bd0d156bab57822408a",
    "0xc03f1204350a2ef9b367252ef92aa46a95091b946b7622185b09d1b3cfa5097e"
  ],
  "microblocks_accepted": [],
  "microblocks_streamed": []
}

The orphaned anchor block https://stacks-node-api-microblocks.testnet.stacks.co/extended/v1/block/0x688e370f23b1b7b99f700ffab2800c68f81f3e05fd99b2be1a5866a9a41d9e64

{
  "canonical": false,
  "height": 1305,
  "hash": "0x688e370f23b1b7b99f700ffab2800c68f81f3e05fd99b2be1a5866a9a41d9e64",
  "parent_block_hash": "0x2cd4089bb3d7b8093aff328b3a19c7531122cadc8dc54194345a6e051f17bfd1",
  "burn_block_time": 1626683996,
  "burn_block_time_iso": "2021-07-19T08:39:56.000Z",
  "burn_block_hash": "0x000000000000001648899ca8c3e3b8ce9e8a2bd594ad5c2d80ac1a336dd754a3",
  "burn_block_height": 2035440,
  "miner_txid": "0xb2ac5eb4d6c6db73044d02e212b6940f73b7ec6a8d0179510fe2aa746e7e6a0b",
  "parent_microblock_hash": "",
  "parent_microblock_sequence": -1,
  "txs": [
    "0xc03f1204350a2ef9b367252ef92aa46a95091b946b7622185b09d1b3cfa5097e",
    "0x5def7bddbfbaa6f861a481c170738bb94e10711d4427a070aca4f613f4c1301c",
    "0x5f1db9f0caad93d2c151f28180fedcbb9d1db40ca3c6fa838330ee206168e7cf"
  ],
  "microblocks_accepted": [],
  "microblocks_streamed": []
}

The orphaned microblock is marked canonical=true but microblock_canonical=false.
https://stacks-node-api-microblocks.testnet.stacks.co/extended/v1/microblock/0xa272fe0501b770e6dc4c1af8de1059982db7621ff6491f5ae361a7aedbc63dd1

{
  "canonical": true,
  "microblock_canonical": false,
  "microblock_hash": "0xa272fe0501b770e6dc4c1af8de1059982db7621ff6491f5ae361a7aedbc63dd1",
  "microblock_sequence": 0,
  "microblock_parent_hash": "0x2cd4089bb3d7b8093aff328b3a19c7531122cadc8dc54194345a6e051f17bfd1",
  "block_height": 1305,
  "parent_block_height": 1304,
  "parent_block_hash": "0x2cd4089bb3d7b8093aff328b3a19c7531122cadc8dc54194345a6e051f17bfd1",
  "block_hash": "0x70b1ce7c8ac98bbc6bbe2c40688ce22c2047d56764ddb34f1fa611adc6c4ff34",
  "txs": [
    "0xc03f1204350a2ef9b367252ef92aa46a95091b946b7622185b09d1b3cfa5097e"
  ],
  "parent_burn_block_height": 2035438,
  "parent_burn_block_hash": "0x000000000000001dc2bd040dc170817399b7ecc3330f370757db7d88000b19e5",
  "parent_burn_block_time": 1626683602,
  "parent_burn_block_time_iso": "2021-07-19T08:33:22.000Z"
}

This seems like a bug in the orphan restoration logic, because the canonical block appears to have been seen first, likely marked orphaned temporarily by another block before winning out and being restored.
image

I'm still investigating the event replay logs to narrow down the scenario, then hopefully re-create in a test and fix.
@zone117x zone117x added bug Something isn't working P1 Priority 1 Critical functionality not working Impacts all/most customers labels Jul 19, 2021
@zone117x zone117x self-assigned this Jul 19, 2021
@zone117x zone117x mentioned this issue Jul 30, 2021
Merged
1 task
@zone117x zone117x linked a pull request Jul 30, 2021 that will close this issue
fix: re-org bug introduced in microblock implementation #686
Merged
1 task
zone117x added a commit that referenced this issue Jul 30, 2021
@zone117x
fix: bugged re-org logic introduced in microblock implementation #670
10cc023
zone117x added a commit that referenced this issue Aug 2, 2021
@zone117x
fix: bugged re-org logic introduced in microblock implementation #670
ea4d801
zone117x added a commit that referenced this issue Aug 5, 2021
@zone117x
Revert "fix: bugged re-org logic introduced in microblock implementat…
8dc34ef 
…ion #670"

This reverts commit ea4d801.
zone117x added a commit that referenced this issue Aug 6, 2021
@zone117x
fix: bugged re-org logic introduced in microblock implementation #670
7dfc5a9
@zone117x
Copy link
Member Author

zone117x commented Aug 6, 2021

Closed by #686
Hopefully the last re-org bug.

@zone117x zone117x closed this as completed Aug 6, 2021
wileyj added a commit that referenced this issue Aug 10, 2021
@wileyj
Merge remote-tracking branch 'origin/develop' into feat/cleanup
011e98a 
* origin/develop: (96 commits)
  chore: update stacks-blockchain docker image to `2.0.11.2.0`
  feat(rosetta): support stacking, delegate stacking and revoke stacking event
  fix: bugged re-org logic introduced in microblock implementation #670
  feat(rosetta): support passing btc address for rosetta stacking op #672
  feat: singular tx with STX transfer events endpoint #622
  fix(rosetta): contract details in payload stacking transactions
  fix: preprocess bug to remove network from the dummy transaction
  fix: remove token_transfer_recipient_address from stacking transactions
  feat: add get block by burn block height and by burn block hash (#675)
  feat: new database connection options
  fix: bundling issues
  chore: fix json schema doc example type
  chore: update openapi/json-schema summaries and examples with latest microblock related changes
  chore: progress on fixing unit tests
  chore: stacks-node docker image bumps to 2.0.11.2.0-rc1-stretch
  feat: add parent_burn_block_{time,hash,height} to txs and microblock API responses
  chore: fix tests
  fix: include contract-call metadata in mempool/pending transactions
  feat: add explicit `is_unanchored: boolean` flag to tx responses
  chore: remove debug code
  ...
blockstack-devops pushed a commit that referenced this issue Aug 11, 2021
@semantic-release-bot
chore(release): 0.63.0 [skip ci]
a7b42bb 
# [0.63.0](v0.62.3...v0.63.0) (2021-08-11)

### Bug Fixes

* bugged re-org logic introduced in microblock implementation [#670](#670) ([7dfc5a9](7dfc5a9))
* **rosetta:** contract details in payload stacking transactions ([a903cb0](a903cb0))
* all unit and integration tests working with microblock-capable stacks-node ([bf89f6a](bf89f6a))
* bns queries not using the pg query function ([51c762f](51c762f))
* bundling issues ([ab45a15](ab45a15))
* duplicated mempool-tx and mined-tx schemas leading to buggy tx parsing code ([a85dcad](a85dcad))
* fixed invalid URL crash, added route for invalid requests ([81e5bec](81e5bec))
* handling for receiving stale microblock events from stacks-node ([b82b3e0](b82b3e0))
* ignore 'data system is starting' error while connecting to postgres ([f637e8a](f637e8a))
* include contract-call metadata in mempool/pending transactions ([8f36f85](8f36f85))
* issue with client doc generation in gh actions ([3a017f9](3a017f9))
* missing sponsor_address property in some tx responses ([cbe16dd](cbe16dd))
* preprocess bug to remove network from the dummy transaction ([95cd1be](95cd1be))
* preserve logical transaction ordering using INT32_MAX value rather than -1 for batched tx microblock_sequence ([654669c](654669c))
* prevent querying duplicate tx data when existing in both micro-orphaned and unanchored-microblock form ([4903148](4903148))
* remove token_transfer_recipient_address from stacking transactions ([fc95319](fc95319))
* repair bns integration tests ([c0e0a69](c0e0a69))
* repair syntax in sample clarity contracts for debug endpoints ([e576361](e576361))
* various bugs and typing issues with parsing mined and mempool tx db data ([97bb2cb](97bb2cb))
* wip- add try catch and log ([fcb1216](fcb1216))

### Features

* **rosetta:** support passing btc address for rosetta stacking op [#672](#672) ([cf36b8f](cf36b8f))
* **rosetta:** support stacking, delegate stacking and revoke stacking event ([f5190c5](f5190c5))
* add anchored microblock hash array to API anchor block response ([f6a307a](f6a307a))
* add explicit `is_unanchored: boolean` flag to tx responses ([267a5eb](267a5eb))
* add get block by burn block height and by burn block hash ([#675](#675)) ([d002dad](d002dad))
* add microblock metadata to tx byproduct tables ([a3a9605](a3a9605))
* add nonce gap detection and recommended nonce value to /address/{principal}/nonces ([119615e](119615e))
* add parent_burn_block_{time,hash,height} to txs and microblock API responses ([977db77](977db77))
* added microblock metadata to regular/anchor block API responses ([39a8d32](39a8d32))
* allow dangerous `--force` option to drop tables during event-import when migrations had breaking changes ([7f71f2d](7f71f2d))
* anticipated sql schema required for storing microblock data with the ability to handle micro-fork reorgs ([5457a9e](5457a9e))
* API endpoint to return unanchored txs ([6f3aed9](6f3aed9))
* ensure microblock data is marked with the correct anchor-canonical status on receipt of anchor block ([dc89c98](dc89c98))
* env var to enable streaming events to file as they are received ([6114ae0](6114ae0))
* exclude micro-orphaned data from applicable sql queries ([9cff795](9cff795))
* flag microblock tx data as non-canonical when orphaned by an anchor block ([0f2a3ec](0f2a3ec))
* handle microblocks reorgs for micro-forks off the same same unanchored chain tip, e.g. a leader orphaning it's own unconfirmed microblocks ([ecb2c79](ecb2c79))
* handling for the happy-path of microblock-txs accepted in the next anchor block ([8ce3366](8ce3366))
* implement endpoint to get the latest account nonce based off mempool and unanchored or anchored tx data ([0b33bcb](0b33bcb))
* logical ordering of txs and events (e.g. for pagination) using microblock_sequence with tx_index ([0593591](0593591))
* microblocks API endpoints ([19e92ae](19e92ae))
* new database connection options ([d3f23d3](d3f23d3))
* option to "replay" event emitter data via program args ([e0d5c5f](e0d5c5f))
* option to export all raw event observer requests to file via program args, progress on replaying requests through the observer interface ([912113d](912113d))
* parse txs from microblock event payloads and inserted into db ([dc32f4e](dc32f4e))
* populate tx metadata tables (stx transfers, contract deployments, etc) with index_block_hash on microblock acceptance ([e8689b1](e8689b1))
* progress on making unanchored microblock tx data opt-in in API requests and db sql queries ([3057ab3](3057ab3))
* prune txs from mempool on microblock-tx receipt ([36158ba](36158ba))
* refactoring microblock data oprhaning logic into separate functions for usage in streamed micro-fork handling ([60fcd0a](60fcd0a))
* singular tx with STX transfer events endpoint [#622](#622) ([6dbbba6](6dbbba6))
* store raw event observer request payloads in db ([33fe79e](33fe79e))
* storing microblock headers, progress towards storing microblock txs ([6fa003c](6fa003c))
* storing microblock tx events and other metadata ([1871446](1871446))
* support microblock canonical status updating during anchor block re-orgs ([09844c2](09844c2))
* support processing of confirmed-only microblock data, see stacks-network/stacks-core#2668 ([d4b72e8](d4b72e8))
* **debug:** ability to broadcast test txs using unconfirmed chain tip nonce ([ab672f8](ab672f8))
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@zone117x zone117x

Labels
bug Something isn't working P1 Priority 1 Critical functionality not working Impacts all/most customers
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix: re-org bug introduced in microblock implementation hirosystems/stacks-blockchain-api
1 participant
@zone117x