docs: enhance README of projection packages

Author: mkazlauskas

demo/README.md

@@ -1,12 +1,48 @@
 # Cardano JS SDK | projection-typeorm
 
-Project Chain Sync events into PostgreSQL via TypeORM.
+This package is a library of utilities for projecting `ProjectionEvent`s into PostgreSQL database via [TypeORM](https://github.com/typeorm/typeorm).
 
-## Adding new projections
+If you're interested in generic projection types and utilities, see [projection](../projection) package.
 
-1. Create a new mapper that maps the block into something that you want to project (see [withStakeKeys](../projection/src/operators/Mappers/certificates/withStakeKeys.ts) as an example).
-2. Create a new granular TypeORM store (see [storeStakeKeys](./src/operators/storeStakeKeys.ts)) as an example.
+If you're interested in projector application as run by Lace, see [setup](../cardano-services/src/Projection) and [README](../cardano-services/README.md) in `cardano-services` package.
+
+## TypeormStabilityWindowBuffer
+
+Writes the block data into [block_data](./src/entity/BlockData.entity.ts) table and implements [StabilityWindowBuffer](../projection/README.md#stabilitywindowbuffer) interface that is required by [Bootstrap.fromCardanoNode](../projection/README.md#bootstrapfromcardanonode)
+
+### TypeormStabilityWindowBuffer.storeBlockData
+
+This method is intended to be called as part of PostgreSQL transaction that writes all other data from the event. It is important to keep StabilityWindowBuffer consistent with  projection state.
+
+## createTypeormTipTracker
+
+Queries and emits local tip (latest block header) from [block](./src/entity/Block.entity.ts) table. Returns:
+- an operator that should be applied after processing the block
+- an `Observable<TipOrOrgin>` that is required by [Bootstrap.fromCardanoNode](../projection/README.md#bootstrapfromcardanonode)
+
+## withTypeormTransaction
+
+Adds TypeORM context (query runner) to each event and starts a PostgreSQL transaction. Subsequent operators can utilize this context to perform database operations (see [Store Operators](#store-operators)).
 
-### Demo
+## typeormTransactionCommit
 
-See [demo/projection-typeorm.js](../../demo/)
+Commits PostgreSQL transaction started by [withTypeormTransaction](#withtypeormtransaction) and removes TypeORM context from the event object.
+
+## createObservableConnection
+
+Utility to initialize TypeORM data source.
+Returns an Observable that can be used as a dependency for
+[withTypeormTransaction](#withtypeormtransaction), [TypeormStabilityWindowBuffer](#typeormstabilitywindowbuffer) and [createTypeormTipTracker](#createtypeormtiptracker).
+
+## Store Operators
+
+[Each store operator](./src/operators) takes in an `Observable<WithTypeormContext & T>`, where `T` depends on what the specific operator needs. Usually it depends on one or more of the [Mappers](../projection/README.md#mappers).
+
+Most store operators will just write some data into the database and emit the same event object (unchanged). However, they can also add additional context to the events (see [storeAssets](./src/operators/storeAssets.ts) which adds new total supplies for each minted asset).
+
+---
+
+### Adding New Store Operators
+
+1. (optional) Create a new mapper that maps the block into something that you want to project (see [withStakeKeys](../projection/src/operators/Mappers/certificates/withStakeKeys.ts) as an example).
+2. Create a new granular TypeORM store (see [storeStakeKeys](./src/operators/storeStakeKeys.ts)) as an example.

demo/projection-typeorm.js

@@ -1,20 +1,105 @@
 # Cardano JS SDK | projection
 
-Chain Sync event projection utilities.
+This package is a library of generic projection types and utilities.
 
-## Summary
+> [!TIP]
+> [Guide to Projections and Read Models in Event-Driven Architecture](https://event-driven.io/en/projections_and_read_models_in_event_driven_architecture/)
 
-Projection is based on [RxJS](https://rxjs.dev/), where source observable of Chain Sync events is processed with various [operators](./src/operators/).
+If you're interested in projecting into PostgreSQL database, see [projection-typeorm](../projection-typeorm) package.
 
-There are no restrictions what an operator can do - you can utilize the full power of RxJS which makes it very flexible.
+If you're interested in projector application as run by Lace, see [setup](../cardano-services/src/Projection) and [README](../cardano-services/README.md) in `cardano-services` package.
 
-All operators implemented in this package are extending the source event object with extra properties, e.g.
+## Prerequisites
 
-```ts
-Bootstrap.fromCardanoNode({ buffer, cardanoNode, logger }).pipe(
-  Mappers.withStakeKeys(),
-  tap(({ stakeKeys }) => console.log('stakeKeys', stakeKeys)),
-  Mappers.withStakePools(),
-  tap(({ stakePools }) => console.log('stakePools', stakePools))
+In order to understand how `cardano-js-sdk` projections work, you'll first need basic understanding of:
+
+- [Observables](https://github.com/tc39/proposal-observable) (we're using [RxJS](https://rxjs.dev/) implementation)
+- Chain Sync mini protocol (well explained in [Ogmios docs](https://ogmios.dev/mini-protocols/local-chain-sync/))
+
+## Types
+
+### ProjectionEvent\<ExtraProps>
+
+All projections start by obtaining a source/producer<sup>1</sup>, which is an `Observable<ProjectionEvent<{}>>`. These events are very similar to Chain Sync events, but there are some important differences:
+
+1. Block format is compatible with types from `@cardano-sdk/core` package.
+2. Events include some additional properties: `{eraSumaries, genesisParameters, epochNo, crossEpochBoundary}`.
+3. `RollBackward` events include block data (instead of just specifying the rollback point), and are emitted once for **each** rolled back block.
+
+#### ExtraProps (Generic Parameter)
+
+Source observable can be piped through a series of RxJS operators, which may add some properties to the event. In a nutshell, `ProjectionEvent<T> = ProjectionEvent<{}> & T`. For more detail see [Mappers](#mappers).
+
+### StabilityWindowBuffer
+
+Since `ProjectionEvent{eventType=RollBackward}` includes block data, we must store some number (at least `k`, which is the "security parameter") of latest blocks so that if a fork/rollback happens, we can call `StabilityWindowBuffer.getBlock(hash)` in order to build a valid `ProjectionEvent`.
+
+### ObservableCardanoNode
+
+This is our interface to Chain Sync and Local State Query protocols of Cardano node. The only implementation available in `cardano-js-sdk` is `OgmiosObservableCardanoNode`.
+
+## Utilities
+
+### Bootstrap.fromCardanoNode
+
+Creates a projection source (`Observable<ProjectionEvent>`) from:
+
+- [ObservableCardanoNode](#observablecardanonode)
+- [StabilityWindowBuffer](#stabilitywindowbuffer)
+- `Observable<TipOrOrigin>` that emits block header of last processed event (local tip). This is used to find intersection point.
+
+```mermaid
+---
+title: Bootstrap Algorithm
+---
+flowchart LR
+    subscribe[Subscribe] --> find-intersect{Find node\nintersection\nwith local tip}
+    find-intersect --> |Found| start[Start ChainSync\nfrom intersection]
+    start --> get-event{Get event\nfrom node,\ncheck type}
+    get-event --> |RollBackward| get-block[Get block from StabilityWindowBuffer]
+    get-event --> |RollForward| hydrate
+    start --> hydrate[Hydrate event\nwith additional\npropertes]
+    hydrate --> emit[Emit\nProjectionEvent]
+    find-intersect --> |Not Found| get-block[Get block from\nStabilityWindowBuffer]
+    get-block --> rollback[Create\nRollBackward\nevent]
+    rollback --> hydrate
+    rollback --> wait[Wait for\nlocal tip change]
+    wait --> is-rollback-point{Is rollback point?}
+    is-rollback-point --> |No| get-block
+    is-rollback-point --> |Yes| started-sync{Already started\nChain Sync?}
+    started-sync --> |Yes| get-event
+    started-sync --> |No| find-intersect
+```
+
+### Mappers
+
+Projections are typically
+
+1. Interpreting the block (e.g. extracting some relevant properties from all transactions). `Mappers` is a namespace that contains a collection of RxJS operators that process the ProjectionEvent (block) and emit a new event object that has some additional properties.
+2. Doing something with it (e.g. store into the database).
+
+Here's an example:
+
+```typescript
+// source$: Observable<ProjectionEvent<{}>>
+source$.pipe(
+  // policyId is a parameter that should be provided
+  // by handle issuer, e.g. ADA Handle
+  Mappers.withHandles({ policyId }),
+  // type WithHandles = { handles: HandleOwnership; }
+  // Event type can also be inferred (no need for explicit type declaration)
+  tap((event: ProjectionEvent<WithHandles>) => {
+    console.log('Handles found by withHandles operator', event.handles);
+  })
 );
 ```
+
+### Other Utils
+
+#### InMemoryStabilityWindowBuffer
+
+If your application doesn't need persistence, you can use in-memory implementation of [StabilityWindowBuffer](#stabilitywindowbuffer), which can also be used as local tip tracker for Bootstrap (provides `Observable<TipOrOrigin>`).
+
+---
+
+_<sup>1</sup>Currently our projection source observable is not a pure 'Producer', as events have a `requestNext` method that is used to control the source. This is subject to change in the future._

packages/projection-typeorm/README.md

@@ -55,6 +55,20 @@ export type WithEpochBoundary = { crossEpochBoundary: boolean };
 
 export type BootstrapExtraProps = WithNetworkInfo & WithEpochNo & WithEpochBoundary;
 
+/**
+ * All projections start by obtaining a source/producer, which is an `Observable<ProjectionEvent<{}>>`.
+ * These events are very similar to Chain Sync events, but there are some important differences:
+ *
+ * 1. Block format is compatible with types from `@cardano-sdk/core` package.
+ * 2. Events include some additional properties: `{eraSumaries, genesisParameters, epochNo, crossEpochBoundary}`.
+ * 3. `RollBackward` events include block data (instead of just specifying the rollback point),
+ *    and are emitted once for **each** rolled back block.
+ *
+ * #### ExtraProps (Generic Parameter)
+ *
+ * Source observable can be piped through a series of RxJS operators, which may add some properties to the event.
+ * In an nutshell, `type ProjectionEvent<T> = ProjectionEvent<{}> & T`.
+ */
 export type ProjectionEvent<ExtraProps = {}> = UnifiedExtChainSyncEvent<BootstrapExtraProps & ExtraProps>;
 
 export type ProjectionOperator<ExtraPropsIn, ExtraPropsOut = {}> = UnifiedExtChainSyncOperator<
@@ -63,7 +77,7 @@ export type ProjectionOperator<ExtraPropsIn, ExtraPropsOut = {}> = UnifiedExtCha
 >;
 
 /**
- * Keeps track of blocks within stability window (computed as numSlots=3k/f).
+ * Keeps track of blocks within stability window (computed as numSlots=3k/f, or k blocks).
  *
  * With current mainnet protocol parameters (Dec 2022),
  * it can be up to 2160 ("securityParam"/"k") blocks,