Skip to content

Commit 36e74a1

Browse files
sebersolesebersole
committed
HHH-20766 - Improve collection flush handling
* Compact homogeneous collection rows into shared operation templates and row bindings * Preserve per-row binding, result checking and mutation completion when compaction is not safe * Avoid repeated collection inspection during graph planning * Complete collection flush acceptance coverage Signed-off-by: Steve Ebersole <steve@hibernate.org>
1 parent eef67a5 commit 36e74a1

21 files changed

Lines changed: 1059 additions & 73 deletions

‎design/collection-flush-processing-benchmark-results.adoc‎

Lines changed: 63 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -101,3 +101,66 @@ The required-auto-flush headline is noisy, so a separate five-fork
101101
candidate mean was 1.040 times Stage 0 and its median-fork ratio was 0.994,
102102
with candidate allocation approximately 0.9% higher. This does not establish
103103
a collection-processing regression exceeding the specified threshold.
104+
105+
== Stage 5 compact-row characterization
106+
107+
This is a development characterization of `CFS-STG-005`, not a replacement for
108+
the three-fork Stage 0 gate above. It compares the production compact-row
109+
policy with a negative control which forces safe remove-and-create rows back to
110+
the expanded representation.
111+
112+
The `BAG_REMOVE_CREATE_512` fixture loads a basic-valued bag with 256 rows and
113+
adds 256 values. Bag semantics interpret the change as one bulk remove followed
114+
by creation of all 512 current rows. The benchmark stops after graph planning,
115+
so JDBC latency does not obscure row-representation cost. Both runs used one
116+
fork, five one-second warmup iterations, and ten one-second measurement
117+
iterations on the environment recorded above.
118+
119+
[cols="2,1,1,1",options="header"]
120+
|===
121+
| Measurement | Compact | Expanded control | Compact / expanded
122+
123+
| Planning time (`us/op`) | 4.711 | 42.535 | 0.1108
124+
| Planning allocation (`bytes/op`) | 11,976 | 106,208 | 0.1128
125+
| Graph-planning allocation (`bytes/op`) | 5,144 | 99,472 | 0.0517
126+
| Graph operations | 2 | 513 | 0.0039
127+
| Physical row bindings | 513 | 513 | 1.0000
128+
|===
129+
130+
The compact representation reduced observed planning time by 88.9%, total
131+
planning allocation by 88.7%, and graph-planning allocation by 94.8% in this
132+
fixture. The remove and insert templates still represent all 513 physical
133+
bindings: one binding for the bulk remove and 512 independently executed insert
134+
bindings.
135+
136+
The policy remains deliberately conservative. It keeps per-row operations for
137+
non-primary unique constraints, self-referential tables, entity-valued collection
138+
rows, and nonstandard mutation-plan contributors. Those shapes may have
139+
row-specific unique slots, cycle or dependency endpoints, or state-management
140+
work which statement-shape equality alone cannot prove equivalent.
141+
142+
=== Full-stack regression screen
143+
144+
A one-fork graph-queue run of the nine-scenario full-stack matrix was compared
145+
with the mean of the three Stage 4 candidate forks recorded above. This is a
146+
directional regression screen rather than a new confidence-bound gate.
147+
148+
[cols="2,1,1,1",options="header"]
149+
|===
150+
| Scenario | Stage 4 mean (`us/op`) | Stage 5 screen (`us/op`) | Ratio
151+
152+
| `CLEAN_32` | 11.550 | 11.011 | 0.953
153+
| `SET_SPARSE_256` | 18.490 | 16.904 | 0.914
154+
| `SET_CLEAR_256` | 201.266 | 176.073 | 0.875
155+
| `BAG_SPARSE_256` | 282.546 | 252.950 | 0.895
156+
| `LIST_SPARSE_256` | 7.511 | 7.670 | 1.021
157+
| `MAP_SPARSE_256` | 14.951 | 14.190 | 0.949
158+
| `CREATE_16_BY_16` | 1,235.430 | 1,270.319 | 1.028
159+
| `AUTOFLUSH_DISCARDED` | 3.356 | 3.352 | 0.999
160+
| `AUTOFLUSH_REQUIRED` | 77.524 | 77.138 | 0.995
161+
|===
162+
163+
The geometric mean ratio was `0.957`. No scenario moved adversely by 5%; the
164+
two adverse point estimates were 2.1% and 2.8%, both within the ordinary
165+
single-fork variation visible in the earlier gate. The screen therefore found
166+
no broader performance regression requiring rollback or a new three-fork gate.

‎design/collection-flush-processing-spec.adoc‎

Lines changed: 202 additions & 41 deletions
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@
66

77
Status: Approved Specification
88

9-
Revision: 1.2
9+
Revision: 1.4
1010

1111
Approved by: Steve Ebersole
1212

@@ -132,10 +132,21 @@ Revision history::
132132
remove-all/create-all, or row execution. This revision responds to graph
133133
queue benchmarks which exposed the redundant
134134
`CollectionChangeSet -> CollectionDelta -> CollectionChangeSet` path.
135+
* `1.3` -- Replaced aspirational acceptance-suite names with traceability to the
136+
concrete tests which now exercise the implementation. Added explicit partial
137+
and pending coverage markers so an unimplemented acceptance obligation cannot
138+
be mistaken for a passing test. This revision changes test traceability only;
139+
it does not change normative behavior.
140+
* `1.4` -- Mapped the tests added to close the remaining non-cascade
141+
`CFS-SCN-*` acceptance scenarios. Added coverage for move-local ordering,
142+
replacement statistics, failure suppression, database-cascade events,
143+
cross-collection listener mutation, conservative required-flush
144+
initialization, and frozen-state access. `CFS-SCN-010` remains the
145+
acceptance target for the subsequent `CFS-CAS-*` stage.
135146
136147
== Status and Relationship to the Design Proposal
137148

138-
Revision 1.2 is the approved specification for collection processing during a
149+
Revision 1.4 is the approved specification for collection processing during a
139150
Hibernate flush. It is the implementation and conformance baseline, revised in
140151
response to implementation and benchmark evidence.
141152

@@ -171,7 +182,7 @@ Working Specification
171182
-> Approved Specification
172183
----
173184

174-
Revision 1.2 is currently at the final state.
185+
Revision 1.4 is currently at the final state.
175186

176187
== Normative Language
177188

@@ -1851,65 +1862,214 @@ Both queues execute that projection without enumerating the collection again.
18511862

18521863
== Test Traceability
18531864

1854-
[cols="1,2",options="header"]
1865+
[cols="1,4,2",options="header"]
18551866
|===
1856-
| Requirement group | Required named test suite
1867+
| Requirement group | Concrete current coverage | Remaining acceptance coverage
18571868

18581869
| `CFS-CTX-*`, `CFS-TRN-*`
1859-
| `CollectionFlushContextTest` for identity state and
1860-
`CollectionTransitionSelectionTest` for transition tables, plus
1861-
`CollectionFlushTransitionIntegrationTest` against each queue
1870+
| `CollectionEntryFlushStateTest` covers flush-local reached/processed state and
1871+
duplicate visitation. `AbstractCollectionEventTest` covers new, updated,
1872+
removed, replaced, and moved collections through the full stack.
1873+
`CollectionInterpretationQueueExecutionTest` covers create and update through
1874+
both queue implementations.
1875+
| _Partial_: no focused test currently asserts the complete normalization table,
1876+
including loaded/current role-and-key combinations and explicit
1877+
`REMOVE_AND_CREATE` ordering.
18621878

18631879
| `CFS-INT-*`
1864-
| `CollectionMutationInterpreterContractTest` parameterized over transition and
1865-
every supported standard semantics, `CollectionPhysicalMutationTest` for
1866-
exhaustive sealed variants and required payload, and
1867-
`CollectionInterpretationQueueExecutionTest` against both queues. The graph
1868-
execution test MUST detect a second collection comparison, a
1869-
`CollectionDelta -> CollectionChangeSet` reconstruction, and live collection
1870-
reads after freezing.
1880+
| `CollectionMutationInterpreterContractTest` covers interpreter selection,
1881+
create projection, structural freezing, indexed updates, identifier-bag row
1882+
changes, and conservative initialization. `CustomCollectionMutationInterpreterTest`
1883+
covers a specialized uninitialized interpretation. `CollectionInterpretationQueueExecutionTest`
1884+
covers standard set, bag, list, map, and one-to-many execution through both
1885+
queues. `CollectionRowCompactionTest`, `GroupedCollectionInsertBindPlanTest`,
1886+
and `OneToManyFallbackOrdinalTest` cover graph-native compact row projection,
1887+
per-row bindings, and fallback ordinals.
1888+
| No known group-level gap. `CollectionMutationInterpreterContractTest`
1889+
explicitly permits only mutation-generation validity checks after freezing
1890+
and rejects further live collection access.
18711891

18721892
| `CFS-DLT-*`, `CFS-QOP-*`
1873-
| `CollectionDeltaInterpretationTest` parameterized over every supported
1874-
semantics, `QueuedOperationDeltaTest`, `CollectionDeltaValidityTest`, and
1875-
`CustomCollectionMutationInterpreterTest`, plus `QueuedOperationFailureTest`
1876-
for entity-owned physical work and unsuccessful flush completion
1893+
| `CollectionMutationInterpreterContractTest`, `QueuedOperationDeltaTest`,
1894+
`CollectionDeltaValidityTest`, `CustomCollectionMutationInterpreterTest`, and
1895+
`QueuedOperationInitializationTest` cover delta content, provenance-sensitive
1896+
queued commands, validity, orphan projection, and initialization sequencing.
1897+
`ListDelayedOperationTest` and `QueuedOperationFailureTest` cover full-stack
1898+
queued list work and unsuccessful entity-owned physical work.
1899+
| No known group-level gap beyond the scenario-specific cascade integration and
1900+
required-flush custom-semantics cases identified below.
18771901

18781902
| `CFS-ACT-*`, `CFS-LFC-*`
1879-
| `CollectionMutationLifecycleTest` and `OwnerCollectionLifecycleTest` covering
1880-
queue-independent Hibernate events, interceptor hooks, Jakarta Persistence
1881-
callbacks, monitoring, cache, and statistics
1903+
| The `AbstractCollectionEventTest` hierarchy,
1904+
`DetachedMultipleCollectionChangeTest`, and `MergeCollectionEventTest` cover
1905+
Hibernate collection-event pairs. `CollectionAutoFlushLifecycleTest` covers
1906+
interceptor, pre-event, and owner-callback deferral. `AssociationChangeTests`
1907+
and `OwnerUpdateCompletionCoordinatorTest` cover owner callback applicability,
1908+
deduplication, mutation visibility, ordering, and failure. `CollectionExecutionMonitorTest`
1909+
and `BatchingExecutionMonitorTest` cover diagnostic monitoring.
1910+
`CollectionMutationCompletionTest#failedPhysicalWorkSuppressesEverySuccessfulLifecycleConsumer`
1911+
covers joint suppression of state advancement, post-event, cache, queued-work,
1912+
and statistics completion consumers.
1913+
| No known group-level gap.
18821914

18831915
| `CFS-CMP-*`, `CFS-GRP-*`
1884-
| `CollectionMutationCompletionTest`, `CollectionMutationGroupingTest`, and
1885-
`CollectionMutationFixupTest` covering direct, planned, batched, and failed
1886-
execution
1916+
| `CollectionMutationCompletionTest`, `CollectionMutationExecutorCompletionTest`,
1917+
`BatchingExecutionMonitorTest`, `FlushCoordinatorCollectionGroupingTest`,
1918+
`CollectionRowCompactionTest`, and `GroupedCollectionInsertBindPlanTest` cover
1919+
zero/direct/batched completion, failure, queued work, fixups, mutation identity,
1920+
compact row grouping, and per-row binding/result checks.
1921+
| No known group-level gap.
18871922

18881923
| `CFS-AFL-*`
1889-
| `ActionQueueCheckpointTest` as a shared contract suite for every queue and
1890-
`CollectionAutoFlushLifecycleTest`
1924+
| `ActionQueueCheckpointTest` invokes the checkpoint contract for both queues and
1925+
verifies durable orphan-collection removal. `CollectionAutoFlushLifecycleTest`
1926+
and `AutoFlushEventListenerTest` cover required and discarded auto-flush paths.
1927+
| No known group-level gap.
18911928

18921929
| `CFS-CAS-*`
1893-
| `CollectionDeltaOrphanProjectionTest` covering valid/stale reuse,
1894-
no-forced-initialization, standalone fallback, orphan timing, queued orphans,
1895-
iteration, and lazy initialization
1930+
| `CollectionDeltaValidityTest#orphanProjectionContainsOnlyKnownRemovedState`
1931+
and `#mutationGenerationInvalidatesRetainedDelta` cover projection content and
1932+
invalidation primitives.
1933+
| _Pending_: cascade consumption, valid and stale delta reuse, standalone
1934+
fallback, no-forced-initialization, and regression coverage proving unchanged
1935+
orphan timing, queued-orphan behavior, collection iteration, and lazy behavior.
18961936

18971937
| `CFS-COM-*`
1898-
| `CollectionFlushQueueConformanceTest` against both queues and focused
1899-
regression tests named in `CFS-COM-002`
1938+
| `CollectionInterpretationQueueExecutionTest` is the explicit two-queue
1939+
full-stack suite. `ActionQueueCheckpointTest` exercises both queue checkpoint
1940+
implementations. `CrossCollectionListenerMutationTest` runs against both
1941+
queues. The collection event hierarchy and the existing
1942+
delayed-operation, orphan-delete, one-to-many, many-to-many, element-collection,
1943+
and cache regression suites provide broader compatibility coverage.
1944+
| No known acceptance-scenario gap.
19001945

19011946
| `CFS-MIG-*`
1902-
| Static documentation review of the required anchors and final implemented
1903-
signatures in `migration-guide.adoc`; no runtime test is required
1947+
| Static review of `migration-guide.adoc` verifies the required anchors,
1948+
`ActionQueue#checkpoint()` / `#restore(ActionQueueCheckpoint)`, and
1949+
`CollectionSemantics#getCollectionMutationInterpreter()`.
1950+
| No runtime test is required.
19041951
|===
19051952

1906-
The acceptance scenarios map to methods `scenario001NewCollection()` through
1907-
`scenario016CreateAndBagUpdateRowProjection()` in
1908-
`CollectionFlushAcceptanceTest`, executed against both queue implementations
1909-
unless a scenario explicitly concerns one queue's internal representation.
1953+
[cols="1,4,1",options="header"]
1954+
|===
1955+
| Acceptance scenario | Concrete test mapping | Coverage
1956+
1957+
| `CFS-SCN-001` -- New collection
1958+
| `AbstractCollectionEventTest#testSaveParentEmptyChildren` and
1959+
`#testSaveParentOneChild`;
1960+
`CollectionInterpretationQueueExecutionTest.LegacyQueue#executesCreateAndUpdateDeltas`
1961+
and `CollectionInterpretationQueueExecutionTest.GraphQueue#executesCreateAndUpdateDeltas`;
1962+
`CollectionMutationCompletionTest#zeroOperationMutationCompletesWhenSealed`
1963+
| Covered
1964+
1965+
| `CFS-SCN-002` -- Collection move
1966+
| `AbstractCollectionEventTest#testMoveCollectionToDifferentParent` and
1967+
`#testMoveCollectionToDifferentParentFlushMoveToDifferentParent`;
1968+
`CollectionMutationCompletionTest#statementGroupingDoesNotMergeMutationIdentity`
1969+
| Covered
1970+
1971+
| `CFS-SCN-003` -- Update requiring physical replacement
1972+
| `CollectionInterpretationQueueExecutionTest.LegacyQueue#executesCreateAndUpdateDeltas`
1973+
and `CollectionInterpretationQueueExecutionTest.GraphQueue#executesCreateAndUpdateDeltas`;
1974+
`CollectionInterpretationQueueExecutionTest.GraphQueue#executesCompactBagRecreateAcrossJdbcBatches`;
1975+
both queue fixtures of
1976+
`CollectionInterpretationQueueExecutionTest#bagReplacementRecordsOneSemanticUpdate`;
1977+
the update cases in `AbstractCollectionEventTest`
1978+
| Covered
1979+
1980+
| `CFS-SCN-004` -- Uninitialized queued list appends
1981+
| `QueuedOperationDeltaTest#listAppendsShareOnePersistedSizeHandle`;
1982+
`ListDelayedOperationTest#testSimpleAddManaged`;
1983+
`CollectionMutationCompletionTest#queuedWorkWaitsForSuccessfulFlush`;
1984+
`QueuedOperationFailureTest#failedEntityInsertDoesNotFinalizeQueuedCollectionWork`
1985+
| Covered
1986+
1987+
| `CFS-SCN-005` -- Same-shape mutations
1988+
| `CollectionMutationCompletionTest#statementGroupingDoesNotMergeMutationIdentity`;
1989+
`BatchingExecutionMonitorTest#groupedRowsRetainPerRowBindingAndResultChecksWithOneCompletion`;
1990+
`CollectionMutationExecutorCompletionTest#directExecutionVisitsEveryGroupedBindingAndCompletesOnce`
1991+
| Covered
1992+
1993+
| `CFS-SCN-006` -- Zero-SQL update
1994+
| `CollectionMutationCompletionTest#zeroOperationMutationCompletesWhenSealed`;
1995+
`FlushCoordinatorCollectionGroupingTest#noOpCompletionCarriersRemainMutationLocal`
1996+
| Covered
1997+
1998+
| `CFS-SCN-007` -- Batch failure
1999+
| `BatchingExecutionMonitorTest#failedBatchSuppressesSemanticCompletion`;
2000+
`CollectionMutationCompletionTest#failureSuppressesSuccessfulCompletion`;
2001+
`#failedPhysicalWorkSuppressesEverySuccessfulLifecycleConsumer`;
2002+
`QueuedOperationFailureTest#failedEntityInsertDoesNotFinalizeQueuedCollectionWork`;
2003+
`OwnerUpdateCompletionCoordinatorTest#failureSuppressesPostUpdate`
2004+
| Covered
2005+
2006+
| `CFS-SCN-008` -- Discarded auto-flush
2007+
| `CollectionAutoFlushLifecycleTest#lifecycleWaitsForPositiveFlushDecision`;
2008+
`AutoFlushEventListenerTest#testAutoFlushNotRequired`;
2009+
`ActionQueueCheckpointTest#checkpointContract` and
2010+
`#postCheckpointOrphanCollectionRemovalIsDurable`
2011+
| Covered
2012+
2013+
| `CFS-SCN-009` -- Database-cascade remove
2014+
| `OnDeleteCascadeToElementCollectionTest#testCascadingDeleteUnloaded` and
2015+
`#testCascadingDeleteLoaded`, including their direct remove-event assertions;
2016+
collection remove event pairing in
2017+
`AbstractAssociationCollectionEventTest#testDeleteParentButNotChild`
2018+
| Covered
2019+
2020+
| `CFS-SCN-010` -- Cascade orphan-projection reuse
2021+
| `CollectionDeltaValidityTest#orphanProjectionContainsOnlyKnownRemovedState`
2022+
and `#mutationGenerationInvalidatesRetainedDelta` cover the reusable
2023+
projection and its comparison-validity primitive.
2024+
| Pending cascade integration
2025+
2026+
| `CFS-SCN-011` -- Cross-collection listener mutation
2027+
| Both queue fixtures of
2028+
`CrossCollectionListenerMutationTest#laterListenerMutationRepreparesEarlierCollection`
2029+
| Covered
2030+
2031+
| `CFS-SCN-012` -- Owner callback deduplication
2032+
| `AssociationChangeTests#testOwnerElementCollectionChangeGuardrail`,
2033+
`#testOwnerToManyChangeGuardrail`,
2034+
`#ownerPreUpdateCollectionChangeIsIncludedInTheSameFlush`, and
2035+
`#ownerPreUpdateCollectionReplacementIsIncludedInAutoFlush`;
2036+
`OwnerUpdateCompletionCoordinatorTest#waitsForEntityAndEveryCollectionParticipant`
2037+
| Covered
2038+
2039+
| `CFS-SCN-013` -- Initialization consumes queued operations
2040+
| `QueuedOperationInitializationTest#queuedCommandsAreAppliedAndClearedAfterSnapshotCapture`
2041+
| Covered
2042+
2043+
| `CFS-SCN-014` -- Conservative custom-semantics fallback
2044+
| `CollectionMutationInterpreterContractTest#defaultInterpreterRequestsInitializationForUninitializedCustomWrapper`;
2045+
`CustomCollectionMutationInterpreterTest#requiredFlushInitializesForConservativeInterpreterAndThenFreezesCompleteDelta`;
2046+
`CustomCollectionMutationInterpreterTest#specializedInterpreterCanDescribeUninitializedStateWithoutInitialization`;
2047+
`QueuedOperationInitializationTest#queuedCommandsAreAppliedAndClearedAfterSnapshotCapture`
2048+
| Covered
2049+
2050+
| `CFS-SCN-015` -- Graph row-change interpretation
2051+
| `CollectionInterpretationQueueExecutionTest.LegacyQueue#executesCreateAndUpdateDeltas`
2052+
and `CollectionInterpretationQueueExecutionTest.GraphQueue#executesCreateAndUpdateDeltas`;
2053+
`CollectionMutationInterpreterContractTest#createRetainsCurrentRowsWithoutSemanticAdditions`;
2054+
`CollectionMutationInterpreterContractTest#indexedUpdateRetainsOriginalChangeSet`
2055+
and `#changeSetFreezesStructuralLists`
2056+
| Covered
2057+
2058+
| `CFS-SCN-016` -- Create and bag-update row projection
2059+
| `CollectionMutationInterpreterContractTest#createRetainsCurrentRowsWithoutSemanticAdditions`;
2060+
`CollectionInterpretationQueueExecutionTest.GraphQueue#executesCompactCreateAcrossJdbcBatches`
2061+
and `#executesCompactBagRecreateAcrossJdbcBatches`;
2062+
`CollectionRowCompactionTest`; `GroupedCollectionInsertBindPlanTest`
2063+
| Covered
2064+
|===
19102065

1911-
Every `MUST` requirement maps to at least one named test suite above or requires
1912-
an explicitly recorded reason why static verification is sufficient.
2066+
These tables describe current coverage rather than prescribing nonexistent
2067+
monolithic suites. Every non-cascade `CFS-SCN-*` acceptance scenario now maps
2068+
to concrete tests; `CFS-SCN-010` remains the acceptance target for `CFS-CAS-*`.
2069+
A _Partial_ or _Pending_ row remains an open obligation. Before the
2070+
implementation is considered fully conformant, every `MUST` requirement must
2071+
map to a concrete test above or have an explicitly recorded reason why static
2072+
verification is sufficient.
19132073

19142074
== Approved Decisions
19152075

@@ -2187,7 +2347,8 @@ existing cascade traversal architecture.
21872347

21882348
== Development-Team Review and Change Control
21892349

2190-
Revision 1.2 was approved after satisfying these readiness criteria, including
2350+
Revision 1.4 remains approved under the same readiness criteria as revision 1.2,
2351+
including
21912352
acceptance of the resolved `OQ-012` performance gate and the new interpretation
21922353
test and benchmark obligations:
21932354

0 commit comments

Comments
 (0)