feat: (schema - phase 2) Perform Column Statistics Schema Migration #14311

voonhous · 2025-11-21T11:34:44Z

Describe the issue this Pull Request addresses

This PR adds the required changes as stipulated in #14267 - phase 2: Perform Column Statistics Schema Migration.

This change is part of an effort to migrate column statistics handling to use the internal HoodieSchema representation instead of direct Avro Schema objects. This facilitates schema evolution support for column statistics, particularly for the min/max values recorded in the Hudi Metadata Table.

Summary and Changelog

This patch introduces HoodieSchema into the column statistics logic across FileFormatUtils, HoodieTableMetadataUtil, and ValueMetadata to support schema migration and evolution.

Updated methods in FileFormatUtils and HoodieTableMetadataUtil to accept and use HoodieSchema (or map of HoodieSchema) when aggregating column ranges and coercing values to Comparable<?>.
Modified ValueMetadata and ValueType to derive metadata and type information from HoodieSchema instead of Avro Schema, ensuring compatibility with existing Avro-based serialization
Deprecates use of direct Avro Schema in several column statistics-related utility methods in favor of HoodieSchema.

Impact

None, we are just wrapping existing code with a wrapper class and adapting functions that use Avro schema

Risk Level

low. The changes primarily involve replacing direct usage of Avro's Schema with Hudi's HoodieSchema in the metadata logic, which is an internal component. Existing Avro-based serialization is maintained.

Documentation Update

voonhous · 2025-11-22T09:59:51Z

CI is failing due to out of disk space error:

failed to register layer: write /opt/statsd/node_modules/typescript/lib/typescriptServices.js: no space left on device

@rahil-c @the-other-tim-brown This is ready for review

the-other-tim-brown · 2025-11-23T23:44:30Z

hudi-common/src/main/java/org/apache/hudi/common/util/FileFormatUtils.java

          } else {
            // schema is evolving for the column of interest.
-            Schema schema = colsToIndexSchemaMap.get(a.getColumnName());
+            HoodieSchema hoodieSchema = colsToIndexSchemaMap.get(a.getColumnName());


Nitpick: Let's keep the variable name schema

@voonhous just a reminder to push this change

the-other-tim-brown · 2025-11-23T23:47:03Z

hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java

        String fieldName = fieldNameFieldPair.getKey();
        Schema fieldSchema = getNonNullTypeFromUnion(fieldNameFieldPair.getValue().schema());
-        ColumnStats colStats = allColumnStats.computeIfAbsent(fieldName, ignored -> new ColumnStats(getValueMetadata(fieldSchema, indexVersion)));
+        HoodieSchema hoodieFieldSchema = HoodieSchema.fromAvroSchema(fieldSchema);


Can we update all the methods in this class to also take in HoodieSchema?

This may take awhile, there's ~3k LoC, it touches quite abit of tests too, will do it slowly.

Ok, can we add a separate task for this then?

No worries, i am already doing it now. Almost done.

the-other-tim-brown · 2025-11-23T23:49:51Z

hudi-common/src/main/java/org/apache/hudi/stats/ValueType.java

+   * @throws IllegalArgumentException if the schema type is not supported
+   * @since 1.2.0
+   */
+  public static ValueType fromSchema(HoodieSchema schema) {


This should be easier to migrate once we have added the other types in this PR #14312

Let me rebase this PR onto your changes since they are merged.

These changes are merged now

Modified fromSchema to not rely on avro for conversion.

the-other-tim-brown · 2025-11-26T16:07:44Z

hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java

+   * @since 1.2.0
   */
-  public static Comparable<?> coerceToComparable(Schema schema, Object val) {
+  public static Comparable<?> coerceToComparable(HoodieSchema hoodieSchema, Object val) {


nitpick: let's just leave the variable name as schema

the-other-tim-brown · 2025-11-26T18:13:18Z

hudi-common/src/main/java/org/apache/hudi/common/util/FileFormatUtils.java

 public abstract class FileFormatUtils {
  /**
-   * Aggregate column range statistics across files in a partition.
+   * Aggregate column range statistics across files in a partition using HoodieSchema.


I think it would be helpful to add the context that the HoodieSchema is used for properly extracting the stats based on the data type.

the-other-tim-brown · 2025-11-26T18:18:15Z

hudi-common/src/main/java/org/apache/hudi/stats/ValueMetadata.java

+      Schema avroSchema = valueSchema.toAvroSchema();
+      return DecimalMetadata.create((LogicalTypes.Decimal) avroSchema.getLogicalType());


Can this be updated to operate directly on HoodieSchema?

the-other-tim-brown · 2025-11-26T18:18:56Z

hudi-common/src/main/java/org/apache/hudi/stats/ValueType.java

+   */
+  public static ValueType fromSchema(HoodieSchema schema) {
+    // Handle logical types first using instanceof checks on specialized classes
+    if (schema instanceof HoodieSchema.Decimal) {


Instead of using instanceof we can just augment the switch statement below

the-other-tim-brown · 2025-11-26T18:21:34Z

...udi-spark/src/test/java/org/apache/hudi/functional/TestColStatsRecordWithMetadataRecord.java

+    Map<String, HoodieSchema> hoodieSchemaMap = colsToIndexSchemaMap.entrySet().stream()
+        .collect(Collectors.toMap(Map.Entry::getKey, entry -> HoodieSchema.fromAvroSchema(entry.getValue())));


Let's simplify this

Map<String, HoodieSchema> hoodieSchemaMap = Collections.singletonMap(colName, HoodieSchema.create(schemaType));

Let's also update the method signature to take in HoodieSchemaType instead of an avro type to help break away from the avro dependency

hudi-bot · 2025-11-28T10:46:34Z

CI report:

859aa63 UNKNOWN
965332d UNKNOWN
78a7204 UNKNOWN
7d35116 UNKNOWN
bf4a171 UNKNOWN
0084b79 UNKNOWN
9c5452e Azure: SUCCESS

Bot commands

@hudi-bot supports the following commands:

@hudi-bot run azure re-run the last Azure build

voonhous · 2025-11-28T10:50:18Z

Whew, all tests passing, it's ready for a second round of review now.

github-actions bot added the size:M PR with lines of changes in (100, 300] label Nov 21, 2025

voonhous changed the title ~~feat:~~ feat: (phase 2) Perform Column Statistics Schema Migration Nov 21, 2025

voonhous marked this pull request as draft November 21, 2025 16:23

voonhous marked this pull request as ready for review November 21, 2025 16:36

voonhous force-pushed the phase_2_col_stats branch from e2f5d30 to efa976a Compare November 21, 2025 16:37

voonhous marked this pull request as draft November 21, 2025 16:37

voonhous marked this pull request as ready for review November 21, 2025 18:34

voonhous changed the title ~~feat: (phase 2) Perform Column Statistics Schema Migration~~ feat (schema): (phase 2) Perform Column Statistics Schema Migration Nov 21, 2025

voonhous changed the title ~~feat (schema): (phase 2) Perform Column Statistics Schema Migration~~ feat: (schema - phase 2) Perform Column Statistics Schema Migration Nov 22, 2025

the-other-tim-brown reviewed Nov 23, 2025

View reviewed changes

voonhous force-pushed the phase_2_col_stats branch 4 times, most recently from 965332d to 78a7204 Compare November 26, 2025 08:48

voonhous mentioned this pull request Nov 26, 2025

feat: (schema - phase 5) Perform Java Client Core Migration #14340

Open

3 tasks

apache#14267 - phase 2: Perform Column Statistics Schema Migration

94f6e10

voonhous force-pushed the phase_2_col_stats branch from 78a7204 to 94f6e10 Compare November 26, 2025 08:53

the-other-tim-brown reviewed Nov 26, 2025

View reviewed changes

github-actions bot added size:XL PR with lines of changes > 1000 and removed size:M PR with lines of changes in (100, 300] labels Nov 27, 2025

Change method parameters in HoodieTableMetadataUtil to HoodieSchema

2fb147e

voonhous force-pushed the phase_2_col_stats branch 2 times, most recently from 7d35116 to 11ddcc4 Compare November 27, 2025 11:37

Fix type erasure issue due collector + stream usage

bf4a171

voonhous force-pushed the phase_2_col_stats branch 3 times, most recently from c4dc3d0 to 476d16a Compare November 27, 2025 13:15

Address comments

7a9e53f

voonhous force-pushed the phase_2_col_stats branch from 476d16a to 7a9e53f Compare November 27, 2025 14:50

Account for decimal being a bytes type

9c5452e

		Schema avroSchema = valueSchema.toAvroSchema();
		return DecimalMetadata.create((LogicalTypes.Decimal) avroSchema.getLogicalType());

		Map<String, HoodieSchema> hoodieSchemaMap = colsToIndexSchemaMap.entrySet().stream()
		.collect(Collectors.toMap(Map.Entry::getKey, entry -> HoodieSchema.fromAvroSchema(entry.getValue())));

feat: (schema - phase 2) Perform Column Statistics Schema Migration #14311

Are you sure you want to change the base?

feat: (schema - phase 2) Perform Column Statistics Schema Migration #14311

Conversation

voonhous commented Nov 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Describe the issue this Pull Request addresses

Summary and Changelog

Impact

Risk Level

Documentation Update

Uh oh!

voonhous commented Nov 22, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hudi-bot commented Nov 28, 2025

CI report:

Uh oh!

voonhous commented Nov 28, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

voonhous commented Nov 21, 2025 •

edited

Loading