fix: fill schema-added nested columns with typed NULL arrays on read by viirya · Pull Request #2635 · apache/iceberg-rust

viirya · 2026-06-12T23:53:01Z

Which issue does this PR close?

Closes Data files persisted without complex columns in the table schema hit "unexpected target column" on read #2618.

What changes are included in this PR?

When a column of a nested type — list, map, or a struct that itself contains nested children — is added to the table schema after data files were written, reading those older files fails with unexpected target column type List(...). The transformer correctly plans a ColumnSource::Add { value: None, .. } for the missing column, but the helpers that materialize the all-NULL array (create_primitive_array_repeated and create_primitive_array_single_element in arrow/value.rs) only covered primitive types plus structs with primitive-only children, each via a hand-written per-type NULL branch.

This PR replaces all of those NULL branches with a single early return using arrow's new_null_array, which constructs a typed all-NULL array for every Arrow type, including arbitrarily nested ones (the timezone of timestamps and precision/scale of decimals are part of the DataType, so they are preserved). The Some(literal) branches — used for initial_default values and partition constants — are unchanged. Net effect: the two functions shrink by ~180 lines and the unsupported-type failure mode for NULL filling disappears entirely.

Are these changes tested?

New regression test schema_evolution_adds_list_map_and_nested_struct_columns_with_nulls in record_batch_transformer.rs: a file batch containing only id is read against an evolved schema that added xs: list<int>, props: map<string, int>, and s: struct<a: string, ys: list<long>> (the struct's ys child also exercises the nested-children path that the old Struct branch couldn't handle). The test asserts the added columns come back with the evolved schema's Arrow types and null_count == num_rows.

The test fails on main with unexpected target column type List(Int32, ...) — the exact error from the issue — and passes with this change. Full iceberg lib suite (1313 tests) passes; clippy and rustfmt clean.

When a column of a nested type (list, map, or a struct with nested children) is added to the table schema after data files were written, reading those older files failed with "unexpected target column type": the helpers that materialize missing columns only handled primitive types plus structs with primitive-only children, via hand-written per-type NULL branches. Build the all-NULL column with arrow's new_null_array instead, which supports every Arrow type (including arbitrarily nested ones), and drop the per-type NULL branches from create_primitive_array_repeated and create_primitive_array_single_element. Closes apache#2618

advancedxy · 2026-06-15T08:37:31Z

+    // With no value, the single element is NULL. `new_null_array` supports every
+    // Arrow type, including nested ones (list/map/struct), which matters for
+    // columns added by schema evolution after a data file was written (#2618).
+    if prim_lit.is_none() {


Hi @viirya thanks for the fix. We encountered the same issue with a schema evolution case: a list of binary was added, and the original code cannot handle it. We fixed that internally and are about to contribute it back.

Our fix is similar with yours. However during the internal code review process, we noticed the function name create_primitive_array_single_element is no longer valid, it's not just creating primitive array any more, it also creating list/struct/map arrays now. Considering the default value in Iceberg V3, a struct could have default values. I think it would be best to change the signature to create_array_single_element snd passes prim_lit as &Option<Literal>, WDYT?

I think this fix is valid short-term fix. We can get it merged first, and refactor it in a follow-up PR to address the naming issue and default value support in Iceberg V3.

Thanks @advancedxy — agreed on both counts, and thanks for flagging the V3 angle.

You're right that create_primitive_* is now a misnomer since these functions materialize list/map/struct NULLs too. I dug into the V3 default-value direction a bit, and it's actually a slightly larger refactor than just the helper signatures: the value currently threaded into them is Option<PrimitiveLiteral> (via ColumnSource::Add), and generate_transform_operations deliberately drops any non-primitive initial_default today (if let Literal::Primitive(prim) = lit { .. } else { None } in record_batch_transformer.rs). So supporting a struct/nested default would mean widening ColumnSource::Add.value to Option<Literal> and the transformer's default-extraction alongside the rename to create_array_single_element.

Given that, I'd prefer to keep this PR as the focused wraparound/nested-NULL fix and do the rename + Literal widening + V3 default support together in the follow-up, so the type change lands in one coherent step rather than renaming now and re-touching the signature later. Happy to open the follow-up issue/PR for that.

Given that, I'd prefer to keep this PR as the focused wraparound/nested-NULL fix and do the rename + Literal widening + V3 default support together in the follow-up

Agreed. Let's merge this first.

viirya · 2026-06-20T15:56:02Z

cc @kevinjqliu

kevinjqliu

Thanks! A couple questions from me. Would be good to have others take a look too

BTW I also notice there's another PR addressing the same issue (#2649)

kevinjqliu · 2026-06-30T20:55:15Z

    }

+    #[test]
+    fn schema_evolution_adds_list_map_and_nested_struct_columns_with_nulls() {


nit: this test is a bit difficult to read. do you mind ordering so its easier to decipher the test?

we're using an evolved schema (with new optional list/map/struct types) to read the a data file that has the older schema. And we expect that all new optional list/map/struct types to be null.

should we also test for the shape of the null value? e.g. the null struct still has the expected child fields?

kevinjqliu · 2026-06-30T20:55:54Z

+    // With no value, the single element is NULL. `new_null_array` supports every
+    // Arrow type, including nested ones (list/map/struct), which matters for
+    // columns added by schema evolution after a data file was written (#2618).


nit: feels like we're over explaining here, wdyt?

maybe we should document the 1, i think its for new_null_array 's num_rows

kevinjqliu · 2026-06-30T20:58:08Z

                Ok(Arc::new(array))
            }
        }
-        (DataType::Timestamp(TimeUnit::Microsecond, timezone), None) => {


just double checking, do you know if new_null_array handles this and the nanosecond case correctly?

Yes. new_null_array builds from the DataType, and for timestamps the timezone is part of DataType::Timestamp(unit, tz) (and precision/scale are part of Decimal128), so the null array carries the exact same type the old per-type branches produced, including Timestamp(Nanosecond, tz). The thing it does not preserve is RunEndEncoded wrapping, but the NULL-fill path uses the plain schema type (REE is only applied on the Some/constant path via datum_to_arrow_type_with_ree), so there is no regression. A test asserting the full nested DataType (which kevinjqliu also asked for) would lock this in.

kevinjqliu · 2026-06-30T20:58:39Z

-                })?;
-            Ok(Arc::new(array))
-        }
-        (DataType::Struct(fields), None) => {


nice, glad we can get rid of these with the helper method!

kevinjqliu · 2026-06-30T20:59:45Z

 ///
 /// This is used for creating non-constant arrays where we need the same value
 /// repeated for each row.
 pub(crate) fn create_primitive_array_repeated(


maybe a good opportunity for refactoring with create_primitive_array_repeated, i see a lot of repetition. but its outside this pr's scope 😄

mbutrovich

Building on kevinjqliu's note that #2649 addresses the same issue: this is the one to take. #2649 is an additive per-type fix (+80/-1); this PR's new_null_array approach is more general, removes the existing per-type branches rather than extending them, and its test covers a struct whose child is itself a list (s: struct<a, ys: list<long>>) that an enumerate-each-type fix would miss. Recommend landing this and closing #2649 as redundant (the bug still gets fixed and #2649's intent is fully covered here).

Thanks @viirya!

mbutrovich · 2026-06-30T21:21:56Z

                Ok(Arc::new(array))
            }
        }
-        (DataType::Timestamp(TimeUnit::Microsecond, timezone), None) => {


Yes. new_null_array builds from the DataType, and for timestamps the timezone is part of DataType::Timestamp(unit, tz) (and precision/scale are part of Decimal128), so the null array carries the exact same type the old per-type branches produced, including Timestamp(Nanosecond, tz). The thing it does not preserve is RunEndEncoded wrapping, but the NULL-fill path uses the plain schema type (REE is only applied on the Some/constant path via datum_to_arrow_type_with_ree), so there is no regression. A test asserting the full nested DataType (which kevinjqliu also asked for) would lock this in.

@kevinjqliu

Address review feedback on apache#2635: - Extract the evolved schema into `schema_with_added_nested_columns` and reorder the test to read top-to-bottom as a story (old file -> evolved schema -> expect typed NULLs), per @kevinjqliu. - Assert the all-NULL struct still carries its full nested shape (child fields `a` and the nested list `ys`), locking in that the type-preserving new_null_array fill keeps nested children. - Trim the over-explanatory comments on the new_null_array NULL-fill paths and note that the literal `1` is new_null_array's row count.

viirya · 2026-06-30T22:00:07Z

Thanks @kevinjqliu — addressed the feedback in e845ae0:

Test readability / ordering. Extracted the evolved schema into schema_with_added_nested_columns() and reordered the test body so it reads top-to-bottom as the story you described: (1) the old data file with only id, (2) read it against the evolved schema, (3) expect the added list/map/struct columns as typed NULLs. The big nested-schema definition no longer sits in front of the actual logic.
Null shape assertion. Added an assertion that the all-NULL struct still carries its full nested shape — child fields a and the nested list ys — rather than collapsing to an empty struct. This is exactly the property the new_null_array approach guarantees over an enumerate-each-type fix (and what @mbutrovich noted).
new_null_array for timestamp/decimal (your question on the Some timestamp arm). Confirmed: new_null_array builds from the DataType, and timezone is part of DataType::Timestamp(unit, tz) and precision/scale part of Decimal128, so the NULL array carries the exact type the old per-type branches produced. I verified locally:
```
new_null_array(Timestamp(Nanosecond, Some("+08:00")), 1) -> dt = Timestamp(Nanosecond, Some("+08:00")), null
new_null_array(Decimal128(20, 4), 1)                     -> dt = Decimal128(20, 4), null
```
(REE wrapping isn't preserved, but the NULL-fill path uses the plain schema type — REE is only applied on the constant/Some path — so no regression.)
Comments. Trimmed the over-explanatory comments on both NULL-fill paths and noted the literal 1 is new_null_array's row count.
Refactoring create_primitive_array_repeated (the repetition you spotted) and the create_primitive_* rename / Literal widening for V3 default values — agreed these are worth doing, keeping them as the follow-up @advancedxy and I discussed above so this PR stays the focused Data files persisted without complex columns in the table schema hit "unexpected target column" on read #2618 fix.

viirya · 2026-06-30T22:06:05Z

Thanks @mbutrovich for reviewing.

advancedxy reviewed Jun 15, 2026

View reviewed changes

advancedxy approved these changes Jun 16, 2026

View reviewed changes

kevinjqliu reviewed Jun 30, 2026

View reviewed changes

kevinjqliu requested review from CTTY and blackmwk June 30, 2026 21:00

mbutrovich self-requested a review June 30, 2026 21:21

mbutrovich approved these changes Jun 30, 2026

View reviewed changes

Uh oh!

Conversation

viirya commented Jun 12, 2026

Which issue does this PR close?

What changes are included in this PR?

Are these changes tested?

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

viirya commented Jun 20, 2026

Uh oh!

kevinjqliu left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mbutrovich left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

viirya commented Jun 30, 2026

Uh oh!

viirya commented Jun 30, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants