Skip to content

Expose Avro writer schema when building the reader #9460

@mzabaluev

Description

@mzabaluev

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

Currently, the reader schema cannot be created with knowledge of the actual writer schema of the Avro file. Access to the discovered schema would more flexibly support complex resolution and projection use cases that are not covered by the Avro specification. One important instance is compatibility with Spark, which performs custom name-based resolution based on the Avro writer schema and the output schema of the operator, lacking record type names to resolve against.

Describe the solution you'd like

Since the Avro file header is already read as part of the construction of the reader to resolve the reader schema and apply the projection, it's a small change to expose this step in the builder API.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementAny new improvement worthy of a entry in the changelog

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions