schema_update empty in load_packages

[gsarviareply]

dlt version

1.6.0

Describe the problem

In case of the run of a pipeline, with schema contract set to evolve for columns and datatype, the load_package in the output of the pipeline.run() contains the variable schema_update that is always empty, even if a new column is added at the source we extract data from.

Expected behavior

The expected behavior is that, when the schema_contract is evolve, this variable schema_update is populated with the columns that changed.

Steps to reproduce

To replicate the issue:

• Run the pipeline a first time to obtain the original schema and information from a SQL table
• Then add a new column in the source table and populate it
• Rerun the pipeline with the schema_contract set to evolve for columns and datatype
• Check the value of the schema_update variable within the load_packages in the output of the pipeline run

Operating system

Windows

Runtime environment

Local

Python version

3.10

dlt data source

MySQL

dlt destination

No response

Other deployment details

No response

Additional information

No response

[sh-rp]

Please provide a minimal full code exampe that reproduces this so we can take a look. Thanks :)

[sh-rp]

closing for inactivity

[gsarviareply]

I followed the example reported here:

https://github.com/dlt-hub/dlt_demos/blob/main/schema_evolution.ipynb

The difference with the example is that I am reading data from a mysql table. In a first run, the schema is inferred from the mysql table. Afterwards, a new column is added to the table in mysql and then a second pipeline run is performed, specifying the import_schema_path in the pipeline.run() (it is the path where the schema was produced in the first step).