Null value during labeling causes can not save error

[sonalgoyal]

See if the following throws an error and fix accordingly:

1. Create febrl dataset with firstname completely null
2. Run findTrainingData and label
3. Save the labels

[sonalgoyal]

This was reported on Fabric, but lets test locally first.

[padam-prakash]

Issue

For a dataset where any column in the CSV has all null (empty) values, Spark infers the type as VoidType.
Since Parquet does not support VoidType, the write operation fails.

---

Solution

Force such columns to a valid type (e.g., StringType) before writing labelled output.

Example helper function:

def fix_void_columns(df):
    for col in df.columns:
        if df[col].apply(lambda x: x is None).all():
            df[col] = df[col].astype(str)  # cast to string
    return df

Usage (before writing labelled output):

candidate_pairs_pd = fix_void_columns(candidate_pairs_pd)

This case works without any error in local.

[sonalgoyal]

Thanks for looking into this @padam-prakash . When you say this works without any error in local, what does that mean? Does it mean that Zingg CLI on docket or Spark cluster works fine, but this si an issue on notebooks based environments? Please also indicate where the function needs to be introduced in the notebook and on which variable.

[padam-prakash]

When you say this works without any error in local, what does that mean?

It mean that on Zingg CLI , It worked fine. All the phase ran successfully and output saved in proper format.

but this is an issue on notebooks based environments?

Yes, it seems to be a Spark + Parquet issue specific to notebook environments.

Please also indicate where the function needs to be introduced in the notebook and on which variable.

The helper function can be defined in a separate cell, and should be called before writing labelled output, for example:

candidate_pairs_pd = fix_void_columns(candidate_pairs_pd)
zingg.writeLabelledOutputFromPandas(candidate_pairs_pd, args)