Offloading of objects during registration is a difficult to debug trap for inexperienced users

[fg91]

Discussed in #4743

^{Originally posted by fg91 January 18, 2024}
Let us consider an example where 1) the type engine transports an object by offloading it to blob storage (in this case as a pickle file) and 2) where this object is instantiated not in a task but when calling a task in a workflow:

from flytekit import task, workflow


class Config:
    """Something that is transported by the Flyte type engine via pickle."""
    def __init__(self, a) -> None:
        self.a = a


@task
def print_config(config: Config) -> None:
    print(config)


@workflow
def wf():
    print_config(config=Config(a=5))

This workflow fails with:

      File ".../flytekit/core/base_task.py", line 626, in dispatch_execute
        raise type(exc)(msg) from exc

Message:

    Failed to convert inputs of task 'wf.print_config':
  [Errno 2] No such file or directory: '/var/folders/vz/l684gsw57pndcbm_909n9jp40000gn/T/flyte-e45vhbku/raw/91b6f84917aa22176a7ba5a4f4e71fd5/74a1306209f894ebf0fc720fde719b3b'

SYSTEM ERROR! Contact platform administrators.

The reason is that during registration, pyflyte does not realize that the object needs to be uploaded to blob storage. The user would have to proactively configure the raw data prefix.

I would argue that this example is very difficult to understand and debug for users that don't have a clear understanding of Flyte's data model and too "simple" to let users fall into this trap.

As a user I would want flytekit to 1) realize that during registration, files need to be offloaded to blob storage and 2) the backend to specify a default raw data prefix during registration unless I configured it explicitly in my flyte config file.

---

How could this be fixed?

In the FileAccessProvider, we need to prevent that put_raw_data stores offloaded objects locally during registration.

• Is there a way to determine from "the flyte context that we are in registration mode"? Then we could at least catch this error and tell users to configure the raw data prefix.
• Could pyflyte request the raw_data_prefix from flyteadmin if the user didn't set it explicitly and "make the file access provider aware of it"?