Simplify use of local virtual refs

[nbren12]

Problem

An unexpected amount of precise boilerplate is required to open a local file reference inside a local repo.
If I do not provide auth in a very specific format I will get this error:

 a virtual chunk in this repository resolves to the url prefix file:///tmp/tmpu0we8rb0/data/, to be able to fetch the chunk you need to authorize the virtual chunk container when you open/create
  │ the repository, see https://icechunk.io/en/stable/virtual/
  │
  │ context:
  │    0: icechunk::store::get
  │            with key="test_arr/c/0/0" byte_range=From(0)
  │              at icechunk/src/store.rs:198
  │
  ╰─▶ a virtual chunk in this repository resolves to the url prefix file:///path/to/subdir/, to be able to fetch the chunk you need to authorize the virtual chunk container when you open/
      create the repository, see https://icechunk.io/en/stable/virtual/

This error can be avoided by creating a credentials object like this.

    credentials = icechunk.containers_credentials({
        "file:///path/to/subdir": None
    })

In addition the credentials seem quite picky about the path "file:///path/to/subdir". In my experiments it didn't work if I passed the path to a subdir like "file:///path/to". I found this confusing since the virtual_chunk_container seems to tolerate being passed a subpath.

config.set_virtual_chunk_container(
    icechunk.VirtualChunkContainer(
        "file:///path/to",
        icechunk.local_filesystem_store(data_path),
    )
)

Proposed Solution

Local file references should not require configuring any auth.

When required like for remote stores, paths in the auth configuration should have the same meaning as in the virtual chunk container.

Additional context

test_icechunk_file_urls.py

[paraseba]

Hello @nbren12. Thanks for reporting this.

Yes, there is a lot of boilerplate here. The main reason why we ask users to be so explicit about all this is security. Here is one of the attack vectors: A bad actor sets up a repository with one or more Virtual Chunks pointing to file:///home, the they store in the repo a config with the corresponding VirtualChunkContainer. If they trick you to open that repo, and Icechunk didn't ask you explicitly authorize the access, you would be reading from your local filesystem without knowing it, then they only need to trick you (or some system like a server that uses Icechunk) to somehow provide that data.

The current version of the API is very conservative, trying to avoid this issues at the price of usability.

Do you have suggestions on what to do to make it more usable without losing security. Here is one approach we could take: if you define the env variable UNSAFE_VIRTUAL_CHUNKS_OR_SOME_OTHER_SCARY_NAME, local filesystem doesn't require credentials and S3-like use credentials from the environment.

How would you feel about that?

In my experiments it didn't work if I passed the path to a subdir like "file:///path/to"

This is the same issue, currently we ask you to be very explicit so you are not authorizing more than you need.

[nbren12]

Thanks. That's a good point about security. I hadn't considered that, would hate for someone to exfil all my keys that way. It seems like the tip of the iceberg with zarr given the flexibility of the spec and custom codecs. That said, I do like the idea of simplifying the configuration when working with trusted datasets through an environment variable.

Perhaps the only thing needed is to make the error message better---for instance it could suggest the actual lines of code needed to fix. This took me some trial and error and browsing the code base to figure out.

The documentantion is also pretty sparse and potentially inaccurate. I'm not sure it's documented at all that you need to use credentials = icechunk.containers_credentials({"file:///path/to/subdir": None}), and the documentation claims here

No extra configuration is necessary for local filesystem references.

Also all the examples on the virtual datasets only show writing, and no loading an existing icechunk. When creating the icechunk, it's likely the user knows what prefixes they need to authorize, but it's less clear how to do this when reading. Are there any quick ways to figure out which virtual file system prefixes need to be loaded? Would be nice to show an warning message when the Repository is first opened listing any prefixes that need to be authorized.