Allow setting the root of a filesystem

[dhirschfeld]

It would be useful to be able to specify a filesystem with an arbitrary root - e.g.

fs = LocalFileSystem(root='C:/temp')

All operations would then be performed relative to this root - e.g. fs.ls('/') would then list the contents of C:/temp

[martindurant]

This is an interesting idea. pyfilesystem2 has something similar.
Rather than adding this capability to LocalFileSystem or the general spec, I would write a new implementation PrefixFileSystem which operates on any other filesystem and does the path manipulation you would need here. This would be like the caching file systems, with target_protocol and target_options kwargs.
Another idea from pyfilesystems2 would be to generalise this further: to be able to "mount" any other filesystem as some given path, so that, for instance "/s3/" lives alongside normal local files. I'm not certain how useful that is, given the differences between implementations.

[martindurant]

On the other hand, we already have ._strip_protocol() on all filesystems that would be a natural and easy place to put this logic.

[ahirner]

I do like the PrefixFileSystem idea. We have something like that. However, the interaction of ls and _strip_protocol is tricky imo. The former needs to reverse the latter. Then, some operations don't seem to use _strip_protocol so we override major operations like get and rm as well. Furthermore, stripping root might not catch all affected detail fields of ls.

Did you think a PrefixFileSystem would work solely by overriding _strip_protocol? Maybe I'm on the wrong foot here.

[brunorpinho]

Based on what I read on this post, I changed LocalFileSystem and it seems to be working fine based on a few tests.

import os
from fsspec.implementations.local import LocalFileSystem, stringify_path, make_path_posix

class PrefixFileSystem(LocalFileSystem):
    
    def __init__(self, root_path='/', *args, **kwargs):
        self.root_path = stringify_path(root_path)
        super().__init__(*args, **kwargs)
    
    def _strip_protocol(self, path):
        path = stringify_path(path)
        if path.startswith("file://"):
            path = path[7:]
        return os.path.join(self.root_path, path).rstrip("/")

[martindurant]

@brunorpinho , I don't think so, because the output of ls would include the original real paths of files, not their prefixed versions. Lots of other methods depend on ls (glob, find, any recursive-enabled method).

In any case, it would be a shame to provide something like this to the local files only.

[brunorpinho]

@martindurant yep, I see.

[lucmos]

This would be extremely useful with s3 filesystems, so that you can change the fs root to the specified bucket without prefixing the bucket to every path

[martindurant]

Perhaps you could make another implementation that wraps an arbitrary backend filesystem and does all the path manipulations? That's the kind of thing going on with the caching filesystems and ReferenceFileSystem, so there are precedents. Also, universal_path's implementation might be useful using Path-like objects.

[lucmos]

I see thanks for the suggestions!
Do you have any specific pointer for the universal_path and Path-like objects?

[martindurant]

See https://github.com/Quansight/universal_pathlib , and feel free to ask further questions there.

[efiop]

A bit unrelated, but my 2c about Path-like objects: we need to be careful with those, as things might get really slow if you create lots of them. We have this problem with PathInfo's in dvc right now and working on getting rid of those in favor of using strings. It is also easy for anyone writing a new filesystem to accidentally hinder the performance when working with batches of files.

[ahirner]

things might get really slow if you create lots of them

Can you link to an issue? It's unclear if creating PurePath and/or accidental IO is concerning you.

[efiop]

@ahirner It is creating PurePath, not accidental IO. When dealing with hundreds of thousands of files, creating PurePath for each one of those is takes significant time. Just bringing this up for the record, but it is not like this is unusual or not known. I see that @lucmos took good care of that in #745