Skip to content

Commit 66ee7d8

Browse files
committed
avoid isfile calls on find as much as possible
If `walk` is also returning the same file, it should already have been included in the `out`. If `out` is not empty, we can safely assume that it is either a directory or a file. If it's empty, it could be that it is either an empty directory or, the path is a file and was not included from the `walk` results. This time we will make a `isfile` call. This is unfortunate but better than before, where we were making an additional for each find. This was noticed in DVC, which was making remote cache querying slower when traversing through the caches (512 calls instead of 256 :( ).
1 parent c3dac4f commit 66ee7d8

File tree

2 files changed

+2
-2
lines changed

2 files changed

+2
-2
lines changed

fsspec/asyn.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -581,7 +581,7 @@ async def _find(self, path, maxdepth=None, withdirs=False, **kwargs):
581581
if withdirs:
582582
files.update(dirs)
583583
out.update({info["name"]: info for name, info in files.items()})
584-
if (await self._isfile(path)) and path not in out:
584+
if not out and (await self._isfile(path)):
585585
# walk works on directories, but find should also return [path]
586586
# when path happens to be a file
587587
out[path] = {}

fsspec/spec.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -445,7 +445,7 @@ def find(self, path, maxdepth=None, withdirs=False, **kwargs):
445445
if withdirs:
446446
files.update(dirs)
447447
out.update({info["name"]: info for name, info in files.items()})
448-
if self.isfile(path) and path not in out:
448+
if not out and self.isfile(path):
449449
# walk works on directories, but find should also return [path]
450450
# when path happens to be a file
451451
out[path] = {}

0 commit comments

Comments
 (0)