Batch upload directory (#1097)

* Batch upload directory Signed-off-by: Kevin Su <[email protected]> * lint Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> * lint Signed-off-by: Kevin Su <[email protected]> --------- Signed-off-by: Kevin Su <[email protected]>
flyteorg · Aug 29, 2023 · 20064ca · 20064ca
1 parent 344da7d
commit 20064ca
Showing 1 changed file with 18 additions and 0 deletions.
diff --git a/examples/basics/basics/folders.py b/examples/basics/basics/folders.py
@@ -49,6 +49,24 @@ def download_files(csv_urls: List[str]) -> FlyteDirectory:
     return FlyteDirectory(path=str(local_dir))
 
 
+# %% [markdown]
+# :::{note}
+# You can annotate a FlyteDirectory when you want to download/upload the contents of the directory in batches. For example,
+#
+# ```{code-block}
+# @task
+# def t1(directory: Annotated[FlyteDirectory, BatchSize(10)]) -> Annotated[FlyteDirectory, BatchSize(100)]:
+#     ...
+#     return FlyteDirectory(...)
+# ```
+#
+# In the above example flytekit will download all files from the input `directory` in chunks of 10, i.e. first it
+# downloads 10 files, loads them to memory, then writes those 10 to local disk, then it loads the next 10, so on
+# and so forth. Similarly, for outputs, in this case flytekit is going to upload the resulting directory in chunks of
+# 100.
+# :::
+
+
 # %% [markdown]
 # Next, we define a helper function to normalize the columns in-place.
 #