rsync: --filter on .git/info/exclude (skypilot-org#652)

* rsync: --filter on .git/info/exclude

* Update docs.

* Use --exclude-from, and check if git exclude exists

* Update docs

Author: concretevitamin

docs/source/examples/syncing-code-artifacts.rst

@@ -47,7 +47,7 @@ scripts, access checkpoints, etc.).
     For large, multi-gigabyte workdirs, uploading may be slow because the they
     are synced to the remote VM(s) with :code:`rsync`. To exclude large files in
     your workdir from being uploaded, add them to the :code:`.gitignore` file
-    under the workdir.
+    (or a ``.git/info/exclude`` file) under the workdir.
 
 .. note::
 
@@ -94,8 +94,9 @@ For more details, see `this example <https://github.com/sky-proj/sky/blob/master
 
 .. note::
 
-    Items listed in a :code:`.gitignore` file under any local file_mount source
-    are also ignored (the same behavior as handling ``workdir``).
+    Items listed in a :code:`.gitignore` file (or a ``.git/info/exclude`` file)
+    under a local file_mount source are also ignored (the same behavior as
+    handling ``workdir``).
 
 Uploading or reusing large files
 --------------------------------------

docs/source/reference/yaml-spec.rst

@@ -13,12 +13,13 @@ describe all fields available.
 
     # Working directory (optional), synced to ~/sky_workdir on the remote cluster
     # each time launch or exec is run with the yaml file.
+    #
     # NOTE: Sky does not currently support large, multi-gigabyte workdirs as the
     # files are synced to the remote VM with `rsync`. Please consider using Sky
-    # Storage to transfer large datasets and files. If a .gitignore exists anywhere
-    # within the working directory tree, the behavior will match git's behavior
-    # for finding and using .gitignore files. Files and directories included in
-    # a .gitignore file will be ignored by Sky.
+    # Storage to transfer large datasets and files.
+    #
+    # If a .gitignore file (or a .git/info/exclude file) exists in the working
+    # directory, files and directories listed in those files will be ignored.
     workdir: ~/my-task-code
 
     # Number of nodes (optional) to launch including the head node. If not

sky/backends/cloud_vm_ray_backend.py

@@ -131,8 +131,12 @@ def _path_size_megabytes(path: str, exclude_gitignore: bool = False) -> int:
                          'falling back to du -shk')
             pass
     return int(
-        subprocess.check_output(['du', '-sh', '-k', path
-                                ]).split()[0].decode('utf-8')) // (2**10)
+        subprocess.check_output([
+            'du',
+            '-sh',
+            '-k',
+            path,
+        ]).split()[0].decode('utf-8')) // (2**10)
 
 
 class RayCodeGen:
@@ -2147,7 +2151,19 @@ def _rsync_up(
         # to get a total progress bar, but it requires rsync>=3.1.0 and Mac
         # OS has a default rsync==2.6.9 (16 years old).
         rsync_command = ['rsync', '-Pavz']
-        rsync_command.append('--filter=\':- .gitignore\'')
+        # Legend
+        #   dir-merge: ignore file can appear in any subdir, applies to that
+        #     subdir downwards
+        # Note that "-" is mandatory for rsync and means all patterns in the
+        # ignore files are treated as *exclude* patterns.  Non-exclude
+        # patterns, e.g., "!  do_not_exclude" doesn't work, even though git
+        # allows it.
+        rsync_command.append('--filter=\'dir-merge,- .gitignore\'')
+        git_exclude = '.git/info/exclude'
+        if (pathlib.Path(source) / git_exclude).exists():
+            # Ensure file exists; otherwise, rsync will error out.
+            rsync_command.append('--exclude-from=.git/info/exclude')
+
         ssh_options = ' '.join(
             backend_utils.ssh_options_list(ssh_key,
                                            self._ssh_control_name(handle)))