Skip to content

Commit

Permalink
[spark] Stop checking if volume to mount in k8s exists locally (#148)
Browse files Browse the repository at this point in the history
It's entirely possible (and not an error) to create a podspec that
contains paths to volume-mount that do not exist wherever
service_configuration_lib is being invoked from.

As long as we ensure that we don't double-add things (which is only a
problem due to the way that the env var keys are constructed here),
skipping the existence check should be fine.
  • Loading branch information
nemacysts authored Sep 10, 2024
1 parent 363d435 commit 8c896ae
Show file tree
Hide file tree
Showing 2 changed files with 2 additions and 4 deletions.
5 changes: 2 additions & 3 deletions service_configuration_lib/spark_config.py
Original file line number Diff line number Diff line change
Expand Up @@ -223,13 +223,12 @@ def _get_k8s_docker_volumes_conf(

for volume in k8s_volumes:
host_path, container_path, mode = volume['hostPath'], volume['containerPath'], volume['mode']
if os.path.exists(host_path) and host_path not in mounted_volumes:
if host_path not in mounted_volumes:
env.update(_get_k8s_volume(host_path, container_path, mode))
mounted_volumes.add(host_path)
else:
log.warning(
f'Path {host_path} does not exist on this host or it has already been mounted.'
' Skipping this bindings.',
f'Path {host_path} has already been mounted. Skipping this binding.',
)
return env

Expand Down
1 change: 0 additions & 1 deletion tests/spark_config_test.py
Original file line number Diff line number Diff line change
Expand Up @@ -1216,7 +1216,6 @@ def _get_k8s_base_volumes(self):
"""Helper needed to allow tests to pass in github CI checks."""
return [
volume for volume in spark_config.K8S_BASE_VOLUMES
if os.path.exists(volume['containerPath'])
]

@pytest.fixture
Expand Down

0 comments on commit 8c896ae

Please sign in to comment.