Rootless podman file permissions #30681
I have searched the existing issues, both open and closed, to make sure this is not a duplicate report.
The bugFirst of all, thank you for making this app! I am trying to deploy immch using the rootless compose. I was previously using the nixOS package for immich, but switched to using compose2nix so that I could manage versions manually (and not have to wait for nixpkgs to update to the latest immich version). The problem that I am having looks very similar to #29710, where there is a permission error inside the machine learning container when trying to download the model. Since I'm using compose2nix, I've included the systemd start commands for the service and volume creation in place of the docker-compose.yml. Again, I understand this is not the supported deployment, but would appreciate any input/suggestions. The OS that Immich Server is running onnixOS Version of Immich Serverv3.10 Version of Immich Mobile AppN/A Platform with the issue
Device make and modelNo response Your docker-compose.yml content$ cat /nix/store/kfcdix6kr6lr4zka0hq4jhjbvlzpmzqw-unit-script-podman-immich_server-start/bin/podman-immich_server-start
#!/nix/store/7ik8057hajl6vq9j40h3jracxjn0bb5x-bash-5.3p15/bin/bash
set -e
exec podman \
run \
--name=immich_server \
--log-driver=journald \
--cidfile=/run/immich_server/ctr-id \
--cgroups=enabled \
--sdnotify=conmon \
-d \
--replace \
--env-file /run/agenix/immich-env \
-p 2283:2283/tcp \
-u 1026:1026 \
-v /etc/localtime:/etc/localtime:ro \
-v /mnt/immich/upload:/data:rw \
-v immich_thumbnails:/data/thumbs:rw \
-v /run/immich_config.json:/config.json:ro \
--rm \
--pull missing \
'--cap-drop=NET_RAW' \
'--network-alias=immich-server' \
'--network=immich_default' \
'--security-opt=no-new-privileges:true' \
ghcr.io/immich-app/immich-server:v3.1.0
$ cat /nix/store/i0j7s9njfy2ph2n5v93yckgdfkq2azfx-unit-script-podman-immich_redis-start/bin/podman-immich_redis-start
#!/nix/store/7ik8057hajl6vq9j40h3jracxjn0bb5x-bash-5.3p15/bin/bash
set -e
exec podman \
run \
--name=immich_redis \
--log-driver=journald \
--cidfile=/run/immich_redis/ctr-id \
--cgroups=enabled \
--sdnotify=conmon \
-d \
--replace \
-u 1026:1026 \
-v immich_redis_data:/data:rw \
--rm \
--pull missing \
'--cap-drop=NET_RAW' \
'--health-cmd=redis-cli ping || exit 1' \
'--network-alias=redis' \
'--network=immich_default' \
'--security-opt=no-new-privileges:true' \
docker.io/valkey/valkey:9@sha256:8e8d64b405ce18f41b8e5ee20aa4687a8ed0022d1298f2ce31cdcf3a76e09411
$ cat /nix/store/df3rx05j5n1rlxi4a6y9mzxx7w8d48sx-unit-script-podman-immich_postgres-start/bin/podman-immich_postgres-start
#!/nix/store/7ik8057hajl6vq9j40h3jracxjn0bb5x-bash-5.3p15/bin/bash
set -e
exec podman \
run \
--name=immich_postgres \
--log-driver=journald \
--cidfile=/run/immich_postgres/ctr-id \
--cgroups=enabled \
--sdnotify=conmon \
-d \
--replace \
-e POSTGRES_INITDB_ARGS=--data-checksums \
--env-file /run/agenix/immich-env \
-u 1026:1026 \
-v immich_postgres_data:/var/lib/postgresql/data:rw \
--rm \
--pull missing \
'--cap-drop=NET_RAW' \
'--network-alias=database' \
'--network=immich_default' \
'--security-opt=no-new-privileges:true' \
'--shm-size=134217728' \
ghcr.io/immich-app/postgres:14-vectorchord0.4.3-pgvectors0.2.0@sha256:bcf63357191b76a916ae5eb93464d65c07511da41e3bf7a8416db519b40b1c23
$ cat /nix/store/7f4msr67vh6wabm2kypk4c9c1hns1kdx-unit-script-podman-immich_machine_learning-start/bin/podman-immich_machine_learning-start
#!/nix/store/7ik8057hajl6vq9j40h3jracxjn0bb5x-bash-5.3p15/bin/bash
set -e
exec podman \
run \
--name=immich_machine_learning \
--log-driver=journald \
--cidfile=/run/immich_machine_learning/ctr-id \
--cgroups=enabled \
--sdnotify=conmon \
-d \
--replace \
--env-file /run/agenix/immich-env \
-u 1026:1026 \
-v immich_ml_config:/.config:rw \
-v immich_ml_dotcache:/.cache:rw \
-v immich_ml_model_cache:/cache:rw \
--rm \
--pull missing \
'--cap-drop=NET_RAW' \
'--network-alias=immich-machine-learning' \
'--network=immich_default' \
'--security-opt=no-new-privileges:true' \
ghcr.io/immich-app/immich-machine-learning:v3.1.0
$ cat /nix/store/8xl9j1bcsmg5q6vwd3g6s684akkkcr5q-unit-script-podman-volume-immich_ml_model_cache-start/bin/podman-volume-immich_ml_model_cache-start
#!/nix/store/7ik8057hajl6vq9j40h3jracxjn0bb5x-bash-5.3p15/bin/bash
set -e
podman volume inspect immich_ml_model_cache || podman volume create immich_ml_model_cache \
--gid=1026 \
--uid=1026
$ cat /nix/store/3b38sgcji240q07g4ncn6qv6a8gvd23j-unit-script-podman-volume-immich_ml_dotcache-start/bin/podman-volume-immich_ml_dotcache-start
#!/nix/store/7ik8057hajl6vq9j40h3jracxjn0bb5x-bash-5.3p15/bin/bash
set -e
podman volume inspect immich_ml_dotcache || podman volume create immich_ml_dotcache \
--gid=1026 \
--uid=1026
$ cat /nix/store/750pb7rl7dzwfm30fm40glnkgpl6ig66-unit-script-podman-volume-immich_ml_config-start/bin/podman-volume-immich_ml_config-start
#!/nix/store/7ik8057hajl6vq9j40h3jracxjn0bb5x-bash-5.3p15/bin/bash
set -e
podman volume inspect immich_ml_config || podman volume create immich_ml_config \
--gid=1026 \
--uid=1026Your .env contentDB_PASSWORD=<>
DB_USERNAME=postgres
DB_DATABASE_NAME=immich
POSTGRES_PASSWORD=<>
POSTGRES_USER=postgres
POSTGRES_DB=immich
TZ=America/New_YorkReproduction steps
Relevant log output[08/09/26 16:09:47] INFO Starting gunicorn 25.3.0
[08/09/26 16:09:47] INFO Listening at: http://[::]:3003 (8)
[08/09/26 16:09:47] INFO Using worker: immich_ml.config.CustomUvicornWorker
[08/09/26 16:09:47] INFO Booting worker with pid: 14
[08/09/26 16:09:49] WARNING mkdir -p failed for path
/usr/src/.config/matplotlib: [Errno 13] Permission
denied: '/usr/src/.config'
[08/09/26 16:09:49] WARNING Matplotlib created a temporary cache directory at
/tmp/matplotlib-t2vgov4j because there was an issue
with the default path
(/usr/src/.config/matplotlib); it is highly
recommended to set the MPLCONFIGDIR environment
variable to a writable directory, in particular to
speed up the import of Matplotlib and to better
support multiprocessing.
[08/09/26 16:09:50] INFO Started server process [14]
[08/09/26 16:09:50] INFO Waiting for application startup.
[08/09/26 16:09:50] INFO Created in-memory cache with unloading after 300s
of inactivity.
[08/09/26 16:09:50] INFO Initialized request thread pool with 6 threads.
[08/09/26 16:09:50] INFO Application startup complete.
[08/09/26 16:10:45] INFO Downloading textual model 'ViT-B-32__openai' to
/cache/clip/ViT-B-32__openai/textual/model.onnx.
This may take a while.
Fetching 11 files: 0%| | 0/11 [00:00<?, ?it/s]Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
[08/09/26 16:10:45] WARNING Warning: You are sending unauthenticated requests
to the HF Hub. Please set a HF_TOKEN to enable
higher rate limits and faster downloads.
{"timestamp":"2026-08-09T20:10:45.939047Z","level":"ERROR","fields":{"message":"Error logging to file \"/usr/src/.cache/huggingface/xet/logs/xet_20260809T161045936-0400_14.log\" (Permission denied (os error 13)); falling back to console logging."},"filename":"/home/runner/work/xet-core/xet-core/xet_runtime/src/logging/init.rs","line_number":58}
Fetching 11 files: 36%|███▋ | 4/11 [00:00<00:00, 12.19it/s]
[08/09/26 16:10:46] WARNING Failed to load textual model 'ViT-B-32__openai'.
Clearing cache.
[08/09/26 16:10:46] INFO Cleared cache directory for model
'ViT-B-32__openai'.
[08/09/26 16:10:46] INFO Downloading textual model 'ViT-B-32__openai' to
/cache/clip/ViT-B-32__openai/textual/model.onnx.
This may take a while.
Fetching 11 files: 36%|███▋ | 4/11 [00:00<00:00, 25.06it/s]
[08/09/26 16:10:46] ERROR Exception in ASGI application
╭─────── Traceback (most recent call last) ───────╮
│ /usr/src/immich_ml/main.py:244 in load │
│ │
│ 241 │ │ return model │
│ 242 │ │
│ 243 │ try: │
│ ❱ 244 │ │ return await run(_load, model) │
│ 245 │ except (OSError, InvalidProtobuf, Bad │
│ 246 │ │ log.warning(f"Failed to load {mod │
│ '{model.model_name}'. Clearing cache.") │
│ 247 │ │ model.clear_cache() │
│ │
│ /usr/src/immich_ml/main.py:219 in run │
│ │
│ 216 │ if thread_pool is None: │
│ 217 │ │ return func(*args, **kwargs) │
│ 218 │ partial_func = partial(func, *args, * │
│ ❱ 219 │ return await asyncio.get_running_loop │
│ 220 │
│ 221 │
│ 222 async def load(model: InferenceModel) -> │
│ │
│ /usr/local/lib/python3.11/concurrent/futures/th │
│ read.py:58 in run │
│ │
│ /usr/src/immich_ml/main.py:231 in _load │
│ │
│ 228 │ │ │ raise HTTPException(500, f"Fa │
│ 229 │ │ with lock: │
│ 230 │ │ │ try: │
│ ❱ 231 │ │ │ │ model.load() │
│ 232 │ │ │ except FileNotFoundError as e │
│ 233 │ │ │ │ if model.model_format == │
│ 234 │ │ │ │ │ raise e │
│ │
│ /usr/src/immich_ml/models/base.py:50 in load │
│ │
│ 47 │ │ │ return │
│ 48 │ │ self.load_attempts += 1 │
│ 49 │ │ │
│ ❱ 50 │ │ self.download() │
│ 51 │ │ attempt = f"Attempt #{self.load_a │
│ else "Loading" │
│ 52 │ │ log.info(f"{attempt} {self.model_ │
│ '{self.model_name}' to memory") │
│ 53 │ │ self.session = self._load() │
│ │
│ /usr/src/immich_ml/models/base.py:43 in │
│ download │
│ │
│ 40 │ │ if not self.cached: │
│ 41 │ │ │ model_type = self.model_type. │
│ 42 │ │ │ log.info(f"Downloading {model │
│ {self.model_path}. This may take a while. │
│ ❱ 43 │ │ │ self._download() │
│ 44 │ │
│ 45 │ def load(self) -> None: │
│ 46 │ │ if self.loaded: │
│ │
│ /usr/src/immich_ml/models/base.py:75 in │
│ _download │
│ │
│ 72 │ │ │ ModelFormat.RKNN: ["*.armnn"] │
│ 73 │ │ } │
│ 74 │ │ │
│ ❱ 75 │ │ snapshot_download( │
│ 76 │ │ │ f"immich-app/{clean_name(self │
│ 77 │ │ │ cache_dir=self.cache_dir, │
│ 78 │ │ │ local_dir=self.cache_dir, │
│ │
│ /opt/venv/lib/python3.11/site-packages/huggingf │
│ ace_hub/utils/_validators.py:88 in _inner_fn │
│ │
│ 85 │ │ │
│ 86 │ │ kwargs = smoothly_deprecate_legac │
│ 87 │ │ │
│ ❱ 88 │ │ return fn(*args, **kwargs) │
│ 89 │ │
│ 90 │ return _inner_fn # type: ignore │
│ 91 │
│ │
│ /opt/venv/lib/python3.11/site-packages/huggingf │
│ ace_hub/_snapshot_download.py:456 in │
│ snapshot_download │
│ │
│ 453 │ │ │ ) │
│ 454 │ │ ) │
│ 455 │ │
│ ❱ 456 │ thread_map( │
│ 457 │ │ _inner_hf_hub_download, │
│ 458 │ │ filtered_repo_files, │
│ 459 │ │ desc=tqdm_desc, │
│ │
│ /opt/venv/lib/python3.11/site-packages/tqdm/con │
│ trib/concurrent.py:69 in thread_map │
│ │
│ 66 │ │ [default: max(32, cpu_count() + 4 │
│ 67 │ """ │
│ 68 │ from concurrent.futures import Thread │
│ ❱ 69 │ return _executor_map(ThreadPoolExecut │
│ 70 │
│ 71 │
│ 72 def process_map(fn, *iterables, **tqdm_kw │
│ │
│ /opt/venv/lib/python3.11/site-packages/tqdm/con │
│ trib/concurrent.py:51 in _executor_map │
│ │
│ 48 │ │ # share lock in case workers are │
│ 49 │ │ with PoolExecutor(max_workers=max │
│ 50 │ │ │ │ │ │ initargs=(lk,)) │
│ ❱ 51 │ │ │ return list(tqdm_class(ex.map │
│ **kwargs)) │
│ 52 │
│ 53 │
│ 54 def thread_map(fn, *iterables, **tqdm_kwa │
│ │
│ /opt/venv/lib/python3.11/site-packages/tqdm/std │
│ .py:1181 in __iter__ │
│ │
│ 1178 │ │ time = self._time │
│ 1179 │ │ │
│ 1180 │ │ try: │
│ ❱ 1181 │ │ │ for obj in iterable: │
│ 1182 │ │ │ │ yield obj │
│ 1183 │ │ │ │ # Update and possibly pr │
│ 1184 │ │ │ │ # Note: does not call se │
│ │
│ /usr/local/lib/python3.11/concurrent/futures/_b │
│ ase.py:619 in result_iterator │
│ │
│ /usr/local/lib/python3.11/concurrent/futures/_b │
│ ase.py:317 in _result_or_cancel │
│ │
│ /usr/local/lib/python3.11/concurrent/futures/_b │
│ ase.py:456 in result │
│ │
│ /usr/local/lib/python3.11/concurrent/futures/_b │
│ ase.py:401 in __get_result │
│ │
│ /usr/local/lib/python3.11/concurrent/futures/th │
│ read.py:58 in run │
│ │
│ /opt/venv/lib/python3.11/site-packages/huggingf │
│ ace_hub/_snapshot_download.py:436 in │
│ _inner_hf_hub_download │
│ │
│ 433 │ # have the file locally. │
│ 434 │ def _inner_hf_hub_download(repo_file: │
│ 435 │ │ results.append( │
│ ❱ 436 │ │ │ hf_hub_download( # type: ign │
│ 437 │ │ │ │ repo_id, │
│ 438 │ │ │ │ filename=repo_file, │
│ 439 │ │ │ │ repo_type=repo_type, │
│ │
│ /opt/venv/lib/python3.11/site-packages/huggingf │
│ ace_hub/utils/_validators.py:88 in _inner_fn │
│ │
│ 85 │ │ │
│ 86 │ │ kwargs = smoothly_deprecate_legac │
│ 87 │ │ │
│ ❱ 88 │ │ return fn(*args, **kwargs) │
│ 89 │ │
│ 90 │ return _inner_fn # type: ignore │
│ 91 │
│ │
│ /opt/venv/lib/python3.11/site-packages/huggingf │
│ ace_hub/file_download.py:994 in hf_hub_download │
│ │
│ 991 │ ) │
│ 992 │ │
│ 993 │ if local_dir is not None: │
│ ❱ 994 │ │ return _hf_hub_download_to_local │
│ 995 │ │ │ # Destination │
│ 996 │ │ │ local_dir=local_dir, │
│ 997 │ │ │ # File info │
│ │
│ /opt/venv/lib/python3.11/site-packages/huggingf │
│ ace_hub/file_download.py:1448 in │
│ _hf_hub_download_to_local_dir │
│ │
│ 1445 │ # Otherwise, let's download the file │
│ 1446 │ with WeakFileLock(paths.lock_path): │
│ 1447 │ │ paths.file_path.unlink(missing_o │
│ ❱ 1448 │ │ _download_to_tmp_and_move( │
│ 1449 │ │ │ incomplete_path=paths.incomp │
│ 1450 │ │ │ destination_path=paths.file_ │
│ 1451 │ │ │ url_to_download=url_to_downl │
│ │
│ /opt/venv/lib/python3.11/site-packages/huggingf │
│ ace_hub/file_download.py:1860 in │
│ _download_to_tmp_and_move │
│ │
│ 1857 │ │ │ │
│ 1858 │ │ │ if xet_file_data is not None │
│ 1859 │ │ │ │ logger.debug("Xet Storag │
│ from Xet Storage..") │
│ ❱ 1860 │ │ │ │ xet_get( │
│ 1861 │ │ │ │ │ incomplete_path=tmp_ │
│ 1862 │ │ │ │ │ xet_file_data=xet_fi │
│ 1863 │ │ │ │ │ headers=headers, │
│ │
│ /opt/venv/lib/python3.11/site-packages/huggingf │
│ ace_hub/file_download.py:565 in xet_get │
│ │
│ 562 │ │ │ _prev = current │
│ 563 │ │ │
│ 564 │ │ try: │
│ ❱ 565 │ │ │ with session.new_file_downlo │
│ 566 │ │ │ │ token_refresh_url=xet_fi │
│ 567 │ │ │ │ token_refresh_headers=he │
│ 568 │ │ │ │ custom_headers=xet_heade │
╰─────────────────────────────────────────────────╯
OSError: I/O error: I/O error: Permission denied
(os error 13)
During handling of the above exception, another
exception occurred:
╭─────── Traceback (most recent call last) ───────╮
│ /usr/src/immich_ml/main.py:181 in predict │
│ │
│ 178 │ │ inputs = text │
│ 179 │ else: │
│ 180 │ │ raise HTTPException(400, "Either │
│ ❱ 181 │ response = await run_inference(inputs │
│ 182 │ return ORJSONResponse(response) │
│ 183 │
│ 184 │
│ │
│ /usr/src/immich_ml/main.py:206 in run_inference │
│ │
│ 203 │ │ response[entry["task"]] = output │
│ 204 │ │
│ 205 │ without_deps, with_deps = entries │
│ ❱ 206 │ await asyncio.gather(*[_run_inference │
│ 207 │ if with_deps: │
│ 208 │ │ await asyncio.gather(*[_run_infer │
│ 209 │ if isinstance(payload, Image): │
│ │
│ /usr/src/immich_ml/main.py:200 in │
│ _run_inference │
│ │
│ 197 │ │ │ except KeyError: │
│ 198 │ │ │ │ message = f"Task {entry[' │
│ output of {dep}" │
│ 199 │ │ │ │ raise HTTPException(400, │
│ ❱ 200 │ │ model = await load(model) │
│ 201 │ │ output = await run(model.predict, │
│ 202 │ │ outputs[model.identity] = output │
│ 203 │ │ response[entry["task"]] = output │
│ │
│ /usr/src/immich_ml/main.py:248 in load │
│ │
│ 245 │ except (OSError, InvalidProtobuf, Bad │
│ 246 │ │ log.warning(f"Failed to load {mod │
│ '{model.model_name}'. Clearing cache.") │
│ 247 │ │ model.clear_cache() │
│ ❱ 248 │ │ return await run(_load, model) │
│ 249 │
│ 250 │
│ 251 async def idle_shutdown_task() -> None: │
│ │
│ /usr/local/lib/python3.11/concurrent/futures/th │
│ read.py:58 in run │
│ │
│ /usr/src/immich_ml/main.py:231 in _load │
│ │
│ 228 │ │ │ raise HTTPException(500, f"Fa │
│ 229 │ │ with lock: │
│ 230 │ │ │ try: │
│ ❱ 231 │ │ │ │ model.load() │
│ 232 │ │ │ except FileNotFoundError as e │
│ 233 │ │ │ │ if model.model_format == │
│ 234 │ │ │ │ │ raise e │
│ │
│ /usr/src/immich_ml/models/base.py:50 in load │
│ │
│ 47 │ │ │ return │
│ 48 │ │ self.load_attempts += 1 │
│ 49 │ │ │
│ ❱ 50 │ │ self.download() │
│ 51 │ │ attempt = f"Attempt #{self.load_a │
│ else "Loading" │
│ 52 │ │ log.info(f"{attempt} {self.model_ │
│ '{self.model_name}' to memory") │
│ 53 │ │ self.session = self._load() │
│ │
│ /usr/src/immich_ml/models/base.py:43 in │
│ download │
│ │
│ 40 │ │ if not self.cached: │
│ 41 │ │ │ model_type = self.model_type. │
│ 42 │ │ │ log.info(f"Downloading {model │
│ {self.model_path}. This may take a while. │
│ ❱ 43 │ │ │ self._download() │
│ 44 │ │
│ 45 │ def load(self) -> None: │
│ 46 │ │ if self.loaded: │
│ │
│ /usr/src/immich_ml/models/base.py:75 in │
│ _download │
│ │
│ 72 │ │ │ ModelFormat.RKNN: ["*.armnn"] │
│ 73 │ │ } │
│ 74 │ │ │
│ ❱ 75 │ │ snapshot_download( │
│ 76 │ │ │ f"immich-app/{clean_name(self │
│ 77 │ │ │ cache_dir=self.cache_dir, │
│ 78 │ │ │ local_dir=self.cache_dir, │
│ │
│ /opt/venv/lib/python3.11/site-packages/huggingf │
│ ace_hub/utils/_validators.py:88 in _inner_fn │
│ │
│ 85 │ │ │
│ 86 │ │ kwargs = smoothly_deprecate_legac │
│ 87 │ │ │
│ ❱ 88 │ │ return fn(*args, **kwargs) │
│ 89 │ │
│ 90 │ return _inner_fn # type: ignore │
│ 91 │
│ │
│ /opt/venv/lib/python3.11/site-packages/huggingf │
│ ace_hub/_snapshot_download.py:456 in │
│ snapshot_download │
│ │
│ 453 │ │ │ ) │
│ 454 │ │ ) │
│ 455 │ │
│ ❱ 456 │ thread_map( │
│ 457 │ │ _inner_hf_hub_download, │
│ 458 │ │ filtered_repo_files, │
│ 459 │ │ desc=tqdm_desc, │
│ │
│ /opt/venv/lib/python3.11/site-packages/tqdm/con │
│ trib/concurrent.py:69 in thread_map │
│ │
│ 66 │ │ [default: max(32, cpu_count() + 4 │
│ 67 │ """ │
│ 68 │ from concurrent.futures import Thread │
│ ❱ 69 │ return _executor_map(ThreadPoolExecut │
│ 70 │
│ 71 │
│ 72 def process_map(fn, *iterables, **tqdm_kw │
│ │
│ /opt/venv/lib/python3.11/site-packages/tqdm/con │
│ trib/concurrent.py:51 in _executor_map │
│ │
│ 48 │ │ # share lock in case workers are │
│ 49 │ │ with PoolExecutor(max_workers=max │
│ 50 │ │ │ │ │ │ initargs=(lk,)) │
│ ❱ 51 │ │ │ return list(tqdm_class(ex.map │
│ **kwargs)) │
│ 52 │
│ 53 │
│ 54 def thread_map(fn, *iterables, **tqdm_kwa │
│ │
│ /opt/venv/lib/python3.11/site-packages/tqdm/std │
│ .py:1181 in __iter__ │
│ │
│ 1178 │ │ time = self._time │
│ 1179 │ │ │
│ 1180 │ │ try: │
│ ❱ 1181 │ │ │ for obj in iterable: │
│ 1182 │ │ │ │ yield obj │
│ 1183 │ │ │ │ # Update and possibly pr │
│ 1184 │ │ │ │ # Note: does not call se │
│ │
│ /usr/local/lib/python3.11/concurrent/futures/_b │
│ ase.py:619 in result_iterator │
│ │
│ /usr/local/lib/python3.11/concurrent/futures/_b │
│ ase.py:317 in _result_or_cancel │
│ │
│ /usr/local/lib/python3.11/concurrent/futures/_b │
│ ase.py:449 in result │
│ │
│ /usr/local/lib/python3.11/concurrent/futures/_b │
│ ase.py:401 in __get_result │
│ │
│ /usr/local/lib/python3.11/concurrent/futures/th │
│ read.py:58 in run │
│ │
│ /opt/venv/lib/python3.11/site-packages/huggingf │
│ ace_hub/_snapshot_download.py:436 in │
│ _inner_hf_hub_download │
│ │
│ 433 │ # have the file locally. │
│ 434 │ def _inner_hf_hub_download(repo_file: │
│ 435 │ │ results.append( │
│ ❱ 436 │ │ │ hf_hub_download( # type: ign │
│ 437 │ │ │ │ repo_id, │
│ 438 │ │ │ │ filename=repo_file, │
│ 439 │ │ │ │ repo_type=repo_type, │
│ │
│ /opt/venv/lib/python3.11/site-packages/huggingf │
│ ace_hub/utils/_validators.py:88 in _inner_fn │
│ │
│ 85 │ │ │
│ 86 │ │ kwargs = smoothly_deprecate_legac │
│ 87 │ │ │
│ ❱ 88 │ │ return fn(*args, **kwargs) │
│ 89 │ │
│ 90 │ return _inner_fn # type: ignore │
│ 91 │
│ │
│ /opt/venv/lib/python3.11/site-packages/huggingf │
│ ace_hub/file_download.py:994 in hf_hub_download │
│ │
│ 991 │ ) │
│ 992 │ │
│ 993 │ if local_dir is not None: │
│ ❱ 994 │ │ return _hf_hub_download_to_local │
│ 995 │ │ │ # Destination │
│ 996 │ │ │ local_dir=local_dir, │
│ 997 │ │ │ # File info │
│ │
│ /opt/venv/lib/python3.11/site-packages/huggingf │
│ ace_hub/file_download.py:1448 in │
│ _hf_hub_download_to_local_dir │
│ │
│ 1445 │ # Otherwise, let's download the file │
│ 1446 │ with WeakFileLock(paths.lock_path): │
│ 1447 │ │ paths.file_path.unlink(missing_o │
│ ❱ 1448 │ │ _download_to_tmp_and_move( │
│ 1449 │ │ │ incomplete_path=paths.incomp │
│ 1450 │ │ │ destination_path=paths.file_ │
│ 1451 │ │ │ url_to_download=url_to_downl │
│ │
│ /opt/venv/lib/python3.11/site-packages/huggingf │
│ ace_hub/file_download.py:1860 in │
│ _download_to_tmp_and_move │
│ │
│ 1857 │ │ │ │
│ 1858 │ │ │ if xet_file_data is not None │
│ 1859 │ │ │ │ logger.debug("Xet Storag │
│ from Xet Storage..") │
│ ❱ 1860 │ │ │ │ xet_get( │
│ 1861 │ │ │ │ │ incomplete_path=tmp_ │
│ 1862 │ │ │ │ │ xet_file_data=xet_fi │
│ 1863 │ │ │ │ │ headers=headers, │
│ │
│ /opt/venv/lib/python3.11/site-packages/huggingf │
│ ace_hub/file_download.py:565 in xet_get │
│ │
│ 562 │ │ │ _prev = current │
│ 563 │ │ │
│ 564 │ │ try: │
│ ❱ 565 │ │ │ with session.new_file_downlo │
│ 566 │ │ │ │ token_refresh_url=xet_fi │
│ 567 │ │ │ │ token_refresh_headers=he │
│ 568 │ │ │ │ custom_headers=xet_heade │
╰─────────────────────────────────────────────────╯
RuntimeError: Previous task error: I/O error: I/O
error: Permission denied (os error 13)Additional informationThe immich machine learning container is being started with the following podman command (started by a systemd service) If I run Looking inside the container storage, the mounts show up as being owned by the right user. It looks like matplotlib is trying to use the home/working directory I can see that some files have been written to Details``` $ podman exec -it immich_machine_learning find /cache /cache /cache/clip /cache/clip/ViT-B-16-SigLIP2__webli /cache/clip/ViT-B-16-SigLIP2__webli/README.md /cache/clip/ViT-B-16-SigLIP2__webli/visual /cache/clip/ViT-B-16-SigLIP2__webli/visual/preprocess_cfg.json /cache/clip/ViT-B-16-SigLIP2__webli/.gitattributes /cache/clip/ViT-B-16-SigLIP2__webli/textual /cache/clip/ViT-B-16-SigLIP2__webli/textual/special_tokens_map.json /cache/clip/ViT-B-16-SigLIP2__webli/textual/tokenizer_config.json /cache/clip/ViT-B-16-SigLIP2__webli/models--immich-app--ViT-B-16-SigLIP2__webli /cache/clip/ViT-B-16-SigLIP2__webli/models--immich-app--ViT-B-16-SigLIP2__webli/refs /cache/clip/ViT-B-16-SigLIP2__webli/models--immich-app--ViT-B-16-SigLIP2__webli/refs/main /cache/clip/ViT-B-16-SigLIP2__webli/.cache /cache/clip/ViT-B-16-SigLIP2__webli/.cache/huggingface /cache/clip/ViT-B-16-SigLIP2__webli/.cache/huggingface/CACHEDIR.TAG /cache/clip/ViT-B-16-SigLIP2__webli/.cache/huggingface/.gitignore /cache/clip/ViT-B-16-SigLIP2__webli/.cache/huggingface/download /cache/clip/ViT-B-16-SigLIP2__webli/.cache/huggingface/download/README.md.metadata /cache/clip/ViT-B-16-SigLIP2__webli/.cache/huggingface/download/README.md.lock /cache/clip/ViT-B-16-SigLIP2__webli/.cache/huggingface/download/.gitattributes.metadata /cache/clip/ViT-B-16-SigLIP2__webli/.cache/huggingface/download/config.json.metadata /cache/clip/ViT-B-16-SigLIP2__webli/.cache/huggingface/download/.gitattributes.lock /cache/clip/ViT-B-16-SigLIP2__webli/.cache/huggingface/download/visual /cache/clip/ViT-B-16-SigLIP2__webli/.cache/huggingface/download/visual/model.onnx.lock /cache/clip/ViT-B-16-SigLIP2__webli/.cache/huggingface/download/visual/preprocess_cfg.json.metadata /cache/clip/ViT-B-16-SigLIP2__webli/.cache/huggingface/download/visual/preprocess_cfg.json.lock /cache/clip/ViT-B-16-SigLIP2__webli/.cache/huggingface/download/textual /cache/clip/ViT-B-16-SigLIP2__webli/.cache/huggingface/download/textual/model.onnx.lock /cache/clip/ViT-B-16-SigLIP2__webli/.cache/huggingface/download/textual/special_tokens_map.json.lock /cache/clip/ViT-B-16-SigLIP2__webli/.cache/huggingface/download/textual/tokenizer.json.lock /cache/clip/ViT-B-16-SigLIP2__webli/.cache/huggingface/download/textual/tokenizer_config.json.lock /cache/clip/ViT-B-16-SigLIP2__webli/.cache/huggingface/download/textual/special_tokens_map.json.metadata /cache/clip/ViT-B-16-SigLIP2__webli/.cache/huggingface/download/textual/tokenizer_config.json.metadata /cache/clip/ViT-B-16-SigLIP2__webli/.cache/huggingface/download/config.json.lock /cache/clip/ViT-B-16-SigLIP2__webli/config.json /cache/clip/ViT-B-32__openai /cache/clip/ViT-B-32__openai/README.md /cache/clip/ViT-B-32__openai/models--immich-app--ViT-B-32__openai /cache/clip/ViT-B-32__openai/models--immich-app--ViT-B-32__openai/refs /cache/clip/ViT-B-32__openai/models--immich-app--ViT-B-32__openai/refs/main /cache/clip/ViT-B-32__openai/visual /cache/clip/ViT-B-32__openai/visual/preprocess_cfg.json /cache/clip/ViT-B-32__openai/.gitattributes /cache/clip/ViT-B-32__openai/textual /cache/clip/ViT-B-32__openai/textual/tokenizer.json /cache/clip/ViT-B-32__openai/textual/special_tokens_map.json /cache/clip/ViT-B-32__openai/textual/tokenizer_config.json /cache/clip/ViT-B-32__openai/textual/merges.txt /cache/clip/ViT-B-32__openai/textual/vocab.json /cache/clip/ViT-B-32__openai/.cache /cache/clip/ViT-B-32__openai/.cache/huggingface /cache/clip/ViT-B-32__openai/.cache/huggingface/CACHEDIR.TAG /cache/clip/ViT-B-32__openai/.cache/huggingface/.gitignore /cache/clip/ViT-B-32__openai/.cache/huggingface/download /cache/clip/ViT-B-32__openai/.cache/huggingface/download/README.md.metadata /cache/clip/ViT-B-32__openai/.cache/huggingface/download/README.md.lock /cache/clip/ViT-B-32__openai/.cache/huggingface/download/.gitattributes.metadata /cache/clip/ViT-B-32__openai/.cache/huggingface/download/config.json.metadata /cache/clip/ViT-B-32__openai/.cache/huggingface/download/.gitattributes.lock /cache/clip/ViT-B-32__openai/.cache/huggingface/download/visual /cache/clip/ViT-B-32__openai/.cache/huggingface/download/visual/model.onnx.lock /cache/clip/ViT-B-32__openai/.cache/huggingface/download/visual/preprocess_cfg.json.metadata /cache/clip/ViT-B-32__openai/.cache/huggingface/download/visual/preprocess_cfg.json.lock /cache/clip/ViT-B-32__openai/.cache/huggingface/download/textual /cache/clip/ViT-B-32__openai/.cache/huggingface/download/textual/merges.txt.metadata /cache/clip/ViT-B-32__openai/.cache/huggingface/download/textual/model.onnx.lock /cache/clip/ViT-B-32__openai/.cache/huggingface/download/textual/special_tokens_map.json.lock /cache/clip/ViT-B-32__openai/.cache/huggingface/download/textual/merges.txt.lock /cache/clip/ViT-B-32__openai/.cache/huggingface/download/textual/vocab.json.metadata /cache/clip/ViT-B-32__openai/.cache/huggingface/download/textual/vocab.json.lock /cache/clip/ViT-B-32__openai/.cache/huggingface/download/textual/tokenizer.json.lock /cache/clip/ViT-B-32__openai/.cache/huggingface/download/textual/tokenizer_config.json.lock /cache/clip/ViT-B-32__openai/.cache/huggingface/download/textual/special_tokens_map.json.metadata /cache/clip/ViT-B-32__openai/.cache/huggingface/download/textual/tokenizer.json.metadata /cache/clip/ViT-B-32__openai/.cache/huggingface/download/textual/tokenizer_config.json.metadata /cache/clip/ViT-B-32__openai/.cache/huggingface/download/config.json.lock /cache/clip/ViT-B-32__openai/config.json ```The error in the machine learning service log is different from if the I have also tried using bind mounts instead of using podman volumes, but the result is the same. |
Answered by
reed-foster
Aug 9, 2026
Replies: 1 comment
|
Okay, I just tried changing the mount point inside the container for
|
0 replies
Answer selected by
reed-foster
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Okay, I just tried changing the mount point inside the container for
/.cacheand/.configto be/usr/src/.cacheand/usr/src/.config, and so far no errors. I am able to download the model.I'm waiting for the smart search job to complete now.This resolved the issue; I'm now able to use the smart search and other features that depend on the machine learning container.