./lodestar validator ... hits Error: EROFS: read-only file system, mkdir error

[khteh]

Describe the bug

I am running lodestar as k8s STS with the keystore ConfigMap mounted to /tmp/keystores directory. Running the validator hits the following error:

root@lodestar-0:/usr/app# ./lodestar validator --network holesky --dataDir=/data --importKeystores /tmp/keystores --importKeystoresPassword /tmp/password.txt
Jan-15 07:35:14.821[]                 info: Lodestar network=holesky, version=v1.25.0/191f2f0, commit=191f2f0ac71d4f660d1ffbd04044a58de114c049
Jan-15 07:35:14.823[]                 info: Connecting to LevelDB database path=/data/validator-db
 ✖ Error: EROFS: read-only file system, mkdir '/tmp/keystores/..2025_01_15_07_34_31.4293813640/keystore-m_12381_3600_0_0_0-1736913052.json.lock'
    at Object.mkdirSync (node:fs:1364:26)
    at Object.newFs.<computed> [as mkdir] (/usr/app/node_modules/proper-lockfile/lib/adapter.js:15:42)
    at acquireLock (/usr/app/node_modules/proper-lockfile/lib/lockfile.js:29:16)
    at RetryOperation._fn (/usr/app/node_modules/proper-lockfile/lib/lockfile.js:233:13)
    at RetryOperation.attempt (/usr/app/node_modules/retry/lib/retry_operation.js:112:8)
    at /usr/app/node_modules/proper-lockfile/lib/lockfile.js:232:19
    at resolveCanonicalPath (/usr/app/node_modules/proper-lockfile/lib/lockfile.js:17:16)
    at lock (/usr/app/node_modules/proper-lockfile/lib/lockfile.js:224:5)
    at /usr/app/node_modules/proper-lockfile/lib/adapter.js:53:9
    at lockSync (/usr/app/node_modules/proper-lockfile/index.js:13:42)
root@lodestar-0:/tmp/keystores# ls -al
total 12
drwxrwxrwx 3 root root 4096 Jan 15 07:34 .
drwxrwxrwt 1 root root 4096 Jan 15 07:34 ..
drwxr-xr-x 2 root root 4096 Jan 15 07:34 ..2025_01_15_07_34_31.4293813640
lrwxrwxrwx 1 root root   32 Jan 15 07:34 ..data -> ..2025_01_15_07_34_31.4293813640
lrwxrwxrwx 1 root root   50 Jan 15 07:34 keystore-m_12381_3600_0_0_0-1736913052.json -> ..data/keystore-m_12381_3600_0_0_0-1736913052.json
root@lodestar-0:/tmp/keystores# 

My manifest portion of the ConfigMap mount:

          volumeMounts:
            - name: validator-keystore
              mountPath: /tmp/keystores
              readOnly: false
      volumes:
        - name: validator-keystore
          configMap:
            name: validator-keystore
            defaultMode: 0755

This happens because ConfigMap is mounted RO.

Expected behavior

No error!

Steps to reproduce

No response

Additional context

No response

Operating system

Linux

Lodestar version or commit hash

v1.25.0

[nflaig]

The validator client will load the keystore files from /tmp/keystores which seems to work but it will create a .lock file for each keystore so it need to be able to write to that directory but your filesystem seems to be write only (EROFS: read-only file system). There is currently no way to disable to behavior so it will need the write access.

[khteh]

This will not work on k8s. All ConfigMap are mounted RO on k8s. Why can't it use it's current working directory?

[nflaig]

Why can't it use it's current working directory?

it could do that but there is currently no way to configure the lock file directory

[nflaig]

alternative is to add a hook or prestart script which moves the files from the configmap directory to another one

[khteh]

alternative is to add a hook or prestart script which moves the files from the configmap directory to another one

Yes, this works but please consider improving the working directory in order to work with ConfigMap.

[nflaig]

Yes, this works but please consider improving the working directory in order to work with ConfigMap.

what solution would you like to see for this? we can look into implementing something, I don't think we want to change the default behavior how out lock files work but we can add a feature flag maybe, might make sense if you could open another issues for this to track as a feature request if you have something in mind

[nflaig]

Closing this as it is expected that we can't write lock files in a read-only directory.

alternative is to add a hook or prestart script which moves the files from the configmap directory to another one

this should work for now, we are looking into another solution here #7367