"Permission denied" error when not using the default namespace in hf_interactive notebook

[MichaelClifford]

I've noticed the below permissions error while running the hf_interactive.ipynb notebook demo.

This error occurs when I deploy my RayCluster in any namespace besides the default namespace. Simply using default overcomes the permissions issue. However, this is not behavior we want to support. We need to ensure that the actions of the codeflare-sdk have the required permissions in the correct namespaces.

This issue can be partially circumvented by adding runtime_env = "env_vars": {"HF_HOME":"huggingface"} to your ray.init() command, however, it will just lead to a similar permission issue later on in training.

How can we ensure that the users have the correct permissions in their namespaces?

[KPostOffice]

Is this connected to #125 ?

[KPostOffice]

My current thought is that this is an issue with the upstream image not being built to be compatible with running as an arbitrary user on OpenShift. I'm planning on rebuilding and pushing the image after adding something like:

RUN chgrp -R 0 /home/ray && \
    chmod -R g+rwX /home/ray

as seen here

If this works we can rebuild the images ourselves and push them to our quay org. Then we can see if Ray is open to adding this to their builds

[KPostOffice]

I was able to get the hf_interactive notebook working using the following Dockerfile

FROM ghcr.io/foundation-model-stack/base:ray2.1.0-py38-gpu-pytorch1.12.0cu116-20221213-193103

USER 0

RUN chgrp -R 0 /home/ray && chmod -R g+rwX /home/ray

which is available at quay.io/kpostlet/ray:2.1.0 if anyone else wants to use this for testing you can pass it to your ClusterConfiguration with the image parameter.

cluster = Cluster(ClusterConfiguration(
    ...,
    image='quay.io/kpostlet/ray:2.1.0'
))

[KPostOffice]

This was added but reverted in the ray project. See ray-project/ray#32025 for discussion

[KPostOffice]

My current ideas are that we can either:

1. host a custom image with the correct permissions in the home directory
2. run pods with ray user (I think this is UID=1000)