Add documentation how to switch kfp client for single-user mode

[mr-yaky]

Hi all,

We are trying to test Kubeflow with Kale and Jupyterlab but due to the following issues:
#4377
#4440
kubeflow-kale/kale#204

we can't get it works together.

I also found some discussions that it's possible to use single-user mode for kfp but I have no any idea how to do that yet.
I don't know if this PR will be enough: #4638

Could someone clarify it, please ?

We used next deployment method: https://www.kubeflow.org/docs/started/k8s/kfctl-k8s-istio/

[davidspek]

Hi @mr-yaky, the deployment you are using is already single-user in terms of kubeflow, and I believe this is also true for KFP. However, the issues you link to are regarding multi-user deployments. If you indeed are trying to setup a multi-user environment you will need to follow https://www.kubeflow.org/docs/started/k8s/kfctl-istio-dex/.

Regarding issue #4440, that is regarding the issue that you cannot access the KFP API from within the cluster. Thus, the workaround in the comments of the above issue enables users to access the KFP API through the public endpoint using some policies, but that workaround isn't fully secure. Arrikto has fixed this issue using istio and mTLS, which should be released after KubeCon later this month.

The pull request of mine that you mentioned is aimed at setting the namespace for users in a multi-user environment automatically when they yse the KFP client from within the cluster. Again, this is regarding a multi-user deployment.

I might be mistaken, but I don't think there is a method to change the KFP client to single-user mode. The actual deployment of KFP can be single-user or multi-user, and that will depend on how you deployed kubeflow.

Without knowing what the exact issue is that you are facing, I can confirm that regardless of the issues you mention Kale is able to create and run pipelines (though you might need to use the workaround from #4440).

Something else you might be interested in. I have started to add support for taking snapshots using the generic kubernetes CSI driver to Kale. It is not yet fully functional, but unless you are using MiniKF or Arrikto's Enterprise Kubeflow (which has the fixes for the multi-user issues you linked to alread) which use Rok for storage, it might be important functionality for you. kubeflow-kale/kale#217

[mr-yaky]

Thank you @davidspek for clarifying. Anyway I'm still fighting with same error. So, maybe I have kfp in multy-user mode:

Message: (403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'content-length': '19', 'content-type': 'text/plain', 'date': 'Mon, 09 Nov 2020 16:48:25 GMT', 'server': 'envoy', 'x-envoy-upstream-service-time': '1'})
HTTP response body: RBAC: access denied

I might be mistaken, but I don't think there is a method to change the KFP client to single-user mode. The actual deployment of KFP can be single-user or multi-user, and that will depend on how you deployed kubeflow.

Any ideas how I can check it? I'm really need to solve this problem.

To keep this discussion clear I'll write you on other issue related to RBAC: access denied (kubeflow-kale/kale#204 (comment)) and we can keep this discussion exactly for adding some documentation if it's possible.

[mr-yaky]

Arrikto has fixed this issue using istio and mTLS, which should be released after KubeCon later this month.

I hope so!

[davidspek]

@mr-yaky The RBAC issue you are facing is due to issue #4440. It is not related to KFP being in multi-user as far as I can see. You should be able to use the workaround from the comments #4440 (comment) and #4440 (comment) for now. But I do recommend reading that discussion regarding the security concerns with doing this.

[mr-yaky]

well, I fixed already that issue with adding following to my servicerolebinding config: #4440 (comment)
@davidspek thank you for your support.

[rochaporto]

Are there news for a patch to use kfp from in-cluster notebooks in multi-user deployments?

[PatrickXYS]

I think it depends on @yanniszark 's availability for his PR to add support for mTLS and istio.

[davidspek]

@PatrickXYS SubjectAccessReview has been merged (see #3513), however, it is not clear to me what the state is of Istio mTLS for authentication.

[yanniszark]

@davidspek @PatrickXYS a PR hasn't been opened yet. AFAIK, the plan is to PR the ServiceAccountToken auth functionality after the holidays.

[davidspek]

@yanniszark Awesome, thanks for the update.

[omlomloml]

well, I fixed already that issue with adding following to my servicerolebinding config: #4440 (comment)
@davidspek thank you for your support.

@yanniszark @davidspek Hi guys, just want to clarify, I have a single user kubeflow setup, and I am trying to use kfp sdk inside the cluster, so do I only need to setup the ServiceRoleBinding from my current namespace? or do I also need to setup the Envoy filter? I wasn't sure what is the user name there since I am not using the notebook

Thanks

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically closed because it has not had recent activity. Please comment "/reopen" to reopen it.