refactor(spark): Re-use Spark Python models from PyPI instead of custom dataclasses

[Shekharrajak]

Ref #225

Currently, the SparkClient implementation uses custom dataclasses (Driver, Executor, SparkConnectInfo) defined in kubeflow/spark/types/types.py. We should migrate to using the official Spark Python models from the spark-operator PyPI package once they become available.

• Update imports and type hints throughout the codebase
• Ensure backward compatibility during migration

[github-actions]

🎉 Welcome to the Kubeflow SDK! 🎉

Thanks for opening your first issue! We're happy to have you as part of our community 🚀

Here's what happens next:

• Our team will review your issue soon! cc @kubeflow/kubeflow-sdk-team
• If you'd like to contribute to this issue, check out our Contributing Guide for repo-specific guidelines and the Kubeflow Contributor Guide for general community standards

Join the community:

• Slack: Join our #kubeflow-ml-experience and #kubeflow-trainer Slack channels
• Meetings: Attend the Kubeflow SDK and ML Experience bi-weekly meetings

Feel free to ask questions in the comments if you need any help or clarification!
Thanks again for contributing to Kubeflow! 🙏

[tariq-hasan]

I’d be interested to do the implementation.

/assign

[andreyvelich]

/area spark
/good-first-issue
/assign @tariq-hasan

[google-oss-prow]

@andreyvelich:
This request has been marked as suitable for new contributors.

Please ensure the request meets the requirements listed here.

If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-good-first-issue command.

Details

In response to this:

/area spark
/good-first-issue
/assign @tariq-hasan

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.