k8s launcher python version compatibility (Python 3.7/3.8)

[andrijaperovic]

Running historical retrieval with os.environ["FEAST_SPARK_LAUNCHER"] = "k8s" breaks due to python compatibility with generated spark python file:

kubectl logs feast-m5mz27p6-driver -n spark-operator                                                                            main@d2a800f ✘
++ id -u
+ myuid=0
++ id -g
+ mygid=0
+ set +e
++ getent passwd 0
+ uidentry=root:x:0:0:root:/root:/bin/bash
+ set -e
+ '[' -z root:x:0:0:root:/root:/bin/bash ']'
+ SPARK_CLASSPATH=':/opt/spark/jars/*'
+ env
+ grep SPARK_JAVA_OPT_
+ sort -t_ -k4 -n
+ sed 's/[^=]*=\(.*\)/\1/g'
+ readarray -t SPARK_EXECUTOR_JAVA_OPTS
+ '[' -n '' ']'
+ '[' '' == 2 ']'
+ '[' '' == 3 ']'
+ '[' -n '' ']'
+ '[' -z ']'
+ case "$1" in
+ shift 1
+ CMD=("$SPARK_HOME/bin/spark-submit" --conf "spark.driver.bindAddress=$SPARK_DRIVER_BIND_ADDRESS" --deploy-mode client "$@")
+ exec /usr/bin/tini -s -- /opt/spark/bin/spark-submit --conf spark.driver.bindAddress=10.244.1.33 --deploy-mode client --properties-file /opt/spark/conf/spark.properties --class org.apache.spark.deploy.PythonRunner wasbs://[email protected]/artifacts/44d600e498692f5f4d9e183718f12fbd7925c1b97205373c181486ccab33f667.py --feature-tables W3siZmVhdHVyZXMiOiBbeyJuYW1lIjogImF2Z19kYWlseV90cmlwcyIsICJ0eXBlIjogIklOVDMyIn0sIHsibmFtZSI6ICJhY2NfcmF0ZSIsICJ0eXBlIjogIkZMT0FUIn0sIHsibmFtZSI6ICJjb252X3JhdGUiLCAidHlwZSI6ICJGTE9BVCJ9XSwgInByb2plY3QiOiAiZmVhc3Rkcml2aW5ncG9jdGVzdCIsICJuYW1lIjogImRyaXZlcl9zdGF0aXN0aWNzIiwgImVudGl0aWVzIjogW3sibmFtZSI6ICJkcml2ZXJfaWQiLCAidHlwZSI6ICJJTlQ2NCJ9XSwgIm1heF9hZ2UiOiA4NjQwMCwgImxhYmVscyI6IHt9fSwgeyJmZWF0dXJlcyI6IFt7Im5hbWUiOiAidHJpcHNfdG9kYXkiLCAidHlwZSI6ICJJTlQzMiJ9XSwgInByb2plY3QiOiAiZmVhc3Rkcml2aW5ncG9jdGVzdCIsICJuYW1lIjogImRyaXZlcl90cmlwcyIsICJlbnRpdGllcyI6IFt7Im5hbWUiOiAiZHJpdmVyX2lkIiwgInR5cGUiOiAiSU5UNjQifV0sICJtYXhfYWdlIjogODY0MDAsICJsYWJlbHMiOiB7fX1d --feature-tables-sources W3siZmlsZSI6IHsiZmllbGRfbWFwcGluZyI6IHt9LCAiZXZlbnRfdGltZXN0YW1wX2NvbHVtbiI6ICJkYXRldGltZSIsICJjcmVhdGVkX3RpbWVzdGFtcF9jb2x1bW4iOiAiY3JlYXRlZCIsICJkYXRlX3BhcnRpdGlvbl9jb2x1bW4iOiAiZGF0ZSIsICJwYXRoIjogIndhc2JzOi8vZmVhc3R0ZXN0QGZlYXN0ZHJpdmluZ3BvYy5ibG9iLmNvcmUud2luZG93cy5uZXQvZHJpdmVyX3N0YXRpc3RpY3MiLCAiZm9ybWF0IjogeyJqc29uX2NsYXNzIjogIlBhcnF1ZXRGb3JtYXQifX19LCB7ImZpbGUiOiB7ImZpZWxkX21hcHBpbmciOiB7fSwgImV2ZW50X3RpbWVzdGFtcF9jb2x1bW4iOiAiZGF0ZXRpbWUiLCAiY3JlYXRlZF90aW1lc3RhbXBfY29sdW1uIjogImNyZWF0ZWQiLCAiZGF0ZV9wYXJ0aXRpb25fY29sdW1uIjogImRhdGUiLCAicGF0aCI6ICJ3YXNiczovL2ZlYXN0dGVzdEBmZWFzdGRyaXZpbmdwb2MuYmxvYi5jb3JlLndpbmRvd3MubmV0L2RyaXZlcl90cmlwcyIsICJmb3JtYXQiOiB7Impzb25fY2xhc3MiOiAiUGFycXVldEZvcm1hdCJ9fX1d --entity-source eyJmaWxlIjogeyJmaWVsZF9tYXBwaW5nIjoge30sICJldmVudF90aW1lc3RhbXBfY29sdW1uIjogImV2ZW50X3RpbWVzdGFtcCIsICJjcmVhdGVkX3RpbWVzdGFtcF9jb2x1bW4iOiAiIiwgImRhdGVfcGFydGl0aW9uX2NvbHVtbiI6ICIiLCAicGF0aCI6ICJ3YXNiczovL2ZlYXN0dGVzdEBmZWFzdGRyaXZpbmdwb2MuYmxvYi5jb3JlLndpbmRvd3MubmV0L2FydGlmYWN0cy9hODYxMTg0My0yY2QxLTQ1MjktYTliNi1mMTEwNGExYTEzYjMiLCAiZm9ybWF0IjogeyJqc29uX2NsYXNzIjogIlBhcnF1ZXRGb3JtYXQifX19 --destination eyJmb3JtYXQiOiAicGFycXVldCIsICJwYXRoIjogImFiZnNzOi8vZmVhc3Rwb2NAZmVhc3Rwb2MuZGZzLmNvcmUud2luZG93cy5uZXQvZmVhc3Qvb3V0LzQ5ODQ1MzRhLWJiOGUtNDVmZi1hYzc3LTM2YzE3ZjA2YjRhOCJ9
22/01/23 22:43:01 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
22/01/23 22:43:02 WARN MetricsConfig: Cannot locate configuration: tried hadoop-metrics2-azure-file-system.properties,hadoop-metrics2.properties
  File "/tmp/spark-ac4d3d12-c430-4f11-987a-7a193b8fef54/44d600e498692f5f4d9e183718f12fbd7925c1b97205373c181486ccab33f667.py", line 73
    event_timestamp_column: str,
                          ^
SyntaxError: invalid syntax
log4j:WARN No appenders could be found for logger (org.apache.spark.util.ShutdownHookManager).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.

Upgrading to python3.8 from source in gcr.io/kf-feast/spark-py:v3.0.1 doesn't appear to fix the issue either.
Using historical features example:
https://github.com/Azure/feast-azure/blob/main/cluster/samples/feature_store_azure.ipynb