Merge pull request #111 from ExpediaGroup/test_feast_merge

fix: Sync from open source feast to EG Feast for July, 2023 changes

.github/workflows/nightly-ci.yml

@@ -27,7 +27,7 @@ jobs:
   cleanup_dynamo_tables:
     if: github.repository == 'feast-dev/feast'
     runs-on: ubuntu-latest
-    name: Cleanup dynamo tables which can fail to cleanup
+    name: Cleanup Bigtable / Dynamo tables which can fail to cleanup
     steps:
       - uses: actions/checkout@v2
         with:
@@ -47,9 +47,20 @@ jobs:
       - name: Install Python dependencies
         run: |
           pip install boto3
+          pip install google-cloud-bigtable
           pip install tqdm
-      - name: Run DynamoDB cleanup script
-        run: python infra/scripts/cleanup_dynamo_ci.py
+      - name: Authenticate to Google Cloud
+        uses: 'google-github-actions/auth@v1'
+        with:
+          credentials_json: '${{ secrets.GCP_SA_KEY }}'
+      - name: Set up gcloud SDK
+        uses: google-github-actions/setup-gcloud@v1
+        with:
+          project_id: ${{ secrets.GCP_PROJECT_ID }}
+      - name: Use gcloud CLI
+        run: gcloud info
+      - name: Run DynamoDB / Bigtable cleanup script
+        run: python infra/scripts/cleanup_ci.py
   build-docker-image:
     if: github.repository == 'feast-dev/feast'
     needs: [check_date]

CHANGELOG.md

@@ -1,5 +1,28 @@
 # Changelog
 
+# [0.32.0](https://github.com/feast-dev/feast/compare/v0.31.0...v0.32.0) (2023-07-17)
+
+
+### Bug Fixes
+
+* Added generic Feature store Creation for CLI ([#3618](https://github.com/feast-dev/feast/issues/3618)) ([bf740d2](https://github.com/feast-dev/feast/commit/bf740d2d0ef3b62cf27f25083b5b4d49955b56fb))
+* Broken non-root path with projects-list.json ([#3665](https://github.com/feast-dev/feast/issues/3665)) ([4861af0](https://github.com/feast-dev/feast/commit/4861af0f3206f965ad2f7a4eddf2ebec2df149f8))
+* Clean up snowflake to_spark_df() ([#3607](https://github.com/feast-dev/feast/issues/3607)) ([e8e643e](https://github.com/feast-dev/feast/commit/e8e643e3555127daf37fa4961fdadb0508a041db))
+* Entityless fv breaks with `KeyError: __dummy` applying feature_store.plan() on python ([#3640](https://github.com/feast-dev/feast/issues/3640)) ([ef4ef32](https://github.com/feast-dev/feast/commit/ef4ef32f7466b5d874d537abe8def4731b15dc85))
+* Fix scan datasize to 0 for inference schema ([#3628](https://github.com/feast-dev/feast/issues/3628)) ([c3dd74e](https://github.com/feast-dev/feast/commit/c3dd74e238b5b9e784e3dbe102941c66a63f6686))
+* Fix timestamp consistency in push api ([#3614](https://github.com/feast-dev/feast/issues/3614)) ([9b227d7](https://github.com/feast-dev/feast/commit/9b227d7d44f30d28d1faadc8015f25dc4a6f56b5))
+* For SQL registry, increase max data_source_name length to 255 ([#3630](https://github.com/feast-dev/feast/issues/3630)) ([478caec](https://github.com/feast-dev/feast/commit/478caecc8d61b6070ec03bc13688a83b8c5f5936))
+* Implements connection pool for postgres online store ([#3633](https://github.com/feast-dev/feast/issues/3633)) ([059509a](https://github.com/feast-dev/feast/commit/059509a492d180effb1786713738665e293838e7))
+* Manage redis pipe's context ([#3655](https://github.com/feast-dev/feast/issues/3655)) ([48e0971](https://github.com/feast-dev/feast/commit/48e097130e68241e751bd4be5af7427fffad47cf))
+* Missing Catalog argument in athena connector ([#3661](https://github.com/feast-dev/feast/issues/3661)) ([f6d3caf](https://github.com/feast-dev/feast/commit/f6d3caf8affc74aef7ac489d3e6816d45b30d820))
+* Optimize bytes processed when retrieving entity df schema to 0 ([#3680](https://github.com/feast-dev/feast/issues/3680)) ([1c01035](https://github.com/feast-dev/feast/commit/1c010357affd48616d39f1ad01b872fac946269d))
+
+
+### Features
+
+* Add gunicorn for serve with multiprocess ([#3636](https://github.com/feast-dev/feast/issues/3636)) ([4de7faf](https://github.com/feast-dev/feast/commit/4de7faf7b262d30a9f6795911d8fa97df775fa8d))
+* Use string as a substitute for unregistered types during schema inference ([#3646](https://github.com/feast-dev/feast/issues/3646)) ([c474ccd](https://github.com/feast-dev/feast/commit/c474ccdd23ca8161de5e2958f0a12826c020dc44))
+
 # [0.31.0](https://github.com/feast-dev/feast/compare/v0.30.0...v0.31.0) (2023-04-21)
 
 

go.sum

@@ -701,4 +701,4 @@ inet.af/netaddr v0.0.0-20220811202034-502d2d690317 h1:U2fwK6P2EqmopP/hFLTOAjWTki
 inet.af/netaddr v0.0.0-20220811202034-502d2d690317/go.mod h1:OIezDfdzOgFhuw4HuWapWq2e9l0H9tK4F1j+ETRtF3k=
 rsc.io/pdf v0.1.1/go.mod h1:n8OzWcQ6Sp37PL01nO98y4iUCRdTGarVfzxY20ICaU4=
 sigs.k8s.io/yaml v1.1.0/go.mod h1:UJmg0vDUVViEyp3mgSv9WPwZCDxu4rQW1olrI1uml+o=
-sourcegraph.com/sourcegraph/appdash v0.0.0-20190731080439-ebfcffb1b5c0/go.mod h1:hI742Nqp5OhwiqlzhgfbWU4mW4yO10fP+LoT9WOswdU=
+sourcegraph.com/sourcegraph/appdash v0.0.0-20190731080439-ebfcffb1b5c0/go.mod h1:hI742Nqp5OhwiqlzhgfbWU4mW4yO10fP+LoT9WOswdU=

infra/charts/feast-feature-server/Chart.yaml

@@ -2,7 +2,7 @@ apiVersion: v2
 name: feast-feature-server
 description: Feast Feature Server in Go or Python
 type: application
-version: 0.31.0
+version: 0.32.0
 keywords:
   - machine learning
   - big data

infra/charts/feast-feature-server/README.md

@@ -1,6 +1,6 @@
 # Feast Python / Go Feature Server Helm Charts
 
-Current chart version is `0.31.0`
+Current chart version is `0.32.0`
 
 ## Installation
 
@@ -30,7 +30,7 @@ See [here](https://github.com/feast-dev/feast/tree/master/examples/python-helm-d
 | fullnameOverride | string | `""` |  |
 | image.pullPolicy | string | `"IfNotPresent"` |  |
 | image.repository | string | `"feastdev/feature-server"` | Docker image for Feature Server repository |
-| image.tag | string | `"0.31.0"` | The Docker image tag (can be overwritten if custom feature server deps are needed for on demand transforms) |
+| image.tag | string | `"0.32.0"` | The Docker image tag (can be overwritten if custom feature server deps are needed for on demand transforms) |
 | imagePullSecrets | list | `[]` |  |
 | livenessProbe.initialDelaySeconds | int | `30` |  |
 | livenessProbe.periodSeconds | int | `30` |  |

infra/charts/feast-feature-server/values.yaml

@@ -9,7 +9,7 @@ image:
   repository: feastdev/feature-server
   pullPolicy: IfNotPresent
   # image.tag -- The Docker image tag (can be overwritten if custom feature server deps are needed for on demand transforms)
-  tag: 0.31.0
+  tag: 0.32.0
 
 imagePullSecrets: []
 nameOverride: ""

infra/charts/feast/Chart.yaml

@@ -1,7 +1,7 @@
 apiVersion: v1
 description: Feature store for machine learning
 name: feast
-version: 0.31.0
+version: 0.32.0
 keywords:
   - machine learning
   - big data

infra/charts/feast/README.md

@@ -8,7 +8,7 @@ This repo contains Helm charts for Feast Java components that are being installe
 
 ## Chart: Feast
 
-Feature store for machine learning Current chart version is `0.31.0`
+Feature store for machine learning Current chart version is `0.32.0`
 
 ## Installation
 
@@ -65,8 +65,8 @@ See [here](https://github.com/feast-dev/feast/tree/master/examples/java-demo) fo
 | Repository | Name | Version |
 |------------|------|---------|
 | https://charts.helm.sh/stable | redis | 10.5.6 |
-| https://feast-helm-charts.storage.googleapis.com | feature-server(feature-server) | 0.31.0 |
-| https://feast-helm-charts.storage.googleapis.com | transformation-service(transformation-service) | 0.31.0 |
+| https://feast-helm-charts.storage.googleapis.com | feature-server(feature-server) | 0.32.0 |
+| https://feast-helm-charts.storage.googleapis.com | transformation-service(transformation-service) | 0.32.0 |
 
 ## Values
 

infra/charts/feast/charts/feature-server/Chart.yaml

@@ -1,8 +1,8 @@
 apiVersion: v1
 description: "Feast Feature Server: Online feature serving service for Feast"
 name: feature-server
-version: 0.31.0
-appVersion: v0.31.0
+version: 0.32.0
+appVersion: v0.32.0
 keywords:
 - machine learning
 - big data

infra/charts/feast/charts/feature-server/README.md

@@ -1,6 +1,6 @@
 # feature-server
 
-![Version: 0.31.0](https://img.shields.io/badge/Version-0.31.0-informational?style=flat-square) ![AppVersion: v0.31.0](https://img.shields.io/badge/AppVersion-v0.31.0-informational?style=flat-square)
+![Version: 0.32.0](https://img.shields.io/badge/Version-0.32.0-informational?style=flat-square) ![AppVersion: v0.32.0](https://img.shields.io/badge/AppVersion-v0.32.0-informational?style=flat-square)
 
 Feast Feature Server: Online feature serving service for Feast
 
@@ -17,7 +17,7 @@ Feast Feature Server: Online feature serving service for Feast
 | envOverrides | object | `{}` | Extra environment variables to set |
 | image.pullPolicy | string | `"IfNotPresent"` | Image pull policy |
 | image.repository | string | `"feastdev/feature-server-java"` | Docker image for Feature Server repository |
-| image.tag | string | `"0.31.0"` | Image tag |
+| image.tag | string | `"0.32.0"` | Image tag |
 | ingress.grpc.annotations | object | `{}` | Extra annotations for the ingress |
 | ingress.grpc.auth.enabled | bool | `false` | Flag to enable auth |
 | ingress.grpc.class | string | `"nginx"` | Which ingress controller to use |

infra/charts/feast/charts/feature-server/values.yaml

@@ -5,7 +5,7 @@ image:
   # image.repository -- Docker image for Feature Server repository
   repository: feastdev/feature-server-java
   # image.tag -- Image tag
-  tag: 0.31.0
+  tag: 0.32.0
   # image.pullPolicy -- Image pull policy
   pullPolicy: IfNotPresent
 

infra/charts/feast/charts/transformation-service/Chart.yaml

@@ -1,8 +1,8 @@
 apiVersion: v1
 description: "Transformation service: to compute on-demand features"
 name: transformation-service
-version: 0.31.0
-appVersion: v0.31.0
+version: 0.32.0
+appVersion: v0.32.0
 keywords:
 - machine learning
 - big data

infra/charts/feast/charts/transformation-service/README.md

@@ -1,6 +1,6 @@
 # transformation-service
 
-![Version: 0.31.0](https://img.shields.io/badge/Version-0.31.0-informational?style=flat-square) ![AppVersion: v0.31.0](https://img.shields.io/badge/AppVersion-v0.31.0-informational?style=flat-square)
+![Version: 0.32.0](https://img.shields.io/badge/Version-0.32.0-informational?style=flat-square) ![AppVersion: v0.32.0](https://img.shields.io/badge/AppVersion-v0.32.0-informational?style=flat-square)
 
 Transformation service: to compute on-demand features
 
@@ -13,7 +13,7 @@ Transformation service: to compute on-demand features
 | envOverrides | object | `{}` | Extra environment variables to set |
 | image.pullPolicy | string | `"IfNotPresent"` | Image pull policy |
 | image.repository | string | `"feastdev/feature-transformation-server"` | Docker image for Transformation Server repository |
-| image.tag | string | `"0.31.0"` | Image tag |
+| image.tag | string | `"0.32.0"` | Image tag |
 | nodeSelector | object | `{}` | Node labels for pod assignment |
 | podLabels | object | `{}` | Labels to be added to Feast Serving pods |
 | replicaCount | int | `1` | Number of pods that will be created |

infra/charts/feast/charts/transformation-service/values.yaml

@@ -5,7 +5,7 @@ image:
   # image.repository -- Docker image for Transformation Server repository
   repository: feastdev/feature-transformation-server
   # image.tag -- Image tag
-  tag: 0.31.0
+  tag: 0.32.0
   # image.pullPolicy -- Image pull policy
   pullPolicy: IfNotPresent
 

infra/charts/feast/requirements.yaml

@@ -1,12 +1,12 @@
 dependencies:
 - name: feature-server
   alias: feature-server
-  version: 0.31.0
+  version: 0.32.0
   condition: feature-server.enabled
   repository: https://feast-helm-charts.storage.googleapis.com
 - name: transformation-service
   alias: transformation-service
-  version: 0.31.0
+  version: 0.32.0
   condition: transformation-service.enabled
   repository: https://feast-helm-charts.storage.googleapis.com
 - name: redis

infra/scripts/cleanup_ci.py

@@ -0,0 +1,50 @@
+from time import sleep
+import boto3
+from tqdm import tqdm
+from google.cloud import bigtable
+from google.cloud.bigtable import enums
+
+
+def cleanup_dynamo_ci():
+    db = boto3.resource("dynamodb")
+
+    num_to_delete = 0
+    all_tables = db.tables.all()
+    for table in all_tables:
+        if "integration_test" in table.name:
+            num_to_delete += 1
+    with tqdm(total=num_to_delete) as progress:
+        for table in all_tables:
+            if "integration_test" in table.name:
+                table.delete()
+                progress.update()
+    print(f"Deleted {num_to_delete} CI DynamoDB tables")
+
+
+def cleanup_bigtable_ci():
+    client = bigtable.Client(project="kf-feast", admin=True)
+    instance = client.instance("feast-integration-tests")
+    if instance.exists():
+        print(f"Deleted Bigtable CI instance")
+        instance.delete()
+
+    location_id = "us-central1-f"
+    serve_nodes = 1
+    storage_type = enums.StorageType.SSD
+    cluster = instance.cluster(
+        "feast-integration-tests-c1",
+        location_id=location_id,
+        serve_nodes=serve_nodes,
+        default_storage_type=storage_type,
+    )
+    instance.create(clusters=[cluster])
+    print(f"Created new Bigtable CI tables")
+
+
+def main() -> None:
+    cleanup_dynamo_ci()
+    cleanup_bigtable_ci()
+
+
+if __name__ == "__main__":
+    main()

java/pom.xml

@@ -35,7 +35,7 @@
     </modules>
 
     <properties>
-        <revision>0.31.0</revision>
+        <revision>0.32.0</revision>
         <github.url>https://github.com/feast-dev/feast</github.url>
 
         <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>

java/serving/pom.xml

@@ -243,6 +243,12 @@
       <version>1.12.261</version>
     </dependency>
 
+    <dependency>
+      <groupId>com.amazonaws</groupId>
+      <artifactId>aws-java-sdk-sts</artifactId>
+      <version>1.12.476</version>
+    </dependency>
+
     <dependency>
       <groupId>com.adobe.testing</groupId>
       <artifactId>s3mock-testcontainers</artifactId>

sdk/python/feast/inference.py

@@ -1,5 +1,5 @@
 import re
-from typing import List, Set, Union
+from typing import List, Optional, Set, Union
 
 from feast.data_source import DataSource, PushSource, RequestSource
 from feast.entity import Entity
@@ -119,7 +119,10 @@ def update_feature_views_with_inferred_features_and_entities(
 
     for fv in fvs:
         join_keys = set(
-            [entity_name_to_join_key_map[entity_name] for entity_name in fv.entities]
+            [
+                entity_name_to_join_key_map.get(entity_name)
+                for entity_name in fv.entities
+            ]
         )
 
         # Fields whose names match a join key are considered to be entity columns; all
@@ -137,7 +140,10 @@ def update_feature_views_with_inferred_features_and_entities(
 
         # Respect the `value_type` attribute of the entity, if it is specified.
         for entity_name in fv.entities:
-            entity = entity_name_to_entity_map[entity_name]
+            entity = entity_name_to_entity_map.get(entity_name)
+            # pass when entity does not exist. Entityless feature view case
+            if entity is None:
+                continue
             if (
                 entity.join_key
                 not in [entity_column.name for entity_column in fv.entity_columns]
@@ -181,7 +187,7 @@ def update_feature_views_with_inferred_features_and_entities(
 
 def _infer_features_and_entities(
     fv: FeatureView,
-    join_keys: Set[str],
+    join_keys: Set[Optional[str]],
     run_inference_for_features,
     config,
 ) -> None:

sdk/python/feast/infra/offline_stores/bigquery.py

@@ -693,7 +693,7 @@ def _get_entity_schema(
 ) -> Dict[str, np.dtype]:
     if isinstance(entity_df, str):
         entity_df_sample = (
-            client.query(f"SELECT * FROM ({entity_df}) LIMIT 1").result().to_dataframe()
+            client.query(f"SELECT * FROM ({entity_df}) LIMIT 0").result().to_dataframe()
         )
 
         entity_schema = dict(zip(entity_df_sample.columns, entity_df_sample.dtypes))

sdk/python/feast/infra/offline_stores/redshift.py

@@ -369,7 +369,7 @@ def offline_write_batch(
             s3_resource=s3_resource,
             s3_path=f"{config.offline_store.s3_staging_location}/push/{uuid.uuid4()}.parquet",
             iam_role=config.offline_store.iam_role,
-            table_name=redshift_options.table,
+            table_name=redshift_options.fully_qualified_table_name,
             schema=pa_schema,
             fail_if_exists=False,
         )