Skip to content

Commit a6881dc

Browse files
Manisha4Manisha4
authored
feat: Adding Python classes for new Feature View type SortedFeatureView (#175)
* Adding python classes for new Feature View type SortedFeatureView * Adding python classes for new Feature View type SortedFeatureView * Adding python classes for new Feature View type SortedFeatureView * fixing formatting * Adding tests and fixing formatting * Adding go class * Adding relevant go class * Adding additional testing * getting rid of dummy entity constants * formatting file to fix linting errors * Adding in new changes to support the new feature view type sortedFeatureView * fixing circular dependency * fixing circular dependency * fixing circular dependency by fixing how things are imported * fixing circular dependency by fixing how things are imported * Fixing linting and missed additions of the SortedFeatureView * fixing formatting issues * fixing formatting issues * fixing failing tests * adding SortedFeatureView to the permission proto * Fixing linting errors * Fixing linting errors * Fixing linting errors * Fixing linting errors * Fixing linting errors * Adding in more changes * Adding changes to the workflow to see if it fixes the build failures * fixing linting issues * Committing PR changes * fixing failing tests * removing setup tools from GHA unit_tests workflow * Modifying go class and adding back build addition to workflow * removing cast * changing value type field for SortedFeatureViewSortKeyModel * changing value type field for SortedFeatureViewSortKeyModel * modified ensure_valid method in SortedFeatureView * Added more changes based on PR comments
1 parent df265fc commit a6881dc

28 files changed

+1557
-55
lines changed

‎.github/workflows/unit_tests.yml

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -80,6 +80,8 @@ jobs:
8080
- name: Upgrade pip version
8181
run: |
8282
pip install --upgrade "pip>=22.1,<23"
83+
- name: Upgrade setuptools and wheel
84+
run: pip install --upgrade setuptools wheel
8385
- name: Test
8486
run: make test-go
8587

‎docs/reference/registry/registry-permissions.md

Lines changed: 40 additions & 38 deletions
Original file line numberDiff line numberDiff line change
@@ -3,43 +3,45 @@
33

44
## API Endpoints and Permissions
55

6-
| Endpoint | Resource Type | Permission | Description |
7-
| ------------------------ |---------------------|------------------------| -------------------------------------------------------------- |
8-
| ApplyEntity | Entity | Create, Update, Delete | Apply an entity to the registry |
9-
| GetEntity | Entity | Read | Get an entity from the registry |
10-
| ListEntities | Entity | Read | List entities in the registry |
11-
| DeleteEntity | Entity | Delete | Delete an entity from the registry |
12-
| ApplyDataSource | DataSource | Create, Update, Delete | Apply a data source to the registry |
13-
| GetDataSource | DataSource | Read | Get a data source from the registry |
14-
| ListDataSources | DataSource | Read | List data sources in the registry |
15-
| DeleteDataSource | DataSource | Delete | Delete a data source from the registry |
16-
| ApplyFeatureView | FeatureView | Create, Update, Delete | Apply a feature view to the registry |
17-
| GetFeatureView | FeatureView | Read | Get a feature view from the registry |
18-
| ListFeatureViews | FeatureView | Read | List feature views in the registry |
19-
| DeleteFeatureView | FeatureView | Delete | Delete a feature view from the registry |
20-
| GetStreamFeatureView | StreamFeatureView | Read | Get a stream feature view from the registry |
21-
| ListStreamFeatureViews | StreamFeatureView | Read | List stream feature views in the registry |
22-
| GetOnDemandFeatureView | OnDemandFeatureView | Read | Get an on-demand feature view from the registry |
23-
| ListOnDemandFeatureViews | OnDemandFeatureView | Read | List on-demand feature views in the registry |
24-
| ApplyFeatureService | FeatureService | Create, Update, Delete | Apply a feature service to the registry |
25-
| GetFeatureService | FeatureService | Read | Get a feature service from the registry |
26-
| ListFeatureServices | FeatureService | Read | List feature services in the registry |
27-
| DeleteFeatureService | FeatureService | Delete | Delete a feature service from the registry |
28-
| ApplySavedDataset | SavedDataset | Create, Update, Delete | Apply a saved dataset to the registry |
29-
| GetSavedDataset | SavedDataset | Read | Get a saved dataset from the registry |
30-
| ListSavedDatasets | SavedDataset | Read | List saved datasets in the registry |
31-
| DeleteSavedDataset | SavedDataset | Delete | Delete a saved dataset from the registry |
32-
| ApplyValidationReference | ValidationReference | Create, Update, Delete | Apply a validation reference to the registry |
33-
| GetValidationReference | ValidationReference | Read | Get a validation reference from the registry |
34-
| ListValidationReferences | ValidationReference | Read | List validation references in the registry |
35-
| DeleteValidationReference| ValidationReference | Delete | Delete a validation reference from the registry |
36-
| ApplyPermission | Permission | Create, Update, Delete | Apply a permission to the registry |
37-
| GetPermission | Permission | Read | Get a permission from the registry |
38-
| ListPermissions | Permission | Read | List permissions in the registry |
39-
| DeletePermission | Permission | Delete | Delete a permission from the registry |
40-
| Commit | | None | Commit changes to the registry |
41-
| Refresh | | None | Refresh the registry |
42-
| Proto | | None | Get the proto representation of the registry |
6+
| Endpoint | Resource Type | Permission | Description |
7+
|---------------------------|---------------------|------------------------|-------------------------------------------------|
8+
| ApplyEntity | Entity | Create, Update, Delete | Apply an entity to the registry |
9+
| GetEntity | Entity | Read | Get an entity from the registry |
10+
| ListEntities | Entity | Read | List entities in the registry |
11+
| DeleteEntity | Entity | Delete | Delete an entity from the registry |
12+
| ApplyDataSource | DataSource | Create, Update, Delete | Apply a data source to the registry |
13+
| GetDataSource | DataSource | Read | Get a data source from the registry |
14+
| ListDataSources | DataSource | Read | List data sources in the registry |
15+
| DeleteDataSource | DataSource | Delete | Delete a data source from the registry |
16+
| ApplyFeatureView | FeatureView | Create, Update, Delete | Apply a feature view to the registry |
17+
| GetFeatureView | FeatureView | Read | Get a feature view from the registry |
18+
| ListFeatureViews | FeatureView | Read | List feature views in the registry |
19+
| DeleteFeatureView | FeatureView | Delete | Delete a feature view from the registry |
20+
| GetSortedFeatureView | SortedFeatureView | Read | Get a sorted feature view from the registry |
21+
| ListSortedFeatureViews | SortedFeatureView | Read | List sorted feature views in the registry |
22+
| GetStreamFeatureView | StreamFeatureView | Read | Get a stream feature view from the registry |
23+
| ListStreamFeatureViews | StreamFeatureView | Read | List stream feature views in the registry |
24+
| GetOnDemandFeatureView | OnDemandFeatureView | Read | Get an on-demand feature view from the registry |
25+
| ListOnDemandFeatureViews | OnDemandFeatureView | Read | List on-demand feature views in the registry |
26+
| ApplyFeatureService | FeatureService | Create, Update, Delete | Apply a feature service to the registry |
27+
| GetFeatureService | FeatureService | Read | Get a feature service from the registry |
28+
| ListFeatureServices | FeatureService | Read | List feature services in the registry |
29+
| DeleteFeatureService | FeatureService | Delete | Delete a feature service from the registry |
30+
| ApplySavedDataset | SavedDataset | Create, Update, Delete | Apply a saved dataset to the registry |
31+
| GetSavedDataset | SavedDataset | Read | Get a saved dataset from the registry |
32+
| ListSavedDatasets | SavedDataset | Read | List saved datasets in the registry |
33+
| DeleteSavedDataset | SavedDataset | Delete | Delete a saved dataset from the registry |
34+
| ApplyValidationReference | ValidationReference | Create, Update, Delete | Apply a validation reference to the registry |
35+
| GetValidationReference | ValidationReference | Read | Get a validation reference from the registry |
36+
| ListValidationReferences | ValidationReference | Read | List validation references in the registry |
37+
| DeleteValidationReference | ValidationReference | Delete | Delete a validation reference from the registry |
38+
| ApplyPermission | Permission | Create, Update, Delete | Apply a permission to the registry |
39+
| GetPermission | Permission | Read | Get a permission from the registry |
40+
| ListPermissions | Permission | Read | List permissions in the registry |
41+
| DeletePermission | Permission | Delete | Delete a permission from the registry |
42+
| Commit | | None | Commit changes to the registry |
43+
| Refresh | | None | Refresh the registry |
44+
| Proto | | None | Get the proto representation of the registry |
4345

4446
## How to configure Authentication and Authorization
45-
Please refer the [page](./../../../docs/getting-started/concepts/permission.md) for more details on how to configure authentication and authorization.
47+
Please refer the [page](./../../../docs/getting-started/concepts/permission.md) for more details on how to configure authentication and authorization.

‎go/internal/feast/model/sortedfeatureview.go

Lines changed: 76 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,76 @@
1+
package model
2+
3+
import (
4+
"github.com/feast-dev/feast/go/protos/feast/core"
5+
"github.com/feast-dev/feast/go/protos/feast/types"
6+
)
7+
8+
type SortOrder struct {
9+
Order core.SortOrder_Enum
10+
}
11+
12+
func NewSortOrderFromProto(order core.SortOrder_Enum) *SortOrder {
13+
return &SortOrder{
14+
Order: order,
15+
}
16+
}
17+
18+
func (so *SortOrder) String() string {
19+
switch so.Order {
20+
case core.SortOrder_ASC:
21+
return "ASC"
22+
case core.SortOrder_DESC:
23+
return "DESC"
24+
default:
25+
return "INVALID"
26+
}
27+
}
28+
29+
func (so *SortOrder) ToProto() core.SortOrder_Enum {
30+
return so.Order
31+
}
32+
33+
type SortKey struct {
34+
FieldName string
35+
Order *SortOrder
36+
ValueType types.ValueType_Enum
37+
}
38+
39+
func NewSortKeyFromProto(proto *core.SortKey) *SortKey {
40+
return &SortKey{
41+
FieldName: proto.GetName(),
42+
Order: NewSortOrderFromProto(proto.GetDefaultSortOrder()),
43+
ValueType: proto.GetValueType(),
44+
}
45+
}
46+
47+
type SortedFeatureView struct {
48+
*FeatureView
49+
SortKeys []*SortKey
50+
}
51+
52+
func NewSortedFeatureViewFromProto(proto *core.SortedFeatureView) *SortedFeatureView {
53+
baseFV := &FeatureView{
54+
Base: NewBaseFeatureView(proto.GetSpec().GetName(), proto.GetSpec().GetFeatures()),
55+
Ttl: proto.GetSpec().GetTtl(),
56+
}
57+
58+
// Convert each sort key from the proto.
59+
sortKeys := make([]*SortKey, len(proto.GetSpec().GetSortKeys()))
60+
for i, skProto := range proto.GetSpec().GetSortKeys() {
61+
sortKeys[i] = NewSortKeyFromProto(skProto)
62+
}
63+
64+
return &SortedFeatureView{
65+
FeatureView: baseFV,
66+
SortKeys: sortKeys,
67+
}
68+
}
69+
70+
func (sfv *SortedFeatureView) NewSortedFeatureViewFromBase(base *BaseFeatureView) *SortedFeatureView {
71+
newFV := sfv.FeatureView.NewFeatureViewFromBase(base)
72+
return &SortedFeatureView{
73+
FeatureView: newFV,
74+
SortKeys: sfv.SortKeys,
75+
}
76+
}

‎protos/feast/core/Permission.proto

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -39,6 +39,7 @@ message PermissionSpec {
3939
ON_DEMAND_FEATURE_VIEW = 1;
4040
BATCH_FEATURE_VIEW = 2;
4141
STREAM_FEATURE_VIEW= 3;
42+
SORTED_FEATURE_VIEW= 15;
4243
ENTITY = 4;
4344
FEATURE_SERVICE = 5;
4445
DATA_SOURCE = 6;
@@ -67,4 +68,4 @@ message PermissionSpec {
6768
message PermissionMeta {
6869
google.protobuf.Timestamp created_timestamp = 1;
6970
google.protobuf.Timestamp last_updated_timestamp = 2;
70-
}
71+
}

‎sdk/python/feast/__init__.py

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,7 @@
2323
from .on_demand_feature_view import OnDemandFeatureView
2424
from .project import Project
2525
from .repo_config import RepoConfig
26+
from .sorted_feature_view import SortedFeatureView
2627
from .stream_feature_view import StreamFeatureView
2728
from .value_type import ValueType
2829

@@ -42,6 +43,7 @@
4243
"FeatureService",
4344
"FeatureStore",
4445
"FeatureView",
46+
"SortedFeatureView",
4547
"OnDemandFeatureView",
4648
"RepoConfig",
4749
"StreamFeatureView",

‎sdk/python/feast/base_feature_view.py

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,9 @@
2424
from feast.protos.feast.core.OnDemandFeatureView_pb2 import (
2525
OnDemandFeatureView as OnDemandFeatureViewProto,
2626
)
27+
from feast.protos.feast.core.SortedFeatureView_pb2 import (
28+
SortedFeatureView as SortedFeatureViewProto,
29+
)
2730
from feast.protos.feast.core.StreamFeatureView_pb2 import (
2831
StreamFeatureView as StreamFeatureViewProto,
2932
)
@@ -98,7 +101,12 @@ def proto_class(self) -> Type[Message]:
98101
@abstractmethod
99102
def to_proto(
100103
self,
101-
) -> Union[FeatureViewProto, OnDemandFeatureViewProto, StreamFeatureViewProto]:
104+
) -> Union[
105+
FeatureViewProto,
106+
OnDemandFeatureViewProto,
107+
StreamFeatureViewProto,
108+
SortedFeatureViewProto,
109+
]:
102110
pass
103111

104112
@classmethod

‎sdk/python/feast/diff/registry_diff.py

Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,9 @@
2424
from feast.protos.feast.core.OnDemandFeatureView_pb2 import OnDemandFeatureViewSpec
2525
from feast.protos.feast.core.Permission_pb2 import Permission as PermissionProto
2626
from feast.protos.feast.core.SavedDataset_pb2 import SavedDataset as SavedDatasetProto
27+
from feast.protos.feast.core.SortedFeatureView_pb2 import (
28+
SortedFeatureView as SortedFeatureViewProto,
29+
)
2730
from feast.protos.feast.core.StreamFeatureView_pb2 import (
2831
StreamFeatureView as StreamFeatureViewProto,
2932
)
@@ -110,6 +113,7 @@ def tag_objects_for_keep_delete_update_add(
110113
EntityProto,
111114
FeatureViewProto,
112115
FeatureServiceProto,
116+
SortedFeatureViewProto,
113117
OnDemandFeatureViewProto,
114118
StreamFeatureViewProto,
115119
ValidationReferenceProto,
@@ -180,6 +184,61 @@ def diff_registry_objects(
180184
getattr(new_udf, _udf_field.name),
181185
)
182186
)
187+
elif _field.name == "sort_keys":
188+
# Special handling for sort_keys field (a repeated field)
189+
current_sort_keys = getattr(current_spec, _field.name)
190+
new_sort_keys = getattr(new_spec, _field.name)
191+
if len(current_sort_keys) != len(new_sort_keys):
192+
transition = TransitionType.UPDATE
193+
property_diffs.append(
194+
PropertyDiff(_field.name, current_sort_keys, new_sort_keys)
195+
)
196+
else:
197+
# Iterate through each sort key pair
198+
for idx, (cs, ns) in enumerate(
199+
zip(current_sort_keys, new_sort_keys)
200+
):
201+
if cs.name != ns.name:
202+
transition = TransitionType.UPDATE
203+
property_diffs.append(
204+
PropertyDiff(
205+
f"sort_keys[{idx}].name", cs.name, ns.name
206+
)
207+
)
208+
if cs.value_type != ns.value_type:
209+
transition = TransitionType.UPDATE
210+
property_diffs.append(
211+
PropertyDiff(
212+
f"sort_keys[{idx}].value_type",
213+
cs.value_type,
214+
ns.value_type,
215+
)
216+
)
217+
if cs.default_sort_order != ns.default_sort_order:
218+
transition = TransitionType.UPDATE
219+
property_diffs.append(
220+
PropertyDiff(
221+
f"sort_keys[{idx}].default_sort_order",
222+
cs.default_sort_order,
223+
ns.default_sort_order,
224+
)
225+
)
226+
if cs.tags != ns.tags:
227+
transition = TransitionType.UPDATE
228+
property_diffs.append(
229+
PropertyDiff(
230+
f"sort_keys[{idx}].tags", cs.tags, ns.tags
231+
)
232+
)
233+
if cs.description != ns.description:
234+
transition = TransitionType.UPDATE
235+
property_diffs.append(
236+
PropertyDiff(
237+
f"sort_keys[{idx}].description",
238+
cs.description,
239+
ns.description,
240+
)
241+
)
183242
else:
184243
transition = TransitionType.UPDATE
185244
property_diffs.append(
@@ -342,6 +401,7 @@ def apply_diff_to_registry(
342401
FeastObjectType.FEATURE_VIEW,
343402
FeastObjectType.ON_DEMAND_FEATURE_VIEW,
344403
FeastObjectType.STREAM_FEATURE_VIEW,
404+
FeastObjectType.SORTED_FEATURE_VIEW,
345405
]:
346406
feature_view_obj = cast(
347407
BaseFeatureView, feast_object_diff.current_feast_object

0 commit comments

Comments
 (0)