Store object metadata in metabases for SearchV2 service #3080

cthulhu-rider · 2025-01-13T16:41:22Z

pkg/local_object_storage/metabase/util.go

pkg/local_object_storage/metabase/metadata.go

cthulhu-rider · 2025-02-05T14:17:38Z

it seems working except case when primary filtered attribute is not the 1st requested one. Querying the primary attribute is almost always needed, e.g. with NE/PREFIX/NUM filters. It's only redundant with EQ, but for now a primary attribute request can be required. In the future, if the requirement is relaxed, the client's behavior will not be broken and he will be able to slightly optimize the search query

i also dont plan to support migration and GC in this PR so to not block testing with fresh storages

i need some time to review TODOs, fix linters and beautify the code, but it is rdy overall

codecov · 2025-02-06T11:06:20Z

Codecov Report

Attention: Patch coverage is 71.83099% with 180 lines in your changes missing coverage. Please review.

Project coverage is 22.92%. Comparing base (5279df2) to head (9e2e544).
Report is 19 commits behind head on master.

Files with missing lines	Patch %	Lines
pkg/local_object_storage/metabase/metadata.go	72.36%	115 Missing and 53 partials ⚠️
pkg/local_object_storage/metabase/put.go	55.00%	6 Missing and 3 partials ⚠️
pkg/local_object_storage/metabase/containers.go	0.00%	2 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #3080      +/-   ##
==========================================
+ Coverage   22.47%   22.92%   +0.45%     
==========================================
  Files         751      750       -1     
  Lines       57804    58710     +906     
==========================================
+ Hits        12991    13462     +471     
- Misses      43929    44308     +379     
- Partials      884      940      +56

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

pkg/local_object_storage/metabase/metadata.go

roman-khimov · 2025-02-07T15:28:38Z

pkg/local_object_storage/metabase/metadata.go

+			return nil, "", fmt.Errorf("empty attribute #%d", i)
+		}
+		if attrs[i] == object.FilterContainerID || attrs[i] == object.FilterID {
+			return nil, "", fmt.Errorf("prohibited attribute %s", attrs[i])


These checks (including the ones above of course) need to be separated out, they're about semantic request validation, so the Search handler should perform them first and if successful run searches everywhere (including local) that could fail only for some internal (not related to request itself) reasons.

pkg/local_object_storage/metabase/metadata.go

roman-khimov · 2025-02-07T16:52:18Z

pkg/local_object_storage/metabase/metadata.go

+		primSeekPrefix = primSeekKey[:1+len(primAttr)+len(utf8Delimiter)]
+		valID := primSeekKey[len(primSeekPrefix):]
+		if len(valID) <= oid.Size {
+			return nil, nil, fmt.Errorf("%w: too small VAL_OID len %d", errInvalidCursor, len(valID))


Also a part of the initial search parameter check.

possible with #3080 (comment). Done

pkg/local_object_storage/metabase/metadata.go

carpawell · 2025-02-10T07:22:35Z

pkg/local_object_storage/metabase/metadata.go

+
+// Search selects up to count container's objects from the given container
+// matching the specified filters.
+func (db *DB) Search(cnr cid.ID, fs object.SearchFilters, attrs []string, cursor string, count uint16) ([]client.SearchResultItem, string, error) {


should meta package use SDK's client package?

SDK is a basest in the node

sdk in general -- agree, but the client package -- not sure

pkg/local_object_storage/metabase/metadata.go

roman-khimov · 2025-02-10T12:17:59Z

pkg/local_object_storage/metabase/metadata.go

+			primMatcher == object.MatchNumGT || primMatcher == object.MatchNumGE {
+			var err error
+			if primSeekKey, primSeekPrefix, err = seekKeyForAttribute(primAttr, fs[0].Value()); err != nil {
+				return nil, nil, fmt.Errorf("invalid primary filter value: %w", err)


Shouldn't this be checked before going into searchInBucket?

can be, dont think this is sufficient for now. I already have TODO bout this

pkg/local_object_storage/metabase/metadata.go

There is a need to serve `ObjectService.SearchV2` RPC by the SN. In order not to expand the structure and configuration of the node, the best place to store metadata is metabase. Metabases are extended with per-container object metadata buckets. For each object, following indexes are created: - OID; - attribute->OID; - OID->attribute. Integers are stored specifically to reach lexicographic comparisons without decoding. New `Search` method is provided: it allows to filter out container's objects and receive specified attributes. Count is also limited, op is paged via cursor. In other words, the method follows SearchV2 behavior within single metabase. Refs #3058. Signed-off-by: Leonard Lyubich <[email protected]>

carpawell · 2025-02-11T08:20:13Z

pkg/local_object_storage/metabase/metadata.go

+	if s == "" {
+		return nil, nil
+	}
+	b := make([]byte, 1+base64.StdEncoding.DecodedLen(len(s)))


isnt DecodeString about the same but shorter?

DecodeString creates new buffer on his own, this isnt desired here

carpawell · 2025-02-11T08:39:05Z

pkg/local_object_storage/metabase/metadata.go

+
+// Search selects up to count container's objects from the given container
+// matching the specified filters.
+func (db *DB) Search(cnr cid.ID, fs object.SearchFilters, attrs []string, cursor string, count uint16) ([]client.SearchResultItem, string, error) {


sdk in general -- agree, but the client package -- not sure

cthulhu-rider force-pushed the object-searchv2/metabases branch 4 times, most recently from 9624b00 to 3540889 Compare January 14, 2025 13:17

roman-khimov reviewed Jan 14, 2025

View reviewed changes

pkg/local_object_storage/metabase/util.go Outdated Show resolved Hide resolved

cthulhu-rider force-pushed the object-searchv2/metabases branch 2 times, most recently from f16c1c1 to 8957af6 Compare January 16, 2025 15:38

roman-khimov reviewed Jan 16, 2025

View reviewed changes

cthulhu-rider force-pushed the object-searchv2/metabases branch 3 times, most recently from c29b7e5 to 00c7b83 Compare January 20, 2025 13:56

carpawell mentioned this pull request Jan 20, 2025

Handle meta events #3070

Closed

cthulhu-rider force-pushed the object-searchv2/metabases branch 5 times, most recently from 22664c2 to 4bb5abf Compare January 24, 2025 11:30

cthulhu-rider force-pushed the object-searchv2/metabases branch 3 times, most recently from c2c593b to 0904251 Compare February 5, 2025 14:10

cthulhu-rider force-pushed the object-searchv2/metabases branch 3 times, most recently from eb9036f to 871972a Compare February 6, 2025 11:03

cthulhu-rider force-pushed the object-searchv2/metabases branch 3 times, most recently from 8dffd53 to 5d513b6 Compare February 6, 2025 11:50

cthulhu-rider mentioned this pull request Feb 6, 2025

Implement searchv2 #3058

Open

cthulhu-rider marked this pull request as ready for review February 6, 2025 11:55

cthulhu-rider requested a review from carpawell as a code owner February 6, 2025 11:55

cthulhu-rider requested a review from roman-khimov February 6, 2025 11:56

cthulhu-rider mentioned this pull request Feb 7, 2025

Serve ObjectService.SearchV2 RPC #3111

Merged

cthulhu-rider force-pushed the object-searchv2/metabases branch from 5d513b6 to 400c685 Compare February 7, 2025 10:49

roman-khimov reviewed Feb 7, 2025

View reviewed changes

carpawell reviewed Feb 10, 2025

View reviewed changes

cthulhu-rider mentioned this pull request Feb 10, 2025

Consider lib usage for metadata integer encoding #3115

Open

cthulhu-rider force-pushed the object-searchv2/metabases branch 5 times, most recently from 8085434 to c59f430 Compare February 10, 2025 11:21

roman-khimov reviewed Feb 10, 2025

View reviewed changes

cthulhu-rider force-pushed the object-searchv2/metabases branch from c59f430 to 9e2e544 Compare February 10, 2025 14:16

roman-khimov approved these changes Feb 10, 2025

View reviewed changes

cthulhu-rider requested a review from carpawell February 11, 2025 06:26

This was referenced Feb 11, 2025

Support ROOT and PHY filters in SearchV2 #3116

Closed

Fill SearchV2 metadata on metabase migration #3117

Closed

Clean up SearchV2 metadata on object removals #3118

Closed

carpawell approved these changes Feb 11, 2025

View reviewed changes

cthulhu-rider merged commit 64dac4c into master Feb 11, 2025
21 of 22 checks passed

cthulhu-rider deleted the object-searchv2/metabases branch February 11, 2025 08:48

cthulhu-rider mentioned this pull request Feb 13, 2025

Store integer ID->Attribute indexes in text format inside metadata bucket #3130

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Store object metadata in metabases for SearchV2 service #3080

Store object metadata in metabases for SearchV2 service #3080

cthulhu-rider commented Jan 13, 2025 •

edited

Loading

cthulhu-rider commented Feb 5, 2025

codecov bot commented Feb 6, 2025 •

edited

Loading

roman-khimov Feb 7, 2025

cthulhu-rider Feb 10, 2025

roman-khimov Feb 7, 2025

cthulhu-rider Feb 10, 2025 •

edited

Loading

carpawell Feb 10, 2025

cthulhu-rider Feb 10, 2025

carpawell Feb 11, 2025

roman-khimov Feb 10, 2025

cthulhu-rider Feb 10, 2025 •

edited

Loading

carpawell Feb 11, 2025

cthulhu-rider Feb 11, 2025

carpawell Feb 11, 2025

Store object metadata in metabases for SearchV2 service #3080

Store object metadata in metabases for SearchV2 service #3080

Conversation

cthulhu-rider commented Jan 13, 2025 • edited Loading

cthulhu-rider commented Feb 5, 2025

codecov bot commented Feb 6, 2025 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cthulhu-rider Feb 10, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cthulhu-rider Feb 10, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cthulhu-rider commented Jan 13, 2025 •

edited

Loading

codecov bot commented Feb 6, 2025 •

edited

Loading

cthulhu-rider Feb 10, 2025 •

edited

Loading

cthulhu-rider Feb 10, 2025 •

edited

Loading