Add MongoDB indices for analysis fields #2079
Conversation
This massively improves load time when accessing an individual analysis result in CAPEv2 on large MongoDB instances.
| "procmemory.file_ref", | ||
| ] | ||
| for item in items: | ||
| mongo_create_index( |
There was a problem hiding this comment.
This will take a very long time for a lot of CAPE installations - terabyte scale is common. Probably better to do this in the utils somewhere and emit a warning if the indices aren't found.
There was a problem hiding this comment.
Where exactly do you suggest we should put this?
There was a problem hiding this comment.
I'd suggest as a new module in utils, mongodb_indices or something appropriately named.
In the startup module you can check the indices are there and emit a warning if they're missing.
Note too, the difference between doing it in startup and utils. Putting it in utils means we can add indices out-of-band, while CAPE continutes to run. Great way of making things faster incrementally without bringing CAPE down by touching startup.
|
this is obsolete now, i have added indexes for md5 and sha256 + rewrote the searching |
3 participants
This PR adds
keywordindices for certain fields in theanalysiscollection. This massively improves load time when accessing an individual analysis result in CAPE on large MongoDB instances.CAPE uses one large OR-query when loading an analysis result which is why we opted for a separate index for every field on which the aggregation is run. The query also looks for exact matches so the fields are indexed as
keywordinstead oftext.