Skip to content

Files

Latest commit

99bcd31 · Sep 27, 2022

History

History

retrieval

Image Retrieval

The image retrieval pipeline consists of several distinct steps:

ResNet50 inference to generate database and query set arrays

We utilize a native PyTorch inference to generate the database and query sets for image retrieval. An example for performing retrieval on ImageNet-4K with an MRL pretrained model (trained with PyTorch DDP) is provided below:

python ../inference/pytorch_inference.py --retrieval --path=path_to_model/final_weights.pt \
--retrieval_array_path='output_path/' --dataset=1K --mrl

This will generate train (database) and val (query) arrays for the vector representations and labels for ImageNet-1K, ie 1K_val_mrl1_e0_ff2048-X.npy, 1K_val_mrl1_e0_ff2048-y.npy, 1K_train_mrl1_e0_ff2048-X.npy, and 1K_train_mrl1_e0_ff2048-y.npy, which will be saved to disk in the output_path/ directory.

Database Index and Search

The arrays generated above are used to create an index file of the database, as shown in faiss_nn.ipynb. FAISS requires an index type (Exact L2, HNSW32) and a shortlist length k to search the indexed database for each query in the query set. Note that we utilize GPU search for exact search, which is currently unsupported for HNSW. The k -length shortlist for all desired representation sizes is saved to disk to be used for downstream adaptive retrieval or metric computation as, for example, neighbors/mrl/exactl2_16dim_2048shortlist_1K.csv.

Adaptive Retrieval

In an attempt to achieve equivalent performance to shortlisting at higher representation sizes with reduced MFLOPs, we perform adaptive retrieval by, for example, retrieving a shortlist k = 200 with rep. size D r = 16 followed by reranking with a higher capacity representation D s = 2048 on ImageNetV2 representations. The reranked k-NN shortlist is saved to disk as 8dim-reranked16_200shortlist_V2_exactl2.csv as shown in reranking.ipynb.

In an attempt to remove supervision in choosing D r and D s , we utilize Funnel Retrieval. Funnel retrieval thins out the initial shortlist by a repeated re-ranking and shortlisting with a series of increasing capacity representations. For example, retrieval with D r = 8 followed by a funnel with Rerank Cascade = [16, 32, 64, 128, 2048] and Shortlist Cascade = [200, 100, 50, 25, 10] would be saved as
8dim-cascade[16,32,64,128,2048]_shortlist[200,100,50,25,10]_1K_exactl2.csv

Metric Computation

We compute mAP@k, precision@k, recall@k, and top-k accuracy of the k-NN shortlist for various values of k , as shown in compute_metrics.ipynb, with flags for model configuration, dataset, index type, and retrieval configuration required. The script loads database labels 1K_train_mrl1_e0_ff2048-y.npy and query labels 1K_val_mrl1_e0_ff2048-y.npy alongside the k-NN shortlist generated via FAISS retrieval (neighbors/mrl/exactl2_16dim_2048shortlist_1K.csv as in the example above) or after reranking/funnel retrieval in the steps above.