This repository has been archived by the owner on Jan 5, 2024. It is now read-only.
-
Notifications
You must be signed in to change notification settings - Fork 20
add visual search #51
Open
shashikg
wants to merge
9
commits into
brain-score:master
Choose a base branch
from
shashikg:master
base: master
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
9 commits
Select commit
Hold shift + click to select a range
cabd80d
added option to provide input size to base models
shashikg 4b382c3
activation layers for visual search model
shashikg de2ddd3
add test_search
shashikg ba40323
example for visual search benchmark
shashikg 4f6f270
changes to gitignore
shashikg 6edada7
added visual search models inside ml_pool
shashikg f9e9438
Merge branch 'master' of https://github.com/brain-score/candidate_mod…
shashikg 3cdd50f
Merge branch 'brain-score-master'
shashikg 76ef12e
visual search - waldo and natural design
shashikg File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,2 +1,7 @@ | ||
| *.pyc | ||
| *.DS_Store | ||
| __pycache__/ | ||
| .ipynb_checkpoints | ||
| build/* | ||
| dist/* | ||
| candidate_models.egg-info/* |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,9 @@ | ||
| import itertools | ||
|
|
||
| from brainscore.submission.ml_pool import ModelLayers | ||
|
|
||
| layers = { | ||
| 'vgg-16': [f'block{i + 1}_pool' for i in range(3,5)], | ||
| } | ||
|
|
||
| visual_search_layer = ModelLayers(layers) | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -349,4 +349,4 @@ | |
| }, | ||
| "nbformat": 4, | ||
| "nbformat_minor": 1 | ||
| } | ||
| } | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,122 @@ | ||
| { | ||
| "cells": [ | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": 1, | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "import warnings\n", | ||
| "warnings.simplefilter(\"ignore\")\n", | ||
| "\n", | ||
| "try:\n", | ||
| " from tensorflow.python.util import module_wrapper as deprecation\n", | ||
| "except ImportError:\n", | ||
| " from tensorflow.python.util import deprecation_wrapper as deprecation\n", | ||
| "deprecation._PER_MODULE_WARNING_LIMIT = 0" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": 2, | ||
| "metadata": {}, | ||
| "outputs": [ | ||
| { | ||
| "name": "stdout", | ||
| "output_type": "stream", | ||
| "text": [ | ||
| "WARNING:tensorflow:\n", | ||
| "The TensorFlow contrib module will not be included in TensorFlow 2.0.\n", | ||
| "For more information, please see:\n", | ||
| " * https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md\n", | ||
| " * https://github.com/tensorflow/addons\n", | ||
| " * https://github.com/tensorflow/io (for I/O related ops)\n", | ||
| "If you depend on functionality not listed there, please file an issue.\n", | ||
| "\n" | ||
| ] | ||
| } | ||
| ], | ||
| "source": [ | ||
| "from candidate_models import score_model\n", | ||
| "from candidate_models.model_commitments import brain_translated_pool\n", | ||
| "import numpy as np" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": 3, | ||
| "metadata": {}, | ||
| "outputs": [ | ||
| { | ||
| "name": "stderr", | ||
| "output_type": "stream", | ||
| "text": [ | ||
| "Using TensorFlow backend.\n", | ||
| "activations: 100%|██████████| 320/320 [00:01<00:00, 231.07it/s]\n", | ||
| "layer packaging: 100%|██████████| 1/1 [00:00<00:00, 110.52it/s]\n", | ||
| "activations: 100%|██████████| 320/320 [01:01<00:00, 5.20it/s]\n", | ||
| "layer packaging: 100%|██████████| 1/1 [00:00<00:00, 2.07it/s]\n", | ||
| "visual search stimuli: 100%|██████████| 300/300 [01:02<00:00, 4.78it/s]\n", | ||
| "comparing with human data: 100%|██████████| 15/15 [00:06<00:00, 2.49it/s]" | ||
| ] | ||
| }, | ||
| { | ||
| "name": "stdout", | ||
| "output_type": "stream", | ||
| "text": [ | ||
| "<xarray.Score (aggregation: 2)>\n", | ||
| "array([0.923438, 0.005427])\n", | ||
| "Coordinates:\n", | ||
| " * aggregation (aggregation) <U6 'center' 'error'\n", | ||
| "Attributes:\n", | ||
| " raw: <xarray.Score (aggregation: 2)>\\narray([0.407328, ...\n", | ||
| " ceiling: <xarray.Score (aggregation: 2)>\\narray([0.4411, ...\n", | ||
| " model_identifier: vgg-16\n", | ||
| " benchmark_identifier: klab.Zhang2018-object_search\n" | ||
| ] | ||
| }, | ||
| { | ||
| "name": "stderr", | ||
| "output_type": "stream", | ||
| "text": [ | ||
| "\n" | ||
| ] | ||
| } | ||
| ], | ||
| "source": [ | ||
| "identifier = 'vgg-16'\n", | ||
| "model = brain_translated_pool[identifier]\n", | ||
| "score = score_model(model_identifier=identifier, model=model, benchmark_identifier='klab.Zhang2018-object_search')\n", | ||
| "print(score)" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [] | ||
| } | ||
| ], | ||
| "metadata": { | ||
| "kernelspec": { | ||
| "display_name": "Python 3", | ||
| "language": "python", | ||
| "name": "python3" | ||
| }, | ||
| "language_info": { | ||
| "codemirror_mode": { | ||
| "name": "ipython", | ||
| "version": 3 | ||
| }, | ||
| "file_extension": ".py", | ||
| "mimetype": "text/x-python", | ||
| "name": "python", | ||
| "nbconvert_exporter": "python", | ||
| "pygments_lexer": "ipython3", | ||
| "version": "3.7.6" | ||
| } | ||
| }, | ||
| "nbformat": 4, | ||
| "nbformat_minor": 2 | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,13 @@ | ||
| import pytest | ||
| from pytest import approx | ||
|
|
||
| from candidate_models.model_commitments import brain_translated_pool | ||
| from brainscore.benchmarks import benchmark_pool | ||
| import numpy as np | ||
| import matplotlib.pyplot as plt | ||
| import brainscore | ||
|
|
||
| def test_search(): | ||
| model = brain_translated_pool['vgg-16'] | ||
| score = score_model(model_identifier='vgg-16', model=model, benchmark_identifier='klab.Zhang2018-object_search') | ||
| assert score.raw.sel(aggregation='center') == approx(0.407328, abs=.005) |
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I would create another vgg-16 BrainModel from the same underlying BaseModel that makes these exact commitments
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The issue is that the "target_image" size is just 28x28 pixels. So we will need to exclusively define which layer will work for a specific ML model. So isn't it much better that we commit these layers in a separate py file? i.e. which layer to use for which ML model?