Skip to content

Commit ed98d93

Browse files
authored
feat: Add GroundingDINO backend (#350)
* Add GroundingDINO backend * Clean comments * Feature/shondle sam with dino (#352) * Add baseline SAM code * Add TO DOs * Add SAM integration fixes * Add regular SAM * Add SAM capability and instructions * Update README.md * Remove whitespace * Remove whitespace * Adding env variables and code cleaning * Fix environment variable issues * Change truncation of ID * Remove exec
1 parent 0d36cc1 commit ed98d93

File tree

6 files changed

+560
-0
lines changed

6 files changed

+560
-0
lines changed
Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
FROM python:3.11-slim
2+
3+
WORKDIR /app
4+
5+
RUN apt-get -y update \
6+
&& apt-get install -y git \
7+
&& apt-get install -y wget \
8+
&& apt-get -y install ffmpeg libsm6 libxext6 libffi-dev python3-dev gcc
9+
10+
11+
ENV PYTHONUNBUFFERED=True \
12+
PORT=9090 \
13+
WORKERS=2 \
14+
THREADS=4
15+
16+
COPY requirements.txt .
17+
RUN pip install --no-cache-dir -r requirements.txt
18+
19+
RUN git clone https://github.com/IDEA-Research/GroundingDINO.git
20+
WORKDIR /app/GroundingDINO
21+
RUN pip -q install -e .
22+
RUN mkdir weights
23+
WORKDIR /app/GroundingDINO/weights
24+
RUN wget -q https://github.com/IDEA-Research/GroundingDINO/releases/download/v0.1.0-alpha/groundingdino_swint_ogc.pth
25+
26+
WORKDIR /app
27+
RUN wget -q https://github.com/ChaoningZhang/MobileSAM/raw/master/weights/mobile_sam.pt
28+
RUN wget -q https://dl.fbaipublicfiles.com/segment_anything/sam_vit_h_4b8939.pth
29+
30+
31+
COPY . ./
32+
33+
34+
CMD gunicorn --preload --bind :$PORT --workers $WORKERS --threads $THREADS --timeout 0 _wsgi:app
Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,59 @@
1+
https://github.com/HumanSignal/label-studio-ml-backend/assets/106922533/d1d2f233-d7c0-40ac-ba6f-368c3c01fd36
2+
3+
4+
5+
## GroundingDINO Backend Integration
6+
7+
Use text prompts for zero-shot detection of objects in images! Specify the detection of any object and get State of the Art results without any model fine tuning! In addition, get segmentation predictions from SAM with just text prompts!
8+
9+
See [here](https://github.com/IDEA-Research/GroundingDINO) for more details about the pretrained GroundingDINO model.
10+
11+
12+
Quickstart
13+
14+
1. Make sure docker is installed
15+
2. Edit `docker-compose.yml` to include your LABEL_STUDIO_ACCESS_TOKEN found in the Label Studio software, and the LABEL_STUDIO_HOST which includes the address on which the frontend is hosted on.
16+
17+
Example-
18+
- `LABEL_STUDIO_HOST=http://123.456.7.8:8080`
19+
- `LABEL_STUDIO_ACCESS_TOKEN=c9djf998eii2948ee9hh835nferkj959923`
20+
21+
3. Run `docker compose up`
22+
4. Check the IP of your backend using `docker ps` and add it to the Machine Learning backend in the Label Studio software. Usually this is on `http://localhost:9090`.
23+
24+
5. Edit the labelling config to the below.
25+
26+
```
27+
<View>
28+
<Image name="image" value="$image"/>
29+
<Style>
30+
.lsf-main-content.lsf-requesting .prompt::before { content: ' loading...'; color: #808080; }
31+
</Style>
32+
<View className="prompt">
33+
<TextArea name="prompt" toName="image" editable="true" rows="2" maxSubmissions="1" showSubmitButton="true"/>
34+
</View>
35+
<RectangleLabels name="label" toName="image">
36+
<Label value="cats" background="yellow"/>
37+
</RectangleLabels>
38+
</View>
39+
```
40+
41+
This may be adjusted to your needs, but please keep the promp section and some rectangle labels.
42+
43+
6. Go to an image task in one of your projects. Turn on the Auto-annotation switch. Then, type in the prompt box and press add. After this, you should receive your predictions. See the video above for a demo.
44+
45+
46+
## Using GroundingSAM
47+
48+
Combine the Segment Anything Model with your text input to automatically generate mask predictions!
49+
50+
To do this, set `USE_SAM=True` before running.
51+
52+
If you want to use a more efficient version of SAM, set `USE_MOBILE_SAM=True` as well.
53+
54+
55+
## Other Environment Variables
56+
57+
Adjust `BOX_THRESHOLD` and `TEXT_THRESHOLD` values in the Dockerfile to a number between 0 to 1 if experimenting. Defaults are set in `dino.py`. See explanation of these values in this [section](https://github.com/IDEA-Research/GroundingDINO#star-explanationstips-for-grounding-dino-inputs-and-outputs).
58+
59+
If you want to use SAM models saved from either directories, you can use the `MOBILESAM_CHECKPOINT` and `SAM_CHECKPOINT` as shown in the Dockerfile.
Lines changed: 114 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,114 @@
1+
import os
2+
import argparse
3+
import json
4+
import logging
5+
import logging.config
6+
7+
logging.config.dictConfig({
8+
"version": 1,
9+
"formatters": {
10+
"standard": {
11+
"format": "[%(asctime)s] [%(levelname)s] [%(name)s::%(funcName)s::%(lineno)d] %(message)s"
12+
}
13+
},
14+
"handlers": {
15+
"console": {
16+
"class": "logging.StreamHandler",
17+
"level": os.getenv('LOG_LEVEL'),
18+
"stream": "ext://sys.stdout",
19+
"formatter": "standard"
20+
}
21+
},
22+
"root": {
23+
"level": os.getenv('LOG_LEVEL'),
24+
"handlers": [
25+
"console"
26+
],
27+
"propagate": True
28+
}
29+
})
30+
31+
from label_studio_ml.api import init_app
32+
from dino import DINOBackend
33+
34+
35+
_DEFAULT_CONFIG_PATH = os.path.join(os.path.dirname(__file__), 'config.json')
36+
37+
38+
def get_kwargs_from_config(config_path=_DEFAULT_CONFIG_PATH):
39+
if not os.path.exists(config_path):
40+
return dict()
41+
with open(config_path) as f:
42+
config = json.load(f)
43+
assert isinstance(config, dict)
44+
return config
45+
46+
47+
if __name__ == "__main__":
48+
parser = argparse.ArgumentParser(description='Label studio')
49+
parser.add_argument(
50+
'-p', '--port', dest='port', type=int, default=9090,
51+
help='Server port')
52+
parser.add_argument(
53+
'--host', dest='host', type=str, default='0.0.0.0',
54+
help='Server host')
55+
parser.add_argument(
56+
'--kwargs', '--with', dest='kwargs', metavar='KEY=VAL', nargs='+', type=lambda kv: kv.split('='),
57+
help='Additional LabelStudioMLBase model initialization kwargs')
58+
parser.add_argument(
59+
'-d', '--debug', dest='debug', action='store_true',
60+
help='Switch debug mode')
61+
parser.add_argument(
62+
'--log-level', dest='log_level', choices=['DEBUG', 'INFO', 'WARNING', 'ERROR'], default=None,
63+
help='Logging level')
64+
parser.add_argument(
65+
'--model-dir', dest='model_dir', default=os.path.dirname(__file__),
66+
help='Directory where models are stored (relative to the project directory)')
67+
parser.add_argument(
68+
'--check', dest='check', action='store_true',
69+
help='Validate model instance before launching server')
70+
71+
args = parser.parse_args()
72+
73+
# setup logging level
74+
if args.log_level:
75+
logging.root.setLevel(args.log_level)
76+
77+
def isfloat(value):
78+
try:
79+
float(value)
80+
return True
81+
except ValueError:
82+
return False
83+
84+
def parse_kwargs():
85+
param = dict()
86+
for k, v in args.kwargs:
87+
if v.isdigit():
88+
param[k] = int(v)
89+
elif v == 'True' or v == 'true':
90+
param[k] = True
91+
elif v == 'False' or v == 'false':
92+
param[k] = False
93+
elif isfloat(v):
94+
param[k] = float(v)
95+
else:
96+
param[k] = v
97+
return param
98+
99+
kwargs = get_kwargs_from_config()
100+
101+
if args.kwargs:
102+
kwargs.update(parse_kwargs())
103+
104+
if args.check:
105+
print('Check "' + DINOBackend.__name__ + '" instance creation..')
106+
model = DINOBackend(**kwargs)
107+
108+
app = init_app(model_class=DINOBackend)
109+
110+
app.run(host=args.host, port=args.port, debug=args.debug)
111+
112+
else:
113+
# for uWSGI use
114+
app = init_app(model_class=DINOBackend)

0 commit comments

Comments
 (0)