Skip to content

Commit 08898b3

Browse files
kevinwuTTkevinwuTT
committed
Update README with new model stats
1 parent 47ef3ba commit 08898b3

File tree

1 file changed

+121
-23
lines changed

1 file changed

+121
-23
lines changed

‎README.md

Lines changed: 121 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -8,17 +8,17 @@ This project allows to run PyTorch code on [Tenstorrent](https://tenstorrent.com
88

99
The table below summarizes the results of running various ML models through our TTNN compiler. For each model, we track whether the run was successful, the number of operations before and after conversion, the number of `to_device` and `from_device` operations, performance metrics, and accuracy.
1010

11-
| Model | Run Success | Torch Ops Before (Unique Ops) | Torch Ops Remain (Unique Ops) | To/From Device Ops | Original Run Time (ms) | Compiled Run Time (ms) | Accuracy (%) |
12-
|:------------------------------------|:--------------|:--------------------------------|:--------------------------------|:---------------------|-------------------------:|:-------------------------|:---------------|
13-
| [Mnist (Eval)](tests/models/mnist) | ✘ | 14 (8) | 5 (4) | 16 | 36.12 | N/A | N/A |
14-
| [Mnist (Train)](tests/models/mnist) | ✅ | 14 (8) | 7 (5) | 14 | 114.49 | 2742.8 | 81.75 |
15-
| [ResNet18](tests/models/resnet) | ✅ | 70 (9) | 42 (4) | 47 | 2094.6 | 10950.18 | 99.99 |
16-
| [Bloom](tests/models/bloom) | ✘ | 1407 (29) | N/A | N/A | 9127.68 | N/A | N/A |
17-
| [YOLOS](tests/models/yolos) | ✘ | 964 (28) | N/A | N/A | 1353.22 | N/A | N/A |
18-
| [Llama](tests/models/llama) | ✘ | 3 (3) | 1 (1) | 5 | 52926.3 | N/A | N/A |
19-
| [BERT](tests/models/bert) | ✅ | 1393 (21) | 537 (4) | 1607 | 65342 | 61028.65 | 98.64 |
20-
| [Falcon](tests/models/falcon) | ✘ | 3 (3) | 1 (1) | 5 | 47738.8 | N/A | N/A |
21-
| [GPT-2](tests/models/gpt2) | ✘ | 748 (31) | N/A | N/A | 2287.61 | N/A | N/A |
11+
| Model | Run Success | Torch Ops Before (Unique Ops) | Torch Ops Remain (Unique Ops) | To/From Device Ops | Original Run Time (ms) | Compiled Run Time (ms) | Accuracy (%) |
12+
|:------------------------------------|:--------------|:--------------------------------|:--------------------------------|---------------------:|-------------------------:|:-------------------------|:---------------|
13+
| [Mnist (Eval)](tests/models/mnist) | ✅ | 14 (8) | 5 (4) | 16 | 36.62 | 477.12 | 98.33 |
14+
| [Mnist (Train)](tests/models/mnist) | ✅ | 14 (8) | 7 (5) | 14 | 117.66 | 2571.71 | 68.54 |
15+
| [ResNet18](tests/models/resnet) | ✅ | 70 (9) | 42 (4) | 47 | 2025.6 | 10350.92 | 99.99 |
16+
| [Bloom](tests/models/bloom) | ✅ | 1407 (29) | 627 (12) | 1375 | 28876.2 | 65763.95 | 45.77 |
17+
| [YOLOS](tests/models/yolos) | ✅ | 964 (28) | 409 (11) | 919 | 1516.37 | 45219.57 | 71.71 |
18+
| [Llama](tests/models/llama) | ✅ | 5 (4) | 3 (2) | 3 | 168160 | 186989.25 | 34.41 |
19+
| [BERT](tests/models/bert) | ✅ | 1393 (21) | 564 (7) | 1488 | 64262 | 57553.72 | 98.64 |
20+
| [Falcon](tests/models/falcon) | ✘ | 3 (3) | 2 (2) | 5 | 39306.6 | N/A | N/A |
21+
| [GPT-2](tests/models/gpt2) | ✘ | 748 (31) | 330 (15) | 530 | 5889.45 | N/A | N/A |
2222

2323
### Explanation of Metrics
2424

@@ -47,7 +47,7 @@ The table below summarizes the results of running various ML models through our
4747
| aten.max_pool2d_with_indices.default | ✘ | 1 |
4848
| aten.relu.default | ✅ | 3 |
4949
| aten.t.default | ✅ | 2 |
50-
| aten.view.default | ✘ | 1 |
50+
| aten.view.default | ✅ | 1 |
5151
#### Mnist (Train)
5252
| aten ops | status | count |
5353
|:-------------------------------------|:---------|--------:|
@@ -58,7 +58,7 @@ The table below summarizes the results of running various ML models through our
5858
| aten.native_dropout.default | ✘ | 2 |
5959
| aten.relu.default | ✅ | 3 |
6060
| aten.t.default | ✅ | 2 |
61-
| aten.view.default | ✘ | 1 |
61+
| aten.view.default | ✅ | 1 |
6262
#### ResNet18
6363
| aten ops | status | count |
6464
|:--------------------------------------------------|:---------|--------:|
@@ -70,43 +70,141 @@ The table below summarizes the results of running various ML models through our
7070
| aten.mean.dim | ✅ | 1 |
7171
| aten.relu.default | ✅ | 17 |
7272
| aten.t.default | ✅ | 1 |
73-
| aten.view.default | ✘ | 1 |
73+
| aten.view.default | ✅ | 1 |
74+
#### Bloom
75+
| aten ops | status | count |
76+
|:-------------------------------|:---------|--------:|
77+
| aten._softmax.default | ✅ | 24 |
78+
| aten._to_copy.default | ✘ | 54 |
79+
| aten._unsafe_view.default | ✘ | 24 |
80+
| aten.add.Tensor | ✅ | 96 |
81+
| aten.addmm.default | ✅ | 96 |
82+
| aten.arange.start | ✘ | 1 |
83+
| aten.baddbmm.default | ✅ | 24 |
84+
| aten.bmm.default | ✅ | 24 |
85+
| aten.clone.default | ✅ | 96 |
86+
| aten.cumsum.default | ✘ | 1 |
87+
| aten.embedding.default | ✅ | 1 |
88+
| aten.expand.default | ✅ | 2 |
89+
| aten.full.default | ✅ | 1 |
90+
| aten.lift_fresh_copy.default | ✘ | 1 |
91+
| aten.masked_fill.Scalar | ✘ | 26 |
92+
| aten.mm.default | ✅ | 1 |
93+
| aten.mul.Tensor | ✅ | 146 |
94+
| aten.native_layer_norm.default | ✅ | 50 |
95+
| aten.permute.default | ✅ | 48 |
96+
| aten.pow.Tensor_Tensor | ✘ | 1 |
97+
| aten.rsub.Scalar | ✘ | 1 |
98+
| aten.select.int | ✘ | 72 |
99+
| aten.slice.Tensor | ✘ | 78 |
100+
| aten.sub.Tensor | ✅ | 1 |
101+
| aten.t.default | ✅ | 97 |
102+
| aten.tanh.default | ✅ | 24 |
103+
| aten.transpose.int | ✅ | 48 |
104+
| aten.unsqueeze.default | ✘ | 6 |
105+
| aten.view.default | ✅ | 363 |
106+
#### YOLOS
107+
| aten ops | status | count |
108+
|:-------------------------------|:---------|--------:|
109+
| aten._softmax.default | ✅ | 12 |
110+
| aten._to_copy.default | ✘ | 2 |
111+
| aten._unsafe_index.Tensor | ✘ | 16 |
112+
| aten.add.Tensor | ✅ | 71 |
113+
| aten.addmm.default | ✅ | 78 |
114+
| aten.arange.default | ✘ | 4 |
115+
| aten.bmm.default | ✅ | 24 |
116+
| aten.cat.default | ✘ | 2 |
117+
| aten.clamp.default | ✅ | 32 |
118+
| aten.clone.default | ✅ | 50 |
119+
| aten.convolution.default | ✘ | 1 |
120+
| aten.div.Tensor | ✘ | 12 |
121+
| aten.expand.default | ✅ | 50 |
122+
| aten.floor.default | ✘ | 2 |
123+
| aten.gelu.default | ✅ | 12 |
124+
| aten.mul.Tensor | ✅ | 82 |
125+
| aten.native_layer_norm.default | ✅ | 25 |
126+
| aten.permute.default | ✅ | 48 |
127+
| aten.relu.default | ✅ | 4 |
128+
| aten.rsub.Scalar | ✘ | 10 |
129+
| aten.select.int | ✘ | 1 |
130+
| aten.sigmoid.default | ✅ | 1 |
131+
| aten.slice.Tensor | ✘ | 12 |
132+
| aten.sub.Tensor | ✅ | 36 |
133+
| aten.t.default | ✅ | 78 |
134+
| aten.transpose.int | ✅ | 15 |
135+
| aten.unsqueeze.default | ✅ | 1 |
136+
| aten.view.default | ✅ | 283 |
74137
#### Llama
75-
| aten ops | status | count |
76-
|:-----------------------|:---------|--------:|
77-
| aten.arange.start | ✘ | 1 |
78-
| aten.embedding.default | ✅ | 1 |
79-
| aten.unsqueeze.default | ✅ | 1 |
138+
| aten ops | status | count |
139+
|:----------------------|:---------|--------:|
140+
| aten._to_copy.default | ✘ | 1 |
141+
| aten.mm.default | ✅ | 1 |
142+
| aten.t.default | ✅ | 1 |
143+
| aten.view.default | ✅ | 2 |
80144
#### BERT
81145
| aten ops | status | count |
82146
|:-------------------------------|:---------|--------:|
83147
| aten._softmax.default | ✅ | 24 |
84-
| aten._to_copy.default | ✅ | 1 |
148+
| aten._to_copy.default | ✘ | 1 |
85149
| aten.add.Tensor | ✅ | 74 |
86150
| aten.addmm.default | ✅ | 145 |
87151
| aten.bmm.default | ✅ | 48 |
88152
| aten.clone.default | ✅ | 99 |
89-
| aten.div.Tensor | ✅ | 24 |
153+
| aten.div.Tensor | ✘ | 24 |
90154
| aten.embedding.default | ✅ | 3 |
91155
| aten.expand.default | ✅ | 96 |
92156
| aten.gelu.default | ✅ | 24 |
93157
| aten.mul.Tensor | ✅ | 1 |
94158
| aten.native_layer_norm.default | ✅ | 49 |
95159
| aten.permute.default | ✅ | 96 |
96-
| aten.rsub.Scalar | ✅ | 1 |
160+
| aten.rsub.Scalar | ✘ | 1 |
97161
| aten.slice.Tensor | ✘ | 4 |
98162
| aten.split.Tensor | ✘ | 1 |
99163
| aten.squeeze.dim | ✘ | 2 |
100164
| aten.t.default | ✅ | 145 |
101165
| aten.transpose.int | ✅ | 24 |
102166
| aten.unsqueeze.default | ✅ | 2 |
103-
| aten.view.default | ✘ | 530 |
167+
| aten.view.default | ✅ | 530 |
104168
#### Falcon
105169
| aten ops | status | count |
106170
|:-----------------------|:---------|--------:|
107171
| aten.arange.start | ✘ | 1 |
108172
| aten.embedding.default | ✅ | 1 |
109173
| aten.unsqueeze.default | ✅ | 1 |
174+
#### GPT-2
175+
| aten ops | status | count |
176+
|:-------------------------------|:---------|--------:|
177+
| aten._softmax.default | ✅ | 12 |
178+
| aten._to_copy.default | ✘ | 2 |
179+
| aten.add.Tensor | ✅ | 61 |
180+
| aten.addmm.default | ✅ | 48 |
181+
| aten.arange.default | ✘ | 1 |
182+
| aten.arange.start | ✘ | 1 |
183+
| aten.argmax.default | ✘ | 1 |
184+
| aten.bmm.default | ✅ | 24 |
185+
| aten.clone.default | ✅ | 49 |
186+
| aten.div.Tensor | ✘ | 12 |
187+
| aten.embedding.default | ✅ | 2 |
188+
| aten.eq.Scalar | ✘ | 1 |
189+
| aten.expand.default | ✅ | 48 |
190+
| aten.full.default | ✘ | 24 |
191+
| aten.index.Tensor | ✘ | 1 |
192+
| aten.mm.default | ✅ | 1 |
193+
| aten.mul.Tensor | ✅ | 49 |
194+
| aten.native_layer_norm.default | ✅ | 25 |
195+
| aten.permute.default | ✅ | 48 |
196+
| aten.pow.Tensor_Scalar | ✅ | 12 |
197+
| aten.remainder.Scalar | ✘ | 1 |
198+
| aten.rsub.Scalar | ✘ | 1 |
199+
| aten.slice.Tensor | ✘ | 50 |
200+
| aten.split.Tensor | ✘ | 12 |
201+
| aten.sub.Tensor | ✘ | 1 |
202+
| aten.t.default | ✅ | 1 |
203+
| aten.tanh.default | ✅ | 12 |
204+
| aten.transpose.int | ✅ | 12 |
205+
| aten.unsqueeze.default | ✅ | 3 |
206+
| aten.view.default | ✅ | 221 |
207+
| aten.where.self | ✘ | 12 |
110208

111209

112210
## Quickstart

0 commit comments

Comments
 (0)