Skip to content

Commit 54e45e3

Browse files
kevinwuTTkevinwuTT
authored
Generate metrics data with pydantic models (#51)
* Generate aten op conversion status tables for each model * Generate data with pydantic models * Move pydantic model schema to a separate file
1 parent b88b88d commit 54e45e3

File tree

16 files changed

+580
-300
lines changed

16 files changed

+580
-300
lines changed

‎README.md

Lines changed: 90 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -8,17 +8,17 @@ This project allows to run PyTorch code on [Tenstorrent](https://tenstorrent.com
88

99
The table below summarizes the results of running various ML models through our TTNN compiler. For each model, we track whether the run was successful, the number of operations before and after conversion, the number of `to_device` and `from_device` operations, performance metrics, and accuracy.
1010

11-
| Model | Run Success | Torch Ops Before (Unique Ops) | Torch Ops Remain (Unique Ops) | To/From Device Ops | Original Run Time (s) | Compiled Run Time(s) | Accuracy |
12-
|:------------------------------------|:--------------|:--------------------------------|:--------------------------------|:---------------------|------------------------:|:-----------------------|:-----------|
13-
| [Mnist (Eval)](tests/models/mnist) | ✘ | 14 (8) | 5 (4) | 12 | 0.01 | N/A | N/A |
14-
| [Mnist (Train)](tests/models/mnist) | ✅ | 14 (8) | 7 (5) | 14 | 0.01 | 2.52 | 0.64 |
15-
| [ResNet18](tests/models/resnet) | ✅ | 70 (9) | 42 (4) | 42 | 1.78 | 9.46 | 1.0 |
16-
| [Bloom](tests/models/bloom) | ✘ | N/A | N/A | N/A | 5.58 | N/A | N/A |
17-
| [YOLOS](tests/models/yolos) | ✘ | N/A | N/A | N/A | 0.18 | N/A | N/A |
18-
| [Llama](tests/models/llama) | ✘ | 3 (3) | 1 (1) | 5 | 38.21 | N/A | N/A |
19-
| [BERT](tests/models/bert) | ✅ | 1393 (21) | 489 (4) | 1340 | 62 | 36.17 | 0.99 |
20-
| [Falcon](tests/models/falcon) | ✘ | 3 (3) | 1 (1) | 5 | 34.81 | N/A | N/A |
21-
| [GPT-2](tests/models/gpt2) | ✘ | N/A | N/A | N/A | 1.04 | N/A | N/A |
11+
| Model | Run Success | Torch Ops Before (Unique Ops) | Torch Ops Remain (Unique Ops) | To/From Device Ops | Original Run Time (ms) | Compiled Run Time (ms) | Accuracy (%) |
12+
|:------------------------------------|:--------------|:--------------------------------|:--------------------------------|:---------------------|-------------------------:|:-------------------------|:---------------|
13+
| [Mnist (Eval)](tests/models/mnist) | ✘ | 14 (8) | 5 (4) | 12 | 11.04 | N/A | N/A |
14+
| [Mnist (Train)](tests/models/mnist) | ✅ | 14 (8) | 7 (5) | 14 | 18.01 | 2922.51 | 85.88 |
15+
| [ResNet18](tests/models/resnet) | ✅ | 70 (9) | 42 (4) | 45 | 1772.4 | 8398.87 | 99.99 |
16+
| [Bloom](tests/models/bloom) | ✘ | 1407 (29) | N/A | N/A | 5602.6 | N/A | N/A |
17+
| [YOLOS](tests/models/yolos) | ✘ | 964 (28) | N/A | N/A | 209.04 | N/A | N/A |
18+
| [Llama](tests/models/llama) | ✘ | 3 (3) | 1 (1) | 5 | 38255.4 | N/A | N/A |
19+
| [BERT](tests/models/bert) | ✅ | 1393 (21) | 537 (4) | 1388 | 61919.4 | 52814.88 | 98.64 |
20+
| [Falcon](tests/models/falcon) | ✘ | 3 (3) | 1 (1) | 5 | 35014.3 | N/A | N/A |
21+
| [GPT-2](tests/models/gpt2) | ✘ | 748 (31) | N/A | N/A | 1033.47 | N/A | N/A |
2222

2323
### Explanation of Metrics
2424

@@ -27,12 +27,88 @@ The table below summarizes the results of running various ML models through our
2727
**Torch Ops Before (Unique Ops)**: The total number of operations used by the model in the original Torch implementation. The number in parenthesis represents the total unique ops.
2828
**Torch Ops Remain (Unique Ops)**: The total number of operations used after conversion to TTNN. The number in parenthesis represents the total unique ops.
2929
**To/From Device Ops**: The number of `to/from_device` operations (data transfer to/from the device).
30-
**Original Run Time (s)**: Execution time (in seconds) of the model before conversion.
31-
**Compiled Run Time(s)**: Execution time (in seconds) of the model after conversion.
32-
**Accuracy**: Model accuracy on a predefined test dataset after conversion.
30+
**Original Run Time (ms)**: Execution time (in seconds) of the model before conversion.
31+
**Compiled Run Time (ms)**: Execution time (in seconds) of the model after conversion.
32+
**Accuracy (%)**: Model accuracy on a predefined test dataset after conversion.
3333
***
3434
**NOTE:** The total number of ops currently reflect only the first graph of a model. This will be fixed in a future update to include all graphs.
3535

36+
***
37+
38+
### Op conversion status per model
39+
40+
#### Mnist (Eval)
41+
| aten ops | status | count |
42+
|:-------------------------------------|:---------|--------:|
43+
| aten._log_softmax.default | ✘ | 1 |
44+
| aten.addmm.default | ✅ | 2 |
45+
| aten.clone.default | ✅ | 2 |
46+
| aten.convolution.default | ✘ | 2 |
47+
| aten.max_pool2d_with_indices.default | ✘ | 1 |
48+
| aten.relu.default | ✅ | 3 |
49+
| aten.t.default | ✅ | 2 |
50+
| aten.view.default | ✘ | 1 |
51+
#### Mnist (Train)
52+
| aten ops | status | count |
53+
|:-------------------------------------|:---------|--------:|
54+
| aten._log_softmax.default | ✘ | 1 |
55+
| aten.addmm.default | ✅ | 2 |
56+
| aten.convolution.default | ✘ | 2 |
57+
| aten.max_pool2d_with_indices.default | ✘ | 1 |
58+
| aten.native_dropout.default | ✘ | 2 |
59+
| aten.relu.default | ✅ | 3 |
60+
| aten.t.default | ✅ | 2 |
61+
| aten.view.default | ✘ | 1 |
62+
#### ResNet18
63+
| aten ops | status | count |
64+
|:--------------------------------------------------|:---------|--------:|
65+
| aten._native_batch_norm_legit_no_training.default | ✘ | 20 |
66+
| aten.add.Tensor | ✅ | 8 |
67+
| aten.addmm.default | ✅ | 1 |
68+
| aten.convolution.default | ✘ | 20 |
69+
| aten.max_pool2d_with_indices.default | ✘ | 1 |
70+
| aten.mean.dim | ✅ | 1 |
71+
| aten.relu.default | ✅ | 17 |
72+
| aten.t.default | ✅ | 1 |
73+
| aten.view.default | ✘ | 1 |
74+
#### Llama
75+
| aten ops | status | count |
76+
|:-----------------------|:---------|--------:|
77+
| aten.arange.start | ✘ | 1 |
78+
| aten.embedding.default | ✅ | 1 |
79+
| aten.unsqueeze.default | ✅ | 1 |
80+
#### BERT
81+
| aten ops | status | count |
82+
|:-------------------------------|:---------|--------:|
83+
| aten._softmax.default | ✅ | 24 |
84+
| aten._to_copy.default | ✅ | 1 |
85+
| aten.add.Tensor | ✅ | 74 |
86+
| aten.addmm.default | ✅ | 145 |
87+
| aten.bmm.default | ✅ | 48 |
88+
| aten.clone.default | ✅ | 99 |
89+
| aten.div.Tensor | ✅ | 24 |
90+
| aten.embedding.default | ✅ | 3 |
91+
| aten.expand.default | ✅ | 96 |
92+
| aten.gelu.default | ✅ | 24 |
93+
| aten.mul.Tensor | ✅ | 1 |
94+
| aten.native_layer_norm.default | ✅ | 49 |
95+
| aten.permute.default | ✅ | 96 |
96+
| aten.rsub.Scalar | ✅ | 1 |
97+
| aten.slice.Tensor | ✘ | 4 |
98+
| aten.split.Tensor | ✘ | 1 |
99+
| aten.squeeze.dim | ✘ | 2 |
100+
| aten.t.default | ✅ | 145 |
101+
| aten.transpose.int | ✅ | 24 |
102+
| aten.unsqueeze.default | ✅ | 2 |
103+
| aten.view.default | ✘ | 530 |
104+
#### Falcon
105+
| aten ops | status | count |
106+
|:-----------------------|:---------|--------:|
107+
| aten.arange.start | ✘ | 1 |
108+
| aten.embedding.default | ✅ | 1 |
109+
| aten.unsqueeze.default | ✅ | 1 |
110+
111+
36112
## Quickstart
37113

38114
The `torch_ttnn` module has a `backend` function, which can be used with the `torch.compile()`.

‎docs/README.md.in

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,12 @@ The table below summarizes the results of running various ML models through our
1212

1313
{explanations_md}
1414

15+
***
16+
17+
### Op conversion status per model
18+
19+
{aten_ops_md}
20+
1521
## Quickstart
1622

1723
The `torch_ttnn` module has a `backend` function, which can be used with the `torch.compile()`.

‎requirements-dev.txt

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,3 +5,4 @@ pre-commit==3.0.4
55
transformers==4.38.0
66
pandas==2.0.3
77
Pillow==10.3.0
8+
pydantic==2.8.2

‎tests/models/bert/test_bert.py

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -47,7 +47,9 @@ def decode_output(outputs):
4747
answer_before = decode_output(outputs_before)
4848

4949
# Compile model with ttnn backend
50-
option = torch_ttnn.TorchTtnnOption(device=device, metrics_path=metrics_path)
50+
option = torch_ttnn.TorchTtnnOption(
51+
device=device, gen_graphviz=True, metrics_path=metrics_path
52+
)
5153
m = torch.compile(m, backend=torch_ttnn.backend, options=option)
5254

5355
# Run inference with the compiled model

‎tests/models/bloom/test_bloom.py

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -35,7 +35,9 @@ def decode_output(outputs):
3535
decoded_output_before = decode_output(outputs_before)
3636

3737
# Compile model with ttnn backend
38-
option = torch_ttnn.TorchTtnnOption(device=device, metrics_path=metrics_path)
38+
option = torch_ttnn.TorchTtnnOption(
39+
device=device, gen_graphviz=True, metrics_path=metrics_path
40+
)
3941
m = torch.compile(m, backend=torch_ttnn.backend, options=option)
4042

4143
# Run inference with the compiled model

‎tests/models/falcon/test_falcon.py

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -35,7 +35,9 @@ def decode_output(outputs):
3535
decoded_output_before = decode_output(outputs_before)
3636

3737
# Compile model with ttnn backend
38-
option = torch_ttnn.TorchTtnnOption(device=device, metrics_path=metrics_path)
38+
option = torch_ttnn.TorchTtnnOption(
39+
device=device, gen_graphviz=True, metrics_path=metrics_path
40+
)
3941
m = torch.compile(m, backend=torch_ttnn.backend, options=option)
4042

4143
# Run inference with the compiled model

‎tests/models/gpt2/test_gpt2.py

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -36,7 +36,9 @@ def decode_output(outputs):
3636
decoded_output_before = decode_output(outputs_before)
3737

3838
# Compile model with ttnn backend
39-
option = torch_ttnn.TorchTtnnOption(device=device, metrics_path=metrics_path)
39+
option = torch_ttnn.TorchTtnnOption(
40+
device=device, gen_graphviz=True, metrics_path=metrics_path
41+
)
4042
m = torch.compile(m, backend=torch_ttnn.backend, options=option)
4143

4244
# Run inference with the compiled model

‎tests/models/llama/test_llama.py

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -35,7 +35,9 @@ def decode_output(outputs):
3535
decoded_output_before = decode_output(outputs_before)
3636

3737
# Compile model with ttnn backend
38-
option = torch_ttnn.TorchTtnnOption(device=device, metrics_path=metrics_path)
38+
option = torch_ttnn.TorchTtnnOption(
39+
device=device, gen_graphviz=True, metrics_path=metrics_path
40+
)
3941
m = torch.compile(m, backend=torch_ttnn.backend, options=option)
4042

4143
# Run inference with the compiled model

‎tests/models/mnist/test_mnist.py

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -57,7 +57,9 @@ def test_mnist_train(device):
5757
)
5858

5959
# Compile model with ttnn backend
60-
option = torch_ttnn.TorchTtnnOption(device=device, metrics_path=metrics_path)
60+
option = torch_ttnn.TorchTtnnOption(
61+
device=device, gen_graphviz=True, metrics_path=metrics_path
62+
)
6163
m = torch.compile(m, backend=torch_ttnn.backend, options=option)
6264

6365
# Run train with the compiled model
@@ -93,7 +95,9 @@ def test_mnist_eval(device):
9395
)
9496

9597
# Compile model with ttnn backend
96-
option = torch_ttnn.TorchTtnnOption(device=device, metrics_path=metrics_path)
98+
option = torch_ttnn.TorchTtnnOption(
99+
device=device, gen_graphviz=True, metrics_path=metrics_path
100+
)
97101
m = torch.compile(m, backend=torch_ttnn.backend, options=option)
98102

99103
# Run inference with the compiled model

‎tests/models/resnet/test_resnet.py

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -23,7 +23,9 @@ def test_resnet(device):
2323
)
2424

2525
# Compile the model
26-
option = torch_ttnn.TorchTtnnOption(device=device, metrics_path=metrics_path)
26+
option = torch_ttnn.TorchTtnnOption(
27+
device=device, gen_graphviz=True, metrics_path=metrics_path
28+
)
2729
option.gen_graphviz = True
2830
model = torch.compile(model, backend=torch_ttnn.backend, options=option)
2931

‎tests/models/yolos/test_yolos.py

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -42,7 +42,9 @@ def decode_output(outputs):
4242
decoded_output_before = decode_output(outputs_before)
4343

4444
# Compile model with ttnn backend
45-
option = torch_ttnn.TorchTtnnOption(device=device, metrics_path=metrics_path)
45+
option = torch_ttnn.TorchTtnnOption(
46+
device=device, gen_graphviz=True, metrics_path=metrics_path
47+
)
4648
m = torch.compile(m, backend=torch_ttnn.backend, options=option)
4749

4850
# Run inference with the compiled model

0 commit comments

Comments
 (0)