Skip to content

Commit 25878dc

Browse files
mlcommons-botmlcommons-bot
committed
Auto-merge updates from auto-update branch
2 parents 07fb97f + ce23289 commit 25878dc

File tree

14 files changed

+465
-465
lines changed

14 files changed

+465
-465
lines changed

‎open/MLCommons/measurements/gh_action-reference-cpu-pytorch_v2.6.0-default_config/README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,3 @@
11
| Model | Scenario | Accuracy | Throughput | Latency (in ms) |
22
|---------------|------------|------------------------------------|--------------|-------------------|
3-
| llama2-70b-99 | offline | (61.7021, 37.9679, 39.3617, 610.0) | 0.401 | - |
3+
| llama2-70b-99 | offline | (61.7021, 37.9679, 39.3617, 610.0) | 0.406 | - |

‎open/MLCommons/measurements/gh_action-reference-cpu-pytorch_v2.6.0-default_config/llama2-70b-99/offline/README.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,7 @@ pip install -U mlcflow
1616

1717
mlc rm cache -f
1818

19-
mlc pull repo gateoverflow@mlperf-automations --checkout=08ce3123f46ca9d1d4a403a96650f0b008e000b8
19+
mlc pull repo gateoverflow@mlperf-automations --checkout=03a8a5ac4e767293a1029bdeb20d7cd24ce4289a
2020

2121

2222
```
@@ -43,4 +43,4 @@ Model Precision: fp32
4343
`TOKENS_PER_SAMPLE`: `610.0`, Required accuracy for closed division `>= 265.005` and `<= 323.895`
4444

4545
### Performance Results
46-
`Samples per second`: `0.40113`
46+
`Samples per second`: `0.406358`

‎open/MLCommons/measurements/gh_action-reference-cpu-pytorch_v2.6.0-default_config/llama2-70b-99/offline/accuracy_console.out‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
INFO:datasets:PyTorch version 2.6.0+cpu available.
22
Loading dataset...
33
Finished loading dataset.
4-
Loading checkpoint shards: 0%| | 0/15 [00:00<?, ?it/s]Loading checkpoint shards: 7%|▋ | 1/15 [00:01<00:23, 1.68s/it]Loading checkpoint shards: 13%|█▎ | 2/15 [00:03<00:23, 1.83s/it]Loading checkpoint shards: 20%|██ | 3/15 [00:05<00:22, 1.89s/it]Loading checkpoint shards: 27%|██▋ | 4/15 [00:07<00:21, 1.91s/it]Loading checkpoint shards: 33%|███▎ | 5/15 [00:09<00:18, 1.89s/it]Loading checkpoint shards: 40%|████ | 6/15 [00:11<00:16, 1.86s/it]Loading checkpoint shards: 47%|████▋ | 7/15 [00:12<00:14, 1.84s/it]Loading checkpoint shards: 53%|█████▎ | 8/15 [00:15<00:13, 1.95s/it]Loading checkpoint shards: 60%|██████ | 9/15 [00:18<00:14, 2.34s/it]Loading checkpoint shards: 67%|██████▋ | 10/15 [00:22<00:15, 3.02s/it]Loading checkpoint shards: 73%|███████▎ | 11/15 [00:25<00:11, 2.77s/it]Loading checkpoint shards: 80%|████████ | 12/15 [00:27<00:07, 2.54s/it]Loading checkpoint shards: 87%|████████▋ | 13/15 [00:28<00:04, 2.33s/it]Loading checkpoint shards: 93%|█████████▎| 14/15 [00:31<00:02, 2.31s/it]Loading checkpoint shards: 100%|██████████| 15/15 [00:31<00:00, 1.74s/it]Loading checkpoint shards: 100%|██████████| 15/15 [00:31<00:00, 2.11s/it]
4+
Loading checkpoint shards: 0%| | 0/15 [00:00<?, ?it/s]Loading checkpoint shards: 7%|▋ | 1/15 [00:01<00:23, 1.68s/it]Loading checkpoint shards: 13%|█▎ | 2/15 [00:03<00:23, 1.84s/it]Loading checkpoint shards: 20%|██ | 3/15 [00:05<00:22, 1.86s/it]Loading checkpoint shards: 27%|██▋ | 4/15 [00:07<00:20, 1.86s/it]Loading checkpoint shards: 33%|███▎ | 5/15 [00:09<00:18, 1.87s/it]Loading checkpoint shards: 40%|████ | 6/15 [00:11<00:16, 1.85s/it]Loading checkpoint shards: 47%|████▋ | 7/15 [00:12<00:14, 1.83s/it]Loading checkpoint shards: 53%|█████▎ | 8/15 [00:15<00:13, 1.93s/it]Loading checkpoint shards: 60%|██████ | 9/15 [00:17<00:11, 1.98s/it]Loading checkpoint shards: 67%|██████▋ | 10/15 [00:19<00:09, 1.99s/it]Loading checkpoint shards: 73%|███████▎ | 11/15 [00:24<00:12, 3.05s/it]Loading checkpoint shards: 80%|████████ | 12/15 [00:26<00:08, 2.78s/it]Loading checkpoint shards: 87%|████████▋ | 13/15 [00:28<00:05, 2.54s/it]Loading checkpoint shards: 93%|█████████▎| 14/15 [00:30<00:02, 2.36s/it]Loading checkpoint shards: 100%|██████████| 15/15 [00:31<00:00, 1.75s/it]Loading checkpoint shards: 100%|██████████| 15/15 [00:31<00:00, 2.07s/it]
55
INFO:Llama-70B-MAIN:Starting Benchmark run
66
/home/mlcuser/venv/mlc/lib/python3.10/site-packages/transformers/generation/configuration_utils.py:628: UserWarning: `do_sample` is set to `False`. However, `temperature` is set to `0.6` -- this flag is only used in sample-based generation modes. You should set `do_sample=True` or unset `temperature`.
77
warnings.warn(
@@ -22,7 +22,7 @@ IssueQuery started with 1 samples
2222
IssueQuery done
2323
Saving outputs to run_outputs/q0.pkl
2424
Samples run: 1
25-
BatchMaker time: 0.007257223129272461
26-
Inference time: 749.6236867904663
27-
Postprocess time: 0.07573413848876953
28-
==== Total time: 749.7066781520844
25+
BatchMaker time: 0.0076906681060791016
26+
Inference time: 776.3055598735809
27+
Postprocess time: 0.07918000221252441
28+
==== Total time: 776.3924305438995

‎open/MLCommons/measurements/gh_action-reference-cpu-pytorch_v2.6.0-default_config/llama2-70b-99/offline/mlc-deps.mmd‎

Lines changed: 26 additions & 26 deletions
Original file line numberDiff line numberDiff line change
@@ -7,16 +7,16 @@ graph TD
77
app-mlperf-inference,d775cac873ee4231_(_reference,_llama2-70b-99,_pytorch,_cpu,_test,_r5.0-dev_default,_bfloat16,_offline_) --> pull,git,repo
88
get-mlperf-inference-utils,e341e5f86d8342e5 --> get,mlperf,inference,src
99
app-mlperf-inference,d775cac873ee4231_(_reference,_llama2-70b-99,_pytorch,_cpu,_test,_r5.0-dev_default,_bfloat16,_offline_) --> get,mlperf,inference,utils
10-
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_pytorch,_llama2-70b-99,_offline,_cpu,_bfloat16_) --> detect,os
10+
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_cpu,_llama2-70b-99,_pytorch,_offline,_bfloat16_) --> detect,os
1111
detect-cpu,586c8a43320142f7 --> detect,os
12-
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_pytorch,_llama2-70b-99,_offline,_cpu,_bfloat16_) --> detect,cpu
13-
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_pytorch,_llama2-70b-99,_offline,_cpu,_bfloat16_) --> get,sys-utils-cm
14-
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_pytorch,_llama2-70b-99,_offline,_cpu,_bfloat16_) --> get,python
12+
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_cpu,_llama2-70b-99,_pytorch,_offline,_bfloat16_) --> detect,cpu
13+
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_cpu,_llama2-70b-99,_pytorch,_offline,_bfloat16_) --> get,sys-utils-cm
14+
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_cpu,_llama2-70b-99,_pytorch,_offline,_bfloat16_) --> get,python
1515
get-generic-python-lib,94b62a682bc44791_(_torch_) --> get,python3
16-
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_pytorch,_llama2-70b-99,_offline,_cpu,_bfloat16_) --> get,generic-python-lib,_torch
16+
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_cpu,_llama2-70b-99,_pytorch,_offline,_bfloat16_) --> get,generic-python-lib,_torch
1717
get-generic-python-lib,94b62a682bc44791_(_torchvision_) --> get,python3
18-
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_pytorch,_llama2-70b-99,_offline,_cpu,_bfloat16_) --> get,generic-python-lib,_torchvision
19-
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_pytorch,_llama2-70b-99,_offline,_cpu,_bfloat16_) --> get,ml-model,llama2,raw,_pytorch
18+
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_cpu,_llama2-70b-99,_pytorch,_offline,_bfloat16_) --> get,generic-python-lib,_torchvision
19+
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_cpu,_llama2-70b-99,_pytorch,_offline,_bfloat16_) --> get,ml-model,llama2,raw,_pytorch
2020
get-preprocessed-dataset-openorca,5614c39cb1564d72_(_validation,_mlcommons_) --> get,sys-utils-cm
2121
get-preprocessed-dataset-openorca,5614c39cb1564d72_(_validation,_mlcommons_) --> get,python3
2222
get-generic-python-lib,94b62a682bc44791_(_package.pyarrow_) --> get,python3
@@ -26,45 +26,45 @@ graph TD
2626
get-generic-python-lib,94b62a682bc44791_(_package.transformers_) --> get,python3
2727
get-preprocessed-dataset-openorca,5614c39cb1564d72_(_validation,_mlcommons_) --> get,generic-python-lib,_package.transformers
2828
get-preprocessed-dataset-openorca,5614c39cb1564d72_(_validation,_mlcommons_) --> download-and-extract,_rclone,_url.mlc-inference:mlcommons-inference-wg-public/open_orca
29-
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_pytorch,_llama2-70b-99,_offline,_cpu,_bfloat16_) --> get,preprocessed,dataset,openorca,_validation,_mlcommons
29+
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_cpu,_llama2-70b-99,_pytorch,_offline,_bfloat16_) --> get,preprocessed,dataset,openorca,_validation,_mlcommons
3030
generate-mlperf-inference-user-conf,3af4475745964b93 --> detect,os
3131
detect-cpu,586c8a43320142f7 --> detect,os
3232
generate-mlperf-inference-user-conf,3af4475745964b93 --> detect,cpu
3333
generate-mlperf-inference-user-conf,3af4475745964b93 --> get,python
3434
generate-mlperf-inference-user-conf,3af4475745964b93 --> get,mlcommons,inference,src
3535
get-mlperf-inference-sut-configs,c2fbf72009e2445b --> get,cache,dir,_name.mlperf-inference-sut-configs
3636
generate-mlperf-inference-user-conf,3af4475745964b93 --> get,sut,configs
37-
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_pytorch,_llama2-70b-99,_offline,_cpu,_bfloat16_) --> generate,user-conf,mlperf,inference
38-
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_pytorch,_llama2-70b-99,_offline,_cpu,_bfloat16_) --> get,loadgen
39-
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_pytorch,_llama2-70b-99,_offline,_cpu,_bfloat16_) --> get,mlcommons,inference,src
40-
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_pytorch,_llama2-70b-99,_offline,_cpu,_bfloat16_) --> get,mlcommons,inference,src
37+
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_cpu,_llama2-70b-99,_pytorch,_offline,_bfloat16_) --> generate,user-conf,mlperf,inference
38+
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_cpu,_llama2-70b-99,_pytorch,_offline,_bfloat16_) --> get,loadgen
39+
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_cpu,_llama2-70b-99,_pytorch,_offline,_bfloat16_) --> get,mlcommons,inference,src
40+
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_cpu,_llama2-70b-99,_pytorch,_offline,_bfloat16_) --> get,mlcommons,inference,src
4141
get-generic-python-lib,94b62a682bc44791_(_package.psutil_) --> get,python3
42-
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_pytorch,_llama2-70b-99,_offline,_cpu,_bfloat16_) --> get,generic-python-lib,_package.psutil
42+
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_cpu,_llama2-70b-99,_pytorch,_offline,_bfloat16_) --> get,generic-python-lib,_package.psutil
4343
get-generic-python-lib,94b62a682bc44791_(_package.transformers_) --> get,python3
44-
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_pytorch,_llama2-70b-99,_offline,_cpu,_bfloat16_) --> get,generic-python-lib,_package.transformers
44+
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_cpu,_llama2-70b-99,_pytorch,_offline,_bfloat16_) --> get,generic-python-lib,_package.transformers
4545
get-generic-python-lib,94b62a682bc44791_(_package.datasets_) --> get,python3
46-
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_pytorch,_llama2-70b-99,_offline,_cpu,_bfloat16_) --> get,generic-python-lib,_package.datasets
46+
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_cpu,_llama2-70b-99,_pytorch,_offline,_bfloat16_) --> get,generic-python-lib,_package.datasets
4747
get-generic-python-lib,94b62a682bc44791_(_package.sentencepiece_) --> get,python3
48-
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_pytorch,_llama2-70b-99,_offline,_cpu,_bfloat16_) --> get,generic-python-lib,_package.sentencepiece
48+
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_cpu,_llama2-70b-99,_pytorch,_offline,_bfloat16_) --> get,generic-python-lib,_package.sentencepiece
4949
get-generic-python-lib,94b62a682bc44791_(_package.protobuf_) --> get,python3
50-
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_pytorch,_llama2-70b-99,_offline,_cpu,_bfloat16_) --> get,generic-python-lib,_package.protobuf
50+
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_cpu,_llama2-70b-99,_pytorch,_offline,_bfloat16_) --> get,generic-python-lib,_package.protobuf
5151
get-generic-python-lib,94b62a682bc44791_(_package.accelerate_) --> get,python3
52-
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_pytorch,_llama2-70b-99,_offline,_cpu,_bfloat16_) --> get,generic-python-lib,_package.accelerate
52+
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_cpu,_llama2-70b-99,_pytorch,_offline,_bfloat16_) --> get,generic-python-lib,_package.accelerate
5353
get-generic-python-lib,94b62a682bc44791_(_package.absl-py_) --> get,python3
54-
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_pytorch,_llama2-70b-99,_offline,_cpu,_bfloat16_) --> get,generic-python-lib,_package.absl-py
54+
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_cpu,_llama2-70b-99,_pytorch,_offline,_bfloat16_) --> get,generic-python-lib,_package.absl-py
5555
get-generic-python-lib,94b62a682bc44791_(_package.evaluate_) --> get,python3
56-
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_pytorch,_llama2-70b-99,_offline,_cpu,_bfloat16_) --> get,generic-python-lib,_package.evaluate
56+
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_cpu,_llama2-70b-99,_pytorch,_offline,_bfloat16_) --> get,generic-python-lib,_package.evaluate
5757
get-generic-python-lib,94b62a682bc44791_(_package.nltk_) --> get,python3
58-
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_pytorch,_llama2-70b-99,_offline,_cpu,_bfloat16_) --> get,generic-python-lib,_package.nltk
58+
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_cpu,_llama2-70b-99,_pytorch,_offline,_bfloat16_) --> get,generic-python-lib,_package.nltk
5959
get-generic-python-lib,94b62a682bc44791_(_package.numpy_) --> get,python3
60-
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_pytorch,_llama2-70b-99,_offline,_cpu,_bfloat16_) --> get,generic-python-lib,_package.numpy
60+
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_cpu,_llama2-70b-99,_pytorch,_offline,_bfloat16_) --> get,generic-python-lib,_package.numpy
6161
get-generic-python-lib,94b62a682bc44791_(_package.rouge-score_) --> get,python3
62-
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_pytorch,_llama2-70b-99,_offline,_cpu,_bfloat16_) --> get,generic-python-lib,_package.rouge-score
62+
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_cpu,_llama2-70b-99,_pytorch,_offline,_bfloat16_) --> get,generic-python-lib,_package.rouge-score
6363
get-generic-python-lib,94b62a682bc44791_(_package.more-itertools_) --> get,python3
64-
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_pytorch,_llama2-70b-99,_offline,_cpu,_bfloat16_) --> get,generic-python-lib,_package.more-itertools
64+
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_cpu,_llama2-70b-99,_pytorch,_offline,_bfloat16_) --> get,generic-python-lib,_package.more-itertools
6565
get-generic-python-lib,94b62a682bc44791_(_package.compressed_tensors_) --> get,python3
66-
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_pytorch,_llama2-70b-99,_offline,_cpu,_bfloat16_) --> get,generic-python-lib,_package.compressed_tensors
66+
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_cpu,_llama2-70b-99,_pytorch,_offline,_bfloat16_) --> get,generic-python-lib,_package.compressed_tensors
6767
detect-cpu,586c8a43320142f7 --> detect,os
6868
benchmark-program,19f369ef47084895 --> detect,cpu
6969
benchmark-program-mlperf,cfff0132a8aa4018 --> benchmark-program,program
70-
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_pytorch,_llama2-70b-99,_offline,_cpu,_bfloat16_) --> benchmark-mlperf
70+
app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_cpu,_llama2-70b-99,_pytorch,_offline,_bfloat16_) --> benchmark-mlperf

‎open/MLCommons/measurements/gh_action-reference-cpu-pytorch_v2.6.0-default_config/llama2-70b-99/offline/mlc-deps.png‎

-1.75 KB
Loading

0 commit comments

Comments
 (0)