TEST01 is failing for pointpainting reference implementation - no accuracy samples in the log

[arjunsuresh]

[2025-07-30 17:15:33,005 customize.py:519 INFO] - /home/mlcuser/venv/mlc/bin/python3  '/home/mlcuser/MLC/repos/local/cache/get-git-repo_inference-src_168234dc/inference/compliance/nvidia/TEST01/run_verification.py'  -r '/home/mlcuser/MLC/repos/local/cache/get-mlperf-inference-results-dir_d5af2b93/valid_results/596a81fc0ea1-reference-gpu-pytorch-v2.2.2-cu126/pointpainting/singlestream' -c  '/home/mlcuser/MLC/repos/local/cache/get-mlperf-inference-results-dir_d5af2b93/valid_results/596a81fc0ea1-reference-gpu-pytorch-v2.2.2-cu126/pointpainting/singlestream/TEST01'  -o  '/home/mlcuser/MLC/repos/local/cache/get-mlperf-inference-results-dir_d5af2b93/valid_results/596a81fc0ea1-reference-gpu-pytorch-v2.2.2-cu126/pointpainting/singlestream'
Parsing arguments.
Verifying accuracy. This might take a while...
Reading accuracy mode results...
Reading performance mode results...
num_acc_log_entries = 39987
num_acc_log_duplicate_keys = 0
num_acc_log_data_mismatch = 0
num_perf_log_entries = 0
num_perf_log_qsl_idx_match = 0
num_perf_log_data_mismatch = 0
num_missing_qsl_idxs = 0
TEST FAIL

[hanyunfan]

@anandhu-eng could you help to take a look.

[anandhu-eng]

I'm unable to reproduce the exact error Arjun has experienced.

Here are my findings:

1. Verify accuracy of test-01 is passing

Verifying accuracy. This might take a while...
Reading accuracy mode results...
Reading performance mode results...
num_acc_log_entries = 39987   
num_acc_log_duplicate_keys = 0
num_acc_log_data_mismatch = 0 
num_perf_log_entries = 86
num_perf_log_qsl_idx_match = 86
num_perf_log_data_mismatch = 0
num_missing_qsl_idxs = 0
TEST PASS

2. The SingleStream performance run is getting over after the set value for min_duration due to which the run is processing lesser amount of queries than the generated_query_count by loadgen(8198 instead of 400001 generated by loadgen). Just wanted to confirm if this is an expected behavior? Do we have a proper documentation stating the behavior of config arguments like the min_duration?

mlperf_log_summary.txt

root@f169d3ac30fa:/workspace# cat /root/MLC/repos/local/cache/get-mlperf-inference-results-dir_5f29854a/valid_results_tmp3/6c80ff2bf1f7-reference-gpu-pytorch-v2.2.2-cu126/pointpainting/singlestream/TEST01/mlperf_log_summary.txt 
================================================
MLPerf Results Summary
================================================
SUT name : PySUT
Scenario : SingleStream
Mode     : PerformanceOnly
99.9th percentile latency (ns) : 880340777
Result is : VALID
  Min duration satisfied : Yes
  Min queries satisfied : Yes
  Early stopping satisfied: Yes
Early Stopping Result:
 * Processed at least 6636 queries (8198).
 * Would discard 0 highest latency queries.
 * Early stopping 99.9th percentile estimate: 1619852546
 * Early stopping 99.0th percentile estimate: 809879588

================================================
Additional Stats
================================================
QPS w/ loadgen overhead         : 2.05
QPS w/o loadgen overhead        : 2.08

Min latency (ns)                : 326276092
Max latency (ns)                : 1619852546
Mean latency (ns)               : 481041215
50.00 percentile latency (ns)   : 463879833
90.00 percentile latency (ns)   : 561518394
95.00 percentile latency (ns)   : 639295313
97.00 percentile latency (ns)   : 706540620
99.00 percentile latency (ns)   : 791369929
99.90 percentile latency (ns)   : 880340777

================================================
Test Parameters Used
================================================
samples_per_query : 1
target_qps : 50
target_latency (ns): 0
max_async_queries : 1
min_duration (ms): 4000000
max_duration (ms): 0
min_query_count : 100
max_query_count : 0
qsl_rng_seed : 1780908523862526354
sample_index_rng_seed : 14771362308971278857
schedule_rng_seed : 18209322760996052031
accuracy_log_rng_seed : 720381539243781796
accuracy_log_probability : 0
accuracy_log_sampling_target : 256
print_timestamps : 0
performance_issue_unique : 0
performance_issue_same : 0
performance_issue_same_index : 0
performance_sample_count : 1024

1 warning encountered. See detailed log.

No errors encountered during test.

generated query count from mlperf_log_details.txt

root@f169d3ac30fa:/workspace# cat /root/MLC/repos/local/cache/get-mlperf-inference-results-dir_5f29854a/valid_results_tmp3/6c80ff2bf1f7-reference-gpu-pytorch-v2.2.2-cu126/pointpainting/singlestream/TEST01/mlperf_log_detail.txt | grep generated_query_count
:::MLLOG {"key": "generated_query_count", "value": 400001, "time_ms": 194.579646, "namespace": "mlperf::logging", "event_type": "POINT_IN_TIME", "metadata": {"is_error": false, "is_warning": false, "file": "loadgen.cc", "line_no": 427, "pid": 90361, "tid": 90361}}

User conf

root@f169d3ac30fa:/workspace# cat /root/MLC/repos/local/cache/get-mlperf-inference-results-dir_5f29854a/valid_results_tmp3/6c80ff2bf1f7-reference-gpu-pytorch-v2.2.2-cu126/pointpainting/singlestream/performance/run_1/user.conf 
pointpainting.SingleStream.target_latency = 20
pointpainting.SingleStream.min_duration = 4000000 

[hanyunfan]

@arjunsuresh Could you take a look on this run. Anandhu mentioned he cannot duplicate the issue.

@pgmpablo157321 Could you help to take a look and check if the maximum duration set to zero is a expected behavior

[anandhu-eng]

I did another run with larger number of samples, one thing I noticed comparing the previous run was the num_perf_log_entries is 47 for the run with 29926 samples whereas for 8198 samples, it is 86.

Please find the logs below:

User conf:

pointpainting.SingleStream.target_latency = 100
pointpainting.SingleStream.min_duration = 15000000

mlperf log summary:

================================================
MLPerf Results Summary
================================================
SUT name : PySUT
Scenario : SingleStream
Mode     : PerformanceOnly
99.9th percentile latency (ns) : 922635271
Result is : VALID
  Min duration satisfied : Yes
  Min queries satisfied : Yes
  Early stopping satisfied: Yes
Early Stopping Result:
 * Processed at least 6636 queries (29926).
 * Would discard 16 highest latency queries.
 * Early stopping 99.9th percentile estimate: 964408524
 * Early stopping 99.0th percentile estimate: 839326173

================================================
Additional Stats
================================================
QPS w/ loadgen overhead         : 1.99
QPS w/o loadgen overhead        : 2.00

Min latency (ns)                : 340023455
Max latency (ns)                : 1641281348
Mean latency (ns)               : 499168056
50.00 percentile latency (ns)   : 476411066
90.00 percentile latency (ns)   : 581840059
95.00 percentile latency (ns)   : 715115379
97.00 percentile latency (ns)   : 763183978
99.00 percentile latency (ns)   : 833139615
99.90 percentile latency (ns)   : 922635271

================================================
Test Parameters Used
================================================
samples_per_query : 1
target_qps : 10
target_latency (ns): 0
max_async_queries : 1
min_duration (ms): 15000000
max_duration (ms): 0
min_query_count : 100
max_query_count : 0
qsl_rng_seed : 1780908523862526354
sample_index_rng_seed : 14771362308971278857
schedule_rng_seed : 18209322760996052031
accuracy_log_rng_seed : 720381539243781796
accuracy_log_probability : 0
accuracy_log_sampling_target : 256
print_timestamps : 0
performance_issue_unique : 0
performance_issue_same : 0
performance_issue_same_index : 0
performance_sample_count : 1024

1 warning encountered. See detailed log.

No errors encountered during test.

verify accuracy:

Verifying accuracy. This might take a while...
Reading accuracy mode results...
Reading performance mode results...
num_acc_log_entries = 39987
num_acc_log_duplicate_keys = 0
num_acc_log_data_mismatch = 0
num_perf_log_entries = 47
num_perf_log_qsl_idx_match = 47
num_perf_log_data_mismatch = 0
num_missing_qsl_idxs = 0
TEST PASS

run command(inside MLC docker):

mlcr run-mlperf,inference,_all-modes,_full,_r5.1-dev --model=pointpainting --implementation=reference --framework=pytorch --category=edge --scenario=SingleStream --execution_mode=valid --compliance=yes --device=cuda --quiet --singlestream_target_latency=100 --min_duration=15000000 --division=closed --env.MLC_MLPERF_USE_MAX_DURATION=no

[hanyunfan]

@arjunsuresh @pgmpablo157321 Could you take a look at the latest info @anandhu-eng provided and my precious comments

@arjunsuresh Could you take a look on this run. Anandhu mentioned he cannot duplicate the issue.

@pgmpablo157321 Could you help to take a look and check if the maximum duration set to zero is a expected behavior

[arjunsuresh]

@hanyunfan Sorry I missed the notification.

Thank you @anandhu-eng for testing.

"one thing I noticed comparing the previous run was the num_perf_log_entries is 47 for the run with 29926 samples whereas for 8198 samples, it is 86."

Is there any known reason for this behaviour? Could it be the case that for some number of samples the num_perf_log_entries become even 0?

[hanyunfan]

Thanks @arjunsuresh

I'm completely new to this workload, beside what Arjun is asking, I'm wondering could this be caused by the early stopping mechanism?