Skip to content

Detect and report invalid predefined lists#2507

Closed
iarspider wants to merge 25 commits into
masterfrom
run-ib-pr-matrix-list
Closed

Detect and report invalid predefined lists#2507
iarspider wants to merge 25 commits into
masterfrom
run-ib-pr-matrix-list

Conversation

@iarspider
Copy link
Copy Markdown
Contributor

No description provided.

@cmsbuild
Copy link
Copy Markdown
Contributor

cmsbuild commented May 16, 2025

cms-bot internal usage

@iarspider iarspider force-pushed the run-ib-pr-matrix-list branch from 2df6ad6 to e66deb5 Compare May 19, 2025 09:10
@cmsbuild
Copy link
Copy Markdown
Contributor

Pull request #2507 was updated.

@iarspider
Copy link
Copy Markdown
Contributor Author

please test workflows 1.0,invalidlist

@cmsbuild
Copy link
Copy Markdown
Contributor

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-aab350/46236/summary.html
COMMIT: e66deb5
CMSSW: CMSSW_15_1_X_2025-05-18-2300/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cms-bot/2507/46236/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

  • No significant changes to the logs found
  • Reco comparison results: 8 differences found in the comparisons
  • DQMHistoTests: Total files compared: 50
  • DQMHistoTests: Total histograms compared: 4038193
  • DQMHistoTests: Total failures: 3
  • DQMHistoTests: Total nulls: 0
  • DQMHistoTests: Total successes: 4038170
  • DQMHistoTests: Total skipped: 20
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.0 KiB( 49 files compared)
  • Checked 218 log files, 188 edm output root files, 50 DQM output files
  • TriggerResults: no differences found

@iarspider
Copy link
Copy Markdown
Contributor Author

please test workflows 1.0,invalidlist

@cmsbuild
Copy link
Copy Markdown
Contributor

Pull request #2507 was updated.

@iarspider
Copy link
Copy Markdown
Contributor Author

please abort

@cmsbuild
Copy link
Copy Markdown
Contributor

Pull request #2507 was updated.

@iarspider
Copy link
Copy Markdown
Contributor Author

please test workflows 1.0,invalidlist

@cmsbuild
Copy link
Copy Markdown
Contributor

Pull request #2507 was updated.

@cmsbuild
Copy link
Copy Markdown
Contributor

-1

Failed Tests: RelVals
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-aab350/46248/summary.html
COMMIT: 9097e31
CMSSW: CMSSW_15_1_X_2025-05-19-2300/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cms-bot/2507/46248/install.sh to create a dev area with all the needed externals and cmssw changes.

RelVals

ERROR Running runTheMatrix for '-s -l 9.0,101.0,1306.0,10224.0,25202.0,250202.181'
+ echo 'ERROR : all workflow lists were not recognized, and no additional workflows were requested'
ERROR : all workflow lists were not recognized, and no additional workflows were requested
ERROR Running runTheMatrix for '-l invalidlist,1.0 '

@cmsbuild
Copy link
Copy Markdown
Contributor

cmsbuild commented Aug 8, 2025

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-aab350/47606/summary.html
COMMIT: 88d224d
CMSSW: CMSSW_15_1_X_2025-08-07-2300/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cms-bot/2507/47606/install.sh to create a dev area with all the needed externals and cmssw changes.

Invalid workflow lists

The following workflow names were not recognized and not ran:

  • invalidlist

Comparison Summary

The workflows 2500.3001 have different files in step1_dasquery.log than the ones found in the baseline. You may want to check and retrigger the tests if necessary. You can check it in the "files" directory in the results of the comparisons

Summary:

  • You potentially removed 1 lines from the logs
  • Reco comparison results: 0 differences found in the comparisons
  • DQMHistoTests: Total files compared: 50
  • DQMHistoTests: Total histograms compared: 4076122
  • DQMHistoTests: Total failures: 35
  • DQMHistoTests: Total nulls: 0
  • DQMHistoTests: Total successes: 4076067
  • DQMHistoTests: Total skipped: 20
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.0 KiB( 49 files compared)
  • Checked 218 log files, 188 edm output root files, 50 DQM output files
  • TriggerResults: no differences found

@iarspider
Copy link
Copy Markdown
Contributor Author

Invalid workflow lists

The following workflow names were not recognized and not ran:

* invalidlist

@smuzaffar

@smuzaffar
Copy link
Copy Markdown
Contributor

please test workflows invalidlist,9999999.0

@cmsbuild
Copy link
Copy Markdown
Contributor

-1

Failed Tests: RelVals
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-aab350/47641/summary.html
COMMIT: 88d224d
CMSSW: CMSSW_15_1_X_2025-08-10-0000/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cms-bot/2507/47641/install.sh to create a dev area with all the needed externals and cmssw changes.

RelVals

ValueError: Undefined workflows: 9999999.0'

@cmsbuild
Copy link
Copy Markdown
Contributor

Pull request #2507 was updated.

@iarspider
Copy link
Copy Markdown
Contributor Author

please test workflows invalidlist,9999999.0

@cmsbuild
Copy link
Copy Markdown
Contributor

-1

Failed Tests: RelVals
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-aab350/47651/summary.html
COMMIT: dd37cd0
CMSSW: CMSSW_15_1_X_2025-08-11-1100/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cms-bot/2507/47651/install.sh to create a dev area with all the needed externals and cmssw changes.

RelVals

ValueError: Undefined workflows: 9999999.0

@smuzaffar
Copy link
Copy Markdown
Contributor

smuzaffar commented Aug 11, 2025

test parameters:

  • baseline = self
  • enable = nano,gpu

@smuzaffar
Copy link
Copy Markdown
Contributor

please test workflows invalidlist,9999999.0 for CMSSW_15_1_DEVEL_X

@smuzaffar
Copy link
Copy Markdown
Contributor

please test workflows invalidlist,1.0,2.0 for CMSSW_15_1_ROOT636_X

@cmsbuild
Copy link
Copy Markdown
Contributor

Pull request #2507 was updated.

@smuzaffar
Copy link
Copy Markdown
Contributor

please test workflows invalidlist,1.0,2.0,3.0 for CMSSW_15_1_ROOT636_X

@cmsbuild
Copy link
Copy Markdown
Contributor

Pull request #2507 was updated.

@cmsbuild
Copy link
Copy Markdown
Contributor

Pull request #2507 was updated.

@smuzaffar
Copy link
Copy Markdown
Contributor

please test workflows invalidlist,1.0,2.0,3.0 for CMSSW_15_1_ROOT6_X

…gnize the gpu,nano workflows with need -w name opt
@smuzaffar
Copy link
Copy Markdown
Contributor

please test workflows invalidlist,1.0,2.0,3.0 for CMSSW_15_1_ROOT6_X

@cmsbuild
Copy link
Copy Markdown
Contributor

Pull request #2507 was updated.

@cmsbuild
Copy link
Copy Markdown
Contributor

Pull request #2507 was updated.

@smuzaffar
Copy link
Copy Markdown
Contributor

please test workflows invalidlist,1.0,2.0,3.0 for CMSSW_15_1_GEANT4_X

@smuzaffar
Copy link
Copy Markdown
Contributor

please test workflows invalidlist,1.0,2.0,3.0 for CMSSW_15_0_X

@cmsbuild
Copy link
Copy Markdown
Contributor

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-aab350/47687/summary.html
COMMIT: 825ff8e
CMSSW: CMSSW_15_0_X_2025-08-11-2300/el8_amd64_gcc12
Additional Tests: CUDA,NANO,ROCM
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cms-bot/2507/47687/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

CUDA Comparison Summary

Summary:

NANO Comparison Summary

Summary:

  • You potentially added 388 lines to the logs
  • ROOTFileChecks: Some differences in event products or their sizes found
  • Reco comparison results: 0 differences found in the comparisons
  • DQMHistoTests: Total files compared: 22
  • DQMHistoTests: Total histograms compared: 88860
  • DQMHistoTests: Total failures: 0
  • DQMHistoTests: Total nulls: 0
  • DQMHistoTests: Total successes: 88860
  • DQMHistoTests: Total skipped: 0
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.0 KiB( 21 files compared)
  • Checked 134 log files, 80 edm output root files, 22 DQM output files
  • TriggerResults: no differences found

ROCM Comparison Summary

Summary:

  • No significant changes to the logs found
  • Reco comparison results: 60 differences found in the comparisons
  • DQMHistoTests: Total files compared: 7
  • DQMHistoTests: Total histograms compared: 53444
  • DQMHistoTests: Total failures: 3936
  • DQMHistoTests: Total nulls: 0
  • DQMHistoTests: Total successes: 49508
  • DQMHistoTests: Total skipped: 0
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.0 KiB( 6 files compared)
  • Checked 24 log files, 30 edm output root files, 7 DQM output files
  • TriggerResults: found differences in 3 / 6 workflows

Nano size comparison Summary:

Sample kb/ev ref kb/ev diff kb/ev ev/s/thd ref ev/s/thd diff rate mem/thd ref mem/thd
2500.001_NANOmc106Xul16v2 3.085 3.085 0.000 ( +0.0% ) 3.85 3.73 +3.3% 2.705 2.601
2500.002_NANOmc106Xul17v2 3.144 3.144 0.000 ( +0.0% ) 3.60 3.51 +2.4% 3.187 3.053
2500.003_NANOmc106Xul18v2 3.097 3.097 0.000 ( +0.0% ) 3.65 3.58 +2.1% 3.180 3.044
2500.011_NANOdata106Xul16v2 1.539 1.539 0.000 ( +0.0% ) 6.44 6.20 +3.9% 2.825 2.665
2500.012_NANOdata106Xul17v2 1.925 1.925 0.000 ( +0.0% ) 4.27 4.14 +3.2% 3.032 2.819
2500.013_NANOdata106Xul18v2 1.890 1.890 0.000 ( +0.0% ) 5.13 4.96 +3.6% 2.938 2.742
2500.021_NANOmcUL16APVreMINI 0.022 0.022 0.000 ( +0.0% ) 1.87 1.81 +3.5% 2.941 2.554
2500.022_NANOmcUL16reMINI 0.022 0.022 0.000 ( +0.0% ) 1.79 1.70 +5.6% 2.940 2.536
2500.023_NANOmcUL17reMINI 0.022 0.022 0.000 ( +0.0% ) 1.61 1.56 +3.3% 2.662 2.518
2500.024_NANOmcUL18reMINI 0.022 0.022 0.000 ( +0.0% ) 1.43 1.38 +4.0% 3.067 2.701
2500.031_NANOdataUL16APVreMINI 0.035 0.035 0.000 ( +0.0% ) 1.63 1.55 +4.9% 2.884 2.651
2500.032_NANOdataUL16reMINI 0.036 0.036 0.000 ( +0.0% ) 1.61 1.56 +3.7% 2.881 2.566
2500.033_NANOdataUL17reMINI 0.037 0.037 0.000 ( +0.0% ) 1.57 1.48 +6.1% 2.932 2.690
2500.034_NANOdataUL18reMINI 0.036 0.036 0.000 ( +0.0% ) 1.57 1.49 +5.5% 2.918 2.674
2500.101_NANOmc130X 3.220 3.220 0.000 ( +0.0% ) 4.40 4.27 +3.2% 3.115 2.983
2500.111_NANOdata130Xrun3 1.799 1.799 0.000 ( +0.0% ) 6.51 6.39 +2.0% 2.813 2.691
2500.131_ScoutingNANOdata130Xrun3 0.771 0.771 0.000 ( +0.0% ) 33.88 33.12 +2.3% 1.661 1.646
2500.221_muPOGNANOmc140X 2.065 2.065 0.000 ( +0.0% ) 5.76 5.53 +4.2% 2.245 2.028
2500.222_EGMNANOmc140X 3.876 3.876 0.000 ( +0.0% ) 4.09 3.94 +4.0% 2.603 2.322
2500.223_BTVNANOmc140X 10.607 10.607 0.000 ( +0.0% ) 2.60 2.52 +3.3% 2.820 2.517
2500.224_jmeNANOmc140X 6.666 6.666 0.000 ( +0.0% ) 1.16 1.13 +3.2% 2.680 2.422
2500.226_lepTrackInfoNANOmc140X 3.570 3.570 0.000 ( +0.0% ) 4.14 3.94 +5.0% 2.604 2.331
2500.227_ScoutingNANOmc140X 1.486 1.486 0.000 ( +0.0% ) 15.89 15.55 +2.2% 1.930 1.850
2500.228_ScoutingNANOwithPromptmc140X 4.474 4.474 0.000 ( +0.0% ) 3.42 3.30 +3.6% 2.900 2.708
2500.231_muPOGNANO140Xrun3 1.469 1.469 0.000 ( +0.0% ) 8.17 7.72 +5.8% 2.571 2.313
2500.232_EGMNANOdata140Xrun3 2.623 2.623 0.000 ( +0.0% ) 6.00 5.66 +6.0% 2.931 2.642
2500.233_BTVNANOdata140Xrun3 5.612 5.612 0.000 ( +0.0% ) 3.80 3.61 +5.2% 3.153 2.819
2500.234_jmeNANOdata140Xrun3 3.976 3.976 0.000 ( +0.0% ) 1.50 1.44 +4.2% 2.975 2.649
2500.236_lepTrackInfoNANOdata140Xrun3 2.413 2.413 0.000 ( +0.0% ) 5.95 5.68 +4.9% 2.911 2.620
2500.237_ScoutingNANOdata140Xrun3 1.101 1.101 0.000 ( +0.0% ) 30.11 29.01 +3.8% 1.651 1.476
2500.238_ScoutingNANOwithPromptdata140Xrun3 2.783 2.783 0.000 ( +0.0% ) 5.23 5.02 +4.1% 3.141 2.941
2500.241_l1DPGNANO140Xrun3 9.404 9.404 0.000 ( +0.0% ) 7.53 7.08 +6.3% 1.931 1.801
2500.242_muDPGNANO140Xrun3 10.331 10.331 0.000 ( +0.0% ) 1.68 1.61 +4.5% 1.727 1.673
2500.243_muDPGNANOBkg140Xrun3 2.712 2.712 0.000 ( +0.0% ) 12.76 15.40 -17.2% 1.063 1.034
2500.244_hcalDPGNANO140Xrun3 486.016 486.016 0.000 ( +0.0% ) 1.13 1.13 -0.0% 1.689 1.561
2500.245_hcalDPGCalibNANO140Xrun3 826.413 826.413 0.000 ( +0.0% ) 1.53 1.52 +0.9% 1.666 1.550
2500.251_hcalDPGMCNANO140Xrun3 645.333 645.333 0.000 ( +0.0% ) 1.67 1.65 +1.4% 1.780 1.721
2500.261_NANOmc2024reMINI 0.021 0.021 0.000 ( +0.0% ) 1.62 1.61 +0.8% 3.093 2.712
2500.271_NANOdata2024reMINI 0.036 0.036 0.000 ( +0.0% ) 1.66 1.62 +2.8% 2.879 2.652
2500.301_NANOmc150X 3.217 3.217 0.000 ( +0.0% ) 18.75 17.99 +4.2% 2.706 2.541
2500.311_NANOdata150X 2.144 2.144 0.000 ( +0.0% ) 37.27 35.22 +5.8% 2.442 2.205
2500.312_NANOdata150X 2.630 2.630 0.000 ( +0.0% ) 30.32 27.14 +11.7% 2.529 2.247
2500.321_muPOGNANOmc150X 2.255 2.255 0.000 ( +0.0% ) 16.77 16.26 +3.1% 2.593 2.360
2500.322_EGMNANOmc150X 4.111 4.111 0.000 ( +0.0% ) 15.35 14.79 +3.8% 2.691 2.433
2500.323_BTVNANOmc150X 10.552 10.552 0.000 ( +0.0% ) 5.11 4.85 +5.2% 2.915 2.627
2500.324_jmeNANOmc150X 6.760 6.760 0.000 ( +0.0% ) 1.41 1.34 +5.4% 2.879 2.594
2500.325_jmeNANOrePuppimc150X 6.763 6.763 0.000 ( +0.0% ) 1.30 1.28 +1.8% 3.155 2.942
2500.326_lepTrackInfoNANOmc150X 3.721 3.721 0.000 ( +0.0% ) 16.33 15.62 +4.5% 2.693 2.599
2500.327_ScoutingNANOmc150X 1.562 1.562 0.000 ( +0.0% ) 19.66 19.16 +2.6% 2.455 2.354
2500.328_ScoutingNANOwithPromptmc150X 4.774 4.774 0.000 ( +0.0% ) 10.87 10.48 +3.7% 2.933 2.688
2500.329_BPHNANOmc150X 4.109 4.109 0.000 ( +0.0% ) 17.84 17.00 +4.9% 2.752 2.519
2500.331_muPOGNANO150Xrun3 1.125 1.125 0.000 ( +0.0% ) 32.85 28.84 +13.9% 2.358 2.086
2500.332_EGMNANOdata150Xrun3 2.985 2.985 0.000 ( +0.0% ) 26.99 25.28 +6.8% 2.442 2.115
2500.333_BTVNANOdata150Xrun3 7.971 7.971 0.000 ( +0.0% ) 6.18 5.82 +6.2% 2.662 2.318
2500.334_jmeNANOdata150Xrun3 4.349 4.349 0.000 ( +0.0% ) 1.71 1.65 +3.9% 2.602 2.337
2500.335_jmeNANOrePuppidata150Xrun3 4.332 4.332 0.000 ( +0.0% ) 1.46 1.42 +2.9% 2.874 2.568
2500.336_lepTrackInfoNANOdata150Xrun3 2.625 2.625 0.000 ( +0.0% ) 28.97 27.61 +4.9% 2.433 2.109
2500.337_ScoutingNANOdata150Xrun3 1.737 1.737 0.000 ( +0.0% ) 31.92 30.90 +3.3% 1.665 1.485
2500.338_ScoutingNANOwithPromptdata150Xrun3 2.601 2.601 0.000 ( +0.0% ) 18.24 17.64 +3.4% 2.703 2.461
2500.339_BPHNANOdata150Xrun3 2.604 2.604 0.000 ( +0.0% ) 35.83 34.33 +4.4% 2.488 2.268
2500.901_TTbarMINIAOD10.6_UL18v2 1.819 1.819 0.000 ( +0.0% ) 35.28 32.99 +6.9% 1.878 1.793
2500.902_TTbarMINIAOD14.0 1.665 1.665 0.000 ( +0.0% ) 33.35 30.04 +11.0% 1.458 1.456
2500.911_DYToLL_M-50_13TeV_pythia8 14.345 14.345 0.000 ( +0.0% ) 8.51 8.30 +2.5% 1.091 1.047
2500.912_DYToll01234Jets_5f_LO_MLM_Madgraph_LHE_13TeV 0.310 0.171 0.140 ( +81.9% ) 2.06 1.98 +3.8% 0.847 0.849
2500.913_TTbar_Pow_LHE_13TeV 0.110 0.110 0.000 ( +0.0% ) 2.73 2.64 +3.4% 0.852 0.852

@smuzaffar
Copy link
Copy Markdown
Contributor

closing this in favor of #2547

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants