Profiling CESM with Intel VTune on cheyenne

1. Generating a profiling report

After having created a CESM case, make the following changes:

• Change the relevant executable command, e.g., mpiexec_mpt, in env_mach_specific.xml as follows:

amplxe-cl --collect=hotspots --result-dir=result mpiexec_mpt

• Then execute:

$ ./case.setup --keep env_mach_specific.xml
$ ./case.build

Run the case after the above changes. A result directory should appear in RUNDIR while the case is running. This directory includes the raw profiling data generated by Vtune.

Note: You can add options to the above amplxe-cl command to customize the generated reports. To get the full list of options: $ amplxe-cl -help collect

Note 2: Normally, one should be able to control the PEs that are being profiled via the options like -cpu-mask and -target-pid, but these may not work on cheyenne, possibly because Vtune doesn't like the default MPI library, mpt, on cheyenne. A workaround is to set the ROOTPE of the target component(s) to 0, and assign all other components to a different PE group. (Note: Edit this wiki accordingly if you have successfully used the -cpu-mask or -target-pid options on cheyenne.)

2. Analyzing the report

After having run the CESM case with the profiler, there are several options to analyze the report:

1. GUI: run the amplxe-gui command on a cheyenne terminal session and open the generated report.
2. Or, CLI: To generate a human-readable text file (result.txt), for instance, run the following command in the RUNDIR:

amplxe-cl --report=gprof-cc --result-dir=result --format=text --report-output=result.txt

3. Or, create a simple visual call graph: https://software.intel.com/content/www/us/en/develop/articles/making-visual-call-graphs-from-intel-vtune-amplifier-output.html

---

A comprehensive Intel VTune documentation: https://software.intel.com/content/www/us/en/develop/tools/oneapi/components/vtune-profiler.html#gs.4kxg06