CPU and Max RSS Analysis tools #6663

ChrisPaulBennett · 2025-03-12T09:14:38Z

This apart of 3 pull requests for adding CPU time and Max RSS analysis to the Cylc UI.

This adds the Max RSS and CPU time (as measured by cgroups) to the table view, box plot and time series views.

This adds a python profiler script. This profiler will will be ran by cylc in the same crgroup as the cylc task. It will periodically poll cgroups and save data to a file. Cylc will then store these values in the sql db file.

Linked to;
cylc/cylc-ui#2100
cylc/cylc-uiserver#675

Check List

I have read CONTRIBUTING.md and added my name as a Code Contributor.
Contains logically grouped changes (else tidy your branch by rebase).
Does not contain off-topic changes (use other PRs for other changes).
Applied any dependency changes to both setup.cfg (and conda-environment.yml if present).
Tests are included (or explain why tests are not needed).
Changelog entry included if this is a change that can affect users
Cylc-Doc pull request opened if required at cylc/cylc-doc/pull/XXXX.
If this is a bug fix, PR should be raised against the relevant ?.?.x branch.

oliver-sanders

🎉

cylc/flow/etc/job.sh

cylc/flow/job_file.py

cylc/flow/etc/job.sh

cylc/flow/scripts/profile.py

tests/functional/jobscript/02-profiler.t

Changed the name of the profiler module. Linting Profiler sends KB instead of bytes Time Series now working CPU/Memory Logging working

Initial profiler implementation (non working) Changed the name of the profiler module. Linting Profiler sends KB instead of bytes Time Series now working CPU/Memory Logging working Adding profiler unit tests updating tests Fail gracefully if cgroups cannot be found Revert "Fail gracefully if cgroups cannot be found" This reverts commit 92e1e11c9b392b4742501d399f191f590814e95e. Linting Modifying unit tests Linting Changed the name of the profiler module. Profiler sends KB instead of bytes Time Series now working

Co-authored-by: Hilary James Oliver <[email protected]>

* The COPYING file appears to have moved from `dist-info/COPYING` into `dist-info/licenses/COPYING`.

Co-authored-by: Hilary James Oliver <[email protected]>

* Attempt to fix flaky test. * Cut out shell profile files to omit some spurious stderr.

Co-authored-by: Hilary James Oliver <[email protected]>

* Attempt to fix flaky test. * Cut out shell profile files to omit some spurious stderr.

Revert "Fail gracefully if cgroups cannot be found" This reverts commit 92e1e11c9b392b4742501d399f191f590814e95e. Adding profiler unit tests updating tests

…lc_profiler

oliver-sanders · 2025-08-18T15:07:13Z

cylc/flow/etc/job.sh

+###############################################################################
+# Save the data using cylc message and exit the profiler
+cylc__kill_profiler() {
+    if [[ -n "${profiler_pid:-}" && -d "/proc/${profiler_pid}" ]]; then


The POSIX standard is a set of interfaces that all compliant operating systems must follow (Mac OS included).

The /proc filesystem ain't POSIX, it's a Linux specific thing, hence the fun you had with Mac OS testing.

The most POSIX way to determine if a process is running that I could find is ps -p:

Suggested change

if [[ -n "${profiler_pid:-}" && -d "/proc/${profiler_pid}" ]]; then

if [[ -n "${profiler_pid:-}" ]] && ps -p "$profiler_pid" >/dev/null;

You can see the POSIX description for the ps command here:

https://pubs.opengroup.org/onlinepubs/9699919799/utilities/ps.html

The index of commands is here:

https://pubs.opengroup.org/onlinepubs/9699919799/utilities/

oliver-sanders · 2025-08-18T15:25:38Z

cylc/flow/scripts/profiler.py

+max_rss_location = None
+cpu_time_location = None
+cgroup_version = None
+comms_timeout = None


Unused global variables:

Suggested change

max_rss_location = None

cpu_time_location = None

cgroup_version = None

comms_timeout = None

oliver-sanders · 2025-08-18T15:26:09Z

cylc/flow/scripts/profiler.py

+    global comms_timeout
+    # Register the stop_profiler function with the signal library
+    signal.signal(signal.SIGINT, stop_profiler)
+    signal.signal(signal.SIGHUP, stop_profiler)
+    signal.signal(signal.SIGTERM, stop_profiler)
+
+    comms_timeout = options.comms_timeout


Unused global variables:

Suggested change

global comms_timeout

# Register the stop_profiler function with the signal library

signal.signal(signal.SIGINT, stop_profiler)

signal.signal(signal.SIGHUP, stop_profiler)

signal.signal(signal.SIGTERM, stop_profiler)

comms_timeout = options.comms_timeout

# Register the stop_profiler function with the signal library

signal.signal(signal.SIGINT, stop_profiler)

signal.signal(signal.SIGHUP, stop_profiler)

signal.signal(signal.SIGTERM, stop_profiler)

oliver-sanders · 2025-08-18T15:26:59Z

cylc/flow/scripts/profiler.py

+    # HPC uses cgroups v2 and SPICE uses cgroups v1
+    global cgroup_version
+    if Path.exists(Path(cgroup_location + cgroup_name)):
+        cgroup_version = 2
+        return cgroup_version
+    elif Path.exists(Path(cgroup_location + "/memory" + cgroup_name)):
+        cgroup_version = 1
+        return cgroup_version
+    else:
+        raise FileNotFoundError("Cgroup not found at " +
+                                cgroup_location + cgroup_name)


Unused global variables:

(also Met Office specific comment)

Suggested change

# HPC uses cgroups v2 and SPICE uses cgroups v1

global cgroup_version

if Path.exists(Path(cgroup_location + cgroup_name)):

cgroup_version = 2

return cgroup_version

elif Path.exists(Path(cgroup_location + "/memory" + cgroup_name)):

cgroup_version = 1

return cgroup_version

else:

raise FileNotFoundError("Cgroup not found at " +

cgroup_location + cgroup_name)

if Path.exists(Path(cgroup_location + cgroup_name)):

return 2

elif Path.exists(Path(cgroup_location + "/memory" + cgroup_name)):

return 1

else:

raise FileNotFoundError("Cgroup not found at " +

cgroup_location + cgroup_name)

oliver-sanders · 2025-08-18T15:27:15Z

cylc/flow/scripts/profiler.py

+def get_cgroup_name():
+    """Get the cgroup directory for the current process"""
+
+    # fugly hack to allow functional tests to use test data


oliver-sanders · 2025-08-18T15:27:38Z

cylc/flow/scripts/profiler.py

+    global max_rss_location
+    global cpu_time_location
+    if version == 2:
+        max_rss_location = location + name + "/" + "memory.peak"
+        cpu_time_location = location + name + "/" + "cpu.stat"
+        return Process(
+            cgroup_memory_path=location +
+            name + "/" + "memory.peak",
+            cgroup_cpu_path=location +
+            name + "/" + "cpu.stat")
+
+    elif version == 1:
+        max_rss_location = (location + "/memory" +
+                            name + "/memory.max_usage_in_bytes")
+        cpu_time_location = (location + "/cpu" +
+                             name + "/cpuacct.usage")
+        return Process(
+            cgroup_memory_path=location + "/memory" +
+            name + "/memory.max_usage_in_bytes",
+            cgroup_cpu_path=location + "/cpu" +
+            name + "/cpuacct.usage")


Unused global variables:

Suggested change

global max_rss_location

global cpu_time_location

if version == 2:

max_rss_location = location + name + "/" + "memory.peak"

cpu_time_location = location + name + "/" + "cpu.stat"

return Process(

cgroup_memory_path=location +

name + "/" + "memory.peak",

cgroup_cpu_path=location +

name + "/" + "cpu.stat")

elif version == 1:

max_rss_location = (location + "/memory" +

name + "/memory.max_usage_in_bytes")

cpu_time_location = (location + "/cpu" +

name + "/cpuacct.usage")

return Process(

cgroup_memory_path=location + "/memory" +

name + "/memory.max_usage_in_bytes",

cgroup_cpu_path=location + "/cpu" +

name + "/cpuacct.usage")

if version == 2:

return Process(

cgroup_memory_path=location +

name + "/" + "memory.peak",

cgroup_cpu_path=location +

name + "/" + "cpu.stat")

elif version == 1:

return Process(

cgroup_memory_path=location + "/memory" +

name + "/memory.max_usage_in_bytes",

cgroup_cpu_path=location + "/cpu" +

name + "/cpuacct.usage")

These global variables are used. They are used in the stop_profiler function.
I can't see a way to pass it arguments when it is called by the signal library. Using globals is the only way I can see to do it

ChrisPaulBennett marked this pull request as draft March 12, 2025 09:19

oliver-sanders reviewed Mar 12, 2025

View reviewed changes

oliver-sanders added this to the 8.x milestone Mar 12, 2025

oliver-sanders assigned ChrisPaulBennett Mar 12, 2025

This was referenced Mar 13, 2025

CPU and Max RSS Analysis tools cylc/cylc-ui#2100

Open

CPU and Max RSS Analysis tools cylc/cylc-uiserver#675

Open

ChrisPaulBennett force-pushed the cylc_profiler branch 2 times, most recently from fb1b12b to c5d30b3 Compare March 21, 2025 11:37

Initial profiler implementation (non working)

246a4c1

Changed the name of the profiler module. Linting Profiler sends KB instead of bytes Time Series now working CPU/Memory Logging working

ChrisPaulBennett force-pushed the cylc_profiler branch 2 times, most recently from 4a6dbe8 to 30a7bb0 Compare April 1, 2025 15:31

ChrisPaulBennett and others added 19 commits April 2, 2025 09:34

GH Actions: use explicit bash shell & other defaults

171f7ee

host-select: fix compatibility with force-condemned hosts

7d50d0b

Update cylc/flow/cfgspec/globalcfg.py

8608088

Co-authored-by: Hilary James Oliver <[email protected]>

actions: update build action to support python 3.13

060056b

* The COPYING file appears to have moved from `dist-info/COPYING` into `dist-info/licenses/COPYING`.

host-select: fix compatibility with force-condemned hosts

a0cdcd6

Update cylc/flow/cfgspec/globalcfg.py

3d4a187

Co-authored-by: Hilary James Oliver <[email protected]>

tests: convert unittest to pytest

3057edd

tests/u: test_subprocpool.py::test_run_command_writes_to_err

a6ecd29

* Attempt to fix flaky test. * Cut out shell profile files to omit some spurious stderr.

Time Series now working

e5119b4

Profiler sends KB instead of bytes

9f593fc

Modifying unit tests

5f84c2f

host-select: fix compatibility with force-condemned hosts

47a4a98

Update cylc/flow/cfgspec/globalcfg.py

356abff

Co-authored-by: Hilary James Oliver <[email protected]>

tests: convert unittest to pytest

6ed1770

tests/u: test_subprocpool.py::test_run_command_writes_to_err

4cfcffe

* Attempt to fix flaky test. * Cut out shell profile files to omit some spurious stderr.

Linting

1e5b804

Fail gracefully if cgroups cannot be found

38c31f5

Revert "Fail gracefully if cgroups cannot be found" This reverts commit 92e1e11c9b392b4742501d399f191f590814e95e. Adding profiler unit tests updating tests

tests: convert unittest to pytest

339af9c

ChrisPaulBennett added 4 commits July 9, 2025 13:25

Kill profiler more reliably

15c29fa

Added unit test coverage

c09aa6f

linting

a29cf82

linting

172b453

ChrisPaulBennett requested a review from oliver-sanders July 10, 2025 12:59

ChrisPaulBennett added 23 commits July 21, 2025 10:11

Corrected cgroup versions

59200fc

Merge branch 'refs/heads/master' into cylc_profiler

35daea9

Updating unit tests

9189ac3

Updating unit tests

31f2c18

Merge branch 'refs/heads/master' into cylc_profiler

baaa1a6

Updating unit tests

038348a

Updating unit tests

2975ea4

Merge branch 'master' into cylc_profiler

0448e70

Merge branch 'master' into cylc_profiler

3cff4a1

Merge remote-tracking branch 'ChrisPaulBennett/cylc_profiler' into cy…

c8f3ab5

…lc_profiler

updating .mailmap

5e994fb

Updating .mailmap

5c8585e

testing macos

0c68cec

testing macos

7eddcf2

testing macos

2cd9991

testing macos

c1a5686

testing macos

2bc9ec6

testing macos

8ca760d

testing macos

ebbb8f0

testing macos

6f65ebd

testing macos

54553c1

testing macos

556f3ba

testing macos

f70c1d3

oliver-sanders reviewed Aug 18, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

CPU and Max RSS Analysis tools #6663

CPU and Max RSS Analysis tools #6663

Uh oh!

ChrisPaulBennett commented Mar 12, 2025 •

edited by oliver-sanders

Loading

Uh oh!

oliver-sanders left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

oliver-sanders Aug 18, 2025

Uh oh!

oliver-sanders Aug 18, 2025

Uh oh!

oliver-sanders Aug 18, 2025

Uh oh!

oliver-sanders Aug 18, 2025

Uh oh!

oliver-sanders Aug 18, 2025

Uh oh!

oliver-sanders Aug 18, 2025

Uh oh!

ChrisPaulBennett Sep 12, 2025

Uh oh!

Uh oh!

	if [[ -n "${profiler_pid:-}" && -d "/proc/${profiler_pid}" ]]; then
	if [[ -n "${profiler_pid:-}" ]] && ps -p "$profiler_pid" >/dev/null;

CPU and Max RSS Analysis tools #6663

Are you sure you want to change the base?

CPU and Max RSS Analysis tools #6663

Uh oh!

Conversation

ChrisPaulBennett commented Mar 12, 2025 • edited by oliver-sanders Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

oliver-sanders left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

oliver-sanders Aug 18, 2025

Choose a reason for hiding this comment

Uh oh!

oliver-sanders Aug 18, 2025

Choose a reason for hiding this comment

Uh oh!

oliver-sanders Aug 18, 2025

Choose a reason for hiding this comment

Uh oh!

oliver-sanders Aug 18, 2025

Choose a reason for hiding this comment

Uh oh!

oliver-sanders Aug 18, 2025

Choose a reason for hiding this comment

Uh oh!

oliver-sanders Aug 18, 2025

Choose a reason for hiding this comment

Uh oh!

ChrisPaulBennett Sep 12, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ChrisPaulBennett commented Mar 12, 2025 •

edited by oliver-sanders

Loading