From 583515d2402809311a05f4dc53752501db1a83be Mon Sep 17 00:00:00 2001 From: Oliver Sanders Date: Thu, 4 Sep 2025 16:59:51 +0100 Subject: [PATCH 1/4] improve broadcast documentation --- src/dictionaries/words | 1 + src/tutorial/furthertopics/broadcast.rst | 11 ++ .../running-workflows/cylc-broadcast.rst | 135 ++++++++++++++++++ .../running-workflows/dynamic-behaviour.rst | 21 --- src/user-guide/running-workflows/index.rst | 2 +- 5 files changed, 148 insertions(+), 22 deletions(-) create mode 100644 src/user-guide/running-workflows/cylc-broadcast.rst delete mode 100644 src/user-guide/running-workflows/dynamic-behaviour.rst diff --git a/src/dictionaries/words b/src/dictionaries/words index d11139ea7c..c1c323fbb8 100644 --- a/src/dictionaries/words +++ b/src/dictionaries/words @@ -12,6 +12,7 @@ basename baz boolean booleans +broadcasted cfg changelog changelogs diff --git a/src/tutorial/furthertopics/broadcast.rst b/src/tutorial/furthertopics/broadcast.rst index d949c7c335..47956944f5 100644 --- a/src/tutorial/furthertopics/broadcast.rst +++ b/src/tutorial/furthertopics/broadcast.rst @@ -135,3 +135,14 @@ e.g:: Stop the workflow:: cylc stop tutorial-broadcast + + +Further Reading +--------------- + +The :ref:`cylc-broadcast` section in the user guide contains more detail +on broadcasts. + +The GUI contains an "Edit Runtime" utility which loads the task's configuration +into a form. Any changes you make are then broadcasted to the task. +See :ref:`interventions.edit-a-tasks-configuration`. diff --git a/src/user-guide/running-workflows/cylc-broadcast.rst b/src/user-guide/running-workflows/cylc-broadcast.rst new file mode 100644 index 0000000000..8ed09f83fd --- /dev/null +++ b/src/user-guide/running-workflows/cylc-broadcast.rst @@ -0,0 +1,135 @@ +.. _cylc-broadcast: + +Cylc Broadcast +============== + +Cylc "Broadcasts" allow us to override task's :cylc:conf:`[runtime]` +settings within a running workflow. + +Broadcasts can target specific cycles, families or tasks. + +Broadcasts can be helpful for: + +* Quickly developing tasks without having to + :ref:`edit and reload the workflow configuration `. +* Sending small amounts of data from a running task to other upcoming tasks (e.g. file paths). +* Orchestrating production workflows from an external system. + +Broadcasts which target specific cycles will eventually be +:ref:`expired ` when no longer needed. + +Otherwise, broadcasts last for the life of the workflow and will persist if +the workflow is shutdown and restarted (unless manually "cleared"). + +.. seealso:: + + :ref:`broadcast-tutorial` + + +Issuing Broadcasts +------------------ + +CLI +^^^ + +Some examples of issuing broadcasts using the ``cylc broacast`` CLI command: + +.. code-block:: bash + + # set or update the environment variable "ANSWER" for all tasks + cylc broadcast myworkflow -s '[environment]ANSWER=42' + + # amend the directives for all tasks in the cycle "2000" + cylc broadcast myworkflow -p 2000 -s '[directives]--memory=2GB' + + # change the platform all tasks in the family "FOO" will submit on + cylc broadcast myworkflow -n FOO -s 'platform=my-hpc' + +The ``cylc broadcast`` command can be run from within tasks as a means to +communicate small amounts of data back to the scheduler for subsequent tasks to +use. Note that this will not work for remote task platforms which have been +configured to use polling - +:cylc:conf:`global.cylc[platforms][]communication method = poll`. + +For more information, run ``cylc broadcast --help``. + + +GUI +^^^ + +Broadcasts can also be issued from the GUI in a similar way using the +"Broadcast" command. + +Additionally, the GUI provides a utility called "Edit Runtime" which loads +the tasks configuration into a form. Any changes you make are then broadcasted +to the task: + +.. image:: ../interventions/edit-a-tasks-configuration.gui.gif + :width: 75% + :align: center + +| + + +.. _user_guide.broadcast.expiry: + +Expiry +------ + +Broadcasts which target specific cycles will eventually expire (i.e. be +deleted) as the workflow moves on. Otherwise they would gradually accumulate +over the life of the workflow (note broadcasts are persisted when the workflow +restarts). + + +Expiry Point +^^^^^^^^^^^^ + +Broadcasts are only expired after they are no longer required by tasks. +The exact point at which a broadcast is expired depends on two things: + +* The oldest cycle in the workflow to contain + :term:`active tasks `. +* The longest cycling :term:`recurrence` in the workflow. + +Broadcasts which are older than the oldest cycle point to contain active tasks +*minus* the duration of the longest recurrence will be cleared. + +For example, for the following workflow: + +.. code-block:: cylc + + [scheduling] + [[graph]] + P1Y = foo + P2Y = bar + P3Y = baz + +The longest cycling recurrence is ``P3Y``. + +If there were no more tasks left running in the cycle ``2000``, then broadcasts +for cycles earlier than ``1997`` (``2000 - P3Y``) would be expired. + +This arrangement has been designed such that broadcasts should always be +present for the previous instance of a task in case you want to re-run it. + + +Broadcasting To Historical Cycles +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Broadcasts targeting historical cycles may be expired as soon as they are +issued as the result of broadcast expiry. + +Broadcast expiry does not occur while the workflow is paused. If you want to +broadcast to a historical cycle before re-running it, first pause the workflow, +then trigger the tasks, then resume the workflow, e.g: + +.. code-block:: bash + + cylc pause my-workflow + cylc broadcast my-workflow -p 2000 -s ... + cylc trigger my-workflow + cylc play my-workflow + + +.. TODO: document sub-workflows diff --git a/src/user-guide/running-workflows/dynamic-behaviour.rst b/src/user-guide/running-workflows/dynamic-behaviour.rst deleted file mode 100644 index 0b892513b3..0000000000 --- a/src/user-guide/running-workflows/dynamic-behaviour.rst +++ /dev/null @@ -1,21 +0,0 @@ -.. _cylc-broadcast: - -Cylc Broadcast --------------- - -The ``cylc broadcast`` command overrides task :cylc:conf:`[runtime]` -settings in a running scheduler. You can think of it as communicating -new configuration settings (including information via environment variables) to -selected upcoming tasks via the scheduler. - -See ``cylc broadcast --help`` for detailed information. - -Broadcast settings targeting a specific cycle point expire as the workflow -moves on. Otherwise they persist for lifetime of the run, and across restarts, -unless cleared with another invocation of the command. - -.. seealso:: - - :ref:`broadcast-tutorial` - -.. TODO: document sub-workflows diff --git a/src/user-guide/running-workflows/index.rst b/src/user-guide/running-workflows/index.rst index ad3d2604d7..af4cdd4898 100644 --- a/src/user-guide/running-workflows/index.rst +++ b/src/user-guide/running-workflows/index.rst @@ -13,7 +13,7 @@ Running Workflows reflow workflow-run-modes scheduler-log-files - dynamic-behaviour + cylc-broadcast authentication-files workflow-databases advanced From 80c4b176df7cceea6aa24fea7c91c105a17b6dbe Mon Sep 17 00:00:00 2001 From: Oliver Sanders Date: Wed, 17 Sep 2025 13:44:05 +0100 Subject: [PATCH 2/4] Apply suggestions from code review Co-authored-by: Hilary James Oliver --- .../running-workflows/cylc-broadcast.rst | 25 +++++++++---------- 1 file changed, 12 insertions(+), 13 deletions(-) diff --git a/src/user-guide/running-workflows/cylc-broadcast.rst b/src/user-guide/running-workflows/cylc-broadcast.rst index 8ed09f83fd..3ff3c4513c 100644 --- a/src/user-guide/running-workflows/cylc-broadcast.rst +++ b/src/user-guide/running-workflows/cylc-broadcast.rst @@ -13,7 +13,7 @@ Broadcasts can be helpful for: * Quickly developing tasks without having to :ref:`edit and reload the workflow configuration `. * Sending small amounts of data from a running task to other upcoming tasks (e.g. file paths). -* Orchestrating production workflows from an external system. +* Reconfiguring production workflows while they are running. Broadcasts which target specific cycles will eventually be :ref:`expired ` when no longer needed. @@ -57,8 +57,8 @@ For more information, run ``cylc broadcast --help``. GUI ^^^ -Broadcasts can also be issued from the GUI in a similar way using the -"Broadcast" command. +Broadcasts can also be issued from the GUI in a similar way by choosing +"Broadcast" from the task menu. Additionally, the GUI provides a utility called "Edit Runtime" which loads the tasks configuration into a form. Any changes you make are then broadcasted @@ -77,22 +77,21 @@ Expiry ------ Broadcasts which target specific cycles will eventually expire (i.e. be -deleted) as the workflow moves on. Otherwise they would gradually accumulate -over the life of the workflow (note broadcasts are persisted when the workflow -restarts). +deleted) as the workflow moves on, to avoid gradual accumulation +(note broadcasts are persisted when the workflow restarts). Expiry Point ^^^^^^^^^^^^ -Broadcasts are only expired after they are no longer required by tasks. +Broadcasts expire once are they are no longer required by upcoming tasks. The exact point at which a broadcast is expired depends on two things: * The oldest cycle in the workflow to contain :term:`active tasks `. * The longest cycling :term:`recurrence` in the workflow. -Broadcasts which are older than the oldest cycle point to contain active tasks +Broadcasts which are older than the oldest :term:`active cycle point` *minus* the duration of the longest recurrence will be cleared. For example, for the following workflow: @@ -110,8 +109,8 @@ The longest cycling recurrence is ``P3Y``. If there were no more tasks left running in the cycle ``2000``, then broadcasts for cycles earlier than ``1997`` (``2000 - P3Y``) would be expired. -This arrangement has been designed such that broadcasts should always be -present for the previous instance of a task in case you want to re-run it. +This is designed to keep broadcasts as far back as the previous instance +of each task, in case you want to re-run it. Broadcasting To Historical Cycles @@ -120,9 +119,9 @@ Broadcasting To Historical Cycles Broadcasts targeting historical cycles may be expired as soon as they are issued as the result of broadcast expiry. -Broadcast expiry does not occur while the workflow is paused. If you want to -broadcast to a historical cycle before re-running it, first pause the workflow, -then trigger the tasks, then resume the workflow, e.g: +However, broadcast expiry does not occur while the workflow is paused, so +you can pause the workflow, do the broadcast, trigger the tasks, and then +resume the workflow, e.g: .. code-block:: bash From 7a821e4201e53281f63ce2901bf03a24ea7a4f1b Mon Sep 17 00:00:00 2001 From: Oliver Sanders Date: Fri, 19 Sep 2025 16:09:39 +0100 Subject: [PATCH 3/4] Apply suggestions from code review Co-authored-by: Hilary James Oliver Co-authored-by: Tim Pillinger <26465611+wxtim@users.noreply.github.com> --- src/user-guide/running-workflows/cylc-broadcast.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/src/user-guide/running-workflows/cylc-broadcast.rst b/src/user-guide/running-workflows/cylc-broadcast.rst index 3ff3c4513c..d5a98f1dff 100644 --- a/src/user-guide/running-workflows/cylc-broadcast.rst +++ b/src/user-guide/running-workflows/cylc-broadcast.rst @@ -3,7 +3,7 @@ Cylc Broadcast ============== -Cylc "Broadcasts" allow us to override task's :cylc:conf:`[runtime]` +Cylc "Broadcasts" allow us to override the :cylc:conf:`[runtime]` settings of tasks. settings within a running workflow. Broadcasts can target specific cycles, families or tasks. @@ -84,8 +84,8 @@ deleted) as the workflow moves on, to avoid gradual accumulation Expiry Point ^^^^^^^^^^^^ -Broadcasts expire once are they are no longer required by upcoming tasks. -The exact point at which a broadcast is expired depends on two things: +Broadcasts expire once they are no longer required by upcoming tasks. +The exact point at which a broadcast is expired depends on: * The oldest cycle in the workflow to contain :term:`active tasks `. From 4b47093f6c0103f4140361ef4a929875de603f76 Mon Sep 17 00:00:00 2001 From: Hilary Oliver Date: Sat, 20 Sep 2025 17:45:52 +1200 Subject: [PATCH 4/4] Delete borked line. --- src/user-guide/running-workflows/cylc-broadcast.rst | 1 - 1 file changed, 1 deletion(-) diff --git a/src/user-guide/running-workflows/cylc-broadcast.rst b/src/user-guide/running-workflows/cylc-broadcast.rst index d5a98f1dff..5087e78d6e 100644 --- a/src/user-guide/running-workflows/cylc-broadcast.rst +++ b/src/user-guide/running-workflows/cylc-broadcast.rst @@ -4,7 +4,6 @@ Cylc Broadcast ============== Cylc "Broadcasts" allow us to override the :cylc:conf:`[runtime]` settings of tasks. -settings within a running workflow. Broadcasts can target specific cycles, families or tasks.