Module graceful shutdown support #667

rameshraghupathy · 2025-08-25T00:09:19Z

Provide support for SmartSwitch DPU module graceful shutdown.

Description

Single source of truth for transitions
- All components now use sonic_platform_base.module_base.ModuleBase helpers:
  - set_module_state_transition(db, name, transition_type)
  - clear_module_state_transition(db, name)
  - get_module_state_transition(db, name) -> dict
  - is_module_state_transition_timed_out(db, name, timeout_secs) -> bool
- Eliminates duplicated logic and race-prone direct Redis writes.
Correct table everywhere
- Standardized on CHASSIS_MODULE_TABLE (replaces CHASSIS_MODULE_INFO_TABLE).
- HLD mismatch addressed in code (HLD fix tracked separately).
Ownership & lifecycle
- The initiator of an operation (startup/shutdown/reboot) sets:
  - state_transition_in_progress=True
  - transition_type=<op>
  - transition_start_time=<utc-iso8601>
- The platform (set_admin_state()) is responsible for clearing:
  - state_transition_in_progress=False
  - optionally transition_end_time=<epoch> (or similar end stamp).
- CLI pre-clears only when a prior transition is timed out.
Timeouts & policy
- Platform JSON path only: /usr/share/sonic/device/{plat}/platform.json; else constants.
- Typical production values used:
  - startup: 180s, shutdown: 180s (≈ graceful_wait 60s + power 120s), reboot: 120s.
- Graceful wait (e.g., waiting for “Graceful shutdown complete”) is a platform policy and implemented inside platform set_admin_state()—not in ModuleBase.
Boot behavior
- chassisd on start:
  1. Clears stale flags once (centralized sweep).
  2. Runs set_initial_dpu_admin_state() which marks transitions via ModuleBase before calling platform set_admin_state().
  3. Leaves clearing to the platform or to well-defined status transitions (ONLINE/OFFLINE) where appropriate.
gNOI shutdown daemon
- Listens on CHASSIS_MODULE_TABLE and triggers only when:
  - state_transition_in_progress=True and transition_type=shutdown.
- Never clears the flag (ownership stays with the platform).
- Bounded RPC timeouts and robust Redis access (swsssdk/swsscommon).
CLI (config chassis modules …)
- Uses ModuleBase APIs for all set/get/timeout checks.
- If a previous transition is stuck, is_module_state_transition_timed_out() → auto-clear then proceed.
- Sets transition at the start of startup/shutdown; platform clears on completion.
- Fabric card flow retained; edits are surgical.
Redis robustness
- Helpers handle both stacks (swsssdk/swsscommon); no hset(mapping=...) usage.
- Consistent HGETALL/HSET paths; resilient to connector differences.
Race reduction & consistency
- Centralized writes prevent multi-writer races.
- All transition writes include transition_start_time; clears may add an end stamp.
- Existing PCI/file-lock logic left intact; unrelated behavior unchanged.
Change scope
- Minimal, targeted diffs.
- No background tasks added, no broad refactors beyond transition handling.
- Behavior changes are limited to making transition semantics correct and uniform across repos.

Please refer to the HLD and related PRs:
HLD: # 1991 sonic-net/SONiC#1991
sonic-host-services: sonic-net/sonic-host-services#255
sonic-platform-common: sonic-net/sonic-platform-common#567
Module graceful shutdown support #4031 sonic-net/sonic-utilities#4031

How Has This Been Tested?
Issue the "config chassis modules shutdown DPUx" command
Verify the DPU module is gracefully shut by checking the logs in /var/log/syslog on both NPU and DPU

mssonicbld · 2025-08-25T00:09:26Z

/azp run

azure-pipelines · 2025-08-25T00:09:35Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2025-08-25T01:39:06Z

/azp run

azure-pipelines · 2025-08-25T01:39:15Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2025-08-26T03:24:07Z

/azp run

azure-pipelines · 2025-08-26T03:24:17Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2025-09-01T03:58:41Z

/azp run

azure-pipelines · 2025-09-01T03:58:49Z

Azure Pipelines successfully started running 1 pipeline(s).

gpunathilell · 2025-09-05T18:24:01Z

sonic-chassisd/scripts/chassisd

+                    module.set_module_state_transition(v2, module_name, "startup")
+                except Exception as e:


When is the module state transition from here cleared?

gpunathilell · 2025-09-05T18:25:42Z

sonic-chassisd/scripts/chassisd

+                wants_up = (admin_state != 'down')
+                not_online = (str(operational_state).lower()
+                              != str(ModuleBase.MODULE_STATUS_ONLINE).lower())
+                if wants_up and not_online and v2:
+                    try:
+                        module.set_module_state_transition(v2, module_name, "startup")
+                        self.log_info(f"Marked startup transition for {module_name} at boot")
+                    except Exception as e:
+                        self.log_error(f"Failed to set startup transition for {module_name}: {e}")


Do we need this section, this function was only present to handle the case were the CONFIG_DB entry is not present, in which case dark mode is implied and we power off the DPUs

gpunathilell · 2025-09-05T18:27:07Z

sonic-chassisd/scripts/chassisd

-            ModuleTransitionFlagHelper().set_transition_flag(module_name)
-            try_get(self.module_updater.chassis.get_module(module_index).set_admin_state, admin_state, default=False)
+            # Only run pre-shutdown on DOWN path
+            try_get(module.module_pre_shutdown, default=False)


module_pre_shutdown is supposed to be called if we need to power off the DPU, or if we need to stay in dark mode, as the exisitng code was present in the same format, please align

gpunathilell · 2025-09-05T18:37:13Z

sonic-chassisd/scripts/chassisd

+                    # Clear transition flag in STATE_DB via ModuleBase centralized API
+                    try:
+                        module_obj = self.chassis.get_module(module_index)
+                        module_obj.clear_module_state_transition(self._state_v2, key)


@rameshraghupathy Clearing on operational state change is a problem, this can be intentional or unintentional, moreover, there is a transition from online to offline to online when we do a reboot, this will clear the transition during that state change, which is not the expected behavior

mssonicbld · 2025-09-08T19:20:04Z

/azp run

azure-pipelines · 2025-09-08T19:20:14Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2025-09-09T19:47:34Z

/azp run

azure-pipelines · 2025-09-09T19:47:43Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2025-09-09T20:11:42Z

/azp run

azure-pipelines · 2025-09-09T20:11:51Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2025-09-09T21:14:24Z

/azp run

azure-pipelines · 2025-09-09T21:14:34Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2025-09-10T00:05:13Z

/azp run

mssonicbld · 2025-10-02T00:35:57Z

/azp run

azure-pipelines · 2025-10-02T00:36:06Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2025-10-02T00:37:18Z

/azp run

azure-pipelines · 2025-10-02T00:37:27Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2025-10-02T00:58:16Z

/azp run

azure-pipelines · 2025-10-02T00:58:25Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2025-10-02T02:29:46Z

/azp run

azure-pipelines · 2025-10-02T02:29:55Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2025-10-02T02:50:14Z

/azp run

azure-pipelines · 2025-10-02T02:50:23Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2025-10-02T02:50:48Z

/azp run

azure-pipelines · 2025-10-02T02:50:57Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2025-10-02T03:07:11Z

/azp run

azure-pipelines · 2025-10-02T03:07:20Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2025-10-02T03:21:07Z

/azp run

azure-pipelines · 2025-10-02T03:21:15Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2025-10-02T03:47:25Z

/azp run

azure-pipelines · 2025-10-02T03:47:35Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2025-10-02T04:24:16Z

/azp run

azure-pipelines · 2025-10-02T04:24:26Z

Azure Pipelines successfully started running 1 pipeline(s).

Refactored for graceful shutdown

ecd68c9

This was referenced Aug 25, 2025

Module graceful shutdown support sonic-net/sonic-host-services#255

Open

Module graceful shutdown support sonic-net/sonic-utilities#4031

Open

Module graceful shutdown support sonic-net/sonic-platform-common#567

Open

Refactored for graceful shutdown

b022801

Refactored for graceful shutdown, fixing UT

ef2f282

Refactored for graceful shutdown, fixing UT

98c2146

gpunathilell suggested changes Sep 5, 2025

View reviewed changes

Refactored for graceful shutdown, fixing UT - Final round of tweaks

b959da9

Refactored for graceful shutdown, fixing UT - Final round of tweaks

5b2c0f6

rameshraghupathy force-pushed the graceful-shutdown branch from dfad0fc to 5b2c0f6 Compare September 9, 2025 20:11

Refactored for graceful shutdown, fixing UT - Final round of tweaks

b02d32e

Refactored for graceful shutdown, fixing UT - Final round of tweaks

fde2fd4

rameshraghupathy force-pushed the graceful-shutdown branch from 7836e1f to 3169b9d Compare October 2, 2025 00:37

Fixing test failures

500f3d3

Improving coverage

8762717

Improving coverage

ddf58cc

rameshraghupathy force-pushed the graceful-shutdown branch from eedae8c to ddf58cc Compare October 2, 2025 02:50

Merge branch 'sonic-net:master' into graceful-shutdown

9223003

Improving coverage

8cd0c59

Improving coverage

75e0d47

Improving coverage

b424cb7

Fixing test failures

663c496

		module.set_module_state_transition(v2, module_name, "startup")
		except Exception as e:

Module graceful shutdown support #667

Are you sure you want to change the base?

Module graceful shutdown support #667

Uh oh!

Conversation

rameshraghupathy commented Aug 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Uh oh!

mssonicbld commented Aug 25, 2025

Uh oh!

azure-pipelines bot commented Aug 25, 2025

Uh oh!

mssonicbld commented Aug 25, 2025

Uh oh!

azure-pipelines bot commented Aug 25, 2025

Uh oh!

mssonicbld commented Aug 26, 2025

Uh oh!

azure-pipelines bot commented Aug 26, 2025

Uh oh!

mssonicbld commented Sep 1, 2025

Uh oh!

azure-pipelines bot commented Sep 1, 2025

Uh oh!

gpunathilell Sep 5, 2025

Choose a reason for hiding this comment

Uh oh!

gpunathilell Sep 5, 2025

Choose a reason for hiding this comment

Uh oh!

gpunathilell Sep 5, 2025

Choose a reason for hiding this comment

Uh oh!

gpunathilell Sep 5, 2025

Choose a reason for hiding this comment

Uh oh!

mssonicbld commented Sep 8, 2025

Uh oh!

azure-pipelines bot commented Sep 8, 2025

Uh oh!

mssonicbld commented Sep 9, 2025

Uh oh!

azure-pipelines bot commented Sep 9, 2025

Uh oh!

mssonicbld commented Sep 9, 2025

Uh oh!

azure-pipelines bot commented Sep 9, 2025

Uh oh!

mssonicbld commented Sep 9, 2025

Uh oh!

azure-pipelines bot commented Sep 9, 2025

Uh oh!

mssonicbld commented Sep 10, 2025

Uh oh!

mssonicbld commented Oct 2, 2025

Uh oh!

azure-pipelines bot commented Oct 2, 2025

Uh oh!

mssonicbld commented Oct 2, 2025

Uh oh!

azure-pipelines bot commented Oct 2, 2025

Uh oh!

mssonicbld commented Oct 2, 2025

Uh oh!

azure-pipelines bot commented Oct 2, 2025

Uh oh!

mssonicbld commented Oct 2, 2025

Uh oh!

azure-pipelines bot commented Oct 2, 2025

Uh oh!

mssonicbld commented Oct 2, 2025

Uh oh!

azure-pipelines bot commented Oct 2, 2025

Uh oh!

mssonicbld commented Oct 2, 2025

Uh oh!

azure-pipelines bot commented Oct 2, 2025

Uh oh!

mssonicbld commented Oct 2, 2025

rameshraghupathy commented Aug 25, 2025 •

edited

Loading