Skip to content

Kdump non-functional due to syntax and semantic errors in kdump remote feature #4115

@arista-hpandya

Description

@arista-hpandya

The kdump remote SSH feature has multiple bugs that prevent it from functioning correctly. When a device boots up the kdump configuration set in the /etc/default/kdump-tools is incorrect and this causes the crash files to not be generated when a kernel panic takes place.

List of bugs

  1. Typo in variable name - options.verbos instead of options.verbose causes runtime errors when using --ssh_path flag

    • Error: Exception['Namespace' object has no attribute 'verbos']
  2. Incorrect sed regex patterns - Missing ^ anchors cause unintended line modifications

    • The pattern s/#SSH/SSH/ matches both #SSH and #SSH_KEY, causing SSH_KEY to accumulate multiple # prefixes with each toggle
    • After one cycle of remote false→true, SSH_KEY becomes ##SSH_KEY instead of SSH_KEY
    • Each subsequent cycle adds another #, making the feature progressively more broken
    • This causes kdump to try to send kdump over ssh but it fails. The end result is that the core file is not stored under /var/crash or to a remote server
  3. Redundant SSH_KEY sed operations - Duplicate commands that are unnecessary and cause issues

    • Both SSH and SSH_KEY lines are toggled by separate sed commands
    • The first sed command already matches both lines, making the second redundant
    • This redundancy contributes to the accumulating # prefix problem
  4. SSH path validation failure - The default value in kdump-tools config breaks path updates

    • Default config has: #SSH_KEY="<path>"
    • The read_ssh_path() function validates that paths start with /
    • The literal string "<path>" fails this validation
    • All subsequent attempts to update SSH path fail due to this initial validation error
  5. Incorrect sed delimiter - The / delimiter conflicts with forward slashes in SSH key paths

    • Pattern: s/#*SSH_KEY=.*/SSH_KEY=\"%s\"/ fails when path contains /
    • Should use | delimiter: s|#*SSH_KEY=.*|SSH_KEY=\"%s\"|

Steps to Reproduce

Sanitize with latest master and check the journalctl logs:

Oct 29 21:34:24 ctn102 hostcfgd[8142]: sonic-kdump-config: error: unrecognized arguments: false
Oct 29 21:34:24 ctn102 hostcfgd[6737]: ['sonic-kdump-config', '--remote', 'false'] - failed: return code - 2, output:
                                       None
Oct 29 21:34:24 ctn102 hostcfgd[8193]: Error! Exception['Namespace' object has no attribute 'verbos'] occured while processing the command sonic-kdump-config --ssh_path.
Oct 29 21:34:24 ctn102 hostcfgd[6737]: ['sonic-kdump-config', '--ssh_path', '/a/b/c'] - failed: return code - 1, output:
                                       None

The kdump-tools config will also be incorrect by default:

SSH="user@localhost"
#SSH_KEY="<path>"

Trigger a kernel panic at this point and notice that there are no crash files created:
echo c | sudo tee /proc/sysrq-trigger

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    Status

    No status

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions