slurm_account error when the account is a builtin python function

[grabear]

Software Versions

slurm 23.02.6

Python:         3.11.9

snakefmt                                  0.10.3        The uncompromising Snakemake code formatter
snakemake                                 9.10.0        Workflow management system to create reproducible and scalable data analyses
snakemake-executor-plugin-cluster-generic 1.0.9
snakemake-executor-plugin-cluster-sync    0.1.5         A Snakemake executor plugin for cluster jobs that are executed synchronously.
snakemake-executor-plugin-slurm           1.7.0         A Snakemake executor plugin for submitting jobs to a SLURM cluster.
snakemake-executor-plugin-slurm-jobstep   0.3.0         A Snakemake executor plugin for running srun jobs inside of SLURM jobs (meant for internal use by snakemake-executor-plugin-slurm)
snakemake-interface-common                1.21.0        Common functions and classes for Snakemake and its plugins
snakemake-interface-executor-plugins      9.3.9         This package provides a stable interface for interactions between Snakemake and its executor plugins.
snakemake-interface-logger-plugins        1.2.4         Logger plugin interface for snakemake
snakemake-interface-report-plugins        1.2.0         The interface for Snakemake report plugins.
snakemake-interface-scheduler-plugins     2.0.1         Scheduler plugin interface for snakemake
snakemake-interface-storage-plugins       4.2.3         This package provides a stable interface for interactions between Snakemake and its storage plugins.
snakemake-storage-plugin-s3               0.3.6         A Snakemake storage plugin for S3 API storage (AWS S3, MinIO, etc.)
snakeviz                                  2.2.2         A web-based viewer for Python profiler output

Describe the bug
When using slurm_account in my profile under default-resources, I get a strange error when using the "dir" account on my local cluster. It seems to be calling python's builtin dir function. After talking with support I also tried using account, and also setting dir with ("dir", 'dir') and without quotes, but I had the same issue. Additionally, I tried setting it from the command line, which I think I was doing wrong, so I can try that again if that's helpful. In the end I had to settle with using slurm_extra: --account dir and that worked just fine.

Logs
Here's the log I got from the main snakemake process:

Select jobs to execute...
InputFunctionException in rule cram_quality_control_mod_samtools_idxstats_by_patient in file "cram_quality.smk", line 214:
Error:
  TypeError: dir() takes no keyword arguments
Wildcards:
  id=S2529182_BG
Traceback:
 (rule cram_quality_control_mod_samtools_idxstats_by_patient, line 334, cram_quality.smk)

Additional context

[cmeesters]

Hi,

Thank you for reporting this issue!

This is a particularly silly issue, particularly considering, that it should not arise, because I thought we fixed this in the past.

I made a PR (#355) edit: PR #350, because other implementations might lead to conflicts /edit. Please check, whether it works for you.

Cheers
Chris

[cmeesters]

The issue ought to be fixed in v1.8.. Please check and give feedback.

Also, no dir did not cause the issue because it is a Python built-in, but rather because it is a shell built-in. As we submit jobs not by using pragmas (this special #SBATCH -A dir comments) but via a shell command, e.g. sbatch ... -A dir prior to the fix, this would not work. Now, we submit with ... -A 'dir'.

If I may remark to your cluster's admins: To use a master slurm shell script which uses $(sbatch --dependency=$JOBID -A dir ...) you are also better off with non-built-in functions. In any case, using shell built-ins for account or partition names (or features, qos, etc.) is not a good idea.

Cheers
Chris

[grabear]

Hi @cmeesters thanks! A little busy here so I missed this last week. I'll give it a try and let you know.

[grabear]

Hi @cmeesters. This may not be an issue with the slurm plugin? I tested before and after updating to v1.8.0 and still got the same error when using slurm_account.

I also made a super minimal example with just a rule that "touches" an output file and use --verbose to debug:

snakemake --profile ../config/profiles/slurm/ --verbose
Using profile ../config/profiles/slurm/ for setting default command line arguments.
host:
Building DAG of jobs...
test.txt: True 0
shared_storage_local_copies: True
remote_exec: False
Submitting maximum 40 job(s) over 1.0 second(s).
SLURM run ID: 
Using shell: /usr/bin/bash
Provided remote nodes: 750
Job stats:
job      count
-----  -------
all          1
total        1

Resources before job selection:
Ready jobs: 1
Select jobs to execute...
unlocking
removing lock
removing lock
removed all locks
Full Traceback (most recent call last):
  File "/pipeline/.venv/lib/python3.11/site-packages/snakemake/rules.py", line 1073, in apply
    res, _ = self.apply_input_function(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/pipeline/.venv/lib/python3.11/site-packages/snakemake/rules.py", line 720, in apply_input_function
    res, incomplete = apply_func(res)
                      ^^^^^^^^^^^^^^^
  File "/pipeline/.venv/lib/python3.11/site-packages/snakemake/rules.py", line 712, in apply_func
    raise e
  File "/pipeline/.venv/lib/python3.11/site-packages/snakemake/rules.py", line 691, in apply_func
    value = func(wildcards_arg, **_aux_params)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: dir() takes no keyword arguments

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/pipeline/.venv/lib/python3.11/site-packages/snakemake/cli.py", line 2149, in args_to_api
    dag_api.execute_workflow(
  File "/pipeline/.venv/lib/python3.11/site-packages/snakemake/api.py", line 642, in execute_workflow
    workflow.execute(
  File "/pipeline/.venv/lib/python3.11/site-packages/snakemake/workflow.py", line 1420, in execute
    raise e
  File "/pipeline/.venv/lib/python3.11/site-packages/snakemake/workflow.py", line 1416, in execute
    success = self.scheduler.schedule()
              ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/pipeline/.venv/lib/python3.11/site-packages/snakemake/scheduling/job_scheduler.py", line 380, in schedule
    raise e
  File "/pipeline/.venv/lib/python3.11/site-packages/snakemake/scheduling/job_scheduler.py", line 316, in schedule
    run = self.job_selector(needrun)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/pipeline/.venv/lib/python3.11/site-packages/snakemake/scheduling/job_scheduler.py", line 545, in job_selector
    self.validate_job(job)
  File "/pipeline/.venv/lib/python3.11/site-packages/snakemake/scheduling/job_scheduler.py", line 605, in validate_job
    value = job.scheduler_resources.get(name, 0)
            ^^^^^^^^^^^^^^^^^^^^^^^
  File "/pipeline/.venv/lib/python3.11/site-packages/snakemake/jobs.py", line 514, in scheduler_resources
    return self._get_scheduler_resources()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/pipeline/.venv/lib/python3.11/site-packages/snakemake/jobs.py", line 112, in _get_scheduler_resources
    set(self.resources.keys())
        ^^^^^^^^^^^^^^
  File "/pipeline/.venv/lib/python3.11/site-packages/snakemake/jobs.py", line 500, in resources
    self._resources = self.rule.expand_resources(
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/pipeline/.venv/lib/python3.11/site-packages/snakemake/rules.py", line 1126, in expand_resources
    value = apply(name, res, threads=threads)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/pipeline/.venv/lib/python3.11/site-packages/snakemake/rules.py", line 1083, in apply
    raise InputFunctionException(e, rule=self, wildcards=wildcards)
snakemake.exceptions.InputFunctionException: Error:
  TypeError: dir() takes no keyword arguments
Wildcards:

Traceback:
 (rule all, line 1, /pipeline/test/Snakefile)

InputFunctionException in rule all in file "/pipeline/test/Snakefile", line 1:
Error:
  TypeError: dir() takes no keyword arguments
Wildcards:

Traceback:
 (rule all, line 1, /pipeline/test/Snakefile)

Additionally, I don't think this is using the dir shell utility based on the traceback and the error that's thrown:

TypeError: dir() takes no keyword arguments

[grabear]

Here is what my Snakefile looks like:

rule all:
   output: "test.txt"
   shell:
        "touch {output[0]}"

Here is what my profile looks like:

show-failed-logs: true
printshellcmds: true
# Execution Settings
scheduler: "greedy"
executor: "slurm"
latency-wait: 500
keep-going: true
rerun-incomplete: true
slurm-keep-successful-logs: true
# Behavior
retries: 3
max-status-checks-per-second: 5 #defaults to 10
max-jobs-per-timespan: "40/1s" # defaults to 100/1s
slurm-keep-successful-logs: true
# Resource Settings
jobs: 750
default-resources:
#  slurm_extra: "--account dir"
  slurm_account: "dir"

[cmeesters]

Indeed. The full error log suggests, that a resource is interpreted as a Python function. It could not have stemmed from a subprocess of Python. My bad.

I suppose, you already checked renaming your account (which would then crash the plugin, but just as a check)?

I am not sure, this is easy to solve. Currently travelling, I will ask for help, later.

[grabear]

Changing my account from "dir" to "bcbb" actually works.

[cmeesters]

Changing my account from "dir" to "bcbb" actually works.

I know. ;-)

I think, I will just ask whether we want to “fix” this or rather throw a big fat error, prohibiting the use of build-ins (shell or Python) as potential resources.