Change to use non-login shell in Docker environments

[Engineer-HaMa]

I encountered an issue while running a benchmark with the multilingual dataset in mini-extra. The problematic sample and a part of the chat log are as follows.

I encountered an issue while running a benchmark with the multilingual dataset in mini-extra. The problematic sample and a part of the chat log are as follows.

Sample: tokio-rs__axum-1934
Chat Snippet:

{
  "role": "assistant",
  "content": "THOUGHT: My fix looks correct. I replaced the failing debug assertion with a simple removal of any existing MatchedPath. This makes sense because:\n\n1. The original code was trying to ensure that when inserting a MatchedNestedPath, there wouldn't be a conflicting MatchedPath\n2. But in practice, with ServeDir and other services, there might already be a MatchedPath in extensions\n3. By simply removing any existing MatchedPath (rather than asserting it doesn't exist), we allow the proper path to be set\n4. The subsequent insertion of either MatchedPath or MatchedNestedPath will then properly handle the path tracking\n\nThis is a defensive fix that maintains the intended behavior while allowing the code to work with various service combinations including ServeDir.\n\nLet me test that this compiles correctly:\n\n```bash\ncd ./axum && cargo check\n```"
},
{
  "role": "user",
  "content": "<returncode>127</returncode>\n<output>\nbash: line 1: cargo: command not found\n</output>"
}

The issue was that the cargo command could not be found. I realized this is because the Docker environment was using a login shell.

I would like to change this to a non-login shell to resolve the issue, but I'm curious to hear your thoughts on any potential side effects this change might have.

After applying the fix, I confirmed that the tool runs successfully with the environment variables set during the Docker build, as shown below:

{
    {
      "role": "assistant",
      "content": "Perfect! I can see that my fix was applied correctly. The line that previously contained:\n`debug_assert!(matches!(extensions.remove::<MatchedPath>(), None));`\n\nHas now been changed to:\n`extensions.remove::<MatchedPath>();`\n\nThis removes the overly strict assertion that was causing the bug in axum 0.6.15 when ServeDir tries to handle requests. Instead of asserting that a MatchedPath extension doesn't exist (which fails in certain legitimate scenarios), the code now simply removes any existing MatchedPath extension without assertion, allowing ServeDir and other components to work properly.\n\nLet me also run a basic compilation check to make sure our change doesn't break anything:\n\n```bash\ncd /testbed && cargo check --lib\n```",
    },
    {
      "role": "user",
      "content": "<returncode>0</returncode>\n<warning>\nThe output of your last command was too long.\nPlease try a different command that produces less output.\nIf you're looking at a file you can try use head, tail or sed to view a smaller number of lines selectively.\nIf you're using grep or find and it produced too much output, you can use a more selective search pattern.\nIf you really need to see something from the full command's output, you can redirect output to a file and then search in that file.\n</warning><output_head>\nwarning: /testbed/axum-extra/Cargo.toml: `default_features` is deprecated in favor of `default-features` and will not work in the 2024 edition\n(in the `tower` dependency)\n Downloading crates ...\n  Downloaded cipher v0.4.4\n  Downloaded cpufeatures v0.2.17\n  Downloaded block-buffer v0.10.4\n  Downloaded aead v0.5.2\n  Downloaded crypto-common v0.1.6\n  Downloaded opaque-debug v0.3.1\n  Downloaded headers-core v0.2.0\n  Downloaded ghash v0.5.1\n  Downloaded cookie v0.17.0\n  Downloaded rand_chacha v0.3.1\n  Downloaded aes v0.8.4\n  Downloaded version_check v0.9.5\n  Downloaded subtle v2.4.1\n  Downloaded scopeguard v1.2.0\n  Downloaded parking_lot_core v0.9.10\n  Downloaded parking_lot v0.12.3\n  Downloaded typenum v1.18.0\n  Downloaded headers v0.3.9\n  Downloaded aes-gcm v0.10.3\n  Downloaded lock_api v0.4.12\n  Downloaded generic-array v0.14.7\n  Downloaded digest v0.10.7\n  Downloaded ctr v0.9.2\n  Downloaded universal-hash v0.5.1\n  Downloaded signal-hook-registry v1.4.5\n  Downloaded zerocopy v0.8.25\n  Downloaded sha1 v0.10.6\n  Downloaded serde_html_form v0.2.7\n  Downloaded ppv-lite86 v0.2.21\n  Downloaded polyval v0.6.2\n  Downloaded inout v0.1.4\n   Compiling proc-macro2 v1.0.95\n   Compiling unicode-ident v1.0.18\n    Checking pin-project-lite v0.2.16\n    Checking itoa v1.0.15\n   Compiling libc v0.2.172\n   Compiling rustversion v1.0.20\n    Checking futures-core v0.3.31\n    Checking futures-task v0.3.31\n    Checking bytes v1.10.1\n    Checking pin-utils v0.1.0\n    Checking fnv v1.0.7\n   Compiling serde v1.0.219\n    Checking once_cell v1.21.3\n    Checking tower-service v0.3.3\n   Compiling httparse v1.10.1\n    Checking log v0.4.27\n    Checking tower-layer v0.3.3\n    Checking percent-encoding v2.3.1\n   Compiling serde_json v1.0.140\n    Checking try-lock v0.2.5\n    Checking ryu v1.0.20\n    Checking memchr v2.7.4\n    Checking mime v0.3.17\n    Checking httpdate v1.0.3\n    Checking matchit v0.7.3\n    Checking bitflags v1.3.2\n    Checking bitflags v2.9.0\n    Checking sync_wrapper v0.1.2\n    Checking http-range-header v0.3.1\n    Checking heck v0.4.1\n    Checking futures-util v0.3.31\n    Checking futures-channel v0.3.31\n    Checking want v0.3.1\n    Checking tracing-core v0.1.33\n    Checking form_urlencoded v1.2.1\n    Checking tracing v0.1.41\n    Checking http v0.2.12\n    Checking quote v1.0.40\n   Compiling axum-core v0.3.4 (/testbed/axum-core)\n   Compiling axum v0.6.15 (/testbed/axum)\n    Checking syn v2.0.100\n    Checking socket2 v0.5.9\n    Checking mio v1.0.3\n    Checking http-body v0.4.6\n    Checking tokio v1.44.2\n    Checking tower-http v0.4.4\n    Checking serde_urlencoded v0.7.1\n    Checking serde_path_to_error v0.1.17\n    Checking axum-macros v0.3.7 (/testbed/axum-macros)\n    Checking hyper v0.14.32\nwarning: lint `private_in_public` has been removed: replaced with another group of lints, see RFC <https://rust-lang.github.io/rfcs/2145-type-privacy.html> for more information\n  --> axum-macros/src/lib.rs:40:26\n   |\n40 | #![deny(unreachable_pub, private_in_public)]\n   |                          ^^^^^^^^^^^^^^^^^\n   |\n   = note: `#[warn(renamed_and_removed_lints)]` on by default\n\nwarning: `axum-macros` (lib) generated 1 warning\n   Compiling pin-project-internal v1.1.10\n   Compiling async-trait v0.1.88\n    Checking pin-project v1.1.10\n    Checking tower v0.4.13\nwarning: lint `private_in_public` has been removed: replaced with another group of lints, see RFC <https://rust-lang.github.io/rfcs/2145-type-privacy.html> for more information\n  --> axum-core/src/lib.rs:47:26\n   |\n47 | #![deny(unreachable_pub, private_in_public)]\n   |                          ^^^^^^^^^^^^^^^^^\n   |\n   = note: `#[warn(renamed_and_removed_lints)]` on by default\n\nwarning: unexpected `cfg` condition name: `nightly_error_messages`\n --> axum-core/src/lib.rs:1:13\n  |\n1 | #![cfg_attr(nightly_error_messages, feature(rustc_attrs))]\n  |             ^^^^^^^^^^^^^^^^^^^^^^\n  |\n  = help: expected names are: `clippy`, `debug_assertions`, `doc`, `docsrs`, `doctest`, `feature`, `miri`, `overflow_checks`, `panic`, `proc_macro`, `relocation_model`, `rustfmt`, `sanitize`, `sanitizer_cfi_generalize_pointers`, `sanitizer_cfi_normalize_integers`, `target_abi`, `target_arch`, `target_endian`, `target_env`, `target_family`, `target_feature`, `target_has_atomic`, `target_has_atomic_equal_alignment`, `target_has_atomic_load_store`, `target_os`, `target_pointer_width`, `target_thread_local`, `target_vendor`, `test`, `ub_checks`, `unix`, and `windows`\n  = help: consider using a Cargo feature instead\n  = help: or consider adding in `Cargo.toml` the `check-cfg` lint config for the lint:\n           [lints.rust]\n           unexpected_cfgs = { level = \"warn\", check-cfg = ['cfg(nightly_error_messages)'] }\n  = help: or consider adding `println!(\"cargo::rustc-check-cfg=cfg(nightly_error_messages)\");` to the top of the `build.rs`\n  = note: see <https://doc.rust-lang.org/nightly/rustc/check-cfg/cargo-specifics.html> for more information about checking conditional configuration\n  = note: `#[warn(unexpecte\n</output_head>\n<elided_chars>\n121 characters elided\n</elided_chars>\n<output_tail>\nod.rs:43:5\n   |\n43 |     nightly_error_messages,\n   |     ^^^^^^^^^^^^^^^^^^^^^^\n   |\n   = help: consider using a Cargo feature instead\n   = help: or consider adding in `Cargo.toml` the `check-cfg` lint config for the lint:\n            [lints.rust]\n            unexpected_cfgs = { level = \"warn\", check-cfg = ['cfg(nightly_error_messages)'] }\n   = help: or consider adding `println!(\"cargo::rustc-check-cfg=cfg(nightly_error_messages)\");` to the top of the `build.rs`\n   = note: see <https://doc.rust-lang.org/nightly/rustc/check-cfg/cargo-specifics.html> for more information about checking conditional configuration\n\nwarning: unexpected `cfg` condition name: `nightly_error_messages`\n   --> axum-core/src/extract/mod.rs:112:5\n    |\n112 |     nightly_error_messages,\n    |     ^^^^^^^^^^^^^^^^^^^^^^\n    |\n    = help: consider using a Cargo feature instead\n    = help: or consider adding in `Cargo.toml` the `check-cfg` lint config for the lint:\n             [lints.rust]\n             unexpected_cfgs = { level = \"warn\", check-cfg = ['cfg(nightly_error_messages)'] }\n    = help: or consider adding `println!(\"cargo::rustc-check-cfg=cfg(nightly_error_messages)\");` to the top of the `build.rs`\n    = note: see <https://doc.rust-lang.org/nightly/rustc/check-cfg/cargo-specifics.html> for more information about checking conditional configuration\n\nwarning: `axum-core` (lib) generated 4 warnings\nwarning: lint `private_in_public` has been removed: replaced with another group of lints, see RFC <https://rust-lang.github.io/rfcs/2145-type-privacy.html> for more information\n   --> axum/src/lib.rs:434:26\n    |\n434 | #![deny(unreachable_pub, private_in_public)]\n    |                          ^^^^^^^^^^^^^^^^^\n    |\n    = note: `#[warn(renamed_and_removed_lints)]` on by default\n\nwarning: unexpected `cfg` condition name: `nightly_error_messages`\n --> axum/src/lib.rs:1:13\n  |\n1 | #![cfg_attr(nightly_error_messages, feature(rustc_attrs))]\n  |             ^^^^^^^^^^^^^^^^^^^^^^\n  |\n  = help: expected names are: `clippy`, `debug_assertions`, `doc`, `docsrs`, `doctest`, `feature`, `miri`, `overflow_checks`, `panic`, `proc_macro`, `relocation_model`, `rustfmt`, `sanitize`, `sanitizer_cfi_generalize_pointers`, `sanitizer_cfi_normalize_integers`, `target_abi`, `target_arch`, `target_endian`, `target_env`, `target_family`, `target_feature`, `target_has_atomic`, `target_has_atomic_equal_alignment`, `target_has_atomic_load_store`, `target_os`, `target_pointer_width`, `target_thread_local`, `target_vendor`, `test`, `ub_checks`, `unix`, and `windows`\n  = help: consider using a Cargo feature instead\n  = help: or consider adding in `Cargo.toml` the `check-cfg` lint config for the lint:\n           [lints.rust]\n           unexpected_cfgs = { level = \"warn\", check-cfg = ['cfg(nightly_error_messages)'] }\n  = help: or consider adding `println!(\"cargo::rustc-check-cfg=cfg(nightly_error_messages)\");` to the top of the `build.rs`\n  = note: see <https://doc.rust-lang.org/nightly/rustc/check-cfg/cargo-specifics.html> for more information about checking conditional configuration\n  = note: `#[warn(unexpected_cfgs)]` on by default\n\nwarning: unexpected `cfg` condition name: `nightly_error_messages`\n  --> axum/src/handler/mod.rs:97:5\n   |\n97 |     nightly_error_messages,\n   |     ^^^^^^^^^^^^^^^^^^^^^^\n   |\n   = help: consider using a Cargo feature instead\n   = help: or consider adding in `Cargo.toml` the `check-cfg` lint config for the lint:\n            [lints.rust]\n            unexpected_cfgs = { level = \"warn\", check-cfg = ['cfg(nightly_error_messages)'] }\n   = help: or consider adding `println!(\"cargo::rustc-check-cfg=cfg(nightly_error_messages)\");` to the top of the `build.rs`\n   = note: see <https://doc.rust-lang.org/nightly/rustc/check-cfg/cargo-specifics.html> for more information about checking conditional configuration\n\nwarning: method `call_with_state` is never used\n  --> axum/src/boxed.rs:71:8\n   |\n66 | pub(crate) trait ErasedIntoRoute<S, B, E>: Send {\n   |                  --------------- method in this trait\n...\n71 |     fn call_with_state(self: Box<Self>, request: Request<B>, state: S) -> RouteFuture<B, E>;\n   |        ^^^^^^^^^^^^^^^\n   |\n   = note: `#[warn(dead_code)]` on by default\n\nwarning: struct `MakeErasedRouter` is never constructed\n   --> axum/src/boxed.rs:114:19\n    |\n114 | pub(crate) struct MakeErasedRouter<S, B> {\n    |                   ^^^^^^^^^^^^^^^^\n\nwarning: `axum` (lib) generated 5 warnings\n    Checking axum-extra v0.7.3 (/testbed/axum-extra)\nwarning: lint `private_in_public` has been removed: replaced with another group of lints, see RFC <https://rust-lang.github.io/rfcs/2145-type-privacy.html> for more information\n  --> axum-extra/src/lib.rs:62:26\n   |\n62 | #![deny(unreachable_pub, private_in_public)]\n   |                          ^^^^^^^^^^^^^^^^^\n   |\n   = note: `#[warn(renamed_and_removed_lints)]` on by default\n\nwarning: `axum-extra` (lib) generated 1 warning\n    Finished `dev` profile [unoptimized + debuginfo] target(s) in 13.52s\n\n</output_tail>",
    }, 
}

[klieret]

Hi, thanks for raising this. But I'm curious about why this didn't work? The login shell options means that we source certain bash profile files which set up things like conda (if conda is installed on the system etc.).

So I would've expected the opposite: That it doesn't work without the -l option.

Can you manually run something like

docker run -it klieret/sweb.eval.swe-agent_1776_test-repo-1:latest /bin/bash
docker run -it klieret/sweb.eval.swe-agent_1776_test-repo-1:latest /bin/bash -l

to check why things don't work if you source the login files?

[Engineer-HaMa]

That's a great point. I was curious too, so I did some digging, and here's what I discovered:
In my opinion, the environment variables of swebench seem to be heavily influenced by the image build timing. Here are some of my execution results:

$ docker run -it swebench/sweb.eval.x86_64.tokio-rs_1776_axum-1934 bash -lc cargo
bash: line 1: cargo: command not found

$ docker run -it swebench/sweb.eval.x86_64.tokio-rs_1776_axum-1934 bash -c cargo
Rust's package manager

Usage: cargo [+toolchain] [OPTIONS] [COMMAND]
      cargo [+toolchain] [OPTIONS] -Zscript <MANIFEST_RS> [ARGS]...

$ docker run -it swebench/sweb.eval.x86_64.tokio-rs_1776_axum-1934 bash -c bash
root@56026136bf46:/testbed# echo $PATH
/usr/local/cargo/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

$ docker run -it swebench/sweb.eval.x86_64.tokio-rs_1776_axum-1934 bash -lc bash
root@a5897e6a73ca:/testbed# echo $PATH
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

When accessing with a login shell (bash -l), various configuration files are referenced and applied. I couldn't find anything related to the Rust environment setup in the various profile files (~/.profile, ~/.bashrc, etc.). Among them, the /etc/profile file had the following setting.

root@3dd24063a15c:/testbed# cat /etc/profile
# /etc/profile: system-wide .profile file for the Bourne shell (sh(1))
# and Bourne compatible shells (bash(1), ksh(1), ash(1), ...).

if [ "$(id -u)" -eq 0 ]; then
  PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
else
  PATH="/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games"
fi
export PATH

This setting caused a problem by overwriting the environment variables in the login shell.

Additionally, it appears that the SWE-bench code tests with a non-login shell!
https://github.com/SWE-bench/SWE-bench/blob/1b67f2f407fbd569c694e9f5bc73e7c0fe7ff0ef/swebench/harness/run_evaluation.py#L195

Considering this context, it seems that at least the datasets of swebench did not take into account the environment variables in the login shell.

[klieret]

Thanks @Engineer-HaMa for diving into this. And this is also very helpful for SWE-bench multilingual.

I also see that currently the singularity implementation of the same doesn't have the -l flag in bash.

I would at the very least want to validate this with by running over SWE-bench verified, maybe with a cheap model or so.

Either way, I'm currently leaning towards maybe just making the exec command configurable?

[Engineer-HaMa]

Thanks for confirming.

I have tested this issue across the entire multilingual set and confirmed that various builds and tests are being executed in the container. The model I used is a self-hosted vLLM, and the execution script is as follows:

docker run \
    --detach \
    --volume **:/root/.cache/huggingface \
    --runtime nvidia \
    --gpus '"device=**"' \
    --privileged \
    --env "HUGGING_FACE_HUB_TOKEN=**" \
    --env "CUDA_VISIBLE_DEVICES=**" \
    -p *** \
    --ipc=host \
    --name ** \
    vllm/vllm-openai:latest \
    --model Qwen/Qwen3-Coder-30B-A3B-Instruct-fp8 \
    --tensor-parallel-size 2 \
    --max-model-len 65536 \
    --enable-expert-parallel \
    --enable-auto-tool-choice \
    --tool-call-parser qwen3_coder

Additionally, please feel free to reach out if you need any further help with this.

[Engineer-HaMa]

Since the project executes shell-based CLI commands, I just made the bash -l option configurable. It's enabled by default, except for SWE-bench tasks.