Add ApptainerWorkspace implementation for rootless container support

[neubig]

Description

This PR implements ApptainerWorkspace, a container-based workspace that uses Apptainer (formerly Singularity) instead of Docker. This addresses the need for rootless container execution in HPC and shared computing environments where Docker may not be available or permitted.

✨ Critical Bug Fix (2025-10-24): Discovered and fixed a bug where the initial implementation incorrectly used apptainer build ... docker-daemon://image, which required Docker to be running. This defeated the entire purpose of Apptainer! The fix changes to apptainer pull docker://image which pulls directly from Docker registries without needing Docker daemon. This is the key feature that makes Apptainer valuable.

✨ Additional Fixes (2025-10-24):

• Switched to exec mode: Changed from Apptainer instance mode to direct apptainer exec for better compatibility in environments without systemd/FUSE
• Fixed authentication: RemoteWorkspace now properly includes API key in default HTTP client headers, fixing 401 errors when creating conversations
• Environment variable support: ApptainerWorkspace now reads SESSION_API_KEY from environment and passes it to RemoteWorkspace

Fixes #891

Key Features

• ✅ No root privileges required for container execution
• ✅ No Docker daemon required - pulls directly from registries using apptainer pull
• ✅ Works without systemd/FUSE - uses direct exec mode instead of instances
• ✅ Converts Docker images to Apptainer SIF format with intelligent caching
• ✅ Full RemoteWorkspace API compatibility - drop-in replacement for DockerWorkspace
• ✅ Automatic port management and health checking
• ✅ Directory mounting and environment variable forwarding
• ✅ Proper authentication via SESSION_API_KEY
• ✅ Comprehensive documentation and usage examples
• ✅ Actually tested end-to-end with Apptainer 1.3.5 - see demo log

Implementation Details

Files Added

1. openhands-workspace/openhands/workspace/apptainer/workspace.py (378 lines)

   • Main ApptainerWorkspace class implementation
   • Image preparation using native apptainer pull (no Docker required!)
   • Container lifecycle management using apptainer exec
   • Health checking and logging
   • Authentication support via SESSION_API_KEY

2. openhands-workspace/openhands/workspace/apptainer/__init__.py

   • Module initialization and exports

3. openhands-workspace/openhands/workspace/apptainer/README.md

   • Comprehensive documentation
   • Usage examples for all three image source options
   • Configuration reference
   • Troubleshooting guide

4. examples/02_remote_agent_server/05_convo_with_apptainer_sandboxed_server.py

   • Complete working example
   • Demonstrates workspace setup, agent conversations, and cleanup

5. tests/workspace/test_apptainer_workspace.py

   • Import and inheritance tests
   • Field definition validation
   • All tests passing ✅

6. apptainer_workspace_demo.log

   • Complete log showing successful end-to-end operation
   • Demonstrates container startup, health checks, bash commands, and conversation creation
   • Shows authentication working correctly

Files Modified

• openhands-workspace/openhands/workspace/__init__.py
  • Added ApptainerWorkspace to exports
• openhands-workspace/openhands/workspace/apptainer/workspace.py
  • Fixed to use apptainer pull instead of Docker daemon
  • Switched to apptainer exec instead of instance mode for better compatibility
  • Added SESSION_API_KEY environment variable support
• openhands-sdk/openhands/sdk/workspace/remote/base.py
  • Fixed to include API key headers in HTTP client initialization

Usage

Option 1: Pre-built Server Image (Recommended for HPC)

from openhands.workspace import ApptainerWorkspace

# Pulls from Docker Hub/GHCR without needing Docker!
with ApptainerWorkspace(
    server_image="ghcr.io/openhands/agent-server:main-python",
    host_port=8010,
) as workspace:
    result = workspace.execute_command("echo 'Hello from Apptainer!'")
    print(result.stdout)

Option 2: Build from Base Image (Requires Docker for initial build)

# Note: This still requires Docker to build the agent-server initially,
# but can then be used on systems without Docker
with ApptainerWorkspace(
    base_image="nikolaik/python-nodejs:python3.12-nodejs22",
    host_port=8010,
) as workspace:
    result = workspace.execute_command("python --version")
    print(result.stdout)

Option 3: Use Existing SIF File

with ApptainerWorkspace(
    sif_file="/path/to/agent-server.sif",
    host_port=8010,
) as workspace:
    result = workspace.execute_command("ls -la")
    print(result.stdout)

Testing

All tests pass successfully:

$ uv run pytest tests/workspace/test_apptainer_workspace.py -v

tests/workspace/test_apptainer_workspace.py::test_apptainer_workspace_import PASSED [ 33%]
tests/workspace/test_apptainer_workspace.py::test_apptainer_workspace_inheritance PASSED [ 66%]
tests/workspace/test_apptainer_workspace.py::test_apptainer_workspace_field_definitions PASSED [100%]

============================== 3 passed in 0.13s ===============================

End-to-End Testing with Actual Apptainer

Successfully tested the complete example with Apptainer 1.3.5. See apptainer_workspace_demo.log for full details:

✅ Image Preparation

• Successfully pulled Docker image without Docker daemon
• Created SIF file: /root/.apptainer_cache/ghcr.io_openhands_agent-server_main-python.sif
• Cached for future use

✅ Container Execution

• Container starts successfully using apptainer exec mode
• Server starts and listens on port 8010
• Health endpoint responds correctly (HTTP 200)

✅ Command Execution

Command 'echo 'Hello from Apptainer sandboxed environment!' && pwd' completed with exit code 0
Output: Hello from Apptainer sandboxed environment!
/workspace

✅ Authentication

• SESSION_API_KEY properly passed to RemoteWorkspace
• API key included in HTTP client headers
• Conversation created successfully (no 401 errors)

✅ API Endpoints

• /health endpoint: ✅ Working
• /api/bash/start_bash_command: ✅ Working
• /api/conversations: ✅ Working (with auth)
• /api/conversations/{id}/run: ✅ Working (with auth)

All pre-commit hooks pass:

• ✅ Ruff format
• ✅ Ruff lint
• ✅ PEP8 style check (pycodestyle)
• ✅ Type check with basedpyright

Comparison: ApptainerWorkspace vs DockerWorkspace

Feature	DockerWorkspace	ApptainerWorkspace
Root privileges	Required (usually)	Not required
Docker daemon	Required	Not required ✨
systemd/FUSE	Not needed	Not needed ✨
Container runtime	Docker	Apptainer
Image format	Docker images	SIF (from Docker)
Use case	General development	HPC, shared systems
Port mapping	Native	Host networking
Exec mode	Process model	Direct exec

Prerequisites

Users need to install Apptainer: https://apptainer.org/docs/user/main/quick_start.html

On Ubuntu/Debian:

sudo apt-get install -y apptainer

Or build from source:

wget https://github.com/apptainer/apptainer/releases/download/v1.3.5/apptainer-1.3.5.tar.gz
tar xzf apptainer-1.3.5.tar.gz
cd apptainer-1.3.5
./mconfig && make -C builddir && sudo make -C builddir install

Why Apptainer?

As mentioned in issue #891, Docker requires root access which is often not available or permitted in:

• High-Performance Computing (HPC) environments
• Shared multi-user systems
• Security-sensitive environments
• Academic computing clusters
• Environments without systemd or FUSE support

Apptainer was specifically designed for these use cases and provides:

• Rootless container execution (no sudo needed)
• No Docker daemon required - pulls directly from registries
• Works without systemd/FUSE - direct exec mode
• Better security isolation for multi-tenant systems
• Native HPC integration
• Full compatibility with Docker images

Technical Implementation Notes

Exec Mode vs Instance Mode

Initially implemented using Apptainer instance mode (apptainer instance start), but discovered this requires systemd and/or FUSE which may not be available in all environments. Switched to direct execution mode (apptainer exec) which:

• Works in any environment
• Simpler lifecycle management
• Direct process control (can kill directly)
• No dependency on systemd/FUSE

Authentication Flow

ApptainerWorkspace discovers SESSION_API_KEY from environment and passes it to RemoteWorkspace, which now properly includes it in the HTTP client's default headers. This ensures all API requests (including conversation creation) are properly authenticated.

Demo Log

See apptainer_workspace_demo.log for the complete end-to-end test output showing:

• ✅ Apptainer 1.3.5 detected
• ✅ Image pull without Docker (using cached SIF)
• ✅ Container startup with uvicorn server
• ✅ Health endpoint responding
• ✅ Bash command execution succeeding
• ✅ Conversation creation with authentication
• ✅ Agent initialization and tool setup

Checklist

• Implementation follows the same pattern as DockerWorkspace
• Full RemoteWorkspace API compatibility
• Comprehensive documentation and README
• Usage example included and tested
• Test suite added and passing
• Pre-commit hooks passing
• Type hints and Pydantic validation
• Proper error handling and logging
• Demo log included showing full functionality
• Tested end-to-end with actual Apptainer 1.3.5 ✨
• Verified Docker-free operation ✨
• Verified systemd/FUSE-free operation ✨
• Verified authentication working ✨

Next Steps

After merging, users can:

1. Install Apptainer on their systems
2. Use ApptainerWorkspace as a drop-in replacement for DockerWorkspace
3. Run the example to see it in action
4. Deploy in HPC environments without root access, Docker, systemd, or FUSE

---

Agent Server images for this PR

• GHCR package: https://github.com/OpenHands/agent-sdk/pkgs/container/agent-server

Variants & Base Images

Variant	Base Image	Docs / Tags
golang	golang:1.21-bookworm	Link
java	eclipse-temurin:17-jdk	Link
python	nikolaik/python-nodejs:python3.12-nodejs22	Link

Pull (multi-arch manifest)

docker pull ghcr.io/openhands/agent-server:73648e4-python

Run

docker run -it --rm \
  -p 8000:8000 \
  --name agent-server-73648e4-python \
  ghcr.io/openhands/agent-server:73648e4-python

All tags pushed for this build

ghcr.io/openhands/agent-server:73648e4-golang
ghcr.io/openhands/agent-server:v1.0.0a4_golang_tag_1.21-bookworm_binary
ghcr.io/openhands/agent-server:73648e4-java
ghcr.io/openhands/agent-server:v1.0.0a4_eclipse-temurin_tag_17-jdk_binary
ghcr.io/openhands/agent-server:73648e4-python
ghcr.io/openhands/agent-server:v1.0.0a4_nikolaik_s_python-nodejs_tag_python3.12-nodejs22_binary

The 73648e4 tag is a multi-arch manifest (amd64/arm64); your client pulls the right arch automatically.

[github-actions]

Coverage Report •

File	Stmts	Miss	Cover	Missing
openhands-sdk/openhands/sdk/workspace/remote
base.py	38	6	84%	82–84, 102–104
openhands-workspace/openhands/workspace/apptainer
workspace.py	180	133	26%	26, 28–34, 36, 43, 45–47, 49–52, 144–146, 150, 155–156, 158, 160–161, 163–165, 168–169, 174–176, 182–184, 187–191, 193, 196–197, 200, 203–204, 207–208, 211, 215–217, 223–225, 227, 230–232, 234–236, 238, 241, 247–249, 253–254, 259–262, 265–269, 277, 291, 299–301, 305–315, 317–320, 324–325, 327–333, 336, 338, 343–344, 348, 352, 356, 360, 362–364, 367–377, 379–380
TOTAL	11036	4940	55%

[neubig]

ℹ️ Note on check-examples CI Failure

The check-examples test is currently failing because it checks that all example files are documented in the OpenHands/docs repository. This PR adds a new example file:

• examples/02_remote_agent_server/05_convo_with_apptainer_sandboxed_server.py

This example file demonstrates the new ApptainerWorkspace functionality and needs to be documented in the docs repository. Once documentation is added there, this check will pass.

Action Items

• Add documentation for the Apptainer workspace example in the docs repository
• Document the new ApptainerWorkspace class and its usage

All other CI checks are passing ✅

[openhands-ai]

Looks like there are a few issues preventing this PR from being merged!

• GitHub Actions are failing:
  • Check Documented Examples

If you'd like me to help, just leave a comment, like

@OpenHands please fix the failing actions on PR #892 at branch `openhands/apptainer-workspace-891`

Feel free to include any additional details that might help me get this PR into a better state.

_{^{You can manage your notification settings}}

[blacksmith-sh]

[Automatic Post]: It has been a while since there was any activity on this PR. @neubig, are you still working on it? If so, please go ahead, if not then please request review, close it, or request that someone else follow up.