Problem

When using Vault Agent injection with ToolHive's Kubernetes operator, secrets are injected into the proxy runner pod at /vault/secrets, but they are not being passed to the actual workload StatefulSet pod.

Symptoms

• Vault agent successfully injects secrets into the proxy pod (visible in pod description)
• Secrets are mounted at /vault/secrets in the proxy pod
• Environment variables are missing in the workload StatefulSet pod
• Workload pod crashes with missing environment variables

Example Configuration

apiVersion: toolhive.stacklok.dev/v1alpha1
kind: MCPServer
metadata:
  name: unsplash
spec:
  image: 937028213865.dkr.ecr.us-east-1.amazonaws.com/mcp-unsplash:test-6
  resourceOverrides:
    proxyDeployment:
      podTemplateMetadataOverrides:
        annotations:
          vault.hashicorp.com/agent-inject: "true"
          vault.hashicorp.com/role: "toolhive-mcp-workloads"
          vault.hashicorp.com/agent-inject-secret-unsplash-config: "toolhive/mcp-unsplash"
          vault.hashicorp.com/agent-inject-template-unsplash-config: |
            {{- with secret "toolhive/mcp-unsplash" -}}
            UNSPLASH_ACCESS_KEY="{{ .Data.data.UNSPLASH_ACCESS_KEY }}"
            {{- end -}}

Root Cause

The bug is in cmd/thv-proxyrunner/app/execution.go in the runWithFileBasedConfig function. This function loads the RunConfig from /etc/runconfig/runconfig.json (which includes EnvFileDir set to /vault/secrets by the operator), but it never processes the EnvFileDir field to actually read the environment files.

Code Flow

1. Operator detects Vault annotations (cmd/thv-operator/controllers/mcpserver_runconfig.go:216-218)

   • Sets EnvFileDir: "/vault/secrets" in the RunConfig
   • Serializes RunConfig to ConfigMap

2. Proxy runner loads config (cmd/thv-proxyrunner/app/execution.go:99-140)

   • Loads RunConfig from /etc/runconfig/runconfig.json
   • EnvFileDir field is present but never processed
   • Calls workloadManager.RunWorkload(ctx, config) directly

3. Workload manager (pkg/workloads/manager.go)

   • Doesn't process EnvFileDir either
   • Just passes config to runner

When This Was Introduced

Commit: 90993f60 (September 30, 2025)
PR: #1952
Title: "change the default behaviour of proxyrunner"

This commit introduced the runWithFileBasedConfig function but missed the critical step of processing EnvFileDir that was already present in the flags-based approach (runWithFlagsBasedConfig at line 56).

Solution

Add code to process EnvFileDir in runWithFileBasedConfig before deploying the workload.

Implementation

Add this code block in cmd/thv-proxyrunner/app/execution.go after line 119 (after applying k8s-pod-patch and before environment variable validation):

// Process env file directory if specified (e.g., for Vault secrets)
if config.EnvFileDir != "" {
    var err error
    config, err = config.WithEnvFilesFromDirectory(config.EnvFileDir)
    if err != nil {
        return fmt.Errorf("failed to load environment files from directory %s: %w", config.EnvFileDir, err)
    }
}

How It Works

1. Operator detects Vault → Sets EnvFileDir: "/vault/secrets" in RunConfig
2. Proxy runner loads config → Reads RunConfig from ConfigMap
3. NEW: Process EnvFileDir → Reads all files in /vault/secrets and merges them into config.EnvVars
4. Deploy workload → Environment variables are passed to the StatefulSet pod

The WithEnvFilesFromDirectory method reads all files in the directory, parses them as KEY=VALUE format, and merges them into the EnvVars map, which then gets passed to the workload pod.

Testing

After applying this fix:

1. Deploy an MCPServer with Vault annotations
2. Verify vault secrets are injected into proxy pod at /vault/secrets
3. Check that environment variables are present in the workload StatefulSet pod
4. Verify the workload pod starts successfully

Related Files

• Bug location: cmd/thv-proxyrunner/app/execution.go:99-140
• Operator sets EnvFileDir: cmd/thv-operator/controllers/mcpserver_runconfig.go:216-218
• RunConfig structure: pkg/runner/config.go:87-88
• Environment file processing: pkg/runner/env_files.go:15-102
• WithEnvFilesFromDirectory method: pkg/runner/config.go:415-422