Secure Local AI Agents in Sandbox VMs

Articles 5 Minutes

Nilbox, Caddy, LMStudio and Hermes

How I got a Hermes agent, sandboxed in a VM via nilbox, talking to a locally-hosted model in LM Studio — without a single API key ever touching the sandbox.

The Goal

I wanted to run Hermes Agent — an autonomous AI agent — inside nilbox, a desktop sandbox that runs agents in a real VM so they can’t touch the host filesystem or exfiltrate real credentials. Instead of pointing it at a cloud model, I wanted it talking to my own local model running in LM Studio, on the same Windows 11 machine hosting the VM. Fully private, fully local, nothing leaving the box.

It took a full afternoon of layered debugging to get there. Here’s the journey, in case it saves someone else the same afternoon.

Layer 1: Getting Hermes to Actually Use the Local Model

First hurdle: pointing Hermes at a custom OpenAI-compatible endpoint. Hermes has a known bug (tracked upstream) where the interactive hermes model picker’s “Custom endpoint” option silently falls through to OpenRouter instead of your configured endpoint — throwing a confusing empty API key, set OPENROUTER_API_KEY error that has nothing to do with your actual setup.

Fix: hand-edit ~/.hermes/config.yaml with a named custom provider rather than a bare one:

yaml

providers:
  local:
    base_url: "http://<host-address>:1234/v1"
    api_key: "no-key-required"
    default_model: "<your-model-name>"
model:
  default: "custom:local"

Even with this fixed in config, a related bug means fresh sessions can still silently reset to the broken bare-custom provider. The reliable workaround: run /model inside every new session and explicitly select custom:local before sending a message.

Layer 2: The VM Can’t Reach the Host’s LAN Address

With provider resolution sorted, the next error was a 502 Bad Gateway. nilbox’s architecture is deliberately locked down: the VM has no external network interface at all — every outbound request is forced through a proxy running on the host, over VSOCK, so a compromised agent can never touch the network directly.

Testing directly inside the VM (bypassing Hermes entirely) with curl confirmed the request wasn’t even reaching my LAN — it was dying at nilbox’s own proxy, immediately, with an empty response. Confirmed by testing the same endpoint from the Windows host itself, where it worked perfectly. The proxy was the failure point, not the model server or Windows Firewall.

Layer 3: Understanding “BYPASS” Mode

nilbox’s domain-allowlist has a per-domain toggle that, on hover, reveals itself as “RAW TLS Tunnel — proxy does not inspect or inject tokens.” This matters because nilbox’s default behavior for allowed domains is to terminate and re-establish TLS itself (so it can swap in real API tokens transparently). That’s great for api.openai.com. It’s a problem for a plain-HTTP local server, which was never using TLS in the first place — the interception machinery had nothing valid to intercept.

Layer 4: Plain HTTP Doesn’t Play Nice With a TLS-Oriented Proxy

Even with BYPASS enabled for the target address, plain HTTP still failed. The proxy’s raw-tunnel mode appears to assume genuine TLS traffic throughout. The real fix: stop giving it plain HTTP altogether, and wrap the local model server in actual TLS.

Solution: front LM Studio with Caddy, using its tls internal directive to mint a locally-trusted self-signed certificate automatically:

lmstudio.host:8443 {
    tls internal
    reverse_proxy 127.0.0.1:1234 {
        flush_interval -1
    }
}

(That flush_interval -1 turns out to matter enormously — more on that below.)

Layer 5: Two Separate Hosts Files, Two Separate Resolvers

Getting this Caddy front-end reachable required patience with hostname resolution in two entirely separate places:

  • The Linux VM’s /etc/hosts, so curl running inside the sandbox could resolve lmstudio.host.
  • The Windows host’s own hosts file (C:\Windows\System32\drivers\etc\hosts), because nilbox’s proxy process runs on the host, not inside the guest — and it needed to resolve that same hostname independently, using Windows’ own resolver, to open its outbound tunnel connection.

Missing either one produced a different, oddly-specific-looking failure — a good reminder that in a setup with this many layers, the same symptom (502, or “could not resolve host”) can originate from completely different components depending on which side of the VM boundary you’re debugging.

Layer 6: Certificate Trust, Twice Over

Once Caddy was reachable and resolvable, the next failures were trust-related:

  • The VM needed Caddy’s local root CA certificate imported into its trust store (update-ca-certificates), so curl inside the guest would accept Caddy’s certificate as valid.
  • Along the way, a request that hit nilbox’s default (non-bypass) inspection mode surfaced a certificate issued by NilBox Inspect CA rather than Caddy’s own CA — a clear tell that a given request wasn’t actually being tunneled raw, but intercepted, because the exact hostname on that specific request hadn’t been separately added to the allowlist.

A useful diagnostic in hindsight: checking the TLS certificate issuer in a verbose curl (curl -v) output at each stage told us immediately whether a request was being genuinely bypassed (issuer: Caddy) or silently intercepted (issuer: NilBox Inspect CA) — much faster than guessing from the error message alone.

Layer 7: The Real Villain — Response Buffering

With every layer of the tunnel finally working — VM → nilbox bypass proxy → Caddy → LM Studio — responses came back correctly, but painfully slowly: over six minutes for what should have been a quick reply.

The cause: Caddy’s reverse proxy buffers streamed responses by default rather than forwarding each chunk as it arrives. LM Studio streams tokens back as Server-Sent Events, but none of them were reaching Hermes until the entire response had finished generating and Caddy released it all at once. Adding flush_interval -1 to the Caddyfile fixed this immediately — six minutes became seventeen seconds.

The Final, Working Chain

Hermes Agent (in nilbox VM, no real keys ever present)
        │  HTTPS, tunneled raw via nilbox BYPASS mode
        ▼
Caddy (Windows host, real self-signed TLS, immediate flushing)
        │  plain HTTP, loopback only
        ▼
LM Studio (Windows host, serving a local model)

Lessons Worth Keeping

  • Isolate layer by layer. Test each hop independently (host → local server, VM → proxy, proxy → host) rather than only testing the full chain through the agent — the error messages from the agent were consistently the least informative link in the whole chain.
  • Check what a security-focused proxy actually promises. A tool built to protect against a compromised sandbox exfiltrating credentials will make very different tradeoffs than a general-purpose reverse proxy — assumptions about TLS, domain-matching, and interception need to be understood, not guessed at.
  • Buffering masquerades as “slowness.” If a working request takes drastically longer than it should, check for a proxy silently accumulating a streamed response before checking the model or hardware.
  • Small, young projects are still worth the investment — but expect to hit some genuinely undocumented edges, and file issues when you do. This setup surfaced at least one real gap (BYPASS mode’s assumptions around plain HTTP / private-network targets) worth reporting upstream.

Stack: AMD Ryzen 9600X / Radeon RX 9060 XT (16GB VRAM) · Windows 11 · LM Studio · Caddy · nilbox · Hermes Agent

Published

Leave a comment