codellm: extracting code execution into a standalone OpenAI-compatible service

TL;DR

Today fleet runs code in-process: an executor: code role extracts fenced blocks from a rendered body and runs them under an in-process Linux sandbox (internal/agentruntime/runcode.go + sandbox_linux.go). We move that execution into a standalone codellm service that speaks the OpenAI /v1/chat/completions protocol and pretends to be an LLM: it receives chat messages containing markdown-with-code, executes the fenced blocks, and returns the writes as OpenAI tool_calls (write_note/patch_note/finish). Because fleet's agentruntime.LLM interface already talks to any OpenAI-compatible endpoint (internal/agentruntime/llm.go:49, openai_llm.go), a code role becomes an ordinary executor: llm run pointed at codellm's base_url — and fleet's existing scope enforcement (ScopedKB + write_patterns) applies the returned tool calls unchanged. codellm holds no vault, no KB, no auth, no secrets — that emptiness is the isolation win. The multi-block streaming pipeline and the block-by-block debug seam move inside codellm and stay steppable. The runcode/sandbox/interpreters subsystem leaves fleet.

Chosen return contract: tool_calls, not JSON-in-content (the alternative reintroduces a code-specific parse branch in fleet, defeating the whole point). Hard invariant: code-role runs are pinned to MaxSteps: 1 and codellm must always emit finish — otherwise fleet's stateless loop re-calls codellm, which re-extracts the same blocks and re-applies every write up to the step ceiling (see "The re-execution hazard" below). Delivery bag rides the request as a labeled message; env passthrough is dropped, which means secret-dependent code roles are unsupported under codellm (not "migrate the secret into the bag" — that would push secrets into the HTTP request). Top 3 risks: (1) determinism is only "no external nondeterminism," not bit-reproducibility — the sandbox controls fs/net/resources but not clock/PRNG; (2) codellm is an unauthenticated code-execution oracle that can write arbitrary attacker-controlled bytes to any in-scope path (scope checks the path, never the content), unless the fleet↔codellm channel is locked down (mTLS/shared token/loopback); (3) apply-error semantics diverge between the two paths (a bad patch is a hard 502 today, a soft success under the loop) and must be reconciled.

Why this came up

executor: code and the exec(program, code) tool both run untrusted code inside the fleet process, confined only by an in-process re-exec sandbox (MaybeRunSandboxChild at internal/agentruntime/sandbox_linux.go:45, called first in cmd/fleet/main.go:36). That couples three concerns that should be separable:

• Blast radius. The code runs in the same address space and namespace as fleet, which holds the JWT secret, the LLM API key, and the admin HAT. The sandbox scrubs the child's env (buildChildEnv, coderun.go:388) and re-execs into namespaces, but the confinement lives in the same binary it is protecting against.
• Reuse. The clean interface fleet already has for "give me writes" is agentruntime.LLM.Chat returning tool_calls. Code execution is the odd path that bypasses it (handler.go:376 branches to f.codeRunner instead of agentruntime.Run).
• Debuggability. The owner wants a future debugger that steps the markdown block-by-block and inspects the pipe between blocks. That model already exists as a seam (pipelineDebugTaps, runcode.go:159; /debug/run-block, internal/fleet/debug.go) but is buried inside fleet.

Extracting code execution behind an OpenAI-compatible endpoint collapses all three: fleet loses a special path, the sandbox moves to a service whose container is the blast-radius boundary, and the block/pipe stepping becomes a first-class API on that service.

Current-state map: what moves, stays, or is deleted

The wire protocol

codellm is a real OpenAI chat-completions endpoint. Fleet reaches it through the same OpenAILLM it uses for real models (openai_llm.go:36), so the request is whatever agentruntime.Run already emits, and the response must be a standard ChatCompletionResponse.

POST /v1/chat/completions
{
  "model": "codellm",                       // role model or a fixed id; codellm echoes it
  "messages": [
    { "role": "system",
      "content": "You are a scoped trip2g micro-agent...\n<role body with ```python ... ``` blocks>" },
    { "role": "system", "name": "fleet_input",
      "content": "{\"changed_files\":[...],\"attached_notes\":[...],\"depth\":1,\"now\":\"2026-07-14T..\"}" },
    { "role": "user", "content": "Begin." }
  ],
  "tools": [ /* write_note, patch_note, finish schemas (runtime.go:450) */ ]
}

{
  "id": "codellm-1a2b3c",
  "object": "chat.completion",
  "choices": [{
    "index": 0,
    "message": {
      "role": "assistant",
      "content": "",
      "tool_calls": [
        { "id": "call_0", "type": "function",
          "function": { "name": "write_note",
            "arguments": "{\"path\":\"concepts/x.md\",\"content\":\"...\"}" } },
        { "id": "call_1", "type": "function",
          "function": { "name": "patch_note",
            "arguments": "{\"path\":\"index.md\",\"find\":\"a\",\"replace\":\"b\"}" } },
        { "id": "call_2", "type": "function",
          "function": { "name": "finish", "arguments": "{\"answer\":\"done\"}" } }
      ]
    },
    "finish_reason": "tool_calls"
  }],
  "usage": { "prompt_tokens": 0, "completion_tokens": 0, "total_tokens": 0 }
}

The codellm service shape

New binary cmd/codellm + new package internal/codellm (housing the moved coderun/runcode/sandbox/interpreters). Deployed as a separate container. Same repo initially (shares Go types and the sandbox re-exec trick); can graduate to its own repo later without protocol change.

Endpoints

• POST /v1/chat/completions — the OpenAI-compatible surface described above. Non-streaming first; SSE streaming is optional and not needed for the contract (fleet wants the final tool_calls).
• GET /v1/models — optional, returns a single synthetic model for client compatibility.
• GET /healthz — liveness.
• Loopback-only debug surface (moved from internal/fleet/debug.go): GET /debug/blocks, POST /debug/run-block, GET / (the step UI). Bound to loopback via ValidateLoopbackAddr (debug.go:223), never on the public port.

Sandbox lives here. cmd/codellm/main.go calls MaybeRunSandboxChild() first (the re-exec sandbox works by re-execing the current binary with a marker env var, sandbox_linux.go:45), so it re-execs itself, not fleet. Per-block runs still get the full native posture (namespaces + Landlock + rlimits + uid drop). --allowed-programs, --sandbox, --sandbox-network, --max-stdout-bytes, --interpreters move to codellm's flags.

Staying a real OpenAI endpoint. Because codellm accepts the standard request and returns a standard tool_calls response, fleet can point the same code role at a real LLM base_url instead and it "just works" as an actual AI agent. The code executor becomes swappable with a model — which is the deeper payoff of the whole design, and a good conformance test: run the same role against codellm and against a stub OpenAI server and diff the applied changes.

The debugger seam (design only — do not build now)

The owner's hard constraint: a future debugger must step the markdown block-by-block and inspect what is in the pipe between blocks (e.g. to write/validate a payload mid-pipeline). The design must not preclude it. codellm's internal execution model therefore treats each block and its inter-block pipe edge as an addressable, inspectable step — never a fused opaque run. The seam is half-built today — be precise about what exists, what is dead, and what moves:

1. Per-block build (exists, moves). runPipeline builds each block as a separate builtBlock with its own context/cancel and its own stdout/stderr buffers (runcode.go:161). Blocks are not merged into one command. This structure is what keeps stepping possible and must be carried over intact — do not collapse runPipeline into a single subprocess.
2. Tap-able edges (exists but DEAD CODE, moves + must be tested). Each inter-block connection is an io.Pipe; wirePipeline (runcode.go:220) threads an optional io.TeeReader per edge via pipelineDebugTaps []*limitedBuffer (runcode.go:159). No caller passes a non-nil tap — the parameter is always nil (runcode.go:90, :190), so the TeeReader path is never exercised and is untested. It is a latent seam, not a working one; the earlier claim that it is "already exercised for debug capture" was wrong. It must move and gain a test that actually drives a non-nil tap, or it will rot before the debugger arrives.
3. Step API (exists, only the inline path moves). /debug/run-block runs a single block with a caller-supplied stdin (the previous block's stdout) and returns stdout/stderr/exit/timing (debug.go:182); /debug/blocks enumerates (debug.go:94). The actual stepping mechanism today is this single-block RunBlock+stdin, not the TeeReader pipeline. But both endpoints have a ?path=/Role mode that looks the role up in fleet's live role registry (debug.go:102, :145, via f.roleByKey) — that binding is fleet-specific and cannot move to codellm, which has no registry. Only the inline ?body=/Body mode survives the extraction. codellm's debug surface therefore accepts inline markdown only; resolving a role name stays a fleet concern (fleet can render a role and POST the body to codellm's debug endpoint if needed).

Where a future debugger hooks: a stateful /debug/step session that (a) runs block i, (b) exposes the captured inter-block buffer for inspection and edit, (c) feeds the (possibly edited) buffer as block i+1's stdin. The primitives (per-block build, per-edge tap, single-block-with-stdin runner) exist; the future work is making them stateful/interactive and first making the tap seam a tested, live path rather than dead code.

Isolation model

The point is moving isolation from in-process confinement to a container boundary, in two layers:

• Layer 1 — the codellm container. codellm runs the untrusted code, so its container is the blast-radius boundary: its own PID/mount/network namespace, a cgroup memory/CPU limit, default-deny egress, no host mounts, dropped capabilities. codellm holds no long-lived KB handle, no auth, and (given the env-passthrough drop) no fleet secrets — so a compromise has far less to steal than in-process execution. But "nothing to steal" is overstated, and two exposures remain real:
  • Note content transits the boundary. Every request carries the rendered role body plus the delivery bag, which for change deliveries includes ChangedFiles and for cron/attach includes AttachedNotes — i.e. note content, not just metadata (buildInputBag, handler.go:227). A compromised or wiretapped codellm sees whatever notes the role's scope feeds it. This is why the channel must be authenticated/encrypted (risk 2), not merely "empty."
  • Scope checks the path, never the content. ScopedKB.Write/Patch enforce only that the path matches write_patterns (scope.go:141, :152); they do not inspect the bytes. A buggy or compromised codellm can therefore write arbitrary attacker-controlled content to any in-scope path. Scope confinement bounds where it can write, not what. The container boundary limits exfiltration; it does not make a misbehaving codellm safe to trust for content integrity — that is the fleet operator's residual risk.
• Layer 2 — the per-block sandbox (moved sandbox_linux.go): empty net + PID + mount namespaces, private /proc, Landlock FS confinement to the run workdir, rlimits, NO_NEW_PRIVS, uid drop. Still applied per code block, fail-closed in native mode.

Determinism — state it precisely. "100% deterministic" holds only under: no network egress (SandboxPolicy.Network defaults to false, sandbox.go:40), controlled resources, and a throwaway workdir per run (coderun.go:160). That guarantees no external nondeterminism (no network, no cross-run FS contamination). It does not guarantee bit-reproducibility: the sandbox does not freeze the wall clock or seed the PRNG, so time.now() / random() still vary. If a consumer needs reproducibility, codellm must additionally inject a fixed clock/seed — not designed here, flagged as an open question.

Container/userns interaction (gotcha). The per-block sandbox needs unprivileged user namespaces, which some container runtimes disable. If userns is unavailable inside the codellm container, native mode fails closed (refuses to run). Either the container must permit userns, or codellm relies on the container as the sole boundary and runs blocks in besteffort/off — an explicit, logged operator choice, not a silent downgrade.

Backward-compat and cutover

Migration of executor: code roles — zero role edits. Keep executor: code as author-facing sugar; swap only the backend. Fleet holds two LLM clients: f.llm (real model, cmd/fleet/main.go:80) and a new f.codeLLM (an OpenAILLM pointed at --codellm-base-url). execRole's code branch (handler.go:376) builds an agentruntime.Input pointed at codellm instead of calling f.codeRunner. The Input must be fully specified — Run hard-requires both MaxTokens > 0 and MaxSteps > 0 (runtime.go:110, :113) or it returns an error immediately, so a naive migration that omits them breaks every code role:

agentruntime.Input{
    Instruction:   p.Instr,                      // rendered role body (markdown+code)
    WritePatterns: p.Role.WritePatterns,         // unchanged; scope stays in fleet
    Tools:         []string{"write_note", "patch_note", "finish"}, // or role's declared tools
    MaxTokens:     codeRoleMaxTokens,             // must be > 0; a small fixed floor (codellm reports 0 usage)
    MaxSteps:      1,                             // HARD: single-shot; caps the re-execution hazard
    InputBag:      p.InputBag,                    // the delivery bag (new field)
    LLM:           f.codeLLM,
    KB:            kb,
}

MaxSteps: 1 is the re-execution defense (above); MaxTokens just has to be positive (codellm returns zero usage, so the value is a formality but cannot be 0). Role notes are untouched; the executor: code keyword now means "single-step llm-run against codellm."

Two migration constraints roles must satisfy (checklist for the reconciler's dry-run, main.go:546):

1. The role's effective tool set must include write_note/patch_note/finish (empty tools: already yields the full default set at runtime.go:334, so most roles need nothing).
2. write_patterns must be set for any role that writes — otherwise ScopedKB denies every change (the existing deny-all trap, role.go:109), same as today.
3. No env_passthrough/env_prefix. A role that declares either (role.go:64-65) depends on a secret reaching its code child; that capability does not exist under codellm (see the env-passthrough drop). Such roles cannot migrate and must stay on the in-process path or be redesigned. The reconciler dry-run should flag them; see the pre-Phase-4 audit.

Does the exec(program, code) tool move too? Separate, smaller concern. exec (makeExecInvoker, runtime.go:372) is a tool an LLM role calls mid-reasoning, not a whole-role code run. Initial cutover targets executor: code only and leaves exec in-process. Consequence: to fully delete the sandbox from fleet, exec must also move (Phase 5) — until then fleet keeps sandbox_linux.go solely for exec. This is a genuine scope decision, flagged not papered over: shipping Phases 1–4 removes the code-role path and the debugger need; Phase 5 finishes the job by routing exec through codellm and dropping the sandbox from fleet entirely.

Feature flag / rollout. --codellm-base-url empty → keep the in-process RunCode path (old behavior). Non-empty → route executor: code through codellm. Per-deploy toggle enables A/B parity checks before deletion. Optionally per-role opt-in during transition.

Fleet config after cutover. Adds --codellm-base-url (and --codellm-api-key or mTLS material if codellm authenticates callers). --allowed-programs, --sandbox, --sandbox-network, --max-stdout-bytes, --interpreters move to codellm; fleet keeps them transitionally only for exec until Phase 5.

Phased implementation

Each phase is independently shippable and reviewable (a candidate PR).

• Phase 1 — protocol + service scaffold. New cmd/codellm + internal/codellm. Implement POST /v1/chat/completions that extracts blocks and runs them in-process (reuse a copy of coderun/runPipeline), returning tool_calls. Ship with a golden-file conformance suite: request → response schema, {changes}→tool_calls mapping, error→422. TDD applies strongly here — the wire contract is the spec; write the golden request/response tests first. No fleet change yet.
• Phase 2 — move sandbox + interpreters + debug surface. Move sandbox*.go, interpreters.json, and the /debug/* endpoints into codellm; wire MaybeRunSandboxChild() first in cmd/codellm/main.go. Port sandbox_linux_test.go (TDD: sandbox behavior is security-critical). codellm now sandboxes each block and is steppable via the debug API.
• Phase 3 — fleet cutover behind a flag. Add f.codeLLM + the InputBag field on agentruntime.Input and its message builder. Build the code-role Input with MaxSteps: 1 and a positive MaxTokens (above). Route executor: code through codellm when --codellm-base-url is set. Add parity tests: the same role note produces the same applied changes via the in-process path and via codellm (run both, diff res.Changes), including the apply-error path (a bad-find patch must fail both ways) and a test asserting the code-role Input has MaxSteps == 1. Roll out with the flag off, flip per-deploy.
• Pre-Phase-4 audit. Before deleting the in-process path, grep prod role notes for env_passthrough: / env_prefix: usage (the reconciler dry-run, or a one-off scan of the roles/ folder). Any live role using them cannot run under codellm and would break silently on cutover — resolve each (redesign or keep in-process) before removal.
• Phase 4 — remove the executor: code in-process branch. Delete the codeRunner field (fleet.go:24) and the code branch in execRole (handler.go:376); executor: code now routes only through codellm. Fleet still imports RunBlock/sandbox*.go at this point — the exec tool depends on them (runtime.go:384) — so this phase removes the code-role path, not the packages.
• Phase 5 (deferred) — move exec and finish the isolation. Route the exec tool through codellm, then delete sandbox*.go/coderun.go/runcode.go from fleet, drop MaybeRunSandboxChild() from cmd/fleet/main.go:36, and remove the sandbox/program flags. Only now does fleet stop importing the code-exec/sandbox packages entirely; fleet no longer executes any code.

Risks and open questions

1. Determinism ≠ reproducibility. The sandbox controls fs/net/resources but not wall-clock/PRNG. "Deterministic" means "no external nondeterminism," not bit-reproducible. If a consumer needs reproducibility, codellm must inject a fixed clock/seed — undesigned. Open.
2. codellm is a code-execution oracle, and note content crosses to it. An unauthenticated /v1/chat/completions is RCE-as-a-service; the fleet↔codellm channel must be authenticated (shared token / mTLS) or loopback/private-network only. Note content (ChangedFiles/AttachedNotes) transits the request, and a compromised codellm can write arbitrary content to any in-scope path (scope checks path, not bytes — scope.go:141). "Empty service" reduces the theft surface but does not eliminate content-exposure or content-integrity risk. Must decide auth before exposing.
3. tool_call mapping fidelity + apply-error divergence. {changes}→write_note/patch_note must be lossless (path/content/find/replace), finish must carry the answer and stay last (and always be present — the re-execution invariant), and the migrated role must declare the write tools + write_patterns or the loop's allowlist rejects codellm's calls (runtime.go:180). Separately, apply errors (bad-find patch) are a hard 502 today but a soft skip under the loop — decide and pin the canonical behavior (recommend codellm returns 422 to preserve hard-fail). Both covered by parity tests; the highest-churn correctness surface.
4. Streaming over HTTP / timeouts. The block→block pipeline stays inside codellm (good), but the fleet↔codellm call is one request/response held open for the whole run. The trip2g→fleet→codellm timeout chain must align: the role's EffectiveTimeoutSeconds (default 300, role.go:89) must bound codellm's run and fleet's HTTP client to codellm, or a long pipeline is cut mid-run. Wire all three.
5. Small prod VM. Another always-on service + container on a box already under memory pressure (see prod_box_constraints). The RLIMIT_AS default of 4 GiB (sandbox.go:52) is virtual address space, not RSS — a Node process reserves far more virtual than it resident-uses, so that number is not a direct OOM predictor and should not be read as "4 GiB per run." The real control is a concurrency cap on codellm (max simultaneous block runs) sized to the box's actual RAM, plus off-peak deploys; rlimits are a backstop, not the budget.
6. How codellm is itself sandboxed/deployed. The container is the real boundary; the in-process per-block sandbox needs unprivileged userns, which the container runtime may disable (see Isolation gotcha). Decide: permit userns in the container, or treat the container as the sole boundary with besteffort.
7. System-prompt wrapper. agentruntime.Run wraps the body in the micro-agent system prompt + "Begin." (runtime.go:81). codellm ignores the wrapper and scans for fenced blocks — works because the wrapper has no fences, but slightly hacky. Alternative: a dedicated Input mode that sends the raw body without the agent wrapper. Minor; decide during Phase 3.