Skip to content

Running the worker ​

codemode-worker is a small binary that runs one program and exits. The process isolation starts one per run, so a program executes in its own process instead of inside your application. This page is how to install it, configure it, check it is working, and run it in a container.

You do not need it for development. Leave SANDBOX_ISOLATION unset (or inprocess) and programs run in-process with goja; see Development and production.

What changes when you turn it on ​

SANDBOX_ISOLATION=inprocess (default)     SANDBOX_ISOLATION=process

your app                                  your app                         codemode-worker
 ├─ agent.Run                              ├─ agent.Run                      └─ goja
 ├─ goja ← program runs here               ├─ registry, roles, budgets   ◄──►   program runs here
 └─ ExecuteFunc                            └─ ExecuteFunc                 JSON lines on stdin/stdout
In-processWorker
Language, prompt, programsgoja / Starlark / Luathe same
A program that allocates foreverexhausts your app's memorythe worker exits; your app returns an error
Your environment (API keys, DB URLs)visible to the process the program runs inthe worker gets an empty environment
Registry check, roles, call budget, response limit, timeoutin your appstill in your app
Debugging with breakpointsin ExecuteFunc and the enginein ExecuteFunc only
Cost per runnonea process start (milliseconds)
Numbers in the resultint64 or float64always float64 (results cross JSON)

The important row is the fourth. Everything that decides what a program may do stays in your application. The worker only interprets; every capability call it makes is a message your application checks before anything runs.

Step 1: install it ​

From your application's module (the directory with its go.mod):

sh
go get -tool github.com/mukezhz/codemode/cmd/codemode-worker   # once
go install github.com/mukezhz/codemode/cmd/codemode-worker     # → $GOBIN/codemode-worker

The first command records the worker in your go.mod as a tool, at the same codemode version as the library you already use, and fetches the interpreters it links (all three, even if your app imports one). The second builds it at exactly that version. Upgrade codemode with go get and the worker follows.

Keep the worker on the same codemode version as your application. The two speak a versioned protocol; a mismatch fails every run with protocol version N, worker speaks M, and sandbox.New reports it at startup.

Other ways to get it:

sh
# Inside the codemode repository
make worker            # → bin/codemode-worker

# A release binary: pick the release matching the codemode version in your go.mod
VERSION=$(go list -m -f '{{.Version}}' github.com/mukezhz/codemode)
curl -L -o codemode-worker \
  "https://github.com/mukezhz/codemode/releases/download/$VERSION/codemode-worker-linux-amd64"
chmod +x codemode-worker

Release binaries exist for tagged versions only (linux and darwin, amd64 and arm64, with SHA256SUMS). If your go.mod points at an untagged commit, use the go install route.

In a container image, build both in one stage so the versions cannot drift (this needs the go get -tool line above committed in your go.mod):

dockerfile
FROM golang:1.27 AS build
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 go build -o /out/app . && \
    CGO_ENABLED=0 go build -o /out/codemode-worker github.com/mukezhz/codemode/cmd/codemode-worker

FROM gcr.io/distroless/static
COPY --from=build /out/app /app
COPY --from=build /out/codemode-worker /usr/local/bin/codemode-worker
ENTRYPOINT ["/app"]

go build github.com/mukezhz/codemode/cmd/codemode-worker uses the version in your go.mod, which is what keeps them matched.

Step 2: import the engine ​

Blank-import sandbox/subprocess and the language engine. The language engine is still needed in your app: it writes the prompt.

go
import (
	_ "github.com/mukezhz/codemode/sandbox/goja"       // prompt and in-process runs
	_ "github.com/mukezhz/codemode/sandbox/subprocess" // SANDBOX_ISOLATION=process
)

Importing both costs nothing at runtime. Configuration chooses which one runs, so the same binary serves development and production.

Step 3: configure it ​

From the environment, with sandbox.NewFromEnv:

sh
SANDBOX_ENGINE=goja                             # the language: goja, starlark or lua
SANDBOX_ISOLATION=process                       # run it in codemode-worker
SANDBOX_WORKER=/usr/local/bin/codemode-worker   # optional when it is on PATH
SANDBOX_MAX_HEAP_BYTES=268435456                # 256 MiB, the default
SANDBOX_TIMEOUT=10s
SANDBOX_MAX_CALLS=25
go
runner, err := sandbox.NewFromEnv(sandbox.Options{})

Or in code, which wins over the environment:

go
runner, err := sandbox.New(sandbox.Options{
	Engine:       "goja",
	Isolation:    "process",
	Worker:       []string{"/usr/local/bin/codemode-worker"},
	MaxHeapBytes: 256 << 20,
	Timeout:      10 * time.Second,
	MaxCalls:     25,
})
defer runner.Close()
OptionEnvDefaultMeaning
EngineSANDBOX_ENGINEgojathe language: goja, starlark, lua
IsolationSANDBOX_ISOLATIONinprocessprocess runs every program in a worker
WorkerSANDBOX_WORKERcodemode-worker on PATHargv that starts a worker; split on spaces from the env
WorkerContainerSANDBOX_WORKER_CONTAINERfalsetrue when Worker starts a container (step 5)
MaxHeapBytesSANDBOX_MAX_HEAP_BYTES256 MiBworker exits when its heap passes this
TimeoutSANDBOX_TIMEOUT30 swhole program, enforced by your app
MaxCallsSANDBOX_MAX_CALLS50capability calls per program, counted by your app
MaxResponseBytes—1 MiBeach capability response, and the final value

Nothing else changes: agent.Run, mcpserver.New and runner.Run take the runner exactly as before.

Step 4: check it is working ​

sandbox.New checks for you. With process isolation it starts one worker and runs return 1 through it before returning, so a missing binary, a version mismatch, a bad docker run or a missing image is an error from New, at startup, not a failure on the first user request. It also refuses settings that look like a mistake (see Troubleshooting).

Log the runner's name at startup. It states what is enforcing the boundary:

go
slog.Info("sandbox", "runner", runner.Name())
goja / javascript (child process, 10s timeout, 25 call budget, 256 MiB heap)

If it says in-process, the environment did not reach the binary.

Then run one program by hand, with no application involved:

sh
echo '{"v":1,"type":"run","engine":"goja","source":"return 1+1","limits":{"timeout_ms":1000,"max_calls":1,"max_response_bytes":1024,"max_heap_bytes":67108864}}' \
  | codemode-worker
{"type":"done","value":2}

And check that a runaway program costs only the worker:

go
_, err := runner.Run(ctx, mode, `const a = []; while (true) a.push(new Array(1e6).fill(1));`)
// err: sandbox: memory limit of 268435456 bytes exceeded

What the worker does on each platform ​

macOSLinuxLinux in a container
Own process, empty environment, empty working directory✓✓✓
Heap cap (MaxHeapBytes)✓✓✓ plus --memory
Killed with its whole process group on timeout✓✓the container exits on its own
CPU and open-file limits—✓✓
Seccomp: no sockets, exec, file opens, ptrace, mounts—✓✓
No network, private mounts (user namespaces)—where allowed✓ --network none
Separate kernel——with --runtime runsc (gVisor)

"Where allowed": many kernels and container platforms disable unprivileged user namespaces. The worker then runs without them rather than failing, and the other rows still hold.

Step 5: a container per run ​

For tenants you do not trust, start each worker in its own container. Same protocol, same configuration, different argv.

Build the worker image (scratch: the binary and nothing else):

sh
docker build -f deploy/worker.Dockerfile -t ghcr.io/you/codemode-worker .

Point the runner at it:

sh
SANDBOX_ENGINE=goja
SANDBOX_ISOLATION=process
SANDBOX_WORKER_CONTAINER=true
SANDBOX_WORKER="docker run -i --rm --network none --read-only --cap-drop ALL \
  --security-opt no-new-privileges --user 65534:65534 \
  --memory 256m --memory-swap 256m --cpus 0.5 --pids-limit 32 \
  --runtime runsc ghcr.io/you/codemode-worker"
FlagWhy
-ithe protocol is stdin/stdout
--rmone run, one container, nothing reused
--network none, --read-onlynothing to reach, nothing to write
--cap-drop ALL, no-new-privileges, --user 65534no privileges to escalate from
--memory, --memory-swapthe hard memory cap behind MaxHeapBytes
--cpus, --pids-limitone program cannot starve the node
--runtime runscgVisor: a separate kernel; omit only where it is not installed

SANDBOX_WORKER_CONTAINER=true is required. Without it your app wraps the docker command in new namespaces, which cuts it off from the Docker socket.

Each run pays a container start, roughly 150–400 ms. A worker that outlives its timeout (because the docker client was killed) exits on its own one second later.

The Docker socket is root

Your application needs to start containers. Mounting /var/run/docker.sock into an application container gives that container root on the node. Run the application on a VM or node you treat as privileged, or set SANDBOX_WORKER to a runtime that does not need a root daemon (podman run ..., nerdctl run ...).

Test it the way CI does:

sh
make test-container                # the whole engine suite, each run in a container
make test-container RUNTIME=runsc  # under gVisor

Troubleshooting ​

ErrorCauseFix
process isolation needs codemode-worker: install it on PATH or set SANDBOX_WORKERworker not found on PATHinstall it, or set SANDBOX_WORKER to its path
codemode-worker check failed (...): ...the startup check could not run a program through the worker: wrong path, not a worker, version mismatch, bad docker run, missing imagethe error after the colon says which; fix it and restart
sandbox isolation "process" is not registered: add _ ".../sandbox/subprocess"SANDBOX_ISOLATION=process without the importblank-import sandbox/subprocess (step 2)
sandbox engine "goja" is not registered: add _ ".../sandbox/goja"only sandbox/subprocess importedblank-import the language engine too (step 2)
a worker is configured but isolation is inprocessSANDBOX_WORKER or SANDBOX_WORKER_CONTAINER set, SANDBOX_ISOLATION notset SANDBOX_ISOLATION=process; this is refused because running in-process here is the unsafe way to be wrong
worker container is set but no worker command isSANDBOX_WORKER_CONTAINER=true aloneset SANDBOX_WORKER to the docker run argv
protocol version 1, worker speaks 2app and worker from different codemode versionsbuild both from the same go.mod
sandbox: memory limit of N bytes exceededthe program allocated past MaxHeapBytesthe program's fault; raise the cap only if real data needs it
sandbox stopped: context deadline exceededthe program ran past Timeoutsame; it is a runaway or the timeout is too tight
worker protocol error: ...the worker sent something malformed or died mid-runthe stderr tail is attached; a crash here is a bug worth reporting
worker protocol error: EOF with permission denied ... docker in the stderr tailthe docker CLI was wrapped in namespaces, or the app cannot reach the daemonset SANDBOX_WORKER_CONTAINER=true; check the app's access to the runtime
stderr: seccomp unavailable, running without itkernel or emulator cannot load filtersexpected under emulation; on a real host, check the kernel config

How it works, briefly ​

The worker reads one run message with the source and limits, applies its own limits, then runs the same engine your app would have run in-process. That engine is given a CodeMode whose Execute writes a call line and waits for the result line. Your app answers each call with its real CodeMode, so the registry, roles, Recorder and OnEvent all work unchanged. A done or error line ends the run, and your app kills the worker's process group either way.

Your app trusts nothing the worker says. It counts calls itself, checks every message's type and id, caps every line's size, and enforces the timeout from outside. A program that escapes the interpreter controls the worker, and the worker still has nothing to call but your app's checked bridge.

Design record: notes/decisions/0005-subprocess-runner.md.