Deploying the harness

Who does what:

On SignalPilot Cloud, the harness is already deployed. Evals are included on every paid plan; everything you need is on the Evals page. Skip to Running evals on Cloud.
Self-hosting? You supply the runner image, the evidence bucket, an agent credential, and, for write tasks, a branch provider. That is the rest of this page.

What a run needs

Requirement	Why
Runner image	Executes each task. Ships the Claude CLI, the SignalPilot plugin (skills and verifier agents) and a dbt toolchain; `psql`, `git`, and `curl` must be present for setup scripts and project delivery.
Agent credential	`SP_EVAL_CLAUDE_TOKEN` (OAuth) or `SP_EVAL_ANTHROPIC_KEY`.
Evidence bucket (S3 or MinIO)	Transcripts, logs, captures. A run refuses to start without one.
An execution backend	Docker Engine locally; Vercel sandboxes (`SP_EVAL_EXECUTION_BACKEND=vercel`) in cloud mode.
A pinned warehouse connection	Chosen per workspace on the Evals page.
A branch provider (write tasks only)	Xata connection, or Postgres admin DSN + parent database.

Turning it on

The feature is off until SP_EVAL_RUNNER_IMAGE is set. Minimum viable self-hosted configuration:

.env: minimum eval configuration

SP_EVAL_RUNNER_IMAGE=your-registry/sp-eval-runner:1.4.0
SP_EVAL_CLAUDE_TOKEN=...            # or SP_EVAL_ANTHROPIC_KEY
SP_EVAL_S3_BUCKET=sp-eval-evidence
SP_EVAL_S3_ENDPOINT=http://minio:9000
SP_EVAL_S3_ACCESS_KEY=...
SP_EVAL_S3_SECRET_KEY=...
SP_EVAL_MCP_URL=http://gateway:3300/mcp

Add a branch provider if the set has write tasks. SP_EVAL_PG_ADMIN_DSN is a libpq connection URI: postgresql scheme, the admin role and its password in the userinfo section, and the maintenance database as the path:

.env: local Postgres branch provider

SP_EVAL_PG_ADMIN_DSN=postgresql://warehouse-host:5432/postgres
SP_EVAL_PG_PARENT_DB=northwind

The local provider has two hard prerequisites. Read both before pointing it at a database you care about.

:::danger Forking terminates every session on the parent database Branches are forked with CREATE DATABASE <branch> TEMPLATE <parent>, which Postgres refuses while the template has sessions. The provider responds by running pg_terminate_backend against every connection to the parent database, including connections that have nothing to do with evals, and retrying, up to six times. Point SP_EVAL_PG_PARENT_DB at a database whose connections are expendable, and do not share that server with anything production. :::

The second prerequisite is that PUBLIC must not hold CONNECT on the other databases of that server. Each task gets a fresh least-privilege role, and the provider checks what that role can still reach; if it can reach anything besides its own branch, it aborts with:

Postgres eval isolation is not configured: PUBLIC CONNECT lets the task role reach <databases>. Revoke PUBLIC CONNECT on every non-eval non-template database before enabling the local branch provider.

revoke PUBLIC CONNECT on a database

REVOKE CONNECT ON DATABASE some_other_db FROM PUBLIC;

The reaper that cleans up abandoned branches matches eval-* databases across the whole server, not per organization. A dedicated Postgres server for eval branches is the configuration this was designed for.

Building a runner image

Dockerfile.eval-runner

FROM node:22-slim
# Claude CLI, psql/git/curl for setup scripts, msodbcsql18 for dbt-sqlserver
RUN apt-get update && apt-get install -y --no-install-recommends     git ca-certificates curl gnupg postgresql-client python3 python3-venv libstdc++6 &&   ... msodbcsql18 unixodbc from packages.microsoft.com ... &&   npm install -g @anthropic-ai/claude-code
# dbt toolchain, same adapter set as the notebook image
RUN python3 -m venv /opt/sp-eval/.venv &&   /opt/sp-eval/.venv/bin/pip install --no-cache-dir uv &&   /opt/sp-eval/.venv/bin/uv pip install --python /opt/sp-eval/.venv/bin/python     dbt-core dbt-duckdb dbt-postgres dbt-snowflake dbt-sqlserver dbt-redshift
ENV PATH="/opt/sp-eval/.venv/bin:$PATH"
RUN useradd -m -u 10002 evalrunner && mkdir -p /work && chown evalrunner:evalrunner /work
# SignalPilot plugin: skills + verifier agents, loaded with --plugin-dir
COPY --chown=evalrunner:evalrunner signalpilot-plugin/.claude-plugin/ /opt/signalpilot-plugin/.claude-plugin/
COPY --chown=evalrunner:evalrunner signalpilot-plugin/skills/ /opt/signalpilot-plugin/skills/
COPY --chown=evalrunner:evalrunner signalpilot-plugin/agents/ /opt/signalpilot-plugin/agents/
ENV SP_AGENT_PLUGIN_PATH=/opt/signalpilot-plugin
USER evalrunner
WORKDIR /work
CMD ["claude", "--version"]

The full file is Dockerfile.eval-runner at the repository root; build it from there with docker build -f Dockerfile.eval-runner -t sp-eval-runner:latest .. The runner passes --plugin-dir $SP_AGENT_PLUGIN_PATH to the Claude CLI when that directory exists and puts the dbt venv on PATH, so an eval repo no longer has to vendor .claude/skills or install dbt in a setup script. Older images without either still work; the flags are skipped.

postgresql-client is not optional if any task ships a setup script: those scripts exist to mutate the branch, and without psql every one of them dies after the branch has already been forked. Use SP_EVAL_SETUP_IMAGE if you would rather keep script tooling in a separate image.

For cloud mode, push the image to Vercel Container Registry with VERCEL_REGISTRY=registry.vercel.com/<team> bash scripts/build-eval-runner-image-vercel.sh; it prints the digest-pinned SP_EVAL_RUNNER_IMAGE=...@sha256:... line to set.

Local (Docker) mode

Eval containers run on the host Docker daemon:

SP_EVAL_DOCKER_SOCKET: socket mounted into the gateway (default /var/run/docker.sock).
SP_EVAL_DOCKER_NETWORK: network the containers join. It must be able to reach the gateway at SP_EVAL_MCP_URL.
SP_EVAL_PROJECTS_DIR: root under which local-path eval sets may live (mounted read-only). Paths outside it are refused.
SP_EVAL_PROJECTS_HOST_DIR: the same directory as the host sees it. Required for local-path eval sets, because Docker binds resolve on the host, not inside the gateway container.

Local mode also accepts a filesystem path as the eval repo, useful while authoring a set. Cloud mode does not.

Container sizing in local mode is fixed: the agent container gets 2 GiB and 2 CPUs, a setup or teardown container gets 4 GiB and 4 CPUs, and /work, /tmp and /repo are 512 MB tmpfs mounts. The dbt project tarball unpacks into that 512 MB /work. A very large project is the one thing likely to hit it.

Cloud (Vercel) mode

When SP_DEPLOYMENT_MODE=cloud, the Docker path is unreachable by construction: the backend selector raises rather than falling back, so a misconfigured deployment can never hand eval workloads the host daemon. Cloud mode requires SP_EVAL_EXECUTION_BACKEND=vercel (plus VERCEL_TOKEN, VERCEL_TEAM_ID and VERCEL_PROJECT_ID). Each task becomes a short-lived, isolated Vercel sandbox microVM booted from the digest-pinned SP_EVAL_RUNNER_IMAGE, pushed with scripts/build-eval-runner-image-vercel.sh. The microVM has fixed CPU and memory, restricted egress, and no access to the gateway's database or object storage networks. The harness enforces:

Digest-pinned images. SP_EVAL_RUNNER_IMAGE and SP_EVAL_SETUP_IMAGE must end in @sha256:<64 hex>. A floating tag would let what runs in the sandbox change without the configuration changing. Look one up with crane digest <image> or docker buildx imagetools inspect <image>.
Secrets ride the exec environment only. Credentials never enter the sandbox creation spec, which is readable back from the provider API.
A public MCP URL. Sandboxes run in Vercel's network, not next to the gateway, so SP_EVAL_MCP_URL must be reachable from the internet.
Bounded lifetime. The sandbox is destroyed when the task ends, with the provider's execution time limit as backstop.
GitHub HTTPS repositories only. Cloud mode refuses local paths and SSH URLs for both the eval repo and the project repository.

Local-path eval sets and host bind mounts are refused on this backend; use a git eval repo with a self-contained setup script.

Running evals on Cloud

Nothing to deploy. What you control:

Enablement. Evals are on for every organization on a paid plan. If the page shows a plan prompt, your organization is on the free plan; if it shows a deployment notice, the deployment is not wired to run evals.
The eval repo and the project repository. Cloud mode accepts https://github.com/… only, no other host, no local paths, no SSH URLs. The same rule applies to both repositories. The project repository can pin a branch; leave it empty for the repository default branch.
Private repos. Connect GitHub from Settings → GitHub and install the app on the repositories you want to use. The gateway then clones them with a short-lived installation token scoped to your organization; a repository that your installation does not cover is refused, not silently attempted. Public repos need no setup. The credentialed URL never leaves the gateway; sandboxes receive a presigned tarball instead of a git credential.
The pinned connection. Add the warehouse under Connections, then pin it on the Evals page. Read-only credentials are correct unless you run write tasks; write tasks need a connection whose provider can fork branches.
Notification emails, on the same form.

Environment variables

Every variable the eval harness reads. See the configuration reference for the rest of the gateway.

Core

Variable	Default	Description
`SP_EVAL_RUNNER_IMAGE`	unset	Image with the Claude CLI. Empty disables the feature. Must be digest-pinned in cloud mode.
`SP_EVAL_SETUP_IMAGE`	runner image	Image for setup/teardown script containers. Digest-pinned in cloud mode.
`SP_EVAL_EXECUTION_BACKEND`	`""`	Empty runs tasks on the local Docker Engine. `vercel` runs them on the hosted sandbox backend. Cloud mode requires `vercel`.
`SP_EVAL_MCP_URL`	`http://gateway:3300/mcp`	MCP endpoint the sandboxes call back on.
`SP_EVAL_CLAUDE_TOKEN`	unset	`CLAUDE_CODE_OAUTH_TOKEN` passed to task containers. Falls back to `CLAUDE_KEY_1` if unset.
`SP_EVAL_ANTHROPIC_KEY`	unset	Alternative: `ANTHROPIC_API_KEY` for task containers.
`SP_EVAL_TIMEOUT_SECONDS`	`600`	Per-task agent container timeout.
`SP_EVAL_PROJECT_ENV`	`""`	Extra `KEY=VALUE` entries (newline- or comma-separated) injected into agent containers, for example read-only warehouse credentials a `profiles.yml` reads via `env_var`. Gateway-controlled variables always win. Setup and teardown containers do not receive these; give those what they need through the manifest's `setup.env_file`.

Execution: local Docker

Variable	Default	Description
`SP_EVAL_DOCKER_SOCKET`	`/var/run/docker.sock`	Docker Engine socket mounted into the gateway.
`SP_EVAL_DOCKER_NETWORK`	`signalpilot_eval_runtime`	Network eval containers join; must reach the gateway.
`SP_EVAL_PROJECTS_DIR`	`/eval-projects`	Root for local-path eval sets, mounted read-only.
`SP_EVAL_PROJECTS_HOST_DIR`	unset	Host path of the above; required to bind-mount local eval repos into script containers.
`SP_EVAL_SETUP_HOST_ROOT`	unset	Declared but not currently used by any code path. Leave unset.

Execution: cloud Vercel

Variable	Default	Description
`SP_EVAL_EXECUTION_BACKEND`	unset	`vercel` runs each task on an ephemeral Vercel sandbox. Required in cloud mode; also honoured locally.
`VERCEL_TOKEN` / `VERCEL_TEAM_ID` / `VERCEL_PROJECT_ID`	unset	Sandbox provider credentials. The backend refuses to start without all three.

Evidence store

Variable	Default	Description
`SP_EVAL_S3_BUCKET`	unset	Bucket for transcripts, setup logs, captures, exports. Empty blocks every run.
`SP_EVAL_S3_ENDPOINT`	unset	S3 endpoint. Set for MinIO or another S3-compatible service; leave unset for a hosted S3 service.
`SP_EVAL_S3_RUNNER_ENDPOINT`	unset	Optional GET-only endpoint embedded in URLs handed to sandboxes, while the gateway keeps using the private endpoint.
`SP_EVAL_S3_REGION`	`us-east-1`	Region.
`SP_EVAL_S3_ACCESS_KEY` / `SP_EVAL_S3_SECRET_KEY`	unset	Dedicated credentials, deliberately separate from BYOK or warehouse credentials.

Branch provider

Variable	Default	Description
`SP_EVAL_PG_ADMIN_DSN`	unset	Admin DSN used to fork branch databases when the pinned connection is not a Xata connection.
`SP_EVAL_PG_PARENT_DB`	unset	Parent database each branch is templated from.

Xata-backed sets need no variables: the project, branch, and control-plane credentials come from the pinned connection's stored extras.

Limits and quotas

Variable	Default	Description
`SP_EVAL_MAX_PARALLEL_TASKS`	`4`	Tasks in flight within a run. Bounds node capacity and model spend.
`SP_EVAL_MAX_BRANCHES`	`50`	Ceiling on live eval branches, enforced before each fork.
`SP_EVAL_BRANCH_STORAGE_DELTA_BYTES`	`5368709120` (5 GiB)	How far a branch may outgrow its parent before the task fails.
`SP_EVAL_ARTIFACT_BYTES_PER_RUN`	`1073741824` (1 GiB)	Artifact budget per run; captures past it are marked truncated.
`SP_EVAL_CAPTURE_FULL_MAX_BYTES`	`268435456` (256 MiB)	Estimated-size ceiling for a `full` capture; over it the capture is refused up front. Note a separate hard ceiling of 128 MiB applies to any single capture file, so a value above that leaves a band where captures truncate instead of being refused.
`SP_EVAL_SETUP_TIMEOUT_SECONDS`	`1800`	Default setup/teardown timeout; a manifest may lower it.

Regression notifications

Variable	Default	Description
`SP_EVAL_REGRESSION_DROP_PCT`	`10.0`	Points below the trailing median that count as a regression.
`SP_EVAL_SMTP_HOST`	`""`	SMTP host. Empty falls back to the platform's default mail transport.
`SP_EVAL_SMTP_PORT`	`1025`	SMTP port.
`SP_EVAL_NOTIFY_FROM`	`evals@your-domain.example`	From address on regression email.

Eval-set uploads

Optional intake path for customers mailing in an eval set as a zip (/evals/upload).

Variable	Default	Description
`SP_EVAL_UPLOADS_BUCKET`	unset	Bucket for uploaded archives. Empty disables the page's backend.
`SP_EVAL_UPLOADS_MAX_MB`	`8192`	Upload ceiling.
`SP_EVAL_UPLOADS_S3_ENDPOINT`	unset	S3 endpoint for uploads.
`SP_EVAL_UPLOADS_S3_PUBLIC_ENDPOINT`	unset	Endpoint the browser uses for multipart PUTs.
`SP_EVAL_UPLOADS_S3_REGION`	unset	Region.
`SP_EVAL_UPLOADS_S3_ACCESS_KEY` / `…_SECRET_KEY`	unset	Upload credentials.
`SP_EVAL_UPLOADS_NOTIFY_EMAIL`	unset	Where upload notifications go.
`SP_EVAL_UPLOADS_NOTIFY_FROM`	`eval-uploads@your-domain.example`	From address.
`SP_EVAL_UPLOADS_SMTP_HOST` / `…_SMTP_PORT`	`""` / `1025`	SMTP for upload notifications.

Variable	Relevance
`SP_DEPLOYMENT_MODE`	`cloud` requires `SP_EVAL_EXECUTION_BACKEND=vercel` (the Vercel backend runs the digest-pinned `SP_EVAL_RUNNER_IMAGE`), requires digest-pinned images, and restricts repos to GitHub HTTPS.
`SP_BACKEND_URL`	When set, the organization's plan decides access to evals (paid plans only). When unset, the local organization is unlimited.
`SP_GITHUB_APP_*`	Needed to clone private eval or project repositories.

Isolation and data handling

Worth knowing before you point this at production data:

Per-task credentials. Each task container gets its own API key, bound server-side to that run, task, connection, and knowledge overlay, and expiring after 3 hours. The agent cannot widen its own scope with headers; the binding is read from the stored key, never from the request. Keys are revoked when the task ends and when the run ends.
Read-only by default. Governed MCP blocks DDL/DML at parse time. Write access exists only as a branch DSN, only for write tasks, only on a disposable branch.
Redaction. Transcripts and script logs are scrubbed before storage: the exact secrets the harness injected, DSN user-info, auth headers, and long base64-looking blobs. It is a heuristic, not a guarantee. A secret the harness never knew about (one your setup script prints, say) can survive it.
The project tarball is deleted once every task has fetched it, and is excluded from run exports.
Retention prunes artifacts to the last 10 runs and transcripts to the last 100 runs per workspace; accuracy history is kept.

Troubleshooting

Symptom	Cause
`runner disabled — SP_EVAL_RUNNER_IMAGE unset on gateway`	The feature is off.
`Eval evidence bucket is not configured (SP_EVAL_S3_BUCKET)`	No evidence store.
`An eval connection pin is required` / `does not exist in this workspace`	Set Connection on the Evals page to a real connection name.
`402 {"error": "plan_required"}`	The organization is on the free plan.
`503 {"error": "not_available_in_deployment", "capability": "evals"}`	The runner image, evidence bucket, or (in cloud mode) the `vercel` backend is not configured.
`only https://github.com/ repositories are allowed`	Cloud mode; use a GitHub HTTPS URL.
`local paths must live under /eval-projects`	Move the set under `SP_EVAL_PROJECTS_DIR`.
`no branch provider: pin a Xata connection …`	The set has write tasks but no provider is configured.
`warehouse does not match the eval set's build fingerprint`	Rebuild the warehouse, or clear `build_fingerprint` in the manifest.
`psql: not found` in a setup log	The setup image lacks `postgresql-client`.
Every write task is `SETUP_FAILED`	Read the setup log in the evidence store; it is stored even when the agent never ran.
`Too many eval runs in flight`	Two runs already active on the gateway.

What a run needs​

Turning it on​

Building a runner image​

Local (Docker) mode​

Cloud (Vercel) mode​

Running evals on Cloud​

Environment variables​

Core​

Execution: local Docker​

Execution: cloud Vercel​

Evidence store​

Branch provider​

Limits and quotas​

Regression notifications​

Eval-set uploads​

Related gateway variables​

Isolation and data handling​

Troubleshooting​