Deploying the harness

Who does what:

• On SignalPilot Cloud, the harness is already deployed. Evals are opt-in per workspace. Ask us to enable yours, then everything you need is on the Evals page. Skip to Running evals on Cloud.
• Self-hosting? You supply the runner image, the evidence bucket, an agent credential, and, for write tasks, a branch provider. That is the rest of this page.

What a run needs

Requirement	Why
Runner image with the Claude CLI	Executes each task. psql, git, and curl must be present for setup scripts and project delivery.
Agent credential	SP_EVAL_CLAUDE_TOKEN (OAuth) or SP_EVAL_ANTHROPIC_KEY.
Evidence bucket (S3 or MinIO)	Transcripts, logs, captures. A run refuses to start without one.
An execution backend	Docker Engine locally, or the hosted sandbox backend (SP_EVAL_EXECUTION_BACKEND=vercel).
A pinned warehouse connection	Chosen per workspace on the Evals page.
A branch provider (write tasks only)	Xata connection, or Postgres admin DSN + parent database.

Turning it on

The feature is off until SP_EVAL_RUNNER_IMAGE is set. Minimum viable self-hosted configuration:

.env: minimum eval configuration

SP_EVAL_RUNNER_IMAGE=your-registry/sp-eval-runner:1.4.0
SP_EVAL_CLAUDE_TOKEN=...            # or SP_EVAL_ANTHROPIC_KEY
SP_EVAL_S3_BUCKET=sp-eval-evidence
SP_EVAL_S3_ENDPOINT=http://minio:9000
SP_EVAL_S3_ACCESS_KEY=...
SP_EVAL_S3_SECRET_KEY=...
SP_EVAL_MCP_URL=http://gateway:3300/mcp

Add a branch provider if the set has write tasks. SP_EVAL_PG_ADMIN_DSN is a libpq connection URI: postgresql scheme, the admin role and its password in the userinfo section, and the maintenance database as the path:

.env: local Postgres branch provider

SP_EVAL_PG_ADMIN_DSN=postgresql://warehouse-host:5432/postgres
SP_EVAL_PG_PARENT_DB=northwind

The local provider has two hard prerequisites. Read both before pointing it at a database you care about.

:::danger Forking terminates every session on the parent database Branches are forked with CREATE DATABASE <branch> TEMPLATE <parent>, which Postgres refuses while the template has sessions. The provider responds by running pg_terminate_backend against every connection to the parent database, including connections that have nothing to do with evals, and retrying, up to six times. Point SP_EVAL_PG_PARENT_DB at a database whose connections are expendable, and do not share that server with anything production. :::

The second prerequisite is that PUBLIC must not hold CONNECT on the other databases of that server. Each task gets a fresh least-privilege role, and the provider checks what that role can still reach; if it can reach anything besides its own branch, it aborts with:

Postgres eval isolation is not configured: PUBLIC CONNECT lets the task role reach <databases>. Revoke PUBLIC CONNECT on every non-eval non-template database before enabling the local branch provider.

revoke PUBLIC CONNECT on a database

REVOKE CONNECT ON DATABASE some_other_db FROM PUBLIC;

The reaper that cleans up abandoned branches matches eval-* databases across the whole server, not per organization. A dedicated Postgres server for eval branches is the configuration this was designed for.

Building a runner image

Dockerfile.eval-runner

FROM node:22-slim
RUN apt-get update && apt-get install -y --no-install-recommends \
      git ca-certificates curl postgresql-client && \
    rm -rf /var/lib/apt/lists/* && \
    npm install -g @anthropic-ai/claude-code
RUN useradd -m -u 10002 evalrunner
USER evalrunner
WORKDIR /work
CMD ["claude", "--version"]

postgresql-client is not optional if any task ships a setup script: those scripts exist to mutate the branch, and without psql every one of them dies after the branch has already been forked. Use SP_EVAL_SETUP_IMAGE if you would rather keep script tooling in a separate image.

Local (Docker) mode

Eval containers run on the host Docker daemon:

• SP_EVAL_DOCKER_SOCKET: socket mounted into the gateway (default /var/run/docker.sock).
• SP_EVAL_DOCKER_NETWORK: network the containers join. It must be able to reach the gateway at SP_EVAL_MCP_URL.
• SP_EVAL_PROJECTS_DIR: root under which local-path eval sets may live (mounted read-only). Paths outside it are refused.
• SP_EVAL_PROJECTS_HOST_DIR: the same directory as the host sees it. Required for local-path eval sets, because Docker binds resolve on the host, not inside the gateway container.

Local mode also accepts a filesystem path as the eval repo, useful while authoring a set. Cloud mode does not.

Container sizing in local mode is fixed: the agent container gets 2 GiB and 2 CPUs, a setup or teardown container gets 4 GiB and 4 CPUs, and /work, /tmp and /repo are 512 MB tmpfs mounts. The dbt project tarball unpacks into that 512 MB /work. A very large project is the one thing likely to hit it.

Cloud mode

When SP_DEPLOYMENT_MODE=cloud, the Docker path is unreachable by construction: the backend selector raises rather than falling back, so a misconfigured deployment can never hand eval workloads the host daemon. Set SP_EVAL_EXECUTION_BACKEND=vercel to run each task on the hosted sandbox backend. Every task becomes a short-lived, isolated microVM with fixed CPU and memory, restricted egress, and no access to the gateway's database or object storage networks. Two rules apply:

• Digest-pinned images. SP_EVAL_RUNNER_IMAGE and SP_EVAL_SETUP_IMAGE must end in @sha256:<64 hex>. A floating tag would let what runs in the sandbox change without the configuration changing. Look one up with crane digest <image> or docker buildx imagetools inspect <image>.
• GitHub HTTPS repositories only. Cloud mode refuses local paths and SSH URLs for the eval repo and the manifest's project_repo.

Running evals on Cloud

Nothing to deploy. What you control:

1. Enablement. Evals are allow-listed per workspace, and every eval route additionally requires platform-staff access. If the page reports evals are not enabled for your workspace, that is what it means.
2. The eval repo. Cloud mode accepts https://github.com/… only, no other host, no local paths, no SSH URLs. The same rule applies to the manifest's project_repo.
3. Private repos. Connect GitHub from Settings → GitHub and install the app on the repositories you want to use. The gateway then clones them with a short-lived installation token scoped to your organization; a repository that your installation does not cover is refused, not silently attempted. Public repos need no setup. The credentialed URL never leaves the gateway; sandboxes receive a presigned tarball instead of a git credential.
4. The pinned connection. Add the warehouse under Connections, then pin it on the Evals page. Read-only credentials are correct unless you run write tasks; write tasks need a connection whose provider can fork branches.
5. Notification emails, on the same form.

Environment variables

Every variable the eval harness reads. See the configuration reference for the rest of the gateway.

Core

Variable	Default	Description
SP_EVAL_RUNNER_IMAGE	unset	Image with the Claude CLI. Empty disables the feature. Must be digest-pinned in cloud mode.
SP_EVAL_SETUP_IMAGE	runner image	Image for setup/teardown script containers. Digest-pinned in cloud mode.
SP_EVAL_EXECUTION_BACKEND	""	Empty runs tasks on the local Docker Engine. vercel runs them on the hosted sandbox backend. Cloud mode requires vercel.
SP_EVAL_MCP_URL	http://gateway:3300/mcp	MCP endpoint the sandboxes call back on.
SP_EVAL_CLAUDE_TOKEN	unset	CLAUDE_CODE_OAUTH_TOKEN passed to task containers. Falls back to CLAUDE_KEY_1 if unset.
SP_EVAL_ANTHROPIC_KEY	unset	Alternative: ANTHROPIC_API_KEY for task containers.
SP_EVAL_ALLOWED_ORGS	""	Comma-separated org ids allowed to use evals. Empty denies everyone in cloud mode and allows the single local tenant in local mode.
SP_EVAL_TIMEOUT_SECONDS	600	Per-task agent container timeout.
SP_EVAL_PROJECT_ENV	""	Extra KEY=VALUE entries (newline- or comma-separated) injected into agent containers, for example read-only warehouse credentials a profiles.yml reads via env_var. Gateway-controlled variables always win. Setup and teardown containers do not receive these; give those what they need through the manifest's setup.env_file.

Execution: local Docker

Variable	Default	Description
SP_EVAL_DOCKER_SOCKET	/var/run/docker.sock	Docker Engine socket mounted into the gateway.
SP_EVAL_DOCKER_NETWORK	signalpilot_eval_runtime	Network eval containers join; must reach the gateway.
SP_EVAL_PROJECTS_DIR	/eval-projects	Root for local-path eval sets, mounted read-only.
SP_EVAL_PROJECTS_HOST_DIR	unset	Host path of the above; required to bind-mount local eval repos into script containers.
SP_EVAL_SETUP_HOST_ROOT	unset	Host directory that a manifest's setup.mounts paths resolve under. Required when a set mounts files into its setup container.

Evidence store

Variable	Default	Description
SP_EVAL_S3_BUCKET	unset	Bucket for transcripts, setup logs, captures, exports. Empty blocks every run.
SP_EVAL_S3_ENDPOINT	unset	S3 endpoint. Set for MinIO or another S3-compatible service; leave unset for a hosted S3 service.
SP_EVAL_S3_RUNNER_ENDPOINT	unset	Optional GET-only endpoint embedded in URLs handed to sandboxes, while the gateway keeps using the private endpoint.
SP_EVAL_S3_REGION	us-east-1	Region.
SP_EVAL_S3_ACCESS_KEY / SP_EVAL_S3_SECRET_KEY	unset	Dedicated credentials, deliberately separate from BYOK or warehouse credentials.

Branch provider

Variable	Default	Description
SP_EVAL_PG_ADMIN_DSN	unset	Admin DSN used to fork branch databases when the pinned connection is not a Xata connection.
SP_EVAL_PG_PARENT_DB	unset	Parent database each branch is templated from.

Xata-backed sets need no variables: the project, branch, and control-plane credentials come from the pinned connection's stored extras.

Limits and quotas

Variable	Default	Description
SP_EVAL_MAX_PARALLEL_TASKS	4	Tasks in flight within a run. Bounds node capacity and model spend.
SP_EVAL_MAX_BRANCHES	50	Ceiling on live eval branches, enforced before each fork.
SP_EVAL_BRANCH_STORAGE_DELTA_BYTES	5368709120 (5 GiB)	How far a branch may outgrow its parent before the task fails.
SP_EVAL_ARTIFACT_BYTES_PER_RUN	1073741824 (1 GiB)	Artifact budget per run; captures past it are marked truncated.
SP_EVAL_CAPTURE_FULL_MAX_BYTES	268435456 (256 MiB)	Estimated-size ceiling for a full capture; over it the capture is refused up front. Note a separate hard ceiling of 128 MiB applies to any single capture file, so a value above that leaves a band where captures truncate instead of being refused.
SP_EVAL_SETUP_TIMEOUT_SECONDS	1800	Default setup/teardown timeout; a manifest may lower it.

Regression notifications

Variable	Default	Description
SP_EVAL_REGRESSION_DROP_PCT	10.0	Points below the trailing median that count as a regression.
SP_EVAL_SMTP_HOST	""	SMTP host. Empty falls back to the platform's default mail transport.
SP_EVAL_SMTP_PORT	1025	SMTP port.
SP_EVAL_NOTIFY_FROM	evals@your-domain.example	From address on regression email.

Eval-set uploads

Optional intake path for customers mailing in an eval set as a zip (/evals/upload).

Variable	Default	Description
SP_EVAL_UPLOADS_BUCKET	unset	Bucket for uploaded archives. Empty disables the page's backend.
SP_EVAL_UPLOADS_MAX_MB	8192	Upload ceiling.
SP_EVAL_UPLOADS_S3_ENDPOINT	unset	S3 endpoint for uploads.
SP_EVAL_UPLOADS_S3_PUBLIC_ENDPOINT	unset	Endpoint the browser uses for multipart PUTs.
SP_EVAL_UPLOADS_S3_REGION	unset	Region.
SP_EVAL_UPLOADS_S3_ACCESS_KEY / …_SECRET_KEY	unset	Upload credentials.
SP_EVAL_UPLOADS_NOTIFY_EMAIL	unset	Where upload notifications go.
SP_EVAL_UPLOADS_NOTIFY_FROM	eval-uploads@your-domain.example	From address.
SP_EVAL_UPLOADS_SMTP_HOST / …_SMTP_PORT	"" / 1025	SMTP for upload notifications.

Related gateway variables

Variable	Relevance
SP_DEPLOYMENT_MODE	cloud requires the hosted sandbox backend, digest-pinned images, and GitHub HTTPS repos.
SP_ADMIN_USER_IDS	Only these users pass the platform-staff check on eval routes.
SP_GITHUB_APP_*	Needed to clone private eval or project repositories.

Isolation and data handling

Worth knowing before you point this at production data:

• Per-task credentials. Each task container gets its own API key, bound server-side to that run, task, connection, and knowledge overlay, and expiring after 3 hours. The agent cannot widen its own scope with headers; the binding is read from the stored key, never from the request. Keys are revoked when the task ends and when the run ends.
• Read-only by default. Governed MCP blocks DDL/DML at parse time. Write access exists only as a branch DSN, only for write tasks, only on a disposable branch.
• Redaction. Transcripts and script logs are scrubbed before storage: the exact secrets the harness injected, DSN user-info, auth headers, and long base64-looking blobs. It is a heuristic, not a guarantee. A secret the harness never knew about (one your setup script prints, say) can survive it.
• The project tarball is deleted once every task has fetched it, and is excluded from run exports.
• Retention prunes artifacts to the last 10 runs and transcripts to the last 100 runs per workspace; accuracy history is kept.

Troubleshooting

Symptom	Cause
runner disabled — SP_EVAL_RUNNER_IMAGE unset on gateway	The feature is off.
Eval evidence bucket is not configured (SP_EVAL_S3_BUCKET)	No evidence store.
An eval connection pin is required / does not exist in this workspace	Set Connection on the Evals page to a real connection name.
Evals are not enabled for this workspace	The org is not in SP_EVAL_ALLOWED_ORGS.
Platform staff access required	The caller is not in SP_ADMIN_USER_IDS.
only https://github.com/ repositories are allowed	Cloud mode; use a GitHub HTTPS URL.
local paths must live under /eval-projects	Move the set under SP_EVAL_PROJECTS_DIR.
no branch provider: pin a Xata connection …	The set has write tasks but no provider is configured.
warehouse does not match the eval set's build fingerprint	Rebuild the warehouse, or clear build_fingerprint in the manifest.
psql: not found in a setup log	The setup image lacks postgresql-client.
Every write task is SETUP_FAILED	Read the setup log in the evidence store; it is stored even when the agent never ran.
Too many eval runs in flight	Two runs already active on the gateway.