Run Harbor tasks on Sailboxes

Run every Harbor trial in its own Sailbox with one flag

Harbor is a framework for evaluating and training agents on sandboxed tasks. Each trial runs an agent against one task inside an isolated environment, then a verifier scores the result. Harbor includes environments for Docker on your own machine and for several cloud sandboxes, and loads any other environment by import path. Sail provides one:

sail.harbor:SailboxEnvironment

Pass that import path to Harbor and every trial’s environment becomes a Sailbox. Tasks, agents, verifiers, and datasets are unchanged.

Quickstart

Install sail and harbor in the same Python environment (Harbor needs Python 3.12 or newer) and set your Sail API key. Then run Harbor’s hello-world dataset with the oracle agent, which runs each task’s reference solution and doesn’t need a model key:

pip install sail harbor
export SAIL_API_KEY=sk_your_key_here

harbor run -d hello-world@1.0 --agent oracle \
  --env sail.harbor:SailboxEnvironment -o ./jobs

Harbor creates a Sailbox, runs the task’s solution in it, runs the verifier, writes the trial’s results under ./jobs, and terminates the Sailbox when the trial ends.

A real evaluation is the same command with a real agent, a model, and more trials in flight. Each trial gets its own Sailbox, so the machine running Harbor is no longer the limit on concurrency:

harbor run -d terminal-bench@2.0 --agent <agent> --model <model> \
  --env sail.harbor:SailboxEnvironment -n 32

A job config file (harbor run -c job.yaml) selects the environment the same way:

environment:
  import_path: sail.harbor:SailboxEnvironment

Credentials and app

The process that runs Harbor needs a Sail credential: set SAIL_API_KEY, or run sail auth login once on that machine. Sailboxes are created in the harbor app in your organization, which is created on first use. Set SAIL_APP to use a different app. Each Sailbox is named harbor- followed by the trial’s session id, and sail box list --app harbor (or the app you chose) lists the ones a run created.

Task images

Harbor decides where an environment’s image comes from, and Sail follows the same rules as Harbor’s other cloud providers.

• A task that includes an environment/Dockerfile has it built into a Sailbox image. The build happens once per organization: the first trial of that task waits for it, and later trials start from the cached image. --force-build rebuilds it. See Build an image from a Dockerfile for what a Dockerfile can contain.
• A task that declares a docker_image in its task.toml runs on that image, pulled from its registry. Docker-style short references work: python:3.11 means docker.io/library/python:3.11. The registry must be one Sail supports (docker.io, ghcr.io, public.ecr.aws, or quay.io), and the image must be Debian or Ubuntu based. See Bring your own base image for the full requirements.
• A task can have both. Harbor treats docker_image as the published build of the Dockerfile and runs on that image unless you pass --force-build.
• A task that includes an environment/docker-compose.yaml runs as a Docker Compose project.

Docker Compose projects

The project runs inside the Sailbox. Commands run in the task’s main service, and Harbor’s per-service operations (exec, download, stop) reach the other services. If the task also has a docker_image or a Dockerfile, main runs on that image, and the compose file’s main entry adds settings such as environment or ports.

Sail builds the service images into the Sailbox image, cached and rebuilt the same way as a Dockerfile build. The compose files follow the rules in Build an image from a Compose project. A project that uses an unsupported feature fails before a Sailbox is created, and the error identifies the feature.

Compose variables

• ${VAR} takes its value from the environment of the machine running Harbor, then from the project’s .env, then from the default in the expression.
• DOCKER_* and COMPOSE_* names are never read from the machine, and an env_file path written with ${VAR} is rejected.
• The build sees .env and the machine variables that pick a service’s image: the ones under image, build, platform, and extends, and any .env itself uses. Other machine variables are passed to the project only when it starts in the Sailbox, so a secret a service’s environment reads from the machine never enters the build.

Sizing

A task’s CPU, memory, and storage requests pick the smallest Sailbox size that covers them: s (1 vCPU), m (4 vCPU), or l (8 vCPU). A request above the size’s default memory or disk ceiling raises that ceiling, up to the size’s maximum. A task that doesn’t declare resources gets an s Sailbox. A task that requests more than the largest size provides fails up front, before a Sailbox is created, and the error names the request and that size’s limits.

The size is a ceiling, not a reservation: a Sailbox is billed for the CPU, memory, and disk it actually uses, so a task that requests 8 CPUs and uses one pays for one.

Cleanup

By default, harbor run deletes each environment when its trial ends, which terminates the Sailbox. A termination that fails is reported in that trial’s result, like any other trial error.

With --no-delete, Harbor stops each environment without deleting it. The Sailbox goes to sleep with its filesystem and processes intact, and billing stops until it is resumed. Terminate those Sailboxes yourself when you are done with them, with sail box terminate <id> or from the dashboard.

A trial whose environment fails to start terminates its Sailbox before the error is reported, even with --no-delete. If that termination fails, the trial’s normal cleanup stops the Sailbox the same way it would after a completed trial.

Unsupported task features

Harbor checks a task’s needs against the environment before starting it, so a task that needs something unsupported fails up front with a clear error instead of running without the feature. Sailboxes do not offer:

• GPUs, TPUs, or Windows environments.
• IPv6 allowlist entries. A Sailbox does not reach the internet over IPv6.

The egress policy reference has the allowlist entry rules.

Mounts declared for a task are ignored, except in a Compose task, where they are bound into its main service from the Sailbox’s filesystem.