Skip to main content
Give your coding agent one prompt to run your Harbor evals on Sail, with every trial in its own Sailbox. It’s one flag on harbor run, and your tasks, agents, and datasets stay as they are. Once you’re finished, you’ll have:
  • A passing smoke run of Harbor’s hello-world task in a Sailbox, no model key needed.
  • Your own eval, such as terminal-bench@2.0, running with the agent, model, and concurrency you pick, with every trial’s results under ./jobs.
Why Harbor on Sail?
  • Every trial gets its own Sailbox, so the machine running Harbor isn’t the limit on how many trials you run at once.
  • You don’t have to size your tasks. A task’s CPU, memory, and storage requests become limits on its Sailbox, and you pay for the CPU, memory, and disk each trial actually uses. A task that asks for 8 CPUs and uses one is charged for one.
  • A task’s Dockerfile or Docker Compose project is built once for your organization. Every later trial starts from the cached image.
Share this page with your coding agent or paste the prompt below.

Manual steps

1. Install Harbor and the Sail SDK

You’ll need Python 3.12 or newer and a Sail API key. Install both packages in the same environment.
If you’ve already run sail auth login on this machine, you can skip the export.

2. Run the smoke test

Run Harbor’s hello-world task with the oracle agent. It runs the task’s reference solution, so you don’t need a model key.
--env sail.harbor:SailboxEnvironment is the only Sail-specific part. Harbor creates a Sailbox, runs the solution and the verifier in it, writes the results under ./jobs, and terminates the Sailbox. Look for one trial, no exceptions, and a mean reward of 1.

3. Run your eval

Swap in your dataset, agent, and model, and set the model’s API key in the same shell (ANTHROPIC_API_KEY for claude-code, for example). harbor run --help lists the agents.
-n is how many trials run at once. Without -l, Harbor runs every task in the dataset, so add -l 5 to keep a first run small.If you use a job config (harbor run -c job.yaml), select Sail there:

Settings

Pass any of these with --ek on the command line or under kwargs in a job config.

Task images

Harbor decides where a task’s image comes from. Sail follows the same rules as Harbor’s other cloud providers.A task with both a Dockerfile and a docker_image runs on the docker_image. --force-build builds the Dockerfile instead, and rebuilds a cached image.

Docker Compose projects

The whole project runs inside the Sailbox. Commands run in the task’s main service, and Harbor still reaches the other services. Sail builds the service images into the Sailbox image, cached the same way as a Dockerfile build. Your compose files need to follow the rules in Build an image from a Compose project.
  • ${VAR} or $VAR in the task’s compose files takes its value from the variables the trial adds, then the task’s env entries, then the environment of the machine running Harbor. Compose fills anything still unset from the project’s .env, then the default in the expression.
  • The task’s env entries and the trial’s variables are passed to the project whether or not a compose file references them.
  • DOCKER_HOST and the other variables that select a Docker daemon (DOCKER_CONTEXT, DOCKER_CONFIG, DOCKER_CERT_PATH, DOCKER_TLS, and DOCKER_TLS_VERIFY) are not taken from the machine, even when a compose file references them. A task env entry or trial variable with one of these names is still passed as written.
  • The build sees the same variables as the start.
  • Commands in main use non-login Bash and keep the image’s PATH. Commands in the other services use sh.
  • If the task also has a docker_image or a Dockerfile, main runs on that image, and the compose file’s main entry adds settings such as environment or ports.

Open a shell in a task’s environment

You get a shell in the task’s working directory with the task’s variables, the same as the agent’s commands (in main, for a Compose project). The Sailbox is terminated when you exit.

Gotchas

The task asked for something a Sailbox can’t give it, and the error names what. Sailboxes don’t offer GPUs, TPUs, Windows, or IPv6 allowlist entries. A task can ask for up to 64 CPUs, 256 GiB of memory, and 1024 GiB of storage. Harbor’s limit and guarantee enforcement policies fail too, so leave them at their default or set request. A Compose project that uses a feature Sail doesn’t support also fails here.
Outside a Compose project, a Sailbox doesn’t run the image’s ENTRYPOINT or CMD, so a service they’d start never comes up and the check runs out of retries. Move the service into a Compose project. A sidecar runs its image’s ENTRYPOINT and CMD. In main, the ENTRYPOINT runs too, but Harbor replaces the CMD with sleep infinity, so a service the CMD would start needs its own command under main in the task’s compose file.
Run sail box list --app harbor --exclude-status terminated. Harbor terminates each Sailbox when its trial ends. If that fails, the trial’s result stands and Harbor logs a warning that names the Sailbox. Terminate it with sail box terminate <id> or from the dashboard. If a run might get killed, set max_lifetime_seconds and Sail cleans up on its own.
Pass --no-delete. The Sailbox goes to sleep with its filesystem and processes intact, and you aren’t billed while it sleeps. Harbor’s debug log names the Sailbox. Open it with sail box shell <id> and terminate it when you’re done. A rerun starts a new Sailbox, and a trial that fails during setup terminates its Sailbox even with --no-delete.
A Sailbox is a separate machine, so the mount can’t attach your directory. Outside a Compose project, a writable mount’s target is created on the Sailbox, if it isn’t there already, for the task to fill. A read-only mount is skipped. In a Compose project, every bind mount’s source becomes a directory on the Sailbox at its target path, and main sees it there.