harbor run, and your tasks,
agents, and datasets stay as they are.
Once you’re finished, you’ll have:
- A passing smoke run of Harbor’s
hello-worldtask in a Sailbox, no model key needed. - Your own eval, such as
terminal-bench@2.0, running with the agent, model, and concurrency you pick, with every trial’s results under./jobs.
- Every trial gets its own Sailbox, so the machine running Harbor isn’t the limit on how many trials you run at once.
- You don’t have to size your tasks. A task’s CPU, memory, and storage requests become limits on its Sailbox, and you pay for the CPU, memory, and disk each trial actually uses. A task that asks for 8 CPUs and uses one is charged for one.
- A task’s Dockerfile or Docker Compose project is built once for your organization. Every later trial starts from the cached image.
Copy agent prompt (recommended):
Share this page with your coding agent or paste the prompt below.Manual steps
1. Install Harbor and the Sail SDK
You’ll need Python 3.12 or newer and a Sail API key. Install both packages in the same environment.sail auth login on this machine, you can skip the
export.2. Run the smoke test
Run Harbor’shello-world task with the oracle agent. It runs the task’s
reference solution, so you don’t need a model key.--env sail.harbor:SailboxEnvironment is the only Sail-specific part. Harbor
creates a Sailbox, runs the solution and the verifier in it, writes the
results under ./jobs, and terminates the Sailbox. Look for one trial, no
exceptions, and a mean reward of 1.3. Run your eval
Swap in your dataset, agent, and model, and set the model’s API key in the same shell (ANTHROPIC_API_KEY for claude-code, for example).
harbor run --help lists the agents.-n is how many trials run at once. Without -l, Harbor runs every task in
the dataset, so add -l 5 to keep a first run small.If you use a job config (harbor run -c job.yaml), select Sail there:Settings
Pass any of these with--ek on the command line or under kwargs in a job
config.Task images
Harbor decides where a task’s image comes from. Sail follows the same rules as Harbor’s other cloud providers.A task with both a Dockerfile and a
docker_image runs on the
docker_image. --force-build builds the Dockerfile instead, and rebuilds a
cached image.Docker Compose projects
The whole project runs inside the Sailbox. Commands run in the task’smain
service, and Harbor still reaches the other services. Sail builds the service
images into the Sailbox image, cached the same way as a Dockerfile build. Your
compose files need to follow the rules in
Build an image from a Compose project.Compose variables and shells
Compose variables and shells
${VAR}or$VARin the task’s compose files takes its value from the variables the trial adds, then the task’senventries, then the environment of the machine running Harbor. Compose fills anything still unset from the project’s.env, then the default in the expression.- The task’s
enventries and the trial’s variables are passed to the project whether or not a compose file references them. DOCKER_HOSTand the other variables that select a Docker daemon (DOCKER_CONTEXT,DOCKER_CONFIG,DOCKER_CERT_PATH,DOCKER_TLS, andDOCKER_TLS_VERIFY) are not taken from the machine, even when a compose file references them. A taskenventry or trial variable with one of these names is still passed as written.- The build sees the same variables as the start.
- Commands in
mainuse non-login Bash and keep the image’sPATH. Commands in the other services usesh. - If the task also has a
docker_imageor a Dockerfile,mainruns on that image, and the compose file’smainentry adds settings such asenvironmentorports.
Open a shell in a task’s environment
main, for a Compose project). The
Sailbox is terminated when you exit.Gotchas
A trial fails before any Sailbox is created
A trial fails before any Sailbox is created
The task asked for something a Sailbox can’t give it, and the error names
what. Sailboxes don’t offer GPUs, TPUs, Windows, or IPv6
allowlist entries. A task can ask for
up to 64 CPUs, 256 GiB of memory, and 1024 GiB of storage. Harbor’s
limit and guarantee enforcement policies fail too, so leave them at
their default or set request. A Compose project that uses a feature Sail
doesn’t support also fails here.A healthcheck that passes with Docker never passes here
A healthcheck that passes with Docker never passes here
Outside a Compose project, a Sailbox doesn’t run the image’s
ENTRYPOINT
or CMD, so a service they’d start never comes up and the check runs out
of retries. Move the service into a Compose project. A sidecar runs its
image’s ENTRYPOINT and CMD. In main, the ENTRYPOINT runs too, but
Harbor replaces the CMD with sleep infinity, so a service the CMD
would start needs its own command under main in the task’s compose
file.How do I know a run didn't leave Sailboxes behind?
How do I know a run didn't leave Sailboxes behind?
Run
sail box list --app harbor --exclude-status terminated. Harbor
terminates each Sailbox when its trial ends. If that fails, the trial’s
result stands and Harbor logs a warning that names the Sailbox. Terminate
it with sail box terminate <id> or from the dashboard. If a run might
get killed, set max_lifetime_seconds and Sail cleans up on its own.How do I keep a Sailbox to debug a trial?
How do I keep a Sailbox to debug a trial?
Pass
--no-delete. The Sailbox goes to sleep with its filesystem and
processes intact, and you aren’t billed while it sleeps. Harbor’s debug
log names the Sailbox. Open it with sail box shell <id> and terminate it
when you’re done. A rerun starts a new Sailbox, and a trial that fails
during setup terminates its Sailbox even with --no-delete.A task mounts a directory from my machine
A task mounts a directory from my machine
A Sailbox is a separate machine, so the mount can’t attach your directory.
Outside a Compose project, a writable mount’s target is created on the
Sailbox, if it isn’t there already, for the task to fill. A read-only
mount is skipped. In a Compose project, every bind mount’s source becomes
a directory on the Sailbox at its target path, and
main sees it there.