Docker quickstart¶

Run the SDK without installing Python or PyTorch on your host. For the CPU quickstart, allow 8 GB of RAM and 10 GB of free disk, with Docker Engine or Docker Desktop and Compose v2 or newer.

From the repository root:

docker compose run --build --rm laya

This builds the checkout, runs the sample request on CPU and prints JSON covering choice, score and noul. The first request downloads the selected public Hugging Face checkpoint; no account is needed. Allow several minutes for its first download. Weights stay in a named volume. Subsequent runs use docker compose run --rm laya.

Predictions and confidence still need evaluation on your workload. See the benchmark limits.

For ARM64 hosts, DGX Spark and Apple Silicon, see ARM64 and DGX Spark containers.

NVIDIA GPU / CUDA¶

Install a compatible NVIDIA driver and configure Docker with the NVIDIA Container Toolkit. The GPU image uses PyTorch CUDA 13.0 wheels. Check your GPU's compute capability and driver against PyTorch's supported builds; older cards may require a different build. Allow additional disk space for CUDA layers. VRAM needs depend on the checkpoint, batch size and input length.

docker compose -f compose.yaml -f compose.cuda.yaml run --build --rm laya

The override selects GPU 0 and defaults to LAYA_DEVICE=cuda. Set LAYA_GPU_ID to another host index or UUID. That GPU appears as device 0 inside the container. Check access without downloading weights:

docker compose -f compose.yaml -f compose.cuda.yaml run --rm laya python -c \
  'import torch; assert torch.cuda.is_available(); print(torch.cuda.get_device_name(0)); print(torch.ones(1, device="cuda").cpu())'

The sample rejects unavailable CUDA before loading a checkpoint. Laya can still fall back to CPU after a memory or inference error, so inspect its warnings. Rebuild when switching between CPU and CUDA configurations.

The image sets TORCH_DISABLE_NATIVE_JIT=1. PyTorch 2.14 otherwise replaces some eager CUDA ops with Triton kernels that it compiles on the first inference, which needs a C compiler the slim image does not carry: the container reports healthy and then fails every request (#365). The stock kernels give the same answers at the same latency. Set the same variable on a bare-metal install if predict fails with Failed to find C compiler.

This uses Compose GPU reservations. Windows requires Docker Desktop's supported WSL2 GPU setup. Apple MPS, AMD/ROCm and Intel GPU containers are outside this quickstart; use CPU unless you configure and validate another backend.

Configuration¶

Set Compose variables in your shell, a local .env file, or the service's environment block. Don't commit secrets in .env. Runtime variables also work with docker run -e; Compose-only settings are identified below.

Compose forwards the runtime variables except HF_HOME, which stays aligned with its fixed cache mount, and except LAYA_MPS_AMP_MIN_ROWS, the MPS row gate, which no image here can reach because no container here can select MPS. If overriding HF_HOME in docker run or your own Compose file, provide a matching mount writable by UID 10001. Direct Docker builds select PyTorch with --build-arg TORCH_INDEX=cu130; runtime -e cannot change the installed wheel.

LAYA_MODEL=english OMP_NUM_THREADS=2 docker compose run --build --rm laya

docker build -t laya:local .
docker run --rm -e LAYA_MODEL=english -e OMP_NUM_THREADS=2 \
  -v laya-model-cache:/home/laya/.cache/huggingface laya:local

For your own request:

docker compose run --rm --volume "$PWD/request.json:/inputs/request.json:ro" \
  --env LAYA_REQUEST_FILE=/inputs/request.json laya

For a commented configuration with request, checkpoint and secret-file mounts, see compose.example.yml:

docker compose -f compose.yaml -f compose.example.yml run --build --rm laya

Add -f compose.cuda.yaml before run for NVIDIA GPUs. The example is an override of compose.yaml, so cache and image settings stay in one place.

Secret files¶

HF_TOKEN_FILE reads a mounted UTF-8 file at startup, trims surrounding whitespace and takes precedence over HF_TOKEN. Unreadable, empty or invalid files stop startup without printing their contents. The file must be readable by UID 10001. _FILE applies only to supported secrets, not every setting.

With HF_TOKEN_PATH pointing to an existing host file outside the checkout:

docker compose run --rm --volume "$HF_TOKEN_PATH:/run/secrets/hf_token:ro" \
  --env HF_TOKEN_FILE=/run/secrets/hf_token laya

Docker secrets or Kubernetes Secret volumes can supply the same file. Values are loaded into the process environment at startup; restart after changing a file. Never use tokens as build arguments or bake them into images. Public checkpoints need no token.

Fine-tuned checkpoints¶

This image runs inference. Fine-tuning happens outside it — the fine-tuning notebook runs the whole loop on Kaggle's free 2xT4 GPUs and exports a checkpoint this image can serve. Background and open questions about the training interface stay in #4 and #26.

Point LAYA_CHECKPOINT_PATH to an absolute host directory containing rl_agent_config.json, model.safetensors and matching tokenizer files:

docker compose run --rm --volume "$LAYA_CHECKPOINT_PATH:/models/custom" \
  --env LAYA_MODEL_PATH=/models/custom laya

Use a working copy writable by UID 10001 because the loader may update tokenizer configuration. A LoRA adapter alone is not a complete checkpoint. Leave LAYA_MODEL=auto when setting LAYA_MODEL_PATH; an explicit alias and local path are mutually exclusive. The local-path response comes from the Agent and has no Router routing metadata. These settings also work with the CUDA override. Evaluate fine-tuned checkpoints on held-out examples before relying on them.

Models from ModelScope¶

When a host cannot reach huggingface.co, the checkpoint can come from ModelScope and be baked into the image at build time. One argument selects which checkpoint, and it defaults to the multilingual one. The Compose override adds the prefetch arguments to both services and keeps the container off the Hub:

docker compose -f compose.yaml -f compose.http.yaml -f compose.modelscope.yaml up --build laya-serve

For NVIDIA, add -f compose.cuda.yaml before up; it repeats its own arguments for both services, so the ordering between the two overrides does not matter. Plain Docker takes the arguments directly:

docker build --build-arg MODELSCOPE_MODEL=multilingual \
  -t laya:local .
docker run --rm -e HF_HUB_OFFLINE=1 -p 127.0.0.1:8000:8000 laya:local laya-serve

One prerequisite on a host that has run this before. The baked weights land in $HF_HOME/hub inside the image, under the cache directory compose.yaml mounts the model-cache volume at (/home/laya/.cache/huggingface), and Docker seeds a named volume from the image only while that volume is empty. A volume left over from the Hub-based quickstart holds the older Hub snapshot, it is never re-seeded, and the baked weights stay invisible behind it: the loader resolves refs/main to the old Hub commit and the container answers with the weights that were already downloaded, as if the rebuild had changed nothing. Point the deployment at an empty cache volume -- docker compose down --volumes with the same Compose files and the same LAYA_CACHE_VOLUME, or LAYA_CACHE_VOLUME=<name> for a fresh one. With HF_HUB_OFFLINE=1 a missing or divergent ref is a load failure with no network to fall back to, but the prerequisite is the same.

docker/prefetch_modelscope.py lists the repository on modelscope.cn, downloads the checkpoint's own files -- the same set laya/agent.py asks the Hub for, so no sibling checkpoint is pulled -- and writes them into the image's hub cache the way snapshot_download lays out a snapshot. Nothing else changes: Agent, the Router that laya-serve builds, laya.cli and the integrations keep their repo ids and resolve them to the baked snapshot, so a container built this way needs no network at all. Each file's size is checked against what the repository reports before the snapshot is published, and its SHA-256 as well when the repository publishes one. A size or digest mismatch fails the build. A repository that publishes no digest leaves the check to size alone, which cannot detect a same-size substitution.

A type expands to that checkpoint's path inside the bundled repository, which is what the Router and the one-shot quickstart load by default, so a build that names one type serves it with no further change:

MODELSCOPE_MODEL=english docker compose -f compose.yaml -f compose.http.yaml -f compose.modelscope.yaml up --build laya-serve
MODELSCOPE_MODEL=all docker compose -f compose.yaml -f compose.http.yaml -f compose.modelscope.yaml up --build laya-serve

all is the whole family, about 2.4 GB of weights. The override also sets LAYA_MODELS=multilingual, because LAYA_PRELOAD=1 with the default list would try to build every checkpoint and fail on the first one that was not baked; set LAYA_MODELS to the list you baked when you bake more, and MODELSCOPE_MODEL=all when a deployment really does serve the family.

Several types can be named at once -- MODELSCOPE_MODEL="english multilingual" bakes both, about 1.5 GB -- which is what a serving image usually wants: the Router chooses between the English and the multilingual checkpoint on its own, and one it was not given answers 500 inference failed with does not contain 'rl_agent_config.json' in the log. Checkpoints that share a repository always bake into one snapshot, because a cached revision resolves to a single directory; the root checkpoint and each subfolder are both in it.

Beyond the types the argument also takes repo[:subfolder] specs, comma- or space-separated, which is how the mirror's standalone repositories (laya, laya-multilingual, laya-typed-decisions) or a fine-tuned checkpoint is baked. A standalone repository is what Agent("convaiinnovations/laya-multilingual") loads directly; the Router's default is the bundled path, so a type is normally what a serving image wants.

Two details about pins and provenance. The build prints the commit the snapshot is keyed by, which is the tip of the revision that was baked -- pass that SHA as revision= or LAYA_REVISION to pin a load to exactly what was baked. A mirror repository can hold files from several uploads, so that tip is the one repository-wide key there is. Hub-side pins do not describe a mirror snapshot: reviewed names Hugging Face commits, and the SHA-256 digest map is keyed to Hub artifact hashes, so neither applies here, and no digest pin exists for a mirror snapshot either. The build already refuses a download that does not match the mirror's own reported size -- and its digest, when the mirror publishes one -- but this checks consistency with mirror metadata, not an independently pinned digest. The weights come from whichever mirror account the argument names, which is its own supply-chain decision for the deployment to make.

Development and cleanup¶

Open a Python prompt with docker compose run --rm laya python. To run the existing routing/criteria checks and secret-file tests against your checkout without downloading weights:

docker compose run --rm --volume "$PWD:/workspace:ro" --workdir /workspace laya \
  sh -ec 'python tests/test_router.py; python tests/test_criteria.py; python tests/test_docker_entrypoint.py'

Rebuild with --build after changing source or the bundled example. The image runs as UID/GID 10001. New named volumes inherit the image cache directory's ownership; host directories must be writable by that UID. Keep model caches writable for tokenizer compatibility updates.

--rm removes completed containers. docker compose down retains the cache. To delete downloaded weights, run docker compose down --volumes using the same Compose files and LAYA_CACHE_VOLUME setting. The next request downloads them again; don't remove a cache shared with another project.

HTTP serving¶

The image ships laya-serve, so the same build that runs the one-shot quickstart can serve the Jev-compatible API. compose.http.yaml adds it as a second service and leaves laya alone:

docker compose -f compose.yaml -f compose.http.yaml up --build laya-serve
curl -s localhost:8000/health
curl -s localhost:8000/v1/systemone -H 'content-type: application/json' \
  --data @examples/docker/request.json

For NVIDIA, add the CUDA override. It repeats the build args and the device reservation for laya-serve, because laya-serve is a separate service and overrides for laya never reach it:

docker compose -f compose.yaml -f compose.http.yaml -f compose.cuda.yaml up --build laya-serve

up keeps the service running in the foreground; -d detaches. Weights go to the same named model-cache volume as the quickstart, so serving after a quickstart run starts with the checkpoints already on disk. Stop with docker compose ... down, using the same Compose files.

The port is published on 127.0.0.1 only. The API has no authentication until LAYA_API_KEY is set, so set a key before exposing it with LAYA_BIND_ADDRESS=0.0.0.0, and put a TLS reverse proxy in front for remote clients. /health does not require authentication in either case, so the healthcheck below keeps working; with a key set it answers an unauthenticated caller {"status": "ok"} and withholds the checkpoint, revision and device fields, which need the bearer.

The service has a healthcheck on /health. docker compose ... up -d --wait laya-serve waits for HTTP liveness. With the Compose default LAYA_PRELOAD=0, it can return with no checkpoint loaded; the first valid inference request loads the checkpoint it needs. With LAYA_PRELOAD=1, the server builds the selected checkpoints before listening, but preloading does not run a model forward.

These are three separate checks: /health confirms the HTTP process responds, an authenticated /health response lists resident checkpoints, and a successful authenticated POST /v1/systemone confirms that the requested inference ran. A healthy container or a nonempty loaded list alone does not establish successful inference. The Docker workflow's HTTP job checks startup, loopback publication and authentication with preload disabled and invalid request bodies; it does not run a model forward.

/health reports device as the device a resident checkpoint actually computes on, which is not always what LAYA_DEVICE asked for: a checkpoint that wants a GPU it cannot get falls back to CPU silently and still answers correctly. checkpoint_devices names each loaded checkpoint, and device_is_preference is true only while nothing is resident, so a deployment that quietly lost its GPU says so instead of echoing its own configuration back.

Server configuration¶

These apply to the laya-serve service only.

For example, set LAYA_ROOT_PATH=/laya when publishing the API under /laya. The proxy must strip that prefix before forwarding to the container; this setting updates FastAPI's generated URLs and does not change the internal /health or /v1/systemone routes.

LAYA_PRELOAD defaults to 0 here rather than the package default of 1, because preloading makes the first boot download all three checkpoints. Set it to 1 for a long-running deployment so the first request does not pay for the build.

LAYA_PORT sets both the published host port and the port the server binds, so the two cannot drift. Change one place to move the service:

LAYA_PORT=9000 docker compose -f compose.yaml -f compose.http.yaml up --build laya-serve

Bearer token from a file¶

LAYA_API_KEY_FILE is read once at startup, moved into LAYA_API_KEY, and the _FILE variable is removed before the server execs. Prefer this to putting the key in the environment:

docker compose -f compose.yaml -f compose.http.yaml run --rm \
  --volume "$PWD/laya_api_key:/run/secrets/laya_api_key:ro" \
  -e LAYA_API_KEY_FILE=/run/secrets/laya_api_key \
  --service-ports laya-serve