Use Docker Model Runner with Compose Bridge

Table of contents

Compose Bridge supports model-aware deployments. It can deploy and configure Docker Model Runner, a lightweight service that hosts and serves machine LLMs.

This reduces manual setup for LLM-enabled services and keeps deployments consistent across Docker Desktop and Kubernetes environments.

If you have a models top-level element in your compose.yaml file, Compose Bridge:

  • Automatically injects environment variables for each model’s endpoint and name.
  • Configures model endpoints differently for Docker Desktop vs Kubernetes.
  • Optionally deploys Docker Model Runner in Kubernetes when enabled in Helm values

Configure model runner settings

When deploying using generated Helm Charts, you can control the model runner configuration through Helm values.

# Model Runner settings modelRunner:  # Set to false for Docker Desktop (uses host instance)  # Set to true for standalone Kubernetes clusters  enabled: false  # Endpoint used when enabled=false (Docker Desktop)  hostEndpoint: "http://host.docker.internal:12434/engines/v1/"  # Deployment settings when enabled=true  image: "docker/model-runner:latest"  imagePullPolicy: "IfNotPresent"  # GPU support  gpu:  enabled: false  vendor: "nvidia" # nvidia or amd  count: 1  # Node scheduling (uncomment and customize as needed)  # nodeSelector:  # accelerator: nvidia-tesla-t4  # tolerations: []  # affinity: {}   # Security context  securityContext:  allowPrivilegeEscalation: false  # Environment variables (uncomment and add as needed)  # env:  # DMR_ORIGINS: "http://localhost:31246"  resources:  limits:  cpu: "1000m"  memory: "2Gi"  requests:  cpu: "100m"  memory: "256Mi"  # Storage for models  storage:  size: "100Gi"  storageClass: "" # Empty uses default storage class  # Models to pre-pull  models:  - ai/qwen2.5:latest  - ai/mxbai-embed-large

Deploying a model runner

Docker Desktop

When modelRunner.enabled is false, Compose Bridge configures your workloads to connect to Docker Model Runner running on the host:

http://host.docker.internal:12434/engines/v1/

The endpoint is automatically injected into your service containers.

Kubernetes

When modelRunner.enabled is true, Compose Bridge uses the generated manifests to deploy Docker Model Runner in your cluster, including:

  • Deployment: Runs the docker-model-runner container
  • Service: Exposes port 80 (maps to container port 12434)
  • PersistentVolumeClaim: Stores model files

The modelRunner.enabled setting also determines the number of replicas for the model-runner-deployment:

  • When true, the deployment replica count is set to 1, and Docker Model Runner is deployed in the Kubernetes cluster.
  • When false, the replica count is 0, and no Docker Model Runner resources are deployed.