Running Docker Containers

Run GPU-accelerated Docker containers on a virtual machine with the NVIDIA Container Toolkit.

Run GPU-accelerated Docker containers on a Hyperstack virtual machine within minutes. Docker packages an application and its dependencies into a container that runs the same way on any machine, and the NVIDIA Container Toolkit gives those containers direct access to the virtual machine's GPU. Hyperstack offers OS images with Docker and the NVIDIA Container Toolkit already installed, so you can deploy a GPU container host with no manual setup.

This tutorial walks through deploying a virtual machine with Docker preinstalled, connecting over SSH, and running a GPU-accelerated container. It also covers running multi-container workloads with Docker Compose and installing Docker yourself with a cloud-init script when you need a custom base image.

Step 1: Deploy a Virtual Machine with Docker

The fastest way to get a GPU container host is to deploy from an OS image that ships with Docker and the NVIDIA Container Toolkit preinstalled. These images carry a with Docker suffix.

Open the deployment page

In the Hyperstack console, navigate to Virtual Machines in the sidebar under Cloud, then click Deploy New Virtual Machine.
Select a GPU flavor

Choose the flavor that matches your workload. A single-GPU flavor is a cost-effective starting point for learning the workflow.

Choose a GPU flavor for GPU containers
Containers can only access a GPU if the virtual machine has one. Select a GPU flavor to run the GPU-accelerated container later in this tutorial.
Select an environment

Choose the environment in which to deploy the virtual machine. Each environment belongs to a region, so the list only offers environments compatible with your selected flavor.
Choose an OS image with Docker

In the Select OS Image section, on the OS Images tab, select the Ubuntu tile, then choose a version ending in with Docker from its dropdown, such as Server 24.04 LTS R570 CUDA 12.8 with Docker.

What the "with Docker" images include
These images come with Docker Engine, the Docker Compose plugin, and the NVIDIA Container Toolkit already installed and configured, and the default ubuntu user is already a member of the docker group. The included NVIDIA driver and CUDA version are listed in the image name, such as R570 CUDA 12.8.
Select an SSH key

Choose an SSH key for the virtual machine, then enable the SSH Access toggle to allow incoming SSH traffic on port 22. You connect to the container host over SSH in the next section.
Assign a public IP address

Enable the Assign Public IP toggle. A public IP address is required to reach the virtual machine over SSH.
Deploy the virtual machine

Review the configuration and price breakdown, then click Deploy.

Your virtual machine is now deploying. Once it reaches the ACTIVE state, connect to it over SSH to start working with Docker.

Step 2: Connect to Your Virtual Machine

With the virtual machine running, connect over SSH to reach the Docker command line.

Wait for the virtual machine to become active

On the Virtual Machines page, wait until the virtual machine shows the ACTIVE status and a public IP address. Services can take a short time to finish initializing after the machine becomes active.
Connect over SSH

From your local terminal, connect as the ubuntu user. Replace [public-ip] with the public IP address from the virtual machine's details page and [path-to-private-key] with the path to your private key file.
Connect to the virtual machine over SSH
```
ssh -i [path-to-private-key] ubuntu@[public-ip]
```
For more detail on SSH access, see Connecting to an Ubuntu VM.
Confirm Docker is installed

Check the Docker Engine and Docker Compose versions to confirm both are available:
Check the Docker and Compose versions
```
docker --version
docker compose version
```
The output reports the installed versions, similar to the following:
Example output
```
Docker version 29.1.4, build 0e6fee6
Docker Compose version v5.0.1
```

You now have a shell on a virtual machine with Docker ready to use. Next, run a GPU-accelerated container to confirm containers can reach the GPU.

Step 3: Run a GPU-Accelerated Container

Confirm that a container can access the GPU by running nvidia-smi inside an official NVIDIA CUDA image. The --gpus all flag passes every GPU on the virtual machine through to the container, and --rm removes the container when it exits.

Run nvidia-smi inside a container
docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi

Docker pulls the image on first use, then prints the GPU table from inside the container:

Example output
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 570.195.03             Driver Version: 570.195.03     CUDA Version: 12.8     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA RTX A4000               On  |   00000000:00:06.0 Off |                    0 |
| 41%   34C    P8              7W /  140W |       1MiB /  15352MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

Seeing the GPU listed inside the container confirms the NVIDIA runtime is working. You can now run any GPU-accelerated image, such as framework containers from the NVIDIA NGC catalog.

Run Multi-Container Workloads with Docker Compose

Docker Compose defines and runs multi-container applications from a single file. To give a Compose service GPU access, reserve GPU devices under deploy.resources.reservations.devices, as described in the Docker Compose GPU support guide.

Create a Compose file

On the virtual machine, create a directory named gpu-demo, then add a file named compose.yaml inside it with the following content. The capabilities: [gpu] field is required for GPU reservations, and count: all requests every GPU on the host.

compose.yaml
services:
  gpu-test:
    image: nvidia/cuda:12.4.1-base-ubuntu22.04
    command: nvidia-smi
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

Start the service

From the gpu-demo directory, start the service:

Run the Compose service
docker compose up

Compose creates and runs the container, which prints the GPU table and then exits. The container name is prefixed with the directory name:

Example output
 Container gpu-demo-gpu-test-1  Created
gpu-test-1  | | NVIDIA-SMI 570.195.03             Driver Version: 570.195.03     CUDA Version: 12.8     |
gpu-test-1  | |   0  NVIDIA RTX A4000               On  |   00000000:00:06.0 Off |                    0 |
gpu-test-1 exited with code 0

Stop and remove the service

When the service has finished, remove its containers and network:
Tear down the Compose service
```
docker compose down
```

Managing your virtual machine

Virtual machines bill for as long as they are running. When you're finished, hibernate the virtual machine to reduce charges, or delete it if you no longer need it. See VM Status and State Management for lifecycle options.

Install Docker on a Custom Image with Cloud-Init

If you need a base image that does not include Docker, install it automatically at first boot with a cloud-init script. The script below installs Docker Engine, the Docker Compose plugin, and the NVIDIA Container Toolkit, then configures the Docker runtime for GPU access.

Use an image with the NVIDIA driver

The NVIDIA Container Toolkit gives containers access to the GPU, but it relies on the NVIDIA driver being present on the virtual machine. Deploy from an image that includes the driver and CUDA, such as Ubuntu Server 24.04 LTS R570 CUDA 12.8, so GPU containers work after the script runs.

Console
API

Deploy a virtual machine as described in Step 1, but choose a plain CUDA image without the with Docker suffix. Before deploying, add the cloud-init script:

Open the cloud-init field

On the deployment page, click Configure Additional Settings to expand the panel, then scroll to the Cloud-init Script section.

Add the install script

Select the Bash format, then paste the following script into the field:

Install Docker and the NVIDIA Container Toolkit
#!/bin/bash
set -eux

# Install Docker Engine from Docker's official apt repository
apt-get update
apt-get install -y ca-certificates curl gnupg
install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
chmod a+r /etc/apt/keyrings/docker.asc
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu $(. /etc/os-release && echo "$VERSION_CODENAME") stable" > /etc/apt/sources.list.d/docker.list
apt-get update
apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

# Allow the default 'ubuntu' user to run docker without sudo
usermod -aG docker ubuntu

# Install the NVIDIA Container Toolkit from NVIDIA's official repository
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -fsSL https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' > /etc/apt/sources.list.d/nvidia-container-toolkit.list
apt-get update
apt-get install -y nvidia-container-toolkit

# Configure the Docker runtime to use the NVIDIA runtime and restart Docker
nvidia-ctk runtime configure --runtime=docker
systemctl restart docker

Deploy and wait for the script to finish

Finish configuring the virtual machine, then click Deploy. The script runs at first boot and takes a short time to complete after the machine becomes ACTIVE. Connect over SSH, then confirm it finished:
Check cloud-init status
```
sudo cloud-init status --wait
```

Pass the same script to the Create Virtual Machine API in the user_data field. The example below reads the script from a file named user_data.txt and submits it as part of the request. Replace the environment, image, flavor, and key names with your own values.

# Convert the cloud-init script to a single-line JSON string
user_data_content=$(jq -Rs '.' < user_data.txt)

# Create the VM with the script as user data
curl --location 'https://infrahub-api.Hyperstack.com/v1/core/virtual-machines' \
--header 'Content-Type: application/json' \
--header "api_key: $API_KEY" \
--data '{
  "name": "docker-host",
  "environment_name": "your-environment",
  "image_name": "Ubuntu Server 24.04 LTS R570 CUDA 12.8",
  "flavor_name": "your-flavor",
  "key_name": "your-keypair",
  "count": 1,
  "assign_floating_ip": true,
  "security_rules": [
      {
          "direction": "ingress",
          "protocol": "tcp",
          "ethertype": "IPv4",
          "remote_ip_prefix": "0.0.0.0/0",
          "port_range_min": 22,
          "port_range_max": 22
      }
  ],
  "user_data": '"$user_data_content"'
}'

After the script finishes, verify the installation with the steps in Step 3.

Troubleshooting

Find solutions to common issues you might hit while following this tutorial. Select an issue to expand its solution:

`docker` returns "permission denied" on the Docker socket

If docker commands fail with permission denied while trying to connect to the Docker daemon socket, your shell session does not yet have the docker group membership added by the install script. Log out and reconnect over SSH to start a new session, run newgrp docker in the current session, or prefix the command with sudo.

`docker run --gpus all` fails with "could not select device driver"

If the GPU flag fails with could not select device driver "" with capabilities: [[gpu]], the NVIDIA runtime is not registered with Docker. This happens on images where the NVIDIA Container Toolkit is not installed or configured. Install the toolkit, register it with sudo nvidia-ctk runtime configure --runtime=docker, and restart Docker with sudo systemctl restart docker. Deploying from a with Docker image avoids this step entirely.

`nvidia-smi` inside a container reports "Failed to initialize NVML"

If a container running nvidia-smi returns Failed to initialize NVML: Unknown Error, confirm the GPU is visible on the host by running nvidia-smi directly on the virtual machine. If the host command fails as well, the GPU initialized incorrectly at boot. Reboot the virtual machine from its details page with Hard Reboot, wait for the ACTIVE state, then run the container again.

Deployment fails with "Not Enough Stock"

GPU availability is allocated in real time, so a flavor can run out of stock between checking and deploying. Choose a flavor that has stock, select a different region, or retry the deployment.

Back to top

Was this page helpful?

Step 1: Deploy a Virtual Machine with Docker​

Open the deployment page​

Select a GPU flavor​

Select an environment​

Choose an OS image with Docker​

Select an SSH key​

Assign a public IP address​

Deploy the virtual machine​

Step 2: Connect to Your Virtual Machine​

Wait for the virtual machine to become active​

Connect over SSH​

Confirm Docker is installed​

Step 3: Run a GPU-Accelerated Container​

Run Multi-Container Workloads with Docker Compose​

Create a Compose file​

Start the service​

Stop and remove the service​

Install Docker on a Custom Image with Cloud-Init​

Open the cloud-init field​

Add the install script​

Deploy and wait for the script to finish​

Troubleshooting​

docker returns "permission denied" on the Docker socket

docker run --gpus all fails with "could not select device driver"

nvidia-smi inside a container reports "Failed to initialize NVML"

Deployment fails with "Not Enough Stock"

Back to top

Step 1: Deploy a Virtual Machine with Docker

Open the deployment page

Select a GPU flavor

Select an environment

Choose an OS image with Docker

Select an SSH key

Assign a public IP address

Deploy the virtual machine

Step 2: Connect to Your Virtual Machine

Wait for the virtual machine to become active

Connect over SSH

Confirm Docker is installed