Running Multi-GPU vLLM

Run a large language model across several GPUs on one virtual machine with vLLM tensor parallelism.

A large language model that does not fit in a single GPU's memory can still run on one virtual machine by splitting it across several GPUs. vLLM does this with tensor parallelism: it shards each layer's weights across the GPUs you give it, so the GPUs hold one model together and serve it behind a single OpenAI-compatible API.

This tutorial walks through deploying a multi-GPU virtual machine, serving Qwen/Qwen2.5-72B-Instruct across four GPUs with one command-line flag, and querying the model from your local machine.

Other ways to serve a model

If your model fits on a single GPU, Running vLLM covers the same workflow on one GPU. To scale across multiple nodes with load balancing and high availability, see Deploy an LLM with vLLM on Kubernetes. For a managed, zero-infrastructure option, AI Studio serves open-source models through a serverless OpenAI-compatible API.

Step 1: Deploy a multi-GPU virtual machine

Tensor parallelism spreads the model across the GPUs on one virtual machine, so the first decision is a flavor whose GPUs hold the model together. Add up the VRAM: a 16-bit model needs roughly two GB of GPU memory per billion parameters for the weights, plus headroom for the key-value cache. Qwen/Qwen2.5-72B-Instruct is about 145 GB in 16-bit precision, so this tutorial uses four H100 GPUs (320 GB combined).

In Hyperstack, navigate to the Virtual Machines page and click Deploy New Virtual Machine.
In the GPU Flavor section, select the H100-80G-PCIe flavor and choose 4x from its GPU-count dropdown, which gives four H100 GPUs with 320 GB of combined VRAM. Any multi-GPU flavor works as long as its combined VRAM holds your model. See flavors for the VRAM of each GPU.
For the OS image, choose an Ubuntu image that includes CUDA drivers and Docker, such as Ubuntu Server 24.04 LTS R570 CUDA 12.8 with Docker. This image ships the NVIDIA driver, the CUDA toolkit, Docker, and the NVIDIA Container Toolkit, so the vLLM container can reach every GPU with no further setup.
Select an SSH key, enable the SSH Access toggle so port 22 is reachable, and enable the Assign Public IP toggle so the virtual machine gets a public IP address. For full deployment options, see the getting started guide.
Click Deploy. The virtual machine reaches the ACTIVE state in a few minutes.

Once the virtual machine is ACTIVE, connect to it over SSH to start the server.

Step 2: Connect to the virtual machine

Connect over SSH to run the vLLM container and, later, to forward the API port to your local machine.

Find the virtual machine's public IP in the Public IP column on the Virtual Machines page.
Execute the following command, replacing <path-to-ssh-key> with the path to your private SSH key and <vm-ip-address> with the public IP. If you downloaded the key from the console, restrict its permissions first or SSH will refuse it:
Set key permissions and connect
```
chmod 400 <path-to-ssh-key>
ssh -i <path-to-ssh-key> ubuntu@<vm-ip-address>
```

With a session open on the virtual machine, you can start the vLLM server.

Step 3: Start the vLLM server with tensor parallelism

The official vllm/vllm-openai container image runs the vLLM OpenAI-compatible API server. The --tensor-parallel-size flag tells vLLM how many GPUs to shard the model across.

A 145 GB model does not fit on the virtual machine's root disk, which is around 96 GB. Store the downloaded weights on the larger ephemeral drive instead. Create a directory on the ephemeral drive for the model cache:
Create a cache directory on the ephemeral drive
```
sudo mkdir -p /ephemeral/hf && sudo chmod 777 /ephemeral/hf
```
Start the server with the command below. The --tensor-parallel-size 4 flag shards the model across all four GPUs:
Start the vLLM server across four GPUs
```
sudo docker run -d --name vllm \
  --gpus all \
  -p 8000:8000 \
  --ipc=host \
  -v /ephemeral/hf:/root/.cache/huggingface \
  vllm/vllm-openai:latest \
  --model Qwen/Qwen2.5-72B-Instruct \
  --tensor-parallel-size 4
```
The flags pass every GPU through to the container (--gpus all), publish the API on port 8000 (-p 8000:8000), grant the shared memory vLLM needs for inter-GPU communication (--ipc=host), and cache the downloaded weights on the ephemeral drive so later restarts skip the download (-v ...).

Set --tensor-parallel-size to the number of GPUs on the virtual machine. The value must divide the model's attention head count evenly, so it is almost always a power of two. A four-GPU flavor uses --tensor-parallel-size 4; a two-GPU flavor uses --tensor-parallel-size 2.

Gated models
Qwen/Qwen2.5-72B-Instruct is open and downloads without an account. Models with gated access, such as the original Meta Llama weights, require a Hugging Face token. Add -e HF_TOKEN=<your-token> before the image name in the command.
The server downloads the model and loads a shard onto each GPU before it accepts requests. This takes several minutes on first run; subsequent starts are faster because the weights are cached. Watch the logs until the line Application startup complete appears:
Follow the server logs
```
sudo docker logs -f vllm
```
Confirm the server is ready. When the health endpoint returns 200, the model is serving:
Check server health
```
curl -o /dev/null -w "%{http_code}\n" http://localhost:8000/health
```
Confirm the model is sharded across every GPU. Each GPU should show a similar amount of memory in use:
Check GPU memory across all GPUs
```
nvidia-smi --query-gpu=index,memory.used,memory.total --format=csv
```
The output lists one row per GPU, each holding part of the model:
Example output
```
index, memory.used [MiB], memory.total [MiB]
0, 76243 MiB, 81559 MiB
1, 76171 MiB, 81559 MiB
2, 76203 MiB, 81559 MiB
3, 76115 MiB, 81559 MiB
```
GPU memory usage
vLLM reserves about 90% of each GPU's memory for the model and key-value cache by default, so a near-full reading on every GPU is expected. Lower this with --gpu-memory-utilization if you need to leave headroom.

With the model serving across all four GPUs, you can query it from your local machine.

Step 4: Query the model

The server exposes an OpenAI-compatible API on port 8000. Because port 8000 is not open in the virtual machine's firewall, reach it securely from your local machine with an SSH port-forward rather than exposing the endpoint publicly.

Open a new terminal on your local machine (not inside the SSH session). Run the following command to forward local port 8000 to the server. Keep this terminal open while you send requests:
Forward the API port to your local machine
```
ssh -i <path-to-ssh-key> -L 8000:localhost:8000 ubuntu@<vm-ip-address>
```

In a second terminal on your local machine, send a chat request with curl:

Send a chat request
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen2.5-72B-Instruct",
    "messages": [
      {"role": "system", "content": "You are a helpful assistant."},
      {"role": "user", "content": "In one sentence, what is tensor parallelism in large language model inference?"}
    ]
  }'

The response contains the generated message and token counts:

Example response
{
  "id": "chatcmpl-a61ff459a5cb8233",
  "object": "chat.completion",
  "model": "Qwen/Qwen2.5-72B-Instruct",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "Tensor parallelism in large language model inference involves splitting the model's tensors across multiple GPUs to enable efficient parallel computation and handle models that exceed the memory capacity of a single GPU."
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 34,
    "total_tokens": 70,
    "completion_tokens": 36
  }
}

Because the API is OpenAI-compatible, you can point the OpenAI Python client at the server by changing the base_url. Install the package if you have not already (pip install openai), then run the following. The api_key is required by the client but unused unless you start the server with --api-key:
Query with the OpenAI Python client
```
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")

response = client.chat.completions.create(
    model="Qwen/Qwen2.5-72B-Instruct",
    messages=[{"role": "user", "content": "Say hello in exactly three words."}],
)
print(response.choices[0].message.content)
```

Scaling further

Tensor parallelism works across the GPUs of a single virtual machine. To split a model by layers instead, for example across GPUs without a high-speed interconnect, add --pipeline-parallel-size. To serve across multiple nodes with load balancing and high availability, see Deploy an LLM with vLLM on Kubernetes. For serverless inference with no virtual machine to manage, AI Studio serves open-source models through the same OpenAI-compatible API shape.

Managing your virtual machine

Virtual machines bill for as long as they are running. When you're finished, hibernate the virtual machine to reduce charges, or delete it if you no longer need it. See VM Status and State Management for lifecycle options.

Troubleshooting

Find solutions to common issues you might hit while following this tutorial. Select an issue to expand its solution:

The model download fills the disk and the container exits

Large models do not fit on the root disk. Store the weights on the ephemeral drive by creating a cache directory there and mounting it into the container with -v /ephemeral/hf:/root/.cache/huggingface, as shown in Step 3. Check the available space on each disk with df -h.

vLLM reports an attention head or GPU count error on startup

--tensor-parallel-size must equal the number of GPUs on the virtual machine, and it must divide the model's attention head count evenly. Set it to a power of two that matches your GPU count, such as 2, 4, or 8. Confirm the GPU count with nvidia-smi --list-gpus.

Requests fail or hang right after starting the container

The server accepts requests only after the model finishes downloading and loading onto every GPU. Until then, curl to the endpoint is refused. Follow sudo docker logs -f vllm and wait for Application startup complete, then confirm http://localhost:8000/health returns 200.

The container reports out of memory while loading the model

The combined VRAM of the GPUs is too small for the model. Deploy a flavor with more GPUs or more GPU memory, choose a smaller model, or lower --gpu-memory-utilization to leave less headroom for the key-value cache.

You cannot reach port 8000 from your local machine

Port 8000 is not open in the virtual machine's firewall by design, so the model is not exposed publicly. Query it through the SSH port-forward shown in Step 4. Verify the tunnel session is still open.

Back to top

Was this page helpful?

Step 1: Deploy a multi-GPU virtual machine​

Step 2: Connect to the virtual machine​

Step 3: Start the vLLM server with tensor parallelism​

Step 4: Query the model​

Troubleshooting​