Running vLLM

Serve open-source LLMs behind an OpenAI-compatible API with vLLM on a dedicated GPU virtual machine.

vLLM is a high-throughput inference engine that serves open-source large language models behind an OpenAI-compatible API. Running it on a dedicated GPU virtual machine gives you full control over the model, the hardware, and the endpoint, with no per-token pricing and no resource oversubscription.

This tutorial walks through deploying a GPU virtual machine, starting a vLLM server with the official container image, and querying a running model from your local machine.

Other ways to serve a model

This tutorial covers self-managed serving on a single GPU. To serve a model that does not fit on one GPU by sharding it across several GPUs on one virtual machine, see Running Multi-GPU vLLM. To scale across multiple nodes with load balancing and high availability, see Deploy an LLM with vLLM on Kubernetes. For a managed, zero-infrastructure option, AI Studio serves open-source models through a serverless OpenAI-compatible API.

Step 1: Deploy a GPU virtual machine

vLLM loads the entire model into GPU memory, so the first decision is a flavor with enough VRAM for the model you intend to serve. The steps below deploy a virtual machine that is ready to run the vLLM container.

In Hyperstack, navigate to the 'Virtual Machines' page and click "Deploy New Virtual Machine".
Select a GPU flavor with enough VRAM to hold your model. A GPU with at least 16 GB of VRAM runs a small model such as Qwen/Qwen2.5-1.5B-Instruct. Larger models need proportionally more memory. See flavors for the VRAM of each GPU.
For the OS image, choose an Ubuntu image that includes CUDA drivers and Docker, such as Ubuntu Server 24.04 LTS R570 CUDA 12.8 with Docker. This image ships the NVIDIA driver, the CUDA toolkit, Docker, and the NVIDIA Container Toolkit, so the vLLM container can reach the GPU with no further setup.
Select an SSH key, enable the SSH Access toggle so port 22 is reachable, and enable the Assign Public IP toggle so the virtual machine gets a public IP address. For full deployment options, see the getting started guide.
Click "Deploy". The virtual machine reaches the ACTIVE state in a few minutes.

Once the virtual machine is ACTIVE, connect to it over SSH to start the server.

Step 2: Connect to the virtual machine

Connect over SSH to run the vLLM container and, later, to forward the API port to your local machine.

Find the virtual machine's public IP in the 'PUBLIC IP' column on the 'Virtual Machines' page.
Execute the following command, replacing <path-to-ssh-key> with the path to your private SSH key and <vm-ip-address> with the public IP. If you downloaded the key from the console, restrict its permissions first or SSH will refuse it:
Set key permissions and connect
```
chmod 400 <path-to-ssh-key>
ssh -i <path-to-ssh-key> ubuntu@<vm-ip-address>
```

With a session open on the virtual machine, you can start the vLLM server.

Step 3: Start the vLLM server

The official vllm/vllm-openai container image runs the vLLM OpenAI-compatible API server. This avoids installing Python, CUDA, or vLLM on the host directly.

Start the server with the command below. Replace the model with any Hugging Face model that fits your GPU.
Start the vLLM server
```
sudo docker run -d --name vllm \
  --gpus all \
  -p 8000:8000 \
  --ipc=host \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:latest \
  --model Qwen/Qwen2.5-1.5B-Instruct
```
The flags set up GPU access (--gpus all), publish the API on port 8000 (-p 8000:8000), grant the shared memory vLLM needs (--ipc=host), and cache downloaded model weights on the virtual machine so later restarts skip the download (-v ...).
Gated models
Qwen/Qwen2.5-1.5B-Instruct is open and downloads without an account. Models with gated access, such as the original Meta Llama weights, require a Hugging Face token. Add -e HF_TOKEN=<your-token> before the image name in the command, like this:
Start the vLLM server with a Hugging Face token
sudo docker run -d --name vllm \ --gpus all \ -p 8000:8000 \ --ipc=host \ -v ~/.cache/huggingface:/root/.cache/huggingface \ -e HF_TOKEN=<your-token> \ vllm/vllm-openai:latest \ --model <gated-model-name>
The server downloads the model and loads it into GPU memory before it accepts requests. This takes a few minutes on first run; subsequent starts are faster because the weights are cached. Watch the logs until the line Application startup complete appears:
Follow the server logs
```
sudo docker logs -f vllm
```
Confirm the server is ready. When the health endpoint returns 200, the model is serving:
Check server health
```
curl -o /dev/null -w "%{http_code}\n" http://localhost:8000/health
```
GPU memory usage
vLLM reserves about 90% of GPU memory for the model and key-value cache by default, so a near-full GPU reading is expected even for a small model. Lower this with --gpu-memory-utilization if you need to leave headroom.

With the server healthy, you can query the model.

Step 4: Query the model

The server exposes an OpenAI-compatible API on port 8000. Because port 8000 is not open in the virtual machine's firewall, reach it securely from your local machine with an SSH port-forward rather than exposing the endpoint publicly.

Open a new terminal on your local machine (not inside the SSH session). Run the following command to forward local port 8000 to the server. Keep this terminal open while you send requests:
Forward the API port to your local machine
```
ssh -i <path-to-ssh-key> -L 8000:localhost:8000 ubuntu@<vm-ip-address>
```

In a second terminal on your local machine, send a chat request with curl:

Send a chat request
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen2.5-1.5B-Instruct",
    "messages": [
      {"role": "system", "content": "You are a helpful assistant."},
      {"role": "user", "content": "In one sentence, what is a GPU used for in machine learning?"}
    ]
  }'

The response contains the generated message and token counts:

Example response
{
  "id": "chatcmpl-a04d379ff238bcb9",
  "object": "chat.completion",
  "model": "Qwen/Qwen2.5-1.5B-Instruct",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "A GPU is used in machine learning to accelerate computations and speed up training processes by utilizing its parallel processing capabilities."
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 33,
    "total_tokens": 56,
    "completion_tokens": 23
  }
}

Because the API is OpenAI-compatible, you can point the OpenAI Python client at the server by changing the base_url. Install the package if you have not already (pip install openai), then run the following. The api_key is required by the client but unused unless you start the server with --api-key:
Query with the OpenAI Python client
```
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")

response = client.chat.completions.create(
    model="Qwen/Qwen2.5-1.5B-Instruct",
    messages=[{"role": "user", "content": "Say hello in exactly three words."}],
)
print(response.choices[0].message.content)
```

Scaling and managed alternatives

To serve a model that does not fit on one GPU, shard it across several GPUs on one virtual machine with Running Multi-GPU vLLM. To scale across multiple nodes with load balancing and high availability, see Deploy an LLM with vLLM on Kubernetes. For serverless inference with no virtual machine to manage, AI Studio serves open-source models through the same OpenAI-compatible API shape.

Managing your virtual machine

Virtual machines bill for as long as they are running. When you're finished, hibernate the virtual machine to reduce charges, or delete it if you no longer need it. See VM Status and State Management for lifecycle options.

Troubleshooting

Find solutions to common issues you might hit while following this tutorial. Select an issue to expand its solution:

Requests fail or hang right after starting the container

The server accepts requests only after the model finishes downloading and loading into GPU memory. Until then, curl to the endpoint is refused. Follow sudo docker logs -f vllm and wait for Application startup complete, then confirm http://localhost:8000/health returns 200.

You cannot reach port 8000 from your local machine

Port 8000 is not open in the virtual machine's firewall by design, so the model is not exposed publicly. Query it through the SSH port-forward shown in Step 4. Verify the tunnel session is still open.

The container exits or reports out of memory

The model is too large for the GPU's VRAM. Deploy a flavor with more GPU memory, choose a smaller model, or lower --gpu-memory-utilization.

Back to top

Was this page helpful?

Step 1: Deploy a GPU virtual machine​

Step 2: Connect to the virtual machine​

Step 3: Start the vLLM server​

Step 4: Query the model​

Troubleshooting​