Running an LLM on Kubernetes

Deploy an LLM across multiple GPUs with vLLM and serve it through an OpenAI-compatible API on Kubernetes.

Deploy a Large Language Model across multiple GPUs and serve it through an OpenAI-compatible API on a Hyperstack Kubernetes cluster. vLLM is a high-throughput inference engine for LLMs, and running it on Kubernetes lets you distribute the model across several GPU worker nodes for availability and scale.

This tutorial walks through connecting to your cluster's bastion server, deploying vLLM with the Llama-3-8B model, and sending a test request to confirm the model is serving.

Step 1: Connect to the Bastion Server

The bastion server is the secure gateway to your Kubernetes cluster. You connect to it over SSH, then run every kubectl and helm command from there.

Prerequisites

Before you begin, make sure you have:

A running Hyperstack Kubernetes cluster. If you don't have one, follow How to deploy a Kubernetes cluster first.
The SSH key pair associated with the cluster, available on your local machine.

Set your connection variables

Set BASTION_IP_ADDRESS to the bastion server's IP address, shown on your cluster's details page in the Hyperstack console, and KEYPAIR_PATH to the path of the SSH key pair associated with the cluster on your local machine.
```
BASTION_IP_ADDRESS="<bastion-ip>"
KEYPAIR_PATH="<path-to-ssh-key>"
```
note
The deployment in this tutorial uses the NousResearch build of Llama-3-8B, which you can pull without a Hugging Face account or token. To use Meta's original model instead, pass an HF_TOKEN environment variable to the deployment.
Clear any stale SSH host key

If you have previously connected to a different cluster at the same IP address, remove the old host key to avoid an SSH host-key conflict:
```
ssh-keygen -R "$BASTION_IP_ADDRESS"
```

Connect to the bastion server

Connect over SSH:

ssh -i "$KEYPAIR_PATH" ubuntu@"$BASTION_IP_ADDRESS"

With a shell open on the bastion server, you can manage the cluster with kubectl. Next, deploy vLLM to the cluster.

Step 2: Deploy vLLM to the Cluster

Deploy vLLM as a Kubernetes Deployment and expose it with a Service. Run these commands from the bastion server.

Create a namespace

Create a dedicated namespace for the vLLM resources:
```
kubectl create ns vllm-ns
```

Create the deployment manifest

The following manifest deploys vLLM with four pod replicas using the vllm/vllm-openai:latest image, each serving the NousResearch/Meta-Llama-3-8B-Instruct model on one GPU (four GPUs in total). It includes a rolling update strategy, and liveness and readiness probes so Kubernetes only routes traffic to pods that are ready.

note

Set replicas to the number of GPUs in your cluster, between 1 and 8, with one GPU per replica. For example, a cluster with four GPUs uses a replicas value of 4.

cat <<EOF > vllm_deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: vllm-app
  name: vllm
  namespace: vllm-ns
spec:
  replicas: 4
  selector:
    matchLabels:
      app: vllm-app
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: vllm-app
    spec:
      containers:
      - command:
        - python3
        - -m
        - vllm.entrypoints.openai.api_server
        - --model
        - NousResearch/Meta-Llama-3-8B-Instruct
        image: vllm/vllm-openai:latest
        imagePullPolicy: Always
        livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /health
            port: 8000
            scheme: HTTP
          initialDelaySeconds: 240
          periodSeconds: 5
          successThreshold: 1
          timeoutSeconds: 1
        name: vllm-openai
        ports:
        - containerPort: 8000
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /health
            port: 8000
            scheme: HTTP
          initialDelaySeconds: 240
          periodSeconds: 5
          successThreshold: 1
          timeoutSeconds: 1
        resources:
          limits:
            nvidia.com/gpu: "1"
          requests:
            nvidia.com/gpu: "1"
        volumeMounts:
        - mountPath: /root/.cache/huggingface
          name: cache-volume
      volumes:
      - emptyDir: {}
        name: cache-volume
EOF

Create the service manifest

Create a Service to expose the deployment inside the cluster:

cat <<EOF > vllm_service.yaml
apiVersion: v1
kind: Service
metadata:
  labels:
    app: vllm-app
  name: vllm-openai-svc
  namespace: vllm-ns
spec:
  ports:
  - port: 8000
    protocol: TCP
    targetPort: 8000
  selector:
    app: vllm-app
  type: ClusterIP
EOF

Apply the manifests

Apply the deployment and service:

kubectl apply -f vllm_deployment.yaml
kubectl apply -f vllm_service.yaml

Check the deployment status

Confirm the deployment is progressing and the pods are starting:
```
kubectl describe deployments -n vllm-ns
```
The pods take a few minutes to pull the image and download the model before they report ready. Track them with kubectl get pods -n vllm-ns -w.
Forward the vLLM port

Once the pods are ready, forward the vLLM service to port 8000:
```
kubectl port-forward svc/vllm-openai-svc 8000:8000 -n vllm-ns
```

Your LLM is now serving on the cluster, reachable at http://localhost:8000 on the bastion server. Next, test the model by sending it a request.

Step 3: Test the Model

With the vLLM service running and port-forwarded, send it a request to confirm the model is serving correctly.

Open a second connection to the bastion

The kubectl port-forward command from the previous step occupies its terminal. Open a second terminal and connect to the bastion server again so you can send a request while the forward stays active:
```
ssh -i "$KEYPAIR_PATH" ubuntu@"$BASTION_IP_ADDRESS"
```

Send a test request

Use curl to send a completion request to the model through the forwarded port:

curl -X POST http://localhost:8000/v1/completions \
-H "Content-Type: application/json" \
-d '{
  "model": "NousResearch/Meta-Llama-3-8B-Instruct",
  "prompt": "Explain the benefits of Kubernetes.",
  "max_tokens": 50
}'

Check the response

The model returns a JSON response containing the generated text:

{
  "id": "cmpl-8655f02c6a0e3df4",
  "object": "text_completion",
  "created": 1781558213,
  "model": "NousResearch/Meta-Llama-3-8B-Instruct",
  "choices": [
    {
      "index": 0,
      "text": " What are the key features of Kubernetes that make it a popular choice for container orchestration? Kubernetes is an open-source container orchestration system for automating the deployment, scaling, and management of containerized applications.",
      "logprobs": null,
      "finish_reason": "length",
      "stop_reason": null
    }
  ],
  "usage": {
    "prompt_tokens": 8,
    "total_tokens": 58,
    "completion_tokens": 50
  }
}

You have deployed a Large Language Model on Hyperstack Kubernetes, served across multiple GPU worker nodes through an OpenAI-compatible API. From here you can scale the cluster to add GPU capacity, or point your application at the service endpoint.

Load balancing across replicas

By default, the Kubernetes Service distributes requests across the vLLM replicas using round-robin distribution. This is effective for stateless inference. Workloads that need session persistence, uneven load handling, or more advanced routing may require a dedicated ingress or load balancer.

Troubleshooting

Find solutions to common issues you might hit while following this tutorial. Select an issue to expand its solution:

`kubectl` fails with `connection to the server localhost:8080 was refused`

The bastion server receives its cluster credentials only once the cluster reaches the ACTIVE state. If you connect while the cluster is still provisioning, kubectl has no configuration and falls back to localhost:8080. Confirm the cluster shows the ACTIVE status, as described in Cluster Statuses, then reconnect to the bastion.

Pods stay in `Pending`

Confirm your cluster has enough GPU worker nodes for the replicas count. Each replica requests one GPU, so a replicas value higher than the number of available GPUs leaves the extra pods unschedulable. Run kubectl get nodes to list nodes and kubectl describe pod <pod-name> -n vllm-ns to see the scheduling reason.

A pod stays in `ContainerCreating` or shows a brief `ImagePullBackOff`

The vllm/vllm-openai image is several gigabytes, and each pod then downloads the model on first start, so a pod can take several minutes to become ready. A transient ImagePullBackOff during the initial pull is retried automatically. The readiness and liveness probes allow for this with a 240-second initialDelaySeconds. Watch progress with kubectl get pods -n vllm-ns -w and check a pod's logs with kubectl logs <pod-name> -n vllm-ns.

`curl` to `localhost:8000` is refused

Confirm the kubectl port-forward command is still running in its terminal. It must stay active for the local port to forward to the service.
Confirm the pods are ready with kubectl get pods -n vllm-ns. The service has no ready endpoints until at least one pod passes its readiness probe.

Back to top

Was this page helpful?

Step 1: Connect to the Bastion Server​

Prerequisites​

Set your connection variables​

Clear any stale SSH host key​

Connect to the bastion server​

Step 2: Deploy vLLM to the Cluster​

Create a namespace​

Create the deployment manifest​

Create the service manifest​

Apply the manifests​

Check the deployment status​

Forward the vLLM port​

Step 3: Test the Model​

Open a second connection to the bastion​

Send a test request​

Check the response​

Troubleshooting​

kubectl fails with connection to the server localhost:8080 was refused

Pods stay in Pending

A pod stays in ContainerCreating or shows a brief ImagePullBackOff

curl to localhost:8000 is refused

Back to top

Step 1: Connect to the Bastion Server

Prerequisites

Set your connection variables

Clear any stale SSH host key