Running Ollama

Run large language models on a GPU virtual machine with Ollama for a private, low-latency endpoint.

Ollama is an open-source tool for running large language models locally. It wraps model management, GPU scheduling, and a REST API into a single container, so you can pull a model and start serving it with two commands. Running Ollama on a dedicated GPU virtual machine gives you a private, low-latency model endpoint with no usage limits or per-token costs.

By the end of this tutorial, you will have Ollama running on a GPU virtual machine and a working OpenAI-compatible endpoint you can query from your local machine.

Other ways to serve a model

This tutorial covers self-managed serving on a single virtual machine. For a managed, zero-infrastructure option, AI Studio serves open-source models through a serverless OpenAI-compatible API. To serve larger models or handle more concurrent requests, pair Ollama with Open WebUI for a browser-based chat interface, or see Deploy an LLM with vLLM on Kubernetes for a scalable multi-GPU setup.

Step 1: Deploy a GPU virtual machine

Ollama loads the model into GPU memory, so the virtual machine needs enough VRAM for the model and its context window. A model uses more VRAM than its download size suggests, since Ollama also allocates a key-value cache for the context window. Choose a GPU with headroom and confirm the loaded footprint with ollama ps, shown in Step 4.

In Hyperstack, navigate to the Virtual Machines page and click Deploy New Virtual Machine.
Select a GPU flavor with enough VRAM for your model. See flavors for the VRAM of each GPU.
For the OS image, choose an Ubuntu image that includes CUDA drivers and Docker, such as Ubuntu Server 24.04 LTS R570 CUDA 12.8 with Docker. This image ships the NVIDIA driver, the CUDA toolkit, Docker, and the NVIDIA Container Toolkit, so the Ollama container can reach the GPU with no further setup.
Select an SSH key, enable the SSH Access toggle so port 22 is reachable, and enable the Assign Public IP toggle so the virtual machine gets a public IP address. For full deployment options, see the getting started guide.
Click Deploy. The virtual machine reaches the ACTIVE state in a few minutes.

Once the virtual machine is ACTIVE, connect to it over SSH to start Ollama.

Step 2: Connect to the virtual machine

Connect over SSH to run the Ollama container and, later, to forward the API port to your local machine.

Find the virtual machine's public IP in the PUBLIC IP column on the Virtual Machines page.
Run the following command, replacing <path-to-ssh-key> with the path to your private SSH key and <vm-ip-address> with the public IP. If you downloaded the key from the console, restrict its permissions first or SSH will refuse it:
Set key permissions and connect
```
chmod 400 <path-to-ssh-key>
ssh -i <path-to-ssh-key> ubuntu@<vm-ip-address>
```

With a shell open on the virtual machine, you can start Ollama.

Step 3: Start Ollama and pull a model

Ollama's official container image includes the server, the model manager, and full GPU support. Starting the container launches the API server; then you pull a model into it.

Start the Ollama container:
Start the Ollama container
```
sudo docker run -d \
  --gpus=all \
  -p 11434:11434 \
  -v /home/ubuntu/ollama:/root/.ollama \
  --name ollama \
  --restart always \
  ollama/ollama:latest
```
The flags pass all GPUs through to the container (--gpus=all), publish the API on port 11434 (-p 11434:11434), and store downloaded model weights in /home/ubuntu/ollama so they persist across container restarts (-v ...). The --restart always policy restarts Ollama automatically if the virtual machine reboots.
Pull a model. This example uses llama3.2, a 2 GB model that fits on any GPU. The weights are saved to the volume mount from the previous step, so subsequent starts skip the download:
Pull a model
```
sudo docker exec ollama ollama pull llama3.2
```
The model downloads and is ready to serve when the command returns.

With Ollama running and the model downloaded, you can query it from your local machine.

Step 4: Query the model

Ollama listens on port 11434. Because that port is not open in the virtual machine's firewall, reach it securely from your local machine with an SSH port-forward rather than exposing the endpoint publicly.

Open a new terminal on your local machine (not inside the SSH session). Run the following command to forward local port 11434 to the Ollama server. Keep this terminal open while you send requests:
Forward the API port to your local machine
```
ssh -i <path-to-ssh-key> -L 11434:localhost:11434 ubuntu@<vm-ip-address>
```

In a second terminal on your local machine, send a chat request with curl:

Send a chat request
curl http://localhost:11434/api/chat \
  -d '{
    "model": "llama3.2",
    "messages": [
      {"role": "user", "content": "In one sentence, what is a GPU used for in machine learning?"}
    ],
    "stream": false
  }'

The response contains the generated message and timing information:

Example response
{
  "model": "llama3.2",
  "created_at": "2026-06-12T18:38:31.580406194Z",
  "message": {
    "role": "assistant",
    "content": "A GPU is widely used in machine learning due to its ability to handle high-volume matrix operations and parallel processing, making it ideal for training deep neural networks."
  },
  "done": true,
  "done_reason": "stop",
  "prompt_eval_count": 34,
  "eval_count": 37
}

Confirm the model is running on the GPU. In any terminal connected to the virtual machine, run:
Check GPU usage
```
sudo docker exec ollama ollama ps
```
The output shows the model name, its VRAM footprint, and whether it is using the GPU or CPU:
Example output
```
NAME               ID              SIZE     PROCESSOR    CONTEXT    UNTIL
llama3.2:latest    a80c4f17acd5    17 GB    100% GPU     131072     4 minutes from now
```
GPU memory usage
Ollama reserves VRAM for the model weights plus a KV cache for the active context window. The UNTIL timestamp shows when Ollama will unload the model from VRAM after the last request. You can adjust this with the OLLAMA_KEEP_ALIVE environment variable.
Ollama also exposes an OpenAI-compatible API at /v1/chat/completions, so you can use the OpenAI Python client without changing your application code. Install the package if you have not already (pip install openai), then run the following. Set api_key to any non-empty string since Ollama does not enforce authentication by default:
Query with the OpenAI Python client
```
from openai import OpenAI

client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")

response = client.chat.completions.create(
    model="llama3.2",
    messages=[{"role": "user", "content": "Say hello in exactly three words."}],
)
print(response.choices[0].message.content)
```

Pull more models

Run sudo docker exec ollama ollama pull <model> on the virtual machine to add more models. Browse available models at ollama.com/library. Each model stays cached on the volume mount, and Ollama loads the appropriate one automatically when a request specifies it.

Scaling and managed alternatives

For serverless inference with no virtual machine to manage, AI Studio serves open-source models through the same OpenAI-compatible API shape.

Managing your virtual machine

Virtual machines bill for as long as they are running. When you're finished, hibernate the virtual machine to reduce charges, or delete it if you no longer need it. See VM Status and State Management for lifecycle options.

Troubleshooting

Find solutions to common issues you might hit while following this tutorial. Select an issue to expand its solution:

The model runs on CPU instead of GPU

If ollama ps shows 100% CPU instead of 100% GPU, the container was started without the --gpus=all flag. Stop and remove the container, then run the docker run command from Step 3 again with the flag included:

Remove and restart the container
sudo docker rm -f ollama

You cannot reach port 11434 from your local machine

Port 11434 is not open in the virtual machine's firewall by design, so the endpoint is not publicly accessible. Query it through the SSH port-forward shown in Step 4 and verify the tunnel terminal is still open.

The container exits after a reboot

If Ollama does not restart after a virtual machine reboot, confirm the container was started with --restart always. Check the restart policy with:

Check restart policy
sudo docker inspect ollama --format '{{.HostConfig.RestartPolicy.Name}}'

If the output is not always, remove the container and run the docker run command again with --restart always included.

Back to top

Was this page helpful?

Step 1: Deploy a GPU virtual machine​

Step 2: Connect to the virtual machine​

Step 3: Start Ollama and pull a model​

Step 4: Query the model​

Troubleshooting​