Skip to main content

Fine-Tuning LLMs with Unsloth

Fine-tuning an LLM with Unsloth and QLoRA on a GPU virtual machine.

Unsloth is an open-source library that makes QLoRA fine-tuning faster and more memory-efficient than the baseline Hugging Face stack. This tutorial walks through deploying a GPU virtual machine, installing Unsloth, fine-tuning Qwen3-8B with QLoRA on the Alpaca instruction dataset, and running inference with the saved adapter to confirm the fine-tune worked.

The result is a LoRA adapter: a small set of trained weight deltas that you load alongside the frozen base model at inference time. The adapter is only about a hundred megabytes, not a full copy of the 8B model.

AI Studio managed fine-tuning

This tutorial covers self-managed fine-tuning on your own GPU virtual machine. For a no-infrastructure option, AI Studio provides managed fine-tuning through a web interface and API, with no VM to provision or maintain. See the AI Studio fine-tuning docs if that path fits your use case.

Step 1: Deploy a GPU virtual machine​

Qwen3-8B QLoRA training requires a GPU with at least 16 GB of VRAM, though 48 GB gives enough headroom to use a batch size of 2 with gradient accumulation and avoid out-of-memory errors at the default max_seq_length of 2048.

Prerequisites​

  • A Hyperstack account with an SSH key imported. See the getting started guide if you have not created an environment and SSH key yet.

  1. In Hyperstack, navigate to the Virtual Machines page and click Deploy New Virtual Machine.

  2. Select a GPU flavor. The L40 and RTX-A6000 families with at least 48 GB of VRAM are recommended for this tutorial. See flavors for a full list with VRAM per GPU.

  3. For the OS image, select an Ubuntu image that includes CUDA drivers. Choose Ubuntu Server 22.04 LTS R570 CUDA 12.8 or the equivalent Ubuntu 24.04 variant. Unsloth requires CUDA to be available on the host.

  4. Select your SSH key, enable the SSH Access toggle, and enable the Assign Public IP toggle.

  5. Click Deploy. The virtual machine reaches the ACTIVE state in a few minutes.

Ephemeral storage for model downloads

Some flavors include a large ephemeral drive (725 GB on n3-L40x1). If your flavor includes one, it is automatically mounted at /ephemeral. Use it to cache model weights so the root disk does not fill up. See Ephemeral Drive Mounting for details.

With the virtual machine ACTIVE, connect over SSH to set up the training environment.

Step 2: Connect to the virtual machine​

  1. Find the virtual machine's public IP in the PUBLIC IP column on the Virtual Machines page.

  2. If you downloaded the private key from the console, restrict its permissions before using it:

    Set key permissions
    chmod 400 <path-to-ssh-key>

  3. Connect to the virtual machine. Replace <path-to-ssh-key> with your private key path and <vm-ip-address> with the public IP:

    SSH into the virtual machine
    ssh -i <path-to-ssh-key> ubuntu@<vm-ip-address>

You now have a shell on the virtual machine. Next, install Unsloth and its dependencies.

Step 3: Install Unsloth​

Unsloth installs via pip. The command below pulls Unsloth and all required dependencies, including bitsandbytes for 4-bit quantization, trl for the supervised fine-tuning trainer, and datasets for loading the training data.

  1. Set the Hugging Face cache directory to the ephemeral drive if your flavor has one, so model weights do not fill the root disk. If your flavor has no ephemeral drive, skip this step and the cache goes to ~/.cache/huggingface on the root disk:

    Set Hugging Face cache to ephemeral (optional)
    export HF_HOME=/ephemeral/hf
    mkdir -p /ephemeral/hf
    Security note

    This directory is on the ephemeral drive, which is wiped when the virtual machine is deleted. It is suitable for temporary model caches during training but not for long-term storage.

  2. Install Unsloth:

    Install Unsloth
    pip install unsloth

    Unsloth detects your CUDA version and installs compatible PyTorch, Triton, and other dependencies automatically.

With Unsloth installed, write the training script.

Step 4: Write the training script​

Save the following script as train.py on the virtual machine. It loads Qwen3-8B in 4-bit (QLoRA), applies LoRA adapters to the attention and feed-forward layers, trains on the Alpaca instruction dataset for 60 steps, saves the adapter, and runs a quick inference check.

Create the training script
cat > ~/train.py << 'EOF'
import os

# Cache model weights on ephemeral if available, otherwise use home directory
if os.path.isdir("/ephemeral"):
    os.environ["HF_HOME"] = "/ephemeral/hf"
    os.makedirs("/ephemeral/hf", exist_ok=True)

from unsloth import FastLanguageModel
from datasets import load_dataset
from trl import SFTConfig, SFTTrainer
import torch

# ── 1. Load model ──────────────────────────────────────────────────────────────
model, tokenizer = FastLanguageModel.from_pretrained(
    model_name = "unsloth/Qwen3-8B-unsloth-bnb-4bit",
    max_seq_length = 2048,
    dtype = None,
    load_in_4bit = True,
)

# ── 2. Apply QLoRA adapters ────────────────────────────────────────────────────
model = FastLanguageModel.get_peft_model(
    model,
    r = 16,
    target_modules = ["q_proj", "k_proj", "v_proj", "o_proj",
                      "gate_proj", "up_proj", "down_proj"],
    lora_alpha = 16,
    lora_dropout = 0,
    bias = "none",
    use_gradient_checkpointing = "unsloth",
    random_state = 3407,
)

# ── 3. Prepare dataset ─────────────────────────────────────────────────────────
dataset = load_dataset("unsloth/alpaca-cleaned", split="train")

alpaca_prompt = """Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.

### Instruction:
{}

### Input:
{}

### Response:
{}"""

EOS_TOKEN = tokenizer.eos_token

def formatting_prompts_func(examples):
    texts = []
    for instruction, inp, output in zip(
        examples["instruction"], examples["input"], examples["output"]
    ):
        texts.append(alpaca_prompt.format(instruction, inp, output) + EOS_TOKEN)
    return {"text": texts}

dataset = dataset.map(formatting_prompts_func, batched=True)

# ── 4. Train ───────────────────────────────────────────────────────────────────
trainer = SFTTrainer(
    model = model,
    tokenizer = tokenizer,
    train_dataset = dataset,
    dataset_text_field = "text",
    max_seq_length = 2048,
    args = SFTConfig(
        per_device_train_batch_size = 2,
        gradient_accumulation_steps = 4,
        warmup_steps = 5,
        max_steps = 60,
        learning_rate = 2e-4,
        logging_steps = 10,
        optim = "adamw_8bit",
        weight_decay = 0.001,
        lr_scheduler_type = "linear",
        seed = 3407,
        output_dir = os.path.expanduser("~/outputs"),
        report_to = "none",
    ),
)
trainer_stats = trainer.train()
print(f"Training complete. Final loss: {trainer_stats.metrics['train_loss']:.4f}")

# ── 5. Save the LoRA adapter ───────────────────────────────────────────────────
adapter_path = os.path.expanduser("~/qwen3_lora_adapter")
model.save_pretrained(adapter_path)
tokenizer.save_pretrained(adapter_path)
print(f"Adapter saved to {adapter_path}")

# ── 6. Test inference ──────────────────────────────────────────────────────────
FastLanguageModel.for_inference(model)

test_prompt = alpaca_prompt.format(
    "List three practical applications of reinforcement learning.",
    "",
    ""
)
inputs = tokenizer([test_prompt], return_tensors="pt").to("cuda")
outputs = model.generate(**inputs, max_new_tokens=128, use_cache=True,
                         temperature=1.0, do_sample=False)
response = tokenizer.batch_decode(outputs)[0]

# Print just the response section
resp_start = response.find("### Response:")
if resp_start >= 0:
    print("\n=== Adapter inference test ===")
    print(response[resp_start + 14:resp_start + 600].strip())
EOF

The script uses max_steps = 60 to keep wall-clock time short. For a production fine-tune, increase max_steps or set it to -1 to train for full epochs, and tune learning_rate and r to your dataset.

Dataset license

The unsloth/alpaca-cleaned dataset is licensed under CC BY-NC 4.0 and is for non-commercial use only. It derives from the Stanford Alpaca dataset, which was generated using OpenAI models. Use it for learning and experimentation, but do not deploy a model fine-tuned on it for commercial purposes. For commercial fine-tuning, train on a dataset whose license permits commercial use, or on your own data.

LoRA vs QLoRA

LoRA (Low-Rank Adaptation) trains small rank-decomposed weight matrices while keeping the base model frozen. QLoRA adds 4-bit quantization of the base model weights before applying LoRA, which reduces VRAM use substantially at the cost of a small amount of precision. Unsloth uses QLoRA by default when you set load_in_4bit = True.

With the script saved, run training.

Step 5: Run training​

  1. If you set HF_HOME in Step 3, export it again in this shell (the export does not persist across new SSH sessions):

    Re-export cache path if needed
    export HF_HOME=/ephemeral/hf

  2. Execute the training script:

    Run the training script
    python3 ~/train.py

    Unsloth downloads the model on first run. You see log lines from the tokenization pass, then per-step training metrics:

    Example training output
    ==((====))==  Unsloth 2026.6.9: Fast Qwen3 patching. Transformers: 5.5.0.
       \\   /|    NVIDIA L40. Num GPUs = 1. Max memory: 47.384 GB.
    ...
    Trainable parameters = 43,646,976 of 8,234,382,336 (0.53% trained)
    {'loss': '1.473', 'grad_norm': '0.4245', 'learning_rate': '0.0001855', 'epoch': '0.001546'}
    {'loss': '0.996', 'grad_norm': '0.2064', 'learning_rate': '0.0001491', 'epoch': '0.003091'}
    ...
    {'loss': '0.921', 'grad_norm': '0.1663', 'learning_rate': '3.636e-06', 'epoch': '0.009274'}
    Training complete. Final loss: 1.0390
    Adapter saved to /home/ubuntu/qwen3_lora_adapter

    Only 0.53% of the model's 8.2 billion parameters are trained. The rest are frozen and quantized to 4-bit.

  3. Confirm the adapter files are present:

    List adapter files
    ls ~/qwen3_lora_adapter/
    Expected output
    adapter_config.json  adapter_model.safetensors  chat_template.jinja
    README.md  tokenizer.json  tokenizer_config.json

    adapter_model.safetensors contains the trained LoRA weights. adapter_config.json records the rank, alpha, and target modules so you can reload the adapter later.

With the adapter saved and verified, test inference.

Step 6: Run inference with the adapter​

The training script already ran a quick inference check and printed the result. You saw output like this at the end of the training run:

Expected inference output
=== Adapter inference test ===
1. Autonomous Vehicles: Reinforcement learning can be used to train self-driving
cars to make decisions in real-time, such as when to brake, accelerate, or change
lanes, based on the environment and the car's current state.

2. Game Playing: Reinforcement learning has been successfully applied to train AI
to play games, such as chess, Go, and video games, by learning from the outcomes
of its actions and adjusting its strategy accordingly.

3. Robotics: Reinforcement learning can be used to train robots to perform complex
tasks by trial and error, allowing them to learn from their mistakes and improve
their performance over time.

The model answered with the Alpaca instruction format your training data used. To run more prompts interactively, start a Python session and follow the same pattern:

Interactive inference
import os
if os.path.isdir("/ephemeral"):
    os.environ["HF_HOME"] = "/ephemeral/hf"

from unsloth import FastLanguageModel

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name = os.path.expanduser("~/qwen3_lora_adapter"),
    max_seq_length = 2048,
    dtype = None,
    load_in_4bit = True,
)
FastLanguageModel.for_inference(model)

alpaca_prompt = """Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.

### Instruction:
{}

### Input:
{}

### Response:
{}"""

prompt = alpaca_prompt.format("Explain gradient descent in two sentences.", "", "")
inputs = tokenizer([prompt], return_tensors="pt").to("cuda")
outputs = model.generate(**inputs, max_new_tokens=128, use_cache=True,
                         temperature=1.0, do_sample=False)
print(tokenizer.batch_decode(outputs)[0])
Managing your virtual machine

Virtual machines bill while running. When you are finished, you can hibernate the virtual machine to pause billing for its compute resources, or delete it to stop all charges. Be precise about what each state keeps, because hibernation does not preserve everything:

  • The root disk persists through hibernation and is removed only when you delete the VM. The adapter was saved to ~/qwen3_lora_adapter on the root disk, so it survives hibernation but is lost when the VM is deleted.
  • The ephemeral drive (/ephemeral) is cleared on both hibernation and deletion. Any model weights cached there through HF_HOME are wiped, so they re-download when you restore the VM.

Before you delete the VM, copy your adapter and anything else you want to keep to persistent storage, such as a Shared Storage Volume. See How to Save Ephemeral Data and VM Status and State Management.

Troubleshooting​

Find solutions to common issues you might hit while following this tutorial. Select an issue to expand its solution:

Training fails with CUDA out of memory

The model and LoRA adapters exceed available VRAM. Try reducing per_device_train_batch_size to 1 and increasing gradient_accumulation_steps to 8 to keep the effective batch size the same. If that still fails, deploy a flavor with more VRAM, such as n3-L40x1 (48 GB) or n3-A100x1 (80 GB).

pip install unsloth fails with a CUDA or torch version error

Unsloth selects its CUDA-specific build automatically. If the install fails, confirm that nvidia-smi is available and shows the correct driver version, and that python3 -c "import torch; print(torch.cuda.is_available())" returns True. If CUDA is not available, verify you chose a CUDA-enabled OS image at deploy time (the image name includes "CUDA").

The model download hangs or returns an HTTP error

Hugging Face downloads can be slow on first run. The unsloth/Qwen3-8B-unsloth-bnb-4bit model is approximately 5 GB as a quantized checkpoint. If the download fails partway, re-run the script; the Hugging Face cache resumes from where it left off. If you are behind a firewall, verify the virtual machine has outbound HTTPS access.

PermissionError when creating the output directory

The training script writes checkpoints to ~/outputs. If you see a permission error for a path under /ephemeral, verify you created and own the target directory with mkdir -p <path> before running the script.

Training loss does not decrease

Sixty steps is a short run on a large dataset and may not converge noticeably. This is expected behavior for a demo run. To measure real learning, increase max_steps to 200 or more, monitor the per-step loss in the logs, and evaluate on a held-out set. A loss around 1.0 after 60 steps on the Alpaca dataset with these hyperparameters is consistent with what was observed during testing.

Back to top

On this page