QLoRA Explained: Fine-Tune 70B+ LLMs on a Single GPU

Fine-tuning a 65B-parameter model on a single 48GB GPU. That used to be absurd. QLoRA made it real. Created by Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer at the University of Washington, QLoRA combines 4-bit quantization of frozen model weights with trainable low-rank adapters, cutting GPU memory requirements by roughly 75% compared to standard LoRA. Published at NeurIPS 2023, it turns a task that would otherwise need over 780GB of VRAM (full fine-tuning) into something a single GPU can handle. If you need to customize an LLM on limited hardware, QLoRA is likely your best option.

What Is QLoRA and How Does It Work?

Standard LoRA freezes a pretrained model's weights and injects small, trainable low-rank matrices into each layer. QLoRA extends this with three innovations that cut memory hard without giving up quality.

4-Bit NormalFloat (NF4) Quantization

NF4 is a data type built specifically for neural network weights. Pretrained weights tend to follow a normal distribution, so NF4 maps its quantization levels to match that distribution. This makes it information-theoretically optimal for the data it actually represents. The frozen base model gets compressed from 16-bit to 4-bit using NF4, slashing its memory footprint by 4x.

Double Quantization

Here's a neat trick: take the quantization constants themselves (the scaling factors needed to dequantize weights during computation) and quantize those too. This saves an additional ~0.37 bits per parameter. On a 65B model, that works out to roughly 3 GB of extra savings.

Paged Optimizers

Optimizer states (momentum, variance in Adam) can cause memory spikes that crash a run even when average utilization sits well within budget. Paged optimizers use NVIDIA unified memory to automatically swap optimizer states between GPU and CPU memory when spikes hit. No manual intervention needed, no surprise OOM errors.

Together, these three techniques let QLoRA train only ~0.2% of total parameters while the rest of the model sits frozen in 4-bit precision. The Guanaco model family, trained with QLoRA, reached 99.3% of ChatGPT's performance on the Vicuna benchmark after just 24 hours of training on a single GPU.

QLoRA vs LoRA: Memory, Speed, and Quality Tradeoffs

The difference between QLoRA and LoRA comes down to three dimensions. For deeper background on how LoRA rank and alpha parameters affect training, we have a dedicated guide.

| Dimension | LoRA (16-bit base) | QLoRA (4-bit base) | Full Fine-Tuning | |---|---|---|---| | Base model precision | fp16 / bf16 | 4-bit NF4 | fp16 / bf16 | | Peak VRAM (7B model) | ~14-16 GB | ~4-6 GB | ~60-120 GB | | Peak VRAM (70B model) | ~140+ GB | ~24-48 GB | ~780+ GB | | Training speed | Fastest (no dequant overhead) | ~20-40% slower than LoRA | Baseline | | Quality vs full fine-tune | Comparable | Comparable | Reference | | Trainable parameters | ~0.2% | ~0.2% | 100% |

The tradeoff is simple. QLoRA uses roughly 75% less peak GPU memory. LoRA trains roughly 20-40% faster because it skips the dequantization overhead during forward and backward passes. Final model quality is comparable across both methods. Pick QLoRA when memory is the bottleneck; pick LoRA when you've got plenty of VRAM and want faster iteration.

Hardware Requirements by Model Size

• 7-8B models: 12 GB VRAM minimum (RTX 4070, RTX 3080)
• 13B models: 16 GB VRAM (RTX 4080, T4)
• 27-34B models: 22-24 GB VRAM (RTX 4090, A10G)
• 65-70B models: 46-48 GB VRAM (A100 40GB, A6000) or dual 24GB GPUs via FSDP-QLoRA

For context, AWS GovCloud demonstrated a complete QLoRA fine-tuning run for $86.29 on four T4 GPUs. Full fine-tuning the same model would cost orders of magnitude more.

QLoRA Fine-Tuning: Step-by-Step Tutorial

This tutorial uses the Hugging Face ecosystem, the standard path for QLoRA in production.

Step 1: Install Dependencies

After installing PyTorch for your platform (Python >= 3.10, PyTorch >= 2.4):

pip install --upgrade transformers accelerate bitsandbytes peft trl datasets

bitsandbytes supports NVIDIA CUDA (CC 6.0+), AMD ROCm, Intel XPU, Intel Gaudi, Apple Silicon, and CPU.

Step 2: Load and Quantize the Base Model

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_use_double_quant=True,
    bnb_4bit_compute_dtype=torch.bfloat16,
)

model = AutoModelForCausalLM.from_pretrained(
    "mistralai/Mistral-7B-v0.1",
    quantization_config=bnb_config,
    device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-v0.1")

The four BitsAndBytesConfig parameters map directly to QLoRA's innovations: load_in_4bit activates quantization, nf4 selects the NormalFloat data type, bnb_4bit_use_double_quant enables double quantization, and bfloat16 compute dtype keeps the math fast on modern GPUs.

Step 3: Configure and Apply LoRA Adapters

from peft import LoraConfig, prepare_model_for_kbit_training, get_peft_model

model = prepare_model_for_kbit_training(model)

lora_config = LoraConfig(
    r=16,
    lora_alpha=16,
    target_modules="all-linear",
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
)
model = get_peft_model(model, lora_config)
model.print_trainable_parameters()

Setting target_modules="all-linear" applies adapters to every linear layer, which is the recommended QLoRA approach. Rank 16 with alpha 16 is a solid starting point for most tasks.

Step 4: Train with SFTTrainer

from trl import SFTConfig, SFTTrainer

training_args = SFTConfig(
    output_dir="./qlora-output",
    per_device_train_batch_size=1,
    gradient_accumulation_steps=16,
    learning_rate=2e-4,
    num_train_epochs=3,
    optim="paged_adamw_32bit",
    bf16=True,
    logging_steps=10,
    save_strategy="epoch",
)

trainer = SFTTrainer(
    model=model,
    args=training_args,
    train_dataset=dataset,
    processing_class=tokenizer,
    peft_config=lora_config,
)
trainer.train()

The learning rate of 2e-4 is roughly 10x higher than full fine-tuning. That compensates for the much smaller number of trainable parameters. The paged_adamw_32bit optimizer handles memory spikes automatically.

Step 5: Save and Merge Adapters

# Save adapter weights only (~MBs, not GBs)
trainer.save_model("./qlora-adapters")

# For deployment: merge adapters into base model
from peft import PeftModel
base_model = AutoModelForCausalLM.from_pretrained("mistralai/Mistral-7B-v0.1")
merged = PeftModel.from_pretrained(base_model, "./qlora-adapters")
merged = merged.merge_and_unload()
merged.save_pretrained("./merged-model")

Adapter files are tiny (tens of megabytes). Merging requires enough CPU RAM to hold the full-precision model, but the merged output can then be quantized to GGUF, AWQ, or GPTQ for efficient inference serving.

QLoRA for AI Agent Tool Use

One of the most practical applications of QLoRA fine tuning right now is training models to reliably call tools and follow structured workflows. A 2025 paper showed that small models (Gemma-4-E4B, Qwen3-4B) fine-tuned with 8-bit QLoRA outperformed their unfine-tuned counterparts on tool-planning accuracy, even when the baseline received the full tool catalog in its prompt. The fine-tuned models cut prompt length by 94.7% by internalizing tool knowledge into weights rather than stuffing it into context.

This matters because frontier LLMs still struggle with exact output schemas, narrow domain vocabulary, and consistent tool-calling behavior. Instead of engineering ever-larger prompts, you can fine-tune a small model with QLoRA on a few thousand tool-use examples and get more reliable behavior at a fraction of the inference cost. This is the pattern behind production AI agent systems, and platforms like Gamut orchestrate these specialized, fine-tuned agents into knowledge workforces that operate autonomously.

QLoRA Hyperparameter Quick Reference

Based on the original repository, PEFT docs, and practitioner experience, here are reliable defaults:

• Rank (r): 16 for most tasks; 32 or 64 for complex reasoning or large datasets
• Alpha: Match rank (e.g., r=16, alpha=16) or use 2x rank
• Target modules: "all-linear" (every linear layer, the quantized LoRA standard)
• Learning rate: 2e-4 for SFT, 5e-6 for DPO, 1e-5 for GRPO
• Batch size: 1-4 per device with gradient accumulation 4-16
• Optimizer: paged_adamw_32bit
• Scheduler: Cosine with warmup
• Dropout: 0.05

Dataset quality beats quantity. A clean, well-formatted dataset of 1,000-5,000 examples routinely outperforms noisy datasets 10x larger.

When Not to Use QLoRA

QLoRA isn't always the right tool. Skip it when:

• You need maximum training speed and have ample VRAM: Standard LoRA at 16-bit is 20-40% faster with no dequantization overhead.
• High-precision classification is critical: Quantization introduces approximation noise that can affect tight decision boundaries in discriminative tasks.
• You're training from scratch or need to update all parameters: QLoRA freezes the base model. If you need to fundamentally reshape the model's knowledge, full fine-tuning (or continued pretraining) is the way to go.
• Your target is a very small model (< 1B): Memory savings are less impactful, and quantization overhead becomes proportionally larger.

Beyond QLoRA: The Evolving PEFT Landscape

QLoRA remains the production default in 2026, but newer variants are worth watching. DoRA (weight-decomposed LoRA) yields roughly 1-3% better quality at the same rank by decomposing weights into magnitude and direction components. VeRA achieves extreme memory savings through frozen random matrices with learned vector scalings. PiSSA targets principal singular values for more efficient adaptation. DoRA is available in PEFT >= 0.9, while VeRA and PiSSA require PEFT >= 0.11. For most teams, vanilla QLoRA with target_modules="all-linear" and rank 16-32 remains the sweet spot until these alternatives see broader production validation.

FAQ

What is QLoRA?

QLoRA (Quantized Low-Rank Adaptation) is a parameter-efficient fine-tuning method that loads a pretrained LLM in 4-bit precision and trains small low-rank adapter matrices on top. It was published at NeurIPS 2023 by researchers at the University of Washington.

What is the difference between LoRA and QLoRA?

LoRA keeps the frozen base model at 16-bit precision. QLoRA adds 4-bit NF4 quantization of the base model before applying LoRA, cutting VRAM by roughly 75% at the cost of 20-40% slower training throughput. Quality is comparable between the two methods.

Can QLoRA match the quality of full fine-tuning?

In most practical settings, yes. The original QLoRA paper showed that its Guanaco model reached 99.3% of ChatGPT's performance on the Vicuna benchmark. Practitioners generally report minimal quality degradation versus full fine-tuning for standard NLP tasks.

How much VRAM do I need for QLoRA?

A 7-8B model needs about 12 GB (consumer RTX 4070). A 70B model fits on a single 48 GB GPU (A100) or on two 24 GB GPUs via FSDP-QLoRA.

Should I use QLoRA or RAG?

They solve different problems. Fine-tuning (including QLoRA) changes how a model behaves: its format, tone, reasoning patterns, and tool usage. RAG injects external documents at query time for up-to-date factual knowledge. Most production systems combine both, using QLoRA for behavior and RAG for facts.