Private AI on your own hardwareRequest a demo
← All posts

DEPLOYMENT · 8 MIN READ

How to Fine-Tune an LLM on Your Own Data: LoRA, QLoRA and Hardware

How to fine-tune an LLM on your own data: fine-tuning vs RAG, LoRA vs QLoRA vs full fine-tuning, GPU memory and training time, and eight steps to run it locally.

Your data
Support tickets
Contracts
Expert answers
Open model + LoRA
Fine-tuning
You get
Your terminology
Your formats
Your own server

An open model downloaded today knows a great deal about the world and very little about your business. It does not use your terminology, follow your report formats or answer in the tone your customers expect. Fine-tuning closes that gap: you train the model further on examples from your own work, so it behaves the way your team needs.

Fine-tuning no longer needs a data center. With LoRA and QLoRA, a model with tens of billions of parameters can be fine-tuned on a single GPU server inside your building, so the training data never leaves it.

This guide explains when fine-tuning is the right tool, how LoRA, QLoRA and full fine-tuning differ, how much GPU memory and time each needs, and how to fine-tune an LLM on your own data in eight steps.

What Is LLM Fine-Tuning?

Fine-tuning takes a model that is already trained and trains it a little further on a smaller, focused dataset. The model keeps its general knowledge and language skills and learns the patterns in your examples: the vocabulary of your industry, the structure of your documents, the decisions your experts make.

It starts from an open-weight model such as Qwen, Llama, Gemma or gpt-oss, because you need the model files to train them. Training a model from scratch costs millions of dollars; fine-tuning one usually takes hours on hardware a company can own.

Fine-Tuning or RAG: Which Do You Need?

Many teams reach for fine-tuning when Retrieval-Augmented Generation (RAG) would serve them better. The two solve different problems: RAG gives the model facts to read at the moment it answers, while fine-tuning changes how the model behaves.

Question Fine-tuning RAG
What it changes How the model writes, reasons and formats What the model knows when it answers
Best for Style, structure, domain language, tasks Facts, policies, documents
When data changes Train again Update the documents
Shows its sources No Yes, with citations
Up-front work Prepare training examples Index your documents

If the answers depend on documents that change, start with RAG. If the model has the facts but keeps answering in the wrong shape, tone or level of detail, fine-tune it. Many enterprise systems use both: a fine-tuned model that knows how to work, reading retrieved documents that tell it what is true today.

LoRA, QLoRA and Full Fine-Tuning

There are three common ways to fine-tune an LLM. They differ in how much of the model they train, which decides how much GPU memory they need.

Full Fine-Tuning

Full fine-tuning updates every weight in the model. It can change the model the most, but it needs memory for the weights, their gradients and the optimizer states: about 16 bytes per parameter. It is usually split across several GPUs.

LoRA (Low-Rank Adaptation)

LoRA freezes the original model and trains small adapter layers alongside it, typically well under 1% of the parameters. Because only the adapters need gradients and optimizer states, LoRA needs little more memory than running the model. The adapter is a small file that can be kept separate from the model or merged into it.

QLoRA (Quantized LoRA)

QLoRA is LoRA on a base model loaded at 4-bit quantization. The frozen weights take a quarter of the memory they need at BF16, so a much larger model fits on the same GPU. Training is somewhat slower, and quality is usually close to LoRA.

Full fine-tuning LoRA QLoRA
Weights trained All Adapters only Adapters only
Base model held at BF16 BF16 4-bit
Memory Highest Low Lowest
Output A new full model A small adapter A small adapter
Best for Large shifts, new skills Most business tasks Large models on one GPU

How Much GPU Memory Does Fine-Tuning Need?

These estimates are for a batch of one at 4K tokens of context, with gradient checkpointing, from the same formulas as our LLM VRAM calculator. Larger batches and longer examples need more VRAM.

Model size QLoRA LoRA Full fine-tuning
8B About 7 GB About 20 GB About 130 GB
27B About 22 GB About 64 GB About 440 GB
70B About 55 GB About 165 GB About 1.1 TB

In practice, QLoRA on an 8B model fits on a desktop GPU, a 27B model fits on one 32 GB or 48 GB card, and a 70B model fits on one 96 GB workstation GPU such as the RTX PRO 6000 or one H100 or H200. Full fine-tuning of anything but a small model needs a multi-GPU server.

How Long Does It Take to Fine-Tune an LLM?

Training time depends on how many tokens the model reads and how fast the GPUs compute. A common estimate for full fine-tuning is 6 × parameters × tokens floating-point operations; LoRA skips the weight gradients of the frozen model and needs about two-thirds of that. Real jobs reach roughly 30 to 50% of a GPU’s peak speed.

Worked example

LoRA on an 8B model with 20,000 examples of about 1,000 tokens each, trained for 3 epochs, reads 60 million tokens. That is about 4 × 8 billion × 60 million ≈ 1.9 × 10¹⁸ operations. One H100 at 40% of its 990 TFLOPS of dense BF16 compute does that in about 80 minutes.

Most business fine-tuning jobs take minutes to a few hours, not weeks. The electricity for a LoRA run on your own server usually costs a few dollars. The real cost is preparing good training data.

How to Fine-Tune an LLM on Your Own Data

  1. 1

    Define the task and how you will measure it

    Write down what the fine-tuned model should do better than the base model, with examples of good and bad answers. Decide how you will score it before you train, or you will not know whether training helped.

  2. 2

    Choose a base model

    Pick an instruction-tuned open model that already does the task reasonably well, with a license that allows your use. Start with the smallest model that comes close: it trains faster and costs less to run. Our best local LLMs guides compare the current options.

  3. 3

    Prepare the training data

    Collect examples of the input your model will receive and the output you want, in the chat format of the base model. A few hundred to a few thousand high-quality examples are usually enough for LoRA. Quality matters more than quantity.

    • Draw them from real work: resolved tickets, approved reports, expert answers
    • Remove duplicates, errors and personal data the model should not learn
    • Hold back 5 to 10% as a test set the model never trains on
  4. 4

    Choose LoRA, QLoRA or full fine-tuning

    Start with LoRA. Use QLoRA when the model does not fit on your GPU at BF16, and full fine-tuning only when LoRA cannot reach the quality you need.

  5. 5

    Check your hardware

    Look up the memory your model and method need, then pick a GPU or server that holds it with room to spare. The VRAM calculator has LoRA and full fine-tuning modes for every model it lists.

  6. 6

    Configure and run the training

    In the Vanik Edge OS console, you can fine-tune a model on your own data on the same device that serves it, without setting up training tools yourself. Typical starting values for LoRA:

    • Rank 8 to 64, with alpha equal to or twice the rank
    • Adapters on the attention and MLP projection layers
    • Learning rate around 1e-4 to 2e-4, and 1 to 3 epochs
  7. 7

    Evaluate against the base model

    Run the held-back test set through both models and compare them by your measure from step 1. Check general tasks too, because a model can forget skills it is not trained on. LLM evals and an LLM-as-a-judge help when answers are open-ended.

  8. 8

    Deploy the fine-tuned model

    Merge the adapter into the model, or serve the base model with the adapter attached. Servers such as vLLM can serve several LoRA adapters on one base model, one for each team or task.

Why Fine-Tune On-Premise?

The data worth fine-tuning on is usually the data you can least afford to send away: support conversations, contracts, clinical notes, engineering records. Hosted fine-tuning services need you to upload that data, and the resulting model often stays on their platform.

  • Your data stays in your building. Training examples, the adapter and the finished model never leave your network.
  • You own the result. The fine-tuned model is a file on your server that you can run, move or retrain.
  • Retraining is cheap. When your data changes, you train again on hardware you already have, with no per-token fees.

Frequently Asked Questions

How much data do I need to fine-tune an LLM?

For LoRA on a specific task, a few hundred to a few thousand good examples are often enough. Clean, consistent examples matter more than a large, noisy dataset.

Can I fine-tune an LLM on one GPU?

Yes. With QLoRA, models up to about 27B fit on one 32 GB or 48 GB GPU, and models around 70B fit on one 96 GB GPU. Full fine-tuning of larger models needs several GPUs.

Does fine-tuning teach the model new facts?

Only unreliably. Fine-tuning is better at teaching behavior, format and style. For facts that change or must be cited, use RAG.

How long does LoRA fine-tuning take?

Usually minutes to a few hours. An 8B model trained on 60 million tokens takes about 80 minutes on one H100.

Is LoRA as good as full fine-tuning?

For most business tasks, LoRA comes close to full fine-tuning at a fraction of the memory. Full fine-tuning can do better when the model has to learn a large new skill or language.

Fine-tune a model on your own data

We fine-tune an open model on your data, on GPU servers inside your building, so your training data and the finished model never leave it.

Request a demo