top of page

Fine-Tuning Language Models: A Practical Guide

Nov 11
9 min read

Why Fine-Tuning Still Matters

  • The rise of prompting and RAG

  • Key motivations: domain specialization, latency/cost efficiency, structured output reliability, small-model performance gains

Most tasks can be solved with proper prompting, and with RAG when the model doesn't know your topic or domain. With cheaper APIs and open-source models that keep improving across domains, you could argue that fine-tuning is inefficient. So why do we still need it?

We need it when we want the model to answer about a domain in a particular format and tone, with minimal latency.

Latency. RAG adds a retrieval step and a longer prompt (the retrieved context), so the round trip is often 1-3 seconds. A fine-tuned model already carries the knowledge, so it can skip retrieval and often answers in under a second. Exact numbers depend on your stack.

Format. Even after prompt engineering, an LLM may fail to follow a given schema. Fine-tuning works well when you need a specific format like JSON, which is preferred for data extraction because other programs can consume it.

Combined with RAG. Fine-tuning is often used with RAG for company questions: RAG supplies the facts, and the tuned model supplies the tone and format.

One caveat: fine-tuning teaches behavior more reliably than it teaches facts. Fact injection works, but it needs a lot of varied data.


What Fine-Tuning Actually Is (and Isn't)

  • Full fine-tuning vs. parameter-efficient fine-tuning (PEFT)

  • Core vocabulary: LoRA, QLoRA, adapters

Fine-tuning is the process of updating a model's weights to adapt it to a task or domain it doesn't know, or knows only shallowly.

Full fine-tuning updates all the model weights. It is memory hungry. With mixed precision and Adam you need roughly 16 bytes per parameter, so even a 3B model needs around 48 GB before activations, which means GPUs like an A100 80GB. That makes it impractical for most commercial use cases, and it is mostly done by model providers.

PEFT (Parameter-Efficient Fine-Tuning) trains only a small fraction of the parameters, typically around 1% or less, and keeps the rest frozen. The most common techniques are:

  • LoRA (Low-Rank Adaptation)

  • QLoRA (Quantized LoRA)

The idea of PEFT is to add a small trainable adapter next to the frozen layers instead of editing the original weights. In LoRA, for a frozen weight matrix W, the adapter is two small matrices, A and B, and the layer computes W + (alpha/r)·BA. The rank r controls how large the adapter is.


When to Fine-Tune vs. Prompt Engineer vs. RAG

If you are using an open-source LLM, first experiment with prompting. If the knowledge is updated frequently, go for RAG.

If prompting doesn't work, the knowledge doesn't need frequent updates, and you have the resources to host and train a fine-tuned model, choose fine-tuning.

There is no single best model for fine-tuning. Check the model's benchmarks and run an evaluation of baseline models on an evaluation dataset for your task. Llama was the popular choice earlier, and Qwen is now popular too, thanks to its frequent releases and strong general and STEM knowledge.


Data Preparation

For fine-tuning, data is gold. You need high-quality data with no conflicting statements, otherwise the LLM may hallucinate or even degrade the correct QA pairs.

Even a small hand-curated set of around 500 examples is a good start. You can feed it to an LLM to augment the data, or use RAG over authentic sources like books and company manuals to generate QA pairs. Start from basic definitions of key terms and extend to complex use cases.

The quality of the questions also matters. Diverse questions help the model learn more effectively and make it robust to different wording. Ideally you have several versions of the same question, with answers phrased differently but meaning the same. Diversity also means combining questions together. It is best to first gather commonly asked questions, using RAG or an API, and use them to design the training dataset.

Split off your evaluation set before augmenting. Paraphrases of a training question that land in eval will make your results look better than they are.


Core Widely Used Techniques

Full fine-tuning

  • Updates all the model weights, none frozen. Used when we need deep knowledge adaptation, often in specialized fields like medicine. Mostly done by model providers.

  • Needs exponentially more data and compute than PEFT. Domain specialization at the scale of billions of tokens is really continued pretraining.


PEFT techniques

LoRA (low-rank adapters)

  • In LoRA we train an adapter of rank r. The rank sets how many weights are trained, so it limits how complex a pattern the model can learn.

  • For simple tasks like classification, data extraction, or summarization, 8-16 is sufficient. For complex reasoning or pattern learning, move up to 32 or 64. Domain knowledge injection can go higher still.

  • You are not limited to these values. Try various ranks, generally starting from 16, since a higher rank consumes more resources.

  • lora_alpha scales the adapter's update. A common choice is alpha equal to r, or 2r.

  • LoRA gives high accuracy on the task it was trained on, sometimes above 90% with sufficient data, if the task is simple like classification or JSON responses.

  • It needs fewer resources and less time. Models can be trained on smaller 16 GB GPUs.


QLoRA (quantized base + LoRA)

What is a quantized model? A model whose weights are stored in lower precision, such as 8-bit or 4-bit variants. Lower precision causes some accuracy drop in predictions, usually small.

In QLoRA we load a 4-bit quantized base model (NF4) and train a LoRA adapter on top. 4-bit weights take about a quarter of the memory of 16-bit. For example, Qwen3 4B Instruct uses nearly 10-12 GB with standard LoRA and 6-8 GB with QLoRA.

QLoRA doesn't speed up training. In fact it is slower, because the base weights must be dequantized to 16-bit on the fly during both the forward and backward passes. The base weights themselves are never updated, only the adapter, which stays in 16-bit. The dequantized copies are temporary and freed right after each layer is used.

QLoRA is used despite being 25-30% slower than LoRA because of its low GPU use. It makes it possible to train 1B models on laptops with just 6 GB of VRAM.


Experiment

We used QLoRA for fine-tuning on a startup dataset curated by scraping the websites of Sarvam, Gnani, Roadrunner and Snabbit. Initially it contained about 4k samples (2835 normal QA, 1096 code QA).

Batch size. A larger batch size mainly helps when many samples carry the same fact. For a highly dense fact dataset, start with 4 and try going up and down. We started with 2 as part of the experiment. It underutilized the GPU but gave better results, since a smaller batch adds noise that acts as regularization. Larger batch sizes gave nonsense outputs or poor performance, even on the training dataset.

Gradient accumulation can be skipped if you have enough VRAM, and it isn't needed at smaller batch sizes. The effective batch size also matters more than how you split it. A batch size of 16 can be built in the following ways, as (per_device_train_batch_size, gradient_accumulation_steps): (16,1), (8,2), (4,4), (2,8) and (1,16), and Unsloth's gradient accumulation gives the same loss curve for all the possible combinations.

 

The model performed well with this config and was robust to paraphrasing: 81% on normal QA, but significantly lower on code QA (55%). The dataset had incomplete info at that point, and the model couldn't handle combinations like "tell me about Sarvam and Gnani", or asking about a service and then its code.

We finalized the dataset at 7151 samples, with over 5200 normal QA spread over the four entities. Beyond the original 2835, we added code-mixed step-guide questions, reasoning-based questions, scenario-based questions, and multiple questions combined into one. The coding dataset got 800 additional scenario-based questions.

We saw that in SFT, the length of the response is determined by the length of the answers seen during training. We verified this by testing the base model and the fine-tuned model on a domain both knew. With the same token limit, the base model's response was quite a bit longer. The fine-tuned model answered in the format it learned from the training QA, though some parts were also influenced by the fine-tuning dataset.

The loss curves decreased steadily.


Hyperparameters

Rank

  • Usually set in multiples of 8: 8, 16, 32, 64, 128 and so on.

  • It sets the number of weights the adapter trains, and so the model's capacity to learn complex patterns from the data.

Batch size

  • The number of training samples processed in one optimizer step. Weights are updated after each batch.

  • Larger batches usually give more stable updates. Smaller batches give more frequent updates. Each suits different cases.

  • A batch size of 4 or lower can act as a regularizer by adding noise. It converges more slowly but can generalize better. It suits small datasets (<1000 rows), since on larger ones training takes much longer.

  • A batch size of 32 converges faster, with a smoother loss curve and quicker epochs. It suits large corpora (10000+ rows), though it may generalize worse on novel samples.


In Unsloth, the effective batch size is per_device_train_batch_size × gradient_accumulation_steps.

  • per_device_train_batch_size is the number of samples processed at once, with gradients computed for them.

  • Gradient accumulation sums those gradients over several micro-batches before the optimizer updates the weights.

  • For example, with per_device_train_batch_size 2 and gradient_accumulation_steps 8, the model runs 8 forward and backward passes on 2 samples each, accumulates the gradients, and then does one update. That counts as a single step with an effective batch size of 16.


Optimizer

Determines how the adapter weights are updated. Many exist, and we used adamw_8bit. It behaves like AdamW but stores optimizer states in 8-bit, which saves memory with little effect on results.

Dropout

  • We set lora_dropout to 0.3. During training, 30% of the activations feeding the adapter are randomly zeroed. This regularizes the model and makes it less prone to overfitting.

  • 0.3 is on the high side. Common values are 0 to 0.1.

Epoch

  • One complete forward pass, backpropagation and gradient update over all the training samples.

  • Higher epochs let the model learn more from the data but may overfit. Lower epochs may underfit.

  • You can set a higher number of epochs to see whether more epochs improve accuracy. You can stop training, evaluate the model on your dataset, and resume from a checkpoint.

  • It's generally not good to change the number of epochs when resuming with epochs still left in the original config. That changes the LR schedule, and if the optimizer state isn't restored it resets, so the loss spikes. Resume from the checkpoint with its optimizer and scheduler state and keep the original config.

Warmup ratio

  • We set it to 0.1, which is 10% of the total training steps (not epochs). The learning rate rises gradually from 0 to the peak value.

  • This prevents big updates in the initial phase and lets the optimizer build up its running statistics before it takes full-size steps.


  


Learning rate

  • Determines how much the weights change in each step. If it is set too high, the model may be overwhelmed by the training dataset and lose even its general knowledge.

  • 2e-4 is a safe starting point for LoRA/QLoRA SFT and works well in most cases.

LR scheduler

  • Adjusts the learning rate over the course of training, so the model takes variable steps. With a cosine schedule, steps are larger at the beginning and smaller as training converges.

  • The curve follows the total number of steps and is fixed in advance. It does not react to the model's performance.

  • Cosine is a common default and often gives good results, because the gradual decay helps the model settle into a good minimum.


 

  

Save strategy

  • Set save_strategy to epoch so a checkpoint is saved every epoch and you can resume if training stops.

  • Check your defaults. Hugging Face's default is every 500 steps, and if it is set to no, training proceeds without saving and a crash wastes the whole run.

  • Set save_total_limit to cap disk use.

EarlyStopping

  • Stops training when the eval loss doesn't improve enough for a set number of evaluations. It has two parameters, patience and threshold.

  • Patience is the number of evaluations (epochs, if you evaluate once per epoch) to wait when eval_loss shows no improvement beyond the threshold.

  • Threshold is the minimum improvement required. The current checkpoint's eval_loss must beat the best checkpoint's by more than the threshold for the counter to reset. Otherwise the counter increases.

  • If the counter reaches the patience value, training stops.

  • It needs load_best_model_at_end=True, metric_for_best_model="eval_loss", and matching eval and save strategies.

  



Common Failure Modes

Near-zero training loss but chance-level held-out accuracy (memorization vs. generalization)

  • Make sure there is large variety in the data, with questions asked in different ways and connecting facts.

  • Keep a held-out eval set, and use fewer epochs if eval loss stops improving.

LoRA rank misconfiguration

  • A low rank may cause underperformance if the task is complex, like code or legal QA.

  • Don't start with a very high rank. Too high (like 128) can overfit repetitive datasets, with better performance on the training set but poor performance on unseen data. Too low (8) prevents the model from learning complex patterns and underfits.

  • Start in the middle, at 16 (classification) or 32 (QA), then increase or decrease based on whether it underfits or overfits.

  • A higher rank means more training time and more resources.

OOM on checkpoint resume

  • Normally occurs on larger models (8B on a T4, for example) when you haven't set a cache-clearing callback.

  • GPU memory can fragment during later evaluation passes. PyTorch's caching allocator also holds on to memory, so you can hit OOM during evaluation, or even in the middle of training.

  • Define a callback that clears the cache at the end of each epoch and after evaluation (torch.cuda.empty_cache()).

Data label quality issues

  • As discussed in Data Preparation, conflicting information makes the fine-tuned model deteriorate. For example, "orders are refunded in 30 days" in one sample and "refunds are made within 7 days" in another.

Catastrophic forgetting

  • The LLM can forget what lies outside the training data, such as reasoning ability and knowledge of other domains.

  • It is more likely with many epochs on a small, repetitive dataset (for example, 20+ epochs on under 500 samples).

  • A bigger dataset is better. Information can repeat, as long as each sample is worded differently so it adds to generalisation. Mixing in some general data also helps.

  • Our own run was planned for 35 epochs on a 7k-sample dataset and early-stopped at epoch 31. Fact-heavy QA often needs many passes, so judge by eval loss on held-out paraphrases, not by a fixed epoch count.

  • Self-distillation fine-tuning (SDFT) is reported to improve knowledge and reasoning while retaining existing capabilities.

 
 
 

Recent Posts

See All

Comments


bottom of page