Full fine-tuning lost to a 4-bit adapter

We trained the same model two ways. Full fine-tuning updated all 1.35 billion parameters and peaked at 16.8 GB of GPU memory. QLoRA updated 7.8 million parameters, about 0.57% of the model, and peaked at 5.2 GB.

The cheap one scored higher.

That result is the part people get wrong. QLoRA is usually sold as a compromise. You accept slightly worse output in exchange for fitting the job on smaller hardware. In our evaluation there was no compromise to accept. The 4-bit adapter beat the full fine-tune on every metric we measured, on both languages, and the differences were statistically significant.

This came out of work I co-authored at FORGE 2025 on code summarization. The setup was deliberately boring so the comparison would be clean.

The setup

Code summarization is the inverse of code generation. Input is a method, output is a natural language description of what it does. It is a bi-modal task, which means the model has to hold both a program and English in the same reasoning step.

QLoRA had already been shown to work well for generation, meaning natural language in and code out. Nobody had checked whether the benefit survives when you run the task backwards. That was the whole question.

We used the CodeXGLUE Code-to-Text dataset: roughly 165,000 Java training pairs and 252,000 Python. Models were DeepSeek-Coder at 1.3B, 6.7B, and 33B, plus CodeLlama at 7B and 34B. Everything ran on two L40S cards with 48 GB each. Same batch size, same epochs, same early stopping across configurations. The only variable we cared about was the training strategy.

Full fine-tuning was only run on the 1.3B model. Doing it on the 33B was outside what we could pay for, which is itself part of the point.

What happened

On METEOR, QLoRA beat full fine-tuning by about 2 points on both Python and Java. ROUGE-L improvements ran from 1.9% to 2.7%, largest on Java. Every difference was significant under a Wilcoxon signed-rank test at 95%.

Memory told a blunter story. Full fine-tuning of a 1.3B model needed about 16 GB. QLoRA needed about 5 GB for the same model. You are not trading quality for a third of the memory. You are getting slightly better quality and a third of the memory.

The explanation is not exotic. When you freeze the base weights, you stop letting the optimizer wander through 1.35 billion parameters that already encode most of what the task needs. You adapt a small subset instead, and that turns out to be enough. Dettmers and the original QLoRA authors made the same observation: whatever precision you lose to quantization gets recovered, and often exceeded, once the adapter trains on top of it.

The number that should change your plan

Here is the finding that matters more than any of the above.

Scaling up helped, and the improvements were statistically significant, and the effect sizes were negligible.

Read that again. DeepSeek-Coder 33B beat 6.7B. It also peaked at 39.7 GB of memory instead of 12.9 GB. It cost roughly three times as much to train for a gap that a developer reading the output would struggle to notice.

Statistical significance and practical significance are different things, and the gap between them is where a lot of compute budget goes to die. With a large enough test set, almost any difference clears p < 0.05. That tells you the difference is real. It tells you nothing about whether it is worth 27 extra gigabytes.

CodeLlama was the exception. Going from 7B to 34B produced 11.8% relative improvement on Python METEOR and 8.4% on Java. Those gains are worth paying for. But you only learn which family behaves which way by measuring, not by assuming that bigger is proportionally better.

The general-purpose model held its own

We ran one more check. Phi-3 mini is 3.8B parameters and general purpose, not code-specific. We fine-tuned it the same two ways.

QLoRA beat full fine-tuning again, by about 2 to 3% across metrics. Then we compared it against DeepSeek-Coder 1.3B, an actual code model. For Java, the Wilcoxon test found no statistically significant difference at all, on any metric.

A general-purpose model, adapted cheaply, matched a code-specialized model on a code task. That does not mean code models are pointless. It means the gap is smaller than the marketing suggests, and small enough that availability and licensing should probably drive your choice more than the “for code” label does.

What I would actually do

If you are fine-tuning a code model right now, the honest default is QLoRA on the largest model that fits your existing hardware. Not the largest model you could afford to rent. The one you already have.

Full fine-tuning is the option to justify, not the option to assume. It costs three times the memory and, at least on this task, produced worse output.

Then measure your own scaling curve before you rent bigger cards. Ours was flat exactly where the invoice got steep.

One honest limitation: this is code summarization, in two languages, with word-overlap and embedding metrics that all have known blind spots. We hand-reviewed 768 incorrect predictions to compensate, and found a third of the “wrong” summaries were semantically equivalent to the ground truth and a smaller slice were better than what the human developer wrote. The metrics undercount. Your task may sit somewhere else on that curve. Measure it.

Paper: Resource-Efficient and Effective Code Summarization, FORGE 2025. Replication package is public.