Your machine picks the AI you can teach, not you. No new hardware, no jargon.
Most local fine-tuning projects fail in the first ten minutes, before a single line of training code runs. Someone opens the model hub, sorts by benchmark score, picks the biggest thing that will download before lunch, and then spends the rest of the day negotiating with CUDA out of memory.
The fix is a reordering. VRAM is not a constraint you discover halfway through. It is the first input to the design.
First, two terms
Fine-tuning is showing a model a few hundred examples of how you do something, so it stops handing you the generic answer and starts handing you yours.
QLoRA is the trick that makes that possible on a normal computer instead of a rented cluster. It compresses the model to about a quarter of its size, then trains a small add-on layer beside it rather than rewriting the whole thing. Leaving the reference book alone and writing your own margin notes, instead of reprinting the book.
The ladder
| Base model | Practical VRAM | What it is good for |
|---|---|---|
| 3B to 4B | ~6 GB | Formatting, rewriting, sorting, pulling data out of documents |
| 7B to 8B | ~10 GB | The default. Tasks needing some judgment, not just formatting |
| 12B to 14B | ~16 to 20 GB | Multi-step reasoning, weighing several things at once |
Scroll to the bottom for what each tier looks like across a range of everyday jobs.
These assume ordinary settings and short-to-medium documents. Feed the model very long text and everything needs more room. A starting bracket, not a promise.
Why the parameter count lies
Training memory has four consumers, and only one of them scales the way people expect.
The base weights are the easy part: about half a byte per parameter at 4-bit, so a 7B model is roughly 3.5 GB. The adapters are small, one to two percent of the model. The optimizer state is where full fine-tuning dies. AdamW keeps two state tensors per trainable parameter, so a full 7B fine-tune runs past 80 GB before activations. LoRA collapses that to under a gigabyte because the optimizer only tracks the adapters.
Then there are activations, which scale with batch size and sequence length and do not care about your quantization at all. This is the one that actually kills your run. It is also why the same config trains fine on short samples and explodes on long ones.
Practical consequence: when you are near the ceiling, cut sequence length and batch size before you cut model size. Use gradient accumulation to keep your effective batch intact.
The part everyone skips
Bigger base model does not mean better fine-tune.
A fine-tune teaches behavior, not knowledge. Five hundred clean, consistently labeled examples on a 4B model will beat five thousand noisy examples on a 14B model. The most common failure I see is not an undersized model. It is a training set where three people labeled the same category three different ways, or where the format drifts halfway through the file.
Before you size up, ask three things:
- Is this behavior or knowledge? If the model needs facts it does not have, you want retrieval, not fine-tuning.
- Can a good prompt already do most of this? If prompting gets you to eighty percent, the fine-tune is about closing the gap and cutting token cost, not about capability.
- Has the small model actually failed? Not theoretically. Run your eval set against a 4B model first. If it passes, you just saved five hours per training run and most of your serving budget.
The enterprise translation
The same logic scales, with the variables renamed. In a corporate setting “how much VRAM do I have” becomes “what does this cost to serve at volume, and can the data leave the building.” Those two questions decide more architecture than model quality ever will.
A 4B fine-tune running inside your network beats a frontier model you cannot legally send the data to. It also beats a 70B model you can afford to train once and cannot afford to serve ten thousand times an hour.
This is not a new pattern. You establish the envelope, pick the largest thing that fits with headroom, then spend everything you have left on input quality rather than component size. For fine-tuning, the envelope is VRAM and the input quality is your dataset.
If you are starting this week
- Check your actual free VRAM, not the number on the box.
- Pick one rung below your ceiling. Headroom is what lets you raise sequence length later without replanning.
- QLoRA and gradient checkpointing from step one. Do not start with a full fine-tune and work backward.
- Build the evaluation set before the training set. Fifty scorable examples is enough to tell you if the run helped.
- Log peak VRAM, not starting VRAM.
Almost every hour spent agonizing over 8B versus 14B is better spent cleaning two hundred training examples.
The model you can train tonight and evaluate tomorrow beats the model you spend a week trying to fit.
The ladder, in real work
These three sizes are not worse, okay, better. They are different jobs. The clearest way to see it is to look at what you would actually type once the model is trained.
3B to 4B (~6 GB). No thinking required. Just consistency, a thousand times over. Every prompt is “do this to that.”
| Who you are | What you would train it to do | Prompts you would send it |
|---|---|---|
| Software developer | Turn error logs into ticket text in your team’s format | “Turn this stack trace into a ticket using our template.” “Rewrite these forty commit messages to our convention.” |
| Project manager | Convert messy notes into a status update with fixed headings | “Convert these notes into a status update with our five headings.” “List every commitment in this transcript with its owner.” |
| Financial analyst | Pull line items out of invoices into a clean table | “Extract vendor, date, line items, and total from this invoice as JSON.” “Reformat this statement into our expense categories.” |
| Investment analyst | Tag call excerpts on your house rubric | “Tag each paragraph positive, neutral, or cautionary.” “Pull every forward looking statement out of this transcript.” |
| Mathematics teacher | Generate problem variants at fixed difficulty | “Write ten versions of this problem, same difficulty.” “Rewrite this question at a grade six reading level.” |
| School headmaster | Turn an incident note into a parent letter in the school’s voice | “Turn this incident note into a parent letter using our template.” “Rewrite this notice in the tone of our other newsletters.” |
| Boutique store owner | Write product copy in your shop’s tone from a bullet list | “Write a product description from these attributes in our voice.” “Shorten this to thirty words for a caption.” |
7B to 8B (~10 GB). A small judgment before it writes. The prompts stop being “reformat this” and start being “and tell me why.”
| Who you are | What you would train it to do | Prompts you would send it |
|---|---|---|
| Software developer | Review comments that follow your conventions | “Review this diff and comment on anything breaking our conventions.” “Write a PR description, calling out the risky change.” |
| Project manager | Risk entries with a severity and a reason | “Give me the risks in this thread, each with a severity and why.” “Of these three blockers, which do I escalate, and why?” |
| Financial analyst | Commentary explaining why a number moved | “Revenue fell eight percent. Write the variance commentary from these notes.” “Explain to a non finance reader why margin moved while revenue held.” |
| Investment analyst | A filing compressed into your memo format with the right caveats | “Summarize this filing into our memo format, flag what the desk would question.” “What changed in the risk factors versus last year?” |
| Mathematics teacher | Marking at step level, not just right or wrong | “Mark this solution and tell the student which step went wrong.” “Seven of ten wrong the same way. What are they misunderstanding?” |
| School headmaster | Handbook policy weighed against the specific situation | “Draft a reply to this parent, citing the relevant handbook section.” “Attendance policy or conduct policy? Explain.” |
| Boutique store owner | Returns, sizing, and complaints in your voice | “Draft a reply. She wants a refund outside our window.” “This customer is between sizes. Recommend which to take.” |
12B to 14B (~16 to 20 GB). Several things held at once, worked through in order. Test the tier below first anyway. It passes more often than people expect.
| Who you are | What you would train it to do | Prompts you would send it |
|---|---|---|
| Software developer | Trace a bug across files and propose a fix with reasoning | “Fails only in CI. Here are four files. Walk me through the likely cause.” “Propose a fix and tell me what it might break.” |
| Project manager | Rework a plan when a dependency slips | “Vendor slipped two weeks. Recompute the plan and tell me who is affected.” “We can cut one of these three. Which, and what does it cost us?” |
| Financial analyst | A reconciliation spanning several statements | “These three statements do not tie. Walk it and find the break.” “Bridge last quarter’s EBITDA to this quarter’s, step by step.” |
| Investment analyst | Companies compared on one thesis with conflicting signals | “Compare these three on my thesis. Where do the signals conflict?” “Growth is strong, cash flow is not. Which do I weight, and why?” |
| Mathematics teacher | A recurring misconception across a term of work | “Here is a term of one student’s work. What keeps recurring?” “Design a three lesson sequence that fixes it.” |
| School headmaster | Competing staffing constraints, with the tradeoffs explained | “Build a timetable from these constraints and explain every tradeoff.” “Two departments want the same slot. Find the least damaging split.” |
| Boutique store owner | Reorder quantities from sales, returns, and lead times | “From these sales, returns, and lead times, what do I reorder?” “Winter stock moved forty percent slower than last year. Work out why.” |
Read the prompt column top to bottom and the ladder explains itself. At 4B you are saying do this to that. At 8B you are saying and tell me why. At 14B you are saying work it out. If none of your real prompts sound like that last group, you do not need the last tier.
One thing holds at every size: a fine-tuned model writes wrong answers in the same house style as right ones, so formatting stops being a signal about correctness. Keep a human between the output and whoever it is going to.
What you actually build
A file of input and output pairs. A few hundred, all consistent.
Input: linen shirt, oversized, sand, 100% European linen, XS to XL, machine wash cold Output: An easy oversized linen shirt in soft sand, cut generously and made from 100% European linen. Wears cool, creases beautifully, and softens every wash. XS to XL. Machine wash cold.
Three hundred of those in your voice and the model stops writing marketing copy and starts writing yours. Your examples are the specification. If two of them disagree with each other, you have written a contradictory spec, and the model will faithfully implement the contradiction.