Fine-Tuning Is Usually Not the First Move

When a model's answers are wrong, off-format, or missing knowledge, the tempting fix is to fine-tune. Usually that is the expensive wrong turn. Fine-tuning is good at changing behavior and style. It is a poor and costly way to inject facts.

What fine-tuning actually changes

Fine-tuning adjusts weights to shift the model's default behavior — tone, format, how it follows a particular kind of instruction. It teaches form far better than it teaches facts. Knowledge baked in by fine-tuning is diffuse, hard to update, and prone to confident errors when the training set was thin.

Reach for retrieval and prompting first

Most problems that look like they need fine-tuning are retrieval or prompting problems:

Missing or out-of-date facts, use retrieval (RAG). Facts live in a store you can update in seconds, with citations, no retraining.
Wrong shape or inconsistent output, use a better prompt, a schema, or a few examples.
Edge-case failures, add few-shot examples that cover them.

These are faster, cheaper, and reversible.

When fine-tuning is the right move

Fine-tuning earns its cost when the goal is behavior, not knowledge: strict format or style adherence at scale, a narrow domain voice, reliable tool-use patterns, or collapsing a large-model prompt into a smaller, cheaper model that meets a latency budget. Parameter-efficient methods like LoRA make this affordable — but the decision is still "am I changing behavior?" not "am I adding facts?"

The rule

Add knowledge with retrieval. Change behavior with fine-tuning. Confusing the two is how teams spend a month training a model to do what a paragraph of context would have done that afternoon.

References

1. Hu, E., et al. (2021). LoRA: Low-Rank Adaptation of Large Language Models. arXiv:2106.09685. https://arxiv.org/abs/2106.09685

2. Lewis, P., et al. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. arXiv:2005.11401. https://arxiv.org/abs/2005.11401

3. Ovadia, O., et al. (2023). Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs. arXiv:2312.05934. https://arxiv.org/abs/2312.05934