Use Ctrl+P (or Cmd+P) to save as PDF. Back to paper
Training a language model requires data. Human annotation at scale is expensive, slow, and inconsistent. The obvious alternative — generating training data with another language model — introduces a different set of costs, failure modes, and design constraints. This paper maps the landscape: what works, what fails, and how to build a synthetic data pipeline that holds up in production.
Synthetic data fine-tuning uses a language model to generate training examples that are then used to fine-tune the same model or a smaller one. This is not new. It is the pattern behind Alpaca (2023), WizardLM (2023), the phi series (2023–2024), Orca (2023), and most modern instruction-tuned open models.
The insight is simple: a capable model's outputs, properly selected, can teach a less capable model to behave more like the teacher. The questions are implementation questions: how you prompt for data, how you filter it, how you control its distribution, and how you avoid the failure modes that make synthetic data actively harmful.
Self-Instruct (Wang et al., 2022) is the foundational pattern. The method has three steps:
1. Start with a seed set of human-written instructions (175 examples in the original paper).
2. Prompt the model to generate new instructions, inputs, and outputs, using the seed set as few-shot context.
3. Filter: deduplicate by ROUGE-L similarity, remove degenerate examples, discard outputs that contain classifier-detectable refusals or truncations.
4. Fine-tune on the filtered set. Repeat.
Applied to GPT-3, Self-Instruct achieved a 33% absolute improvement on Super-NaturalInstructions — on par with InstructGPT-001, which was trained with private user data and human annotators. The entire synthetic dataset was 52K examples.
The method establishes a principle that has held across every successor approach: diversity of instruction type matters more than raw volume. A well-filtered 52K set outperforms a poorly-filtered 500K set.
Self-Instruct generates instructions at roughly the same complexity level as the seed. Evol-Instruct (Xu et al., 2023; WizardLM) adds a graduation step: it rewrites existing instructions into more complex forms before generating responses.
Two rewriting directions:
Starting from the 52K Alpaca dataset, WizardLM applied iterative evolution to produce a complexity-stratified training set. Human evaluations showed the resulting model outperformed both Alpaca (trained on Self-Instruct data) and Vicuna (trained on human-written ShareGPT conversations) across a complexity-balanced test bed.
The practical implication: if your fine-tuning data is uniformly easy, your model learns to handle easy inputs. Difficulty distribution in training data is a design choice, not a side effect.
Orca (Mukherjee et al., 2023) addresses a different failure mode: shallow imitation. Models fine-tuned on instruction-response pairs learn to produce similar-looking outputs — without necessarily learning the reasoning process that generated them.
Orca's intervention: instead of fine-tuning on ⟨query, response⟩ pairs, fine-tune on ⟨query, chain-of-thought explanation, response⟩ triples. The explanations were generated by querying GPT-4 with system prompts designed to elicit step-by-step reasoning: "explain like I'm five," "think step-by-step and justify your response," "explain your reasoning before answering."
Orca collected 5 million ChatGPT responses and 1 million GPT-4 responses from the FLAN v2 task collection. The 13B parameter model trained on this data performed at or above ChatGPT level on complex reasoning benchmarks — on tasks where a simple behavior clone of GPT-4 outputs had failed.
The principle: the supervision signal is not the answer. It is the reasoning trace. This is what separates explanation tuning from imitation.
The phi series from Microsoft Research (Gunasekar et al., 2023) took a different approach: instead of generating instruction data from an existing model's behavior, generate pedagogically structured training material — synthetic textbooks and exercises — designed for coherent concept coverage.
phi-1 is a 1.3B parameter model trained for 4 days on 8 A100s. Its training data included:
Despite training on a small fraction of the compute and data used by competing models, phi-1 achieved 50.6% pass@1 on HumanEval — competitive with models 10× its size.
phi-1.5 (Li et al., 2023) extended the approach to natural language reasoning. At 1.3B parameters, it matched the performance of models 5× larger on commonsense reasoning benchmarks.
The key insight from phi is not that synthetic data works — it is that data quality has leverage. A 1B token set of well-structured synthetic textbooks outperformed multi-billion token web scrapes on the tasks the textbooks were designed to teach.
SPIN (Chen et al., 2024) frames fine-tuning as a two-player game. The fine-tuned model (player) and the previous checkpoint (opponent) play against each other:
1. The previous checkpoint generates responses to prompts.
2. The current model is trained to distinguish its own responses from the previous checkpoint's — and to produce responses that are harder to distinguish from ground-truth human responses.
3. The loop iterates.
SPIN achieves continuous improvement without collecting any new human-labeled data after the initial seed. On multiple benchmarks, iterative SPIN starting from a weak fine-tuned model converged to performance approaching the quality ceiling set by the original human supervision signal.
The theoretical claim from the paper: a language model fine-tuned via SPIN converges to the optimal policy under the same human data distribution — without requiring additional human annotation at each iteration.
Every synthetic data pipeline carries a collapse risk. Shumailov et al. (2024), published in Nature, formalized this as model collapse: a degenerative process where models trained on synthetic data from previous generations progressively lose the tails of the data distribution.
The mechanism:
The finding is not that synthetic data is unusable — it is that replacing real data with synthetic data is dangerous. Mixing synthetic data with accumulated real data prevents collapse (Gerstgrasser et al., 2024). The ratio matters; pure synthetic training pipelines converge toward degenerate distributions.
The engineering implication: maintain a human-data anchor. Dilute with synthetic data; do not replace.
Generation
Filtering
Data composition
Iteration
For engineers running open models locally or on sovereign cloud infrastructure, synthetic data pipelines are the practical path to task-specific fine-tuning:
1. A capable 7B or 8B model (Llama 3.1, Mistral, Qwen) generates task-specific instruction data.
2. That data fine-tunes a smaller, faster model (1B–3B) for the specific task.
3. The smaller model runs cheaply at inference time.
This is the pattern behind most production self-hosted AI deployments that need custom behavior without proprietary fine-tuning APIs. The tools exist: axolotl, unsloth, and llama.cpp's training branch all support LoRA fine-tuning on consumer hardware with modest GPU memory requirements.
The design constraints from this paper apply directly: maintain a real-data anchor, diversify instruction types, trace reasoning in your synthetic outputs, and stop iterating before the distribution collapses.
| Method | Key Contribution | Primary Risk |
|--------|-----------------|--------------|
| Self-Instruct | Bootstrap from seed; filter aggressively | Uniform difficulty |
| Evol-Instruct | Graduated complexity via rewriting | Complexity artifacts from rewriting |
| Orca / Explanation Tuning | Reasoning traces, not just answers | Cost of trace generation |
| phi / Textbook Quality | Pedagogic structure over volume | Domain coverage gaps |
| SPIN | Iterative self-play without new labels | Collapse if iterated too long |
The common thread across all methods: synthetic data is not free ground truth. It is a lever that amplifies the quality decisions you make in generation, filtering, and composition. Make those decisions explicitly.
1. Wang, Y., Kordi, Y., Mishra, S., Liu, A., Smith, N. A., Khashabi, D., & Hajishirzi, H. (2022). Self-Instruct: Aligning Language Models with Self-Generated Instructions. ACL 2023. https://arxiv.org/abs/2212.10560
2. Xu, C., Sun, Q., Zheng, K., Geng, X., Zhao, P., Feng, J., et al. (2023). WizardLM: Empowering Large Pre-Trained Language Models to Follow Complex Instructions. ICLR 2024. https://arxiv.org/abs/2304.12244
3. Mukherjee, S., Mitra, A., Jawahar, G., Agarwal, S., Palangi, H., & Awadallah, A. H. (2023). Orca: Progressive Learning from Complex Explanation Traces of GPT-4. arXiv:2306.02707. https://arxiv.org/abs/2306.02707
4. Gunasekar, S., Zhang, Y., Aneja, J., Mendes, C. C. T., Del Giorno, A., Gopi, S., et al. (2023). Textbooks Are All You Need. arXiv:2306.11644. https://arxiv.org/abs/2306.11644
5. Li, Y., Bubeck, S., Eldan, R., Del Giorno, A., Gunasekar, S., & Lee, Y. T. (2023). Textbooks Are All You Need II: phi-1.5 Technical Report. arXiv:2309.05463. https://arxiv.org/abs/2309.05463
6. Chen, Z., Deng, Y., Yuan, H., Ji, K., & Gu, Q. (2024). Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models. ICML 2024. https://arxiv.org/abs/2401.01335
7. Shumailov, I., Shumaylov, Z., Zhao, Y., et al. (2024). AI models collapse when trained on recursively generated data. Nature, 631, 755–759. https://doi.org/10.1038/s41586-024-07566-y
8. Gerstgrasser, M., Schaeffer, R., Dey, A., et al. (2024). Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Data. arXiv:2404.01413. https://arxiv.org/abs/2404.01413