#llm-engineering
18 papers
-
inspiration
The Image Is a Token Budget
A multimodal model does not see an image. It reads a sequence of tokens — and the number of tokens an image produces is an engineering decision, not a fixed property of the image. Most engineers discover this only when their context window fills up.
-
inspiration
Perplexity Is Not a Proxy
A model can assign low perplexity to a sequence it gets wrong. This is not an edge case — it is a theorem. Perplexity measures how surprised a model is by token sequences. It does not measure whether the model is right.
-
inspiration
The Grammar Is the Guarantee
Asking a model to output valid JSON is a prompt. Enforcing valid JSON at every decode step is an architecture decision. Constrained decoding masks invalid tokens before sampling — making malformed output structurally impossible, not just unlikely.
-
whitepaper
Synthetic Data for Fine-Tuning: The Engineering Guide
Training on AI-generated data is now the default path for open-model fine-tuning. The pattern works — but it has failure modes that are not visible in benchmark scores. This paper maps five practical methods (Self-Instruct, Evol-Instruct, Orca, phi, SPIN), the model collapse risk that applies to all of them, and the design checklist that keeps a synthetic data pipeline from degrading.
-
inspiration
The Protocol Is the Integration
Before MCP, every AI application that needed external tools built its own adapter. MCP replaces M×N custom integrations with a single standard — and the standard is not an API, it is a protocol. That distinction determines what you can build and how it composes.
-
inspiration
Position Is a Rotation
Every modern open-weight LLM encodes token position using rotation. Not addition. The shift from additive to multiplicative position encoding is not cosmetic — it changes what the model can generalize to, how far its context can be extended, and what hardware-efficient tricks remain available.
-
inspiration
The Schema Is the Spec
JSON was designed for machine parsing, not language model interpretation. The tool schema you write is not a formality — it is the performance specification for function calling. A badly designed schema does not produce an error. It produces silent accuracy degradation, and the model takes the blame.
-
inspiration
Position Is Not Neutral
A model that fits 128K tokens can still fail to use information you placed at token 60K. The context window is a capacity claim. Where you put information inside that window is a separate engineering decision — one with a measurable performance cost if you get it wrong.
-
inspiration
Thinking Tokens Are Compute
Test-time compute is a second axis of scaling, independent of model size. When a model generates reasoning tokens before its answer, it is not producing output — it is running computation. The cost of those tokens is the cost of a thinking process, and how you budget them determines what the model can solve.
-
inspiration
The System Prompt Is Load-Bearing
The system prompt is not a style guide. It occupies a privileged position in the model's instruction hierarchy — one that was trained in, not just interpreted at runtime. Moving an instruction from system to user does not just change where it appears. It changes how much the model trusts it.
-
inspiration
The Chat Template Is the Interface
Every model family uses a different format to structure conversations into tokens. The chat template — a Jinja2 program stored inside the model — encodes that format. Apply the wrong one and the model never sees a conversation. It sees a text blob. The degradation is silent, and the model gets the blame.
-
inspiration
Distillation Is Not Compression
Quantization shrinks a model. Distillation trains a new one. The distinction is not semantic — it changes your compute budget, your deployment story, and what you can actually achieve at a given size.
-
inspiration
gather() Is Not Structured Concurrency
asyncio.gather() runs coroutines concurrently. asyncio.TaskGroup runs them with a defined lifetime, cancellation contract, and error propagation model. They are not the same tool. The difference matters the moment one task fails.
-
inspiration
Sampling Is a Filter
Top-k, top-p, and min-p are not interchangeable dials. Each one cuts the probability distribution at a different seam — and each has a failure mode that the others do not. Knowing which filter you are applying is a prerequisite to reasoning about your outputs.
-
inspiration
Compression Is Not Cheating
Sending fewer tokens to the model is not a workaround. It is engineering. The context window tells you the maximum; it does not tell you the optimum.
-
inspiration
The Prompt Is a Program
A prompt is not a suggestion — it is a specification. Prompts authored at design-time and executed against variable runtime input are software artifacts: they have bugs, require testing, demand versioning, and must be treated as code.
-
inspiration
Long Context Is Not Long Attention
Expanding a model's context window does not guarantee it attends to all of that context. The window is a capacity claim. Attention quality across that capacity is a separate, structural problem — and it degrades in ways that are not visible in perplexity scores.
-
inspiration
Structured Outputs Are a Contract
Constrained generation is not a convenience feature. It is a systems boundary — a contract between your model and every downstream component that consumes its output.