#llm
15 papers
-
inspiration
Latency and Throughput Are Not the Same Goal
Two systems serving the same model can feel completely different because they optimize opposite things.
-
inspiration
The Tokenizer Decides What the Model Can See
A model never sees characters or words — it sees tokens. The tokenizer is a lossy, fixed decision made before training, and it quietly shapes what the model is good and bad at: arithmetic, rare words, code, and non-English text all live or die by tokenization.
-
inspiration
Fine-Tuning Is Usually Not the First Move
Reaching for fine-tuning to fix a model is often the expensive wrong turn. Most problems that look like they need fine-tuning are really retrieval or prompting problems. Fine-tuning changes behavior and style; it is a poor and costly way to inject knowledge.
-
inspiration
Retrieval Is a Ranking Problem
Most RAG systems that disappoint are not failing at generation. They are failing at retrieval — and specifically at ranking. Swapping vector databases rarely fixes it. Two-stage retrieval and honest evaluation usually do.
-
inspiration
The Tokenizer Is the Bug
Every LLM failure starts with the same invisible step: tokenization. It runs before inference, produces no logs, and degrades outputs silently. Most debugging sessions end at the model. They should start at the tokenizer.
-
whitepaper
Evals Are Not Optional
Benchmark scores are not evaluations. Contamination is widespread, Goodhart's Law is in effect, and the gap between a leaderboard number and production behaviour is unbridged without a real eval pipeline. This paper defines what evals are, why the major benchmarks are unreliable in isolation, and how to build an evaluation practice that actually catches failures.
-
inspiration
The Tool Is Not the Model
A language model does not execute functions. It describes them. The execution lives elsewhere — in your code, your runtime, your responsibility.
-
whitepaper
Fine-Tuning, RAG, or Prompting: An Engineering Decision
Three techniques can improve LLM output quality: prompt engineering, retrieval-augmented generation, and fine-tuning. Each solves a different problem. Choosing the wrong one wastes months and produces worse results than the right one done simply.
-
inspiration
Temperature Is Not Creativity
Temperature is a probability reshaper, not a creativity dial. Calling it a creativity parameter is a category error — one that leads to misconfigured systems and wasted inference budget.
-
inspiration
Prompt Caching Is Free Money
Every time your app resends the same system prompt, you pay to compute it again. Prompt caching eliminates that cost by reusing precomputed KV tensors across requests. It requires no code changes and delivers up to 90% input token savings.
-
inspiration
The Model Is Not the Agent
An LLM does not call tools. It requests them. The loop is the agent — and most broken agents are broken loops, not broken models.
-
inspiration
Speculative Decoding: The Free Tokens
Speculative decoding cuts inference latency 2–3x without changing a single output token. The gain is real. So is the catch.
-
whitepaper
Choosing Your Inference Engine: llama.cpp, Ollama, and vLLM
llama.cpp, Ollama, and vLLM are not interchangeable. They solve different problems at different scales. This paper maps the architectural differences, performance characteristics, and deployment tradeoffs to help you pick the right engine for your workload — and understand why the wrong choice costs you in ways that are hard to undo.
-
inspiration
Quantization Is a Design Decision
Quantization is not just compression. It is a tradeoff you are making about accuracy, speed, and memory — and it belongs in your architecture docs, not your deployment scripts.
-
inspiration
Context Is Not Memory
A large context window does not make an LLM remember. It makes it attend. The distinction changes how you build.