Sampling: inverting the request direction
In everything covered so far, the client requests and the server responds. Sampling flips that: it's a mechanism for a server to ask the client's LLM to generate something on the server's behalf, mid-way through handling some other request. A server might be executing a multi-step task and hit a point where it needs the model to summarize an intermediate result, classify something, or decide what to do next, without the server itself needing to have any LLM access or API keys of its own — it borrows the client's model connection for that one generation and continues.
This matters because it keeps model access, and the cost, rate limits, and user consent that come with it, centralized at the client, even when the actual work is happening deep inside a server-side process. A server author doesn't need to provision their own model API key or worry about which model the user prefers; they ask the client to sample on their behalf, and the client (typically with the user's visibility into what's being asked) handles the actual generation.