Prompt caching

Prompt caching is a feature that some model providers offer. When the start of a prompt is identical from one call to the next, the provider can reuse the work it has already done on that prefix. The result is usually lower cost and faster responses for the repeated part. What counts as the prefix, how long it is kept and how it is priced depend on the provider, so the provider's documentation is the reference.

The design rule is simple: put the stable material first and the changing material last. Instructions, policy text and tool definitions rarely change, so they belong at the front. The user's question and the retrieved passages change on every call, so they come after.

In practice

I order the prompt deliberately and check the cache hit rate in the logs. A prompt that changes its first line on every call gets no benefit. I also keep timestamps and random identifiers out of the prefix, because they break the match.

Caching changes cost, not quality. It is still measured against the evaluation set, because a mistake in the prefix can change behaviour for every call that uses it.

Example

[Composite] An assistant put the current date at the top of every prompt. The cache never matched. Moving the date to the end of the prompt, and keeping the instructions unchanged at the front, produced a steady cache hit rate and lower cost per call.

Common mistakes

• Putting volatile content such as dates, identifiers or retrieved passages at the start.
• Assuming caching is free of the usual token costs. Check the provider's pricing.
• Measuring cost without checking the hit rate.