How Do You Ship a Prompt Change Safely?
On this page
Problem statement
"Your team wants to change the system prompt of a production assistant used by 200,000 people a day, to make answers friendlier. Last time a prompt change caused a spike in complaints. How would you ship this change safely?"
Prompts behave like code: a one-line change can alter behaviour everywhere. This question tests whether you treat them that way.
Why it happens
Prompt changes go wrong for predictable reasons:
- No version tracking: nobody knows which prompt produced a bad answer.
- Testing on a few examples: the change fixes the motivating case and breaks others (see Evals and LLM-as-Judge).
- Side effects: "be friendlier" also makes answers longer, which costs more, slows responses and can push out key details.
- Interactions: the prompt is tuned for one model version; a provider update or another team's change shifts behaviour.
- All-at-once release: every user gets the change before anyone sees the live metrics.

Try it yourself first
You have the full problem. Sketch your answer, then sign up free to read the complete walkthrough.
- The complete design, flow by flow, with pictures
- Deep dives into the hard parts
- What mid, senior and staff candidates are expected to say
- Full company question lists
Free forever. No card needed.