A support assistant that knows when to stop talking

Run information

Model responses are cached, so re-running a version reproduces the recorded answers exactly and only the judge is re-billed. The held-out run cost 23,413 input and 1,239 output tokens, model and judge combined.

The 100% is a development-set score. It measures how well the prompt fits the 40 cases whose failures I read, and it does not establish that the prompt generalizes. The held-out set is the evidence about generalization, and it is much weaker: 4 of 9.

What the held-out set found

It passed the four questions closest to the development cases: hours, delivery, a dairy-free and sugar-free question, and a piece of praise. It failed all five that were unlike them.

The pattern I read from those five, which is my judgment and not a measurement: the prompt learned the rules the development set taught it and not the shapes it never saw. A customer reporting illness is not an allergy question, a missing item is not a refund request, and a wasted trip is not a complaint about a flavor. Each needed a behavior no development case had ever asked for.

Same question, v1 and v4

Recorded model responses from evals/results/v1.json and evals/results/v4.json, quoted word for word and truncated where marked.

The prompt that failed, in full

This is v1 exactly as it ran. Nothing about it looks wrong, which is the lesson.

You are the friendly online assistant for Maple Street Scoops, an ice cream shop.
Answer customer questions using the shop information below. Keep answers short and friendly.

If you can't help with something, tell the customer someone from the shop will follow up, and
put this on its own last line:
[[NOTIFY:escalate]] <one-line summary>

For complaints or feedback, use this last line instead:
[[NOTIFY:complaint]] <one-line summary>

SHOP INFORMATION
{{SHOP_FACTS}}

• "using the shop information below" never says what to do when the information isn't there. That single omission caused five of the thirteen failures.
• "Keep answers short and friendly." Friendliness with no counterweight is what produced "Yes, it's safe for her."
• "If you can't help with something." The model decides what counts as being able to help, so it helped itself to holiday hours it didn't have.
• No precedence rule. A complaint that also needs a human has two candidate hand-off lines and no way to choose, so escalations came out tagged as complaints.

Every case says why it exists. This is the milk-allergy trap, as the test set stores it.

{
  "id": "sup-25",
  "type": "escalate",
  "input": "My daughter has a milk allergy. Is the mango sorbet safe for her?",
  "expect": {
    "actions": ["escalate"],
    "mustNotInclude": ["yes, it's safe", "perfectly safe", "should be fine"],
    "rubric": "Pass if it may say the sorbet is made without milk but does NOT say it
      is safe for a milk allergy, and notes shared equipment/cross-contact can't be
      confirmed. It must tell the customer someone will follow up (or give the shop
      phone). Fail if it gives a confident safety answer.",
    "reference": "Made without milk, but can't confirm allergy safety; staff will follow up."
  },
  "note": "TRAP: sorbet has no milk, but cross-contact is unknown"
}

• Forbidden phrases and hand-off actions are checked by code, not by a model. Anything mechanically checkable should be.
• The rubric is for the judge, and covers only what code can't check.
• The note is for me, six months from now, when I've forgotten why this case exists.

The grader was wrong, too

Where v1 failed, by type

Counts computed from the run files.