Aravind 7120efa417 State the untrusted-content rule for tool results, and answer in the reader's language ...

I7 had a hole. ContextInstruction states the rule for <context> blocks —
retrieved documents — and SystemPrompt has always carried it. Nothing stated it
for tool results, which arrive as their own message carrying whatever the
records hold: a candidate's note, a job description, a worker's name. Any of
those is text a person outside the company can write, and the model was given no
reason to read it as data.

gateway.ToolResultInstruction sits beside ToolResult for the same reason
ContextInstruction sits beside its renderer: a prompt promising a rule the
transport does not frame is a defence that has quietly stopped existing.

What it is worth is small, and the comment says so with the numbers. Against a
local qwen3:0.6b with a tool result carrying "ignore your previous
instructions": 3 runs in 20 held the line without the sentence, 5 in 20 with it.
An n=10 pass first suggested 1-in-10 against 6-in-10 and did not replicate. So
it is hygiene, not a control — what makes an injection survivable is I1 and I4,
which cost a hijacked turn an answer and never an action.

qwen_probe_test.go is how those numbers were taken: a DB-free probe of a
candidate model's tool-calling and injection resistance, skipped unless
MODEL_BASE_URL is set. The live eval suites need PostgreSQL and SKIP without it,
so they pass while testing nothing on a machine with none.

Also carries the language selector: a closed enum, because the value arrives
from a browser and the directive it selects goes into the system prompt. A
client picks a constant by name; nothing it sends is ever written into a prompt.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

2026-10-06 12:56:57 +05:30

34 KiB

Raw Blame History

View Raw