State the untrusted-content rule for tool results, and answer in the reader's language

I7 had a hole. ContextInstruction states the rule for <context> blocks —
retrieved documents — and SystemPrompt has always carried it. Nothing stated it
for tool results, which arrive as their own message carrying whatever the
records hold: a candidate's note, a job description, a worker's name. Any of
those is text a person outside the company can write, and the model was given no
reason to read it as data.

gateway.ToolResultInstruction sits beside ToolResult for the same reason
ContextInstruction sits beside its renderer: a prompt promising a rule the
transport does not frame is a defence that has quietly stopped existing.

What it is worth is small, and the comment says so with the numbers. Against a
local qwen3:0.6b with a tool result carrying "ignore your previous
instructions": 3 runs in 20 held the line without the sentence, 5 in 20 with it.
An n=10 pass first suggested 1-in-10 against 6-in-10 and did not replicate. So
it is hygiene, not a control — what makes an injection survivable is I1 and I4,
which cost a hijacked turn an answer and never an action.

qwen_probe_test.go is how those numbers were taken: a DB-free probe of a
candidate model's tool-calling and injection resistance, skipped unless
MODEL_BASE_URL is set. The live eval suites need PostgreSQL and SKIP without it,
so they pass while testing nothing on a machine with none.

Also carries the language selector: a closed enum, because the value arrives
from a browser and the directive it selects goes into the system prompt. A
client picks a constant by name; nothing it sends is ever written into a prompt.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
2026-10-06 12:56:57 +05:30
parent a372340281
commit 7120efa417
9 changed files with 554 additions and 3 deletions

View File

@@ -290,7 +290,19 @@ func (m *ModelExecutor) executeRun(
// fill its trajectory with the same note.
var toolsWithheld bool
system := SystemPrompt(agent)
// The language the reader chose, resolved once for the whole run rather
// than per step: a run that answered its third turn in a different language
// from its first would be a bug, not a feature. An unrecognised tag is
// recorded and falls back to English — the reader loses a preference they
// may not have set, which is the cheap failure, and somebody is told.
lang, known := ParseLanguage(string(input.Language))
if !known {
rec.Error("runtime.unknown_language",
fmt.Sprintf("%q is not a language this build answers in; used %s",
string(input.Language), lang.Name()))
}
system := SystemPrompt(agent, lang)
if smalltalk {
// Taking the evidence away removes the citations; it does not by itself
// shorten the reply, because the agent's own instructions still
@@ -794,7 +806,11 @@ func terminationMessage(t Termination) string {
// retrieval, the retrieved chunks go into a delimited block in a *user*
// message — not into this string — and the standing instruction below is what
// makes that delimiter mean something.
func SystemPrompt(agent *Agent) string {
//
// `lang` does not weaken that. It is a Language, so the only strings it can
// contribute are the constants in language.go — the caller's two-letter tag
// selects one and is never itself written here. See Language.
func SystemPrompt(agent *Agent, lang Language) string {
var b strings.Builder
b.WriteString("You are ")
@@ -826,8 +842,26 @@ func SystemPrompt(agent *Agent) string {
b.WriteString(knowledge.ContextInstruction)
b.WriteString("\n\n")
// The same boundary for the other channel untrusted text arrives on.
// Retrieval is not the only one: a tool result carries whatever the records
// hold, and a person who can type into the platform can put a sentence
// there. Stated unconditionally, like the one above, because the rule has
// to be established before the content arrives rather than alongside it.
b.WriteString(gateway.ToolResultInstruction)
b.WriteString("\n\n")
b.WriteString("State a figure only where the records you were given show it. " +
"When you cannot answer from them, say so rather than estimating.")
// Last, and deliberately so. Everything above it is English — the agent's
// own instructions, the standing rules, and every tool result that will
// arrive later — so a language instruction placed earlier is one the rest
// of the prompt spends thousands of tokens arguing against. Nearest the
// question is where it holds.
if directive := lang.Directive(); directive != "" {
b.WriteString("\n\n")
b.WriteString(directive)
}
return b.String()
}