AI safety / Evaluation / RAG / False positives / Research · 5 min read
When a safety score hides the cost
A defense reached zero attack success in one test. The more useful question was what happened to the clean answers.
Muhammad Musa
The attractive number
In a paired experiment on llama3.2:3b, our fused defense stopped the measured T2 attacks. Attack success was 12% without a defense, 38% with a prompt-only defense and 0% with the fused defense. There were 50 attack-paired queries. It is a striking comparison, and an easy place to stop reading.
I do not think it is a sufficient account of the result. The same filter can remove malicious instructions and useful evidence. A retrieval system that avoids an attack by losing the information needed to answer has traded one failure for another. I wanted that tradeoff to be visible in the evaluation.
Give every percentage its denominator
The initial clean-query measure was deliberately strict: count a query whenever the defense removes at least one retrieved chunk. The fixed-threshold fused defense triggered that condition for 76% of 50 clean queries. That does not mean it removed 76% of chunks, and it does not mean 76% of answers were wrong.
Each query retrieved five chunks. Removing one of those five can leave enough information to answer; removing the only relevant passage can do much more damage. Combining both events into one flag makes the result easy to count but difficult to interpret. I kept the original measure and added more specific ones.
Three views of the cost
The first view counted removed chunks over all retrieved chunks: 31%. The second counted clean queries where removal coincided with a degraded answer: 32%. The third used a pairwise model judge to compare answers with and without the defense, independent of whether a removal occurred: 34% were judged worse.
These figures describe different populations and conditions. They refine the original 76% signal rather than replacing it with a more flattering score. In the public analysis, the definitions sit next to the results so a reader can tell exactly which event each percentage counts.
removed chunks / all retrieved chunks
queries with removal and degradation / clean queries
queries judged degraded / clean queriesThe evaluator needs scrutiny too
I compared saved defense-on and defense-off answers and randomized their A/B order, recording that assignment so it could be inspected. This reduces a source of position bias; it does not turn a model judge into ground truth. The report also records how ties, refusals and parse failures were handled.
The study remains small and specific. A single seed, one clean-query set and one target model cannot establish deployment safety. Other parts of the project exposed additional limits: a fragmented attack could fail because its pieces were not retrieved together, and a proposed defense signal could contribute almost no learned weight. Those details change what the headline is allowed to mean.
What I would carry into the next experiment
I would preserve the retrieved passages, filter decisions and answers as first-class results. Then I would measure security and utility separately across more seeds, domains and model sizes. If a decision changes the denominator, I would make that change explicit rather than compare the numbers as though they were the same experiment.
The useful outcome was not a claim that I had solved prompt injection. It was a clearer account of where a defense helped, where it hurt, and which next experiments could distinguish those effects. That is a much better starting point for a system someone might actually use.