Language-model security
What does a defense cost?
I tested how poisoned source material changes a model’s answers, then measured what the defenses broke along the way.
My role
I co-authored this two-person study and led the advanced attacks, fused defense, ablations and evaluation. Waleed Arshad built the initial pipeline scaffold and early attack modules.
Where it stands
Scoped research study at UW–Madison with public code and recorded results. The defense is not deployment-ready.
What changed.
A defense that reduced measured attack success also removed useful evidence. Reporting both changed the conclusion.
The threat model
The attacker could add passages to the retrieved corpus, but could not edit the user’s question or the system prompt. I studied the point where useful retrieved text becomes an instruction the generator should never have followed.
What I built
I developed obfuscated, semantic, fragmented and defense-aware attacks, a classifier-based fused defense, and an evaluation pipeline that preserves the retrieved evidence and generated answers. The classifier combines several signals; one proposed signal contributed almost no learned weight, which matters when describing the final mechanism.
The result worth keeping
In the paired T2 ablation on llama3.2:3b, attack success was 12% without a defense, 38% with a prompt-only defense and 0% with the fused defense, across 50 attack-paired queries. Those are results for that experiment, not a universal ranking of defenses.
The cost behind zero
On 50 clean queries, the fixed-threshold fused defense removed at least one retrieved chunk for 76% of queries. A later analysis separated the denominators: 31% of retrieved chunks were removed, 32% of queries combined removal with a degraded answer, and 34% were judged to have worse answers. Those metrics describe different effects; they are not interchangeable.
The limits
The corpus, model choices and attack families constrain the result. An attack that never retrieves its required fragments has not proved a defense strong. A model judge also has its own biases. I kept those limitations beside the scores rather than treating them as footnotes.
The stack
- Python
- PyTorch
- ChromaDB
- Transformers
- scikit-learn
- Ollama