Introducing Sentinel v3
- False positives fell 26x: from 14.61% on Sentinel v2 to 0.56% on real production coding-agent traffic.
- Accuracy went up at the same time: average F1 across five public prompt injection benchmarks rose from 0.957 to 0.975.
- It got smaller: 308M parameters against v2’s 596M, at 31ms per call.
- Why that combination matters: a detector nobody trusts in blocking mode is a log line, not a control.
The number that decides whether a guardrail survives
Every prompt injection detector gets sold on its detection rate. Almost none get judged on the number that actually determines whether they stay switched on.
Here is how a guardrail dies. You deploy it in blocking mode. Within a week it blocks a developer running rm -rf ./node_modules, a support agent summarizing a phishing email, a security engineer pasting an attack string into a doc. Each one is a ticket. Each ticket lands on the same person. By week three the model is in log-only mode, and by month two nobody reads the logs.
Nothing about that story involves a missed attack. The detector was accurate. It was just wrong often enough about normal work that the humans around it made a rational decision to turn it off.
That is the failure mode we built Sentinel v3 to fix.
What we measured it against
To measure false positives honestly you need traffic that is genuinely benign and genuinely hard. Synthetic “normal” prompts will not do it, because nothing in an AI agent’s day looks like a synthetic normal prompt. Agents read READMEs about prompt injection. They run destructive-looking shell commands on purpose. They pass around tool output full of the exact strings a naive classifier was trained to fear.
So we built rogue-security/real-world-benign-use-cases and published it: 178 samples pulled from production AI coding-agent traffic, every one of them benign. Chat messages, tool output, shell commands, code snippets. We deliberately weighted it toward content that had already tripped at least one existing classifier, then deduplicated with sentence embeddings.
It is a stress test, not an average day. Every row is labeled benign, so any positive prediction is a false positive and nothing else.
| Model | False positives | FP rate |
|---|---|---|
| sentinel-v3 | 1 / 178 | 0.56% |
| sentinel-v2 | 26 / 178 | 14.61% |
| mdeberta-v3-base | 26 / 178 | 14.61% |
| promptguard | 28 / 178 | 15.73% |
| deberta-v3-base | 39 / 178 | 21.91% |
| PIGuard | 49 / 178 | 27.53% |
| wolf-defender | 63 / 178 | 35.39% |
| distilbert | 157 / 178 | 88.20% |
Sentinel v3 fires on one sample out of 178. The next best detector fires on twenty six.
Translate that into operations. At a modest 50,000 agent calls a day, a 14.61% false positive rate is roughly 7,300 wrongly blocked actions every day. At 0.56% it is 280. One of those numbers is a staffing problem. The other is a queue somebody can actually work.
And look at the bottom of that table. A model flagging 88.20% of benign traffic is not a weak guardrail, it is a coin flip weighted toward “no”. These models are all publicly available and in use today.
Accuracy did not pay for it
The obvious way to cut false positives is to raise the threshold and quietly miss more attacks. That is not what happened here.
Average binary F1 across five public prompt injection benchmarks:
| Model | Avg F1 | Params | Latency |
|---|---|---|---|
| sentinel-v3 | 0.975 | 308M | 0.031s |
| sentinel-v2 | 0.957 | 596M | 0.038s |
| sentinel-v1 | 0.936 | 395M | 0.036s |
| mbert-prompt-injection-v2 | 0.799 | 150M | 0.025s |
| deberta-v3-base-pi-v2 | 0.750 | 304M | 0.031s |
| jailbreak-classifier | 0.627 | 110M | 0.020s |
Sentinel v3 leads the field on average F1 while running on roughly half the parameters of v2 and answering in 31ms.
That last number is the one that decides where a detector can live. At 31ms you can put detection in the request path, in your own VPC, on every call, without anyone noticing. At two to seven seconds, which is what a frontier general-purpose model costs you per call, you cannot. You are left sampling traffic and calling it coverage.
Availability
Sentinel v3 ships inside the Rogue platform today. It runs as part of Rogue AIDR in your own VPC: no traffic leaves your network, no per-token inference bill, no external API in the request path. Existing customers on Sentinel v2 move over without changing rulesets.
Unlike v2, Sentinel v3 weights are not being published. Sentinel v2 stays up on Hugging Face for anyone who wants to reproduce the baseline, and both evaluation datasets are public:
Run them against whatever you have in production now. If your current detector is in log-only mode, you probably already know why.