Introducing Sentinel v3
- False positives fell 26x: from 14.61% on Sentinel v2 to 0.56% on our 178-sample difficult benign dataset.
- Accuracy went up at the same time: average F1 across five public prompt injection benchmarks rose from 0.957 to 0.975.
- It got smaller: 308M parameters against v2's 596M.
False positives in blocking mode
False positives interrupt legitimate work. A detector that repeatedly blocks shell commands or security research creates review work, and teams may disable blocking to get their jobs done. We built Sentinel v3 to reduce those interruptions.
What we measured it against
Agents read attack examples, run destructive-looking commands, and handle tool output that resembles an injection. We wanted to test those cases.
We built rogue-security/real-world-benign-use-cases and published it: 178 samples pulled from production AI coding-agent traffic, every one of them benign. Chat messages, tool output, shell commands, code snippets. We deliberately weighted it toward content that had already tripped at least one existing classifier, then deduplicated with sentence embeddings.
It is a stress test, not an average day. Every row is labeled benign, so any positive prediction is a false positive and nothing else.
| Model | False positives | FP rate |
|---|---|---|
| sentinel-v3 | 1 / 178 | 0.56% |
| sentinel-v2 | 26 / 178 | 14.61% |
| mdeberta-v3-base | 26 / 178 | 14.61% |
| promptguard | 28 / 178 | 15.73% |
| deberta-v3-base | 39 / 178 | 21.91% |
| PIGuard | 49 / 178 | 27.53% |
| wolf-defender | 63 / 178 | 35.39% |
| distilbert | 157 / 178 | 88.20% |
Sentinel v3 fires on one sample out of 178. The next best detector fires on twenty six.
The measured difference is 25 fewer false positives in this 178-sample stress test. Estimate production impact separately on representative traffic from the intended deployment.
The comparison applies to this dataset and evaluation setup. A model's behavior on representative traffic may differ.
Detection accuracy
We also measured detection accuracy to check whether fewer false positives came at the cost of missed attacks.
Average binary F1 across five public prompt injection benchmarks:
| Model | Avg F1 | Params |
|---|---|---|
| sentinel-v3 | 0.975 | 308M |
| sentinel-v2 | 0.957 | 596M |
| sentinel-v1 | 0.936 | 395M |
| mbert-prompt-injection-v2 | 0.799 | 150M |
| deberta-v3-base-pi-v2 | 0.750 | 304M |
| jailbreak-classifier | 0.627 | 110M |
Sentinel v3 had the highest average F1 among the models in this comparison and uses about half as many parameters as v2. Measure inference latency on the hardware, input lengths, and batch sizes you plan to deploy.
Availability
Sentinel v3 ships inside the Rogue platform today. It runs as part of Rogue AIDR in your own VPC: no traffic leaves your network, no per-token inference bill, no external API in the request path. Existing customers on Sentinel v2 move over without changing rulesets.
Unlike v2, Sentinel v3 weights are not being published. Sentinel v2 stays up on Hugging Face for anyone who wants to reproduce the baseline, and both evaluation datasets are public:
Run them against whatever you have in production now. If your current detector is in log-only mode, you probably already know why.