What a 25 Percent Score Taught Me About Testing by Nicolas

What a 25 Percent Score Taught Me About Testing by Nicolas - October 2026 Scholarship Essay

This summer I built a tool called Gatekeep. It sits between a user and an AI chatbot and checks every request before it goes out. The first version looked for passwords, API keys and personal information. Later I added a second check for prompt injection, which is when text hidden inside a document or a message tells an AI model to ignore its instructions and do something else.

The question I wanted to answer was simple. Does it work?

To find out, I wrote my own set of 100 test examples. Fifty-two were injection attacks, sorted by technique: instructions that override the model's rules, attempts to leak its hidden system prompt, fake role changes, and text disguised with invisible characters or lookalike letters. The other 48 were normal requests that happen to use some of the same trigger words. The detector caught all 52 attacks and flagged none of the 48 normal requests. That looked like a finished result.

Then I ran it against a public dataset that someone else built, called deepset/prompt-injections. It caught 15 of 60 attacks, which is 25 percent. The same code produced both numbers.

The only thing that changed was who wrote the examples. I had written attacks that my own detector already knew how to recognize, because I wrote the detector and the test myself. The public dataset used phrasing I hadn't thought of. My test measured what I had imagined, not what attackers actually do.

I read more after that. Research from this year shows that detectors like mine can be beaten by an attacker who is allowed to keep trying against them. One 2025 study broke most of twelve published defenses with a success rate above 90 percent once the attacker could adapt. So the 25 percent was not a bug that more rules would fix. It showed me what this kind of tool can do and what it can't.

The second number I watched was false alarms. A security tool that blocks normal work gets turned off. I tested it against 339 harmless prompts that contain scary-sounding words. It wrongly flagged 7 of them, about 2 percent. I decided that number mattered more than the catch rate. A missed attack can be improved later, but a tool people disable protects no one.

Two things changed in the project because of this. Gatekeep now logs and warns by default instead of blocking, and the person using it picks where to set the cutoff. I also wrote the results into a plain report that ships with the code, including the 25 percent, so anyone who uses it can see how it performs before they trust it.

I came away with a different idea of what a test is. A score only means something if someone other than the author wrote the questions. Now I look for an outside dataset before I trust my own numbers, and I report the weak result next to the strong one. After graduation I want to work in security operations, where saying clearly what a tool misses is part of the job.