Detection Validation Pipeline

Why I built it

Security teams catch attackers by writing detection rules: small queries that look for suspicious behavior in logs, like PowerShell being launched with an encoded command. Plenty of teams now keep those rules in Git and review them like code. But the usual automated check only confirms that a rule parses.

A rule can parse perfectly and still be useless. It might miss the abbreviated -e flag attackers actually type, or fire every time an admin script sets -ExecutionPolicy. Neither problem shows up until the rule is in production, either silently missing attacks or flooding analysts with noise.

I wanted a pipeline that answers the question a reviewer actually has on a pull request: does this rule catch the attack, and does it stay quiet otherwise?

How it works

When a pull request adds or changes a rule, CI figures out which rules changed and starts one isolated job per rule. Each job runs in its own Docker network with no internet access.

1. A simulator generates telemetry modeled on the rule’s mapped Atomic Red Team test, plus a baseline of normal activity.
2. A shipper loads the rule into a mock SIEM and sends it the events.
3. A validator waits until it can confirm the events arrived, then checks the alerts and returns an exit code.

A rule passes only if three things are true for that run: it’s tagged with the same ATT&CK technique its test exercises, it fires on the attack, and it doesn’t fire on the normal baseline. Here’s what that looks like for a real rule and for the test fixtures I wrote to make sure the validator can say no:

This is the actual output for the “too broad” fixture:

Rule: Negative control: too broad (00000000-0000-4000-8000-000000000002)
Test: T1059.001 PowerShell Command Execution (a538de64-1c74-46ed-aa60-b995ed302598)
  [PASS] Technique mapping: test exercises T1059.001; rule declares ['T1059.001']
  [PASS] True positive: 1 alert(s) on 2 attack event(s) from 'PowerShell Command Execution'
  [FAIL] No false positive: 1 alert(s) on 4 benign event(s)
      matched: powershell.exe -ExecutionPolicy RemoteSigned -File C:\Ops\inventory.ps1
RESULT: FAIL

Three exit codes instead of two

Pass and fail aren’t enough on their own, because “zero alerts” can mean two very different things. Either the rule is too narrow, or the logs never arrived. If a SIEM outage shows up as a failed rule, people learn to ignore the check.

So there are three outcomes. 0 means pass. 1 means the rule is wrong and its author needs to fix it. 2 means there’s no trustworthy verdict, for example missing telemetry, a SIEM that’s down, or a rule using syntax my engine doesn’t support. That last case matters. When the engine hits a clause it can’t evaluate, it returns an error instead of skipping the clause, because skipping part of a rule changes what it means and could produce a false pass.

Simulated logs on PRs, real attacks after merge

Simulated telemetry is fast and harmless, so it can gate every pull request. But simulation can’t prove that Windows actually logs what I modeled.

A second workflow runs the real Atomic tests on a disposable Windows VM with Sysmon, collects the logs, and scores every rule against them. It only runs on pushes to main and manual runs, never on pull requests, because code from a pull request should never get to choose what executes on a machine.

Real machines are noisy, and helper processes often log before the attack does. Instead of labeling events by time window, I label them by process tree: only the process running the test’s command and its children count as attack events.

Real logs corrected my simulation at least once. Exported Sysmon data showed %VAR% written as %%VAR%% inside a command line, so I updated the simulated events to match.

The rules

To give the project a focus, I wrote most of the rules for video-game tampering: license bypass, mod or crack injection, memory tampering, bootleg copies, and license-key abuse. They protect FakeQuest, a harmless stand-in game with a pretend license check that I wrote for this, so nothing touches a real game, real cracks, or real anti-cheat software. The license domain uses the reserved .invalid name, and the “memory tool” closes its handle without reading or writing anything.

Rule	ATT&CK	Reads	Agrees with Elasticsearch
PowerShell launched with encoded command (Defender XDR)	T1059.001	Process creation	Not compared (KQL)
PowerShell launched with encoded command	T1059.001	Process creation	12 of 13
PowerShell launched with short encoded flag	T1059.001	Process creation	13 of 13
Command shell obfuscated with variable substring	T1059.003	Process creation	13 of 13
License server redirected through the hosts file	T1565.001	Process creation	15 of 15
Game loaded a system DLL from outside the Windows folder	T1574.001	DLL load	14 of 15
Process requested write access to game memory	T1055	Process access	15 of 15
Game running from an unofficial build	T1554	Process creation	14 of 14
License key guessing from one address	T1110	License server log	15 of 15
One license key on too many machines	T1078	License server log	14 of 14

The two license-abuse rules count activity over time using standard Sigma correlations. They count different values, not attempts. A player retrying the same mistyped key shouldn’t look like someone guessing 20 different keys. I included two negative controls that count attempts instead, and the pipeline fails both of them on normal players.

Measuring what gets past

Passing against one version of an attack doesn’t say much about the next one. I wrote ten harmless variations of the game and license attacks and recorded what each rule does with them. On real logs, the rules originally caught 2 of 10. After the fixes they catch 5 of 10.

Variation	Before	Now	What changed or why not
Edit hosts file with PowerShell’s Add-Content	Missed	Caught	The rule now knows PowerShell’s file-writing commands
Edit hosts file from a script	Missed	Missed	The command line only shows the script name; needs file monitoring
Plant the DLL as uppercase WINMM.DLL	Caught	Caught	Matching ignores case
Plant the DLL next to a renamed game	Missed	Missed	Logging every DLL load is too noisy, so logging keys off the game’s name
Request full access to game memory	Missed	Missed	Launchers request full access too, so it’s left out on purpose
Request write-only access to game memory	Missed	Caught	The rule now covers write access without read access
Run the bootleg copy under a new file name	Missed	Caught	The rule now reads the name stored inside the file
Guess keys while changing the machine ID	Caught	Caught	The rule counts by address
Guess keys from many addresses	Missed	Missed	Needs thresholds tuned on real traffic
Share a key while reporting the same machine ID	Missed	Missed	The rule trusts the ID the client reports

Any result that differs from the record fails the run in either direction. That way known gaps can stay on record without blocking every build, but a regression still fails, and an improvement forces me to update the record.

I did the same for false alarms. Eleven normal scenarios, like a family sharing a PC or a launch day with 60 players, are scored against every rule. Before tuning, 4 of 11 set off no rules. After raising the key-guessing threshold from 10 different keys to 20, a small LAN event with 12 typos stopped triggering it, and 5 of 11 are clean. The cost is that a real guesser now gets 19 tries before an alert instead of 9. The remaining false alarms each have a reason: an official update that isn’t on the hash list yet looks like a bootleg, and a legitimate mod loads the same way a crack does.

Checking my own engine

Every rule passes or fails on engines I wrote, so a pass could mean the engine and the rule share the same misunderstanding. To check that, each real run loads its logs into an actual Elasticsearch instance, converts the Sigma rules with the standard pySigma converter, and compares the results.

125 of 127 results agree. The comparison also turned up problems I never would have found testing against my own engine:

• The converted queries are case-sensitive, even though Sigma rules are supposed to ignore case. In Elasticsearch, two rules miss an uppercase WINMM.DLL and PowerShell’s -EncodedCommand. Those are the two recorded differences.
• The converter’s like patterns are rejected outright when a rule contains a Windows path with backslashes.
• Its rlike patterns have to match the entire value and turn \d into a literal d, so the obfuscation rule never fired at all.

My comparison script corrects the last two. These findings apply to pySigma-backend-elasticsearch 1.1.6, the version I tested.

Treating the rule as untrusted

The repository is public, so anyone can open a pull request containing a rule. I treated every rule as untrusted input. Workflows have read-only permissions and use pull_request rather than pull_request_target, so forks never see secrets. Actions are pinned to full commit SHAs. YAML is only loaded with safe_load. Test containers run as a non-root user with a read-only filesystem, all Linux capabilities dropped, memory and process limits, and no network. Values reach the shell through environment variables instead of inline template expressions, which blocks script injection through file names.

Timeline

Most of it landed over two days, through ten pull requests.

Date	What landed
Nov 4	Pipeline, first PowerShell rules, and the overbroad-rule demo
Nov 4	Real Windows logs from a disposable VM
Nov 4	Game tampering rules, license-abuse correlations, bootleg-copy detection
Nov 4	Evasion scorecard
Nov 5	Noise scoring across eleven normal scenarios
Nov 5	Elasticsearch comparison
Nov 5	Public dashboard rebuilt from each run

What it doesn’t prove

• Coverage is small: two Atomic tests, four game tests, and two license tests.
• A real pass covers one clean Windows Server VM with a minimal Sysmon config. It says nothing about false positives on real user activity.
• License traffic is scripted from one machine, so every request has the same address. Real thresholds would need real player traffic.
• Converted rules can behave differently in a real SIEM, as the Elasticsearch comparison shows, so they need testing wherever they run.
• Half the evasion variations still get through.

What I’d do next

The gaps point to the next steps: file monitoring for hosts-file edits made from scripts, signature checks so legitimate overlays stop tripping the memory rule, an approved-mods list, and thresholds tuned against real traffic instead of scripts.