Cross-checking the matcher against Chainsaw

2026-07-09. Commands to reproduce at the end.

detsema’s unit tests prove the matcher agrees with my reading of the Sigma spec. To test it against someone else’s reading, I ran the same rules over the same logs in Chainsaw, WithSecure’s Sigma scanner for EVTX files, and compared matches event by event.

Setup

• Rules: the public SigmaHQ corpus.
• Logs: EVTX-ATTACK-SAMPLES, 278 real attack captures, 3,241 Sysmon events.
• Scope: Sysmon events only, where rule field names match log field names as-is, so a disagreement is about rule evaluation rather than field mapping. Scored only rules both tools ran: supported by detsema, loaded by Chainsaw, applied to the event IDs Chainsaw’s own mapping routes them to. 1,754 rules.

Result

detsema reproduced every Chainsaw match. I verified the 242 extra matches by hand: all are events the rules match per the spec, and all involve OR lists (a selection written as a list of alternatives), which Chainsaw mis-evaluated in two ways. Reported with reproduction steps in chainsaw#232; the maintainer confirmed both patterns came from a single bug and fixed it for v2.16.1.

The two failure patterns

An alternative naming a field the event doesn’t have (148 of the 242). A standard SigmaHQ idiom matches a process by OriginalFileName or by Image path. Captures from older Sysmon versions have no OriginalFileName field; the Image alternative should carry the match, and in Chainsaw it doesn’t:

Rule: Whoami.EXE Execution From Privileged Process
Event: whoami.exe running as SYSTEM, capture without an OriginalFileName field

rule as written  (OriginalFileName OR Image ends with \whoami.exe)  ->  Chainsaw: no match   detsema: match
same rule, OriginalFileName alternative deleted                     ->  Chainsaw: match      detsema: match

An added alternative that matches nothing removes existing matches. A rule matched 22 events in one file via Image|endswith: '\calc.exe'. Appending a second alternative that matches nothing in that file drops Chainsaw to 0. An OR alternative can add matches, not remove them:

OR list: [ Image ends with \calc.exe ]                                   ->  Chainsaw: 22   detsema: 22
OR list: [ Image ends with \calc.exe,  CommandLine contains ' -e* JAB' ] ->  Chainsaw:  0   detsema: 22

Both reductions are single-line edits to real SigmaHQ rules, metadata untouched.

Conclusions

• The matcher held up against an independent implementation on real logs; every disagreement resolved against the other tool, and the other tool’s maintainer agreed. This is one comparison on one log source. I’ll repeat it against Hayabusa; until then, validation tracks what is and isn’t proven.
• If you hunt with Chainsaw, upgrade to v2.16.1 when it lands. On 2.16.0 and earlier, rules using OR lists can miss events, most often on captures from older Sysmon versions (chainsaw#232, confirmed and fixed).

Reproduce

Harness, comparison script and step-by-step commands: contrib/sigma-diff. Needs chainsaw, evtx_dump, Go and Python 3.

Versions: detsema v0.1.2, Chainsaw 2.16.0, evtx_dump 0.12.2, SigmaHQ/sigma fe2a6e8, EVTX-ATTACK-SAMPLES 4ceed2f.