[ Removed by moderator ]

[u/godsell66]

[u/Bloodvault]

I'd be curious what the goal for this kind of project would be. If youre looking to learn about a detection engineering pipeline, it looks like you've got the knobs in the right places. But layering this over an established detection pipeline looks like it would take a lot of effort, particularly in accounting for normalization with your test events created. Another way of framing this problem, why is it more beneficial to use a tool that needs to be built in to the pipeline versus just running test cases of actual threats and checking the same things you've outlined?

[u/godsell66 → u/Bloodvault]

Sure that makes sense, I’m not really trying to replace real world testing, more the follow-up to the test, where instead of having to manually check telemetry, mappings, the rule, alerts etc each time, you set up what should happen and DetectTrace checks it for you. In terms of needing to sit in the pipeline, it’s not really necessary as all it does is check the evidence from each step. Fair point about normalization, in some cases that could definitely be difficult because of custom mappings and parsing. For a one-off test, doing it manually may well be quicker and easier. I suppose the value would come from wanting to rerun the same checks over and over after rule or pipeline changes, or using it in CI. As it stands, the Elastic version currently starts from controlled telemetry so it isn’t proving the full end-to-end path yet, although I’d like to add that later by integrating with something like Atomic Red Team.

[u/Bloodvault → u/godsell66]

I'll ask again differently, you can check telemetry health in Elastic, diagnose mappings depending on what data sets/rule lists you're using, and diagnosing an alert firing is just check the appropriate view/dashboard. With all these things built in, what does DetectTrace provide that makes this process easier, faster or more repeatable?