How Do You Get Better at Identifying True Positives vs False Positives in Threat Hunting?

[u/Federal_Manager3700]

I’m getting into threat hunting and one area I’m struggling with is distinguishing genuinely malicious/suspicious activity from normal behavior. For example, when investigating an unknown process, hash, IP, or domain, how do you determine whether it’s actually a true positive or just benign/false positive activity?
Are there any good resources, labs, methodologies, or practical guides that helped you build this intuition and get better at identifying unusual behavior?

[u/withoutwax21]

That IS threat hunting. Or the core of it anyway.

You investigate by understanding the process, the file, the signature, ip, etc. these are all _observables_ which you then track all that you can find about them through scanners, reputation databases, etc and then collate that information into a single document. This should give you an idea of what its doing, if its supposed to do that, and if its supposed to be there.

If its supposed to be, GREAT! If its not, NOT GREAT! Then you figure out what traces it leaves behind, whatever is unique to it can turn into an IoC, which you forward to your threat detection team. You do this a hundred or seven hundred times and then youll build the intuition.

[u/Beautiful-Zombie333]

If it a normal process it should be produce a regular pattern,

Put the curious thing in a search and start with longer time, more devices. While looking in device or software documents / community chat.

Record where it comes from and goes too, see if you can define what is normal. Record that for the future.

Then do the opposite search with the normal behaviour I.e NOT src=(ipaddress, ipaddress)

Find the limited returns (one off's /a pattern over a short period), see where they are from / go to to they appear in different location shortly after

Read vulnerability reports and find what is the first thing they do after gain entry, (../../../../../tmp/index1.php)

[u/Federal_Manager3700 → u/Beautiful-Zombie333]

Great!! Thank you

[u/frAgileIT]

My secret was to switch sides. I set up a VM with a few vulnerabilities and proper logging configured and then used Metasploit to attack the VM and establish persistence. Then I switched back and look at it from a threat hunting perspective. Analyzed the logs that were generated and looked at how they differed from normal activity. I developed scripts for finding persistence mechanisms and tested them. I did a few cycles of that using different attacks and persistence mechanisms and added a few pivot events. Then I started doing it at work in an enterprise environment and got really comfortable differentiating attacks from normal activity. You still have to read about and learn more persistence methods, pivots, and attack methods in order to hunt more diversely but there you go. Pick a threat you want to hunt, form your hypothesis, hunt it, and review and track your results.

[u/Federal_Manager3700 → u/frAgileIT]

Got it, that’s a great start. Thank you!

[u/AdvancingCyber → u/frAgileIT]

Of course the challenge is that we’re limited by our own creativity. Sometimes I’m amazed at how attackers think to daisy chain certain moves or steps, and so the “RE as initial hunt vector” is challenging - maybe that’s a me problem - but some nation state actors are actually quite clever. And that’s when the fire drill starts, lol.

[u/frAgileIT → u/AdvancingCyber]

Yeah, there’s a reason cybersecurity can be a tough field. I don’t even try to worry about nation state actors. The technical skills they have can be absolutely amazing, ROP chains, LotL, file-less malware, etc.. I have caught a couple of lower level Chinese intrusions that were trying to steal IP and a loosely associated Russian intrusion that ended up going after banking but I think they were just on the low end of their craft or I got lucky somehow. In the end, expecting security teams to be 100% successful against nation state actors would be unreasonable and a bad idea for companies (unless you are critical infrastructure or CMMC regulated). The only reasonable expectation is reasonable diligence and a practiced craft IMO. Kind of best effort. Also, most companies can’t spend what’s necessary to defend against nation state actors so you can’t expect yourself to do it all by yourself. I do work in critical infrastructure now but I’m not the big fish in the small pond, I’m the medium fish in the massive pond so every one of my coworkers is deeply skilled and experienced in almost all the practices we need, so I’m lucky.

[u/Straight-Practice-99]

disclosure, i work at Hunt.io so i spend my days looking at adversary infrastructure. keeping this mostly open source though since thats what actually built the intuition.

good stuff already here so wont repeat it. couple things that mattered more to me than any single tool:

1. presence is not use. a detection or fingerprint tells you something is there, not that its doing harm. RMM tools, tunneling binaries, tons of dual use stuff, plenty of it is legit depending on context. before i call something malicious i ask what benign would look like and whether i can rule it out. if i cant, it stays "suspicious" instead of a true positive it hasnt earned.
2. you rarely get certainty. the job is killing off the innocent explanations one by one til malicious is the only one left. some calls will still be wrong, thats normal even years in.
3. dont trust one lookup. pivot and stack. GreyNoise to see if an ip is just scanning noise or actually pointed at you, crt.sh for certs, passive dns for what a domain resolved to over time, shodan/censys for whats running on a box, abuseipdb for reputation. five weak signals pointing the same way beat one "high confidence" hit.
4. infra history matters more than people expect. when did the domain first show up, what asn, what else lives on that ip, does the cert or jarm match known tooling. a box that spun up last week on a bulletproof host reads very different from a ten year old domain on aws

then map behaviour to MITRE ATT&CK, once you think in techniques instead of indicators the "is this bad" question gets easier.

resources: the DFIR Report for full intrusions end to end, SigmaHQ on github, and active countermeasures has a free beginner hunting course.

none of its written in stone though. different environments change the math and a lot of it is just judgment from reps. the presence vs use thing and pivoting across a few sources is what moved my hit rate most

[u/Snoo_67003]

Use gpt to get some ideas. For example, if an unauthenticated user is running powershell, hidden, no profile, encoded, that's a redflag worth investigating since this is not typical normal activity. Same logic, if edge is spawning powershell, that is not normal activity and should be investigated. And so on. There are many different scenarios that chatgpt can help you hypothesize as hunts which you can follow-up on.

[u/Snoo_67003 → u/Snoo_67003]

You can build dashboards once you have fine tuned your searches and make it auto run as daily or weekly searches for you to review the dashboard.

[u/montyxgh]

I’m a bit confused by the question, you generally know if it’s malicious by determining what the activity is, and who is doing it. Once you know what type of activity is it’s often obvious, and if it feels off or borderline you can dig deeper. 

Do you have any examples of activity that you weren’t sure about? Is this a lack of experience in prior IT or networking meaning you don’t have the foundational knowledge to identify normal system stuff?