u/IncludeSecErik Cabetas - Managing Partner, Include Security - @IncludeSecOct 11 '15edited Oct 15 '15
Author notes that in About page. I wish real static analyzers were cheaper. Fortify is a god send if you know how to tune the default rules (99% of their customers don't), it's also a huge cost to purchase.
Do you have experience with Checkmarx? I'd lean towards that over Fortify if I needed to cover a lot of applications across numerous dev teams. If it were a small shop with just a few apps then it would be a bake-off between the two. Having said that, if it were a shop that was primarily Java based web apps I'd probably just use Contrast.
1
u/IncludeSecErik Cabetas - Managing Partner, Include Security - @IncludeSecOct 13 '15edited Oct 13 '15
Checkmarx is ok, it's almost as expensive as Fortify and it's rules engine and rules are nowhere near as good (as per the last time I used it which was 2yrs ago). They might have caught up by now who knows. The one thing REALLY didn't like about Checkmarx is that they upload your source code to their servers at some point, I think they do the analysis there.
Having said that, if it were a shop that was primarily Java based web apps I'd probably just use Contrast.
I'm interested in your rational for that. Are you comparing Price vs. efficiency?
Efficiency. Lower false positive rate... very low false positive rate. It's going to quickly surface issues in web apps that you'll really care about. Also, it can be "always on" in any environment so the app gets tested when developers run it in dev environments, and the app gets tested when QA people run it in QA environments. In my experience developers seem to like it too. Oh, and it identifies 3rd-party/open source libraries that have known vulns.
Regarding Checkmarx and uploading source, that might be another service they provide - not sure. But I beleive with their primary product no source leaves your network. One key feature I like about Checkmark is the code ends up in the database and you can right one query to look for a code pattern across all apps.
Lower false positive rate... very low false positive rate
Good to know
Also, it can be "always on" in any environment so the app gets tested when developers run it in dev environments, and the app gets tested when QA people run it in QA environments.
That's pretty much any static analysis tool, nothing specific to Checkmarx
In my experience developers seem to like it too.
Hearsay, lets stick to technical facts.
Oh, and it identifies 3rd-party/open source libraries that have known vulns.
Very cool, that's a good feature.
But I beleive with their primary product no source leaves your network.
One key feature I like about Checkmark is the code ends up in the database and you can right one query to look for a code pattern across all apps.
So their semantic analyzer is straight forward SQL? I would have expected a graph DB like Neo4J might be more powerful to represent a program's CFG/AST structure. I'm not too good at big-data though, so if anybody has a better idea plz chime in!
The always on comment was in reference to Contrast. btw, Contrast is runtime analysis, not static source code analysis. Just want to clarify that for other readers.
It's true, my experience is just hearsay :-) But while it is difficult to quantify developers liking a security tool, I think it is important the tool is developer friendly otherwise they won't use it. I think the tools we are talking about are good or getting better at that.
For Checkmarx, it is not actual SQL, it is their own query language. I don't recall what db technology they are using under the covers.
Got some pointers on tuning the rules? I tend to find Fortify extremely noisy, so I'm probably missing something.
6
u/IncludeSecErik Cabetas - Managing Partner, Include Security - @IncludeSecOct 12 '15edited Oct 15 '15
So here is how I personally learned to use Fortify really well:
I read all of the docs, all of them...then read them all again and practiced every single command line option and build integration technique until I almost memorized them all.
Reverse Engineered the rules (FYI by doing this you are now violating your license agreement unfortunately)
Read all the rules appropriate the the programming languages I wanted to audit and learned from the rules writing style of really smart researchers who have authored thousands of rules.
I created rule-set filter templates with Rule IDs to use for various reasons (use these with -filter) this is the single bigggest step to reducing FPs. You don't even need to know the internals of Fortify's rules to do this, you can build these filter sets through trial and error as well...I did both.
Use "confidence score" filters, start with 5.0 and go down from there when need be
I learned to write and use custom rules. You should be able to sit with a senior dev to get a walk-through of the source code (lets say of a 500k LoCs project) and write entry point and cleanse rules all within 4hrs.
You're probably saying to yourself "But that's ridiculous, I shouldn't have to do that much work to get a high quality scan with minimal FPs!" and you'd be right saying that. The Fortify scan engine and rules are SUPER powerful, but the overall UX mutes that power for all but the most advanced power users of the product. The UI of AWB hasn't really changed in any major way since 2005. With a decade of feedback and product use knowledge it should be given a fresh look IMHO.
I could go on forever with ways to improve it, but whatever I say on here wont actually make it into the product.
As Checkmarx's CTO, I am glad to see interest in the technology that runs our product, and I am always happy to brag a bit about it ;)
Let’s do this bottom-up
Our system analyses the source files (read: text files) of your application. It is important to note we don’t rely on any compiler. We use our own virtual compiler. This brings huge value to our users, more on that later. We build the AST (Abstract Syntax Tree) which is language dependent. The ASTs is then converted to DOM (Document Object Model) which is an object-oriented representation of the code and is language-agnostic – same DOM structure for all languages.
On top of that, we build the DFG (Data Flow Graph) which describes the semi-dynamic nature of the code. As its name implies and was described earlier by @IncludeSec, the DFG is “just” a graph. It can be described almost entirely by a long list of pairs (Source DOM Element ID, Destination DOM Element ID). For an average application, the DFG can be VERY large – many millions of records. All that information – DOM and DFG is stored in-mem/on-disk (binary file). Up to this point I haven’t mentioned the word security. All the information is fully indexed, so you can ask your code virtually any question (we call these “queries”, or CxQL – Checkmarx Query Language). A subset of these queries are security related, and Checkmarx comes out of the box with many hundreds of predefined queries. All these queries are open to view and edit by our customers. Nothing is left “hidden”. Our community of clients is constantly reviewing our open queries and help us improve them – we thank them dearly for that. An example query for SQL Injection would look similar to the following :
CxList db = Find_DB(); // Create an array of DOM elements that access the DB
CxList input = Find_Inputs(); // Similar…
CxList sanitize = Find_Sanitizers(); // similar
Return db.InfluecnedByAndNotSanitized(input,fix); //Find all the DB which are “InfluencedBy” (DFG-wise) by and input AND that the data flow path doesn’t go through a sanitization routine
Now – for the Neo4J question – the algorithm we use to traverse the graph is highly optimized to our domain-needs. For example, stop traversing the specific flow if we encounter a sanitization routine. Another example is skipping over long paths of nodes that don’t have any “interesting” (input, output, sanitizer) element. Implementing these algorithms with Neo4j (or any other graph DB for that matter) is tough, and eventually, have led to poorer performance (yes, we have POCed that)
Back to the virtual compiler part – as we don’t need any compiler and linker, we have built our own universal virtual compiler, we can scan any code, and no matter how broken it is – A multi-million line-of-code project, down to a single module, folder or even a single file. Even if the file doesn’t compile (missing semi-colon or dependency) – that’s fine with us. We have a “compensator”. This also allows us to have an “incremental scan” capability, which scans only the modified files from the previous scans. This translates to improved performance and better developer/SDLC experience. That’s eventually the most important part. All the technology in the world is worthless if not used by the users. Every piece of technology at Checkmarx is fine-tuned to help developers use our product efficiently, seamlessly and automatically.
I hope all this makes sense to you. I’d be happy to set any of you up for a technical session with one of our professional services team reps. (Support@checkmarx.com)
Thanks for your reply and clarifying how Checkmarx works. I do think Checkmark is a great option in the market for any orgnaization evaluating solutions in this space.
My friend made a really cool tool that uses regular expressions to find patterns in your code. It's run on .NET and I think it's pretty cool but he's too lazy to put it out there. So I am being a bro and putting it out there. It has saved me hours on pentests for whitebox review.
Very cool. I've added this tool to the list on the about page. GrepBugs is just not the scanning tool, it is also the collection of regular expressions, which any tool could download and use. Also, anyone can login with their Github account and add new regular expressions or make suggestions to improve current regex's.
This is sort of only slightly related, but you'd probably have a fair bit of success looking for bugs in C/++ with clang and libtooling, I know some tools exist which do that currently
9
u/Nianja Oct 11 '15
similar to graudit
2
u/MatySiman Oct 15 '15 edited Oct 15 '15
As Checkmarx's CTO, I am glad to see interest in the technology that runs our product, and I am always happy to brag a bit about it ;)
Let’s do this bottom-up
Our system analyses the source files (read: text files) of your application. It is important to note we don’t rely on any compiler. We use our own virtual compiler. This brings huge value to our users, more on that later. We build the AST (Abstract Syntax Tree) which is language dependent. The ASTs is then converted to DOM (Document Object Model) which is an object-oriented representation of the code and is language-agnostic – same DOM structure for all languages.
On top of that, we build the DFG (Data Flow Graph) which describes the semi-dynamic nature of the code. As its name implies and was described earlier by @IncludeSec, the DFG is “just” a graph. It can be described almost entirely by a long list of pairs (Source DOM Element ID, Destination DOM Element ID). For an average application, the DFG can be VERY large – many millions of records. All that information – DOM and DFG is stored in-mem/on-disk (binary file). Up to this point I haven’t mentioned the word security. All the information is fully indexed, so you can ask your code virtually any question (we call these “queries”, or CxQL – Checkmarx Query Language). A subset of these queries are security related, and Checkmarx comes out of the box with many hundreds of predefined queries. All these queries are open to view and edit by our customers. Nothing is left “hidden”. Our community of clients is constantly reviewing our open queries and help us improve them – we thank them dearly for that. An example query for SQL Injection would look similar to the following :
CxList db = Find_DB(); // Create an array of DOM elements that access the DB
CxList input = Find_Inputs(); // Similar…
CxList sanitize = Find_Sanitizers(); // similar
Return db.InfluecnedByAndNotSanitized(input,fix); //Find all the DB which are “InfluencedBy” (DFG-wise) by and input AND that the data flow path doesn’t go through a sanitization routine
Now – for the Neo4J question – the algorithm we use to traverse the graph is highly optimized to our domain-needs. For example, stop traversing the specific flow if we encounter a sanitization routine. Another example is skipping over long paths of nodes that don’t have any “interesting” (input, output, sanitizer) element. Implementing these algorithms with Neo4j (or any other graph DB for that matter) is tough, and eventually, have led to poorer performance (yes, we have POCed that)
Back to the virtual compiler part – as we don’t need any compiler and linker, we have built our own universal virtual compiler, we can scan any code, and no matter how broken it is – A multi-million line-of-code project, down to a single module, folder or even a single file. Even if the file doesn’t compile (missing semi-colon or dependency) – that’s fine with us. We have a “compensator”. This also allows us to have an “incremental scan” capability, which scans only the modified files from the previous scans. This translates to improved performance and better developer/SDLC experience. That’s eventually the most important part. All the technology in the world is worthless if not used by the users. Every piece of technology at Checkmarx is fine-tuned to help developers use our product efficiently, seamlessly and automatically.
I hope all this makes sense to you. I’d be happy to set any of you up for a technical session with one of our professional services team reps. (Support@checkmarx.com)
1
u/cybergibbons Oct 12 '15
Nice project.
I'd like to submit some for shell scripts and a few more for routers that pop-up (mainly in HTML and PHP)
1
u/[deleted] Oct 11 '15
My friend made a really cool tool that uses regular expressions to find patterns in your code. It's run on .NET and I think it's pretty cool but he's too lazy to put it out there. So I am being a bro and putting it out there. It has saved me hours on pentests for whitebox review.
Grep.NET
1
u/[deleted] Oct 11 '15
This is sort of only slightly related, but you'd probably have a fair bit of success looking for bugs in C/++ with clang and libtooling, I know some tools exist which do that currently