Hello,
it has been a while since I posted something (nothing unusual), but I really wanted to touch on a sensitive/controversial topic. Firstly, the blog just represent my personal opinion and not that of my employer, so do not draw any conclusions!
So, to start the debate, I have a question:
Do you expect a black-box pentest to find *all* vulnerabilities?
–
I will comment on this with mainly with a black-box app pentest in mind but the same logic apply to other forms of pentest too (in my opinion that is)
—
My thoughts: Any pentest vendor you choose, they would try to find as many vulnerabilities as they possibly can. Most security consultancy companies (at-least all good ones) follow a set methodology, some sort of check-list, a number of in-house /commercial tools and other related material to ensure 2 most important things :
1. consistency
2. coverage
Pentesters are not robots, they are humans and different people have different expertise and different skills and of-course some are more skilled in one area than others. The methodology, in-house/commercial tools, check-lists etc help achieve some level of consistency between various pentesters.
Time factor: Black Box pentests are usually a function of time. That is you only get a few days to assess the security of a particular application. New vulnerabilities/technology makes our job more complicated/interesting. Most tests nowadays are scoped based around clients budget they have for security testing and hence the scope is limited to find as many vulnerabilities as one possibly can in that amount of time. The more time you will allow to find vulnerabilities, the more likely it is that new vulnerabilities will be found.
What is a black box pentest: The nature of black-box pentest is such that it can never guarantee that there are no more security flaws other than those reported in the pentest report. For a more thorough assessment, source code auditing is recommended. Black box pentest is done primarily to provide assurance that if a reputed pentest vendor cannot find anything major wrong with the application’s security than its less likely to suffer from high risk issue(s). I can easily code up applications with critical security flaws which can only be identified when the source code is available and these are nearly impossible to find in a black box pentest.
The length vs the breadth: Should you find a high risk vulnerability, how much time can you spend in exploiting it. Should you exploit it? Of-course, you should safely exploit it to demonstrate the true impact of the vulnerability. Often exploitation of one vulnerability leads to discovery of another vulnerability. E.g. exploiting a vulnerable file upload functionality can give you access to source code and could lead to discovery of a SQL Injection issue. But exactly how much time can you spend exploiting it. Can you afford to miss a Local file include vulnerability because you spent too much time exploiting some other vulnerability? Thus, when a critical issue has been identified I would always recommend that the retest don’t just focus on the 1 issue but at-least some more testing is done to ensure that the test has received a decent coverage.
The out-of-box thinking: Pentest by nature involves creative thinking. The more familiarized the pentesters are with the application, the results will be so much better. Most pentesters would start a pentest by getting familiarized with the application. If I perform a pentest of the same application which i tested six months ago, then I would have already had an understanding of the application’s behavior/security/logic/input validation etc and I can then spend some time to come up with some new innovative hacks. But unfortunately, the way it works, If i do find a clever hack in the recent test than sadly, I will be asked the question “Why the Hell did I miss in the first pentest”…….
Technology and Vulnerability move on: Again, even if you have not changed any line of code in your application it does not mean that the pentest will not find anything new this time. E.g. the world did not know of the padding oracle attack till 2 or so years ago; but because I know it now, I will identify and report it now. It does not mean I missed it then..
Thus there are so many factors which one should consider when requesting a pentest and set expectations accordingly. Also the pentester needs to consider several factors to ensure that they provide the best possible result in the allocated time. Hopefully, this also highlights the need for regular pentest.
Hope my rant dont upset too many people. Would love to hear what the other guys from the industry think about this….
From the customer-side of a pen test (I’m the guy who hires pen testers in to test my apps) everything that you say makes sense. Anyone who thinks that a manual pen test (especially a time-boxed test) will find all of the real vulnerabilities in their app is hopelessly naive. What am I looking for in a pen test? A better understanding of risks and threats that we can feedback into how we design and develop software.
I understand that you can’t test everything and that how you test and how well you test will change over time as you learn more about the app and your tools and your craft. And I understand that the results of a pen test, like any test, depend on many factors like the approach that you take and the tools that you use or even how much sleep you’ve had lately, and on luck as well as skill and experience.
If you can find real exploitable bugs in my app in a few days then we are doing something wrong and we need to understand what this is and what we need to fix – in the design and code and in the way that we design, build, deploy and test software. I want to know how hard each bug was to find and how exploitable it was or how exploitable you think it would be if you had more time.
Sure, I want you to find as many bugs as you can so that we can fix them, but what I mostly want is information. I want help in understanding what you’ve found and where else I should look for other problems so that I know what I have to fix and so that I can fix the code and how we make it. I think that these are the expectations that you need to set – that your testing won’t be/can’t be exhaustive, that you are going to help identify areas of risk, and that it’s up to the development team to look deeper into what needs to be fixed: not just the code, but more careful design and programming, better tools and training, whatever.