Security bugs are fundamentally different than quality bugs

This topic has come up a few times this year in question period: arguments that quality bugs and security bugs ‘have equal value’, that security testing and QA are ‘the same thing’, that security testing should ‘just be performed by QA’ and that ‘there’s no specific skillset’ required to do security testing versus QA. This post will explain why I fundamentally disagree with all of those statements.

First some definitions.

A software bug is an error, flaw, failure or fault in a computer program or system that causes it to produce an incorrect or unexpected result, or to behave in unintended ways.

A security bug is specifically a bug that causes a vulnerability. A vulnerability is a weakness which can be exploited by a Threat Actor, such as an attacker, to perform unauthorized actions within a computer system.

QA looks for software bugs (any kind); security testers look for vulnerabilities. This is the main difference, their goals.

Just as all women are human beings, but not all human beings are women; while all security bugs are defects, not all defects are security bugs.

Now let’s dissect each of the claims above.

1. Quality bugs and security bugs ‘have equal value’.

If a security bug can lead to a low risk vulnerability, it does not have ‘the same value’ as a non-security-related bug that is making the system crash over and over. The same as if a security bug is creating a situation of a potential data breach, or worse, it’s not equivalent to the fonts not matching from page to page. I am of the opinion that security bugs are more likely to be able to cause catastrophic business harm than a regular bug, due to the fact that if your system has fallen under the control of a malicious actor, creativity is the only limit. Malicious actors never cease to amaze me with the damage they can do.

Someone is wearing camouflage. — #MSIgniteTheTour, Toronto, 2019

2. Security testing and QA are ‘the same thing’

The goals of security testing and quality assurance testing are different, which I feel makes them obviously different (if they were the same, why would they not be called the same thing?), however I want to get deeper into this idea.

Security is a part of quality.

I often say, “Security is a part of quality.”, because I believe this to be true. You cannot have a high-quality product that is insecure; it is an oxymoron. If an application is fast, beautiful and does everything the client asked for, but someone breaks into the first day that it is released, I don’t think you will find anyone willing to call it a high-quality application.

There are many different types of testing;

· Unit Testing — small automate-able tests to verify small units of code (functions/subroutines) to verify it does the one thing it is supposed to do.

· Integration Testing — test between different components to ensure they work well together. Larger than unit tests, but less intense than end to end tests.

· End-To-End Testing — ensuring the flow of your application from start to finish is as expected

· User Acceptance Testing (UAT) — manual and/or automated testing of client requirements (often used interchangeably with ‘QA’),

· User Experience Testing (UX) — verifying that the application or product is easy to use and understand from a user perspective)

· Regression Testing — verifying that new changes have not broken anything that was already tested, a ‘retesting’ of all previously released functionality

· Stress/Performance/Load Testing — verifying your application can handle large amounts of usage/traffic while continuing to perform well, generally performed using software tools (although each of these three have slight differences, they are all generally lumped in together)

· Security Testing — a mix of manual and automated testing, using one or more tools, with the aim to find vulnerabilities within applications.

There are more types of testing, but I think you get the point.

Some or all of these types of testing can be used to verify that a product is of high quality, and security is just one part. Therefor, security testing and QA are not ‘the same thing.’

3. Security testing ‘should be performed by QA’

For each one of the types of testing listed above a different skillset is required. All of them require patience, attention to detail, basic technical skills, and the ability to document what you have found in a way that the software developers will understand and be able to fix the issue(s). That is where the similarities end. Each one of these types of testing requires different experience, knowledge, and tools, often meaning you need to hire different resources to perform the different tasks. Also, we can’t concentrate on everything at once and still do a great job at each one of them.

Although theoretically you could find one person who is both skilled and experienced in all of these areas, it is rare, and that person would likely be costly to employ as a full-time resource. This is one reason that people hired for general software testing are not often also tasked with security testing. Another reason is that people who have the experience and skills in order to perform thorough and complete security testing are currently a rarity, there is a skill shortage, while as an industry we are lucky to have quite a number of skilled QA professionals, making them easier to hire and staff. Lastly, the amount of time, training and experience that it takes to become a security tester, versus a general software tester, is more difficult to acquire.

Training on how to perform security testing is extremely expensive and difficult to find, it generally takes longer to learn it as a skill than other types of testing, and there are fewer opportunities to get into that industry, when compared to QA. Thus it is more difficult to become a security tester, when compared to general testing. Scarce resources, high demand, expensive training means it costs significantly more to hire security testers than it does to hire general software testers.

All of these facts lead up to the reality that it is cost-prohibitive to have staff your QA team with professionals who are skilled and experienced in both QA and security testing. This also means that you are creating a single point of failure for testing in your organization, which will not save you money in the long run.

#MSIgniteTheTour, Toronto, 2019

Another point on this topic; those who work in the security industry are likely to have a preference for their area of focus, security, and may be unwilling to perform other types of work outside their area of concentration (people who specialize generally want to work within their area of specialization, whenever possible, and security testing is a specialization).

4. ‘There’s no specific skillset’ required to do security testing versus QA.

First of all, I feel this statement is insulting to QA testers, as though they do not have a specific skillset that makes them good at what they do. I don’t believe that to be true. I suspect that when people make this argument that it is out of frustration with our industry, because I honestly cannot fathom someone thinking that security testing does not require specific experience, training or skills; otherwise there would be no skills shortage and it would not be a high-paying job. Security testing is a specialization within the field of testing, just as there are specializations within any field, and by definition it requires more knowledge and training to form the skillset in order to do the job.

I do not intend to downplay the value of QA testing, only to explain that quality assurance is different from ensuring that a product is secure. I should also say that I feel that hacking is sometimes glorified in television, the media and our industry as a whole, in a way that isn’t logical to me. Security testing is very important, but I do not believe that hackers are superior to other professionals who work in IT. In fact, I choose to focus my career on AppSec, DevSecOps and other types of defence, because I truly believe that it is more important that we write secure code than we ‘hack all the things’. Security is so much more than just security testing (ethical hacking), it is secure design, secure coding, threat modelling, etc.

I feel comments like this (#4) are not based on facts, but feelings, and it’s difficult to debate with someone when that is the case.

It is okay if we disagree on this topic. Debate is good and healthy, and I would love to hear your feelings, thoughts and ideas in the comments.

At this point I’d like to remind you all that security is everybody’s job. Not only is it everyone’s responsibility to do their job in the most secure way they know how, but having many different people look at something with security in mind can help us find new and different problems that may have otherwise been missed.

For content like this and more, check out my book, Alice and Bob Learn Application Security and my online community, We Hack Purple!

VAs, Scans and PenTests; not the same thing

I’d like to define a couple of subjects that seem to be confused often in the industry of application security; Vulnerability Assessment (VA), Vulnerability Scan (VA Scan) and Penetration Test (PenTest). They are often used interchangeably, and the differences do not seem to be well-understood; I have seen this misunderstanding used against many clients who have purchased these services and am hoping clear definitions will help us all.

Vulnerability Assessment (VA) (sometimes called a security assessment) is an assessment of the security of a system, in attempts to find all possible vulnerabilities. It generally involves using multiple scanning tools, manual exploration and evaluation, as well as examination of all security controls (a lock on a door, a login screen, or input validation are all security controls). The Assessor does not exploit vulnerabilities that are found (for instance they see the door is unlocked, but they do not enter), they just report them, along with information on how to fix each of the vulnerabilities. This sometimes includes a security review of the design and/or threat modelling, questionnaires or interviews, and generally takes days or weeks, not hours or minutes. Sometimes the security assessor will create a proof of concept (POC) to explain a vulnerability with more clarity, but to be clear, that is not the focus of this exercise.

In the past when I was hired to do a penetration test, I would often describe a VA, as if that’s what they wanted, and they would say “yes, do that”. My contract would say “PenTest”, but I would conduct a vulnerability assessment.

In the past I often had requests for “a quick VA” or “VA Scan”, which as it turns out meant “one scan with a vulnerability assessment tool” and no other activities, such a manual investigation of the results. This can be done in as few as a few hours or even minutes if your target is small, and the person performing the task does not need advanced training or skills to perform the task. There are many VA tools on the market; Nessus, Nexpose, OpenVAs and Azure Security Center (for Azure cloud infrastructure only) are all used for scanning infrastructure, while Microsoft Security Risk Detection, Burp Suite, Zed Attack Proxy, NetSparker, Acunetix, AppScan, and App Spider are for scanning web apps. Doing “a quick scan” with any of these tools will net you a list of vulnerabilities, and many of them will be true positives (as opposed to false positives); it is most certainly a worthwhile venture. It is not, however, as thorough as a Vulnerability Assessment or Penetration Test, and there will remain many other issues that are not uncovered if you leave it at that.

I also enjoy infrastructure as code, from time to time

Penetration Test is another beast entirely. A PenTest seeks to find vulnerabilities and then exploit them, to prove real-world risk. Sometimes penetration tests can cause damage (exploits, if not done very carefully, can leave a mess), and sometimes the scope of a PenTest can call for the tester to collect “trophies” to prove they did the things they claim.

It is very rare that I write an exploit or feel the need to exploit vulnerabilities I find when testing*. Most of the times in my career when I have exploited something everyone just ended up pissed off at me; from the first PenTest I ever did as a sub-contract when I ruined a live prod server and the person that hired me had to explain what happened, to creating proof of concept exploits that embarrass management into doing “the right thing”, to breaking a Drupal CMS site so badly that they had to restore the database AND the app server (Drupal CMS itself was completely unusable) from backup. It’s nice that I impressed people, but I honestly would prefer to spend that extra time helping the developers fix what I have found and re-testing the fixes, rather than showing off whatever talent I have for burning things down.

Special note on ethics: I have seen many consultants who offer these services pass off a quick scan as a full VA or Pentest, charging for 10 days what took them only 1 day to perform. I have also seen many of these same consultants sub-contract this work out to others who they pay less (and with who they share your sensitive data with!), but they do not credit these individuals in the reports or contracts resulting in you having no idea who had access to your systems and data. When writing contracts for such services it would be wise to be explicit in what you are paying for, as well as who will do the work and what information must remain confidential. I am sad to report that I have met many consultants who have bragged about doing these types of (in my opinion) unethical practices. Buyer beware.

I would suggest that performing a proper VA against all of your custom applications as well as large COTS implementations (Customizable Off The Shelf system, such as SharePoint) is a best practice for Enterprise businesses. Not only would you be amazed at the things that you find, (assuming you fix the issues) you will have taken serious measures to avoiding a data breach in the future, as insecure software is still, sadly, the top reason for data breaches (as per the Verizon Breach Reports 2016, 2017, and 2018).

I hope this article helps instill a bit of clarity in our industry.

When I did testing, I did exploit XSS using alert boxes, regularly, because it’s 100% safe to do so. And also blind SQL with timers and errors, but to be clear I am very careful to only perform safe exploits when testing. I can feel myself putting my foot right into my mouth with this note…

For content like this and more, check out my book, Alice and Bob Learn Application Security and my online community, We Hack Purple!