Software development and testing go hand in hand. And in the era of agile software development, with quick releases of small iterations, you should do testing more and more frequently.

In order to perform effective testing, you need to know about the different types of testing and when you should use them.

In this article, I'll discuss some of the tests available to you to help you ensure the operability, integrity, and security of your products and apps.

The Software Testing Pyramid

The Software Testing Pyramid. Enjoy this free graphic and share it on your blog or Twitter.

The software testing pyramid covers all stages of the software development life cycle (SDLC). It extends from unit testing at the base, through to integration testing, and concludes with functional testing at the apex.

There is no set distribution among these types of testing. Instead, you should determine which tests best suit your individual needs. In order to make these decisions about the types of testing you need, you should balance their cost, how long they'll take, and how many resources they'll require.

Agile software developers also use software testing quadrants that categorize tests based on whether they are business-facing or technology-facing, and whether they critique the product or support the team.

Unit testing, for example, is a technology-facing test that supports the team, whereas usability testing is a business-facing test that critiques the product.

Let's go over some important types of testing now.

Unit Testing Definition

Unit testing involves testing individual code components rather than the code as a whole. It verifies the operation of all your component logic to identify bugs early in the SDLC, which allows you to correct errors before further development.

Unit testing is known as “white box” testing, because testing occurs with full knowledge of the application's structure and environment.

One example of unit testing is to create mock objects for testing sections of code, such as functions with variables that have not yet been made.

const mocha = require('mocha')
const chai = require('chai')  // It is an assertion library
describe('Test to check add function', function(){
  it('should add two numbers', function(){
    (add(2,3)).should.equal(5)  //Checking that 2+3 should equal 5 using the given add function
  });
});

Integration Testing Definition

A step up from unit testing is integration testing, which combines individual components and tests them as groups. Integration testing identifies issues in how the individual components interact with each other to see if the code meets all its functional specifications.

Integration testing differs from unit testing in that it focuses on modules and components working independently in relation to the overall group. On the other hand, unit testing focuses on isolating the modules or components before testing.

The point of integration testing is to expose any issues or vulnerabilities in the software between integrated modules or components.

As a more simplified example, if you were to perform an integration test of an email service you’re building, you would need to test the individual components such as Composing Mail, Saving Drafts, Sending, Moving to Inbox, Logging Out, and so on.

You would perform a unit test of the individual features first, followed with the integration test for each of the functions that are related.

End-to-end Testing Definition

At the top of the pyramid is end-to-end (E2E) testing. As its name suggests, end-to-end testing replicates the full operation of the application in order to test all of the application’s connections and dependencies. This includes network connectivity, database access, and external dependencies.

You conduct E2E testing in an environment that simulates the actual user environment.

You can determine the success of an E2E test using several metrics, including a Status of Test (to be tracked with a visual, such as a graph), and a Status and Report (which must display the execution status and any vulnerabilities or defects discovered).

Types of Software Testing

Within the levels of the testing pyramid are a wide variety of specific processes for testing various application functions and features, as well as application integrity and security.

Application Security Testing Definition

One of the most important types of testing for applications is application security testing. Security testing helps you identify application vulnerabilities that could be exploited by hackers and correct them before you release your product or app.

There are a range of application security tests available to you with different tests that are applicable at different parts of the software development life cycle.

You can find different types of application security testing at different levels of the testing pyramid. Each test has its own strengths and weaknesses. You should use the different types of testing together to ensure their overall integrity.

Static Application Security Testing (SAST) Definition

You should use static application security testing (SAST) early in the SDLC. This is an example of unit testing.

SAST reflects the developer’s knowledge, including the general design and implementation of the application, and it is therefore white box, or inside out, testing.

SAST analyzes the code itself rather than the final application, and you can run it without actually executing the code.

Image source

According to the security analysts at Cloud Defense,

“SAST checks your code for violation of security rules and compares the found vulnerabilities between the source and target branches...you'll then get notified if your project’s dependencies are affected by newly disclosed vulnerabilities.”

Once you're aware of vulnerabilities, you can resolve them before the final application build.

You should apply SAST in the development phase of your software projects. A good approach for you will be to design and write your applications to include SAST scans into your development workflow.

Dynamic Application Security Testing (DAST) Definition

On the other end of the spectrum is dynamic application security testing (DAST), which tests the fully compiled application. You design and run these tests without any knowledge of the underlying structures or code.

Because DAST applies the hacker’s perspective, it is known as black box, or outside in, testing.

DAST operates by attacking the running code and seeking to exploit potential vulnerabilities. DAST may employ such common attack techniques as cross-site scripting and SQL injection.

DAST is used late in the SDLC and is an example of integration security testing. While slow (a complete DAST test of a complete application can take five to seven days on average), it will reveal to you the most likely vulnerabilities in your applications that hackers would exploit.

Interactive Application Security Testing Definition

Interactive application security testing (IAST) is a newer testing methodology that combines the effectiveness of SAST and DAST while overcoming the issues associated with these more established tests.

IAST conducts continuous real-time scanning of an application for errors and vulnerabilities using an inserted monitoring agent. Even though IAST operates in a running application, it is considered an early SDLC test process.

Regardless of what type of software you’re looking to test, IAST is best used in a QA (Quality Assurance) environment, or an environment that is designed to replicate production as closely as possible without your clients or customers actually accessing it.

Compatibility Testing Definition

Compatibility testing assesses how your application operates and how secure it is on various devices and environments, including mobile devices and on different operating systems.

Compatibility testing can also assess whether a current version of software is compatible with other software versions. Version testing can be backward or forward facing.

Image Source

Examples of compatibility testing include:

• browser testing (checking to make sure your website or mobile site is fully compatible with different browsers)
• mobile testing (making sure your application is compatible with iOS and Android)
• or software testing (if you’re going to be creating multiple software applications that need to be interacting with one another, you’ll need to conduct compatibility testing to ensure that they actually do so).

Beyond the Software Testing Pyramid

Modified versions of the testing pyramid can include a level that's next to or above end-to-end testing. This level consists of tests focused on the application user.

Performance Testing Definition

You need to know how the application will work in a variety of different conditions, and this is the purpose of performance testing. Performance testing can model various loads and stresses to assess the robustness of the application. The type of performance testing is based on the applied conditions.

An example of performance sting is load testing, which determines the maximum load applied to the system at the time of a crash.

Another example like scalability testing, on the other hand, applies a gradually increasing load to the system to assess ways to accommodate the added system stresses.

And spike testing assesses the effect of applying sudden large load changes to the system.

You should conduct performance testing on any software system before you put it to market. Test it against stability, scalability, and speed so you can identify what to fix before going live.

Usability Testing Definition

Testing the actual use of the application interface is an important task. It is one thing to understand if the application functions as designed. It is another thing to understand if the design itself is acceptable to users. This is where usability testing comes in.

With usability testing, developers can assess user reactions to specific application features and functions. This includes features that you may know in advance will be less desirable from the user perspective but are necessary for strong security and proper operation (like strong password requirements).

Usability testing is not so much about cosmetic issues or fixing grammar errors in any written text (although both of those issues are certainly important in their own right). Instead it's about how easy the completed application is to use by the end user.

Conclusion

Testing is not just something the QA division should do after you have finished developing an application. It's also important part of the software development process.

Knowing what tests are available to you and how they work will help you ensure your application functions well, is secure, and is acceptable to the end user.

Over the years, I’ve sat through dozens of usability tests run by design agencies. Clients have asked me to oversee the tests to make sure that the agency really puts their design through its paces. This is a good thing as it shows that usability testing is now becoming a mainstream activity in the design community. But many of the usability tests I’ve sat through have been so poorly designed that it’s difficult to draw any meaningful conclusions from them. No wonder that Fast Company mistakenly believe that user centred design doesn’t work.

Picture a usability test

If I ask you to picture one of these usability tests, you’ll probably conjure an image of a participant behind a one-way mirror, with video cameras and screen recording software. Although your picture would be accurate, there’s something missing: the hand of an experimental psychologist (or an experienced user researcher) checking that other factors are in place behind the scenes. You need this guiding hand because the technology can often obscure the key goals and principles of usability testing. If I wear a white coat and dangle a stethoscope around my neck, it doesn’t make me a medic. Similarly, if I record a picture-in-picture video of someone using a web site, it doesn’t mean I’m running a usability test.

Here are 4 principles of usability testing that have been absent in many of the tests I’ve observed.

• Screen for behaviours not demographics.
• Test the red routes.
• Focus on what people do, not what they say.
• Don’t ask users to redesign the interface.

Screen for behaviours not demographics

Participants are often recruited using a demographic screener that focuses on aspects such as gender and age. To understand why this is a mistake, next time you come across a Norman Door — a door with a handle on it that screams ‘Pull!’ but the architect decided to make people push it — take a seat nearby and watch a handful of people use it.

You’ll find that you won’t need to observe many people before you see that there’s a problem with the design of the door. It won’t matter if the people you observe are men or women, young or old, tall or short — virtually everyone will experience this problem. The ones that don’t experience the problem have probably used the door before, and know what to expect.

This short observation tells us a lot about the criteria you should use when writing the perfect participant screener. Trying to balance gender, age and other demographic factors is not just impossible — because of the small sample sizes used in a usability test — but pointless. These factors have a negligible impact on the usability of a system (unless it’s aimed exclusively at women, or seniors).

Instead, recruit users based on their behaviour: people’s previous experience with the domain that you’re testing. If you screen participants based on their demographic characteristics you’ll end up compromising on the kinds of domain knowledge you need. You’ll end up with a demographically representative, but behaviourally biassed, sample.

Test the red routes

Whatever kind of usability test you run — moderated or unmoderated, remote or lab-based — they all share one feature: participants carry out tasks with the system. There are 6 main categories of usability test task but whatever kind you choose you need to make sure they focus on the red routes — tasks that are critical both to the user and the organisation.

In contrast, a scenario like ‘Have a look around the home page and tell me what you think’ — one that I’ve seen used more than once in recent usability tests — is one of a series of usability test tasks you should avoid. The only people who arrive at a home page, look around at the design, and pass judgement on it are other designers. Real people don’t behave that way because real people have specific objectives in mind when visiting a site and it’s those objectives you need to test your site against.

Focus on what people do, not what they say

Lab-based usability tests use a small number of participants: 5 is a common sample size and more than 20 is rare, except in unmoderated, remote usability tests. These small sample sizes work because usability tests focus on cognitive, problem solving behaviours and when it comes to how the brain works, people really aren’t that different from each other.

Problems occur when usability testers shoehorn other research objectives into their usability tests. For example, questions like, ‘How much would you pay for this service?’, ‘What kind of brand values does this site elicit?’ or ‘What are your feelings about this design?’ are totally meaningless when asked in the context of a usability test. You might as well ask people who they will vote for at the next election: you’ll get an answer but the small sample size means it won’t have any predictive value. A more subtle kind of bias occurs when you ask participants to introspect on why they did a particular behaviour. There’s no point asking ‘Why?’ in a usability test because people can’t reliably introspect into their motivations.

Don’t ask users to redesign the interface

When participants struggle with some aspect of the user interface, it’s very tempting to ask them how they would like it designed. But if design was that easy, we could all go home and leave it to participants to develop the next generation of user interfaces. So, after the participant has failed to choose the appropriate button, it’s frustrating to hear the moderator ask, ‘What would have made that button more obvious to you?’ The inevitable answer from the participant is, ‘Make it bigger,’ or even that old chestnut, ‘Make it a different colour.’

Asking people to redesign things introduces solutions into a process that’s designed to find problems. Just when you had them in ‘problem mode’, this line of questioning shifts their thinking. It tends to close down promising investigative threads and opportunities and you then risk not properly understanding the problems. Once users are shown a solution, or are asked to think of one for themselves, they get fixated on it and they struggle to think of anything else. The truth is that participants don’t know what’s possible, and asking them to generate design solutions will make them focus on the blindingly obvious or get them to focus on solutions that may work in one context but may not work in yours.

Conclusion: Avoid cargo cult usability testing

The physicist Richard Feynman once wrote about cargo cult science, where researchers adopt the paraphernalia of doing scientific activity but forget its core principles of empiricism, integrity and avoidance of bias. In the same way, people sometimes adopt the paraphernalia of usability testing, such as the one-way mirror and the video cameras, but forget the core principles of doing user research. Get those core principles right and you can run a great usability test with just a pencil and paper.

Originally published at www.userfocus.co.uk.