Agile Testing Cost Reduction using Pairwise Technique

19 - 20 August 2016 Bangalore www.xpconference.in

#XPIndia2016 Pairwise Testing in Agile World Classic Combinatorial Test case design technique that can be applied to agile testing to save testing cost & time.

#XPIndia2016 How many combinations does it take to test this?

#XPIndia2016 Four fundamental challenges of software testing Source : Cem Kaner • Complete testing is impossible • Testers misallocate resources because they fall for the company’s process myths • Test groups operate under multiple missions, often conflicting, rarely articulated • Test groups often lack skilled programmers, and a vision of appropriate projects that would keep programming testers challenged

#XPIndia2016 Complete testing is IMPOSSIBLE (Cem Kaner) • Test every possible input to every variable • Test every possible combination of inputs to every combination of variables • Test every possible sequence through the program • Test every hardware / software configuration, including configurations of servers not under your control • Test every way in which the user might try to use the program

#XPIndia2016 Motivation for Pairwise Testing • Wallace and Kuhn of NIST (National Institute of Standards & Technology) determined that 98% of the reported software defects in recalled medical devices could have been detected by testing all pairs of parameter settings. D.R. Wallace, D.R. Kuhn, “Failure Modes in Medical Device Software: An Analysis of 15 Years of Recall Data,” Intl. Journal of Reliability, Quality, and Safety Engineering, vol. 8, no. 4.

#XPIndia2016 Essence of Pairwise Testing Total Possible Configurations: 4 x 2 x 5 x 3 x 2 x 2 x 4 = 1920 Parameter Name Values No of Values Age 10, 14, 30, 50 4 Gender Male, Female 2 Certainty Probable, Doubtful, Possible, Established, Suspected 5 Severity Minor, Moderate, Major 3 Food True, False 2 Ethanol True, False 2 Medication COUMADIN 10 MG ORAL TABS, ASPIRIN 325 MG ORAL TABS, STENDRA TABS, ABILIFY TABS 4

#XPIndia2016 Software Failure Analysis  The simplest bugs in a system are generally triggered by a single input parameter E.g., failure occurs if • pressure < 10 (1-way interaction)  The next simplest category of bugs consists of those dependent on interactions between pairs of parameters E.g., failure occurs if • pressure < 10 & volume > 300 (2-way interaction)

#XPIndia2016 Software Failure Analysis  Bugs involving interactions between 3 or more parameters are progressively less common, while at the same time progressively more expensive  Most complex failure required 4-way interaction

#XPIndia2016 Software Failure Analysis – Medical Devices

#XPIndia2016 The Pairwise Technique Don’t test all combinations. Test all pairs.  Pairwise testing – an approach to combinatorial testing that executes a pairwise test data set.  Pairwise test data set - A set of test cases that covers all combinations of the selected test data values for every pair of a system’s input variables.  Pairwise testing is also known as 2-way testing; other approaches include 3-way, 4-way, … , n-way.

#XPIndia2016 An Example X={A, B} Y={C, D} Z={E, F} There are 2^3 = 8 Combinations System S

#XPIndia2016 All Combinations (Test Cases) Test ID X Y Z T1 A C E T2 A C F T3 A D E T4 A D F T5 B C E T6 B C F T7 B D E T8 B D F X={A, B} Y={C, D} Z={E, F}

#XPIndia2016 Pairwise Reductions by hand You can get rid of test case T2, since  AC is covered in T1  CF is covered in T6  AF is covered in T4 What other test cases can you get rid of?

#XPIndia2016 Pairwise Reductions by hand Test ID X Y Z T1 A C E T4 A D F T6 B C F T7 B D E

#XPIndia2016 Pairwise Reductions Number of inputs Number of selected test data values Total Number of combinations Size of pairwise test set 3 2 8 4 7 2 128 8 13 3 1594323 18 40 3 1.2 x 1019 21

#XPIndia2016 Pairwise Reduction - Tools

#XPIndia2016 TestCover  Commercial off the shelf tool  Algorithm based on Bell Labs research  Mathematically optimal among other tools  Determines test scenarios/cases based on input  Test Factor  Test Factor Values  Documented Constraints  https://testcover.com/sub/index.php

#XPIndia2016 TestCover Example Test Factor Number of Values Test Factor Values Operating system 4 XP, Vista, MacOS, Linux Display resolution 3 Low, medium, high Connection 4 wi-fi, dsl, cable, lan Browser 4 IE, Firefox, Opera, Safari Application 3 App1, App2, App3 Total Configurations = 576

#XPIndia2016 The TestCover Request Header Test Factor Test Factor Values Partition

#XPIndia2016 The Reduced Pairwise Subset

#XPIndia2016 Another Example – with Constraints Header Block Block Block Partition

#XPIndia2016 Caveat • Pairwise testing will not be effective if you choose the wrong input test data values • We often assume defects can be immediately observed. In fact, there may be slow internal corruption occurring • High-use (including defaults) or high-risk combinations probably don’t get enough attention Note: These dangers are not exclusive to pairwise testing. They affect all testing efforts

References  http://www.pairwise.org  https://testcover.com/sub/index.php  http://csrc.nist.gov/groups/SNS/acts/documents/verify07.pdf  http://www.bcs.org/upload/pdf/lcopeland-070312.pdf

Agile Testing Cost Reduction using Pairwise Technique

More Related Content

What's hot

Viewers also liked

Similar to Agile Testing Cost Reduction using Pairwise Technique

More from XP Conference India

Recently uploaded

Agile Testing Cost Reduction using Pairwise Technique