Risk based regression testing approach

Risk based regression testing approach

• 1.
  By Asim Ali Presentedat Agile Testing Summit – Feb 21st 2017 in Charlotte, NC, USA www.testingsummit.com Pre-Requisite to Test Automation – Risk based Regression Testing Approach 1 2/21/2017
• 2.
  By Asim Ali Contents 2 RegressionTesting 1 What is Regression Testing? 2 Why Regression Testing is Needed 3 Challenges with Regression Testing and Possible Solutions 4 Regression Testing Categories High, Medium & Low 5 Building a Regression Test Suite 6 Maintaining a Regression Test Suite 7 Test Case selection approach for Regression Testing 8 Questions? Following are ………………………………
• 3.
  By Asim Ali Whatis Regression Testing? • Regression testing ensure that only code that meets specific quality standards is promoted to production or to next level of testing. • It is the repeated running of test cases to verify software modification has not broken previously functional code, ensuring that defects found are relevant to the new code. • Common methods of regression testing include rerunning previously run tests and checking whether program behavior has changed and whether previously fixed faults have re-emerged. • Regression tests include smoke tests, as well as additional tests that may verify specific functionality. Using a testing tool, these test details are run following each new release of code in the test environment. Any necessary rework to the code is performed, the code is re-released, and the regression tests are run until the code meets the expected results. 3
• 4.
  By Asim Ali4 Why Regression Testing is Needed?  When system code is changed as a result of: • Change requested by respective clients • System issues reported by client/participant/SBU • System maintenance  When one or more of the following have occurred: • More Functionality may be added to the system; • More complexity may be added to the system; • New bugs may be introduced; • New vulnerabilities may be introduced in the system; • System may tend to become more and more fragile with each change
• 5.
  By Asim Ali5 Expectations from Regression Testing  Regression Testing will attempt to verify that • The system works as specified even after the changes/additions/modifications were made to it • The original functionality continues to work as specified even after changes/additions/modification to the system • The changes/additions/modification to the system have not introduced any new bugs
• 6.
  By Asim Ali6 Challenges with Regression Testing & Possible Solution Challenges • Regression suites can often contain thousands and thousands of tests as they are built up over releases of software, especially when dealing with a legacy product. • At the end of each release, the size of the regression test suite will increase as well as the overall time to execute. Possible Solution – Create Regression Categories • In order to overcome with this challenge we can break the overall regression test suite into following categories: • High • Medium • Low • It is recommended to get a sign-off from BA’s, Users or SME’s to ensure that the grouping is correct. The candidates for each of the regression category will be updated regularly based on future releases.
• 7.
  By Asim Ali7 Regression Categories: High • Select around 10% of the test cases from overall regression suites • A small subset of test cases will cover basic End-to-End functionality for each of the features/flows in a given application. • High regression test cases are use during Quick Regression. They can be executed in a short duration and provide high degree of confidence in the overall quality of the software release. • Best practices on selecting High regression test cases: - Core functionality of the system (e.g., Rating, Web Quotes etc); - Areas that are highly visible to users (e.g., verification of content on the web, navigation, links etc.); - Areas where frequent defects were found in the past; - Areas which have undergone many/recent code changes;
• 8.
  By Asim Ali8 Regression Categories: Medium • Around 30% of the test cases from overall regression suite • The Medium group consists of test cases other than those selected for High regression. • They usually represent main & alternate flows as well as exception conditions for the application. The main focus is to select those test cases that have repeatedly found problems on previous releases. • Best practices on the selection of Medium Regression Test Cases from the overall Test Suite is to spread test cases selection across components/areas with the following types: • Negative Test Cases • Boundary Value Test Cases • Complex Test Cases • Some positive test cases
• 9.
  By Asim Ali9 Regression Categories: Low • Around 60% of the test cases from overall regression suite • The Low group consists of all the remaining test cases that are not candidates for High and Medium regression. • Low tests are executed in order to ensure completeness. We allow time for them in the schedule during a major release when High and Medium testing is complete.
• 10.
  By Asim Ali10 Building a Regression Test Suite 1/2 • Review all existing requirements that are impacted due to the project changes and identify test scenarios and test cases • Study historical test scenarios & test cases (if any) from any archive tool (excel sheets, word documents, test management tools such as HP Quality Center) and extract relevant test cases • Review the defect management tool and extract all defects relevant to the requirements being tested. Identify test cases that will test these defects. • Build a requirements traceability matrix for all test cases. • Hold discussions with BA’s / Users / SME’s to identify High, Medium & Low criticality repeatable end-to-end Business test cases / scenarios • Create a modular repository of these test cases, based on product category. Group test cases by Product and then Product Category to make it easy to be accessed when needed. Review Defects Review Requirements Identify Test Cases Identify data conditions Review & Approve Test Cases
• 11.
  By Asim Ali11 Building a Regression Test Suite 2/2 TC# TC Title Miscellane ous columns for TC Product Category High Medium Low TC_01 Title of the test case TC_01 DocProd Letters X TC_02 Title of the test case TC_02 DocProd Letters X TC_03 Title of the test case TC_03 DocProd Letters X TC_04 Title of the test case TC_04 DocProd Forms X TC_05 Title of the test case TC_05 DocProd Forms X TC_06 Title of the test case TC_06 DocProd E-Sign X TC_07 Title of the test case TC_07 PC Auto Rating X TC_08 Title of the test case TC_08 PC Auto Rating X TC_09 Title of the test case TC_09 PC Auto Quote Summary X A sample table is given below if Test Cases are written in Excel format (In HPQC a field ‘Regression Priority’ field with three choices of ‘High’, ‘Medium’ & ‘Low’ needs to be added):
• 12.
  By Asim Ali12 Maintaining a Regression Test Suite Review Defects Review Changes Identify Test Cases Identify data conditions Review & Approve Test Cases Review existing Test Cases • Review all existing requirements that are impacted due to the project changes and identify test scenarios and test cases • Study historical test scenarios & test cases (if any) from any archive tool (excel sheets, word documents, test management tools such as HP Quality Center) and extract relevant test cases • Review the defect management tool and extract all defects relevant to the requirements being tested. Identify test cases that will test these defects. • Build a requirements traceability matrix for all test cases. • Hold discussions with BA’s / Users / SME’s to identify High, Medium & Low criticality repeatable end-to-end Business test cases / scenarios • Create a modular repository of these test cases, based on product category. Group test cases by Product and then Product Category to make it easy to be accessed when needed.
• 13.
  Questions? @Copyright 2017: AsimAli About Speaker Asim Ali is a Thought Process Leader with 20 years of expertise in DevOps & Cloud Solution Architecture and Test Management. Experience working with top tier IT consulting companies such as IBM, PwC, CAPGEMINI, CGI Group, Xerox & NTT DATA. He holds MBA from Duke University, MEng in Computer Science from ENSIMAG, France and acquired certifications such as AWS Solutions Architect, PMP-Project Management Professional, CMST-Certified Manager of Software Testing, CMST- Certified Manager of Software Testing, ITIL, CCNA (Cisco Certified Network Associate) & MCSE (Microsoft Certified Systems Engineer). LinkedIn: https://www.linkedin.com/in/asimali1973 Email: asim_@Hotmail.com