Exploratory Testing: Make It Part of Your Test Strategy

Exploratory Testing: Make It Part of Your Test Strategy

• 1.
  BW12 Concurrent Session 11/11/15 4:15pm “Exploratory Testing: MakeIt Part of Your Test Strategy” Presented by: Kevin Dunne QA Symphony Brought to you by: 340 Corporate Way, Suite 300, Orange Park, FL 32073 888-268-8770 · 904-278-0524 · info@techwell.com · www.techwell.com
• 2.
  Kevin Dunne QA Symphony Asdirector of product strategy at QASymphony, Kevin Dunne works closely with customers and thought leaders to help produce the next generation of exploratory testing tools. Kevin is passionate about agile and specifically about exploratory testing and its ability to help foster cross-functional communication and collaboration between all areas of the engineering organization. In a previous role, he was responsible for customer success for QASymphony where he was able to participate in and observe several successful exploratory testing implementations. Prior to joining QASymphony, Kevin managed testing teams at Deloitte, working on projects for large Fortune 500 and government customers.
• 3.
  Exploratory Testing: MakeIt Part of Your Test Strategy Kevin Dunne Director of Product Strategy QASymphony Agenda 1.  What is exploratory testing (ET)? 2.  What is NOT ET? 3.  Why should we do ET? 4.  How can I integrate ET into my test strategy? 5.  ET best practices – do’s and don’ts
• 4.
  What is exploratorytesting? 1.  Parallel test planning, test design, and test execution 2.  Specific yet flexible 3.  Aligned towards investigation of potential opportunities 4.  Values depth and attention to detail during testing 5.  Fosters knowledge sharing and awareness Parallel Planning, Design & Execution Unlike traditional testing techniques, planning, design, and execution happen concurrently, allowing efficiencies of time as well as flexibility in approach Plan Design Execute Report Plan DesignExecute Report
• 5.
  Specific yet Flexible Exploratorytesting provides a specific lens through which to perform testing – whether that be a user persona, functionality, criteria (i.e. localization), etc. However, it allows testers to use the tool as an end user would, not necessarily as the product owner envisioned it Manual Scripted Testing I tested the application as the script prescribed Exploratory Testing I tested the application as the end user would Investigating Opportunities Exploratory testing rewards testers who identify unknown areas of “opportunity” within the application, as they are essential in maintaining a backlog of future test charters Manual Scripted Testing Exploratory Testing
• 6.
  Knowledge Sharing Exploratory testingrelies on knowledge sharing to reach full potential – developing testers who understand the impact of more areas of the application allows them to identify more areas of risk and opportunity Plan DesignExecute Report Transfer Learning Example Questions to Ask •  Have you seen this before? •  What am I not considering? •  Why would someone do this? •  How would you have tested this? What is Not Exploratory Testing 1.  Exploratory testing is NOT unstructured testing 2.  Exploratory testing is NOT the only form of testing 3.  Exploratory testing is NOT throwaway work 4.  Exploratory testing is NOT impossible in a regulated environment
• 7.
  ET is NOTUnstructured Testing While exploratory testing allows for flexibility in the exact path of the application that is tested, it is NOT unstructured, in that it still contains parameters such as: 1.  A goal of the exploration 2.  A log of the activity performed 3.  A lens through which the testing is performed (i.e. a user persona) Performing exploratory testing without involving some parameters such as the above allows a greater risk of unsuccessful implementation of exploratory testing ET is NOT the Only Form of Testing Exploratory testing is best suited as a complement to automated and manual scripted test cases. It can feed these types of testing to create greater depth in testing, and also to identify any potential gaps in coverage. Potential New Feature Testing Cycle Code Developer Unit Test Exploratory Testing Manual Scripted Test Automation Regression Test Exploratory Testing Feature “Delivered” “Let’s make sure this is worth writing scripts against yet” “Let’s make sure were still testing all aspects of this”
• 8.
  ET is NOTThrowaway Work Exploratory testing does NOT need to be extra work done on top of other testing methods – it can count on its own towards testing progress and coverage if properly accounted for. Some of the necessary information needed to manage it: 1.  Charter 2.  Session Sheet 3.  Oral report 4.  Debrief 5.  Data Files 6.  Logs "Any testing approach is manageable when you choose to manage it.” – Michael Bolton http://www.developsense.com/blog/2010/01/exploratory-testing-is-accountable/ ET is NOT Impossible in a Regulated Environment Contrary to rumor and popular belief, exploratory testing is not only allowed in most regulated environments, it is also essential.
• 9.
  Why Should wedo ET? A 2007 controlled study found that: •  Testing with test cases vs. exploratory testing take almost 7 times longer, due to the amount of time needed to write the tests and report results on them •  Testing with test cases vs. exploratory testing finds more defects, and does not miss many (if any) critical or severe defects in comparison to test case testing •  Testing with test cases causes more false defect reports vs. exploratory testing Study link: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.167.3696&rep=rep1&type=pdf Adding ET to your Test Strategy 1.  Paired Testing – get quick feedback from product experts early in the development cycle 2.  Team Based ET – rally the entire team around product learning and exploration 3.  UAT – focus valuable SME time on value added activity rather than documentation 4.  Beta Testing – maximize the returns of beta testers, minimize duplication and uncovered areas 5.  Replacing Traditional Testing – shift “stale” manual scripted tests to drive net new exploration and feedback
• 10.
  Traditional Problems inSiloed Testing Unsure if problem is code or requirements •  Testing is removed from development and requirements management Bottlenecks in handoffs between test and dev •  Submitting defects, assisting in defect reproduction, notifying tester of defect ready for retest Too Much Time Passing Between Dev and test •  Developer is notified of bugs in their code days after testing has been completed Benefits of Paired Testing Close integration of development and test •  Minimize the risk of miscommunication and incorrect interpretation of requirements Immediate feedback on new code •  Code is tested right away, meaning developers have less time to continue to build on bad code that will need to be rewritten Fix Problems Immediately and Move On •  Reduce the number of defects opened, decreasing the number of issues that would make it into production
• 11.
  Team Based ET Exploratorytesting is something that the whole team can benefit from: 1.  Easy to Learn – don’t need to be proficient in many tools, languages, etc. 2.  Benefits from multiple perspectives and viewpoints 3.  Quick to start up and scale – reduced overhead to get process set up and running NOTE: just because some team members are not experienced testers, that does not mean that you throw the basics of exploratory testing out the window! UAT with ET UAT Challenge ET Benefit UATer’s are unfamiliar test case syntax and need continual clarification Allow UATer’s to perform the business flows they know well without test scripts UATer’s are not trained on test case management, automation tools, etc. Focus UATer’s time on learning how to document proper defects, reduce time to ramp UATer’s have a shorter attention span – they are not used to testing 6-8 hrs. per day Allow UATer’s to veer off the rails from time to time and investigate areas of interest UATer’s have a short period of time in which to provide feedback Ensure that as much of the UATer’s time as possible is dedicated to ET
• 12.
  Beta Testing ChallengesSolved Through ET Problem #1 - Even the most efficient beta testing shops rarely get feedback from more than 30% of their users – meaning that 70% of the beta testers provide no feedback Solution – provide specific ET charters to beta testers. Charters will keep the testers focused on key objectives and will drive accountability that will increase participation Problem #2 – There is a segment of use cases are either under or over tested, leaving bugs undiscovered in production Solution – prioritize particular features and use intelligent assignment of the specific charters to make sure adequate coverage is planned across appropriate environments and devices Traditional Beta vs. ET Beta Testing Traditional Beta ET Beta Where should I focus testing? At the users discretion, where they’d like According to their assigned charter How many times will this feature get tested? We don’t really know As many times as it is assigned out to users Will this feature get tested on all environments? We don’t really know We will assign it to environments needing coverage Are testers focusing their efforts on new features being released? We don’t really know Yes, assuming we assign them to work in those particular areas
• 13.
  Replace Traditional Testingwith ET Workplace choice improves the employee experience, and adding exploratory testing to the mix allows testers to have choice many times per day Replace Traditional Testing with ET There are typically two strategies, which can be used in conjunction to begin replacing manual scripted tests with exploratory ones: 1.  Look for tests that have resulted in the lowest failure rates, lowest defect detection rates, or both. In priority order, transfer these tests to exploratory test charters, and monitor the defect detection rates from the transition. 2.  Look for tests that take the longest to run, and are run the most frequently. In priority, transfer these tests to exploratory charters, monitor the time per execution, and ensure that defect detection rates stay constant or improve.
• 14.
  How to beststructure our exploratory testing? Introducing exploratory testing within a framework will greatly increase the odds of success, and will reduce fear and uncertainty among the practitioners as well as executives. Session Based Test Management is a popular framework for this, as it tracks all the important data on testing: More info on SBTM: http://www.satisfice.com/articles/sbtm.pdf •  Session charter (includes a mission statement, and areas to be tested) •  Testers involved Date and time executed •  Task breakdown •  Data files •  Test notes •  Issues •  Bugs Resources/Thought Leaders •  James Bach - http://www.satisfice.com/ •  Jonathan Bach - https://jonbox.wordpress.com/ •  Michael Bolton - http://www.developsense.com/ •  Paul Holland - http://testingthoughts.com/blog/author/ testthought •  Keith Klain - http://qualityremarks.com/ •  Brian Osman - https://bjosman.wordpress.com/
• 15.
  Questions? Kevin Dunne kevindunne@qasymphony.com Twitter: @kevindunneQA Linkedin:www.linkedin.com/in/kevindunneQA Blog: http://www.qasymphony.com/blog/