Questions for successful test automation projects

Questions for successful test automation projects

• 1.
  QUESTIONS FOR SUCCESSFUL TESTAUTOMATION PROJECTS Daniel Ionita SEETEST 2016
• 3.
  WHAT IS THEMISSION OF THE PROJECT? • Think BIG, take a more strategical approach • Relate to it with clear and measurable objectives • Involve appropriate stakeholders, including sponsors and beneficiaries Remember as how software changes so the mission may change and your test automation must adapt!
• 4.
  WHY ARE WENEEDING TEST AUTOMATION? • This is related to the quality expectations and analysis of existing testing and that translates into many questions: • What is the current status of your testing? • Can you make your testing scalable? • Can testing keep up with development? • How fast do you want to deliver? • How often do you want to have a specific coverage of testing performed? … and many more
• 5.
  SHOW ME WHY? ROIROE ROI = (CTM- CTA)/ CTA*100 Cost of manual testing (CTM) = Cost of manually testing as many times as the automated test will be executed Cost of test automation(CTA) = Cost of hardware + Cost of software + Cost of developing the test automation + Cost of executing test automation+ Cost of maintaining the test automation ROE= (Efficiency gain – Investment)/Investment*100 Efficiency gain through test automation = Manual test execution and analysis time of tests that will be automated Investment = Test automation development, execution, analysis and maintenance time
• 6.
  WHO WILL DOTHE AUTOMATION? • Take in consideration • Software testing skills • Software development skills • Knowledge of test automation scripting/programming language • Knowledge of the application under test • Knowledge of the business
• 7.
  WHAT TYPE OFTESTING DO YOU TARGET WITH TEST AUTOMATION? • Unit testing • Regression Testing • Performance Testing • Exploratory Testing(mostly to support it) • Random Testing • Security Testing • Usability Testing
• 8.
  WHAT TYPE OFAUTOMATION? • Unit tests layer – testing the code • Service layer – testing the services of the application independent from its user interface • UI – testing the application through UI
• 9.
  WHAT TO AUTOMATE? •M - must have this – critical to the current delivery • S - should have this if all possible – important but not necessary for delivery • C - could have this but not necessary – desirable but not necessary • W - won’t have this - dropped or reconsidered for inclusion • Other factors: • what takes the most time regarding test execution • priorities of tests • coverage expected (breadth or depth) • tests/requirements age • what are the most tedious and repetitive tests …..
• 10.
  WHAT IS THETEST AUTOMATION INPUT AND OUTPUT DESIRED? Test design Test artifacts Hardware Software Interaction with AUT Traceability Auditing Risk information Artifacts to support manual testing
• 11.
  WHAT IS THECOVERAGE EXPECTED FOR THE TEST ENVIRONMENT? • Hardware • Compiler • OS • Databases • Browser • Communication protocols • File system …
• 12.
  WHAT TYPE OFFEEDBACK IS NEEDED? • Goal is to offer the fastest possible feedback to the project stakeholders • The information should drive actions • The information needs to target a diverse audience • Reporting • focus on test automation results • communication channels • visualization of test results • Metrics(do not use the metrics as objectives) • costs of test automation • effectiveness of test automation
• 13.
  REPORTING • Information usefulfor test automation engineers • highlighting exceptions and providing alerts • troubleshooting data results at granular and macro level • encouraging actions to improve effectiveness and efficiency • supporting maintenance of tests • Information useful for managers • aggregated info supporting decision making • highlight risks especially at product level • integration data analysis( defect tracker, test case management, coverage …) • Information useful for software test engineers • highlight risks at granular and macro level for tests and products
• 14.
  METRICS • A usefulmetrics should be accurate and related to your goals, objectives. • “Don’t measure anything unless data helps you make a better decision or change your actions.” Seth Godin • Test automation metrics • ROI/ROE • Test automation coverage • Test automation flakiness • Mean time of investigation and Resolve the problem • Cycle time for test automation • Project metrics that may be impacted by test automation • Defect removal efficiency • Customer defect trends post-release • Lead time Do not use the metrics as targets and evaluating performance, use them to for continuous improvement.
• 15.
  HOW? • The W?questions should drive the How? Cone of uncertainty Programming process pyramid Steve McConnell Code Complete, Second Editionhttp://www.construx.com/
• 16.
  METHOD? • A goodmethod as for putting everything together is ATAM(Architecture Tradeoff Analysis Method) • In order to proceed with the architecture analysis and design we need to focus on the following main things: architectural significant requirements, constraints and understanding different architectural tactics. • The W? questions will give you the business drivers, architectural significant requirements and constraints. • The main principle of the method is that is a risk- mitigation process focusing on finding the trade-offs and sensitivity points to drive you in choosing the suitable architecture.
• 17.
  ANALYSIS? • Architectures aremostly driven or shaped by quality attributes requirements. • The quality attributes used in general for ATAM are the following: availability, interoperability,modifiability,performance,security,t estability,usability. • Based on the business drivers, architecturally significant requirements and this quality attributes a utility tree will be built as to generate concrete scenarios. Those will be prioritized from the business and architectural impact to drive the architectural design analysis. • As for that analysis you will need to know architectural approaches, architectural tactics, design patterns. Utility Tree
• 18.
  TEST AUTOMATION FRAMEWORKS? •A test automation framework is an integrated system that sets the rules of automation of a specific product. • It is defined as the set of assumptions, concepts, and practices that constitute a work platform or support for automated testing. • Linear – basic record and playback approach • Modular – abstraction or encapsulation of scripts • Data driven – externalization of test data from scripts • Keyword driven – application-independent automation • Hybrid – combination of several techniques
• 19.
  ADVANTAGES AND DISADVANTAGES? Test frameworkAdvantages Disadvantages Linear test framework Low technical knowledge Requires little training Ease of automation Short setup time Brittle tests Recording is usually repeated for each test Not respecting software engineering standards Modular test framework Reducing the number of test scripts Greater flexibility and higher scalability Easier maintenance Requires technical expertise Increased time spent for automation Data driven test framework model Reduced maintenance Less amount of code written Low data redundancy Requires expertise in handling test tool and test data files Maintenance of test data Keyword driven test model Less technical expertise in automating test cases Low maintenance cost Portable architecture between applications and tools Extensive technical and tool skills knowledge for framework development High cost of implementation Hybrid test Framework Minimized maintenance Takes advantages of the other frameworks strengths High complexity High technical expertise