Maximizing UI Automation – A Case Study

Maximizing UI Automation – A Case Study

• 1.
  Maximizing UI Automation –A Case Study Noam Kfir | ALM Senior Consultant | Sela Group Copyright © Sela Group 2012
• 2.
  The Company  Hi-techsecurity industry  Hundreds of employees  Offices and customers all over the world  Customers include governments and large corporations Copyright © Sela Group 2012
• 3.
  Process Characteristics  Waterfallmethodology  10 years in development  30 programmers  12 manual testers  No automated tests  Very few automated processes Copyright © Sela Group 2012
• 4.
  Product Overview  Intelligence– gathering, managing, dispersing, analyzing  Mission-critical  More than 800 concurrent users  Real time client/server application  Many complex input forms and reports  Multi-lingual and bi-directional Copyright © Sela Group 2012
• 5.
  The Product isComplex  More than a million LOC  3rd party control libraries (Telerik, Janus, Infragistics, and more)  Very complex application flow, many screens  WinForms embedded in WPF  Clients communicate directly with the DB instead of via the server  Integration with other company products  Difficult to debug due to security considerations  The company cannot examine client data  Limited logging Copyright © Sela Group 2012 The Large Hadron Collider (LHC) is a particle accelerator located at CERN, near Geneva, Switzerland. It lies in a tunnel under France and Switzerland.
• 6.
  The Big Problem The end users report too many bugs  The product fails to handle the expected load  The client is dissatisfied and threatens to end the relationship  The company does not trust their product  Every QA cycle is very long (lots of “full regressions”)  The company reevaluates its quality control strategy Copyright © Sela Group 2012 Problem Solving We can’t solve problems by using the same kind of thinking we used when we created them…
• 7.
  Short Term Solutions On-site QA  On-site development  On-site fixes – not synchronized with source control  Lots of hacks and improvised solutions  Endless “lessons learned”  Off with their heads… Copyright © Sela Group 2012
• 8.
  Long Term Solutions Full transparency  Improve communication with the customer  Reports – more, richer, live  Improve and add automated builds  Gated check-ins  Nightly builds  Full builds  Automated deployment  Much more testing Copyright © Sela Group 2012
• 9.
  New Testing Strategy Unit testing  Use TFS lab environments  Easier to reproduce bugs  Snapshots  Traceability  Manual testing  End-to-end scenarios  Sanity and full regressions  Load testing – realistic conditions  Server API testing  Automated UI testing Copyright © Sela Group 2012
• 10.
  Attempt 1: CodedUI (2010) The Good  Intention-aware recording  Support for multiple UI technologies  Full TFS build integration including full support for lab environments  Extensible object recognition  Very low learning curve for programmers  Same language as the product – C# The Bad  Incomplete tooling  Object recognition needs improvement (Janus and pure graphical controls)  Very difficult for manual testers Copyright © Sela Group 2012
• 11.
  Coded UI (2010)ROI – After 2 Man-Months  Included with VS Ultimate  0 end-to-end tests  Excellent infrastructure Copyright © Sela Group 2012
• 12.
  Attempt 2: TestComplete Better object recognition  Designed for manual testers, good for building simple keyword-driven tests  Inconsistent object recognition  Very difficult scripting platform, weird language, very poor documentation  Not the same language as the product, C# possible with poor COM Interop implementation  Poor integration with TFS  Steep learning curve Copyright © Sela Group 2012 The Good The Bad
• 13.
  Test Complete ROI– After 2 Man-Months  $15,000 for several IDE licenses  $400 per build agent  30 end-to-end tests  No infrastructure or reusable scripts  Agents too expensive for load and stress testing Copyright © Sela Group 2012
• 14.
  Attempt 3: Ranorex Excellent object recognition  Designed for both testers and programmers, has IDE and .NET API  Partial TFS lab build process integration  Unlimited agents usable for load and stress tests  TFS integration needs some work  Tests need maintenance  Medium learning curve Copyright © Sela Group 2012 The Good The Bad
• 15.
  Ranorex ROI –After 2 Man-Months  $22,000 for 5 IDE licenses  Unlimited agents  60 end-to-end tests  Rich reusable infrastructure  After 3 man-months  120 end-to-end tests Copyright © Sela Group 2012
• 16.
  Comparison Matrix Coded UITestComplete Ranorex Target Programmers Testers Both Cost Part of VS Ultimate $15,000 $22,000 Agents Unlimited $400 per agent Unlimited Learning Curve Easy Difficult Medium TFS Integration Yes No Partial After 2 Man-months 0 tests (due to Janus) Good infrastructure 30 tests No infrastructure 60 tests Good infrastructure Copyright © Sela Group 2012
• 17.
  The Ranorex Solution Excellent object recognition for multiple UI technologies  Load and stress tests use same UI automation platform  Reporting under one umbrella  Bugs are caught much sooner  Better collaboration between testers and programmers  Customers can see and run the UI automation tests  Fewer defects and deployment errors in production  Fewer on-site team members and sessions  Adopted by other product teams Copyright © Sela Group 2012
• 18.
  DEMO UI Automation withRanorex Copyright © Sela Group 2012
• 19.
  Thank You! Email: noamkfir@sela.co.il Blog:http://noam.kfir.cc Twitter: @NoamKfir Copyright © Sela Group 2012