Downloaded 10 times
Uploaded byAnand Bagmar
Does your functional automation really add value?
The document discusses the value of functional automation in software testing, highlighting the importance of faster feedback, CI/CD enablement, and effective test execution strategies. It explores challenges in traditional automated testing frameworks and suggests a shift towards leveraging AI and ML in test automation. Key metrics for measuring product quality and test effectiveness are also outlined, encouraging a focus on meaningful metrics rather than merely fulfilling requests.
Does your functional automation really add value?
- 1. Does your functionalautomation really add value? @BagmarAnand Anand Bagmar Software Quality Evangelist
- 2.
- 3.
- 4.
- 5. • Faster feedback •Enabler of CI-CD • Work with interesting tools / technologies Value of Automation @BagmarAnand
- 6. Be a “developer” withoutthe pressure of sending code to production @BagmarAnand
- 7.
- 8. How do yousetup your test execution machine? @BagmarAnand Manual setup? Scripted setup
- 9. Do your testsrun in CI ? @BagmarAnand No Yes
- 10. If tests runin CI, do they run automatically? @BagmarAnand Manual trigger Auto triggers
- 11. How long doesit take to run your functional tests? @BagmarAnand > 30 min < 30 min
- 12. For multiple platforms,do you implement separate tests? @BagmarAnand Duplicated tests Single test
- 13. What is thetest passing percentage? @BagmarAnand Various reasons of failure Failures related to product issues
- 14. Do you automatically“rerun” the failing tests? @BagmarAnand Yes No
- 15. How do youdebug and get to the root cause of failures? @BagmarAnand Rerun test All info available to debug
- 16. Do you thinkabout Patterns / Code Quality / DRY principles / etc? @BagmarAnand What patterns? Yes
- 17. How do youvalidate the UI / UX as part of the e2e tests? @BagmarAnand Manual verification Intelligent tooling
- 18.
- 19. What is theway forward? @BagmarAnand
- 20. Bugs still escape @BagmarAnand Traditionalautomated testing frameworks are not built for modern apps
- 21.
- 22.
- 23. Bugs escape becauseour approach to testing is incorrect @BagmarAnand
- 24.
- 25. Test Automation Inthe world of AI & ML @BagmarAnand
- 26. @BagmarAnand Test Automation Inthe world of AI & ML https://www.infoq.com/articles/test-automation-ai-ml/
- 27. Unit (xUnit /JavaScript) Manual / Exploratory Web Service UI / e2e Business-facing Tests Technology-facing Tests @BagmarAnand Test Pyramid Slow More integration Expensive Fast More isolation Cheap
- 28. Unit (xUnit /JavaScript) Manual / Exploratory Web Service UI / e2e Business-facing Tests Technology-facing Tests @BagmarAnand Slow More integration Expensive Fast More isolation Cheap Workflow API Contract Consumer Provider Test Pyramid
- 29.
- 30. @BagmarAnand Design Patterns inAutomation
- 31. • Page-Object • BusinessLayer • Composition • Factory Method • Singleton -> DO NOT USE UNLESS YOU ARE VERY SURE Commonly used Design Patterns @BagmarAnand
- 32. • Page-Object • BusinessLayer • Composition • Factory Method • Singleton -> DO NOT USE UNLESS YOU ARE VERY SURE Commonly used Design Patterns @BagmarAnand
- 33. Architecture of e2e AutomationFramework @BagmarAnand
- 34. gradle / groovy ManageDevices Wifi Management Optimize Device Utilization Appium Server Management Test Specifications Business Rules (cucumber-jvm) Business Flows Implement Domain functionality (actions, verifications, rules) Resources Entities / Models Pages / Screens Trigger e2e Tests in Jenkins Server - Jenkins File - Node Management Archive artefacts - logs, screenshots, videos, logcat, Appium logs, etc - Cucumber reports ‘x’ Jenkins Agents on ‘y’ Mac Minis Helpers / Utilities (Android / iOS) Driver utilities Loggers File utilities Custom Reporter Screenshot utilities ADB utilities Video capture Analytics GPU Profiling Command Executor Log files from Devices Enrich Test Execution Report Manage Failing Tests Upload to TTA
- 35. How do youmeasure the quality of your product? @BagmarAnand
- 36. • Test CaseEffectiveness = (Number of defects detected / Number of test cases run) x 100 • Test Case Productivity = (Number of Test Cases / Efforts Spent for Test Case Preparation) • Test Coverage = Number of detected faults/number of predicted defects. • Test Code Coverage = Produce code coverage by e2e / UI tests • Requirement Coverage = (Number of requirements covered / Total number of requirements) x 100 • Test Design Coverage = (Total number of requirements mapped to test cases / Total number of requirements) x 100 • Test Execution Coverage = (Total number of executed test cases or scripts / Total number of test cases or scripts planned to be executed) x 100 • %ge of work completed & yet to be completed • Time to complete the remaining work @BagmarAnand (Traditional) Test Metrics
- 37. USE METRICS THATADD VALUE, NOT JUST BECAUSE SOMEONE IS ASKING FOR IT @BagmarAnand
- 38. • Does yourfunctional automation really add value? • https://essenceoftesting.blogspot.com/2020/07/does-your-functional- automation-really.html • Tracking functional coverage from your api / functional UI (e2e) tests • https://essenceoftesting.blogspot.com/2020/03/tracking-functional- coverage.html • Test Automation in the World of AI & ML • https://www.infoq.com/articles/test-automation-ai-ml/ Resources @BagmarAnand
- 39.