Downloaded 16 times
![“The number one reason people give us for not upgrading to the latest version of WordPress is fear that their plugins won’t be compatible.” [WordPress] 9](https://image.slidesharecdn.com/icse2012-testconfessions-astudyoftestingpracticesforplug-insystemspdfversion-120608020833-phpapp01/75/ICSE-2012-Test-Confessions-A-study-of-testing-practices-for-plug-in-systems-9-2048.jpg)
![“Thanks, but I think we have given up on Eclipse and Bugzilla integration.” [Eclipse – Bug ID: 268207] 10](https://image.slidesharecdn.com/icse2012-testconfessions-astudyoftestingpracticesforplug-insystemspdfversion-120608020833-phpapp01/75/ICSE-2012-Test-Confessions-A-study-of-testing-practices-for-plug-in-systems-10-2048.jpg)
![Other forms of testing are less popular “We think that with a high “QF-tests [were] too rigid when test coverage through unit the system was evolving” tests, integration tests are not necessary.” P20 P14 23](https://image.slidesharecdn.com/icse2012-testconfessions-astudyoftestingpracticesforplug-insystemspdfversion-120608020833-phpapp01/75/ICSE-2012-Test-Confessions-A-study-of-testing-practices-for-plug-in-systems-23-2048.jpg)
![Version testing is minimal “A lot of people put version ranges […], and they say they can run with 3.3 up to version 4.0 […].” P13 “But I’m willing to bet that 99% of the people do not test that their stuff works.” 28](https://image.slidesharecdn.com/icse2012-testconfessions-astudyoftestingpracticesforplug-insystemspdfversion-120608020833-phpapp01/75/ICSE-2012-Test-Confessions-A-study-of-testing-practices-for-plug-in-systems-28-2048.jpg)
![Testing not needed? “Simply fetch the latest and you'll end up in a mess!” [Ian Bull on updating problems with plug-in systems] 31](https://image.slidesharecdn.com/icse2012-testconfessions-astudyoftestingpracticesforplug-insystemspdfversion-120608020833-phpapp01/75/ICSE-2012-Test-Confessions-A-study-of-testing-practices-for-plug-in-systems-31-2048.jpg)
![Versioning, cross-plug-in testing not needed? “Sometimes we update, but there is always the risk that it will break something and then you have to do extensive [manual] testing.” P9 P12 “I do not even have a chance to test [all possible combinations]. There are too many operating systems, there are too many Eclipse versions.” 32](https://image.slidesharecdn.com/icse2012-testconfessions-astudyoftestingpracticesforplug-insystemspdfversion-120608020833-phpapp01/75/ICSE-2012-Test-Confessions-A-study-of-testing-practices-for-plug-in-systems-32-2048.jpg)
![Manual Testing “Tests that I do are very simple manual tests, the real tests are coming from the users, that are doing all kind of different things with [x].“—P9 38](https://image.slidesharecdn.com/icse2012-testconfessions-astudyoftestingpracticesforplug-insystemspdfversion-120608020833-phpapp01/75/ICSE-2012-Test-Confessions-A-study-of-testing-practices-for-plug-in-systems-38-2048.jpg)
![Developer Involvement “We’re a framework. If the user downloads a new version and lets his application run with it, then this is already like a test.” P20 P11 “Perhaps it is not our own product, but our product relies on this other product. So it is normal to improve [it].” 40](https://image.slidesharecdn.com/icse2012-testconfessions-astudyoftestingpracticesforplug-insystemspdfversion-120608020833-phpapp01/75/ICSE-2012-Test-Confessions-A-study-of-testing-practices-for-plug-in-systems-40-2048.jpg)
Uploaded byMichaela Greiler
ICSE 2012: Test Confessions - A study of testing practices for plug-in systems
1. The document summarizes a study on testing practices for plug-in systems. It examines how developers test plug-ins and the challenges of testing cross-plugin integration and different versions. 2. The study found that unit testing is widely used for plug-ins but other types of testing like integration and system testing are less common. Testing cross-plugin interactions and different versions is also limited. 3. Barriers to more rigorous plug-in testing include unclear responsibilities, lack of expertise, and long test times. Developers compensate by relying on community testing and involvement to identify issues.
ICSE 2012: Test Confessions - A study of testing practices for plug-in systems
- 1. Test Confessions: AStudy of Testing Practices for Plug-In Systems Michaela Greiler, Arie van Deursen and Margaret-Anne Storey 1
- 2. Testing Practices forPlug-In Systems? 2
- 3.
- 4. extending without the needof change assemble new products loaded at runtime 4
- 5.
- 6. complex compositions integrating multipleplug-ins different developers 6
- 7.
- 8.
- 9. “The number onereason people give us for not upgrading to the latest version of WordPress is fear that their plugins won’t be compatible.” [WordPress] 9
- 10. “Thanks, but Ithink we have given up on Eclipse and Bugzilla integration.” [Eclipse – Bug ID: 268207] 10
- 11.
- 12. How are plug-inspecific Which test practices are integration issues prevalent in testing of (i.e. versioning, configurations) plug-in-based systems? tested? Practices? Plug-in Testing? What are challenges experienced when testing plug-in based systems? Challenges? Research Questions 12
- 13. Survey of existingapproaches – over 200 resources LITERATURE STUDY? 13
- 14. A Grounded TheoryStudy Systematic procedure to discover a theory from (qualitative) data 14
- 15. Objective Increase understandingof what testers and developers think and do when it comes to testing plug-in-based systems. 15
- 16. Study Interviews with 25experienced Eclipse developers or testers representing well- known open and closed source projects. 16
- 17. Practices? Plug-in Testing? Are there compensation strategies to support the Challenges? testing of plug-ins? Compensation? Research Questions 17
- 18. Triangulation 1. Resonance @ EclipseCon 2. Survey among 151 developers 18
- 19.
- 20. Focus on UnitTesting 20
- 21. Focus on UnitTesting Focus on Unit Testing “Unit testing is where “Ultimately, unit test are our you find the most bugs” best friends” P18 P14 “At least 70% of our test effort is spent on unit testing.” 21
- 22. Integration 20 Tests 14 22
- 23. Other forms oftesting are less popular “We think that with a high “QF-tests [were] too rigid when test coverage through unit the system was evolving” tests, integration tests are not necessary.” P20 P14 23
- 24. 1. Finding: (Automated)unit testing is widely adopted; Integration, system, UI and acceptance testing are much less automated SURVEY 24
- 25.
- 26. Cross-plug-in integration? Testing platformor plug-in versioning? 26
- 27. Cross plug-in testingis optional “We handle problems between several plug-ins in a bug- driven way” P18 P19 “We have no automated tests for cross plug-in testing, but we do manual testing.” 27
- 28. Version testing isminimal “A lot of people put version ranges […], and they say they can run with 3.3 up to version 4.0 […].” P13 “But I’m willing to bet that 99% of the people do not test that their stuff works.” 28
- 29. Testing combinations orversions? 43% don’t test integration of different products only 3% test this thoroughly 55% don’t test for platform versions only 4% test this thoroughly 63% don’t test for dependency versions only 10% test this thoroughly 29
- 30.
- 31. Testing not needed? “Simplyfetch the latest and you'll end up in a mess!” [Ian Bull on updating problems with plug-in systems] 31
- 32. Versioning, cross-plug-in testingnot needed? “Sometimes we update, but there is always the risk that it will break something and then you have to do extensive [manual] testing.” P9 P12 “I do not even have a chance to test [all possible combinations]. There are too many operating systems, there are too many Eclipse versions.” 32
- 33.
- 34. Plug-In Integration TestingBarriers 34
- 35. Findings: Barriers • Responsibilityfor integration unclear • Lack of ownership • Insufficient plug-in testing knowledge • Test execution too long 35
- 36.
- 37. Self-Hosting “Eating your owndog food” 37
- 38. Manual Testing “Tests that I do are very simple manual tests, the real tests are coming from the users, that are doing all kind of different things with [x].“—P9 38
- 39. “Testing is doneby the user-community and they are rigorous about it. We have more than 10,000 installations per month. If there is a bug it gets reported P12 immediately.“ 39
- 40. Developer Involvement “We’rea framework. If the user downloads a new version and lets his application run with it, then this is already like a test.” P20 P11 “Perhaps it is not our own product, but our product relies on this other product. So it is normal to improve [it].” 40
- 41. A Prerequisite Openness Communication Release Management Extensibility Feedback Manual Testing Automated Testing Requirements Alpha & Beta Tester Downstream Projects & Release Train 41
- 42. Summary: Findings 1. (Automated)unit testing is widely adopted; Integration, system, UI and acceptance testing are much less automated 2. The plug-in nature has little direct impact on test practices 3. Barriers to adopt techniques include unclear ownership, responsibilities, and test effort & execution time 4. Limited integration testing is compensated by community 42
- 43. Plug-in specific testingsupport Provide a test modus Rewarding community Centralized compatibility information 43
- 44. More Details? Michaela Greiler,Arie van Deursen & Margaret-Anne Storey. “Test Confessions: A Study of Testing Practices for Plug-in Systems” Contact: m.s.greiler@tudelft.nl arie.vandeursen@tudelft.nl mstorey@uvic.ca 44