20060712 automated model based testing of community-driven open-source gui applications

20060712 automated model based testing of community-driven open-source gui applications

• 1.
  Automated Model-Based Testingof Community- Driven Open-Source GUI Applications Zheng-Wen Shen 2006/07/12 1
• 2.
  Reference • “Automated Model-BasedTesting of Community-Driven Open-Source GUI Applications” – Qing Xie and Atif Memon – 22nd International Conference on Software Maintenance (ICSM 2006), Philadelphia, PA, USA, Sep. 25-27, 2006. 2
• 3.
  Outline • 1. Introduction • 2. Testing Loops • 3. Overview of GUI Model • 4. Experiment • 5. Conclusions 3
• 4.
  1. Introduction (1/3) • Open-source software (OSS) development on world-wide web. (WWW) – Communities of programmers distributed world-wide – Unprecedented OSS code churn rates 4
• 5.
  Problems on OSS (2/3) • Little direct inter-developer communication – CVS commit log message, bug reports, change-requests, and comments • Developers work on loosely coupled parts of the application code. – Local change has inadvertently broken other parts of the overall software code 5
• 6.
  Key idea (3/3) • To create concentric testing loops Specific GUI testing goals, Comprehensive GUI testing resource usage, and target developer sub-groups for Smoke testing error reports Crash testing 6
• 7.
  2. Testing Loops •CR Tools Test cases  fragile – Input event sequence can no longer execute on the GUI – The expected output stored with the test case becomes obsolete. • Model-based techniques – Generate and maintain test cases automatically during OSS evolution – Employs GUI models to generate test cases. 7
• 8.
  8
• 9.
  3. GUI Model (1/5) • An event-flow graph (EFG) model represents all possible event sequences that may be executed on a GUI. – All executable paths of the software • Microsoft Word (total 4210 events) – 80 events open menus, 346 events open windows, 196 events close windows, and the remaining 3588 events interact with the underlying code. 9
• 10.
  Event-Interaction Graph (2/5) • Test interactions between loosely-coupled parts of an OSS • System-interaction events = Non- structural events + close windows events • Test cases consist of event-flow-paths that start and end with system-interaction events, without any intermediate system- interaction events. 10
• 11.
  Path in anEFG and EIG (3/5) 11
• 12.
  Test Case Generation (4/5) 1. Test cases are short  generate and execute very quickly 2. Only consists of system-interaction events 3. The expected state is stored only for system-interaction events 4. All system-interaction events are executed; most of the GUI’s functionality is covered. 5. Each test case is independent and the suite can be distributed. 12
• 13.
  Test Oracle Creation (5/5) • Oracles for Crash Tests – Crashes during test execution may be used to identify serious problems in the software. • Oracles for Smoke Tests – Software does what it was doing before modifications – Reference Testing • Oracles for Comprehensive Testing – Specifications-based approach – Precondition + Effect 13
• 14.
  4. Experiment • Dopopular web based community- driven GUI–based OSS have problems that can be detected by our automated techniques? • Do these problems persist across multiple versions of the OSS? 14
• 15.
  Notice • Execute fully-automaticcrash testing process on applications and report problems – A crash  An uncaught exception • Determine how long these problems have been in the application code • We overall process executed without any human intervention in 5-8 hours. 15
• 16.
  Subject Applications • FreeMind, a premier mind-mapping software – 0.0.2, 0.1.0, 0.4, 0.7.1, 0.8.0RC5, 0.8.0 • Gantt Project, a project scheduling application – 1.6, 19.11, 1.10.3, 1.11, 1.11.1, 2.pre1 • JMSN, a pure java Microsoft MSN messenger clone – 0.9a, 0.9.2, 0.9.7, 0.9.8b7, 0.9.9b1 • Crossword Sage, a tool for creating (and solving) professional looking crosswords with powerful word suggestion capabilities – 0.1, 0.2, 0.3.0, 0.3.1, 0.3.2, and 0.3.5 16
• 17.
  FreeMind Bugs 1. NullPointerExceptionwhen trying to open a non-existent file (0.0.2, 0.1.0) 2. FileNotFoundException when trying to save a file with a very long le name (0.0.2, 0.1.0, 0.4) 3. NullPointerException when clicking on some buttons on the main toolbar when no le is open (0.1.0); 4. NullPointerException when clicking on some menu items if no le is open (0.1.0, 0.4, 0.7.1, 0.8.0RC5) 5. NullPointerException when trying to save a “blank” file (0.1.0) 6. NullPointerException when adding a new node after toggling folded node (0.4) 7. FileNotFoundException when trying to import a nonexistent file (0.4, 0.7.1, 0.8.0RC5, 0.8.0) 8. FileNotFoundException when trying to export a file with a very long le name (0.7.1, 0.8.0RC5, 0.8.0) 9. NullPointerException when trying to split a node in “Edit a long node” window (0.7.1, 0.8.0RC5, 0.8.0) 10.NumberFormatException when setting non-numeric input while expecting a number in “preferences setting” window (0.8.0RC5, 0.8.0) 17
• 18.
  Gantt Project Bugs 1.NumberFormatException when setting non-numeric inputs while expecting a number in “New task” window (1.6) 2. FileNotFoundException when trying to open a nonexistent file (1.6) 3. FileNotFoundException when trying to save a file with a very long le name (1.6, 1.9.11, 1.10.3, 1.11, 1.11.1, 2.pre1) 4. NullPointerException after confirming any preferences setting (1.9.11) 5. NullPointerException when trying to save the content to a server (1.9.11) 6. NullPointerException when trying to import a nonexistent file (1.9.11, 1.10.3, 1.11, 1.11.1, 2.pre1) 7. InterruptedException when trying to open a new window (1.10.3) 8. Runtime error when trying to send e-mail (1.11, 1.11.1, 2.pre1) 18
• 19.
  JMSN Bugs 1. InvocationTargetExceptionwhen trying to refresh the buddy list (0.9a, 0.9.2) 2. FileNotFoundException when trying to submit a bug/request report because the submission page doesn't exist (0.9a, 0.9.2, 0.9.5, 0.9.7, 0.9.8b7, 0.9.9b2); 3. NullPointerException when trying to check the validity of the login data (0.9.7, 0.9.8b7, 0.9.9b2) 4. SocketException and NullPointerException when stopping a socket that has been started (0.9.8b7, 0.9.9b2) 19
• 20.
  Crossword Sage Bugs •NullPointerException in Crossword Builder when trying to delete a word (0.3.0, 0.3.1) • NullPointerException in Crossword Builder when trying to suggest a new word (0.3.0, 0.3.1, 0.3.2, 0.3.5) • NullPointerException in Crossword Builder when trying to write a clue for a word (0.3.0, 0.3.1, 0.3.2, 0.3.5) • NullPointerException when loading a new crossword file (0.3.5) • NullPointerException when splitting a word (0.3.5) • NullPointerException when publishing the crossword (0.3.5) 20
• 21.
  4.1 Results 21
• 22.
  Results • The somebug existed across applications  shared open-source GUI components (FileSave)  sanitize inputs • Many bugs are persistent across versions • There are fewer bugs in the first version than in later versions  consistent with our experience 22
• 23.
  the reasons forcrash 1. Invalid text input: validity, size 2. Widget enabled when it should be disabled 3. Object declared but not initialized 4. Obsolete external resources 23
• 24.
  5. Conclusions (1/2) • Recognition the nature of the WWW – enables the separation of GUI testing steps by level of automation, feedback, and resource utilization. • Demonstration that resources may be better utilized by defining a concentric loop-based GUI testing approach. • Demonstration that popular GUI-based OSS developed on the WWW have flaws that can be detected by our fully automated approach. 24
• 25.
  Conclusions (2/2) • A more detailed study of the overall benefits of this technique • Extended subject application  test TerpOffice incrementally • Web application that have complex back- ends • The interaction between three test loops – Whether one loop can benefit from the execution of the inner loops – Study the need of additional loops 25