3.
SOFTWARE TEST TYPES AND
LEVELS
Table of Contents
Test Levels
Component Testing (Unit testing)
Integration Testing
System Testing
Acceptance Testing
Table of Contents (2)
Test Types
Risk-Based Testing
Functional Testing
Non-functional Testing
Structural Testing
Testing Related to Changes:
Re-testing and Regression Testing
Maintenance Testing
Component Testing
Main Terms
Component testing
Testing separate components of the software
Software units (components)
Modules, units, programs, functions
Classes – in Object Oriented Programming
Respective tests are called:
Module, unit, program or class tests
In practice usually involves the programmer who wrote the code
Defects are typically fixed as soon as they are found
without formally recording the incidents found
Units vs. Components
Unit
The smallest compilable component
Component
A unit is a component
The integration of one or more components is a component
Test Objects
Individual testing
Components are tested individually
Isolated from all other software components
Isolation
Prevents external influences on the components
Component test checks aspects internal to the
component
Interaction with neighbors is not performed
Component Testing Helpers
Stubs
In Component testing called components are replaced
with stubs, simulators, or trusted components
A skeletal or special-purpose implementation of a software
component, used to develop or test a component that calls or is
otherwise dependent on it. It replaces a called component.
Drivers
Calling components are replaced with drivers or trusted super-
components
A software component or test tool that replaces a component that
takes care of the control and/or the calling of a component or system.
Integration Testing
Integration
Composing units to form larger structural units and
subsystems
Done by developers, testers, or special integration
teams
Supposes that components are already tested
individually
Levels of Integration Testing
Component integration testing
Expose defects in the interfaces and interaction between
integrated components
Also called
○ “Integration test in the small”
System integration testing
Testing the integration of systems and packages
Testing interfaces to external organizations
Also called
○ “Integration test in the large”
Off-the-shelf Products
Standard, existing components used with some
modification
Usually not subject of component testing
Must be tested for integration
Why Integration Testing?
After assembling the components new fault may occur
Testing must confirm that all components collaborate
correctly
The main goal - exposing faults
In the interfaces
In the interaction between integrated components
Some Typical Problems
Wrong interface formats
Incompatible interface formats
Wrong files format
Typical faults in data exchange
Syntactically wrong or no data
Different interpretation of received data
Timing problems
Integration Approaches
There are different approaches for integration testing
The Big Bang approach
○ all components or systems are integrated simultaneously
○ The main disadvantage: difficult to trace the cause of failures
The incremental approach
○ The main disadvantage: time-consuming
Incremental Approaches
The Top-Down approach
The high level logic and flow are tested first - the low level
components are tested later
The Bottom-Up approach
Opposite to the Top-Down approach
The main disadvantage - the high level or the most complex
functionalities are tested late
System Testing
Why System Testing
Previous tests were done against technical
specifications
The system test
Looks at the system from another perspective
○ Of the customer
○ Of the future user
Many functions and system characteristics result from
the interaction of all system components
Test Environment
System testing requires specific test environment
Hardware
System software
Device driver software
Networks
External systems
Test Environment (2)
A common mistake is testing in the customer’s
operational environment
Failures may cause damage to the system
No control on the environment
○ Parallel processes may influence
○ The test can hardly be reproduced
Common Problems
Unclear or missing system requirements
Missing specification of the system's correct behavior
Missed decisions
Not reviewed and not approved requirements
Project failure possible
Realization might turn to be in the wrong direction
Acceptance Testing
The Main Idea
The focus is on the customer's perspective and
judgment
Especially for customer specific software
The customer is actually involved
The only test he can understand
Might have the main responsibility
Performed in a customer’s like environment
As similar as possible to the target environment
New issues may occur
Forms of Acceptance Testing
Typical aspects of acceptance testing:
Contract fulfillment verification
User acceptance testing
Operational (acceptance) testing
Field test (alpha and beta testing)
Contract Fulfillment Verification
Testing according to the contract
Is the development / service contract accomplished
Is the software free of (major) deficiencies
Acceptance criteria
Determined in the development contract
Any regulations that must be adhered to
○ Governmental, legal, or safety regulations
User Acceptance Testing
The client might not be the user
Every user group must be involved
Different user groups may have different expectations
Rejection even by a single user group may be problematic
Acceptance In Advance
Acceptance tests can be executed within lower test
levels
During integration
○ E.g. a commercial off-the-shelf software
During component testing
○ For component’s usability
Before system testing
○ Using a prototype
○ For new functionality
Operational (Acceptance) Testing
Acceptance by the system administrators
Testing backup/restore cycles
Disaster recovery
User management
Maintenance tasks
Security vulnerabilities
Field Testing
Software may be run on many environments
All variations cannot be represented in a test
Testing with representative customers
Alpha testing
○ Takes place at the developer's site
○ A cross-section of potential users and members of the developer's organization are
invited
○ Developers observe the users and note problems
Beta testing
○ Sends the system to a cross-section of users who install it and
use it under real-world working conditions
○ The users send records of incidents with the system to the development organization
Summary
Component testing - The testing of individual software components.
Integration testing - Testing performed to expose defects in the
interfaces and in the interactions between integrated components or
systems.
System testing - The process of testing an integrated system to
verify that it meets specified requirements.
Acceptance testing - Formal testing with respect to user needs,
requirements, and business processes conducted to determine
whether or not a system satisfies the acceptance criteria and to
enable the user, customers or other authorized entity to determine
whether or not to accept the system.
Test Types
Risk-Based Testing
Risk
Risk
The possibility of a negative or undesirable outcome or event
Any problem that may occur
○ Would decrease perceptions of product quality or project
success
Types of Risk
Two main types of risk are concerned
Product (quality) risks
○ The primary effect of a potential problem is on the product
quality
Project (planning) risks
○ The primary effect is on the project success
Levels of Risk
Not all risks are equal in importance
Factors for classifying the level of risk:
Likelihood of the problem occurring
○ Arises from technical considerations
○ E.g. programming languages used, bandwidth of connections,
etc.
Impact of the problem in case it occurs
○ Arises from business considerations
○ E.g. financial loss, number of users affected, etc.
35
Levels of Risk - Chart
Prioritization of Effort
Effort is allocated proportionally to the level of risk
The more important risks are tested first !!!
Product Risks:
What to Think About
Which functions and attributes are critical (for the
success of the product)?
How visible is a problem in a function or attribute?
(For customers, users, people outside)
How often is a function used?
Can we do without?
Functional Testing
Functional Testing
Functional testing verifies the system's input–output
behavior
Black box testing methods are used
The test bases are the functional requirements
Functional Requirements
They specify the behavior of the system
“What" the system must be able to do?
Define constraints on the system
Requirements Specifications
Functional requirements must be documented
Requirements management system
Text based Software Requirements Specification (SRS)
Example:
http://www.slideshare.net/KrishnasaiGudavalli/software
-requirements-specification-17173967
Requirements-based Testing
Requirements are used as the basis for testing
At least one test case for each requirement
Usually more than one is needed
Mainly used in:
System testing
Acceptance testing
Functional testing types
Smoke
Sanity
Regression
Re-testing
System testing - it’s functional part
Unit testing
UAT
Non-functional Testing
Testing the System Attributes
“How well" or with what quality the system should carry
out its function
Attributive characteristics:
Reliability
Usability
Efficiency
Testability of Requirements
Nonfunctional requirements are often not clearly
defined
How would you test:
“The system should be easy to operate”
“The system should be fast”
Requirements should be expressed in a testable way
Make sure every requirement is testable
○ Make it early in the development process
Nonfunctional Tests
Performance test
Processing speed and response time
Load test
Behavior in increasing system loads
○ Number of simultaneous users
○ Number of transactions
Stress test
Behavior when overloaded
Nonfunctional Tests (2)
Volume test
Behavior dependent on the amount of data
Testing of security
Against unauthorized access
Service attacks
Stability
Mean time between failures
Failure rate with a given user profile
Nonfunctional Tests (3)
Robustness test
Response
Examination of exception handling and recovery to errors
Compatibility and data conversion
Compatibility to given systems
Import/export of data
50
Nonfunctional Tests (4)
Different configurations of the system
Back-to-back testing
Usability test
Ease of learning the system
Ease and efficiency of operation
Understandability of the system
51
Structural Testing
Examining the Structure
Often referred to as ‘white-box’ or ‘glass-box’ testing
Uses information about the internal code structure or
architecture
Tools can be used to measure the code coverage of
elements, such as statements or decisions
53
Structure Testing Application
Mostly used for:
Component testing
Integration testing
Can also be applied at:
System integration
Acceptance testing
54
Testing Related to Changes:
Re-testing and Regression Testing
Re-testing
After a defect is detected and fixed, the software
should be re-tested
To confirm that the original defect has been successfully
removed
This is called confirmation
56
What is Regression Testing
Retest of a previously tested program
Needed after modifications of the program
Testing for newly introduced faults
As a result of the changes made to the system
May be performed at all test levels
57
Tests Reusability
Test cases, used in regression testing, run many times
They have to be well documented and reusable
Strong candidates for test automation
58
Volume of the Regression Test
How extensive a regression test should be?
There are a few levels of testing extensity:
1. Defect retest (confirmation testing)
○ Rerunning tests that have detected faults
2. Testing altered functionality
○ Only changed or corrected parts
59
Volume of the Regression Test (2)
There are a few levels of testing extensity:
3. Testing new functionality
○ Testing newly integrated program parts
4. Complete regression test
○ Testing the whole system
60
Unexpected Side Effects
The main trouble of software
The code complexity
Altered or new code parts may affect unchanged code
Testing only code, that is changed, is not enough
61
Complete Regression Test
The only way to be sure (as possible)
System environment changes
Also require regression testing
Could have effects on every part of the system
Too time consuming and costly
Not achievable in a reasonable cost
Impact analysis of the changes is needed
62
Maintenance Testing
What Do We Maintain?
Software does not wear out and tear
Some design faults already exist
Bugs are about to be revealed
A software project does not end with the first
deployment
Once installed, it will often be used for years or decades
It will be changed, updated, and extended many times
64
What Do We Maintain? (2)
New versions
Each time a correction is made - a new version of the original
product is created
Testing the changes can be difficult
Outdated or missing system specifications
65
Main Types Of Maintenance
Adaptive maintenance
Product is adapted to new operational conditions
Corrective maintenance
Defects being eliminated
66
Common Reasons For Maintenance
The system is run under new operating conditions
Not predictable and not planned
The customers express new wishes
Rarely occurring special cases
Not anticipated by design
New methods and classes need to be written
Rarely occurring crashes reported
67
Testing After Maintenance
Anything new or changed should be tested
Regression testing is required
The rest of the software should be tested for side effects
What if the system is unchanged?
Testing is needed even if only the environment is changed
The scope is related to the risk of the change, the size of the
existing system and to the size of the change
May be done at any or all test levels and for any or all test
types
Key activity – impact analysis
68
Triggers for Maintenance Testing
Planned modifications
Enhancement modifications (adapting software to the user's wishes, for
instance by supplying new functions or enhancing performance)
Changes of environment (planned operating system or database
upgrades)
Corrective and emergency changes
Migration (from one platform to another)
Should include operational test of the new environment
Retirement of a system
May include the testing of data migration or archiving if long data-
retention periods are required
Definitions
Maintenance testing - Testing the changes to an
operational system or the impact of a changed
environment to an operational system.
Maintainability testing - The process of testing to
determine the maintainability of a software product.
Links
http
://softwaretestingfundamentals.com/differences-betwee
n-black-box-testing-and-white-box-testing/
http://technologyconversations.com/2013/12/11/black-b
ox-vs-white-box-testing/
