Shift-Left Testing: QA in a DevOps World by David Laulusa

Shift-Left Testing QA in a DevOps World David Laulusa @dlaulusa Feb 25, 2020

About Me • Brigham Young University, BS/EE • Princeton University, MS/EE • 20+ yrs in industry • Development/QA • Startups, Large Enterprises, Permanent/Contract • Telecommute • DevOpsDays Columbus

Agenda Inject quality earlier in the process

You cannot inspect quality into a product. - Harold F. Dodge

Inspection does not improve the quality, nor guarantee quality. Inspection is too late. The quality, good or bad, is already in the product. - William Deming

Quality is not an act. It is a habit. - Aristotle

Agenda • Testing Principles • Testing Partners • Test

Continuous Delivery is the NEW Agile Satisfy the customer through early and continuous delivery of valuable software - Agile Manifesto

F1 Design Dev Test Deploy F2 Design Dev Test Deploy B1 Design Dev Test Deploy B2 Design Dev Test Deploy F3 Design Dev Test Deploy B3 Design Dev Test Deploy

Shift-Left Testing Static Unit Integration End-to-End Acceptance

Bugs are cheaper when caught young. - Larry Smith

Testing a System System Expected Outputs Inputs

Input Domain • The input domain contains all possible inputs. • Choose a finite set of values. • Include valid, invalid and special values • Include values that represent “normal use” (domain-specific knowledge) • Test selection is about sampling the input space in a cost-effective manner.

Equivalence Classes • All members of an equivalence class contribute to the fault detection in the same way. • The combination of all equivalence classes must cover the entire domain. • Equivalence classes must not overlap. • Select one value from each Equivalence Class. • Choosing more than one value does not increase fault detection.

Range • One class with values inside the range, and two with values outside the range. • If the range were 10 <= x <=20. 1.Equivalence Class 1: all numbers between 10 and 20, inclusive 2.Equivalence Class 2: all numbers less than 10 3.Equivalence Class 3: all numbers greater than 20 4.You could also do letters and special characters.

String • At least one containing all legal strings and one containing all illegal strings. • For example, let fname: string be a variable to denote a first name. 1. Valid name {Sue} 2. Special characters {ε} 3. Number {7} 4. Too long {Too many characters so it would not be able to fit into the database}

Enumeration • Each value in a separate class. • And then any other value. • For example, consider auto_color ∈ {red, green , blue}. • The following classes are generated, {{red}, {green}, {blue}, {blah}}

Array • One class containing all legal arrays, one containing only the empty array, and one containing arrays larger than the expected size. • For example, consider int[] aName = new int [3]. • The following classes are generated: 1. {[]} 2. {[-10, 20]} 3. {[-9, 0, 12, 15]}

Boundary-Value Analysis Programmers often make mistakes in processing values at and near the boundaries of equivalence classes. • min-1, min, min+1 • max-1, max, max+1 • 0, null, empty string

Combinatorial Testing • Many defects only occur when a combination of inputs or events occur that interact with each other. • However, if you were to test every combination of inputs it may take years especially with today’s complex systems. • How do you choose which combination of inputs? • Combinatorial testing is a technique for reducing the number of test cases without drastically compromising functional coverage, in order to get more ‘bang for the buck’.

Testing a System X (1,2) Y (Q,R) Z (5,6) System

Combinatorial Testing Where T = 2, Pairwise Testing Test Case X Y Z TC1 1 Q 5 TC2 1 R 5 TC3 1 Q 6 TC4 1 R 6 TC5 2 Q 5 TC6 2 R 5 TC7 2 Q 6 TC8 2 R 6

Test Case X Y Z TC1 1 Q 5 TC2 1 R 5 TC3 1 Q 6 TC4 1 R 6 TC5 2 Q 5 TC6 2 R 5 TC7 2 Q 6 TC8 2 R 6 Combinatorial Testing Where T = 2, Pairwise Testing

Combinatorial Testing Where T = 2, Pairwise Testing Test Case X Y Z TC1 1 Q 5 TC4 1 R 6 TC6 2 R 5 TC7 2 Q 6 TC8 2 R 6

Combinatorial Testing Where T = 2, Pairwise Testing Test Case X Y Z TC1 1 Q 5 TC4 1 R 6 TC6 2 R 5 TC7 2 Q 6

Combinatorial Testing Where T = 2, Pairwise Testing Test Case X Y Z TC1 1 Q 5 TC4 1 R 6 TC6 2 R 5 TC7 2 Q 6 • Going from 8 -> 4 isn’t that big a deal • Reducing # tests 50% is not that big a deal either • Reducing by hand is tedious

Android Options Parameter Values HardKeyboardHidden No, Undefined, Yes KeyboardHidden No, Undefined, Yes Keyboard 12Key, NoKeys, Qwerty, Undefined NavigationHidden No, Undefined, Yes Navigation Dpad, NoNav, Trackball, Undefined, Wheel Orientation Landscape, Portrait, Square, Undefined ScreenLayout_Long Mask, No, Undefined, Yes ScreenLayout_Size Large, Mask, Normal, Small, Undefined TouchScreen Finger, NoTouch, Stylus, Undefined 3*3*4*3*5*4*4*5*4 = 172,800 combinations

Combinatorial Testing to the Rescue T # Tests % of Total (172k) % Defects Found 2 29 0.02% 76% 3 137 0.08% 95% 4 625 0.4% 97% 5 2532 1.5% 99% 6 9168 5.3% 100% • The savings as a percentage of exhaustive testing depends on the number of parameters and the # of values for the parameters. • With larger systems the savings can be enormous

“There’s an App for that” • AllPairs (perl) • PICT (Microsoft) • ACTS (NIST)

When Not to Use Combinatorial Testing • When you really do have to test all combinations • This assumes a uniform distribution across all combinations. • Some combinations are more important than others. • Some combinations are impossible.

T-Way - Effective in Finding Those Pesky Bugs 0 10 20 30 40 50 60 70 80 90 100 1 2 3 4 5 6 # Interacting Parameters Leading to Failure Browser Bugs (194) Server Bugs (171)

Fly With Us – We found 90% of the defects!

Dependency Injection In software engineering, dependency injection is a software design pattern that implements inversion of control for resolving dependencies. A dependency is an object that can be used (a service). An injection is the passing of a dependency to a dependent object (a client) that would use it.

sub method { my $self = shift; my $obj = MyProject::Class->new(); my $x = $obj->get_something(); . . . return($x); }

sub method { my $self = shift; my $obj = shift; my $x = $obj->get_something(); . . . return($x); }

Grooming the Backlog • Make sure that there are sane completion requirements • Is it testable?

Why Do We Unit Test? static OSStatus SSLVerifySignedServerKeyExchange(SSLContext *ctx, bool isRsa, SSLBuffer signedParams, uint8_t *signature, UInt16 signatureLen) { OSStatus err; ... if ((err = SSLHashSHA1.update(&hashCtx, &serverRandom)) != 0) goto fail; if ((err = SSLHashSHA1.update(&hashCtx, &signedParams)) != 0) goto fail; goto fail; if ((err = SSLHashSHA1.final(&hashCtx, &hashOut)) != 0) goto fail; ... fail: SSLFreeBuffer(&signedHashes); SSLFreeBuffer(&hashCtx); return err; }

Why Do We Unit Test? • It gives you confidence that your code works as you expect it to work. • Easily refactor your code to make it cleaner and more efficient. • Saves time because unit tests help prevent regressions from being introduced and released. • Once a bug is found, you can write a unit test for it, you can fix the bug, and the bug can never make it to production again because the unit tests will catch it in the future. (shift left testing) • Implicit documentation

How Do We Unit Test? • Unit tests test units in isolation, in a dedicated, controlled environment. • Each unit test creates in own environment by instantiating only the classes needed to execute one test, putting them in a known state, then it invokes the method to be tested and verify the outcome. • This verification is done by assertions on the behavior of the method (as opposed to its implementation). • Performing the verification on the behavior and not on the implementation is important as this allows modifying the implementation without breaking the unit tests, and therefore using the unit tests as a safety net for the modification.

What Do We Unit Test? • Generally a method. • Need to write tests for the public methods. • Could write tests for private methods, but it means you might have to re-write these tests when you re-factor. • No need to write tests for trivial setter/getters.

Anatomy of a Unit Test • Set up all conditions for testing. • Call the method (or Trigger) being tested. • Verify that the results are correct. • Clean up modified records.

Unit Tests • Make each test orthogonal (i.e., independent) to all the others • Don’t make unnecessary assertions • Test only one code unit at a time • Mock out all external services and state • Avoid unnecessary preconditions • Unit tests should ideally take less than 5 min to run. If it approaches 10 min, then we can break them up and run sets in parallel. • Unit tests will run each time code is checked in. • Builds cannot progress unless all unit tests pass.

A Test is not a unit test if: • It talks to the database • It communicates across the network • It touches the file system • It can't run at the same time as any of your other unit tests • You have to do special things to your environment (such as editing config files) to run it.

Test-Driven Development • write a failing test • write just enough code to make it pass • refactor the code to clean it up

TDD • TDD is a design process, not a testing process. • Unit tests help to shape your design. • TDD is a robust way of designing software components (“units”) interactively so that their behavior is specified through unit tests. • TDD helps you to deliver software components that individually behave according to your design. • A suite of good unit tests is immensely valuable: it documents your design, and makes it easier to refactor and expand your code while retaining a clear overview of each component’s behavior.

Code Coverage • Code coverage is a measure used to describe the degree to which the source code of a program is executed when a particular test suite runs. • A program with high code coverage, measured as a percentage, has had more of its source code executed during testing which suggests it has a lower chance of containing undetected software bugs compared to a program with low code coverage.

Code Coverage • Function coverage • Statement coverage • Branch coverage • Conditional coverage

Code Coverage sub foo { my($x, $y) = @_; if ($x eq ‘string’ && $y < 10) { return 1; } else { return 0; } }

Profiling Your Code Profiling is a form of dynamic program analysis that can measure things like: • memory usage • time complexity of a program • usage of particular instructions • frequency and duration of function calls

Profiling Your Code • optimization • algorithm • loops • expensive function calls, including system calls • lazy building

Acceptance Testing - Example Test Specification: • Free shipping a month before Christmas

Test Code @Test public void orderLessThenAMonthBeforeChristmasShouldGiveZeroShipping() { Calendar purchaseDate = GregorianCalendar.getInstance(); purchaseDate.set(Calendar.YEAR, 2014); purchaseDate.set(Calendar.MONTH, Calendar.NOVEMBER); purchaseDate.set(Calendar.DAY_OF_MONTH, 24); int christmasEve = 24; int expectedShipping = 0; String expectedCurrency = “USD"; Date purchase = purchaseDate.getTime(); Shipping shipping = new Shipping(purchase, christmasEve); int actualCost = shipping.getCost(); assertThat(actualCost, is(expectedShipping)); }

Cucumber - Layers Execute Specification (Gherkin) Step Definitions - Glue (47 languages) System Under Test

Cucumber - Specification Scenario: Free shipping a month before Christmas Given a customer that Christmas is celebrated 24 of December When a customer buys a book on 2014-12-10 Then the shipping cost should be $0

Cucumber - Step Definitions @When("^a customer buys a book on (.*)$") public void buyBook(@Format("yyyy-MM-dd") Date purchaseDate) throws Throwable { this.purchaseDate = purchaseDate; }

Mutation Testing • White Box Testing • Mutate (change) certain statements in the source code and check if the test cases are able to find the errors. • This is called killing the mutant. Test suites are measured by the percentage of mutants that they kill. • New tests can be designed to kill additional mutants.

Other Types of Testing • Security Testing • Performance Testing • Compliance Testing • Resiliency Testing

PARTNER WITH EXTERNAL STAKEHOLDERS

Testing in Production • Logging • Server Monitoring • Application Performance Monitoring • Health Check

Burger World Burger: How Many?: Here: To Go: Lettuce: Tomato: Pickle: Onions: Mayo: Ketchup: Mustard: Instructions: Order Cancel

THANKS David Laulusa @dlaulusa

Shift-Left Testing: QA in a DevOps World by David Laulusa

More Related Content

What's hot

Similar to Shift-Left Testing: QA in a DevOps World by David Laulusa

More from QA or the Highway

Recently uploaded

Shift-Left Testing: QA in a DevOps World by David Laulusa