How to Build Your Own Test Automation Framework?

How to Build Your Own Test Automation Framework? Dmitry Buzdin TAPOST 18 October 11th/12th

Working as a consultant • DevOps (Jenkins, Docker, Cloud etc) • Test Automation (TDD, BDD, ATDD, ?DD) • Software Architecture (aka Microservices) • Java/Web Development @buzdin About Trainer

Agenda • What is expected from a test framework? • What makes a test framework? • How to make one? • What should be considered?

Lets Create a Test Automation Framework!

Requirements • Minimal cost of ownership • No vendor lock-in • Large user base • Integration with existing tools • Longevity • Transparency

Assembling open source test framework based on Java

Framework vs Library • Frameworks have key distinguishing features that separate them from normal libraries: • inversion of control: In a framework, unlike in libraries or in standard user applications, the overall program's flow of control is not dictated by the caller, but by the framework. • extensibility: A user can extend the framework - usually by selective overriding; or programmers can add specialized user code to provide specific functionality. • non-modifiable framework code: The framework code, in general, is not supposed to be modified, while accepting user- implemented extensions. In other words, users can extend the framework, but should not modify its code. https://en.wikipedia.org/wiki/Software_framework

Where would you find   the building blocks?

VS How to know if the component is good or bad?

We need some architecture guidelines

ISTQB Generic Test Automation Architecture

Not sure if this is related to the real world automated testing?

Test Definition Layer Test Procedures Test Cases Test Conditions Test Data Test Library

Test Library • Collection of reusable elements to compose test cases • Directly related to business logic of SUT • The most valuable asset, which should outlive framework changes

Test Step @Step public void login(String username, String password) { // performs login } @Step public void navigateToPage(String page) { // navigates browser to specified page } Test Library == Test Steps

Test Step Benefits • Using minimal features of the actual framework • Independent from each other • Parametrised with inputs Test Framework should support technology agnostic test steps

Test Conditions • Test conditions are methods to check expectations • Important selection factors: • Possibility to write/find custom assertions • Clarity of failure messages • Additional assertion strategies Test Conditions == Assertions

Fluent Assertions http://joel-costigliola.github.io/assertj/ Readable assertions with auto-completion support

Power Assertions available in Groovy

Soft Assertions @org.junit.Rule public final JUnitSoftAssertions softly; @Step public void ensureAllFieldsAreDisabledForUser() { … softly.assertThat(user.getText()).isEqualTo("user"); softly.assertThat(nameInput.isEnabled()).isFalse(); softly.assertThat(surnameInput.isEnabled()).isFalse(); softly.assertThat(submitButton.isEnabled()).isFalse(); softly.assertThat(logoutButton.isEnabled()).isTrue(); … } All assertion failures are recorded until end of the test

Assertion Groups assertAll("person", () -> assertEquals("John", person.getFirstName()), () -> assertEquals("Doe", person.getLastName()) ); All assertions in a group are evaluated at the same time. You will see all errors in output together. http://junit.org/junit5/

Test Data • Test Data is used to define test inputs for abstract test cases • Test Data can come from different sources and formats

JUnit 5 Data-Driven @ParameterizedTest @CsvFileSource(resources = "/data/users.csv") public void testWithParameters( String user, String password, Integer id) { logger.info("{}:{}:{}", user, password, id); }

Test Cases “A set of input values, execution preconditions, expected results and execution postconditions, developed for a particular objective or test condition, such as to exercise a particular program path or to verify compliance with a specific requirement.” http://glossary.istqb.org/search/test%20case

Test Case in Manual Testing No Test Step Expectation Data 1 User logs in navigated to home page username=“joe” 2 User selects item navigated to item page item=“123” 3 User clicks “Add to Cart” Shopping cart updated 4 User clicks “Go To Cart” navigated to shopping cart 5 User clicks “Checkout” Checkout screen opens

Automated Testing Where are all the test steps?

Proper Test Case @Test public void checkoutTest() { login(“joe”, “password”); assertOnPage(“/home”); viewItem(123); assertOnPage(“/item.123”); addToCart(); assertCartItems(1); goToCart(); … } Test Case contains: • test steps • assertions • test data

Abstract vs Concrete • Abstract (high-level) test cases • Concrete (low-level) test cases = Data-Driven • Relation between them should be kept Test Framework should distinguish between those @TestTemplate public void abstractTest(Data data) { }

Test Procedurs Defines how to select and run a set of test cases? Test Procedures == Test Suites

Testing Assets Test Plans / Test Suites Test Cases Test Steps Test Libraries Test Data Assertions

Test Language •You could write test cases in: •Gherkin •Java/Groovy •Excel •HTML •Markdown All of that could coexist in a single framework!

Gherkin Language Cross-platform specification Cucumber JVM works both with JUnit and TestNG Test Step Test Conditions Test Step Test Case

Plain Language @Test public void ensureAllFieldsAreDisabledForUser() { goToPage(“/login“); login(“janedoe”, “password”); assertPage(“/home”); logout(); } Same test in Java

Multi-Language Framework Gherkin Test Cases Test Steps Test Adapters JUnit Test Cases Cucumber Step Defs Reusing the same test steps, data and assertions

Test Execution Layer Test Execution Test Logging Test Reporting

Test Execution • API for Test Case creation • Extension model • Engine to run the tests • External API to IDE, Build tools and GUI

JUnit 5 Architecture Test Runner Test Engine Test Language

Test Engine Responsibilities • Provides API/Language for Test Definition • Discovers test cases • Knows how to run tests • Provide engine-specific extension points

Test Runner Responsibilities • Integrates with IDE, Build and others tools • Integrates with Test Engines • Runs tests according to test plan • Starts multiple engines if needed • Provides extension points

Test Logging • Recording test execution event for further analysis • Test logs are attached to test reports • Vital to decrease investigation time of failures

System.out.println( “HTTP Result for ” + req.getUrl() + “ is ” + resp.getStatus() ); This isn’t a good test logging strategy!

Logger Frameworks • Possibility to change level of detail (runtime) • Log message tagging • Configurable log message format private static final Logger logger = LoggerFactory.getLogger(“NAME”); logger.info(“Hello, logger!”);

Test Reporting • Has to make sense to your Product Owner • Technical drill-down should be available • Aid in decision making process

Typical Test Report This is a very poor test report

Open Questions • What feature failed? • Was this test failing recently? • Is this the only test with such exception? • Are there any screenshots? • Which test inputs were used? • Which environment?

Test Metadata @Epic("JIRA-123") @Story("JIRA-456") @Feature(“Login") @Description("hello") @Owner("dmitry.buzdin") @Severity(SeverityLevel.NORMAL) @Test public void allureTest() {…}

More Features • Test steps are auto discovered • Test data is captured • All attachments are placed to according places • Historical analysis: flakiness, screenshot diffs

Test Configuration • Loading environment configurations • Configuration profiles • Different configuration sources (json, yaml, xml, external system) • Configuring SUT before test run

OWNER Library http://owner.aeonbits.org/

Dependency Injection • Necessary because of all building blocks • Keeping intermediate results • Keeping virtual user sessions * same name for inversion of control

Dependency Scopes • Test run scope • Test suite scope • Test case scope • User session scope

Test Adaptation Layer GUI API Services Protocols Databases Simulators Emulators

Test Adaptation Layer • Means to interact with SUT • GUI: Web, Mobile, Desktop, Image Recognition • Protocols: HTTP, AMQP, SSH • Database connectors

Test Adapters • Adapters should be configurable • Adapters should be independent from other test framework components • Adapters should be easy to use and replace

Rest Assured http://rest-assured.io/ when(). get("/lotto/{id}", 5). then(). statusCode(200). body("lotto.lottoId", equalTo(5), "lotto.winners.winnerId", containsOnly(23, 54)); Adapter DSL + Inversion of Control + Assertions Easy to use, but hard to integrate with other tools

Sometimes,   simple HttpClient  is a better choice…

Pieces of the Puzzle Configuration Test Reporting Test Engine Assertions Logging Dependency Injection Test Language Test Adapters Test Runner Test Data

Sample Java Stack Component Technology Test Engine JUnit 5 Test Reporting Allure 2 Assertions AssertJ Conﬁgurations OWNER Test Language Cucumber-JVM Logging Logback Test Adapters Rest-assured, Selenide etc. Dependency Injection Guice Data-driven JUnit 5 + Extensions

How it all fits together? • Test Automator value is in knowing all pieces of puzzle and integrating those • Skills and experience required! • Plugins, plugins, plugins…

Tool Evaluation Checklist • Documentation quality • Number of extensions points • Number of plugins • StackOverflow responsiveness • Project statistics • Open standards

Open Standards / APIs • Gherkin language • WebDriver https://www.w3.org/TR/webdriver/ • JUnit 5 Platform • Allure Reporting Result Format

You have to pick and choose components in order to build your framework

October 26th Allure Meetup in Riga

How to Build Your Own Test Automation Framework?

How to Build Your Own Test Automation Framework?

More Related Content

What's hot

Similar to How to Build Your Own Test Automation Framework?

More from Dmitry Buzdin

Recently uploaded

How to Build Your Own Test Automation Framework?