BDD als Ansatz zum Automatisierten Testenvon Embedded GUIs und HMIs

Embedded Testing 2015

Reginald Stadlbauer

About me

Name: Reginald Stadlbauer Company: froglogic GmbH Position: co-founder and CEO Worked as Software Engineer at Trolltech and the KDE project

About froglogic

HQ: Hamburg Founded: 2003 US presence since 2008 Product focus on Squish- Squish GUI Tester (Cross-Platform/Cross-Technology GUI Test Automation)- Squish Coco (C, C++ and C# Code Coverage)

More than 3.000 customers world-wide

Overview

What is GUI testing Challenges for cross-platform and embedded Proposed methodology: BDD Live demo

What is GUI testing

Outside-In Assume user's point of view Automate to spot regressions Combine with profiling, coverage and other analysis and monitoring tools

Why Automate?

Faster- Get results quicker- Run more tests in the same time

Trivial to replay in different configurations Reliable, reproducible and repeatable Relieve testers from monotonous tasks

But...

- Automating GUI tests is not trivial- Typical reason for test effort failures: wrong test approach

Platform Challenge

UIs runs on:

Embedded Linux (several variants) Windows (CE, Embedded, RT) Android iOS QNX VxWorks Nucleus Desktop (Windows, Mac, Linux/Unix) for R&D

Toolkit Challenge

Those play a role:

Native (Android, iOS, ...) Qt QtQuick/QML JavaFx HTML5 ...

Most challenging: combinations of the above.

Platform Solution for Testing

Provided Abstractions by Test Tool

Resolution independence via dedicated UI control support UI abstractions Synchronization methods Distributed environment Reusable functions/objects

Virtualization

Target hardware- the real thing- limited number- harder to automate

Virtual systems- VMware, Virtual Box, qemu- emulator vs. simulator- easy replication, resets and automation- Simulation of hardware features, limitations and events.

Capture and replay

Produces massive test scripts Not readable Not maintainable No code re-use possible Brittle against changes in the UI

Solution: Scripting, Refactoring and Abstractions

Proposed Solution

BDD for test logic and enforcing abstraction Implementation: Specialized UI test driver- Scriptable- Supports used UI controls- Supports remote testing

What is BDD / BDT?

“BDD is a second-generation, outside-in, pull-based, multiple-stakeholder, multiple-scale, high-automation, agile methodology. It describes a cycle of interactions with well-defined outputs, resulting in the delivery of working, tested software that matters.” - Dan North

http://en.wikipedia.org/wiki/Behavior-driven_development

OR...

What is BDD / BDT?

Based on Test Driven Development- Write (failing) test- Implement feature until test passes- Unit-Test level granularity (inside-out)

But- Focus on application's behavior and specification- (Test) Description in a human-readable DSL (Gherkin)- Less focus on implementation details- Clear separation between Tests Logic (Feature Files) and Test Implementation- Allows non-programmers to create tests

Feature Partitioning

Product

Feature A Feature B

Feature C

Feature D Feature E

Components

Features ≠ Units

Feature A Feature B

Feature C

Feature D Feature E

Unit 1

Unit 2

Unit n

Scenarios

Feature A Feature B

Feature C

Feature D Feature E

Scenario D-1Scenario A-1 Scenario E-1

Scenario D-2

Versatile usage of Feature Files

User story / feature specification Communicate with customer / users Documentation of acceptance test Sequence to walk through for manual tests Storyboard for automation of tests

Feature: Aaaaa bbb Scenario: Bbbb ccc Given foo bar zzz And zilch zorro When I do something Then something should

Feature: Aaaaa bbb Scenario: Bbbb ccc Given foo bar zzz And zilch zorro When I do something Then something should

Feature: Aaaaa bbb Scenario: Bbbb ccc Given foo bar zzz And zilch zorro When I do something Then something should

Developer

Tester

Documentation

Customer,Business Owner

Analyst Developer,Tester

Why BDD/BDT

“Test first” development on a higher level Clearly separate test logic from implementation Allow non-programmers to define features & tests Have a common, single language two communicate

Example: Embedded Map App

A Feature File

Feature: Go to location

Scenario: Navigate to address Given the map application is running When I enter the address Alexanderstr 4, 10178 Berlin, Germany Then the address is found in the district Mitte

A Feature File (Step types)

Feature: Go to location

Scenario: Navigate to address Given the map application is running When I enter the address Alexanderstr 4, 10178 Berlin, Germany Then the address is found in the district Mitte

PreconditionActionsExpectations

What is BDD / BDT – The Process

Write a failingFeature Test

MakeFeature Test

pass

Implement/Refactor

Code

Test

Manually Automated

Automating a Behavior Driven Test

Requirements- BDT framework- GUI testing framework- Glue between both

BDT Framework – Generate Skeletons

Parse feature files Generate step definition skeletons (functions and annotations) in preferred language

Test.feature

Feature: Computation

Scenario: Add two numbers

Given the calc App is running

When I enter 8

....

Test.py

@Step("Given the calc App is running")

def step(context):

test.warning("Implement me”)

@Step("When I enter 8")

def step(context):

test.warning("Implement me”)

BDT Framework – Run Feature Files

Parse feature files Execute feature files by mapping to steps to step definitions (functions) Reporting

Test.feature

Feature: Computation

Scenario: Add two numbers

Given the calc App is running

When I enter 8

....

Test.py

@Step("Given the calc App is running")

def step(context):

[...]

@Step("When I enter 8")

def step(context):

[...]

Testing framework

Support the specific programming language Support the specific UI technology of AUT Scripting support / integration options Tooling for convenient test creation, maintenance and debugging

Test.py

@Step("Given the calc App is running")

def step(context):

startApplication("calc”)

@Step("When I enter 8")

def step(context):

click("8”)

BDT Frameworks

Cucumber JBehave Behave SpecFlow RSpec Lettuce Squish GUI Tester ...

GUI Testing Frameworks

Squish GUI Tester HP QTP Rational Functional Tester Selenium ...

Live Demo – QML Map App

Q & A

Questions? Email reggie@froglogic.com

Free and supported trial of Squish at http://www.froglogic.com/evaluate

About Squish GUI Tester

Cross-Platform / Cross-GUI-Technology Test Automation- Windows, Linux, Mac OS X, Unix, Embedded, RTOSes, Mobile- Qt/QML/QtQuick, Java (Swing/AWT, SWT/RCP, JavaFx), Web, MFC, WinForms, WPF, iOS, Cocoa, Carbon,

Android, Tk, Flex, …

Object-based GUI object identification Record & replay Powerful scripting (JavaScript, Python, Ruby, Tcl, Perl) Eclipse-based IDE Built-in BDD framework and support

Batch-testing via command-line tools Remote/distributed testing architecture Integrations: Microsoft ALM, HP QC/ALM, Rational RQM, Seapine TCM, SpiraTest, MKS, XStudio,

Jenkins, Hudson, TeamCity, Bamboo, Robot Framework, JUnit, Maven, …