BDD Frameworks in focus

Hello Carsten and Christoph. You wrote an article titled "Not to be underestimated" about Behavior-Driven Frameworks for the October 2020 issue of iX.
What is the article about?

Carsten Weise: Behaviour Driven Development is a software development method in which requirements are specified semiformally as so-called behaviours. These can be used as acceptance criteria and are thus suitable for test guidance. BDD frameworks support the implementation of behaviors in automated test cases. In the article we give an overview of current BDD frameworks, e.g. Cucumber, Gauge, Catch2 and many others.

Christoph Singer: Since such a framework has to fit the technology used in the development of the software product, we cover a wide range of frameworks in the article: different operating systems, programming languages like Java, C++, Javascript, C#, Python, etc., and also different approaches to the specification of a Behaviour - especially Gherkin, but also e.g. the describe/it approach. -.

Carsten Weise: To help the reader choose the right tool, we evaluate tools based on a comprehensive set of criteria. These are criteria such as supported test objects (web applications, GUIs, APIs, etc.), the connection to development environments and also other details of the technical implementation. We explain in the article where we see the strengths and weaknesses of the frameworks. I can reveal at this point that we have found all frameworks useful. Christoph and I implemented the same example with all tools to get a feel for working with the frameworks.

Christoph Singer: It was important to us that we get to grips with each tool intensively by implementing a concrete example. This also enables us to make a statement about how quickly one can familiarize oneself with the respective tool and how good and helpful the documentation or other available help sources are. In our example, we have chosen the approach of a multi-layer architecture, which we have tested a lot. This also points out the advantages of separating specification and technical implementation. Should it be necessary to replace the automation tool due to technology changes, discontinuation of a tool or similar, the already formulated test cases can still be used.

Carsten Weise: Our goal in the article was also to make the reader understand that a framework is not a no-brainer: you have to get into the technical details of the framework in order to use it efficiently. To make the basic technical structure of the frameworks understandable, we related their architectures to the generic test automation architecture (gTAA), which is used e.g. in the ISTQB as part of the "Test Automation Developer" curriculum as a basis for test automation.

You just mentioned gherkin, what is that?

Carsten Weise: Gherkin is a method to specify Behaviours. You can call Gherkin a de-facto standard that is supported by almost all tools. In Gherkin, behaviors are described as naturally as possible with the help of keywords "Given" - the preconditions -, "When" - the trigger -, and "Then" - the expected result. Behind the keywords are descriptions of the steps needed to implement a Behaviour. The natural language description of these steps should be based on the user's logic, and should be easy for the user to understand.

For example, withdrawing money from an ATM could be described as a Behaviour in Gherkin like this:

Given an ATM showing a login screen
When I log in with my debit card
And enter an amount to withdraw from my account
Then the ATM pays out the desired amount

The steps in the Behaviour - e.g. "I log in with my debit card" - have to be translated into executable functions or scripts by a test automator. Then the BDD framework can be used to automatically test the Behaviour, ensuring that the applications meet their requirements.

Christoph Singer: Due to the different levels of abstraction of the layers, each of the people involved in the testing process can work within the domain specific language (DSL) that they are used to. Thus, a specialist tester describes the test cases in his familiar language and the test automation specialist takes over the technical implementation. This also counteracts the often existing bottleneck between many test cases to be automated and too few test automators.

What does the approach have to do with the agile environment?

Carsten Weise: In Agile, test automation is very important - due to the many iterations, the software product also goes through many changes, which means we need frequent regression testing in particular. You can only achieve the high beat rate that you aim for in Agile software development if you use test automation extensively. Behaviours are close to a natural language specification - turning a user story into a behaviour is just a small step.

Christoph Singer: Agile software development involves intensive collaboration between the development team and the customer. This requires a formulation of requirements that is understood by the programmers, by the testers, and of course especially by the users or customers of the software. BDD is, of course, ideally suited for this purpose. In addition, the agile principles aim, among other things, at fast, short release cycles and functioning software as the most important measure of progress of the current software status. By using a BDD test automation framework, already automated steps do not have to be reprogrammed, but can be reused. This makes it possible to implement automated test cases quickly and to provide developers with rapid feedback on the current status of the software and its functionality.

What does imbus do within the framework of BDD?

Christoph Singer: We also use the frameworks in our test automation projects. Here, we develop solutions tailored specifically to the customer and his infrastructure, i.e. we support the entire test automation process: from the creation of an automation strategy that fits the project, to tool evaluation with a pilot project to find the right framework, implementation of the framework, and roll-out, which also includes coaching, for example. We also offer the option of outsourcing the entire automation process to our imbus test center.

Carsten Weise: In our software testing training courses, e.g. our ISTQB Certified Tester training courses, we also address BDD as a development method and testing approach. In our ISTQB Test Automation Developer training, we deal with a generic architecture approach for test automation, which is an excellent fit for building BDD frameworks.

We thank you for the interview!

Further content:

Certified Tester trainings

Courses for test automation developers

imbus test automation

iX Heise Article