How do we test an application manually

5 things you should know about regression testing

Why regression tests are not technical tests

The purpose and benefit of a technical test is to uncover technical errors in the code. After the test has been carried out, we can be reasonably certain that the functionality that is checked in the test has also been implemented accordingly. Technical tests are absolutely necessary if a functionality has been added or changed. But are professional tests also good regression tests?

A regression test has a different objective than a technical test. In contrast to a technical test, a regression test does not want to find any errors! If the code contains a technical error, but the software is already being used by the customer, it is often the case that the error must not be corrected. Errors had to be built in again after they were simply fixed without consulting the customer ...

But if a regression test finds no errors, what does it do? The goal of a regression test is technical consistency. The regression test ensures that after an adjustment, the unchanged parts of the software still function as they did before the adjustment - regardless of their technical correctness.

You probably know the saying: If the only tool you have is a hammer, everything looks like a nail. The fact that we use technical tests as regression tests is solely due to the fact that we have no tool support for what we actually need: consistency checks. Technical tests "fix" the behavior of the application. This is good if you don't want the application's behavior to change. In fact, the behavior of the application should change constantly. And it is precisely then that these "fixed" tests quickly turn from help into ballast. Which in turn is exactly the reason for the shape of the well-known "test pyramid".

What we actually need

In order to recognize technical regressions after a change, we do not actually need any technical tests. It is completely sufficient to recognize changes - similar to a version control system. To accept the current state as correct or at least as given and to define your tests based on it is not a new finding. Michael Feathers describes this approach in his book "Working effectively with legacy code" and calls it the "characterization test" [2]. However, this pure "recognition of changes" is not enough. What makes a version control system so effective is the ability to simply "commit" these changes. And this is exactly what we need for our regression test system.

The end of Git

When reviewing a code change, the developer has to evaluate several aspects: the code quality, the changes to the structure of the code ... but also the effects on the behavior of the software. That, in turn, is a mental transfer task. And practice shows that people regularly fail at this. Well-known version control systems such as Git and SVN are great tools for recognizing changes at file level and thus managing static code. But their area of ​​competence ends with the dynamic behavior of the software. Automated tests are usually used to fill this known gap. They answer the question of how a change affects and changes the behavior of the application. Unit tests define this behavior relatively "close" to the code. UI tests, on the other hand, are very far away from the respective functions - which causes the changes to add up. As a result, many changes have an impact on the UI, which means that it changes "constantly".

Oh, how good that nobody knows that my LoginButton is hot!

Since the attributes of UI elements can change, it is important to select attributes that are as stable as possible and to use them again across individual tests. This is exactly what is done, for example, in the now well-known Page Objects pattern or in key word mapping. B. Cucumber propagated. If an attribute then changes, it only needs to be adjusted at one point, which means that we can minimize the maintenance effort for action targets and test rules. In order for this to work, however, a relatively large amount of manual effort is made in advance to create the abstractions. As mentioned earlier, this only reduces the impact of the problem, but does not solve the problem in itself.

The version control for the behavior

How about a new idea, both radical and simple? Instead of manually specifying the correct behavior of the application with technical tests, we simply record the current behavior of the application. When the tests are "played back", the recorded behavior is then simply compared with the current behavior and all differences are presented to the user. At this point it should be mentioned that the appearance of the application, as well as any information displayed, is part of the behavior. As with a conventional version management system, the user basically has three options: he can approve and confirm the behavior, he can reject it or he can ignore it permanently. If he confirms the new behavior of the software, this corresponds to a commit in the version management system: The behavior is adopted, saved and checked against this behavior in the future. If he rejects the new behavior, this simply means that the system has served its purpose and has discovered an error. This must then be rectified. In rare cases there are still a few UI elements that are constantly changing: date displays, version numbers, etc. These can simply be permanently excluded from the comparison (similar to temporary files or log files in version management systems).

When redundancy is a good thing

With conventional GUI test tools, one concentrates as much as possible on a stable attribute. If this changes, the test breaks and you have to maintain it. If you were to choose several attributes at once, you would have to maintain each time if only one of them changes. However, if updating does not mean any effort, then the identification of UI elements via several attributes suddenly means that the tests become more robust! If an attribute changes, the UI element can still be correctly identified using the remaining attributes. So the question is how you can organize the maintenance efficiently.

The puzzle advantage

Imagine you have the same puzzle twice. Your child only got their hands on one of them and completely painted over the picture. Now you are comparing a single puzzle piece of the original puzzle and want to find the corresponding changed and painted over puzzle piece. Probably not an easy task. However, if you compare the complete original puzzle with the complete modified puzzle, the task is made much easier. It is the same with the regression test: If you know all the elements of the previous state, it is much easier to make a 1-to-1 assignment and see which elements are "left over". This makes the tests even more robust against changes.

What should actually happen when you press this button?

Documentation of the current situation

This type of testing has another advantage: the current state is documented and documented in minute detail. This is not only useful if you are developing medical device software, but in many other cases as well. Agile methods want to break the water head in software development by bringing developers and users around the same table. Ideally, if this approach works well, it means that nothing needs to be documented ... which is often the case. But if after six months the requirements and the people in the team have changed and the rest of the application has to be tested (mostly manually) for a new feature, the question often arises what the target state actually looks like. What exactly is supposed to happen when you press this button? Which data should actually be displayed in this field? If you ask yourself these questions, you can congratulate yourself on a very lean and (hopefully) efficient software development process. And for test automation, you should try a new testing approach right away.

Why only now?

If testing this way is such a good idea, why haven't we been doing it for a long time? Well one of the reasons is because of the tool support. So that this approach does not involve a lot of "false positives"produces - in other words, errors that are actually not errors at all - one condition must be met as far as possible: the repeatability of the execution. Every change, even if it is only the time, potentially leads to changes that are not. Ever According to the situation and complexity of the test systems, that used to be a problem. Today we have Docker and the like - building a stable test environment is much easier than it was a few years ago. A second point is the availability of computing capacity Computers usually with a complex application at the edge of their possibilities - today it is no problem to run a virtualization of a simulation of a container in a virtualization in the cloud. This means that more and cheaper GUI tests can be automated than ever in the history of the GUI -Test.

Where the journey leads

Most people agree that the future of testing (like the future of virtually everything else) lies in artificial intelligence. So the question is not what the future of testing looks like - the question is when that future will come and what the path to it will be. Since a technical error is only in the eye of the beholder and is therefore very dependent on the context, technical testing can only be taken over by a "strong" AI, i.e. human-like and generally valid intelligence. This challenge is called the "oracle problem" in research. However, if we specify tests without checking rules, in which the current status is documented and compared with a later version, then suddenly an AI no longer has to be "strong". You no longer have to recognize a technical error as such, which is then made by the user after a change. The AI ​​only needs to run the application and record all use cases, including exceptions and special cases, if possible. And you can even train an AI today!

ReTest GmbH is a Karlsruhe-based start-up that deals with AI-based testing and is developing a tool for a new testing approach, which we call "difference testing". We have just released version 1.0 and are now in the process of releasing parts of our tool as open source. We would be happy if you visit us to download our tool, try it out and give us feedback. Or visit us at CeBIT in Hall 7, Stand D13.

The time has come to try a new testing approach!

  1. World Quality Report 2016-17
  2. M. Feathers, 2013: Working effectively with legacy code, Prentice Hall

Dr. Jeremias Roessler

Dr. Jeremias Rößler is a software engineer and an expert in creating and maintaining large, complex software systems. In 2013 he did his doctorate at the chair for software technology at Saarland University.
>> Read more