So far, we have assumed that failures would be discovered and fixed by a single programmer during development. But what if the user who discovers a bug is different from the developer who eventually fixes it? In this case, users have to report bugs, and one needs to ensure that reported bugs are systematically tracked. This is the job of dedicated bug tracking systems, which we will discuss (and demo) in this chapter.
Prerequisites
# ignore
if 'CI' in os.environ:
# Can't run this in our continuous environment,
# since it can't run a headless Web browser
sys.exit(0)
# ignore
#
# WARNING: Unlike the other chapters in this book,
# this chapter should NOT BE RUN AS A NOTEBOOK:
#
# * It will delete ALL data from an existing
# local _Redmine_ installation.
# * It will create new users and databases in an existing
# local _MySQL_ installation.
#
# The only reason to run this notebook is to create the book chapter,
# which is the task of Andreas Zeller (and possibly some translators).
# If you are not Andreas, you should exactly know what you are doing.
assert os.getenv('USER') == 'zeller'
So far, we have always assumed an environment in which software failures would be discovered by the very developers responsible for the code – that is, failures discovered (and fixed) during development. However, failures can also be discovered by third parties, such as
In all these cases, developers need to be informed about the fact that the program failed; if they won't know that a bug exists, it will be hard to fix it. This means that we have to set up mechanisms for reporting bugs – manual ones and/or automated ones.
Let us start with the information a developer requires to fix a bug. In a 2008 study \cite{Bettenburg2008}, Bettenburg et al. asked 872 developers from the Apache, Eclipse, and Mozilla projects to complete a survey on the most important information they need. From top to bottom, these were as follows:
This is a list of steps by which the failure would be reproduced. For instance:
- I started the program using
$ python Debugger.py my_code.py
.- Then, at the
(debugger)
prompt, I enteredrun
and pressed the ENTER key.
The easier it will be for the developer to reproduce the bug, the higher the chances it will be effectively fixed. Reducing the steps to those relevant to reproduce the bug can be helpful. But at the same time, the main problem experienced by developers as it comes to bug reports is incomplete information, and this especially applies to the steps to reproduce.
These give hints on which parts of the code were active at the moment the failure occurred.
I got this stack trace:
Traceback (most recent call last):
File "Debugger.py", line 2, in <module>
handle_command("run")
File "Debugger.py", line 3, in handle_command
scope = s.index(" in ")
ValueError: substring not found (expected)
Even though stack traces are useful, they are seldom reported by regular users, as they are difficult to obtain (or to find if included in log files). Automated crash reports (see below), however, frequently include them.
Test cases that reproduce the bug are also seen as important:
I can reproduce the bug using the following code:
import Debugger
Debugger.handle_command("run")
Non-developers hardly ever report test cases.
What the bug reporter observed as a failure.
The program crashed with a
ValueError
.
In many cases, this mimics the stack trace or the steps to reproduce the bug.
Screenshots can further illustrate the failure.
Here is a screenshot of the
Debugger
failing in Jupyter.
Screenshots are helpful for certain bugs, such as GUI errors.
What the bug reporter expected instead.
I expected the program not to crash.
Perhaps surprisingly, the information that was seen as least relevant for developers was:
The relative low importance of these fields may be surprising as entering them is usually mandated in bug report forms. However, in \cite{Bettenburg2008}, developers stated that
[Operating System] fields are rarely needed as most [of] our bugs are usually found on all platforms.
This not meant to be read as these fields being totally irrelevant, as, of course, there can be bugs that occur only on specific platforms. Also, if a bug is reported for an older version, but is known to be fixed in a more current version, a simple resolution would be to ask the user to upgrade to the fixed version.
If a program crashes, it can be a good idea to have it automatically report the failure to developers. A common practice is to have a crashing program show a bug report dialog allowing the user to report the crash to the vendor. The user is typically asked to provide additional details on how the failure came to be, and the crash report would be sent directly to the vendor's database:
The automatic report typically includes a stack trace and configuration information. These two do not reveal too many sensitive details about the user, yet already can help a lot in fixing a bug. In the interest of transparency, the user should be able to inspect all information sent to the vendor.
Besides stack traces and configuration information, such crash reporters could, of course, collect much more - say the data the program operated on, logs, recorded steps to reproduce, or automatically recorded screenshots. However, all of these will likely include sensitive information; and despite their potential usefulness, it is typically better to not collect them in the first place.
When writing an issue report, it is important to look at it from the developer's perspective. Developers not only require information on how to reproduce the bug; they also want to work effectively and efficiently. The following aspects will all help developers with that:
At the developer's end, all issues reported need to be tracked – that is, they have to be registered, they have to be checked, and of course, they have to be addressed. This process takes place via dedicated database systems, so-called bug tracking systems.
The purposes of an issue tracking system include
Let us illustrate how these steps work, using the popular Redmine
issue tracking system.
This is what the Redmine tracker starts with:
# ignore
redmine_gui.get(redmine_url + '/login')
screenshot(redmine_gui)
After we login, we see our account:
# ignore
redmine_gui.find_element(By.ID, "username").send_keys("admin")
redmine_gui.find_element(By.ID, "password").send_keys("admin001")
redmine_gui.find_element(By.NAME, "login").click()
screenshot(redmine_gui)
We start with a list of projects.
# ignore
redmine_gui.get(redmine_url + '/projects')
screenshot(redmine_gui)