Tracking Bugs

So far, we have assumed that failures would be discovered and fixed by a single programmer during development. But what if the user who discovers a bug is different from the developer who eventually fixes it? In this case, users have to report bugs, and one needs to ensure that reported bugs are systematically tracked. This is the job of dedicated bug tracking systems, which we will discuss (and demo) in this chapter.

from bookutils import YouTubeVideo
YouTubeVideo("bJzHYzvxHm8")

Prerequisites

import os
import sys

Synopsis

To use the code provided in this chapter, write

>>> from debuggingbook.Tracking import <identifier>

and then make use of the following features.

This chapter provides no functionality that could be used by third-party code.

Reporting Issues

So far, we have always assumed an environment in which software failures would be discovered by the very developers responsible for the code – that is, failures discovered (and fixed) during development. However, failures can also be discovered by third parties, such as

  • Testers whose job it is to test the code of developers
  • Other developers using the code
  • Users running the code as it is in production

In all these cases, developers need to be informed about the fact that the program failed; if they won't know that a bug exists, it will be hard to fix it. This means that we have to set up mechanisms for reporting bugs – manual ones and/or automated ones.

What Goes in a Bug Report?

Let us start with the information a developer requires to fix a bug. In a 2008 study [Bettenburg et al, 2008], Bettenburg et al. asked 872 developers from the Apache, Eclipse, and Mozilla projects to complete a survey on the most important information they need. From top to bottom, these were as follows:

Steps to Reproduce (83%)

This is a list of steps by which the failure would be reproduced. For instance:

  1. I started the program using $ python Debugger.py my_code.py.
  2. Then, at the (debugger) prompt, I entered run and pressed the ENTER key.

The easier it will be for the developer to reproduce the bug, the higher the chances it will be effectively fixed. Reducing the steps to those relevant to reproduce the bug can be helpful. But at the same time, the main problem experienced by developers as it comes to bug reports is incomplete information, and this especially applies to the steps to reproduce.

Stack Traces (57%)

These give hints on which parts of the code were active at the moment the failure occurred.

I got this stack trace:

Traceback (most recent call last):
  File "Debugger.py", line 2, in <module>
    handle_command("run")
  File "Debugger.py", line 3, in handle_command
    scope = s.index(" in ")
ValueError: substring not found (expected)

Even though stack traces are useful, they are seldom reported by regular users, as they are difficult to obtain (or to find if included in log files). Automated crash reports (see below), however, frequently include them.

Test Cases (51%)

Test cases that reproduce the bug are also seen as important:

I can reproduce the bug using the following code: ```python import Debugger

Debugger.handle_command("run") ```

Non-developers hardly ever report test cases.

Observed Behavior (33%)

What the bug reporter observed as a failure.

The program crashed with a ValueError.

In many cases, this mimics the stack trace or the steps to reproduce the bug.

Screenshots (26%)

Screenshots can further illustrate the failure.

Here is a screenshot of the Debugger failing in Jupyter.

Screenshots are helpful for certain bugs, such as GUI errors.

Expected Behavior (22%)

What the bug reporter expected instead.

I expected the program not to crash.

Configuration Information (< 12%)

Perhaps surprisingly, the information that was seen as least relevant for developers was:

  • Version (12%)
  • Build information (8%)
  • Product (5%)
  • Operating system (4%)
  • Component (3%)
  • Hardware (0%)

The relative low importance of these fields may be surprising as entering them is usually mandated in bug report forms. However, in [Bettenburg et al, 2008], developers stated that

[Operating System] fields are rarely needed as most [of] our bugs are usually found on all platforms.

This not meant to be read as these fields being totally irrelevant, as, of course, there can be bugs that occur only on specific platforms. Also, if a bug is reported for an older version, but is known to be fixed in a more current version, a simple resolution would be to ask the user to upgrade to the fixed version.

Reporting Crashes Automatically

If a program crashes, it can be a good idea to have it automatically report the failure to developers. A common practice is to have a crashing program show a bug report dialog allowing the user to report the crash to the vendor. The user is typically asked to provide additional details on how the failure came to be, and the crash report would be sent directly to the vendor's database:

The automatic report typically includes a stack trace and configuration information. These two do not reveal too many sensitive details about the user, yet already can help a lot in fixing a bug. In the interest of transparency, the user should be able to inspect all information sent to the vendor.

Besides stack traces and configuration information, such crash reporters could, of course, collect much more - say the data the program operated on, logs, recorded steps to reproduce, or automatically recorded screenshots. However, all of these will likely include sensitive information; and despite their potential usefulness, it is typically better to not collect them in the first place.

Effective Issue Reporting

When writing an issue report, it is important to look at it from the developer's perspective. Developers not only require information on how to reproduce the bug; they also want to work effectively and efficiently. The following aspects will all help developers with that:

  • Have a concise summary. A good summary should quickly and uniquely identify a bug. Make it easy to understand such that the reader can know if the bug has been reported and/or fixed.
  • Be clear and concise. Provide necessary information (as shown above) and avoid any extras. Use meaningful sentences and simple words. Structure the report into enumerations and bullet lists.
  • Do not assume context. Make no assumption that the developer knows all about your bug. Find out whether similar issues have been reported before and reference them.
  • Avoid commanding tones. Developers enjoy autonomy in their work, and coming across as too authoritative hurts morale.
  • Avoid sarcasm. Same as above. If you think you can get a volunteer to fix a bug in an open source program for you with sarcasm or commands – good luck with that.
  • Do not assume mistakes. Do not assume some developer (or anyone) has made a mistake. In many cases, issues can be resolved on your side.
  • One issue per report. If you have multiple issues, split them in multiple reports; this makes it easier to process them.

An Issue Tracker

At the developer's end, all issues reported need to be tracked – that is, they have to be registered, they have to be checked, and of course, they have to be addressed. This process takes place via dedicated database systems, so-called bug tracking systems.

The purposes of an issue tracking system include

  • to collect and store all issue reports;
  • to check the status of issues at all times; and
  • to organize the debugging and development process.

Let us illustrate how these steps work, using the popular Redmine issue tracking system.

Setting up Redmine

To install Redmine, we followed the instructions at https://gist.github.com/johnjohndoe/2763243. These final steps initialize the database:

import subprocess
import os
import sys
def with_ruby(cmd: str, inp: str = '', timeout: int = 30, show_stdout: bool = False) -> None:
    print(f"$ {cmd}")
    shell = subprocess.Popen(['/bin/sh', '-c',
        f'''rvm_redmine=$HOME/.rvm/gems/ruby-2.7.2@redmine; \
rvm_global=$HOME/.rvm/gems/ruby-2.7.2@global; \
export GEM_PATH=$rvm_redmine:$rvm_global; \
export PATH=$rvm_redmine/bin:$rvm_global/bin:$HOME/.rvm/rubies/ruby-2.7.2/bin:$HOME/.rvm/bin:$PATH; \
cd $HOME/lib/redmine && {cmd}'''],
                             stdin=subprocess.PIPE,
                             stdout=subprocess.PIPE,
                             stderr=subprocess.PIPE,
                             universal_newlines=True)
    try:
        stdout_data, stderr_data = shell.communicate(inp, timeout=timeout)
    except subprocess.TimeoutExpired:
        shell.kill()
#         stdout_data, stderr_data = shell.communicate(inp)
#         if show_stdout:
#             print(stdout_data, end="")
#         print(stderr_data, file=sys.stderr, end="")
        raise

    print(stderr_data, file=sys.stderr, end="")
    if show_stdout:
        print(stdout_data, end="")
def with_mysql(cmd: str, timeout: int = 2, show_stdout: bool = False) -> None:
    print(f"sql>{cmd}")
    sql = subprocess.Popen(["mysql", "-u", "root",
                           "--default-character-set=utf8mb4"],
                           stdin=subprocess.PIPE,
                           stdout=subprocess.PIPE,
                           stderr=subprocess.PIPE,
                           universal_newlines=True)
    try:
        stdout_data, stderr_data = sql.communicate(cmd + ';',
                                                   timeout=timeout)
    except subprocess.TimeoutExpired:
        sql.kill()
#         stdout_data, stderr_data = sql.communicate(inp)
#         if show_stdout:
#             print(stdout_data, end="")
#         print(stderr_data, file=sys.stderr, end="")
        raise

    print(stderr_data, file=sys.stderr, end="")
    if show_stdout:
        print(stdout_data, end="")
with_ruby("bundle config set without development test")
$ bundle config set without development test
with_ruby("bundle install")
$ bundle install
The dependency tzinfo-data (>= 0) will be unused by any of the platforms Bundler is installing for. Bundler is installing for ruby but the dependency is only for x86-mingw32, x64-mingw32, x86-mswin32. To add those platforms to the bundle, run `bundle lock --add-platform x86-mingw32 x64-mingw32 x86-mswin32`.
The dependency ffi (>= 0) will be unused by any of the platforms Bundler is installing for. Bundler is installing for ruby but the dependency is only for x86-mingw32, x64-mingw32, x86-mswin32. To add those platforms to the bundle, run `bundle lock --add-platform x86-mingw32 x64-mingw32 x86-mswin32`.
with_ruby("pkill sql; sleep 5")
$ pkill sql; sleep 5
try:
    with_ruby("mysql.server start", show_stdout=True)
except subprocess.TimeoutExpired:
    pass  # Can actually start without producing output
$ mysql.server start
with_mysql("drop database redmine")
sql>drop database redmine
with_mysql("drop user 'redmine'@'localhost'")
sql>drop user 'redmine'@'localhost'
with_mysql("create database redmine character set utf8")
sql>create database redmine character set utf8
with_mysql("create user 'redmine'@'localhost' identified by 'my_password'")
sql>create user 'redmine'@'localhost' identified by 'my_password'
with_mysql("grant all privileges on redmine.* to 'redmine'@'localhost'")
sql>grant all privileges on redmine.* to 'redmine'@'localhost'
with_ruby("bundle exec rake generate_secret_token")
$ bundle exec rake generate_secret_token
with_ruby("RAILS_ENV=production bundle exec rake db:migrate")
$ RAILS_ENV=production bundle exec rake db:migrate
with_ruby("RAILS_ENV=production bundle exec rake redmine:load_default_data", '\n')
$ RAILS_ENV=production bundle exec rake redmine:load_default_data
Starting Redmine
import os
import time
from multiprocess import Process
def run_redmine(port: int) -> None:
    with_ruby(f'exec rails s -e production -p {port} > redmine.log 2>&1',
              timeout=3600)
def start_redmine(port: int = 3000) -> Tuple[Process, str]:
    process = Process(target=run_redmine, args=(port,))
    process.start()
    time.sleep(5)

    url = f"http://localhost:{port}"
    return process, url
redmine_process, redmine_url = start_redmine()
$ exec rails s -e production -p 3000 > redmine.log 2>&1
Remote Control with Selenium

To produce this book, we use Selenium to interact with user interfaces and obtain screenshots.

Selenium is a framework for testing Web applications by automating interaction in the browser. Selenium provides an API that allows one to launch a Web browser, query the state of the user interface, and interact with individual user interface elements. The Selenium API is available in a number of languages; we use the Selenium API for Python.

A Selenium web driver is the interface between a program and a browser controlled by the program.

from selenium import webdriver

The following code starts a Firefox browser in the background, which we then control through the web driver.

BROWSER = 'firefox'
with_ruby("pkill Firefox.app firefox-bin")
$ pkill Firefox.app firefox-bin

Note: If you don't have Firefox installed, you can also set BROWSER to 'chrome' to use Google Chrome instead.

# BROWSER = 'chrome'

When running this outside of Jupyter notebooks, the browser is headless, meaning that it does not show on the screen.

from bookutils import rich_output
# HEADLESS = not rich_output()
HEADLESS = True
def start_webdriver(browser: str = BROWSER, headless: bool = HEADLESS,
                    zoom: float = 4.0) -> WebDriver:
    if browser == 'firefox':
        options = webdriver.FirefoxOptions()
    if browser == 'chrome':
        options = webdriver.ChromeOptions()

    if headless and browser == 'chrome':
        options.add_argument('headless')
    else:
        options.headless = headless

    # Start the browser, and obtain a _web driver_ object such that we can interact with it.
    if browser == 'firefox':
        # For firefox, set a higher resolution for our screenshots
        options.set_preference('layout.css.devPixelsPerPx', repr(zoom))
        redmine_gui = webdriver.Firefox(options=options)

        # We set the window size such that it fits
        redmine_gui.set_window_size(500, 600)  # was 1024, 600

    elif browser == 'chrome':
        redmine_gui = webdriver.Chrome(options=options)
        redmine_gui.set_window_size(1024, 510 if headless else 640)

    return redmine_gui
redmine_gui = start_webdriver(browser=BROWSER, headless=HEADLESS)
/var/folders/n2/xd9445p97rb3xh7m1dfx8_4h0006ts/T/ipykernel_35479/676888690.py:11: DeprecationWarning: headless property is deprecated, instead use add_argument('-headless')
  options.headless = headless  # type: ignore

We can now interact with the browser programmatically. First, we have it navigate to the URL of our Web server:

redmine_gui.get(redmine_url)

To see what the "headless" browser displays, we can obtain a screenshot. We see that it actually displays the home page.

from IPython.display import display, Image
Image(redmine_gui.get_screenshot_as_png())
Screenshots with Drop Shadows

By default, our screenshots are flat. We add a drop shadow to make them look nicer. With help from https://graphicdesign.stackexchange.com/questions/117272/how-to-add-drop-shadow-to-a-picture-via-cli

import tempfile
def drop_shadow(contents: bytes) -> bytes:
    with tempfile.NamedTemporaryFile() as tmp:
        tmp.write(contents)
        convert = subprocess.Popen(
            ['convert', tmp.name,
            '(', '+clone', '-background', 'black', '-shadow', '50x10+15+15', ')',
            '+swap', '-background', 'none', '-layers', 'merge', '+repage', '-'],
            stdout=subprocess.PIPE, stderr=subprocess.PIPE)
        stdout_data, stderr_data = convert.communicate()

    if stderr_data:
        print(stderr_data.decode("utf-8"), file=sys.stderr, end="")

    return stdout_data
def screenshot(driver: WebDriver, width: int = 500) -> Any:
    return Image(drop_shadow(redmine_gui.get_screenshot_as_png()), width=width)
screenshot(redmine_gui)
First Registration at Redmine
redmine_gui.get(redmine_url + '/login')
screenshot(redmine_gui)
redmine_gui.find_element(By.ID, "username").send_keys("admin")
redmine_gui.find_element(By.ID, "password").send_keys("admin")
redmine_gui.find_element(By.NAME, "login").click()
time.sleep(2)
if redmine_gui.current_url.endswith('my/password'):
    redmine_gui.get(redmine_url + '/my/password')
    redmine_gui.find_element(By.ID, "password").send_keys("admin")
    redmine_gui.find_element(By.ID, "new_password").send_keys("admin001")
    redmine_gui.find_element(By.ID, "new_password_confirmation").send_keys("admin001")
    display(screenshot(redmine_gui))
    redmine_gui.find_element(By.NAME, "commit").click()
redmine_gui.get(redmine_url + '/logout')
redmine_gui.find_element(By.NAME, "commit").click()

This is what the Redmine tracker starts with:

After we login, we see our account: