I get asked a lot if it really makes sense to write tests for software. The general answer is yes, but it depends with who you are working with, the complexity, life cycle of the project and many other factors. I would argue that is very hard to have a maintainable system without some sort of automated tests. That is especially true if the system is evolving rapidly. Even if a language has a good typing system and good static analysis tools, writing tests that validate user stories.

Even we as craftsman can't agree on them. I hope that this article will convince you otherwise. This article is intended for people that would like to try them out, but are looking for a practical example, why and how to start.

I encountered multiple problems that I could not solve without automatic tests. It would take a lot of time to test all the edge cases and it required me to use some more advanced approaches that are typically not used when implementing business logic. An example of this would be meta-programing or working with protocols/interfaces/traits. You usually want to avoid using such features of languages, but in certain situations such as replacing an old part of complex code with new implementation you have to use it to ensure the same API for the developers.

I usually write tests after code being tested, but when the development experience is important and you kinda have an idea how the API for your library would look like, I would recommend you to do TDD.

We will take python as an example. Python is a simple language, but you can modify the behavior of objects in some interesting way. We will use basic features of a library called pytest for unit tests and we will take a quick look at doc tests.

Let's implement a simple ConfigStore

Sometime when you work on a project, you want to have more control over configuration values. Maybe you don't want to decouple code that is run in a web framework, but also without context of a web requests. An example of that would be a cronjob, a job executed by a queue worker or something else.

I am not suggesting you go with this specific implementation. My point is that sometimes you need to do things, that are out of you comfort zone. Firstly, I recommend that we isolate the implementation into a class, function, module (python file) or package (python directory). So you can replace the implementation more easily in the feature, but also reason about the implementation. Secondly, write some unit tests for the unit of isolation.

If we know what we want, we can write tests and explore different implementations. Or we can try building something and then realizing that maybe we need to write tests, otherwise we are not capable of implementing a solution. That happened to me in this case.

We will define a ConfigStore class. The instance of the class supports getting and setting a settings value pair in 2 different ways. First way is with object attributes (in JavaScript property e.g. config_store.fu) and the second with dictionary like behavior (e.g. config_store["biz"]). In python those two ways of doing it require a different implementation unlike in JavaScript.

>>> config_store = ConfigStore()
>>> config_store.fu = "bar"
>>> config_store["biz"] = "baz"
>>> config_store.biz
'baz'
>>> config_store["fu"]
'bar'

In addition to that we want to provide a mechanism that freezes such object. To prevent modifying the config store at runtime. Similar to JavaScript Object.freeze() in strict mode

>>> config_store.freeze()
>>> config_store.bat = "man"
Traceback (most recent call last):
  [...]
  File "main.py", line 36, in __setattr__
    raise TypeError("ConfigStore is frozen, it does not support attribute assignment")
TypeError: ConfigStore is frozen, it does not support attribute assignment

Everything in python is an object

In python everything is an object. Objects interact with each other and python interpreter will look at some special methods to determine how interactions should behave. Some people will call this a protocol based approach to solving problems. In some specific domains that approach can be very powerful (for example if you are implementing a card game, but the card comparison depends on the context).

In our case we can define setters and getters and return the values that we want. In order to do that we will have to define some special methods in python. I will not explain what each of those magic methods does, but feel free to click on the function name and read the official docs. Here is a list of methods we will have to implement:

• object.__getattr__ get attribute
• object.__setattr__ set attribute
• object.__getitem__ get item
• object.__setitem__ set item

It's important to realize at this point that this is not something most of us do on a day to day basis.

Initial implementation

So the implementation is simple right ...lets create a main.py file with the implementation.

# XXX broken
class ConfigStore:
    def __init__(self):
        # is config store frozen
        self._frozen = False
        # internal storage
        self._config = dict()

    def __getattr__(self, name):
        "Allow getting with config_store.key_"
        try:
            return self._config[name]
        except KeyError:
            raise AttributeError

    def __setattr__(self, name, value):
        "Allow setting with config_store.key_"
        if self._frozen is True:
            raise TypeError("ConfigStore is frozen, it does not support attribute assignment")
        new_config = self._config.copy()
        new_config[name] = value
        self._config = new_config

    def __getitem__(self, key):
        "Allow getting with config_store[key_]"
        return self._config[key]

    def __setitem__(self, key, value):
        "Allow setting with config_store[key_]"
        if self._frozen is True:
            raise TypeError("ConfigStore is frozen, it does not support item assignment")
        self._config[key] = value

    def freeze(self):
        self._frozen = True

It's not that simple. The main problem is with the __setattr__ method. The main problem is when you are defining attributes you call the __setattr__ and the code will loop until an RecursionError is raised. At least at this point it would make sense to write some tests that will help you develop.

Unit tests

We will add unit tests, that test our class:

import pytest

# ConfigStore implementation

class TestConfigStore:

    def test_set_get_attribute(self):
        config_store = ConfigStore()
        config_store.FOO = "bar"
        assert config_store.FOO == "bar"

    def test_set_get_item(self):
        config_store = ConfigStore()
        config_store["FOO"] = "bar"
        assert config_store["FOO"] == "bar"

    def test_freeze(self):
        config_store = ConfigStore()
        config_store.LOVE = "crab"

        config_store.freeze()

        assert config_store.LOVE == "crab"
        assert config_store["LOVE"] == "crab"

        with pytest.raises(TypeError):
            config_store["FOO"] = "bar"
        with pytest.raises(TypeError):
            config_store.BIZ = "baz"


if __name__ == "__main__":
    pytest.main(["-s", "main.py"])

With the tests we cover all the user-stories defined above. And we can work towards correctly breaking out of the __setattr__ method. As you can see writing unit tests with pytest is not super scary.

So we come up with a better implementation of ConfigStore:

class ConfigStore:
    def __init__(self):
        # is config store frozen
        self._frozen = False
        # internal storage
        self._config = dict()

    def __getattr__(self, name):
        "Allow getting with config_store.key_"
        try:
            return self._config[name]
        except KeyError:
            raise AttributeError

    def __setattr__(self, name, value):
        "Allow setting with config_store.key_"
        # avoid recursion
        if name in ("_config", "freeze", "_frozen") or name.startswith("__"):
            super().__setattr__(name, value)
            return
        if self._frozen is True:
            raise TypeError("ConfigStore is frozen, it does not support attribute assignment")
        new_config = self._config.copy()
        new_config[name] = value
        super().__setattr__("_config", new_config)

    def __getitem__(self, key):
        "Allow getting with config_store[key_]"
        return self._config[key]

    def __setitem__(self, key, value):
        "Allow setting with config_store[key_]"
        if self._frozen is True:
            raise TypeError("ConfigStore is frozen, it does not support item assignment")
        self._config[key] = value

    def freeze(self):
        self._frozen = True

So this is a better implementation. We call the objects built-in __setattr__ method for internal use. For me implementing something like this is hard. I can't imagine it doing it without tests.

Doctest

Add a start of this article we defined how ConfigStore should behave. Such definitions can be used as documentation. It can greatly speed up library adoption. But the problem with documentation is, that with time it's gets outdated. Wouldn't be nice to have a way of testing the documentation against the implementation.

In python that can simply be done with the built-in module called doctest.

Lets take a quick look how to do it in our case. At the top we will define a __doc__ string. That can simply be done by adding a string to the file. We also need to mimic the output of python console. doctest will then compare the string with the output of the python interpreter.

In order to have a working example you can run yourself we will run doctest inside python. But it's common to have it evoked in a shell.

"""
Simple ConfigStore that supports setting, getting and freezing
of key value pairs.

>>> config_store = ConfigStore()
>>> config_store.fu = "bar"
>>> config_store["biz"] = "baz"
>>> config_store.biz
'baz'
>>> config_store["fu"]
'bar'
>>> config_store.freeze()
>>> config_store.bat = "man"
Traceback (most recent call last):
  [...]
  File "main.py", line 36, in __setattr__
    raise TypeError("ConfigStore is frozen, it does not support attribute assignment")
TypeError: ConfigStore is frozen, it does not support attribute assignment
"""
import pytest

# ConfigStore implementation

# TestConfigStore implementation

if __name__ == "__main__":
    import doctest

    pytest.main(["-s", "main.py"])
    print('doctest: {r.attempted} tested, {r.failed} failed'.format(r=doctest.testmod()))

Conclusion

In this article we showed why tests are important. I am a huge believer that you should write them. I hope that this example convinced you to start writing them. Below you will find a working example, feel free to change it and play around. I will be super happy, if you have any questions, just write them in comments below.

Working example: https://repl.it/@brodul/ConfiguratorMadness