Automated Testing Performed by Developers

I finished my thesis about a month ago and it was checked and graded just recently. Title of the thesis is “Automated Testing Performed by Developers”. It consists of a literary review that forms a theoretical basis for the action research. The thesis is available online.

I’m pretty happy how it turned out and I learned a lot while doing it. There are of course many things that I would want to fix, change or expand, but there is only a limited amount of time to write it. The thesis could be summed up as “testing fun, testing good, testing hard”.

Now I just have to finish one more course before graduating.

Refactoring tests

Work with the magic system continues. Last time I added ability to create effects for spells. This time I work on creating those effects. Since the test for this step is really close to the previous step and the code required to make it pass is just couple of lines, I’ll concentrate on writing about something else: refactoring tests.

The test I wrote is as follows:

def test_triggering_effect(self):
    """
    Casting a spell should trigger the effect
    """
    effects_factory = mock(EffectsFactory)
    effect = mock(Effect)
    dying_rules = mock(Dying)
    when(effects_factory).create_effect(key = 'healing wind',
                                        target = self.character).thenReturn(effect)

    effect_handle = EffectHandle(trigger = 'on spell hit',
                                 effect = 'healing wind',
                                 parameters = {},
                                 charges = 1)

    spell = (SpellBuilder()
                .with_effect_handle(effect_handle)
                .with_target(self.character)
                .build())

    spell.cast(effects_factory, 
               dying_rules)
    
    verify(effect).trigger(dying_rules)

You probably notice how similar it is to the code I wrote in the previous step. It didn’t bother me when I was working on making it pass, but as soon as I was done with that, it was time to clean things up.

First step was to move these two similar tests to a separate test class called TestSpellEffects. The maintenance is easier when similar tests or tests for certain functionality are close to each other.

Next I extracted the duplicate code from each test and moved it into a setup function:

def setup(self):
    """
    Setup test cases
    """
    self.character = (CharacterBuilder()
                            .build())

    self.effects_factory = mock(EffectsFactory)
    self.effect = mock(Effect)
    self.dying_rules = mock(Dying)
    when(self.effects_factory).create_effect(key = 'healing wind',
                                        target = self.character).thenReturn(self.effect)

    self.effect_handle = EffectHandle(trigger = 'on spell hit',
                                      effect = 'healing wind',
                                      parameters = {},
                                      charges = 1)

    self.spell = (SpellBuilder()
                    .with_effect_handle(self.effect_handle)
                    .with_target(self.character)
                    .build())

The setup function is run once for each test and it is perfect place for code that would be duplicated otherwise. It’s good idea to pay close attention what actually ends into setup and try to keep it as small as possible. The more there is code in setup function the harder to read the tests seem to me.

After this change the actual tests were much smaller:

def test_creating_effect(self):
    """
    Casting a spell should create effects it has
    """
    self.spell.cast(self.effects_factory,
                    self.dying_rules)

    verify(self.effects_factory).create_effect(key = 'healing wind',
                                               target = self.character)

def test_triggering_effect(self):
    """
    Casting a spell should trigger the effect
    """
    self.spell.cast(self.effects_factory, 
                    self.dying_rules)
    
    verify(self.effect).trigger(self.dying_rules)

Much nicer than the first version. This highlights the fact that the test code is equally important as the production code and deserves equal attention and care.

Changes for this step are in two commits: ad2bdda513bdbd170ac7927fabcf0632b7e8658a and adfa2540bc90ae28187bb09caeba62a4f16adc50.

Effects of spells

The next step with spell casting takes me to effects. There is already a effect subsystem in place that is being used for healing potions and spider poisonings for example. Effects can be one off or they can have a duration (finite or even infinite). So it seemed like a good idea to reuse those effects for spells. It gives the possibility to have immediate spell effects (like magic missiles and fire balls) or longer lasting ones (blessings and curses).

First step is to create some effects, so I will start with a test:

def test_creating_effect(self):
    """
    Casting a spell should create effects it has
    """
    effects_factory = mock(EffectsFactory)
    effect = mock(Effect)
    dying_rules = mock(Dying)
    when(effects_factory).create_effect(key = 'healing wind',
                                        target = self.character).thenReturn(effect)

    effect_handle = EffectHandle(trigger = 'on spell hit',
                                 effect = 'healing wind',
                                 parameters = {},
                                 charges = 1)

    spell = (SpellBuilder()
                .with_effect_handle(effect_handle)
                .with_target(self.character)
                .build())

    spell.cast(effects_factory, 
               dying_rules)

    verify(effects_factory).create_effect(key = 'healing wind',
                                          target = self.character)

Essential parts are where we tell the effect_factory to return a mocked effect when it is being called with correct parameters. EffectHandles are sort of specifications that tell when to create what kind of effect. In this case, when ‘on spell hit’ event happens, we want to create a healing wind. EffectFactory on the other hand can be used to create effect by name, which in this case is ‘healing wind’.

I’m using a SpellBuilder to create instance of Spell object. The reason for this is to decouple tests and the code they are testing to a degree. Obviously there has to be some sort of connectivity, but I try to limit it a bit. If internal representation of spells change, it is enough to change the builder and all the tests are hopefully up and running again (at least if the verification part of the test works).

Executing tests at this point gives an error:

Traceback (most recent call last):
  File "C:\Python32\lib\site-packages\nose\case.py", line 198, in runTest
    self.test(*self.arg)
  File "C:\programming\pyHack\src\pyherc\test\unit\test_spells.py", line 85, in test_triggering_effect
    target = self.character)
  File "C:\Python32\lib\site-packages\mockito\invocation.py", line 98, in __call__
    verification.verify(self, len(matched_invocations))
  File "C:\Python32\lib\site-packages\mockito\verification.py", line 51, in verify
    raise VerificationError("\nWanted but not invoked: %s" % (invocation))
mockito.verification.VerificationError:
Wanted but not invoked: create_effect()

One noteworthy thing about how mockito-python verifies interactions between objects. If I have a verification like:

verify(effects_factory).create_effect(key = 'healing wind',
                                      target = character)

It does not match to a call like this:

effects_factory.create_effect('healing wind',
                              character)

If the verification is specified with keyword arguments, then the call must be made with keyword arguments:

effects_factory.create_effect(key = 'healing wind',
                              target = character)

This is a limitation in mockito-python and so far I have not found a workaround for it. It is something that needs to be remembered when working with keyword arguments.

Another advantage of reusing effects subsystem is that I already have an EffectsCollection class ready that is used to store and query Effects and EffectHandles. Instead of writing all that logic again, I can just reuse it in Spell class:

@logged
def get_effect_handles(self, trigger = None):
    """
    Get effect handles

    :param trigger: optional trigger type
    :type trigger: string

    :returns: effect handles
    :rtype: [EffectHandle]
    """
    return self.effects.get_effect_handles(trigger)

In order to make the test pass, I made following modifications to Spell class:

@logged
def cast(self, effects_factory):
    """
    Cast the spell

    :param effects_factory: factory for creating effects
    :type effects_factory: EffectsFactory
    """
    handles = self.effects.get_effect_handles('on spell hit')
    effects = []

    for target in self.target:
        for handle in handles:
            effects.append(effects_factory.create_effect(
                                            key = handle.effect,
                                            target = target))

At this point the test passes. The test verifies only that the EffectsFactory was called with parameters that it should be called with. It does not verify what is being done with the results of the call. That is left for the next step, when I want to write the code to actually trigger the created effects.

Code for this change (including the SpellBuilder) can be found in commit 62512fb694a93e81e792ec2cf41c1d10901f4c32.

Does it make sense to take this tiny steps? Couldn’t I have just written a test that checks that the effects being created are being triggered and hit two birds with a single stone? I could have done that and I think that would have been as valid approach as dividing it to two separate tests. I prefer to work with very small steps and small tests. While they help me to pinpoint the exact spot where the system is misbehaving, the drawback is the amount of code I have to write.

Intermission: wiring things for behave

After reflecting the status of the building pieces for spell casting, I realised that I can wire things a bit further on behave side.

I started by fixing a step definition for behave:

@when('{caster_name} casts {spell_and_target}')
def impl(context, caster_name, spell_and_target):
    # Simon casts magic missile on Uglak
    # Simon casts healing wind
    
    caster = get_character(context, caster_name)
    spell = spell_and_target

    make(caster, cast_spell(spell_name = spell))

Since healing wind spell will heal only the caster, I make some assumptions here (like not caring about who I’m casting the spell at). I used cutesy here too, just like in the melee combat, in order to keep the step definition short and decoupled from the actual implementation details.

Implementation for cast_spell – function is as follows:

class CastSpell():
    """
    Class representing casting a spell
    """
    def __init__(self, spell_name):
        """
        Default constructor
        
        :param spell_name: name of the spell to cast
        :type spell_name: string
        """
        super(CastSpell, self).__init__()
        self.spell_name = spell_name
    
    def __call__(self, caster):
        """
        Performs the casting
        
        :param caster: character doing the casting
        :type caster: Character
        """
        caster.old_values = {}
        caster.old_values['hit points'] = caster.hit_points

        action_factory = (ActionFactoryBuilder()
                                    .with_move_factory()
                                    .with_attack_factory()
                                    .with_drink_factory()
                                    .with_inventory_factory()
                                    .with_dying_rules()
                                    .with_spellcasting_factory()
                                    .build())

        caster.cast(direction = 1,
                    spell_name = self.spell_name, 
                    action_factory = action_factory)

def cast_spell(spell_name):
    """
    Cast a spell

    :param spell_name: name of the spell to cast
    """
    action = CastSpell(spell_name)
    return action

This hopefully decouples behave steps enough from actual implementation, in case they change later. CastSpell class is used to cast spells and it is callable (__call__ method has been implemented). This allows me to store context about the action in the object’s attributes and still call it like it would be a function. Another option would of course be to create a function and handle context parameters with closures.

cast_spell – function is just a convenient way of constructing CastSpell object with given parameters.

I’m keeping track of casters hit points, because further down the line I want to verify that they change after the spell has been cast. Later on I should add other attributes like mana points for tracking.

I’m also creating a new instance of ActionFactory. That is an object that is used to create actions, like moving, fighting and casting spells. Having a builder object makes it easy to construct the ActionFactory.

Final step was to implement one more behave step:

@then('{character_name} should be in full health')
def impl(context, character_name):
    character = get_character(context, character_name)

    assert_that(character.hit_points, equal_to(character.max_hp))

The step is short and it verifies that the character in question is in full health.

After running the behave I got following output:

  @wip
  Scenario: Healing                       # features\magic.feature:29
    Given Simon is Wizard                 # features\helpers\events.py:35
    And Uglak is Goblin                   # features\helpers\events.py:35
    And Uglak is almost dead              # features\helpers\events.py:35
    And Uglak is standing in room         # features\helpers\context.py:35
    And Simon is standing away from Uglak # <string>:50
    Given Simon is almost dead            # features\helpers\events.py:35
    When Simon casts healing wind         # <string>:24
    Then Simon should be in full health   # <string>:109
      Assertion Failed: 
      Expected: <5>
           but: was <1>

So Uglak and Simon, both in weak condition, are standing in a room. Simon casts healing wind, but does not have full health. Every step should have been implemented at this point, meaning I can keep coding and running the test to see when the code satisfies it.

Code for this change can be found from commit 3a32c172c65f19e7fcee9360eda4c301994ad8e8.

Targetting spells

My next goal was to start working with actual spell objects. I sketched my ideas on a paper and decided that there would be a single Spell class that could be parametrized to handle various spells. And the first step to creating that class would be targeting characters with spells.

I started writing following test:

    def test_target_single(self):
        """
        Targeting a single character should be possible
        """
        level = LevelBuilder().build()
        
        character = (CharacterBuilder()
                        .with_level(level)
                        .with_location((5, 5))
                        .build())

        spell_generator = SpellGeneratorBuilder().build()

        spell = spell_generator.create_spell(spell_name = 'healing wind', 
                                             target = character)

        assert_that(character, is_in(spell.target))

Nothing too complicated. My focus will be on healing wind spell and after I get it working, I can start from the beginning and see what needs to be changed to get magic missile working. You may notice that target attribute is a list of some sort, instead of just a single entity. I’m jumping forward a bit here, but I know already that I want spells that can target multiple characters and felt that it makes sense to write the system to support that from the beginning.

The code needed to make the test pass is simple:

class Spell():
    """
    Class to represent spells
    
    .. versionadded:: 0.9
    """
    
    def __init__(self):
        """
        Default constructor
        """
        self.target = []

class SpellGenerator():
    """
    Factory for creating spells
    
    .. versionadded:: 0.9
    """
    @logged
    def __init__(self):
        """
        Default constructor
        """
        pass
     
    @logged
    def create_spell(self, spell_name, target):
        """
        Create a spell
         
        :param spell_name: name of the spell
        :type spell_name: string
        :param target: target of the spell
        :type target: Character
        :returns: ready to use spell
        :rtype: Spell
        """
        new_spell = Spell()
        new_spell.target.append(target)
 
        return new_spell

SpellGenerator will be the most used interface for the spell subsystem. It can be used to create all the different spells.

Code for this can be found from commit 112e9a77654490e0c8875c23572aa561bf35462d

Cyber-Dojo

Jon Jagger has built an excellent web application for practicing test driven development. The environment consists of multiple languages and exercises that are easily accessible via web browser. I really like the concept because it eliminates the need to install any development environments or libraries before you can get started. This would be pretty perfect for demonstrating and training TDD together with a team, since everybody would immediately have the same system setup without any hassle. Cyber-dojo also comes with ability to review how code has changed over time and what the effect those changes have had on the test results. At the top of the screen there’s history of traffic lights and it’s easy to get hooked with that familiar red-green – cycle.

Lets play test driven development

As a part of course 1400 (Advanced Technologies), I should spend some time watching screen casts or presentations relating to my thesis and make notes for rest of the group. I got tipped that there’s really interesting series called “Lets play test driven development“. Each episode is short, which means that I can keep watching them when I have small slot of time available, instead of committing 1 hour or more.

As a fan of test driven development, it will be really interesting to see how other developer is doing it.