Testing with Hy and Nose

I have been using Nose for the longest time and it’s my go-to tool for running tests. Nose is very no-nonsense and has very sensible defaults (to me at least).

Writing a test in Hy that gets discovered and executed with Nose is pretty straightforward. One just needs a correct naming convention and everything happens more or less automatically. However, by applying some standard lisp practices, one can cut down amount of the code needed quite a bit. In this post, I’ll present how my style of writing tests have evolved over time and what kind of measures I took in order to write minimum amount of code.

Continue reading

Running nosetests for Hy

Getting nosetest to work with Hy was easier than I thought in the end. The trick I used was to create a Python module that will act as an entry point. In that module I first import Hy, which in turn lets me to import my test modules:

import hy
from .test_person_repository import *

After that I can run nosetests and end up with:

tuukka@new-loft:~/programming/python/sandvalley/src/sandvalley/tests$ nosetests
..
----------------------------------------------------------------------
Ran 2 tests in 0.015s

OK

Pretty nifty I must say. Having to import everything from modules in a entry point module is a bit hassle, but that’s something that I should be able to automate.

The same method can be used to call Hy code from Python program.

Automated Testing Performed by Developers

I finished my thesis about a month ago and it was checked and graded just recently. Title of the thesis is “Automated Testing Performed by Developers”. It consists of a literary review that forms a theoretical basis for the action research. The thesis is available online.

I’m pretty happy how it turned out and I learned a lot while doing it. There are of course many things that I would want to fix, change or expand, but there is only a limited amount of time to write it. The thesis could be summed up as “testing fun, testing good, testing hard”.

Now I just have to finish one more course before graduating.

Refactoring tests

Work with the magic system continues. Last time I added ability to create effects for spells. This time I work on creating those effects. Since the test for this step is really close to the previous step and the code required to make it pass is just couple of lines, I’ll concentrate on writing about something else: refactoring tests.

The test I wrote is as follows:

def test_triggering_effect(self):
    """
    Casting a spell should trigger the effect
    """
    effects_factory = mock(EffectsFactory)
    effect = mock(Effect)
    dying_rules = mock(Dying)
    when(effects_factory).create_effect(key = 'healing wind',
                                        target = self.character).thenReturn(effect)

    effect_handle = EffectHandle(trigger = 'on spell hit',
                                 effect = 'healing wind',
                                 parameters = {},
                                 charges = 1)

    spell = (SpellBuilder()
                .with_effect_handle(effect_handle)
                .with_target(self.character)
                .build())

    spell.cast(effects_factory, 
               dying_rules)
    
    verify(effect).trigger(dying_rules)

You probably notice how similar it is to the code I wrote in the previous step. It didn’t bother me when I was working on making it pass, but as soon as I was done with that, it was time to clean things up.

First step was to move these two similar tests to a separate test class called TestSpellEffects. The maintenance is easier when similar tests or tests for certain functionality are close to each other.

Next I extracted the duplicate code from each test and moved it into a setup function:

def setup(self):
    """
    Setup test cases
    """
    self.character = (CharacterBuilder()
                            .build())

    self.effects_factory = mock(EffectsFactory)
    self.effect = mock(Effect)
    self.dying_rules = mock(Dying)
    when(self.effects_factory).create_effect(key = 'healing wind',
                                        target = self.character).thenReturn(self.effect)

    self.effect_handle = EffectHandle(trigger = 'on spell hit',
                                      effect = 'healing wind',
                                      parameters = {},
                                      charges = 1)

    self.spell = (SpellBuilder()
                    .with_effect_handle(self.effect_handle)
                    .with_target(self.character)
                    .build())

The setup function is run once for each test and it is perfect place for code that would be duplicated otherwise. It’s good idea to pay close attention what actually ends into setup and try to keep it as small as possible. The more there is code in setup function the harder to read the tests seem to me.

After this change the actual tests were much smaller:

def test_creating_effect(self):
    """
    Casting a spell should create effects it has
    """
    self.spell.cast(self.effects_factory,
                    self.dying_rules)

    verify(self.effects_factory).create_effect(key = 'healing wind',
                                               target = self.character)

def test_triggering_effect(self):
    """
    Casting a spell should trigger the effect
    """
    self.spell.cast(self.effects_factory, 
                    self.dying_rules)
    
    verify(self.effect).trigger(self.dying_rules)

Much nicer than the first version. This highlights the fact that the test code is equally important as the production code and deserves equal attention and care.

Changes for this step are in two commits: ad2bdda513bdbd170ac7927fabcf0632b7e8658a and adfa2540bc90ae28187bb09caeba62a4f16adc50.

Effects of spells

The next step with spell casting takes me to effects. There is already a effect subsystem in place that is being used for healing potions and spider poisonings for example. Effects can be one off or they can have a duration (finite or even infinite). So it seemed like a good idea to reuse those effects for spells. It gives the possibility to have immediate spell effects (like magic missiles and fire balls) or longer lasting ones (blessings and curses).

First step is to create some effects, so I will start with a test:

def test_creating_effect(self):
    """
    Casting a spell should create effects it has
    """
    effects_factory = mock(EffectsFactory)
    effect = mock(Effect)
    dying_rules = mock(Dying)
    when(effects_factory).create_effect(key = 'healing wind',
                                        target = self.character).thenReturn(effect)

    effect_handle = EffectHandle(trigger = 'on spell hit',
                                 effect = 'healing wind',
                                 parameters = {},
                                 charges = 1)

    spell = (SpellBuilder()
                .with_effect_handle(effect_handle)
                .with_target(self.character)
                .build())

    spell.cast(effects_factory, 
               dying_rules)

    verify(effects_factory).create_effect(key = 'healing wind',
                                          target = self.character)

Essential parts are where we tell the effect_factory to return a mocked effect when it is being called with correct parameters. EffectHandles are sort of specifications that tell when to create what kind of effect. In this case, when ‘on spell hit’ event happens, we want to create a healing wind. EffectFactory on the other hand can be used to create effect by name, which in this case is ‘healing wind’.

I’m using a SpellBuilder to create instance of Spell object. The reason for this is to decouple tests and the code they are testing to a degree. Obviously there has to be some sort of connectivity, but I try to limit it a bit. If internal representation of spells change, it is enough to change the builder and all the tests are hopefully up and running again (at least if the verification part of the test works).

Executing tests at this point gives an error:

Traceback (most recent call last):
  File "C:\Python32\lib\site-packages\nose\case.py", line 198, in runTest
    self.test(*self.arg)
  File "C:\programming\pyHack\src\pyherc\test\unit\test_spells.py", line 85, in test_triggering_effect
    target = self.character)
  File "C:\Python32\lib\site-packages\mockito\invocation.py", line 98, in __call__
    verification.verify(self, len(matched_invocations))
  File "C:\Python32\lib\site-packages\mockito\verification.py", line 51, in verify
    raise VerificationError("\nWanted but not invoked: %s" % (invocation))
mockito.verification.VerificationError:
Wanted but not invoked: create_effect()

One noteworthy thing about how mockito-python verifies interactions between objects. If I have a verification like:

verify(effects_factory).create_effect(key = 'healing wind',
                                      target = character)

It does not match to a call like this:

effects_factory.create_effect('healing wind',
                              character)

If the verification is specified with keyword arguments, then the call must be made with keyword arguments:

effects_factory.create_effect(key = 'healing wind',
                              target = character)

This is a limitation in mockito-python and so far I have not found a workaround for it. It is something that needs to be remembered when working with keyword arguments.

Another advantage of reusing effects subsystem is that I already have an EffectsCollection class ready that is used to store and query Effects and EffectHandles. Instead of writing all that logic again, I can just reuse it in Spell class:

@logged
def get_effect_handles(self, trigger = None):
    """
    Get effect handles

    :param trigger: optional trigger type
    :type trigger: string

    :returns: effect handles
    :rtype: [EffectHandle]
    """
    return self.effects.get_effect_handles(trigger)

In order to make the test pass, I made following modifications to Spell class:

@logged
def cast(self, effects_factory):
    """
    Cast the spell

    :param effects_factory: factory for creating effects
    :type effects_factory: EffectsFactory
    """
    handles = self.effects.get_effect_handles('on spell hit')
    effects = []

    for target in self.target:
        for handle in handles:
            effects.append(effects_factory.create_effect(
                                            key = handle.effect,
                                            target = target))

At this point the test passes. The test verifies only that the EffectsFactory was called with parameters that it should be called with. It does not verify what is being done with the results of the call. That is left for the next step, when I want to write the code to actually trigger the created effects.

Code for this change (including the SpellBuilder) can be found in commit 62512fb694a93e81e792ec2cf41c1d10901f4c32.

Does it make sense to take this tiny steps? Couldn’t I have just written a test that checks that the effects being created are being triggered and hit two birds with a single stone? I could have done that and I think that would have been as valid approach as dividing it to two separate tests. I prefer to work with very small steps and small tests. While they help me to pinpoint the exact spot where the system is misbehaving, the drawback is the amount of code I have to write.

Targetting spells

My next goal was to start working with actual spell objects. I sketched my ideas on a paper and decided that there would be a single Spell class that could be parametrized to handle various spells. And the first step to creating that class would be targeting characters with spells.

I started writing following test:

    def test_target_single(self):
        """
        Targeting a single character should be possible
        """
        level = LevelBuilder().build()
        
        character = (CharacterBuilder()
                        .with_level(level)
                        .with_location((5, 5))
                        .build())

        spell_generator = SpellGeneratorBuilder().build()

        spell = spell_generator.create_spell(spell_name = 'healing wind', 
                                             target = character)

        assert_that(character, is_in(spell.target))

Nothing too complicated. My focus will be on healing wind spell and after I get it working, I can start from the beginning and see what needs to be changed to get magic missile working. You may notice that target attribute is a list of some sort, instead of just a single entity. I’m jumping forward a bit here, but I know already that I want spells that can target multiple characters and felt that it makes sense to write the system to support that from the beginning.

The code needed to make the test pass is simple:

class Spell():
    """
    Class to represent spells
    
    .. versionadded:: 0.9
    """
    
    def __init__(self):
        """
        Default constructor
        """
        self.target = []

class SpellGenerator():
    """
    Factory for creating spells
    
    .. versionadded:: 0.9
    """
    @logged
    def __init__(self):
        """
        Default constructor
        """
        pass
     
    @logged
    def create_spell(self, spell_name, target):
        """
        Create a spell
         
        :param spell_name: name of the spell
        :type spell_name: string
        :param target: target of the spell
        :type target: Character
        :returns: ready to use spell
        :rtype: Spell
        """
        new_spell = Spell()
        new_spell.target.append(target)
 
        return new_spell

SpellGenerator will be the most used interface for the spell subsystem. It can be used to create all the different spells.

Code for this can be found from commit 112e9a77654490e0c8875c23572aa561bf35462d

Getting closer to the source of magic

The current plan is to have spell casting use effects sub-system and make Spell object a composite that has a list of effects. In order to make construction of Spells easy, I’m planning on using a factory and call it while creating the action. Following code captures this in the form of a test:

def test_spell_is_created_with_a_factory(self):
    """
    When creating a spell casting action, spell should be created
    """
    spell_factory = SpellFactoryBuilder().build()
    when(spell_factory).create_spell('healing wind').thenReturn(mock())
    spellcasting_factory = (SpellCastingFactoryBuilder()
                                        .with_spell_factory(spell_factory)
                                        .build())
    
    spellcasting_factory.get_action(
                              SpellCastingParameters(self,
                                                     direction = 1, 
                                                     spell_name = 'healing wind'))
    
    verify(spell_factory).create_spell('healing wind')

Essentially, when SpellCastingFactory is used to create a spell called “healing wind”, the SpellFactory should receive a call to make a spell called “healing wind”. I’m creating spell_factory with SpellFactoryBuilder (so I have a real instance and not just a mock object), but then I’m using mockito to tell it to return mock when create_spell – method is called. I would like to use mock, but for some reason I couldn’t get it working with the code in SpellCastingFactoryBuilder:

def with_spell_factory(self, spell_factory):
    """
    Configure spell factory to use
    """
    if spell_factory == None:
        self.use_real_spell_factory = True
    else:
        if hasattr(spell_factory, 'build'):
            self.spell_factory = spell_factory.build()
        else:
            self.spell_factory = spell_factory
    return self

When creating a SpellCastingFactory with the builder, I can supply it with a spell_factory to use. I the spell_factory has method “build”, it will be called and the result is used as a factory, otherwise the supplied object is used. For mock object, hasattr will always return True, so this approach does not work.

Following modification to code makes the test to pass:

@logged
def get_action(self, parameters):
    """
    Create a spell casting action

    :param parameters: parameters used to control creation
    :type parameters: SpellCastingParameters
    """
    spell = self.spell_factory.create_spell(parameters.spell_name)
    
    return SpellCastingAction(caster = parameters.caster, 
                                       direction = parameters.direction,
                                       spell = spell)

Again, really small step forward, but the progression is steady and the direction looks good. I’m going to leave details about finding who the spell hits and how much the mana is subtracted later (both probably ending up in the SpellCastingAction) and concentrate on getting the spell part done.

Code for the changes can be found in the following commit.

pre-commit hook in git

I have been using git as my version control system since beginning with pyherc. I haven’t been using anything really advanced features. Most complex things I have been doing are having couple branches and few merges.

Recently I started thinking that I could try out using pre-commit hook for something useful and created quick and dirty script:

#!/bin/sh
nosetests
code=$?
exit $code

Now, everytime I commit code into my local repository, unit tests are run automatically and commit is aborted if they don’t pass. This is ok for now, since tests take only couple of seconds to run, but I wouldn’t want to use this with test suite that takes 10 seconds.

Doctests

I have known about doctest for quite a while, but never got around to try them out. Today I wrote my first one:

def add_to_tick(self, cost):
    """
    Add cost of action to characters tick, while taking characters speed into account

    For example:
    >> pc = Character(None, None, None)
    >> pc.tick = 5
    >> pc.speed = 2
    >> pc.add_to_tick(5)
    15

    Args:
    cost: Cost of action in ticks

    Returns:
    New ticks
    """
    self.tick = self.tick + (self.speed * cost)
    return self.tick

After running it through nosetests –with-doctest –with-xunit and mangling the xml to html, I end up with:

pyherc.data.model.Character
add_to_tick Ok 0.093

Nothing really fancy and certainly nothing that I couldn’t achieve with normal unit tests. But the key to here is that I’m not testing the code, I’m testing the documentation (namely the code example). If I were to add a new parameter to the function and not update the example, the test would fail.

I’m most likely not going to use doctest very extensively, but maybe there are place or two to use it.

Shortening the feedback loop

One of the reason I like doing test driven development is the really short feedback loop it offers (there are many others, but this post is about the feedback loop). Basic idea is:

  • Write a test that demonstrates the lack of the wanted feature
  • Start developing

While developing (Repeat until acceptance test is passed):

  • Write failing unit test
  • Make the test pass
  • Refactor

All the time, you keep running all the tests that have been written before. This gives confidence that nothing is accidentally broken while the new stuff is being developed and code is being refactored. As soon as the acceptance test passes, feature is ready. Of course it might lack some detail that the acceptance test doesn’t cover, but for those you can always write more acceptance tests.

This is really nifty and I like doing it. While working with .Net code, cycle is like this (in more detailed level):

  • Write test, compile, run tests
  • Code to make the test pass, compile, run tests
  • Refactor, compile, run tests

While working with Python, cycle gets shorter:

  • Write test, run tests
  • Code to make the test pass, run tests
  • Refactor, run tests

This morning while biking to work, I had an idea that it would be really cool if the test suite at the local workstation would work in a similar way to test suite in the continuous integration.

I’m of course not the first one to think this. Two first hits were autonose¬†and nosy. I tried autonose first, but couldn’t get it to install because of mismatch in md5 hash. With nosy I was more lucky and got it installed and quickly set up. I still need to manually start it by simply typing:

nosy

In my top level source folder and system starts itself and runs all the tests found inside the folder hierarchy. After that, everytime a source file is changed tests are automatically run. For fully automated solution on Linux, there’s nosier.

After this, cycle is shortened even more:

  • Write test
  • Make test pass
  • Refactor

For .Net side, there is of course equivalent tools. First one I found is called NCrunch and looks quite promising. I haven’t tested that, but I probably should.