Hy, Nose and Hypothesis

I recently came across a tool called Hypothesis that immediately sparked my interest. Their documentation describe it as:

Hypothesis is a Python library for creating unit tests which are simpler to write and more powerful when run, finding edge cases in your code you wouldn’t have thought to look for. It is stable, powerful and easy to add to any existing test suite.

Stable, powerful and easy to add are all words that I like. And if you know me, you quickly guessed that I wanted a Hy interface for this new shiny tool.

Translating examples in their getting started docs provided me the following:

(with-decorator (given :s (text))
  (with-decorator (example :s "")
    (defn test-decode-inverts-encode [s]
      (assert-that (-> (encode s)
                       (decode))
                   (is- (equal-to s))))))

Lots and lots of nested things, with the test name buried somewhere in the middle. Luckilly, I had started working on a testing DSL in my earlier blog post and I decided to expand it to support some of the features of Hypothesis. Disclaimer: I haven’t even read through the whole documentation proprerly, so I’m probably missing some of the finer points of Hypothesis. I’m pretty much just looking how to create simple DSL on Hy, that can leverage features shown in the example.

What I wanted to be able to write and execute is this:

(fact "decode inverts encode"
      (variants :s (text))
      (sample :s "")
      (assert-that (decode (encode s))
                   (is- (equal-to s))))

Testing function parameters are automatically generated from variants special form and two decorators are controlled by variants and sample special forms. Both forms are optional, but sample can’t be present without variants form.

Building such a macro seemed somewhat daunting task, so I broke it into parts and started with:

(defmacro fact [desc &rest code]
  (-> (create-code-block)
      (create-func-definition)
      (create-sample-decorator)
      (create-given-decorator)))

Easy enough, right? Rest is just filling in details. I chose to build the resulting code from inside out, starting with the actual code block and then adding function definition around it. Last two steps add decorators if they’re needed. This was easier than starting with decorators and then trying to insert more code into correct position inside of them.

Before tackling the juicy parts of the code, I wrote couple helper functions that are used to detect if variants or sample forms are present. Both are defined inside of the macro, so I have access to them during the compilation:

  (defn variants? []
    "check if variants are specified"
    (= 'variants (first (first code))))

  (defn samples? []
    "check if samples are specified"
    (if (>= (len code) 2)
          (= 'sample (first (second code)))
          false))

As they aren’t meant to be reusable, both functions directly access to variables in enclosing scope (macrodef that is). There should be more error handling and parameter checking in the code. Currently it’s far too easy to break the macro by feeding it funny looking code. Last one of the helpers is group that can be used to group a sequence to sublists of specified length. This implementation has a problem though and won’t work in Python 2.7, due to a bug in Hy. This function comes handy, when we want to separate argument values from their names (for example).

  (defn group [seq &optional [n 2]]
    "group list to lists of size n"
    (setv val [])
    (for [x seq]
      (.append val x)
      (when (>= (len val) n)
        (yield val)
        (setv val [])))
    (when val (yield val)))

First step is to create code block which is executed in test. It needs to be able to handle three different cases, depending on if variants or samples are present. Essentially, it just skips over those forms and outputs everything else.

  (defn create-code-block []
    "create test function body"
    (cond [(and (variants?)
                (samples?)) (rest (rest code))]
          [(variants?) (rest code)]
          [true code]))

Wrapping the code block inside a function definition comes next. Function name is constructed by appending “test_” and user supplied test description together and replacing spaces with underscores. This gives us a understandable function name that is also valid from point of view of normal Python code. Parameter list is constructed by taking keywords from variants special form and turning them into normal strings and then in HySymbols. If no variants is present, we just use an empty list. Final touch is to make sure that the function doc string matches to description given by user, as this is what they’ll use to identify test in the results.

  (defn create-func-definition [res]
    "create function header and splice in res"
    (let [[fn-name (HySymbol (.join "" ["test_" (.replace (str desc) " " "_")]))]
          [param-list (if (variants?)
                        (list (ap-map (HySymbol (name (first it)))
                                      (group (rest (first code)))))
                        `[])]]
      `(defn ~fn-name ~param-list
         ~desc
       ~@res)))

If samples special form is present, the function is then wrapped inside a decorator. If not, we’re not doing anything in this step. Since I chose to have this part of the DSL match 1 to 1 with Hypothesis, the code is quite simple: Just add correct decorator and splice in contents of samples special form and function so far. Final step is to repeat the process with variants and then we’re done:

  (defn create-sample-decorator [res]
    "create decorator for sample data and splice in res"
    (if (samples?)
      `(with-decorator (example ~@(rest (second code)))
         ~res)
      res))

  (defn create-given-decorator [res]
    "create decorator for test data generators and splice in res"
    (if (variants?)
      `(with-decorator (given ~@(rest (first code)))
         ~res)
      res))

At this point, the example test translated from Hypothesis docs is runnable and works just fine. As an exercise, I wrote couple more tests (or facts, however you want to put it) about group function:

(fact "grouping an empty list will return an empty list"
      (assert-that (list (group []))
                   (has-length 0)))

(fact "groups of grouped list are equal or less than max size"
      (variants :seq (lists (text) :min-size 1)
                :n (integers :min-value 1))
      (ap-each (group seq n)
               (assert-that it (has-length (less-than-or-equal-to n)))))

(fact "flattening grouped sequence produces original sequence"
      (variants :seq (lists (text))
                :n (integers :min-value 1))
      (sample :seq [] :n 1)
      (assert-that (flatten (group seq n))
                   (is- (equal-to seq))))

Final code along with the examples is available in this gist. As mentioned, it lacks all the error handling and probably some of the features of Hypothesis aren’t supported. But it’s a start and I’m planning to try it on writing tests for pyherc in the future.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s