Testing Hy macros

I like my code tested and working. I like it even better when it’s done automatically. But recently I was faced with a new kind of code that proved to be rather different beast to tackle: macros. Problem with testing them is that macros usually capture some common, general pattern and give it a name. If one were to use the same approach as with testing function, they would feed specific parameters to macro and assert that the generated code looks like what it should look like. This could soon lead into brittle, hard to maintain tests, since tests would specify very precisely what the generated code should look like. Even minor changes to implementation would break most of the tests, even if the functionality of the generated code wouldn’t change.

Better option (depending on the case again) would be to write code that uses that macro and assert that the code has correct functionality. Depending on the complexity of the macro this might require more than single test in order to cover all possibilities. This is the way I usually test my macros, when I bother testing them at all. Most of the time I just assume that if there are any problems with the macro implementation, tests covering code where it is used will catch problems.

But what about if I wanted to test that macro-error is called in certain case? macro-error will stop macro expansion after all. If I have a test function with code that calls macro-error during the macro expansion, the test function will never get executed, thus no any kind of assertions about the resulting code can be made (since there is no resulting code). I started digging source code of Hy and first checked implementation of macro-error. So HyMacroExpansionError is being raised when macro expansion fails. But how to capture an exception that gets raised before the test code is running?

The answer was simple (I got pointed to correct solution on #hy at freenode): use macroexpand inside try form and catch the error:

(setv ok False)
  (macroexpand '(cond (= 1 1) True))
  (catch [e HyMacroExpansionError]
    (setv ok True)))
(when (not ok) (assert Fail)

Notice that I’m quoting the offending macro, so it doesn’t get expanded before macroexpand is called during test execution. After getting this far, rest was just a mechanical excercise of writing small macro that captures this pattern and polishing it a bit. Previous code could then be written as:

(fact "macro errors can be asserted"
      (assert-macro-error "cond branches need to be a list"
                          (cond (= 1 1) true)))

This not only checks that the macro-error is called, but also checks that reported reason matched to the expected one. After couple rounds of polish, I ended up with the following version (this one still uses now obsolete let):

(defmacro/g! assert-macro-error [error-str code]
  `(let [[~g!result (try
                      (import [hy.errors [HyMacroExpansionError]])
                      (macroexpand (quote ~code))
                      "no exception raised")
                     (catch [~g!e HyMacroExpansionError]
                       (if (= (. ~g!e message) ~error-str)
                         (.format "expected: '{0}'\n  got: '{1}'"
                                  (. ~g!e message)))))]]
     (when ~g!result
       (assert false ~g!result))))

The assert-macro-error is included in the 0.2 version of archimedes.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s