Writing macros with Hy

I have recently been working with some macros with Hy and wanted to write down some of things I have noticed or learned.

In lisp, code is essentially just a large tree (or bunch of lists inside of other lists). These lists are called s-expressions or sexprs for short. Following simple example demonstrates this:

=> (- (+ 2 (* 3 4)) (+ 1 1))
12

Functions work the same way, first element is the function being called and rest of the elements are parameters for that function:

=> (cut [0 1 2 3 4 5] 2 5)
[2 3 4]

One of the advantages of this system is that lists are generally pretty easy to manipulate. One can insert, remove and modify elements easily. If we consider a regular hy program, it’s just a collection of lists that define the code of the program. And this is where macros start to come into play.

A list macro is small snippet of code that can produce code or data (they’re essentially a same thing anyway). This lets programmer to automate writing some parts of the code. Instead of typing similar snippet of code over and over again, programmer writes a macro that does this for them. Since macro is an actual executable piece of code, it can inspect the parameters given to it and use them in its internal logic. They’re really great tool if you want to add new features to the language and extend it in ways that help you to write your programs more elegantly with less effort.

Consider following:

(if (= counter 5) (do (print "we have reached full speed!")
                      (setv warning-level :high)))

If counter has reached 5, we want to display a message and set warning-level to :high. That do form is there to group print and setv together. Without it, setv would be treated as a else branch for the if statement. This is pretty common pattern: when something is true, do multiple things. In fact, it’s so common, that Hy has special form for it: when.

(when (= counter 5)
      (print "we have reached full speed!")
      (setv warning-level :high))

However, when is implemented as a macro. When Hy encounters it in source code, macro is executed and it’s given three parameters: (= counter 5), (print “we have reached full speed!”) and (setv warning-level :high)). Macro then outputs a bit of extra code and inserts the parameters in correct places in that code.

Similar is if-not macro, which works just like if, but in inverted fashion:

(if-not (= motor :started)
        (print "We should get started")
        (print "Ready to go"))

If motor has not been started, print a message urging to get started. Otherwise just report that we’re ready to go. Of course one could write this with regular if statement (in fact, after macro has done its work, there will be just a regular if statement), but sometimes it’s useful to emphasize the different order. In any case, result of this macro is following:

(if (not (= motor :started))
    (print "We should get started")
    (print "Ready to go"))

Macros aren’t limited on outputting code. Since code and data are pretty interchangeable in lisps, macros can be used to output data as well.

Anatomy of macros

Lets write a macro that can be used to double its input and try it out:

=> (defmacro double [x] `(* 2 ~x))
=> (double 2)
4

This is simple macro, accepting a single parameter. To check actual output of it, we can use macroexpand:

=> (macroexpand '(double 2))
'(* 2 2)

So for input of “2”, it will output “(* 2 2)”, which of course evaluates to 4. But how does it do that?

` stands for quasiquote. It is used to output code (data), instead of executing it. ~ stands for unquote. It is used to output value of a name, instead of the name and works only inside of quasiquote. If we didn’t unquote x, we would be outputting “(* 2 x)”, which wouldn’t be what we want.

We can also achieve similar result with following macro:

=> (defmacro double [x] (* 2 x))
=> (double 2)
4

Looks similar enough, but lets check with macroexpand:

=> (macroexpand '(double 2))
4

Notice the difference? Instead of generating code to perform calculation, macro performs calculation and outputs the result. This is possible because “2” is a literal and the value is known while the macro is being run. If we were to try and double result of a function with the macro, things wouldn’t work out as we might expect them to work out:

=> (macroexpand '(double (+ 3 3)))
'[+ 3 3 + 3 3]

Why is this? As earlier hinted, macros are run before the actual code execution begins. So our macro dutifully takes “(+ 3 3)” and multiplies that with 2, resulting with “(+ 3 3 + 3 3)”, which isn’t valid Hy code at all. As somebody eloquently put it “the key, is that you must always keep in mind that the macro is executed now, in your present time, while the functions will be called in the year 2345 by Capt Picard.” Macro only sees s-expression it is supposed to manipulate, not the result on executing any part of it.

Macroexpansion

As mentioned earlier, macros are executed (expanded) before the code is executed. This phase is repeated until there are no macros left to expand. This means, that it’s perfectly fine to write a macro, that outputs another macro, which gets executed in its turn. Suppose, we wanted to write when-not macro, which works in the similar way as if-not:

=> (defmacro when-not [test &rest body] `(when (not ~test) ~@body))
=> (macroexpand '(when-not false (print "success")))
'(if (not False) (do (print "success")))

As you can see, our little when-not macro got expanded all the way to if-not-do construct, that does exactly what we wanted it to do. Our when-not macro outputted form that had when macro in it, which in turn got expanded to if-do form. If we wanted to see individual steps, we could use macroexpand-1 form like following:

=> (macroexpand-1 '(when-not false (print "success")))
'(when (not False) (print "success"))
=> (macroexpand-1 '(when (not false) (print "success")))
'(if (not False) (do (print "success")))

macroexpand-1 performs only one expansion, even if the result has macros in it. It and macroexpand are pretty useful tools when things aren’t working quite as they should and you want to have a closer look to results of macro expansion.

Splicing

You might have noticed a funny looking form in the previous example ~@. This is called unquote-splicing. It works sort of same like unquote, except that it expects a list, which it then splices into result. Consider following examples and their difference:

=> `(1 2 3 ~`(4 5 6))
(1 2 3 (4 5 6))
=> `(1 2 3 ~@`(4 5 6))
(1 2 3 4 5 6)

In the first example, value of second part is inserted into first list as a single element. The result is a nested list, with last element being a list in itself. In the second example, value of the second part is spliced into first list. The result is a list of six elements. The distinction is important for example when working with &rest parameter. &rest parameter is special kind of parameter that allows function to have 0 to n parameters in addition to the normal ones. All extra parameters are available as a single list. We can illustrate this with following examples:

=> (defmacro when-2 [test &rest body] `(if ~test (do ~body)))
=> (macroexpand '(when-2 True (print "hello there") (print "nice to see you")))
'(if* True (do [(print "hello there") (print "nice to see you")]) (if))
=> (when-2 true (print "hello there") (print "nice to see you"))
hello there
nice to see you
[None, None]

Our when-2 macro works otherwise in the same way as the standard when, except it uses unquote insted of unquote splicing. The result is that value of parameter is inserted, thus the final code will have a list, with two print s-expressions inside of it. When executed, messages are printed to screen and value of said list is returned.

More complex macros with logic in them

So far our macros have been pretty small and simple. Lets up the challenge a bit and write a macro that has logic in it. As an exercise, we’ll write macro called when-∧ (by the way, you can use UTF-8 characters in many places with Hy. I like the flexibility that they give in naming.) that accepts list of predicates and body of code. If all predicates are true, body will be executed. In addition to this, as an extra challenge, we don’t want to have and form in the final result if only one predicate was given. Following is an example of usage of our macro:

(when-∧ [(= power :on)
         (> available-power 9000)
         (= coast :clear)]
        (print "all values are true")
        (print "executing phase shift"))

And following is one way of write the macro we want:

(defmacro when-∧ [tests &rest body]
  (if (= (len tests) 1)
      `(if (first ~tests) (do ~@body))
      `(if (and ~@tests) (do ~@body))))

Notice how we don’t start body of macro with quasiquote. Instead of that, we have if form that checks if there are one or more tests present. Depending on the amount of tests, macro outputs code with or without and form. Also, pay attention to the fact that tests parameter isn’t unquoted in the if form. There is no need for that, since we’re not inside a quasiquote. Lets verify that the macro actually works:

=> (macroexpand '(when-∧ [(= power :on)
...                       (> available-power 9000)
...                       (= coast :clear)]
...                      (print "all values are true")
...                      (print "executing phase shift")))
'(if (and (= power :on) (> available_power 9000)
          (= coast :clear)) (do (print "all values are true")
     (print "executing phase shift")))

I had to break output of macroexpand on multiple lines, but so far it looks good.

=> (macroexpand '(when-∧ [(= power :on)] (print "power is on")))
'(if* (and (= power :on) (> available_power 9000)
(= coast :clear)) (do (print "all values are true")
(print "executing phase shift"))
(if))

With just a single test, the and form is omitted, just like we wanted. But, there’s extra call to first, in order to extract the first and only test from the list. Lets see if we can get rid of that too with a little bit of extra work:

(defmacro when-∧ [tests &rest body]
  (if (= (len tests) 1)
      (do (setv only-test (first tests))
          `(if ~only-test (do ~@body)))
      `(if (and ~@tests) (do ~@body))))

So in the revised version we are explicitly taking the first item of tests parameter and assigning it to only-test. When outputting the code, we place value of only-test in it to achieve the result we want. Another, cleaner looking option is to use ~@tests, which will automatically do the correct thing, regardless amount of the tests (This I only figured out after writing the more complex version of the macro). Testing with macroexpand reveals that now the macro works as we want it to work:

=> (macroexpand '(when-∧ [(= power :on)] (print "power is on")))
'(if (= power :on) (do (print "power is on")))

Symbols and name collisions

While writing macros, it pays to keep in mind how and where they might be used. If the generated code includes any symbols, those symbols are not going to live in isolation from rest of the program. Sometimes this interaction is wanted, while sometimes it’s best avoided. Good example of wanted interaction can be found in Hy’s anaphoric macros module. When used, these macros add a new symbol it that can be used. For example, an anaphoric version of common map-function can be used as following:

=> (require hy.contrib.anaphoric)
=> (list (ap-map (+ 2 it) [1 2 3 4 5]))
[3, 4, 5, 6, 7]

The macro walks through the iterable (a list of 5 elements in this case), binds current value to symbol it and calls supplied form. Results are collected to a list that is returned.

But when a name collision isn’t a desirable thing? Consider following contrived example:

(defmacro mix-up [input-data]
`(ap-each ~input-data
         (do (setv data (* 2 it))
             (if (> 4 data) (print data)
                            (print it)))))

Macro will output code that iterates through elements in a list, printing either value of the element or double of it. See how in the middle of ap-each loop we’re assigning a value to data symbol? In isolation this will work, but if there is already a symbol named data, it will get overwritten.

=> (setv data :do-not-touch)
=> (mix-up [1 2 3 4 5 6])
2
2
3
4
5
6
=> data
12

Luckily, there’s an easy way around this with defmacro/g!. It’s a different version of our trusty defmacro that can automatically generate us unique symbols. Every symbol that starts with g! will have a new name generated to it.

(defmacro/g! mix-up [input-data]
`(ap-each ~input-data
         (do (setv ~g!data (* 2 it))
             (if (> 4 ~g!data) (print ~g!data)
                               (print it)))))

After this change, our example works:

=> (setv data :do-not-touch)
=> (mix-up [1 2 3 4 5 6])
2
2
3
4
5
6
=> data
:do-not-touch'

We can see new symbols, if we check generated code with macroexpand

=> (macroexpand '(mix-up [1 2 3 4 5 6]))
'(for* [it [1 2 3 4 5 6]] (do (setv :data_1273 (* 2 it))
   (if (> 4 :data_1273) (print :data_1273) (print it))))

This was somewhat lengthy post and I still only scraped the surface on writing macros. Hopefully there are no omissions, mistakes or any other problems with the text. If you do spot any or have feed back to give, I would be glad to hear it.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s