Some hylarious macro shenanigans

Just a word of warning: what I’m about to describe here makes even less sense than what I usually write. It’s probably very bad idea for various reasons and definitely shouldn’t be used in production code. However, it might come handy in some limited situations and offer short term gains that justify the bad things. Use your own judgement.

As you might know, Hy is gearing towards the next release and there will be some major, code breaking changes. Things like let and defclass will have changed syntax and some aliases will be dropped (true doesn’t work anymore, only True). After talking with various people, I started thinking “what kind of system could help me to port old code to new one?”. And of course I turned for macros. So, here I’m going to show basic implementation for a macro that takes Hy code written in old syntax and transforms it for the new syntax. One can then wrap all existing code in that macro, update Hy and things hopefully work right out of box. Then it’s just matter of manually fixing things on the new syntax, while having test suite to check that there has not been any errors in process. Sounds easy enough, right?

Very basic skeleton for our macro:

(defmacro update-to-0.12 [&rest forms]
  "take possibly old code and output new code"

  (print "Running experimental update-to-0.12 macro. Consistency not quaranteed.")
  `(do ~@(map update-to-new forms)))

Since we want to be able to update all code in a file, without having to wrap each and every expression inside of our macro, we’ll define it to take 0 or more forms to transform. After printing a warning about experimental status of our macro, we’ll call update-to-new for each entry in forms and output result as new code.

update-to-new is going to dispatch to various functions that do the actual heavy lifting. I was entertaining idea of data driven structure, where functions would be registered into some central location and dispatch were automatic, but decided against it. I’m not expecting this code to grow much larger and it can always be restructured later:

  (defn update-to-new [form]
    "take symbol or expression and transform it to new syntax if needed"
    (if (symbol? form) (update-symbol form)
        (let? form) (update-let form)
        (require? form) (update-require form)
        (with? form) (update-with form)
        (defclass? form) (update-defclass form)
        (expression? form) (update-expression form)
        (list? form) (update-list form)
        (dictionary? form) (update-dict form)
        form))

Bare symbols (nil, true and such) have their own handling in the beginning of the block. After that come more complex expressions like let, require and with. After these are more generic containers (expressions, lists and dictionaries). If no rule matches, we just return what ever was passed into and hope for the best.

I’m not going to paste all code for detecting different forms as they’re very similar as the one handling let below:

  (defn expression? [form]
    "is given form HyExpression?"
    (is (type form) HyExpression)) 

  (defn let? [form]
    "is given form let?"
    (and (expression? form)
         (= (first form) 'let)))

Transforming symbols from old to new is probably the easiest of the bunch. It’s for taking care of transformation of: False -> false and throw -> raise. As such, the code is just mapping of old and new symbols and comparison:

  (defn update-symbol [symbol]
    "take single symbol and replace it with new"
    (setv mapping {'nil 'None
                   'true 'True
                   'false 'False
                   'progn 'do
                   'throw 'raise
                   'catch 'expect
                   'defun 'defn
                   'lisp-if 'lif
                   'lisp-if-not 'lif-not})
    (if (in symbol mapping)
      (do (report symbol)
          (get mapping symbol))
      symbol))

Default is to return symbol without translating it to anything. Only if there’s a match in mapping dictionary, the new one will be returned instead.

Require is a bit of special case of the bunch. In previous versions of Hy, require loaded all macros from specified package. In the new version, syntax is same as with import. One can now require just specific macros, all macros or all macros, but leave them in the package’s namespace. I could have replaces (require foo) with (require [foo [*]]), but that only works if I can be sure that the input is in old syntax. As there is no way to test it, I’m just emitting a warning, without doing any transformations.

Updating let is where things get a bit more complex. Here we have to detect if code is written in old or new style and possibly translate old to new syntax. Following snippet shows difference between the old and new syntax:

(let [[a 1]
      [b 2]
      [c (+ a b)]]
  (print a b c))

(let [a 1
      b 2
      c (+ a b)]
  (print a b c))

It’s worth noting that the value being bound to a symbol might be a value of an expression. And that expression could be complicated and contain code written in the old syntax (it could even be a nested let). For this reason, we have to recursively process values and transform them into new syntax where needed.

Main part of the routine is this check and unrolling list-of-lists to be just a simple list:

    (if (new-form? form)
      form
      (do (report form)
          (setv new-binding [])
          (setv body (update-to-new (cut form 2)))
          (for [x (second form)] (add-to new-binding x))
          `(let ~new-binding ~@body)))

Couple helpers that the update-let uses are short:

    (defn new-form? [form]
      "is this let expression in new syntax?"
      (setv binding-form (second form))
      (not (all (map list? binding-form))))

    (defn add-to [new-form elem]
      (.append new-form (first elem))
      (.append new-form (update-to-new (second elem))))

new-form? detects if given let is using list of lists do define bindings or if it’s simple list. add-to is helper that is used to unroll list of lists into simple list.

update-defclass is quite similar to update-let, so I’m not copying it here. Main difference is in shape of expression being handled, otherwise it’s just the same unrolling list-of-lists into simple list as update-let is too.

There are couple special cases left that the routine doesn’t yet handle. In following example, we have expression that itself isn’t written in syntax that needs transforming, but part of it (those two lets) is:

(+ (let [[a 1]]
     a)
   (let [[a 2]]
     a))

To handle such a case, we need to have routine that recursively traverses any given expression and calls update-new to each element in it:

  (defn update-expression [form] 
    "take expression and transform it to new syntax if needed"
    `(~@(map update-to-new form)))

Another special case are literals, specifically lists and dictionaries:

[nil true false]

{:a nil :b true :c false}

Handling such cases is almost identical to update-expression, only enclosing symbols differ:

   (defn update-list [form] 
     "take list and update each element if needed"
     `[~@(map update-to-new form)])

That’s about it. With such a macro it’s possible to run (to a degree) code that is in old syntax:

=> (update-to-0.12
...  (let [[a 1]
...        [b 5]
...        [c nil]]
...    (if c
...      (+ a b)
...      (* a b))))
5

However, there’s couple improvements we can do. If we’re updating only one form or symbol, the macro really wouldn’t need to wrap the result in do. This is easy to fix with single if:

  (if (> (len forms) 1)
    `(do ~@(map update-to-new forms))
    (update-to-new (first forms)))

Another thing is that update-to-0.12 is rather mouthful and requires a set of parenthesis to wrap the code being updated. More space conserving solution is to use reader macro:

(defreader U [form]
  `(update-to-0.12 ~form))

After this, we can perform upgrade to a single symbol or form with simple syntax:

=> #U(let [[a 1]
...        [b 5]
...        [c nil]]
...    (if c
...      (+ a b)
...      (* a b)))
5

Ideas, critique or thoughts? I’m interested to hear what you have to say about this. Below is the complete macro once more:

(defmacro update-to-0.12 [&rest forms]
  "take possibly old code and output new code"

  (defn list? [form]
    "is given form HyList?"
    (is (type form) HyList))

  (defn dictionary? [form]
    "is given form HyDict"
    (is (type form) HyDict))
    
  (defn expression? [form]
    "is given form HyExpression?"
    (is (type form) HyExpression)) 

  (defn let? [form]
    "is given form let?"
    (and (expression? form)
         (= (first form) 'let)))

  (defn require? [form]
    "is given form require?"
    (and (expression? form)
         (= (first form) 'require)))

  (defn with? [form]
    "is given form with?"
    (and (expression? form)
         (= (first form) 'with)))

  (defn defclass? [form]
    "is given form defclass?"
    (and (expression? form)
         (= (first form) 'defclass)))
         
  (defn report [form]
    "report that old syntax was detected"
    (print "replacing old syntax:" form))

    
  (defn update-symbol [symbol]
    "take single symbol and replace it with new"
    (setv mapping {'nil 'None
                   'true 'True
                   'false 'False
                   'progn 'do
                   'throw 'raise
                   'catch 'expect
                   'defun 'defn
                   'lisp-if 'lif
                   'lisp-if-not 'lif-not})
    (if (in symbol mapping)
      (do (report symbol)
          (get mapping symbol))
      symbol))

      
  (defn update-let [form]
    "take let expression and transform it to new syntax if needed"
   
    (defn new-form? [form]
      "is this let expression in new syntax?"
      (setv binding-form (second form))
      (not (all (map list? binding-form))))

    (defn add-to [new-form elem]
      (.append new-form (first elem))
      (.append new-form (update-to-new (second elem))))
     
    (if (new-form? form)
      form
      (do (report form)
          (setv new-binding [])
          (setv body (update-to-new (cut form 2)))
          (for [x (second form)] (add-to new-binding x))
          `(let ~new-binding ~@body))))

          
  (defn update-require [form]
    "take require expression, output a warning and return original expression"
    (print "can't update require: " form)
    form)

    
  (defn update-with [form]
    "take with form and transform it to new syntax if needed"
    (print "with update not implemented yet!")
    form)

    
  (defn update-defclass [form]
    "take defclass form and transform it to new syntax if needed"
    
    (defn new-form? [form]
      "is this defclass expression in new syntax"
      (setv attributes (get form 3))
      (if (= (len attributes) 0) True
          (list? (first attributes)) False
          True))

    (defn add-to [new-form elem]
      (.append new-form (first elem))
      (.append new-form (update-to-new (second elem))))
          
    (if (new-form? form)
      form
      (do (report form)
          (setv new-attributes [])
          (for [x (get form 3)] (add-to new-attributes x))
          `(defclass ~(second form) ~(get form 2) ~new-attributes))))

    
  (defn update-expression [form] 
    "take expression and transform it to new syntax if needed"
    `(~@(map update-to-new form)))

    
   (defn update-list [form] 
     "take list and update each element if needed"
     `[~@(map update-to-new form)])


    (defn update-dict [form]
      "take dictionary and update each element if needed"
      `{~@(map update-to-new form)})
     
  (defn update-to-new [form]
    "take symbol or expression and transform it to new syntax if needed"
    (if (symbol? form) (update-symbol form)
        (let? form) (update-let form)
        (require? form) (update-require form)
        (with? form) (update-with form)
        (defclass? form) (update-defclass form)
        (expression? form) (update-expression form)
        (list? form) (update-list form)
        (dictionary? form) (update-dict form)
        form))
 
  (print "Running experimental update-to-0.12 macro. Consistency not quaranteed.")
  (if (> (len forms) 1)
    `(do ~@(map update-to-new forms))
    (update-to-new (first forms))))

(defreader U [form]
  `(update-to-0.12 ~form))
Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s