History of let

Let is rather integral part of lisps as it is used to define lexical scope for a one or more symbols and bind values to them. However, since version 0.13.0 Hy doesn’t have one. Lets have a look what lead to this decision.

We have to remember that Hy is first and foremost just a clever preprocessor that translates lisp syntax into Python AST. Therefore we’re bound by the rules of Python, albeit we can sometimes try to work around them.

In order to examine what happens under the hood, we launch Hy REPL with –spy switch. This causes REPL to show to what kind of Python code our Hy code was translated, before executing it:

$ hy --spy
hy 0.12.1 using CPython(default) 3.4.3 on Linux
=> (+ 1 2 3)
((1 + 2) + 3)
6
=>

Like I mentioned earlier, Hy is bound by the rules of Python. Hy translates let to a function that gets called immediately:

=> (let [a 1 b 2]
...  (+ a b))

def _hy_anon_fn_1():
    a = 1
    b = 2
    (a, b)
    return (a + b)
_hy_anon_fn_1()

3

This is a handy trick that ensures that even if a or b had been defined before, they would have new values inside the let and any changes made there wouldn’t affect the original values:

=> (let [a 1 b 2]
...  (let [a 2 c 3]
...    (+ a b c)))

def _hy_anon_fn_2():
    a = 1
    b = 2
    (a, b)

    def _hy_anon_fn_1():
        a = 2
        c = 3
        (a, c)
        return ((a + b) + c)
    return _hy_anon_fn_1()
_hy_anon_fn_2()

7

Here lies many subtle and not so subtle obstacles that one needs to navigate in case they would want to use this system. They tended to pop up now and then, raise lots of confusion and require debugging and work.

For example, if you had a variable outside of let block that you wanted to mutate while inside of the let block, you had to remember that there were invisible function in the play. Otherwise you ended up modifying wrong variable:

=> (defn test []
...  (setv a 1)
...  (let [b 2]
...    (setv a b))
...  a)

def test():
    a = 1

    def _hy_anon_fn_1():
        b = 2
        a = b
        return a
    _hy_anon_fn_1()
    return a

=> (test)

test()

1

In the previous example, there are two symbols name to a, the outer and the inner. Changes to the inner one aren’t reflected on the outer one and thus incorrect value of 1 is returned instead of the expected 2. The solution to this problem is to use nonlocal:

=> (defn test []
...  (setv a 1)
...  (let [b 2]
...    (nonlocal a)
...    (setv a b))
...  a)

def test():
    a = 1

    def _hy_anon_fn_1():
        b = 2
        nonlocal a
        a = b
        return a
    _hy_anon_fn_1()
    return a

=> (test)

test()

2

Hy can’t put the nonlocal there by default, because then there wouldn’t be much point in having let blocks to begin with. Programmer has to know about the hidden function and plan their code around this fact.

Another error (as shown in issue 1212) happened when symbol being bound in let had same name as symbol in the outer scope:

=> (defn test [x]
...  (let [x (inc x)]
...    (print x)))

from hy.core.language import inc

def test(x):

    def _hy_anon_fn_1():
        x = inc(x)
        return print(x)
    return _hy_anon_fn_1()

=> (test 2)

test(2)

Traceback (most recent call last):
  File "/usr/lib/python3.4/code.py", line 90, in runcode
    exec(code, self.locals)
  File "<input>", line 1, in <module>
  File "<input>", line 1, in test
  File "<input>", line 2, in _hy_anon_fn_1
UnboundLocalError: local variable 'x' referenced before assignment

The fix suggested in the GitHub issue would have fixed this error, but instead of applying yet another patch over the sore point, let was completely removed.

Since let introduces an invisible function, placing one inside a for loop and breaking out is asking for trouble as the break keyword is allowed only inside of a loop. The following example demonstrates this:

=> (for [x (range 10)]
...   (let [res (+ x 1)]
...     (if (> x 5)
...   (break)
...   (print res))))

from hy.core.language import range
for x in range(10):

    def _hy_anon_fn_1():
        res = (x + 1)
        if (x > 5):
            break
            _hy_anon_var_1 = None
        else:
            _hy_anon_var_1 = print(res)
        return _hy_anon_var_1
    _hy_anon_fn_1()

Traceback (most recent call last):
...
  File "<input>", line 1
SyntaxError: 'break' outside loop

The easiest way to fix this is to get rid of let (but then there isn’t local scope anymore of course):

=> (for [x (range 10)]
...   (setv res (+ x 1))
...   (if (> x 5)
... (break)
... (print res)))

from hy.core.language import range
for x in range(10):
    res = (x + 1)
    if (x > 5):
        break
        _hy_anon_var_1 = None
    else:
        _hy_anon_var_1 = print(res)

1
2
3
4
5
6

Even more obscure bug was highlighted in issue 619. Essentially, the invisible function created by let can lead to infinite loop when combined with while and yield. First the incorrect version:

=> (defn test []
...   (while True
...     (let [foo "bar"]
...       (yield foo))))

def test():
    while True:

        def _hy_anon_fn_1():
            foo = 'bar'
            return yield foo
        _hy_anon_fn_1()

=> (test)

test()

^CTraceback (most recent call last):
  File "/usr/lib/python3.4/code.py", line 90, in runcode
    exec(code, self.locals)
  File "<input>", line 1, in <module>
  File "<input>", line 257, in test
KeyboardInterrupt

As you can see, the resulting Python code has an infinite loop and inside of the loop there’s a new function (introduced by let) and inside of that there’s the yield that the programmer expected to yield “bar”. But because the yield is inside of the function that gets called as the last step of the while loop, the resulting generator object doesn’t return execution outside of while. Instead a new generator object is created indefinitely. One way to fix this is to reorder the code:

=> (defn test []
...   (let [foo "bar"]
...     (while True
...       (yield foo))))

def test():

    def _hy_anon_fn_1():
        foo = 'bar'
        while True:
            yield foo
    return _hy_anon_fn_1()

=> (test)

test()

<generator object _hy_anon_fn_1 at 0xb663fa54>

While this works, it’s not exactly the same end result. What if the let block contained more logic and can foo should be evaluated each time before yielding it? That behaviour is not possible with the latter code.

Most likely I forgot some edge or corner cases with let in this blog posting and most likely we didn’t even saw them all. The end result was in any case that the let was removed because it was causing trouble all the time and nobody managed to come up with a reasonable solution that would have fixed all the known issues.

Advertisements

5 thoughts on “History of let

  1. In case i ever find myself designing a programming language, what facilities would Python have to support in order to allow ‘let’ to work? It seems to me that in theory Hy could solve the problem with which variables are to be marked ‘nonlocal’ by doing a lot of additional code analysis (is this correct?).

    I gather that the main remaining problem is that break, continue, yield, async, await work differently if you covertly introduce a new function scope. Would it be sufficient if scopes could have explicit labels and break, continue, yield took an argument that said which label to break/continue/yield out of? Would that solve the problem with async, await too?

    If Hy did extensive code analysis to determine where to introduce ‘nonlocal’, and if Python’s break/continue/yield took this extra argument, would implementation of ‘let’ by Hy then become possible or are there still additional impediments that i am missing?

    • Thanks for the reply. Answering took a while as I had to ponder about things in more detail and be extra careful not to commit any big plunders.

      Extensive code analysis probably can’t reliable detect when to place nonlocal and when not. Sometimes programmer wants to introduce a new symbol that shadows existing one, sometimes they want to modify value of already existing one that is from outer scope. In order the system to detect what the programmer wants to do in a specific case, it would have to understand semantics of the problem the programmer is trying to solve and how they’re trying to solve it. It would require cognitive capabilities close to human. It really boils down to the fact that while sometimes we would like to have that nonlocal keyword there, knowing what the programmer had in mind when writing partical piece of code is really tricky for computers to do.

      Extra labels probably made it at least a little bit easier to deal with the covert function. If we had a situation like this (formatting probably breaks though):

      .(for [x (range 10)]
      …(let [res (+ x 1)]
      …..(if (> x 5)
      …(break)
      (print res))))

      System would have to detect that break is inside a let and instead of just placing break in the resulting output, generate more sophisticated solution. In theory, it could use return to exit function and signal with special return value that next action should be breaking out of loop. This way the break wouldn’t be inside a function, but outside it, thus in correct scope:

      .for x in range(10):
      …..def _hy_anon_fn_1():
      ……..res = (x + 1)
      ……..if (x > 5):
      …………return “break requested”
      ……..else:
      …………_hy_anon_var_1 = print(res)
      ……..return _hy_anon_var_1

      …._hy_anon_var_2 = _hy_anon_fn_1()
      …..if _hy_anon_var_2 = “break requested”:
      ………break
      …..else:
      ………return _hy_anon_var_2

      And you would have to keep in mind that there might be multiple lets nested each other, each introducing a new function that we first need to escape before issuing break command. I haven’t really looked into what problem we had with async/await, but I imagine it being something similar.

      It’s rather hairy problem, with lots and lots of corner cases and I suspect that we didn’t even encounter all of them by the time it was decided that let should be dropped. We really wanted to have let, but it was just too much of trouble for half-working solution that generated many strange and confusing bugs.

      • Thank you so much for the detailed response! I think I understand it a little better now.

    • I remember context managers being mentioned in regards to “let”, but I couldn’t find any discussion on GitHub tickets. Maybe it happened in IRC or maybe I misremember. In any case, I gave this some pondering, as I had completely forgotten this avenue of thought. But you’re completely right, “with” statement could be used to create sub-scope.

      I think, that it would be possible to build some sort of system using context managers to handle scope, but it would be rather complicated. If I’m not missing something crucial, variable holding the context manager should be named the same as the variable introduced by let. And every time “setv” (or any other function really) tries to access a variable, it should first check if there’s a context manager and use the value store in it instead of a regular variable. This system wouldn’t have problem with “yield”, but I’m thinking it would be rather complicated piece of code and interoperability with Python would be tricky.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s