A lisp macro virgin tells all

I finished my first lisp macro, and I want to tell the world.

I’ll talk about what a lisp macro is, and what makes it unique in the world of programming, how it’s a technique only possible in lisp. I’ll then take you through an example.

So firstly, what’s a lisp macro, and why would you want to write one?

So, you may have seen lisp programs before, and you’ll recognise them instantly — Larry Wall, the inventor of [Perl][], said they had all the aesthetic appeal of a bowl of porridge mixed with toenail clippings;

(defun accumulate (combiner lst initial)
(let ((accum initial))
(dolist (i lst)
(setf accum (funcall combiner accum i)))

He has a point. They are butt-ugly. But hell, the best he came up with is [Perl][], so he can `$_@++` right off. (I’m pretty sure that’s valid Perl, too 😉 )

It’s ugly, in an aesthetic way, but it’s amazingly practical. It’s got an engineering beauty to it. If you look at that snippet above, you’ll notice that the whole program is made out of exactly three types of symbols;

* open parenthesis: `(`
* close parenthesis: `)`
* symbols, like `defun`, `accum` and `setf`

All simple lisp programs are like this. Just brackets to group stuff together, and stuff that needs grouping. Compare that with C#, where you might find;

* parenthesis for;
* function calls; `print(“hello”)`
* special forms; `using(OdbcConnection con = …)`
* semi-colon to end statements; `int x = 1;`
* curly brackets for;
* code blocks; `{ /* code block */ }`
* array initialisers; `string[] words = { “hello”, “world” };`
* square brackets for array indexing; `x[3] = 4`;

and the list goes on. I gave up because there are too many to list.

So lisp has this seriously small syntactic footprint. You can have a thing, or a group of things in brackets. It’s simple. It’s *so* simple that you can start doing crazy stuff in lisp that you just can’t do otherwise. That crazy stuff goes by the name of macros.

I can write a program that takes a chunk of lisp (remember, just a thing or a list of things), cuts it up, and reassembles it. That creates new lisp code.

So imagine you do a lot of work on three-dimensional arrays. You find yourself, over and over, writing nested loops that say;

for x in range(100):
for y in range(100):
for z in range(100):
# do something to matrix[x,y,z]

And frankly, you’re bored of typing it over and over. What you really want to do is something like;

for {x 100, y 100, z 100}:
# do something to matrix[x,y,z]

You want a brand new bit of syntax for multiple-value looping. Can you add it to python? Nope. C? Nope. Java? Nope.

But now look at the lisp version;

I could, theoretically, write this

(domanytimes (x 100 y 100 z 100)

and, because it’s just a list of stuff, I can chop and change that into this new bit of lisp;

(dotimes (x 100)
(dotimes (y 100)
(dotimes (z 100)

I’ll show you how in a second, but notice what’s possible — I can write my own looping construct (`domanytimes`) and lisp will rewrite it into many simpler looping construct (the built-in `dotimes`).

Is that particularly special? Well, yeah. I’ve written new syntax. I’ve defined a new way of looping that is no different from the standard loops. I’ve basically added something new to the language. Lisp is now better at dealing with multi-dimensional loops. Try adding a new loop to ruby, or javascript. Make python understand

for x in range(100), y in range(100), z in range(100):
# body here

and you’ll find you can’t.

So I’ve made my version of lisp a bit better at handling loops. If I were writing database code, I could make lisp better at writing SQL statements or data access layers. C# recently got built-in DAL logic with [LINQ][], and it’s great, but only the C# team can write it. Whereas a lisper could write this sort of code;

(sql-select (ID NAME) from PROJECT where (DUEDATE > TODAY))

and it’s do basically the same thing as [LINQ][].

So that’s the why’s and wherefores. Here’s the how of the `domanytimes` macro.

`domanytimes` takes two parts; the loop variables `(x 100 y 100 z 100)` and whatever body you want to execute. We’re going to write a program that skims two elements from the front of the loop variables (say, `x` and `100`) and uses them to write a built-in `dotimes` loop; so a program which converts

(domanytimes (x 100 y 100) body)


(dotimes (x 100)
(domanytimes (y 100) body))

and then again to give you

(dotimes (x 100)
(dotimes (y 100)

Here’s the `domanytimes` macro, in all it’s eye-bleeding horror;

(defmacro domanytimes (loop-list &body body)
“allows you to write (domanytimes (x 10 y 10) …)
instead of (dotimes (x 10) (dotimes (y 10)) body ))”
(if (eq (length loop-list) 0)
;; we have our form to execute
`(progn ,@body)
;; we have more loops to arrange
(let ((fst (car loop-list))
(snd (cadr loop-list))
(rst (cddr loop-list)))
`(dotimes (,fst ,snd)
(domanytimes ,rst ,@body)))))

There. Wasn’t that fun? 😉

It looks nasty, I know. All lisp looks nasty. But it’s actually created something new in the language. As far as I understand it, lisp has survived for fifty years basically because the macro system lets you write macros which can add any new kind of syntax you like. You can write knock up a set of macros to [implement OO][clos], and suddenly lisp is OO. You can know up macros for manipulating lazy lists, and suddenly lisp has a [lazy evaluation][lazy]. You can knock up data access layer macros, and it’s got a version of [LINQ][linq]. There seems to be nothing you can’t hack lisp into being.

And if you want to know how the hell that works, I’d recommend [Practical Common Lisp][pcl], which is online and free.

[linq]: http://msdn2.microsoft.com/en-gb/netframework/aa904594.aspx
[lazy]: http://en.wikipedia.org/wiki/Lazy_evaluation
[clos]: http://en.wikipedia.org/wiki/CLOS
[perl]: http://www.perl.com/
[pcl]: http://gigamonkeys.com/book/

lisp, the beautiful hydra

I’m currently learning the programming language, [Lisp][]. If you’re not a programmer, you may wish to simply ignore this post…

Lisp doesn’t have the mindshare it deserves. At fifty, it is the second-oldest programming language in the world, after [Fortran][]. I’m starting to see the whole history of programming as a struggle between the elder brother, Fortran, and the younger sister, Lisp.

[Lisp]: http://en.wikipedia.org/wiki/Lisp_programming_language
[Fortran]: http://en.wikipedia.org/wiki/Fortran

Fortran Vs Lisp

Programming is, at it’s core, an attempt to write down the solution to a problem so precisely that a mechanical device can perform the solution. We start with our own ideas, then use the program and a compiler to give us machine instructions;

> ideas -> program -> machine instructions

There are two approaches; one is to find better ways to describe machine instructions, which is what FORTRAN does. The other is to find better ways to describe ideas, which is what Lisp does. These two languages established for us an axis; Almost every programming language thereafter fits somewhere in between these two giants.

So far, my impression is that lisp is an excellent language, let down by it’s awful libraries and tools; compared to Ruby, Python, Java, or C#, it just doesn’t have the libraries, and getting extant libraries installed is a dog. Worse, there is no canonical implementation, which means that your code may or may not work on someone else’s lisp; even worse news for libraries. It’s a pity, because the language itself seems beautiful and powerful. Meh.

A programming language only a mother could love

For the last few days, I’ve been trying to get my head around Common Lisp. Here are my first impressions.

It’s a language that is both incredibly beautiful and awfully ugly. I don’t know quite how it manages it. First, some of the ugly;

#'(lambda (x y) (+ x y))
‘(1 2))

See? Ugly. (The code is from Paul Graham’s free book, [On Lisp][].) It calculates the same as this C code;

return 1 + 2;

But, of course, all that `#'(lambda` stuff is doing more than the C; in fact, it’s doing more than C possibly can, which is where the beauty comes in.

It seems to me that Lisp has some fantastic fundamental ideas and some really awful choices in naming and syntax. For example, here’s the way you set a variable in the lisp console;

CL-USER> (setq x 4)

`setq`! not `:=`, not `assign`, not `set`; `setq`. That’s why I mean by awful naming.

Another; here’s how to get the first item in a list;

(car list)

again, `car`? Admittedly, you _could_ use `first`, which is much more sensible, but the Lisp world seems to have settled on `car`, which stands for the ‘contents of the address register’, referring to a memory register in a long-dead machine from the 1970s. Awful naming.

However, the punctuation is great. It’s just that all the words suck.

If I were an author, and an editor gave me that advice, I’d quit. Wouldn’t you? So tell me something. Why is it that Lisp’s punctuation seems to hold the most profound and important programming concepts?

For example; the quote character — ‘ — allows you to freeze-dry code into something you can pass around like any other variable. If I want to multiply a global variable, x, by two, I’d say this;

CL-USER> (setq x 2)
CL-USER> (* x 2)

Thats sort-of-equivalent to this c-like code;

int x = 2;
int doublex = x * 2;

But now I’m going to use the quote symbol…

CL-USER> (setq x 2)
CL-USER> (setq doublex ‘(* x 2))
(* X 2)
CL-USER> (eval doublex)

I set the global variable `x` to 2. Then I set the variable `doublex` to be a piece of code which doubles the current value of `x`. Then, I evaluate the code in `doublex`, which gives me four.

The fun part is, you can’t do this in C — I’ve assigned arbitrary code to a variable. So now I can pass functions around like variables. Isn’t that just like function pointers in C? A little. But Lisp goes way beyond.

The awful glory is that the code is just a list; it’s the three symbols `*`, `x` and `2`. And since lisp can edit lists, and programs are lists, lisp can edit programs. And it’s no trickier to edit programs than it is to change arrays.

I’ll say that again. _Lisp programs can edit programs as easily as C can edit arrays._

Watch this. I’m going to examine code to see if it’s a multiplication.

;; `x` is a multiplier if it starts with the `*` symbol
(defun is-multiplier? (x)
(eq (first x) ‘*))

;; is doublex a multiplier? returns true.
(is-multiplier? doublex)

Just by checking for the `*` symbol, I can tell you things about the code.

And if we decided that we actually wanted to calculate x+2, rather than x*2? Well, we just change a piece of the program

;; change the first item in ‘doublex’ to a plus.
(setf (first doublex) ‘+)

This is stuff you just can’t do in other languages. I’m going to keep looking into Lisp. It seems worthwhile.

[On Lisp]: http://www.paulgraham.com/onlisp.html