CL-WHO - Yet another Lisp Markup Language

Abstract
There are plenty of Lisp Markup Languages out there - every Lisp programmer seems to write at least one during his career - and CL-WHO (where WHO means "with-html-output" for want of a better acronym) is probably just as good or bad as the next one. They are all more or less similar in that they provide convenient means to convert S-expressions intermingled with code into (X)HTML, XML, or whatever but differ with respect to syntax, implementation, and API. So, if you haven't made a choice yet, check out the alternatives as well before you begin to use CL-WHO just because it was the first one you came across. (Was that repelling enough?) If you're looking for a slightly different approach you might also want to look at HTML-TEMPLATE.
I wrote this one in 2002 although at least Tim Bradshaw's htout and AllegroServe's HTML generation facilities by John Foderaro of Franz Inc. where readily available. Actually, I don't remember why I had to write my own library - maybe just because it was fun and didn't take very long. The syntax was obviously inspired by htout although it is slightly different.
I've since used CL-WHO successfully in several web applications in conjuction with mod_lisp and LispWorks or CMUCL so it doesn't seem to have any significant bugs. If you use it like I do, that is. YMMV. (OK, to be completely honest, I've made a few modifications for this public release and have probably broken a couple of things. That's why I begin to count with version 0.1.0.)
CL-WHO tries to create efficient code in that it makes constant strings as long as possible. In other words, the code generated by the CL-WHO macros will usually be a sequence of WRITE-STRING forms for constant parts of the output interspersed with arbitrary code inserted by the user of the macro. CL-WHO will make sure that there aren't two adjacent WRITE-STRING forms with constant strings - see examples below. CL-WHO's output is intended to create XHTML rather than 'plain' HTML.
CL-WHO is intended to be portable and should work with all conforming Common Lisp implementations. Let me know if you encounter any problems.
It comes with a BSD-style license so you can basically do with it whatever you want.

Example usage
Download and installation
Syntax and Semantics
The CL-WHO dictionary
Acknowledgements

Example usage

Let's assume that *HTTP-STREAM* is the stream your web application is supposed to write to. Here are some contrived code snippets together with the Lisp code generated by CL-WHO and the resulting HTML output.

(with-html-output (*http-stream*)
  (loop for (link . title) in '(("http://zappa.com/" . "Frank Zappa")
                                ("http://marcusmiller.com/" . "Marcus Miller")
                                ("http://www.milesdavis.com/" . "Miles Davis"))
        do (htm (:a :href link
                  (:b (str title)))
                :br)))

Frank Zappa
Marcus Miller
Miles Davis

;; Code generated by CL-WHO

(let ((*http-stream* *http-stream*))
  (progn
    nil
    (loop for (link . title) in '(("http://zappa.com/" . "Frank Zappa")
                                  ("http://marcusmiller.com/" . "Marcus Miller")
                                  ("http://www.milesdavis.com/" . "Miles Davis"))
          do (progn
               (write-string "<a href='" *http-stream*)
               (princ link *http-stream*)
               (write-string "'><b>" *http-stream*)
               (princ title *http-stream*)
               (write-string "</b></a><br />" *http-stream*)))))

(with-html-output (*http-stream*)
  (:table :border 0 :cellpadding 4
   (loop for i below 25 by 5
         do (htm
             (:tr :align "right"
              (loop for j from i below (+ i 5)
                    do (htm
                        (:td :bgcolor (if (oddp j)
                                        "pink"
                                        "green")
                             (fmt "~@R" (1+ j))))))))))

I	II	III	IV	V
VI	VII	VIII	IX	X
XI	XII	XIII	XIV	XV
XVI	XVII	XVIII	XIX	XX
XXI	XXII	XXIII	XXIV	XXV

;; Code generated by CL-WHO

(let ((*http-stream* *http-stream*))
  (progn
    nil
    (write-string "<table border='0' cellpadding='4'>" *http-stream*)
    (loop for i below 25 by 5
          do (progn
               (write-string "<tr align='right'>" *http-stream*)
               (loop for j from i below (+ i 5)
                     do (progn
                          (write-string "<td bgcolor='" *http-stream*)
                          (princ (if (oddp j) "pink" "green") *http-stream*)
                          (write-string "'>" *http-stream*)
                          (format *http-stream* "~@r" (1+ j))
                          (write-string "</td>" *http-stream*)))
               (write-string "</tr>" *http-stream*)))
    (write-string "</table>" *http-stream*)))

(with-html-output (*http-stream*)
  (:h4 "Look at the character entities generated by this example")
   (loop for i from 0
         for string in '("Fête" "Sørensen" "naïve" "Hühner" "Straße")
         do (htm
             (:p :style (conc "background-color:" (case (mod i 3)
                                                    ((0) "red")
                                                    ((1) "orange")
                                                    ((2) "blue")))
              (htm (esc string))))))

Look at the character entities generated by this example

Fête

Sørensen

naïve

Hühner

Straße

;; Code generated by CL-WHO

(let ((*http-stream* *http-stream*))
  (progn
    nil
    (write-string
     "<h4>Look at the character entities generated by this example</h4>"
     *http-stream*)
    (loop for i from 0 for string in '("Fête" "Sørensen" "naïve" "Hühner" "Straße")
          do (progn
               (write-string "<p style='" *http-stream*)
               (princ (conc "background-color:"
                            (case (mod i 3)
                              ((0) "red")
                              ((1) "orange")
                              ((2) "blue")))
                      *http-stream*)
               (write-string "'>" *http-stream*)
               (progn (write-string (escape-string string) *http-stream*))
               (write-string "</p>" *http-stream*)))))

Download and installation

CL-WHO together with this documentation can be downloaded from http://weitz.de/files/cl-who.tgz. The current version is 0.3.0.

If you're on Debian you should probably use the cl-who Debian package which is available thanks to Kevin Rosenberg.

CL-WHO comes with simple system definitions for MK:DEFSYSTEM and asdf so you can either adapt it to your needs or just unpack the archive and from within the CL-WHO directory start your Lisp image and evaluate the form (mk:compile-system "cl-who") (or the equivalent one for asdf) which should compile and load the whole system. Installation via asdf-install should also be possible.

If for some reason you don't want to use MK:DEFSYSTEM or asdf you can just LOAD the file load.lisp or you can also get away with something like this:

(loop for name in '("packages" "who")
      do (compile-file (make-pathname :name name
                                      :type "lisp"))
         (load name))

Note that on CL implementations which use the Python compiler (i.e. CMUCL, SBCL, SCL) you can concatenate the compiled object files to create one single object file which you can load afterwards:

cat {packages,who}.x86f > cl-who.x86f

(Replace ".x86f" with the correct suffix for your platform.)

Syntax and Semantics

CL-WHO is essentially just one macro, WITH-HTML-OUTPUT, which transforms the body of code it encloses into something else obeying the following rules (which we'll call transformation rules) for the body's forms:

A string will be printed verbatim. To be more precise, it is transformed into a form which'll print this string to the stream the user provides.
"foo" => (write-string "foo" s)
(Here and for the rest of this document the red arrow means '... will be converted to code equivalent to ...' where equivalent means that all output is sent to the "right" stream.)

Each list beginning with a keyword is transformed into an XHTML tag of the same (downcased) name by the following rules:

If the list contains nothing but the keyword, the resulting tag will be empty.
(:br) => (write-string "<br />" s)

The initial keyword can be followed by another keyword which will be interpreted as the name of an attribute. The next form which will be taken as the attribute's value. (If there's no next form it'll be as if the next form had been NIL.) The form denoting the attribute's value will be treated as follows. (Note that the behaviour with respect to attributes is incompatible with versions earlier than 0.3.0!)

If it is a string it will be printed literally.

(:td :bgcolor "red") => (write-string "<td bgcolor='red' />" s)

If it is T the attribute's value will be the attribute's name (following XHTML convention to denote attributes which don't have a value in HTML).
(:td :nowrap t) => (write-string "<td nowrap='nowrap' />" s)
If it is NIL the attribute will be left out completely.
(:td :nowrap nil) => (write-string "<td />" s)
If it is a constant form it will be inserted into the resulting string as if printed with the format string "~A" at macro expansion time.
(:table :border 3) => (write-string "<table border='3' />" s)

If it is any other form it will be left as is and later evaluated at run time and printed with PRINC unless the value is T or NIL which will be treated as above. (It is the application developer's job to provide the correct printer control variables.)

;; simplified example, see function CHECKBOX below

(:table :border (+ 1 2)) => (write-string "<table border='" s)
                              (princ (+ 1 2) s)
                              (write-string "<' />" s)

Once an attribute/value pair has been worked up another one can follow, i.e. if the form following an attribute's value is again a keyword it will again be treated as an attribute and so on.
(:table :border 0 :cellpadding 5 :cellspacing 5) => (write-string "<table border='0' cellpadding='5' cellspacing='5' />" s)

The first form following either the tag's name itself or an attribute value which is not a keyword determines the beginning of the tag's content. This and all the following forms are subject to the transformation rules we're just describing.

(:p "Paragraph") => (write-string "<p>Paragraph</p>" s)
(:p :class "foo" "Paragraph") => (write-string "<p class='foo'>Paragraph</p>" s)
(:p :class "foo" "One" " " "long" " " "sentence") => (write-string "<p class='foo'>One long sentence</p>" s)
(:p :class "foo" "Visit " (:a :href "http://www.cliki.net/" "CLiki"))
    => (write-string "<p class='foo'>Visit <a href='http://www.cliki.net/'>CLiki</a></p>" s)

Here's a slightly more elaborate example:

* (defun checkbox (stream name checked &optional value)
    (with-html-output (stream)
      (:input :type "checkbox" :name name :checked checked :value value)))

CHECKBOX
* (with-output-to-string (s) (checkbox s "foo" t))

"<input type='checkbox' name='foo' checked='checked' />"
* (with-output-to-string (s) (checkbox s "foo" nil))

"<input type='checkbox' name='foo' />"
* (with-output-to-string (s) (checkbox s "foo" nil "bar"))

"<input type='checkbox' name='foo' value='bar' />"
* (with-output-to-string (s) (checkbox s "foo" t "bar"))

"<input type='checkbox' name='foo' checked='checked' value='bar' />"

A keyword alone will be treated like a list containing only this keyword.
:hr => (write-string "<hr />" s)
A form which is neither a string nor a keyword nor a list beginning with a keyword will be left as is except for the following substitutions:
- Forms that look like (str form1 form*) will be substituted with (princ form1 s). (Note that all forms behind form1 are ignored.)
  (loop for i below 10 do (str i)) => (loop for i below 10 do (princ i s))
- Forms that look like (fmt form*) will be substituted with (format s form*).
  (loop for i below 10 do (fmt "~R" i)) => (loop for i below 10 do (format s "~R" i))
- Forms that look like (esc form1 form*) will be substituted with (write-string (escape-string form1) s).
- If a form looks like (htm form*) then each of the forms will be subject to the transformation rules we're just describing.
  (loop for i below 100 do (htm (:b "foo") :br)) => (loop for i below 100 do (progn (write-string "<b>foo</b><br />" s)))
That's all. Note in particular that CL-WHO knows nothing about HTML or XHTML, i.e. it doesn't check whether you mis-spelled tag names or use attributes which aren't allowed. CL-WHO doesn't care if you use, say, :foobar instead of :hr.

The CL-WHO dictionary

CL-WHO exports the following symbols:

[Macro]
with-html-output (var &optional stream &key prologue indent) declaration* form* => result*

This is the main macro of CL-WHO. It will transform its body by the transformation rules described in Syntax and Semantics such that the output generated is sent to the stream denoted by var and stream. var must be a symbol. If stream is NIL it is assumed that var is already bound to a stream, if stream is not NIL var will be bound to the form stream which will be evaluated at run time. prologue should be a string (or NIL for the empty string which is the default) which is guaranteed to be the first thing sent to the stream from within the body of this macro. If prologue is T the prologue string is the value of *PROLOGUE*. CL-WHO will usually try not to insert any unnecessary whitespace in order to save bandwidth. However, if indent is true line breaks will be inserted and nested tags will be intended properly. The value of indent - if it is an integer - will be taken as the initial indentation. If it is not an integer it is assumed to mean 0. The results are the values returned by the forms.
Note that the keyword arguments prologue and indent are used at macro expansion time.
* (with-html-output (*standard-output* nil :prologue t)
    (:html (:body "Not much there"))
    (values))
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"><html><body>Not much there</body></html>
* (with-html-output (*standard-output*)
    (:html (:body :bgcolor "white"
             "Not much there"))
    (values))
<html><body bgcolor='white'>Not much there</body></html>
* (with-html-output (*standard-output* nil :prologue t :indent t)
    (:html (:body :bgcolor "white"
             "Not much there"))
    (values))
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html>
  <body bgcolor='white'>
    Not much there
  </body>
</html>

[Macro]
with-html-output-to-string (var &optional string-form &key element-type prologue indent) declaration* form* => result*

This is just a thin wrapper around WITH-HTML-OUTPUT. Indeed, the wrapper is so thin that the best explanation probably is to show its definition:

(defmacro with-html-output-to-string ((var &optional string-form
                                           &key (element-type 'character)
                                                prologue
                                                indent)
                                      &body body)
  "Transform the enclosed BODY consisting of HTML as s-expressions
into Lisp code which creates the corresponding HTML as a string."
  `(with-output-to-string (,var ,string-form :elementy-type ,element-type)
    (with-html-output (,var nil :prologue ,prologue :indent ,indent)
      ,@body)))

Note that the results of this macro are determined by the behaviour of WITH-OUPUT-TO-STRING.

[Macro]
show-html-expansion (var &optional stream &key prologue indent) declaration* form* => <no values>

This macro is intended for debugging purposes. It'll print to *STANDARD-OUTPUT* the code which would have been generated by WITH-HTML-OUTPUT had it been invoked with the same arguments.

* (show-html-expansion (s)
    (:html
     (:body :bgcolor "white"
      (:table
       (:tr
        (dotimes (i 5)
          (htm (:td :align "left"
                (str i)))))))))
(LET ((S S))
  (PROGN
    (WRITE-STRING
      "<html><body bgcolor='white'><table><tr>" S)
    (DOTIMES (I 5)
      (PROGN
        (WRITE-STRING "<td align='left'>" S)
        (PRINC I S)
        (WRITE-STRING "</td>" S)))
    (WRITE-STRING "</tr></table></body></html>" S)))

[Special variable]
*prologue*

This is the prologue string which will be printed if the prologue keyword argument to WITH-HTML-OUTPUT is T. Its initial value is
"<!DOCTYPE html PUBLIC \"-//W3C//DTD XHTML 1.0 Strict//EN\" \"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd\">"

[Symbol]
esc
[Symbol]
fmt
[Symbol]
htm
[Symbol]
str

These are just symbols with no bindings associated with them. The only reason they are exported is their special meaning during the transformations described in Syntax and Semantics.

[Function]
escape-string string &key test => escaped-string

This function will accept a string string and will replace every character for which test returns true with its XML character entity. test must be a function of one argument which accepts a character and returns a generalized boolean. The default is the value of *ESCAPE-CHAR-P*. Note the ESC shortcut described in Syntax and Semantics.
* (escape-string "<Hühner> 'naïve'")
"&lt;H&#xFC;hner&gt; &apos;na&#xEF;ve&apos;"
* (with-html-output-to-string (s)
    (:b (esc "<Hühner> 'naïve'")))
"<!DOCTYPE html PUBLIC \"-//W3C//DTD XHTML 1.0 Strict//EN\" \"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd\"<b>&lt;H&#xFC;hner&gt; &apos;na&#xEF;ve&apos;</b>"

[Special variable]
*escape-char-p*

This is the default for the test keyword argument to ESCAPE-STRING. Its initial value is
#'(lambda (char)
    (or (find char "<>&'\"")
        (> (char-code char) 127)))

[Function]
escape-string-minimal string => escaped-string
[Function]
escape-string-minimal-plus-quotes string => escaped-string
[Function]
escape-string-iso-8859 string => escaped-string
[Function]
escape-string-all string => escaped-string

These are convenience function based on ESCAPE-STRING. They are defined as follows:

(defun escape-string-minimal (string)
  "Escape only #\<, #\>, and #\& in STRING."
  (escape-string string :test #'(lambda (char) (find char "<>&"))))

(defun escape-string-minimal-plus-quotes (string)
  "Like ESCAPE-STRING-MINIMAL but also escapes quotes."
  (escape-string string :test #'(lambda (char) (find char "<>&'\""))))

(defun escape-string-iso-8859 (string)
  "Escapes all characters in STRING which aren't defined in
ISO-8859. This of course assumes that STRING is an ISO-8859 string."
  (escape-string string :test #'(lambda (char)
                                  (or (find char "<>&'\"")
                                      (> (char-code char) 255)))))

(defun escape-string-all (string)
  "Escapes all characters in STRING which aren't in the 7-bit ASCII
character set."
  (escape-string string :test #'(lambda (char)
                                  (or (find char "<>&'\"")
                                      (> (char-code char) 127)))))

[Function]
conc &rest string-list => string

Utility function to concatenate all arguments (which should be strings) into one string. Meant to be used mainly with attribute values.
* (conc "This" " " "is" " " "a" " " "sentence")
"This is a sentence"
* (with-html-output-to-string (s)
    (:div :style (conc "padding:"
                       (format nil "~A" (+ 3 2)))
     "Foobar"))
"<div style='padding:5'>Foobar</div>"

Acknowledgements

Thanks to Tim Bradshaw and John Foderaro for the inspiration provided by their libraries mentioned above. Thanks to Jörg-Cyril Höhle for his suggestions with respect to attribute values.

$Header: /usr/local/cvsrep/cl-who/doc/index.html,v 1.15 2003/08/02 13:26:27 edi Exp $

BACK TO MY HOMEPAGE