• chevron_right

    Forever Stable Branch

    Michał "phoe" Herda · 3 days ago - 20:24 edit

#CommonLisp #Lisp

I wrote a kinda-long post in which:

  • I try to give an overview of how I perceive the current troublesome situation regarding ASDF and SBCL and everyone and everything else,
  • I try to brainstorm and describe some ways forward and out of this impasse.

Because of its length (and because of current Movim technical issues wrt rendering Markdown and Common Lisp), it's on its own separate page. You can read it here.

  • chevron_right

    The mystery of :UNINTERN

    Michał "phoe" Herda · 6 days ago - 19:39 edit · 8 minutes

#CommonLisp #Lisp

> Let us say, for the sake of simplicity, a fence or gate erected across a road. The more modern type of reformer goes gaily up to it and says, 'I don't see the use of this; let us clear it away.' To which the more intelligent type of reformer will do well to answer: 'If you don't see the use of it, I certainly won't let you clear it away. Go away and think. Then, when you can come back and tell me that you do see the use of it, I may allow you to destroy it.' > > -- Wikipedia - Chesterton's fence

Advertisement time

UIOP:DEFINE-PACKAGE is the part of UIOP that I personally use the most - it fills (IMO) the biggest hole in the Common Lisp package system, which is CLHS Macro DEFPACKAGE saying:

> If the new definition is at variance with the current state of that package, the consequences are undefined; (...)

This means that removing an export from a DEFPACKAGE can cause your implementation to wag a finger at you, and also ignore your attempt at removing it.

CL-USER> (defpackage #:foo (:use) (:export #:bar))

CL-USER> (defpackage #:foo (:use) (:export))
;; WARNING: FOO also exports the following symbols:
;;   (FOO:BAR)
;; See also:
;;   The ANSI Standard, Macro DEFPACKAGE
;;   The SBCL Manual, Variable *ON-PACKAGE-VARIANCE*

CL-USER> (loop for sym being the external-symbols of :foo 
               collect sym)

The solution is to manually call UNEXPORT on FOO::BAR, at which point SBCL will calm down and let you evaluate the second DEFPACKAGE form in peace.

DEFINE-PACKAGE, in the same situation, will do "the right thing" (read: the thing I personally expect it to) and adjust the package's export list to be consistent with the one provided to it.

CL-USER> (uiop:define-package #:foo (:use) (:export #:bar))

CL-USER> (uiop:define-package #:foo (:use) (:export))

CL-USER> (loop for sym being the external-symbols of :foo 
               collect sym)

There's plenty of other useful options, such as :MIX, :REEXPORT and all, but there's one of them that looks... A bit off.

Mystery time

The option :UNINTERN is specified to call CL:UNINTERN on some symbols when the package is defined.

Hold up, wait a second, though. Uninterning symbols? During package definition?

When a package is defined for the first time, there are no symbols to unintern. This means that this option is only useful when a package already exists, and therefore UIOP:DEFINE-PACKAGE is used to redefine it.

This, and uninterning cannot be used to achieve "partial :use", that is, to remove symbols from packages that are :used in the current package in order to only "use a part of" this other package. That simply isn't doable in Common Lisp - :use imports all of the symbols exported by another package, except those that are explicitly :shadowed.

So, again, what's the point? Scroll down only if you'd like the mystery to be spoiled to you.

Story time

Let's assume a very simple situation:

(defpackage #:bar
  (:export #:symbol))

We have a single package which exports a single symbol. That package was created by some software which we use, and the symbol BAR:SYMBOL is useful to us in some way.

And then, while our Lisp image is still running, we'd like to upgrade this software to a new version. That is, we'd like to load a new version of that software and disregard the old one. In the new version of our software, the package structure looks like this:

(defpackage #:foo
  (:export #:symbol))

(defpackage #:bar
  (:use #:foo)
  (:export #:symbol))

It seems that the symbol named SYMBOL was moved into another package, possibly because that is where the implementation of that symbol has been moved to. Oh well, looks understandable from a software architecture point of view!

...and then trying to load the upgraded version will fail at the very beginning. Worse - it might fail, since we have just stepped into undefined behavior area, as stated in the beginning of this post.

In particular, DEFPACKAGE FOO will be evaluated without any problem, but a keen eye will notice an error which will be signaled the moment we evaluate DEFPACKAGE BAR. The currently existing package contains its own version of the symbol named SYMBOL, whereas the new requirement is to :USE the package FOO, which has its own symbol named SYMBOL - a classic package name conflict.

What is the producer of this piece of software to do now in order to ensure a smooth transition?

One way forward is to DELETE-PACKAGE before moving on with the upgrade, but that's pretty explosive - if BAR exported any other symbols, naming e.g. class definitions, then this means trouble for us. Another way forward is to manually call UNINTERN before calling DEFPACKAGE, but only if the package already exists - and that is a little bit messy.

And this is exactly the problem that is meant to be solved by UIOP:DEFINE-PACKAGE. In particular, this utility is capable of automatically changing the structure of the underlying package to resolve conflicts in favor of the newly added symbols. We can simply use it as a drop-in replacement for DEFPACKAGE, like this:

(defpackage #:foo
  (:export #:symbol))

(uiop:define-package #:bar
  (:use #:foo)
  (:export #:symbol))

That change allows this code to compile and load without errors. In particular, we can verify that BAR:SYMBOL correctly resolves to the new symbol from package FOO:

CL-USER> 'bar:symbol

So, that's one upgrading problem less, solved by using UIOP:DEFINE-PACKAGE instead of DEFPACKAGE.

...but, uh, what about DEFINE-PACKAGE :UNINTERN? That's still not the end of the story.

Edge case time

Let us assume that you are the developer of Lisp software who is working on it and you are testing the scenario in which you upgrade one version of software to another. The technique described above works well with regard to upgrading software, but let's say that your package definition looked like this:

(defpackage #:foo
  (:intern #:some #:totally-random #:stuff))

And you want to replace it with the following:

(uiop:define-package #:foo
  (:intern #:some #:totally-randomized #:stuff))

The explanation is that TOTALLY-RANDOM was a symbol that was useful (and used) in the previous version of software, but the new version uses something better, which also has a better name - TOTALLY-RANDOMIZED.

And all is fine and well, until you go into your REPL and see this:


The syntax completion is suggesting the old symbol even though it no longer bears any meaning. It means that you, as the programmer, need to hit the key to navigate downwards and select the proper symbol, which can annoy you to no avail. That's a pet peeve.

But it also means that you have the possibility of introducing bugs into the system by using the old version of a function - or, worse, breaking the build by using a symbol that is only present on systems upgraded from the old version and not ones which had the new version loaded start from scratch.

That's actually scary.

And that's the concrete edge case solved by :UNINTERN!

(uiop:define-package #:foo
  (:intern #:totally-randomized)
  (:unintern #:totally-random))

Using this fixes the syntax completion:


Evaluating this :UNINTERN option inside DEFINE-PACKAGE will either be a no-op (if the symbol doesn't exist, e.g. when defining the package from scratch) or automatically unintern the old symbol from the system (if it exists, e.g. when upgrading the package to a newer version).

In particular, the second option will happen even if the current shape of the source code no longer has any other mentions of it and even if this :UNINTERN call seems to make no sense.

In this context, :UNINTERN is something protecting the programmer from a danger that may no longer be relevant for current versions of the software, but was once something that the programmer considered important enough to remove during a software upgrade. This :UNINTERN should stay in the source code for however long it is supported to make upgrades from the versions of software which still used this symbol to the current version.

Hell of an edge case, eh? As always, it's an edge case until you hit it and need a tool for solving it - and :UNINTERN fits that description pretty damn well.

And let's not think about the scenario where your software needs to reintroduce that symbol later on, possibly for different purposes... and support all the upgrade paths along the way.

This, and I heard that it's useful when developing, especially with one-package-per-file style (which also includes ASDF's package-inferred systems); I heard that it's more convenient to jump to the top of the file, add a (:UNINTERN #:FOO) clause to the UIOP:DEFINE-PACKAGE there, reevaluate the form, remove the clause, and keep on hacking, rather than change Emacs buffers in order to jump into the REPL and evaluate a (UNINTERN '#:FOO) form there.

Personally, though, I don't share the sentiment - I can use C-↓ or C-↑ anywhere in the file to go out of whatever form my cursor is in, write a (UNINTERN '#:FOO), C-c C-c that form to get Slime to evaluate it, and then delete the form and continue hacking.


UIOP:DEFINE-PACKAGE's :UNINTERN option is useful in the rare and obscure situations when all of the following are true:

  • you are hot-patching an existing Lisp image and do not want to restart it,
  • you need to redefine a package (possibly as a part of a software upgrade),
  • you need to ensure that, after such a redefinition, a symbol with a given name is not internal in a given package.

This is useful e.g. for avoiding invalid syntax completions inside your Lisp image.


Thanks to Robert Goldman and Phoebe Goldman for helping me solve the mystery of :UNINTERN.

Thanks to Francis St-Amour for his long and painful review of this post.

Thanks to Catie from #lispcafe on Libera Chat and Gnuxie for shorter, less painful reviews of this post.

  • chevron_right

    Macroexpand-time branching

    Michał "phoe" Herda · Sunday, 9 January - 23:31 · 3 minutes

#CommonLisp #Lisp

Let's consider the following function:

(defun make-adder (x huge-p)
  (lambda (y) (+ x y (if huge-p 1000 0))))

The result of calling (MAKE-ADDER 10) closes over HUGE-P and makes a runtime check for its value.

CL-USER> (disassemble (make-adder 10 nil))
; disassembly for (LAMBDA (Y) :IN MAKE-ADDER)
; Size: 65 bytes. Origin: #x53730938                          ; (LAMBDA (Y) :IN MAKE-ADDER)
; 38:       488975F8         MOV [RBP-8], RSI
; 3C:       488BD3           MOV RDX, RBX
; 3F:       E8EC012DFF       CALL #x52A00B30                  ; GENERIC-+
; 44:       488B75F8         MOV RSI, [RBP-8]
; 48:       4881FE17011050   CMP RSI, #x50100117              ; NIL
; 4F:       BFD0070000       MOV EDI, 2000
; 54:       B800000000       MOV EAX, 0
; 59:       480F44F8         CMOVEQ RDI, RAX
; 5D:       E8CE012DFF       CALL #x52A00B30                  ; GENERIC-+
; 62:       488BE5           MOV RSP, RBP
; 65:       F8               CLC
; 66:       5D               POP RBP
; 67:       C3               RET
; 68:       CC10             INT3 16                          ; Invalid argument count trap
; 6A:       6A20             PUSH 32
; 6C:       E8FFFA2CFF       CALL #x52A00470                  ; ALLOC-TRAMP
; 71:       5B               POP RBX
; 72:       E958FFFFFF       JMP #x537308CF
; 77:       CC10             INT3 16                          ; Invalid argument count trap

It would be better for performance if the test was only made once, in MAKE-ADDER, rather than on every call of the adder closure. MAKE-ADDER could then return one of two functions depending on whether the check succeeds.

(defun make-adder (x huge-p)
  (if huge-p
      (lambda (y) (+ x y 1000))
      (lambda (y) (+ x y 0))))

A brief look at the disassembly of this fixed version shows us that we're right:

CL-USER> (disassemble (make-adder 10 nil))
; disassembly for (LAMBDA (Y) :IN MAKE-ADDER)
; Size: 21 bytes. Origin: #x53730BC7                          ; (LAMBDA (Y) :IN MAKE-ADDER)
; C7:       488BD1           MOV RDX, RCX
; CA:       E861FF2CFF       CALL #x52A00B30                  ; GENERIC-+
; CF:       31FF             XOR EDI, EDI
; D1:       E85AFF2CFF       CALL #x52A00B30                  ; GENERIC-+
; D6:       488BE5           MOV RSP, RBP
; D9:       F8               CLC
; DA:       5D               POP RBP
; DB:       C3               RET

Still, with more flags than one, this style of writing code is likely to become unwieldy. For three flags, we would need to write something like this for the runtime version:

(defun make-adder (x huge-p enormous-p humongous-p)
  (lambda (y) (+ x y
                 (if huge-p 1000 0)
                 (if enormous-p 2000 0)
                 (if humongous-p 3000 0))))

But it would look like this for the macroexpand-time version:

(defun make-adder (x huge-p enormous-p humongous-p)
  (if huge-p
      (if enormous-p
          (if humongous-p
              (lambda (y) (+ x y 1000 2000 3000))
              (lambda (y) (+ x y 1000 2000 0)))
          (if humongous-p
              (lambda (y) (+ x y 1000 0 3000))
              (lambda (y) (+ x y 1000 0 0))))
      (if enormous-p
          (if humongous-p
              (lambda (y) (+ x y 0 2000 3000))
              (lambda (y) (+ x y 0 2000 0)))
          (if humongous-p
              (lambda (y) (+ x y 0 0 3000))
              (lambda (y) (+ x y 0 0 0))))))

The total number of combinations for n boolean flags is 2^n, making it hard to write and maintain code with so many branches. This is where WITH-MACROEXPAND-TIME-BRANCHING comes into play. Using it, we can write our code in a way that looks similar to the runtime-check version:

(defun make-adder (x huge-p enormous-p humongous-p)
  (with-macroexpand-time-branching (huge-p enormous-p humongous-p)
    (lambda (y) (+ x y
                   (macroexpand-time-if huge-p 1000 0)
                   (macroexpand-time-if enormous-p 2000 0)
                   (macroexpand-time-if humongous-p 3000 0)))))

This code gives us the clarity of runtime-checked version and the performance of a macroexpand-time-checked version. A total of eight versions of the body (and therefore, eight possible LAMBDA forms) are generated. At runtime, only one of them is selected, based on the boolean values of the three flags we provided.

Three conditional operators are provided - MACROEXPAND-TIME-IF, MACROEXPAND-TIME-WHEN, and MACROEXPAND-TIME-UNLESS, mimicking the syntax of, respectively, IF, WHEN, and UNLESS.

It is possible to use the variable *MACROEXPAND-TIME-BRANCH-BYPASS* for bypassing macroexpand-time branching; this is useful e.g. when trying to read the macroexpansions or when debugging. If that variable is set to true, the behavior of the macroexpander is modified:

  • MACROEXPAND-TIME-IF expands into an IF form,
  • MACROEXPAND-TIME-WHEN expands into a WHEN form,
  • MACROEXPAND-TIME-UNLESS expands into an UNLESS form.



Grab the code from GitHub.

  • chevron_right


    Michał "phoe" Herda · Monday, 3 January - 18:55

#CommonLisp #Lisp

Someone noted that they'd like a stream that can append to an existing stream with a fill pointer, like the stream that with-output-to-string can produce, except with indefinite extent. A little bit of Lisp hackery produced something that seems to work, even if greatly untested (yet).

;;; A fill pointer output stream with indefinite extent

(defclass fill-pointer-output-stream
  ((string :accessor fill-pointer-output-stream-string :initarg :string))
  (:default-initargs :string (a:required-argument :string)))

(defmethod trivial-gray-streams:stream-line-column
    ((stream fill-pointer-output-stream)))

(defmethod trivial-gray-streams:stream-start-line-p
    ((stream fill-pointer-output-stream)))

(defmethod trivial-gray-streams:stream-write-char
    ((stream fill-pointer-output-stream) char)
  (vector-push-extend char (fill-pointer-output-stream-string stream)))


CL-USER> (let* ((string (make-array 0 :element-type 'character :fill-pointer 0))
                (stream (make-instance 'fill-pointer-output-stream :string string)))
           (write-string "asdf" stream)
           (close stream)
  • chevron_right

    Current Common Lisp IRC situation

    Michał "phoe" Herda · Wednesday, 26 May, 2021 - 15:22 edit

Because of the upheaval at Freenode, I've migrated to Libera Chat along with a bunch of other Lisp programmers. We have used that as a chance to make some small changes to the channel structure:

  • #commonlisp is the on-topic Common Lisp channel (formerly #lisp),
  • #lisp is the somewhat on-topic discussion about all Lisp dialects (formerly ##lisp),
  • the rest of the channel names should work the same.

The first two lines of the above were mentioned because #lisp on Freenode used to have a non-trivial volume of people asking questions about Scheme or Emacs Lisp due to the too-generic channel name. Naming the Common Lisp channel #commonlisp resolves this issue, at the cost of sacrificing a lucrative and attractive five-character channel name.

  • chevron_right

    When CL:TENTH is not enough

    Michał "phoe" Herda · Thursday, 20 August, 2020 - 14:21 edit

#Lisp #CommonLisp

;;;; License: don't use this at home

(defmacro defnth (n)
  (let ((var (gensym "LIST"))
        (name (intern (substitute #\- #\Space (format nil "~:@(~:R~)" n)))))
    `(defun ,name (,var) (nth (1- ,n) ,var))))

CL-USER> (defnth 1234)

CL-USER> (one-thousand-two-hundred-thirty-fourth (loop for i from 1 below 2000 collect i))
  • chevron_right

    Call for review: package-local nicknames on CLISP

    Michał "phoe" Herda · Monday, 25 May, 2020 - 20:04 edit

#CommonLisp #Lisp

Joram Schrijver, known as mood on Freenode, has implemented package-local nicknames on CLISP.

If anyone is anyhow familiar with the CLISP codebase and is capable of reviewing that merge request, please review it - it is an important last step in pushing support for package-local nicknames across all of the Common Lisp ecosystem. (They are already supported by SBCL, CCL, ECL, Clasp, ABCL, ACL, and will be supported in LispWorks 7.2)

  • chevron_right

    Parentheses and indentation

    Michał "phoe" Herda · Monday, 18 May, 2020 - 19:23 edit · 6 minutes

#CommonLisp #Lisp

Claim: You know you've got used to reading Lisp when you no longer care about the parentheses and instead read Lisp by indentation. And this is how it is supposed to be read.

(Warning: this post has a slight rant tone to it.)

Let us consider three versions of read-file-into-string, a Common Lisp utility function adapted from the Alexandria source code. The questions are: How are they different? How do they work? What do they say about the code that is executed?

;;; Version A

(defun read-file-into-string (pathname &key (buffer-size 4096) external-format)
  (flet ((read-stream-content-into-string (stream)
           (check-type buffer-size (integer 1))
           (let ((*print-pretty* nil)
                 (buffer (make-array buffer-size :element-type 'character)))
             (with-output-to-string (datum)
               (loop :for bytes-read := (read-sequence buffer stream)
                     :do (write-sequence buffer datum :start 0 :end bytes-read)
                     :while (= bytes-read buffer-size))))))
    (with-open-file (stream pathname :direction :input
                                     :external-format external-format)
      (read-stream-content-into-string stream :buffer-size buffer-size))))
;;; Version B

(defun read-file-into-string ((pathname &key (buffer-size 4096) external-format)))
  (flet (read-stream-content-into-string (stream)
          (check-type buffer-size (integer 1)
          (let ((*print-pretty* nil))
                (buffer (make-array buffer-size :element-type 'character))
            (with-output-to-string (datum)))
              (loop :for bytes-read := (read-sequence buffer stream)
                    :do (write-sequence buffer datum :start 0 :end bytes-read))
                    :while (= bytes-read buffer-size)))
    (with-open-file ((stream pathname :direction :input
                                      :external-format external-format)))
      (read-stream-content-into-string stream :buffer-size buffer-size)))))))
;;; Version C

(defun read-file-into-string (pathname &key (buffer-size 4096) external-format)
  (flet ((read-stream-content-into-string (stream)
           (check-type buffer-size (integer 1))
    (let ((*print-pretty* nil)
      (buffer (make-array buffer-size :element-type 'character)))
      (with-output-to-string (datum)
        (loop :for bytes-read := (read-sequence buffer stream)
              :do (write-sequence buffer datum :start 0 :end bytes-read)
              :while (= bytes-read buffer-size))))))
        (with-open-file (stream pathname :direction :input
                                         :external-format external-format)
          (read-stream-content-into-string stream :buffer-size buffer-size))))

You are free to check these in a Common Lisp REPL in case of doubts.

The answer is that A and B tell the same story to the programmer, even though B won't compile. Many starting and ending parentheses in version B have been removed, duplicated, or displaced, which makes that code incomprehensible to the Lisp compiler.

C, however, does compile and work just like A does, and the Lisp compiler will not see any difference between forms from A and C. This is because C is a copy of A with broken indentation. The only thing that differs is the whitespace at the begining of each line.

To a Lisp programmer, version C is much more dangerous than B: while trying to evaluate the code from version B provides immediate feedback (it won't compile, it's broken code!), version C will instead silently work in a way that is not expected.

The intent conveyed by version A is that most of the space is taken by a local function, which is why most of the middle is indented more heavily than the bottom lines that form the actual body of read-file-into-string. Version C instead assumes that the only thing done by the local function is a check-type assertion - it is the only form indented in a way that denotes the body of a local function. The rest of function body implies that we first call some function named buffer on a freshly created array. Then, we open a with-output-to-string context, and perform everything else - which are the loop iteration and the subsequent with-open-file form - inside that context.

Such indentation is actively hostile towards the programmer, as I have intentionally created it to be misleading; it is therefore unlikely to find it in Lisp code that is actively used. Still, it is a proof of concept that it is possible to mislead a Lisp programmer, either by someone who either actively tries to do it or by one who is novice enough to not know better - and therefore, indirectly, a proof that indentation pays a crucial role in understanding and parsing Lisp code by humans.

To make another proof, we can take this example in another direction, a very extreme one this time. We will take the code from version A and remove all the parentheses from it (except where they are required to parse the context), leaving only indenation in place.

;;; Version D

defun read-file-into-string pathname &key (buffer-size 4096) external-format
  flet read-stream-content-into-string stream
         check-type buffer-size integer 1
         let *print-pretty* nil
             buffer make-array buffer-size :element-type 'character
           with-output-to-string datum
             loop :for bytes-read := read-sequence buffer stream
                  :do write-sequence buffer datum :start 0 :end bytes-read
                  :while = bytes-read buffer-size
    with-open-file stream pathname :direction :input
                                   :external-format external-format
      read-stream-content-into-string stream :buffer-size buffer-size

Suddenly, we get something strangely pythonesque. Code scopes are no longer defined by parentheses and instead they are defined purely by whitespace. Lisp programmers might also be put off by the sudden lack of parentheses.

And yet, this example - at least to me - reads very similar to the Lisp code from variant A. Again, this is because the indentation for both pieces of code is identical: it is clear where a given block or subblock goes deeper, where it continues at the same depth, and where it ends, and this is the information that a Lisp programmer uses when parsing code meant for human consumption.

There's a valid point that needs to be taken into account here - that one needs to be somewhat proficient Lisp semantics in order to know the argument counts for each form that is executed. In the above example, one needs to know that make-array takes one mandatory argument and may then take a number of keyword arguments, that write-sequence takes two mandatory arguments and keyword args, that check-type takes a variable and a type, and so on. Such knowledge comes only from using the language in depth, but, thankfully, it is knowledge that relates to the right side of each line of such program, rather than to the left part. And the left is where the indentation is.

When writing Lisp, two tasks are not meant to be done by humans: managing parentheses and indenting code. The programmer's focus should be on what happens between the parentheses and whose meaning and order is provided by indentation.

  • When I write Lisp, I do not pay any attention about the indentation; emacs automatically indents my code for me as I write it thanks to electric-indent and aggressive-indent.
  • When I write Lisp, I do not need to pay any attention to closing or counting parentheses: emacs automatically inserts them in pairs and prevents me from deleting a lone paren thanks to smartparens, and I have a visual color cue that informs me about the depth of a given paren thanks to rainbow-delimiters.
  • When I write Lisp, I do not need to pay much attention to where exactly I insert a new subform: if I misplace a form within my Lisp expression, emacs will automatically indent it for me, and I will notice that it is not indented at the level where I expected it to be. I can then fix its position, again, thanks to smartparens.

This is also why I consider writing Lisp code to be almost impossible without support from an editor that performs these tasks for the programmer. Even more, I consider sharing Lisp code to be impossible if that code is not formatted correctly, because then this code will not be indented correctly and will therefore either require other Lisp programmers who read it to put extra cognitive load to try and understand it, or, worse - it will give them outright wrong and misleading information about what a given piece of Lisp code does.

Oh, and while we're at it, Lisp also solves the tabs-versus-spaces debate. It is impossible to indent Lisp with tabs only, unless either tabs are exactly one-space wide or one chooses the worst possible option of both at the same time in the same lines.

So, spaces it is.