• chevron_right

      Call for review: package-local nicknames on CLISP

      Michał "phoe" Herda · Monday, 25 May, 2020 - 20:04 edit

    #CommonLisp #Lisp

    Joram Schrijver, known as mood on Freenode, has implemented package-local nicknames on CLISP.

    If anyone is anyhow familiar with the CLISP codebase and is capable of reviewing that merge request, please review it - it is an important last step in pushing support for package-local nicknames across all of the Common Lisp ecosystem. (They are already supported by SBCL, CCL, ECL, Clasp, ABCL, ACL, and will be supported in LispWorks 7.2)

    • chevron_right

      Parentheses and indentation

      Michał "phoe" Herda · Monday, 18 May, 2020 - 19:23 edit · 6 minutes

    #CommonLisp #Lisp

    Claim: You know you've got used to reading Lisp when you no longer care about the parentheses and instead read Lisp by indentation. And this is how it is supposed to be read.

    (Warning: this post has a slight rant tone to it.)

    Let us consider three versions of read-file-into-string, a Common Lisp utility function adapted from the Alexandria source code. The questions are: How are they different? How do they work? What do they say about the code that is executed?

    ;;; Version A
    (defun read-file-into-string (pathname &key (buffer-size 4096) external-format)
      (flet ((read-stream-content-into-string (stream)
               (check-type buffer-size (integer 1))
               (let ((*print-pretty* nil)
                     (buffer (make-array buffer-size :element-type 'character)))
                 (with-output-to-string (datum)
                   (loop :for bytes-read := (read-sequence buffer stream)
                         :do (write-sequence buffer datum :start 0 :end bytes-read)
                         :while (= bytes-read buffer-size))))))
        (with-open-file (stream pathname :direction :input
                                         :external-format external-format)
          (read-stream-content-into-string stream :buffer-size buffer-size))))
    ;;; Version B
    (defun read-file-into-string ((pathname &key (buffer-size 4096) external-format)))
      (flet (read-stream-content-into-string (stream)
              (check-type buffer-size (integer 1)
              (let ((*print-pretty* nil))
                    (buffer (make-array buffer-size :element-type 'character))
                (with-output-to-string (datum)))
                  (loop :for bytes-read := (read-sequence buffer stream)
                        :do (write-sequence buffer datum :start 0 :end bytes-read))
                        :while (= bytes-read buffer-size)))
        (with-open-file ((stream pathname :direction :input
                                          :external-format external-format)))
          (read-stream-content-into-string stream :buffer-size buffer-size)))))))
    ;;; Version C
    (defun read-file-into-string (pathname &key (buffer-size 4096) external-format)
      (flet ((read-stream-content-into-string (stream)
               (check-type buffer-size (integer 1))
        (let ((*print-pretty* nil)
          (buffer (make-array buffer-size :element-type 'character)))
          (with-output-to-string (datum)
            (loop :for bytes-read := (read-sequence buffer stream)
                  :do (write-sequence buffer datum :start 0 :end bytes-read)
                  :while (= bytes-read buffer-size))))))
            (with-open-file (stream pathname :direction :input
                                             :external-format external-format)
              (read-stream-content-into-string stream :buffer-size buffer-size))))

    You are free to check these in a Common Lisp REPL in case of doubts.

    The answer is that A and B tell the same story to the programmer, even though B won't compile. Many starting and ending parentheses in version B have been removed, duplicated, or displaced, which makes that code incomprehensible to the Lisp compiler.

    C, however, does compile and work just like A does, and the Lisp compiler will not see any difference between forms from A and C. This is because C is a copy of A with broken indentation. The only thing that differs is the whitespace at the begining of each line.

    To a Lisp programmer, version C is much more dangerous than B: while trying to evaluate the code from version B provides immediate feedback (it won't compile, it's broken code!), version C will instead silently work in a way that is not expected.

    The intent conveyed by version A is that most of the space is taken by a local function, which is why most of the middle is indented more heavily than the bottom lines that form the actual body of read-file-into-string. Version C instead assumes that the only thing done by the local function is a check-type assertion - it is the only form indented in a way that denotes the body of a local function. The rest of function body implies that we first call some function named buffer on a freshly created array. Then, we open a with-output-to-string context, and perform everything else - which are the loop iteration and the subsequent with-open-file form - inside that context.

    Such indentation is actively hostile towards the programmer, as I have intentionally created it to be misleading; it is therefore unlikely to find it in Lisp code that is actively used. Still, it is a proof of concept that it is possible to mislead a Lisp programmer, either by someone who either actively tries to do it or by one who is novice enough to not know better - and therefore, indirectly, a proof that indentation pays a crucial role in understanding and parsing Lisp code by humans.

    To make another proof, we can take this example in another direction, a very extreme one this time. We will take the code from version A and remove all the parentheses from it (except where they are required to parse the context), leaving only indenation in place.

    ;;; Version D
    defun read-file-into-string pathname &key (buffer-size 4096) external-format
      flet read-stream-content-into-string stream
             check-type buffer-size integer 1
             let *print-pretty* nil
                 buffer make-array buffer-size :element-type 'character
               with-output-to-string datum
                 loop :for bytes-read := read-sequence buffer stream
                      :do write-sequence buffer datum :start 0 :end bytes-read
                      :while = bytes-read buffer-size
        with-open-file stream pathname :direction :input
                                       :external-format external-format
          read-stream-content-into-string stream :buffer-size buffer-size

    Suddenly, we get something strangely pythonesque. Code scopes are no longer defined by parentheses and instead they are defined purely by whitespace. Lisp programmers might also be put off by the sudden lack of parentheses.

    And yet, this example - at least to me - reads very similar to the Lisp code from variant A. Again, this is because the indentation for both pieces of code is identical: it is clear where a given block or subblock goes deeper, where it continues at the same depth, and where it ends, and this is the information that a Lisp programmer uses when parsing code meant for human consumption.

    There's a valid point that needs to be taken into account here - that one needs to be somewhat proficient Lisp semantics in order to know the argument counts for each form that is executed. In the above example, one needs to know that make-array takes one mandatory argument and may then take a number of keyword arguments, that write-sequence takes two mandatory arguments and keyword args, that check-type takes a variable and a type, and so on. Such knowledge comes only from using the language in depth, but, thankfully, it is knowledge that relates to the right side of each line of such program, rather than to the left part. And the left is where the indentation is.

    When writing Lisp, two tasks are not meant to be done by humans: managing parentheses and indenting code. The programmer's focus should be on what happens between the parentheses and whose meaning and order is provided by indentation.

    • When I write Lisp, I do not pay any attention about the indentation; emacs automatically indents my code for me as I write it thanks to electric-indent and aggressive-indent.
    • When I write Lisp, I do not need to pay any attention to closing or counting parentheses: emacs automatically inserts them in pairs and prevents me from deleting a lone paren thanks to smartparens, and I have a visual color cue that informs me about the depth of a given paren thanks to rainbow-delimiters.
    • When I write Lisp, I do not need to pay much attention to where exactly I insert a new subform: if I misplace a form within my Lisp expression, emacs will automatically indent it for me, and I will notice that it is not indented at the level where I expected it to be. I can then fix its position, again, thanks to smartparens.

    This is also why I consider writing Lisp code to be almost impossible without support from an editor that performs these tasks for the programmer. Even more, I consider sharing Lisp code to be impossible if that code is not formatted correctly, because then this code will not be indented correctly and will therefore either require other Lisp programmers who read it to put extra cognitive load to try and understand it, or, worse - it will give them outright wrong and misleading information about what a given piece of Lisp code does.

    Oh, and while we're at it, Lisp also solves the tabs-versus-spaces debate. It is impossible to indent Lisp with tabs only, unless either tabs are exactly one-space wide or one chooses the worst possible option of both at the same time in the same lines.

    So, spaces it is.

    • chevron_right


      Michał "phoe" Herda · Wednesday, 15 April, 2020 - 20:19

    #Lisp #CommonLisp

    I have managed to create a portability library that successfully overrides the standard Lisp debugger and allows the programmer to supply their own code in its stead.

    Purpose: a basic building block that allows one to implement a portable, custom debugger for Common Lisp systems in place of the original system-provided one.

    Roughly tested on SBCL, CCL, ECL, Clasp, CLISP, ABCL, LW, and ACL.

    TRIVIAL-CUSTOM-DEBUGGER> (with-debugger ((lambda (condition hook)
                                               (declare (ignore hook))
                                               (format t ";; Debugging a ~S!~%" 
                                                       (type-of condition))
                                               (throw :handled t)))
                               (list (catch :handled (error 'error))
                                     (catch :handled (break))
                                     (let ((*break-on-signals* 'error))
                                       (catch :handled (signal 'error)))
                                     (catch :handled 
                                       (invoke-debugger (make-condition 'error)))))
    ;; Debugging a ERROR!
    ;; Debugging a SIMPLE-CONDITION!
    ;; Debugging a SIMPLE-CONDITION!
    ;; Debugging a ERROR!
    (T T T T)

    See it on GitHub.

    • chevron_right

      Common Lisp: downcase Lisp files

      Michał "phoe" Herda · Wednesday, 15 April, 2020 - 20:15 edit

    #Lisp #CommonLisp

    If you ever encounter a Lisp file that is written with upcased symbols, you might want to be able to easily convert it to downcase. That's my implementation of that.

    (defun case-lisp-file (pathname case-function)
      (with-input-from-file (input pathname)
        (with-output-to-string (output)
          (loop for char = (read-char input nil nil)
                while char
                do (case char
                      (princ char output)
                      (let ((next-char (read-char input)))
                        (case next-char
                           (princ next-char output)
                           (princ (read-char input) output))
                          (t (unread-char next-char input)))))
                      (unread-char char input)
                      (princ (read-line input) output)
                      (terpri output))
                     ((#\" #\|)
                      (unread-char char input)
                      (prin1 (read input) output))
                     (t (write-char (funcall case-function char) output)))))))
    (defun upcase-lisp-file (pathname)
      "Upcases a Common Lisp source file."
      (case-lisp-file pathname #'char-upcase))
    (defun downcase-lisp-file (pathname)
      "Downcases a Common Lisp source file."
      (case-lisp-file pathname #'char-downcase))

    Thanks to Rainer Joswig and to travv0 for noticing and implementing fixes for that.

    • chevron_right

      Brackets in Lisp

      Michał "phoe" Herda · Sunday, 28 April, 2019 - 08:41 edit

    #CommonLisp #Lisp

    (set-macro-character #\[
      (lambda (stream char)
        (declare (ignore char))
        (read-delimited-list #\] stream t)))
    (set-macro-character #\]
      (lambda (stream char)
        (declare (ignore stream char))
        (error "unmatched bracket on input")))
    CL-USER> [defun frobnicate [foo bar]
               [list [+ foo bar] [- foo bar]
                     [* foo bar] [/ foo bar]]]
    CL-USER> [frobnicate 42 20]
    (62 22 840 21/10)
    • chevron_right

      The case against IF-LET*

      Michał "phoe" Herda · Monday, 15 April, 2019 - 11:49 edit · 1 minute

    #CommonLisp #Lisp #Alexandria

    Someone asked a question on #lisp on Freenode: since WHEN-LET has an analogous WHEN-LET*, why IF-LET has no analogous IF-LET*?

    Here is my attempt at answering it.

    Suppose we have a following code block:

    (if-let ((x 42) 
             (y nil)
             (z :foo))
      (list :true x y z)
      (list :false x y z))

    All of X, Y, and Z are bound to the respective values: 42, NIL, and :FOO. At least one of the values is false. Therefore, it should be obvious that the result of evaluating this code would be the list (:FALSE 42 NIL :FOO).

    Now, let us slightly modify this code:

    (if-let* ((x 42) 
              (y nil)
              (z :foo))
      (list :true x y z)
      (list :false x y z))

    We no longer use IF-LET but instead we use the hypothetical sequential version, IF-LET*. We evaluate X and Y. By definition, we are supposed to not evaluate the following variables at this moment, since Y evaluated to false, and we instead execute the false branch.

    The big question is: what is the value of Z in the false branch?

    It is certainly not :FOO because we have not managed to evaluate that binding yet. In fact, that lexical variable should be unbound since there is no logical value for it, but there is no such thing as an unbound lexical variable in Lisp. In particular, this might mean that we encounter any number of logically unbound variables in the false branch. We could work around it by setting the not-yet-bound lexical variables to some predefined value (which would greatly complicate the code and introduce a lot of mess) or we would make the false branch unable to use any of the lexical bindings, which makes the sequential IF-LET* greatly inconsistent with the parallel IF-LET.

    In other words: having IF-LET* would be inconsistent. None of the variables are available in the lexical scope of the false branch, and that should be part of IF-LET*'s contract.

    • chevron_right


      Michał "phoe" Herda · Tuesday, 25 December, 2018 - 16:42 edit · 5 minutes

    #CommonLisp #Lisp #MOP

    (This blogpost is also known as "WTF is VALIDATE-SUPERCLASS and why do I have to define methods on it to get a metaclass to work?".)

    Let us assume that we have a set of superclasses. In this case, let us define two of them, but the number does not matter: can be zero, can be one, can be five.

    Additionally, let us define a new subclass of STANDARD-CLASS, that we will call our metaclass.

    (defclass superclass-1 () ())
    (defclass superclass-2 () ())
    (defclass metaclass (standard-class) ())

    Let us attempt to create a subclass of these two superclasses, while also using our newly created metaclass.

    (defclass subclass (superclass-1 superclass-2) ()
      (:metaclass metaclass))

    This signals an error.

    The class #<STANDARD-CLASS SUPERCLASS-1> was specified as a superclass of 
    the class #<METACLASS SUBCLASS {100377A0D3}>, but the metaclasses

    There are four distinct classes mentioned in the error message. First of all, there is one of our superclasses, SUPERCLASS-1, then, there is the class object named SUBCLASS that we are trying to define. Then, there are two more class objects, STANDARD-CLASS and METACLASS.

    The magical combination of parentheses that causes the above DEFCLASS form to evaluate successfully is this mysteriously-looking method:

    (defmethod validate-superclass
        ((class metaclass) (superclass standard-class))

    Let us analyze why this particular method fixes our problem and allows us to define classes with the metaclass of our choosing.

    The MOP description of VALIDATE-SUPERCLASS says that this function is called with two arguments: class, and superclass, to determine whether the class superclass is suitable for use as a superclass of class.

    In other words, if Common Lisp Object System is supposed to permit a situation when superclass becomes a superclass of class, this function needs to return true. If such a situation is forbidden, this function must return false.

    The MOP says that the default system-defined method on VALIDATE-SUPERCLASS returns true only in three cases:

    1. if the superclass argument is the class named T,
    2. if the class of the class argument is the same as the class of the superclass argument, or
    3. if the class of one of the arguments is STANDARD-CLASS and the class of the other is FUNCALLABLE-STANDARD-CLASS.

    The first case is trivial; every class is a subclass of the class T, therefore it would be pointless to return false on

    The third case is there to assure that standard objects and funcallable standard objects can interoperate. A subclass of a funcallable class does not have to be funcallable, and the other way around, too: a subclass of a non-funcallable class may be funcallable.

    The second case is what allows the basics of CLOS class definitions to work. Whenever we define a new class with DEFCLASS without any superclasses, we define a new instance of STANDARD-CLASS that has STANDARD-OBJECT as its superclass. STANDARD-OBJECT is an instance of STANDARD-CLASS; the newly defined class is an instance of STANDARD-CLASS; therefore, VALIDATE-SUPECLASS returns true. Whenever we define a new class with any number of superclasses that itself are also instances of STANDARD-CLASS, the same thing happens: each of the superclasses is an instance of STANDARD-CLASS, just like the class we are defining right now, which allows VALIDATE-SUPERCLASS to return true in each single case.

    The issue happens when we specify the :METACLASS option to DEFCLASS. :METACLASS states that the new class should be an instance of the provided class. If we pass the option (:METACLASS FOO-CLASS), then the newly instantiated class metaobject will be an instance of FOO-CLASS.

    But, remember that we also need to specify superclasses for that class! Unless we explicitly specify a superclass, then it is going to default to STANDARD-OBJECT, in which case, VALIDATE-SUPERCLASS will be called on a pair of arguments: the class we are attempting to create, an instance of FOO-CLASS, and the class STANDARD-OBJECT, an instance of STANDARD-CLASS, which is stated to be its superclass.

    Neither of the three cases the default method fit this combination, therefore the method returns NIL. The validation fails, and an error is signaled from the DEFCLASS macro.

    This is why we explicitly need to state that we, the programmers, want class STANDARD-OBJECT to be a valid superclass for instances of FOO-CLASS. This is the purpose of the VALIDATE-SUPERCLASS method defined on FOO-CLASS as the class we want to instantiate and STANDARD-OBJECT as the superclass we want to assign to the class.

    In other words, the way the aforementioned MOP usage works this way is because VALIDATE-SUPERCLASS serves a greater function than mere playing with metaclasses - it is the central MOP mechanism for declaring which combinations of classes and subclasses are allowed in the Lisp image. Methods on that function which may be specialized in any way, including methods that use EQL-specializers and perform arbitrary computation.

    For an example of the above, let us state that the class named FOO may be subclassed only by classes whose name contain the letter E.

    (defmethod sb-mop:validate-superclass
        ((class class) (superclass (eql (find-class 'foo))))
      (find-if (lambda (x) (member x '(#\E #\e)))
               (symbol-name (class-name class))))

    The above snippet denies (defclass bar (foo) ()), but permits (defclass fred (foo) ()).

    It is also possible to imagine a class hierarchy where a programmer decides, for any reason, that a class named BAR should not be allowed to be subclassed at all. The following snippet is enough to achieve that:

    (defmethod sb-mop:validate-superclass
        ((class class) (superclass (eql (find-class 'bar))))

    (To add to the confusion, the error message we get from SBCL in such a case is not very helpful, as it states, the metaclasses #1=#<STANDARD-CLASS COMMON-LISP:STANDARD-CLASS> and #1# are incompatible. I have raised a bugticket on that edge case.)

    Summing up, VALIDATE-SUPERCLASS is a general mechanism for describing which kinds of superclass-subclass relations we want to see in the Lisp image. Practice has shown that this function is particularly often used in context of permitting instances of non-STANDARD-CLASSes to inherit from instances of STANDARD-CLASSes - in other words, allowing metaclasses to be instantiated. The way it is done is via defining methods on this function, specialized on the metaclass we want to instantiate, and the superclass that we want to assign to the instances of that metaclass - either a particular superclass of our choosing, or STANDARD-OBJECT as the default. These methods must return true whenever we want to permit a particular combination to happen.