(This is a repost of an old blog post of mine from Teknik.)

#gif #common-lisp #skippy

Animated GIF files are most often optimized - if frame 2 only changes a small region of frame 1, then the surplus data is not duplicated, and instead only a rectangle containing the changed pixels is saved in the file, along with the offset information.

This is good for compression, but bad for programmers. For the purpose of image processing, I needed a way to "flatten" the GIF frames, as in, get all the full-sized frames. And for this, it turned out, I needed a full GIF renderer, because after the first frame, the output of frame X always depends on the output of frame X-1.

I utilized the SKIPPY library for GIF processing, but found out that it did not have a GIF renderer. So it turned out that I need to write one, which I did. And now Common Lisp has a GIF renderer written in portable Common Lisp.

It still needs a bit of work to support the little-used "Restore To Previous" GIF disposal mode, but works well enough otherwise.

See the code here. I might package it separately if there is demand for it. UPDATE: It is packaged and was submitted to Quicklisp.

(This is a repost of an old blog post of mine from Teknik.)

#alexandria #common-lisp

I had a hash table in a running Common Lisp image that I wanted to print readably while preserving object identity (mostly to avoid data duplication).

Alexandria and Bike from #lisp came to the rescue.

;;; (ql:quickload :alexandria)

(defun print-hash-table-readably (hash-table
                                  &optional (stream *standard-output*))
  "Prints a hash table readably using ALEXANDRIA:ALIST-HASH-TABLE."
  (let ((test (hash-table-test hash-table))
        (*print-circle* t)
        (*print-readably* t))
    (format stream "#.(ALEXANDRIA:ALIST-HASH-TABLE '(~%")
    (maphash (lambda (k v) (format stream "   (~S . ~S)~%" k v)) hash-table)
    (format stream "   ) :TEST '~A)" test)
    hash-table))

(This is a repost of an old blog post of mine from Teknik.)

#qtools #qt #common-lisp

I looked for a quick way of turning vectors of uint8s into Qtools's QImages for purpose of displaying them.

This is a quick macro that I hacked up to do the job.

;;; (ql:quickload :qtools)

(defmacro with-qimage-from-vector ((image-var vector width height) &body body)
  "Establishes a lexical environment, in which IMAGE-VAR is bound to a QImage
created from the ARGB contents of VECTOR, with width WIDTH and height HEIGHT."
  (with-gensyms (length v)
    `(let* ((,length (array-dimension ,vector 0))
            (,v (make-static-vector ,length :initial-contents vector)))
       (unwind-protect
            (with-finalizing
                ((,image-var (q+:make-qimage
                              (static-vector-pointer ,v)
                              ,width ,height
                              (q+:qimage.format_argb32))))
              ,@body)
         (free-static-vector ,v)))))

(This is a repost of an old blog post of mine from Teknik.)

#ultraspec #clus #common-lisp #documentation

Foreword/Confession

tl;dr I have started the Common Lisp UltraSpec project some time ago. I am responsible for it being stuck in place.

When I started the CLUS, as a one-man project back in 2016, I have never expected that it would generate so much, how do you say it in English. Echo? Answers? Agreement? Need, hidden by years of being stuck to the CLHS? I don't exactly know, but even at the European Lisp Symposium 2017 I presented CLUS at, I got multiple offers for help and support, even financial. It was obvious that the Lisp world in 2017 needs such a project

Then real life came. I don't really want to excuse myself in this blog post since I could have organized my life better and worked on CLUS easier, but I needed to straighten out multiple things that required attention. I also needed to rest, relax and reorganize larger chunks of my life - CLUS as a project was pushed aside, in a very messy, unfinished and, lo and behold, undocumented state.

Thanks to Michael Fiano, who prodded me back into life, I have decided that it's been enough time of not accepting help from all the people who offered me their help with CLUS. It's time to make this project fully alive again.

I hereby offer this short writeup as something that describes the current state of CLUS and will, along with my further work, serve as a basis of pulling the UltraSpec out of the stagnancy that I allowed it to slip into.

TODO

Diffs

tl;dr What are the differences between CLUS text and dpANS3 text?

One of the weak points of my process of converting TeX into DokuWiki markup was - I fixed various things and tidbits along the way. This means that CLUS right now is perhaps better documentation but it is no longer consistent with the specification, as there are differences at unknown points in text compared to the original text. It had become a task to find and list these differences.

Thanks to the monumental work by Kamil Ziemian and Ricardo M. H. da Silva, the diff files have been generated and are now being reviewed, with the differences listed. Thanks to this effort, we might soon have this subtask done.

Format

tl;dr What format should CLUS use for its data?

One of the questions after my ELS talk was, is there a formal grammar for the DokuWiki format. Obviously, there's not, and the format itself is good enough for displaying things on DokuWiki, but quite bad for everything else. It is a combination of the original DokuWiki format and some ugly hacks that I have made to display it the way I wanted it to.

This format is not perfect and has its obvious flaws. I have been thinking of various formats: TeX, S-expressions, Markdown, DokuWiki, but all of them don't seem good enough for the job. Perhaps Markdown would work, with some extensions, but then we'd need means of parsing its extended form and displaying it on various places. The format should be advanced enough to handle displaying of monstrocities like Macro DEFSTRUCT - the single most complicated page of the specification.

This question is open and I will be glad to accept any sensible ideas.

Displaying engine

tl;dr How should CLUS be displayed to the end users?

DokuWiki is a wiki that I use for displaying CLUS content. It is used exclusively as a displaying engine that also has a search capability. It does the job, but is not perfect, and most notably, requires us to use DokuWiki format for the text files.

Are there any better ones around?

What next?

tl;dr What should be done after finishing the CL part of CLUS

As someone on Lisp Discord noted, the sidebar of CLUS contains a lot of topics that are not related to the CL standard - libraries, extensions, substandards, implementations, books. I originally imagined CLUS to be an effort to collect information from the whole Common Lisp universe. Obviously, we first need to finish CL in order to have a foundation, but my dream was CLUS unifying various pieces of documentation from all over the Lisp universe.

Decisions and conclusions

Live chat

We obviously need means of organizing work by means of a live chat where we can discuss matters

Courtesy of lonjil, we have linked the #clus-general channel of Lisp Discord server and #clus @ Freenode, so it is possible to discuss work both on IRC and Discord. This is where I have decided to settle for CLUS-related discussion.

You're invited.

Splitting of work

There are several areas in which CLUS needs help - outlined above in the TODO section. I do not want to organize how work should be split among willing people in this post, as it's much better discussed live, on the IRC/Discord channel linked above - and that's what I'd like to focus on during the next few days.

Follow-up

I will write more on the topic in 48 hours from now. (lonjil, prepare your killswitch.)

(This is a repost of an old blog post of mine from Teknik. This blog post was split into multiple parts to accomodate for Movim's size limitations; use hypelinks in the table of contents to navigate.)

Why is on-boarding new developers so difficult? For instance, why isn’t Roswell linked to from every About Common Lisp page?

Baggers

Warning: Very much my own opinion

• CL is enjoyable to learn with a good environment
• Emacs/Vim are the only places with good environments
• Learning Emacs/Vim and CL at the same is super hard

When atom-slime is stable I'll defintely try using that to teach people but currently (last checked 1 month ago) it's way too unstable. Please don't recommend it yet.

Roswell isnt recommended by most as most people don't use it. I'm certainly in that camp too, once I had a way of getting set up that was simple I stopped looking for other ways to do it (I made a youtube video on this process which has been fairly popular).

Also teaching is hard and the longer it has been since you struggled, the harder is can be to recall what made things difficult. The 'churn' of people in the community feels low and so it seems that many of the people around have been in for a very long time.

dzecniv

I'm not sure the community is aware of this difficulty. I hope that the CL Cookbook help in fixing that.

For me the difficulty was not about Emacs that I was familiar with, but the extreme lack of good, up to date and enjoyable resources. Do you imagine that many projects don't even have documentation (uiop points to its sources, series...), don't even have syntax highlighting for lisp snippets (common-lisp.net), is old and clunky (CLHS), that lots of stuff doesn't point to modern tools (all the old books) or is outdated? :( Hopefully we'll keep on counting on the cl-cookbook and the awesome-list, which have quite improved lately.

KDallas_Multipass

I would say that there are more books that provide value to the CL community in spite of being old than there are books that are way outdated, due I think largely to the lack of change in the standard. I too thought that the CLHS seemed clunky, and there are parts of it that I don't understand, but for the most part when I need it I put my cursor over a symbol and get the clhs page on the symbol and I'm good to go. It took me a while to find the section where loop's semantics are explained rather than just the grammar, and I go back to that a lot. I imagine there are other such references that I haven't found yet. Regarding up-to-date resources, I found PCL to be a great start, it shows you how to get real stuff done. The only thing that might be out of date is the practical about parsing mp3's and streaming them, and thats only because I have a hard time believing that people have libraries of mp3's sitting around anymore, or maybe thats just me. I also recommend Edi Weitz' cookbook for an example of more modern excursions, esp w.r.t. the section on concurrency, where he walks through using a library that lets you specify a graph of computation steps, allowing concurrent operations to proceed when called for, and synchronizing when necessary.

Aidenn0

Roswell didn't exist when many of the "About Common Lisp" pages were written. There are probably still some pages out there that mention "asdf-install" which is a failed precursor to Quicklisp.

Also, many of the advantages of Lisp development are in the IDE and tooling. This means Emacs if you want something stable. People say "I just want to program in lisp, not learn a whole new editor." This is a framing problem I think, because nobody complained about firing up TurboPascal for Pascal programming.

cannabis_detox

For people who feel they are great programmers and are just switching languages: PCL was designed for this purpose.

I can't speak for those guys. I was shit when I started lisp. I had used a lot of languages. I had made a lot of money off of software that I had written. After all, good code has nothing to do with good software. But I knew I was shit, programming was just way too hard. So I'm gonna talk about the people who were like me when I started out, and why they would hate and more importantly struggle to use lisp.

As I've alluded, I've been on my Lisp journey for a couple of years and I've gone from knowing nothing about lisp to the point where I can actually argue why I want to use lisp instead of languages like c# or python. To me this issue is pretty simple. Lisp's lack of popularity makes it much more difficult to glue things together in order to create software. You can do it, but it pales in comparison to how easy it is to do in Python, which is the best at this in my experience. If I want to write a program in python, I can google how to do it and you bet your ass there will be a library that already does everything I want to do (and more). In addition, I'll see at least 2 stack overflow questions on "how to do x". I'm not saying you can't glue things together in lisp, you absolutely can, and I think lisp actually simplifes so many things, but there is undeniably so much more in your way if you're a newb just trying to make something.

Let's get concrete! Lisp's standard library (despite being massive) does not provide you with enough to make a basic application just by gluing some high level functions together. Heck, even writing a file is a fucking mess. (defun write-file (file string)) pls. Imagine you know the filename and what you want to put in the file. That should be enough. Nope. You've gotta copy paste this mess that has something to do with something called a stream (whatever the fuck that is). Manipulating strings? Forget it! (regular expressions?! Now we have 2 problems! Lisp doesn't support regex out of the box? Make that 3 problems!) I haven't even mentioned the parenthesis. Programming lisp without rainbow parentheses is a surefire way to give up on it. If you are not a proficient programmer (ie the way I was before I learned how to apply recursion. ty little lisper), rolling your own functions to do string manipulation is a huge vacuum of your time and energy for something that you feel should already be there. Of course, once you understand how to program, it's easy to roll your own, and honestly when it comes to text, I've been thinking about it all my life, I don't need to reinvent how I think about it. I just need the computer to understand me. So if anything, I'd rather have my own functions that match my thought process instead of trying to reshape my mind to conform to regular expressions or the standard way of manipulating strings that everyone uses so you have to too. Lisp is so much about getting the computer to come to you instead of the other way around. But this is pointless when you're just trying to make something. When you're new and you're just trying to make something, it just feels like, "Why isn't this here? Doesn't everyone have to use strings? Why hasn't someone added this shit yet?" It's so frustrating that the only way someone is really going to put up with all that is if they really believe there's a payoff in the long term. Luckily, there is. Lisp rewards deep study. For example let's look at code substitution. Code substitution is great. It allows you to choose your inputs which is a very poweful thing. However, code substitution where your code is in a stupid simple data structure and you have a huge library dedicated to using that data structure, well that's not great, that's fucking beautiful. But you don't learn that in the same day. You learn about code substitution. You learn about manipulating lists. You realize that your code is a list (multiple times). And piece by piece it all comes together and you can start to see the tapestry those smug lispers wouldn't shut up about. It's fucking beautiful but it's not something you get any value out of when you're first starting out and just trying to make something. Lisp also requires blind faith and trust. You will benefit immensely from suspending belief, doing things the lisp way, and asking why do lispers do things this way. Why do we use let? There must be a good reason. Use it. Ask why. Trust that there's a smart reason for doing so. There is. And if you can't find a good reason, great! You're using lisp, do it your way instead. You're not forced to do things the lisp way. And I didn't even talk about development environments. There's just too much there, but basically it's another one of those long term rewards. I wrote my own toy IDE with LTK in ~30 lines of code that I didn't even realize could update itself in real time. So it's not even about learning EMACS SLIME. The language itself makes it easy to make really great development environments. And when you think about it, shouldn't every language? But when you're new who cares. just give me the IDE. Give me the libraries I need. Let me make my application. And suddenly all of these "features" I bragged about just stand in your way. None of that talk about beauty and reward matters to a gal who just wants to automatically download new episodes of one piece. The only thing she's thinking is why is this shit so pointlessly complicated and difficult. Even if you were to fix all of the problems I mentioned, you still can't compete with googling "how do i do x in python".

blue-dog-blues

My journey was a little bit different. I have always loved computing history read up on the exploits of ancient engineers as much as I can. Out of that reading I came to appreciate the following:

• The beauty of the Lisp programming language.
• The beauty of the APL programming language.
• Just how unbeleivably smart the designers of Lisp and APL were. We are not talking normal smart here. We are talking 100 years ahead of the curve smart. Orders of magnitude smarter than smart people. Then consider that McCarthy and Iverson were not the only intellects of that caliber in the computing field in the 50s 60s and 70s. Computer science was bloody full of them!
• Just how large of a canyon separates the everyday smart people in comp sci from the true geniuses like McCarthy and Iverse et al. Holy crap. Watching the smart re-invent over and over again some random subset of ideas already inventet so long ago is truly a thing to behold. It's eye watering.
• Just how little we have progressed. Computer science has not really progressed much in the last 30 years. Most of the fruits of computer science that we use today were invented in the 50s 60s and 70s. All we do now is more of it and faster. Instead, to make sure the hardware folks get the well deserved spotlight, all of the innovation seems to have gone to hardware (not just better, faster, smaller and more of the same BUT new as well!).

So, for every complaint about Lisp my mind just scrolls back to any of intellectual giants of the past in the field of computer science and thank my lucky stars I was born after them to be able to enjoy the fruits of the genius. Because, what would programming look like without McCarthy? Would someone else invent Lisp or APL without them?

I think the world would be exactly as it is today, only a little bit poorer becasue we would probably only have a subset of the ideas discovered by these people. Everyone would be programming in some language that is essentially a cobbled together hot pile of garbage ALMOST ALWAYS containing only a subset of the ideas expressed in Lisp. A language invented 50 years ago for crying out loud.

The resulting conclusion though is that, there are truly two different demographics exploring Lisp:

• The pragmatist - need to deliver a real product yesterday, don't have time for bullshit, gotta pay some bills. Real world economics rules the day, real programmers deliver product.
• The admirer - real world economics either don't matter or have bypassed us and we spend our time in magical fields sown for us FOR FREE by giants that are either long gone or slowly passing away.

I preffer the latter than the former, perhaps one day we will manage the economics side of things to make a pragmattic Lisp experience have a lower barrier to entry.

Lastly, I never look down on the pragmatist. They have delivered 99.9% of the worlds code, not us. I am just glad I don't have to suffer their toil. Lisp makes it fun.

phoe

The documentation and materials are scattered and not that easy to find. I'm trying to fix that with the UltraSpec.

Why hasn’t the Common Lisp Foundation updated the common-lisp.net home page since 2015?

phoe

I have no idea. The Foundation has been alive in general, but it seems thay have other priorities than that. Why don't you ask them? #common-lisp.net @ Freenode. EDIT: The website has been updated in 2018.