It may be tempting at the start of a new project to create the first database tables manually, or write SQL scripts that you run manually, especially when you first have to spend a significant amount of time on sifting through all the migration libraries and then some more to get it working properly.

Going through this process did slow me down at the start of the project but I was determined to use a migration tool because hunting inexplicable bugs that only happen in production just to find out there is a definition mismatch between the production and development databases is not fun. Using such a tool also motivates you to write both the setup and teardown steps for each table while the current design is still fresh in your mind.

At first I considered a standalone migration tool because I expect them to be very good at that single task. However, learning the idiosyncrasies of a new tool and trying to make it fit seamlessly into my development workflow seemed like more trouble than it is worth.

I decided to stick with a Common Lisp library and found the following seven that work with PostgreSQL and/or Postmodern:

• Postmodern-passenger-pigeon
• database-migrations
• cl-migrations
• Mito
• Crane
• Orizuru-orm
• cl-mgr

I quickly discounted Crane and Mito because they are ORM (Object Relational Mapper) libraries which are way more complex than a dedicated migration library. Development on Crane have stalled some time ago and I don't feel it is mature enough for frictionless use yet. Mito declares itself as being in Alpha state; also not mature enough yet.

I only stumbled onto cl-mgr and Orizuru-orm long after making my decision so I did not investigate them seriously. Orizuru-orm is in any case an ORM which I would have discounted because it is too complex for my needs. CL-mgr looks simple, which is a good thing. It is based on cl-dbi which makes it a good candidate if you foresee switching databases but even if I discovered it sooner I would have discounted it for the same reason as CL-migrations.

CL-migrations looks very promising. It is a simple library focusing only on migrations. It uses clsql to interface with the database which bothered me because I already committed to using Postmodern and I try to avoid adding a lot of unused code to my projects. The positive side is that it interfaces to many different databases so it is a good candidate if you are not committed to using Postmodern. It is also a stable code base with no outstanding bug reports.

The two projects I focused on was Postmodern-passenger-pigeon and Database-migrations because they both use Postmodern for a database interface.

Postmodern-passenger-pigeon was in active development at the time and it seemed safer to use than Database-migrations because it can do dry runs, which is a very nice feature when you are upgrading your production database and face the possibility of losing data when things go awry. Unfortunately I could not get it working within a reasonable amount of time.

I finally settled on Database-migrations. It is a small code base, focused on one task, it is mature and it uses Postmodern so it does not pull in a whole new database interface into my project. There are however some less positive issues.

The first issue is a hindrance during development. Every time the migrations ASDF system (or the file containing it, as ASDF prefers that all systems be defined in a single file) is recompiled it adds all the defined migrations to the migrations list. Though each one will only be applied once to the DB it is still bothersome. One can then clear the list with (setf database-migrations::*migrations* nil) but then only newly modified migration files will be added. The solution then is to touch the .asd file after clearing the migrations list.

The second negative point is quite dangerous. The downgrade function takes a target version as parameter, with a default target of 0. This means that if you execute downgrade without specifying a target version you delete your whole database.

I am currently using Database-migrations and it works well for me. If for some reason I need to switch I will use cl-migrations.

Using Database-migrations

To address the danger of unintentionally deleting my database I created a wrapper function that does both upgrade and downgrade, and it requires a target version number.

Another practical issue I discovered is that upgrades and downgrades happen in the same order as they are defined in the migration file. If you create two tables in a single file where table 2 depends on table 1 then you can not revert / downgrade because Database-migrations will attempt to delete table 1 before table 2. The solution here is to use the def-queries-migration macro (instead of def-query-migration) which defines multiple queries simultaneously . If you get overwhelmed by a single definition that defines multiple tables the other option is to stick with one migration definition per file.

New projects:

• anypool — General-purpose pooling library — BSD 2-Clause
• avl-tree — An implementation of the AVL tree data structure. — MIT
• cl-aubio — Aubio bindings for Common Lisp — GPLv3
• cl-interval — Intervals, interval trees — NewBSD, LLGPL
• cl-liballegro — Allegro 5 game programming library bindings for Common Lisp — Allegro 5 - http://alleg.sourceforge.net/license.html
• cl-mime-from-string — A one function library to return a mime-type based on the file extension found at the end of a string. ie abc.txt -> text/plain. The common types implemented are from https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/MIME_types/Common_types — MIT
• cl-telegram-bot — Telegram Bot API, based on sovietspaceship's work but mostly rewritten. — MIT
• dns-client — A client for the DNS protocol. — zlib
• feeder — RSS, Atom and general feed parsing and generating — zlib
• perceptual-hashes — Perceptual hash algorithms for images — 2-clause BSD
• portable-condition-system — A portable condition system for Common Lisp — CC0
• ten — Template System for Common Lisp — MIT
• trivial-custom-debugger — Allows arbitrary functions to become the standard Lisp debugger — MIT
• trivial-with-current-source-form — Helps macro writers produce better errors for macro users — GPLv3
• vom-json — A json-formatted logger for vom — MIT
• vp-trees — Perceptual hash algorithms for images — 2-clause BSD

Updated projects: 3b-bmfont, 3bgl-shader, 3bmd, 3bz, a-cl-logger, alexandria, april, async-process, bdef, bp, ccldoc, chunga, cl+ssl, cl-ana, cl-capstone, cl-cffi-gtk, cl-collider, cl-containers, cl-environments, cl-gamepad, cl-gserver, cl-inotify, cl-markless, cl-pack, cl-patterns, cl-python, cl-rdkafka, cl-shlex, cl-sparql, cl-str, cl-tui, cl-utils, cl-webkit, clinenoise, clip, closer-mop, concrete-syntax-tree, croatoan, cserial-port, dartscltools, defenum, deploy, dexador, diff-match-patch, dissect, djula, doubly-linked-list, easy-routes, eclector, escalator, fast-generic-functions, fast-io, flexi-streams, flexichain, float-features, fset, fucc, functional-trees, fxml, gendl, graph, gtirb, hu.dwim.computed-class, hu.dwim.def, hu.dwim.perec, hu.dwim.presentation, hu.dwim.quasi-quote, hu.dwim.walker, hu.dwim.web-server, hunchentoot-multi-acceptor, ironclad, keystone, lispqr, literate-lisp, maiden, maxpc, mcclim, mmap, modularize, mutility, nodgui, numcl, numpy-file-format, origin, osicat, overlord, parachute, patchwork, petalisp, petri, phoe-toolbox, plump, policy-cond, polisher, postmodern, pzmq, qtools, quilc, qvm, roan, rpcq, s-graphviz, s-http-client, s-http-server, s-sysdeps, s-utils, sanity-clause, scalpl, sealable-metaobjects, sel, select-file, serapeum, sketch, skippy-renderer, snappy, softdrink, spinneret, staple, stumpwm, sucle, swank-client, swank-crew, tooter, trace-db, trivial-features, trivial-file-size, vgplot, woo.

Removed projects: cl-password-store, fomus, rfc3339-timestamp, rpc4cl.

All the removed projects are removed because they no longer build. For the first two (cl-password-store and fomus), I was unable to get a response from the authors. The other two (rfc3339-timestamp, rpc4cl) the author was responsive, but has abandoned the projects.

To get this update, use (ql:update-dist "quicklisp"). Enjoy!

A number of people support Quicklisp with a monthly contribution through PayPal. I recently set up a Quicklisp Patreon page as an alternative - if you are interested in supporting Quicklisp, feel free to check it out.

As any Common Lisp coder knows, a REPL is an incredibly useful tool. It can be used not just for development, but for running all sorts of tasks. Personally, I don't bother making my Lisp tools into executable scripts and just run them directly from SLIME. As such, any operation that requires leaving the REPL is quite inconvenient. For me, one such operation was viewing image files, for example in conjunction with my match-client:match tool. So lately I've been researching various methods to incorporate this functionality into the normal REPL workflow. Below, I present 3 methods that can be used to achieve this.

Open in external program

This one's easy. When you want to view a file, launch an external process with your favorite image viewer. On Windows a shell command consisting of the image filename would launch the associated application, on Linux it's necessary to provide the name of the image viewer.

(defvar *image-app* nil) ;; set it to '("eog") or something

(defun view-file-native (file)
  (let ((ns (uiop:native-namestring file)))
    (uiop:launch-program (if *image-app*
                             (append *image-app* (list ns))
                             (uiop:escape-shell-token ns)))))

Note that uiop:launch-program is used instead of uiop:run-program. The difference is that launch- is non-blocking - you can continue to work in your REPL while the image is displayed, whereas run- will not return until you close the image viewer.

Also note that when the first argument to run/launch-program is a string, it is not escaped, so I have to do it manually. And if the first argument is a list, it must be a program and a list of its arguments, so merely using (list ns) wouldn’t work on Windows.

Inline image in REPL

The disadvantage of the previous method is that the external program might steal focus, appear on top of your REPL and disrupt your workflow. And it's well known that Emacs can do everything, including viewing images, so why not use that?

In fact, SLIME has a plugin specifically for displaying images in REPL, slime-media. However it’s difficult to find any information on how to use it. Eventually I figured out that SWANK (SLIME’s CL backend) needs to send an event :write-image with appropriate arguments and slime-media's handler will display it right in the REPL. The easiest way is to just send the file path. The second argument is the resulting image's string value. If you copy-paste (sorry, "kill-yank") it in the repl, it would act just like if you typed this string.

(swank::send-to-emacs '(:write-image "/path/to/test.png" "test"))

You can even send raw image data using this method. I don't have anything on hand to generate raw image data so here's some code that reads from a file, converts it to a base64 string and sends it over SWANK.

(with-open-file (in "/path/to/test.png" :direction :input  :element-type '(unsigned-byte 8))
                (let* ((arr (make-array (file-length in) :element-type '(unsigned-byte 8)))
                       (b64 (progn (read-sequence arr in) (cl-base64:usb8-array-to-base64-string arr))))
                  (swank::send-to-emacs `(:write-image ((:data ,b64 :type swank-io-package::png)) "12345"))))

Note that the first argument to :write-image must be a list with a single element, which is itself a plist containing :data and :type keys. :data must be a base64-encoded raw image data. :type must be a symbol in swank-io-package. It’s not exactly convenient, so if you’re going to use this functionality a helper function/macro might be necessary.

Image in a SLIME popup buffer

Inline images are not always convenient. They can’t be resized, and will take up as much space as is necessary to display them. Meanwhile EMACS itself has a built-in image viewer (image-mode) which can fit images to width or height of a buffer. And SLIME has a concept of a “popup buffer” which is for example used by macroexpander (C-c C-m) to display the result of a macro expansion in a separate window.

Interestingly, slime-media.el defines an event :popup-buffer but it seems impossible to trigger it from SWANK. It is however a useful code reference for how to create the popup buffer in ELisp. This time we won’t bother with “events” and just straight up execute some ELisp code using swank::eval-in-emacs. However by default, this feature is disabled on Emacs-side, so you’ll have to set Emacs variable slime-enable-evaluate-in-emacs to t in order for this method to work.

Also Emacs must be compiled with ImageMagick for the resizing functionality to work.

Anyway, the code to view file in the popup buffer looks like this:

(defun view-file-slime (file &key (bufname "*image-viewer*"))
  (let ((ns (namestring file)))
    (swank::eval-in-emacs
     `(progn
        (slime-with-popup-buffer (,bufname :connection t :package t)
          (insert-image (create-image ,ns))
          (image-mode)
          (setf buffer-file-name ,ns)
          (not-modified)
          (image-toggle-display-image))
        ;; try to resize the image after the buffer is displayed
        (with-current-buffer ,bufname (image-toggle-display-image))))))
    ))

Arriving to this solution has required reading image-mode’s source code to understand what exactly makes image-mode behave just like if the image file was opened in Emacs via C-x C-f. First off, image-mode can be a major and a minor mode - and the minor mode is not nearly as useful. slime-with-popup-buffer has a :mode keyword argument but it would cause image-mode to be set before the image is inserted, and it will be a minor mode in this case! Therefore (image-mode) must be called after insert-image.

Next, the buffer must satisfy several conditions in order to get image data from the filename and not from the buffer itself. Technically it shouldn’t be necessary, but I couldn’t get auto resizing to work when data-p is true. So I set buffer-file-name to image’s filename and set not-modified flag on.

Next, image-toggle-display-image is called to possibly resize the image according to image-mode settings. It's called outside of slime-with-popup-buffer for the following reason: the buffer might not yet be visible and have any specific dimensions assigned to it, and therefore resizing will do nothing.

Here’s an example of how calling this function looks in Emacs.

The position of the popup buffer depends on whether the original Emacs window is wide enough or not. I think it looks better when it’s divided vertically. Use M-x image-transform-fit-to-height or M-x image-transform-fit-to-width to set up the auto-resizing method (it gets remembered for future images). Unfortunately there’s no way to fit both height and width, at least with vanilla Emacs. I prefer fit-to-width because in case the image is too tall, it is possible to scroll the image vertically with M-PgDn and M-PgUp from the other buffer. Unlike other image-mode buffers, this buffer supports a shortcut q to close itself, as well as various SLIME shortcuts, for example C-c C-z to return to the REPL.

That's it for now, hope you enjoyed this overview and if you happen to know a better way to display images in Emacs, I would be interested to hear about it.

Native pink trillium, aka wake-robin, presumably named such because they bloom in spring when the robins are warbling like mad? ‘Wake-robin’ is just a great name for a flower. 

Heh. K Tempest Bradford happened across an old photo of me and Neil — I’m in a sari, so I’m guessing this was at the Hugos. We look very young, so at least a decade ago — maybe two. 

Slow morning, it feels like. I did get up reasonably early, but ended up spending an hour sorting through exercise clothes, trying to figure out what fits well, what’s worn out, etc. Since I’m constantly going in and out of my house gardening these days, I want clothes that I’m comfortable being seen in, which … Continue reading "Slow Morning"

Packed up the first of the local Mother’s Day treat packages — lily-of-the-valley soap and bath salts, baked goodies and confections. And locals get a little potted lily-of-the-valley too. Supplies limited — I think I can do 4 more before I run out of cookies. I’m done with baking for a bit…need to get back … Continue reading "There Will Be Dragons"

Really serious about staying out of airplanes now. Two airplane trips this year. A low key summer not feeling terrific. Lots of time doing more local stuff and not regretting it at all. Here is the short list of places. Four states. Stars indicate multiple visits to the exact same place. Past years: 2018, 2017, […]

So I got all my wrap-ups written and here’s the short list libraries visited books read places stayed other events (i.e. timeline of big events in 2019) I’m a few days behind last year but more or less on schedule. And like last year, January is a Wikipedia month, staying busy writing articles and making […]

January is a hunkering month. I get out when I can, stay home when I can’t, keep the sun box pointed at my face, and work on Wikipedia, learning new things and giving other people the chance to learn about my world. The #Lib1Ref campaign is happening, I’m barely paying attention to it, but it’s as […]

So I’d been kind of waiting until I had a positive “Hey I’m feeling better!” update. I had my head down, doing my thing, being basically okay but a little crabby at having felt kinda lousy since August. And two things happened. 1. COVID-19 2. The medicine my doc gave me for GERD (or something […]

This is the Call for Nominations for the 2019 Theszie Awards (the rec.sport.pro-wrestling Awards). To nominate candidates for all categories, you may use this form. Nominations are due by January 5, 2019. Finally, to see previous years’ results, click here for 2018, click here for 2017, here for 2016, here for 2015, here for 2014, […]

This is the Call for Votes for the 2019 RSPW (Theszie) Awards. You can vote here. The Theszies are the oldest fan awards in pro wrestling history, going back to 1990 (when Mr. Perfect quite appropriately won Best Wrestler and Junkyard Dog v. Ric Flair at Clash of the Champions XI won Worst Match). They […]

Welcome to the results of the 2019 Theszies / Rec.sport.pro-wrestling Awards. (Editor’s note: Some of you may be wondering “what took so long?” The answer is: 1.) Immediately after the voting period ended, my wife and I went on a previously-scheduled vacation to Italy, which was great, and 2.) then when we got back we […]

The Worst Things For Sale is Drew's blog. It updates every day. Subscribe to the Worst Things For Sale RSS!