Knowing that you've found signs of life beyond Earth may not be as clear-cut and simple as one might think.

New research suggests liquid water might be hiding under the surface of Mars. Could life be there too?

NASA is underfunding planetary exploration relative to recommendations made by the National Academies Decadal Survey report, resulting in mission delays and cancelations.

The Planetary Society and Lerner Publishing Group have teamed up to bring young readers an engaging series of books that make space science fun and accessible.

China is in the early stages of planning a huge space observatory to help answer the matter of whether we are alone in the galaxy.

Looking at life on Earth from another perspective.

Looking to the past to guide the search for life.

Life thrives on Earth, and we even send evidence of our presence out into the Solar System. Is anyone out there looking for us?

NASA is preparing to launch its flagship mission to explore Jupiter’s moon Europa. Launching sometime in October 2024 and arriving in 2030, the mission will explore the icy world with a subsurface ocean that scientists think could have conditions favorable to life.

For a mission that doesn’t aim to find alien life, Europa Clipper may come surprisingly close.

A Martian cloud atlas, LightSail wins big, and multiple missions coast toward launch.

Europa Clipper will soon head for Jupiter's icy, potentially habitable moon. Without the advocacy efforts of The Planetary Society and our members, the mission may never have been possible.

This week we look forward to launches, gaze at glowing auroras, and get creative with glitter.

NASA’s Europa Clipper spacecraft launched today aboard a SpaceX Falcon Heavy rocket from NASA’s Kennedy Space Center in Cape Canaveral, Florida.

It’s been a great year for space exploration. Now you get to pick the highlights.

How the inclusion of Katie Taylor v Amanda Serrano on the bill legitimises the carnival of Mike Tyson v Jake Paul in Texas

We just made the decision to ship the remaining copies of Feast to us. There are about 800 left, out of the original 2000 copy print run. Which is a lot of copies, but given that we were originally scheduled to launch March 2020, and had to cancel an entire summer’s worth of scheduled bookstore […]

Here is Beorn in his full fury. He has been given a Matt varnish that really helped. This proved even more effective once his mouth, eyes and nose was given a coat of gloss varnish. This brought him to life and gave him a sparkle in his eye.

I ended up putting more and more greenstuff on him until he was completely covered. It's good to have him covered in thick, shaggy fur as he looks more wild and rugged.


The Eagles are coming! Eagles and White wolves get their various layers of flock.
A lovely new Reaper miniatures Dwarf joins the latest group of Dwarves. Reaper make great 'hero' figures to adorn any warband. They can be quite large sculpts so it can be a risky business ordering them, this one is perfect though.
Beorn's base has to be 24cm wide for the game I'm playing, so two side bases were made. These bring the base up to the right measurement and allow for more bear carnage.
Beorn with his slobbering maw but cute adorable shiny eyes.
More goblin wreckage. Wargs too have not escaped the Bears fury and lie amongst the fallen. A few goblins still cling to life.
It was fun to sculpt great claw wounds in the orcs clothes and equipment. Everything has been torn and crushed to pieces.
Beorn's base with some Bodyguard of Bolg behind him. I'm not sure I'll have time to finish this big orcs before the weekend.

Bard of Lake town with his new fixed banner. This banner was a bit too tall for the storage box and the spearpoint snapped off, twice!. So using my flattened brush bristle technique, he's got a new much tougher one. The plastic used for brush bristles make them fantastic for super glue!

As Bard needs to be free to roam around, another replacement base was needed. Here is Bard with a fellow command stand. Bard is another Reaper Miniatures figure and is slightly bigger than most. This is fine as he is a hero and it suits his manly character.

More carnage, this poor Warg has been disemboweled. This is pretty grim but I did have fun sculpting it. Facing a giant werebear was never going to end well. Again, a coat of gloss helps bring it to life 
Another shot of the basing production line.
Bard of Esgaroth.

 The blue theme really helped to tie these militia type troops together. I tested the new banner tip and it comes below the storage box height, hopefully it won't get crushed again.

Earlier this year, I started working through the online book Ray Tracing In One Weekend (Book 1). I have been following along with it in Common Lisp, and I have been extending it all from 3-dimensional to n-dimensional.

I reproduced 4-dimensional versions of all of the book images which you can see on my weekend-raytracer github page.

Here is the final image. This is a 250-samples-per-pixel, 640x360x10 image plane of three large hyperspheres (one mirrored, one diffuse, one glass) atop a very large, diffuse hypersphere. Also atop this very large hypersphere are a bunch of smaller hyperspheres of varying colors and materials. The image is rendered with some defocus-blur.

Final image of 4-dimensional scene

Caveat: This depends on a patched version of the policy-cond library that is not in the current Quicklisp distribution but should be in the next.

 New projects: 

• adp-github — ADP extension to generate github markdown files. — MIT
• adp-plain — Add Documentation, Please... using plain text. An extension of ADP to generate files with barely additional features. — MIT
• allioli — Alliolification — MIT
• alternate-asdf-system-connections — Allows for ASDF system to be connected so that auto-loading may occur. This is a fork of asdf-system-connections and incorporates a load-system-driven mechanism for loading dependencies and also loads the dependencies of the connections. — MIT
• cbor — CBOR encoder/decoder — MIT
• charje.documentation — Documentation is an opinionated yet customizable docstring parsing library. — AGPL V3 or any later version
• chipi — House automation bus in Common Lisp — Apache-2
• cl-aseprite — Aseprite file format parser — GPLv3
• cl-astar — A heavily optimized yet flexible A* pathfinding algorithm implementation — MIT
• cl-ceigen-lite — A Common Lisp wrapper around CEIGEN-LITE - which is itself a C wrapper around the C++ Eigen library. — MIT
• cl-cf — Computations using continued fractions — GPL-3
• cl-concord — CONCORD implementation based on Common Lisp — LGPL
• cl-duckdb — CFFI wrapper around the DuckDB C API — MIT License
• cl-fastcgi — FastCGI wrapper for Common Lisp — BSD License
• cl-flx — Rewrite emacs-flx in Common Lisp — MIT
• cl-frugal-uuid — Common Lisp UUID library with zero dependencies — MIT License
• cl-gog-galaxy — A wrapper for the GOG Galaxy SDK — zlib
• cl-lc — List comprehensions — MIT
• cl-naive-ptrees — Functions to make it easier to work with plist(s) and plist trees. Works with plist(s) pairs as units and not as individual list items. — MIT
• cl-qoa — An implementation of the Quite Okay Audio format. — zlib
• cl-reddit — Reddit client api library — BSD
• cl-resvg — An up-to-date bindings library for the resvg SVG rendering library — zlib
• cl-trivial-clock — Common Lisp library to get accurate wall-clock times on multiple platforms — MIT License
• clack-cors — A Clack middleware to set CORS related HTTP headers. — Unlicense
• clack-prometheus — Clack middleware to serve stats in Prometheus format. — Unlicense
• clith — Common Lisp wITH macro. A general WITH macro. — MIT
• clj-arrows — Implements Clojure-styled threading/transformation macros. — MIT
• clos-encounters — A collection of OOP patterns benefiting from the CLOS MOP. — Unlicense
• coalton — An efficient, statically typed functional programming language that supercharges Common Lisp. — MIT
• cocoas — A toolkit library to help deal with CoreFoundation, Cocoa, and objc — zlib
• com.danielkeogh.graph — A fast an reliable graph library. — MIT
• fast-mpsc-queue — Multi-Producer Single-Consumer queue implementation. — MIT
• file-finder — File finder. Enable rapid file search, inspection and manipulation. — GPL3+
• golden-utils — A utility library. — MIT
• hiccl — HTML generator for Common Lisp — MIT
• hsx — Hypertext S-expression — MIT
• hunchentoot-stuck-connection-monitor — Monitors hunchentoot connections and logs the connections stuck in the same state for a long time (due to slow or inactive clients and network stream timeouts that hunchentoot tries to utilize not working properly). Offers an option to shutdown the stuck connections sockets manually or automatically, thus unblocking the connection threads and preventing thread and socket leak. See https://github.com/edicl/hunchentoot/issues/189 — BSD-2-Clause
• incless — A portable and extensible Common Lisp printer implementation (core) — BSD
• inravina — A portable and extensible Common Lisp pretty printer. — MIT
• invistra — A portable and extensible Common Lisp FORMAT implementation — BSD
• knx-conn — KNXnet/IP implementation in Common Lisp — GNU GPL, version 3
• machine-state — Retrieve machine state information about CPU time, memory usage, etc. — zlib
• myweb — simple web server written in common lisp for educational reasons — LGPLv3
• noisy — Perlin noise for arbitrary numbers of dimensions. — MIT
• nontrivial-gray-streams — A compatibility layer for Gray streams including extensions — MIT
• open-with — Open a file in a suitable external program — zlib
• openai-openapi-client — Openai API client — AGPLv3+
• openrpc — CI for Common Lisp OpenRPC library. — BSD
• parse-number-range — Parses LOOP's convenient "for-as-arithmetic" syntax into 5 simple values: from, to, limit-kind (:inclusive, :exclusive or nil if unbounded), by (step) and direction (+ or -)). Further related utilities are provided. Intended for easy implementation of analogous functionality in other constructs. — Public Domain
• precise-time — Precise time measurements — zlib
• pregexp — Portable regular expressions for Common Lisp — MIT-like
• progressons — Display a progress bar on one line. — MIT
• quaviver — A portable and extensible floating point string library — MIT
• quilc — A CLI front-end for the Quil compiler — Apache License 2.0 (See LICENSE.txt)
• qvm — An implementation of the Quantum Abstract Machine. — Apache License 2.0 (See LICENSE.txt)
• random-sampling — Functions to generate random samples with various distributions — zlib
• rs-dlx — Knuth's Algorithm X with dancing links. — Modified BSD License
• scrapycl — The web scraping framework for writing crawlers in Common Lisp. — Unlicense
• smoothers — Statistical methods to create approximating functions that attempt to capture important patterns in the data, while leaving out noise or other fine-scale structures/rapid phenomena. — MS-PL
• trivial-adjust-simple-array — A tiny utility to change array size ensuring it is simple. — MIT
• trivial-system-loader — A system installation/loading abstraction for Common Lisp — MIT
• trivial-toplevel-commands — Trivial Toplevel Commands allows to define toplevel commands available on most implementations in a portable fashion. — BSD-3 Clause
• trivial-toplevel-prompt — Portability library to customize REPL prompts. — BSD-3 Clause
• utf8-input-stream — A UTF-8 string input stream over a binary stream for Common Lisp — MIT
• whereiseveryone.command-line-args — Automatically create a command-line-argument parser for a given Common Lisp function definition. — AGPL v3 or any later version

Updated projects: 3b-bmfont, 3bgl-shader, 3bmd, 3d-math, 3d-spaces, 40ants-asdf-system, 40ants-slynk, access, acclimation, action-list, adhoc, adopt, adp, agnostic-lizard, alexandria, alexandria-plus, anatevka, anypool, april, arc-compat, architecture.builder-protocol, array-utils, arrow-macros, assoc-utils, async-process, atomics, auto-restart, aws-sdk-lisp, babel, bdef, bike, binary-structures, binding-arrows, birch, blackbird, bordeaux-threads, calm, carrier, caveman, ccldoc, cephes.cl, cepl, cerberus, cffi, cffi-object, cffi-ops, chanl, chunga, ci, ci-utils, ciao, cl-6502, cl-algebraic-data-type, cl-all, cl-ansi-term, cl-async, cl-atelier, cl-autowrap, cl-base32, cl-bmas, cl-bmp, cl-bnf, cl-brewer, cl-buchberger, cl-cmark, cl-collider, cl-colors2, cl-confidence, cl-containers, cl-cookie, cl-csv, cl-custom-hash-table, cl-cxx-jit, cl-data-structures, cl-dbi, cl-digraph, cl-dot, cl-enchant, cl-environments, cl-fast-ecs, cl-fbx, cl-fluent-logger, cl-form-types, cl-forms, cl-freetype2, cl-gamepad, cl-github-v3, cl-gltf, cl-gobject-introspection, cl-graph, cl-grip, cl-gserver, cl-hamcrest, cl-hash-util, cl-html-readme, cl-i18n, cl-info, cl-ini, cl-ipfs-api2, cl-kanren, cl-lib-helper, cl-liballegro, cl-liballegro-nuklear, cl-log, cl-markless, cl-marshal, cl-migratum, cl-mixed, cl-modio, cl-mount-info, cl-mpg123, cl-mssql, cl-mustache, cl-mysql, cl-neovim, cl-netpbm, cl-oju, cl-opengl, cl-opensearch-query-builder, cl-opus, cl-patterns, cl-plus-ssl-osx-fix, cl-ppcre, cl-project, cl-protobufs, cl-pslib, cl-pslib-barcode, cl-rashell, cl-readline, cl-sat.minisat, cl-sdl2-image, cl-sdl2-mixer, cl-sdl2-ttf, cl-sendgrid, cl-sentry-client, cl-skkserv, cl-smtp, cl-ssh-keys, cl-steamworks, cl-str, cl-svg, cl-telegram-bot, cl-threadpool, cl-tiled, cl-torrents, cl-tqdm, cl-transducers, cl-transit, cl-unicode, cl-unification, cl-unix-sockets, cl-utils, cl-vectors, cl-vorbis, cl-wavefront, cl-webdriver-client, cl-webkit, cl-webmachine, cl-who, clack, clack-pretend, clad, classimp, clast, clath, clavier, clazy, clerk, clgplot, climacs, clingon, clip, clj-con, clj-re, clobber, clog, clog-ace, clog-collection, clog-plotly, clog-terminal, clohost, closer-mop, clss, cluffer, clunit2, clx, cmd, codata-recommended-values, codex, coleslaw, collectors, colored, com-on, common-lisp-jupyter, commondoc-markdown, compiler-macro-notes, conduit-packages, consfigurator, contextl, croatoan, ctype, cytoscape-clj, damn-fast-priority-queue, dartscluuid, data-frame, data-lens, datafly, dbus, decompress, defenum, definer, definitions, deflate, defmain, deploy, depot, deptree, dexador, dissect, djula, dns-client, doc, docs-builder, dsm, dufy, easter-gauss, easy-audio, easy-macros, easy-routes, eclector, equals, erjoalgo-webutil, erudite, esrap, event-emitter, external-program, external-symbol-not-found, fare-csv, fare-scripts, fast-http, fast-websocket, file-attributes, file-notify, file-select, filesystem-utils, fiveam, fiveam-matchers, flexi-streams, float-features, flow, fn, fset, functional-trees, fuzzy-dates, gadgets, generic-cl, github-api-cl, glfw, glsl-toolkit, harmony, hashtrie, helambdap, http2, hunchentoot, imago, in-nomine, inferior-shell, introspect-environment, ironclad, jose, js, json-mop, jsonrpc, jzon, khazern, lack, lass, lemmy-api, letv, lichat-protocol, lichat-tcp-client, linear-programming, lisp-binary, lisp-chat, lisp-critic, lisp-pay, lisp-stat, lispcord, lla, local-time, log4cl-extras, logging, lru-cache, magicl, maiden, maidenhead, manifolds, math, mcclim, memory-regions, messagebox, method-combination-utilities, mgl-pax, misc-extensions, mito, mk-defsystem, mmap, mnas-package, mnas-string, moira, multiposter, mutility, mutils, named-closure, ndebug, neural-classifier, new-op, nibbles, nibbles-streams, ningle, nodgui, north, numerical-utilities, nytpu.lisp-utils, omglib, ook, open-location-code, openapi-generator, orizuru-orm, overlord, papyrus, parachute, parse-number, pathname-utils, petalisp, phos, picl, plot, plump, plump-sexp, pngload, policy-cond, polymorphic-functions, postmodern, ppath, prometheus-gc, psychiq, purgatory, py4cl, py4cl2, py4cl2-cffi, qlot, qoi, query-fs, quick-patch, quickhull, quri, random-state, reblocks, reblocks-auth, reblocks-file-server, reblocks-lass, reblocks-navigation-widget, reblocks-parenscript, reblocks-prometheus, reblocks-typeahead, reblocks-ui, reblocks-websocket, rove, s-dot2, sandalphon.lambda-list, sb-fastcgi, sc-extensions, sel, select, serapeum, shasht, shop3, si-kanren, sketch, slime, slite, sly, snooze, spinneret, staple, static-vectors, statistics, stepster, stmx, stripe, swank-crew, swank-protocol, sxql, symath, system-locale, taglib, teddy, ten, testiere, tfeb-lisp-hax, tfm, tiny-routes, tooter, trivia, trivial-arguments, trivial-clipboard, trivial-file-size, trivial-gray-streams, trivial-main-thread, trivial-octet-streams, trivial-package-locks, trivial-package-manager, trivial-sanitize, trivial-shell, type-templates, typo, uax-15, uiop, usocket, vellum, vellum-binary, vellum-csv, vellum-postmodern, verbose, vernacular, vom, websocket-driver, winhttp, with-branching, with-contexts, woo, xhtmlambda, xml-emitter, yason, zippy, zpb-ttf.

Removed projects: abstract-arrays, ahungry-fleece, cl-cheshire-cat, cl-darksky, cl-epoch, cl-naive-store, convolution-kernel, dense-arrays, extensible-compound-types, extensible-optimizing-coerce, fast-generic-functions, flac-metadata, freebsd-ffi, listoflist, luckless, one-more-re-nightmare, postmodern-localtime, stumpwm-dynamic-float, stumpwm-sndioctl, unicly.

To get this update, use:

 (ql:update-dist "quicklisp")

Sorry this update took so long. My goal is to resume monthly releases.

Enjoy!

It's no secret that I'm an aficionado of Lisp. It's my go to language, especially when I don't know what I'm doing. I call it research and prototyping, but it's really just playing around until something works.

We had a need for some auditing of some of our databases at work. They ought to agree with each other and with what GitHub and CircleCI think. It took a couple of weeks part time to prototype a solution in Common Lisp. It showed that the databases were in 99% agreement and found the few points of disagreement and anomalies that we ought to fix or look out for.

I want to integrate this information into a dashboard on one of our tools. I prototyped this by spinning up a Common Lisp microservice that returns the information in JSON format.

But management prefers that new services are written in golang. It would be easier for me to rewrite the service in golang than to try to persuade others to use Common Lisp. It also gives me the opportunity to compare the two languages head to head on a real world problem.

No, this is not a fair comparison. When I wrote the Lisp code I was exploring the problem space and prototyping. I'm much more experienced with Lisp than with golang. The golang version has the advantage that I know what I want to do and how to do it. In theory, I can just translate the Common Lisp code into golang. But then again, this is a “second system” which is not a prototype and has slightly larger scope and fuller requirements. So this cannot be a true head to head comparison.

The first point of comparison is macros (or lack thereof). I generally don't use a lot of macros in Common Lisp, but they come in handy when I do use them. One macro I wrote is called audit-step, which you can wrap around any expresion and it prints out a message before and after the expression is evaluated. The steps are numbered in sequence, and nested steps get nested numbers (like step 2.3.1). If you wrap the major function bodies with this macro, you get a nice trace of the call sequence in the log.

Golang doesn't have macros, but it has first class functions. It's easy enough to write a function that takes a function as an argument and wraps it to output the trace messages. In fact, the macro version in Common Lisp just rewrites the form into such a function call. But the macro version hides a level of indentation and a lambda. In golang, my major functions all start with

func MajorFunction (args) int {
        return AuditStep("MajorFunction", "aux message", func() int {
                // body of MajorFunction
                // Actual code goes here.
        })    
}

The bodies of all my major functions are indented by 16 spaces, which is a little much.

I like higher order functions. I can write one higher order function and parameterize it with functions that handle the specific cases. In my auditing code, one such workhorse function is called collate. It takes a list of objects and creates a table that maps values to all objects in the list that contain that value. To give an example, imaging you have a list of objects that all have a field called foo. The foo field is a string. The collate function can return a table that maps strings to all objects that have that string in the foo field.

collate is very general. It takes a list of objects and four keyword arguments. The :key argument is a function that extracts the value to collate on. The :test argument is a function that compares two keys (it defaults to eql if not specified). The :merger argument is a function to add the mapped object to its appropriate collection in the table (it defaults to adjoin). The :default argument specifies the initial value of a collection in the table (it defaults to nil).

The :merger function is the most interesting. It takes the key and the object and the current value of the table at that key. It returns the new value of the table at that key. The default merger function is adjoin, which adds the object to the collection at the key if it is not already there. But you can specify a different merger function. For example, if you want to count the number of objects at each key, you can specify a merger function that increments a counter.

The functional arguments to the collate function are often the results of other higher order functions. For example, the :key argument is often the result of composing selector functions. The :merger argument is often the result of composing a binary merge function with a unary transformer function. The transformer function is often the result of composing a number of primitive selectors and transformers.

In Common Lisp, it is quite easy to write these higher order functions. We can compose two unary functions with the compose2 function:

(defun compose2 (f g)
  (lambda (x) (funcall f (funcall g x)))

and then compose as many functions as we like by fold-left of compose2 starting with the identity function:

(defun compose (&rest fs)
  (fold-left #'compose2 #'identity fs))

We can compose a binary function with a unary function in three ways: we can pipe the output of the binary function into the unary function, or we can pipe the output of the unary function into one or the other of the inputs of the binary function.

(defun binary-compose-output (f g)
  (lambda (x y) (funcall f (funcall g x y))))

(defun binary-compose-left (f g)
  (lambda (x y) (funcall f (funcall g x) y)))

(defun binary-compose-right (f g)
  (lambda (x y) (funcall f x (funcall g y))))

The collate function can now assume that a lot of the work is done by the :key and :merger functions that are passed in. It simply builds a hash table and fills it:

(defun collate (item &key (key #'identity) (test #'eql) (merger (merge-adjoin #'eql)) (default nil))
  (let ((table (make-hash-table :test test)))
    (dolist (item items table)
      (let ((k (funcall key item)))
        (setf (gethash k table) (funcall merger (gethash k table default) item))))))

(defun merge-adjoin (test)
  (lambda (collection item)
    (adjoin item collection :test test)))

So suppose, for example, that we have a list of records. Each record is a three element list. The third element is a struct that contains a string. We want a table mapping strings to the two element lists you get when you strip out the struct. This is easily done with collate:

(collate records
  :key (compose #'get-string #'third)
  :test #'equal      ; or #'string= if you prefer
  :merger (binary-compose-right (merge-adjoin #'equal) #'butlast))

The audit code reads lists of records from the database and from GitHub and from CircleCI and uses collate to build hash tables we can use to quickly walk and validate the data.

Translating this into golang isn't quite so easy. Golang has first class function, true, but golang is a statically typed language. This causes two problems. First, the signature of the higher order functions includes the types of the arguments and the return value. This means you cannot just slap on the lambda symbol, you have to annotate each argument and the return value. This is far more verbose. Second, higher order functions map onto parameterized (generic) types. Generic type systems come with their own little constraint language so that the computer can figure out what concrete types can correctly match the generic types. This makes higher order functions fairly unweildy.

Consider compose2. The functions f and g each have an input and output type, but the output type of g is the input type of f so only three types are involved

func Compose2[T any, U any, V any](f func(U) V, g func(T) U) func(T) V {
	return func(x T) V {
		return f(g(x))
	}
}

If want to compose three functions, we can write this:

func Compose3[T any, U any, V any, W any](f func(V) W, g func(U) V, h func(T) U) func(T) W {
	return func(x T) W {
		return f(g(h(x)))
	}
}

The generic type specifiers take up as much space as the code itself.

I don't see a way to write an n-ary compose function. It would have to be dynamically parameterized by the intermediate types of all the functions it was composing.

For the collate function, we can write this:

func Collate[R any, K comparable, V any](
	list *Cons[R],
	keyfunc func(R) K,
	merger func(V, R) V,
	defaultValue V) map[K]V {
	answer := make(map[K]V)
	for list != nil {
		key := keyfunc(list.Car)
		probe, ok := answer[key]
		if !ok {
			probe = defaultValue
		}
		answer[key] = merger(probe, list.Car)
		list = list.Cdr
	}
	return answer
}

We have three types to parameterize over: the type of the list elements (i.e. the record type) R, the type of the key K, and the type of the value V. The key type is needs to be constrained to be a valid key in a map, so we use the comparable constraint. Now that we have the types, we can annotate the arguments and return value. The list we are collating is a list of R elements. The key function takes an R and returns a K. The merger takes an existing value of type V and the record of type R and returns a new value of type V.

The magic of type inference means that I do not have to annotate all the variables in the body of the function, but the compiler cannot read my mind and infer the types of the arguments and return value. Golang forces you to think about the types of arguments and return values at every step of the way. Yes, one should be aware of what types are being passed around, but it is a burden to have to formally specify them at every step. I could write the Common Lisp code without worrying too much about types. Of couse the types would have to be consistent at runtime, but I could write the code just by considering what was connected to what. In golang, the types are in your face at every function definition. You not only have to think about what is connected to what, you have to think about what sort of thing is passed through the connection.

I'm sure that many would argue that type safety is worth the trouble of annotation. I don't want to argue that it isn't. But the type system is cumbersome, awkward, and unweildy, especially when you are trying to write higher order functions.

It is taking me longer to write the golang version of the audit service than it did to write the Common Lisp version. There are several reasons. First, I am more experienced with Common Lisp than golang, so the right Common Lisp idioms just come to mind. I have to look up many of the golang idioms. Second, the golang code is trying to do more than the Common Lisp code. But third, golang itself introduces more friction than Common Lisp. Programs have to do more than express the algorithm, they have to satisfy the type system.

There are more points of comparison between the two languages. When I get frustrated enough, I'll probably write another post.

Last week I finished a new service written in Common Lisp. It now runs in production© every mornings, and it expands the set of services I offer to clients.

It’s the 4th service of this kind that I developed: - they are not big - but have to be done nonetheless, and the quicker the better (they each amount to 1k to 2k lines of Lisp code), - they are not part of a super advanced domain that requires Common Lisp superpowers - I am the one who benefits from CL during development, - I could have written them in Python - and conversely nothing prevented me from writing them in Common Lisp.

So here lies the goal of this post: illustrate that you don’t need to need a super difficult problem to use Common Lisp. This has been asked many times, directly to me or on social media :)

At the same time, I want to encourage you to write a little something about how you use Common Lisp in the real world. Sharing creates emulation. Do it! If you don’t have a blog you can simply write in a new GitHub repository or in a Gist and come share on /r/lisp. We don’t care. Thanks <3

We’ll briefly see what my scripts do, what libraries I use, how I deploy them, what I did along the way.

Needless to say that I dogfooded my CIEL (beta) meta-library and scripting tool for all those projects.

Table of Contents

• Scripts n°4 and 2 - shaping and sending data - when you can write Lisp on the side
  • SFTP
  • Deploying
  • Script n°2 and simple FTP
• Scripts n°3 and 1 - complementary web apps
• Lasting words
• Links

Scripts n°4 and 2 - shaping and sending data - when you can write Lisp on the side

My latest script needs to read data from a DB, format what’s necessary according to specifications, and send the result by SFTP.

In this case I read a DB that I own, created by a software that I develop and host. So I could have developed this script in the software itself, right? I could have, but I would have been tied to the main project’s versioning scheme, quirks, and deployment. I rather had to write this script on the side. And since it can be done on the side, it can be done in Common Lisp.

I have to extract products and their data (price, VAT...), aggregate the numbers for each day, write this to a file, according to a specification.

To read the DB, I used cl-dbi. I didn’t format the SQL with SxQL this time like in my web apps (where I use the Mito light ORM), but I wrote SQL directly. I’m spoiled by the Django ORM (which has its idiosyncrasies and shortcomings), so I double checked the different kinds of JOINs and all went well.

I had to group rows by some properties, so it was a great time to use serapeum:assort. I left you an example here: https://dev.to/vindarel/common-lisps-group-by-is-serapeumassort-32ma

Dates have to be handled in different formats. I used local-time of course, and I still greatly appreciate its lispy formatter syntax:

(defun date-yymmddhhnnss (&optional date stream)
  (local-time:format-timestring stream
                                (or date (local-time:now))
                                :format
                                '((:year 4)
                                  (:month 2)
                                  (:day 2)
                                  (:hour 2)
                                  (:min 2)
                                  (:sec 2)
                                  )))

the 2 in (:month 2) is to ensure the month is written with 2 digits.

Once the file is written, I have to send it to a SFTP server, with the client’s codes.

I wrote a profile class to encapsulate the client’s data as well as some functions to read the credentials from either environment variables, the file system, or a lisp variable. I had a top-level profile object for ease of testing, but I made sure that my functions formatting or sending data required a profile parameter.

(defun send-stock (profile &key date) ...)
(defun write-stock (profile filename) ...)

Still nothing surprising, but it’s tempting to only use global parameters for a one-off script. Except the program grows and you pay the mess later.

SFTP

To send the result through SFTP, I had to make a choice. The SFTP command line doesn’t make it possible to give a password as argument (or via an environment variable, etc). So I use lftp (in Debian repositories) that allows to do that. In the end, we format a command like this:

lftp sftp://user:****@host  -e "CD I/; put local-file.name; bye"

You can format the command string and run it with uiop:run-program: no problem, but I took the opportunity to release another utility:

• https://github.com/vindarel/lftp-wrapper

First, you create a profile object. This one-liner reads the credentials from a lispy file:

(defvar profile (make-profile-from-plist (uiop:read-file-form "CREDS.lisp-expr"))

then you define the commands you’ll want to run:

(defvar command (put :cd "I/" :local-filename "data.csv"))
;; #<PUT cd: "I/", filename: "data.csv" {1007153883}>

and finally you call the run method on a profile and a command. Tada.

Deploying

Build a binary the classic way (it’s all on the Cookbook), send it to your server, run it.

(during a testing phase I have deployed “as a script”, from sources, which is a bit quicker to pull changes and try again on the server)

Set up a CRON job.

No Python virtual env to activate in the CRON environment...

Add command line arguments the easy way or with the library of your choice (I like Clingon).

Script n°2 and simple FTP

My script #2 at the time was similar and simpler. I extract the same products but only take their quantities, and I assemble lines like

EXTRACTION STOCK DU 11/04/2008
....978202019116600010000001387
....978270730656200040000000991

For this service, we have to send the file to a simple FTP server.

We have a pure Lisp library for FTP (and not SFTP) which works very well, cl-ftp.

It’s a typical example of an old library that didn’t receive any update in years and so that looks abandoned, that has seldom documentation but whose usage is easy to infer, and that does its job as requested.

For example we do this to send a file:

(ftp:with-ftp-connection (conn :hostname hostname
                                   :username username
                                   :password password
                                   :passive-ftp-p t)
      (ftp:store-file conn local-filename filename))

I left you notes about cl-ftp and my SFTP wrapper here:

• https://dev.to/vindarel/ftp-and-sftp-clients-for-common-lisp-1c3b

Scripts n°3 and n°1 - specialized web apps

A recent web app that I’m testing with a couple clients extends an existing stock management system.

This one also was done in order to avoid a Python monolith. I still needed additions in the Python main software, but this little app can be independent and grow on its own. The app maintains its state and communicates it with a REST API.

 

It gives a web interface to their clients (so my clients’ clients, but not all of them, only the institutional) so that they can:

• search for products
• add them in shopping carts
• validate the cart, which sends the data to the main software and notifies the owner, who will work on them.

The peculiarities of this app are that:

• there is no user login, we use unique URLs with UUIDs in the form: http://command.client.com/admin-E9DFOO82-R2D2-007/list?id=1
• I need a bit of file persistence but I didn’t want the rigidity of a database so I am using the clache library. Here also, not a great activity, but it works©. I persist lists and hash-tables. Now that the needs grow and the original scope doesn’t cut it any more, I wonder how long I’ll survive without a DB. Only for its short SQL queries VS lisp code to filter data.

I deploy a self-contained binary: code + html templates in the same binary (+ the implementation, the web server, the debugger...), with Systemd.

I wrote more on how to ship a standalone binary with templates and static assets with Djula templates here:

• https://lisp-journey.gitlab.io/blog/lisp-for-the-web-build-standalone-binaries-foreign-libraries-templates-static-assets/

I can connect to the running app with a Swank server to check and set parameters, which is super helpful and harmless.

It is possible to reload the whole app from within itself and I did it with no hiccups for a couple years, but it isn’t necessary the most reliable, easiest to set up and fastest method. You can do it, but nobody forces you to do this because you are running CL in production. You can use the industry’s boring and best practices too. Common Lisp doesn’t inforce a “big ball of mud” approach. Develop locally, use Git, use a CI, deploy a binary...

Every thing that I learned I documented it along the way in the Cookbook ;)

Another app that I’ll mention but about which I also wrote earlier is my first web app. This one is open-source. It still runs :)

 

In this project I had my friend and colleague contribute five lines of Lisp code to add a theme switcher in the backend that would help him do the frontend. He had never written a line of Lisp before. Of course, he did so by looking at my existing code to learn the existing functions at hand, and he could do it because the project was easy to install and run.

(defun get-template(template &optional (theme *theme*))
  "Loads template from the base templates directory or from the given theme templates directory if it exists."
  (if (and (str:non-blank-string-p theme)
           (probe-file (asdf:system-relative-pathname "abstock" (str:concat "src/templates/themes/" theme "/" template))))
      ;; then
      (str:concat "themes/" theme "/" template)
      ;; else :D
      template))

He had to annotate the if branches :] This passed the code review.

Lasting words

The 5th script/app is already on the way, and the next ones are awaiting that I open their .docx specification files. This one was a bit harder but the Lisp side was done sucessfully with the efficient collaboration of another freelance lisper (Kevin to not name him).

All those tasks (read a DB, transform data...) are very mundane.

They are everywhere. They don’t always need supercharged web framework or integrations.

You have plenty of opportunities to make yourself a favor, and use Common Lisp in the wild. Not counting the super-advanced domains where Lisp excels at ;)

---

Links

• https://lispcookbook.github.io/cl-cookbook/
• awesome-cl
• companies using Common Lisp in production (at least the ones we know)
• Common Lisp course in videos – it helps me, and you ;) I added 9 videos about CLOS last month, and more are coming. It’s 86 minutes of an efficient code-first approach, out of 7+ hours of total content in the course. After this chapter you know enough to read the sources of the Hunchentoot web server or of the Kandria game.

I have done some preliminary Common Lisp exploration prior to this course but had a lot of questions regarding practical use and development workflows. This course was amazing for this! I learned a lot of useful techniques for actually writing the code in Emacs, as well as conversational explanations of concepts that had previously confused me in text-heavy resources. Please keep up the good work and continue with this line of topics, it is well worth the price! [Preston, October of 2024]

A comment on my previous post said,

The most difficult thing when coming to a different language is to leave the other language behind. The kind of friction experienced here is common when transliterating ideas from one language to another. Go (in this case) is telling you it just doesn't like to work like this.

Try writing simple Go, instead of reaching for Lisp idioms. Then find the ways that work for Go to express the concepts you find.

That's not at all how I approach programming.

A friend of mine once paid me a high compliment. He said, “Even your C code looks like Lisp.”

When I write code, I don't think in terms of the language I'm using, I think in terms of the problem I'm solving. I'm a mostly functional programmer, so I like to think in terms of functions and abstractions. I mostly reason about my code informally, but I draw upon the formal framework of Lambda Calculus. Lambda Calculus is a simple, but powerful (and universal) model of computation.

Programming therefore becomes a matter of expressing the solution to a problem with the syntax and idioms of the language I'm using. Lisp was inspired by Lambda Calculus, so there is little friction in expressing computations in Lisp. Lisp is extensible and customizable, so I can add new syntax and idioms as desired.

Other languages are less accommodating. Some computations are not easily expressable in the syntax of the language, or the semantics of the language are quirky and inconsistent. Essentially, every general purpose fourth generation programming language can be viewed as a poorly-specified, half-assed, incomplete, bug-ridden implementation of half of Common Lisp. The friction comes from working around the limitations of the language.

 .

The kind people at Reactor Magazine have posted my two latest Mongolian Wizard stories, one yesterday and the other today. Thursday's "Halcyon Afternoon" took place during a rare moment of peace for Franz-Karl Ritter. But in today's "Dragons of Paris," it's warfare as usual. 

 .

So Tim Sullivan and I are, back in the early eighties, intensely browsing the science fiction paperback section of a bookstore when the young woman running the place comes up and brightly asks, "Are you interested in science fiction?"

Tim kind of shuffles his feet and, looking down at them, says, "Not really."

"Don't let him kid you!" I say. "This is Tim Sullivan, the famous science fiction writer. You may or may not have his books in stock, but you've definitely sold a lot of them."

"Really!" the clerk says, and addressing him directly, "How did you decide to become a science fiction writer?"

"Well," Tim replies, "I suck at art or music, and I'm not any good with my hands, so..."

And now you know why you probably have never heard of this good man.

The last time I saw Timothy R. Sullivan was at Gardner Dozois's funeral. He had lost a little weight and shaved off his unfortunate mustache and, to everyone's surprise, it turned out that he was handsome. Not that that mattered to anyone when he and Gregory Frost shared rooms on Brown Street. They two and Gardner Dozois and I were the beating heart of science fiction in Philly back then. We were, in the parlance of Saturday Night Live, "wild and crazy guys."

So the news that Tim died recently, of congestive heart failure, leaves me mourning not only him but a time in my life when we were all undiscovered geniuses only a matter of months away from the astonished recognition and accolades of a grateful world.

Old people like to say that youth is wasted on the young.  They're full of it. We all had great fun, great plans, and a heartfelt appreciation of how lucky we were to have such friends as each other. Somewhere in there, we managed to write a lot of worthwhile fiction.

Tim was a solid writer. He was a finalist for the Nebula Award. And he and I collaborated on a story, "Fantasies," which, it must be admitted, was not much of a much. He had a good start on a writing career when he veered into movies, acting in Somtow Sucharitkul's The Laughing Dead and co-writing and starring in Twilight of the Dogs, both ultra-low budget endeavors. He moved to California and then to Florida and focused on  scriptwriting and we fell out of touch. I regret that.

Rather than mope about the loss of someone who was a very good friend (we had lunch together when our friendship was new and when Tim objected to me picking up the check, I said, "Be honest. You're a writer, a creator. Don't you honestly feel that the world owes you a living?" Tim thought about it and replied, "Yes." I got out my wallet and said, "I've been authorized by the world to say: Fuck you. You're lucky to get a sandwich"), I would like to celebrate those days when we all knew we were the best thing about to happen to literature ever.

And you know what? I am authorized by the world to say we were.

Rest in peace, Tim. You never got your just deserts. But maybe nobody ever does.

Above, l-r: Gregory Frost, Gardner Dozois, Tim Sullivan, John Kessel (not a Philadelphian, but visiting), and me. Those were the days.

*

*I

Holiday of Hearts



View Comic!

World Wrestling



View Comic!

The implication of that



View Comic!

1. Why Virat Kohli, Jasprit Bumrah were missing from Perth nets; India ramp up privacy amid Manchester United-like security  Hindustan Times
2. Virat Kohli in focus: Intense net session begins for upcoming Test series against Australia  The Times of India
3. Virat Kohli in Australia for BGT: A timeline  India Today
4. Black veil of secrecy: India begin training in privacy in Perth  ESPNcricinfo
5. India to play intra-squad warm-up match at WACA on Friday ahead of Australia Tests but BCCI denies public viewing  Hindustan Times

1. Kanguva Box Office Day 1 Advance Sales: 568% Higher Than Suriya’s Last Release ET In USA, Selling 5,460 Tix/Hr In India!  Koimoi
2. Kanguva morning show: Tamilians miffed as other states get earlier shows for Suriya, Bobby Deol's film  Hindustan Times
3. 5 Reasons To Watch Suriya and Bobby Deol Starrer 'Kanguva' In Theatres  Zee News
4. Kanguva’s second half to be very crisp  123telugu
5. Kanguva advance booking: Suriya's film earns over Rs 4 crore ahead of release  India Today

1. Melania Trump's 'I am the most bullied person' video viral as she 'rejects' Jill Biden's tea invite  The Times of India
2. Melania Trump declines invite to meet Jill Biden over Mar-a-Lago raids when FBI ’snooped through her wardrobe’  Mint
3. Jimmy Kimmel again mocks Donald Trump for Melania's ‘interesting’ decision to…  Hindustan Times
4. Biden Invites Trump To White House, But Why Is Melania Refusing To Go?  News18
5. Melania Trump To Skip White House Meeting With Jill Biden: Report  NDTV

1. 3 Children, 3 Women Missing After 10 Suspected Kuki Militants Killed In Encounter In Manipur's Jiribam  NDTV
2. Manipur on boil: 2 more bodies found, 6 missing  The Times of India
3. Additional paramilitary forces rushed to Manipur amid spike in ethnic violence  Hindustan Times
4. Letters to The Editor — November 13, 2024  The Hindu
5. 2 men found dead, 6 of family missing day after militants killed in Manipur  India Today

Full Article