Packages, systems, modules, libraries - WTF?

Bart Simpson at the blackboard

At least once per week (my personal estimate) some "newbie" on c.l.l or a Lisp-related mailing list seems to be confused about packages one way or the other. They talk about "loading" packages, "requiring" packages, they wonder why, after loading a system, they still have to use package markers, and so on. This is a pet peeve of mine, and I think it might even be one of the reasons why newcomers find using Lisp libraries harder than it actually is.

As I usually end up trying to write a somewhat helpful explanation and as, naturally, all these explanations are very similar, I finally created this page, so the next time I can just point to it instead of typing the same thing over and over again.

First of all, you probably have to clear your mind. The term "package" is pretty much overloaded. Some Linux distributions like Debian or Gentoo have "packages", programming languages like Java, Perl, or Python have "packages", you name it. It is very likely that you came to Lisp with a preconceived notion of what a "package" is or should be. Forget about it while you're here.



In Common Lisp, a package is a first-class object the semantics of which are clearly defined by the language standard. In fact, from all the terms discussed on this page, it is the only one that (in the context of Common Lisp) has an unambiguous definition. Packages are, loosely speaking, containers for symbols. They are there to help you create separate namespaces within your programs, so to say.

Common Lisp has functions and macros to create, modify, investigate, and delete packages. A very good introduction to packages (and symbols) can be found in chapter 21 of Peter Seibel's excellent Lisp tutorial Practical Common Lisp. The definitive resource is chapter 11 of the (online version of) the ANSI Common Lisp specification.

That's basically all there is about packages. Specifically, you don't load packages. You can LOAD Lisp code which in turns creates a package, but that's something substantially different.

Also, if your Lisp complains that a specific package could not be found, that means that the package as a Lisp object isn't in your image (i.e. FIND-PACKAGE would return NIL) because it hasn't been created yet. It does not mean that the implementation was somehow searching the file system and failed. (A frequent reason for this kind of failure is that things happen in the wrong order. More about that below.)



Systems, as opposed to packages, are something the ANSI standard doesn't even mention. Nevertheless, most experienced Lisp programmers know the term because they will know and use some kind of system definition tool. The most prominent one nowadays is ASDF (used by the majority of open source Lisp libraries); another well-known system definition tool, and a lot older than ASDF, is MK:DEFSYSTEM. Several vendors also distribute their own system definition tools together with their Lisps, see for example LispWorks' Common Defsystem.

A system in this sense is, very loosely speaking, a bunch of code plus some instructions how it should be treated, for example which other systems it depends on, what should be loaded and/or compiled first, and so on. In other words, a system definition tool is similar in purpose to, say, make or Ant.

As an aside, system definition tools can usually do much more than that - Common Defsystem can for example integrate COM type library files, and ASDF is fully extensible and has been used, amongst other things, to compile C files. It is also routinely used to define test suites for the systems it describes.

And, by the way, although ASDF is pretty popular it's not ubiquitous. It comes pre-installed with Lisps like SBCL, OpenMCL, or AllegroCL, and it can probably be loaded into every other Lisp, but that doesn't make it a part of Common Lisp. Basically, it is just a bunch of code with no exact specification of what it's supposed to do and with different release versions which are likely not compatible with each other. Go figure...



The standard does mention modules, but only in passing. The two important things you need to know about REQUIRE, PROVIDE, and *MODULES* is that this functionality is deprecated and implementation-dependent. The fact that it's deprecated shouldn't bother you much. Every Common Lisp nowadays has these functions, and the chances that a new ANSI standard will emerge and all the implementations will suddenly drop these functions are, cough, very slim. What should bother you, though, is that REQUIRE might be convenient, but isn't portable. (If you don't care about portability, that's fine of course.)

For example, in LispWorks you can use

(require "foreign-parser")
to load a parser that can read C declarations, but that won't work on, say, OpenMCL. Likewise, you can use
(require :asdf)
to load ASDF on OpenMCL, but not on LispWorks.

Some Lisps offer hooks so that you can tweak the way REQUIRE works, and there are extensions like common-lisp-controller that couple REQUIRE with ASDF, but generally a module is some implementation-dependent thingy you shouldn't confuse with (ASDF) systems (not to mention with packages).



There's probably no clear definition for what a library is, but most people will think of it as a collection of code that is supposed to perform one or more specific tasks and is distributed as a whole, usually as some kind of compressed archive that can be downloaded somewhere. Actually, this vague term is I think the most appropriate one if you want to talk about Lisp software. Most Lisp libraries nowadays come with (ASDF) system definitions, but they don't have to. Depending on how you received them, it could be possible that they will pretend to be a module in your Lisp, but they don't have to. Also, a library will usually define one or more Lisp packages, but not necessarily so.

And, due to conventions and maybe lack of imagination, it can and will happen that the library "FOO" comes with a definition for a system "FOO" that you can maybe load as a module named "FOO". And once the code is loaded you'll have a new Lisp package called "FOO". These are four different things which just happen to have the same name! I admit that this is confusing, but I hope that the paragraphs above helped to clarify the situation a bit.


But my code still doesn't work!

A related issue is that often people complain that they can't compile a Lisp file containing code like this:

;; this line could also be (require :cl-ppcre)
(asdf:oos 'asdf:load-op :cl-ppcre)

(defun my-simple-number-scanner (string)
  (cl-ppcre:scan "^[0-9]+$" string))
Why is that? And why can I load this file if I can't compile it? And why can I compile it after I've loaded it? Isn't that strange?

No, it's not. The compiler will read the first form (which is an instruction to compile - if necessary - and load the CL-PPCRE system), but it won't execute it. The compiler is only interested in compiling code, after all. Once it is done with the first form, it'll proceed to the second form, the function definition. At this point it will eventually complain as the Lisp reader which tries to read this form encounters the sequence of characters "cl-ppcre:scan" which is supposed to denote an external symbol of the CL-PPCRE Lisp package, but there is no CL-PPCRE package yet. The process of loading the CL-PPCRE system will, amongst other things, create the CL-PPCRE package, but that hasn't happened yet. Read chapter 3 of the CLHS for more.

You could use EVAL-WHEN here to instruct the compiler to load the CL-PPCRE system before it reads the second form. You should rather find a better way to organize your code, though. The first form is basically a declaration that your code has a dependency on the CL-PPCRE system. This shouldn't be in the same file as your Lisp code. Write your own system definition for your code and put the dependencies there.

$Header: /usr/local/cvsrep/,v 1.13 2008/04/22 09:52:49 edi Exp $