Core User Manual
This manual documents the CC core software suite. It is intended for users/developers and assumes a intermediate level understanding of computer programming concepts such as OOP, FP, IDEs, and the shell.
The core is a Lisp development system used to build next-gen applications. It is composed of interconnected modules providing an exceptionally powerful development ecosystem.
1. Introduction
Welcome to the core, the primary software development ecosystem of CC.
The core provides the foundation of all applications we produce. It is optimized for rapid development of next-gen applications.
The core is highly opinionated - and these opinions are consistently hard-coded into our development practices.
For example:
- Linux is the only platform worth supporting
- We make no efforts to support cross-compilation or porting of our software to devices or operating systems the developer can't fully control such as Windows/MacOS/iOS/Android. We always assume the developer is running a GNU/Linux host.
- Portability is over-rated
- CL implementations (lisps) share a base language but many parts of the ANSI CL spec are left up to interpretation: threads, OS, GC, MOP, etc
- Many lisp packages attempt to account for as many lisps as possible via conditional compilation and wrappers - this clutters up the code base, is difficult to test, and is error-prone.
- Instead of accounting for multiple compilers, we realize that we only need one - SBCL currently. The core does not contain any support for alternate lisps.
- 'One true lisp' is all we need, and we're only concerned with internal consistency - we are never bound by the choices in implementation made by other lisps (although there is much to learn from them).
- Introspection is under-rated
- Black boxes (proprietary codes) are disrespectful to developers. Computer systems should be transparent by nature and make minimal efforts to 'hide' internal representations of data and code.
- Never hide a part of the system, unless you have a very good reason for doing so.
- Follow the Bleeding Edge
- We pull and rebuild the latest changes for all of our dependencies frequently
- We host mirrors of all our major dependencies
- Core development is always based on latest changes, can optionally downgrade for release builds where stability is valued
- incorporate pre-release APIs and our own custom patches as needed.
2. Overview
The core itself is not an application but is complex enough that it merits its own dedicated manual. There is library-specific and API documentation available for the core, so what we are concerned with here is the overall design, common patterns, and architecture of the core environment.
This manual is often technical, but occassionally theoretical in nature and intended for software developers with expertise in Lisp, GNU/Linux, and Emacs.
The Core is a Lisp system. Common Lisp is our language of choice and there is a decent amount of Emacs Lisp code too.
2.1. Source Code
ls -Ah
| api.org |
| info.org |
| man.org |
| readme.org |
- Common Lisp
TheSTDlibrary lives under the std directory, low-level FFI library bindings such asROCKSDBunder the alien directory and high-level user libraries likeRDBunder the lib directory. The skel directory contains the source of our multi-purpose project toolSKEL. - Emacs Lisp
The etc/emacs directory contains a complete Emacs configuration.
2.2. Images
Images refer to the binary file resulting from a call to
SAVE-LISP-AND-DIE. According to SBCL this file contains:
enough information to restart a Lisp process later in the same state, in the file of the specified name.
See the function docs for details - there are many options available which determine how the image can be used directly or dynamically from other programs.
The core images we produce are executables which contain the full
SBCL runtime at their disposal. Note that these images can be quite
large - multiple gigabytes in some cases, and so care should be taken
when enabling additional build-time features.
Note that image-based development is quite unique. You start inside
your Lisp implementation's core image which contains the Lisp runtime
in its executable. The runtime in SBCL is compiled C code which you
may be included with the images you save to make them executable, or
you can remove it to save space. Restarting an image in the exact
same state you saved it in is as simple as executing sbcl with the
--core argument or executing the saved core file (with runtime)
directly.
The development experience is exceptionally fluid when fully appreciated. You have everything at your fingertips, always fully compiled and ready to load into memory at a moment's notice.
Images are concatenative - they always build on an existing image. We start with the ANSI LISP 'kernel' image, then we save it, extend it, and repeat indefinitely. The natural evolution of this concept is to have your entire system in a Lisp image. It is a 'maximalist' philosophy which is deeply ingrained into the very idea of Lisp, although ultimately relegated to the long-forgotten Lisp Machines.
Image-based development does come with its own set of foot guns - it is more complex by nature and easy to end up in an unusable state when you compile in faulty code. However the biggest conflict that arises from images is that they are anti-UNIX.
GNU/Linux distributions today come with hundreds of binaries. The vast majority of these are single-use tools with simple text interfaces which aide themselves to composability via the shell. Lisp images are not single-use tools - they always contain a complete Lisp runtime in addition to any extensions you've added - so if you want to use them like UNIX binaries with one image per application or tool, you're duplicating a tremendous amount of code in each image1.
One of the goals of the core is to realize a full appreciation of
images and image-based development, starting from GNU/Linux Userspace
and working our way towards service management and then the
kernel. The general strategy is to reduce the OS environment while
extending our Lisp image. We aim to first serve as a default user
login shell, and ultimately challenge SystemD for PID1. If we are
successful, we will have a complete Lisp system compatible with the
Linux kernel.
3. Getting Started
3.1. Dependencies
To ensure dependencies on an existing system you should refer to the infra manual.
A PKGBUILD recipe is planned but not ready AO . Note
that RocksDB should be installed via the infra project (the default
extra/rocksdb package is not compatible).
3.1.1. Required Dependencies
The following dependencies are required - Arch Linux is assumed but other distros should work.
- linux-headers
- gcc
- sbcl
- emacs
- rocksdb (built WITHOUT jemalloc)
- gflags
- intel-oneapi-tbb
- zstd
- mercurial
- openssl
3.1.2. Optional Dependencies
Some of these dependencies may already be installed on your system, others will need to be installed manually or via infra.
- cuda
- evdev
- jack
- libjpeg-turbo
- xkbcommon
- keyutils
- sndfile
- libssh2
- btrfsutils
- gstreamer
- tree-sitter
- blake3
- liburing
- alsa
- ffmpeg
- chromaprint
- openblas (blas/lapack)
3.2. Bootstrap
Bootstrapping from SBCL is quite easy. Once all required dependencies are installed, just run bootstrap.lisp:
# clone the source code from a mirror
git clone
cd core
sbcl --script bootstrap.lisp
sudo .stash/core --script etc/script/install.lisp
Release tarballs are planned but not available AO .
3.3. Configuration
The next thing you'll want to take care of is user configuration. You
can save your configuration at your preferred XDG-compliant location
(~/.config/corerc is recommended).
;;; .corerc --- core init file -*- mode: common-lisp; -*-
(in-package :user)
(init :xdg) ;; apply user XDG config
(init :term) ;; detect current terminal
(init :log :level :debug) ;; enable global logger
;; (init :sys :sysdefs (sysdefs path-to-core t))
The only other development dependency which needs configuring is
Emacs. This can be done via the skel tool, but keep in mind that
this will overwrite the XDG-compliant emacs configuration directory
(~/.config/emacs) if it exists:
skel make install-emacs
#:debug 0.157 ; (#<rule install-emacs>)
3.3.1. Core
The core is a single binary which dispatches on argv[0] from the CLI. Each unique symlink to the core binary is treated as a separate dispatch to a 'main' function like so:
(define-multi-main dispatch-core
(progn (in-package :core-lisp)
#+nil ...)
(:skel (bin/skel::start-skel))
(:homer (bin/homer::start-homer))
(:mpk (bin/mpk::start-mpk))
#+nil ...)
The default path (when argv[0]=core) accepts a lisp file as
configuration just like ~/.sbclrc, corerc respected the
XDG_CONFIG_HOME variable though and may be located in
~/.config/corerc.
The current apps that read a configuration file on startup are:
- skel - project management:
skelrc - homer - home management:
homerc - mpk - media production kit:
mpkrc
By default they are located under XDG_CONFIG_HOME.
These apps all use the same configuration object protocol from the
OBJ/CONFIG package and are printed to various S-Expression. Most
commonly the 'pretty' format is used which is simply a plist with
implied outer parentheses:
:key1 val1
:key2 val2
Canonically that would be:
(:key1 val1
:key2 val2)
3.3.2. Emacs
If you don't already have your own Emacs config you can install the
default CC Emacs configuration using the skel command.
- AO the core should is intended to be hacked on using this config only but is subject to change in the future.
this will overwrite
$XDG_CONFIG_HOME/emacs/skel make install-emacssudo: a terminal is required to read the password; either use the -S option to read from standard input or configure an askpass helper sudo: a password is required sudo: a terminal is required to read the password; either use the -S option to read from standard input or configure an askpass helper sudo: a password is required sudo: a terminal is required to read the password; either use the -S option to read from standard input or configure an askpass helper sudo: a password is required #:debug 6.870001 ; (#<SIMPLE-RULE install-emacs>)
- the vanilla load sequence is applied first, starting with early-init.el if provided by the user then init.el
- init.el loads default.el which initializes all required MELPA packages and local libraries from etc/emacs/.
- the site
config.elis loaded next (optional, user controlled) - additionally load
user-custom-fileif it exists, defaulting to the name of the current user (foo.elwhen$USER=foo) - finally load the
custom-filecustom.elif it exists
- Defaults
Our Emacs configuration philosophy follows the same principles found elsewhere in the core - prioritizing built-ins/internals, and first-class support for modern Lisp development.
3.4. Demos
Core demo applications, examples, and tutorials can be found in the demo project with documentation available online.
3.5. Documentation
AO full-auto API docs are planned but not available. They will be located under the api directory.
lib docs will be move here
3.6. Testing
Each module in the core contains its own test suite located in the
tests.lisp file or tests directory and isolated to its own
system. The system is always the module name with a suffix of
/tests - for example the IO/TESTS system is responsible for
testing the IO module system.
3.6.1. Regression Tests
Testing the core can be done from the CLI using skel:
# skel compile tests
skel run tests
or from lisp:
(test-system :core)
3.7. Contributing
The core is a solo experiment and I don't plan on accepting contributions in the near future. The core is its own private island, isolated from the rest of the Common Lisp ecosystem and in almost all cases contributions would be better directed towards other packages.
See Awesome CL for a list of projects which could use additional support, and take note the comments scattered throughout the core source code which often reference popular packages from the ecosystem.
4. Concepts
4.1. Systems
4.2. Logical Hosts
Logical Pathnames are used sparingly in the core source but their
use is encouraged by users. Several logical hosts are pre-defined in
addition to the SBCL SYS host.
(list-all-logical-host-names)
| ETC | MEDIA | MPK | PACKY | SKEL | SRV | SYS | USER | USR | VAR |
4.3. Reader
4.8. Protocols
Protocols are keyed groupings of symbols, sort of like light-weight
packages.
(fmt-tree nil (find-module :std :proto :kernel) :plist t :layout :down)
:KERNEL
├─ :PACKAGE
│ ╰─ :STD/PRIM
├─ :DESCRIPTION
│ ╰─ "Standard funcallable object extensions."
├─ :VARIABLES
│ ╰─ (*KERNEL*)
├─ :CONDITIONS
│ ╰─ (KERNEL-INIT-ERROR NO-KERNEL-ERROR)
├─ :ACCESSORS
│ ╰─ (KERNEL KERNEL-INFO KERNEL-DOCUMENTATION)
├─ :METHODS
│ ╰─ (KERNEL-EXPRESSION)
├─ :FUNCTIONS
│ ╰─ (CHECK-KERNEL MAKE-KERNEL)
├─ :MACROS
│ ╰─ (DEFKERNEL)
├─ :PREDICATES
│ ╰─ (KERNELP)
╰─ :CLASSES
╰─ (KERNEL-CLASS KERNEL-OBJECT)
Protocols are designated by a key with an associated plist (the cdr of
(find-module ... :proto ...)). The list of available plist keys in
a protocol are identified by a variable:
std/defsys::*protocol-keyword-imports*
- :METHODS
- :FUNCTIONS
- :TYPES
- :VARIABLES
- :CONSTANTS
- :PARAMETERS
- :MACROS
- :CONDITIONS
- :RESTARTS
- :ACCESSORS
- :PREDICATES
- :CLASSES
- :STRUCTS
- :DECLARATIONS
- :GLOBALS
5. Skel
skel help
skel v0.1.1:4da8a49ea455+ --- A universal project development tool. usage: skel [opts] <command> [<arg>]
The skel tool is an extensible program designed around the idea of
projects.
5.1. Core
The skel/core module contains all internal components of the skel
system. Despite the name, much of this code is glue code which
stitches together multiple library protocols.
Most important data structures are represented with classes, slots,
and methods such as the SK-PROJECT class which is based on the
OBJ:PROJECT class (a generic project type).
(mapcar 'name (components (find-component :core (find-system :skel))))
- condition
- header
- var
- obj
- component
- project
- schema
- db
- log
- util
5.2. Configuration
The skel tool can be configured at runtime using the CLI flags, or
via local configuration files. The configuration files are in the
skelfile format and are assigned the following locations by default:
*system-skelrc*/etc/skelrc*user-skelrc*$XDG_CONFIG_HOME/skelrc
Configurations are applied to the same object in sequence - starting with the system configuration and then the user config. Both are optional.
Example ~/.config/skelrc:
cat ~/.config/skelrc
;;; skelrc
:name "Richard Westhaver"
:email "richard.westhaver@gmail.com"
:version "0.1.0"
:vc :hg
:logger (:level :info)
:stash ".stash"
:store "/opt/store"
:scripts ("~/.stash/scripts")
5.3. Projects
Projects are the root objects of the skel system. A project consists
of a list of slots which are populated automatically or by the
user. Once a project is initialized the slots can be accessed and a
set of high-level operations can be performed.
The easiest way to initialize a project is with a skelfile. These
files are parsed as plists and generate sk-project objects based
on their contents.
The simplest project looks like this:
:name hello-world
Don't worry, this is still lisp - the parentheses are implied. All we actually need is a name to generate an object, and we can fill in the blanks later as the project develops.
Here is a more verbose example, still with only metadata:
;;; skelfile --- core skelfile -*- mode: skel; -*-
:name "core"
:author "Richard Westhaver <ellis@rwest.io>"
:version "0.1.0"
:license "MPL"
:description "The Compiler Company Core"
:vc :hg
:tags ("core")
:docs ((:org "readme") (:org "install") (:org "tests") (:org "todo"))
:include ("emacs/emacs.sk")
Another example:
;;; skelfile @ 2023-10-08.02:37:25 -*- mode: skel; -*-
:name skel
:author "ellis"
:version "0.1.0"
:description "a hacker's project compiler"
:license "MPL"
:vc :hg
:tags ("lisp")
:bind ((target :skel/cli))
:rules ((build () (print (asdf:make target)))
(clean () #$rm -rf */*.fasl$#))
:docs ((:org "readme"))
:components
((:elisp "sk"))
:stash "~/.stash"
The currently accessible sk-project can be easily inferred via the
init protocol:
(init :skel) ;; find the top-level skel project
(describe *project*)
#<SKEL-PROJECT ORG :components 1 :rules 19>
[standard-object]
Slots with :INSTANCE allocation:
NAME = ORG
HOOK = #<unbound slot>
PROVIDE = #<unbound slot>
REQUIRE = #<unbound slot>
PATH = #P"/home/ellis/src/org/skelfile"
AUTHOR = ("Richard Westhaver" . "ellis@rwest.io")
VERSION = "0.1.0"
TAGS = (ORG DOCS NOTES META PLAN ARCHIVE GRAPH)
LINKS = #<unbound slot>
DESCRIPTION = "CC Org Files"
LICENSE = "MPL"
AST = NIL
ID = 1784914079056115652
VC = #<HG-REPO {1203CA8FE3}>
SRC = #P"/home/ellis/src/org/"
STASH = #P"/home/ellis/src/org/.stash/"
STORE = #P"/home/ellis/src/org/.stash/store/"
CACHE = #P"/home/ellis/src/org/.stash/cache/"
COMPONENTS = #(#<EMACS-LISP-FILE :ID 1091-ba75-0b6c-698b>)
BIND = ((SUBMODS (LIST "graph" "meta" "plan" "docs" "notes" "archive")))
RULES = #(#<SIMPLE-RULE default build publish deploy dist-html dist-source dis..
5.4. Components
In addition to the core objects we provide a set of COMPONENT
definitions which convert core objects to a corresponding external
representation. For example the MAKEFILE class defined in
comp/makefile.lisp can be built from or consumed by a project.
(mapcar 'name (components (find-component :comp (find-system :skel))))
- lisp
- make
- rust
- python
- emacs
- shell
- box
- pod
- infer
5.5. Rules
Skel rules are the Lisp equivalent of Makefile rules. They are
almost always executed at the CLI via the make command.
Rules are a project slot so you can list them with show:
skel show rules
#<SIMPLE-RULE default build publish deploy dist-html dist-source dist-repos> #<SIMPLE-RULE build build-css update-docs build-graph build-docs build-agenda> #<SIMPLE-RULE build-graph> #<SIMPLE-RULE build-agenda> #<SIMPLE-RULE update-docs> #<SIMPLE-RULE build-docs> #<SIMPLE-RULE clean> #<SIMPLE-RULE build-css> #<SIMPLE-RULE deploy-css> #<SIMPLE-RULE browse> #<SIMPLE-RULE publish> #<SIMPLE-RULE deploy-ssh> #<SIMPLE-RULE deploy-cloudflare> #<SIMPLE-RULE deploy deploy-css> #<SIMPLE-RULE upstream> #<SIMPLE-RULE dist-repos> #<SIMPLE-RULE dist-source clean> #<SIMPLE-RULE push> #<SIMPLE-RULE dist-html>
Unlike makefile rules we don't need to specify a target (although
one may be provided for build caching).
Each rule is associated with a function, or more precisely a closure
with bindings determined by the bind slot of a project:
skel show bind
(SUBMODS (LIST graph meta plan docs notes archive))
The RULE class is defined in the PROJECT protocol:
(let ((class (find-class 'project:rule)))
(values
(make-instance class)
(mapcar 'name (class-slots class))))
#<RULE 0> (ID AST)
5.5.1. Simple Rules
(let ((class (find-class 'project:simple-rule)))
(values
(make-instance class)
(mapcar 'name (class-slots class))))
#<SIMPLE-RULE NIL> (ID AST SOURCE OBJ/PROJECT::TARGET)
5.5.2. Interactive Rules
(let ((class (find-class 'project:interactive-rule)))
(values
(make-instance class)
(mapcar 'name (class-slots class))))
#<INTERACTIVE-RULE 0> (ID AST INTERACTIVE OBJ/PROJECT::ARGS)
Interactive rules implement the COMMAND protocol and accept a
lambda-list and INTERACTIVE declaration in the body. They are
commonly used as project-specific subcommands.
5.6. Apps
The skel system is further extended by built-in applications which
specialize on skel objects in project domains of particular interest.
There are currently 4 app entry points in addition to skel itself:
- krypt
- Secrets Management
- packy
- Package Management
- mpk
- Media Management
- homer
- Home Management
- TODO Krypt
logbook
- State "TODO" from
- WIP Packy
packyis a package management tool which is based on thepkgfileclass. Our pkgfiles are read from special lisp files similar toskelfiles.Packages may be installed by compiling these objects into PKGBUILDs which are consumed by
makepkgon Arch Linux systems.The packy tool will eventually replace pacman functionality but this is still a ways off and not a high priority.
- Package Format
- tar.zst
- component-specific formats
- Package Index
- packy.compiler.company
- eventually direct support for lang-specific package managers - cargo/pypi/gobuild/elpa/etc.
- Package Format
- TODO MPK
logbook
- State "TODO" from
- TODO Homer
logbook
- State "TODO" from
6. Hacking
Lisp is by nature a hacking-friendly language, and every effort is
made to preserve those properties throughout the core. As a result it
is easy to change any symbol and modify any place in a running core
system, although the consequences may range from unexpected to
catastrophic if attempts are made to use or modify
internals. Especially when dealing with alien objects it's just as
easy to corrupt your Lisp image as it is to modify it.
When hacking on the source refer to the module docs and take note of the inline file comments. For work-in-progress modules take note of the core project tasks.
Footnotes:
Some Lisp applications developers do actually prefer this - taking GNU/Linux and the UNIX philosophy as a constant in any self-respecting OS.