Core User Manual

This manual documents the CC core software suite. It is intended for users/developers and assumes a intermediate level understanding of computer programming concepts such as OOP, FP, IDEs, and the shell.

The core is a Lisp development system used to build next-gen applications. It is composed of interconnected modules providing an exceptionally powerful development ecosystem.

1. Introduction

Welcome to the core, the primary software development ecosystem of CC.

The core provides the foundation of all applications we produce. It is optimized for rapid development of next-gen applications.

The core is highly opinionated - and these opinions are consistently hard-coded into our development practices.

For example:

  • Linux is the only platform worth supporting
    • We make no efforts to support cross-compilation or porting of our software to devices or operating systems the developer can't fully control such as Windows/MacOS/iOS/Android. We always assume the developer is running a GNU/Linux host.
  • Portability is over-rated
    • CL implementations (lisps) share a base language but many parts of the ANSI CL spec are left up to interpretation: threads, OS, GC, MOP, etc
    • Many lisp packages attempt to account for as many lisps as possible via conditional compilation and wrappers - this clutters up the code base, is difficult to test, and is error-prone.
    • Instead of accounting for multiple compilers, we realize that we only need one - SBCL currently. The core does not contain any support for alternate lisps.
    • 'One true lisp' is all we need, and we're only concerned with internal consistency - we are never bound by the choices in implementation made by other lisps (although there is much to learn from them).
  • Introspection is under-rated
    • Black boxes (proprietary codes) are disrespectful to developers. Computer systems should be transparent by nature and make minimal efforts to 'hide' internal representations of data and code.
    • Never hide a part of the system, unless you have a very good reason for doing so.
  • Follow the Bleeding Edge
    • We pull and rebuild the latest changes for all of our dependencies frequently
    • We host mirrors of all our major dependencies
    • Core development is always based on latest changes, can optionally downgrade for release builds where stability is valued
    • incorporate pre-release APIs and our own custom patches as needed.

2. Overview

The core itself is not an application but is complex enough that it merits its own dedicated manual. There is library-specific and API documentation available for the core, so what we are concerned with here is the overall design, common patterns, and architecture of the core environment.

This manual is often technical, but occassionally theoretical in nature and intended for software developers with expertise in Lisp, GNU/Linux, and Emacs.

The Core is a Lisp system. Common Lisp is our language of choice and there is a decent amount of Emacs Lisp code too.

2.1. Source Code

ls -Ah
api.org
info.org
man.org
readme.org
  • Common Lisp
    The STD library lives under the std directory, low-level FFI library bindings such as ROCKSDB under the alien directory and high-level user libraries like RDB under the lib directory. The skel directory contains the source of our multi-purpose project tool SKEL.
  • Emacs Lisp
    The etc/emacs directory contains a complete Emacs configuration.

2.2. Images

Images refer to the binary file resulting from a call to SAVE-LISP-AND-DIE. According to SBCL this file contains:

enough information to restart a Lisp process later in the same state, in the file of the specified name.

See the function docs for details - there are many options available which determine how the image can be used directly or dynamically from other programs.

The core images we produce are executables which contain the full SBCL runtime at their disposal. Note that these images can be quite large - multiple gigabytes in some cases, and so care should be taken when enabling additional build-time features.

Note that image-based development is quite unique. You start inside your Lisp implementation's core image which contains the Lisp runtime in its executable. The runtime in SBCL is compiled C code which you may be included with the images you save to make them executable, or you can remove it to save space. Restarting an image in the exact same state you saved it in is as simple as executing sbcl with the --core argument or executing the saved core file (with runtime) directly.

The development experience is exceptionally fluid when fully appreciated. You have everything at your fingertips, always fully compiled and ready to load into memory at a moment's notice.

Images are concatenative - they always build on an existing image. We start with the ANSI LISP 'kernel' image, then we save it, extend it, and repeat indefinitely. The natural evolution of this concept is to have your entire system in a Lisp image. It is a 'maximalist' philosophy which is deeply ingrained into the very idea of Lisp, although ultimately relegated to the long-forgotten Lisp Machines.

Image-based development does come with its own set of foot guns - it is more complex by nature and easy to end up in an unusable state when you compile in faulty code. However the biggest conflict that arises from images is that they are anti-UNIX.

GNU/Linux distributions today come with hundreds of binaries. The vast majority of these are single-use tools with simple text interfaces which aide themselves to composability via the shell. Lisp images are not single-use tools - they always contain a complete Lisp runtime in addition to any extensions you've added - so if you want to use them like UNIX binaries with one image per application or tool, you're duplicating a tremendous amount of code in each image1.

One of the goals of the core is to realize a full appreciation of images and image-based development, starting from GNU/Linux Userspace and working our way towards service management and then the kernel. The general strategy is to reduce the OS environment while extending our Lisp image. We aim to first serve as a default user login shell, and ultimately challenge SystemD for PID1. If we are successful, we will have a complete Lisp system compatible with the Linux kernel.

3. Getting Started

3.1. Dependencies

To ensure dependencies on an existing system you should refer to the infra manual.

A PKGBUILD recipe is planned but not ready AO . Note that RocksDB should be installed via the infra project (the default extra/rocksdb package is not compatible).

3.1.1. Required Dependencies

The following dependencies are required - Arch Linux is assumed but other distros should work.

  • linux-headers
  • gcc
  • sbcl
  • emacs
  • rocksdb (built WITHOUT jemalloc)
    • gflags
    • intel-oneapi-tbb
  • zstd
  • mercurial
  • openssl

3.1.2. Optional Dependencies

Some of these dependencies may already be installed on your system, others will need to be installed manually or via infra.

  • cuda
  • evdev
  • jack
  • libjpeg-turbo
  • xkbcommon
  • keyutils
  • sndfile
  • libssh2
  • btrfsutils
  • gstreamer
  • tree-sitter
  • blake3
  • liburing
  • alsa
  • ffmpeg
  • chromaprint
  • openblas (blas/lapack)

3.2. Bootstrap

Bootstrapping from SBCL is quite easy. Once all required dependencies are installed, just run bootstrap.lisp:

# clone the source code from a mirror
git clone https://codeberg.org/c-c/core
cd core
sbcl --script bootstrap.lisp
sudo .stash/core --script etc/script/install.lisp

Release tarballs are planned but not available AO .

3.3. Configuration

The next thing you'll want to take care of is user configuration. You can save your configuration at your preferred XDG-compliant location (~/.config/corerc is recommended).

;;; .corerc --- core init file -*- mode: common-lisp; -*-
(in-package :user)
(init :xdg) ;; apply user XDG config
(init :term) ;; detect current terminal
(init :log :level :debug) ;; enable global logger
;; (init :sys :sysdefs (sysdefs path-to-core t))

The only other development dependency which needs configuring is Emacs. This can be done via the skel tool, but keep in mind that this will overwrite the XDG-compliant emacs configuration directory (~/.config/emacs) if it exists:

skel make install-emacs
#:debug 0.157
; (#<rule install-emacs>)

3.3.1. Core

The core is a single binary which dispatches on argv[0] from the CLI. Each unique symlink to the core binary is treated as a separate dispatch to a 'main' function like so:

(define-multi-main dispatch-core
  (progn (in-package :core-lisp)
         #+nil ...)
  (:skel (bin/skel::start-skel))
  (:homer (bin/homer::start-homer))
  (:mpk (bin/mpk::start-mpk))
  #+nil ...)

The default path (when argv[0]=core) accepts a lisp file as configuration just like ~/.sbclrc, corerc respected the XDG_CONFIG_HOME variable though and may be located in ~/.config/corerc.

The current apps that read a configuration file on startup are:

  • skel - project management: skelrc
  • homer - home management: homerc
  • mpk - media production kit: mpkrc

By default they are located under XDG_CONFIG_HOME.

These apps all use the same configuration object protocol from the OBJ/CONFIG package and are printed to various S-Expression. Most commonly the 'pretty' format is used which is simply a plist with implied outer parentheses:

:key1 val1
:key2 val2

Canonically that would be:

(:key1 val1
 :key2 val2)

3.3.2. Emacs

If you don't already have your own Emacs config you can install the default CC Emacs configuration using the skel command.

  • AO the core should is intended to be hacked on using this config only but is subject to change in the future.
  • this will overwrite $XDG_CONFIG_HOME/emacs/

    skel make install-emacs
    
    sudo: a terminal is required to read the password; either use the -S option to read from standard input or configure an askpass helper
    sudo: a password is required
    sudo: a terminal is required to read the password; either use the -S option to read from standard input or configure an askpass helper
    sudo: a password is required
    sudo: a terminal is required to read the password; either use the -S option to read from standard input or configure an askpass helper
    sudo: a password is required
    #:debug 6.870001
    ; (#<SIMPLE-RULE install-emacs>)
    
  • the vanilla load sequence is applied first, starting with early-init.el if provided by the user then init.el
  • init.el loads default.el which initializes all required MELPA packages and local libraries from etc/emacs/.
  • the site config.el is loaded next (optional, user controlled)
  • additionally load user-custom-file if it exists, defaulting to the name of the current user (foo.el when $USER=foo)
  • finally load the custom-file custom.el if it exists
  1. Defaults

    Our Emacs configuration philosophy follows the same principles found elsewhere in the core - prioritizing built-ins/internals, and first-class support for modern Lisp development.

3.4. Demos

Core demo applications, examples, and tutorials can be found in the demo project with documentation available online.

3.5. Documentation

AO full-auto API docs are planned but not available. They will be located under the api directory.

lib docs will be move here

3.6. Testing

Each module in the core contains its own test suite located in the tests.lisp file or tests directory and isolated to its own system. The system is always the module name with a suffix of /tests - for example the IO/TESTS system is responsible for testing the IO module system.

3.6.1. Regression Tests

Testing the core can be done from the CLI using skel:

# skel compile tests
skel run tests

or from lisp:

(test-system :core)

3.7. Contributing

The core is a solo experiment and I don't plan on accepting contributions in the near future. The core is its own private island, isolated from the rest of the Common Lisp ecosystem and in almost all cases contributions would be better directed towards other packages.

See Awesome CL for a list of projects which could use additional support, and take note the comments scattered throughout the core source code which often reference popular packages from the ecosystem.

4. Concepts

4.1. Systems

4.2. Logical Hosts

Logical Pathnames are used sparingly in the core source but their use is encouraged by users. Several logical hosts are pre-defined in addition to the SBCL SYS host.

(list-all-logical-host-names)
ETC MEDIA MPK PACKY SKEL SRV SYS USER USR VAR

4.3. Reader

4.4. Printer

4.4.1. Annotations

4.5. CLOS

4.5.1. Metaclasses

4.5.2. Templates

4.7. Macros

4.7.1. DSLs

4.7.2. Memoization

4.8. Protocols

Protocols are keyed groupings of symbols, sort of like light-weight packages.

(fmt-tree nil (find-module :std :proto :kernel) :plist t :layout :down)
:KERNEL
 ├─ :PACKAGE
 │   ╰─ :STD/PRIM
 ├─ :DESCRIPTION
 │   ╰─ "Standard funcallable object extensions."
 ├─ :VARIABLES
 │   ╰─ (*KERNEL*)
 ├─ :CONDITIONS
 │   ╰─ (KERNEL-INIT-ERROR NO-KERNEL-ERROR)
 ├─ :ACCESSORS
 │   ╰─ (KERNEL KERNEL-INFO KERNEL-DOCUMENTATION)
 ├─ :METHODS
 │   ╰─ (KERNEL-EXPRESSION)
 ├─ :FUNCTIONS
 │   ╰─ (CHECK-KERNEL MAKE-KERNEL)
 ├─ :MACROS
 │   ╰─ (DEFKERNEL)
 ├─ :PREDICATES
 │   ╰─ (KERNELP)
 ╰─ :CLASSES
     ╰─ (KERNEL-CLASS KERNEL-OBJECT)

Protocols are designated by a key with an associated plist (the cdr of (find-module ... :proto ...)). The list of available plist keys in a protocol are identified by a variable:

std/defsys::*protocol-keyword-imports*
  • :METHODS
  • :FUNCTIONS
  • :TYPES
  • :VARIABLES
  • :CONSTANTS
  • :PARAMETERS
  • :MACROS
  • :CONDITIONS
  • :RESTARTS
  • :ACCESSORS
  • :PREDICATES
  • :CLASSES
  • :STRUCTS
  • :DECLARATIONS
  • :GLOBALS

5. Skel

skel help
skel v0.1.1:4da8a49ea455+ --- A universal project development tool.
usage: skel [opts] <command> [<arg>]

The skel tool is an extensible program designed around the idea of projects.

5.1. Core

The skel/core module contains all internal components of the skel system. Despite the name, much of this code is glue code which stitches together multiple library protocols.

Most important data structures are represented with classes, slots, and methods such as the SK-PROJECT class which is based on the OBJ:PROJECT class (a generic project type).

(mapcar 'name (components (find-component :core (find-system :skel))))
  • condition
  • header
  • var
  • obj
  • component
  • project
  • print
  • schema
  • db
  • log
  • util

5.2. Configuration

The skel tool can be configured at runtime using the CLI flags, or via local configuration files. The configuration files are in the skelfile format and are assigned the following locations by default:

*system-skelrc*
/etc/skelrc
*user-skelrc*
$XDG_CONFIG_HOME/skelrc

Configurations are applied to the same object in sequence - starting with the system configuration and then the user config. Both are optional.

Example ~/.config/skelrc:

cat ~/.config/skelrc
;;; skelrc
:name "Richard Westhaver"
:email "richard.westhaver@gmail.com"
:version "0.1.0"
:vc :hg
:logger (:level :info)
:stash ".stash"
:store "/opt/store"
:scripts ("~/.stash/scripts")

5.3. Projects

Projects are the root objects of the skel system. A project consists of a list of slots which are populated automatically or by the user. Once a project is initialized the slots can be accessed and a set of high-level operations can be performed.

The easiest way to initialize a project is with a skelfile. These files are parsed as plists and generate sk-project objects based on their contents.

The simplest project looks like this:

:name hello-world

Don't worry, this is still lisp - the parentheses are implied. All we actually need is a name to generate an object, and we can fill in the blanks later as the project develops.

Here is a more verbose example, still with only metadata:

;;; skelfile --- core skelfile -*- mode: skel; -*-
:name "core"
:author "Richard Westhaver <ellis@rwest.io>"
:version "0.1.0"
:license "MPL"
:description "The Compiler Company Core"
:vc :hg
:tags ("core")
:docs ((:org "readme") (:org "install") (:org "tests") (:org "todo"))
:include ("emacs/emacs.sk")

Another example:

;;; skelfile @ 2023-10-08.02:37:25 -*- mode: skel; -*-
:name skel
:author "ellis"
:version "0.1.0"
:description "a hacker's project compiler"
:license "MPL"
:vc :hg
:tags ("lisp")
:bind ((target :skel/cli))
:rules ((build () (print (asdf:make target)))
        (clean () #$rm -rf */*.fasl$#))
:docs ((:org "readme"))
:components
((:elisp "sk"))
:stash "~/.stash"

The currently accessible sk-project can be easily inferred via the init protocol:

(init :skel) ;; find the top-level skel project
(describe *project*)
#<SKEL-PROJECT ORG :components 1 :rules 19>
  [standard-object]

Slots with :INSTANCE allocation:
  NAME                           = ORG
  HOOK                           = #<unbound slot>
  PROVIDE                        = #<unbound slot>
  REQUIRE                        = #<unbound slot>
  PATH                           = #P"/home/ellis/src/org/skelfile"
  AUTHOR                         = ("Richard Westhaver" . "ellis@rwest.io")
  VERSION                        = "0.1.0"
  TAGS                           = (ORG DOCS NOTES META PLAN ARCHIVE GRAPH)
  LINKS                          = #<unbound slot>
  DESCRIPTION                    = "CC Org Files"
  LICENSE                        = "MPL"
  AST                            = NIL
  ID                             = 1784914079056115652
  VC                             = #<HG-REPO {1203CA8FE3}>
  SRC                            = #P"/home/ellis/src/org/"
  STASH                          = #P"/home/ellis/src/org/.stash/"
  STORE                          = #P"/home/ellis/src/org/.stash/store/"
  CACHE                          = #P"/home/ellis/src/org/.stash/cache/"
  COMPONENTS                     = #(#<EMACS-LISP-FILE :ID 1091-ba75-0b6c-698b>)
  BIND                           = ((SUBMODS (LIST "graph" "meta" "plan" "docs" "notes" "archive")))
  RULES                          = #(#<SIMPLE-RULE default build publish deploy dist-html dist-source dis..

5.4. Components

In addition to the core objects we provide a set of COMPONENT definitions which convert core objects to a corresponding external representation. For example the MAKEFILE class defined in comp/makefile.lisp can be built from or consumed by a project.

(mapcar 'name (components (find-component :comp (find-system :skel))))
  • lisp
  • make
  • rust
  • python
  • emacs
  • shell
  • box
  • pod
  • infer

5.5. Rules

Skel rules are the Lisp equivalent of Makefile rules. They are almost always executed at the CLI via the make command.

Rules are a project slot so you can list them with show:

skel show rules
#<SIMPLE-RULE default build publish deploy dist-html dist-source dist-repos>
#<SIMPLE-RULE build build-css update-docs build-graph build-docs build-agenda>
#<SIMPLE-RULE build-graph>
#<SIMPLE-RULE build-agenda>
#<SIMPLE-RULE update-docs>
#<SIMPLE-RULE build-docs>
#<SIMPLE-RULE clean>
#<SIMPLE-RULE build-css>
#<SIMPLE-RULE deploy-css>
#<SIMPLE-RULE browse>
#<SIMPLE-RULE publish>
#<SIMPLE-RULE deploy-ssh>
#<SIMPLE-RULE deploy-cloudflare>
#<SIMPLE-RULE deploy deploy-css>
#<SIMPLE-RULE upstream>
#<SIMPLE-RULE dist-repos>
#<SIMPLE-RULE dist-source clean>
#<SIMPLE-RULE push>
#<SIMPLE-RULE dist-html>

Unlike makefile rules we don't need to specify a target (although one may be provided for build caching).

Each rule is associated with a function, or more precisely a closure with bindings determined by the bind slot of a project:

skel show bind
(SUBMODS (LIST graph meta plan docs notes archive))

The RULE class is defined in the PROJECT protocol:

(let ((class (find-class 'project:rule)))
  (values 
   (make-instance class)
   (mapcar 'name (class-slots class))))
#<RULE 0>
(ID AST)

5.5.1. Simple Rules

(let ((class (find-class 'project:simple-rule)))
  (values 
   (make-instance class)
   (mapcar 'name (class-slots class))))
#<SIMPLE-RULE NIL>
(ID AST SOURCE OBJ/PROJECT::TARGET)

5.5.2. Interactive Rules

(let ((class (find-class 'project:interactive-rule)))
  (values 
   (make-instance class)
   (mapcar 'name (class-slots class))))
#<INTERACTIVE-RULE 0>
(ID AST INTERACTIVE OBJ/PROJECT::ARGS)

Interactive rules implement the COMMAND protocol and accept a lambda-list and INTERACTIVE declaration in the body. They are commonly used as project-specific subcommands.

5.6. Apps

The skel system is further extended by built-in applications which specialize on skel objects in project domains of particular interest.

There are currently 4 app entry points in addition to skel itself:

krypt
Secrets Management
packy
Package Management
mpk
Media Management
homer
Home Management
  1. TODO Krypt
    logbook
    • State "TODO" from
  2. WIP Packy

    packy is a package management tool which is based on the pkgfile class. Our pkgfiles are read from special lisp files similar to skelfiles.

    Packages may be installed by compiling these objects into PKGBUILDs which are consumed by makepkg on Arch Linux systems.

    The packy tool will eventually replace pacman functionality but this is still a ways off and not a high priority.

    1. Package Format
      • tar.zst
      • component-specific formats
    2. Package Index
      • packy.compiler.company
      • eventually direct support for lang-specific package managers - cargo/pypi/gobuild/elpa/etc.
  3. TODO MPK
    logbook
    • State "TODO" from
  4. TODO Homer
    logbook
    • State "TODO" from

6. Hacking

Lisp is by nature a hacking-friendly language, and every effort is made to preserve those properties throughout the core. As a result it is easy to change any symbol and modify any place in a running core system, although the consequences may range from unexpected to catastrophic if attempts are made to use or modify internals. Especially when dealing with alien objects it's just as easy to corrupt your Lisp image as it is to modify it.

When hacking on the source refer to the module docs and take note of the inline file comments. For work-in-progress modules take note of the core project tasks.

Footnotes:

1

Some Lisp applications developers do actually prefer this - taking GNU/Linux and the UNIX philosophy as a constant in any self-respecting OS.