How `lib` is composed

A caisson flake’s lib is canonically built by a call to caisson-core.mkLib. This page traces what happens between that call and the finished attrset: what goes in, the order things apply, the rules that decide conflicts, and how to drive the same machinery without mkLib when you want to.

What goes in

mkLib is given a set of declarations, and nothing to compose over: the seed is the empty attribute set, and nixpkgs’ library arrives as an entry like everything else.

defaultEcosystemSrc: a function of the tree’s sources, returning the tree’s default source per ecosystem key (see Ecosystem sources). An attribute set given in place of the function is refused. A tree declares nixpkgs-lib under this key for a mirror such as nix-community/nixpkgs.lib, for a flake that carries no nixpkgs, or for a nixpkgs checkout pinned under another input name. The nixpkgs-lib integration of caisson, at lib-overlays/nixpkgs-lib in the caisson source tree, is the entry that imports the lib directory of the source resolved under the nixpkgs-lib key, with the fixed point upstream computes. It loads the source declared by the flake whose mkLib call composes it, including when that flake consumes caisson as a project. The caisson integrations import the entry. An overlay outside caisson that calls upstream’s functions imports it by key from the caisson source tree: (mkLibOverlay (closure-inputs.caisson.outPath + "/lib-overlays/nixpkgs-lib")) // { key = "nixpkgs-lib"; }. A flake that declares no source fails only where that entry is composed, with a message naming the declaration.
libOverlays: the flake’s overlay registrations, built with lib.caisson-core.mkLibOverlay of the core lib the function receives, or registered directly when already built (another flake’s export, for example).
modules: the flake’s class-keyed module registrations.
projects: whole upstream contributions, each carrying exported overlays and modules that register under <project>/<name>.

The registry also holds the entries caisson-core publishes, caisson-core (the machinery) and nixpkgs-lib, under those names. Neither is selectable, and a registration under either name replaces the entry for every overlay that imports it, so a polyfill over upstream’s library or a replacement of the core is an ordinary registration.

Registration and application are separate steps: libOverlayImports selects which of the registered overlays apply to this flake’s lib (default: all of them), and registration also feeds export, so a flake can register overlays for downstream consumers that it does not apply to itself.

The stages

mkLib builds the library in stages, each a new fixpoint over the seed, because some of its arguments are functions of a library and can only see what is built before them:

The core lib is the caisson-core entries with the lib overlay registry recorded on its manifest. libOverlayImports receives it and picks entries from that registry.
The bootstrap lib is the core entries plus the selection, the sequence below without the module registrations. modules and configs receive it, so they can use every integration’s constructors, and its manifest lacks what they register.
The registered lib is the whole sequence below, with the module, configuration and package overlay registries on its manifest. pkgSets receives it, because a package config selects from those registries, and its manifest lacks pkgSets.
The full lib adds the package configs to the manifest, and mkLib returns it.

Each stage records a manifest at lib.caisson-core.libManifest, and the history in each manifest begins with the history of the stage before it. Entries registered at the earlier stages still close over the full lib, so a module reaches its siblings through closure-lib.caisson.modules, which reads the registry from the manifest of that lib.

The sequence

The selected overlays are applied in one pass, wrapped by entries caisson adds itself:

{ }
  -> caisson-core namespace injection (the `caisson-core` entry)
  -> selected overlays (flattened, imports first; the `nixpkgs-lib`
     entry wherever an overlay imports it)
  -> consumed projects' modules
  -> the flake's module registrations
  -> the manifest

The first added overlay injects the machinery and the empty module registry under caisson-core. The last records the manifest, the composition’s self-description, at lib.caisson-core.libManifest (beside pkgsManifest and evalManifest, the manifests of the other evaluation phases, null until a package set or a module evaluation fills them in). Module registrations apply after every selected overlay so that a local name always beats a same-named contribution from an overlay or a consumed project.

Before application, each selected overlay is flattened: a built overlay is an { imports, overlay } value, and the flattening walks depth-first with imports before the overlay itself. This guarantees that anything an overlay depends on has already applied when its overlay function runs, regardless of registration order. Within mkLib, imports guarantee order, not uniqueness: an overlay that two registrations both import is applied once per appearance, which is harmless for overlays that follow the merge conventions.

The fold

Application is the standard Nix overlay contract, folded over the sequence above. Each overlay = final: prev: { ... } receives prev, everything accumulated so far, and final, the finished fixpoint. For an attribute defined by two overlays, the later definition wins, and prev gives it the earlier definition to build on, which is what the my-flake = (prev.my-flake or { }) // { ... } merge convention relies on.

These consequences of the fixpoint are worth knowing:

An overlay’s output attribute names must not depend on final; a fixpoint whose shape depends on itself diverges.
The base library is contributed as an opaque value. Overriding one of its attributes changes what readers of the composed library see, and does not change what the base’s internals call; a function patched for everything downstream of the base has to be patched in the base source you pass in.

Composing without mkLib

mkLib is a convenience over caisson-core.compose, which works on entries, overlay-shaped pieces with identity:

{
  key = "example.base";   # stable identity: a string, or null
  imports = [ ];          # entries this entry depends on
  overlay = final: prev: { greet = name: "hello, ${name}"; };
}

Keys change the rules. A keyed entry applies once no matter how many entries import it; the first occurrence of a key fixes its position and the last occurrence supplies its value, so mentioning a key again replaces that entry wholesale; replacement is the only override mechanism. An entry with key = null cannot be imported, applies after the whole keyed world in list order, and can never be replaced by another entry, because replacement addresses keys and it has none; keyless entries are a consumer’s private patch layer. Import cycles terminate (a key already on the walk’s path is skipped) and grant the cycle’s members no ordering relative to each other.

Identity is what makes patching a dependency reliable. A polyfill imports the entry it patches, which guarantees the target is present and already applied when the polyfill reads prev:

polyfill = {
  key = "example.backport";
  imports = [ base ];
  overlay = final: prev: {
    concatLines = prev.concatLines or (lines: prev.concatStringsSep "\n" lines + "\n");
  };
};

The prev.concatLines or ... shape adds the function only where the base does not already provide it, so the same entry composes correctly over old and new bases.

The exact contract (walk order, the position a replacement takes, metadata) is specified in caisson-core, where the code lives.

caisson exports its contributions in entry form through lib.composition.entriesFor:

caisson.lib.composition.entriesFor {
  # a directory importable as nixpkgs' lib, e.g. "${nixpkgs}/lib"
  # or "${nixpkgs-lib}/lib" for the nixpkgs.lib mirror
  ecosystemSrc = "${inputs.nixpkgs-lib}/lib";
}
# => { base, caisson-lib,
#      flake-parts, tooling,
#      nixpkgs, nixos, nixos-minimal, home-manager,
#      colmena, terranix, system-manager }

base contributes the nixpkgs library the composition builds on; caisson-lib imports it and contributes the caisson-core machinery, the same injection mkLib performs; the rest are the integrations and tooling, each importing caisson-lib.

Where the composed `lib` goes

lib.caisson.flake-parts.mkTopConfiguration passes the composed library to flake-parts as specialArgs.lib, so modules receive it as their ordinary lib argument.

The lib output of the flake is the composed library. By convention, the lib overlays of a flake named my-flake define their values under my-flake, and the lib overlays of other flakes do not define values there (see Library overlays). The values of my-flake are in the output at lib.my-flake.

The manifest of the flake is in the output too, at lib.caisson-core.evalManifest. The manifest is published in lib because the other flake outputs each have a fixed shape. The manifest holds what the flake exports, as outputs.exports: the lib overlays, package overlays and modules that the exported selectors of the flake chose.

producer is a flake that exports a lib overlay. consumer is a flake that lists producer in projects.

consumer reads what producer exports from the manifest of producer, and ignores the flake outputs of producer. consumer adds the lib overlay to the library that consumer composes.

The lib overlay was built with mkLibOverlay, in the flake whose source holds the overlay. mkLibOverlay gave the overlay closure-inputs and closure-lib: the pinned sources and the composed library of the flake that called mkLibOverlay. The overlay keeps both in the library of consumer.

By default, producer exports the lib overlays that producer registers itself. For those, the flake that called mkLibOverlay is producer.

producer can also list a flake upstream in projects, and name a lib overlay of upstream in the exported selector of producer. For that lib overlay, the flake that called mkLibOverlay is upstream. In the library of consumer, the overlay keeps the pinned sources and the composed library of upstream.

Inspecting the result

nix repl .
# :p lib.my-flake                     -- your namespaced additions
# :p lib.my-flake.helper 41           -- call a function

nix eval .#lib --apply builtins.attrNames

Keyboard shortcuts

caisson