Testing

The conventions and tools for testing a caisson flake, as used by the test suite of this repository (tests/ here is a worked example of all of it).

Unit tests

tests/unit/ holds pure evaluation tests, wired into checks; this repository runs them with nix-unit. Anything that can be asserted by evaluating lib belongs here.

When nix-unit runs as a flake check, it runs inside a Nix build. The check is a derivation. The builder of that derivation runs nix-unit, and nix-unit evaluates the flake that holds the tests.

Nix runs every builder in a sandbox. Inside the sandbox, the builder can see a path in the Nix store only when the derivation names that path as a dependency, and the builder has no network access. So Nix cannot find a flake input that the tests need, and it cannot download the input either. The check fails with the message unable to download.

The option nix-unit.inputs prevents that. It is an attribute set of input names and store paths. nix-unit reads each input it lists from the store path given, and the builder can see those store paths. Every flake input that the tests read has to be in the set.

A flake that the tests compose as a project has inputs too. Those inputs also have to be in the set, under names such as caisson/caisson-core. lib.caisson.realizeInputs builds the whole set from the inputs of the test flake:

perSystem.nix-unit.inputs = lib.caisson.realizeInputs inputs;

With this, the test flake does not need follows to point the inputs of its inputs back at inputs it already lists. The reference entry says when to give the function fewer inputs.

Integration tests: consumption from the outside

The strongest test of a flake framework is what a consumer experiences, so integration tests are nested flakes under tests/integration/ that take the project as an input and assert that composition behaves as documented. They are written as completely normal flakes:

{
  inputs = {
    # Standalone equivalent (without shared deps infrastructure):
    #   nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
    #   flake-parts.url = "github:hercules-ci/flake-parts";
    #   parent.url = "github:example/my-flake";

    deps.url = "path:../../dependencies";
    parent.url = "path:../../..";

    nixpkgs.follows = "deps/nixpkgs";
    flake-parts.follows = "deps/flake-parts";
  };

  outputs = inputs@{ parent, ... }: {
    # consume `parent` the way any downstream flake would
  };
}

tests/dependencies

A small lock-bearing flake whose only job is to pin the inputs that tests (and formatters) need, so the main flake.lock stays free of test-only pins. The checks partition takes its inputs as (lib.caisson.pins.flake-compat ./tests/dependencies).sources, the flake-compat pin reader, which resolves the lock without evaluating the flake.

Evaluating the nested flakes

Hand-threading a nested flake’s inputs (the recursive self fixpoint, the resolved input graph) is the genuinely hard part, and callConsumerFlake owns it. In the parent’s checks:

let
  consumerPool = {
    inherit (inputs) flake-parts nixpkgs;
    deps = inputs.self;      # the tests/dependencies flake
    parent = self;           # the flake under test
  };

  minimalConsumer = lib.my-flake.callConsumerFlake {
    path = self.outPath + "/tests/integration/minimal-consumer";
    pool = consumerPool;
  };
in
{
  checks = minimalConsumer.checks.${system};
}

The nested flake’s declared inputs resolve by name (overrides first, then follows chains, then the pool), and an unresolvable input throws an error naming it and what to do. The contract is deliberately explicit, in the same spirit as closed inputs: nothing is fetched, no lock is read, and you supply exactly the input graph you mean the test to see. One consumer’s outputs can feed another’s overrides, so chains of consumers (a flake consuming a flake that consumes yours) are plain data flow.

See callConsumerFlake for the full signature.

Gating evaluation cost

Beyond correctness, checks can gate what evaluation costs; see Evaluation weight.