Development with Nix: Python

Posted on May 23, 2020

This is a tutorial for people working on a Python project that uses Nix or interested in using Nix to manage Python packages. For a more detailed introduction to Nix, I recommend Nix Pills.

Python is particularly messy when it comes to package management: You have pip installing packages on the system Python installation, and you have tools like virtualenv creating “virtual environments” to isolate sets of packages. Finally, you may use a version manager (e.g. pyenv) to handle different version of the language.

Nix takes a different approach by “flattening” the multiple levels of package management. With Nix, you declare “I need Python 3.8 with pandas and statsmodels” in a file containing an expression1 that produces a derivation2. Nix takes the derivation expression and builds the final Python package containing all dependencies.

Go to https://nixos.org/nix, click on “Get Nix”, and follow the instructions to install Nix.

The Nix Shell

The nix-shell command uses a Nix expression to start a fully provisioned shell environment. These shell environments do not depend or affect the external operating system environment which makes them ideal for development.

You use the pkgs.mkShell function to create the shell environment. First, create a new shell.nix file with the following contents:

{ pkgs ? import (fetchTarball https://github.com/nixos/nixpkgs/archive/nixpkgs-unstable.tar.gz) {} }:

pkgs.mkShell {
  buildInputs = [ pkgs.python38 ];
}

The above expression is a function expression that takes a pkgs3 parameter with a default value set to the Nix package collection at GitHub4. The pkgs collection also include the “standard library” of Nix functions that you use to build packages. It then calls pkgs.mkShell with the buildInputs attribute set to a list containing the Python 3.8 package from the package collection.

If Nix used Python instead of its own language, your shell expression would look something like this:

import nix

def shell(pkgs=None):
    if not pkgs:
        pkgs = nix.import(nix.fetchTarball("https://github.com/nixos/nixpkgs/archive/nixpkgs-unstable.tar.gz"))

    nix.mkShell({"buildInputs": [pkgs.python38]})

Anyways, you can now run nix-shell from the same directory of the shell.nix file to start a shell with Python:

[~/foo]$ nix-shell
unpacking 'https://github.com/nixos/nixpkgs/archive/nixpkgs-unstable.tar.gz'...
...

[nix-shell:~/foo]$ which python
/nix/store/kvr9hsw3fp5ba9zcp9ga814xmc6s87wm-python3-3.8.1/bin/python

[nix-shell:~/foo]$ python --version
Python 3.8.1

If you want to use a different value for pkgs, you can pass it from the command line with the --argstr option:

[~/foo]$ nix-shell --argstr pkgs "import (fetchTarball https://example.com/my/custom/pkgs.tar.gz) {}"

Now that you have a Nix shell with Python, you can start adding packages to it.

Adding Packages From The Collection

The Nix package collection contains pre-written derivations for the most important Python Packages. To query the collection, use nix-env:

[~/foo]$ nix-env -f 'https://git.io/Jf0cc' -qaP
...

The https://git.io/Jf0cc URL redirects to the tarball URL of the nixpkgs-unstable branch of nixpkgs used above.

Querying the full package collection takes a bit of time. You are interested in Python packages only, and you can narrow down the search to the python38Packages set within the collection:

[~/foo]$ nix-env -f 'https://git.io/Jf0cc' -qaPA python38Packages
unpacking 'https://git.io/Jf0cc'...
python38Packages.addic7ed-cli                           addic7ed-cli-1.4.6
python38Packages.avahi                                  avahi-0.7
python38Packages.behave                                 behave-1.2.6
python38Packages.boost                                  boost-1.69.0
...

The right column shows the name attribute of each package (which is usually composed of the package name and version). You are interested in the left column that shows the attributes of the python38Packages set representing each package.

You could add packages from above directly to the buildInputs list, but that would install each package in isolate and they would not see each other, and what you want is a Python distribution with the package installed inside it.

To do that, use the Python buildEnv derivation to build a new Python and override its extraLibs attribute with the packages you want:

{ pkgs ? import (fetchTarball https://git.io/Jf0cc) {} }:

let
  customPython = pkgs.python38.buildEnv.override {
    extraLibs = [ pkgs.python38Packages.ipython ];
  };
in

pkgs.mkShell {
  buildInputs = [ customPython ];
}

You can check that Python and IPython are built in the same package by looking at the paths of their executables:

[~/foo]$ nix-shell
...

[nix-shell:~/foo]$ which python
/nix/store/rixay24bilcj65fk87zpwjn1v19lygc1-python3-3.8.2-env/bin/python

[nix-shell:~/foo]$ which ipython
/nix/store/rixay24bilcj65fk87zpwjn1v19lygc1-python3-3.8.2-env/bin/ipython

That hash-like prefix in the path is determined by the derivation inputs thus making the package purely functional. The Nix store serves as local package cache, and if you start the nix-shell a second time with the same derivation, you will notice that nothing gets built again because the hash matches a derivation/package already present in the store.

Adding Packages From Elsewhere

Not all Python packages are in the Nix package collection, and eventually you will have to create a derivation from scratch (e.g. a package from the PyPI archive or a GitHub repository). For that, you use the buildPythonPackage function inside the Python package set.

This is how you package grpclib in your Python environment:

{ pkgs ? import (fetchTarball https://git.io/Jf0cc) {} }:

let
  grpclib = pkgs.python38Packages.buildPythonPackage rec {
    pname = "grpclib";
    version = "0.3.1";

    src = pkgs.python38Packages.fetchPypi {
      inherit pname version;
    };

    propagatedBuildInputs = with pkgs.python38Packages; [ h2 multidict ];

    doCheck = false;
  };

  customPython = pkgs.python38.buildEnv.override {
    extraLibs = [ grpclib ];
  };
in

pkgs.mkShell {
  buildInputs = [ customPython ];
}

There is a bit of new syntax here. The rec keyword before the set passed to buildPythonPackage turns it into a recursive set so its attributes can reference each other. The with keyword before the propagatedBuildInputs list is just a syntax helper to avoid writing the full “pkgs.python38Packages.” prefix before each entry in the list.

In the derivation to a custom Python package, you set the src attribute to a fetchPypi call that pulls the source files from the PyPI archive, and the propagatedBuildInputs to its list of dependencies. The “propagated” prefix makes dependencies indirectly available through the dependency graph.

Finally, you set the doCheck attribute to false to indicate that you are not interested in running tests when building the package. To run them, you would need to add the corresponding test dependencies.

If you run the derivation above, you will get the following error:

[~/foo]$ nix-shell
...
error: fetchurl requires a hash for fixed-output derivation: mirror://pypi/g/grpclib/grpclib-0.3.1.tar.gz
(use '--show-trace' to show detailed location information)

A “fixed-output” derivation uses a hash of a file instead of an input derivation to determine if it has changed. This is useful when the URL of a source file/package changes (but not its contents) and you don’t have to rebuild every other package that depends on it.

Use nix-prefetch-url with the URL from the error message to download the file and get its hash:

[~/foo]$ nix-prefetch-url mirror://pypi/g/grpclib/grpclib-0.3.1.tar.gz
...
path is '/nix/store/ribsza3xgg8vj7l36l5z0g17r4mxp364-grpclib-0.3.1.tar.gz'
0dw7jzw3pf2ckzrl808ayqvk9yqjhc45rj8qhmdxifv4ynwnyjam

Now change the fetchPypi call to include the hash:

# ...

{
  # ...
  src = pkgs.python38Packages.fetchPypi {
    inherit pname version;
    sha256 = "0dw7jzw3pf2ckzrl808ayqvk9yqjhc45rj8qhmdxifv4ynwnyjam";
  };
  # ...
}

You should now be able to start the Nix shell with a Python installation that includes grpclib 0.3.1.

Using Helper Tools

Once you start adding more packages, writing derivations and updating hashes by hand becomes too much work. Fortunately auxiliary tools exist to help automate the process, and one of them is mach-nix.

mach-nix has a few different use cases, but one of them is creating a Python derivation with packages using the same requirements.txt syntax of Pip. Here is the same Python 3.8 derivation with grpclib from above but using mach-nix:

{ pkgs ? import (fetchTarball https://git.io/Jf0cc) {} }:

let
  mach-nix = import (
    builtins.fetchGit {
      url = "https://github.com/DavHau/mach-nix/";
      ref = "2.0.0";
    }
  );

  customPython = mach-nix.mkPython {
    python = pkgs.python38;
    requirements = ''
      grpclib==0.3.1
    '';
  };
in

pkgs.mkShell {
  buildInputs = [ customPython ];
}

Internally, mach-nix maintains a dependency graph of all dependencies in the PyPI archive together with their hashes. It then parses the requirements string and writes the right derivations for each package. You can find more details on the project repository.

You should now be able to provision any Python environment in a fully reproducible manner with Nix.


  1. An expression in Nix, just like in other programming languages, is an statement that produces a value. Most of the time, you will be writing expressions that somewhere call to the derivation function which produces a derivation (i.e. the build action). ↩︎

  2. The derivation is the “building block” of Nix. It is a bit of an overloaded term because it means the build specification of a package and also the name of the built-in function that produces the build specification. There is also the intermediate *.drv file containing the stripped-down representation of build specification. ↩︎

  3. The example uses pkgs as the parameter name, but it could be any other name. ↩︎

  4. The import call takes another derivation (here the output of fetchTarball) and a set of arguments to pass to it (here an empty set). ↩︎