Building Go With Nix Without Checking In a Hash

Intro #

If you build Go with Nix, there is a line like this somewhere in your repository:

vendorHash = "sha256-/Gsqc8rEptMBItqeb/N/gE4V3iUGZa8k1GqUR1+togY=";

It is one hash over all of your dependencies. buildGoModule wants it, and it is wrong every time a dependency changes.

What happens then depends on the machine. One that has never built the project downloads the dependencies, notices the mismatch and prints the hash it got. One that has built it before still has the old vendor directory in the store. Nix finds a store path for the hash it was told to expect, downloads nothing, and the complaint comes from Go instead:

go: inconsistent vendoring in /private/tmp/nix-build-hello-0.1.0.drv-0/9nbl2zkb...-source:
	github.com/google/[email protected]: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt

	To ignore the vendor directory, use -mod=readonly or -mod=mod.
	To sync the vendor directory, run:
		go mod vendor

Neither suggestion helps. The fix is the hash, and to learn the new one you put a wrong one in on purpose and build again:

error: hash mismatch in fixed-output derivation '/nix/store/fp6ivk5p...-hello-0.1.0-go-modules.drv':
         specified: sha256-AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=
            got:    sha256-KpF1MtAYYg7Rl0sS84KSsK33QcdYtrAPK27EBbMurTM=

Then you copy the got: line into the Nix file, build a third time, and commit.

I have never liked this, so I wrote gonixgo. It builds Go programs with Nix one package per derivation, so dependencies are compiled once instead of on every build. And there is no Nix hash for them to check in.

go.sum and vendorHash #

What bothers me about vendorHash is that the repository already has this information. go.sum holds a hash for every module the build can touch, Go checks each download against it, and it changes in the same commit as go.mod.

vendorHash repeats what go.sum already says, only in a form that Nix can check. Nix does not let a build near the network unless the hash of the result is known in advance, and the hashes in go.sum are in Go’s own format. Somebody has to translate. With buildGoModule that somebody is me, copying a line out of an error message.

gomod2nix at least has a command for it, which writes one hash per module into a gomod2nix.toml. That is still a generated file which I have to remember to regenerate and commit.

Rebuilds #

buildGoModule compiles everything in one derivation. Change a line in main.go and it is a different derivation, so all of it runs again, starting from an empty Go build cache. Every dependency is compiled again and the standard library with them, although none of those changed.

go2nix #

Numtide’s go2nix fixes the rebuilds. It asks go list for the package graph and compiles every package in a derivation of its own with go tool compile, then links with go tool link. go build is never called. Edit a package and Nix rebuilds that package, whatever imports it, and the link.

I like this a lot, it is how I want Go builds in Nix to work. With its plugin go2nix can even go without a lockfile and take the module hashes from go.sum and the module cache.

The requirements are where it gets heavy for me. To create those derivations Nix has to know the package graph while it evaluates, and the graph comes from go list. go2nix has two ways to get it. The default is a plugin that adds a builtin to the Nix evaluator, and it has to be built against the exact Nix that loads it, 2.34 or newer. The other one runs go list at build time and needs recursive-nix, ca-derivations and dynamic-derivations enabled.

One of the two has to be set up on every machine that builds the project: my own, CI, the build server, the machine of whoever works with me. The one I am writing this on has Nix 2.26. For a small Go service that is more than I want to ask for. go2nix has good reasons for both. I still wanted something that works on the Nix that is already installed, even if it is uglier.

The Flake #

This is the whole flake.nix of a Go project built with gonixgo:

{
  inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixos-26.05";
  inputs.gonixgo.url = "github:draganm/gonixgo";

  outputs = { nixpkgs, gonixgo, ... }:
    let
      system = "aarch64-darwin";
      pkgs = nixpkgs.legacyPackages.${system};
      goEnv = gonixgo.lib.mkGoEnv { inherit pkgs; };
    in {
      packages.${system}.default = goEnv.buildGoApplication {
        pname = "hello";
        src = ./.;
      };
    };
}

There is no hash in it, and nothing next to it that has to change when go.mod changes. The build command is longer than usual:

nix build --option allow-unsafe-native-code-during-evaluation true

The option is as dangerous as it sounds, and there is a section about it further down.

builtins.exec #

Nix has been able to run a program during evaluation for years. builtins.exec runs a command and evaluates whatever it prints as a Nix expression. It only exists when allow-unsafe-native-code-during-evaluation is set, so it does not come up much.

The usual way to get the output of a program into an evaluation is import-from-derivation, and it does not work here. A derivation that runs go list needs the modules, a sandboxed build can only download them if it has their hash, and the hash is what I wanted to get rid of.

gonixgo is a Go program that Nix builds during evaluation and then runs through builtins.exec. It calls go list on the source and prints a function. For a hello world with one direct dependency, and with most of the fields cut out, the output looks like this:

b: rec {
  modules = {
    "github.com/fatih/[email protected]" = b.fetchModule {
      name = "gomod-github.com-fatih-color-v1.18.0";
      hash = "sha256-pP5y72FSbi4j/BjyVq/XbAOFjzNjMxZt2R/lFFxGWvY=";
      # ...
    };
    # ...
  };
  packages = {
    "github.com/fatih/color" = b.compile {
      name = "gopkg-github.com-fatih-color-v1.18.0";
      src = modules."github.com/fatih/[email protected]";
      goFiles = [ "color.go" "doc.go" ];
      deps = [ packages."github.com/mattn/go-colorable" packages."github.com/mattn/go-isatty" ];
      # ...
    };
    "example.com/hello" = b.compile {
      name = "golocal-example.com-hello";
      src = b.localDir { name = "gosrc-example.com-hello"; files = [ "main.go" ]; };
      goFiles = [ "main.go" ];
      deps = [ packages."example.com/hello/internal/greet" packages."github.com/fatih/color" ];
      # ...
    };
    # ...
  };
  bins = {
    "hello" = b.link {
      name = "gobin-hello";
      main = packages."example.com/hello";
      # ...
    };
  };
}

The argument b is a set of four builder functions from an 81-line Nix file, and each of them makes an ordinary derivation. From here on it is plain Nix. Every package is compiled with go tool compile, the binary is linked with go tool link, and go version -m on the result prints the same as it does for a go build -trimpath of the same source.

The function is never written to a file, it is printed again on every evaluation.

gonixgo itself is built with buildGoModule, by the way. It imports nothing outside the standard library, so the hash there is null and stays that way.

Module Hashes #

There is still a hash in that output, because Nix insists on one, but it is computed and nobody has to type it.

go list needs the source of every package, so before it answers Go downloads whatever is missing into the module cache and checks it against go.sum, the way it always does. gonixgo then hashes each module directory the way Nix would, and from that works out the store path which a fetch derivation with that hash is going to have. If the path does not exist yet, it copies the directory there with nix store add.

When Nix later gets to the derivation that is supposed to fetch fatih/color, the output is already there and nothing runs. Every module is downloaded once, by Go, with whatever credentials Go has on that machine, so for private modules no token has to go into the Nix store. The fetch derivation only does something when it is built on a machine that did not evaluate it. Then it downloads through GOPROXY, with the environment of whatever builds it, usually the Nix daemon, and a module that can only be fetched with git will fail.

This is what adding a dependency looks like now. I ran go get github.com/google/uuid, called it from main.go, and built:

these 4 derivations will be built:
  gopkg-github.com-google-uuid-v1.6.0.drv
  golocal-example.com-hello.drv
  gobin-hello.drv
  hello.drv

That is the new package, my main package, the link, and the derivation that collects the binaries. There is no fetch in the list because it already happened, and there was no Nix file to update.

Rebuild Times #

fatih/color is not in that list, and neither are its dependencies or the standard library. They were compiled once, on an earlier build, and they stay compiled until one of them changes.

To put a number on it I made a small service that imports grpc, cobra and the Prometheus client, which comes to 126 third-party packages from 14 modules. I built it both ways on an M4 Pro:

The rebuild times are the median of five runs. The service, the flake that builds it both ways and the timing script are in the gonixgo repository, under examples/rebuild-times.

After the one-line change buildGoModule compiles all 126 dependencies and the standard library again. gonixgo does this:

these 3 derivations will be built:
  golocal-example.com-svc.drv
  gobin-svc.drv
  svc.drv

Less than eight seconds is fine. But it is time spent compiling dependencies, so it grows with them, and this was a small program on a fast machine.

The 0.7 seconds with nothing changed include go list, which runs on every evaluation.

The first build goes the other way. 126 packages as 126 derivations took 17 seconds, and go build inside a single derivation does the whole job in 7. The standard library is a derivation as well and takes another 7.5 seconds, once per Go version. You pay the 17 once per version of a dependency, and again for all of them when Go or gonixgo itself changes.

A file that no package uses rebuilds nothing at all. Each package gets a filtered copy of the source with only the files go list reported for it, so editing the README, or the flake, changes none of the derivations.

Parallel Builds #

Part of those 17 seconds is my Nix configuration. It has max-jobs = 1, which is the default, so the 126 packages were built one after another. With -j auto the same first build takes 13 seconds.

Because every package is its own derivation, the work also does not have to stay on one machine. Nix can hand derivations to remote builders, and a service like nixbuild.net will run as many of them in parallel as it can. buildGoModule gives a remote builder one big derivation, and there is nothing in it to spread out. I have not tried gonixgo with a remote builder yet. For 126 packages it would hardly be worth it, for a monorepo with a few thousand it might be.

The Unsafe Flag #

allow-unsafe-native-code-during-evaluation means what it says. While it is on, every Nix expression you evaluate can run any program as you, with no sandbox around it. That includes a flake you only wanted to look at with nix flake show. Pass it on the command line for projects you trust and keep it out of nix.conf.

A flake cannot switch it on for you through nixConfig. I think that is the right decision, and it means everybody who builds the project has to type it. nix flake check wants it too when the package sits under packages, and nix flake show wants --allow-import-from-derivation on top of it, because the tool is built during evaluation. And nothing built this way is ever going into nixpkgs.

Evaluation also depends on more than the source now. It reads the Go module cache and goes to the network when a module is missing. There is also no Nix hash in the repository for a reviewer to look at. What you trust is go.sum plus the module cache of the machine that evaluates, which is what go build on that machine trusts. If you need hashes in a file that somebody signs off, use go2nix with its lockfile.

direnv #

use flake evaluates the development shell, so a shell that contains anything built with gonixgo needs the option as well. direnv hands extra arguments on to Nix, which makes this a one-line change in the .envrc:

use flake . --option allow-unsafe-native-code-during-evaluation true

The option is then on for the evaluation of the shell and for nothing else. A nix build typed into that shell still wants the flag.

To have it on for every Nix command in the project directory, export it instead:

export NIX_CONFIG="allow-unsafe-native-code-during-evaluation = true"
use flake

With that a plain nix build works, and so does nix flake check. So does a nix run of somebody else’s flake from that directory, which is the reason I would start with the first form.

In both cases direnv refuses to load an .envrc that is new or has changed until you run direnv allow, so the option cannot get switched on by a git pull without you being asked first.

Limitations #

It is a few days old. Packages that use cgo are rejected and so are replace directives. Tests are not run. Cross-compilation, go.work and vendor/ directories are not supported.

The service above ran into the first of these right away. The Prometheus client has a cgo file on macOS, so both builds in the comparison were made with CGO_ENABLED = 0, and with tests off for buildGoModule. I have only ever run it on an Apple Silicon Mac. Linux is untested.

Alternatives #

If one derivation is fast enough for you and the hash does not bother you, stay with buildGoModule. It is what nixpkgs uses and it needs nothing. If you can get the plugin or the experimental features onto every machine that builds, go2nix is much further along than my thing: it runs tests and handles cgo, and it does not need an option with “unsafe” in its name. gonixgo sits between the two, and for now only for pure Go programs.

The code is at github.com/draganm/gonixgo. If you try it on Linux, I would like to hear how it went.