Manage a set of Nix Haskell package overrides, somewhat automatically.

Put another way, it aids definition and validation of properties of every override in a set:

• validity (whether it breaks things)
• necessity (whether it’s needed to keep things working, along with a proof)
• actuality (whether the override status is up to date with the upstream, Hackage and Nixpkgs)

Definitions are managed somewhat conveniently, like:

nh hackage foo 1.2.14 nh hackage foo # Pull the latest one nh upstream bar nh upstream bar ghc-8.4 nh unmerged bar concerned-citizen PR-number 

1. Init a package DB:

   nh init 

   ..which will ask you to create a config file first – follow the instuctions.

2. Add overrides:

   nh hackage lens # easy: there's a Hackage release, get the latest 

   nh import microlens-th # import metadata for microlens-th from Hackage cabal file nh upstream microlens-th master # Fetch the tip of upstream's master (any refspec goes) nh chdir microlens-th microlens-th # -- in case the Cabal metadata was wrong nh set-issue microlens-th 222 # Specify the relevant upstream issue ID 

   nh set-upstream hedgehog hedgehogqa nh unmerged hedgehog gwils 134 [GIT-REV] # the third argument to "unmerged" is the Github PR # 

3. Observe the changes:

   nh show lens nh show microlens-th nh show hedgehog 

   ..and you always can generate overrides.nix from the very latest package database – although it happens transparently most of the time:

   nh apply nh override lens # ..to see the individual override 

   The --disable-ghc-configuration switch will position your override set as a replacement to the global, version-specific GHC configuration.

   If you set TARGET_NIXPKGS in the .nh configuration file to a non-empty value, nh will emit non-local overrides directly into the Nixpkgs GHC configuration for the selected GHC version.

4. Validate if the overridden packages build now:

   nh acme # Build all defined overrides nh build NAME.. # ..as an alternative. 

   ..and..

   nh progress # ..to watch progress in another terminal 

5. See what the audit tool thinks about the situation:

   nh audit --skip-acme [NAME..] # Defaults to everything, once more 

   It should complain, at the very least because the trim part was not run, and so there’s no proof that any of the overrides are necessary.

   Note that the nt audit subcommand provides suggestions, that can be auto-executed when the --auto-fix option is supplied.

6. Let’s try trimming the set:

   nh trim [NAME..] 

   That would take a while, and also it won’t do anything to the recorded overrides – it merely collected information, as an intermediate step.

   Let’s execute on that information, then:

   nh execute-trims 

7. Audit again:

   nh audit 

   ..and this is probably going to fail, because the trim is often over-eager. So, some of the overrides need to be reintroduced, and the figuring-out-what-and-why part is manual (albeit assisted). Thankfully, we can record the manually-collected information in the database (which would prevent trimming of this override in the future):

   nh set-explanation broken-attribute doCheck "....." 

   Rinse, repeat – and remember that individual attributes can be rebuilt using nh baseline or nh build.

   Also, explanations (aka proofs of override) can be shown per-override with:

   nh explain happy src 

Finally: look at suite.sh for inspiration.

First, we describe the general flow of fixes, to establish a terminology.

1. Fixes are generally born on a third-party Github repo, and they are expected to be submitted upstream via pull request.
2. The PR gets merged upstream.
3. Upstream cuts a release, bumping the package version in the cabal file.
4. Upstream performs a Hackage upload.
5. Nixpkgs imports Hackage, adding a versioned package-attribute_1_2_1_0.
6. Nixpkgs promotest the versioned package-attribute_1_2_1_0 to package-attribute, which completes the cycle.
7. Nixpkgs also supports non-source tweaks (jailbreaking out of restrictive version bounds, test and Haddock generation disables).

nh maps the above into a status, per attribute:

unmerged

phases #1

upstreamed

phases #2 and #3

hackaged

phase #4

shadowed

phase #5 (after shadow attributes – those shadowing non-versioned ones)

config

not-really-phase #7

1. nh tracks the aforementioned attribute status and content of the attribute overrides in a package database (aka PKGDB). This is just a file-system directory – but it’s better to version it in Git, to be able to recover, when nh goes off-rails and breaks overrides.
2. The result is delivered in the form of a Nix file defining a GHC package set override (customarily called overrides.nix).

   This trivially-structured, generated file is then supposed to be imported into another, static Nix file called packages.nix, which then forms a proper GHC package set. That one can be passed to nix-build.

3. The major package DB operations that nh provides are:

   acme

   Build every attribute in the override set using a proxy, that depends on everything overridden (really, acme).

   trim

   Try to remove overrides, one by one, and record the results of those attempts in the package DB – trying to deducing whether these overrides are necessary. It is a heuristic.

   execute-trims

   Modify the package DB in accordance with the trim step. This effectively removes any overrides that weren’t found necessary. This is also error-prone (more things are sometimes removed than is feasible).

   audit

   Verify every attribute against a set of status-dependent invariants, that ensure:

   • the override necessity (along with the existence of proof), and
   • the override being up-to-date.

4. nh keeps as much build information as possible, and that includes store derivation links, store source links, override expressions and build logs for every build attempt that takes place. In particular every attribute build attempt happens in three phases, handled separately:
   • attribute instantiation
   • dependency pre-build
   • build of the attribute itself

$ nh x hackage funcmp 1.9 downloading ‘http://hackage.haskell.org/package/funcmp-1.9.tar.gz’... [0/0 KiB, 0.0 KiB/s] path is ‘/nix/store/akhnn03wfi3jlx2rqgwjdz07qpz983iz-funcmp-1.9.tar.gz’ - 1d5appkjhajb9ndv2gwnfz8lw2w53v8baajzmrhg26ihzj1bkch8 - https://hackage.haskell.org/package/funcmp-1.9 $ nh set-explanation funcmp src funcmp.def/meta.src.explanation: Needed for (<>) in prelude $ nh jailbreak deepseq-generics $ nh set-explanation deepseq-generics jailbreak deepseq-generics.def/meta.jailbreak.explanation: https://github.com/haskell-hvr/deepseq-generics/pull/4 $ nh import securemem # this fetches metadata like repoName, upstream, chdir etc. $ nh unmerged securemem shlevy 12 6168d90b00bfc6a559d3b9160732343644ef60fb - 06dhx1z44j5gshpdlsb4aryr3g4was3x4c2sgv1px8j57zrvlypx - https://github.com/vincenthz/hs-securemem/commit/6168d90b00bfc6a559d3b9160732343644ef60fb

def

definitions

meta

non-override metadata

over

overrides

hackage, github

src-specific information, per-attribute-override

cache

override cache, per-attribute

build

build output information: logs, expressions, derivations

Usage: nh [--cls] [--nixpkgs] [--trace] [--debug] [--quiet] SUBCMD [SUBARGS..] NOTE: if --nixpkgs is passed, non-local overrides instead serve as definition for /home/deepfire/nixpkgs/pkgs/development/haskell-modules/configuration-ghc-8.4.x.nix PKGDB: forall-defined-edit TYPE FIELD Interactively edit all FIELD definitions of TYPE Metadata (non-override): ls-meta ATTR List attribute's metadata (as opposed to overrides meta ATTR META Print a single metadata entry of an attribute set-meta ATTR META VAL Set a single metadata entry of an attribute edit-meta ATTR META Edit the current attribute's meta value using readline disable ATTR[.OVER] Disable all/single overrides for an attribute enable ATTR[.OVER] Re-enable previously disabled overrides with-disabled-attrs ATTR.. Disable all listed attribute overrides and pause; Re-enable on exit or newline in stdin ls-disabled List all disabled attributes set-explanation ATTR OVER VAL Manually supply explanation for an override's existence set-erdeps ATTR 'ATTR..' Set attribute's essential rev-deps that must keep working chdir ATTR SUBDIR Change directory before build; "" removes the override local ATTR Mark ATTR as local: not subject for Nixpkgs GHC configuration nonlocal ATTR Remove marking of ATTR as local Override manipulation (low level): remove ATTR[.OVER] Remove specified overrides ls-over ATTR List attribute's overrides ls-input-overs ATTR List attribute's input overrides get ATTR OVER Get an attribute's override value set ATTR OVER VAL Set an attribute's override value; "" removes the override edit ATTR OVER Edit the current attribute's value using readline set-input-over ATTR INPUT VAL Set ATTR's override for INPUT edit ATTR OVER Edit the current attribute's value using readline check ATTR Disable an existing dontCheck override dontCheck ATTR Disable tests haddock ATTR Disable an existing dontHaddock override dontHaddock ATTR Disable Haddock generation jailbreak ATTR Turn on jailbreaking dontJailbreak ATTR Disable an existing jailbreak override {library,executable,test}Haskell ATTR [ATTR..] Specify extra *HaskellDepends; "" removes the override add-patch ATTR SHA256 URL Add a patch to ATTR Status: status ATTR Print status of a single attribute ls-shadowed List all attributes with status 'shadowed' ls-hackaged ...'hackaged' ls-upstreamed ...'upstreamed' ls-unmerged ...'unmerged' ls-config ...'config' Nix-level inferences: drv ATTR Store derivation for a single override pprint-drv ATTR Pretty-print ATTR's derivation (requires nix-derivation-pretty) src ATTR Store source derivation for a single override src-drv ATTR Store source derivation of ATTR src-url ATTR Source URL of ATTR inputs ATTR ATTR's store inputs deps | refs | references ATTR ATTR's store drv dependencies rdeps | referrers ATTR ATTR's store reverse drv dependencies realise-drv ATTR Realise ATTR's derivation drv-pprint STORE-DRV Pretty-print a Nix-stored .drv file src-drv ATTR Store source derivation of ATTR src-drv-url STORE-DRV Source URL of a Nix-stored source-.drv file drv-inputs STORE-DRV Store inputs for a Nix-stored .drv file drv-refs | drv-references STORE-DRV Store .drv references for a Nix-stored .drv file deriver-of STORE-PATH Store .drv for a Nix store path. Will fail if built non-locally PKGDB emission to Nix overrides: over | override | show-override ATTR Print the attribute's override defined by PKGDB apply [--reuse-cache] Apply all overrides via /home/deepfire/overrides.nix cache [--require-descs] Regenerate override cache show-cache ATTR Print the cached text of attribute's override (DEBUG) General: ls [REGEX] List all overridden attributes info ATTR Overview of an attribute's PKGDB overview [ATTR..] List overridden attributes, grouped by status + relevant info Hackage: import ATTR Scrape ATTR's Cabal file from Hackage for some properties cabal ATTR Print the latest released cabal file for ATTR hackage ATTR [RELEASE=upstream-latest] Override to a Hackage release Github: github ATTR [REF] Override ATTR to its latest upstream Github commit unmerged ATTR USER PR# [REV=HEAD] Override to a 3rd-party Github commit upstream ATTR [REV=HEAD] Override to an upstream Github commit set-upstream ATTR GITHUB-USER Specify an attribute's Github upstream username edit-upstream ATTR Edit an attribute's Github upstream username set-pr ATTR PR# Set the PR# of an attribute's Github override set-issue ATTR ISSUE# Set the Issue# of an attribute's Github override set-repoName ATTR REPO Set an attribute's Github repository name edit-repoName ATTR Edit an attribute's Github repository name Build & results: instantiate [--reuse-cache] [ATTR..] Instantiate overridden attrs (or specified subset) acme [--reuse-cache] Build everything at once, collecting all failures build [COMMON-OPTS] ATTR Build a single attribute with current overrides log ATTR [OVER=baseline] Obtain trim build logs for a single override failure ATTR [OVER=baseline] Obtain trim failure kind of an override failure-log ATTR [OVER=baseline] Obtain trim failure log of an override failure-type ATTR [OVER=baseline] Obtain trim failure type of an override proof ATTR [OVER] Print an override's proof of necessity. When OVER is empty, print context. Override database maintenance: trim [--reuse-cache] [ATTR..] Suggest a reduction to the override set (or specified subset) trim-override ATTR OVER Attempt trimming a specific override of a given attribute show-trims Show the trim suggestion execute-trims Execute the suggestion audit [--autofix] [--autoonly] [--skip-acme] [--reuse-{overrides,cache}] [ATTR..] Sanity check the overridden attrs (or specified subset). --autofix applies suggestions extra-validation-attributes Edit the set of attributes validated regardless of being overridden edit-fixed-content Edit the static part of the GHC configuration Nix shell: shell Nix shell with up-to-date overrides (shell.nix required) shell-for ATTR Nix shell for building ATTR cabal-shell Nix shell from a cabal file (nothing else required) clone-upstream-fixer-shell Nix shell from a cabal file (nothing else required) try-fix ATTR Push the current commit and try the fix find-module NAME Convenience alias for 'ghc-pkg find-module NAME' list-packages ... Convenience alias for 'ghc-pkg list ... describe-package ATTR Convenience alias for 'ghc-pkg describe ATTR package-modules ATTR List ATTR's exposed modules phases ATTR Print ATTR's build phases Miscellanea: eval BASH-EXPR Passthrough, to execute anything defined. loop-hunter Detect attribute loops: nix-shell 2>&1 | nh loop-hunter ls-builds List active builds progress [LOG] Live summary of new, complete and failing builds watch Observe the current build, as it hits the logs.. ghc Shell with current GHC prefetch-ghc GITREV Prefetch a GHC revision less-ghc-config [NEEDLE] Run less on the Nixpkgs GHC configuration git OPTIONS.. ARGS.. Run git inside controlled Nixpkgs nixpkgs-diff [(base-head|base-master|head-master] [REF] Diff of current GHC configuration