Generic dependency managers

May 3, 2021 — June 17, 2022

computers are awful
python
Figure 1: Reproducible deployment error

How do you install the right versions of everything for some python code you are developing? How to deploy that sustainably? How to share it with others.

1 Containerization

An obvious frontrunner on popularity alone. See containerization.

2 Homebrew

See homebrew.

3 Nix

A package manager which also manages dependencies.

4 Vagrant

Figure 2: Dependency import error

vagrant: Virtual machines as dependency managers.

If you are a developer, Vagrant will isolate dependencies and their configuration within a single disposable, consistent environment, without sacrificing any of the tools you are used to working with (editors, browsers, debuggers, etc). Once you or someone else creates a single Vagrantfile, you just need to vagrant up and everything is installed and configured for you to work. Other members of your team create their development environments from the same configuration, so whether you are working on Linux, Mac OS X, or Windows, all your team members are running code in the same environment, against the same dependencies, all configured the same way

I didn’t find that clear, but this example was illustrative:

$ vagrant init hashicorp/bionic64
$ vagrant up
  Bringing machine 'default' up with 'virtualbox' provider...
  ==> default: Importing base box 'hashicorp/bionic64'...
  ==> default: Forwarding ports...
  default: 22 (guest)
  => 2222 (host) (adapter 1)
  ==> default: Waiting for machine to boot...

$ vagrant ssh
  vagrant@bionic64:~$ _

5 Spack

Figure 3

For HPC specifically, there is spack.

Spack is a package manager for supercomputers, Linux, and macOS. It makes installing scientific software easy. With Spack, you can build a package with multiple versions, configurations, platforms, and compilers, and all of these builds can coexist on the same machine.

Spack isn’t tied to a particular language; you can build a software stack in Python or R, link to libraries written in C, C++, or Fortran, and easily swap compilers. Use Spack to install in your home directory, to manage shared installations and modules on a cluster, or to build combinatorial versions of software for testing.

Bonus feature: also lets us prototype on macOS. I do not know what particular HPC affordances it offers.

6 asdf

Have not yet used. asdf-vm/asdf: Extendable version manager with support for Ruby, Node.js, Elixir, Erlang & more.

7 direnv

direnv is an extension for your shell. It augments existing shells with a new feature that can load and unload environment variables depending on the current directory.

Use cases

  • Load 12factor apps environment variables
  • Create per-project isolated development environments
  • Load secrets for deployment

This does indeed create a toolkit for developing isolated environments with specific packages

8 Build tools

Modern build tools will also manage dependencies. e.g. google’s Bazel spend so much energy doing this when you build tensorflow that I become suspicious that it has mirred the entire internet to mu local hard drive

9 Cheating

  • intoli/exodus: Painless relocation of Linux binaries–and all of their dependencies–without containers.

    Exodus is a tool that makes it easy to successfully relocate Linux ELF binaries from one system to another. This is useful in situations where you don’t have root access on a machine or where a package simply isn’t available for a given Linux distribution. For example, CentOS 6.X and Amazon Linux don’t have packages for Google Chrome or aria2. Server-oriented distributions tend to have more limited and outdated packages than desktop distributions, so it’s fairly common that one might have a piece of software installed on their laptop that they can’t easily install on a remote machine.