How do you install the right versions of everything for some python code you are developing? How to deploy that sustainably? How to share it with others.
An obvious frontrunner on popularity alone. See containerization.
A package manager which also manages dependencies.
If you are a developer, Vagrant will isolate dependencies and their configuration within a single disposable, consistent environment, without sacrificing any of the tools you are used to working with (editors, browsers, debuggers, etc). Once you or someone else creates a single Vagrantfile, you just need to
vagrant upand everything is installed and configured for you to work. Other members of your team create their development environments from the same configuration, so whether you are working on Linux, Mac OS X, or Windows, all your team members are running code in the same environment, against the same dependencies, all configured the same way
I didn’t find that clear, but this example was illustrative:
$ vagrant init hashicorp/bionic64 $ vagrant up Bringing machine 'default' up with 'virtualbox' provider... ==> default: Importing base box 'hashicorp/bionic64'... ==> default: Forwarding ports... default: 22 (guest) => 2222 (host) (adapter 1) ==> default: Waiting for machine to boot... $ vagrant ssh vagrant@bionic64:~$ _
For HPC specifically, there is spack.
Spack is a package manager for supercomputers, Linux, and macOS. It makes installing scientific software easy. With Spack, you can build a package with multiple versions, configurations, platforms, and compilers, and all of these builds can coexist on the same machine.
Spack isn’t tied to a particular language; you can build a software stack in Python or R, link to libraries written in C, C++, or Fortran, and easily swap compilers. Use Spack to install in your home directory, to manage shared installations and modules on a cluster, or to build combinatorial versions of software for testing.
Bonus feature: also lets us prototype on macOS. I do not know what particular HPC affordances it offers.
Have not yet used. asdf-vm/asdf: Extendable version manager with support for Ruby, Node.js, Elixir, Erlang & more.
direnv is an extension for your shell. It augments existing shells with a new feature that can load and unload environment variables depending on the current directory.
- Load 12factor apps environment variables
- Create per-project isolated development environments
- Load secrets for deployment
This does indeed create a toolkit for developing isolated environments with specific packages
Modern build tools will also manage dependencies. e.g. google’s Bazel spend so much energy doing this when you build tensorflow that I become suspicious that it has mirred the entire internet to mu local hard drive
Exodus is a tool that makes it easy to successfully relocate Linux ELF binaries from one system to another. This is useful in situations where you don't have root access on a machine or where a package simply isn't available for a given Linux distribution. For example, CentOS 6.X and Amazon Linux don't have packages for Google Chrome or aria2. Server-oriented distributions tend to have more limited and outdated packages than desktop distributions, so it's fairly common that one might have a piece of software installed on their laptop that they can't easily install on a remote machine.