Configuring python with “.env” files

This is mostly python tips for now, but it could potentially be cross-platform. Most important for me as part of a configuring ML.

There are many, many python tools to load environment variables from local files. A good place to find generic resources on that is “Twelve-Factor App” configuration. But that includes eleven more factors than I personally care about because I am not a web developer; I just want the environment config part.

One system I have used is dotenv. dotenv allows easy configuration through OS environment variables or text files in the parent directory.

PRO-TIP: there are lots of packages with similar names but dissimilar functions.

pip install python-dotenv # or
conda install -c conda-forge python-dotenv

Also very similar, henriquebastos/python-decouple. sloria/environs: simplified environment variable parsing Dynaconf is very sophisticated, and comes closer to a full configuration system like hydra.

Let us imagine we are using basic dotenv for now for concreteness Then we can be indifferent to whether files came from an FS config or an environment variable.

import os.path
from dotenv import load_dotenv
load_dotenv()  # take environment variables from .env.
# Code of your application, which uses environment variables (e.g. from `os.environ` or
# `os.getenv`) as if they came from the actual environment.
# substituting a var into a path:
DATA_FILE_PATH = os.path.expandvars('$DATA_PATH/$DATA_FILE')
# getting a var with a default fallback
FAVOURITE_PIZZA_TOPPING = os.getenv('FAVOURITE_PIZZA_TOPPING', 'cheese')

There is a CLI too; its most useful feature is executing arbitrary stuff with the correct environment variable set.

pip install "python-dotenv[cli]"
dotenv run my_cool_script.py

AFAICS this should mean that dotenv is not restricted to python; those .env files can be used in any language. This only works for running python scripts AFAICT.