Configuring python with “.env” files

This is mostly python tips for now, but it could potentially be cross-platform. Most important for me as part of a configuring ML.

There are many, many python tools to load environment variables from local files. A good place to find generic resources on that is “Twelve-Factor App” configuration. But that includes eleven more factors than I personally care about because I am not a web developer; I just want the environment config part.

One system I have used is dotenv. dotenv allows easy configuration through OS environment variables or text files in the parent directory.

pip install python-dotenv # or
conda install -c conda-forge python-dotenv

Also similar, henriquebastos/python-decouple, sloria/environs. Dynaconf is sophisticated, and comes closer to a full configuration system like hydra, and as such is too much for me

Let us imagine we are using basic dotenv for now for concreteness Then we can be indifferent to whether files came from an FS config or an environment variable.

import os.path
from dotenv import load_dotenv
load_dotenv()  # take environment variables from .env.
# Code of your application, which uses environment variables (e.g. from `os.environ` or
# `os.getenv`) as if they came from the actual environment.
# substituting a var into a path:
DATA_FILE_PATH = os.path.expandvars('$DATA_PATH/$DATA_FILE')
# getting a var with a default fallback
FAVOURITE_PIZZA_TOPPING = os.getenv('FAVOURITE_PIZZA_TOPPING', 'cheese')

The datafile .env is just a text file with lines like

DATA_PATH=/home/username/data
DATA_FILE=foo.csv
FAVOURITE_PIZZA_TOPPING=anchovies

There is a CLI too; its most useful feature is executing arbitrary stuff with the correct environment variable set.

pip install "python-dotenv[cli]"
dotenv run my_cool_script.py

AFAICS this should mean that dotenv is not restricted to python; those .env files can be used in any language. This only works for running python scripts AFAICT.