Numerical python

zarr

Zarr is a format for the storage of chunked, compressed, N-dimensional arrays. …

Highlights:

• Create N-dimensional arrays with any NumPy dtype.
• Chunk arrays along any dimension.
• Compress and/or filter chunks using any NumCodecs codec.
• Store arrays in memory, on disk, inside a Zip file, on S3, …
• Read an array concurrently from multiple threads or processes.
• Write to an array concurrently from multiple threads or processes.
• Organize arrays into hierarchies via groups.

Resembles HDF5 but makes fewer assumptions about the storage backend and more assumptions about the language frontend.

Dask

dask is a flexible library for parallel computing in Python.

See the howto for some examples of use cases.

Dask is composed of two parts:

1. Dynamic task scheduling optimized for computation. This is similar to Airflow, Luigi, Celery, or Make, but optimized for interactive computational workloads.

2. “Big Data” collections like parallel arrays, dataframes, and lists that extend common interfaces like NumPy, Pandas, or Python iterators to larger-than-memory or distributed environments. These parallel collections run on top of dynamic task schedulers.

Dask emphasizes the following virtues:

• Familiar: Provides parallelized NumPy array and Pandas DataFrame objects

• Flexible: Provides a task scheduling interface for more custom workloads and integration with other projects.

• Native: Enables distributed computing in pure Python with access to the PyData stack.

• Fast: Operates with low overhead, low latency, and minimal serialization necessary for fast numerical algorithms

• Scales up: Runs resiliently on clusters with 1000s of cores

• Scales down: Trivial to set up and run on a laptop in a single process

• Responsive: Designed with interactive computing in mind, it provides rapid feedback and diagnostics to aid humans

HDF5

h5py provides a simple, easy and efficient interface to HDF5 files, which are an easy way of loading and saving arrays of numbers and indeed arbitrary data. I use this a lot but sometimes I get glitches on macos with passing around serialised H5py objects, which works fine on Linux.

xarray

xarray is a system which labels arrays, which is useful in keeping track of them for statistical analysis.

Xarray introduces labels in the form of dimensions, coordinates and attributes on top of raw NumPy-like multidimensional arrays, which allows for a more intuitive, more concise, and less error-prone developer experience.

This data model is borrowed from the netCDF file format, which also provides xarray with a natural and portable serialization format. NetCDF is very popular in the geosciences, and there are existing libraries for reading and writing netCDF in many programming languages, including Python.

This system seems to be a developing lingua franca for the differentiable learning frameworks.

Compiling

e.g. for performance or invoking external binaries.

See compiling python.

Displaying numbers legibly

Easy, but documentation is hard to find.

print("{:10.4f}".format(x))

np.set_printoptions(edgeitems=5,
  linewidth=85, precision=4,
  suppress=True, threshold=500)

np.set_printoptions(edgeitems=3, infstr='inf',
  linewidth=75, nanstr='nan', precision=8,
  suppress=False, threshold=1000, formatter=None)

Tips

from numpy.random import get_state, set_state, seed

class Seed(object):
  """
  context manager for reproducible seeding.

  >>> with Seed(5):
  >>>   print(np.random.rand())

  0.22199317108973948
  """
  def __init__(self, seed):
    self.seed = seed
    self.state = None

  def __enter__(self):
    self.state = get_state()
    seed(self.seed)

  def __exit__(self, exc_type, exc_value, traceback):
    set_state(self.state)