An optimised and flexible data format from the Big Science applications of the 90s.
Inbuilt compression and indexing support for things too big for memory.
A good default with wide support and good performance.
For numerical data very simple.
The table definition when writing structured data is booooring and it mangles text encodings if you aren’t careful, which ends up meaning a surprising amount of time can be lost to writing schemas if there is highly structured data to store.
Built-in lossless compression in HDF5 is not impressive on floats, and their lossy compression is bad. Recent HDF5 supports filters providing fancier methods.
I am currently doing a lot of heave HDF5 processing, so this is cruffy.
potentially useful — compression algorithms specialised on smooth fields
- LLNL/H5Z-ZFP: A registered ZFP compression plugin for HDF5
- LLNL/zfp: Compressed numerical arrays that support high-speed random access
Of these only ZFP is supported in hdf5plugin making it easy to use from python.
HDF5 with multiple processes is complicated; may be worthwhile.
Useful tools: visualisers.
To install hdf5py using the homebrew HDF5 libraries (useful for Apple Silicon or other weird architectures) try this tip:
brew install hdf5 HDF5_DIR="$(brew --prefix hdf5)" pip install --no-binary=h5py h5py #or HDF5_DIR="$(brew --prefix hdf5)" pip install --no-build-isolation h5py
I wonder how this interacts with parallelism? How about when writing? It sounds like that is a supported means of getting multiple writers if we are careful.