What is Hystorian ?
Keeping track of the post-treatement done on our data can sometimes be cumbersome.
Hystorian is a project born from our AFM group to solve this issue. It is a python package with two main functionalities:
Structure of an hdf5 file
An hdf5 file has a tree-like structure consisting of groups (folders) and datasets. Each groups and datasets can have attributes attached to them. An attribute is a small metadata containing informations about the corresponding dataset or group. Here we use them to store general information about the data or parameters of the post-processing.
The tree-structure we use in Hystorian is the following :
- A dataset group, containing all the raw data you want to regroup inside a single hdf5.
- A metadata group, containing the metadatas of the initial files.
- A process group, containing the data processed from the original data, or from other processed data.
- A type group, containing the filetype of the original files.
Our goal was to streamline the process and make the usage of Hystorian as seamless as possible for people already using Python. The key to this is the
m_apply function. It allows the application of any usual python function working on array on a datasets inside an hdf5 and to write the results into the same hdf5, while keeping all the key arguments of the function stored with the datasets inside the hdf5 file. Tutorials below show you how to use it in real case examples.
A few tutorial were written by R.Bulanadi and can be found in the examples folder of the project, or in the links below: