SIONlib

SIONlib is a library for writing and reading data from many thousands of parallel tasks/processes into or from one or a small number of physical files. Only the file open and close functions are collective, while each task or process can access files independently. This lends itself to drop-in I/O replacement for applications that use standard I/O in a process-local way – where each parallel process writes to a different file. SIONlib can be easily used as a replacement for standard I/O APIs (e.g. POSIX, ANSI-C), as file access is performed using SIONlib equivalents to standard C-I/O with similar semantic. SIONlib bundles this data into one or a few physical files to avoid sequential bottlenecks in the filesystem, e. g., metadata processing and contention. The process-per-file picture is maintained for the application, so each process has access to its logical file only. This enables in-place parallelisation for simple I/O schemes and is particularly suited for checkpoint and restart files. Figure 1 illustrates the operation of SIONlib for the case of a single physical file.

In DEEP-SEA, SIONlib is being adapted to enable the efficient use of Exascale resources and MSA for I/O. For applications with a suitable I/O pattern that can scale compute easily, SIONlib can be a straightforward option for I/O scaling at Exascale. In DEEP-SEA, SIONlib will be extended with the ability to decide where to store intermediate data in different levels of node local storage architecture, providing buffering and caching capabilities that result in more efficient I/O patterns with regards to filesystem access and network contention. Alongside this, SIONlib will optimise internal data flow on heterogeneous systems by modifying the I/O forwarding layer recently developed in the DEEP-EST projects.