Search NASA⌕ Search

Engineering topics

Hanford, Nathan

Publications and source records attributed to Hanford, Nathan.

Working with Multiple MPIs: Overcoming ABI Incompatibility

Scientific applications rely on third-party libraries, which may have multiple implementations. While sharing an application programming interface (API), many of these implementations do not have a shared application binary interface (ABI) and require recompiling. Recompiling can be a long and complex process and sometimes not even an option when the application is shipped binary only. ABI incompatibility strikes at the heart of portability, productivity, and performance by (1) impeding application execution across different HPC and Cloud systems; (2) adding developer hours rebuilding an application; and (3) not taking advantage of host-optimized libraries. This tutorial teaches attendees a portable way to address ABI incompatibility in MPI using the Wi4MPI library. Wi4MPI translates the ABI dynamically from the MPI library used to build the application to a different MPI library available at run time. With Wi4MPI, HPC practitioners can break the portability barrier imposed by ABI incompatibility, potentially increase performance, and increase user productivity. The tutorial is broken down in three components: (1) Understanding ABI compatibility in MPI; (2) Translating MPI libraries dynamically; and (3) Applying dynamic translation to key use cases in HPC, including Containers. If you use more than one MPI library or supercomputer, this tutorial is for you.

97 MATHEMATICS AND COMPUTING↗

Productivity frameworks for HPC

Productivity Frameworks for HPC will include container recipes, build recipes, continuous integration scripts, and other software aimed at testing the portability of containerized HPC software across platforms and interconnects. In particular, it tests the utility of bind-mounting at the MPI layer (rather than the underlying fabric layer) to leverage a standardized protocol and avoid various technical debt and vendor lock-in. Since MPIs are often ABI-incompatible, trampolines such as the open-source Wi4MPI will be tested when such cases arise.

Hanford, Nathan↗

Calibers: A Bandwidth Calendaring Paradigm for Science Workflows

Many scientific workflows require large data transfers between distributed instrument facilities, storage and computing resources. To ensure that these resources are maximally utilized, R&E networks connecting these resources must ensure that an inherently unpredictable network behaves predictably. In practice, this amounts to the per-application over-provisioning of network resources in an attempt to guarantee that adequate throughput is provided to users. This often results in resource under-utilization over time. One promising solution is the use of deadlines and bandwidth calendaring. In this approach, "fair" resource allocation is replaced with deadline-based resource allocation. However, these approaches often suffer from issues in efficiently regulating resource allocation and failure modes. Therefore, our solution, Calibers, approaches bandwidth calendaring and deadline-awareness in a different way. Calibers uses shaping, metering, and pacing at the edge of the network and end-system to provide participating clients the ability to schedule bandwidth reservations without having to worry about network noise from non-participating clients. Calibers can also fail back to the fair resource allocation of underlying transport protocols if necessary. For example, if a non-participating flow somehow enters the core of the network, or a sudden network change causes the available bandwidth to be exceeded, the underlying transport protocol congestion avoidance implementation will be able to handle the congestion as it normally would. Furthermore, Calibers provides a novel simulation method and resource allocation algorithm.

97 MATHEMATICS AND COMPUTING↗