Search NASA⌕ Search

SEARCH · Search NASA

Results for “Parallel processing”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 1,297 records · Page 72

HBMax: Optimizing Memory Efficiency for Parallel Influence Maximization on Multicore Architectures

The goal of influence maximization is to select k most-influential vertices or seeds in a network, where influence is defined by a given diffusion process. The problem has a number of important applications such as viral marketing, information spread, and epidemic control. Although computing optimal seed set is NP-Hard, due to the submodular nature of the problem efficient approximation algorithms exist. However, even state-of-the-art parallel implementations are limited by a sampling step that incurs large memory footprints. This in turn limits the problem size reach and approximation quality. In this work, we study the memory footprint of the sampling process collecting reverse reachability information in the IMM algorithm over large real-world social networks. We present an adaptive and memory-efficient optimization approach for a state-of-the-art multi-threaded parallel influence maximization algorithm. Our approach,HuffMax, uses a portion of the reverse reachable (RR) sets collected by the algorithm to learn the characteristics of the graph. Then, it compresses the intermediate reverse reachability information with Huffman coding, and queries directly on the compressed data to preserve the memory savings obtained through compression. We also propose an efficient sampling strategy based on the distribution of RR sets, which can further reduce the computation time for typical social networks with long-tail distributions. Considering a NUMA architecture, we scale up our solution on 128-core CPUs and reduce the memory footprint by up to 45.7% with negligible time overhead (or even faster) and without perceivable loss of accuracy.

Chen, Xinyu↗

High Temperature Syngas Cleanup Technology Scale-up and Demonstration Project

Gasification is a technology for clean energy conversion of diverse feedstocks into a wide variety of useful products such as chemicals, fertilizers, fuels, electric power, and hydrogen. Existing technologies can be employed to clean the syngas from gasification processes to meet the demands of such applications, but they are expensive to build and operate and consume a significant fraction of overall parasitic energy requirements, thus lowering overall process efficiency. RTI International has developed a warm syngas desulfurization process (WDP) utilizing a transport-bed reactor design and a proprietary attrition-resistant, high-capacity solid sorbent with excellent performance replicated at lab, bench, and pilot scales. Results indicated that WDP technology can improve both efficiency and cost of gasification plants. The WDP technology achieved ~99.9% removal of total sulfur (as either H 2 S or COS) from coal-derived syngas at temperatures as high as 600°C and over a wide range of pressures (20-80 bar, pressure independent performance) and sulfur concentrations. Based on the success of these tests, RTI negotiated a cooperative agreement with the U.S. Department of Energy for precommercial testing of this technology at Tampa Electric Company’s Polk Power Station IGCC facility in Tampa, Florida. The project scope also included a sweet water-gas-shift process for hydrogen enrichment and an activated amine process for 90+% total carbon capture. Because the activated amine process provides some additional non-selective sulfur removal, the integration of these processes was expected to reduce overall sulfur in the syngas to sub-ppmv concentrations, suitable for most syngas applications. The overall objective of this project was to mitigate the technical risks associated with the scale up and integration of the WDP and carbon dioxide capture technologies, enabling subsequent commercial-scale demonstration. The warm syngas cleanup pre-commercial test unit was designed and constructed on schedule and under budget and was operated for approximately 1,500 total hours utilizing ~20% of the IGCC’s total syngas as feed (~1.5 MM scfh of dry syngas). The WDP system reduced total sulfur levels to ~10 ppmv (~99.9% removal) from raw syngas that contained as high as 14,000 ppmv of total sulfur. The integration of WDP with the activated amine process enabled further reduction of total sulfur in the final treated syngas to the anticipated sub-ppmv concentrations (>99.99% removal), suitable for stringent syngas applications such as chemicals, fertilizers, and fuels. Techno-economic assessments by RTI and by third parties indicate potential for significant (up to 50%) capital and operating cost reductions for the entire syngas cleanup block when WDP technology is integrated with a broad spectrum of conventional and emerging carbon capture or acid gas removal technologies. This final scientific/technical report covers the pre-FEED, FEED, EPC, commissioning, and operation phases of this project, as well as system performance results. In addition, the report addresses other parallel-funded R&D efforts focused on development and testing of trace contaminant removal process (TCRP) sorbents, a direct sulfur recovery process (DSRP), and a novel sorbent for warm carbon dioxide capture, as well as pre-FEED, FEED, and techno-economic studies to consider the potential benefit for use of WDP for polygeneration of electric power and ammonia/urea fertilizers.

01 COAL, LIGNITE, AND PEAT↗

Evaluation of emerging parallel optical link technology for high energy physics

Modern particle detectors utilize optical fiber links to deliver event data to upstream trigger and data processing systems. Future detector systems can benefit from the development of dense arrangements of high speed optical links emerging from industry advancements in transceiver technology. Supporting data transfers of up to 120 Gbps in each direction, optical engines permit assembly of the optical transceivers in close proximity to ASICs and FPGAs. Test results of some of these parallel components will be presented including the development of pluggable FPGA Mezzanine Cards equipped with optical engines to provide to collaborators on the Versatile Link Common Project for the HI-LHC at CERN.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗

Delta: Data Reduction for Integrated Application Workflows

Integrated Application Workflows (IAWs) run multiple simulation workflow components concurrently on an HPC resource connecting these components using compute area resources and compensating for any performance or data processing rate mismatches. These IAWs require high frequency and high volume data transfers between compute nodes and staging area nodes during the lifetime of a large parallel computation. The available network band-width between the two areas may not be enough to efficiently support the data movement. As the processing power available to compute resources increases, the requirements for this data transfer will become more difficult to satisfy and perhaps will not be satisfiable at all since network capabilities are not expanding at a comparable rate. Furthermore, energy consumption in HPC environments is expected to grow by an order of magnitude as exascale systems become a reality. The energy cost of moving large amounts of data frequently will contribute to this issue. It is necessary to reduce the volume of data without reducing the quality of data when it is being processed and analyzed. Delta resolves the issue by addressing the lifetime data transfer operations. Delta removes subsequent identical copies of already transmitted data during transfers and restores those copies once the data has reached the destination. Delta is able to identify duplicated information and determine the most space efficient way to represent it. Initial tests show about 50% reduction in data movement while maintaining the same data quality and transmission frequency.

97 MATHEMATICS AND COMPUTING↗

A parallel workflow implementation for PEST version 13.6 in high-performance computing for WRF-Hydro version 5.0: a case study over the midwestern United States

The Weather Research and Forecasting Hydrological (WRF-Hydro) system is a state-of-the-art numerical model that models the entire hydrological cycle based on physical principles. As with other hydrological models, WRF-Hydro parameterizes many physical processes. Hence, WRF-Hydro needs to be calibrated to optimize its output with respect to observations for the application region. When applied to a relatively large domain, both WRF-Hydro simulations and calibrations require intensive computing resources and are best performed on multimode, multicore high-performance computing (HPC) systems. Typically, each physics-based model requires a calibration process that works specifically with that model and is not transferrable to a different process or model. The parameter estimation tool (PEST) is a flexible and generic calibration tool that can be used in principle to calibrate any of these models. In its existing configuration, however, PEST is not designed to work on the current generation of massively parallel HPC clusters. To address this issue, we ported the parallel PEST to HPCs and adapted it to work with WRF-Hydro. The porting involved writing scripts to modify the workflow for different workload managers and job schedulers, as well as to connect the parallel PEST to WRF-Hydro. To test the operational feasibility and the computational benefits of this first-of-its-kind HPC-enabled parallel PEST, we developed a case study using a flood in the midwestern United States in 2013. Results on a problem involving the calibration of 22 parameters show that on the same computing resources used for parallel WRF-Hydro, the HPC-enabled parallel PEST can speed up the calibration process by a factor of up to 15 compared with commonly used PEST in sequential mode. The speedup factor is expected to be greater with a larger calibration problem (e.g., more parameters to be calibrated or a larger size of study area).

54 ENVIRONMENTAL SCIENCES↗

Electrochemical Reduction of Flue Gas Carbon Dioxide to Commercially Viable C2-C4 Products (Final Report)

This is the final scientific/technical report for a DOE project focused on the electrochemical conversion of CO 2 in non-aqueous solvents to novel products. Electrochemical reduction of CO 2 provides an attractive route to produce valuable fuels and chemicals that can simultaneously lower greenhouse gas emissions when powered by renewable electricity. While recent technological advances have shown the feasibility of industrial CO 2 electroreduction, many challenges remain to improve this technology and expand the list of economically viable products. The vast majority of electrochemical CO 2 reduction research has been conducted in aqueous media under neutral to alkaline conditions, leading to commonly reported products including carbon monoxide, formic acid, methane, methanol, ethylene, acetic acid, and ethanol. In comparison, non-aqueous media for CO 2 reduction has been underexplored but represents a possible avenue to yield new products and improved operating conditions. The aim of the project was to convert waste CO 2 in the form of flue gas to a multicarbon C2 - C4 chemical product in a reactor designed to achieve economically competitive values of current density and selectivity. The project strived to advance the technology readiness of an electrochemical CO 2 reduction process in alcohol solvents from the proof-of-concept stage to a device capable of meeting performance metrics for commercial viability. In the initial plan, the University of Louisville researchers were to focus on investigating the electrochemical process and improving the faradaic efficiency for novel C2 – C4 species, while also working on a parallel effort to build a practical electrolysis reactor to markedly increase the CO 2 reduction current density. The reactor development effort also aimed to engineer a dual-electrolyte feed strategy with non-aqueous catholyte and aqueous anolyte to promote water oxidation as the coupling anodic half-reaction to enable a sustainable and economical overall process. At the outset, the University of North Dakota was to investigate the feasibility of operating directly from coal-derived flue gas without separate capture and purification. The research team sought to determine impurity effects and test mitigation strategies, as well as engineer the gaseous feed system for high reactor tolerance to lower CO 2 concentration. In the last half year of the project, the focus was planned to shift to integrating the advances in the catalysis, electrochemical conditions, reactor design, and flue gas compatibility into a fully functional device and improve it for maximum current density and faradaic efficiency for C2 – C4 species. Knowledge of the full system components, constraints, and maximum performance was then to be used as the basis for a thorough technoeconomic analysis (TEA) and life cycle analysis (LCA) at the end of the project.

01 COAL, LIGNITE, AND PEAT↗

Genten: Software for Generalized Tensor Decompositions v. 1.0.0

Tensors, or multidimensional arrays, are a powerful mathematical means of describing multiway data. This software provides computational means for decomposing or approximating a given tensor in terms of smaller tensors of lower dimension, focusing on decomposition of large, sparse tensors. These techniques have applications in many scientific areas, including signal processing, linear algebra, computer vision, numerical analysis, data mining, graph analysis, neuroscience and more. The software is designed to take advantage of parallelism present emerging computer architectures such has multi-core CPUs, many-core accelerators such as the Intel Xeon Phi, and computation-oriented GPUs to enable efficient processing of large tensors.

Phipps, EricT.↗

Dynamics of rapidly spinning blob-filaments: Fluid theory with a parallel kinetic extension

Blob-filaments (or simply “blobs”) are coherent structures formed by turbulence and sustained by nonlinear processes in the edge and scrape-off layer (SOL) of tokamaks and other magnetically confined plasmas. The dynamics of these blob-filaments, in particular, their radial motion, can influence the scrape-off layer width and plasma interactions with both the divertor target and with the main chamber walls. Motivated by recent results from the XGC1 gyrokinetic simulation code reported on elsewhere [J. Cheng et al., Nucl. Fusion 63, 086015 (2023)], a theory of rapidly spinning blob-filaments has been developed for this work. The theory treats blob-filaments in the closed flux surface region or the region that is disconnected from sheaths in the SOL. It extends previous work by treating blob spin, arising from partially or fully adiabatic electrons, as the leading-order effect and retaining inertial (ion charge polarization) physics in next order. Spin helps to maintain blob coherency and affects the blob's propagation speed. Dipole charge polarization, treated perturbatively, gives rise to blob-filaments with relatively slow radial velocity, comparable to that observed in the simulations. The theory also treats the interaction of rapidly spinning blob-filaments with a zonal flow layer. It is shown analytically that the flow layer can act like a transport barrier for these structures. Finally, parallel electron kinetic effects are incorporated into the theory. Various asymptotic parameter regimes are discussed, and asymptotic expressions for the radial and poloidal motion of the blob-filaments are obtained.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗

SWAP-Assembler: scalable and efficient genome assembly towards thousands of cores

Background: There is a widening gap between the throughput of massive parallel sequencing machines and the ability to analyze these sequencing data. Traditional assembly methods requiring long execution time and large amount of memory on a single workstation limit their use on these massive data. Results: This paper presents a highly scalable assembler named as SWAP-Assembler for processing massive sequencing data using thousands of cores, where SWAP is an acronym for Small World Asynchronous Parallel model. In the paper, a mathematical description of multi-step bi-directed graph (MSG) is provided to resolve the computational interdependence on merging edges, and a highly scalable computational framework for SWAP is developed to automatically preform the parallel computation of all operations. Graph cleaning and contig extension are also included for generating contigs with high quality. Experimental results show that SWAP-Assembler scales up to 2048 cores on Yanhuang dataset using only 26 minutes, which is better than several other parallel assemblers, such as ABySS, Ray, and PASHA. Results also show that SWAP-Assembler can generate high quality contigs with good N50 size and low error rate, especially it generated the longest N50 contig sizes for Fish and Yanhuang datasets. Conclusions: In this paper, we presented a highly scalable and efficient genome assembly software, SWAPAssembler. Compared with several other assemblers, it showed very good performance in terms of scalability and contig quality. This software is available at: https://sourceforge.net/projects/swapassembler

59 BASIC BIOLOGICAL SCIENCES↗

High-count rate effects in event processing for XRISM/ Resolve X-ray microcalorimeter: I. Ground test

The spectroscopic performance of an X-ray microcalorimeter is compromised at high count rates. We utilize the Resolve X-ray microcalorimeter onboard the XRISM satellite to examine the effects observed during high-count rate measurements and propose modeling approaches to mitigate them. We specifically address the following instrumental effects that impact performance: CPU limit, pile-up, and untriggered electrical cross-talk. Experimental data at high count rates were acquired during ground testing using the flight model instrument and a calibration X-ray source. In the experiment, data processing not limited by the performance of the onboard CPU was run in parallel, which cannot be done in orbit. This makes it possible to access the data degradation caused by limited CPU performance. We use these data to develop models that allow for a more accurate estimation of the aforementioned effects. To illustrate the application of these models in observation planning, we present a simulated observation of GX 13+1. Understanding and addressing these issues is crucial to enhancing the reliability and precision of X-ray spectroscopy in situations characterized by elevated count rates.

47 OTHER INSTRUMENTATION↗

Final Report DE-SC0006997; PI Sharp; Coupled Biological and Micro-XAS/XRF Analysis of In Situ Uranium Biogeochemical Processes

Project Overview: The impact of the original seed award was substantially increased by leveraging a postdoctoral fellowship (Marie Curie Postdoctoral Fellowship) and parallel funds from (A) synergistic project supported by NSF and (B) with DOE collaborators (PI’s Ranville and Williams) as well as no-cost extension that greatly increased the impact and publications associated with the project. In aligning with SBR priorities, the project’s focus was extended more broadly to explore coupled biogeochemical analysis of metal (im)mobilization processes beyond uranium with a foundation in integrating microbial ecology with geochemical analyses. This included investigations of arsenic and zinc during sulfate reducing conditions in addition to direct microbial reduction of metals. Complimentary work with NSF funding and collaborative DOE interactions further increased the project scope to investigate metal (im)mobilization coupled to biogeochemical perturbations in forest ecosystems with an emphasis on coupled carbon and metal biogeochemistry. In total, the project was highly impactful and resulted in 9 publications and directly supported salary/tuition for 3 graduate students at various stages of their academic careers as well as my promotion to Associate Professor. In going forward, findings provided inspiration for a two subsequent proposals with collaborators at Lawrence Berkeley Laboratory and others that are currently in review (as of March 2016).

54 ENVIRONMENTAL SCIENCES↗

The LISE package: solvers for static and time-dependent superfluid local density approximation equations in three dimensions

Nuclear implementation of the density functional theory (DFT) is at present the only microscopic framework applicable to the whole nuclear landscape. The extension of DFT to superfluid systems in the spirit of the Kohn-Sham approach, the superfluid local density approximation (SLDA) and its extension to time-dependent situations, time-dependent superfluid local density ap- proximation (TDSLDA), have been extensively used to describe various static and dynamical problems in nuclear physics, neutron star crust, and cold atom systems. In this paper, we present the codes that solve the static and time-dependent SLDA equations in three-dimensional coordinate space without any symmetry restriction. These codes are fully parallelized with the message passing interface (MPI) library and take advantage of graphic processing units (GPU) for accelerating execution. The dynamic codes have checkpoint/restart capabilities and for initial conditions one can use any generalized Slater determinant type of wave function. The code can describe a large number of physical problems: nuclear fission, collisions of heavy ions, the interaction of quantized vor- tices with nuclei in the nuclear star crust, excitation of superfluid fermion systems by time dependent external fields, quantum shock waves, domain wall generation and propagation, the dynamics of the Anderson-Bogoliubov-Higgs mode, dynamics of fragmented condensates, vortex rings dynamics, generation and dynamics of quantized vortices, their crossing and recombinations and the incipient phases of quantum turbulence.

Jin, Shi↗

A PETSc Parallel-in-Time Solver based on MGRIT algorithm

We address the development of a modular implementation of the MGRIT (MultiGrid-In-Time) algorithm to solve linear and nonlinear systems that arise from the discretization of evolutionary models with a parallel-in-time approach in the context of the PETSc (the Portable, Extensible Toolkit for Scientific computing) library. Our aim is to give the opportunity of predicting the performance gain achievable when using the MGRIT approach instead of the Time Stepping integrator (TS). To this end, we analyze the performance parameters of the algorithm that provide a-priori the best number of processing elements and grid levels to use to address the scaling of MGRIT, regarded as a parallel iterative algorithm proceeding along the time dimension.

Mele, Valeria↗

FISH: A THREE-DIMENSIONAL PARALLEL MAGNETOHYDRODYNAMICS CODE FOR ASTROPHYSICAL APPLICATIONS

FISH is a fast and simple ideal magnetohydrodynamics code that scales to {approx}10,000 processes for a Cartesian computational domain of {approx}1000{sup 3} cells. The simplicity of FISH has been achieved by the rigorous application of the operator splitting technique, while second-order accuracy is maintained by the symmetric ordering of the operators. Between directional sweeps, the three-dimensional data are rotated in memory so that the sweep is always performed in a cache-efficient way along the direction of contiguous memory. Hence, the code only requires a one-dimensional description of the conservation equations to be solved. This approach also enables an elegant novel parallelization of the code that is based on persistent communications with MPI for cubic domain decomposition on machines with distributed memory. This scheme is then combined with an additional OpenMP parallelization of different sweeps that can take advantage of clusters of shared memory. We document the detailed implementation of a second-order total variation diminishing advection scheme based on flux reconstruction. The magnetic fields are evolved by a constrained transport scheme. We show that the subtraction of a simple estimate of the hydrostatic gradient from the total gradients can significantly reduce the dissipation of the advection scheme in simulations of gravitationally bound hydrostatic objects. Through its simplicity and efficiency, FISH is as well suited for hydrodynamics classes as for large-scale astrophysical simulations on high-performance computer clusters. In preparation for the release of a public version, we demonstrate the performance of FISH in a suite of astrophysically orientated test cases.

79 ASTRONOMY AND ASTROPHYSICS↗

Parallel PIPS-SBB: multi-level parallelism for stochastic mixed-integer programs

PIPS-SBB is a distributed-memory parallel solver with a scalable data distribution paradigm. It is designed to solve MIPs with a dual-block angular structure, which is characteristic of deterministic-equivalent Stochastic Mixed-Integer Programs (SMIPs). In this paper, we present two different parallelizations of Branch & Bound (B&B), implementing both as extensions of PIPS-SBB, thus adding an additional layer of parallelism. In the first of the proposed frameworks, PIPS-PSBB, the coordination and load-balancing of the different optimization workers is done in a decentralized fashion. This new framework is designed to ensure all available cores are processing the most promising parts of the B&B tree. The second, ug[PIPS-SBB,MPI], is a parallel implementation using the Ubiquity Generator (UG), a universal framework for parallelizing B&B tree search that has been sucessfully applied to other MIP solvers. We show the effects of leveraging multiple levels of parallelism in potentially improving scaling performance beyond thousands of cores.

97 MATHEMATICS AND COMPUTING↗

Multi-Source Machine Learning and Thermoplastics Enhanced Aerostructure Manufacturing (mTEAM)

RTX Technology Research Center (RTRC), together with Collins Aerospace (Collins) and Oak Ridge National Laboratory (ORNL) has developed an Artificial Intelligence (AI) / Machine Learning (ML) guided solution to advance the manufacturing and assembly of high performance and lightweight thermoplastic composite (TPC) aerospace products. The solution aims to lower risk, cost and lead time for induction heating based welding and consolidation processes for TPC structure. The cost and lead time of part and material specific process development for induction welding (IW) and induction consolidation will be reduced by replacing traditional empirical methods with optimization methods that merge AI/ML and physics-based process simulations and process experiments with sensing and controls. TPC-IW process development is empirical in nature, and uncertainties in material & process behavior exist near & far from the induction coil. Physics-based simulations can be leveraged directly for process optimization but can be too computationally expensive to run in high fidelity and real time to do robust process optimization. The key impact of successful TPC induction consolidation and welding is cost & lead time reduction for part & material specific consolidation and welding recipes. This is an enabler for more rapid deployment of TPC structures via joining assembly, which can reduce energy & cost intensive usage of autoclaves & ovens. The solution aimed to advance the U.S. Department of Energy’s interests in using thermoplastics and automation in composite manufacturing for improvement of products for existing markets via increased production speeds, reduced costs, and lowered use of energy. Welded TPC structures can offer significant weight & energy savings for high-value commercial aerospace & industrial applications compared to metal & thermoset composite structures assembled by mechanical fastening and/or adhesive bonding. The project was organized into two Budget Periods. Budget Period 1 (BP1) was 15 months and its goal was to perform ML process optimization framework development & deployment on lab-coupon aerostructure components. A Go/No-Go Review was performed at the end of BP1 to verify fulfilment of key tasks & milestones to justify a Go Decision to move into the next Budget Period. Budget Period 2 (BP2) was 12 months and its goal was the deployment of the ML framework for ML process optimization of pilot industrial scale aerostructure components. The overall project aim was to develop & demonstrate ML-enhanced modeling framework that learns process-property mapping from multiple data sources at different fidelities. During BP1, the team accomplished key tasks & milestones to demonstrate the concept of multi-source ML for TPC aerostructure consolidation and assembly. First, the team completed documentation of induction based TPC heating requirements including baseline metrics to compare measured results against. Next the team completed demonstration of data generation from physics-based simulations for ML surrogate model generation and demonstrated the integration of physics-based simulation data into multi-source AI/ML algorithms. In parallel, the team established the lab-coupon scale induction welding system and completed a process to label and reduce generated data from physics-based simulation and experiments for ML surrogate models to enable multi-source ML model training & testing. To complete BP1, the team integrated physics-based simulation data and experimental data into multi-source ML algorithms. This was based on the team completing ML deployment of the induction welding on a lab system at RTRC and AI/ML deployment on existing induction welding line at Collins. ORNL visited both Collins and RTRC sites to witness the TPC induction welding process. Then, ORNL designed and constructed a new version of their vision-based sensing system better adapted to acquire process signals of the TPC induction welding process for process anomaly and defect detection. In BP2, the team accomplished key tasks & milestones to scale up multi-source ML for TPC aerostructure consolidation and assembly from the lab-coupon scale to the pilot-industrial scale. In BP2, the team demonstrated real time anomaly & defect detection via experiments performed by ORNL & RTRC. The team completed ML-optimization heating trials for TPC induction consolidation at Collins, and the team confirmed pilot industrial scale experimental data from Collins was compatible with the developed ML pipeline from RTRC. The team completed sub-element scale ML process optimization demonstration at RTRC, where the team leveraged RTRC’s robotic TPC welding setup to de-risk the ML process optimization by performing ML analysis of recorded temperatures to account for complex part features. Then, the team applied its ML-derived control strategies and ML process optimization framework at Collins to the pilot-industrial scale on a demo skin-stiffener part representative of a nacelle aerostructure fan cowl section. The key innovation is the AI/ML framework enabling effective process development of high performance, lightweight, energy efficient TPCs for composite aircraft structures.

36 MATERIALS SCIENCE↗

End-to-end data movement using MPI-IO over routed terabits infrastructures. In: NDM '13 Proceedings of the Third International Workshop on Network-Aware Data Management, Article No. 9

Scientific discovery is nowadays driven by large-scale simulations running on massively parallel high-performance computing (HPC) systems. These applications each generate a large amount of data, which then needs to be post-processed for example for data mining or visualization. Unfortunately, the computing platform used for post processing might be different from the one on which the data is initially generated, introducing the challenge of moving large amount of data between computing platforms. This is especially challenging when these two platforms are geographically separated since the data needs to be moved between computing facilities. This is even more critical when scientists tightly couple their domain specific applications with a post processing application. The paper presents a solution for the data transfer between MPI applications using a dedicated wide area network (WAN) terabit infrastructure. The proposed solution is based on parallel access to data files and the Message Passing Interface (MPI) over the Common Communication Infrastructure (CCI) for the data transfer over a routed infrastructure. In the context of this research, the Energy Sciences Network (ESnet) of the U.S. Department of Energy (DOE) is targeted for the transfer of data between DOE national laboratories.

Vallée, Geoffroy↗