Search NASA⌕ Search

SEARCH · Search NASA

Results for “distributed computing”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 91 records · Page 5

A Scalable Graph Analytics Framework for Programming with Big Data in R (pbdR)

Many disciplines such as biology, economics, engineering, physics, and the social sciences represent their data as graphs to capture patterns, trends, and associations. There are are many commercially available graph libraries in different programming languages to analyze these complex graphs. But there is no distributed graph library package in R - the popular statistical programming language to analyze graphs that bigger than a single machine's memory. Many domain experts prefer R over the numerous other alternatives. Towards this, we present a distributed graph analytics framework for R called programming with big graph using R (pBGR.) Our proposed framework leverages the Programming with Big Data in R (pbdR) ecosystem that provides scalable R packages for distributed computing in data science. We present an early prototype implementation of this framework using the distributed-memory parallel graph library CombBLAS and evaluate the framework's performance on leadership class computing platforms. Our experimental results demonstrate that the proposed framework is capable of performing large-scale parallel graph mining through the easyto-use R language. This enhanced graph processing capability coupled with other statistical tools already available in R, should be valuable to many domain experts.

Hasan, S M Shamimul↗

Scientific Data Management Beyond Traditional Computing Boundaries

Scientific data management is undergoing a fundamental transformation driven by the convergence of artificial intelligence (AI)/machine learning workflows, distributed computing and storage environments, and exponential data growth. Here, we analyze how these developments address current limitations while enabling new capabilities for cross-facility collaboration and AI-driven research.

Widener, Patrick [Oak Ridge National Laboratory (O↗

Use Case-Informed Framework for Utility Cloud Migration

This white paper presents a comprehensive methodology for assessing utilities’ cloud postures and frameworks. It aims to produce a roadmap and strategy for a cloud-enabled grid future, providing guidance for integrators, asset owners, and operators. Instead of offering a yes or no answer for cloud implementation, this framework offers strategic guidance on responsibly preparing for and deploying cloud solutions. This paper delves into cloud-service models pertinent to the electric sector, dissecting the shared responsibility model and elucidating what on-premise infrastructure as a service (IaaS), platform as a service (PaaS), and software as a service (SaaS) entail. A pivotal consideration within the context of the shared responsibility model is the allocation of responsibility for foundational security aspects—a decision that will be informed by a comprehensive risk assessment. The ensuing discussion will present a checklist of certifications necessary for a secure cloud transition, equipping utilities with the knowledge to navigate this digital transformation with confidence and with a strategic roadmap. Furthermore, the paper outlines gaps in understanding the U.S. government’s role in shaping technology development and responsible use. Its purpose is to aid decision-making by offering support for risk-informed solutions that benefit those managing assets and operating in the cloud environment. The primary objective is to enhance the resilience and future readiness of a decarbonized electric grid, with cloud solutions as one viable option. The paper synthesizes information on current and future grid architectures and applications, considering both conservative and progressive energy transitions, along with scalable and distributed computing considerations.

24 POWER TRANSMISSION AND DISTRIBUTION↗

The environmental impact, carbon emissions and sustainability of computing in the ATLAS experiment

ATLAS, a general-purpose experiment at the Large Hadron Collider (LHC), makes use of a large internationally-distributed computing infrastructure, including over 10 6 TB of managed data on disk and tape and almost one million simultaneously running CPU cores. Upgrades for the High-Luminosity LHC (HL-LHC) will increase the required computing resources by a factor of 3–4 by the beginning of the 2030s, and by an order of magnitude before the conclusion of data taking at the beginning of the 2040s. These resources are spread over around 100 computing sites worldwide. Efforts are underway within the experiment to evaluate and mitigate various aspects of the environmental impact of the sites, with the additional long-term goal of making recommendations to the sites that will significantly reduce the total expected environmental impact in the HL-LHC era. These efforts take several forms: building awareness in the experiment community, adjusting aspects of the computing policy, and modifications of data center configurations, either in ways that take advantage of particular features of ATLAS workloads or in generic ways that reduce the environmental impact of the computing resources. This paper describes the ongoing investigations and approaches that have already provided useful and actionable outcomes.

Aad, G. [CNRS/IN2P3] (ORCID:0000000266654934)↗

Numerical Investigation of Enhanced Dehumidification Processes By Using Dielectrophoresis Principles in Moist Airflows

Dispersed particle-laden flows are encountered in many building and industrial applications, such as flow in a fluidized bed, hydrocarbon transportation in pipelines, and the fouling of air-cooled heat exchangers (Kuruneru et al., 2016; Ray et al., 2019; Wang et al., 2019). Computational fluid dynamic (CFD) models have been developed in recent years to depict particle-fluid and particle-particle interactions in laminar or turbulent flows with increasing accuracy and stability. One particular particle-laden system of interest for moisture control is electrically-enhanced condensation in air and water droplet flows. Electrically-enhanced condensation consists of the use of highly charged water droplets injected in the moist air. The droplets become electric seeds that attract polar water vapor molecules to their surfaces and promote condensation. The nucleation and growth of the charged droplets deplete the vapor phase near a droplet, which is compensated for by the dielectrophoresis flow and diffusion. Dielectrophoresis flow involves surrounding vapor at a distance of about 10 to 100 nm for droplets charged by an electrospray compared to ~2 nm for a single electron charge in a droplet. As the vapor molecules collapse on the surface of the droplets, their initial electrical charge decreases with time due to the neutralization of the ions. While the physics of this phenomena is well known, engineering models for predicting the condensation rates are not available. This work computationally investigates dehumidification of moist airflow in a converging rectangular duct. The objective is to develop an engineering model that predicts water vapor condensation by employing dielectrophoresis principles. We construct a Computational Fluid Dynamics (CFD) model of the duct with electrically-enhanced condensation. The model is implemented in the open-source software OpenFOAM. We utilize the Multi-Phase Particle-In-Cell (MP-PIC) method coupled with a Population Balance Equation (PBE) approach to simulate the particle-laden system. This methodology is an Eulerian-Lagrangian approach used to simulate the droplets' behavior in the humid air. The MP-PIC approach (Andrews and O'Rourke, 1996) mitigates the computational cost by parceling several fundamental particles with similar properties (such as types, sizes, and temperature) into one computational particle. Thus, the billions of particles can be substituted by millions of computational particles without significant loss of information. The PBE was considered with the Lagrangian frame to combine the particle distribution function used in MP-PIC (Kim et al., 2020). This approach preserves mass and energy conservation between the phases in the Eulerian and Lagrangian structures. The PBE in this procedure was directly linked to the discrete parcels, making the simulation of the particle distribution computationally efficient and robust. The MP-PIC-PBE approach used in the present work was applied to the dehumidification of air. Water droplets were injected in the air stream and forced to grow according to experimentally derived correlation. The experiments were conducted on a converging duct with the same geometry and boundary conditions used to build the CFD model. This approach enabled us to approximate the effect of dielectrophoresis phenomena on the droplet and air interface. This presentation will discuss the details of the new CFD model built for the duct, the implementation of the model in OpenFOAM CFD programming language, and the experimental validation of the newly developed model. The results revealed a moderate yet measurable increase in droplet diameter due to water vapor condensation at the vapor-liquid interface of the electrically charged droplets' surface. The seed water droplet particles grew in size by capturing the water vapor in the surrounding air. The OpenFOAM model predicted reductions of humidity in the air from 5 to 10 percent.

Yel Mahi, Maliha↗

Flexible User-Defined Domain Decomposition in Kilometer-Scale E3SM Land Model Simulation

The Energy Exascale Earth System Model (E3SM) Land Model (ELM) has been extended to kilometer-scale (km-ELM) resolutions, enabling high-fidelity simulations of terrestrial processes at 1 km x 1 km grid spacing. In ELM, domain decomposition partitions the computational domain across processors, ensuring efficient parallel execution. Currently, round-robin decomposition is applied, providing a straightforward way to distribute computational workload. As ELM continues evolving at the kilometer-scale (km-scale), particularly with integrating lateral flow modeling, decomposition strategies must also account for the increased workload and data movement. This paper introduces a flexible user-defined domain decomposition framework, allowing users to customize domain partitioning based on application requirements. The impact of different decomposition strategies is evaluated across various applications concerning computation, communication, and I/O. Results demonstrate that while 1D partitioning yields superior I/O performance, k-nearest neighbors (KNN) clustering effectively reduces inter-process communication overhead. This study lays the groundwork for scalable partitioning in large-scale land surface simulations, enhancing next-generation Earth system modeling.

Wang, Dali [ORNL] (ORCID:0000000168065108)↗

P SI4 1.4: Open-source software for high-throughput quantum chemistry

PSI4 is a free and open-source ab initio electronic structure program providing implementations of Hartree–Fock, density functional theory, many-body perturbation theory, configuration interaction, density cumulant theory, symmetry-adapted perturbation theory, and coupled-cluster theory. Most of the methods are quite efficient, thanks to density fitting and multi-core parallelism. The program is a hybrid of C++ and Python, and calculations may be run with very simple text files or using the Python API, facilitating post-processing and complex workflows; method developers also have access to most of PSI4 's core functionalities via Python. Job specification may be passed using The Molecular Sciences Software Institute (MolSSI) QCSCHEMA data format, facilitating interoperability. We announce a rewrite of our top-level computation driver, and concomitant adoption of the MolSSI QCARCHIVE INFRASTRUCTURE project, makes the latest version of PSI4 well suited to distributed computation of large numbers of independent tasks. The project has fostered the development of independent software components that may be reused in other quantum chemistry programs.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗

Software Quality Assurance for High Performance Computing Containers

Software containers are a key channel for delivering portable and reproducible scientific software in high performance computing (HPC) environments. HPC environments are different from other types of computing environments primarily due to usage of the message passing interface (MPI) and drivers for specialized hard- ware to enable distributed computing capabilities. This distinction directly impacts how software containers are built for HPC applications and can complicate software quality assurance efforts including portability and performance. This work introduces a strategy for building containers for HPC applications that adopts layering as a mechanism for software quality assurance. The strategy is demonstrated across three different HPC systems, two of them petaflops scale with entirely different interconnect technologies and/or processor chipsets but running the same container. Performance consequences of the containerization strategy are found to be less than 5-14% while still achieving portable and reproducible containers for HPC systems.

97 MATHEMATICS AND COMPUTING↗

Cloud Services Enable Efficient AI-Guided Simulation Workflows across Heterogeneous Resources

Applications which fuse machine learning and simulation are rarely best served by a single computing resource. Highly parallel simulation codes are best deployed on super- computers, while AI tasks used to decide which simulations to perform may be best suited to specialized accelerators. Here we present a Function-as-a-Service (FaaS) system for executing complex, distributed computational campaigns that achieves performance parity with conventional workflow systems without the complexities of secure network connections between compute providers. One innovation enabling high performance is a subsystem that directly moves task data between sites, separate from the cloud-hosted FaaS system used to distribute task instructions. We also introduce a flexible scheduling system that allows us access factor of 2 trade offs between the amount of resources required to solve a problem at each compute site. We anticipate that this system will upgrade multi-site applications from demonstration projects to routine practice in computational science.

Ward, Logan↗

Explainable Neural Architecture Search (XNAS)

Code for the paper Learning Interpretable Models Through Multi-Objective Neural Architecture Search by Zachariah Carmichael, Tim Moon, and Sam Ade Jacobs. Monumental advances in deep learning have led to unprecedented achievements across a multitude of domains. While the performance of deep neural networks is indubitable, the architectural design and interpretability of such models are nontrivial. Research has been introduced to automate the design of neural network architectures through neural architecture search (NAS). Recent progress has made these methods more pragmatic by exploiting distributed computation and novel optimization algorithms. However, there is little work in optimizing architectures for interpretability. To this end, we propose a multiobjective distributed NAS framework that optimizes for both task performance and introspection. We leverage the non-dominated sorting genetic algorithm (NSGA-II) and explainable AI (XAI) techniques to reward architectures that can be better comprehended by humans. The framework is evaluated on several image classification datasets. We demonstrate that jointly optimizing for introspection ability and task error leads to more disentangled architectures that perform within tolerable error.

Carmichael, ZachariahJ↗

Extreme-scale stochastic optimization and simulation via learning-enhanced decomposition and parallelization (Final Technical Report)

Stochastic optimization and simulation models ubiquitously arise in designing and operating complex service/engineering systems. They can be extreme in scale due to high-dimensional data and decisions, and can also involve decisions made sequentially in response to newly revealed data, both causing significant computational challenge. The objective of this research is to explore a unified framework that integrates machine learning with discrete optimization and risk-averse modeling, to improve the efficiency of decomposition paradigms for stochastic optimization and simulations at extreme scale. The models we consider represent a broad class of complex decision-making problems, where 0-1 or continuous decisions are made before and/or after knowing multiple sources of uncertainties that could be correlated. We will employ machine learning methods to dynamically decide and prioritize computational procedures, including cut generation, branching, and bounding of the optimal objective. Furthermore, the research will shed new lights on the traditional decomposition algorithms for extreme-scale computing. Deliverables of the research include new modeling and computational methods for advancing the state-of-the-art research in optimization and simulation, bringing many relevant risk-averse, data-driven optimization problems in practice within the range of tractability. Examples include distributed computing server scheduling and sensor deployment for monitoring critical infrastructures. Success in this effort will enable progress in solving multiple extreme-scale problems in the complex system design and operations arising from DoE missions in energy, environment, and national security.

24 POWER TRANSMISSION AND DISTRIBUTION↗

Path-accelerated stochastic molecular dynamics: Parallel-in-time integration using path integrals

Massively parallel computer architectures create new opportunities for the performance of long-time scale molecular dynamics (MD) simulations. Here, we introduce the path-accelerated molecular dynamics method that takes advantage of distributed computing to reduce the wall-clock time of MD simulation via parallelization with respect to stochastic MD time steps. The marginal distribution for the time evolution of a system is expressed in terms of a path integral, enabling the use of path sampling techniques to numerically integrate MD trajectories. By parallelizing the evaluation of the path action with respect to time and by initializing the path configurations from a nonequilibrium distribution, the algorithm enables significant speedups in terms of the length of MD trajectories that can be integrated in a given amount of wall-clock time. The method is demonstrated for Brownian dynamics, although it is generalizable to other stochastic equations of motion including open systems. We apply the method to two simple systems, a harmonic oscillator and a Lennard-Jones liquid, and we show that in comparison to the conventional Euler integration scheme for Brownian dynamics, the new method can reduce the wall-clock time for integrating trajectories of a given length by more than three orders of magnitude in the former system and more than two in the latter. This new method for parallelizing MD in the dimension of time can be trivially combined with algorithms for parallelizing the MD force evaluation to achieve further speedup.

Rosa-Raíces, Jorge L.↗

How fast can one resize a distributed file system?

Efficient resource utilization becomes a major concern as large-scale distributed computing infrastructures keep growing in size. Malleability, the possibility for resource managers to dynamically increase or decrease the amount of resources allocated to a job, is a promising way to save energy and costs. However, state-of-the-art parallel and distributed storage systems have not been designed with malleability in mind. The reason is mainly the supposedly high cost of data transfers required by resizing operations. Nevertheless, as network and storage technologies evolve, old assumptions about potential bottlenecks can be revisited. In this study, we evaluate the viability of malleability as a design principle for a distributed storage system. We specifically model the minimal duration of the commission and decommission operations. To show how our models can be used in practice, we evaluate the performance of these operations in HDFS, a relevant state-of-the-art distributed file system. We show that the existing decommission mechanism of HDFS is good when the network is the bottleneck, but can be accelerated by up to a factor 3 when storage is the limiting factor. We also show that the commission in HDFS can be substantially accelerated. With the highlights provided by our model, we suggest improvements to speed both operations in HDFS. We discuss how the proposed models can be generalized for distributed file systems with different assumptions and what perspectives are open for the design of efficient malleable distributed file systems.

97 MATHEMATICS AND COMPUTING↗

Toward A High-Performance Emulation Platform for Brain-Inspired Intelligent Systems Exploring Dataflow-Based Execution Model and Beyond

Brain-inspired computing is a novel computing technology based on neural morphological engineering, which draws lessons from methods of human brain information processing and storage. Combining with the high-performance computing (HPC) platform, they constitute the foundation of general artificial intelligence. However, current brain HPC platforms generally suffer from slow speed, poor scalability, and high energy consumption, which severely restrain its potential and circumscribe the development of general artificial intelligence. The dataflow model was first proposed in the 1970s, providing a novel idea for the development of HPC. In addition, the dataflow model shares similar information processing mechanisms with human's neural system, which makes dataflow models naturally suit the emulation of brain-inspired computing. Based on the contemporary progress of the dataflow model, the Codelet model was proposed. Through a fine-grained asynchronous program execution and resource allocation, the Codelet model successfully realized the distributed computing on the heterogeneous system, effectively improved the computing power and speed, and open up a new path to overcome the shortcomings of the existing high-performance computing technology. We propose a dataflow-based emulation platform, aiming at providing high-performance computing technology support for general brain-inspired intelligent system, as well as using characteristics of dataflow models to fully explore the potential of brain-inspired intelligence. As an example, we will select a convolutional neural network (LeNet5) that already has a spectacular user base to initially verify the superiority and feasibility of our proposal.

brain-inspired intelligent system↗

Characterization of ECRAM materials and devices

As the limits of Moore’s Law approaches, new computing paradigms are developing to breakthrough this bottleneck. One such computer architecture is neuromorphic computing, which models the brain. Electrochemical random-access memory (ECRAM) is a low power and energy efficient memory due to characteristics, such as in-memory compute, ion modulation of the channel conductance, and computation distribution with large scale array integration.

97 MATHEMATICS AND COMPUTING↗

Adaptive Runtime Features For Distributed Graph Algorithms

Performance of distributed graph algorithms can benefit greatly by forming rapport between algorithmic abstraction and the underlying runtime system that is responsible for scheduling work and exchanging messages. However, due to their dynamic and irregular nature of computation, distributed graph algorithms written in different programming models impose varying degree of workload pressure on the runtime. To cope with such vastly different workload characteristics, a runtime has to make several trade-offs. One such trade-off arises, for example, when the runtime scheduler has to choose among alternatives such as whether to execute algorithmic work, or progress the network by probing network buffers, or throttle sending messages (termed flow control). This tradeoff decides between optimizing the throughput of a runtime scheduler by increasing the rate of execution of algorithmic work, and reducing the latency of the network messages. Another trade-off exists when a decision has to be made about when to send aggregated messages in buffers (message coalescing). This decision chooses between trading off latency for network bandwidth and vice versa. At any instant, such trade-offs emphasize either on improving the quantity of work being executed (by maximizing the scheduler throughput) or on improving the quality of work (by prioritizing better work). However, encoding static policies for different runtime features (such as flow control, coalescing) can prevent graph algorithms from achieving their full potential, thus can undermine the actual performance of a distributed graph algorithm . In this paper, we investigate runtime support for distributed graph algorithms in the context of two paradigms: variants of wellknown Bulk-Synchronous Parallel model and asynchronous programming model. We explore generic runtime features such as message coalescing (aggregation) and flow control and show that execution policies of these features need to be adjusted over time to make a positive impact on the execution time of a distributed graph algorithm. Since synchronous and asynchronous graph algorithms have different workload characteristics, not all of such runtime features may be good candidates for adaptation. Each of these algorithmic paradigms may require different set of features to be adapted over time. We demonstrate which set of feature(s) can be useful in each case to achieve the right balance of work in the runtime layer. Existing implementation of different graph algorithms can benefit from adapting dynamic policies in the underlying runtime.

Firoz, Jesun S.↗

SORA: Scalable Overlap-graph Reduction Algorithms for Genome Assembly using Apache Spark in the Cloud

The advent of high-throughput DNA sequencing techniques has permitted very high quality de novo assemblies of genomes, but raise an issue of requiring large amounts of computer memory to resolve the large genome graphs generated by most overlap graph de novo assemblers. To address these limitations, we present a novel algorithmic approach; Scalable Overlap-graph Reduction Algorithms (SORA). SORA adapts string graph reduction algorithms for the genome assembly using a distributed computing platform. To efficiently compute coverage for enormous paths in the graphs, SORA uses Apache Spark which is a cluster-based engine designed on top of Hadoop to handle very large datasets in the cloud. The experimental results show that SORA can process a nearly one billion edge graph in a distributed cloud cluster as well as smaller graphs on a local cluster with a short turnaround time. Moreover, our algorithms scale almost linearly with increasing numbers of virtual instances in the cloud. SORA is freely available for download at https://github.com/BioHPC/SORA/.

Paul, Alexander↗

Semi-inclusive deep-inelastic scattering, parton distributions, and fragmentation functions at a future electron-ion collider

We present a quantitative assessment of the impact a future electron-ion collider would have in the determination of parton distribution functions in the proton and parton-to-hadron fragmentation functions through semi-inclusive deep-inelastic electron-proton scattering data. Specifically, we estimate the kinematic regions for which the forthcoming data are expected to have the most significant impact in the precision of these distributions, computing the respective correlation and sensitivity coefficients. Using a reweighting technique for the sets of simulated data with their realistic uncertainties for two different center-of-mass energies, we analyze the resulting new sets of parton distribution functions and fragmentation functions, which have significantly reduced uncertainties.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗