Search NASA⌕ Search

SEARCH · Search NASA

Results for “parallel, scaling”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 757 records · Page 42
All sourcesNASA NTRSDOE OSTI
DOE OSTI2026

Advancing specialized biofoundries via automated adaptive laboratory evolution

Adaptive laboratory evolution (ALE) is a powerful strategy for improving microbial phenotypes by harnessing natural selection under defined environmental conditions. Through applying selection regimes, beneficial mutations accumulate, enabling the generation of strains with enhanced properties. However, conventional ALE is labor-intensive and difficult to scale, limiting reproducibility and broader discovery of evolutionary principles. Recent advances in robotics, automation, and computational infrastructure are transforming ALE into a scalable, data-rich experimental paradigm. Automated platforms enable standardized and complex protocols, real-time monitoring, and highly parallel evolution campaigns, improving consistency while generating longitudinal datasets that reveal convergent adaptive mechanisms. Here, we discuss the role of specialized biofoundries in advancing automated ALE and enabling large-scale evolutionary engineering. We review major automated ALE formats and outline key design principles for effective ALE biofoundries, highlighting how automated ALE can support autonomous experimentation and AI-guided strain engineering.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2022

The Astronomy Commons Platform: A Deployable Cloud-based Analysis Platform for Astronomy

Abstract We present a scalable, cloud-based science platform solution designed to enable next-to-the-data analyses of terabyte-scale astronomical tabular data sets. The presented platform is built on Amazon Web Services (over Kubernetes and S3 abstraction layers), utilizes Apache Spark and the Astronomy eXtensions for Spark for parallel data analysis and manipulation, and provides the familiar JupyterHub web-accessible front end for user access. We outline the architecture of the analysis platform, provide implementation details and rationale for (and against) technology choices, verify scalability through strong and weak scaling tests, and demonstrate usability through an example science analysis of data from the Zwicky Transient Facility’s 1Bn+ light-curve catalog. Furthermore, we show how this system enables an end user to iteratively build analyses (in Python) that transparently scale processing with no need for end-user interaction. The system is designed to be deployable by astronomers with moderate cloud engineering knowledge, or (ideally) IT groups. Over the past 3 yr, it has been utilized to build science platforms for the DiRAC Institute, the ZTF partnership, the LSST Solar System Science Collaboration, and the LSST Interdisciplinary Network for Collaboration and Computing, as well as for numerous short-term events (with over 100 simultaneous users). In a live demo instance, the deployment scripts, source code, and cost calculators are accessible. 4 4 http://hub.astronomycommons.org/

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2019

Extreme Scale De Novo Metagenome Assembly

Metagenome assembly is the process of transforming a set of short, overlapping, and potentially erroneous DNA segments from environmental samples into the accurate representation of the underlying microbiomes's genomes. State-of-the-art tools require big shared memory machines and cannot handle contemporary metagenome datasets that exceed Terabytes in size. In this paper, we introduce the MetaHipMer pipeline, a high-quality and high-performance metagenome assembler that employs an iterative de Bruijn graph approach. MetaHipMer leverages a specialized scaffolding algorithm that produces long scaffolds and accommodates the idiosyncrasies of metagenomes. MetaHipMer is end-to-end parallelized using the Unified Parallel C language and therefore can run seamlessly on shared and distributed-memory systems. Experimental results show that MetaHipMer matches or outperforms the state-of-the-art tools in terms of accuracy. Ultimately, MetaHipMer scales efficiently to large concurrencies and is able to assemble previously intractable grand challenge metagenomes. We demonstrate the unprecedented capability of MetaHipMer by computing the first full assembly of the Twitchell Wetlands dataset, consisting of 7.5 billion reads - size 2.6 TBytes.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2021

Implementing a neural network interatomic model with performance portability for emerging exascale architectures

The two main thrusts of computational science are increasingly accurate predictions and faster calculations; to this end, the zeitgeist in molecular dynamics (MD) simulations is pursuing machine learned and data driven interatomic models, e.g. neural network potentials, and novel hardware architectures, e.g. GPUs. Current implementations of neural network potentials are orders of magnitude slower than traditional interatomic models and while looming exascale computing offers the ability to run large, accurate simulations with these models, achieving portable performance for MD with new and varied exascale hardware requires rethinking traditional algorithms, using novel data structures, and library solutions. We re-implement a neural network interatomic model in CabanaMD, an MD proxy application, built on libraries developed for performance portability. Our implementation shows significantly improved thread scaling in this complex kernel as compared to a current LAMMPS implementation, across both strong and weak scaling. Our single-source solution enables simulations up to 20 million atoms on a single CPU node and 4 million atoms with improved performance on a single GPU. Furthermore, we also explore parallelism and data layout choices (using flexible data structures called AoSoAs) and their effect on performance, seeing up to ~50% and ~5% improvements in performance on a GPU by choosing the right level of parallelism and data layout respectively.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2020

Toward Large-Scale Image Segmentation on Summit

Semantic segmentation of images is an important computer vision task that emerges in a variety of application domains such as medical imaging, robotic vision and autonomous vehicles to name a few. While these domain-specific image analysis tasks involve relatively small image sizes (~ 10 2 × 10 2 ), there are many applications that need to train machine learning models on image data with extents that are orders of magnitude larger (~10 4 × 10 4 ). Training deep neural network (DNN) models on large extent images is extremely memory-intensive and often exceeds the memory limitations of a single graphical processing unit, a hardware accelerator of choice for computer vision workloads. Here, an efficient, sample parallel approach to train U-Net models on large extent image data sets is presented. Its advantages and limitations are analyzed and near-linear strong-scaling speedup demonstrated on 256 nodes (1536 GPUs) of the Summit supercomputer. Using a single node of the Summit supercomputer, an early evaluation of a recently released model parallel framework called GPipe is demonstrated to deliver ~ 2X speedup in executing a U-Net model with an order of magnitude larger number of trainable parameters than reported before. Performance bottlenecks for pipelined training of U-Net models are identified and mitigation strategies to improve the speedups are discussed. Together, these results open up the possibility of combining both approaches into a unified scalable pipelined and data parallel algorithm to efficiently train U-Net models with very large receptive fields on data sets of ultra-large extent images.

Seal, Sudip↗
DOE OSTI2021

Proactive Operations and Investment Planning via Stochastic Optimization to Enhance Power Systems’ Extreme Weather Resilience

We present scalable stochastic optimization approaches for improving power systems’ resilience to extreme weather events. We consider both proactive redispatch and transmission line hardening as alternatives for mitigating expected load shed due to extreme weather, resulting in large-scale stochastic linear programs (LPs) and mixed-integer linear programs (MILPs). We solve these stochastic optimization problems with progressive hedging (PH), a parallel, scenario-based decomposition algorithm. Our computational experiments indicate that our proposed method for enhancing power system resilience can provide high-quality solutions efficiently. With up to 128 scenarios on a 2,000-bus network, the operations (redispatch) and investment (hardening) resilience problems can be solved in approximately 6 min and 2 h of wall-clock time, respectively. Additionally, we solve the investment problems with up to 512 scenarios, demonstrating that the approach scales very well with the number of scenarios. Moreover, the method produces high quality solutions that result in statistically significant reductions in expected load shed. Our proposed approach can be augmented to incorporate a variety of other operational and investment resilience strategies, or a combination of such strategies.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2019

XScan: An Integrated Tool for Understanding Open Source Community-Based Scientific Code

Many scientific communities have adopted community-based models that integrate multiple components to simulate whole system dynamics. The community software projects’ complexity, stems from the integration of multiple individual software components that were developed under different application requirements and various machine architectures, has become a challenge for effective software system understanding and continuous software development. The paper presents an integrated software toolkit called X-ray Software Scanner (in abbreviation, XScan) for a better understanding of large-scale community-based scientific codes. Our software tool provides support to quickly summarize the overall information of scientific codes, including the number of lines of code, programming languages, external library dependencies, as well as architecture-dependent parallel software features. The XScan toolkit also realizes a static software analysis component to collect detailed structural information and provides an interactive visualization and analysis of the functions. We use a large-scale community-based Earth System Model to demonstrate the workflow, functions and visualization of the toolkit. We also discuss the application of advanced graph analytics techniques to assist software modular design and component refactoring.

Zheng, Weijian↗
DOE OSTI2017

Shock-driven transition to turbulence: Emergence of power-law scaling

Here, we consider two cases of interaction between a planar shock and a cylindrical density interface. In the first case (planar normal shock), the axis of the gas cylinder is parallel to the shock front and baroclinic vorticity deposited by the shock is predominantly two dimensional (directed along the axis of the cylinder). In the second case, the cylinder is tilted, resulting in an oblique shock interaction and a fully-three-dimensional shock-induced vorticity field. Furthermore, the statistical properties of the flow for both cases are analyzed based on images from two orthogonal visualization planes, using structure functions of the intensity maps of fluorescent tracer premixed with heavy gas. And at later times, these structure functions exhibit power-law-like behavior over a considerable range of scales. Manifestation of this behavior is remarkably consistent in terms of dimensionless time τ defined based on Richtmyer's linear theory within the range of Mach numbers from 1.1 to 2.0 and the range of gas cylinder tilt angles with respect to the plane of the shock front (0–30°).

42 ENGINEERING↗
DOE OSTI2021

MemXCT: Design, Optimization, Scaling, and Reproducibility of X-Ray Tomography Imaging

Here, this work extends our previous research entitled "MemXCT: Memory-centric X-ray CT Reconstruction with Massive Parallelization" that was originally published at SC19 conference (Hidayetoglu et al., 2019) with reproducibility of the computational imaging performance. X-ray computed tomography (XCT) is regularly used at synchrotron light sources to study the internal morphology of materials at high resolution. However, experimental constraints, such as radiation sensitivity, can result in noisy or undersampled measurements. Further, depending on the resolution, sample size and data acquisition rates, the resulting noisy dataset can be in the order of terabytes. Advanced iterative reconstruction techniques can produce high-quality images from noisy measurements, but their computational requirements have made their use an exception rather than the rule. We propose a novel memory-centric approach that avoids redundant computations at the expense of additional memory complexity. We develop a memory-centric iterative reconstruction system, MemXCT, that uses an optimized SpMV implementation with two-level pseudo-Hilbert ordering and multi-stage input buffering. We evaluate MemXCT on various supercomputer architectures involving KNL and GPU. MemXCT can reconstruct a large (11Kx11K) mouse brain tomogram in 10 seconds using 4096 KNL nodes (256K cores). The results presented in our original article at the SC19 were based on large-scale supercomputing resources. The MemXCT application was selected for the Student Cluster Competition (SCC) Reproducibility Challenge and evaluated on a variety of cloud computing resources by universities around the world in the SC20 conference. We summarize the results of the top-ranked SCC Reproducibility Challenge teams and identify the most pertinent measures for ensuring the reproducibility of our experiments in this article.

47 OTHER INSTRUMENTATION↗
DOE OSTI2021

Dependence of Solar Wind Proton Temperature on the Polarization Properties of Alfvénic Fluctuations at Ion-kinetic Scales

In this work, we use fluctuating magnetic helicity to investigate the polarization properties of Alfvénic fluctuations at ion-kinetic scales in the solar wind as a function of β p , the ratio of proton thermal pressure to magnetic pressure, and θ vB , the angle between the proton flow and local mean magnetic field, B 0 . Using almost 15 yr of Wind observations, we separate the contributions to helicity from fluctuations with wavevectors, k, quasi-parallel and oblique to B 0 , finding that the helicity of Alfvénic fluctuations is consistent with predictions from linear Vlasov theory. This result suggests that the nonlinear turbulent fluctuations at these scales share at least some polarization properties with Alfvén waves. Furthermore, we also investigate the dependence of proton temperature in the β p –θ vB plane to probe for possible signatures of turbulent dissipation, finding that it correlates with θ vB . The proton temperature parallel to B 0 is higher in the parameter space where we measure the helicity of right-handed Alfvénic fluctuations, and the temperature perpendicular to B 0 is higher where we measure left-handed fluctuations. This finding is inconsistent with the general assumption that by sampling different θ vB in the solar wind we can analyze the dependence of the turbulence distribution on θ kB , the angle between k and B 0 . After ruling out both instrumental and expansion effects, we conclude that our results provide new evidence for the importance of local kinetic processes that depend on θ vB in determining proton temperature in the solar wind.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2018

Distributed X-ray photon correlation spectroscopy data reduction using Hadoop MapReduce

Multi-speckle X-ray photon correlation spectroscopy (XPCS) is a powerful technique for characterizing the dynamic nature of complex materials over a range of time scales. XPCS has been successfully applied to study a wide range of systems. Recent developments in higher-frame-rate detectors, while aiding in the study of faster dynamical processes, creates large amounts of data that require parallel computational techniques to process in near real-time. Here, an implementation of the multi-tau and two-time autocorrelation algorithms using the Hadoop MapReduce framework for distributed computing is presented. In conclusion, the system scales well with regard to the increase in the data size, and has been serving the users of beamline 8-ID-I at the Advanced Photon Source for near real-time autocorrelations for the past five years.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
DOE OSTI2018

Modeling of Dendritic Evolution of Continuously Cast Steel Billet with Cellular Automaton

In order to predict the dendritic evolution during the continuous steel casting process, a simple mechanism to connect the heat transfer at the macroscopic scale and the dendritic growth at the microscopic scale was proposed in the present work. As the core of the across-scale simulation, a two-dimensional cell automaton (CA) model with a decentered square algorithm was developed and parallelized. Apart from nucleation undercooling and probability, a temperature gradient was introduced to deal with the columnar-to-equiaxed transition (CET) by considering its variation during continuous casting. Based on the thermal history, the dendritic evolution in a 4 mm × 40 mm region near the centerline of a SWRH82B steel billet was predicted. The influences of the secondary cooling intensity, superheat, and casting speed on the dendritic structure of the billet were investigated in detail. The results show that the predicted equiaxed dendritic solidification of Fe-5.3Si alloy and columnar dendritic solidification of Fe-0.45C alloy are consistent with in situ experimental results [Yasuda et al. Int J Cast Metals Res 22:15–21 (2009); Yasuda et al. ISIJ Int 51:402–408 (2011)]. Moreover, the predicted dendritic arm spacing and CET location agree well with the actual results in the billet. The primary dendrite arm spacing of columnar dendrites decreases with increasing secondary cooling intensity, or decreasing superheat and casting speed. Meanwhile, the CET is promoted as the secondary cooling intensity and superheat decrease. However, the CET is not influenced by the casting speed, owing to the adjusting of the flow rate of secondary spray water. Compared with the superheat and casting speed, the secondary cooling intensity can influence the cooling rate and temperature gradient in deeper locations, and accordingly exerts a more significant influence on the equiaxed dendritic structure.

36 MATERIALS SCIENCE↗
DOE OSTI2018

Time-dependent behavioral data from zebrafish reveals novel signatures of chemical toxicity using point of departure analysis

High-content imaging of embryonic zebrafish behavior can be used as a screening approach to rapidly evaluate the relative potential for chemicals to cause toxicity. However, most statistical methods applied to these data so far transform movement values to incidence-based “hits” and calculate lowest effect levels (LELs), which loses individual fish resolution of behavior and defies hazard ranking due to reliance on applied dose levels. Here, we have developed and parallelized a workflow to calculate benchmark dose (BMD) values from dynamic, high-content zebrafish behavior data that scales for high-throughput chemical screening. To appropriately capture the dynamics of zebrafish response from light to dark stimulus, we summarized time-dependent data using both area under the curve and immediate change at the transition point into two novel metrics that characterized abnormal behavior as a function of chemical concentration. The BMD workflow was applied to calculate BMD 10 values of 1,060 ToxCast chemicals for 24 zebrafish endpoints, including behavior, mortality and morphology. The BMD 10 values provided better precision and separation than LELs for clustering chemicals since they were derived from models that best-fit their concentration-response curves. Clustering analysis revealed behavioral signatures as the most sensitive endpoints. High concordance in chemical activity was observed between ToxCast assays and zebrafish behavioral endpoints, however ToxPi analysis indicated that rankings based on in vitro data was not a reliable predictor of in vivo rankings for lower potency chemicals. The methods developed in this work will enable the use of high-content zebrafish behavioral screening data for BMD analysis in toxicological hazard assessment.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2017

Energy-efficient mapping of large-scale workflows under deadline constraints in big data computing systems

Large-scale workflows for big data analytics have become a main consumer of energy in data centers where moldable parallel computing models such as MapReduce are widely applied to meet high computational demands with time-varying computing resources. The granularity of task partitioning in each moldable job of such big data workflows has a significant impact on energy efficiency, which remains largely unexplored. In this paper, we analyze the properties of moldable jobs and formulate a workflow mapping problem to minimize the dynamic energy consumption of a given workflow request under a deadline constraint in big data systems. Since this problem is strongly NP-hard, we design a fully polynomial-time approximation scheme (FPTAS) for a special case with a pipeline-structured workflow on a homogeneous cluster and a heuristic for the generalized problem with an arbitrary workflow on a heterogeneous cluster. The performance superiority of the proposed solution in terms of dynamic energy saving and deadline missing rate is illustrated by extensive simulation results in comparison with existing algorithms, and further validated by real-life workflow implementation and experimental results in Hadoop/YARN systems.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2021

Dual-pace transient heat conduction in vertically aligned carbon nanotube arrays induced by structure separation

Vertically aligned arrays of carbon nanotubes (CNTs) are attractive for a wide range of macroscopic applications which can exploit the remarkable properties of individual nanotubes. In this work, an abnormal behavior of CNT bundles in heat conduction is discovered under the transient electro-thermal characterization. The measured voltage change over the sample shows a dual-pace thermal response (DTR), which could not be fitted using a single thermal diffusivity heat transfer model. Instead, two thermal diffusivities are determined from the DTR phenomenon. Three rounds of cryogenic experiments are conducted to investigate the physics behind DTR phenomenon. It starts to emerge when the temperature is reduced to a certain level. After two rounds of cryogenic experiments, the DTR phenomenon becomes permanent from 295 K to 10 K. The nano-scale structure separation induced by the increased thermal strain leads to macroscale structure separation, which results in two parallel heat conduction paths responsible for the DTR phenomenon. By building a new parallel heat transfer model taking both the transient and steady-state electrical and thermal response into consideration, the area ratio of separated CNTs is determined. Furthermore, this result uncovers the existence of coiled morphology and nano-structure evolution of CNTs under cryogenic state and its effect on thermal and electrical transport, especially on their transient behaviors. Knowledge of the thermal transport in these CNTs arrays is important considering the fact that transient thermal response strongly affects their mechanical, optical, and electrical behaviors.

42 ENGINEERING↗
DOE OSTI2021

The LSBmax algorithm for boosting resilience of electric grids post (N‐2) contingencies

Abstract A computationally improved algorithm is presented to find the best transmission switching (TS) candidate for boosting resilience of electricity grids subject to ( N ‐2) contingencies. Here, resilience is computed as the reduction in load shed after the above‐mentioned ( N‐ ) contingencies. TS is a planned line outage, and past research shows that changing the transmission system's topology changes the power flow and removes post contingency violations. Finding the best TS candidate in a computationally suitable time for effectively boosting resilience is a challenge. The best TS candidate is found using a novel heuristic method by decreasing the search space based on proximity to the bus with the maximum load shedding (LSB). The LSB algorithm is faster than existing algorithms in the literature; and, it is compatible with both the AC and DC optimal power flow formulations. To validate the authors' claims of speedup and accuracy, two metrics are used to analyze the results from the IEEE 39‐bus and 118‐bus systems. Finally, the inherent parallelism of the LSB algorithm is leveraged on a high‐performance computing platform and applied to the large‐scale Polish 2383‐bus test system to validate scalability in both size and speedup in computation time.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2024

Flowing plasma rearrangement in the presence of static perturbing fields

Charged particles interacting with electromagnetic waves have a portion of their energy tied up in wave-driven oscillations. When these waves are localized to the exhaust of linear magnetic confinement systems, this ponderomotive effect can be utilized to enhance particle confinement. The same effect can be derived for particles moving via an E×B drift into a region of a static perturbation to the electromagnetic fields which has a large wave vector component in the direction of the motion. In this work, we use a simplified slab model to self-consistently solve for the electromagnetic fields within the fluid flowing plasma of a static flute-like (k∥=0) perturbation and evaluate the resulting ponderomotive potential. We find that two types of perturbations can exist within the flowing plasma, which are an O wave and an X wave in the frame moving with the fluid. In the case of tenuous plasma, these perturbations are magnetostatic or electrostatic multipole-analog perpendicular to the guiding magnetic field in the lab frame, respectfully. For denser plasmas, the O wave-like perturbation is screened at the electron skin depth scale, and the X wave-like perturbation is a combination of a similar perpendicular electric perturbation and parallel magnetic perturbation. The ponderomotive potential generated in the X wave-like case is gyrofrequency-dependent and can be used as either potential barriers or potential wells, depending on the direction of the flow velocity.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗
DOE OSTI2020

AI-optimized detector design for the future Electron-Ion Collider: the dual-radiator RICH case

Advanced detector R&D requires performing computationally intensive and detailed simulations as part of the detector-design optimization process. Here, we propose a general approach to this process based on Bayesian optimization and machine learning that encodes detector requirements. As a case study, we focus on the design of the dual-radiator Ring Imaging Cherenkov (dRICH) detector under development as a potential component of the particle-identification system at the future Electron-Ion Collider (EIC). The EIC is a US-led frontier accelerator project for nuclear physics, which has been proposed to further explore the structure and interactions of nuclear matter at the scale of sea quarks and gluons. We show that the detector design obtained with our automated and highly parallelized framework outperforms the baseline dRICH design within the assumptions of the current model. Our technique can be applied to any detector R&D, provided that realistic simulations are available.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗