SEARCH · Search NASA

Results for “data intensive computing”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 55 records · Page 3

All sources NASA NTRS DOE OSTI

DOE OSTI2018

Computing environments for reproducibility: Capturing the “Whole Tale”

The act of sharing scientific knowledge is rapidly evolving away from traditional articles and presentations to the delivery of executable objects that integrate the data and computational details (e.g., scripts and workflows) upon which the findings rely. This envisioned coupling of data and process is essential to advancing science but faces technical and institutional barriers. The Whole Tale project aims to address these barriers by connecting computational, data-intensive research efforts with the larger research process transforming the knowledge discovery and dissemination process into one where data products are united with research articles to create "living publications" or tales. The Whole Tale focuses on the full spectrum of science, empowering users in the long tail of science, and power users with demands for access to big data and compute resources. We report here on the design, architecture, and implementation of the Whole Tale environment.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2014

Collective input/output under memory constraints

Compared with current high-performance computing (HPC) systems, exascale systems are expected to have much less memory per node, which can significantly reduce necessary collective input/output (I/O) performance. In this study, we introduce a memory-conscious collective I/O strategy that takes into account memory capacity and bandwidth constraints. The new strategy restricts aggregation data traffic within disjointed subgroups, coordinates I/O accesses in intranode and internode layers, and determines I/O aggregators at run time considering memory consumption among processes. We have prototyped the design and evaluated it with commonly used benchmarks to verify its potential. The evaluation results demonstrate that this strategy holds promise in mitigating the memory pressure, alleviating the contention for memory bandwidth, and improving the I/O performance for projected extreme-scale systems. Given the importance of supporting increasingly data-intensive workloads and projected memory constraints on increasingly larger scale HPC systems, this new memory-conscious collective I/O can have a significant positive impact on scientific discovery productivity.

collective input/output↗

DOE OSTI2015

Simplified predictive models for CO 2 sequestration performance assessment

CO2 sequestration in deep saline formations is increasingly being considered as a viable strategy for the mitigation of greenhouse gas emissions from anthropogenic sources. In this context, detailed numerical simulation based models are routinely used to understand key processes and parameters affecting pressure propagation and buoyant plume migration following CO2 injection into the subsurface. As these models are data and computation intensive, the development of computationally-efficient alternatives to conventional numerical simulators has become an active area of research. Such simplified models can be valuable assets during preliminary CO2 injection project screening, serve as a key element of probabilistic system assessment modeling tools, and assist regulators in quickly evaluating geological storage projects. We present three strategies for the development and validation of simplified modeling approaches for CO2 sequestration in deep saline formations: (1) simplified physics-based modeling, (2) statisticallearning based modeling, and (3) reduced-order method based modeling. In the first category, a set of full-physics compositional simulations is used to develop correlations for dimensionless injectivity as a function of the slope of the CO2 fractional-flow curve, variance of layer permeability values, and the nature of vertical permeability arrangement. The same variables, along with a modified gravity number, can be used to develop a correlation for the total storage efficiency within the CO2 plume footprint. Furthermore, the dimensionless average pressure buildup after the onset of boundary effects can be correlated to dimensionless time, CO2 plume footprint, and storativity contrast between the reservoir and caprock. In the second category, statistical “proxy models” are developed using the simulation domain described previously with two approaches: (a) classical Box-Behnken experimental design with a quadratic response surface, and (b) maximin Latin Hypercube sampling (LHS) based design with a multidimensional kriging metamodel fit. For roughly the same number of simulations, the LHS-based metamodel yields a more robust predictive model, as verified by a k-fold cross-validation approach (with data split into training and test sets) as well by validation with an independent dataset. In the third category, a reduced-order modeling procedure is utilized that combines proper orthogonal decomposition (POD) for reducing problem dimensionality with trajectory-piecewise linearization (TPWL) in order to represent system response at new control settings from a limited number of training runs. Significant savings in computational time are observed with reasonable accuracy from the PODTPWL reduced-order model for both vertical and horizontal well problems – which could be important in the context of history matching, uncertainty quantification and optimization problems. The simplified physics and statistical learning based models are also validated using an uncertainty analysis framework. Reference cumulative distribution functions of key model outcomes (i.e., plume radius and reservoir pressure buildup) generated using a 97-run full-physics simulation are successfully validated against the CDF from 10,000 sample probabilistic simulations using the simplified models. The main contribution of this research project is the development and validation of a portfolio of simplified modeling approaches that will enable rapid feasibility and risk assessment for CO2 sequestration in deep saline formations.

54 ENVIRONMENTAL SCIENCES↗

DOE OSTI2013

EINSTEIN@HOME DISCOVERY OF FOUR YOUNG GAMMA-RAY PULSARS IN FERMI LAT DATA

We report the discovery of four gamma-ray pulsars, detected in computing-intensive blind searches of data from the Fermi Large Area Telescope (LAT). The pulsars were found using a novel search approach, combining volunteer distributed computing via Einstein@Home and methods originally developed in gravitational-wave astronomy. The pulsars PSRs J0554+3107, J1422–6138, J1522–5735, and J1932+1916 are young and energetic, with characteristic ages between 35 and 56 kyr and spin-down powers in the range 6 × 10{sup 34}—10{sup 36} erg s{sup –1}. They are located in the Galactic plane and have rotation rates of less than 10 Hz, among which the 2.1 Hz spin frequency of PSR J0554+3107 is the slowest of any known gamma-ray pulsar. For two of the new pulsars, we find supernova remnants coincident on the sky and discuss the plausibility of such associations. Deep radio follow-up observations found no pulsations, suggesting that all four pulsars are radio-quiet as viewed from Earth. These discoveries, the first gamma-ray pulsars found by volunteer computing, motivate continued blind pulsar searches of the many other unidentified LAT gamma-ray sources.

79 ASTRONOMY AND ASTROPHYSICS↗

DOE OSTI2026

Report of the 2026 Workshop on Next-Generation Ecosystems for Scientific Computing: Harnessing Community, Software, and AI for Cross-Disciplinary Team Science

Scientific computing is undergoing rapid transformation as advances in artificial intelligence, heterogeneous computing, automation, and data-intensive research reshape not only computational tools but also the institutions, workforce models, and collaborative practices that support scientific discovery. This report synthesizes insights from the 2026 Workshop on Next-Generation Ecosystems for Scientific Computing, the second in a three-year series focused on strengthening scientific computing ecosystems through socio-technical co-design. Workshop discussions identified four interdependent strategic themes: software ecosystems for AI-enabled scientific discovery; trust, validation, and traceability; human-AI teaming and paradigm shifts; and workforce, pedagogy, and governance. The report translates these themes into eight priorities for community action spanning shared research infrastructure, trust and traceability, user experience, human-AI teaming, workforce development, cross-sector coordination, stewardship and sustainability, and evaluation of scientific value. Together, these priorities outline directions for building scientific computing ecosystems that remain trustworthy, sustainable, innovative, and resilient as AI assumes a growing role in scientific work.

AI↗

DOE OSTI2019

Active Subspace Methods for Data-Intensive Inverse Problems

Fostered by advances in computing, simulation and data science have emerged as the third and fourth paradigms enabling scientific discovery. Many mathematical challenges arise when integrating these paradigms to produce new knowledge. One of the most important challenges is the inverse or calibration problem: find the inputs of a simulation such that its outputs agree with a given set of measurements. As both the cost of the complex simulations and the volume of measurement data sets increase, intuition-driven trial-and-error methods for calibration quickly become untenable; modern scientists need more computationally efficient and mathematically rigorous methods. When measurements and simulated predictions contain uncertainty, the Bayesian perspective on statistical inference provides a popular mathematical framework for developing computational methods to solve inverse problems. Markov chain Monte Carlo (MCMC) methods are a mainstay of this framework; they characterize a probability distribution on the space of model inputs that is conditioned on the given measurements. However, despite recent advances, MCMC methods remain impractical for expensive simulations with more than a handful of inputs. At its heart, MCMC searches the space of inputs for corresponding outputs that match measurements, and so it is afflicted with the curse of dimensionality. Additionally, MCMC methods become less efficient as the volume of the measurement data increases, particularly if the measurements are redundant. To make MCMC—and Bayesian methods, in general—amenable to today’s computer models with millions of inputs and terabytes of measurement data, one must devise methods to reduce the dimension of both the simulation inputs and the measurements. For many models used in practice, the outputs vary primarily on a low-dimensional manifold of the nominally high-dimensional input space. Recently developed active subspace methods discover this low-dimensional manifold and exploit it to enable analyses typically suited for low-dimensional settings (e.g., statistical inversion of complex systems). We hypothesize that active subspace methods can provide the necessary dimension reduction, enabling existing Bayesian methods to find solutions for a large class of otherwise intractable statistical inverse problems. We will test the hypothesis with a series of mathematical analyses and computational experiments that will proceed along three research veins: (i) integrating active subspace methods into methods for statistical inversion, (ii) addressing current challenges in active subspace methods, and (iii) incorporating methods for nonlinear dimensionality reduction into active subspace methods. We will apply the methods we develop to real inverse problems in chemical kinetics and turbulent flame modeling. Our team combines expertise in methods for inverse problems, dimension reduction, and reduced order modeling. With a history of active collaboration and thoughtful engagement, we bring enthusiasm and fresh perspectives to a well studied subject. We are excited by the potential opportunity to work together to extend and apply active subspace methods to data-intensive inverse problems.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2016

Accurate optimization of amino acid form factors for computing small-angle X-ray scattering intensity of atomistic protein structures

Structure modelling via small-angle X-ray scattering (SAXS) data generally requires intensive computations of scattering intensity from any given biomolecular structure, where the accurate evaluation of SAXS profiles using coarse-grained (CG) methods is vital to improve computational efficiency. To date, most CG SAXS computing methods have been based on a single-bead-per-residue approximation but have neglected structural correlations between amino acids. To improve the accuracy of scattering calculations, accurate CG form factors of amino acids are now derived using a rigorous optimization strategy, termed electron-density matching (EDM), to best fit electron-density distributions of protein structures. This EDM method is compared with and tested against other CG SAXS computing methods, and the resulting CG SAXS profiles from EDM agree better with all-atom theoretical SAXS data. By including the protein hydration shell represented by explicit CG water molecules and the correction of protein excluded volume, the developed CG form factors also reproduce the selected experimental SAXS profiles with very small deviations. Taken together, these EDM-derived CG form factors present an accurate and efficient computational approach for SAXS computing, especially when higher molecular details (represented by the q range of the SAXS data) become necessary for effective structure modelling.

Tong, Dudu↗

DOE OSTI2014

A Large-Scale, High-Resolution Hydrological Model Parameter Data Set for Climate Change Impact Assessment for the Conterminous US

To extend geographical coverage, refine spatial resolution, and improve modeling efficiency, a computation- and data-intensive effort was conducted to organize a comprehensive hydrologic dataset with post-calibrated model parameters for hydro-climate impact assessment. Several key inputs for hydrologic simulation—including meteorologic forcings, soil, land class, vegetation, and elevation—were collected from multiple best-available data sources and organized for 2107 hydrologic subbasins (8-digit hydrologic units, HUC8s) in the conterminous United States at refined 1/24° (~4 km) spatial resolution. Using high-performance computing for intensive model calibration, a high-resolution parameter dataset was prepared for the macro-scale Variable Infiltration Capacity (VIC) hydrologic model. The VIC simulation was driven by DAYMET daily meteorological forcing and was calibrated against USGS WaterWatch monthly runoff observations for each HUC8. The results showed that this new parameter dataset may help reasonably simulate runoff at most US HUC8 subbasins. Based on this exhaustive calibration effort, it is now possible to accurately estimate the resources required for further model improvement across the entire conterminous United States. We anticipate that through this hydrologic parameter dataset, the repeated effort of fundamental data processing can be lessened, so that research efforts can emphasize the more challenging task of assessing climate change impacts. The pre-organized model parameter dataset will be provided to interested parties to support further hydro-climate impact assessment.

Oubeidillah, Abdoul A.↗

DOE OSTI2015

Flexible HPC Environments [Poster]

New computational paradigms require new tools and environments. A new class of data-analysis tools are available for scientists. These tools do not integrate well with traditional HPC resource management systems. Differing and often conflicting requirements exist. Some fields require validated environments including high energy physics and cosmology. By adopting cloud technologies in an HPC environment we are merging the compute-intensive and data-intensive worlds in one converged system. HPC Systems will support a Larger Set of Technologies. Scientists will be able to leverage the latest tools on HPC resources at previously unavailable scales. User-defined environments will enable rapid experimentation and adoption of emerging tools. HPC systems will support a consolidated workflow from simulation to analysis and discovery. HPC systems will have increased utility to a broader set of users.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2020

Accelerating geostatistical modeling using geostatistics-informed machine Learning

Ordinary Kriging (OK) is a popular geostatistical algorithm for spatial interpolation and estimation. The computational complexity of OK changes quadratically and cubically for memory and speed, respectively, given the number of data. Therefore, it is computationally intensive and also challenging to process a large set of data, especially in three-dimensional (3D) cases. This paper develops a geostatistics-informed machine learning (GIML) model to improve the efficiency of OK by reducing the number of points required to be estimated using OK. Specifically, only a very few of the unknown points are estimated by OK to get the weights and estimations, which are used as the training dataset. Moreover, the governing equations of OK are used to guide our proposed machine learning to better reproduce the spatial distributions. Our results show that the proposed GIML can reduce the computational time of OK by at least one order of magnitude. The effectiveness of the GIML is evaluated and compared using a 2D case. Furthermore, we demonstrate its efficiency and robustness by considering a different number of training samples on various 3D simulation grids.

58 GEOSCIENCES↗

DOE OSTI2016

The Medical Science DMZ

Objective We describe use cases and an institutional reference architecture for maintaining high-capacity, data-intensive network flows (e.g., 10, 40, 100 Gbpsþ) in a scientific, medical context while still adhering to security and privacy laws and regulations. Materials and Methods High-end networking, packet filter firewalls, network intrusion detection systems. Results We describe a “Medical Science DMZ” concept as an option for secure, high-volume transport of large, sensitive data sets between research institutions over national research networks. Discussion The exponentially increasing amounts of “omics” data, the rapid increase of high-quality imaging, and other rapidly growing clinical data sets have resulted in the rise of biomedical research “big data.” The storage, analysis, and network resources required to process these data and integrate them into patient diagnoses and treatments have grown to scales that strain the capabilities of academic health centers. Some data are not generated locally and cannot be sustained locally, and shared data repositories such as those provided by the National Library of Medicine, the National Cancer Institute, and international partners such as the European Bioinformatics Institute are rapidly growing. The ability to store and compute using these data must therefore be addressed by a combination of local, national, and industry resources that exchange large data sets. Maintaining data-intensive flows that comply with HIPAA and other regulations presents a new challenge for biomedical research. Recognizing this, we describe a strategy that marries performance and security by borrowing from and redefining the concept of a “Science DMZ”—a framework that is used in physical sciences and engineering research to manage high-capacity data flows. Conclusion By implementing a Medical Science DMZ architecture, biomedical researchers can leverage the scale provided by high-performance computer and cloud storage facilities and national high-speed research networks while preserving privacy and meeting regulatory requirements.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2020

Usable Data Abstractions for Next-Generation Scientific Workflows

Data- and computationally-intensive scientific research, such as numerical simulations and inversions or the training of large neural networks in machine learning applications, that are well suited for HPC environments also often require expert insight and evaluation throughout the computation which can be greatly facilitated with the use of interactive computing tools, such as those in the Jupyter ecosystem. HPC workflows and interactive workflows are typically treated as orthogonal, however, the next generation of research will require both. The first challenge we face in this project is thus designing the right level of abstractions to allow interactive capabilities in the JupyterLab environment to allow the working scientist to flexibly explore and query their data at multiple levels, with a minimal amount of customization required of the underlying optimized codes. In addition to these questions regarding the high-level representation of data for interactive use in HPC, we tackled two additional issues that are part of the entire lifecycle of research and that become particularly acute in HPC contexts: how to improve the experience of interfacing with the HPC system's scheduling environment for a scientist focused on exploratory questions, and how can that scientist then best share the results of their work with others in a self-contained, reproducible manner.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2026

Machine Learning Based Metamodel for Faster Life Cycle Assessment of Large Portfolio of Buildings

Managing a large portfolio of buildings involves decisions on reuse, retrofit, renovation, rehabilitation, and new construction, influenced by trade-offs between performance metrics such as cost, time, and operational flexibility over the building's life cycle. Traditional life cycle assessment tools for evaluating these metrics can be labor- and compute-intensive, requiring extensive data and modeling for each building. Metamodels (or surrogate models) using machine learning have been explored as faster alternatives, but training these models has been hindered by the limited availability of comprehensive data on key life cycle metrics. Recent advancements in machine learning, particularly deep learning techniques like zero-shot and few-shot learning, allow models to learn from sparse or limited data. We propose a machine learning-based metamodel that leverages these techniques for rapid estimation of key building life cycle metrics. This presentation will cover the model architecture, data collection, training, and validation processes, along with an ongoing case study applied to a large portfolio of buildings. We will discuss the model's performance in terms of accuracy, compute time, limitations, and its potential for expanding to additional life cycle metrics. This data-driven approach offers a promising direction for the rapid evaluation of large building portfolios.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

DOE OSTI2015

Large-Scale Compute-Intensive Analysis via a Combined In-situ and Co-scheduling Workflow Approach

Large-scale simulations can produce tens of terabytes of data per analysis cycle, complicating and limiting the efficiency of workflows. Traditionally, outputs are stored on the file system and analyzed in post-processing. With the rapidly increasing size and complexity of simulations, this approach faces an uncertain future. Trending techniques consist of performing the analysis in situ, utilizing the same resources as the simulation, and/or off-loading subsets of the data to a compute-intensive analysis system. We introduce an analysis framework developed for HACC, a cosmological N-body code, that uses both in situ and co-scheduling approaches for handling Petabyte-size outputs. An initial in situ step is used to reduce the amount of data to be analyzed, and to separate out the data-intensive tasks handled off-line. The analysis routines are implemented using the PISTON/VTK-m framework, allowing a single implementation of an algorithm that simultaneously targets a variety of GPU, multi-core, and many-core architectures.

Big data↗

DOE OSTI2023

Distributed Quantum Learning with co-Management in a Multi-tenant Quantum System

The rapid advancement of quantum computing has pushed classical designs into the quantum domain, breaking physical boundaries for computing-intensive and data-hungry applications with the hope that some systems may provide a quantum speedup. For example, variational quantum algorithms have been proposed for quantum neural networks to train deep learning models on qubits, achieving promising results. Existing quantum learning architectures and systems rely on single, monolithic quantum machines with abundant and stable resources, such as qubits. However, fabricating a large, monolithic quantum device is considerably more challenging than producing an array of smaller devices. In this paper, we investigate a distributed quantum system that combines multiple quantum machines into a unified system. We propose DQuLearn, which divides a quantum learning task into multiple subtasks. Each subtask can be executed distributively on individual quantum machines, with the results looping back to classical machines for subsequent training iterations. Additionally, our system supports multiple concurrent clients and dynamically manages their circuits according to the runtime status of quantum workers. Through extensive experiments, we demonstrate that DQuLearn achieves similar accuracies with significant runtime reduction, by up to 68.7% and an increase per-second circuit processing speed, by up to 3.99 times, in a 4-worker multi-tenant setting.

quantum computing↗

DOE OSTI2019

Active Subspace Methods for Data-Intensive Inverse Problems (Final Report)

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2015

Load‐balanced and locality‐aware scheduling for data‐intensive workloads at extreme scales

Summary Data‐driven programming models such as many‐task computing (MTC) have been prevalent for running data‐intensive scientific applications. MTC applies over‐decomposition to enable distributed scheduling. To achieve extreme scalability, MTC proposes a fully distributed task scheduling architecture that employs as many schedulers as the compute nodes to make scheduling decisions. Achieving distributed load balancing and best exploiting data locality are two important goals for the best performance of distributed scheduling of data‐intensive applications. Our previous research proposed a data‐aware work‐stealing technique to optimize both load balancing and data locality by using both dedicated and shared task ready queues in each scheduler. Tasks were organized in queues based on the input data size and location. Distributed key‐value store was applied to manage task metadata. We implemented the technique in MATRIX, a distributed MTC task execution framework. In this work, we devise an analytical suboptimal upper bound of the proposed technique, compare MATRIX with other scheduling systems, and explore the scalability of the technique at extreme scales. Results show that the technique is not only scalable but can achieve performance within 15% of the suboptimal solution. Copyright © 2015 John Wiley & Sons, Ltd.

Wang, Ke↗

DOE OSTI2015

Integration of Russian Tier-1 Grid Center with High Performance Computers at NRC-KI for LHC experiments and beyond HENP

The LHC experiments are preparing for the precision measurements and further discoveries that will be made possible by higher LHC energies from April 2015 (LHC Run2). The need for simulation, data processing and analysis would overwhelm the expected capacity of grid infrastructure computing facilities deployed by the Worldwide LHC Computing Grid (WLCG). To meet this challenge the integration of the opportunistic resources into LHC computing model is highly important. The Tier-1 facility at Kurchatov Institute (NRC-KI) in Moscow is a part of WLCG and it will process, simulate and store up to 10% of total data obtained from ALICE, ATLAS and LHCb experiments. In addition Kurchatov Institute has supercomputers with peak performance 0.12 PFLOPS. The delegation of even a fraction of supercomputing resources to the LHC Computing will notably increase total capacity. In 2014 the development a portal combining a Tier-1 and a supercomputer in Kurchatov Institute was started to provide common interfaces and storage. The portal will be used not only for HENP experiments, but also by other data- and compute-intensive sciences like biology with genome sequencing analysis; astrophysics with cosmic rays analysis, antimatter and dark matter search, etc.

97 MATHEMATICS AND COMPUTING↗