Search NASA⌕ Search

SEARCH · Search NASA

Results for “data staging”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 55 records · Page 3
All sourcesNASA NTRSDOE OSTI
NASA NTRS2006

NASA Earth Sciences Data Support System and Services for the Northern Eurasia Earth Science Partnership Initiative

The presentation describes the recently awarded ACCESS project to provide data management of NASA remote sensing data for the Northern Eurasia Earth Science Partnership Initiative (NEESPI). The project targets integration of remote sensing data from MODIS, and other NASA instruments on board US-satellites (with potential expansion to data from non-US satellites), customized data products from climatology data sets (e.g., ISCCP, ISLSCP) and model data (e.g., NCEP/NCAR) into a single, well-architected data management system. It will utilize two existing components developed by the Goddard Earth Sciences Data & Information Services Center (GES DISC) at the NASA Goddard Space Flight Center: (1) online archiving and distribution system, that allows collection, processing and ingest of data from various sources into the online archive, and (2) user-friendly intelligent web-based online visualization and analysis system, also known as Giovanni. The former includes various kinds of data preparation for seamless interoperability between measurements by different instruments. The latter provides convenient access to various geophysical parameters measured in the Northern Eurasia region without any need to learn complicated remote sensing data formats, or retrieve and process large volumes of NASA data. Initial implementation of this data management system will concentrate on atmospheric data and surface data aggregated to coarse resolution to support collaborative environment and climate change studies and modeling, while at later stages, data from NASA and non-NASA satellites at higher resolution will be integrated into the system.

Leptoukh, Gregory↗
DOE OSTI2021

Design and Performance of Kokkos Staging Space toward Scalable Resilient Application Couplings

With the growing number of applications designed for heterogeneous HPC devices, application programmers and users are finding it challenging to compose scalable workflows as ensembles of these applications, that are portable, performant and resilient. The Kokkos C++ library has been designed to simplify this cumbersome procedure by providing an intra-application uniform programming model and portable performance. However, assembling multiple Kokkos-enabled applications into a complex workflow is still a challenge. Although Kokkos enables a uniform programming model, the inter-application data exchange still remains a challenge from both performance and software development cost perspectives. In order to address this issue, we propose Kokkos data staging memory space, an extension of Kokkos' data abstraction (memory space) for heterogeneous computing systems. This new abstraction allows to express data on a virtual shared-space for multiple Kokkos applications, thus extending Kokkos to support inter-application data exchange to build an efficient application workflow. Additionally, we study the effectiveness of asynchronous data layout conversions for applications requiring different memory access patterns for the shared data. Our preliminary evaluation with a synthetic benchmark indicate the effectiveness of this conversion adapted to three different scenarios representing access frequency and use patterns of the shared data.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2025

In-Transit Data Transport Strategies for Coupled AI-Simulation Workflow Patterns

Coupled AI-Simulation workflows are becoming the major workloads for HPC facilities, and their increasing complexity necessitates new tools for performance analysis and prototyping of new in-situ workflows. We present SimAI-Bench, a tool designed to both prototype and evaluate these coupled workflows. In this paper, we use SimAI-Bench to benchmark the data transport performance of two common patterns on the Aurora supercomputer: a one-to-one workflow with co-located simulation and AI training instances, and a many-to-one workflow where a single AI model is trained from an ensemble of simulations. For the one-to-one pattern, our analysis shows that node-local and DragonHPC data staging strategies provide excellent performance compared Redis and Lustre file system. For the many-to-one pattern, we find that data transport becomes a dominant bottleneck as the ensemble size grows. Our evaluation reveals that file system is the optimal solution among the tested strategies for the many-to-one pattern.

Tummalapalli, Harikrishna [Argonne National Labora↗
DOE OSTI2023

Optimizing Data Movement for GPU-Based In-Situ Workflow Using GPUDirect RDMA

The extreme-scale computing landscape is increasingly dominated by GPU-accelerated systems. At the same time, in-situ workflows that employ memory-to-memory inter-application data exchanges have emerged as an effective approach for leveraging these extreme-scale systems. In the case of GPUs, GPUDirect RDMA enables third-party devices, such as network interface cards, to access GPU memory directly and has been adopted for intra-application communications across GPUs. In this paper, we present an interoperable framework for GPU-based in-situ workflows that optimizes data movement using GPUDirect RDMA. Specifically, we analyze the characteristics of the possible data movement pathways between GPUs from an in-situ workflow perspective, and design a strategy that maximizes throughput. Furthermore, we implement this approach as an extension of the DataSpaces data staging service, and experimentally evaluate its performance and scalability on a current leadership GPU cluster. The performance results show that the proposed design reduces data-movement time by up to 53% and 40% for the sender and receiver, respectively, and maintains excellent scalability for up to 256 GPUs.

Zhang, Bo↗
DOE OSTI2021

Understanding collective human movement dynamics during large-scale events using big geosocial data analytics

Conventional approaches for modeling human mobility pattern often focus on human activity and movement dynamics in their regular daily lives and cannot capture changes in human movement dynamics in response to large-scale events. With the rapid advancement of information and communication technologies, many researchers have adopted alternative data sources (e.g., cell phone records, GPS trajectory data) from private data vendors to study human movement dynamics in response to large-scale natural or societal events. Big geosocial data such as georeferenced tweets are publicly available and dynamically evolving as real-world events are happening, making it more likely to capture the real-time sentiments and responses of populations. However, precisely-geolocated geosocial data is scarce and biased toward urban population centers. In this research, we developed a big geosocial data analytical framework for extracting human movement dynamics in response to large-scale events from publicly available georeferenced tweets. The framework includes a two-stage data collection module that collects data in a more targeted fashion in order to mitigate the data scarcity issue of georeferenced tweets; in addition, a variable bandwidth kernel density estimation(VB-KDE) approach was adopted to fuse georeference information at different spatial scales, further augmenting the signals of human movement dynamics contained in georeferenced tweets. To correct for the sampling bias of georeferenced tweets, we adjusted the number of tweets for different spatial units (e.g., county, state) by population. To demonstrate the performance of the proposed analytic framework, we chose an astronomical event that occurred nationwide across the United States, i.e., the 2017 Great American Eclipse, as an example event and studied the human movement dynamics in response to this event. Finally, this analytic framework can easily be applied to other types of large-scale events such as hurricanes or earthquakes.

54 ENVIRONMENTAL SCIENCES↗
NASA NTRSDate not supplied

Transcriptomics Processing Pipelines for Space Biology: An Open Source and Consensus-Driven Approach

Transcriptomics holds significant value in elucidating the relationship between gene expression, experimental factors, biological factors, and various types of omics data. Enhancing our understanding of these connections is paramount for foundational biology, which plays a pivotal role in devising solutions for challenges pertinent to both space travel and terrestrial life. The NASA GeneLab project, part of the Open Science Data Repository (OSDR.nasa.gov), seeks to accelerate space biology research through cataloging and democratizing ‘omics data, including transcriptomics. Since raw omics data are largely inaccessible to non-bioinformaticians, GeneLab works with the scientific community via the Open Science Analysis Working Groups (AWGs) to develop standard processing pipelines to generate and publish processed data. Unlike raw data, processed data have greater immediate value to diverse users with varying technical backgrounds and computational capabilities. Standardizing processing workflows is essential to match the pace of raw data generation, ensure reproducibility, and enable standardized processed data for comparison across datasets. As of June 2023, transcriptomics studies comprise over half of GeneLab datasets hosted on the OSDR, including data from bulk RNA-seq and Affymetrix or Agilent 1-Channel DNA microarray assays. In collaboration with the AWGs, GeneLab developed consensus processing pipelines for these transcriptomics data types that includes quality control, background correction (microarray only), data normalization and quantification, culminating in the detection and annotation of differentially expressed genes. The work presented here describes Nextflow implementations of GeneLab’s consensus transcriptomics pipelines that automates and accelerates processing of these datasets. In addition to the core data processing, these workflows also include raw data staging and a robust verification and validation program to identify errors in real-time, stop additional downstream computation, and preserve computational resources. These workflows are used to generate GeneLab processed data hosted on the OSDR, and are publicly available as open source software for others to use at: https://github.com/nasa/GeneLab_Data_Processing.

Jonathan Oribello↗
NASA NTRS1986

HOST turbine heat transfer subproject overview

The experimental part of the turbine heat transfer subproject consists of six large experiments, which are highlighted in this overview, and three of somewhat more modest scope. One of the initial efforts was the stator airfoil heat transfer program. The non-film cooled and the showerhead film cooled data have already been reported. The gill region film cooling effort is currently underway. The investigation of secondary flows in a 90 deg curved duct, was completed. The first phase examined flows with a relatively thin inlet boundary layer and low free stream turbulence. The second phase studied a thicker inlet boundary layer and higher free stream turbulence. A comparison of analytical and experimental cross flow velocity vectors is shown for the 60 deg plane. Two experiments were also conducted in the high pressure facility. One examined full coverage film cooled vanes, and the other, advanced instrumentation. The other three large experimental efforts were conducted in a rotation reference frame. An experiment to obtain gas path airfoil heat transfer coefficients in the large, low speed turbine was completed. Single-stage data with both high and low-inlet turbulence were taken. The second phase examined a one and one-half stage turbine and focused on the second vane row. Under phase 3 aerodynamic quantities such as interrow time-averaged and rms values of velocity, flow angle, inlet turbulence, and surface pressure distribution were measured.

Gladden, Herbert J.↗
DOE OSTI2025

HDF5 in the exascale era: Delivering efficient and scalable parallel I/O for exascale applications

Accurately modeling real-world systems requires scientific applications at exascale to generate massive amounts of data and manage data storage efficiently. However, parallel input and output (I/O) faces challenges due to new application workflows and the state-of-the-art memory, interconnect, and storage architectures considered in exascale designs. The storage hierarchy has expanded with node-local persistent memory, solid-state storage, and traditional disk and tape-based storage, thus requiring efficiency at each layer and much more efficient data movement among these layers. This paper discusses how the ExaHDF5 project improved the I/O performance and data management for exascale architectures by enhancing HDF5, a widely used parallel I/O library. The team developed an Asynchronous I/O Virtual Object Layer (VOL) connector that allowed overlapping I/O with computation. They also created a Cache VOL to complement asynchronous I/O by incorporating fast storage layers, such as burst buffer and node-local storage, into the parallel I/O workflow through caching and staging data. Additionally, the team enabled data aggregation and I/O at the node level by using a Subfiling Virtual File Driver (VFD). To demonstrate superior I/O performance with HDF5 at exascale, the ExaHDF5 team collaborated with several exascale applications. In this paper, we show I/O performance improvements for three applications: Cabana (a particle-based simulation library), EQSIM (a regional earthquake simulation software), and E3SM (a climate system modeling library).

Asynchronous I/Ol↗
DOE OSTI2022

Building an Integrated Ecosystem of Computational and Observational Facilities to Accelerate Scientific Discovery

Future scientific discoveries will rely on flexible ecosystems that incorporate modern scientific instruments, high performance computing resources, parallel distributed data storage, and performant networks across multiple, independent facilities. In addition to connecting physical resources, such an ecosystem presents many challenges in logistics and accessibility, especially in orchestrating computations and experiments that span across leadership computing systems and experimental instruments. Past efforts have typically been application-specific or limited to interfaces for computing resources. This paper proposes a general framework for integrating computation resources and instrument operations, addressing challenges in code development/execution, data staging and collection, software stack, control mechanisms, resource authorization and governance, and hardware integration. We also describe a demonstration use case wherein a Bayesian optimization algorithm running on an edge computing resource guides a scanning probe microscope to autonomously and intelligently characterize a material sample. This science edge ecosystem framework will provide a blueprint for federating multi-institutional, disparate resources and orchestrating scientific workflows across them to enable next-generation discoveries.

Somnath, Suhas↗
DOE OSTI2020

Day-ahead photovoltaic power production forecasting methodology based on machine learning and statistical post-processing

A main challenge towards ensuring large-scale and seamless integration of photovoltaic systems is to improve the accuracy of energy yield forecasts, especially in grid areas of high photovoltaic shares. The scope of this paper is to address this issue by presenting a unified methodology for hourly-averaged day-ahead photovoltaic power forecasts with improved accuracy, based on data-driven machine learning techniques and statistical post-processing. More specifically, the proposed forecasting methodology framework comprised of a data quality stage, data-driven power output machine learning model development (artificial neural networks), weather clustering assessment (K-means clustering), post-processing output optimisation (linear regressive correction method) and the final performance accuracy evaluation. The results showed that the application of linear regression coefficients to the forecasted outputs of the developed day-ahead photovoltaic power production neural network improved the performance accuracy by further correcting solar irradiance forecasting biases. The resulting optimised model provided a mean absolute percentage error of 4.7% when applied to historical system datasets. Finally, the model was validated both, at a hot as well as a cold semi-arid climatic location, and the obtained results demonstrated close agreement by yielding forecasting accuracies of mean absolute percentage error of 4.7% and 6.3%, respectively. Finally, the validation analysis provides evidence that the proposed model exhibits high performance in both forecasting accuracy and stability.

42 ENGINEERING↗
DOE OSTI2023

Towards elastic in situ analysis for high-performance computing simulations

In situ analysis and visualization have grown increasingly popular for enabling direct access to data from high-performance computing (HPC) simulations. As a simulation progresses and interesting physical phenomena emerge, however, the data produced may become increasingly complex, and users may need to dynamically change the type and scale of in situ analysis tasks being carried out and consequently adapt the amount of resources allocated to such tasks. To date, none of the production in situ analysis frameworks offer such an elasticity feature, and for good reason: the assumption that the number of processes could vary during run time would force developers to rethink software and algorithms at every level of the in situ analysis stack. In this paper we present Colza, a data staging service with elastic in situ visualization capabilities. We demonstrate the use of Colza with the Deep Water Impact and the AMR-Wind simulations, coupling them with the ParaView Catalyst and Ascent in situ libraries, and show that Colza enables dynamic rescaling of these widely-used frameworks with no interruption to the simulation or staging service. Here, we highlight the challenges of enabling such elasticity, which requires overcoming these frameworks' reliance on MPI, using distinct engineering approaches, namely dependency injection and dependency overload. To the best of our knowledge, this work is the first to enable elastic in situ visualization capabilities for HPC applications on top of existing production analysis tools.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2024

An Efficient Checkpointing System for Large Machine Learning Model Training

As machine learning models increase in size and complexity rapidly, the cost of checkpointing in ML training became a bottleneck in storage and performance (time). For example, the latest GPT-4 model has massive parameters at the scale of 1.76 trillion. It is highly time and storage consuming to frequently writes the model to checkpoints with more than 1 trillion floating point values to storage. This work aims to understand and attempt to mitigate this problem. First, we characterize the checkpointing interface in a collection of representative large machine learning/language models with respect to storage consumption and performance overhead. Second, we propose the two optimizations: i) A periodic cleaning strategy that periodically cleans up outdated checkpoints to reduce the storage burden; ii) A data staging optimization that coordinates checkpoints between local and shared file systems for performance improvement.

machine learning, artificial intelligence↗
DOE OSTI2024

Offline Power Systems Applications Enabled by Phasor Measurement Units: Technical Assistance to the Power Sectors of Southeast Asia

This report provides a brief overview of several offline (non-real-time) applications facilitated by high-resolution time-synchronized measurements recorded by phasor measurement units (PMUs). The high reporting rate and time-synchronization of PMU records provide a detailed view of power system dynamics, enabling electric utilities to obtain a better understanding of their systems. In this report, the following applications have been reviewed: Power plant model validation, System model validation, Ringdown oscillation analysis, Frequency response analysis, Postmortem analysis of disturbance events Along with a brief technical background of the applications above, applicable North American Electric Reliability Corporation (NERC) standards have been discussed, and examples of implementation in North American organizations have been provided. Implementing several of the discussed applications may need a preliminary stage of data gathering from multiple entities, and several frameworks and process flows have been formulated by organizations around the world for this purpose. However, the data-gathering stage has not been considered in the scope of the present report.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗
NASA NTRS1977

Hardware report for experiment M133, sleep monitoring on Skylab

The purpose of the experiment was objectively to measure sleep quantity and quality during prolonged space flight with the use of automatic equipment for onboard analyses of electroencephalographic (EEG) and electro-oculographic (EOG) activity and telemetry of sleep stage data.

Booher, C. R.↗
NASA NTRS2008

Parallel-Processing Software for Creating Mosaic Images

A computer program implements parallel processing for nearly real-time creation of panoramic mosaics of images of terrain acquired by video cameras on an exploratory robotic vehicle (e.g., a Mars rover). Because the original images are typically acquired at various camera positions and orientations, it is necessary to warp the images into the reference frame of the mosaic before stitching them together to create the mosaic. [Also see "Parallel-Processing Software for Correlating Stereo Images," Software Supplement to NASA Tech Briefs, Vol. 31, No. 9 (September 2007) page 26.] The warping algorithm in this computer program reflects the considerations that (1) for every pixel in the desired final mosaic, a good corresponding point must be found in one or more of the original images and (2) for this purpose, one needs a good mathematical model of the cameras and a good correlation of individual pixels with respect to their positions in three dimensions. The desired mosaic is divided into slices, each of which is assigned to one of a number of central processing units (CPUs) operating simultaneously. The results from the CPUs are gathered and placed into the final mosaic. The time taken to create the mosaic depends upon the number of CPUs, the speed of each CPU, and whether a local or a remote data-staging mechanism is used.

Klimeck, Gerhard↗
NASA NTRS2011

Three-Dimensional Model for Preservation and Restoration of Architectural Heritage

Thc aim of the research will be to create a model, three-dimensional mathematical. implementation. consultation and assistance to "large" restoration projects that will assist the structural analysis, allowing easier display of dynamic strain. analysis and lighting noise. It could also be a valuable tool for decision support. therefore. may simulate several possible scenarios for intervention, This model appears therefore an excellent support for recovering. ordering and monitoring information about materials and data (stage of restoration. photographs. sampling points. results of diagnostic tests, etc.) collected dynamically during the "life" of the cultural heritage. allowing to document its complete history

Marchis, Elena↗
NASA NTRS2013

A Simple GPU-Accelerated Two-Dimensional MUSCL-Hancock Solver for Ideal Magnetohydrodynamics

We describe our experience using NVIDIA's CUDA (Compute Unified Device Architecture) C programming environment to implement a two-dimensional second-order MUSCL-Hancock ideal magnetohydrodynamics (MHD) solver on a GTX 480 Graphics Processing Unit (GPU). Taking a simple approach in which the MHD variables are stored exclusively in the global memory of the GTX 480 and accessed in a cache-friendly manner (without further optimizing memory access by, for example, staging data in the GPU's faster shared memory), we achieved a maximum speed-up of approx. = 126 for a sq 1024 grid relative to the sequential C code running on a single Intel Nehalem (2.8 GHz) core. This speedup is consistent with simple estimates based on the known floating point performance, memory throughput and parallel processing capacity of the GTX 480.

graphics processing units↗
NASA NTRS1974

Hightip-speed, low-loading transonic fan stage. Part 2: Data compilation

Tests were conducted on a high-tip-speed low-loading transonic fan stage to determine the performance and inlet flow distortion tolerance of the design. Test data were recorded for overall and blade element performance with both uniform and distorted inlet flows. A tabular summary of the data and a representative selection of the computer data reduction sheets are presented.

Ware, T. C.↗