Search NASA⌕ Search

SEARCH · Search NASA

Results for “data access”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 505 records · Page 28

Preliminary design for a standard 10 sup 7 bit Solid State Memory (SSM)

A modular concept with three separate modules roughly separating bubble domain technology, control logic technology, and power supply technology was employed. These modules were respectively the standard memory module (SMM), the data control unit (DCU), and power supply module (PSM). The storage medium was provided by bubble domain chips organized into memory cells. These cells and the circuitry for parallel data access to the cells make up the SMM. The DCU provides a flexible serial data interface to the SMM. The PSM provides adequate power to enable one DCU and one SMM to operate simultaneously at the maximum data rate. The SSM was designed to handle asynchronous data rates from dc to 1.024 Mbs with a bit error rate less than 1 error in 10 to the eight power bits. Two versions of the SSM, a serial data memory and a dual parallel data memory were specified using the standard modules. The SSM specification includes requirements for radiation hardness, temperature and mechanical environments, dc magnetic field emission and susceptibility, electromagnetic compatibility, and reliability.

Hayes, P. J.↗

Non-Imaging Software/Data Analysis Requirements

The analysis software needs of the non-imaging planetary data user are discussed. Assumptions as to the nature of the planetary science data centers where the data are physically stored are advanced, the scope of the non-imaging data is outlined, and facilities that users are likely to need to define and access data are identified. Data manipulation and analysis needs and display graphics are discussed.

Source record↗

A New Look at Data Usage by Using Metadata Attributes as Indicators of Data Quality

This study reviews the key metrics (users, distributed volume, and files) in multiple ways to gain an understanding of the significance of the metadata. Characterizing the usability of data by key metadata elements, such as discipline and study area, will assist in understanding how the user needs have evolved over time. The data usage pattern based on product level provides insight into the level of data quality. In addition, the data metrics by various services, such as the Open-source Project for a Network Data Access Protocol (OPeNDAP) and subsets, address how these services have extended the usage of data. Over-all, this study presents the usage of data and metadata by metrics analyses, which may assist data centers in better supporting the needs of the users.

metadata↗

Visualization of Near Real-Time Global Cloud Composites (GCC): Integration in ArcGIS

The NASA Langley Satellite ClOud and Radiation Property retrieval System (SatCORPS) team provides low latency LEO and GEO satellite derived cloud and radiation products to end users for use in near real-time (NRT) applications. Hourly mosaics are fused from LEO (SNPP, JPSS-1, AQUA, MODIS) and GEO Satellite imagers (GOES-West, GOES-East, Metesat-11, Metesat-8, and Himawari-8) to create global cloud composites (GCC). SatCORPS GCC data are integrated into Esri ArcGIS system and transformed into Analysis Rady Data (ARD) to provide more efficient data access to support disaster management, weather diagnoses/forecasting, and Earth Sciences remote sensing applications. GCC data are exposed as RESTful APIs, ArcGIS Image Services, and Open Geospatial Consortium (OGC) Web Mapping Services/Web Coverage Services to provide a variety of end points for integration into user applications. We will preview the new interactive SatCORPS GCC web visualization tool and discuss initial integration of GCC product into NASA Airborne Mission Tool Suite (MTS) to support NASA and NOAA field campaigns.

GIS↗

Land Use Manager Application Ensures Protectiveness Following Remediation on the Oak Ridge Reservation - 13355

Long-term stewardship is the set of activities necessary to return contaminated land to safe and beneficial use. The activities include physical and legal controls to prevent inappropriate exposure to contamination left in place at a site. It is the longest phase of the Department of Energy's Environmental Management Program and ensures the protection of human health and the environment for varied end uses. At the Department of Energy's Oak Ridge Reservation an automated program has been developed and implemented that tracks the multitude of long-term stewardship activities. The Oak Ridge Reservation is a large site that currently has over 50 actions requiring long-term stewardship activities. The Oak Ridge Reservation consists primarily of three plant sites, and long-term stewardship will enable these sites to be leased to private entities (East Tennessee Technology Park), modernized for an evolving national security mission (Y-12 National Security Complex), and revitalized to continue multi-disciplinary research (Oak Ridge National Laboratory). The varied site end uses of the individual plant sites coupled with the multitude of controls required by leaving waste in place presents challenges. A single remedial action may include surveillance and maintenance activities, media monitoring, property record notices as well as physical controls such as fences and signs. Thus, the array of long-term stewardship activities is complex and intermingled (over 200 inspections each year at various frequencies are required currently) and requires an effective tracking program, termed the Land Use Manager. The Land Use Manager is a web-based data management application for use by personnel responsible for implementing, maintaining, and verifying engineering and land use controls on the Oak Ridge Reservation. The program is a data entry and tracking tool, as well as a notification tool. The status and performance of engineering and land use controls are checked annually for evaluation in the required Remediation Effectiveness Report, and the automated Land Use Manager collects, maintains, tracks, notifies, monitors, and manages the information necessary to perform this evaluation. Land Use Manager tracks site information including type of contamination, regulatory requirements, locates land use controls; provides information on inspections, certification, and reporting; and provides reports. Most data access features, e.g., view, print, query, and download, are available to all users; however, data input, updating, and editing are restricted to the personnel directly responsible for monitoring and inspection. The Land Use Manager application was developed for the Department of Energy Oak Ridge Office by URS - CH2M Oak Ridge LLC, Restoration Services Incorporated, and MIJARA Corporation to meet the specific needs of long-term stewardship tracking on the Oak Ridge Reservation. The successful implementation of long-term stewardship enables the future government and private activities being planned on the Oak Ridge Reservation to proceed. (authors)

54 ENVIRONMENTAL SCIENCES↗

On-Line Access to Weather Satellite Imagery and Image Manipulation Software

Advanced Very High Resolution Radiometer and Geostationary Operational Environmental Satellite Imagery, received by antennas located at the University of Colorado, are made available to the Internet users through an on-line data access system. Created as a 'test bed' data system for the National Aeronautics and Space Administration's future Earth Observing System Data and Information System, this test bed provides an opportunity to test both the technical requirements of an on-line data system and the different ways in which the general user community would employ such a system. Initiated in December 1991, the basic data system experienced four major evolutionary changes in response to user requests and requirements. Features added with these changes were the addition of on-line browse, user subsetting, and dynamic image processing/navigation. Over its lifetime the system has grown to a maximum of over 2500 registered users, and after losing many of these users due to hardware changes, the system is once again growing with its own independent mass storage system.

Emery, William J.↗

Production Operations Management System (POMS) for Fermilab Experiments

The Production Operations Management System (POMS) software allows production teams and analysis groups across multiple Fermilab experiments to launch, modify and monitor large scale campaigns of related Monte Carlo or data processing jobs, and currently manages the majority of production computing of experiments at Fermilab.POMS provides a web service interface that enables automated job submission on distributed resources according to customers’ requests, as well as subsequent monitoring of those submissions as well as recovery of failed submissions, debugging and record keeping.POMS interfaces with existing HEP data access, processing, movement and monitoring tools at Fermilab, including Jobsub (for job submission), SAM (for data management), and Landscape (for monitoring), and in combination with them handles the creation of intermediate datasets of files for multistage campaigns.POMS visualizes these campaigns with an interactive GUI interface. An important feature of POMS is a one-to-one connection between the GUI campaign visualizer and a text based representation of the complete campaign configuration. An extensible library of template campaign configurations is available. The templates are user modifiable and map cleanly to and from the GUI description of the campaign.This paper will discuss the current usage, history, and some technical details of POMS, as well as planned future extensions, particularly interfacing with Rucio as well as SAM for file movement.

Mengel, Marc↗

Dual Channel Dual Staging: Hierarchical and Portable Staging for GPU-Based In-Situ Workflow

In-situ workflows have emerged as an attractive approach for addressing data movement challenges at very large scales. Since GPU-based architectures dominate the HPC landscapes, porting these in-situ workflows, and, specifically, the inter-application data exchange, to GPU-based systems can be challenging. Technologies such as GPUDirect RDMA (GDR), which is typically used for I/O in GPU applications as an optimization that circumvents the CPU overhead, can be leveraged to support bulk data exchanges between GPU applications. However, current GDR design often lacks performance portability across HPC clusters built with different hardware configurations. Furthermore, the local CPU may also be effectively used as an auxiliary communication mechanism to offload data exchanges. In this paper, we present a dual channel dual staging approach for efficient, scalable, and performance-portable inter-application data exchange for in-situ workflows. This approach exploits the data access pattern within in-situ workflows along with the inherent execution asynchrony to accelerate data exchanges and, at the same time, improve performance portability. Specifically, the dual channel dual staging method leverages both the local CPU and the remote data staging server to build a hierarchical joint staging area and uses this staging area to transform blocking inter-application bulk data exchanges into best-effort local data movements between GPU and CPU. The dual channel dual staging is implemented as a portability extension of the Dataspaces-GPU staging framework. We present an experimental evaluation of its performance, portability, and scalability using this implementation on three leadership GPU clusters. The evaluation results demonstrate that the dual channel dual staging method saves up to 75% in data-exchange time compared to host-based, GDR, and alternate portable designs, while maintaining scalability (up to 512 GPUs) and performance portability across the three platforms.

Zhang, Bo [University of Utah]↗

Error-controlled Progressive Retrieval of Scientific Data under Derivable Quantities of Interest

The unprecedented amount of scientific data has introduced heavy pressure on the current data storage and transmission systems. Progressive compression has been proposed to mitigate this problem, which offers data access with on-demand precision. However, existing approaches only consider precision control on primary data, leaving uncertainties on the quantities of interest (QoIs) derived from it. In this work, we present a progressive data retrieval framework with guaranteed error control on derivable QoIs. Our contributions are three-fold. (1) We carefully derive the theories to strictly control QoI errors during progressive retrieval. Our theory is generic and can be applied to any QoIs that can be composited by the basis of derivable QoIs proved in the paper. (2) We design and develop a generic progressive retrieval framework based on the proposed theories, and optimize it by exploring feasible progressive representations. (3) We evaluate our framework using five real-world datasets with a diverse set of QoIs. Experiments demonstrate that our framework can faithfully respect any user-specified QoI error bounds in the evaluated applications. This leads to over 2.02× performance gain in data transfer tasks compared to transferring the primary data while guaranteeing a QoI error that is less than 1E-5.

Wu, Xuan↗

CoREC: Scalable and Resilient In-memory Data Staging for In-situ Workflows

The dramatic increase in the scale of current and planned high-end HPC systems is leading new challenges, such as the growing costs of data movement and IO, and the reduced mean time between failures (MTBF) of system components. In-situ workflows, i.e., executing the entire application workflows on the HPC system, have emerged as an attractive approach to address data-related challenges by moving computations closer to the data, and staging-based frameworks have been effectively used to support in-situ workflows at scale. However, the resilience of these staging-based solutions has not been addressed, and they remain susceptible to expensive data failures. Furthermore, naive use of data resilience techniques such as n-way replication and erasure codes can impact latency and/or result in significant storage overheads. In this article, we present CoREC, a scalable and resilient in-memory data staging runtime for large-scale in-situ workflows. CoREC uses a novel hybrid approach that combines dynamic replication with erasure coding based on data access patterns. It also leverages multiple levels of replications and erasure coding to support diverse data resiliency requirements. Furthermore, the article presents optimizations for load balancing and conflict-avoiding encoding, and a low overhead, lazy data recovery scheme. We have implemented the CoREC runtime and have deployed with the DataSpaces staging service on leadership class computing machines and present an experimental evaluation in the article. Here, the experiments demonstrate that CoREC can tolerate in-memory data failures while maintaining low latency and sustaining high overall storage efficiency at large scales.

97 MATHEMATICS AND COMPUTING↗

Dynamic CCS-EJ-SJ Database and Web Application - What's New

At the 2024 FECM/NETL Carbon Management Research Project Review Meeting, within the Carbon Transport and Storage Breakout Session 3, the presentation "Dynamic CCS-EJ-SJ Database and Web Application - What's New" highlights the critical tool designed to integrate environmental and social justice considerations into Carbon Capture and Storage (CCS) projects. Key features include an interactive dashboard for data access and visualization, which supports stakeholders in making informed decisions regarding CCS implementation, and updated data layers. The latest version enhances data integration and usability, providing a comprehensive resource for assessing the social and environmental impacts of CCS projects. There are 7 categories in the CCS EJSJ v2 database (released 03/31/2024): environmental justice, energy justice, economic justice, social justice, ecosystem assets, clean energy, and infrastructure. Most of the layers within each category have been updated in this version. As compared to the old database, there are 3 new categories in the v2 database: ecosystem assets, clean energy, and infrastructure.

Sharma, Maneesh↗

Tropical Rainfall Measuring Mission (TRMM)

Contents include the following: 1. Hurricane Isabel. TRMM composite images of Hurricane Isabel. TRMM application to monitoring potential floods. 2.TRMM Orbit and Instruments. TRMM Ground Track for 35" Orbit With Ground Truth Verification Sites. 3. TRMM Science Data Products at the GES DAAC. 4. TRMM Data Access and Services at GES DAAC. TRMM Online Visualization and Analysis System. Selected Recent TRMM Data Applications From Around the World.

Rui, Hualan↗

Challenges in Obtaining and Visualizing Satellite Level 2 Data in GIS

Satellite data products are important for a wide variety of applications that can bring far-reaching benefits to the science community and the broader society. These benefits can best be achieved if the satellite data are well utilized and interpreted. Unfortunately, this is not always the case, despite the abundance and relative maturity of numerous satellite data products provided by NASA and other organizations. One way to help users better understand the satellite data is to provide data along with Images, including accurate pixel coverage area delineation, and science team recommended quality screening for individual geophysical parameters. However, there are challenges of visualizing remote sensed non-gridded products: (1) different geodetics of space-borne instruments (2) data often arranged in a long-track and a cross-track axes (3) spatially and temporally continuous data chunked into granule files: data for a portion (or all) of a satellite orbit (4) no general rule of resampling or interpolations to a grid (5) geophysical retrieval only based on pixel center location without shape information. In this presentation, we will unravel a new Goddard Earth Sciences Data and Information Services Center (GES DISC) Level 2 (L2) visualization on-demand service. The service's front end provides various visualization and data accessing capabilities, such as overlay and swipe of multiply variables and subset and download of data in different formats. The backend of the service consists of Open Geospatial Consortium (OGC) standard-compliant Web Mapping Service (WMS) and Web Coverage Service. The infrastructure allows inclusion of outside data sources served in OGC compliant protocols and allows other interoperable clients, such as ArcGIS clients, to connect to our L2 WCS/WMS.

GI↗

An Update on GPM and TRMM Data Services at NASA GES DISC

The Goddard Earth Sciences (GES) Data and Information Services Center (DISC) is home to data archives of the NASA-JAXA Global Precipitation Measurement (GPM), the Tropical Rainfall Measuring Mission (TRMM), and other NASA missions and projects. To maximize the use of NASA data products in scientific research and applications as well as for societal benefits, we provide data and information services that make datasets easy to find and use through simplification of data access for users at all levels around the world. Over the years, user-friendly data services have been developed at GES DISC, including data subsetting, format conversion, online visualization and analysis (i.e. Giovanni), user support system, etc. We routinely analyze questions, feedback, and use cases from users and algorithm developers around the world as well as best practices and new technology to improve existing services and formulate new data services. Interaction between users and algorithm developers is an important process for identifying issues in products, collecting user requirements, and improving product quality and usability. Staff members regularly communicate with algorithm developers with user questions and concerns through conferences and workshops. We publish peer-reviewed papers and articles in major Earth science journals and book chapters to describe NASA global and regional precipitation datasets and services with examples. News articles about GPM and TRMM datasets associated with significant events are regularly posted in the GES DISC Web portal and social media. We also actively participate in training activities. In this presentation, we present our latest activities about GPM and TRMM data services, data/service metrics, and future plans at GES DISC.

Liu, Z.↗

An open-access simulated earthquake ground-motion database for an M7 Hayward Fault earthquake in the San Francisco Bay Region

Comprehensive understanding of earthquake ground motions, particularly in the near-fault region of large-magnitude events, is limited by gaps in strong-motion data. This challenge is prominent in areas with high seismic hazard but infrequent large earthquakes where data is sparse and difficult to interpret. These data limitations lead to uncertainties in the development of site-specific ground motions, which are crucial for engineering risk assessments. To address these challenges, physics-based regional-scale ground-motion simulations have been developed. With the emergence of exaflop-scale computing ecosystems, it is now possible to simulate regional earthquake processes at unprecedented fidelity and generate the large number of fault rupture realizations necessary to characterize both intra- and inter-event ground-motion variability. This article introduces a new database of simulated earthquake ground motions, created for applications in earthquake engineering, earthquake planning, and emergency response. The inaugural version of the database features simulated ground motions for a magnitude 7 Hayward Fault earthquake in the San Francisco Bay Region (SFBR), using the EarthQuake SIMulation (EQSIM) simulation framework and the Graves–Pitarka kinematic rupture model. The aim is to provide high-fidelity, spatially dense, three-component motions generated on the Department of Energy’s (DOE) newest generation of graphics processing unit (GPU)-accelerated supercomputers. These motions are being made openly available to the engineering, scientific, and disaster planning communities. In addition, this work develops protocols for the efficient dissemination of these large data sets and emphasizes community engagement to build confidence in their application. This article discusses the methodology behind the data, underlying software verification and validation, scalable data management, and a user interface for data access. The goal is to facilitate widespread use and elicit expert feedback to maximize the utility and exploitation of simulated motions. While the initial focus is on the San Francisco Region, simulations for additional regions will be added as the DOE program progresses.

Simulated ground-motion database↗

Optimized spatial information for 1990, 2000, and 2010 U.S. census microdata

Abstract We report on the successful completion of a project to upgrade the positional accuracy of every response to the 1990, 2000, and 2010 U.S. decennial censuses. The resulting data set, called Optimized Spatial Census Information Linked Across Time (OSCILAT), resides within the restricted-access data warehouse of the Federal Statistical Research Data Center (FSRDC) system where it is available for use with approval from the U.S. Census Bureau. OSCILAT greatly improves the accuracy and completeness of spatial information for older censuses conducted prior to major quality improvements undertaken by the Bureau. Our work enables more precise spatial and longitudinal analysis of census data and supports exact tabulations of census responses for arbitrary spatial units, including tabulating responses from 1990, 2000, and 2010 within 2020 block boundaries for precise measures of change over time for small geographic areas.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗

MDLoader: A Hybrid Model-Driven Data Loader for Distributed Graph Neural Network Training

Scalable data management is essential for processing large scientific dataset on HPC platforms for distributed deep learning. In-memory distributed storage is preferred for its speed, enabling rapid, random, and frequent data access required by stochastic optimizers. Processes use one-sided or collective communication to fetch remote data, with optimal performance depending on (i) dataset characteristics, (ii) training scale, and (iii) interconnection network. Empirical analysis shows collective communication excels with larger mini-batch sizes and/or fewer processes, whereas one-sided communication outperforms at larger scales. We propose MDLoader, a hybrid in-memory data loader for distributed graph neural network training. MDLoader features a model-driven performance estimator that dynamically selects between one-sided and collective communication at the beginning of training using Tree of Parzen Estimators (TPE). Evaluations on NERSC Perlmutter and OLCF Summit show MDLoader outperforms single-backend loaders by up to 2.83 × and predicts the suitable communication method with 96.3% (Perlmutter) and 94.3% (Summit) success rate.

Bae, Jonghyun↗