Search NASA⌕ Search

SEARCH · Search NASA

Results for “DISTRIBUTED DATA PROCESSING”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 253 records · Page 14

SPHEREx: NASA’s Near-Infrared Spectrophotmetric All-Sky Survey

SPHEREx, the Spectro-Photometer for the History of the Universe, Epoch of Reionization, and ices Explorer, isa NASA MIDEX mission planned for launch in 2024. SPHEREx will carry out the first all-sky spectral survey at wavelengths between 0.75μm and 5μm with spectral resolving power ~40 between 0.75 and 3.8μm and ~120 between 3.8 and 5μm At the end of its two-year mission, SPHEREx will provide 14 billion 0.75-to-5μm spectra of each 6."2 x 6."2 pixel on the sky. This paper updates an earlier description of SPHEREx presenting changes made during the mission's Preliminary Design Phase, including a discussion of instrument integration and test flow and a summary of the data processing, analysis, and distribution plans.

Zemcov, Michael↗

Characterizing Wildfires in Western US.: A Cloud-based Case Study for Interdisciplinary Research using NASA Resources

This presentation will demonstrate a case study of interdisciplinary research done in the Amazon Web Services (AWS) cloud platform, in addition to in the local machine. We conduct data analysis next to data by leveraging various cloud-based data in NASA Earthdata Cloud, which are distributed by different missions/NASA Distributed Active Archive Centers (DAACs), and cloud computing resources at NASA. For instance, we directly access multiple datasets stored in the AWS Simple Storage Service (S3) buckets using a Python Jupyter notebook through a JupyterHub interface hosted in AWS (without having to download data), and conduct data analysis next to data in the cloud. We will also show how to share the research results following Open Source policy. This case study characterizes the change in wildfire events in the western United States during the past 20 years. In particular, we focus on the wildfires in California in 2021, one of the most severe wildfire years occurring in the most recent 20 years in California. We will analyze the possible causes of wildfires, such as drought conditions and climate variability, and examine the impacts of wildfires on air quality and atmospheric composition, and on land cover. We will examine the data distributed by the NASA Goddard Earth Sciences Data and Information Services Center (GES DISC), including aerosols and meteorological data from the NASA Modern-Era Retrospective analysis for Research and Applications version 2 (MERRA-2), precipitation from the Global Precipitation Measurement (GPM) and Global Precipitation Climate Project (GPCP), and aerosol index from Ozone Monitoring Instrument (OMI). We also utilize the data distributed by the Physical Oceanography (PO) DAAC, such as Sea Surface Temperature (SST) data from the Group for High Resolution Sea Surface Temperature (GHRSST), and the data distributed by Land Processes (LP) DAAC, such as Normalized Difference Vegetation Index (NDVI).

Xiaohua Pan↗

Overview of the Astrophysics Data System

The Astrophysics Division of NASA has built a geographically- and logically-distributed heterogeneous information system for the dissemination and coordinated multispectral analysis of data from astrophysics missions. The Astrophysics Data System (ADS) is a truly distributed system in which the data and the required processing are physically distributed. To accommodate the anticipated growth and changes in both requirements and technology, the ADS employs a server/client architecture which allows services and data to be added or replaced without having to change the basic architecture or interfaces. Current datasets accessible through the system include all the tabular astronomical data available at each of six existing astrophysics data centers. Additional data nodes, at both NASA data centers and academic institutions, will be added shortly. The future evolution of the system will be driven in large part by user services mounted both by the ADS project itself and by members of the astrophysics community.

Good, John C.↗

Overview of the Astrophysics Data System

The Astrophysics Division of NASA has built a geographically and logically distributed heterogeneous information system for the dissemination and coordinated multispectral analysis of data from astrophysics missions. The Astrophysics Data System (ADS) is a truly distributed system in which the data and the required processing are physically distributed. To accommodate the anticipated growth and changes in both requirements and technology, the ADS employs a server/client architecture which allows services and data to be added or replaced without having to change the basic architecture or interfaces. Current datasets accessible through the system include all the tabular astronomical data available at each of six existing astrophysics data centers. Additional data nodes, at both NASA data centers and academic institutions, will be added shortly. The future evolution of the system will be driven in large part by user services mounted both by the ADS project itself and by members of the astrophysics community.

Good, John C.↗

Monitoring Volcanic Eruptions by OMI/OMPS, ASTER, and MERRA-2

Volcanic lava flows and/or the gas eruptions are the most common characteristics that can be remotely monitored with satellite technology in the global perspective and on different timescales. Atmospheric Sulfur Dioxide (SO2), one of the most abundant gases from volcanic eruptions apart from atmospheric common gases Carbon Dioxide and water vapor, can be directly detected by space-based sensors on satellites. The NASA Goddard Earth Sciences Data and Information Services Center (GES DISC) is one of the 12 Distributed Active Archive Centers (DAACs) within NASA's Earth Observing System Data and Information System (EOSDIS), which archives SO2 data sets from the Nimbus-7 Total Ozone Mapping Spectrometer (TOMS) in 1978, till the ongoing Ozone Monitoring Instrument (OMI) on NASA's EOS-Aura satellite, the Ozone Mapping Profiler Suite (OMPS) Nadir Mapper (NM) on both the Suomi National Polar-Orbiting Partnership (Suomi-NPP or SNPP) and the Joint Polar-orbiting Satellite System-1 (JPSS-1) satellites, into the future JPSS missions. In addition to the standard OMI/Aura and OMPS/S-NPP SO2 products, SO2 products created under the charter of the Making Earth System Data Records for Use in Research Environments (MEaSUREs) project, are also archived at GES DISC, through which NASA enacts to expand understanding the Earth system using consistent data records. The Land Processes Distributed Active Archive Center (LP DAAC) is another EOSDIS's DAAC that provides land data products and operates as a partnership with the U.S. Geological Survey (USGS). The LP DAAC has been archiving the satellite imagery from the Advanced Spaceborne Thermal Emission and Reflection Radiometer (ASTER) aboard NASA's EOS-Terra satellite, a high spatial resolution (15 meters) and 14 band multispectral instrument. The ASTER imagery is one of the land products contributing to the application for monitoring hot spots and land terrain changes caused by volcanic eruption events. The data potential in GES DISC and LP DAAC to monitor volcanic sources of SO2 and the influence of wind fields on the gas plume spread will be demonstrated with the most recent 2018 May-July Kilauea Volcano eruption.

Zeng, Jian↗

Reconstruction of atmospheric pollutant concentrations from remote sensing data - An application of distributed parameter observer theory

The reconstruction of a concentration distribution from spatially averaged and noise-corrupted data is a central problem in processing atmospheric remote sensing data. Distributed parameter observer theory is used to develop reconstructibility conditions for distributed parameter systems having measurements typical of those in remote sensing. The relation of the reconstructibility condition to the stability of the distributed parameter observer is demonstrated. The theory is applied to a variety of remote sensing situations, and it is found that those in which concentrations are measured as a function of altitude satisfy the conditions of distributed state reconstructibility.

Koda, M.↗

Mission operations update for the restructured Earth Observing System (EOS) mission

The National Aeronautics and Space Administration's (NASA) Earth Observing System (EOS) will provide a comprehensive long term set of observations of the Earth to the Earth science research community. The data will aid in determining global changes caused both naturally and through human interaction. Understanding man's impact on the global environment will allow sound policy decisions to be made to protect our future. EOS is a major component of the Mission to Planet Earth program, which is NASA's contribution to the U.S. Global Change Research Program. EOS consists of numerous instruments on multiple spacecraft and a distributed ground system. The EOS Data and Information System (EOSDIS) is the major ground system developed to support EOS. The EOSDIS will provide EOS spacecraft command and control, data processing, product generation, and data archival and distribution services for EOS spacecraft. Data from EOS instruments on other Earth science missions (e.g., Tropical Rainfall Measuring Mission (TRMM)) will also be processed, distributed, and archived in EOSDIS. The U.S. and various International Partners (IP) (e.g., the European Space Agency (ESA), the Ministry of International Trade and Industry (MITI) of Japan, and the Canadian Space Agency (CSA)) participate in and contribute to the international EOS program. The EOSDIS will also archive processed data from other designated NASA Earth science missions (e.g., UARS) that are under the broad umbrella of Mission to Planet Earth.

Kelly, Angelita Castro↗

CMS Tier-0 data processing during the detector commissioning in Run 3

The CMS Tier-0 system is responsible for the prompt processing and distribution of the data collected by the CMS Experiment. A number of upgrades were implemented during the long shutdown 2 of the Large Hadron Collider, which improved the performance and reliability of the system. In this report, these upgrades are discussed and a more detailed description of the Tier-0 system is given. The experience of the data taking during Run 3 detector commissioning as well as the performance of the system are highlighted.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗

Ocean color - Availability of the global data set

The use of satellite observations of ocean color to provide reliable estimates of marine phytoplankton biomass on synoptic scales is examined. An overview is given of the Coastal Zone Color Scanner data processing system. The archiving and distribution of ocean color data are discussed, and NASA-sponsored archive sites are listed.

Feldman, Gene↗

Characterizing the LANDSAT Global Long-Term Data Record

The effects of global climate change are fast becoming politically, sociologically, and personally important: increasing storm frequency and intensity, lengthening cycles of drought and flood, expanding desertification and soil salinization. A vital asset in the analysis of climate change on a global basis is the 34-year record of Landsat imagery. In recognition of its increasing importance, a detailed analysis of the Landsat observation coverage within the US archive was commissioned. Results to date indicate some unexpected gaps in the US-held archive. Fortunately, throughout the Landsat program, data have been downlinked routinely to International Cooperator (IC) ground stations for archival, processing, and distribution. These IC data could be combined with the current US holdings to build a nearly global, annual observation record over this 34-year period. Today, we have inadequate information as to which scenes are available from which IC archives. Our best estimate is that there are over four million digital scenes in the IC archives, compared with the nearly two million scenes held in the US archive. This vast pool of Landsat observations needs to be accurately documented, via metadata, to determine the existence of complementary scenes and to characterize the potential scope of the global Landsat observation record. Of course, knowing the extent and completeness of the data record is but the first step. It will be necessary to assure that the data record is easy to use, internally consistent in terms of calibration and data format, and fully accessible in order to fully realize its potential.

Arvidson, T.↗

The development of a post-test diagnostic system for rocket engines

An effort was undertaken by NASA to develop an automated post-test, post-flight diagnostic system for rocket engines. The automated system is designed to be generic and to automate the rocket engine data review process. A modular, distributed architecture with a generic software core was chosen to meet the design requirements. The diagnostic system is initially being applied to the Space Shuttle Main Engine data review process. The system modules currently under development are the session/message manager, and portions of the applications section, the component analysis section, and the intelligent knowledge server. An overview is presented of a rocket engine data review process, the design requirements and guidelines, the architecture and modules, and the projected benefits of the automated diagnostic system.

Zakrajsek, June F.↗

The development of a post-test diagnostic system for rocket engines

An effort was undertaken by NASA to develop an automated post-test, post-flight diagnostic system for rocket engines. The automated system is designed to be generic and to automate the rocket engine data review process. A modular, distributed architecture with a generic software core was chosen to meet the design requirements. The diagnostic system is initially being applied to the Space Shuttle Main Engine data review process. The system modules currently under development are the session/message manager, and portions of the applications section, the component analysis section, and the intelligent knowledge server. An overview is presented of a rocket engine data review process, the design requirements and guidelines, the architecture and modules, and the projected benefits of the automated diagnostic system.

Zakrajsek, June F.↗

MDLoader: A Hybrid Model-Driven Data Loader for Distributed Graph Neural Network Training

Scalable data management is essential for processing large scientific dataset on HPC platforms for distributed deep learning. In-memory distributed storage is preferred for its speed, enabling rapid, random, and frequent data access required by stochastic optimizers. Processes use one-sided or collective communication to fetch remote data, with optimal performance depending on (i) dataset characteristics, (ii) training scale, and (iii) interconnection network. Empirical analysis shows collective communication excels with larger mini-batch sizes and/or fewer processes, whereas one-sided communication outperforms at larger scales. We propose MDLoader, a hybrid in-memory data loader for distributed graph neural network training. MDLoader features a model-driven performance estimator that dynamically selects between one-sided and collective communication at the beginning of training using Tree of Parzen Estimators (TPE). Evaluations on NERSC Perlmutter and OLCF Summit show MDLoader outperforms single-backend loaders by up to 2.83 × and predicts the suitable communication method with 96.3% (Perlmutter) and 94.3% (Summit) success rate.

Bae, Jonghyun↗

Passive microwave data for snow and ice research - Planned products from the DMSP SSM/I system

Recommendations which have been made for processing and distributing passive microwave data for snow and ice research obtained with the Defense Meteorological Satellite Program (DMSP) Special Sensor Microwave Imager (SSM/I) are discussed. The general objectives for SSM/I data are reviewed, and the sensor and data flow are described. The SSM/I sea ice products are discussed, and algorithm/product validation is addressed. Proposed services and implementation after SSM/I launch are summarized.

Weawer, Ronald↗

Software-Based Challenges of Developing the Future Distribution Grid

The software that the utility industry currently uses may be insufficient to analyze the distribution grid as it rapidly modernizes to include active resources such as distributed generation, switch and voltage control, automation, and increasingly complex loads. Although planners and operators have traditionally viewed the distribution grid as a passive load, utilities and consultants increasingly need enhanced analysis that incorporates active distribution grid loads in order to ensure grid reliability. Numerous commercial and open-source tools are available for analyzing distribution grid systems. These tools vary in complexity from providing basic load-flow and capacity analysis under steady-state conditions to time-series analysis and even geographical representations of dynamic and transient events. The need for each type of analysis is not well understood in the industry, nor are the reasons that distribution analysis requires different techniques and tools both from those now available and from those used for transmission analysis. In addition, there is limited understanding of basic capability of the tools and how they should be practically applied to the evolving distribution system. The study reviews the features and state of the art capability of current tools, including usability and visualization, basic analysis functionality, advanced analysis including inverters, and renewable generation and load modeling. We also discuss the need for each type of distribution grid system analysis. In addition to reviewing basic functionality current models, we discuss dynamics and transient simulation in detail and draw conclusions about existing software?s ability to address the needs of the future distribution grid as well as the barriers to modernization of the distribution grid that are posed by the current state of software and model development. Among our conclusions are that accuracy, data transfer, and data processing abilities are key to future distribution grid modeling, and measured data sources are a key missing element . Modeling tools need to be calibrated based on measured grid data to validate their output in varied conditions such as high renewables penetration and rapidly changing topology. In addition, establishing a standardized data modeling format would enable users to transfer data among tools to take advantage of different analysis features. ?

99 GENERAL AND MISCELLANEOUS↗

NASA's Earth Science Flight Program Meets the Challenges of Today and Tomorrow

NASA's Earth science flight program is a dynamic undertaking that consists of a large fleet of operating satellites, an array of satellite and instrument projects in various stages of development, a robust airborne science program, and a massive data archiving and distribution system. Each element of the flight program is complex and present unique challenges. NASA builds upon its successes and learns from its setbacks to manage this evolving portfolio to meet NASA's Earth science objectives. NASA fleet of 16 operating missions provide a wide range of scientific measurements made from dedicated Earth science satellites and from instruments mounted to the International Space Station. For operational missions, the program must address issues such as an aging satellites operating well beyond their prime mission, constellation flying, and collision avoidance with other spacecraft and orbital debris. Projects in development are divided into two broad categories: systematic missions and pathfinders. The Earth Systematic Missions (ESM) include a broad range of multi-disciplinary Earth-observing research satellite missions aimed at understanding the Earth system and its response to natural and human-induced forces and changes. Understanding these forces will help determine how to predict future changes, and how to mitigate or adapt to these changes. The Earth System Science Pathfinder (ESSP) program provides frequent, regular, competitively selected Earth science research opportunities that accommodate new and emerging scientific priorities and measurement capabilities. This results in a series of relatively low-cost, small-sized investigations and missions. Principal investigators whose scientific objectives support a variety of studies lead these missions, including studies of the atmosphere, oceans, land surface, polar ice regions, or solid Earth. This portfolio of missions and investigations provides opportunity for investment in innovative Earth science that enhances NASA's capability for better understanding the current state of the Earth system. ESM and ESSP projects often involve partnerships with other US agencies and/or international organizations. This adds to the complexity of mission development, but allows for a greater scientific return on NASA's investments. The Earth Science Airborne Science Program provides manned and unmanned aircraft systems that further science and advance the use of satellite data. NASA uses these assets worldwide in campaigns to investigate extreme weather events, observe Earth system processes, obtain data for Earth science modeling activities, and calibrate instruments flying aboard Earth science spacecraft. The Airborne Science Program has six dedicated aircraft and access to many other platforms. The Earth Science Multi-Mission Operations program acquires, preserves, and distributes observational data from operating spacecraft to support Earth Science research focus areas. The Earth Observing System Data and Information System (EOSDIS), which has been in operations since 1994, primarily accomplishes this. EOSDIS acquires, processes, archives, and distributes Earth Science data and information products. The archiving of NASA Earth Science information happens at eight Distributed Active Archive Centers (DAACs) and four disciplinary data centers located across the United States. The DAACs specialize by topic area, and make their data available to researchers around the world. The DAACs currently house over 9 petabytes of data, growing at a rate of 6.4 terabytes per day. NASA's current Earth Science portfolio is responsive to the National Research Council (NRC) 2007 Earth Science Decadal Survey and well as the 2010 NASA Response to President Obama's Climate Plan. As the program evolves into the future it will leverage the lessons learned from the current missions in operations and development, and plan for adjustments to future objectives in response to the anticipated 2017 NRC Decadal Survey.

Flight Program↗

NASA’s Unsteady Pressure-Sensitive Paint Research and Operational Capability Developments

To address challenges in the field of unsteady aerodynamics, NASA has developed a new state-of-the-art capability called Unsteady Pressure-Sensitive Paint (uPSP). It has been developed as an operational surface-pressure measurement capability for deployment in NASA’s AETC ground test facilities. Offering unprecedented spatiotemporal resolution, uPSP is an extremely powerful tool for investigating unsteady separated flows. To accelerate scientific discovery, uPSP data is to be processed and distributed as rapidly as it is acquired. Several demonstrations of the technology collected valuable data that has been used to develop data products, a robust processing pipeline, and other computational capabilities. A collection of papers documents the most recent research and development work on the uPSP technology and capability. This paper summarizes the current state of this effort at NASA.

Pressure-Sensitive Paint↗

NASA’s Unsteady Pressure-Sensitive Paint Research and Operational Capability Developments

To address challenges in the field of unsteady aerodynamics, NASA has developed a new stateof-the-art capability called Unsteady Pressure-Sensitive Paint (uPSP). It has been developed as an operational surface-pressure measurement capability for deployment in NASA’s AETC ground test facilities. Offering unprecedented spatiotemporal resolution, uPSP is an extremely powerful tool for investigating unsteady separated flows. To accelerate scientific discovery, uPSP data is to be processed and distributed as rapidly as it is acquired. Several demonstrations of the technology collected valuable data that has been used to develop data products, a robust processing pipeline, and other computational capabilities. A collection of papers documents the most recent research and development work on the uPSP technology and capability. This paper summarizes the current state of this effort at NASA.

Pressure-Sensitive Paint↗