Search NASA⌕ Search

SEARCH · Search NASA

Results for “Data migration”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 109 records · Page 6
All sourcesNASA NTRSDOE OSTI
NASA NTRS2018

NASA's Earth Observing Data and Information System - Near-Term Challenges

NASA's Earth Observing System Data and Information System (EOSDIS) has been a central component of the NASA Earth observation program since the 1990's. EOSDIS manages data covering a wide range of Earth science disciplines including cryosphere, land cover change, polar processes, field campaigns, ocean surface, digital elevation, atmosphere dynamics and composition, and inter-disciplinary research, and many others. One of the key components of EOSDIS is a set of twelve discipline-based Distributed Active Archive Centers (DAACs) distributed across the United States. Managed by NASA's Earth Science Data and Information System (ESDIS) Project at Goddard Space Flight Center, these DAACs serve over 3 million users globally. The ESDIS Project provides the infrastructure support for EOSDIS, which includes other components such as the Science Investigator-led Processing systems (SIPS), common metadata and metrics management systems, specialized network systems, standards management, and centralized support for use of commercial cloud capabilities. Given the long-term requirements, and the rapid pace of information technology and changing expectations of the user community, EOSDIS has evolved continually over the past three decades. However, many challenges remain. Challenges addressed in this paper include: growing volume and variety, achieving consistency across a diverse set of data producers, managing information about a large number of datasets, migration to a cloud computing environment, optimizing data discovery and access, incorporating user feedback from a diverse community, keeping metadata updated as data collections grow and age, and ensuring that all the content needed for understanding datasets by future users is identified and preserved.

Remote Sensing↗
NASA NTRSDate not supplied

The NASA Merra-2 Reanalysis Products: Data and Tools Used for Aerosol and Air Quality Studies

The NASA Modern-Era Retrospective analysis for Research and Applications Version 2 (MERRA-2) is atmospheric reanalysis data spanning 1980 to present. It has been produced by the NASA Global Modeling and Assimilation Office (GMAO) and is distributed by the NASA Goddard Earth Sciences Data and Information Services Center (GES DISC). MERRA-2 data includes 100 collections of Earth system variables, mainly from the atmospheric model, such as aerosol fields and meteorological fields, radiation fields, and aerosol fields, guided by the assimilation of as many as six million observations every six hours. MERRA-2 has been one of the most popular datasets from NASA and is widely used in interdisciplinary research and applications, with increasing numbers of new users. For example, at least 7000 users accessed MERRA-2 data at GES DISC in the year 2021, ~1000 more users than in the year 2020. In this presentation, we will introduce the MERRA-2 datasets associated with aerosol and air quality studies and use a wildfire case study to demonstrate the data tools developed at GES DISC to analyze and visualize MERRA-2 data, such as Giovanni and the level 3 and level 4 subsetter, and Jupyter Python notebook. We will also update the status of cloud migration of the MERRA-2 data to Amazon Web Services (AWS).

Xiaohua Pan↗
DOE OSTI2024

Deep Learning for Fish Identification from Sonar Data (CRADA 481 Final Report)

In eastern regions of the United States, the American eel is a species of management and regulatory concern because of significant population declines, despite the species’ previous abundance in all tributaries of rivers flowing into the Atlantic Ocean. The American eel is also a candidate for listing under the U.S. Endangered Species Act. While hydropower construction and operation are only one of several factors contributing to this population decline, such a listing could impose additional regulatory challenges for a large number of hydropower projects. In this CRADA project, we improved technologies for identifying migrating eels with the goal of reducing the cost and time required for future American eel hydropower impact assessment and mitigation studies, while maintaining accuracy. We built on results from a previous FOA project (FOA# DE-FOA-0001662), led by the Electric Power Research Institute (EPRI), which developed a highly accurate, deep-learning method for identifying migrating eels from imaging sonar data. The current study aimed to further optimize this deep-learning model, originally designed for image classification, and to develop an object detection software capable of identifying fish from sonar videos in real time, enabling the detection of events like fish migrations and specific species, such as the American eel, at hydropower dams. The data conversion algorithms were packaged as software with a graphical user interface, and the software is evaluated by external collaborators. We focused on the American eel in this project and explored the transferability of the developed deep learning models to the sea lamprey, given the similar body shape and swimming behavior between the two species.

13 HYDRO ENERGY↗
NASA NTRS2017

Data Recovery Effort of Nimbus Era Observations by the NASA GES DISC

NASA launched seven Nimbus meteorological satellites in the 1960s and 70s. These satellites carried instruments for making observations of the Earth in the visible, infrared, ultraviolet, and microwave wavelengths. The original data archive consisted of a combination of magnetic tapes and various film media. As these media are well past their expected end of life, the valuable data they contain are now being migrated to the GES DISC modern online archive. The process involves recovering the digital data files from the tapes as well as scanning images of the data from film strips. This presentation will address the status and challenges of recovering the Nimbus data. The old data products were written on now obsolete hardware systems and outdated file formats. They lack any metadata standards and each product is often written in its own proprietary file structure. This requires creating metadata by reading the contents of the old data files. The job is tedious and laborious, as documentation may be incomplete, data files and tapes are sometimes corrupted, or were improperly copied at the time they were created.

Nimbus↗
NASA NTRSDate not supplied

Cloud-Based Time Series Analysis of Extremes: Use Cases and Applications

"Extreme weather events, such as hurricanes, tornadoes, floods, droughts, heatwaves, and blizzards, can cause widespread damage, disrupting ecosystems, agricultural production, and economies. The frequency and intensity of these events have been increasing, likely due to climate change, raising concerns and the need for more accurate analysis and predictions. NASA's Goddard Earth Sciences Data and Information Services Center (GES DISC) has migrated its long-term historical datasets, including precipitation data from MERRA-2 reanalysis, GLDAS land data assimilation, and IMERG satellite observations, to the cloud. This cloud-stored data enables scientists and researchers to utilize cloud computing for advanced modeling and forecasting of extreme weather events, eliminating the need to download large datasets. In this presentation, we will provide an overview of the cloud-based data and services managed by GES DISC; demonstrate methods for accessing and analyzing time series data stored in the cloud; and compare results across various datasets to address critical questions related to extreme precipitation. We will present use cases including: 1. Determining the average total precipitation in California during January and February from 2000 to 2024, and identifying anomalous precipitation in 2021. 2. Calculating the 10, 20, 50, and 100-year return periods for maximum daily rainfall based on 25 years of historical precipitation data (2000-2024) for Maryland.

time series↗
NASA NTRS2007

NavP: Structured and Multithreaded Distributed Parallel Programming

We present Navigational Programming (NavP) -- a distributed parallel programming methodology based on the principles of migrating computations and multithreading. The four major steps of NavP are: (1) Distribute the data using the data communication pattern in a given algorithm; (2) Insert navigational commands for the computation to migrate and follow large-sized distributed data; (3) Cut the sequential migrating thread and construct a mobile pipeline; and (4) Loop back for refinement. NavP is significantly different from the current prevailing Message Passing (MP) approach. The advantages of NavP include: (1) NavP is structured distributed programming and it does not change the code structure of an original algorithm. This is in sharp contrast to MP as MP implementations in general do not resemble the original sequential code; (2) NavP implementations are always competitive with the best MPI implementations in terms of performance. Approaches such as DSM or HPF have failed to deliver satisfying performance as of today in contrast, even if they are relatively easy to use compared to MP; (3) NavP provides incremental parallelization, which is beyond the reach of MP; and (4) NavP is a unifying approach that allows us to exploit both fine- (multithreading on shared memory) and coarse- (pipelined tasks on distributed memory) grained parallelism. This is in contrast to the currently popular hybrid use of MP+OpenMP, which is known to be complex to use. We present experimental results that demonstrate the effectiveness of NavP.

navigational programming (NavP)↗
NASA NTRS1986

PCDS overview

A brief overview of the philosophy, design, and organization of the Pilot Climate Data System (PCDS) is presented. The design has evolved, over time, to deal with numerous data sets, provide appropriate tools for the manipulation and display of such data, and support a wide variety of users. The user interface to the system is provided by the Transportable Applications Executive (TAE). A user can choose one of the five major subsystems from the root menu display. These subsystems are CATALOG, INVENTORY, DATA ACCESS, DATA MANIPULATION, and GRAPHICS. Each subsystem was discussed and examples were given, demonstrating the flexibility provided by the existing PCDS software, Version 3.3. Three of these subsystems are presently being redesigned to provide a more flexible and friendly structure, to allow more consistency and facilitate the ability to support new data sets. Ongoing developmental efforts to migrate from the Version 3.3 prototype phase of PCDS to the operational Version 4.0 stress increased flexibility and data independence.

Treinish, L.↗
NASA NTRS2019

"Sensor Web Evolution - Webs of Webs for NASA Science - Focus on small Uninhabited Aerial Systems (sUAS)"

This paper will describe the evolution of information collection, derivation and delivery mechanisms in webs of NASA sensor webs, with a focus on recent advancements in small Uninhabited Aerial Systems (sUAS). I will discuss the movement to "Fog Computing", also known as Edge Computing. Fog Computing facilitates the distribution of common operations and networking between edge devices and cloud computing facilities, optimizing the production of actionable intelligence. Initially, sUASs utilized onboard data collection as standard, with minimal data downloaded directly. Information products were derived in conventional computational environments, generally desk top computers, and information products made available to the Science Community in weeks or months. With the increased availability, and increasingly lower costs, of beyond line of sight (BLOS) satellite based communication, transmission rates and data volumes increased, and processing migrated to Cloud based services. Contemporary sUASs are moving some of that information product derivation to on vehicle services, and are creating a distributed Cloud/Fog environment. I will describe the technological advances that have made this possible, including low power multi-core Central Processing Units (CPU), and, more recently, the availability of high end Graphical Processing Units (GPU) that consume only a few watts. Intelligent system software, leveraging these hardware advances, finally allows for information product generation on-board, rather than simple data collection. Additionally, intelligent flight control systems now support mutual vehicle to vehicle collaboration, allowing sUASs to create ad-hoc sensor webs on demand, as required. Also discussed will be the lessons learned by the Authors' development of data systems for NASA's large High Altitude Long Endurance (HALE) UASs like Predator and Global Hawk, and how those lessons are being applied to sUAS development. This paper will focus on application, rather a deep dive into the technology, and will highlight improving data management through these new technologies.

Sensor Web↗
NASA NTRSDate not supplied

UASs in the VOG/Edge/FOG Sensor Web Environment

This paper will describe the evolution of information collection, derivation and delivery mechanisms in sensor webs utilizing Uninhabited Aerial Systems (UAS).We will discuss the movement to "Fog Computing", also known as Edge Computing. Fog Computing facilitates the distribution of common operations and networking between edge devices and cloud computing facilities, optimizing the production of actionable intelligence. Initially, UASs utilized onboard data collection as standard, with minimal data downloaded directly. Information products were derived in conventional computational environments, generally desk top computers, and information products made available to the Science Community in weeks or months. With the increased availability, and increasingly lower costs, of beyond line of sight (BLOS) satellite based communication, transmission rates and data volumes increased, and processing migrated to Cloud based services. Contemporary UASs are moving some of that information product derivation to on vehicle services, and are creating a distributed Cloud/Fog environment. The Author will describe the technological advances that have made this possible, including low power multi-core Central Processing Units (CPU), and, more recently, the availability of high end Graphical Processing Units (GPU) that consume only a few watts. Intelligent system software, leveraging these hardware advances, finally allows for information product generation on-board, rather than simple data collection. Additionally, intelligent flight control systems now support mutual vehicle to vehicle collaboration, allowing UASs to create ad-hoc sensor webs on demand, as required. Also discussed will be the lessons learned by the Authors' development of data systems for NASA's large High Altitude Long Endurance (HALE) UASs like Predator and Global Hawk, and how those lessons are being applied to other UAS development This paper will focus on applications, rather a deep dive into the technology, and will highlight improving data management through these new technologies.

UAS↗
DOE OSTI2025

Voltage cycling as a dynamic operation mode for high temperature electrolysis solid oxide cells

Solid Oxide Electrolysis Cells (SOECs) have emerged as a promising technology for the efficient production of H2 via high-temperature electrolysis. However, power input from dynamic energy sources remains a significant challenge for their long-term stability. It is important to analyze the tolerance of cells under dynamic operation conditions. This study focuses on evaluating the impact of voltage cycling on the performance and durability of electrode-supported SOECs. We explore the operational limits and degradation mechanisms of SOECs subjected to various voltage conditions and find that the cells have high tolerance for dynamic voltage. Voltage cycling between 1.3 V and 1.5 V for 9000 cycles does not damage the cell. Conversely, cycling to higher voltages (≥1.7 V) results in accelerated degradation. Advanced characterization is used to screen for various degradation modes post operation. Within the oxygen electrode, XRD and STEM EDS find compositional and phase evolution in all voltage cycled samples including increased decomposition of the air electrode resulting in cation migration. Microstructural analysis of the fuel electrode from nano-CT data shows minimal change throughout the sample set and no evidence of Ni migration, indicating the fuel electrode is stable and not impacted by cycling to higher voltages within the timeframe studied.

Zhu, Zhikuan↗
DOE OSTI2025

Recent Experience with the CMS Data Management System

The CMS[1] experiment manages a large-scale data infrastructure, currently handling over 200 PB of disk and 500 PB of tape storage and transferring more than 1 PB of data per day on average between various WLCG[2] sites. Utilizing Rucio[3] for high-level data management, FTS[4] for data transfers, and a variety of storage and network technologies at the sites, CMS confronts inevitable challenges due to the system’s growing scale and evolving nature. Key challenges include managing transfer and storage failures, optimizing data distribution across different storages based on production and analysis needs, implementing necessary technology upgrades and migrations, and efficiently handling user requests. The data management team has established comprehensive monitoring to supervise this system and has successfully addressed many of these challenges. The team’s efforts aim to ensure data availability and protection, minimize failures and manual interventions, maximize transfer throughput and resource utilization, and provide reliable user support. This paper details the operational experience of CMS with its data management system in recent years, focusing on the encountered challenges, the effective strategies employed to overcome them and the ongoing challenges as we prepare for future demands.

Öztürk, Hasan [CERN]↗
NASA NTRS1981

A preliminary analysis of the data from experiment 77-13 and final report on glass fining experiments in zero gravity

Thermal fining, thermal migration of bubbles under reduced gravity conditions, and data to verify current theoretical models of bubble location and temperatures as a function of time are discussed. A sample, sodium borate glass, was tested during 5 to 6 minutes of zero gravity during rocket flight. The test cell contained a heater strip; thermocouples were in the sample. At present quantitative data are insufficient to confirm results of theoretical calculations.

Wilcox, W. R.↗
NASA NTRS2018

Ultralow Frequency Waves as an Intermediary for Solar Wind Energy Input Into the Radiation Belts

Enhancements of electron fluxes in the outer radiation belt have been closely linked to increases in solar wind speed and density as well as to prolonged intervals of southward interplanetary magnetic field. Periodic oscillations in the Earth's magnetic field with frequencies in the range of a few millihertz (ultralow frequency or ultralow frequency waves) may be an intermediary through which these solar wind drivers influence radiation belt dynamics due to their potential for resonant interactions with energetic electrons causing the radial migration of resonant electrons. Using data from more than 180 ground magnetometers contributing to the worldwide SuperMAG collaboration, we explore possible relationships between relativistic electron flux variations and the spatial and temporal profiles of ultralow frequency wave power contained in the Pc5 frequency band (2–7 mHz). During 19 geomagnetic storms marked by relativistic (1.5 MeV < E < 6 MeV) electron flux enhancements and 19 storms that led to prolonged electron flux depletions, Pc5 wave power is found penetrating to L shells as low as 2–3. The enhancement of Pc5 wave power starts almost simultaneously with the storm onset. The depth of wave activity penetration was found associated with the strength of geomagnetic activity (Spearman's ρ = 0.54), which is also related to the location of electron flux maximum observed in the recovery phase. Pc5 wave activity persists longer (for up to ≈62 hr) for those storms that produced relativistic electrons. We also investigate the combination of interplanetary conditions necessary to differentiate the response of relativistic electron fluxes to geomagnetic storms. A coupling function that captures the increased reconnection rate at the dayside magnetopause affecting magnetospheric processes which may produce Pc5 wave power offers an additional key to further understanding the outer belt dynamics.

Georgiou, M.↗
NASA NTRS2018

Important Changes to User Access at the NASA CDDIS

The Crustal Dynamics Data Information System (CDDIS) supports data archiving and distribution activities for the space geodesy and geodynamics community. The main objectives of the system are to make space geodesy and geodynamics related data and derived products available in a central archive, to maintain information about the archival of these data, to disseminate these data and information in a timely manner to a global scientific research community, and to provide user based tools for the exploration and use of the archive. Since its inception, the user community has utilized anonymous ftp for accessing and downloading files from the CDDIS archive. Although this protocol allows users to easily automate file downloads, many organizations, data systems, and users have already migrated from ftp or are actively pursuing a move away from the protocol due to problems from a system and security standpoint. Furthermore, U.S. Government agencies have become increasingly concerned about this legacy protocol and ensuring data integrity for the user community have begun recently to disallow the use of the ftp protocol. The CDDIS, operated by NASA GSFC, must therefore address these concerns and provide alternative methods for access to its archive for continued easy and automated download of its contents. This poster will discuss the upcoming changes at CDDIS and provide examples on transitioning from anonymous ftp.

Noll, Carey E.↗
DOE OSTI2024

Site Characterization of the Highest-Priority Geologic Formations for CO2 Storage in Wyoming

The project Site Characterization of the Highest-Priority Geologic Formations for CO2 Storage in Wyoming is one of 9 site characterization projects that were implemented as part of ARRA (American Recovery and Reinvestment Act). Data from this project was used to improve resolution of data in NATCARB in the area of study. Data related to this study has already been incorporated in NATCARB Atlas. The Wyoming Carbon Underground Storage Project (WY-CUSP) consisted of CO2 storage site characterization and evaluation, focusing on Wyoming’s most promising CO2 storage reservoirs (the Pennsylvanian Weber/Tensleep Sandstone and Mississippian Madison Limestone) and premier CO2 storage site (Rock Springs Uplift). Results from the WY-CUSP project suggest the two reservoirs could store up to 17,000 million tons of CO2. The WY-CUSP team drilled a stratigraphic test well and acquired a 3-D seismic survey covering 25 square miles of the Rock Springs Uplift site. The team retrieved 916 feet of core from the 12,810-foot-deep well, along with a complete log suite, borehole images, fluid samples, and other data. Project partners (1) provided continuous visual documentation of the core, including grain size, mineralogy, facies distribution, and porosity; (2) performed continuous permeability and velocity scans of selected reservoir intervals; and (3) chemically analyzed the fluid samples. WY-CUSP scientists integrated seismic attributes with observations from log suites, a VSP survey, core, fluid samples, and laboratory analyses, including continuous permeability scans. From these integrations, researchers constructed 3-D spatial distribution volumes of reservoir and seal properties that represent geological heterogeneity at the targeted CO2 storage site. The WY-CUSP team used this data to perform new CO2 plume migration simulations. Baker Hughes, Inc., completed a series of small-scale, in-situ water injectivity measurements. A database was formed when observations, analyses, and experiments from the stratigraphic test well were integrated. Correlation of these data allowed petrophysical parameters to be extrapolated from the test well out into the storage domain (5x5 mile 3-D seismic survey volume). This resulted in an improved, realistic understanding of performance assessments for potential CO2 storage scenarios. The WY-CUSP team worked on (1) improving CO2 storage resource estimates, (2) establishing long-term integrity and permanence of confining layers, (3) designing a profitable strategy for pressure management, and (4) evaluating the utilization of stored CO2 at the Rock Spring Uplift. Finally, Baker Hughes developed a microseismic baseline for the test site using in-bore geophones to complete field operations.

3-D seismic↗
NASA NTRS2018

Helioseismic Observations of Torsional Oscillations Inside the Sun and Their Potential for Predicting Solar Cycles

The helioseismic analysis of torsional oscillations of the Sun, obtained in 1996-2018 from SOHO (Solar and Heliospheric Observatory) and SDO (Solar Dynamics Observatory), reveals the spatio-temporal dynamics associated with the dynamo process. The data reveal new relationships between the migrating magnetic field patterns observed in synoptic magnetograms and the dynamics of torsional oscillations near the surface and in the interior. In particular, it is found that the evolution of torsional oscillations in the deep convection zone is ahead of the surface magnetic evolution by several years, and that it is related to the extended solar cycle phenomenon previously observed in the solar corona. The data show substantial differences in the torsional oscillation properties between Cycles 23 and 24 indicating fundamental changes in the dynamo regime, and also reveal initiation of Cycle 25. The helioseismology observations of the torsional oscillations open new perspectives for understanding the global dynamo processes inside the Sun, and for predicting the next solar cycle.

Oscillations↗
NASA NTRS2012

Processing NASA Earth Science Data on Nebula Cloud

Three applications were successfully migrated to Nebula, including S4PM, AIRS L1/L2 algorithms, and Giovanni MAPSS. Nebula has some advantages compared with local machines (e.g. performance, cost, scalability, bundling, etc.). Nebula still faces some challenges (e.g. stability, object storage, networking, etc.). Migrating applications to Nebula is feasible but time consuming. Lessons learned from our Nebula experience will benefit future Cloud Computing efforts at GES DISC.

Chen, Aijun↗
NASA NTRS1984

Pipelined digital SAR azimuth correlator using hybrid FFT-transversal filter

A synthetic aperture radar system (SAR) having a range correlator is provided with a hybrid azimuth correlator which utilizes a block-pipe-lined fast Fourier transform (FFT). The correlator has a predetermined FFT transform size with delay elements for delaying SAR range correlated data so as to embed in the Fourier transform operation a corner-turning function as the range correlated SAR data is converted from the time domain to a frequency domain. The azimuth correlator is comprised of a transversal filter to receive the SAR data in the frequency domain, a generator for range migration compensation and azimuth reference functions, and an azimuth reference multiplier for correlation of the SAR data. Following the transversal filter is a block-pipelined inverse FFT used to restore azimuth correlated data in the frequency domain to the time domain for imaging.

Wu, C.↗