Search NASA⌕ Search

SEARCH · Search NASA

Results for “cloud storage”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 37 records · Page 2
All sourcesNASA NTRSDOE OSTI
DOE OSTI2025

ROOT RNTuple and EOS: The Next Generation of Event Data I/O

For several years, the ROOT team is developing the new RNTuple I/O subsystem in preparation of the next generation of collider experiments. Both HL-LHC and DUNE are expected to start data taking by the end of this decade. They pose unprecedented challenges to event data I/O in terms of data rates, event sizes, and event complexity. At the same time, the I/O landscape is becoming more diverse. HPC cluster file systems and object stores, NVMe disk cache layers in analysis facilities, and S3 storage on cloud resources are mixing with traditional XRootD-managed spinning disk pools.The ROOT team will finalize a first production version of the RNTuple binary format by the end of 2024. After this point, ROOT will provide backward compatibility for RNTuple data. This contribution provides an overview of the RNTuple feature set, the related R&D activities and the long-term vision for RNTuple. We report on performance, interface design, tooling, robustness, integration with experiment frameworks, and validation results, as well as recent R&D on parallel reading and writing and exploitation of modern hardware and storage systems. We will give an outlook on possible future features after a first production release.Collaboratively, the IT and EP departments at CERN have launched a formal project within the Research and Computing sector to evaluate the novel data format for physics analysis data utilized in LHC experiments and other fields. This part of the project focuses on validating the scalability of the EOS storage backend during the transition from the over 25 years old TTree production format to the newly developed RNTuple format, using both replicated and erasure-coded storage profiles.

Blomer, Jakob [CERN]↗
NASA NTRS2001

Satellite Retrieval of Atmospheric Water Budget over Gulf of Mexico- Caribbean Basin: Seasonal Variability

This study presents results from a multi-satellite/multi-sensor retrieval system designed to obtain the atmospheric water budget over the open ocean. A combination of hourly-sampled monthly datasets derived from the GOES-8 5 Imager and the DMSP 7-channel passive microwave radiometer (SSM/I) have been acquired for the Gulf of Mexico-Caribbean Sea basin. Whereas the methodology is being tested over this basin, the retrieval system is designed for portability to any open-ocean region. Algorithm modules using the different datasets to retrieve individual geophysical parameters needed in the water budget equation are designed in a manner that takes advantage of the high temporal resolution of the GOES-8 measurements, as well as the physical relationships inherent to the SSM/I passive microwave signals in conjunction with water vapor, cloud liquid water, and rainfall. The methodology consists of retrieving the precipitation, surface evaporation, and vapor-cloud water storage terms in the atmospheric water balance equation from satellite techniques, with the water vapor advection term being obtained as the residue needed for balance. Thus, we have sought to develop a purely satellite-based method for obtaining the full set of terms in the atmospheric water budget equation without requiring in situ sounding information on the wind profile. The algorithm is partly validated by first cross-checking all the algorithm components through multiple-algorithm retrieval intercomparisons. More fundamental validation is obtained by directly comparing water vapor transports into the targeted basin diagnosed from the satellite algorithm to those obtained observationally from a network of land-based upper air stations that nearly uniformly surround the basin. Total columnar atmospheric water budget results will be presented for an extended annual cycle consisting of the months of October-97, January-98, April-98, July-98, October-98, and January-1999. These results are used to emphasize the changing relationship in E-P, as well as in the varying roles of storage and advection in balancing E-P both on daily and monthly time scales and on localized and basin space scales. Results from the algorithm-to-algorithm intercomparisons will also be presented in the context of sensitivity testing to help understand the intrinsic uncertainties in the water budget terms.

Smith, Eric A.↗
NASA NTRS2004

Use of GOES, SSM/I, TRMM Satellite Measurements Estimating Water Budget Variations in Gulf of Mexico - Caribbean Sea Basins

This study presents results from a multi-satellite/multi-sensor retrieval system designed to obtain the atmospheric water budget over the open ocean. A combination of 3ourly-sampled monthly datasets derived from the GOES-8 5-channel Imager, the TRMM TMI radiometer, and the DMSP 7-channel passive microwave radiometers (SSM/I) have been acquired for the combined Gulf of Mexico-Caribbean Sea basin. Whereas the methodology has been tested over this basin, the retrieval system is designed for portability to any open-ocean region. Algorithm modules using the different datasets to retrieve individual geophysical parameters needed in the water budget equation are designed in a manner that takes advantage of the high temporal resolution of the GOES-8 measurements, as well as the physical relationships inherent to the TRMM and SSM/I passive microwave measurements in conjunction with water vapor, cloud liquid water, and rainfall. The methodology consists of retrieving the precipitation, surface evaporation, and vapor-cloud water storage terms in the atmospheric water balance equation from satellite techniques, with the water vapor advection term being obtained as the residue needed for balance. Thus, the intent is to develop a purely satellite-based method for obtaining the full set of terms in the atmospheric water budget equation without requiring in situ sounding information on the wind profile. The algorithm is validated by cross-checking all the algorithm components through multiple- algorithm retrieval intercomparisons. A further check on the validation is obtained by directly comparing water vapor transports into the targeted basin diagnosed from the satellite algorithms to those obtained observationally from a network of land-based upper air stations that nearly uniformly surround the basin, although it is fair to say that these checks are more effective m identifying problems in estimating vapor transports from a leaky operational radiosonde network than in verifying the transport estimates determined from the satellite algorithm system Total columnar atmospheric water budget results are presented for an extended annual cycle consisting of the months of October-97, January-98, April-98, July-98,October-98, and January 1999. These results are used to emphasize the changing relationship in E-P, as well as in the varying roles of storage and advection in balancing E-P both on daily and monthly time scales and on localized and basin space scales. Results from the algorithm-to-algorithm intercomparisons are also presented in the context of sensitivity testing to help understand the intrinsic uncertainties in evaluating the water budget terms by an all-satellite algorithm approach.

Smith, Eric A.↗
NASA NTRS2006

Monthly-Diurnal Water Budget Variability Over Gulf of Mexico-Caribbean Sea Basin from Satellite Observations

This study presents results from a multi-satellite/multi-sensor retrieval system design d to obtain the atmospheric water budget over the open ocean. A combination of hourly-sampled monthly datasets derived from the GOES-8 5-channel Imager, the TRMM TMI radiometer, and the DMSP 7-channel passive microwave radiometers (SSM/I) have been acquired for the combined Gulf of Mexico-Caribbean Sea basin. Whereas the methodology has been tested over this basin, the retrieval system is designed for portability to any open-ocean region. Algorithm modules using the different datasets to retrieve individual geophysical parameters needed in the water budget equation are designed in a manner that takes advantage of the high temporal resolution of the GOES-8 measurements, as well as the physical relationships inherent to the TRMM and SSM/I passive microwave measurements in conjunction with water vapor, cloud liquid water, and rainfall. The methodology consists of retrieving the precipitation, surface evaporation, and vapor-cloud water storage terms in the atmospheric water balance equation from satellite techniques, with the water vapor advection term being obtained as the residue needed for balance. Thus, the intent is to develop a purely satellite-based method for obtaining the full set of terms in the atmospheric water budget equation without requiring in situ sounding information on the wind profile. The algorithm is validated by cross-checking all the algorithm components through multiple-algorithm retrieval intercomparisons. A further check on the validation is obtained by directly comparing water vapor transports into the targeted basin diagnosed from the satellite algorithms to those obtained observationally from a network of land-based upper air stations that nearly uniformly surround the basin, although it is fair to say that these checks are more effective in identifying problems in estimating vapor transports from a "leaky" operational radiosonde network than in verifying the transport estimates determined from the satellite algorithm system. Total columnar atmospheric water budget results are presented for an extended annual cycle consisting of the months of October-97, January-98, April-98, July-98,October-98, and January- 1999. These results are used to emphasize the changing relationship in E-P, as well as in the varying roles of storage and advection in balancing E-P both on daily and monthly time scales and on localized and basin space scales. Results from the algorithm-to-algorithm intercomparisons are also presented in the context of sensitivity testing to help understand the intrinsic uncertainties in evaluating the water budget terms by an all-satellite algorithm approach.

Smith, E. A.↗
NASA NTRSDate not supplied

Surface meteorology and Solar Energy

The Release 5.1 Surface meteorology and Solar Energy (SSE) data contains parameters formulated for assessing and designing renewable energy systems. Parameters fall under 11 categories including: Solar cooking, solar thermal applications, solar geometry, tilted solar panels, energy storage systems, surplus product storage systems, cloud information, temperature, wind, other meteorological factors, and supporting information. This latest release contains new parameters based on recommendations by the renewable energy industry and it is more accurate than previous releases. On-line plotting capabilities allow quick evaluation of potential renewable energy projects for any region of the world. The SSE data set is formulated from NASA satellite- and reanalysis-derived insolation and meteorological data for the 10-year period July 1983 through June 1993. Results are provided for 1 degree latitude by 1 degree longitude grid cells over the globe. Average daily and monthly measurements for 1195 World Radiation Data Centre ground sites are also available. [Mission Objectives] The SSE project contains insolation and meteorology data intended to aid in the development of renewable energy systems. Collaboration between SSE and technology industries such as the Hybrid Optimization Model for Electric Renewables ( HOMER ) may aid in designing electric power systems that employ some combination of wind turbines, photovoltaic panels, or diesel generators to produce electricity. [Temporal_Coverage: Start_Date=1983-07-01; Stop_Date=1993-06-30] [Spatial_Coverage: Southernmost_Latitude=-90; Northernmost_Latitude=90; Westernmost_Longitude=-180; Easternmost_Longitude=180].

Stackhouse, Paul W.↗
NASA NTRS2017

Trade Study: Storing NASA HDF5/netCDF-4 Data in the Amazon Cloud and Retrieving Data via Hyrax Server / THREDDS Data Server

As part of the overall effort to understand implications of migrating ESDIS data and services to the cloud we are testing several common OPeNDAP and HDF use cases against three architectures for general performance and cost characteristics. The architectures include retrieving entire files, retrieving datasets using HTTP range gets, and retrieving elements of datasets (chunks) with HTTP range gets. We will describe these architectures and discuss our approach to estimating cost.

HDF↗
NASA NTRS2018

Earthdata Cloud Analytics Project

This presentation describes a nascent project in NASA to develop a framework to support end-user analytics of NASA's Earth science data in the cloud. The chief benefit of migrating EOSDIS (Earth Observation System Data and Information Systems) data to the cloud is to position the data next to enormous computing capacity to allow end users to process data at scale. The Earthdata Cloud Analytics project will user a service-based approach to facilitate the infusion of evolving analytics technology and the integration with non-NASA analytics or other complementary functionality at other agencies and in other nations.

data services↗
NASA NTRS2017

Giovanni in the Cloud: Earth Science Data Exploration in Amazon Web Services

Giovanni is an exploration tool at the NASA Goddard Earth Sciences Data Information Services Center (GES DISC), providing 22 analysis and visualization services for over 1600 Earth Science data variables. Owing to its popularity, Giovanni has experienced a consistent growth in overall demand, with periodic usage spikes attributed to trainings by education organizations, extensive data analysis in response to natural disasters, preparations for science meetings, etc. Furthermore, the new generation of spaceborne sensors and high resolution models have resulted in an exponential growth in data volume with data distributed across the traditional boundaries of data centers. Seamless exploration of data (without users having to worry about data center boundaries) has been a key recommendation of the GES DISC User Working Group. These factors have required new strategies for delivering acceptable performance. The cloud-based Giovanni, built on Amazon Web Services (AWS), evaluates (1) AWS native solutions to provide a scalable, serverless architecture; (2) open standards for data storage in the Cloud; (3) a cost model for operations; and (4) end-user performance. Our preliminary findings indicate that the use of serverless architecture has a potential to significantly reduce development and operational cost of Giovanni. The combination of using AWS managed services, storage of data in open standards, and schema-on-read data access strategy simplifies data access and analytics, in addition to making data more accessible to the end users of Giovanni through popular programming languages.

Giovanni↗
NASA NTRS2018

Accessing Data Stored in Amazon S3 Using the Hyrax OPeNDAP Server

For three years we have been investigating data storage and retrieval in the Amazon Cloud, striving to optimize both storage structures and (cache-enhanced retrieval processes. These optimizations are hidden from data users, who simply employ the Data Access Protocol (DAP), a popular, Web-based Application Programmer Interface (API) developed by OPeNDAP. We present our collected findings and their realization in the Hyrax data server. Specific techniques include caching data on spinning disk; accessing sharded data in place; optimizing the organizational structures for data within the flat key-value space of Simple Storage Service (S3); and employing an API extension that permits simultaneous operations on many datasets in a single request.

Data access↗
NASA NTRS2018

How to Cloud for Earth Scientists: An Introduction

This presentation is a tutorial on getting started with cloud computing for the purposes of Earth Observation datasets. We first discuss some of the main advantages that cloud computing can provide for the Earth scientist: copious processing power, immense and affordable data storage, and rapid startup time. We also talk about some of the challenges of getting the most out of cloud computing: re-organizing the way data are analyzed, handling node failures and attending.

cloud storage↗
NASA NTRS2017

NASA Update for Unidata Stratcomm

The NASA representative to the Unidata Strategic Committee presented a semiannual update on NASAs work with and use of Unidata technologies. The talk updated Unidata on the program of cloud computing prototypes underway for the Earth Observing System Data and Information System (EOSDIS). Also discussed was a trade study on the use of the Open source Project for a Network Data Access Protocol (OPeNDAP) with Web Object Storage in the cloud.

data systems↗
NASA NTRSDate not supplied

NASA's Implementation of Cloud Services for Human Space Flight

Cloud is a tried-and-true technology used throughout United States government agencies, including the National Aeronautics and Space Administration (NASA). With reliable results and infrequent downtimes, cloud allows for secure remote access, customizability, and streamlined monitoring options, creating an environment for better data integrity and availability. As NASA increasingly migrates functions to the cloud, the Space Communications and Navigation Program (SCaN) program has been investigating how this capability can be leveraged to provide communication services to its users and customers. Currently, missions such as NASA-ISRO Synthetic Aperture Radar (NISAR), Plankton, Aerosol, Cloud, ocean Ecosystem (PACE), and Roman Space Telescope (RST) are planned to incorporate cloud into their data delivery architecture. However, SCaN is looking to expand further. This conversion to using cloud services allows for greater availability of mission data for both robotic and human space flight (HSF)missions. The SCaN program and the Near Space Network (NSN) are working to consolidate resources and create a cloud environment suitable for the entirety of the SCaN program network architecture. SCaN is in the process of finalizing its cloud architecture and soon will be implementing cloud services. The new services used will adhere to federal regulations including Federal Risk and Authorization Management Program (FedRAMP), which is built upon National Institute of Standards and Technology (NIST)documentation. While keeping in mind these security requirements, an auxiliary objective of the cloud integration is to ensure the most cost-efficient solution; providing a scalable, robust and resilient system. Using cloud services, NASA will gain access to better centralized monitoring and management features, along with customizable services on a pay-per-use plan. With the ever-growing NASA mission data volume needs, maintaining ample storage space is another major constraint. Processing and storing such large amounts of data, on the order of terabytes a day, requires dynamic processing capability which is inherently a strength of cloud computing. By routing this data from ground stations through the cloud, there will be greater ease of access for both SCaN and the user community. Artificial intelligence and other built-in cloud functions can also enhance efficiency, improving data processing time. Thereby also allowing for better data availability. As we look to the future of cloud services, NASA will continue to leverage capabilities that will benefit NASA’s ability to provide cost-effective communication services. This paper further outlines the evolution of cloud use by SCaN in the context of Human Space Flight.

cloud storage↗
NASA NTRS2021

On-demand Command and Control of ASTERIA with Cloud-based Ground Station Services

ASTERIA (Arcsecond Space Telescope Enabling Research in Astrophysics) was a 6-unit CubeSat technology demonstration mission that deployed from the International Space Station on November 20th, 2017. After successfully completing its 90-day primary mission that demonstrated arcsecond-level line-of-sight pointing and focal plane thermal stability for exoplanet detection, it entered an extended mission performing onboard software demonstrations to mature technology both in space and on the ground. One of the technologies was a completely cloud-based ground system leveraging Amazon Web Services (AWS) Ground Station service.Announced in December 2018 and launched in May 2019, AWS Ground Station is a fully managed ground station service that aims to reduce the overhead associated with developing and maintaining ground system infrastructure throughout the mission lifecycle. AWS Ground Station makes available the suite of features required for any ground system in support of low-Earth orbit (LEO) and medium-Earth Orbit (MEO) satellite operations on-demand and without setting up or maintaining long-term contracts. Charges are incurred on a per-minute basis for antenna usage during scheduled tracks. Support is available for S-band uplink and downlink, along with X-band narrowband and wideband downlink. Missions that use the service may reserve tracks with any licensed AWS Ground Station antennas located across each service region and have direct access to any AWS services in support of mission operations.The cloud-based architecture built around the AWS Ground Station service greatly enhanced ASTERIA mission operations by enabling end-to-end pass automation, on-demand contact scheduling and contingency planning, along with more efficient data downlink through station availability and station-to-station handovers. It incorporated open-source software, particularly NASA's AMMOS Instrument Toolkit (AIT) and Open Mission Control Technologies (OpenMCT), along with the AWS application programming interfaces (API) to the Ground Station, Elastic Compute Cloud (EC2) and Simple Storage Service (S3) services. After showcasing operability in August 2019, the team continued using and improving this novel ground system architecture until the end of mission in December 2019. This paper describes the cloud-based ground system, how it was designed, tested, and evaluated with an in-orbit spacecraft, the operational capabilities that it enabled, along with lessons learned and recommendations for future missions.

Fesq, Lorraine↗
NASA NTRS2021

On-demand Command and Control of ASTERIA with Cloud-based Ground Station Services

ASTERIA (Arcsecond Space Telescope Enabling Research in Astrophysics) was a 6-unit CubeSat technology demonstration mission that deployed from the International Space Station on November 20th, 2017. After successfully completing its 90-day primary mission that demonstrated arcsecond-level line-of-sight pointing and focal plane thermal stability for exoplanet detection, it entered an extended mission performing onboard software demonstrations to mature technology both in space and on the ground. One of the technologies was a completely cloud-based ground system leveraging Amazon Web Services (AWS) Ground Station service. Announced in December 2018 and launched in May 2019, AWS Ground Station is a fully managed ground station service that aims to reduce the overhead associated with developing and maintaining ground system infrastructure throughout the mission lifecycle. AWS Ground Station makes available the suite of features required for any ground system in support of low-Earth orbit (LEO) and medium-Earth Orbit (MEO) satellite operations on-demand and without setting up or maintaining long-term contracts. Charges are incurred on a per-minute basis for antenna usage during scheduled tracks. Support is available for S-band uplink and downlink, along with X-band narrowband and wideband downlink. Missions that use the service may reserve tracks with any licensed AWS Ground Station antennas located across each service region and have direct access to any AWS services in support of mission operations. The cloud-based architecture built around the AWS Ground Station service greatly enhanced ASTERIA mission operations by enabling end-to-end pass automation, on-demand contact scheduling and contingency planning, along with more efficient data downlink through station availability and station-tostation handovers. It incorporated open-source software, particularly NASA's AMMOS Instrument Toolkit (AIT) and Open Mission Control Technologies (OpenMCT), along with the AWS application programming interfaces (API) to the Ground Station, Elastic Compute Cloud (EC2) and Simple Storage Service (S3) services. After showcasing operability in August 2019, the team continued using and improving this novel ground system architecture until the end of mission in December 2019. This paper describes the cloud-based ground system, how it was designed, tested, and evaluated with an inorbit spacecraft, the operational capabilities that it enabled, along with lessons learned and recommendations for future missions.

Fesq, Lorraine↗
NASA NTRS2017

Archive Management of NASA Earth Observation Data to Support Cloud Analysis

NASA collects, processes and distributes petabytes of Earth Observation (EO) data from satellites, aircraft, in situ instruments and model output, with an order of magnitude increase expected by 2024. Cloud-based web object storage (WOS) of these data can simplify the execution of such an increase. More importantly, it can also facilitate user analysis of those volumes by making the data available to the massively parallel computing power in the cloud. However, storing EO data in cloud WOS has a ripple effect throughout the NASA archive system with unexpected challenges and opportunities. One challenge is modifying data servicing software (such as Web Coverage Service servers) to access and subset data that are no longer on a directly accessible file system, but rather in cloud WOS. Opportunities include refactoring of the archive software to a cloud-native architecture; virtualizing data products by computing on demand; and reorganizing data to be more analysis-friendly.

Lynnes, Christopher↗
NASA NTRS2017

Archive Management of NASA Earth Observation Data to Support Cloud Analysis

NASA collects, processes and distributes petabytes of Earth Observation (EO) data from satellites, aircraft, in situ instruments and model output, with an order of magnitude increase expected by 2024. Cloud-based web object storage (WOS) of these data can simplify the execution of such an increase. More importantly, it can also facilitate user analysis of those volumes by making the data available to the massively parallel computing power in the cloud. However, storing EO data in cloud WOS has a ripple effect throughout the NASA archive system with unexpected challenges and opportunities. One challenge is modifying data servicing software (such as Web Coverage Service servers) to access and subset data that are no longer on a directly accessible file system, but rather in cloud WOS. Opportunities include refactoring of the archive software to a cloud-native architecture; virtualizing data products by computing on demand; and reorganizing data to be more analysis-friendly. Reviewed by Mark McInerney ESDIS Deputy Project Manager.

Lynnes, Christopher↗
NASA NTRS2016

Near Real-Time Flood Monitoring and Impact Assessment Systems

Floods are the costliest natural disaster, causing approximately 6.8 million deaths in the twentieth century alone. Worldwide economic flood damage estimates in 2012 exceed $19 Billion USD. Extended duration floods also pose longer term threats to food security, water, sanitation, hygiene, and community livelihoods, particularly in developing countries. Projections by the Intergovernmental Panel on Climate Change (IPCC) suggest that precipitation extremes, rainfall intensity, storm intensity, and variability are increasing due to climate change. Increasing hydrologic uncertainty will likely lead to unprecedented extreme flood events. As such, there is a vital need to enhance and further develop traditional techniques used to rapidly assess flooding and extend analytical methods to estimate impacted population and infrastructure. Measuring flood extent in situ is generally impractical, time consuming, and can be inaccurate. Remotely sensed imagery acquired from space-borne and airborne sensors provides a viable platform for consistent and rapid wall-to-wall monitoring of large flood events through time. Terabytes of freely available satellite imagery are made available online each day by NASA, ESA, and other international space research institutions. Advances in cloud computing and data storage technologies allow researchers to leverage these satellite data and apply analytical methods at scale. Repeat-survey earth observations help provide insight about how natural phenomena change through time, including the progression and recession of floodwaters. In recent years, cloud-penetrating radar remote sensing techniques (e.g., Synthetic Aperture Radar) and high temporal resolution imagery platforms (e.g., MODIS and its 1-day return period), along with high performance computing infrastructure, have enabled significant advances in software systems that provide flood warning, assessments, and hazard reduction potential. By incorporating social and economic data, researchers can develop systems that automatically quantify the socioeconomic impacts resulting from flood disaster events.

Ahamed, Aakash↗