Search NASA⌕ Search

SEARCH · Search NASA

Results for “cloud storage”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 73 records · Page 4
All sourcesNASA NTRSDOE OSTI
NASA NTRSDate not supplied

Breaking Barriers: Integrating Geo-Leo Aerosol Data with an Open-Source Approach

The scientific community is still examining the novel data from geostationary satellite observations and evaluating methods for effectively fusing the polar observations with various spatial and temporal resolutions. However, the merged data will present a significant ""Big Data"" challenge, including processing, storage, data discoverability, accessibility, and migration within cloud computing environments. We have developed an open-source package to fuse aerosol optical depths (AOD) products from six satellite sensors in the past four years (2019~2023), and this presentation will update our recent progress. Using this Python-based package, we produced a level 3 global (AOD) product in a quarter-degree spatial resolution every half-hour, fusing the Level 2 AOD data with the Dark Target aerosol retrieval algorithm from six satellites: three geostationary (GOES-16/17 and Himawari-8) with high temporal resolution, and three polar orbiting (TERRA/MODIS, AQUA/MODIS, and SNPP-VIIRS) with global coverage. By integrating these observations, the diurnal cycle of global AOD in this fused product can be characterized at local, regional, and global scales. Furthermore, we are committed to openness and transparency by providing our package and its associated functionalities as open-source. Our dedication to adhering to the FAIR, CARE, and TRUST principles ensures that our users can rely on the integrity and ethical standards of our work. For instance of Interoperability, this package fuses remote sensing products on demand into desired temporal and spatial domains. It can be run in a central processing unit (CPU) or a Graphics processing unit (GPU) mode. This package will empower researchers and practitioners to use satellite and sensor data efficiently in various applications and research.

Xiaohua Pan↗
NASA NTRS2017

Trade Study: Storing NASA HDF5/netCDF-4 Data in the Amazon Cloud and Retrieving Data Via Hyrax Server Data Server

This study explored three candidate architectures with different types of objects and access paths for serving NASA Earth Science HDF5 data via Hyrax running on Amazon Web Services (AWS). We studied the cost and performance for each architecture using several representative Use-Cases. The objectives of the study were: Conduct a trade study to identify one or more high performance integrated solutions for storing and retrieving NASA HDF5 and netCDF4 data in a cloud (web object store) environment. The target environment is Amazon Web Services (AWS) Simple Storage Service (S3). Conduct needed level of software development to properly evaluate solutions in the trade study and to obtain required benchmarking metrics for input into government decision of potential follow-on prototyping. Develop a cloud cost model for the preferred data storage solution (or solutions) that accounts for different granulation and aggregation schemes as well as cost and performance trades.We will describe the three architectures and the use cases along with performance results and recommendations for further work.

AWS cost↗
NASA NTRS1982

Infrared remote sensing of the vertical and horizontal distribution of clouds

An algorithm has been developed to derive the horizontal and vertical distribution of clouds from the same set of infrared radiance data used to retrieve atmospheric temperature profiles. The method leads to the determination of the vertical atmospheric temperature structure and the cloud distribution simultaneously, providing information on heat sources and sinks, storage rates and transport phenomena in the atmosphere. Experimental verification of this algorithm was obtained using the 15-micron data measured by the NOAA-VTPR temperature sounder. After correcting for water vapor emission, the results show that the cloud cover derived from 15-micron data is less than that obtained from visible data.

Chahine, M. T.↗
NASA NTRS2003

Observing the Global Water Cycle from Space

This paper presents an approach to measuring all major components of the water cycle from space. The goal of the paper is to explore the concept of using a sensor-web of satellites to observe the global water cycle. The details of the required measurements and observation systems are therefore only an initial approach and will undergo future refinement, as their details will be highly important. Key elements include observation and evaluation of all components of the water cycle in terms of the storage of water-in the ocean, air, cloud and precipitation, in soil, ground water, snow and ice, and in lakes and rivers-and in terms of the global fluxes of water between these reservoirs. For each component of the water cycle that must be observed, the appropriate temporal and spatial scales of measurement are estimated, along with the some of the frequencies that have been used for active and passive microwave observations of the quantities. The suggested types of microwave observations are based on the heritage for such measurements, and some aspects of the recent heritage of these measurement algorithms are listed. The observational requirements are based on present observational systems, as modified by expectations for future needs. Approaches to the development of space systems for measuring the global water cycle can be based on these observational requirements.

Hildebrand, Peter H.↗
NASA NTRS2004

Observing the Global Water Cycle from Space

This paper presents an approach to measuring all major components of the water cycle from space. Key elements of the global water cycle are discussed in terms of the storage of water-in the ocean, air, cloud and precipitation, in soil, ground water, snow and ice, and in lakes and rivers, and in terms of the global fluxes of water between these reservoirs. Approaches to measuring or otherwise evaluating the global water cycle are presented, and the limitations on known accuracy for many components of the water cycle are discussed, as are the characteristic spatial and temporal scales of the different water cycle components. Using these observational requirements for a global water cycle observing system, an approach to measuring the global water cycle from space is developed. The capabilities of various active and passive microwave instruments are discussed, as is the potential of supporting measurements from other sources. Examples of space observational systems, including TRMM/GPM precipitation measurement, cloud radars, soil moisture, sea surface salinity, temperature and humidity profiling, other measurement approaches and assimilation of the microwave and other data into interpretative computer models are discussed to develop the observational possibilities. The selection of orbits is then addressed, for orbit selection and antenna size/beamwidth considerations determine the sampling characteristics for satellite measurement systems. These considerations dictate a particular set of measurement possibilities, which are then matched to the observational sampling requirements based on the science. The results define a network of satellite instrumentation systems, many in low Earth orbit, a few in geostationary orbit, and all tied together through a sampling network that feeds the observations into a data-assimilative computer model.

Hildebrand, P. H.↗
NASA NTRS2012

Comets as Messengers from the Early Solar System - Emerging Insights on Delivery of Water, Nitriles, and Organics to Earth

The question of exogenous delivery of water and organics to Earth and other young planets is of critical importance for understanding the origin of Earth's volatiles, and for assessing the possible existence of exo-planets similar to Earth. Viewed from a cosmic perspective, Earth is a dry planet, yet its oceans are enriched in deuterium by a large factor relative to nebular hydrogen and analogous isotopic enrichments in atmospheric nitrogen and noble gases are also seen. Why is this so? What are the implications for Mars? For icy Worlds in our Planetary System? For the existence of Earth-like exoplanets? An exogenous (vs. outgassed) origin for Earth's atmosphere is implied, and intense debate on the relative contributions of comets and asteroids continues - renewed by fresh models for dynamical transport in the protoplanetary disk, by revelations on the nature and diversity of volatile and rocky material within comets, and by the discovery of ocean-like water in a comet from the Kuiper Belt (cf., Mumma & Charnley 2011). Assessing the creation of conditions favorable to the emergence and sustenance of life depends critically on knowledge of the nature of the impacting bodies. Active comets have long been grouped according to their orbital properties, and this has proven useful for identifying the reservoir from which a given comet emerged (OC, KB) (Levison 1996). However, it is now clear that icy bodies were scattered into each reservoir from a range of nebular distances, and the comet populations in today's reservoirs thus share origins that are (in part) common. Comets from the Oort Cloud and Kuiper Disk reservoirs should have diverse composition, resulting from strong gradients in temperature and chemistry in the proto-planetary disk, coupled with dynamical models of early radial transport and mixing with later dispersion of the final cometary nuclei into the long-term storage reservoirs. The inclusion of material from the natal interstellar cloud is probable, for comets formed in the outer solar system.

Mumma, Michael J.↗
DOE OSTI2024

HPC and Cloud Convergence Beyond Technical Boundaries: Strategies for Economic Sustainability, Standardization, and Data Accessibility

At the IEEE/ACM International Conference for High-Performance Computing, Networking, Storage, and Analysis (SC23), held in Denver, experts discussed the convergence of high-performance computing and cloud computing. Experts explored how this integration could address current scientific computing limitations, enhance computational capabilities, and foster global collaboration while focusing on economic, security, technical, and community challenges and opportunities.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2026

Grid Operator Analytics and Assessment Tools for Inverter- Based Resources Dominated Grid (GOAAT-IBR) Project Update

This presentation provides an update on the OPTIMA GOAAT project, with emphasis on the cloud-native data platform developed in-house to ingest, manage, and operationalize high-resolution power system data. Since our last NASPI presentation, accessible via OSTI ID #2671437, the project team advanced the design and deployment of a scalable architecture capable of handling both synchronized and non-synchronized streams, including PMU, point-on-wave (POW), COMTRADE, and SCADA data. These materials review the project status, recent progress, and key lessons learned. The core of the presentation examines the architecture and engineering of our cloud-native ingestion and data management platform. We then explain how pipelines were designed to collect, normalize, time-align, store, and serve heterogeneous data at scale. We will discuss design choices such as data models, streaming versus batch ingestion, storage tiers, and interoperability with analytics applications. Practical experiences with cloud-native technologies were shared during the event, including benefits, limitations, and integration challenges in a utility environment, along with methods used to improve performance, reduce latency, and optimize resource usage. The presentation also showcases user interface designs and visualization tools that convert raw measurements and analytics results into intuitive, actionable insights for operators and engineers. During the presentation examples were provided demonstrating how visualization, event views, and summarized analytics enhance situational awareness and support operational decision-making. These use cases illustrate how a well-designed data infrastructure can bridge the gap between high-volume measurements and practical grid operations.

Aminifar, Farrokh↗
NASA NTRS1986

Remote sensing of land-surface temperature from HIRS/MSU data

A relaxation algorithm which permits meteorological parameters to be obtained from satellite data, without a priori assumptions about the properties of the other unknowns in the field of view, was developed. Atmospheric temperature profiles, atmospheric humidity, cloud cover, cloud top height, cloud top temperature, sea-surface temperature, land-surface temperature, snow cover, and ice cover are derived. Simultaneous determination of atmospheric and surface thermal structure and the cloud distribution provides information on heat sources and sinks, storage rates, and transport phenomena in the atmosphere. Such information is critical in determining the driving mechanisms for motions in the atmosphere and oceans and in improving numerical weather prediction.

Chahine, M. T.↗
NASA NTRS2019

VISAGE - A Visualization and Exploration Framework for Environmental Data

Diverse airborne and ground-based environmental observations are important technologies for disaster assessment and response, as well as for the validation of environmental satellite observations and atmospheric models which can improve forecasts. The VISAGE (Visualization for Integrated Satellite, Airborne and Ground-based data Exploration) project is working to provide three-dimensional visualization and basic analytics capabilities for such datasets in an interactive user interface. The use of cloud-native, server less technologies for analysis optimized data storage will position VISAGE for integration with other technologies into a Data Analytic Center Framework.

Conover, Helen↗
NASA NTRSDate not supplied

Pyroscopegridding: Geo-Leo Aerosol Data Fusion (an Open-Source Package)

The retrieval of aerosol optical depths (AODs) from sun-synchronous polar orbiting satellites, such as MODISs, VIIRSs, OMI, TROPOMI, etc., has been widely adopted as a method for obtaining information regarding particulate matter (PM) and related atmospheric processes. However, the advent of recently launched geostationary satellites, such as GOES-16/17/18, Himawari-8/9, and Meteosat Third Generation (MTG), has led to an increased temporal resolution of AOD observations (order of 10 minutes), resulting in typically one or more images per hour during daylight hours, compared to the once-per-day observations obtained from LEO satellites. By integrating these observations, the diurnal cycle of global AOD can be characterized at local, regional, and global scales. The scientific community is still examining the novel data from geostationary satellite observations and evaluating methods for effectively merging these observations with differing spatial and temporal resolutions. This presents a significant ""Big Data"" challenge, encompassing not only data storage, but also data discoverability, accessibility, and migration within cloud computing environments. This study presents our attempts at fusing Level 2 aerosol data from six satellites, three of which are geostationary (GOES-16/17 and Himawari-8) and three of which are polar orbiting (TERRA/MODIS, AQUA/MODIS, and SNPP-VIIRS), using the Dark Target aerosol retrieval algorithm. The ability to fuse remote sensing products on demand into desired temporal and spatial domains empowers researchers and practitioners to more efficiently work with satellite and sensor data. It is our hope that through making our open-source package and accompanying functionality available, the scientific community will have improved access to aerosol data processing resources.

Jennifer Wei↗
NASA NTRS2012

Investigation into Cloud Computing for More Robust Automated Bulk Image Geoprocessing

Geospatial resource assessments frequently require timely geospatial data processing that involves large multivariate remote sensing data sets. In particular, for disasters, response requires rapid access to large data volumes, substantial storage space and high performance processing capability. The processing and distribution of this data into usable information products requires a processing pipeline that can efficiently manage the required storage, computing utilities, and data handling requirements. In recent years, with the availability of cloud computing technology, cloud processing platforms have made available a powerful new computing infrastructure resource that can meet this need. To assess the utility of this resource, this project investigates cloud computing platforms for bulk, automated geoprocessing capabilities with respect to data handling and application development requirements. This presentation is of work being conducted by Applied Sciences Program Office at NASA-Stennis Space Center. A prototypical set of image manipulation and transformation processes that incorporate sample Unmanned Airborne System data were developed to create value-added products and tested for implementation on the "cloud". This project outlines the steps involved in creating and testing of open source software developed process code on a local prototype platform, and then transitioning this code with associated environment requirements into an analogous, but memory and processor enhanced cloud platform. A data processing cloud was used to store both standard digital camera panchromatic and multi-band image data, which were subsequently subjected to standard image processing functions such as NDVI (Normalized Difference Vegetation Index), NDMI (Normalized Difference Moisture Index), band stacking, reprojection, and other similar type data processes. Cloud infrastructure service providers were evaluated by taking these locally tested processing functions, and then applying them to a given cloud-enabled infrastructure to assesses and compare environment setup options and enabled technologies. This project reviews findings that were observed when cloud platforms were evaluated for bulk geoprocessing capabilities based on data handling and application development requirements.

Brown, Richard B.↗
NASA NTRSDate not supplied

Cloud Optimized Data Formats

Cloud computing offers the promise of being able to analyze Big Data earth Observations at scale, by allowing scientists to deploy many nodes at once to analyze the data. However, in order to take full advantage of cloud scalability, it is often necessary to reorganize and reformat the data to enable fine-grained, parallel access to the data in Web Object Storage. NASA recently conducted a study of several formats that are optimized for analysis in the cloud: Parquet, zarr, HDF (Hierarchical Data Format) in the Cloud, and Cloud-Optimized GeoTIFF (Tagged Image File Format). They were compared against non-cloud-optimized formats, netCDF (network Common Data Form) and GeoTIFF, with criteria based both on stewardship and analysis performance.

Christopher Lynnes↗
NASA NTRS2015

Intelligent Observation Strategies for Geosynchronous Remote Sensing for Natural Hazards

Geosynchronous satellites offer a unique perspective for monitoring environmental factors important to understanding natural hazards and supporting the disasters management life cycle, namely forecast, detection, response, recovery and mitigation. In the NASA decadal survey for Earth science, the GEO-CAPE mission was proposed to address coastal and air pollution events in geosynchronous orbit, complementing similar initiatives in Asia by the South Koreans and by ESA in Europe, thereby covering the northern hemisphere. In addition to analyzing the challenges of identifying instrument capabilities to meet the science requirements, and the implications of hosting the instrument payloads on commercial geosynchronous satellites, the GEO-CAPE mission design team conducted a short study to explore strategies to optimize the science return for the coastal imaging instrument. The study focused on intelligent scheduling strategies that took into account cloud avoidance techniques as well as onboard processing methods to reduce the data storage and transmission loads. This paper expands the findings of that study to address the use of intelligent scheduling techniques and near-real time data product acquisition of both the coastal water and air pollution events. The topics include the use of onboard processing to refine and execute schedules, to detect cloud contamination in observations, and to reduce data handling operations. Analysis of state of the art flight computing capabilities will be presented, along with an assessment of cloud detection algorithms and their performance characteristics. Tools developed to illustrate operational concepts will be described, including their applicability to environmental monitoring domains with an eye to the future. In the geostationary configuration, the payload becomes a networked thing with enough connectivity to exchange data seamlessly with users. This allows the full field of view to be sensed at very high rate under the control of ground infrastructure, resulting in improved efficiencies, accuracy and science benefits. Hence a remote sensing payload and its data may become one of millions of connected objects in the emerging Internet of Things (IoT), and be as easily accessible by a users smart phone as any other smart appliance.

0↗
NASA NTRSDate not supplied

Characterizing Wildfires in Western US.: A Cloud-based Case Study for Interdisciplinary Research using NASA Resources

This presentation will demonstrate a case study of interdisciplinary research done in the Amazon Web Services (AWS) cloud platform, in addition to in the local machine. We conduct data analysis next to data by leveraging various cloud-based data in NASA Earthdata Cloud, which are distributed by different missions/NASA Distributed Active Archive Centers (DAACs), and cloud computing resources at NASA. For instance, we directly access multiple datasets stored in the AWS Simple Storage Service (S3) buckets using a Python Jupyter notebook through a JupyterHub interface hosted in AWS (without having to download data), and conduct data analysis next to data in the cloud. We will also show how to share the research results following Open Source policy. This case study characterizes the change in wildfire events in the western United States during the past 20 years. In particular, we focus on the wildfires in California in 2021, one of the most severe wildfire years occurring in the most recent 20 years in California. We will analyze the possible causes of wildfires, such as drought conditions and climate variability, and examine the impacts of wildfires on air quality and atmospheric composition, and on land cover. We will examine the data distributed by the NASA Goddard Earth Sciences Data and Information Services Center (GES DISC), including aerosols and meteorological data from the NASA Modern-Era Retrospective analysis for Research and Applications version 2 (MERRA-2), precipitation from the Global Precipitation Measurement (GPM) and Global Precipitation Climate Project (GPCP), and aerosol index from Ozone Monitoring Instrument (OMI). We also utilize the data distributed by the Physical Oceanography (PO) DAAC, such as Sea Surface Temperature (SST) data from the Group for High Resolution Sea Surface Temperature (GHRSST), and the data distributed by Land Processes (LP) DAAC, such as Normalized Difference Vegetation Index (NDVI).

Xiaohua Pan↗
NASA NTRSDate not supplied

The Future of NASA Earth Science in the Commercial Cloud: Challenges and Opportunities

NASA produces a large volume and variety of data products that are used every day to support research, decision making, and education. The widespread use of NASA’s Earth Science data is enabled by NASA’s Earth Science Data System (ESDS) program, which oversees the archiving and distribution of these data and invests in the development of new data systems and tools. However, NASA’s current approach to Earth Science data distribution — based on distributed institutional archives with individual on-premises high-performance computing capabilities — faces some significant challenges, including massive increases in data volume from upcoming missions, a greater need for transdisciplinary science that synthesizes many different kinds of observations, and a push to make science more open, inclusive, and accessible. To address these challenges, NASA is aggressively migrating its Earth Science data and related tools and services into the commercial cloud. Migration of data into the commercial cloud can significantly improve NASA’s existing data system capabilities by (1) providing more flexible options for storage and compute (including rapid, as-needed access to state-of-the-art capabilities); (2) by centralizing and standardizing data access, which gives all of NASA’s institutional data centers access to all of each other’s datasets; and (3) by facilitating “analysis-in-place”, whereby users can bring their own computational workflows and tools to the data rather than having to maintain their own copies of NASA datasets. However, migration to the commercial cloud also poses some significant challenges, including (1) managing costs under a “pay-as-you-go” model; (2) incompatibility with existing tools and data formats with object-based storage and network access; (3) vendor lock-in; (4) challenges with data access for workflows that mix on-premise and cloud computing; and (5) standardization for highly diverse data as is present in NASA’s data archive. I conclude with two examples of recent NASA activities showcasing capabilities enabled by the commercial cloud: An interactive analysis and development platform for analyzing airborne imaging spectroscopy data, and a new collection of tools and services for data discovery, analysis, publication, and data-driven storytelling (Visualization, Exploration, and Data Analysis, VEDA).

Alexey N Shiklomanov↗
NASA NTRS2014

Cultivating an Emergent Earth Observation Analytics Ecosystem in the Cloud

A diverse set of data analytics systems for Earth Observations are sprouting up in the Earth Science community, with a wealth of processing algorithms and analysis methods. There is a similar wealth of data resources available via myriad data providers and clearinghouses, including large institutional systems like the Earth Observing System Data and Information System, Comprehensive Large Scale Array-data Stewardship System, and Federated Earth Observation Missions gateway. With Earth system science driving a need to work with more datasets together, and the community developing more analysis tools (some of them dataset-specific), how can we develop analysis workflows that incorporate far-flung datasets and leverage analysis resources from multiple organizations? Cloud computing points the way toward a solution in two different respects. Firstly, the access to and abstraction of virtually unlimited storage and computing power provides an environment that enables more straightforward means of pulling datasets and analysis resources together. Just as importantly, however, cloud computing serves as an example of an "ecosystem" of interoperating services, since the essence of cloud computing is the presentation of all resources as a service, from hardware to infrastructure to platform to software. This enables the combination of off-the-shelf, diverse services to construct entire systems that emerge out of an equally diverse community of architects and developers. This approach can be similarly applied to the data and analysis resources in the Earth Observation community. By exposing these resources via well understood services, and consuming resources in the same way, different organizations can construct bespoke analysis workflows and systems for their own purposes. The key leap the community needs to make is to develop analysis systems in components that interact with other components via services. The result would be a rich ecosystem of analytics components that can be combined to analyze datasets at scale and in conjunction with other datasets from other sources.

chaos↗
NASA NTRS2015

High Resolution Nature Runs and the Big Data Challenge

NASA's Global Modeling and Assimilation Office at Goddard Space Flight Center is undertaking a series of very computationally intensive Nature Runs and a downscaled reanalysis. The nature runs use the GEOS-5 as an Atmospheric General Circulation Model (AGCM) while the reanalysis uses the GEOS-5 in Data Assimilation mode. This paper will present computational challenges from three runs, two of which are AGCM and one is downscaled reanalysis using the full DAS. The nature runs will be completed at two surface grid resolutions, 7 and 3 kilometers and 72 vertical levels. The 7 km run spanned 2 years (2005-2006) and produced 4 PB of data while the 3 km run will span one year and generate 4 BP of data. The downscaled reanalysis (MERRA-II Modern-Era Reanalysis for Research and Applications) will cover 15 years and generate 1 PB of data. Our efforts to address the big data challenges of climate science, we are moving toward a notion of Climate Analytics-as-a-Service (CAaaS), a specialization of the concept of business process-as-a-service that is an evolving extension of IaaS, PaaS, and SaaS enabled by cloud computing. In this presentation, we will describe two projects that demonstrate this shift. MERRA Analytic Services (MERRA/AS) is an example of cloud-enabled CAaaS. MERRA/AS enables MapReduce analytics over MERRA reanalysis data collection by bringing together the high-performance computing, scalable data management, and a domain-specific climate data services API. NASA's High-Performance Science Cloud (HPSC) is an example of the type of compute-storage fabric required to support CAaaS. The HPSC comprises a high speed Infinib and network, high performance file systems and object storage, and a virtual system environments specific for data intensive, science applications. These technologies are providing a new tier in the data and analytic services stack that helps connect earthbound, enterprise-level data and computational resources to new customers and new mobility-driven applications and modes of work. In our experience, CAaaS lowers the barriers and risk to organizational change, fosters innovation and experimentation, and provides the agility required to meet our customers' increasing and changing needs

big data analysis↗