Search NASA⌕ Search

SEARCH · Search NASA

Results for “cloud computing, data cloud”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 109 records · Page 6
All sourcesNASA NTRSDOE OSTI
NASA NTRS1998

Atmospheric Aerosol Chemical Composition Measurements for the Subsonic Aircraft: Contrail and Cloud Effects Special Study (SUCCESS)

We received funding to provide measurements of the chemical composition of aerosols aboard the NASA Ames DC-8 research aircraft during the SUCCESS mission. These measurements were successfully completed and the final data resides in the Cloud I computer archive at NASA Ames Research Center. The interpretation of the data obtained on this mission over the central United States has been published in special issues of Geophysical Research Letters. The papers with the University of New Hampshire as first author constitute this report and summarize the salient features of our data. The paper by Talbot et al. discusses the impact of vertical transport on free tropospheric chemistry over the the central USA in springtime. This transport was a dominant feature of the aerosol chemistry during SUCCESS. The paper by Dibb et al. discusses aerosol chemistry specifically as it related to free tropospheric sulfate related to jet exhaust and surface sources. Somewhat surprisingly, we found that surface sources appeared to dominant the distribution of aerosol sulfate in the free troposphere. In addition to these first authored papers, researchers from the University of New Hampshire were co-authors on numerous other companion papers in the special issues.

Talbot, Robert W.↗
DOE OSTI2015

Toward Interactive Scenario Analysis and Exploration

As Modeling and Simulation (M&S) tools have matured, their applicability and importance have increased across many national security challenges. In particular, they provide a way to test how something may behave without the need to do real world testing. However, current and future changes across several factors including capabilities, policy, and funding are driving a need for rapid response or evaluation in ways that many M&S tools cannot address. Issues around large data, computational requirements, delivery mechanisms, and analyst involvement already exist and pose significant challenges. Furthermore, rising expectations, rising input complexity, and increasing depth of analysis will only increase the difficulty of these challenges. In this study we examine whether innovations in M&S software coupled with advances in ''cloud'' computing and ''big-data'' methodologies can overcome many of these challenges. In particular, we propose a simple, horizontally-scalable distributed computing environment that could provide the foundation (i.e. ''cloud'') for next-generation M&S-based applications based on the notion of ''parallel multi-simulation''. In our context, the goal of parallel multi- simulation is to consider as many simultaneous paths of execution as possible. Therefore, with sufficient resources, the complexity is dominated by the cost of single scenario runs as opposed to the number of runs required. We show the feasibility of this architecture through a stable prototype implementation coupled with the Umbra Simulation Framework [6]. Finally, we highlight the utility through multiple novel analysis tools and by showing the performance improvement compared to existing tools.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2015

Data Intensive Scientific Workflows on a Federated Cloud: CRADA Final Report

The Fermilab Scientific Computing Division and the KISTI Global Science Experimental Data Hub Center have built a prototypical large-scale infrastructure to handle scientific workflows of stakeholders to run on multiple cloud resources. The demonstrations have been in the areas of (a) Data-Intensive Scientific Workflows on Federated Clouds, (b) Interoperability and Federation of Cloud Resources, and (c) Virtual Infrastructure Automation to enable On-Demand Services.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗
DOE OSTI2023

EDX Spatial: Leveraging cloud and hybrid data management resources for spatial data

In the last few years, the National Energy Technology Laboratory has started leveraging cloud-hosted services for hosting spatial data collections published on the Energy Data eXchange (EDX). Using cloud-hosted storage and compute options offers multiple benefits for visualization and tool development through utilization of spatial data resources. The transition from on-premises services to cloud-hosted services has enabled a few key features: increased accessibility of large derivative datasets, dynamic integration of external authoritative data resources directly from outside entities through representational state transfer application programming interfaces (REST API), and the ability to produce complex mapping applications, dashboards, and online maps. As a result of the shift, NETL has launched EDX Spatial, a platform that leverages on-premises resources combined with cloud compute capabilities to enable enhanced online mapping interfaces and optimize data access. In addition, a unified workflow for handling the public release of spatial data products through EDX has been developed, including standardization of data hosting practices, metadata, symbology, and application elements. This poster reviews the opportunity of leveraging cloud-hosted and hybrid data management solutions for spatial data, and discusses the benefits and lessons learned while leveraging these services through the data repository EDX.

Morkner, Paige↗
DOE OSTI2013

The Energy Efficiency Potential of Cloud-Based Software: A U.S. Case Study

The energy use of data centers is a topic that has received much attention, given that data centers currently account for 1-2% of global electricity use. However, cloud computing holds great potential to reduce data center energy demand moving forward, due to both large reductions in total servers through consolidation and large increases in facility efficiencies compared to traditional local data centers. However, analyzing the net energy implications of shifts to the cloud can be very difficult, because data center services can affect many different components of society’s economic and energy systems.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗
NASA NTRS2018

Advanced Analytics and Big Earth Data

NASA's Earth Science Data Systems process, archive and distribute petabytes of Earth Observation data to a variety of end users. These end users will face dramatically increased data size in the near future, bringing about new challenges and opportunities in analyzing those data. One area of particular ferment currently is Machine Learning. Many Machine Learning methods are black boxes, limiting direct insight into the data's properties. However, they can be used for a variety of data enhancement purposes, such as parameter retrieval, data fusion and image classification and segmentation. The Earth Observing System Data and Information System is also evolving to host large data volumes in the cloud, enabling data proximal analysis. As part of this effort, an Analytics framework is being developed to support and enhance user analysis of the data. By using standards based services in the framework, diverse user communities can be served, while also allowing inter-system collaboration in the analysis process.

Cloud Computing↗
NASA NTRS2013

A Novel Method for Estimating Shortwave Direct Radiative Effect of Above-cloud Aerosols over Ocean Using CALIOP and MODIS Data

This paper describes an efficient and unique method for computing the shortwave direct radiative effect (DRE) of aerosol residing above low-level liquid-phase clouds using CALIOP and MODIS data. It accounts for the overlapping of aerosol and cloud rigorously by utilizing the joint histogram of cloud optical depth and cloud top pressure. Effects of sub-grid scale cloud and aerosol variations on DRE are accounted for. It is computationally efficient through using grid-level cloud and aerosol statistics, instead of pixel-level products, and a pre-computed look-up table in radiative transfer calculations. We verified that for smoke over the southeast Atlantic Ocean the method yields a seasonal mean instantaneous shortwave DRE that generally agrees with more rigorous pixel-level computation within 4%. We have also computed the annual mean instantaneous shortwave DRE of light-absorbing aerosols (i.e., smoke and polluted dust) over global ocean based on 4 yr of CALIOP and MODIS data. We found that the variability of the annual mean shortwave DRE of above-cloud light-absorbing aerosol is mainly driven by the optical depth of the underlying clouds.

CALIOP and MODIS↗
NASA NTRS2014

A Novel Method for Estimating Shortwave Direct Radiative Effect of Above-Cloud Aerosols Using CALIOP and MODIS Data

This paper describes an efficient and unique method for computing the shortwave direct radiative effect (DRE) of aerosol residing above low-level liquid-phase clouds using CALIOP and MODIS data. It accounts for the overlapping of aerosol and cloud rigorously by utilizing the joint histogram of cloud optical depth and cloud top pressure. Effects of sub-grid scale cloud and aerosol variations on DRE are accounted for. It is computationally efficient through using grid-level cloud and aerosol statistics, instead of pixel-level products, and a pre-computed look-up table in radiative transfer calculations. We verified that for smoke over the southeast Atlantic Ocean the method yields a seasonal mean instantaneous shortwave DRE that generally agrees with more rigorous pixel-level computation within 4. We have also computed the annual mean instantaneous shortwave DRE of light-absorbing aerosols (i.e., smoke and polluted dust) over global ocean based on 4 yr of CALIOP and MODIS data. We found that the variability of the annual mean shortwave DRE of above-cloud light-absorbing aerosol is mainly driven by the optical depth of the underlying clouds.

radiative forcing↗
NASA NTRSDate not supplied

High Performance Access to Archival Data Stored in HDF4 and HDF5 on Cloud Object Stores Without Reformatting the Files

Cloud computing offers numerous advantages for users of extensive Earth science data collections. These benefits encompass direct online access to data files and granules from any location, scalable access supporting parallel computing workflows, and flexible computing tools enabling innovative experimentation with processing techniques. However, older archival file formats designed for distinct computing systems hinder efficient access to decade-long time-series data when compared to data stored in modern cloud-optimized formats like Web Object Stores (WOS), exemplified by Amazon Web Services’ Simple Storage Service (S3). We describe DMR++ (Dataset Metadata Response plus plus), a technology facilitating efficient access to HDF5 (Hierarchical Data Format, version 5) and HDF4 files stored on WOS systems without requiring data reformatting. DMR++ achieves performance comparable to technologies like Zarr while preserving the original file structure, a substantial benefit considering the vast quantity of archival files held by organizations such as NASA. Moreover, DMR++ typically outperforms cloud-optimized versions of HDF5. Essentially an XML (Extensible Markup Language) document usually stored alongside the described data, DMR++ can also be generated on-the-fly but is generally created during data staging to the WOS. Archival files that use HDF4/5 often store large arrays of numerical data. The data in these files is often compressed, typically reducing their size by a factor of four or more. To achieve efficient access to portions of those arrays, they are 'chunked' into smaller sub-arrays, each individually compressed. The chunk size is a compromise, where spinning disks can efficiently access data in smaller chunks while S3 favors larger chunks. A simple optimization of aggregating smaller chunks that are stored adjacently, transferring them in a single access and then individually decompressing them will improve performance. NASA data pose an additional challenge: special Application Programmer Interface (API) libraries are often needed to compute some variables. These libraries are incompatible with WOS environments. Our solution involves storing computed values in the DMR++ document or a companion file, making them accessible like other variables and eliminating the need for specialized APIs. We outline specific optimizations for both satellite grid and swath data stored in HDF4-EOS2 (Earth Observing System).

James Gallagher↗
NASA NTRS2017

NASA Wrangler: Automated Cloud-Based Data Assembly in the RECOVER Wildfire Decision Support System

NASA Wrangler is a loosely-coupled, event driven, highly parallel data aggregation service designed to take advantageof the elastic resource capabilities of cloud computing. Wrangler automatically collects Earth observational data, climate model outputs, derived remote sensing data products, and historic biophysical data for pre-, active-, and post-wildfire decision making. It is a core service of the RECOVER decision support system, which is providing rapid-response GIS analytic capabilities to state and local government agencies. Wrangler reduces to minutes the time needed to assemble and deliver crucial wildfire-related data.

decision support↗
DOE OSTI2025

IM3 + EPRI Data Center Load Projections

This dataset contains scenarios of hourly total electricity demand with and without projected loads from data centers over the period 2022-2040. The root projections without data center demands are identical to those documented in Burleyson et al. 2024. In short, those projections encompass hourly electricity demands for 54 Balancing Authorities (BAs) in the United States across a range of eight of weather and socioeconomic scenarios. Refer to the root dataset and accompanying publication, Burleyson et al. 2025, for information about how those projections were generated. For this derivative dataset we used the base loads from the following scenarios: rcp45hotter_ssp3 rcp45hotter_ssp5 rcp85hotter_ssp3 rcp85hotter_ssp5 The root load projections did not reflect the drastic expansion of data centers that has occurred in the last several years to support artificial intelligence and cloud computing. To reflect growth in data center demand, a second set of load projections were created in which we layered in additional data center load projections based on the data center load growth scenarios described in a 2024 report by the Electric Power Research Institute (EPRI): "Powering Intelligence: Analyzing Artificial Intelligence and Data Center Energy Consumption". The EPRI projections from the report are included in this dataset (EPRI_2024_Projections.xlsx). That report contained annual state-level data center load projections for four year-over-year growth rates for data center demands: Low (3.71% annual growth) Moderate (5% annual growth) High (10% annual growth) Higher (15% annual growth) To homogenize the load projections with and without data centers we had to get them to a common scale. The first step was to take the EPRI annual state-level data center energy consumption values and convert them to 8760-hr loads for each year. We did that by assuming a flat (e.g., not weather- or time-sensitive) load profile and distributing the data center loads in each state evenly across all hours in a year. From there the loads were downscaled from the state-level to the county-level using 2019 county-level populations as weights. Finally, the county-level hourly data center loads were summed to the BA-level using the county-to-BA mapping underpinning the root load projections. The net result is 16 (4 weather and socioeconomic scenarios crossed with 4 data center load growth scenarios) unique load projections for the period 2022-2040. The file format follows that of the root dataset with a single additional column "Scaled_TELL_BA_Load_with_DC_MWh" that contains the hourly loads with the added data center loads for a given BA-year-scenario combination. Please refer to the readme file in the root dataset for more information on the file format.

Burleyson, Casey [Pacific Northwest National Labor↗
NASA NTRS2014

A Novel Method for Estimating Shortwave Direct Radiative Effect of Above-Cloud Aerosols Using CALIOP and MODIS Data

This paper describes an efficient and unique method for computing the shortwave direct radiative effect (DRE) of aerosol residing above low-level liquid-phase clouds using CALIOP and MODIS data. It addresses the overlap of aerosol and cloud rigorously by utilizing the joint histogram of cloud optical depth and cloud top pressure while also accounting for subgrid-scale variations of aerosols. The method is computationally efficient because of its use of grid-level cloud and aerosol statistics, instead of pixel-level products, and a pre-computed look-up table based on radiative transfer calculations. We verify that for smoke over the southeast Atlantic Ocean the method yields a seasonal mean instantaneous (approximately 1:30PM local time) shortwave DRE of above cloud aerosol (ACA) that generally agrees with more rigorous pixel-level computation within 4 percent. We also estimate the impact of potential CALIOP aerosol optical depth (AOD) retrieval bias of ACA on DRE. We find that the regional and seasonal mean instantaneous DRE of ACA over southeast Atlantic Ocean would increase, from the original value of 6.4 W m(-2) based on operational CALIOP AOD to 9.6 W m(-2) if CALIOP AOD retrieval are biased low by a factor of 1.5 (Meyer et al., 2013) and further to 30.9 W m(-2) if CALIOP AOD retrieval are biased low by a factor of 5 as suggested in (Jethva et al., 2014). In contrast, the instantaneous ACA radiative forcing efficiency (RFE) remains relatively invariant in all cases at about 53 W m(-2) AOD(-1), suggesting a near linear relation between the instantaneous RFE and AOD. We also compute the annual mean instantaneous shortwave DRE of light-absorbing aerosols (i.e., smoke and polluted dust) over global oceans based on 4 years of CALIOP and MODIS data. We find that the variability of the annual mean shortwave DRE of above-cloud light-absorbing aerosol is mainly driven by the optical depth of the underlying clouds. While we demonstrate our method using CALIOP and MODIS data, it can also be extended to other satellite data sets, as well as climate model outputs.

CALIOP and MODIS↗
DOE OSTI2021

Energy Material Network Data Hubs

In early 2015 the United States Department of Energy conceived of a consortium of collaborative bodies based on shared expertise, data, and resources that could be targeted towards the more difficult problems in energy materials research. The concept of virtual laboratories had been envisioned and discussed earlier in the decade in response to the advent of the Materials Genome Initiative and similar scientific thrusts. To be effective, any virtual laboratory needed a robust method for data management, communication, security, data sharing, dissemination, and demonstration to work efficiently and effectively for groups of remote researchers. With the accessibility of new, easily deployed cloud technology and software frameworks, such individual elements could be integrated, and the required collaboration architecture is now possible. The developers have leveraged open-source software frameworks, customized them, and merged them into a platform to enable collaborative energy materials science, regardless of the geographic dispersal of the people and resources. After five years in operations, the systems are demonstratively an effective platform for enabling research within the Energy Material Networks (EMN). This paper will show the design and development of a secured scientific data sharing platform, the ability to customize the system to support diverse workflows, and examples of the enabled research and results connected with some of the Energy Material Networks.

97 MATHEMATICS AND COMPUTING↗
NASA NTRS2016

Big Data in the Earth Observing System Data and Information System

Approaches that are being pursued for the Earth Observing System Data and Information System (EOSDIS) data system to address the challenges of Big Data were presented to the NASA Big Data Task Force. Cloud prototypes are underway to tackle the volume challenge of Big Data. However, advances in computer hardware or cloud won't help (much) with variety. Rather, interoperability standards, conventions, and community engagement are the key to addressing variety.

data systems↗
DOE OSTI2025

EDX ClaiMM

EDX ClaiMM is a centralized data & analytical platform designed to revolutionize U.S. critical minerals and materials (CMM) activities. By providing a robust digital infrastructure, ClaiMM will accelerate the combination, leveraging, and rapid utilization of vital data, advanced tools, and cutting-edge research advancements in CMM. This adaptive digital research hub connects the CMM community to essential knowledge products and offers access to interoperable datasets, databases, models, software, and tools from the National Energy Technology’s (NETL’s) Energy Data eXchange (EDX) and other authoritative sources, serving both public and private sectors. EDX ClaiMM delivers AI-informed solutions to address fundamental knowledge gaps and fosters the innovation of new techniques for enhanced characterization and recovery of CMMs within the U.S. By leveraging cloud-hosted, scalable digital infrastructure, ClaiMM meets public–private applied energy needs. It equips the CMM community with priority digital resources that harness on-site and cloud compute capabilities, enabling big data storage, advanced processing, analytics, and visualization.

Critical Materials; Critical Minerals; Rare Earth ↗
NASA NTRSDate not supplied

The Machine Learning Showroom: Presentation to OCIO Data Science Summit

Artificial Intelligence/Machine Learning (AI/ML) has become an indispensable tool for descriptive, predictive and prescriptive analytics. Demand for AI/ML models at NASA is outpacing Data Scientist staff. The AI/ML Showroom is an effort to empower NASA professionals to evaluate AI/ML solutions for their problems in a scalable self-help manner, relying on coding examples, reference use cases, digital assistant guides, jam sessions, video training, and pre-configured cloud resources.

machine learning↗
NASA NTRS2018

Generalizing a Data Analysis Pipeline in the Cloud to Handle Diverse Use Cases in NASA's EOSDIS

NASA's Earth Observing System Data and Information System (EOSDIS) is tasked with archiving and distributing Earth Observation data across a range of disciplines, including atmospheric science, oceanography, land processes, natural hazards, solar radiance and even socioeconomic aspects relating to the environment. Driven by rapidly rising data volumes, EOSDIS is migrating to a cloud computing based archive over the next few years. Although this simplifies data management somewhat, the main aim is to provide the data in an environment where end users can bring their analysis to the data rather than attempting to download and manage ever-increasing volumes. To that end, a cloud-based analysis platform is being constructed to enable data transformations, analyses and visualization without egressing the data from the cloud. In this endeavor, we expect a wide variety of users, algorithms and use cases. Consequently, the architecture of this cloud analytics platform is expressly designed to be based on open services, thus fostering an ecosystem that enables the efficient combination of common components with data-specific or analysis-specific components. Reviewed and approved by Andrew Mitchell, ESDIS project manager.

Cloud computing↗