Search NASA⌕ Search

SEARCH · Search NASA

Results for “CLOUD COMPUTING”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 577 records · Page 32
All sourcesNASA NTRSDOE OSTI
DOE OSTI2015

HPCCloud: A Cloud/Web-Based Simulation Environment

Advanced modeling and simulation has enabled the design of a variety of innovative products and the analysis of numerous complex phenomenon. There are significant barriers to widespread adoption of these tools. In particular, advanced modeling and simulation: (1) is considered complex to use, (2) needs in-house expertise, and (3) requires high capital costs. In this paper, we describe the development of an end-to-end, advanced modeling and simulation cloud platform that encapsulates best practices for scientific computing in the cloud, and demonstrate using Hydra-TH as a prototypical application. As an alternative to traditional advanced modeling and simulation workflows, our Web-based approach simplifies the processes, decreases the need for in-house computational science and engineering experts, and lowers the capital investments. In addition to providing significantly improved, intuitive software, the environment offers reproducible workflows where the full lifecycle of data from input to final analyzed results can be saved, shared, and even published.

97 MATHEMATICS AND COMPUTING↗
NASA NTRS1988

A support architecture for reliable distributed computing systems

The Clouds project is well underway to its goal of building a unified distributed operating system supporting the object model. The operating system design uses the object concept of structuring software at all levels of the system. The basic operating system was developed and work is under progress to build a usable system.

Dasgupta, Partha↗
NASA NTRS1993

Low-power millimeter-wave radar observations of the atmosphere

Historically, cloud structures, dynamics, and precipitation processes have been observed and measured with sensors from two different spatial resolutions. Laser-based sensors have volume resolutions on the order of 10(exp -4) to 10(exp -2) cubic meters for a 1-s sample. Radar systems operating at wavelengths between 1 and 10 cm have resolutions on the order of 10(exp 4) to 10(exp 7) cubic meters. The resolutions of micro-wave systems depend primarily on the system RF bandwidth and antenna bandwidth. Both resolution regimes were useful in the study of cloud structures and processes - the former for determining resolution on the individual cloud particle scale and the latter for studying the coarse characteristics of cloud dynamics and structure. There are, however, cloud processes and structures that occur on scales that lie between these two regimes: the process of entrainment, where outside air is brought within the cloud boundaries; the mixing of in-cloud particles with different histories; cloud particle coalescence; and ice formation. The use of mm-wave (30- to 300-GHz) radars offers an opportunity to observe cloud processes at these scales and determine their influence on precipitation development, cloud albedos, cloud lifetimes, chemical cycling of tract substances, aircraft icing, and other meteorological phenomena. To determine the usefulness of a 35-GHz radar for observing these precipitation and cloud processes, a research program was initiated. The objectives of this program are to develop a 35-GHz radar, to measure scattering from precipitation and clouds, and to develop a model to compute scattering from clouds using the finite-difference time-domain (FDTD) technique.

Ronnau, James F.↗
NASA NTRS2018

Evaluating the Suitability of Commercial Clouds for NASA's High Performance Computing Applications: A Trade Study

NASA’s High-End Computing Capability (HECC) Project is periodically asked if it could be more cost effective through the use of commercial cloud resources. To answer the question, HECC’s Application Performance and Productivity (APP) team undertook a performance and cost evaluation comparing three domains: two commercial cloud providers, Amazon and Penguin, and HECC’s in-house resources—the Pleiades and Electra systems. In the study, the APP team used a combination of the NAS Parallel Benchmarks (NPB) and six full applications from NASA’s workload on Pleiades and Electra to compare performance of nodes based on three different generations of Intel Xeon processors—Haswell, Broadwell, and Skylake. Because of export control limitations, the most heavily used applications on Pleiades and Electra could not be used in the cloud; therefore, only one of the applications, OpenFOAM, represents work from the Aeronautics Research Mission Directorate and the Human and Exploration Mission Directorate. The other five applications are from the Science Mission Directorate.

High Performance↗
DOE OSTI2022

Performance of Cloud 3D Solvers in Ice Cloud Shortwave Radiation Closure Over the Equatorial Western Pacific Ocean

Abstract For retrieving cloud optical properties from satellite images or computing these properties from climate model output, computationally efficient treatments of cloud horizontal inhomogeneity include the Monte Carlo Independent Column Approximation (McICA) and the Tripleclouds method. Computationally efficient treatment of cloud horizontal radiation exchanges includes the SPeedy Algorithm for Radiative TrAnsfer through CloUd Sides (SPARTACUS). As a test to derive properties from satellite images, we collocate Moderate Resolution Imaging Spectroradiometer (MODIS) cloud retrievals with near‐nadir Cloud and the Earth's Radiant Energy System (CERES) footprints in July 2008 over an equatorial western Pacific Ocean region to compare the performance of the McICA, Tripleclouds, and SPARTACUS solvers to the conventional plane‐parallel homogeneous (PPH) treatment. PPH overestimates cloud albedo, and the three solvers effectively reduce overestimation with root mean square error of shortwave upwelling irradiance decreasing between 15.72 and 18.53 W m −2 , or about 22%–25%. Although cloud top variability does not get fed into the simulations, all three solvers also reduce the effect of cloud top variability on cloud albedo. Entrapment (energy reflected downward from clouds) and horizontal radiation transfer have opposite effects on the SPARTACUS cloud albedo simulation. The net effect depends on the cloud vertical extent, the unawareness of which limits the performance of the SPARTACUS solver.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2021

Adaptive job and resource management for the growing quantum cloud

As the popularity of quantum computing continues to grow, efficient quantum machine access over the cloud is critical to both academic and industry researchers across the globe. And as cloud quantum computing demands increase exponentially, the analysis of resource consumption and execution characteristics are key to efficient management of jobs and resources at both the vendor-end as well as the client-end. While the analysis and optimization of job / resource consumption and management are popular in the classical HPC domain, it is severely lacking for more nascent technology like quantum computing.This paper proposes optimized adaptive job scheduling to the quantum cloud taking note of primary characteristics such as queuing times and fidelity trends across machines, as well as other characteristics such as quality of service guarantees and machine calibration constraints. Key components of the proposal include a) a prediction model which predicts fidelity trends across machine based on compiled circuit features such as circuit depth and different forms of errors, as well as b) queuing time prediction for each machine based on execution time estimations. Altogether, this proposal is evaluated on simulated IBM machines across a diverse set of quantum applications and system loading scenarios, and is able to reduce wait times by over 3x and improve fidelity by over 40% on specific usecases, when compared to traditional job schedulers.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2015

Next Generation Workload Management System For Big Data on Heterogeneous Distributed Computing

The Large Hadron Collider (LHC), operating at the international CERN Laboratory in Geneva, Switzerland, is leading Big Data driven scientific explorations. Experiments at the LHC explore the fundamental nature of matter and the basic forces that shape our universe, and were recently credited for the discovery of a Higgs boson. ATLAS and ALICE are the largest collaborations ever assembled in the sciences and are at the forefront of research at the LHC. To address an unprecedented multi-petabyte data processing challenge, both experiments rely on a heterogeneous distributed computational infrastructure. The ATLAS experiment uses PanDA (Production and Data Analysis) Workload Management System (WMS) for managing the workflow for all data processing on hundreds of data centers. Through PanDA, ATLAS physicists see a single computing facility that enables rapid scientific breakthroughs for the experiment, even though the data centers are physically scattered all over the world. The scale is demonstrated by the following numbers: PanDA manages O(10 2 ) sites, O(10 5 ) cores, O(10 8 ) jobs per year, O(10 3 ) users, and ATLAS data volume is O(10 17 ) bytes. In 2013 we started an ambitious program to expand PanDA to all available computing resources, including opportunistic use of commercial and academic clouds and Leadership Computing Facilities (LCF). The project titled 'Next Generation Workload Management and Analysis System for Big Data' (BigPanDA) is funded by DOE ASCR and HEP. Extending PanDA to clouds and LCF presents new challenges in managing heterogeneity and supporting workflow. The BigPanDA project is underway to setup and tailor PanDA at the Oak Ridge Leadership Computing Facility (OLCF) and at the National Research Center "Kurchatov Institute" together with ALICE distributed computing and ORNL computing professionals. Our approach to integration of HPC platforms at the OLCF and elsewhere is to reuse, as much as possible, existing components of the PanDA system. Finally, we will present our current accomplishments with running the PanDA WMS at OLCF and other supercomputers and demonstrate our ability to use PanDA as a portal independent of the computing facilities infrastructure for High Energy and Nuclear Physics as well as other data-intensive science applications.

97 MATHEMATICS AND COMPUTING↗
NASA NTRS2012

The Namibia Early Flood Warning System, A CEOS Pilot Project

Over the past year few years, an international collaboration has developed a pilot project under the auspices of Committee on Earth Observation Satellite (CEOS) Disasters team. The overall team consists of civilian satellite agencies. For this pilot effort, the development team consists of NASA, Canadian Space Agency, Univ. of Maryland, Univ. of Colorado, Univ. of Oklahoma, Ukraine Space Research Institute and Joint Research Center(JRC) for European Commission. This development team collaborates with regional , national and international agencies to deliver end-to-end disaster coverage. In particular, the team in collaborating on this effort with the Namibia Department of Hydrology to begin in Namibia . However, the ultimate goal is to expand the functionality to provide early warning over the South Africa region. The initial collaboration was initiated by United Nations Office of Outer Space Affairs and CEOS Working Group for Information Systems and Services (WGISS). The initial driver was to demonstrate international interoperability using various space agency sensors and models along with regional in-situ ground sensors. In 2010, the team created a preliminary semi-manual system to demonstrate moving and combining key data streams and delivering the data to the Namibia Department of Hydrology during their flood season which typically is January through April. In this pilot, a variety of moderate resolution and high resolution satellite flood imagery was rapidly delivered and used in conjunction with flood predictive models in Namibia. This was collected in conjunction with ground measurements and was used to examine how to create a customized flood early warning system. During the first year, the team made use of SensorWeb technology to gather various sensor data which was used to monitor flood waves traveling down basins originating in Angola, but eventually flooding villages in Namibia. The team made use of standardized interfaces such as those articulated under the Open Cloud Consortium (OGC) Sensor Web Enablement (SWE) set of web services was good [1][2]. However, it was discovered that in order to make a system like this functional, there were many performance issues. Data sets were large and located in a variety of location behind firewalls and had to be accessed across open networks, so security was an issue. Furthermore, the network access acted as bottleneck to transfer map products to where they are needed. Finally, during disasters, many users and computer processes act in parallel and thus it was very easy to overload the single string of computers stitched together in a virtual system that was initially developed. To address some of these performance issues, the team partnered with the Open Cloud Consortium (OCC) who supplied a Computation Cloud located at the University of Illinois at Chicago and some manpower to administer this Cloud. The Flood SensorWeb [3] system was interfaced to the Cloud to provide a high performance user interface and product development engine. Figure 1 shows the functional diagram of the Flood SensorWeb. Figure 2 shows some of the functionality of the Computation Cloud that was integrated. A significant portion of the original system was ported to the Cloud and during the past year, technical issues were resolved which included web access to the Cloud, security over the open Internet, beginning experiments on how to handle surge capacity by using the virtual machines in the cloud in parallel, using tiling techniques to render large data sets as layers on map, interfaces to allow user to customize the data processing/product chain and other performance enhancing techniques. The conclusion reached from the effort and this presentation is that defining the interoperability standards in a small fraction of the work. For example, once open web service standards were defined, many users could not make use of the standards due to security restrictions. Furthermore, once an interoperable sysm is functional, then a surge of users can render a system unusable, especially in the disaster domain.

Mandl, Daniel↗
DOE OSTI2021

Prediction of Solar Irradiance and Photovoltaic Solar Energy Product Based on Cloud Coverage Estimation Using Machine Learning Methods

Cloud cover estimation from images taken by sky-facing cameras can be an important input for analyzing current weather conditions and estimating photovoltaic power generation. The constant change in position, shape, and density of clouds, however, makes the development of a robust computational method for cloud cover estimation challenging. Accurately determining the edge of clouds and hence the separation between clouds and clear sky is difficult and often impossible. Toward determining cloud cover for estimating photovoltaic output, we propose using machine learning methods for cloud segmentation. We compare several methods including a classical regression model, deep learning methods, and boosting methods that combine results from the other machine learning models. To train each of the machine learning models with various sky conditions, we supplemented the existing Singapore whole sky imaging segmentation database with hazy and overcast images collected by a camera-equipped Waggle sensor node. We found that the U-Net architecture, one of the deep neural networks we utilized, segmented cloud pixels most accurately. However, the accuracy of segmenting cloud pixels did not guarantee high accuracy of estimating solar irradiance. We confirmed that the cloud cover ratio is directly related to solar irradiance. Additionally, we confirmed that solar irradiance and solar power output are closely related; hence, by predicting solar irradiance, we can estimate solar power output. This study demonstrates that sky-facing cameras with machine learning methods can be used to estimate solar power output. This ground-based approach provides an inexpensive way to understand solar irradiance and estimate production from photovoltaic solar facilities.

14 SOLAR ENERGY↗
DOE OSTI2026

How Cloud is Accelerating Research at NREL

This presentation coincides with AWS's announcement of their new Parallel Computing Service (PCS) which allows for easy creation of HPC-style clusters in their AWS cloud computing platform. I helped them beta test this service before it was made generally available in August. AWS asked if we would be interested in discussing our experience with the PCS service, and our experience with HPC workloads in the cloud in general, so this slideshow discusses a brief history of scientific computing at NREL and shares a bit of our experiences and approach to utilizing cloud services for HPC-style workloads.

97 MATHEMATICS AND COMPUTING↗
NASA NTRS1982

Methods of editing cloud and atmospheric layer affected pixels from satellite data

Practical methods of computer screening cloud-contaminated pixels from data of various satellite systems are proposed. Examples are given of the location of clouds and representative landscape features in HCMM spectral space of reflectance (VIS) vs emission (IR). Methods of screening out cloud affected HCMM are discussed. The character of subvisible absorbing-emitting atmospheric layers (subvisible cirrus or SCi) in HCMM data is considered and radiosonde soundings are examined in relation to the presence of SCi. The statistical characteristics of multispectral meteorological satellite data in clear and SCi affected areas are discussed. Examples in TIROS-N and NOAA-7 data from several states and Mexico are presented. The VIS-IR cluster screening method for removing clouds is applied to a 262, 144 pixel HCMM scene from south Texas and northeast Mexico. The SCi that remain after cluster screening are sited out by applying a statistically determined IR limit.

Nixon, P. R.↗
DOE OSTI2025

If We Build Them, They Will Run: Automated HPC Apps Deployment and Profiling with eBPF in Cloud

The high performance computing (HPC) community is in a period of transition. The rise of AI/ML coupled with a changing landscape of resources deems portability a new metric of performance, and methods to move between on-premises and cloud environments and assess compatibility are paramount. Here we design and test a strategy for bridging the gap between traditional HPC and Kubernetes environments – first containerizing applications, providing automated orchestration to run studies, and packaging the setup with automated means to assess performance using low overhead eXtended Berkeley Packet Filter (eBPF) programs. We first assess different designs for eBPF collection, demonstrating a tradeoff between number of programs deployed on a node and overhead added. We develop 5 low overhead eBPF programs that combine with streaming ML models to assess CPU, futex, TCP, shared memory, and file access across four different builds of an HPC application for CPU and GPU. We use eBPF data to generate insights into the possible underlying etiology of scaling issues. We then assess compatibility of a well-known benchmark, HPCG, across matrices of micro-architectures and optimization levels (217 containers across 24 instance types and over 7500 runs). We provide to the community 30 applications to deploy in our automated setup and perform a scaling study from 4 to a maximum of 256 nodes for both CPU and GPU applications. Finally, we use our gained knowledge about performance to generate compatibility artifacts that are used by a newly developed Kubernetes controller to intelligently select instance type based on optimizing a figure of merit. Along with insights to scaling in this environment with a collection of applications and templates to work from, we provide an overall strategy for approaching HPC application deployment and image selection based on compatibility in cloud.

Computer science↗
NASA NTRSDate not supplied

Progress Towards a Common CERES Cloud Mask Algorithm for MODIS and VIIRS: Evaluation using CALIOP Data

For over two decades The Clouds and the Earth’s Radiant Energy System (CERES) project has endeavored to produce a long-term global climate data record for detecting changes in the Earth’s radiation budget and to improve understanding of how clouds contribute to those changes. CERES incorporates cloud information derived from passive narrowband satellite imaging radiometers, and over the course of the project different instruments have contributed to this effort, namely the Moderate Resolution Imaging Spectroradiometer (MODIS) and the Visible Infrared Imaging Radiometer Suite (VIIRS).Cloud properties have been derived from MODIS on the Aqua satellite for CERES since 2002,and VIIRS on the NOAA-20 satellite launched in 2017 will continue the cloud record once Aqua-MODIS reaches the end of its operational lifetime. However, MODIS and VIIRS have different spectral capabilities and characteristics which complicates our ability to seamlessly transition the record from MODIS to VIIRS. This study evaluates some of the recent algorithm modifications incorporated by the CERES Cloud Working Group towards developing a unified cloud mask algorithm for MODIS and VIIRS that utilizes a set of spectral bands common to both instruments. Cloud-Aerosol Lidar with Orthogonal Polarization (CALIOP) data products are used to evaluate MODIS and VIIRS regional cloud fraction estimates and identify areas where improvements can be made. The cloud mask algorithm relies on computed estimates of cloud-free top-of-atmosphere radiances to differentiate cloudy and cloud-free imager pixels. The unified cloud mask algorithm currently under development uses a reduced number of spectral bands common to both MODIS and VIIRS and so relies on the accuracy of these simulated cloud-free radiances more heavily than predecessor algorithms, i.e., the CERES Edition 4 cloud mask, which use as much spectral information as available on each imager with less consideration for cross-platform consistency. Our validation strategy using CALIOP data is also described. The CALIOP observations are critical for assessing cloud detection accuracies and for independently confirming cloud-free conditions which enables a more robust evaluation of the simulated cloud-free radiances for biases due to factors such as view angle and the spectral dependence of water vapor absorption. These effects are known to differ for the two satellite instruments. Comparisons of MODIS and VIIRS cloud fractions are presented in context with estimates from CALIOP and from the CERES Edition 4 cloud mask algorithm to gauge current progress in developing accurate and consistent MODIS and VIIRS cloud properties for CERES.

CERES↗
NASA NTRS2005

Global, Multi-Year Analysis of Clouds and Earth's Radiant Energy System Terra Observations and Radiative Transfer Calculations

An extended record of the Terra Surface and Atmosphere Radiation Budget (SARB) computed by CERES (Clouds and Earth s Radiant Energy System) is produced in gridded form, facilitating an investigation of global scale direct aerosol forcing. The new gridded version (dubbed FSW) has a spacing of 1 at the Equator. A companion document (Rutan et al. 2005) focuses on advances to (and validation of) the ungridded, footprint scale calculations (dubbed CRS), primarily in clear-sky conditions. While mainly intended to provide observations of fluxes at the top of atmosphere (TOA), CERES (Wielicki et al. 1996) includes a program to also compute the fluxes at TOA, within the atmosphere and at the surface, and also to validate the results with independent ground based measurements (Charlock and Alberta 1996). ARM surface data has been a focus for this component of CERES. To permit the user to infer cloud forcing and direct aerosol forcing with the computed SARB, CERES includes surface and TOA fluxes that have been computed for cloud-free (clear) and aerosol free (pristine) footprints; this accounts for aerosol effects (SW scattering and absorption, and LW scattering, absorption and emission) to both clear and cloudy skies.

Charlock, T. P.↗
NASA NTRS1990

Estimating integrated cloud liquid water from extended time observations of solar irradiance

An analysis technique used to estimate the integrated liquid water content (LWC) from the measured solar irradiance is described. The cloud transmittance is computed by dividing the irradiance measured at some time by a clear sky value obtained at the same time on a cloudless day. From the transmittance and the zenith angle, the cloud LWC is computed using the radiative transfer parameterizations of Stephens et al., (1984). The results are compared with 17 days of mm-wave (20.6 and 31.65 GHz) radiometer measurements made during the First ISCCP Regional Experiment (FIRE) Intensive Field Observation (IFO) in July of 1987.

Fairall, C. W.↗
DOE OSTI2016

Accelerating Science Impact through Big Data Workflow Management and Supercomputing

The Large Hadron Collider (LHC), operating at the international CERN Laboratory in Geneva, Switzerland, is leading Big Data driven scientific explorations. ATLAS, one of the largest collaborations ever assembled in the the history of science, is at the forefront of research at the LHC. To address an unprecedented multi-petabyte data processing challenge, the ATLAS experiment is relying on a heterogeneous distributed computational infrastructure. To manage the workflow for all data processing on hundreds of data centers the PanDA (Production and Distributed Analysis)Workload Management System is used. An ambitious program to expand PanDA to all available computing resources, including opportunistic use of commercial and academic clouds and Leadership Computing Facilities (LCF), is realizing within BigPanDA and megaPanDA projects. These projects are now exploring how PanDA might be used for managing computing jobs that run on supercomputers including OLCF’s Titan and NRC-KI HPC2. The main idea is to reuse, as much as possible, existing components of the PanDA system that are already deployed on the LHC Grid for analysis of physics data. The next generation of PanDA will allow many data-intensive sciences employing a variety of computing platforms to benefit from ATLAS experience and proven tools in highly scalable processing.

97 MATHEMATICS AND COMPUTING↗