Search NASA⌕ Search

SEARCH · Search NASA

Results for “cloud computing, data cloud”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 415 records · Page 23
All sourcesNASA NTRSDOE OSTI
DOE OSTI2023

EPiC-GAN: Equivariant point cloud generation for particle jets

With the vast data-collecting capabilities of current and future high-energy collider experiments, there is an increasing demand for computationally efficient simulations. Generative machine learning models enable fast event generation, yet so far these approaches are largely constrained to fixed data structures and rigid detector geometries. In this paper, we introduce EPiC-GAN - equivariant point cloud generative adversarial network - which can produce point clouds of variable multiplicity. This flexible framework is based on deep sets and is well suited for simulating sprays of particles called jets. The generator and discriminator utilize multiple EPiC layers with an interpretable global latent vector. Crucially, the EPiC layers do not rely on pairwise information sharing between particles, which leads to a significant speed-up over graph- and transformer-based approaches with more complex relation diagrams. We demonstrate that EPiC-GAN scales well to large particle multiplicities and achieves high generation fidelity on benchmark jet generation tasks.

Buhmann, Erik↗
NASA NTRS2022

NASA's Next Generation of Atmospheric Data Science

The Multi-Angle Imager for Aerosols (MAIA) and the Tropospheric Emission: Monitoring of Pollution(TEMPO) are NASA’s next-generation satellite missions for air quality monitoring. These missions will produce high-quality, high-resolution air quality data to support cross-displinary research. The MAIA mission is collaborating with health science researchers and epidemiologists to study the impacts of air quality on health outcomes. TEMPO aims to improve our understanding of tropospheric air pollution chemistry and our ability to make predictions about air quality and climate forcing. TEMPO will offer hourly measurements of tropospheric ozone, aerosols, and clouds focused on North America at high-spatial resolution, while MAIA will produce high-resolution measurements of speciated particulate matter targeting densely populated cities around the globe. Data from these missions will help improve our understanding of the sources, types, and interactions among the aerosols and trace gases that are polluting Earth’s atmosphere, as well as our understanding of the impact of air pollution on pollution on a wide range of important areas including human health, agriculture, weather, and climate change. The challenges of cross-disciplinary research, computationally expensive multi-variate analyses, and high-resolution data at both local and global scales are driving substantial changes across all of NASA’s Distributed Active Archive Centers (DAACs). High resolution data at scales such these requires a new approach to data ingest, archive, and publication. Like other NASA DAACs, the Atmospheric Science Data Center (ASDC), the DAAC that will be responsible for publishing MAIA and TEMPO data products has historically archived and distributed data on premise. DAACs of the future will archive and distribute data in the cloud, enabling them to remake themselves as research-focused data centers that will support on-demand, data-intensive computations for highly accurate retrospective analyses and predictions. Under the new paradigm, data formats and metadata must support on-demand spatial and temporal sub-setting, as well as other data transformation services such as re-gridding and re-sampling. This presentation will discuss work being done to address data formatting and metadata requirements in this dynamic new environment. In addition to the changes in data stewardship practices at the ASDC, the increased focus on supporting scientific research is driving changes in the relationship between DAACs and researchers. While the ASDC will continue to provide first rate data management and stewardship, it is increasingly focused on serving as a partner not only to the science teams that gather and produce the data it publishes, but to the researchers that use that data.

Beth Huffer↗
DOE OSTI2015

Parallel Programming Models and Systems for High Performance Computing. In: Emerging Research in Cloud Distributed Computing Systems

A parallel programming model is an abstraction of a parallel system that allows expression of both algorithms and shared data structures. To accommodate the diversity in parallel system architectures and user requirements, there are a variety of programming models including the models providing a shared memory view or a distributed memory view of the system. The programming models are implemented as libraries, language extensions, or compiler directives. This chapter provides a discussion on programming models and its implementations aimed at application developers, system software researchers, and hardware architects. The first part provides an overview of the programming models. The second part is an in-depth discussion on high-performance networking interface to implement the programming model. The last part of the chapter discusses implementation of a programming model with a case study. Each part of the chapter concludes with a discussion on current research trends and its impact on future architectures.

Venkata, Manjunath Gorentia↗
NASA NTRS2004

The Goddard Profiling Algorithm (GPROF): Description and Current Applications

Atmospheric scientists use different methods for interpreting satellite data. In the early days of satellite meteorology, the analysis of cloud pictures from satellites was primarily subjective. As computer technology improved, satellite pictures could be processed digitally, and mathematical algorithms were developed and applied to the digital images in different wavelength bands to extract information about the atmosphere in an objective way. The kind of mathematical algorithm one applies to satellite data may depend on the complexity of the physical processes that lead to the observed image, and how much information is contained in the satellite images both spatially and at different wavelengths. Imagery from satellite-borne passive microwave radiometers has limited horizontal resolution, and the observed microwave radiances are the result of complex physical processes that are not easily modeled. For this reason, a type of algorithm called a Bayesian estimation method is utilized to interpret passive microwave imagery in an objective, yet computationally efficient manner.

Olson, William S.↗
NASA NTRS1979

Liquid content of the lower clouds of Venus as determined from Mariner 10 radio occultation

S- and X-band radio occulation data obtained during the Mariner 10 flyby of Venus are analyzed to determine the effects of signal attenuation due to the clouds of Venus. The signal-amplitude data are corrected for errors introduced by spacecraft motion and antenna steering during the occultation, absorption-coefficient profiles are computed, and the liquid content of the Venusian clouds is estimated. Excess attenuation effects are found and ascribed to absorption by small liquid particles, consisting of a water-sulfuric acid mixture containing 75% + or - 25% sulfuric acid, in the clouds. Three absorbing cloud layers at respective heights of 68, 60, and 48 km are suggested by the X-band absorptivity profile, but only the 68- and 48-km layers are clearly indicated by the S-band profile. The relatively slow S-band absorption at 60 km is tentatively attributed to an unknown constituent that is much more absorbing in the X-band than in the S-band. The results are compared with nephelometer measurements performed at 0.9 micron by the Venera 9 and 10 probes.

Kliore, A. J.↗
NASA NTRS1982

Satellite measurements of cloud reflectance and optical thickness

The cloud reflectance and the optical thickness have been computed with the aid of atmospheric models from the first eleven months (April 1970-February 1971) of backscattered ultraviolet (BUV) data over the Pacific Ocean. Both the cloud and the optical thickness are derived from the 380 nm channel by assuming that the entire IFOV (200 km) is filled by stratiform clouds. The analysis shows a large variability in the cloud reflectance in both the intertropic conversion zone (ITCZ) and the high latitudes. It is also found that for 90 percent of the time in the tropics, the clouds have an optical thickness of less than 10. An analysis of clouds with optical thickness between 10 and 20 shows that in the tropics the minimum frequency coincides with the dry zone at 2.5 deg S and the maximum frequency for clouds of optical thickness 10-20 is greater in summer than in winter and irrespective of the season; 50 percent of the time the clouds have an optical thickness less than 13.

Ahmad, Z.↗
NASA NTRS2019

Deep Learning Emulation of Atmospheric Correction for Geostationary Sensors

New generation geostationary satellites make reflectance observations available at a continental scale with unprecedented spatiotemporal resolution and spectral range. Generating Earth monitoring products from these observations requires retrieval of the basic parameter, surface reflectance (SR), by atmospheric correction (AC). Algorithms for atmospheric correction, including Multi-Angle Implementation of Atmospheric Correction (MAIAC), are adapted for each sensor and are too computationally complex to be run in real time, relying instead on look-up tables with precomputed values. Machine learning methods, including convolutional neural networks, have demonstrated performance in learning complex, nonlinear mappings and extracting insight from high-dimensional remote sensing data. In this work, we present a deep learning emulator of MAIAC to retrieve both SR and cloud products. Using this adaptation of deep learning-based emulation to remote sensing, we demonstrate stable SR retrieval over a variety of land covers and viewing conditions and accurate cloud detection. Further, a comparison of computation time suggests emulation as a compelling alternative for expensive physical simulation, especially for applications benefited by near-real time data, such as agricultural management and disaster response.

Duffy, Kate↗
NASA NTRS2005

Global, Multi-Year Analysis of Clouds and Earth's Radiant Energy System Terra Observations and Radiative Transfer Calculations

An extended record of the Terra Surface and Atmosphere Radiation Budget (SARB) computed by CERES (Clouds and Earth s Radiant Energy System) is produced in gridded form, facilitating an investigation of global scale direct aerosol forcing. The new gridded version (dubbed FSW) has a spacing of 1 at the Equator. A companion document (Rutan et al. 2005) focuses on advances to (and validation of) the ungridded, footprint scale calculations (dubbed CRS), primarily in clear-sky conditions. While mainly intended to provide observations of fluxes at the top of atmosphere (TOA), CERES (Wielicki et al. 1996) includes a program to also compute the fluxes at TOA, within the atmosphere and at the surface, and also to validate the results with independent ground based measurements (Charlock and Alberta 1996). ARM surface data has been a focus for this component of CERES. To permit the user to infer cloud forcing and direct aerosol forcing with the computed SARB, CERES includes surface and TOA fluxes that have been computed for cloud-free (clear) and aerosol free (pristine) footprints; this accounts for aerosol effects (SW scattering and absorption, and LW scattering, absorption and emission) to both clear and cloudy skies.

Charlock, T. P.↗
NASA NTRS1990

DC-8 mission planning

A number of tasks needed to be performed to operate the DC-8 aircraft in the Arctic Ozone Expedition during 1989. These included development of improved navigational computer programs and performing advanced planning for flight tracks. The missions required further work in the field to plan flight tracks and to establish a data archive. The DC-8 aircraft was operated during the deployment phase of the Arctic Ozone Expedition. Data were collected revealing the presence of extensive systems of polar stratospheric clouds. The data also showed the evolution of the chlorine and nitrogen chemistry within the Arctic vortex. A navigational computer code was improved so that the DC-8 navigators could perform their function more efficiently.

Toon, Owen B.↗
NASA NTRS2017

Trade Study: Storing NASA HDF5/netCDF-4 Data in the Amazon Cloud and Retrieving Data Via Hyrax Server Data Server

This study explored three candidate architectures with different types of objects and access paths for serving NASA Earth Science HDF5 data via Hyrax running on Amazon Web Services (AWS). We studied the cost and performance for each architecture using several representative Use-Cases. The objectives of the study were: Conduct a trade study to identify one or more high performance integrated solutions for storing and retrieving NASA HDF5 and netCDF4 data in a cloud (web object store) environment. The target environment is Amazon Web Services (AWS) Simple Storage Service (S3). Conduct needed level of software development to properly evaluate solutions in the trade study and to obtain required benchmarking metrics for input into government decision of potential follow-on prototyping. Develop a cloud cost model for the preferred data storage solution (or solutions) that accounts for different granulation and aggregation schemes as well as cost and performance trades.We will describe the three architectures and the use cases along with performance results and recommendations for further work.

AWS cost↗
NASA NTRSDate not supplied

Development of a Consistent MODIS and VIIRS Cloud Detection Approach for CERES

A consistent cloud fraction record across various satellite platforms is essential for maintaining a long-term and stable climate data record of Earth's energy budget. With the Aqua satellite nearing the end of its operational lifetime, the continuation of this record relies on utilizing VIIRS observations from NOAA20 for cloud detection in NASA’s Clouds and Earth’s Radiant Energy System (CERES) project. However, integrating data from VIIRS and MODIS instruments poses challenges due to their distinct characteristics, such as varying spatial resolutions and different spectral channels. As a result, deriving consistent cloud properties from these two sensors without introducing artificial discontinuities in the time series remains a complex and challenging task. This paper will present progress toward developing a unified MODIS and VIIRS cloud mask using common channels to produce consistent cloud properties for CERES next edition (Ed5) Earth radiation budget data products. The fundamental approach taken in the CERES cloud mask is to compare the observed radiances to the expected background clear sky radiances. Therefore, one vital step is to compute clear sky radiances with a radiative transfer model that accurately accounts for satellite-specific, spectrally dependent surface reflectance, surface emission, and atmospheric absorption. Refined radiative transfer models and updated ancillary data inputs including surface emissivity maps, IGBP, snow and ice maps are incorporated into the processing framework to improve cloud detection consistency and accuracy. Pixel level cloud mask results and monthly global cloud fraction comparisons between MODIS and VIIRS will be presented to evaluate their consistency. Remaining challenges will be discussed. It is expected that this work will contribute consistent cloud properties for CERES that adequately bridges the MODIS and VIIRS imager data records.

CERES↗
DOE OSTI2020

Joint Genome Institute Analysis Workflow Service (JAWS) v2.7

The U.S. Department of Energy Joint Genome Institute (JGI) has developed the JGI Analysis Workflow Service (JAWS) as a distributed framework to run computational workflows across diverse high-performance computing (HPC) and cloud environments. JAWS enhances the reusability, scalability, and robustness of scientific workflows by orchestrating data movement, code execution, and results retrieval across multiple DOE facilities. At its core, JAWS integrates the Cromwell workflow engine to run workflows expressed in the Workflow Description Language (WDL), ensuring portability and interoperability. To provide consistent runtime environments, JAWS employs container technologies such as Shifter, Apptainer, and Docker. Workflow tasks are managed via HTCondor on HPC backends, while Globus ensures secure and efficient data transfer between sites. JAWS is deployed as a multi-site workflow manager across national laboratory computing facilities, with dedicated instances supporting community projects such as the National Microbiome Data Collaborative (NMDC) and KBase. This distributed, service-oriented architecture enables users to "write once, run anywhere," providing scalable, production-quality workflow execution.

Kirton, Edward↗
NASA NTRS1994

Approaches of comparison for clear-sky radiative fluxes from general circulation models with Earth Radiation Budget Experiment data

In order to compare the clear-sky greenhouse effect and cloud-radiative forcing from general circulation models with Earth Radiation Budget Experiments (ERBE) data, it is necessary to calculate the general circulation model (GCM) clear-sky radiative fluxes in a way consistent with ERBE. This study discusses problems associated with the available methods for clear-sky radiative flux computations in GCMs and proposes a new approach, which uses a statistical relationship between the grid cloud cover and the availability of ERBE clear-sky measurement, established from ERBE pixel data, to sample the model radiative fluxes. Calculations with version 2 of the National Center for Atmospheric Research (NCAR) Community Climate Model using observed sea surface temperature (SST) show good agreement of clear-sky sampling from the proposed method with ERBE sampling. It is also shown that large improvements are achieved in the spatial variability of the model clear-sky radiative fluxes over ocean, with reference to ERBE, by using the new clear-sky sampling method.

Zhang, M. H.↗
NASA NTRS2008

Radiative Susceptibility of Cloudy Atmospheres to Droplet Number Perturbations: 1. Theoretical Analysis and Examples from MODIS

Theoretical and satellite-based assessments of the sensitivity of broadband shortwave radiative fluxes in cloudy atmospheres to small perturbations in the cloud droplet number concentration (N) of liquid water clouds under constant water conditions are performed. Two approaches to study this sensitivity are adopted: absolute increases in N, for which the radiative response is referred to as absolute cloud susceptibility, and relative increases in N or relative cloud susceptibility. Estimating the former is more challenging as it requires an assumed value for either cloud liquid water content or geometrical thickness; both susceptibilities require an assumed relationship between the droplet volume and effective radius. Expanding upon previous susceptibility studies, present radiative calculations include the effect of AN perturbations on droplet asymmetry parameter and single-scattering albedo, in addition to extinction. Absolute cloud susceptibility has a strong nonlinear dependence on the droplet effective radius as expected, while relative cloud susceptibility is primarily dependent on optical thickness. Molecular absorption and reflecting surfaces both reduce the relative contribution of the cloud to the top-of-atmosphere (TOA) flux and therefore also reduce the TOA albedo susceptibility. Transmittance susceptibilities are negative with absolute values similar to albedo susceptibility, while atmospheric absorptance susceptibilities are about an order of magnitude smaller than albedo susceptibilities and can be either positive or negative. Observation-based susceptibility calculations are derived from MODIS pixel-level retrievals of liquid water cloud optical thickness, effective radius, and cloud top temperature; two data granule examples are shown. Susceptibility quantifies the aerosol indirect effect sensitivity in a way that can be easily computed from model fields. As such, susceptibilities derived from MODIS observations provide a higher-order test of model cloud properties used for indirect effect studies. MODIS-derived global distributions of cloud susceptibility and radiative forcing calculations are presented in a companion paper.

Platnick, Steven↗
DOE OSTI2016

Kira: Processing Astronomy Imagery Using Big Data Technology

Scientific analyses commonly compose multiple single-process programs into a dataflow. An end-to-end dataflow of single-process programs is known as a many-task application. Typically, HPC tools are used to parallelize these analyses. In this work, we investigate an alternate approach that uses Apache Spark-a modern platform for data intensive computing-to parallelize many-task applications. We implement Kira, a flexible and distributed astronomy image processing toolkit, and its Source Extractor (Kira SE) application. Using Kira SE as a case study, we examine the programming flexibility, dataflow richness, scheduling capacity and performance of Apache Spark running on the Amazon EC2 cloud. By exploiting data locality, Kira SE achieves a 4.1× speedup over an equivalent C program when analyzing a 1TB dataset using 512 cores on the Amazon EC2 cloud. Furthermore, Kira SE on the Amazon EC2 cloud achieves a 1.8× speedup over the C program on the NERSC Edison supercomputer. A 128-core Amazon EC2 cloud deployment of Kira SE using Spark Streaming can achieve a second-scale latency with a sustained throughput of 800 MB/s. Our experience with Kira demonstrates that data intensive computing platforms like Apache Spark are a performant alternative for many-task scientific applications.

97 MATHEMATICS AND COMPUTING↗
NASA NTRS1971

Apollo contamination analysis

The Apollo 12 photographic observations of clouds produced by venting liquids into space were reduced and analyzed. The number of Baker-Nunn photographs required that film measurement and data processing be automated, and the microdensitometer was adapted to automatic digital operation. Computer programs were prepared to process the image-density measurements. A substantial selection of Apollo 12 data was processed; both hydrogen and oxygen clouds are represented. Some limited data were obtained from Apollo 13 and 14 missions. During the Apollo 12 data analysis, the theory and models of cloud phenomena were refined because the models were initially applied to inferior oxygen data from earlier missions. The reported observations of water dumps from Apollo 12 are also discussed, along with the use of the oxygen and hydrogen data from Apollo 12 photographs with the cloud models.

Source record↗
NASA NTRSDate not supplied

Advancing Open Source Science Initiatives Through Public-Private Partnerships

Collaboration is fundamental to advancing open science within the science community. With the recent developments in technology and research, the establishment of formal partnerships between the private sector and government agencies are needed to bridge the knowledge gaps and expedite the time to actionable science. NASA’s Interagency Implementation and Advanced Concepts Team (IMPACT) seeks to address this challenge by establishing non-reimbursable Space Act Agreements with industry leaders in cloud computing, artificial intelligence (AI) and machine learning. The purpose of these agreements is to advance open source science initiatives in the areas of data discovery, access and use of high value NASA science data sets on the cloud. As well as, jointly work on common research problems to accelerate the development and adoption of new AI technologies. Current success stories include co-locating NASA datasets from multiple science disciplines on one platform using Amazon Web Services Open Data Registry, developing AI Foundation Models for Science with IBM and co-hosting training workshops and tutorials for the science community aimed at providing hands-on experience with using NASA data and AI models on the cloud. In summary, we will present an overview of our partnerships supporting open source science initiatives, describe current activities and lessons learned that may be useful to others considering similar partnerships with the private sector.

Elizabeth Fancher↗
NASA NTRS1999

Development and Validation of a Polar Cloud Algorithm for CERES

The objectives of this project, as described in the original proposal, were to develop an algorithm for diagnosing cloud properties over snow- and ice-covered surfaces, particularly at night, using satellite radiances from the Advanced Very High Resolution Radiometer (AVHRR) and High-resolution Infrared Radiation Sounder (HIRS) sensors. Products from this algorithm include a cloud mask and additional cloud properties such as cloud phase, amount, and height. The SIVIS software package, developed as a part of the CERES project, was originally the primary tool used to develop the algorithm, but as it is no longer supported we have had to pursue a new tool to enable the combination and analysis of collocated radiances from AVHRR and HIRS. This turned out to be a much larger endeavor than we expected, but we now have the data sets collocated (with many thanks to B. Baum for the fundamental code) and we have developed a nighttime cloud detection algorithm. Using this algorithm we have also computed realistic-looking cloud fractions from AVHRR brightness temperatures. A method to identify cloud phase has also been implemented. Atmospheric information from the TIROS Operational Vertical Sounder (TOVS) Polar Pathfinder Data Set, which includes temperature and moisture profiles as well as surface information, provides information required for determining cloud-top height.

Source record↗