Cloud Computing at JPL Science Data Systems
Cloud computing is an approach and tool to support "Big Data".
SEARCH · Search NASA
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
Cloud computing is an approach and tool to support "Big Data".
Digitized optical detectors for cloud data processing onboard satellite using Fourier analysis
Cloud properties are critical for understanding the Earth’s radiation budget and cloud feedbacks. At NASA Langley Research Center, the Satellite ClOud and Radiative Property retrieval System (SatCORPS) provides real-time and historical analyses of clouds derived from Geostationary satellite (GEOsat) data for weather and climate applications. For the Clouds and the Earth’s Radiant Energy System (CERES) program, the global constellation of GEOsats has been analyzed since 2000 to help characterize and account for the diurnal cycle of clouds and their radiative impacts in the CERES climate data record. Obtaining consistent cloud properties over the GEOsat data record during the CERES era is a major objective but a significant challenge considering the diversity of imaging capabilities deployed during that time. The GEOsat data analysis approach for the current CERES Edition-4 (Ed4) data products was focused on accuracy and consistency with MODIS by employing as much spectral information as possible from each satellite. However, the inconsistent use of spectral information across GEOsats led to marked discontinuities in the spatial and temporal record of cloud properties that had to be accounted for post facto in downstream CERES processing. This paper reports progress in developing a new GEOsat analysis system for the next CERES edition (Ed5) that has potential to improve cross-platform consistency and continuity. In this approach, the spectral channel complement is limited to just 3-channels during daytime, ~0.65 µm (VIS), ~3.9 µm (NIR), and ~10.8 µm (IR), common to nearly all of the satellites in the record. At night, a 2-channel approach is taken with the NIR and IR, and ~6.7 µm bands that includes a machine learning approach for optically thick cloud properties. A tradeoff is the potential for reduced accuracy particularly using data from the more advanced satellites that have more spectral channels (e.g. SEVIRI, AHI and ABI) that are known to help improve thin cirrus detection, cloud-aerosol discrimination and estimates in other difficult conditions that challenge cloud remote sensing. The new continuity approach is applied to one month of global GEOSat data for each year of the CERES record since 2000 and compared with the Ed4 GEO and MODIS cloud property time series in order to evaluate the level of improved consistency in the GEOsat record and to assess the accuracy impacts. Cloud fraction will also be assessed with CALIPSO data. Outstanding issues and challenges will be discussed. The results are expected to guide future work needed to develop a more robust GEOsat cloud data record for CERES.
The effect of sensor spatial resolution on satellite-derived estimates of cloud fractional coverage is quantified on the basis of Landsat satellite radiance data. Cloud fraction is found to depend on cloud algorithm as much as it depends on sensor spatial resolution. Even for 28.5-m spatial resolution data, large cloud fraction differences exist between algorithms. Satellite cloud retrieval algorithms depend strongly on sensor spatial resolution and/or on the optical depth of the cloud field. When present, spatial resolution effects are small (less than 0.01) for pixel diameter less than 1/4 km and are large for pixel diameter larger than 1 km. The International Satellite Cloud Climatology Project bispectral threshold gives an increase in cloud fraction of 0.11 as spatial resolution degrades from 20 m to 8 km. The spatial coherence algorithm underestimates boundary layer cloud fraction by 0.18. The use of functional box counting and an assumption of fractal scale invariance overestimates the dependence of cloud fraction for spatial scales below 1 km.
Aerosol Observing System (AOS): cloud condensation nuclei data
The spatial and temporal variability of sea ice concentrations derived from passive microwave data is studied in conjunction with co-registered high resolution infrared and visible satellite data. Cloud free infrared and visible data provide surface temperature and large scale surface characteristics, respectively, that can be used to better understand regional and seasonal fluctuations in ice concentrations. Results from correlation analysis of ice concentration versus surface temperature data show the intuitively expected negative relationship but the strength in the relationship is unexpectedly very strong. In the Antarctic, the correlation is consistently very high spatially when yearly anomalies are used, and not so high in some areas when seasonal anomalies are used, especially during spring and summer. In the monthly anomalies, the correlation is also good, especially in dynamically active regions. The expanse in the anomalies in surface temperature are shown to go way beyond the sea ice regions into the open ocean and continental areas, suggesting strong atmospheric forcing. Weak correlations are normally found in highly consolidated areas, where large changes in temperature do not cause large changes in ice concentration on a short term, and in open ocean polynya areas, where the change in ice concentration may be cause by melt from the underside of the ice. In the Arctic, strong correlations between surface temperature and ice concentration are evident for all seasons except during the summer. In the summer, factors such as meltponding, surface wetness, and ice breakup, as detected by high resolution visible data, contributes to larger uncertainties in the determination of ice concentration and the lack of good correlation of the variables.
The Astromaterials Acquisition & Curation Office at NASA's Johnson Space Center (JSC) is the designated facility for curating all of NASA's extraterrestrial samples. Today, the suite of collections includes the lunar samples from the Apollo missions, cosmic dust particles falling into the Earth's atmosphere, meteorites collected in Antarctica, comet and interstellar dust particles from the Stardust mission, asteroid particles from Japan's Hayabusa mission, solar wind atoms collected during the Genesis mission, and space‐exposed hardware from several missions. To support planetary science research on these samples, JSC's Astromaterials Curation Office hosts NASA's Astromaterials Curation digital repository and data access portal [http://curator.jsc.nasa.gov/], providing descriptions of the missions and collections, and critical information about each individual sample. Our office is designing and implementing several informatics initiatives to better serve the planetary research community. First, we are re‐hosting the basic database framework by consolidating legacy databases for individual collections and providing a uniform access point for information (descriptions, imagery, classification) on all of our samples. Second, we continue to upgrade and host digital compendia that summarize and highlight published findings on the samples (e.g., lunar samples, meteorites from Mars). We host high resolution imagery of samples as it becomes available, including newly scanned images of historical prints from the Apollo missions. Finally we are creating plans to collect and provide new data, including 3D imagery, point cloud data, micro CT data, and external links to other data sets on selected samples. Together, these individual efforts will provide unprecedented digital access to NASA's Astromaterials, enabling preservation of the samples through more specific and targeted requests, and supporting new planetary science research and collaborations on the samples.
Total atmospheric radiation calculations from cloud data using actinometric instruments
This document describes a new Atmospheric Radiation Measurement (ARM) data set, the ARM Cloud Retrieval Ensemble Data Set (ACRED), which is created by assembling nine existing ground-based cloud retrievals of ARM measurements from different cloud retrieval algorithms. The current version of ACRED includes an hourly average of nine ground-based retrievals with vertical resolution of 45 m for 512 layers. The techniques used for the nine cloud retrievals are briefly described in this document. This document also outlines the ACRED data availability, variables, and the nine retrieval products. Technical details about the generation of ACRED, such as the methods used for time average and vertical re-grid, are also provided.
In the year 2001, NASA will launch the satellite TRIANA that will be the first Earth observing mission to provide a continuous, full disk view of the sunlit Earth. As a part of the HPCC Program at NASA GSFC, we have started a project whose objectives are to develop and implement a 3D cloud data assimilation system, by combining TRIANA measurements with model simulation, and to produce accurate statistics of global cloud coverage as an important element of the Earth's climate. For simulation of the atmosphere within this project we are using the NCEP/NOAA operational Eta model. In order to compare TRIANA and the Eta model data on approximately the same grid without significant downscaling, the Eta model will be integrated at a resolution of about 15 km. The integration domain (from -70 to +70 deg in latitude and 150 deg in longitude) will cover most of the sunlit Earth disc and will continuously rotate around the globe following TRIANA. The cloud data assimilation is supposed to run and produce 3D clouds on a near real-time basis. Such a numerical setup and integration design is very ambitious and computationally demanding. Thus, though the Eta model code has been very carefully developed and its computational efficiency has been systematically polished during the years of operational implementation at NCEP, the current MPI version may still have problems with memory and efficiency for the TRIANA simulations. Within this work, we optimize a parallel version of the Eta model code on a Cray T3E and a network of PCs (theHIVE) in order to improve its overall efficiency. Our optimization procedure consists of introducing dynamically allocated arrays to reduce the size of static memory, and optimizing on a single processor by splitting loops to limit the number of streams. All the presented results are derived using an integration domain centered at the equator, with a size of 60 x 60 deg, and with horizontal resolutions of 1/2 and 1/3 deg, respectively. In accompanying charts we report the elapsed time, the speedup and the Mflops as a function of the number of processors for the non-optimized version of the code on the T3E and theHIVE. The large amount of communication required for model integration explains its poor performance on theHIVE. Our initial implementation of the dynamic memory allocation has contributed to about 12% reduction of memory but has introduced a 3% overhead in computing time. This overhead was removed by performing loop splitting in some of the high demanding subroutines. When the Eta code is fully optimized in order to meet the memory requirement for TRIANA simulations, a non-negligeable overhead may appear that may seriously affect the efficiency of the code. To alleviate this problem, we are considering implementation of a new algorithm for the horizontal advection that is computationally less expensive, and also a new approach for marching in time.
A cryogenic, 50 liter volume Planetary Cloud Simulation Chamber has been constructed to permit the laboratory study of the cloud compositions which are likely to be found in the atmospheres of the outer planets. On the basis of available data, clouds composed of water ice, carbon dioxide, and liquid and solid ammonia and methane, both pure and in various mixtures, have been generated. Cloud microphysical observations have been permitted through the use of a cloud particle slide injector and photomicrography. Viewports in the lower chamber have enabled the collection of cloud backscattering data using 633 and 838 nm laser light, including linear depolarization ratios and complete Stokes parameterization. The considerable technological difficulties associated with the collection of angular scattering patterns within the chamber, however, could not be completely overcome.
Cloud-resolving models (CRMs) provide an effective linkage in terms of parameters and scales between observations and the parametric treatments of clouds in global climate models (GCMs). They also represent the best understanding of the physical processes acting to determine cloud system lifecycle. The goal of this project is to improve state-of-the-art CRMs used for studies of cirrus clouds and to establish a relative calibration with GCMs through comparisons among CRMs, single column model (SCM) versions of the GCMs, and observations. This project will compare and evaluate a variety of CRMs and SCMs, under the auspices of the GEWEX Cloud Systems Study (GCSS) Working Group on Cirrus Cloud Systems (WG2), using ARM data acquired at the Southern Great Plains (SGP) site. This poster will report on progress in developing a suitable WG2 case study data set based on the September 26, 1996 ARM IOP case - the Hurricane Nora outflow case. The environmental data (input) will be described as well as the wealth of validating cloud observations. We plan to also show results of preliminary simulations. The science questions to be addressed derive significantly from results of the GCSS WG2 cloud model comparison projects, which will be briefly summarized.
This request is to release LiDAR data files acquired by SRNL R&D Engineering and H-Area Operations and Engineering while performing scans of the interior of the H-Canyon Exhaust tunnel. LiDAR data includes 3-dimensional (3D) point cloud data sets and panoramic digital images of the interior tunnel wall surfaces. Presently two deployments have been completed, the first in November 2019, Fig. 1, and the second in June of 2020. This request is for the release of the data collected during those two deployments. Detailed information on the deployment and data collected can be found in SRS document C-ESR-H-00072, “November 2019 Initial Deployment of LiDAR”. It is planned to perform ongoing scans at approximately 6-month intervals, the purpose of the deployments is to evaluate the usefulness of the data collected to enable quantitative measurements such as tunnel dimensions and rate of surface erosion and as a precursor to a potential deployment of a LiDAR system on the tunnel inspection crawler.
BEACONS aims to demonstrate autonomous deployment of a shipborne system that routinely observes atmospheric, cloud, and aerosol properties. These observations enable scientists to study aerosol-cloud interactions, advancing energy resilience and the Department of Energy's mission to improve model prediction. Key Project Outcomes Autonomous deployment of aerosol and cloud measurement systems aboard a Pasha Hawaii Marjorie C commercial ship. Six-month campaign for continuous, high-resolution aerosol and cloud data collection with minimal crew intervention. Development and testing of instruments in two phases, advancing from essential to complex systems. AI-ready processed data on the BEACONS Data Platform for near-real-time data access. Validation of aerosol-cloud interaction hypotheses for improved modeling. Establishment of best practices for instrument deployment in marine environments. Scientific collaboration to support the U.S. Department of Energy (DOE) Biological and Environmental Research (BER) mission.
The Stratospheric Aerosol and Gas Experiment II (SAGE II) aboard the Earth Radiation Budget Satellite was launched from Shuttle in October 1984. SAGE II is a seven-channel sun-photometer measuring stratospheric aerosols, ozone, water vapor, and nitrogen dioxide during each spacecraft sunrise and sunset. In addition to stratospheric information, mid-tropospheric and higher water vapor, ozone, and aerosol data are being produced in cloud-free regions, and cloud data everywhere else. Aerosol information is being produced at three wavelengths and, together with water vapor data, is providing a global microphysical description of the aerosol.
The Nimbus-7 Global Cloud Climatology (N7GCC) has been produced from measurements made between April 1979 and March 1985 using the Temperature Humidity IR Radiometer and the Total Ozone Mapping Spectrometer on the Nimbus-7 satellite. The N7GCC gives, near local noon and midnight, the fractional area covered by high-level, middle-level, and low-altitude clouds, and the total fractional area covered by all clouds. Statistics for cirrus, deep convective, and warm low-altitude clouds and the cloud and clear-sky radiances with correlative surface temperatures are also included. The N7GCC is compared with other cloud data sets, including the International Satellite Cloud Climatology Project.
This work addresses distributed data management and access dynamically configurable high-speed access to data distributed and shared over wide-area high-speed network environments. An offload engine NIC (network interface card) is proposed that scales at nX10-Gbps increments through 100-Gbps full duplex. The Globus de facto standard was used in projects requiring secure, robust, high-speed bulk data transport. Novel extension mechanisms were derived that will combine these technologies for use by GridFTP, bandwidth management resources, and host CPU (central processing unit) acceleration. The result will be wire-rate encrypted Globus grid data transactions through offload for splintering, encryption, and compression. As the need for greater network bandwidth increases, there is an inherent need for faster CPUs. The best way to accelerate CPUs is through a network acceleration engine. Grid computing data transfers for the Globus tool set did not have wire-rate encryption or compression. Existing technology cannot keep pace with the greater bandwidths of backplane and network connections. Present offload engines with ports to Ethernet are 32 to 40 Gbps f-d at best. The best of ultra-high-speed offload engines use expensive ASICs (application specific integrated circuits) or NPUs (network processing units). The present state of the art also includes bonding and the use of multiple NICs that are also in the planning stages for future portability to ASICs and software to accommodate data rates at 100 Gbps. The remaining industry solutions are for carrier-grade equipment manufacturers, with costly line cards having multiples of 10-Gbps ports, or 100-Gbps ports such as CFP modules that interface to costly ASICs and related circuitry. All of the existing solutions vary in configuration based on requirements of the host, motherboard, or carriergrade equipment. The purpose of the innovation is to eliminate data bottlenecks within cluster, grid, and cloud computing systems, and to add several more capabilities while reducing space consumption and cost. Provisions were designed for interoperability with systems used in the NASA HEC (High-End Computing) program. The new acceleration engine consists of state-ofthe- art FPGA (field-programmable gate array) core IP, C, and Verilog code; novel communication protocol; and extensions to the Globus structure. The engine provides the functions of network acceleration, encryption, compression, packet-ordering, and security added to Globus grid or for cloud data transfer. This system is scalable in nX10-Gbps increments through 100-Gbps f-d. It can be interfaced to industry-standard system-side or network-side devices or core IP in increments of 10 GigE, scaling to provide IEEE 40/100 GigE compliance.
Millimeter Wavelength Cloud Radar (MMCR): monitoring data