Search NASA⌕ Search

SEARCH · Search NASA

Results for “CLOUD DATA”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 235 records · Page 13

National Center for Airborne Laser Mapping (NCALM) LiDAR, Imagery, and DEM data from five NGEE Arctic Sites, Seward Peninsula, Alaska, August 2021

From August 8 through August 16 of 2021, airborne remote sensing data was collected by the National Center for Airborne Laser Mapping (NCALM) in collaboration with NGEE Arctic scientists. Data was collected around five NGEE Arctic study sites on the Seward Peninsula of Alaska: Teller mm 27, Teller mm 47, Kougarok mm 64, Kougarok mm 86, and Council mm 71. A Robinson R44 II helicopter with a RIEGL VQ-580 II airborne laser scanner was used to collect the LiDAR point cloud data for each study site. A Phase One iXM-RS100F camera was integrated with the Riegl sensor to collect RGB imagery. This data package contains LiDAR point clouds (.las), RGB imagery (tif), 1 m or 50 cm Digital Elevation Models (.tif) generated from the LiDAR data, and shapefiles of the .las tiling system for each site (.shp). Two supplemental documents are also included in the package: 1) a report describing data collection details, GNSS corrections, and processing steps and 2) a document describing the LiDAR Classification used (.pdf). This survey was conducted towards the end of the summer on the Seward Peninsula, and can be paired with data collected in April of 2022 during the snow-on campaign "National Center for Airborne Laser Mapping (NCALM) LiDAR and DEM data from two NGEE Arctic Sites, Seward Peninsula, Alaska, Winter 2022" (Singhania et.al, 2023) (NGA314). The Next-Generation Ecosystem Experiments: Arctic (NGEE Arctic), was a research effort to reduce uncertainty in Earth System Models by developing a predictive understanding of carbon-rich Arctic ecosystems and feedbacks to climate. NGEE Arctic was supported by the Department of Energy's Office of Biological and Environmental Research. The NGEE Arctic project had two field research sites: 1) located within the Arctic polygonal tundra coastal region on the Barrow Environmental Observatory (BEO) and the North Slope near Utqiagvik (Barrow), Alaska and 2) multiple areas on the discontinuous permafrost region of the Seward Peninsula north of Nome, Alaska. Through observations, experiments, and synthesis with existing datasets, NGEE Arctic provided an enhanced knowledge base for multi-scale modeling and contributed to improved process representation at global pan-Arctic scales within the Department of Energy's Earth system Model (the Energy Exascale Earth System Model, or E3SM), and specifically within the E3SM Land Model component (ELM).

54 ENVIRONMENTAL SCIENCES↗

Archive Management of NASA Earth Observation Data to Support Cloud Analysis

NASA collects, processes and distributes petabytes of Earth Observation (EO) data from satellites, aircraft, in situ instruments and model output, with an order of magnitude increase expected by 2024. Cloud-based web object storage (WOS) of these data can simplify the execution of such an increase. More importantly, it can also facilitate user analysis of those volumes by making the data available to the massively parallel computing power in the cloud. However, storing EO data in cloud WOS has a ripple effect throughout the NASA archive system with unexpected challenges and opportunities. One challenge is modifying data servicing software (such as Web Coverage Service servers) to access and subset data that are no longer on a directly accessible file system, but rather in cloud WOS. Opportunities include refactoring of the archive software to a cloud-native architecture; virtualizing data products by computing on demand; and reorganizing data to be more analysis-friendly.

Lynnes, Christopher↗

Archive Management of NASA Earth Observation Data to Support Cloud Analysis

NASA collects, processes and distributes petabytes of Earth Observation (EO) data from satellites, aircraft, in situ instruments and model output, with an order of magnitude increase expected by 2024. Cloud-based web object storage (WOS) of these data can simplify the execution of such an increase. More importantly, it can also facilitate user analysis of those volumes by making the data available to the massively parallel computing power in the cloud. However, storing EO data in cloud WOS has a ripple effect throughout the NASA archive system with unexpected challenges and opportunities. One challenge is modifying data servicing software (such as Web Coverage Service servers) to access and subset data that are no longer on a directly accessible file system, but rather in cloud WOS. Opportunities include refactoring of the archive software to a cloud-native architecture; virtualizing data products by computing on demand; and reorganizing data to be more analysis-friendly. Reviewed by Mark McInerney ESDIS Deputy Project Manager.

Lynnes, Christopher↗

An integrated data management and informatics framework for continuous drug product manufacturing processes: A case study on two pilot plants

The pharmaceutical industry continuously looks for ways to improve its development and manufacturing efficiency. In recent years, such efforts have been driven by the transition from batch to continuous manufacturing and digitalization in process development. To facilitate this transition, integrated data management and informatics tools need to be developed and implemented within the framework of Industry 4.0 technology. Here, in this regard, the work aims to guide the data integration development of continuous pharmaceutical manufacturing processes under the Industry 4.0 framework, improving digital maturity and enabling the development of digital twins. This paper demonstrates two instances where a data integration framework has been successfully employed in academic continuous pharmaceutical manufacturing pilot plants. Details of the integration structure and information flows are comprehensively showcased. Approaches to mitigate concerns in incorporating complex data streams, including integrating multiple process analytical technology tools and legacy equipment, connecting cloud data and simulation models, and safeguarding cyber-physical security, are discussed. Critical challenges and opportunities for practical considerations are highlighted.

59 BASIC BIOLOGICAL SCIENCES↗

Validation of AIRS/AMSU Cloud Retrievals Using MODIS Cloud Analyses

The AIRS/AMSU (flying on the EOS-AQUA satellite) sounding retrieval methodology allows for the retrieval of key atmospheric/surface parameters under partially cloudy conditions (Susskind et al.). In addition, cloud parameters are also derived from the AIRS/AMSU observations. Within each AIRS footprint, cloud parameters at up to 2 cloud layers are determined with differing cloud top pressures and effective (product of infrared emissivity at 11 microns and physical cloud fraction) cloud fractions. However, so far the AIRS cloud product has not been rigorously evaluated/validated. Fortunately, collocated/coincident radiances measured by MODIS/AQUA (at a much lower spectral resolution but roughly an order of-magnitude higher spatial resolution than that of AIRS) are used to determine analogous cloud products from MODIS. This allows us for a rather rare and interesting possibility: the intercomparisons and mutual validation of imager vs. sounder-based cloud products obtained from the same satellite positions. First, we present results of small-scale (granules) instantaneous intercomparisons. Next, we will evaluate differences of temporally averaged (monthly) means as well as the representation of inter-annual variability of cloud parameters as presented by the two cloud data sets. In particular, we present statistical differences in the retrieved parameters of cloud fraction and cloud top pressure. We will investigate what type of cloud systems are retrieved most consistently (if any) with both retrieval schemes, and attempt to assess reasons behind statistically significant differences.

Molnar, Gyula I.↗

Comparison of TOMS and AVHRR imagery of volcanic clouds

This project consisted of merging volcanic cloud data from initial studies from TOMS and AVHRR sensors. Because of the different data acquisition platforms and processing algorithms and formats, the combination of the two data required special consideration. Once the registration technique was perfected, several eruptions with data from both sensors were studied. The conclusion showed that, from the initial work, the positions of the volcanic clouds as detected by the two sensors are very similar. Because the AVHRR algorithm is now capable of retrieving masses of ash and sulfate aerosol, the capability of comparing data from the two sensors now allows changes in SO2, H2SO4, and silicated ash in drifting clouds to be described. The results of this study culminated in several presentations, and a master's thesis, which in turn was converted into a paper submitted to the Journal of Geophysical Research. Abstracts of two presentations are attached.

Rose, William I.↗

Design and implementation of a prototype data system for earth radiation budget, cloud, aerosol, and chemistry data

Activities at the NASA Langley Research Center's distributed active archive centers (DAACs) intended to capitalize on existing centers of scientific expertise and to prevent a single point of failure are described. A Version 0 Langley DAAC, a prototype of an Earth Observing System Data and Information System, started archiving and distributing existing datasets on the earth's radiation budget, clouds, aerosols, and tropospheric chemistry in late 1992. The major goals of the LaRC Version 0 effort include to enhance scientific use of existing data; to develop institutional expertise in maintaining and distributing data; to encourage cooperative interagency and international involvement with datasets and research; and to use institutional capability for processing data from previous missions to prepare for processing the future EOS satellite data.

Baum, Bryan A.↗

The Effect of Aerosols and Clouds on the Retrieval of Infrared Sea Surface Temperatures

Comparisons are performed between spatially averaged sea surface temperatures (ASST2) as derived from the second Along-Track Scanning Radiometer (ATSR-2) on board the second European Remote Sensing Satellite (ERS-2) and the NOAA-NASA Advanced Very High Resolution Radiometer (AVHRR) Oceans Pathfinder dataset (MPFSST). Difference maps, MPFSST 2 ASST2, along with the application of a simple statistical regression model to aerosol and cloud data from the Total Ozone Mapping Spectrometer ( TOMS), are used to examine the impact of possible aerosol and cloud contamination. Differences varied regionally, but the largest biases were seen off western Africa. Nighttime and daytime differences off western Africa were reduced from -0.5degrees to -0.2degreesC and from -0.1degrees to 0degreesC, respectively. Significant cloud flagging, based on the model, occurred in the Indian Ocean, the equatorial Pacific, and in the vicinity of the Gulf Stream. Comparisons of the MPFSST and the ASST2 with in situ data from the 2002 version of the World Oceanic Database (WOD02) off western Africa show larger mean differences for the MPFSST. The smallest mean differences occurred for nighttime ASST2 - WOD02 with a value of 0.0degrees +/- 0.4degreesC.

clouds↗

Online anomaly detection for multi‐source VMware using a distributed streaming framework

Summary Anomaly detection refers to the identification of patterns in a dataset that do not conform to expected patterns. Such non‐conformant patterns typically correspond to samples of interest and are assigned to different labels in different domains, such as outliers, anomalies, exceptions, and malware. A daunting challenge is to detect anomalies in rapid voluminous streams of data. This paper presents a novel, generic real‐time distributed anomaly detection framework for multi‐source stream data. As a case study, we investigate anomaly detection for a multi‐source VMware‐based cloud data center, which maintains a large number of virtual machines (VMs). This framework continuously monitors VMware performance stream data related to CPU statistics (e.g., load and usage). It collects data simultaneously from all of the VMs connected to the network and notifies the resource manager to reschedule its CPU resources dynamically when it identifies any abnormal behavior from its collected data. A semi‐supervised clustering technique is used to build a model from benign training data only. During testing, if a data instance deviates significantly from the model, then it is flagged as an anomaly. Effective anomaly detection in this case demands a distributed framework with high throughput and low latency. Distributed streaming frameworks like Apache Storm, Apache Spark, S4, and others are designed for a lower data processing time and a higher throughput than standard centralized frameworks. We have experimentally compared the average processing latency of a tuple during clustering and prediction in both Spark and Storm and demonstrated that Spark processes a tuple much quicker than storm on average. Copyright © 2016 John Wiley & Sons, Ltd.

Solaimani, Mohiuddin↗

Tuning the Performance of a Computational Persistent Homology Package

In recent years, persistent homology has become an attractive method for data analysis. It captures topological features, such as connected components, holes, and voids from point cloud data and summarizes the way in which these features appear and disappear in a filtration sequence. In this project, we focus on improving the performanceof Eirene, a computational package for persistent homology. Eirene is a 5000-line open-source software library implemented in the dynamic programming language Julia. We use the Julia profiling tools to identify performance bottlenecks and develop novel methods to manage them, including the parallelization of some time-consuming functions on multicore/manycore hardware. Empirical results show that performance can be greatly improved.

Persistent Homology↗

Eyes On the Ground: Year 2 Assessment.

The goal of the Eyes On the Ground project is to develop tools to aid IAEA inspectors. Our original vision was to produce a tool that would take three-dimensional measurements of an unknown piece of equipment, construct a semantic representation of the measured object, and then use the resulting data to infer possible explanations of equipment function. We report our tests of a 3-d laser scanner to obtain 3-d point cloud data, and subsequent tests of software to convert the resulting point clouds into primitive geometric objects such as planes and cylinders. These tests successfully identified pipes of moderate diameter and planar surfaces, but also incurred significant noise. We also investigated the IAEA inspector task context, and learned that task constraints may present significant obstacles to using 3-d laser scanners. We further learned that equipment scale and enclosing cases may confound our original goal of equipment diagnosis. Meanwhile, we also surveyed the rapidly evolving field of 3-d measurement technology, and identified alternative sensor modalities that may prove more suitable for inspector use in a safeguards context. We conclude with a detailed discussion of lessons learned and the resulting implications for project goals. Approved for public release; further dissemination unlimited.

47 OTHER INSTRUMENTATION↗

Analysis Ready Data in Analytics Optimized Data Stores for Analysis of Big Earth Data in the Cloud

Cloud computing offers the possibility of making the analysis of Big Data approachable for a wider community due to affordable access to computing power, an ecosystem of usable tools for parallel processing, and migration of many large datasets to archives in the cloud, allowing data-proximal computing. Generally, data analysis acceleration in the cloud comes from running multiple nodes in a split-combine-apply strategy. Data systems such as the Earth Observing System Data and Information System are in a position to "pre-split" the data by storing them in a data store that is optimized for data parallel computing, i.e., an Analytics-Optimized Data Store (AODS). A variety of approaches to AODS are possible, from highly scalable databases to scalable filesystems to data formats optimized for cloud access (e.g., zarr and cloud-optimized datasets), with the optimal choice dependent on both the types of analysis and the geospatial structure of the data. A key question is how much preprocessing of the data to do, both before splitting and as the first part of the apply step. Again, the geospatial structure of the data and the analysis type influence the decision, with the added complexity of the user type. Trans-disciplinary users who are not well-versed in the nuances of quality-filtering and georeferencing of remote sensing orbit/swath/scene data tend to ask for more highly processed data, relying on the data provider to make sensible decisions on preprocessing parameters. (This accounts for the popularity of "Level 3" gridded data, despite the lower spatial resolution it provides.) In this case, data can be preprocessed before the split, resulting in higher performance in the rest of the "apply" step, which can be transformative for use cases such as interactive data exploration at scale. Discipline researchers who are experienced with remote sensing data often prefer more flexibility in customizing the preprocessing data into Analysis Ready Data, resulting in more need for on-the-fly preprocessing.

Lynnes, Christopher↗

Effect of aerosols and clouds on the retrieval of infrared sea surface temperatures

Comparisons are performed between spatially averaged sea surface temperatures (ASST2) as derived from the second Along-Track Scanning Radiometer (ATSR-2) on board the second European Remote Sensing Satellite (ERS-2) and the NOAA–NASA Advanced Very High Resolution Radiometer (AVHRR) Oceans Pathfinder dataset (MPFSST). Difference maps, MPFSST 2 ASST2, along with the application of a simple statistical regression model to aerosol and cloud data from the Total Ozone Mapping Spectrometer (TOMS), are used to examine the impact of possible aerosol and cloud contamination. Differences varied regionally, but the largest biases were seen off western Africa. Nighttime and daytime differences off western Africa were reduced from 20.58 to 20.28C and from 20.18 to 08C, respectively. Significant cloud flagging, based on the model, occurred in the Indian Ocean, the equatorial Pacific, and in the vicinity of the Gulf Stream. Comparisons of the MPFSST and the ASST2 with in situ data from the 2002 version of the World Oceanic Database (WOD02) off western Africa show larger mean differences for the MPFSST. The smallest mean differences occurred for nighttime ASST2 2 WOD02 with a value of 0.08 6 0.48C.

Armstrong, Edward M.↗

Machine learning for automated experimentation in scanning transmission electron microscopy

Abstract Machine learning (ML) has become critical for post-acquisition data analysis in (scanning) transmission electron microscopy, (S)TEM, imaging and spectroscopy. An emerging trend is the transition to real-time analysis and closed-loop microscope operation. The effective use of ML in electron microscopy now requires the development of strategies for microscopy-centric experiment workflow design and optimization. Here, we discuss the associated challenges with the transition to active ML, including sequential data analysis and out-of-distribution drift effects, the requirements for edge operation, local and cloud data storage, and theory in the loop operations. Specifically, we discuss the relative contributions of human scientists and ML agents in the ideation, orchestration, and execution of experimental workflows, as well as the need to develop universal hyper languages that can apply across multiple platforms. These considerations will collectively inform the operationalization of ML in next-generation experimentation.

36 MATERIALS SCIENCE↗

Radiation budget and related measurements in 1985 and beyond

Development of systems for obtaining radiation budget and cloud data is discussed. Instruments for measuring total solar irradiance, total infrared flux, reflected solar flux, and cloud heights and properties are considered. Other topics discussed include sampling by multiple satellites, user identification, and determination of the parameters that need to be measured.

Source record↗

A single field of view method for retrieving tropospheric temperature profiles from cloud-contaminated radiance data

The paper presents a method for retrieving single field of view tropospheric temperature profiles directly from cloud-contaminated radiance data through the use of auxiliary data such as observed shelter temperatures and estimated cloud-top height. A model was formulated to calculate cloud parameters for use with the radiative transport equation at an estimated cloud-top level. The cloud and temperature data are used in conjunction with real and simulated radiance data from NOAA satellites.

Hodges, D. B.↗

Automatic Point Cloud Building Envelope Segmentation (AutoCuBES)

The Auto-CuBES algorithm is based on unsupervised machine learning that automatically labels 3D point cloud data and reduces the time spent in manual segmentation. The algorithm can process high-resolution point clouds and generate a wire-frame building envelope model with a small set of calibration parameters. The algorithm inputs a 3D point cloud generated by commonly available surveying equipment and outputs a wire-frame model of the building envelope. Unsupervised machine learning methods were used to identify facades, windows, and doors while minimizing the number of calibration parameters.

Puente, BryanMaldonado↗

A Fourier approach to cloud motion estimation

A Fourier technique is described for estimating cloud motion from pairs of pictures using the phase of the cross spectral density. The method allows motion estimates to be made for individual spatial frequencies, which are related to cloud pattern dimensions. Results obtained are presented and compared with the results of a Fourier domain cross correlation scheme. Using both artificial and real cloud data show that the technique is relatively sensitive to the presence of mixtures of motions, changes in cloud shape, and edge effects.

Arking, A.↗