Search NASA⌕ Search

SEARCH · Search NASA

Results for “cloud computing, data cloud”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 307 records · Page 17
All sourcesNASA NTRSDOE OSTI
NASA NTRSDate not supplied

TPSAS-NF1676L-11082-DND

Analysis of one decade of radar-lidar and Geostationary Operational Environmental Satellite (GOES) observations at the Atmospheric Radiation Measurement (ARM) Program Southern Great Plains (SGP) site reveals that there is excellent agreement in the long-term mean cloud fractions (CF) derived from the surface and GOES data, and the CF is independent of temporal resolution and spatial scales for grid boxes of size 0.5o to 2.5°. When computed over a 0.5-hr (4-hr) period, cloud frequency of occurrence (FREQ) and amount when present (AWP) derived from the point surface data agree very well with the same quantities determined from GOES for a 0.5o (2.5o) region centered on the ARM SGP site. The values of FREQ (AWP) derived from the radar-lidar observations at a given altitude increase (decrease) as the averaging period increases from 5 min to 6 hours. Similarly, CF at a given altitude increases as the vertical resolution increases from 90 to 1000 m. The profiles of CF have distinct bimodal vertical distributions with a lower peak between 1 and 2 km and a higher one between 8 and 11 km. The 10-year mean total CF, 46.9%, varies seasonally from a summer minimum of 39.8% to a maximum of 54.6% during the winter. The annual mean CF is 1-2% less than that from previous studies, ~48-49%, because fewer clouds occurred during 2005 and 2006, especially during winter. The differences in single- and multi-layered CFs between this study and an earlier analysis can be explained by the different temporal resolutions used in the two studies where single-layered CFs decrease but multilayered CFs increase from a 5-min resolution to a 1-hr resolution. The vertical distribution of nighttime GOES high-cloud tops agrees well with surface observations, but during daytime, fewer high clouds are retrieved by the GOES analysis than seen from the surface observations. The frequencies of occurrence for both daytime and nighttime GOES low-cloud tops are significantly higher than surface observations, but the cloud fractions are in good agreement.

B Xi↗
DOE OSTI2016

AP-Cloud: Adaptive Particle-in-Cloud method for optimal solutions to Vlasov–Poisson equation

We propose a new adaptive Particle-in-Cloud (AP-Cloud) method for obtaining optimal numerical solutions to the Vlasov–Poisson equation. Unlike the traditional particle-in-cell (PIC) method, which is commonly used for solving this problem, the AP-Cloud adaptively selects computational nodes or particles to deliver higher accuracy and efficiency when the particle distribution is highly non-uniform. Unlike other adaptive techniques for PIC, our method balances the errors in PDE discretization and Monte Carlo integration, and discretizes the differential operators using a generalized finite difference (GFD) method based on a weighted least square formulation. As a result, AP-Cloud is independent of the geometric shapes of computational domains and is free of artificial parameters. Efficient and robust implementation is achieved through an octree data structure with 2:1 balance. We analyze the accuracy and convergence order of AP-Cloud theoretically, and verify the method using an electrostatic problem of a particle beam with halo. Simulation results show that the AP-Cloud method is substantially more accurate and faster than the traditional PIC, and it is free of artificial forces that are typical for some adaptive PIC techniques.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗
DOE OSTI2020

Mathematics: The Tao of Data Science

The two pieces, "Ten Research Challenge Areas in Data Science" by Jeannette M. Wing and “Challenges and Opportunities in Statistics and Data Science: Ten Research Areas” by Xuming He and Xihong Lin, provide an impressively complete list of data science challenges from luminaries in the field of data science. They have done an extraordinary job, so this response offers a complementary viewpoint from a mathematical perspective and evangelizes advanced mathematics as a key tool for meeting the challenges they have laid out. Notably, we pick up the themes of scientific understanding of machine learning and deep learning, computational considerations such as cloud computing and scalability, balancing computational and statistical considerations, and inference with limited data. We propose that mathematics is an important key to establishing rigor in the field of data science and as such has an essential role to play in its future.

97 MATHEMATICS AND COMPUTING↗
NASA NTRS2011

Hyperspectral Remote Sensing of Atmosphere and Surface Properties

Atmospheric Infrared Sounder (AIRS), Infrared Atmospheric Sounding Interferometer (IASI), and Cross-track Infrared Sounder (CrIS) are all hyper-spectral satellite sensors with thousands of spectral channels. Top of atmospheric radiance spectra measured by these sensors contain high information content on atmospheric, cloud, and surface properties. Exploring high information content contained in these high spectral resolution spectra is a challenging task due to computation e ort involved in modeling thousands of spectral channels. Usually, only very small fractions (4{10 percent) of the available channels are included in physical retrieval systems or numerical weather forecast (NWP) satellite data assimilations. We will describe a method of simultaneously retrieving atmospheric temperature, moisture, cloud, and surface properties using all available spectral channels without sacrificing computational speed. The essence of the method is to convert channel radiance spectra into super-channels by an Empirical Orthogonal Function (EOF) transformation. Because the EOFs are orthogonal to each other, about 100 super-channels are adequate to capture the information content of the radiance spectra. A Principal Component-based Radiative Transfer Model (PCRTM) developed at NASA Langley Research Center is used to calculate both the super-channel magnitudes and derivatives with respect to atmospheric profiles and other properties. There is no need to perform EOF transformations to convert super channels back to spectral space at each iteration step for a one-dimensional variational retrieval or a NWP data assimilation system. The PCRTM forward model is also capable of calculating radiative contributions due to multiple-layer clouds. The multiple scattering effects of the clouds are efficiently parameterized. A physical retrieval algorithm then performs an inversion of atmospheric, cloud, and surface properties in super channel domain directly therefore both reducing the computational need and preserving the information content of the IASI measurements. The inversion algorithm is based on a non-linear Levenberg-Marquardt method with climatology covariance matrices and a priori information as constraints. One advantage of this approach is that it uses all information content from the hyper-spectral data so that the retrieval is less sensitive to instrument noise and eliminates the need for selecting a sub-set of the channels.

Liu, Xu↗
NASA NTRS2011

Exploring the Universe with WISE and Cloud Computing

WISE is a recently-completed astronomical survey mission that has imaged the entire sky in four infrared wavelength bands. The large quantity of science images returned consists of 2,776,922 individual snapshots in various locations in each band which, along with ancillary data, totals around 110TB of raw, uncompressed data. Making the most use of this data requires advanced computing resources. I will discuss some initial attempts in the use of cloud computing to make this large problem tractable.

Benford, Dominic J.↗
DOE OSTI2011

The Magellan Final Report on Cloud Computing

The goal of Magellan, a project funded through the U.S. Department of Energy (DOE) Office of Advanced Scientific Computing Research (ASCR), was to investigate the potential role of cloud computing in addressing the computing needs for the DOE Office of Science (SC), particularly related to serving the needs of mid- range computing and future data-intensive computing workloads. A set of research questions was formed to probe various aspects of cloud computing from performance, usability, and cost. To address these questions, a distributed testbed infrastructure was deployed at the Argonne Leadership Computing Facility (ALCF) and the National Energy Research Scientific Computing Center (NERSC). The testbed was designed to be flexible and capable enough to explore a variety of computing models and hardware design points in order to understand the impact for various scientific applications. During the project, the testbed also served as a valuable resource to application scientists. Applications from a diverse set of projects such as MG-RAST (a metagenomics analysis server), the Joint Genome Institute, the STAR experiment at the Relativistic Heavy Ion Collider, and the Laser Interferometer Gravitational Wave Observatory (LIGO), were used by the Magellan project for benchmarking within the cloud, but the project teams were also able to accomplish important production science utilizing the Magellan cloud resources.

97 MATHEMATICS AND COMPUTING↗
NASA NTRSDate not supplied

UASs in the VOG/Edge/FOG Sensor Web Environment

This paper will describe the evolution of information collection, derivation and delivery mechanisms in sensor webs utilizing Uninhabited Aerial Systems (UAS).We will discuss the movement to "Fog Computing", also known as Edge Computing. Fog Computing facilitates the distribution of common operations and networking between edge devices and cloud computing facilities, optimizing the production of actionable intelligence. Initially, UASs utilized onboard data collection as standard, with minimal data downloaded directly. Information products were derived in conventional computational environments, generally desk top computers, and information products made available to the Science Community in weeks or months. With the increased availability, and increasingly lower costs, of beyond line of sight (BLOS) satellite based communication, transmission rates and data volumes increased, and processing migrated to Cloud based services. Contemporary UASs are moving some of that information product derivation to on vehicle services, and are creating a distributed Cloud/Fog environment. The Author will describe the technological advances that have made this possible, including low power multi-core Central Processing Units (CPU), and, more recently, the availability of high end Graphical Processing Units (GPU) that consume only a few watts. Intelligent system software, leveraging these hardware advances, finally allows for information product generation on-board, rather than simple data collection. Additionally, intelligent flight control systems now support mutual vehicle to vehicle collaboration, allowing UASs to create ad-hoc sensor webs on demand, as required. Also discussed will be the lessons learned by the Authors' development of data systems for NASA's large High Altitude Long Endurance (HALE) UASs like Predator and Global Hawk, and how those lessons are being applied to other UAS development This paper will focus on applications, rather a deep dive into the technology, and will highlight improving data management through these new technologies.

UAS↗
DOE OSTI2022

Time Sequence Machine Learning-Based Data Intrusion Detection for Smart Voltage Source Converter-Enabled Power Grid

Smart inverters of distributed energy resources can enable cloud computing, condition monitoring, result visualization, remote control, and peer-to-peer energy trading in advanced power systems. However, the advent of data injection attacks in the communication architecture can alter measurement characteristics of power grids and have devastating consequences. In this article, we propose a time sequence machine learning-based anomaly detection methodology for detecting cyber intrusion into control signal setpoints and dc voltage signal measurement bias of the voltage source converter (VSC) in wind generators. We first investigated the effects of four types of denial of service, tampering signal, and stealthy-type data intrusion attacks on smart VSCs and overall wind farms. We then proposed a novel time sequence machine learning-based intrusion detection framework that can be implemented to detect different cyberattacks in the VSCs. The performance of the proposed framework has been compared with that of autoencoder and clustering-based intrusion detection framework. The proposed framework was validated by using the IEEE 39 bus power system in the presence of four wind farms in different locations. Using several metrics for intrusion detection performance, we validated the effectiveness of the proposed framework.

42 ENGINEERING↗
DOE OSTI2015

Progress on the FabrIc for Frontier Experiments project at Fermilab

The FabrIc for Frontier Experiments (FIFE) project is an ambitious, major-impact initiative within the Fermilab Scientific Computing Division designed to lead the computing model for Fermilab experiments. FIFE is a collaborative effort between experimenters and computing professionals to design and develop integrated computing models for experiments of varying needs and infrastructure. The major focus of the FIFE project is the development, deployment, and integration of Open Science Grid solutions for high throughput computing, data management, database access and collaboration within experiment. To accomplish this goal, FIFE has developed workflows that utilize Open Science Grid sites along with dedicated and commercial cloud resources. The FIFE project has made significant progress integrating into experiment computing operations several services including new job submission services, software and reference data distribution through CVMFS repositories, flexible data transfer client, and access to opportunistic resources on the Open Science Grid. Hence, the progress with current experiments and plans for expansion with additional projects will be discussed. FIFE has taken a leading role in the definition of the computing model for Fermilab experiments, aided in the design of computing for experiments beyond Fermilab, and will continue to define the future direction of high throughput computing for future physics experiments worldwide

97 MATHEMATICS AND COMPUTING↗
NASA NTRS2001

GOES-Microburst Products Performance Analysis In The Cape Canaveral And Kennedy Space Center Areas

In the past few years, the capabilities of NOAA Geostationary Operational Environmental Satellites (GOES) have increased dramatically. Hourly vertical sounder data is now generally available, but may be unavailable depending upon cloud conditions, satellite operations, and computer system problems at NOAA's National Environmental Satellite Display and Information Service (NESDIS). Meteorologists at NESDIS have used vertical sounder data to develop experimental products for forecasting the probability of convective downbursts. The two products of interest are the Microburst Day Predictive Index (MDPI), which provides an indication of microburst potential and the WINDEX which is a forecast of maximum winds assuming a microburst does occur. Data analyses were made for the central Florida convective season, that is, the period beginning May I and ending September 30. The MDPI showed significant potential as an aid in forecasting convective downbursts. MDPI calculated from GOES soundings were well correlated with those calculated from Cape Canaveral RAOBs.

Witiw, Michael R.↗
NASA NTRS2002

GOES-Microburst Products Performance Analysis in the Cape Canaveral and Kennedy Space Center Areas

In the past few years, the capabilities of NOAA Geostationary Operational Environmental Satellites (GOES) have increased dramatically. Hourly vertical sounder data is now generally available, but may be unavailable depending upon cloud conditions, satellite operations, and computer system problems at NOAA#s National Environmental Satellite Display and Information Service (NESDIS). Meteorologists at NESDIS have used vertical sounder data to develop experimental products for forecasting the probability of convective downbursts. The two products of interest are the Microburst Day Predictive Index (MDPI), which provides an indication of microburst potential and the WINDEX which is a forecast of maximum winds assuming a microburst does occur. Data analyses were made for the central Florida convective season, that is, the period beginning May 1 and ending September 30. The MDPI showed significant potential as an aid in forecasting convective downbursts. MDPI calculated from GOES soundings were well correlated with those calculated from Cape Canaveral RAOBs.

Witiw, Michael R.↗
NASA NTRS2010

Detection of Multi-Layer and Vertically-Extended Clouds Using A-Train Sensors

The detection of mUltiple cloud layers using satellite observations is important for retrieval algorithms as well as climate applications. In this paper, we describe a relatively simple algorithm to detect multiple cloud layers and distinguish them from vertically-extended clouds. The algorithm can be applied to coincident passive sensors that derive both cloud-top pressure from the thermal infrared observations and an estimate of solar photon pathlength from UV, visible, or near-IR measurements. Here, we use data from the A-train afternoon constellation of satellites: cloud-top pressure, cloud optical thickness, the multi-layer flag from the Aqua MODerate-resolution Imaging Spectroradiometer (MODIS) and the optical centroid cloud pressure from the Aura Ozone Monitoring Instrument (OMI). For the first time, we use data from the CloudSat radar to evaluate the results of a multi-layer cloud detection scheme. The cloud classification algorithms applied with different passive sensor configurations compare well with each other as well as with data from CloudSat. We compute monthly mean fractions of pixels containing multi-layer and vertically-extended clouds for January and July 2007 at the OMI spatial resolution (l2kmx24km at nadir) and at the 5kmx5km MODIS resolution used for infrared cloud retrievals. There are seasonal variations in the spatial distribution of the different cloud types. The fraction of cloudy pixels containing distinct multi-layer cloud is a strong function of the pixel size. Globally averaged, these fractions are approximately 20% and 10% for OMI and MODIS, respectively. These fractions may be significantly higher or lower depending upon location. There is a much smaller resolution dependence for fractions of pixels containing vertically-extended clouds (approx.20% for OMI and slightly less for MODIS globally), suggesting larger spatial scales for these clouds. We also find higher fractions of vertically-extended clouds over land as compared with ocean, particularly in the tropics and summer hemisphere.

Joiner, J.↗
NASA NTRS2006

Estimating Random Errors Due to Shot Noise in Backscatter Lidar Observations

In this paper, we discuss the estimation of random errors due to shot noise in backscatter lidar observations that use either photomultiplier tube (PMT) or avalanche photodiode (APD) detectors. The statistical characteristics of photodetection are reviewed, and photon count distributions of solar background signals and laser backscatter signals are examined using airborne lidar observations at 532 nm using a photon-counting mode APD. Both distributions appear to be Poisson, indicating that the arrival at the photodetector of photons for these signals is a Poisson stochastic process. For Poisson-distributed signals, a proportional, one-to-one relationship is known to exist between the mean of a distribution and its variance. Although the multiplied photocurrent no longer follows a strict Poisson distribution in analog-mode APD and PMT detectors, the proportionality still exists between the mean and the variance of the multiplied photocurrent. We make use of this relationship by introducing the noise scale factor (NSF), which quantifies the constant of proportionality that exists between the root-mean-square of the random noise in a measurement and the square root of the mean signal. Using the NSF to estimate random errors in lidar measurements due to shot noise provides a significant advantage over the conventional error estimation techniques, in that with the NSF uncertainties can be reliably calculated from/for a single data sample. Methods for evaluating the NSF are presented. Algorithms to compute the NSF are developed for the Cloud-Aerosol Lidar and Infrared Pathfinder Satellite Observations (CALIPSO) lidar and tested using data from the Lidar In-space Technology Experiment (LITE). OCIS Codes:

Liu, Zhaoyan↗
DOE OSTI2016

MWR3C physical retrievals of precipitable water vapor and cloud liquid water path

The data set contains physical retrievals of PWV and cloud LWP retrieved from MWR3C measurements during the MAGIC campaign. Additional data used in the retrieval process include radiosondes and ceilometer. The retrieval is based on an optimal estimation technique that starts from a first guess and iteratively repeats the forward model calculations until a predefined convergence criterion is satisfied. The first guess is a vector of [PWV,LWP] from the neural network retrieval fields in the netcdf file. When convergence is achieved the 'a posteriori' covariance is computed and its square root is expressed in the file as the retrieval 1-sigma uncertainty. The closest radiosonde profile is used for the radiative transfer calculations and ceilometer data are used to constrain the cloud base height. The RMS error between the brightness temperatures is computed at the last iterations as a consistency check and is written in the last column of the output file.

54 ENVIRONMENTAL SCIENCES↗
NASA NTRS2007

Validation of GOES-10 Satellite-derived Cloud and Radiative Properties for the MASRAD ARM Mobile Facility Deployment

This poster presentation reviews the process used to validate the GOES-10 satellite derived cloud and radiative properties. The ARM Mobile Facility (AMF) deployment at Pt Reyes, CA as part of the Marine Stratus Radiation Aerosol and Drizzle experiment (MASRAD), 14 March - 14 September 2005 provided an excellent chance to validate satellite cloud-property retrievals with the AMF's flexible suite of ground-based remote sensing instruments. For this comparison, NASA LaRC GOES10 satellite retrievals covering this region and period were re-processed using an updated version of the Visible Infrared Solar-Infrared Split-Window Technique (VISST), which uses data taken at 4 wavelengths (0.65, 3.9,11 and 12 m resolution), and computes broadband fluxes using improved CERES (Clouds and Earth's Radiant Energy System)-GOES-10 narrowband-to-broadband flux conversion coefficients. To validate MASRAD GOES-10 satellite-derived cloud property data, VISST-derived cloud amounts, heights, liquid water paths are compared with similar quantities derived from available ARM ground-based instrumentation and with CERES fluxes from Terra.

Khaiyer, M. M.↗
DOE OSTI2018

Future Online Analysis Platform Workshop Report

Recent technology trends, such as the increasing focus on data management technologies and the emergence of sustainable on-demand computing and commercial cloud facilities provide initial steps and potential building blocks for creating such a compute facility. How do we fill the gap between what we have now and the capabilities we need to make the vision of an online data facility a reality? What research challenges need to be addressed – and can be addressed – in the coming 5-10 years? How should we adjust our vision if they are not solved? Is such a facility compatible with the existing and evolving exascale resources and/or how should they evolve to be compatible? How do “beyond Moore’s law” technologies enable advanced data analysis? Those are the challeges that the Future Platform workshop was organized to address. This report is organized as follows. Section 2 summarizes data challenges and patterns related to the exemplar use cases presented during the first half day of the workshop. Section 3 summarizes the requirements presented by those use cases as well as a synthesis of those requirements into a Future Platform definition achieved on the 3rd half day of the workshop (morning of the 2nd day). Section 4 includes area breakout reports from the areas on the 2nd half day of the workshop (afternoon of the 1st day). Finally Section 5 has a report on recommendation for building methodology and metrics. The report concludes in Section 6.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2022

Development of an end state vision to implement digital monitoring in nuclear plants

Transitioning from an onsite Maintenance & Diagnostics Center to cloud-based services offers many new opportunities with computing power and storage, but also new challenges in terms of networking and security. This report will cover everything required for that transition including data processing and uploading to cloud services, feature selection, model creation, and result visualization for decision making. Although there are several other cloud-based services (e.g. Amazon Web Services and Google Cloud), this report explores Microsoft Azure to simplify nomenclature and maintain a consistent focus. Many of the services offered by Microsoft Azure are also available in the other cloud-based services, and their differences have been recorded in other literature. The Azure services most important to a nuclear power plant including networking & security, storage & databases, and Artificial Intelligence (AI) are reviewed here. Networking covers all aspects related to communication to Azure resources including security, privacy, and redundancy. Storage & databases includes data storage, upgrading, patching, backups, and monitoring. The AI services allows the user access to the machine learning (ML) techniques developed with Azure including automated ML, anomaly detection, computer vision, and natural language processing. This report summaries the features, capabilities, and challenges when using cloud-based services in a user-friendly manner.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗
DOE OSTI2016

A hybrid computational strategy to address WGS variant analysis in >5000 samples

Abstract Background The decreasing costs of sequencing are driving the need for cost effective and real time variant calling of whole genome sequencing data. The scale of these projects are far beyond the capacity of typical computing resources available with most research labs. Other infrastructures like the cloud AWS environment and supercomputers also have limitations due to which large scale joint variant calling becomes infeasible, and infrastructure specific variant calling strategies either fail to scale up to large datasets or abandon joint calling strategies. Results We present a high throughput framework including multiple variant callers for single nucleotide variant (SNV) calling, which leverages hybrid computing infrastructure consisting of cloud AWS, supercomputers and local high performance computing infrastructures. We present a novel binning approach for large scale joint variant calling and imputation which can scale up to over 10,000 samples while producing SNV callsets with high sensitivity and specificity. As a proof of principle, we present results of analysis on Cohorts for Heart And Aging Research in Genomic Epidemiology (CHARGE) WGS freeze 3 dataset in which joint calling, imputation and phasing of over 5300 whole genome samples was produced in under 6 weeks using four state-of-the-art callers. The callers used were SNPTools, GATK-HaplotypeCaller, GATK-UnifiedGenotyper and GotCloud. We used Amazon AWS, a 4000-core in-house cluster at Baylor College of Medicine, IBM power PC Blue BioU at Rice and Rhea at Oak Ridge National Laboratory (ORNL) for the computation. AWS was used for joint calling of 180 TB of BAM files, and ORNL and Rice supercomputers were used for the imputation and phasing step. All other steps were carried out on the local compute cluster. The entire operation used 5.2 million core hours and only transferred a total of 6 TB of data across the platforms. Conclusions Even with increasing sizes of whole genome datasets, ensemble joint calling of SNVs for low coverage data can be accomplished in a scalable, cost effective and fast manner by using heterogeneous computing platforms without compromising on the quality of variants.

59 BASIC BIOLOGICAL SCIENCES↗