Chemistream: Big Data Materials on HPC Clouds
Explore the source record for details and available documents.
cloud computing↗
SEARCH · Search NASA
Results for “cloud computing, data cloud”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
System enhancements of Mesoscale Analysis and Space Sensor (MASS) computer system
The interactive information processing for the mesoscale analysis and space sensor (MASS) program is reported. The development and implementation of new spaceborne remote sensing technology to observe and measure atmospheric processes is described. The space measurements and conventional observational data are processed together to gain an improved understanding of the mesoscale structure and dynamical evolution of the atmosphere relative to cloud development and precipitation processes. A Research Computer System consisting of three primary computers was developed (HP-1000F, Perkin-Elmer 3250, and Harris/6) which provides a wide range of capabilities for processing and displaying interactively large volumes of remote sensing data. The development of a MASS data base management and analysis system on the HP-1000F computer and extending these capabilities by integration with the Perkin-Elmer and Harris/6 computers using the MSFC's Apple III microcomputer workstations is described. The objectives are: to design hardware enhancements for computer integration and to provide data conversion and transfer between machines.
Hickey, J. S.↗
Global Energy and Water Exchanges Program Surface Radiation Budget Release 4 Integrated Product: Algorithm Theoretical Basis Document
This document represents the Algorithm Theoretical Basis Document (ATBD) describing the NASA Langley Research Center's Surface Radiation budget (SRB) developed in collaboration with the World Research Climate Programme's Global Energy and Water Exchanges Program. The top of atmosphere and surface fluxes span from July 1, 1983 through June 30, 2017. The shortwave or solar wavelengths span the entire record. The thermal infrared or longwave fluxes span globally from January 1, 1988 to December 30, 2009. Before 1988, only the land-only fluxes are available. After 2009 only the ocean fluxes are available. This document describes all the inputs and radiative transfer algorithms used to generate the TOA and surface radiative fluxes. The fluxes include upward and downward, all-sky, clear-sky and pristine sky conditions. Surface net fluxes and cloud radiative effect are computed and presented as part of the analysis. Comparisons to other data products are made to assess the top-of-atmosphere and surface fluxes. Additionally, surface measurements are utilized to validation the surface fluxes. The surface validation analysis shows the the radiative fluxes are improved compared to the previous version. Data set cautions are also presented.
surface radiation budget↗
Mitigate: An Adaptive Network Data Anonymization Tool Using Condensation-Based Differential Privacy
Modern network devices collect a large amount of data that can be analyzed to identify bottlenecks, anomalies, cyber-attacks, etc. Therefore, there is often a need to analyze such collections of network data quite often by an external expert or by the research community. However, these collections of data contain sensitive, proprietary information. In order for the network data to be shared, it must first be anonymized. The overall objective of this project is to develop an innovative privacy management tool to anonymize network data and achieve sufficient privacy, acceptable data utility, and efficient data analysis at the same time. No existing anonymization methods can achieve all of these at the same time. The core of this technology is a differential private clustering algorithm that provides strong privacy protection, preserves data properties important for subsequent analysis, and allows the party receiving the anonymized data to conduct analysis directly on anonymized data without the need of decryption or any extra processing. The research carried out was to design, implement and verify a solution to this problem by completing the following tasks: 1) developing the core technology; 2) developing a context based method that automatically recommends fields that must be anonymized; 3) conducted experiments showing superior results using our approach compared to existing tools, and 4) developed an intuitive but basic user interface. The research that was conducted generated novel algorithmic techniques that utilize state-of-the-art methods such as condensation, differential privacy preservation, clustering, automated tuning based on contextual awareness, and recommendation techniques to specify columns to users for anonymization leading to optimal privacy that allows research analysis on the dataset. Experiments were conducted to evaluate the efficacy of these novel algorithmic techniques by performing analysis on original non-anonymized datasets, then conducting analysis on the same yet anonymized datasets and comparing the results of the analyses. Overall, the anonymized analysis results were within 1% of the original results, verifying that the generated technology not only guarantees a high level of privacy but also enables research analysis as if it were conducted on the original dataset. Potential applications of this technology include anonymization of any type of structured network datasets that contain sensitive identifiers, such as IP addresses, that can be used in multiple applications. For example, to create an AI or machine learning model for cyber security, e.g., to detect attacks, or for performance analysis, e.g., identify bottlenecks or predict performance. In addition, a market analysis that was conducted for potential applications of this technology identified a broader range of applications of our anonymization technology beyond the network sector that includes healthcare, banking, insurance, securities, finance (FISB), data brokering, cloud services, ad sales, and government.
97 MATHEMATICS AND COMPUTING↗
ARM Data-Oriented Metrics and Diagnostics Package for Climate Model Evaluation
A Python-based metrics and diagnostics package is currently being developed by the U.S. Department of Energy (DOE) Atmospheric Radiation Measurement (ARM) Infrastructure Team at Lawrence Livermore National Laboratory (LLNL) to facilitate the use of long-term, high-frequency measurements from the ARM Facility in evaluating the regional climate simulation of clouds, radiation, and precipitation. This metrics and diagnostics package computes climatological means of targeted climate model simulation and generates tables and plots for comparing the model simulation with ARM observational data. The Coupled Model Intercomparison Project (CMIP) model data sets are also included in the package to enable model intercomparison as demonstrated in Zhang et al. (2017). The mean of the CMIP model can serve as a reference for individual models. Basic performance metrics are computed to measure the accuracy of mean state and variability of climate models. The evaluated physical quantities include cloud fraction, temperature, relative humidity, cloud liquid water path, total column water vapor, precipitation, sensible and latent heat fluxes, and radiative fluxes, with plan to extend to more fields, such as aerosol and microphysics properties. Process-oriented diagnostics focusing on individual cloud- and precipitation-related phenomena are also being developed for the evaluation and development of specific model physical parameterizations. The version 1.0 package is designed based on data collected at ARM’s Southern Great Plains (SGP) Research Facility, with the plan to extend to other ARM sites. The metrics and diagnostics package is currently built upon standard Python libraries and additional Python packages developed by DOE (such as CDMS and CDAT). The ARM metrics and diagnostic package is available publicly with the hope that it can serve as an easy entry point for climate modelers to compare their models with ARM data. In this report, we first present the input data, which constitutes the core content of the metrics and diagnostics package in section 2, and a user's guide documenting the workflow/structure of the version 1.0 codes, and including step-by-step instruction for running the package in section 3.
54 ENVIRONMENTAL SCIENCES↗
Characterization and classification of vegetation canopy structure and distribution within the Great Smoky Mountains National Park using LiDAR
Vegetation canopy structure is a critically important habit characteristic for many threatened and endangered birds and other animal species, and it is key information needed by forest and wildlife managers for monitoring and managing forest resources, conservation planning and fostering biodiversity. Advances in Light Detection and Ranging (LiDAR) technologies have enabled remote sensing-based studies of vegetation canopies by capturing three-dimensional structures, yielding information not available in two-dimensional images of the landscape pro- vided by traditional multi-spectral remote sensing platforms. However, the large volume data sets produced by airborne LiDAR instruments pose a significant computational challenge, requiring algorithms to identify and analyze patterns of interest buried within LiDAR point clouds in a computationally efficient manner, utilizing state-of-art computing infrastructure. We developed and applied a computationally efficient approach to analyze a large volume of LiDAR data and to characterize and map the vegetation canopy structures for 139,859 hectares (540 sq. miles) in the Great Smoky Mountains National Park. This study helps improve our understanding of the distribution of vegetation and animal habitats in this extremely diverse ecosystem.
Clustering↗
Soil, water, and vegetation conditions in south Texas
The author has identified the following significant results. Software development for a computer-aided crop and soil survey system is nearing completion. Computer-aided variety classification accuracies using LANDSAT-1 MSS data for a 600 hectare citrus farm were 83% for Redblush grapefruit and 91% for oranges. These accuracies indicate that there is good potential for computer-aided inventories of grapefruit and orange citrus orchards with LANDSAT-type MSS data. Mean digital values of clouds differed statistically from those for crop, soil, and water entities, and those for cloud shadows were enough lower than sunlit crop and soil to be distinguishable. The standard errors of estimate for the calibration of computer compatible tape coordinate system (pixel and record) to earth coordinate system (longitude and latitude) for 6 LANDSAT scenes ranged from 0.72 to 1.50 pixels and from 0.58 to 1.75 records.
Wiegand, C. L.↗
The Atmospheric Channels of GLAS: Near Real-Time Global Lidar Remote Sensing of Clouds and Aerosols from Space
The Geoscience Laser Altimeter System (GLAS) will be placed into orbit in 2001 aboard the Ice, Cloud and Land Elevation Satellite (ICESat). From its nearly polar orbit (94 degree inclination), GLAS will provide continuous global measurements of the vertical distribution of clouds and aerosols while simultaneously providing high accuracy topographic profiling of surface features. During the mission, which is slated to last 3 to 5 years, the data collected by GLAS will be in near-real time to produce level 1 and 2 data products at the NASA GLAS Science Computing Facility (SCF) at Goddard Space Flight Center in Greenbelt, Maryland. The atmospheric products include cloud and aerosol layer heights, planetary boundary layer depth, polar stratospheric clouds and thin cloud and aerosol optical depth. These products will be made available to the science community within days of their creation. The processing algorithms must be robust, adaptive, efficient, and clever enough to run autonomously for the widely varying atmospheric conditions that will be encountered. This paper presents an overview of the GLAS atmospheric data products and briefly discusses the design of the processing algorithms.
Palm, Stephen P.↗
Automatic variable selection in ecological niche modeling: A case study using Cassin’s Sparrow (Peucaea cassinii)
MERRA/Max provides a feature selection approach to dimensionality reduction that enables direct use of global climate model outputs in ecological niche modeling. The system accomplishes this reduction through a Monte Carlo optimization in which many independent MaxEnt runs, operating on a species occurrence file and a small set of randomly selected variables in a large collection of variables, converge on an estimate of the top contributing predictors in the larger collection. These top predictors can be viewed as potential candidates in the variable selection step of the ecological niche modeling process. MERRA/Max’s Monte Carlo algorithm operates on files stored in the underlying filesystem, making it scalable to large data sets. Its software components can run as parallel processes in a high-performance cloud computing environment to yield near real-time performance. In tests using Cassin’s Sparrow (Peucaea cassinii) as the target species, MERRA/Max selected a set of predictors from Worldclim’s Bioclim collection of 19 environmental variables that have been shown to be important determinants of the species’ bioclimatic niche. It also selected biologically and ecologically plausible predictors from a more diverse set of 86 environmental variables derived from NASA’s Modern-Era Retrospective Analysis for Research and Applications Version 2 (MERRA-2) reanalysis, an output product of the Goddard Earth Observing System Version 5 (GEOS-5) modeling system. We believe these results point to a technological approach that could expand the use global climate model outputs in ecological niche modeling, foster exploratory experimentation with otherwise difficult-to-use climate data sets, streamline the modeling process, and, eventually, enable automated bioclimatic modeling as a practical, readily accessible, low-cost, commercial cloud service.
John L. Schnase↗
BisQue for 3D Materials Science in the Cloud: Microstructure–Property Linkages
Accelerating the design and development of new advanced materials is one of the priorities in modern materials science. These efforts are critically dependent on the development of comprehensive materials cyber-infrastructures which enable effcient data storage, management, sharing, and collaboration as well as integration of computational tools that help establish processing{structure{property relationships. In this contribution, we present implementation of such computational tools into a cloud-based platform called BisQue (Kvilekval et al., 2010). We first describe the current state of BisQue as an open-source platform for multidisciplinary research in the cloud and its potential for 3D materials science. We then demonstrate how new computational tools, primarily aimed at processing{structure{property relationships, can be implemented into the system. Speci cally, in this work, we develop a module for BisQue that enables microstructure-sensitive predictions of effective yield strength of two-phase materials. Towards this end, we present an implementation of a computationally effcient data-driven model into the BisQue platform. The new module is made available online and can be used from a web browser without any special software and with minimal computational requirements on the user end. The capabilities of the module for rapid property screening are demonstrated in case studies with two different methodologies based on datasets containing 3D microstructure information from (i) synthetic generation and (ii) sampling large 3D volumes obtained in experiments.
36 MATERIALS SCIENCE↗
Tuning the Performance of a Computational Persistent Homology Package
In recent years, persistent homology has become an attractive method for data analysis. It captures topological features, such as connected components, holes, and voids from point cloud data and summarizes the way in which these features appear and disappear in a filtration sequence. In this project, we focus on improving the performanceof Eirene, a computational package for persistent homology. Eirene is a 5000-line open-source software library implemented in the dynamic programming language Julia. We use the Julia profiling tools to identify performance bottlenecks and develop novel methods to manage them, including the parallelization of some time-consuming functions on multicore/manycore hardware. Empirical results show that performance can be greatly improved.
Persistent Homology↗
Reaching new peaks for the future of the CMS HTCondor Global Pool
The CMS experiment at CERN employs a distributed computing infrastructure to satisfy its data processing and simulation needs. The CMS Submission Infrastructure team manages a dynamic HTCondor pool, aggregating mainly Grid clusters worldwide, but also HPC, Cloud and opportunistic resources. This CMS Global Pool, which currently involves over 70 computing sites worldwide and peaks at 350k CPU cores, is employed to successfully manage the simultaneous execution of up to 150k tasks. While the present infrastructure is sufficient to harness the current computing power scales, CMS latest estimates predict a noticeable expansion in the amount of CPU that will be required in order to cope with the massive data increase of the High-Luminosity LHC (HL-LHC) era, planned to start in 2027. This contribution presents the latest results of the CMS Submission Infrastructure team in exploring and expanding the scalability reach of our Global Pool, in order to preventively detect and overcome any barriers in relation to the HL-LHC goals, while maintaining high effciency in our workload scheduling and resource utilization.
72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗
The Edge of Exploration: An Edge Storage and Computing Framework for Ambient Noise Seismic Interferometry Using Internet of Things Based Sensor Networks
Recent technological advances have reduced the complexity and cost of developing sensor networks for remote environmental monitoring. However, the challenges of acquiring, transmitting, storing, and processing remote environmental data remain significant. The transmission of large volumes of sensor data to a centralized location (i.e., the cloud) burdens network resources, introduces latency and jitter, and can ultimately impact user experience. Edge computing has emerged as a paradigm in which substantial storage and computing resources are located at the “edge” of the network. In this paper, we present an edge storage and computing framework leveraging commercially available components organized in a tiered architecture and arranged in a hub-and-spoke topology. The framework includes a popular distributed database to support the acquisition, transmission, storage, and processing of Internet-of-Things-based sensor network data in a field setting. We present details regarding the architecture, distributed database, embedded systems, and topology used to implement an edge-based solution. Lastly, a real-world case study (i.e., seismic) is presented that leverages the edge storage and computing framework to acquire, transmit, store, and process millions of samples of data per hour.
58 GEOSCIENCES↗
A 5G Enabled Adaptive Computing Workflow for Greener Power Grid
5G wireless technology can deliver higher data speeds, ultra low latency, more reliability, massive network capacity, increased availability, and a more uniform user experience to users. It brings additional power to help address the challenges brought by renewable integration and decarbonization. In this paper, a 5G enabled adaptive computing workflow tool has been presented that consists of various computing resources, such as 5G equipment, edge computing, cluster, Graphics processing unit (GPU) and cloud computing, with two examples showing technical feasibility for edge-grid-cloud interaction for real-time monitoring, security assessment, and forecasting. Benefiting from the high data transmission speed and massive connection capability of 5G, the workflow shows its potential to seamlessly integrate various applications at distributed and/or centralized locations to build more complex and powerful functions, with better flexibility.
5G technology, computational workflow, edge comput↗
Improved GEO Derived LW Broadband Fluxes for the Edition5 CERES SYN1Deg Data Product
The NASA Clouds and the Earth's Radiant Energy System (CERES) project provides over 24 years of accurately observed top-of-the-atmosphere (TOA) and surface flux data record for climate monitoring and diagnostic studies. The major level-3 data products include SSF1deg, SYN1deg, FluxByCldTyp, and EBAF data. The CERES SYN1deg product provides 1° gridded observed TOA fluxes and computed surface fluxes based on hourly GEO clouds and derived broadband TOA fluxes. The hourly GEO derived fluxes fill in the temporal gaps between the Terra and Aqua CERES observations. In order to maintain a climate quality dataset the GEO cloud and flux retrievals are consistent over the record. The CERES project is preparing for Edition 5 product providing an opportunity to improve the GEO flux retrieval algorithms. To achieve this, the GEO satellite channel radiances are calibrated against the Aqua-MODIS calibration reference. For SW the GEO channel radiances are then converted to broadband radiances using empirical and theoretical models. The SW broadband radiances are then converted to fluxes using the same CERES angular directional models. For LW the GEO channel radiances are directly converted to LW fluxes utilizing empirical models. The resulting GEO derived broadband fluxes are then normalized to the CERES fluxes regionally by regressing the instantaneous coincident flux pairs. The normalized GEO fluxes are then tied to the CERES instrument calibration, which is very stable over time. This study will attempt to improve the GEO derived LW fluxes based on machine learning technique for the future Edition5 SYN1deg data product.
Moguo Sun↗
Improving Satellite-Based Hotspot Detection Through Deep Learning-Enabled Smoke Recognition
While geostationary satellites, such as the GOES-R series, provide wildland fire hotspot readings at a high temporal resolution, they are prone to false negative readings and decreased confidence. One cause of decreased hotspot confidence is cloud contamination. Smoke produced from wildfire is often misinterpreted as cloud contamination, resulting in inaccurate and unsure sensor readings. To this end, we built a deep learning image segmentation model to identify smoke and cloud in true color satellite images. The model is pre-trained using self-supervised learning on over 10,000 GOES-R images to learn the underlying structure of satellite imagery. Then, the model is fine-tuned on a set of 130 labeled documents using supervised learning. The resulting model performs multi-class image segmentation with 85% accuracy and runs in under a minute on a standard personal computer. When paired alongside hotspot data, the model’s outputs can help increase confidence in wildfire location by identifying cases of cloud contamination that are due to smoke. The resulting model can be deployed in a stand-alone application or bundled in an Open Data Integration for wildland fire management (ODIN) application.
Earth observation↗
National Solar Radiation Database (NSRDB)
The National Solar Radiation Database (NSRDB) is a serially complete collection of meteorological and solar irradiance data sets for the United States and a growing list of international locations for 1998-2023. The NSRDB is updated annually and provides foundational information to support U.S. Department of Energy programs, research, industry and the general public. The NSRDB provides time-series data at 30-minute resolution of resource averaged over surface cells of 0.038 degrees in both latitude and longitude, or nominally 4 km in size. Additionally time series data at 5 minutes for the US and 10 minutes for North, Central and South America at 2 km resolution are produced from the next generation of GOES satellites and made available from 2019. The solar radiation values represent the resource available to solar energy systems. The data was created using cloud properties which are generated using the AVHRR Pathfinder Atmospheres-Extended (PATMOS-x) algorithms developed by the University of Wisconsin. Fast all-sky radiation model for solar applications (FARMS) in conjunction with the cloud properties, and aerosol optical depth (AOD) and precipitable water vapor (PWV) from ancillary source are used to estimate solar irradiance (GHI, DNI, and DHI). The Global Horizontal Irradiance (GHI) is computed for clear skies using the REST2 model. For cloud scenes identified by the cloud mask, FARMS is used to compute GHI and FARMS DNI is used to compute the Direct Normal Irradiance (DNI). The PATMOS-X model uses radiance images in visible and infrared channels from the Geostationary Operational Environmental Satellite (GOES) series of geostationary weather satellites. Ancillary variables needed to run REST2 and FARMS (e.g., aerosol optical depth, precipitable water vapor, and albedo) are derived from NASA's Modern Era-Retrospective Analysis (MERRA-2) dataset. Temperature and wind speed data are also derived from MERRA-2 and provided for use in NREL's System Advisor Model (SAM) to compute PV generation.
aerosol↗
Validation Ice Crystal Icing Engine Test in the Propulsion Systems Laboratory at NASA Glenn Research Center
The Propulsion Systems Laboratory (PSL) is an existing altitude simulation jet engine test facility located at NASA Glenn Research Center in Clevleand, OH. It was modified in 2012 with the integration of an ice crystal cloud generation system. This paper documents the inaugural ice crystal cloud test in PSLthe first ever full scale, high altitude ice crystal cloud turbofan engine test to be conducted in a ground based facility. The test article was a Lycoming ALF502-R5 high bypass turbofan engine, serial number LF01. The objectives of the test were to validate the PSL ice crystal cloud calibration and engine testing methodologies by demonstrating the capability to calibrate and duplicate known flight test events that occurred on the same LF01 engine and to generate engine data to support fundamental and computational research to investigate and better understand the physics of ice crystal icing in a turbofan engine environment while duplicating known revenue service events and conducting test points while varying facility and engine parameters. During PSL calibration testing it was discovered than heated probes installed through tunnel sidewalls experienced ice buildup aft of their location due to ice crystals impinging upon them, melting and running back. Filtered city water was used in the cloud generation nozzle system to provide ice crystal nucleation sites. This resulted in mineralization forming on flow path hardware that led to a chronic degradation of performance during the month long test. Lacking internal flow path cameras, the response of thermocouples along the flow path was interpreted as ice building up. Using this interpretation, a strong correlation between total water content (TWC) and a weaker correlation between median volumetric diameter (MVD) of the ice crystal cloud and the rate of ice buildup along the instrumented flow path was identified. For this test article the engine anti-ice system was required to be turned on before ice crystal icing would occur. The ice crystal icing event, an uncommanded reduction in thrust, was able to be turned on and off by manipulating cloud TWC. A flight test point where no ice crystal icing event occurred was also duplicated in PSL. Physics based computational tools were successfully used to predict tunnel settings to induce ice buildup along the low pressure compression system flow path for several test points at incrementally lower altitudes, demonstrating that development of ice crystal icing scaling laws is potentially feasible. Analysis of PSL test data showed that uncommanded reduction in thrust occurs during ice crystal cloud on operation prior to fan speed reduction. This supports previous findings that the reduction of thrust for this test article is due to ice buildup leading to a restricted airflow from either physical or aerodynamic blockage in the engine core flow path.
Ice Crystal Icing↗