Search NASA⌕ Search

SEARCH · Search NASA

Results for “cloud database”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 55 records · Page 3

Evolving the Land Information System into a Cloud Computing Service

The Land Information System (LIS) was developed to use advanced flexible land surface modeling and data assimilation frameworks to integrate extremely large satellite- and ground-based observations with advanced land surface models to produce continuous high-resolution fields of land surface states and fluxes. The resulting fields are extremely useful for drought and flood assessment, agricultural planning, disaster management, weather and climate forecasting, water resources assessment, and the like. We envisioned transforming the LIS modeling system into a scientific cloud computing-aware web and data service that would allow clients to easily setup and configure for use in addressing large water management issues. The focus of this Phase 1 project was to determine the scientific, technical, commercial merit and feasibility of the proposed LIS-cloud innovations that are currently barriers to broad LIS applicability. We (a) quantified the barriers to broad LIS utility and commercialization (high performance computing, big data, user interface, and licensing issues); (b) designed the proposed LIS-cloud web service, model-data interface, database services, and user interfaces; (c) constructed a prototype LIS user interface including abstractions for simulation control, visualization, and data interaction, (d) used the prototype to conduct a market analysis and survey to determine potential market size and competition, (e) identified LIS software licensing and copyright limitations and developed solutions, and (f) developed a business plan for development and marketing of the LIS-cloud innovation. While some significant feasibility issues were found in the LIS licensing, overall a high degree of LIS-cloud technical feasibility was found.

54 ENVIRONMENTAL SCIENCES↗

Automated data processing architecture for the Gemini Planet Imager Exoplanet Survey

The Gemini Planet Imager Exoplanet Survey (GPIES) is a multiyear direct imaging survey of 600 stars to discover and characterize young Jovian exoplanets and their environments. We have developed an automated data architecture to process and index all data related to the survey uniformly. An automated and flexible data processing framework, which we term the Data Cruncher, combines multiple data reduction pipelines (DRPs) together to process all spectroscopic, polarimetric, and calibration data taken with GPIES. With no human intervention, fully reduced and calibrated data products are available less than an hour after the data are taken to expedite follow up on potential objects of interest. The Data Cruncher can run on a supercomputer to reprocess all GPIES data in a single day as improvements are made to our DRPs. A backend MySQL database indexes all files, which are synced to the cloud, and a front-end web server allows for easy browsing of all files associated with GPIES. To help observers, quicklook displays show reduced data as they are processed in real time, and chatbots on Slack post observing information as well as reduced data products. Together, the GPIES automated data processing architecture reduces our workload, provides real-time data reduction, optimizes our observing strategy, and maintains a homogeneously reduced dataset to study planet occurrence and instrument performance.

79 ASTRONOMY AND ASTROPHYSICS↗

Clear-Sky Probability for the 2023 Annular Solar Eclipse and the 2024 Total Solar Eclipse Using the NREL National Solar Radiation Database

The National Renewable Energy Laboratory (NREL) and collaborators have created a clear-sky probability analysis to help guide viewers to the 2023 annular solar eclipse and the 2024 total solar eclipse. Using cloud and solar data from the NREL National Solar Radiation Database (NSRDB), the analysis provides cloudless-sky probabilities specific to the date, time, and location of each eclipse with a 4-km resolution. Though not intended to be an eclipse weather forecast, the detailed maps can help guide eclipse enthusiasts to likely optimal viewing locations. Additionally, high-resolution data are presented for the centerline of the path of each eclipse, representing the likelihood for cloudless skies and atmospheric clarity. The NSRDB provides industry, academia, and other stakeholders with high-resolution solar irradiance data to support feasibility analyses for photovoltaic and concentrating solar power generation projects.

14 SOLAR ENERGY↗

Virtual machine provisioning, code management, and data movement design for the Fermilab HEPCloud Facility

The Fermilab HEPCloud Facility Project has as its goal to extend the current Fermilab facility interface to provide transparent access to disparate resources including commercial and community clouds, grid federations, and HPC centers. This facility enables experiments to perform the full spectrum of computing tasks, including data-intensive simulation and reconstruction. We have evaluated the use of the commercial cloud to provide elasticity to respond to peaks of demand without overprovisioning local resources. Full scale data-intensive workflows have been successfully completed on Amazon Web Services for two High Energy Physics Experiments, CMS and NOνA, at the scale of 58000 simultaneous cores. This paper describes the significant improvements that were made to the virtual machine provisioning system, code caching system, and data movement system to accomplish this work. The virtual image provisioning and contextualization service was extended to multiple AWS regions, and to support experiment-specific data configurations. A prototype Decision Engine was written to determine the optimal availability zone and instance type to run on, minimizing cost and job interruptions. We have deployed a scalable on-demand caching service to deliver code and database information to jobs running on the commercial cloud. It uses the frontiersquid server and CERN VM File System (CVMFS) clients on EC2 instances and utilizes various services provided by AWS to build the infrastructure (stack). We discuss the architecture and load testing benchmarks on the squid servers. We also describe various approaches that were evaluated to transport experimental data to and from the cloud, and the optimal solutions that were used for the bulk of the data transport. Finally, we summarize lessons learned from this scale test, and our future plans to expand and improve the Fermilab HEP Cloud Facility.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

Clear-Sky Probability for the August 21, 2017, Total Solar Eclipse Using the NREL National Solar Radiation Database

The National Renewable Energy Laboratory (NREL) and collaborators have created a clear-sky probability analysis to help guide viewers of the August 21, 2017, total solar eclipse, the first continent-spanning eclipse in nearly 100 years in the United States. Using cloud and solar data from NREL's National Solar Radiation Database (NSRDB), the analysis provides cloudless sky probabilities specific to the date and time of the eclipse. Although this paper is not intended to be an eclipse weather forecast, the detailed maps can help guide eclipse enthusiasts to likely optimal viewing locations. Additionally, high-resolution data are presented for the centerline of the path of totality, representing the likelihood for cloudless skies and atmospheric clarity. The NSRDB provides industry, academia, and other stakeholders with high-resolution solar irradiance data to support feasibility analyses for photovoltaic and concentrating solar power generation projects.

14 SOLAR ENERGY↗

The Star-formation Law in Galactic High-mass Star-forming Molecular Clouds

We study the star-formation (SF) law in 12 Galactic molecular clouds with ongoing high-mass star-formation (HMSF) activity, as traced by the presence of a bright IRAS source and other HMSF tracers. We define the molecular cloud (MC) associated with each IRAS source using {sup 13}CO line emission, and count the young stellar objects (YSOs) within these clouds using GLIMPSE and MIPSGAL 24 μm Spitzer databases. The masses for high-luminosity YSOs (L {sub bol} > 10 L {sub ⊙}) are determined individually using pre-main-sequence evolutionary tracks and the evolutionary stages of the sources, whereas a mean mass of 0.5 M {sub ⊙} was adopted to determine the masses in the low-luminosity YSO population. The star-formation rate surface density (Σ{sub SFR}) corresponding to a gas surface density (Σ{sub gas}) in each MC is obtained by counting the number of the YSOs within successive contours of {sup 13}CO line emission. We find a break in the relation between Σ{sub SFR} and Σ{sub gas}, with the relation being a power law (Σ{sub SFR}∝Σ{sub gas}{sup N}) with the index N varying between 1.4 and 3.6 above the break. The Σ{sub gas} at the break is between 150–360 M {sub ⊙} pc{sup −2} for the sample clouds, which compares well with the threshold gas density found in recent studies of Galactic star-forming regions. Our clouds treated as a whole lie between the Kennicutt relation and the linear relation for Galactic and extra-galactic dense star-forming regions. We find a tendency for the high-mass YSOs to be found preferentially in dense regions at densities higher than 1200 M {sub ⊙} pc{sup −2} (∼0.25 g cm{sup −2}).

79 ASTRONOMY AND ASTROPHYSICS↗

Workflow-based automatic processing for Internet of Floating Things crowdsourced data

Data from sensors incorporated into mobile devices, such as networked navigational sensors, can be used to capture detailed environmental information. We describe here a workflow and framework for using sensors on boats to construct unique new datasets of underwater topography (bathymetry). Starting with a large number of measurements of position, depth, etc., obtained from such an Internet of Floating Things, we illustrate how, with a specialized protocol, data can be communicated to cloud resources, even when using delayed, intermittent, or disconnected networks. We then propose a method for automatic sensor calibration based on a novel reputation approach. Sampled depth data are interpolated efficiently on a cloud computing platform in order to provide a continuously updated bathymetric database. Our prototype implementation uses the FACE-IT Galaxy workflow engine to manage network communication and exploits the computational power of GPGPUs in a virtualized cloud environment, working with a CUDA-parallel algorithm, for efficient data processing. We report on an initial evaluation involving data from a sailing vessel in Italian coastal waters.

97 MATHEMATICS AND COMPUTING↗

Deep Convection Initiation, Growth, and Environments in the Complex Terrain of Central Argentina during CACTI

This study characterizes the wide range of deep convective cloud life cycles and their relationships with ambient environments observed during the Cloud, Aerosol, and Complex Terrain Interactions (CACTI) field campaign near the Sierras de Córdoba (SDC) range in central Argentina. We develop a novel convective cell tracking database for the entire field campaign using C-band polarimetric radar observations. The cell tracking database includes timing, location, area, depth, merge/split information, microphysical properties, collocated satellite-retrieved cloud properties, and sounding-derived environmental conditions. Results show that the SDC exerts a strong control on convection initiation (CI) and growth. CI preferentially occurs east of the SDC ridge during the afternoon, and cells often undergo upscale growth through the evening as they travel eastward toward the plains. Larger and more intense cells tend to occur in more unstable and humid low-level environments, and surface-based cells are stronger than elevated cells. Midtropospheric relative humidity and vertical wind shear also jointly affect the size and depth of the cells. Rapid cell area growth rates exhibit dependence on both large environmental wind shear and low-level moisture. Evolution of convective cell macro- and microphysical properties are strongly influenced by convective available potential energy and low-level humidity, as well as the presence of other cells in their vicinity. This cell tracking database demonstrates a framework that ties measurements from various platforms centering around convective life cycles to facilitate process understanding of factors that control convective evolution.

54 ENVIRONMENTAL SCIENCES↗

The efficacy of aerosol–cloud radiative perturbations from near-surface emissions in deep open-cell stratocumuli

Aerosol–cloud radiative effects are determined and quantified in simulationsof deep open-cell stratocumuli observed during the VAMOS Ocean-Cloud-Atmosphere-LandStudy Regional Experiment (VOCALS-REx) campaign offthe west coast of Chile. The cloud deck forms in a boundarylayer 1.5 km deep, with cell sizes reaching 50 km in diameter. Global databases of shiptracks suggest that these linear structures are seldom found in boundarylayers this deep. Here, we quantify the changes in cloud radiative propertiesto a continuous aerosol point source moving along a fixed emission linereleasing 10 17 particles per second. We show that a spatially coherentcloud perturbation is not evident along the emission line. Yet our modelsimulates an increase in domain-mean all-sky albedo of 0.05, corresponding toa diurnally averaged cloud radiative effect of 20 W m -2 , given theannual mean solar insolation at the VOCALS-REx site. Therefore, markedchanges in cloud radiative properties in precipitating deep open cells may bedriven by anthropogenic near-surface aerosol perturbations, such as thosegenerated by ships.Furthermore, we demonstrate that these changes in cloud radiative propertiesare masked by the naturally occurring variability within the organised cloudfield. A clear detection and attribution of cloud radiative effects to aperturbation in aerosol concentrations becomes possible when sub-filtering ofthe cloud field is applied, using the spatio-temporal distribution of theaerosol perturbation. Therefore, this work has implications for the detectionand attribution of effective cloud radiative forcing in marine stratocumuli,which constitutes one of the major physical uncertainties within the climatesystem. Our results suggest that ships may sometimes have a substantialradiative effect on marine clouds and albedo, even when ship tracks are notreadily visible.

54 ENVIRONMENTAL SCIENCES↗

Datum: A Scientific Metadata Catalog

The data catalog market is currently flooded with a myriad of different products, but none serve the scientific community well. There are cloud-native tools like Databricks, Snowflake,to on-premise solutions like Collibra and Datahub. The common failing of all these tools however, is their inability to serve the scientific data community directly. Most catalogs are targeted towards financial, health, or user data - not sensor or scientific domain data. They also prioritize integrations that often don’t exist or are just starting to be used in the scientific realm - all while ignoring common scientific tools and file types. Datum is a catalog which targets the scientific data directly, including the tools and networks in which those tools are used. We work with the producers and consumers of the data where they are, targeting cloud and on-premise with a focus on classified networks. Datum is an Erlang/Elixir application. Technical Features Note: The features listed below are still under development and may change, slightly, upon final delivery of the product. File Formats - Datum has the ability to read additional metadata and provides processing pipelines for the following file formats: Plain Text, PDF, LaTeX, HTML, Open Document Format (.odt), XML, CSV/TSV (and other standard delimiters), OpenDocument Database and Spreadsheets, Geo-Referenced TIFF, Common Data Format, HDF/HDF5, LabView TDMS, Excel, DeltaTables, Parquet, Apache Iceberg, Apache Hudi and many others. Metadata Collection - Scanners for the local and networked file systems and cloud storage providers. Network integration with common databases such as MSSQL and MySQL. User Plugin System - Users are able to provide either file processing, metadata extraction, or sampling plugins in the programming language of their choice. Authentication/Authorization -: OIDC integration, SCIM provisioning and EntraID integration out of the box. Full user and group management system with a “least privilege” operating mode. Governance - Customizable data governance platform; dictate and enforce required metadata, enforce data embargos, and enforce user agreements and NDAs before data access. Ability to create health checks on data, rejecting abandoned or poorly curated data and automatically removing it from the search index. Ability for users to submit corrections. Search - Semantic search is a first class citizen. No licenses to expensive, external software required. Integrated use of vectors and vector-based search allows for AI agent integration at all levels of operation. Metadata Model - Display and control data’s lineage and connections to other data and data directories. Data is modeled after a filesystem - an organization instantly recognizable and navigable by most any user. CLI and SDK - Ships with a Command Line Interface (CLI) tool and with a fully-featured Python SDK. This allows for rapid and programmatic use of Datum by every level of user. Minimal Infrastructure - Datum ships as a single executable file and can be run on any operating system and most CPU architectures. Datum has no reliance on external databases, search indexing tools, or other outside services - and it runs equally well on edge computing devices, cloud services, or in a clustered HPC environment.

darrington, john↗

Let’s Unleash the Network Judgment: A Self-Supervised Approach for Cloud Image Analysis

Accurate cloud type identification and coverage analysis are crucial in understanding the Earth’s radiative budget. Traditional computer vision methods rely on low-level visual features of clouds for estimating cloud coverage or sky conditions. Several handcrafted approaches have been proposed; however, scope for improvement still exists. Newer deep neural networks (DNNs) have demonstrated superior performance for cloud segmentation and categorization. These methods, however, need expert engineering intervention in the preprocessing steps—in the traditional methods—or human assistance in assigning cloud or clear sky labels to a pixel for training DNNs. Such human mediation imposes considerable time and labor costs. We present the application of a new self-supervised learning approach to autonomously extract relevant features from sky images captured by ground-based cameras, for the classification and segmentation of clouds. We evaluate a joint embedding architecture that uses self-knowledge distillation plus regularization. We use two datasets to demonstrate the network’s ability to classify and segment sky images—one with ~85,000 images collected from our ground-based camera and another with 400 labeled images from the WSISEG database. We find that this approach can discriminate full-sky images based on cloud coverage, diurnal variation, and cloud base height. Additionally, it semantically segments the cloud areas without labels. The approach shows competitive performance in all tested tasks, suggesting a new alternative for cloud characterization.

54 ENVIRONMENTAL SCIENCES↗

Physics-Guided Machine Learning for Prediction of Cloud Properties in Satellite-Derived Solar Data

With over 20 years of high-resolution surface irradiance data covering most of the western hemisphere, the National Solar Radiation Database (NSRDB) is a vital public data asset. The NSRDB uses a two-step Physical Solar Model (PSM) that explicitly considers the effects of clouds and other atmospheric variables on radiative transfer. High-quality physical and optical cloud properties derived from satellite imagery are perhaps the most important data inputs to the PSM, representing the greatest source of radiation attenuation and scattering. However, traditional methods for cloud property retrieval have their own limitations and are unable to accurately predict cloud properties outside of nominal conditions. We introduce a physics-guided neural network that can accurately predict cloud properties when traditional methods fail or are inaccurate. Using this framework, we show reductions in relative Root Mean Square Error (RMSE) for Global Horizontal Irradiance (GHI) up to 13 percentage points for timesteps that previously had missing or low-quality cloud property data. We expect that this methodology will be effective in improving the quality of cloud property and solar irradiance data in the NSRDB.

cloud properties↗

Physics-Guided Machine Learning for Prediction of Cloud Properties in Satellite-Derived Solar Data: Preprint

With over 20 years of high-resolution surface irradiance data covering most of the western hemisphere, the National Solar Radiation Database (NSRDB) is a vital public data asset. The NSRDB uses a two-step Physical Solar Model (PSM) that explicitly considers the effects of clouds and other atmospheric variables on radiative transfer. High-quality physical and optical cloud properties derived from satellite imagery are perhaps the most important data input to the PSM, representing the greatest source of radiation attenuation and scattering. However, traditional methods for cloud property retrieval have their own limitations and are unable to accurately predict cloud properties outside of nominal conditions. We introduce a physics-guided neural network that can accurately predict cloud properties when traditional methods fail or are inaccurate. Using this framework, we show reductions in relative Root Mean Square Error (RMSE) for Global Horizontal Irradiance (GHI) up to 13 percentage points for timesteps that previously had missing or low-quality cloud property data. We expect that this methodology will be effective in improving the quality of cloud property and solar irradiance data in the NSRDB.

41 EE - Solar Energy Technologies Office (EE-4S)↗

The influence of cloud cover on the reliability of satellite-based solar resource data

Satellite-based solar resource data are often developed and validated by using binary cloudiness categories: clear sky or overcast cloudy sky. To investigate the reliability of solar resource data in partially cloudy conditions, we estimate cloud fraction using two distinct algorithms: a physical retrieval model using surface observed global horizontal irradiance (GHI) and direct normal irradiance (DNI) and a temporal average of cloud mask data estimated by the observed DNI. Our analysis reveals a significant presence of scattered clouds, broken clouds, and mismatches between satellite- and surface-based cloud data at 17 surface sites across the contiguous United States, though confidently clear and cloudy conditions collectively account for more than 70 % of the data. Solar radiation is computed using the National Solar Radiation Database (NSRDB) algorithm and validated using surface observations. Here, our findings suggest that, in the presence of scattered clouds, NSRDB data for clear-sky conditions can be subject to significant overestimation. In cloudy-sky conditions classified by satellite data, DNI computed by the Fast All-sky Radiation Model for Solar applications with DNI (FARMS-DNI) can be underestimated when limited clouds are detected by surface observations. The bias observed in several cloudiness categories indicates that the NSRDB is exceptionally accurate in confidently clear conditions. However, clear-sky conditions with scattered clouds and mismatched cloud data contribute significantly to the overall uncertainties in the NSRDB. Therefore, future improvements in solar resource data should involve development and implementation of satellite-derived cloud fraction and should consider a novel radiative transfer model accounting for amplified cloud reflection. The evaluation within cloudiness categories also provides a physical rationale for the superior performance of FARMS-DNI compared to the Direct Insolation Simulation Code (DISC) in both cloudy-sky and all-sky conditions.

14 SOLAR ENERGY↗

Poplar: a phylogenomics pipeline

Motivation Generating phylogenomic trees from the genomic data is essential in understanding biological systems. Each step of this complex process has received extensive attention and has been significantly streamlined over the years. Given the public availability of data, obtaining genomes for a wide selection of species is straightforward. However, analyzing that data to generate a phylogenomic tree is a multistep process with legitimate scientific and technical challenges, often requiring a significant input from a domain-area scientist. Results We present Poplar, a new, streamlined computational pipeline, to address the computational logistical issues that arise when constructing the phylogenomic trees. It provides a framework that runs state-of-the-art software for essential steps in the phylogenomic pipeline, beginning from a genome with or without an annotation, and resulting in a species tree. Running Poplar requires no external databases. In the execution, it enables parallelism for execution for clusters and cloud computing. The trees generated by Poplar match closely with state-of-the-art published trees. The usage and performance of Poplar is far simpler and quicker than manually running a phylogenomic pipeline. Availability and implementation Freely available on GitHub at https://github.com/sandialabs/poplar. Implemented using Python and supported on Linux.

Koning, Elizabeth [Sandia National Laboratories (S↗

The National Solar Radiation Database (NSRDB): A Brief Overview

This poster presents a high-level overview of the National Solar Radiation Database (NSRDB). The NSRDB uses the physics-based model (PSM), which was developed using: adapted PATMOS-X model for cloud identification and properties, REST-2 model for clear-sky conditions, and NREL's Fast All-sky Radiation Model for Solar Applications (FARMS) for cloudy-sky Global Horizontal Irradiance (GHI) solar irradiance calculations.

14 SOLAR ENERGY↗

Southern Great Plains Ice Nuclei Characterization Experiment Final Campaign Summary

The first ever ice nucleating particle (INP) measurements to be collected at the Southern Great Plains site were made during a period from late April to June 2014, as a trial for possible longer-term measurements at the site. These measurements will also be used to lay the foundation for understanding and parameterizing (for cloud resolving modeling) the sources of these climatically important aerosols as well as to augment the existing database containing this knowledge. Siting the measurements during the spring was intended to capture INP sources in or to this region from plant, soil, dust transported over long distances, biomass burning, and pollution aerosols at a time when they may influence warm-season convective clouds and precipitation. Data have been archived of real-time measurements of INP number concentrations as a function of processing conditions (temperature and relative humidity) during 18 days of sampling that spanned two distinctly different weather situations: a warm, dry and windy period with regional dust and biomass burning influences in early May, and a cooler period of frequent precipitation during early June. Precipitation delayed winter wheat harvesting, preventing intended sampling during that perturbation on atmospheric aerosols. INP concentrations were highest and most variable at all temperatures in the dry period, where we attribute the INP activity primarily to soil dust emissions. Additional offline INP analyses are underway to extend the characterization of INP to cover the entire mixed phase cloud regime from -5°C to -35°C during the full study. Initial comparisons between methods on four days show good agreement and excellent future promise. The additional offline immersion freezing data will be archived as soon as completed under separate funding. Analyses of additional specialized studies for specific attribution of INP to biological and smoke sources are continuing via the National Science Foundation and National Aeronautics and Space Administration funding that helped support instrumentation used for the measurements described herein. Aerosol Observing System aerosol data will be vital to the interpretation and parameterization of results as part of analyses for publications in preparation.

DeMott, PJ↗

Land cover change-induced decline in terrestrial gross primary production over the conterminous United States from 2001 to 2016

As one of the most dynamic aspects of global environmental change, land cover change (LCC) has a profound impact on terrestrial carbon sequestration. However, LCC-induced carbon fluxes are still the most uncertain terms in global and regional carbon budgets. Ecosystem gross primary production (GPP) is the total carbon uptake by vegetation through photosynthesis, serving as a major control on ecosystem function and land carbon balance during and after the modification of the land surface. However, accurately capturing LCC-induced GPP changes requires both high-quality land cover data and controlling for variation driven by other environmental factors such as climate. In this study, we comprehensively examined the effects of LCC on annual GPP trends over the conterminous United States (CONUS) from 2001 to 2016 using the USGS National Land Cover Database, a remote sensing-driven ecosystem model, and the Google Earth Engine cloud computing platform. We designed a series of model experiments to identify LCC effects on GPP by controlling climate effects. During the study period, LCC exerted a strong negative effect on total GPP across the CONUS ([-2.2, -1.8] Tg C yr -2 ), while climate had smaller positive effects ([0.17, 0. 92] Tg C yr -2 ). The LCC-induced reduction of GPP was mainly caused by net forest loss ([-1.98, -1.39] Tg C yr -2 ) and urban expansion ([-2.03, -1.92] Tg C yr -2 ), but was partially offset by increases in crop area ([+0.66, +0.79] Tg C yr -2 ). Ensemble simulations from TRENDY did not capture the strong negative LCC influences on GPP, likely due to limitations of the adopted land use/cover data. Overall, our study provides a novel perspective on LCC-induced GPP changes, which could help to improve our understanding of ecosystem function changes and constrain the estimation of land carbon balance in the context of anthropogenic activity and climate change.

54 ENVIRONMENTAL SCIENCES↗