Search NASA⌕ Search

SEARCH · Search NASA

Results for “cloud database”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 127 records · Page 7
All sourcesNASA NTRSDOE OSTI
DOE OSTI2025

Datum: A Scientific Metadata Catalog

The data catalog market is currently flooded with a myriad of different products, but none serve the scientific community well. There are cloud-native tools like Databricks, Snowflake,to on-premise solutions like Collibra and Datahub. The common failing of all these tools however, is their inability to serve the scientific data community directly. Most catalogs are targeted towards financial, health, or user data - not sensor or scientific domain data. They also prioritize integrations that often don’t exist or are just starting to be used in the scientific realm - all while ignoring common scientific tools and file types. Datum is a catalog which targets the scientific data directly, including the tools and networks in which those tools are used. We work with the producers and consumers of the data where they are, targeting cloud and on-premise with a focus on classified networks. Datum is an Erlang/Elixir application. Technical Features Note: The features listed below are still under development and may change, slightly, upon final delivery of the product. File Formats - Datum has the ability to read additional metadata and provides processing pipelines for the following file formats: Plain Text, PDF, LaTeX, HTML, Open Document Format (.odt), XML, CSV/TSV (and other standard delimiters), OpenDocument Database and Spreadsheets, Geo-Referenced TIFF, Common Data Format, HDF/HDF5, LabView TDMS, Excel, DeltaTables, Parquet, Apache Iceberg, Apache Hudi and many others. Metadata Collection - Scanners for the local and networked file systems and cloud storage providers. Network integration with common databases such as MSSQL and MySQL. User Plugin System - Users are able to provide either file processing, metadata extraction, or sampling plugins in the programming language of their choice. Authentication/Authorization -: OIDC integration, SCIM provisioning and EntraID integration out of the box. Full user and group management system with a “least privilege” operating mode. Governance - Customizable data governance platform; dictate and enforce required metadata, enforce data embargos, and enforce user agreements and NDAs before data access. Ability to create health checks on data, rejecting abandoned or poorly curated data and automatically removing it from the search index. Ability for users to submit corrections. Search - Semantic search is a first class citizen. No licenses to expensive, external software required. Integrated use of vectors and vector-based search allows for AI agent integration at all levels of operation. Metadata Model - Display and control data’s lineage and connections to other data and data directories. Data is modeled after a filesystem - an organization instantly recognizable and navigable by most any user. CLI and SDK - Ships with a Command Line Interface (CLI) tool and with a fully-featured Python SDK. This allows for rapid and programmatic use of Datum by every level of user. Minimal Infrastructure - Datum ships as a single executable file and can be run on any operating system and most CPU architectures. Datum has no reliance on external databases, search indexing tools, or other outside services - and it runs equally well on edge computing devices, cloud services, or in a clustered HPC environment.

darrington, john↗
NASA NTRS2015

Database Design for the Evaluation of On-shore and Off-Shore Storm Characteristics over East Central Florida

Data from multiple sources is needed to investigate lightning characteristics over differing terrain (on-shore vs. off-shore) by comparing natural cloud-to-ground lightning behavior differences depending on the characteristics of attachment mediums. The KSC Lightning Research Database (KLRD) was created to reduce manual data entry time and aid research by combining information from various data sources into a single record for each unique lightning event of interest. The KLRD uses automatic data handling functions to import data from a lightning detection network and identify and record lighting events of interest. Additional automatic functions import data from the NASA Buoy 41009 (located approximately 20 miles off the coast) and the KSC Electric Field Mill network, then match these electric field mill values to the corresponding lightning events. The KLRD calculates distances between each lightning event and the various electric field mills, aids in identifying the location type for each stroke (i.e., on-shore vs. off-shore, etc.), provides statistics on the number of strokes per flash, and produces customizable reports for quick retrieval and logical display of data. Data from February 2014 to date covers 48 unique storm dates with 2295 flashes containing 5700 strokes, of which 2612 are off-shore and 1003 are on-shore. The number of strokes per flash ranges from 1 to 22. The ratio of single to subsequent stroke flashes is 1.29 for off-shore strokes and 2.19 for on-shore strokes.

cloud-to-ground↗
NASA NTRS2006

Bayesian Estimation of Precipitation from Satellite Passive Microwave Observations Using Combined Radar-Radiometer Retrievals

Precipitation estimation from satellite passive microwave radiometer observations is a problem that does not have a unique solution that is insensitive to errors in the input data. Traditionally, to make this problem well posed, a priori information derived from physical models or independent, high-quality observations is incorporated into the solution. In the present study, a database of precipitation profiles and associated brightness temperatures is constructed to serve as a priori information in a passive microwave radiometer algorithm. The precipitation profiles are derived from a Tropical Rainfall Measuring Mission (TRMM) combined radar radiometer algorithm, and the brightness temperatures are TRMM Microwave Imager (TMI) observed. Because the observed brightness temperatures are consistent with those derived from a radiative transfer model embedded in the combined algorithm, the precipitation brightness temperature database is considered to be physically consistent. The database examined here is derived from the analysis of a month-long record of TRMM data that yields more than a million profiles of precipitation and associated brightness temperatures. These profiles are clustered into a tractable number of classes based on the local sea surface temperature, a radiometer-based estimate of the echo-top height (the height beyond which the reflectivity drops below 17 dBZ), and brightness temperature principal components. For each class, the mean precipitation profile, brightness temperature principal components, and probability of occurrence are determined. The precipitation brightness temperature database supports a radiometer-only algorithm that incorporates a Bayesian estimation methodology. In the Bayesian framework, precipitation estimates are weighted averages of the mean precipitation values corresponding to the classes in the database, with the weights being determined according to the similarity between the observed brightness temperature principal components and the brightness temperature principal components of the classes. Because the classes are stratified by the sea surface temperature and the echo-top-height estimator, the number of classes that are considered for retrieval is significantly smaller than the total number of classes, making the algorithm computationally efficient. The radiometer-only algorithm is applied to TMI observations, and precipitation estimates are compared with combined TRMM precipitation radar (PR) TMI reference estimates. The TMI-only algorithm, supported by the empirically derived database, produces estimates that are more consistent with the reference values than the precipitation estimates from the version-6 TRMM facility TMI algorithm. Cloud-resolving model simulations are used to assign a latent heating profile to each precipitation profile in the empirically derived database, making it possible to estimate latent heating using the radiometer-only algorithm. Although the evaluation of latent heating estimates in this study is preliminary, because realistic conditional probability distribution functions are attached to latent heating structures in the algorithm s database, a generally positive impact on latent heating estimation from passive microwave observations is expected.

Grecu, Mircea↗
NASA NTRS2013

A Full Snow Season in Yellowstone: A Database of Restored Aqua Band 6

The algorithms for estimating snow extent for the Moderate Resolution Imaging Spectroradiometer (MODIS) optimally use the 1.6- m channel which is unavailable for MODIS on Aqua due to detector damage. As a test bed to demonstrate that Aqua band 6 can be restored, we chose the area surrounding Yellowstone and Grand Teton national parks. In such rugged and difficult-to-access terrain, satellite images are particularly important for providing an estimation of snow-cover extent. For the full 2010-2011 snow season covering the Yellowstone region, we have used quantitative image restoration to create a database of restored Aqua band 6. The database includes restored radiances, normalized vegetation index, normalized snow index, thermal data, and band-6-based snow-map products. The restored Aqua-band-6 data have also been regridded and combined with Terra data to produce a snow-cover map that utilizes both Terra and Aqua snow maps. Using this database, we show that the restored Aqua-band-6-based snow-cover extent has a comparable performance with respect to ground stations to the one based on Terra. The result of a restored band 6 from Aqua is that we have an additional band-6 image of the Yellowstone region each day. This image can be used to mitigate cloud occlusion, using the same algorithms used for band 6 on Terra. We show an application of this database of restored band-6 images to illustrate the value of creating a cloud gap filling using the National Aeronautics and Space Administration s operational cloud masks and data from both Aqua and Terra.

Gladkova, Irina↗
NASA NTRS2011

Changes in the High-Latitude Topside Ionospheric Vertical Electron-Density Profiles in Response to Solar-Wind Perturbations During Large Magnetic Storms

The latest results from an investigation to establish links between solar-wind and topside-ionospheric parameters will be presented including a case where high-latitude topside electron-density Ne(h) profiles indicated dramatic rapid changes in the scale height during the main phase of a large magnetic storm (Dst < -200 nT). These scale-height changes suggest a large heat input to the topside ionosphere at this time. The topside profiles were derived from ISIS-1 digital ionograms obtained from the NASA Space Physics Data Facility (SPDF) Coordinated Data Analysis Web (CDA Web). Solar-wind data obtained from the NASA OMNIWeb database indicated that the magnetic storm was due to a magnetic cloud. This event is one of several large magnetic storms being investigated during the interval from 1965 to 1984 when both solar-wind and digital topside ionograms, from either Alouette-2, ISIS-1, or ISIS-2, are potentially available.

Benson, Robert F.↗
DOE OSTI2023

Let’s Unleash the Network Judgment: A Self-Supervised Approach for Cloud Image Analysis

Accurate cloud type identification and coverage analysis are crucial in understanding the Earth’s radiative budget. Traditional computer vision methods rely on low-level visual features of clouds for estimating cloud coverage or sky conditions. Several handcrafted approaches have been proposed; however, scope for improvement still exists. Newer deep neural networks (DNNs) have demonstrated superior performance for cloud segmentation and categorization. These methods, however, need expert engineering intervention in the preprocessing steps—in the traditional methods—or human assistance in assigning cloud or clear sky labels to a pixel for training DNNs. Such human mediation imposes considerable time and labor costs. We present the application of a new self-supervised learning approach to autonomously extract relevant features from sky images captured by ground-based cameras, for the classification and segmentation of clouds. We evaluate a joint embedding architecture that uses self-knowledge distillation plus regularization. We use two datasets to demonstrate the network’s ability to classify and segment sky images—one with ~85,000 images collected from our ground-based camera and another with 400 labeled images from the WSISEG database. We find that this approach can discriminate full-sky images based on cloud coverage, diurnal variation, and cloud base height. Additionally, it semantically segments the cloud areas without labels. The approach shows competitive performance in all tested tasks, suggesting a new alternative for cloud characterization.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2021

Physics-Guided Machine Learning for Prediction of Cloud Properties in Satellite-Derived Solar Data

With over 20 years of high-resolution surface irradiance data covering most of the western hemisphere, the National Solar Radiation Database (NSRDB) is a vital public data asset. The NSRDB uses a two-step Physical Solar Model (PSM) that explicitly considers the effects of clouds and other atmospheric variables on radiative transfer. High-quality physical and optical cloud properties derived from satellite imagery are perhaps the most important data inputs to the PSM, representing the greatest source of radiation attenuation and scattering. However, traditional methods for cloud property retrieval have their own limitations and are unable to accurately predict cloud properties outside of nominal conditions. We introduce a physics-guided neural network that can accurately predict cloud properties when traditional methods fail or are inaccurate. Using this framework, we show reductions in relative Root Mean Square Error (RMSE) for Global Horizontal Irradiance (GHI) up to 13 percentage points for timesteps that previously had missing or low-quality cloud property data. We expect that this methodology will be effective in improving the quality of cloud property and solar irradiance data in the NSRDB.

cloud properties↗
DOE OSTI2021

Physics-Guided Machine Learning for Prediction of Cloud Properties in Satellite-Derived Solar Data: Preprint

With over 20 years of high-resolution surface irradiance data covering most of the western hemisphere, the National Solar Radiation Database (NSRDB) is a vital public data asset. The NSRDB uses a two-step Physical Solar Model (PSM) that explicitly considers the effects of clouds and other atmospheric variables on radiative transfer. High-quality physical and optical cloud properties derived from satellite imagery are perhaps the most important data input to the PSM, representing the greatest source of radiation attenuation and scattering. However, traditional methods for cloud property retrieval have their own limitations and are unable to accurately predict cloud properties outside of nominal conditions. We introduce a physics-guided neural network that can accurately predict cloud properties when traditional methods fail or are inaccurate. Using this framework, we show reductions in relative Root Mean Square Error (RMSE) for Global Horizontal Irradiance (GHI) up to 13 percentage points for timesteps that previously had missing or low-quality cloud property data. We expect that this methodology will be effective in improving the quality of cloud property and solar irradiance data in the NSRDB.

41 EE - Solar Energy Technologies Office (EE-4S)↗
DOE OSTI2024

The influence of cloud cover on the reliability of satellite-based solar resource data

Satellite-based solar resource data are often developed and validated by using binary cloudiness categories: clear sky or overcast cloudy sky. To investigate the reliability of solar resource data in partially cloudy conditions, we estimate cloud fraction using two distinct algorithms: a physical retrieval model using surface observed global horizontal irradiance (GHI) and direct normal irradiance (DNI) and a temporal average of cloud mask data estimated by the observed DNI. Our analysis reveals a significant presence of scattered clouds, broken clouds, and mismatches between satellite- and surface-based cloud data at 17 surface sites across the contiguous United States, though confidently clear and cloudy conditions collectively account for more than 70 % of the data. Solar radiation is computed using the National Solar Radiation Database (NSRDB) algorithm and validated using surface observations. Here, our findings suggest that, in the presence of scattered clouds, NSRDB data for clear-sky conditions can be subject to significant overestimation. In cloudy-sky conditions classified by satellite data, DNI computed by the Fast All-sky Radiation Model for Solar applications with DNI (FARMS-DNI) can be underestimated when limited clouds are detected by surface observations. The bias observed in several cloudiness categories indicates that the NSRDB is exceptionally accurate in confidently clear conditions. However, clear-sky conditions with scattered clouds and mismatched cloud data contribute significantly to the overall uncertainties in the NSRDB. Therefore, future improvements in solar resource data should involve development and implementation of satellite-derived cloud fraction and should consider a novel radiative transfer model accounting for amplified cloud reflection. The evaluation within cloudiness categories also provides a physical rationale for the superior performance of FARMS-DNI compared to the Direct Insolation Simulation Code (DISC) in both cloudy-sky and all-sky conditions.

14 SOLAR ENERGY↗
DOE OSTI2025

Poplar: a phylogenomics pipeline

Motivation Generating phylogenomic trees from the genomic data is essential in understanding biological systems. Each step of this complex process has received extensive attention and has been significantly streamlined over the years. Given the public availability of data, obtaining genomes for a wide selection of species is straightforward. However, analyzing that data to generate a phylogenomic tree is a multistep process with legitimate scientific and technical challenges, often requiring a significant input from a domain-area scientist. Results We present Poplar, a new, streamlined computational pipeline, to address the computational logistical issues that arise when constructing the phylogenomic trees. It provides a framework that runs state-of-the-art software for essential steps in the phylogenomic pipeline, beginning from a genome with or without an annotation, and resulting in a species tree. Running Poplar requires no external databases. In the execution, it enables parallelism for execution for clusters and cloud computing. The trees generated by Poplar match closely with state-of-the-art published trees. The usage and performance of Poplar is far simpler and quicker than manually running a phylogenomic pipeline. Availability and implementation Freely available on GitHub at https://github.com/sandialabs/poplar. Implemented using Python and supported on Linux.

Koning, Elizabeth [Sandia National Laboratories (S↗
NASA NTRS2019

Image Labeler: A Web Interface to Catalog Earth Science Events

Advances in machine learning (ML) have made it possible to automatically detect Earth science phenomena from satellite imagery. While useful, ML algorithms typically require an extensive dataset containing labeled images for training. Systematic labeling and management of such datasets is quite cumbersome. With this in mind, we present the Image Labeler. Image Labeler is a fast and scalable cloud-based tool that facilitates the rapid development of Earth science event databases, in order to aid automated ML-based image classification.

Case Study↗
DOE OSTI2017

The National Solar Radiation Database (NSRDB): A Brief Overview

This poster presents a high-level overview of the National Solar Radiation Database (NSRDB). The NSRDB uses the physics-based model (PSM), which was developed using: adapted PATMOS-X model for cloud identification and properties, REST-2 model for clear-sky conditions, and NREL's Fast All-sky Radiation Model for Solar Applications (FARMS) for cloudy-sky Global Horizontal Irradiance (GHI) solar irradiance calculations.

14 SOLAR ENERGY↗
NASA NTRS1996

O2 A Band Studies for Cloud Detection and Algorithm Improvement

Detection of cloud parameters from space-based spectrometers can employ the vibrational bands of O2 in the (sup b1)Sigma(sub +)(sub g) yields X(sub 3) Sigma(sup -)(sub g) spin-forbidden electronic transition manifold, particularly the Delta nu = 0 A band. The GOME instrument uses the A band in the Initial Cloud Fitting Algorithm (ICFA). The work reported here consists of making substantial improvements in the line-by-line spectral database for the A band, testing whether an additional correction to the line shape function is necessary in order to correctly model the atmospheric transmission in this band, and calculating prototype cloud and ground template spectra for comparison with satellite measurements.

Chance, K. V.↗
DOE OSTI2015

Southern Great Plains Ice Nuclei Characterization Experiment Final Campaign Summary

The first ever ice nucleating particle (INP) measurements to be collected at the Southern Great Plains site were made during a period from late April to June 2014, as a trial for possible longer-term measurements at the site. These measurements will also be used to lay the foundation for understanding and parameterizing (for cloud resolving modeling) the sources of these climatically important aerosols as well as to augment the existing database containing this knowledge. Siting the measurements during the spring was intended to capture INP sources in or to this region from plant, soil, dust transported over long distances, biomass burning, and pollution aerosols at a time when they may influence warm-season convective clouds and precipitation. Data have been archived of real-time measurements of INP number concentrations as a function of processing conditions (temperature and relative humidity) during 18 days of sampling that spanned two distinctly different weather situations: a warm, dry and windy period with regional dust and biomass burning influences in early May, and a cooler period of frequent precipitation during early June. Precipitation delayed winter wheat harvesting, preventing intended sampling during that perturbation on atmospheric aerosols. INP concentrations were highest and most variable at all temperatures in the dry period, where we attribute the INP activity primarily to soil dust emissions. Additional offline INP analyses are underway to extend the characterization of INP to cover the entire mixed phase cloud regime from -5°C to -35°C during the full study. Initial comparisons between methods on four days show good agreement and excellent future promise. The additional offline immersion freezing data will be archived as soon as completed under separate funding. Analyses of additional specialized studies for specific attribution of INP to biological and smoke sources are continuing via the National Science Foundation and National Aeronautics and Space Administration funding that helped support instrumentation used for the measurements described herein. Aerosol Observing System aerosol data will be vital to the interpretation and parameterization of results as part of analyses for publications in preparation.

DeMott, PJ↗
DOE OSTI2021

Land cover change-induced decline in terrestrial gross primary production over the conterminous United States from 2001 to 2016

As one of the most dynamic aspects of global environmental change, land cover change (LCC) has a profound impact on terrestrial carbon sequestration. However, LCC-induced carbon fluxes are still the most uncertain terms in global and regional carbon budgets. Ecosystem gross primary production (GPP) is the total carbon uptake by vegetation through photosynthesis, serving as a major control on ecosystem function and land carbon balance during and after the modification of the land surface. However, accurately capturing LCC-induced GPP changes requires both high-quality land cover data and controlling for variation driven by other environmental factors such as climate. In this study, we comprehensively examined the effects of LCC on annual GPP trends over the conterminous United States (CONUS) from 2001 to 2016 using the USGS National Land Cover Database, a remote sensing-driven ecosystem model, and the Google Earth Engine cloud computing platform. We designed a series of model experiments to identify LCC effects on GPP by controlling climate effects. During the study period, LCC exerted a strong negative effect on total GPP across the CONUS ([-2.2, -1.8] Tg C yr -2 ), while climate had smaller positive effects ([0.17, 0. 92] Tg C yr -2 ). The LCC-induced reduction of GPP was mainly caused by net forest loss ([-1.98, -1.39] Tg C yr -2 ) and urban expansion ([-2.03, -1.92] Tg C yr -2 ), but was partially offset by increases in crop area ([+0.66, +0.79] Tg C yr -2 ). Ensemble simulations from TRENDY did not capture the strong negative LCC influences on GPP, likely due to limitations of the adopted land use/cover data. Overall, our study provides a novel perspective on LCC-induced GPP changes, which could help to improve our understanding of ecosystem function changes and constrain the estimation of land carbon balance in the context of anthropogenic activity and climate change.

54 ENVIRONMENTAL SCIENCES↗
NASA NTRS2010

Hyperspectrally-Resolved Surface Emissivity Derived Under Optically Thin Clouds

Surface spectral emissivity derived from current and future satellites can and will reveal critical information about the Earth s ecosystem and land surface type properties, which can be utilized as a means of long-term monitoring of global environment and climate change. Hyperspectrally-resolved surface emissivities are derived with an algorithm utilizes a combined fast radiative transfer model (RTM) with a molecular RTM and a cloud RTM accounting for both atmospheric absorption and cloud absorption/scattering. Clouds are automatically detected and cloud microphysical parameters are retrieved; and emissivity is retrieved under clear and optically thin cloud conditions. This technique separates surface emissivity from skin temperature by representing the emissivity spectrum with eigenvectors derived from a laboratory measured emissivity database; in other words, using the constraint as a means for the emissivity to vary smoothly across atmospheric absorption lines. Here we present the emissivity derived under optically thin clouds in comparison with that under clear conditions.

Zhou, Daniel K.↗
DOE OSTI2022

New limits from microlensing on Galactic black holes in the mass range 10 M ⊙ < M < 1000 M ⊙

We searched for long-duration microlensing events originating from intermediate-mass black holes (BH) in the halo of the Milky Way, using archival data from the EROS-2 and MACHO photometric surveys towards the Large Magellanic Cloud (LMC). We combined data from these two surveys to create a common database of light curves for 14.1 million objects in the LMC, covering a total duration of 10.6 years, with flux series measured in four wide passbands. We carried out a microlensing search on these light curves, complemented by the light curves of 22.7 million objects, observed only by EROS-2 or only by MACHO, over about 7 years, with flux series measured in only two passbands. A likelihood analysis, taking into account the LMC self-lensing and Milky Way disk contributions, allows us to conclude that compact objects with masses in the range 10 – 100 M ⊙ cannot make up more than ~15% of a standard halo total mass (at a 95% confidence level). Our analysis sensitivity weakens for heavier objects, although we still rule out the possibility of ~50% of the halo being made of ~1000 M ⊙ BHs. Combined with previous EROS results, an upper limit of ~15% of the total halo mass can be obtained for the contribution of compact halo objects in the mass range 10 –6 – 10 2 M ⊙ .

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2021

SQL and NoSQL Databases for Cyber Physical Production Systems in Internet of Things for Manufacturing (IoTfM)

Abstract In this paper, the design and performance differences between Relational Database Management Systems (RDBMS) and NoSQL Database Systems are examined, with attention to their applicability for real-world Internet of Things for manufacturing (IoTfM) data. While previous work has extensively compared SQL and NoSQL for both generalized and IoT uses, this work specifically examines the tradeoffs and performance differences for manufacturing applications by using a high-fidelity data set collected from a large US manufacturing firm. Growing an IoT system beyond the pilot stage requires scalable data storage; this work seeks to determine the impact of selected database systems on data write performance at scale. Payload size and message frequency were used as the primary characteristics to maintain model fidelity in simulated clients. As the number of simulated asset clients grow, the data write latency was calculated to determine how both database systems’ performance were affected. To isolate the RDBMS and NoSQL differences, a cloud environment was created using Amazon Web Services (AWS) with two identical data ingestion pipelines: writing data to an RDMBS (1) using AWS Aurora MySQL, and (2) using AWS DynamoDB NoSQL. The findings may provide guidance for further experimentation in large-scale manufacturing IoT implementations.

Gamero, David↗