Search NASA⌕ Search

SEARCH · Search NASA

Results for “evaluation datasets”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records

Advanced Terrestrial Simulator (ATS) evaluation dataset at 7 catchments across the continental United States

This dataset comprises of the input files and other files required for Advanced Terrestrial Simulator (ATS) simulations at 7 catchments across the continental United States. ATS is an integrated surface-subsurface hydrology model. We include Jupyter notebooks (within scripts folder) for individual catchments showing information (including data sources, river network, soil, geology, landuse types etc.) on preparing the machine readable input files. ATS observation output files are provided in the output folder. Figures and analyses (.xlsx sheets) are also provided. The catchments include, Taylor River Upstream (Colorado); (b) Cossatot River (Arkansas); (c) Panther Creek (Alabama); (d) Little Tennessee River (North Carolina and Georgia); (e) Mayo River (Virginia); (f) Flat Brook (New Jersey); (g) Neversink River headwaters (New York). Readme files are provided inside the directories providing more details. Files types include: .xml, .h5, .xlsx, .png, .ipynb, .py, .nc, .txt. All of the files types can be accessed by open source software, details on software requirements are following: .xml (any text editors including notepad and textedit), .h5 (in python using hdf libraries), .xlsx (WPS Office Spreadsheets, OpenOffice Calc, LibreOffice Calc, Microsoft Office etc.), .png (any image viewer), .ipynb (Jupyter notebook), .py (any text editors including notepad and textedit), .nc (using python or other open source software).

54 ENVIRONMENTAL SCIENCES↗

Using the Atmospheric Radiation Measurement (ARM) Datasets to Evaluate Climate Models in Simulating Diurnal and Seasonal Variations of Tropical Clouds

We use the long-term Atmospheric Radiation Measurement (ARM) datasets collected at the three Tropical Western Pacific (TWP) sites as a tropical testbed to evaluate the ability of the Community Atmosphere Model (CAM5) to simulate the various types of clouds, their seasonal and diurnal variations, and their impact on surface radiation. We conducted a series of CAM5 simulations at various horizontal grid spacing (around 2°, 1°, 0.5°, and 0.25°) with meteorological constraints from reanalysis. Model biases in the seasonal cycle of cloudiness are found to be weakly dependent on model resolution. Positive biases (up to 20%) in the annual mean total cloud fraction appear mostly in stratiform ice clouds. Higher-resolution simulations do reduce the positive bias in the frequency of ice clouds, but they inadvertently increase the negative biases in convective clouds and low-level liquid clouds, leading to a positive bias in annual mean shortwave fluxes at the sites, as high as 65 W m-2 in the 0.25° simulation. Such resolution-dependent biases in clouds can adversely lead to biases in ambient thermodynamic properties and, in turn, feedback on clouds. Both the CAM5 model and ARM observations show distinct diurnal cycles in total, stratiform and convective cloud fractions; however, they are out-of-phase by 12 hours and the biases vary by site. Our results suggest that biases in deep convection affect the vertical distribution and diurnal cycle of stratiform clouds through the transport of vapor and/or the detrainment of liquid and ice. We also found that the modelled gridmean surface longwave fluxes are systematically larger than site measurements when the grid that the ARM sites reside in is partially covered by ocean. The modeled longwave fluxes at such sites also lack a discernable diurnal cycle because the ocean part of the grid is warmer and less sensitive to radiative heating/cooling compared to land. Higher spatial resolution is more helpful is this regard. Our testbed approach can be easily adapted for the evaluation of new parameterizations being developed for CAM5 or other global or regional model simulations at high spatial resolutions.

Wang, Hailong↗

Dataset for "Evaluating Deep Learning Approaches for Predictions in Unmonitored Basins with Continental-scale Stream Temperature Models" Willard et al. (2024)

This data release provides all data and code used in the paper " "Evaluating Deep Learning Approaches for Predictions in Unmonitored Basins with Continental-scale Stream Temperature Models" Willard et al. (2024)" to model stream temperature, evaluate, and assess results. The associated manuscript explores current open questions in prediction in ungauged and unmonitored basins concerning top-down versus bottom-up approaches, tradeoffs between data available and input requirements, and the appropriate representation of catchment attributes as inputs to deep learning models. Modeling was done primarily with long short-term memory (LSTM) models, and stream site coverage spans 1362 locations across the conterminous United States. The data is organized into these items items:Code repository and data for the paper " "Evaluating Deep Learning Approaches for Predictions in Unmonitored Basins with Continental-scale Stream Temperature Models" Willard et al. (2024)".Code: stream_temp_ml_regionalization.zip contains the code repositoryData to run the code: - data_dir.zip -- contains all files that should be moved to the "DATA_DIR" variable defined in the "set_env_vars.sh" script in the code repository- metadata_dir.zip -- contains all files that should be moved to the "METADATA_DIR" variable defined in the "set_env_vars.sh" script in the code repository- error_analysis_attribute_and_groundwater_dir.zip - workflows for the extended error analysis by stream attribute and groundwater influenceData produced by the code and used in the paper:- outputs_dir.zip - contains model output and results (outputs_dir/results), model weights (outputs_dir/models), and all other outputs used for the paper including feature importances.To cite this code, please use the following BibTeX or MLA entries:bibtex:@misc{willard2024streamdata, author = {Jared Willard and Fabio Ciulla and Helen Weierbach and Vipin Kumar and Charuleka Varadharajan}, title = {Dataset for "Evaluating Deep Learning Approaches for Predictions in Unmonitored Basins with Continental-scale Stream Temperature Models"}, year = {2024}, doi = {10.15485/2448016}, publisher = {ESS-DIVE Repository}, url = {https://doi.org/10.15485/2448016}}MLA: Willard, Jared, et al. Dataset for "Evaluating Deep Learning Approaches for Predictions in Unmonitored Basins with Continental-scale Stream Temperature Models". 2024. ESS-DIVE Repository, doi:10.15485/2448016.

54 ENVIRONMENTAL SCIENCES↗

Dataset for Evaluation of Extreme Weather Impacts on Utility-Scale Photovoltaic Plant Performance in the United States

This dataset is a fusion of three data types (operations and maintenance tickets, weather data, and production data) that was used to support machine learning analysis and evaluation of drivers for low performance at photovoltaic (PV) sites during compound, extreme weather events. After being processed with machine learning, the data was used in the "Evaluation of Extreme Weather Impacts on Utility-scale Photovoltaic Plant Performance in the United States" manuscript. Additional details are captured in the associated manuscript.

AI↗

Development of a Benchmark Eddy Flux Evapotranspiration Dataset for Evaluation of Satellite-Driven Evapotranspiration Models Over the CONUS

A large sample of ground-based evapotranspiration (ET) measurements made in the United States, primarily from eddy covariance systems, were post-processed to produce a benchmark ET dataset. The dataset was produced primarily to support the intercomparison and evaluation of the OpenET satellite-based remote sensing ET (RSET) models and could also be used to evaluate ET data from other models and approaches. OpenET is a web-based service that makes field-delineated and pixel-level ET estimates from well-established RSET models readily available to water managers, agricultural producers, and the public. The benchmark dataset is composed of flux and meteorological data from a variety of providers covering native vegetation and agricultural settings. Flux footprint predictions were developed for each station and included static flux footprints developed based on average wind direction and speed, as well as dynamic hourly footprints that were generated with a physically based model of upwind source area. The two footprint prediction methods were rigorously compared to evaluate their relative spatial coverage. Data from all sources were post-processed in a consistent and reproducible manner including data handling, gap-filling, temporal aggregation, and energy balance closure correction. The resulting dataset included 243,048 daily and 5,284 monthly ET values from 194 stations, with all data falling between 1995 and 2021. We assessed average daily energy imbalance using 172 flux sites with a total of 193,021 days of data, finding that overall turbulent fluxes were understated by about 12% on average relative to available energy. Multiple linear regression analyses indicated that daily average latent energy flux may be typically understated slightly more than sensible heat flux. This dataset was developed to provide a consistent reference to support evaluation of RSET data being developed for a wide range of applications related to water accounting and water resources management at field to watershed scales.

54 ENVIRONMENTAL SCIENCES↗

Data assimilation and model evaluation experiment datasets

The Institute for Naval Oceanography, in cooperation with Naval Research Laboratories and universities, executed the Data Assimilation and Model Evaluation Experiment (DAMEE) for the Gulf Stream region during fiscal years 1991-1993. Enormous effort has gone into the preparation of several high-quality and consistent datasets for model initialization and verification. This paper describes the preparation process, the temporal and spatial scopes, the contents, the structure, etc., of these datasets. The goal of DAMEE and the need of data for the four phases of experiment are briefly stated. The preparation of DAMEE datasets consisted of a series of processes: (1) collection of observational data; (2) analysis and interpretation; (3) interpolation using the Optimum Thermal Interpolation System package; (4) quality control and re-analysis; and (5) data archiving and software documentation. The data products from these processes included a time series of 3D fields of temperature and salinity, 2D fields of surface dynamic height and mixed-layer depth, analysis of the Gulf Stream and rings system, and bathythermograph profiles. To date, these are the most detailed and high-quality data for mesoscale ocean modeling, data assimilation, and forecasting research. Feedback from ocean modeling groups who tested this data was incorporated into its refinement. Suggestions for DAMEE data usages include (1) ocean modeling and data assimilation studies, (2) diagnosis and theoretical studies, and (3) comparisons with locally detailed observations.

Lai, Chung-Cheng A.↗

The Fire and Smoke Model Evaluation Experiment—A Plan for Integrated, Large Fire–Atmosphere Field Campaigns

The Fire and Smoke Model Evaluation Experiment (FASMEE) is designed to collect integrated observations from large wildland fires and provide evaluation datasets for new models and operational systems. Wildland fire, smoke dispersion, and atmospheric chemistry models have become more sophisticated, and next-generation operational models will require evaluation datasets that are coordinated and comprehensive for their evaluation and advancement. Integrated measurements are required, including ground-based observations of fuels and fire behavior, estimates of fire-emitted heat and emissions fluxes, and observations of near-source micrometeorology, plume properties, smoke dispersion, and atmospheric chemistry. To address these requirements the FASMEE campaign design includes a study plan to guide the suite of required measurements in forested sites representative of many prescribed burning programs in the southeastern United States and increasingly common high-intensity fires in the western United States. Here we provide an overview of the proposed experiment and recommendations for key measurements. The FASMEE study provides a template for additional large-scale experimental campaigns to advance fire science and operational fire and smoke models.

54 ENVIRONMENTAL SCIENCES↗

A globally sampled high-resolution hand-labeled validation dataset for evaluating surface water extent maps

Effective monitoring of global water resources is increasingly critical due to climate change and population growth. Advancements in remote sensing technology, specifically in spatial, spectral, and temporal resolutions, are revolutionizing water resource monitoring, leading to more frequent and high-quality surface water extent maps using various techniques such as traditional image processing and machine learning algorithms. However, satellite imagery datasets contain trade-offs that result in inconsistencies in performance, such as disparities in measurement principles between optical (e.g., Sentinel-2) and radar (e.g., Sentinel-1) sensors and differences in spatial and spectral resolutions among optical sensors. Therefore, developing accurate and robust surface water mapping solutions requires independent validations from multiple datasets to identify potential biases within the imagery and algorithms. However, high-quality validation datasets are expensive to build, and few contain information on water resources. For this purpose, we introduce a globally sampled, high-spatial-resolution dataset labeled using 3 m PlanetScope imagery. Our surface water extent dataset comprises 100 images, each with a size of 1024×1024 pixels, which were sampled using a stratified random sampling strategy covering all 14 biomes. We highlighted urban and rural regions, lakes, and rivers, including braided rivers and coastal regions. We evaluated two surface water extent mapping methods using our dataset – Dynamic World, based on Sentinel-2, and the NASA IMPACT model, based on Sentinel-1. Dynamic World achieved a mean intersection over union (IoU) of 72.16 % and F1 score of 79.70 %, while the NASA IMPACT model had a mean IoU of 57.61 % and F1 score of 65.79 %. Performance varied substantially across biomes, highlighting the importance of evaluating models on diverse landscapes to assess their generalizability and robustness. Our dataset can be used to analyze satellite products and methods, providing insights into their advantages and drawbacks. Our dataset offers a unique tool for analyzing satellite products, aiding the development of more accurate and robust surface water monitoring solutions. The dataset can be accessed via https://doi.org/10.25739/03nt-4f29.

54 ENVIRONMENTAL SCIENCES↗

VLM4Bio: A Benchmark Dataset to Evaluate Pretrained Vision-Language Models for Trait Discovery from Biological Images

Images are increasingly becoming the currency for documenting biodiversity on the planet, providing novel opportunities for accelerating scientific discoveries in the field of organismal biology, especially with the advent of large vision-language models (VLMs). We ask if pre-trained VLMs can aid scientists in answering a range of biologically relevant questions without any additional fine-tuning. In this paper, we evaluate the effectiveness of 12 state-of-the-art (SOTA) VLMs in the field of organismal biology using a novel dataset, VLM4Bio, consisting of 469K question8 answer pairs involving 30K images from three groups of organisms: fishes, birds, and butterflies, covering five biologically relevant tasks. We also explore the effects of applying prompting techniques and tests for reasoning hallucination on the performance of VLMs, shedding new light on the capabilities of current SOTA VLMs in answering biologically relevant questions using images

Maruf, M [Virginia Tech, Blacksburg]↗

BCARS Simulated Phantom Dataset for Evaluation of Processing Pipelines

Broadband coherent anti-Stokes Raman scattering (BCARS) microscopy is a powerful label-free biological imaging technique, but the raw signal requires careful processing. The vibrationally resonant (Raman) fingerprint signal is usually small compared with instrumental noise sources and the nonresonant background (NRB) inherent in the BCARS signal. Fortunately, the NRB exhibits a systematic phase relationship with the coherent Raman response, acting as a heterodyne amplifier for the weak fingerprint signal. Due to this heterodyne effect, the Raman response can be recovered quantitatively and invariantly across different instruments, provided the NRB shape is known. Even with heterodyne amplification, the amplitudes of fingerprint signal components are often comparable to system noise. Singular value decomposition (SVD), which utilizes spatial information, is often employed for additional noise filtering. Consequently, finding optimal processing parameters to properly distinguish the NRB and Raman responses and suppress noise in the complex BCARS signal requires a reference system that realistically represents the spectral and spatial properties of BCARS signals obtained from biological samples. We present a digital tissue phantom that meets these criteria as a tool for testing candidate signal processing pipelines. The digital phantom is generated with simulated hyperspectral Raman images having system-specific noise and background characteristics. Here, we analyze phantom datasets with differing background and signal-to-noise conditions to evaluate their impact on the performance of multiple signal processing pipelines. Specifically, we investigate the application of a Butterworth filter-based routine to directly estimate the NRB from the BCARS signal. Additionally, we evaluate a Lorentzian wavelet transform as an alternative to the Hilbert transform for extracting the Raman spectrum from the BCARS signal. While we demonstrate this phantom for BCARS, it can be used for any spectroscopic Raman imaging approach.

Dixon, Jessica Z. [Georgia Institute of Technology↗

Pavement condition and climatic data in southeast Texas: A dataset for evaluating flood impacts on pavement performance

Effective pavement maintenance is essential for economic stability, optimal network performance, and roadway safety. Achieving this requires thorough evaluation of pavement conditions, including structural integrity, surface roughness, and distress characteristics. Pavement performance indicators play a critical role in influencing vehicle safety and ride quality. Recent advances have emphasized the use of data-driven modeling to anticipate pavement behavior, with the goal of optimizing resource allocation and refining Maintenance and Rehabilitation (M&R) strategies through accurate condition assessment. A foundational requirement for these modeling efforts is the availability of standardized, high-quality datasets that can support robust and reproducible infrastructure analysis. This data article presents a comprehensive dataset assembled to facilitate pavement performance prediction, with a geographic focus on Southeast Texas, particularly the flood-vulnerable area of Beaumont. The dataset encompasses pavement and traffic attributes, meteorological records, flood simulation outputs, ground deformation measurements, and topographic indices, enabling detailed examination of both load-associated and non-load-associated degradation mechanisms. Data preprocessing was performed using ArcGIS Pro, Microsoft Excel, and Python to ensure consistency and usability in data-driven modeling applications, including machine learning workflows. Key contributions of this dataset include its utility in analyzing the climatic and environmental factors affecting pavement conditions, identifying critical predictive features, and enabling in-depth correlation analysis across diverse variables. By filling existing gaps in input variable selection resources, this dataset supports the development of predictive tools for estimating future maintenance demand and enhancing the resilience of pavement networks in flood-impacted areas. The resource highlights the importance of standardized datasets for advancing pavement management practices and provides a robust foundation for ongoing infrastructure performance modeling.

42 ENGINEERING↗

Diagnosis of convective organization and cold pools using ARM datasets and evaluation of a unified convection parameterization (UNICON)

Tropical thunderstorms often cluster together. Studies have suggested that the degree to which the tropical thunderstorms are clustered impacts Earth's energy balance and water cycle, as well as extreme precipitation events. However, the processes controlling the spatial distribution of the thunderstorms are poorly understood and are not properly represented in most computer models for weather and climate prediction. Under the goals of better understanding how convection organizes at the mesoscale and advancing the representation of mesoscale convective organization in global climate models, we i) objectively quantified the degrees of convective organization and diagnosed cold pool processes using ARM field campaign observations, ii) examined the organization processes in storm-resolving model simulations, and iii) evaluated the impacts of mesoscale convective organization in global model simulations. The project yielded a firm reference against which the global model representation of mesoscale convective organization and cold pools can be evaluated against and shed new light into the role of parameterized convective organization in the global model simulation of the basic state and variability. Our results revealed two distinct phases of convective clustering during the two-day rain episodes (Cheng et al. 2018) and a new mechanism through which vertical wind shear in the low-troposphere can aid convective organization over tropical oceans (Cheng et al. 2020). It was demonstrated that the interactive representation of cold pools and mesoscale convective organization is key for global models to successfully simulate both the mean state and intraseasonal variability in the tropics (Ahn et al. 2019; 2020).

54 ENVIRONMENTAL SCIENCES↗

Leveraging NREL's ResStock & ComStock Dataset to Evaluate Building Stock Electrification: Preprint

Residential and commercial buildings accounted for 40% of U.S. energy consumption in 2022 and represent a significant opportunity for decarbonization through energy efficiency and electrification, and for grid planning. Building stock energy modeling is a powerful tool that can evaluate what-if scenarios as utilities, municipalities, policymakers, building owners and others work towards equitable building decarbonization and climate goals. This presentation will highlight several high-impact use cases of the National Renewable Energy Laboratory (NREL)'s highly granular, bottom-up building stock energy modeling tools, ResStock and ComStock. These use cases cover a wide range of project scale, from neighborhood electrification analysis and municipality long-term energy planning, to state energy code development and national policy evaluation. This presentation will showcase specific real-world applications for which ResStock and ComStock have been utilized across the country, including California codes and standards cost-effectiveness analysis, New York City affordable housing electrification cost gap analysis, and California targeted electrification and gas decommissioning analysis. For each use case, this presentation will illustrate how ResStock and ComStock played a crucial role in accurately characterizing regional building stocks, providing discrete and aggregated end-use load shapes, and calculating lifecycle consumption, emissions, and costs for a variety of building electrification strategies and scenarios. Finally, this presentation will demonstrate how the data provided by ResStock and ComStock can help unlock significant outcomes for these use cases, including but not limited to, customer bill impact, incentive and program design, and energy equity analyses.

building stock modeling↗

Evaluation of remote sensing-based evapotranspiration products at low-latitude eddy covariance sites

Remote sensing-based evapotranspiration (ET) products have been evaluated primarily using data from northern middle latitudes; therefore, little is known about their performance at low latitudes. To address this bias, an evaluation dataset was compiled using eddy covariance data from 40 sites between latitudes 30° S and 30° N. The flux data were obtained from the emerging network in Mexico (MexFlux) and from openly available databases of FLUXNET, AsiaFlux, and OzFlux. This unique reference dataset was then used to evaluate remote sensing-based ET products in environments that have been underrepresented in earlier studies. The evaluated products were: MODIS ET (MOD16, both the discontinued collection 5 (C5) and the latest collection (C6)), Global Land Evaporation Amsterdam Model (GLEAM) ET, and Atmosphere-Land Exchange Inverse (ALEXI) ET. Products were compared with unadjusted fluxes (ETorig) and with fluxes corrected for the lack of energy balance closure (ETebc). Three common statistical metrics were used: coefficient of determination (R2), root mean square error (RMSE), and percent bias (PBIAS). The effect of a vegetation mismatch between pixel and site on product evaluation results was investigated by examining the relationship between the statistical metrics and product-specific vegetation match indexes. Evaluation results of this study and those published in the literature were used to examine the performance of the products across latitudes. Differences between the MOD16 collection 5 and 6 datasets were generally smaller than differences with the other products. Performance and ranking of the evaluated products depended on whether ETorig or ETebc was used. When using ETorig, GLEAM generally had the highest R2, smallest PBIAS, and best RMSE values across the studied land cover types and climate zones. Neither MOD16 nor ALEXI performed consistently better than the other. When using ETebc, none of the products stood out in terms of both low bias and strong correlations. The use of ETebc instead of ETorig affected the biases more than the correlations. The product evaluation results showed no significant relationship with the degree of match between the vegetation at the pixel and site scale. The latitudinal comparison showed tendencies of lower R2 (all products) but better PBIAS and normalized RMSE values (MOD16 and GLEAM) for forests at low latitudes than for forests at northern middle latitudes. For non-forest vegetation, the products showed no clear latitudinal differences in performance.

Diego Salazar-Martínez↗

Model Evaluation and Hindcasting: An Experiment with an Integrated Assessment Model

Integrated assessment models have been extensively used for analyzing long term energy and greenhouse emissions trajectories and have influenced key policies on this subject. Though admittedly these models are focused on the long term trajectories, how well these models are able to capture historical dynamics is an open question. In a first experiment of its kind, we present a framework for evaluation of such integrated assessment models. We use Global Change Assessment Model for this zero order experiment, and focus on the building sector results for USA. We calibrate the model for 1990 and run it forward up to 2095 in five year time steps. This gives us results for 1995, 2000, 2005 and 2010 which we compare to observed historical data at both fuel level and service level. We focus on bringing out the key insights for the wider process of model evaluation through our experiment with GCAM. We begin with highlighting that creation of an evaluation dataset and identification of key evaluation metric is the foremost challenge in the evaluation process. Our analysis highlights that estimation of functional form of the relationship between energy service demand, which is an unobserved variable, and its drivers is a significant challenge in the absence of adequate historical data for both the dependent and driver variables. Historical data availability for key metrics is a serious limiting factor in the process of evaluation. Interestingly, service level data against which such models need to be evaluated are itself a result of models. Thus for energy services, the best we can do is compare our model results with other model results rather than observed and measured data. We show that long term models, by the nature of their construction, will most likely underestimate the rapid growth in some services observed in a short time span. Also, we learn that modeling saturated energy services like space heating is easier than modeling unsaturated services like space cooling and understanding that how far a service is from its saturation level is a key question which we probably don’t have an answer to. Finally and most importantly, even if long term models partially miss the short term dynamics, the long term insights provides by these models is fairly robust. We conclude by highlighting that our work is the first step in the much wider process of integrated assessment model evaluation and will hence have its own limitations. Future evaluation research work should build upon this zero order experiment for improving our modeling of human and coupled earth systems.

building energy↗