Search NASA⌕ Search

SEARCH · Search NASA

Results for “model evaluation”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 631 records · Page 35

Predictive analytics of selections of russet potatoes

We explore the application of machine learning algorithms specifically to enhance the selection process of Russet potato (Solanum tuberosum L.) clones in breeding trials by predicting their suitability for advancement. This study addresses the challenge of efficiently identifying high-yield, disease-resistant, and climate-resilient potato varieties that meet processing industry standards. Leveraging manually collected data from trials in the state of Oregon, we investigate the potential of a wide variety of state-of-the-art binary classification models. The dataset includes 1086 clones, with data on 38 attributes recorded for each clone, focusing on yield, size, appearance, and frying characteristics, with several control varieties planted consistently across four Oregon regions from 2013 to 2021. We conduct a comprehensive analysis of the dataset that includes preprocessing, feature engineering, and imputation to address missing values. We focus on several key metrics such as accuracy, F1-score, and Matthews correlation coefficient (MCC) for model evaluation. The top-performing models, namely a feedforward neural network classifier (Neural Net), a histogram-based gradient boosting classifier (HGBC), and a support vector machine classifier (SVM), demonstrate consistent and significant results. To further validate our findings, we conducted a simulation study using the aims, data-generating mechanisms, estimands, methods, and performance measures (ADEMP) framework, simulating different data-generating scenarios to assess model robustness and performance through true positive, true negative, false positive, and false negative distributions, area under the receiver operating characteristic curve (AUC-ROC) and MCC. The simulation results highlight that non-linear models like SVM and HGBC consistently show higher AUC-ROC and MCC than logistic regression, thus outperforming the traditional linear model across various distributions, and emphasizing the importance of model selection and tuning in agricultural trials. Variable selection further enhances model performance and identifies influential features in predicting trial outcomes. The findings emphasize the potential of machine learning in streamlining the selection process for potato varieties, offering benefits such as increased efficiency, substantial cost savings, and judicious resource utilization. Our study contributes insights into precision agriculture and showcases the relevance of advanced technologies for informed decision-making in breeding programs.

60 APPLIED LIFE SCIENCES↗

Global hourly land surface air temperature datasets: inter‐comparison and climate change

ABSTRACT The land surface air temperature ( LSAT ) is one of the fundamental parameters to represent heat transfer and to modulate the moisture cycle between land and atmosphere. Here, we quantify the spatiotemporal characteristics of four newly developed hourly temperature products by merging reanalysis with in situ data Climatic Research Unit ( CRU ). Overall, LSATs from different hourly products are consistent with each other, and their differences are generally smaller in magnitude than biases between hourly products and monthly averaged daily maximum and minimum temperature data from CRU . While the true monthly mean (using hourly values) and the monthly mean (of daily maximum and minimum temperatures) and their seasonality [as represented by the (July–January) differences] differ, their trends agree with each other very well. The polar amplification ratio of average temperature trend north of 65°N to that over global land (excluding Greenland and Antarctica) is also similar among different products, with the annual ratio of around 1.7. The ratio in summer (June–August) is always smaller than the annual value for different periods among all products. Based on the probability distribution functions from the monthly anomalies of different variables, the coldest tenth percentile of temperature in each decade overall increases with time, while the warmest tenth percentile does not vary much from 1950–1979, followed by a rapid increase from 1980–2009. These results and additional sensitivity tests suggest that the 4‐h LSAT products can be widely used for climate analysis, model evaluation, and offline land surface modelling from 1948–2009.

Wang, Aihui↗

Global Chemical Composition of Ambient Fine Particulate Matter for Exposure Assessment

Epidemiologic and health impact studies are inhibited by the paucity of global, long-term measurements of the chemical composition of fine particulate matter. We inferred PM 2.5 chemical composition at 0.1° × 0.1° spatial resolution for 2004–2008 by combining aerosol optical depth retrieved from the MODIS and MISR satellite instruments, with coincident profile and composition information from the GEOS-Chem global chemical transport model. Evaluation of the satellite-model PM 2.5 composition data set with North American in situ measurements indicated significant spatial agreement for secondary inorganic aerosol, particulate organic mass, black carbon, mineral dust, and sea salt. We found that global population-weighted PM 2.5 concentrations were dominated by particulate organic mass (11.9 ± 7.3 μg/m 3 ), secondary inorganic aerosol (11.1 ± 5.0 μg/m 3 ), and mineral dust (11.1 ± 7.9 μg/m 3 ). Secondary inorganic PM 2.5 concentrations exceeded 30 μg/m 3 over East China. Sensitivity simulations suggested that population-weighted ambient PM 2.5 from biofuel burning (11 μg/m 3 ) could be almost as large as from fossil fuel combustion sources (17 μg/m 3 ). In conclusion, these estimates offer information about global population exposure to the chemical components and sources of PM 2.5 .

54 ENVIRONMENTAL SCIENCES↗

Practical Use of Computationally Frugal Model Analysis Methods

Computationally frugal methods of model analysis can provide substantial benefits when developing models of groundwater and other environmental systems. Model analysis includes ways to evaluate model adequacy and to perform sensitivity and uncertainty analysis. Frugal methods typically require 10s of parallelizable model runs; their convenience allows for other uses of the computational effort. We suggest that model analysis be posed as a set of questions used to organize methods that range from frugal to expensive (requiring 10,000 model runs or more). This encourages focus on method utility, even when methods have starkly different theoretical backgrounds. We note that many frugal methods are more useful when unrealistic process-model nonlinearities are reduced. Inexpensive diagnostics are identified for determining when frugal methods are advantageous. Examples from the literature are used to demonstrate local methods and the diagnostics. We suggest that the greater use of computationally frugal model analysis methods would allow questions such as those posed in this work to be addressed more routinely, allowing the environmental sciences community to obtain greater scientific insight from the many ongoing and future modeling efforts

54 ENVIRONMENTAL SCIENCES↗

Spectral Data Fusion From Handheld Laser-Induced Breakdown Spectroscopy (LIBS) and X-ray Fluorescence (XRF) Analyzers for Improved Detection of Cerium in a Simulated Dispersal Accident

Here, this work implements a mid-level data fusion methodology on spectral data from handheld X-ray fluorescence and laser-induced breakdown spectroscopy analyzers to quantify plutonium surrogate (CeO 2 ) contamination in soil samples for the first time. Spectral data from each analyzer were used independently to train supervised machine learning regressions to predict Ce concentration. Fused features from both data sets were then used to train the same models, comparing prediction performance by evaluating model precision and sensitivity. Fusing principal component scores from the two sensors yielded an order of magnitude improvement in precision and sensitivity of predictions made with an artificial neural network, compared to predictions made by models trained on independent sensor data. As a result, a boosted ensemble trained on the fused spectral features yielded an ideal predictor with root-mean-squared error on the order of 10 –6 and calculated limit of detection order 10 –5 wt %.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗

The lunar swirls - Distribution and possible origins

Correlative data analysis and theoretical modeling methods are used to evaluate models involving both meteoroid and cometary impact processes which can explain the correlation in location of the strongest lunar magnetic anomalies with swirl-like high-albedo and low-albedo markings of the Reiner Gamma class. The results indicate that the relatively strong magnetization of basin antipode zones may have favored the formation of swirls in the same regions. The solar wind deflection model is tested by calculating the trajectories of simulated solar wind ions deflected by the Lorentz force in the presence of model crustal magnetic fields.

Hood, L. L.↗

A critical comparison of second order closures with direct numerical simulation of homogeneous turbulence

Recently, several second order closure models have been proposed for closing the second moment equations, in which the velocity-pressure gradient (and scalar-pressure gradient) tensor and the dissipation rate tensor are two of the most important terms. In the literature, these correlation tensors are usually decomposed into a so called rapid term and a return-to-isotropy term. Models of these terms have been used in global flow calculations together with other modeled terms. However, their individual behavior in different flows have not been fully examined because they are un-measurable in the laboratory. Recently, the development of direct numerical simulation (DNS) of turbulence has given us the opportunity to do this kind of study. With the direct numerical simulation, we may use the solution to exactly calculate the values of these correlation terms and then directly compare them with the values from their modeled formulations (models). Here, we make direct comparisons of five representative rapid models and eight return-to-isotropy models using the DNS data of forty five homogeneous flows which were done by Rogers et al. (1986) and Lee et al. (1985). The purpose of these direct comparisons is to explore the performance of these models in different flows and identify the ones which give the best performance. The modeling procedure, model constraints, and the various evaluated models are described. The detailed results of the direct comparisons are discussed, and a few concluding remarks on turbulence models are given.

Shih, Tsan-Hsing↗

Critical comparison of second-order closures with direct numerical simulations of homogeneous turbulence

Recently, several models have been proposed for closing the second-moment equations, in which the velocity-pressure gradient tensor and the dissipation rate tensor are two of the most important terms. In the literature, these correlation tensors are usually decomposed into a so-called rapid term and a return-to-isotropy term. Models of these terms have been used in global flow calculations together with other modeled terms. However, their individual behaviors in different flows have not been fully examined because they are unmeasurable in the laboratory. Recently, the development of direct numerical simulation (DNS) of turbulence has given us the possibility to do this kind of study. With direct numerical simulation, we may use the solution to calculate exactly the values of these correlation terms and then directly compare them with the values from their modeled formulations. In this paper, we make direct comparisons of five representative rapid models and eight return-to-isotropy models using the DNS data of 45 homogeneous flows, which were done by Rogers et al. (1986) and Lee and Reynolds (1985). The purpose of these direct comparisons is to explore the performance of these models in different flows and identify the ones that give the best performance. The paper also describes the modeling procedure, model constraints, and the various evaluated models. The detailed results of the direct comparisons are discussed, and a few concluding remarks on turbulence models are given.

Shih, Tsan-Hsing↗

Global Observations - The Key to Model Development and Improved Assessments

One application of global models is to predict the response of stratospheric ozone to changes in composition and climate. The recent international ozone assessment included results from three-dimensional models with interactions among the dynamical, photochemical and radiative processes that all influence ozone behavior. The physical basis of such models is far more realistic than that of either the one-dimensional (single profile) models of the 1970's and early 1980's or the two-dimensional (latitude height) models of the late 1980's and 1990's. Observations have played a key role in the model development at all stages. This talk will highlight the role of observations in inspiring broad model improvements that have grown from the effort to reproduce observed relationships or processes, for example the correlations between long-lived constituents seen in aircraft data and the deep unmixed descent of mesospheric air into the winter polar vortices seen from satellite. The talk will also trace the evolution of model evaluation from contour plots showing 'good agreement' to the more rigorous process-oriented evaluation of three-dimensional models that is becoming the norm using the wealth of space-based observations obtained from the late 1970's until present.

Douglass, Anne R.↗

The Applied Meteorology Unit: Nineteen Years Successfully Transitioning Research into Operations for America's Space Program

The Applied Meteorology Unit (AMU) provides technology transition and technique development to improve operational weather support to the Space Shuttle and the entire American space program. The AMU is funded and managed by NASA and operated by a contractor that provides five meteorologists with a diverse mix of advanced degrees, operational experience, and associated skills including data processing, statistics, and the development of graphical user interfaces. The AMU's primary customers are the U.S. Air Force 45th Weather Squadron at Patrick Air Force Base, the National Weather Service Spaceflight Meteorology Group at NASA Johnson Space Center, and the National Weather Service Melbourne FL Forecast Office. The AMU has transitioned research into operations for nineteen years and worked on a wide range of topics, including new forecasting techniques for lightning probability, synoptic peak winds,.convective winds, and summer severe weather; satellite tools to predict anvil cloud trajectories and evaluate camera line of sight for Space Shuttle launch; optimized radar scan strategies; evaluated and implemented local numerical models; evaluated weather sensors; and many more. The AMU has completed 113 projects with 5 more scheduled to be completed by the end of 2010. During this rich history, the AMU and its customers have learned many lessons on how to effectively transition research into operations. Some of these lessons learned include collocating with the operational customer and periodically visiting geographically separated customers, operator submitted projects, consensus tasking process, use of operator primary advocates for each project, customer AMU liaisons with experience in both operations and research, flexibility in adapting the project plan based on lessons learned during the project, and incorporating training and other transition assistance into the project plans. Operator involvement has been critical to the AMU's remarkable success and many awards from NASA, the National Weather Association, and two citations from the Navy's Center of Excellence for Best Manufacturing Practices. This paper will present the AMU's proven methods and explain how they may be applied by other organizations to effectively transition research into operations.

Madura, John T.↗

Space Weather Models, Tools and Services at the Community Coordinated Modeling Center

The Community Coordinated Modeling center (CCMC) is a multi-agency partnership to support the research and developmental work necessary to substantially increase space weather modeling capabilities and to facilitate advanced models deployment in forecasting operations. The CCMC conducts unbiased model testing and validation and evaluates model readiness for operational environment. Space weather models and coupled model chains hosted at the CCMC range from the solar corona to the Earth's upper atmosphere. CCMC has developed a number of real-time modeling systems, as well as a large number of modeling and data products tailored to address the space weather needs of NASA's robotic missions. The presentation will demonstrate the rapid progress towards development the system allowing using products derived from space weather models in applications associated with National Space Weather needs. The adaptable Integrated Space Weather Analysis (ISWA) System developed at CCMC for NASA-relevant space weather information combines forecasts based on advanced space weather models hosted at CCMC with concurrent space environment information. The system is also enabling post-impact analysis and flexible dissemination of space weather information.

Kuznetsova, M. M.↗

Validation of Space Weather Models at Community Coordinated Modeling Center

The Community Coordinated Modeling Center (CCMC) is a multi-agency partnership to support the research and developmental work necessary to substantially increase space weather modeling capabilities and to facilitate advanced models deployment in forecasting operations. Space weather models and coupled model chains hosted at the CCMC range from the solar corona to the Earth's upper atmosphere. CCMC has developed a number of real-time modeling systems, as well as a large number of modeling and data products tailored to address the space weather needs of NASA's robotic missions. The CCMC conducts unbiased model testing and validation and evaluates model readiness for operational environment. CCMC has been leading recent comprehensive modeling challenges under GEM, CEDAR and SHINE programs. The presentation will focus on experience in carrying out comprehensive and systematic validation of large sets of. space weather models

Kuznetsova, M. M.↗

Validation of Space Weather Models at Community Coordinated Modeling Center

The Community Coordinated Modeling Center (CCMC) is a multiagency partnership, which aims at the creation of next generation space weather modes. CCMC goal is to support the research and developmental work necessary to substantially increase space weather modeling capabilities and to facilitate advanced models deployment in forecasting operations. The CCMC conducts unbiased model testing and validation and evaluates model readiness for operational environment. The presentation will demonstrate the recent progress in CCMC metrics and validation activities.

Kuznetsova, M. M.↗

Real-Time Kennedy Space Center and Cape Canaveral Air Force Station High-Resolution Model Implementation and Verification

Customer: NASA's Launch Services Program (LSP), Ground Systems Development and Operations (GSDO), and Space Launch System (SLS) programs. NASA's LSP, GSDO, SLS and other programs at Kennedy Space Center (KSC) and Cape Canaveral Air Force Station (CCAFS) use the daily and weekly weather forecasts issued by the 45th Weather Squadron (45 WS) as decision tools for their day-to-day and launch operations on the Eastern Range (ER). For example, to determine if they need to limit activities such as vehicle transport to the launch pad, protect people, structures or exposed launch vehicles given a threat of severe weather, or reschedule other critical operations. The 45 WS uses numerical weather prediction models as a guide for these weather forecasts, particularly the Air Force Weather Agency (AFWA) 1.67 kilometer Weather Research and Forecasting (WRF) model. Considering the 45 WS forecasters' and Launch Weather Officers' (LWO) extensive use of the AFWA model, the 45 WS proposed a task at the September 2013 Applied Meteorology Unit (AMU) Tasking Meeting requesting the AMU verify this model. Due to the lack of archived model data available from AFWA, verification is not yet possible. Instead, the AMU proposed to implement and verify the performance of an ER version of the AMU high-resolution WRF Environmental Modeling System (EMS) model (Watson 2013) in real-time. The tasking group agreed to this proposal; therefore the AMU implemented the WRF-EMS model on the second of two NASA AMU modeling clusters. The model was set up with a triple-nested grid configuration over KSC/CCAFS based on previous AMU work (Watson 2013). The outer domain (D01) has 12-kilometer grid spacing, the middle domain (D02) has 4-kilometer grid spacing, and the inner domain (D03) has 1.33-kilometer grid spacing. The model runs a 12-hour forecast every hour, D01 and D02 domain outputs are available once an hour and D03 is every 15 minutes during the forecast period. The AMU assessed the WRF-EMS 1.33-kilometer domain model performance for the 2014 warm season (May-September). Verification statistics were computed using the Model Evaluation Tools, which compared the model forecasts to observations. The mean error values were close to 0 and the root mean square error values were less than 1.8 for mean sea-level pressure (millibars), temperature (degrees Kelvin), dewpoint temperature (degrees Kelvin), and wind speed (per millisecond), all very small differences between the forecast and observations considering the normal magnitudes of the parameters. The precipitation forecast verification results showed consistent under-forecasting of the precipitation object size. This could be an artifact of calculating the statistics for each hour rather than for the entire 12-hour period. The AMU will continue to generate verification statistics for the 1.33-kilometer WRF-EMS domain as data become available in future cool and warm seasons. More data will produce more robust statistics and reveal a more accurate assessment of model performance. Once the formal task was complete, the AMU conducted additional work to better understand the wind direction results. The results were stratified diurnally and by wind speed to determine what effects the stratifications would have on the model wind direction verification statistics. The results are summarized in the addendum at the end of this report. In addition to verifying the model's performance, the AMU also made the output available in the Advanced Weather Interactive Processing System II (AWIPS II). This allows the 45 WS and AMU staff to customize the model output display on the AMU and Range Weather Operations AWIPS II client computers and conduct real-time subjective analyses. In the future, the AMU will implement an updated version of the WRF-EMS model that incorporates local data assimilation. This model will also run in real-time and be made available in AWIPS II.

Numerical Weather Prediction↗

Multi-Fidelity Heating Prediction of Adaptable, Deployable Entry Placement Technology Vehicles

The objective of this work was to investigate a multi-fidelity modeling approach to accurately and efficiently predict the laminar and turbulent convective heating on adaptable, deployable entry placement technology vehicles in Mars entry. A previously developed co-Kriging based multi-fidelity modeling approach was used to model the laminar and turbulent convective heat fluxes at several surface locations along the vehicle, including the rib sections. The laminar convective heat flux multi-fidelity model was found to have a mean convective heat rate error of approximately 3% when compared to high-fidelity CFD simulations. The turbulent convective heat flux multi-fidelity model was found to have a mean convective heat rate error of approximately 8% when compared to high-fidelity CFD simulations. Compared to a single fidelity model, the multi-fidelity model required approximately one-third the number of high-fidelity model evaluations to obtain the same accuracy level. The computational cost of evaluating the multi-fidelity model was approximately five orders of magnitude less than one high-fidelity model simulation.

Mario Santos↗

Marin County Wildland Fires: Examining Fuel Load and Land Cover Change to Inform Fire Prevention and Suppression Decisions in Marin County, CA

Heightened occurrence of severe wildfires in the Western United States is increasing the need to better understand regions of high potential wild fire severity and develop methodologies for identifying the best locations for fuels reduction and active wildfire suppression, especially in populated regions such as Marin County, California. Marin County, located in the San Francisco Bay Area, has had significant development in the wildland-urban interface and periods of highly wildfire-prone conditions. The NASA DEVELOP team collaborated with Fire Foundry, a Marin-based fire service work force development program, to develop new models to assist with fire management. Using data from Sentinel-2A, Planet Scope, ECOSTRESS, a county-wide LiDAR mapping effort, Landsat 7 Enhanced Thematic Mapper (ETM+), and Landsat 8 Operational Land Imager (OLI), the team developed several input data layers to three models evaluating wild fire severity. One model performed a suitability analysis with weights based on scientific literature, another utilized machine learning based on past fires in Marin and neighboring Sonoma County to predict the difference normalized burn ratio, and the third inputted data layers into the Flam Map tool, which outputs risk categories. The team compared model outputs and, using the best-fit model, performed fuzzy logic analysis to identify specific locations where a fire break could be constructed to interrupt the progress of an active fire. These tools were proven useful and will assist partners in preparing for and managing an active wildfire event.

Suhani Dalal↗

Bayesian Monte Carlo Evaluation of Imperfect (n, 233 U) Data and Model

Conventional nuclear data evaluation methods using generalized linear least squares make the following assumptions: prior and posterior probability distribution functions (PDFs) of all model parameters and data are normal (Gaussian); the linear approximation is sufficiently accurate to minimize the cost function (even for nonlinear models); the model (e.g., of neutron cross section) and experimental data (including covariance data) are without defect and prior PDFs of parameters and measured data are known perfectly. Neglect of covariance between model parameters and measured data in conventional evaluations contributes to imperfections. These assumptions are inherent to the generalized linear least squares minimization method commonly used for resolved resonance region neutron cross section evaluations but are often not justified due to the presence of non-normal PDFs, nonlinear models (e.g., R-matrix formalism), and inherent imperfections in data and models (e.g., imperfect covariance data). Here, these assumptions are removed in a mathematical framework of Bayes’ theorem, which is implemented using the Metropolis-Hastings Monte Carlo method. Most importantly, new parameters are introduced to parameterize discrepancies between the theoretical model and measured data to quantify judgement about discrepancies or imperfections in a reproducible manner. An evaluation of 233U in the eV region using the ENDF-B/VIII.0 library and transmission data (Guber et al.) is presented, and posterior parameters are compared to those obtained by conventional evaluation methods. This example illustrates the effects of removing the most harmful assumption: that of model-data perfection.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗

Legacy of the Madden-Julian Oscillation Working Group

There is a long history of observational and model evaluation of the MJO. Even so, the underlying physics of the MJO remains poorly understood. Initial analyses of the MJO in models concentrated on large-scale diagnosis using such variables as velocity potential and/or zonal winds, and occasionally precipitation or outgoing longwave radiation. The first time a large number of models were systematically evaluated using a common set of diagnostics was in Slingo et al. (1996), who evaluated 15 general circulation models that were contributed to the Atmospheric Model Intercomparison Project (AMIP; Gates et al. 1992). However, the high- LLNL-TR-762421 frequency (daily) model data available to Slingo et al. (1996) were limited, and it was only with the advent of NWP-based reanalyses that a more comprehensive understanding of the 3-dimensional structure and the thermodynamics of the MJO became possible, thus helping to establish the variables relevant to the study of the MJO. International interest in the MJO began to blossom on the heels of the 2006 “Workshop on the Organization and Maintenance of Tropical Convection and the Madden Julian Oscillation,” held at the International Center for Theoretical Physics in Trieste, Italy. With its stated goals of understanding the influence of MJO in the context of the tropical circulation and its remote impacts, and discussion of future directions including the MJO impact on predictive capability, the time was right for a more coordinated attack on understanding and simulating the MJO.

58 GEOSCIENCES↗