SEARCH · Search NASA
Results for “model credibility”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
Simplifying and Visualizing the Ontology of Systems Engineering Models
The credibility of an engineering model is of critical importance in large-scale projects. How concerned should an engineer be when reusing someone else's model when they may not know the author or be familiar with the tools that were used to create it? In this report, the authors advance engineers' capabilities for assessing models through examination of the underlying semantic structure of a model--the ontology. This ontology defines the objects in a model, types of objects, and relationships between them. In this study, two advances in ontology simplification and visualization are discussed and are demonstrated on two systems engineering models. These advances are critical steps toward enabling engineering models to interoperate, as well as assessing models for credibility. For example, results of this research show an 80% reduction in file size and representation size, dramatically improving the throughput of graph algorithms applied to the analysis of these models. Finally, four future problems are outlined in ontology research toward establishing credible models--ontology discovery, ontology matching, ontology alignment, and model assessment.
Credible Practice of Models and Simulations for Healthcare
As computational models penetrate clinical care and drive healthcare policy decisions, there is increasing need to communicate that model creation and testing follows reasonable credible practice. Although verification and validation have been generally accepted as significant components of a model’s credibility, they cannot be assumed to equate to a holistic credible practice, which includes activities that can impact comprehension and in-depth examination inherent in the development and reuse of the models. This presentation introduces the Ten Rules for credible practice of modeling and simulation in healthcare developed from a comparative multidisciplinary analysis and practitioner survey. These rules establish a unified conceptual framework for modeling and simulation design, implementation, evaluation, dissemination and usage across the modeling and simulation life-cycle. Although some of these are common sense guidelines, many are often missed or misconstrued, even by seasoned practitioners.
Toward shell model interactions with credible uncertainties
Background: The nuclear shell model is a powerful framework for predicting nuclear structure observables, but relies on interaction matrix elements fit to experimental data as its inputs. Extending the shell model's applicability, particularly toward dripline nuclei, requires efficient fitting methods and credible uncertainty quantification. Traditional approaches face computational challenges and may underestimate uncertainties. Purpose: We develop and test a framework combining eigenvector continuation and Markov chain Monte Carlo to efficiently fit shell model interaction matrix elements and quantify their uncertainties. Methods: Eigenvector continuation is used to emulate shell model calculations, reducing computational costs. The emulator enables Markov chain Monte Carlo sampling to optimize interaction matrix elements and rigorously assess parametric uncertainties. Here, the framework is benchmarked using the USDB interaction in the 𝑠𝑑 shell. Results: The emulator reproduces the USDB interaction with negligible error, validating its use in shell model fitting applications. However, we find that to obtain credible predictive intervals, the model defect of the shell model itself, rather than experimental or emulator error, must be taken into account in order to obtain credible uncertainties. Conclusions: The proposed framework provides an efficient and rigorous approach for fitting shell model interactions and quantifying uncertainties. Further, the normality assumption used in the past appears sufficient to describe the distribution of interaction matrix elements. However, it is crucial to account for model correlations to avoid underestimating uncertainties.
Connecting Large‐Scale Meteorological Patterns to Extratropical Cyclones in CMIP6 Climate Models Using Self‐Organizing Maps
Abstract Extratropical cyclones (ETCs) are responsible for the majority of cool‐season extreme events in the northeastern United States (NEUS), often leading to high‐impact weather conditions that can have wide‐ranging socioeconomic impacts. Evaluating the ability of climate models to adequately simulate ETC dynamics is essential for improving model performance and increasing confidence in future projections used by stakeholders and policymakers. ETCs are traditionally studied using techniques such as case studies and synoptic typing, however, these approaches can be time‐consuming, require subjective analysis, and do not necessarily identify the coincident large‐scale meteorological patterns (LSMPs). Here, we apply self‐organizing maps (SOMs) as an automated machine‐learning approach to characterize the LSMPs and associated frequency and intensity of discrete ETC events over NEUS. The dominant patterns of geopotential height variability are identified through SOM analysis of five reanalysis products during the last four decades. ETC events are tracked using TempestExtremes and are integrated with SOMs to classify the accumulated cyclone activity (ACA) associated with each pattern. We then evaluate the skill of CMIP6 historical experiments in simulating the LSMPs and ETC events identified in the SOM. Our results identify a robust bias toward more zonal patterns, with models struggling to reproduce the more amplified patterns typically associated with the highest cyclone activity. While model resolution has some impact on simulation credibility, model configuration appears to be more important in LSMP representation. The vast majority of CMIP6 models produce too few ETCs, although model errors are distributed around historical reanalyses when ACA is normalized by storm frequency.
Identifying synoptic controls on boundary layer thermodynamic and cloud properties in a regional forecast model
Although most of our understanding of boundary layer cloudiness is based on idealized, subtropical, barotropic marine environments, boundary layer clouds exist across a range of conditions. In this study, we use the Naval Research Laboratory's Coupled Ocean/Atmosphere Mesoscale Prediction System (COAMPS) and an automated cold-front-relative analysis framework to explore the boundary layer structure associated with low clouds across a transect through the cold front of a midlatitude synoptic cyclone. The model credibly captures boundary layer structure in line with conceptual models of midlatitude cyclones and ground-based observations at Graciosa Island in the Azores. The warm sector is conditionally unstable, with clouds that are too shallow and with too little liquid water, compared to cloud property retrievals from satellite and surface-based instruments. The cold-frontal region exhibits convection associated with weak stability and ascent. Northwest of the cold front, the boundary layer is well mixed, deeper, and capped by a strong inversion maintained by large-scale subsidence. Simulated clouds in frontal and post-frontal regions are mostly too thick, with too much liquid water and too little cloud base drizzle. The post-frontal clouds are associated with grid-scale updrafts, which appear to be the model's attempt to represent mesoscale organization of cellular convection typically observed in the cold sector of midlatitude cyclones. The deep, well-mixed post-frontal boundary layers and cloudiness are maintained by strong surface fluxes, as in cold air outbreaks. Our analysis framework serves as a unique approach to model verification, and our results offer insights into differences in boundary layer cloud behavior between subtropical and synoptic cold-sector regimes.
Preliminary Results for Using Uncertainty and Out-of-distribution Detection to Identify Unreliable Predictions.
As machine learning (ML) models are deployed into an ever-diversifying set of application spaces, ranging from self-driving cars to cybersecurity to climate modeling, the need to carefully evaluate model credibility becomes increasingly important. Uncertainty quantification (UQ) provides important information about the ability of a learned model to make sound predictions, often with respect to individual test cases. However, most UQ methods for ML are themselves data-driven and therefore susceptible to the same knowledge gaps as the models themselves. Specifically, UQ helps to identify points near decision boundaries where the models fit the data poorly, yet predictions can score as certain for points that are under-represented by the training data and thus out-of-distribution (OOD). One method for evaluating the quality of both ML models and their associated uncertainty estimates is out-of-distribution detection (OODD). We combine OODD with UQ to provide insights into the reliability of the individual predictions made by an ML model.
Development and Future of the Ten Simple Rules for Credible Practice of Models and Simulations in HealthCare
In this session, we will review the development and implementation history of the Ten Simple Rules (TSR) for credible practice in models and simulations that support healthcare. Our focus will be initially on the need and background for the Committee on Credible Practice of Models and Simulations in Healthcare (CPMS) as well as the TSR development process and implementation. This will flow into a demonstration of how the TSR has evolved as means of communicating and understanding the model development process, likely successful application and where the model could fail. We will end this presentation with a discussion on future development avenue, the CPMS efforts to develop a comprehensive rubric supporting current TSR implementation, and efforts to support further adoption of the TSR. The audience will be encouraged to comment on the current state of the TSR, share their TSR application experiences, and provide their insights on future development.
A Distinct Atmospheric Mode for California Precipitation
The hydrologic cycle in California is strongly influenced by wet-season (November–April) precipitation. Here, we demonstrate the existence of an influential mode of North Pacific atmospheric pressure variability that regulates wet-season precipitation variability over both northern and southern California. This mode, named as the “California precipitation mode” (CPM), is statistically distinct from other well-known modes of pressure variability such as the Pacific-North American pattern. In addition to controlling wet-season mean precipitation, positive days of the CPM coincide with up to 90% of the extreme (>99th percentile) precipitation days and 76% of detected atmospheric rivers (ARs) days, while the negative days correspond with 60% of the dry days. CMIP6 models capture the CPM remarkably well, including its statistical separation from the other well-known modes of pressure variability. The models also reproduce the CPM's strong association with California wet-season precipitation, giving confidence in the models’ dynamics relating to regional hydrologic extremes. However, the models also exhibit biases in regional hydrologic extremes. The CPM is a useful way to understand the origins of those biases and select the more credible models for further analysis: Models with unrealistically strong gradients in the CPM pressure pattern generally oversimulate larger wet extremes and produce excessively long dry intervals in the historical period. Thus the hydrologic biases can be traced to the particular aspects of North Pacific atmospheric dynamics.
Composite Overwrapped Pressure Vessels (COPV): Developing Flight Rationale for the Space Shuttle Program
Introducing composite vessels into the Space Shuttle Program represented a significant technical achievement. Each Orbiter vehicle contains 24 (nominally) Kevlar tanks for storage of pressurized helium (for propulsion) and nitrogen (for life support). The use of composite cylinders saved 752 pounds per Orbiter vehicle compared with all-metal tanks. The weight savings is significant considering each Shuttle flight can deliver 54,000 pounds of payload to the International Space Station. In the wake of the Columbia accident and the ensuing Return to Flight activities, the Space Shuttle Program, in 2005, re-examined COPV hardware certification. Incorporating COPV data that had been generated over the last 30 years and recognizing differences between initial Shuttle Program requirements and current operation, a new failure mode was identified, as composite stress rupture was deemed credible. The Orbiter Project undertook a comprehensive investigation to quantify and mitigate this risk. First, the engineering team considered and later deemed as unfeasible the option to replace existing all flight tanks. Second, operational improvements to flight procedures were instituted to reduce the flight risk and the danger to personnel. Third, an Orbiter reliability model was developed to quantify flight risk. Laser profilometry inspection of several flight COPVs identified deep (up to 20 mil) depressions on the tank interior. A comprehensive analysis was performed and it confirmed that these observed depressions were far less than the criterion which was established as necessary to lead to liner buckling. Existing fleet vessels were exonerated from this failure mechanism. Because full validation of the Orbiter Reliability Model was not possible given limited hardware resources, an Accelerated Stress Rupture Test of a flown flight vessel was performed to provide increased confidence. A Bayesian statistical approach was developed to evaluate possible test results with respect to the model credibility and thus flight rationale for continued operation of the Space Shuttle with existing flight hardware. A non-destructive evaluation (NDE) technique utilizing Raman Spectroscopy was developed to directly measure the overwrap residual stress state. Preliminary results provide optimistic results that patterns of fluctuation in fiber elastic strains over the outside vessel surface could be directly correlated with increased fiber stress ratios and thus reduced reliability.
Data Fusion for the Development of a Multimodal Freight Transload Facilities Dataset in the U.S.
To withstand the growing demand of commodity volume and its strain on the transportation infrastructure, it is necessary to identify the flow of commodities by route and mode. However, a national multimodal freight routing model does not exist for the U.S. The development of such model requires multiple building blocks, such as virtual representations of roadway, railway, and waterway networks, transload facilities (TFs), and access/egress links. Most of these blocks have a robust database in the U.S., except for the TFs. Here, this paper presents the fusion of dispersed and heterogeneous representations of multimodal TFs into a single, comprehensive, geospatial freight TF dataset. The TF dataset is derived from several sources, including the U.S. Army Corps of Engineers Master Docks Plus, the National Transportation Atlas Database, the Intermodal Association of North America, industry publications, and other public information. First, individual datasets were queried and reconciled. A geocoding/reverse geocoding process was applied to get the best street address and latitude/longitude location for each terminal. Then, duplicate terminals were identified by a fuzzy match algorithm based on terminal name and location, and removed. Validation was performed by visual inspection of random facilities. The main contributions of this work are: a publicly available version of the TF dataset, including facility location and multimodal transfer capability of 9,003 facilities, and an enterprise-version with the same facilities but including commodity handling capabilities. The main purpose of developing the TF dataset is to inform multimodal routing algorithms. The proposed TF dataset allows for credibly modeling the multimodal transfer of commodities within shipment routes.
Investigations of Mechanisms Associated with Nucleate Boiling Under Microgravity Conditions
In this work a building block type of approach is used so that a basic understanding of the processes that contribute to nucleate boiling heat fluxes under microgravity conditions can be developed. This understanding will lead to development of a mechanistic model for nucleate boiling heat transfer which could eventually be used as a design tool in space applications. Task Description Task 1: Fabrication of the Experimental Setup. Under this task, the test section and liquid holding and viewing chambers will be fabricated. Artificial cylinder cavities will be formed on silicon wafers. A single cavity and two or four cavities, with a prescribed spacing and size, will be formed. The desired nucleation wall superheat will be used to determine the size of the mouth of the cavities. Task 2: Experiments. The basic experiments for flow and temperature field around single and multiple (2 or 4 separated or merged bubbles growing on downward facing or inclined surfaces) will be carried out under normal gravity conditions. The experiments will be conducted at one atmosphere pressure, but liquid subcooling will be varied from 0 to 30C. Water and PF-5050 will be used as test liquids. Task 3: Analytical/Numerical Models. In this task, transient temperature and flow field in vapor and liquid will be determined during growth of a single bubble. Analysis will include the evolution of the vapor-liquid interface and development of microlayer underneath the bubbles. For merged bubbles, detailed calculations of flow and temperature field will be carried out for transient shapes of vapor stems supporting a large bubble and the corresponding evaporation rate. Flow and temperature field for a bubble sliding along a heated wall will also be determined. Microgravity conditions will be simulated and a framework of a numerical tool for prediction of nucleate boiling heat fluxes under microgravity conditions will be developed. Task 4: Experiments in a KC-135. To understand bubble growth and detachment behavior of single or large merged bubbles, boiling experiments will be conducted under low gravity (10-2 g) conditions of the aircraft. In these experiments, 'designed' surfaces will be used. Visual observations and heat transfer data will be taken, but holography will not be used. The apparatus used for laboratory experiments will also be employed for experiments in the aircraft. Task 5: Experiments in the Space Shuttle. Effort will be devoted for defining a boiling experiment to be conducted on a 'designed' surface. The experiment will provide microgravity data on bubble growth and departure. These data are needed for development of a credible model for nucleate boiling heat fluxes under microgravity conditions. The heat transfer data will also be obtained and will be used to validate the models.
Community-Engaged Modeling of Urban Flood Adaptation Pathways
Climate change is intensifying the hydrologic cycle, leading to more frequent and severe rainfall-driven (pluvial) flooding in urban areas. In the mid-Atlantic US cities, aging and under-designed stormwater infrastructure is increasingly strained by these events, resulting in recurring damage to property and disruptions to transportation networks. In this study, we combine community engagement with hydrologic modeling to develop and evaluate potential urban flood adaptation strategies. Over a three-year period, local technical experts and community representatives met regularly to discuss flooding concerns, identify priorities, and co-develop adaptation strategies. These discussions informed the development of an urban flooding model (EPA Storm Water Management Model) for the Baltimore Harbor watershed, the focus location of this study. The flooding model integrates complex surface and subsurface stormwater infrastructure data, local expert knowledge, and community insights. We simulate stakeholder-prioritized adaptations, such as green and gray infrastructure strategies. Model results demonstrate that enhanced infrastructure maintenance is the most effective adaptation for reducing flood depths, but has varied effects across the watershed, and can increase flooding in some locations. Spatially concentrated greening provides limited benefit to the watershed as a whole, but moderate benefit in community priority areas. Together, these adaptations have the potential to reduce flood depths by as much as 58% in some locations, greatly reducing property damage and mobility impacts, primary concerns of stakeholders. Future work will implement robust optimization tools to search for adaptations which meet stakeholder objectives and perform highly under varied future climate conditions. This work contributes to the expanding literature on collaborative modeling and demonstrates that community-engaged approaches can enhance model credibility and generate more actionable insights for communities seeking to strengthen climate resilience.
Community-Informed Urban Flood Modeling for Impact Mitigation
Climate change is intensifying the hydrologic cycle, leading to more frequent and severe rainfall-driven (pluvial) flooding in urban areas. In the mid-Atlantic US cities, aging and under-designed stormwater infrastructure is increasingly strained by these events, resulting in recurring damage to property and disruptions to transportation networks. In this study, we combine community engagement with hydrologic modeling to develop and evaluate potential urban flood adaptation strategies. Over a three-year period, local technical experts and community representatives met regularly to discuss flooding concerns, identify priorities, and co-develop adaptation strategies. These discussions informed the development of an urban flooding model (EPA Storm Water Management Model) for the Baltimore Harbor watershed, the focus location of this study. The flooding model integrates complex surface and subsurface stormwater infrastructure data, local expert knowledge, and community insights. We simulate stakeholder-prioritized adaptations, such as green and gray infrastructure strategies. Model results demonstrate that enhanced infrastructure maintenance is the most effective adaptation for reducing flood depths, but has varied effects across the watershed, and can increase flooding in some locations. Spatially concentrated greening provides limited benefit to the watershed as a whole, but moderate benefit in community priority areas. Together, these adaptations have the potential to reduce flood depths by as much as 58% in some locations, greatly reducing property damage and mobility impacts, primary concerns of stakeholders. Future work will implement robust optimization tools to search for adaptations which meet stakeholder objectives and perform highly under varied future climate conditions. This work contributes to the expanding literature on collaborative modeling and demonstrates that community-engaged approaches can enhance model credibility and generate more actionable insights for communities seeking to strengthen climate resilience.
CFD code validation for nozzle flowfields
Renewed interest in hypersonic airbreathing flight has resulted in heavy reliance on computational fluid dynamics for all aspects of aerodynamic and scramjet component modeling. Credible numerical performance predictions require calibration to unit type flowfields. This paper presents results of one such calibration to nonreacting simulated scramjet nozzle flowfields. The experimental test program was performed on a static thrust stand. Computed results are in excellent agreement with measured wall pressure, pitot pressure profiles, and thrust. Agreement with thrust is shown to be within the reported accuracy of the experimental measurements.
Integrated Medical Model Verification, Validation, and Credibility
The Integrated Medical Model (IMM) was designed to forecast relative changes for a specified set of crew health and mission success risk metrics by using a probabilistic (stochastic process) model based on historical data, cohort data, and subject matter expert opinion. A probabilistic approach is taken since exact (deterministic) results would not appropriately reflect the uncertainty in the IMM inputs. Once the IMM was conceptualized, a plan was needed to rigorously assess input information, framework and code, and output results of the IMM, and ensure that end user requests and requirements were considered during all stages of model development and implementation. METHODS: In 2008, the IMM team developed a comprehensive verification and validation (VV) plan, which specified internal and external review criteria encompassing 1) verification of data and IMM structure to ensure proper implementation of the IMM, 2) several validation techniques to confirm that the simulation capability of the IMM appropriately represents occurrences and consequences of medical conditions during space missions, and 3) credibility processes to develop user confidence in the information derived from the IMM. When the NASA-STD-7009 (7009) was published, the IMM team updated their verification, validation, and credibility (VVC) project plan to meet 7009 requirements and include 7009 tools in reporting VVC status of the IMM. RESULTS: IMM VVC updates are compiled recurrently and include 7009 Compliance and Credibility matrices, IMM VV Plan status, and a synopsis of any changes or updates to the IMM during the reporting period. Reporting tools have evolved over the lifetime of the IMM project to better communicate VVC status. This has included refining original 7009 methodology with augmentation from the NASA-STD-7009 Guidance Document. End user requests and requirements are being satisfied as evidenced by ISS Program acceptance of IMM risk forecasts, transition to an operational model and simulation tool, and completion of service requests from a broad end user consortium including Operations, Science and Technology Planning, and Exploration Planning. CONCLUSIONS: The VVC approach established by the IMM project of combining the IMM VV Plan with 7009 requirements is comprehensive and includes the involvement of end users at every stage in IMM evolution. Methods and techniques used to quantify the VVC status of the IMM have not only received approval from the local NASA community but have also garnered recognition by other federal agencies seeking to develop similar guidelines in the medical modeling community.
How to Develop and Interpret a Credibility Assessment of Numerical Models for Human Research: NASA-STD-7009 Demystified
In the wake of the Columbia accident, the NASA-STD-7009 [1] credibility assessment was developed as a unifying platform to describe model credibility and the uncertainties in its modeling predictions. This standard is now being adapted by NASAs Human Research Program to cover a wide range of numerical models for human research. When used properly, the standard can improve the process of code development by encouraging the use of best practices. It can also give management more insight in making informed decisions through a better understanding of the models capabilities and limitations.To a newcomer, the abstractions presented in NASA-STD-7009 and the sheer volume of information that must be absorbed can be overwhelming. This talk is aimed at describing the credibility assessment, which is the heart of the standard, in plain terms. It will outline how to develop a credibility assessment under the standard. It will also show how to quickly interpret the graphs and tables that result from the assessment and how to drill down from the top-level view to the foundation of the assessment. Finally, it will highlight some of the resources that are available for further study.
Verification, Validation and Credibility Assessment of a Computational Model of the Advanced Resistive Exercise Device (ARED)
The Advanced Resistive Exercise Device (ARED) is the resistive exercise device used by astronauts on the International Space Station (ISS) to mitigate bone loss and muscle atrophy due to extended exposure to microgravity (micro g). The Digital Astronaut Project (DAP) has developed a multi-body dynamics model of biomechanics models for use in spaceflight exercise physiology research and operations. In an effort to advance model maturity and credibility of the ARED model, the DAP performed verification, validation and credibility (VV and C) assessment of the analyses of the model in accordance to NASA-STD-7009 'Standards for Models and Simulations'.