Search NASA⌕ Search

SEARCH · Search NASA

Results for “Evaluation Framework”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records
All sourcesNASA NTRSDOE OSTI
NASA NTRS1992

Proposed evaluation framework for assessing operator performance with multisensor displays

Despite aggressive work on the development of sensor fusion algorithms and techniques, no formal evaluation procedures have been proposed. Based on existing integration models in the literature, an evaluation framework is developed to assess an operator's ability to use multisensor, or sensor fusion, displays. The proposed evaluation framework for evaluating the operator's ability to use such systems is a normative approach: The operator's performance with the sensor fusion display can be compared to the models' predictions based on the operator's performance when viewing the original sensor displays prior to fusion. This allows for the determination as to when a sensor fusion system leads to: 1) poorer performance than one of the original sensor displays (clearly an undesirable system in which the fused sensor system causes some distortion or interference); 2) better performance than with either single sensor system alone, but at a sub-optimal (compared to the model predictions) level; 3) optimal performance (compared to model predictions); or, 4) super-optimal performance, which may occur if the operator were able to use some highly diagnostic 'emergent features' in the sensor fusion display, which were unavailable in the original sensor displays. An experiment demonstrating the usefulness of the proposed evaluation framework is discussed.

Foyle, David C.↗
NASA NTRS1999

Evaluation Framework for NASA's Educational Outreach Programs

The objective of the proposed work is to develop an evaluation framework for NASA's educational outreach efforts. We focus on public (rather than technical or scientific) dissemination efforts, specifically on Internet-based outreach sites for children.The outcome of this work is to propose both methods and criteria for evaluation, which would enable NASA to do a more analytic evaluation of its outreach efforts. The proposed framework is based on IRL's ethnographic and video-based observational methods, which allow us to analyze how these sites are actually used.

Berg, Rick↗
NASA NTRSDate not supplied

A Comprehensive eVTOL Performance Evaluation Framework in Urban Air Mobility

In this paper, we developed an open-source simulation framework for the evaluation of electric vertical takeoff and landing vehicles (eVTOLs) in the context of Unmanned Traffic Management (UTM) and under the concept of Urban Air Mobility (UAM). Unlike most existing studies, the proposed framework combines the utilization of UTM and eVTOLs to develop a realistic UAM testing platform. For this purpose, we first develop an UTM simulator to simulate the real-world UAM environment. Then, instead of using a simplified eVOTL model, a high-fidelity eVTOL design tool, namely SUAVE, is employed and an dilation sub-module is introduced to bridge the gap between the UTM simulator and SUAVE eVTOL performance evaluation tool to elaborate the complete mission profile. Based on the developed simulation framework, experiments are conducted and the results are presented to analyze the performance of eVTOLs in the UAM environment.

Mrinmoy Sarkar↗
NASA NTRS2017

Assessment of Intralaminar Progressive Damage and Failure Analysis Using an Efficient Evaluation Framework

Reducing the timeline for development and certification for composite structures has been a long standing objective of the aerospace industry. This timeline can be further exacerbated when attempting to integrate new fiber-reinforced composite materials due to the large number of testing required at every level of design. computational progressive damage and failure analysis (PDFA) attempts to mitigate this effect; however, new PDFA methods have been slow to be adopted in industry since material model evaluation techniques have not been fully defined. This study presents an efficient evaluation framework which uses a piecewise verification and validation (V&V) approach for PDFA methods. Specifically, the framework is applied to evaluate PDFA research codes within the context of intralaminar damage. Methods are incrementally taken through various V&V exercises specifically tailored to study PDFA intralaminar damage modeling capability. Finally, methods are evaluated against a defined set of success criteria to highlight successes and limitations.

Hyder, Imran↗
NASA NTRS2023

A Framework for Evaluating Distributed Electric Propulsion on the SUSAN Electrofan Aircraft

This work presents a framework for evaluating models and algorithms for Distributed Electric Propulsion (DEP) on the SUSAN Electrofan Aircraft. Throughout the development of the SUSAN aircraft, the performance of various configurations of the aircraft will need to be analyzed. However, the static behavior alone is not sufficient to describe the performance of these configurations. Therefore, simulation with fully integrated subsystem models is required. The proposed framework considers the vehicle aerodynamic, propulsion, and control subsystems. The presented framework automatically generates control laws for any vehicle configuration in response to changes in these subsystems. To compare these different vehicle configurations, various time and frequency domain performance metrics are compared. Three different system modifications are used as cases to evaluate this framework. The first modification integrates the propulsion control system with the flight controller to enable differential thrust without stalling the main engine. This evaluation case is used to validate the framework for aircraft configurations with coupled subsystems. The second modification compares the effect of the vertical tail size on open and closed loop performance. This evaluation case is used to validate the framework for controlling different configurations and tuning towards comparable closed loop performance despite changes to the aircraft's aerodynamic model. The third modification implements two different control allocation schemes. This evaluation case demonstrates the framework's ability to evaluate allocation modifications needed to take advantage of DEP. The first evaluation case is used to show that controller integration enables differential thrust, improving realized wingfan bandwidth by up to 40\% in simulation. The second evaluation case demonstrates that the framework can stabilize the reduced tail size aircraft with closed loop control. The third evaluation case demonstrates that a pseudoinverse control allocation scheme improves lateral velocity settling time by approximately 17~seconds over a symmetric-thrust allocation. These cases show that the framework is useful for evaluating the performance of integrated system designs, enabling analyses of new models and algorithms for the SUSAN distributed electric propulsion vehicle.

Nicholas C Ogden↗
NASA NTRS1998

Evaluation of Frameworks for HSCT Design Optimization

This report is an evaluation of engineering frameworks that could be used to augment, supplement, or replace the existing FIDO 3.5 (Framework for Interdisciplinary Design and Optimization Version 3.5) framework. The report begins with the motivation for this effort, followed by a description of an "ideal" multidisciplinary design and optimization (MDO) framework. The discussion then turns to how each candidate framework stacks up against this ideal. This report ends with recommendations as to the "best" frameworks that should be down-selected for detailed review.

Krishnan, Ramki↗
NASA NTRS1993

Sensor fusion display evaluation using information integration models in enhanced/synthetic vision applications

Based on existing integration models in the psychological literature, an evaluation framework is developed to assess sensor fusion displays as might be implemented in an enhanced/synthetic vision system. The proposed evaluation framework for evaluating the operator's ability to use such systems is a normative approach: The pilot's performance with the sensor fusion image is compared to models' predictions based on the pilot's performance when viewing the original component sensor images prior to fusion. This allows for the determination as to when a sensor fusion system leads to: poorer performance than one of the original sensor displays, clearly an undesirable system in which the fused sensor system causes some distortion or interference; better performance than with either single sensor system alone, but at a sub-optimal level compared to model predictions; optimal performance compared to model predictions; or, super-optimal performance, which may occur if the operator were able to use some highly diagnostic 'emergent features' in the sensor fusion display, which were unavailable in the original sensor displays.

Foyle, David C.↗
NASA NTRS2017

SpaceCubeX: A Framework for Evaluating Hybrid Multi-Core CPU FPGA DSP Architectures

The SpaceCubeX project is motivated by the need for high performance, modular, and scalable on-board processing to help scientists answer critical 21st century questions about global climate change, air quality, ocean health, and ecosystem dynamics, while adding new capabilities such as low-latency data products for extreme event warnings. These goals translate into on-board processing throughput requirements that are on the order of 100-1,000 more than those of previous Earth Science missions for standard processing, compression, storage, and downlink operations. To study possible future architectures to achieve these performance requirements, the SpaceCubeX project provides an evolvable testbed and framework that enables a focused design space exploration of candidate hybrid CPU/FPGA/DSP processing architectures. The framework includes ArchGen, an architecture generator tool populated with candidate architecture components, performance models, and IP cores, that allows an end user to specify the type, number, and connectivity of a hybrid architecture. The framework requires minimal extensions to integrate new processors, such as the anticipated High Performance Spaceflight Computer (HPSC), reducing time to initiate benchmarking by months. To evaluate the framework, we leverage a wide suite of high performance embedded computing benchmarks and Earth science scenarios to ensure robust architecture characterization. We report on our projects Year 1 efforts and demonstrate the capabilities across four simulation testbed models, a baseline SpaceCube 2.0 system, a dual ARM A9 processor system, a hybrid quad ARM A53 and FPGA system, and a hybrid quad ARM A53 and DSP system.

Hybrid Flight Architectures↗
NASA NTRS2015

The AgMIP Framework to Evaluate Agricultural Pathways

This talk will describe the community and research framework that AgMIP has built to enable evidence-based adaptation investment. We provide expertise on the ground and connect various disciplines in order to allow specific adaptations to be evaluated for their biophysical and socio-economic ramifications.

agriculture↗
NASA NTRS2009

Towards a Framework for Evaluating and Comparing Diagnosis Algorithms

Diagnostic inference involves the detection of anomalous system behavior and the identification of its cause, possibly down to a failed unit or to a parameter of a failed unit. Traditional approaches to solving this problem include expert/rule-based, model-based, and data-driven methods. Each approach (and various techniques within each approach) use different representations of the knowledge required to perform the diagnosis. The sensor data is expected to be combined with these internal representations to produce the diagnosis result. In spite of the availability of various diagnosis technologies, there have been only minimal efforts to develop a standardized software framework to run, evaluate, and compare different diagnosis technologies on the same system. This paper presents a framework that defines a standardized representation of the system knowledge, the sensor data, and the form of the diagnosis results and provides a run-time architecture that can execute diagnosis algorithms, send sensor data to the algorithms at appropriate time steps from a variety of sources (including the actual physical system), and collect resulting diagnoses. We also define a set of metrics that can be used to evaluate and compare the performance of the algorithms, and provide software to calculate the metrics.

Kurtoglu, Tolga↗
NASA NTRS2004

Integration of Propulsion-Airframe-Aeroacoustic Technologies and Design Concepts for a Quiet Blended-Wing-Body Transport

This paper summarizes the results of studies undertaken to investigate revolutionary propulsion-airframe configurations that have the potential to achieve significant noise reductions over present-day commercial transport aircraft. Using a 300 passenger Blended-Wing-Body (BWB) as a baseline, several alternative low-noise propulsion-airframe-aeroacoustic (PAA) technologies and design concepts were investigated both for their potential to reduce the overall BWB noise levels, and for their impact on the weight, performance, and cost of the vehicle. Two evaluation frameworks were implemented for the assessments. The first was a Multi-Attribute Decision Making (MADM) process that used a Pugh Evaluation Matrix coupled with the Technique for Order Preference by Similarity to Ideal Solution (TOPSIS). This process provided a qualitative evaluation of the PAA technologies and design concepts and ranked them based on how well they satisfied chosen design requirements. From the results of the evaluation, it was observed that almost all of the PAA concepts gave the BWB a noise benefit, but degraded its performance. The second evaluation framework involved both deterministic and probabilistic systems analyses that were performed on a down-selected number of BWB propulsion configurations incorporating the PAA technologies and design concepts. These configurations included embedded engines with Boundary Layer Ingesting Inlets, Distributed Exhaust Nozzles installed on podded engines, a High Aspect Ratio Rectangular Nozzle, Distributed Propulsion, and a fixed and retractable aft airframe extension. The systems analyses focused on the BWB performance impacts of each concept using the mission range as a measure of merit. Noise effects were also investigated when enough information was available for a tractable analysis. Some tentative conclusions were drawn from the results. One was that the Boundary Layer Ingesting Inlets provided improvements to the BWB's mission range, by increasing the propulsive efficiency at cruise, and therefore offered a means to offset performance penalties imposed by some of the advanced PAA configurations. It was also found that the podded Distributed Exhaust Nozzle configuration imposed high penalties on the mission range and the need for substantial synergistic performance enhancements from an advanced integration scheme was identified. The High Aspect Ratio Nozzle showed inconclusive noise results and posed significant integration difficulties. Distributed Propulsion, in general, imposed performance penalties but may offer some promise for noise reduction from jet-to-jet shielding effects. Finally, a retractable aft airframe extension provided excellent noise reduction for a modest decrease in range.

Hill, G. A.↗
NASA NTRS2014

"Efficiency Space" - A Framework for Evaluating Joint Evaporation and Runoff Behavior

At the land surface, higher soil moisture levels generally lead to both increased evaporation for a given amount of incoming radiation (increased evaporation efficiency) and increased runoff for a given amount of precipitation (increased runoff efficiency). Evaporation efficiency and runoff efficiency can thus be said to vary with each other, motivating the development of a unique hydroclimatic analysis framework. Using a simple water balance model fitted, in different experiments, with a wide variety of functional forms for evaporation and runoff efficiency, we transform net radiation and precipitation fields into fields of streamflow that can be directly evaluated against observations. The optimal combination of the functional forms the combination that produces the most skillful stream-flow simulations provides an indication for how evaporation and runoff efficiencies vary with each other in nature, a relationship that can be said to define the overall character of land surface hydrological processes, at least to first order. The inferred optimal relationship is represented herein as a curve in efficiency space and should be valuable for the evaluation and development of GCM-based land surface models, which by this measure are often found to be suboptimal.

GCM↗
NASA NTRSDate not supplied

A Framework for Evaluating Climate Model Performance Metrics

The CMIP5 archive contains future climate projections from over 50 models provided by dozens of modeling centers from around the world. Individual model projections, however, are subject to biases created by structural model uncertainties. As a result, ensemble averaging of multiple models is often used to add value to model projections: consensus projections have been shown to consistently outperform individual models. Previous reports for the IPCC establish climate change projections based on an equal-weighted average of all model projections. However, certain models reproduce climate processes better than other models. Should models be weighted based on performance? Unequal ensemble averages have previously been constructed using a variety of mean state metrics. What metrics are most relevant for constraining future climate projections? This project develops a framework for systematically testing metrics in models to identify optimal metrics for unequal weighting multi-model ensembles. A unique aspect of this project is the construction and testing of climate process-based model evaluation metrics. A climate process-based metric is defined as a metric based on the relationship between two physically related climate variables?e.g., outgoing longwave radiation and surface temperature. Metrics are constructed using high-quality Earth radiation budget data from NASA's Clouds and Earth's Radiant Energy System (CERES) instrument and surface temperature data sets. It is found that regional values of tested quantities can vary significantly when comparing weighted and unweighted model ensembles. For example, one tested metric weights the ensemble by how well models reproduce the time-series probability distribution of the cloud forcing component of reflected shortwave radiation. The weighted ensemble for this metric indicates lower simulated precipitation (up to .7 mm/day) in tropical regions than the unweighted ensemble: since CMIP5 models have been shown to overproduce precipitation, this result could indicate that the metric is effective in identifying models which simulate more realistic precipitation. Ultimately, the goal of the framework is to identify performance metrics for advising better methods for ensemble averaging models and create better climate predictions.

Noel C Baker↗
NASA NTRS1995

A conceptual framework for evaluating variable speed generator options for wind energy applications

Interest in variable speed generating technology has accelerated as greater emphasis on overall efficiency and superior dynamic and control properties in wind-electric generating systems are sought. This paper reviews variable speed technology options providing advantages and disadvantages of each. Furthermore, the dynamic properties of variable speed systems are contrasted with synchronous operation. Finally, control properties of variable speed systems are examined.

Reddoch, T. W.↗
NASA NTRSDate not supplied

Objective Structured Clinical Evaluation (OSCE) of an Artificial Intelligence (AI) Clinical Decision Support System (CDSS) Tool

BACKGROUND Objective Structured Clinical Evaluations (OSCEs) have long been established as a robust methodology for summative assessment of clinical skills and decision-making during medical education. The recent integration of Artificial Intelligence (AI) into clinical decision-making processes has prompted the need for novel evaluation frameworks to assess the efficacy and reliability of AI clinical decision support system (CDSS) tools. This abstract outlines the process of quantitatively evaluating a novel CDSS (“Doc in a Box” Google 2024) trained on curated medical spaceflight data in the psychomotor domain as it interfaces with a human volunteer acting as the crew medical officer (CMO). PURPOSE The AI CDSS under review was developed as part of the Lunar Command and Control Interoperability (LuCCI) project, which is intended to address a gap in how Lunar Surface Systems (LSS) would interoperate across multiple programs, commercial partners, and international partners. The project objective is to define, prototype, integrate, and evaluate an interoperable lunar command, control, data, and software reference architecture to enable autonomy and informatics capability through common standards across LSS. A multi-modal AI-based CDSS compatible with Federated LSS will assist clinicians in diagnosing and managing complex medical conditions by providing evidence-based recommendations through predictive analytics. Given the critical role of decision-support as NASA continues to evolve its Earth-independent medical operations (EIMO), it is imperative to ensure that such AI tools perform reliably and align with clinical standards during progressive lunar and Martian exploration class missions. METHODS The OSCE framework, traditionally used for evaluating human clinicians, was adapted to assess the AI tool's decision-making capabilities in simulated clinical scenarios. In this adapted OSCE, the AI CDSS was tested across a series of structured clinical scenarios designed to mimic real-life spaceflight patient cases. These scenarios included a range of conditions and complexities, allowing for comprehensive assessment of the tool's performance. Key evaluation metrics included accuracy of diagnosis, timeliness of decision-making, and appropriate recommendations for therapies. The OSCE was scored by human physician evaluators who assessed the AI's recommendations in comparison with expert clinicians' medical decision making to ensure alignment with best practices and the standard of care. RESULTS Preliminary results indicate that the AI CDSS demonstrated high accuracy in diagnostic recommendations and decision support across various scenarios. However, certain limitations were noted, such as occasional discrepancies in handling complex or nuanced cases that required a more contextual understanding. Additionally, the tool scored higher on the diagnostic portion of the rubric, with lower scores in the therapeutic recommendations. These findings highlight the importance of continuous refinement and validation of AI tools through rigorous evaluation frameworks like the OSCE. The adaptation of OSCEs for AI tools presents several advantages, including a structured and reproducible approach to evaluation, the ability to test AI systems in diverse clinical scenarios, and the opportunity to benchmark AI performance against established clinical standards to permit charting of future progress as aerospace medicine evolves as a discipline. Remaining challenges include ensuring that these evaluations capture the full spectrum of clinical decision-making scenarios that will be confronted by CMOs during missions and adequately reflecting real-world variability of the austere spaceflight environment. CONCLUSION Employing OSCEs to evaluate AI clinical decision support tools offers a promising approach to validating their clinical utility and efficacy. This methodology not only provides insights into the tool's performance but also fosters ongoing improvement and alignment with standard of care practices. Future research should focus on refining these evaluation processes and addressing limitations to enhance the integration of AI tools in clinical spaceflight settings. REFERENCES Scott S, Hearns V, Barker MA. Testing Clinical Skills: A Look at the OSCE and USMLE Clinical Skills Exams. S D Med. 2019 Oct;72(10):451-453. Majumder MAA, Kumar A, Krishnamurthy K, Ojeh N, Adams OP, Sa B. An evaluative study of objective structured clinical examination (OSCE): students and examiners perspectives. Adv Med Educ Pract. 2019 Jun 5;10:387-397. Karam VY, Park YS, Tekian A, Youssef N. Evaluating the validity evidence of an OSCE: results from a new medical school. BMC Med Educ. 2018 Dec 20;18(1):313.

Ariana M Nelson↗
NASA NTRS2008

Evaluation of Long-Term Cloud-Resolving Model Simulations Using Satellite Radiance Observations and Multi-Frequency Satellite Simulators

This paper proposes a methodology known as the Tropical Rainfall Measuring Mission (TRMM) Triple-Sensor Three-step Evaluation Framework (T3EF) for the systematic evaluation of precipitating cloud types and microphysics in a cloud-resolving model (CRM). T3EF utilizes multi-frequency satellite simulators and novel statistics of multi-frequency radiance and backscattering signals observed from the TRMM satellite. Specifically, T3EF compares CRM and satellite observations in the form of combined probability distributions of precipitation radar (PR) reflectivity, polarization-corrected microwave brightness temperature (Tb), and infrared Tb to evaluate the candidate CRM. T3EF is used to evaluate the Goddard Cumulus Ensemble (GCE) model for cases involving the South China Sea Monsoon Experiment (SCSMEX) and Kwajalein Experiment (KWAJEX). This evaluation reveals that the GCE properly captures the satellite-measured frequencies of different precipitating cloud types in the SCSMEX case but underestimates the frequencies of deep convective and deep stratiform types in the KWAJEX case. Moreover, the GCE tends to simulate excessively large and abundant frozen condensates in deep convective clouds as inferred from the overestimated GCE-simulated radar reflectivities and microwave Tb depressions. Unveiling the detailed errors in the GCE s performance provides the best direction for model improvements.

Matsui, Toshihisa↗