Search NASA⌕ Search

SEARCH · Search NASA

Results for “evaluation methodology”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 73 records · Page 4
All sourcesNASA NTRSDOE OSTI
DOE OSTI2022

A Methodology for Evaluating Operator Usage of Machine Learning Recommendations for Power Grid Contingency Analysis

This work presents the application of a methodology to measure domain expert trust and workload, elicit feedback, and understand the technological usability and impact when a machine learning assistant is introduced into contingency analysis for real-time power grid simulation. The goal of this framework is to rapidly collect and analyze a broad variety of human factors data in order to accelerate the development and evaluation loop for deploying machine learning applications. We describe our methodology and analysis, and we discuss insights gained from a pilot participant about the current usability state of an early technology readiness level (TRL) artificial neural network (ANN) recommender.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗
NASA NTRS2007

Test Methodology to Evaluate the Safety of Materials Using Spark Incendivity

For many years scientists and engineers have been searching for the proper test method to evaluate an electrostatic risk for materials used in hazardous environments. A new test standard created by the International Electrotechnical Commission is a promising addition to conventional test methods used throughout industry. The purpose of this paper is to incorporate this test into a proposed new methodology for the evaluation of materials exposed to flammable environments. However, initial testing using this new standard has uncovered some unconventional behavior in materials that conventional test methods were thought to have reconciled. For example some materials tested at higher humidities were more susceptible to incendive discharges than at lower humidity even though the surface resistivity was lower.

Buhler, Charles↗
DOE OSTI2024

Safety in Artificial Intelligence: Challenges and Opportunities for the U.S. National Labs and Beyond

This report discusses the importance of the critical and underexplored topic of artificial intelligence (AI) safety, as highlighted during the “Strategy Alignment on AI Safety” workshop convened at Lawrence Livermore National Laboratory (LLNL) in April 2024. Through a summary of keynote talks, panel discussions, and breakout sessions, world-leading AI safety experts from academic, industry, national labs, and government agencies clearly agree on the need for and importance of large-scale investments for research and capabilities in AI safety. With the field innovating at unprecedented rates, there is increasing urgency to develop novel evaluation methodologies that allow full considerations of risks/threats of AI technologies in different domains. Quantitative metrics and effective methodologies that can evaluate and audit the “safeness” of how a given AI technology is trained, deployed, or regulated are, at best, nascent for certain scenarios or, more commonly, nonexistent. This maturation gap presents the possibility of serious threats to national security, and further inaction may have serious consequences. Additionally, the gap between the public’s and research community’s perceptions of AI risks/rewards is significant. While numerous voices from the AI community have expressed concern that the risks could be so high that future AI systems could inflict extinction-level damage to humanity if deployed incorrectly, the public largely is aware only of risk in low-impact scenarios. This discrepancy highlights the crucial need for researchers to articulate to governmental bodies what, why, and when various AI risks matter as part of motivating funding requests. Thus, the call to action for this community is to pursue AI safety as a “Big Science” project on a scale comparable to the Manhattan Project. High risks and high payoffs are on the table, but safe AI is a fast-moving target, and large-scale investments are needed to guide development of this technology in a responsible way. We highlight the need for a multilayered solution combining the development of new methods and algorithmic approaches to mitigate threats with an active participation of the government(s) in setting high industry standards and regulations based on state-of-the-art technology. The U.S. Department of Energy (DOE) national laboratories have served as leading institutions for scientific innovation in the U.S. for more than 70 years. Drawing on their expertise in the AI community and their history of safeguarding critical and sensitive information, and as we look to the future, national labs are the best choice for evaluating and safeguarding AI technologies.

97 MATHEMATICS AND COMPUTING↗
NASA NTRS1992

Concept of ASTER calibration requirement

The document of ASTER Calibration Requirement specifies the following items related to spectral and radiometric characteristics of the ASTER instrument: (1) characteristics whose knowledge is specified, (2) requirement for knowledge of the characteristics, (3) methodology for characteristics evaluation, and (4) supplementary information and data related with characteristics evaluation. This document is applicable to the document of the ASTER Instrument Specification on Observational Performances, and will be a part of the ASTER Calibration Plan. ASTER Calibration Requirement is scheduled to establish the concept and framework by March 1992 when the 5th Calibration and Data Validation Panel Meeting is held, and to determine details including requirement values and evaluation methodologies by October 1992 around which the Calibration Peer Review may be held. The ASTER Calibration Plan is planned to finish by the same time.

Ono, A.↗
DOE OSTI2022

Proposed Methodology for Evaluating and Validating TSLs [Slides]

This presentation begins by providing an overview of the research and discusses recent efforts in validating thermal neutron scattering cross sections, including differential cross section measurements at ORNL for evaluation and validation, total cross section measurements at RPI, pulsed-neutron die-away experiments at LLNL, and integral criticality experiments by LLNL. The presentation also proposes a methodology not only for validating thermal scattering files that utilizes all available experimental data, but also for evaluating new libraries as demonstrated on polystyrene. In conclusion, polystyrene evaluation was conducted using multiple experiments, including differential and integral. The proposed methodology has been shown to improve neutron transport and files have been submitted to NNDC for inclusion in ENDF/B-VIII.1 release. MCT-012 & PCM-002 has been relatively insensitive to changes in $\mathcal{S}(α,β)$. The research found that differences in differential results don’t always propagate to differences in integral results, hence why evaluation of differential data important. Future work on the project is also discussed.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
DOE OSTI2022

Simulation, Challenge Testing and Validation of Solutions for Residential and Commercial Occupancy Sensing/Counting and CO2 Sensing (Final Report)

Carbon dioxide sensors used for monitoring and control applications in buildings are known to be sensitive to long term drift and can be affected by variations in temperature, humidity, other gas interferents, pressure and other factors that cannot be corrected through the typical auto-calibration methods often embedded in the sensor electronics. This investigation was part of the Category D defined in the ARPA-E Saving Energy Nationwide in Structures With Occupancy Recognition (SENSOR) Financial Assistance Funding Opportunity Announcement No. DE-FOA-0001737, FFDA No. 81.135, January 18, 2017. The FOA defined target criteria for the evaluation of the Category C commercial CO2 sensors selected for review and development through the ARPA-E SENSOR program and charged the Iowa State Category D team with developing an evaluation methodology for Category C sensors in accordance with these criteria. As part of the work, the Iowa State team chose to engage the standards community in the development of the evaluation methodology. Standard development work in the D22.05 Indoor Air and D22.03 Ambient Atmospheres and Source Emissions subcommittees of ASTM International Committee D22 on Air Quality on the evaluation of low cost sensors and provision of guidance for using indoor carbon dioxide concentrations to evaluate indoor air quality and ventilation is of particular relevance to the Category D efforts. To date, three draft standards with specific input by the Iowa State Category D team have passed the subcommittee balloting stage and are being balloted in December 2022 and January 2023 in the main D22 committee. In addition to the aforementioned standards to which the PI was a contributing author, intellectual property developed in this project pertaining to “An Automated System for Evaluation of the Long Term Performance of Carbon Dioxide Sensors” and “An Automated Ground Truth System for Real-Time Reliability Assessment, Capture of Control Decisions and Energy Savings Measurements for Occupancy Recognition Systems and Other Applications” are the other major deliverables of the project. These standards and the developed intellectual property are expected to benefit the public in a number of ways stemming from their provision of means to benchmark the performance of carbon dioxide and other sensor systems. In particular, there is high interest in techniques to demonstrate the performance of low-cost carbon dioxide sensors which can be used for typical HVAC applications (e.g., demand controlled ventilation, outdoor and indoor air monitoring, estimation of occupancy, etc.).

42 ENGINEERING↗
NASA NTRS1999

Understanding Skill in EVA Mass Handling: An Integrated Methodology for Evaluating Space Suit Mobility and Stability - Volume 4

The empirical investigation of extravehicular activity (EVA) mass handling conducted on NASA's Precision Air-Bearing Floor led to a Phase I SBIR from JSC. The purpose of the SBIR was to design an innovative system for evaluating space suit mobility and stability in conditions that simulate EVA on the surface of the Moon or Mars. The approach we used to satisfy the Phase I objectives was based on a structured methodology for the development of human-systems technology. Accordingly the project was broken down into a number of tasks and subtasks. In sequence, the major tasks were: 1) Identify missions and tasks that will involve EVA and resulting mobility requirements in the near and long term; 2) Assess possible methods for evaluating mobility of space suits during field-based EVA tests; 3) Identify requirements for behavioral evaluation by interacting with NASA stakeholders;.4) Identify necessary and sufficient technology for implementation of a mobility evaluation system; and 5) Prioritize and select technology solutions. The work conducted in these tasks is described in this final volume of the series on EVA mass handling. While prior volumes in the series focus on novel data-analytic techniques, this volume addresses technology that is necessary for minimally intrusive data collection and near-real-time data analysis and display.

McDonald, P. Vernon↗
NASA NTRS1999

A Methodology for Evaluating the Fidelity of Ground-Based Flight Simulators

An analytical and experimental investigation was undertaken to model the manner in which pilots perceive and utilize visual, proprioceptive, and vestibular cues in a ground-based flight simulator. The study was part of a larger research effort which has the creation of a methodology for determining flight simulator fidelity requirements as its ultimate goal. The study utilized a closed-loop feedback structure of the pilot/simulator system which included the pilot, the cockpit inceptor, the dynamics of the simulated vehicle and the motion system. With the exception of time delays which accrued in visual scene production in the simulator, visual scene effects were not included in this study. The NASA Ames Vertical Motion Simulator was used in a simple, single-degree of freedom rotorcraft bob-up/down maneuver. Pilot/vehicle analysis and fuzzy-inference identification were employed to study the changes in fidelity which occurred as the characteristics of the motion system were varied over five configurations i The data from three of the five pilots that participated in the experimental study were analyzed in the fuzzy inference identification. Results indicate that both the analytical pilot/vehicle analysis and the fuzzyinference identification can be used to reflect changes in simulator fidelity for the task examined.

Zeyada, Y.↗
NASA NTRS1999

A Methodology for Evaluating the Fidelity of Ground-Based Flight Simulators

An analytical and experimental investigation was undertaken to model the manner in which pilots perceive and utilize visual, proprioceptive, and vestibular cues in a ground-based flight simulator. The study was part of a larger research effort which has the creation of a methodology for determining flight simulator fidelity requirements as its ultimate goal. The study utilized a closed-loop feedback structure of the pilot/simulator system which included the pilot, the cockpit inceptor, the dynamics of the simulated vehicle and the motion system. With the exception of time delays which accrued in visual scene production in the simulator, visual scene effects were not included in this study. The NASA Ames Vertical Motion Simulator was used in a simple, single-degree of freedom rotorcraft bob-up/down maneuver. Pilot/vehicle analysis and fuzzy-inference identification were employed to study the changes in fidelity which occurred as the characteristics of the motion system were varied over five configurations. The data from three of the five pilots that participated in the experimental study were analyzed in the fuzzy-inference identification. Results indicate that both the analytical pilot/vehicle analysis and the fuzzy-inference identification can be used to reflect changes in simulator fidelity for the task examined.

Zeyada, Y.↗
DOE OSTI2024

Methodology to evaluate design modifications intended to eliminate frosting and high discharge temperatures in air-source heat pumps (ASHPs) in cold climates

Air-source heat pumps (ASHPs) operating in cold climates experience problems with frosting and high refrigerant temperatures. These problems increase energy consumption, and their severity depends on the climatic conditions. In the present paper, a methodology for identifying the prevailing problem between frosting and high discharge temperatures is presented. Three performance indices, the frosting index (FI), the discharge index (DI), and the total loss index (TLI), are proposed to quantify the impacts of frosting and high discharge temperatures on the annual performance of ASHPs in different climatic conditions. The FI and DI show which problem (frosting or high discharge temperature) dominates, and the TLI indicates the combined effect of frosting and high discharge temperatures on the performance of an ASHP. A thermodynamic model of an ASHP coupled with the TRNSYS building simulation tool is used to estimate the performance of an ASHP and the proposed loss indices to estimate the impact of both frosting and high discharge temperatures for 45 cities in Canada. The results can be extended to other parts of the world that experience similar climatic conditions The results reveal that in cities in ASHRAE climatic zones 5 and 6 (classified as cold regions) where the ambient air temperatures are predominantly between -15 °C to 6 °C, ASHPs are heavily impacted by frosting. The problem of high discharge temperatures in ASHPs is predominant in cities in climate zones 7 and 8 (classified as very cold and subarctic regions) where the temperatures are frequently below -20 °C in winter. Among the cities considered, St. John, NL has the highest fraction of heating hours experiencing frosting (90 %), where the annual increase in energy consumption due to frosting is 13.5 % of the annual heating energy consumption. The highest annual increase in energy consumption due to high discharge temperatures is in Isachsen, NU (zone 8), where the increase is 30 % of the annual heating energy consumption. Based on the proposed indices, another index called the performance gain index (PGI) is created, which can be used as a first step to assess the energy-saving potential of design modifications applied to ASHPs to solve the problems of frosting and high discharge temperatures. The PGI will aid in developing climate specific ASHPs. One possible design modification is the use of a two-stage ASHP with an economizer. It is observed that the two-stage ASHP with economizer can mitigate high discharge temperatures and improve performance in very cold and subarctic regions (zones 7 and 8). However, it is not as beneficial in zones 5 and 6, where the impact of high discharge temperatures on performance is minimal and frosting dominates. Finally, a case study, using the PGI to evaluate the economic and environmental effectiveness of a two-stage ASHP with economizer is presented for the city of Saskatoon.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
DOE OSTI2026

A Methodology to Evaluate the Grid Reliability Impact of Oscillations Induced by Large Loads

The rapid growth of hyperscale AI data centers is bringing renewed attention to the reliability risk that sustained forced oscillations pose to bulk power systems, with cyclic computational workloads emerging as a new forcing source. Unlike the broadband, stochastic disturbances from traditional industrial loads such as arc furnaces, AI training and inference facilities can inject large active power swings concentrated at specific frequencies over extended durations - characteristics that existing grid planning practices do not account for. While the North American Electric Reliability Corporation (NERC) has recognized this gap and called for system-level studies of large load interconnections, no standardized methodology exists to screen, simulate, and quantify these risks at the planning stage. This report presents the Risk Assessment Tool for Large Load-induced Events (RATLLE), a Python-based, publicly available script suite developed at the Pacific Northwest National Laboratory to evaluate bulk power system reliability risks from data center-induced oscillations. RATLLE implements a three-module workflow: a screening module that identifies vulnerable interconnection locations and excitable system modes; a simulation module that models cyclic data center load behavior using a commercial positive sequence simulation platform; and an analysis module that computes risk metrics and generates interactive visualization dashboards. The risk metrics, formulated around simulation observables, map oscillation impacts to a three-stage severity scale spanning latent equipment fatigue through imminent cascading failure. The methodology is demonstrated on two Western Electricity Coordinating Council (WECC) system models: a publicly available 240-bus reduced representation and a detailed 2031 Heavy Winter planning case. Case studies illustrate that even modest 50 MW forced oscillations at resonant frequencies can produce wide-area power swings, N-1 security constraint violations, and cascading generator trips through protection actions - outcomes that would not occur under normal operating conditions without oscillations present. The results underscore the need for standardized oscillation impact assessment in large load interconnection studies and provide a reproducible, extensible framework for utilities to adopt or customize within their existing planning workflows.

Biswas, Shuchismita↗
DOE OSTI2020

An evaluation of methodology to determine algal genome completeness

The advancement of sequencing technologies has resulted in a rapid expansion of genome sequencing. One challenge in processing and analyzing this abundance of genomics data is development and application of tools to accurately assess the quality of novel genomes. The quality of a given genome assembly is commonly assessed using the presence of conserved orthologs. However, the applicability and accuracy of identifying conserved orthologs applied to algal genome assemblies, which are uniquely diverse, is unclear. Here, in this study, we analyze the utility of a genome content assessment tool, Benchmarking Universal Single-Copy Orthologs (BUSCO), to evaluate algal genome assembly completeness. We find that Chlorophyta and Stramenopile BUSCO databases are effective tools to analyze genome completeness of these lineages. For other lineages, the Eukaryota database should be utilized until additional lineage-specific databases are developed. Based on these results, suggested best practices for current usage and necessary future improvements are provided.

59 BASIC BIOLOGICAL SCIENCES↗
NASA NTRS1989

Laboratory test methodology for evaluating the effects of electromagnetic disturbances on fault-tolerant control systems

Control systems for advanced aircraft, especially those with relaxed static stability, will be critical to flight and will, therefore, have very high reliability specifications which must be met for adverse as well as nominal operating conditions. Adverse conditions can result from electromagnetic disturbances caused by lightning, high energy radio frequency transmitters, and nuclear electromagnetic pulses. Tools and techniques must be developed to verify the integrity of the control system in adverse operating conditions. The most difficult and illusive perturbations to computer based control systems caused by an electromagnetic environment (EME) are functional error modes that involve no component damage. These error modes are collectively known as upset, can occur simultaneously in all of the channels of a redundant control system, and are software dependent. A methodology is presented for performing upset tests on a multichannel control system and considerations are discussed for the design of upset tests to be conducted in the lab on fault tolerant control systems operating in a closed loop with a simulated plant.

Belcastro, Celeste M.↗
DOE OSTI2023

Guiding Principles for the Rational Design of Hybrid Materials: Use of DFT Methodology for Evaluating Non‐Covalent Interactions in a Uranyl Tetrahalide Model System

Abstract Together with the synthesis and experimental characterization of 14 hybrid materials containing [UO 2 X 4 ] 2− (X=Cl − and Br − ) and organic cations, we report on novel methods for determining correlation trends in their formation enthalpy (ΔH f ) and observed vibrational signatures. Δ H f values were analyzed through isothermal acid calorimetry and a Density Functional Theory+Thermodynamics (DFT+T) approach with results showing good agreement between theory and experiment. Three factors (packing efficiency, cation protonation enthalpy, and hydrogen bonding energy [ ]) were assessed as descriptors for trends in Δ H f . Results demonstrated a strong correlation between and Δ H f , highlighting the importance of hydrogen bonding networks in determining the relative stability of solid‐state hybrid materials. Lastly, we investigate how hydrogen bonding networks affect the vibrational characteristics of uranyl solid‐state materials using experimental Raman and IR spectroscopy and theoretical bond orders and find that hydrogen bonding can red‐shift U≡O stretching modes. Overall, the tightly integrated experimental and theoretical studies presented here bridge the trends in macroscopic thermodynamic energies and spectroscopic features with molecular‐level details of the geometry and electronic structure. This modeling framework forms a basis for exploring 3D hydrogen bonding as a tunable design feature in the pursuit of supramolecular materials by rational design.

36 MATERIALS SCIENCE↗
DOE OSTI2023

Guiding Principles for the Rational Design of Hybrid Materials: Use of DFT Methodology for Evaluating Non‐Covalent Interactions in a Uranyl Tetrahalide Model System

Abstract Together with the synthesis and experimental characterization of 14 hybrid materials containing [UO 2 X 4 ] 2− (X=Cl − and Br − ) and organic cations, we report on novel methods for determining correlation trends in their formation enthalpy (ΔH f ) and observed vibrational signatures. Δ H f values were analyzed through isothermal acid calorimetry and a Density Functional Theory+Thermodynamics (DFT+T) approach with results showing good agreement between theory and experiment. Three factors (packing efficiency, cation protonation enthalpy, and hydrogen bonding energy [ ]) were assessed as descriptors for trends in Δ H f . Results demonstrated a strong correlation between and Δ H f , highlighting the importance of hydrogen bonding networks in determining the relative stability of solid‐state hybrid materials. Lastly, we investigate how hydrogen bonding networks affect the vibrational characteristics of uranyl solid‐state materials using experimental Raman and IR spectroscopy and theoretical bond orders and find that hydrogen bonding can red‐shift U≡O stretching modes. Overall, the tightly integrated experimental and theoretical studies presented here bridge the trends in macroscopic thermodynamic energies and spectroscopic features with molecular‐level details of the geometry and electronic structure. This modeling framework forms a basis for exploring 3D hydrogen bonding as a tunable design feature in the pursuit of supramolecular materials by rational design.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2025

Estimation Methodology to Evaluate Hypothetical Downwind Impacts from Fusion Plants

The continuing move toward establishing fusion systems for power generation and the associated research to that end is prompting examination of the potential health and safety impacts of such plants to the environment and human health. As many fusion facilities will have tritium inventories on site as part of the fusion fuel, evaluating the potential for downwind impacts from fusion facilities or power plants resulting from accident or routine emissions is a general requirement for assessing location and impacts to workers and the public. As part of siting considerations and permitting, the fusion facilities would be evaluated for potential for downwind concentration and dose impacts. For accident assessment scenarios, the downwind impacts are usually modeled as an instantaneous (or near-instantaneous) release of material transported following the wind. A range of meteorological conditions are usually assessed to determine a bounding case which results in a dose exceeding a specified threshold (e.g., 95 th or 99 th percentile; DOE 2015). This report provides initial estimates of the downwind dose impacts from a potential tritium release at a fusion power plant-relevant facility and identifies potential distances required to limit impacts to nearby population. This effort is meant to provide a bounding analysis and theoretical understanding of impacts of tritium releases for facilities subject to various environmental and atmospheric conditions. Using a Gaussian dispersion model to simulate a brief plume, downwind concentration and dose is projected for tritium oxide. Releases are assumed to consist entirely of tritium oxide due to the increased dose impacts from the oxide form relative to the elemental form of tritium. We also briefly identify how climatological conditions could potentially be used to support risk profile determination.

54 ENVIRONMENTAL SCIENCES↗