Search NASA⌕ Search

SEARCH · Search NASA

Results for “evaluation protocol”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 55 records · Page 3
All sourcesNASA NTRSDOE OSTI
DOE OSTI2022

Computational Protocol to Evaluate Electron–Phonon Interactions Within Density Matrix Perturbation Theory

We present a computational protocol, based on density matrix perturbation theory, to obtain non-adiabatic, frequency-dependent electron–phonon self-energies for molecules and solids. Our approach enables the evaluation of electron–phonon interaction using hybrid functionals, for spin-polarized systems, and the computational overhead to include dynamical and non-adiabatic terms in the evaluation of electron–phonon self-energies is negligible. We discuss results for molecules, as well as pristine and defective solids.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2016

PROTOCOL TO EVALUATE THE MOISTURE DURABILITY OF ENERGY-EFFICIENT WALLS

Walls account for about 8% of the energy used in residential buildings. This energy penalty can be reduced with higher insulation levels and increased airtightness. However, these measures can compromise the moisture durability and long-term performance of wall assemblies because they can lead to lower moisture tolerance due to reduced drying potential. To avert these problems, a moisture durability protocol was developed to evaluate the probability that an energy-efficient wall design will experience mold growth. This protocol examines the effects of moisture sources in walls through a combination of simulations and lab experiments, uses the mold growth index as the moisture durability indicator, and is based on a probabilistic approach that utilizes stochastically varying input parameters. The simulation tools used include a new validated method for taking into account the effects of air leakage in wall assemblies This paper provides an overview of the developed protocol, discussion of the probabilistic simulation approach and describes results from the evaluation of two wall assemblies in Climate Zones 2, 4, and 6. The protocol will be used to supply builders with wall designs that are energy efficient, moisture durable and cost-effective.

durability↗
DOE OSTI2025

Implementation and Evaluation of Emission‐Driven Land‐Atmosphere Coupled Simulation in E3SMv2.1

Emissions-driven (prognostic CO 2 ) simulations are essential for representing two-way carbon-climate feedback in Earth System Models. We present an emissions-driven land–atmosphere coupled biogeochemistry (BGC) configuration (BGCLNDATM_progCO2) in version 2.1 of the Energy Exascale Earth System Model (E3SMv2.1). This is the first E3SM configuration that performs land-atmosphere emission-hindcasts. Here, we document its implementation, evaluate the model's performance against observations and other models, and propose a structured evaluation protocol for such emissions-driven simulations. We conducted transient historical simulations (1850–2014) with BGCLNDATM_progCO2 and compare them to reference simulations—a land-atmosphere coupled simulation without BGC and a standalone land simulation with BGC, both using prescribed CO 2 concentrations—and to observations. BGCLNDATM_progCO2 overestimates atmospheric CO 2 concentrations by 11–23 ppm yet stays within the 40-ppm spread CMIP6 emission-driven models and retains physical climate properties comparable to the reference runs. The CO 2 biases are partly attributed to underrepresented oceanic CO 2 uptake and inadequate representations of some terrestrial processes. In general, introducing prognostic CO 2 did not change physical climate metrics at the global scale but had larger regional effects, particularly over land where spatially heterogeneous CO 2 and prognostic leaf area index influenced surface energy balance. Finally, we propose a general evaluation protocol including spin-up assessment, atmospheric CO 2 benchmarking, physical climate evaluation, and land biogeochemical analysis to support scientific rigor and facilitate inter-model comparisons. The new configuration lays the groundwork for future enhancements, including improved terrestrial biogeochemical processes, integrated marine biogeochemistry, and additional human–Earth system interactions. These developments advance E3SM toward fully coupled emissions-driven simulations, enabling more accurate carbon–climate feedback projections and informing mitigation policy by providing physically consistent carbon-budget metrics for mitigation scenarios.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2024

Toward Wireless Smart Grid Communications: An Evaluation of Protocol Latencies in an Open-Source 5G Testbed

Fifth-generation networks promise wide availability of wireless communication with inherent security features. The 5G standards also outline access for different applications requiring low latency, machine-to-machine communication, or mobile broadband. These networks can be advantageous to numerous applications that require widespread and diverse communications. One such application is found in smart grids. Smart grid networks, and Operational Technology (OT) networks in general, utilize a variety of communication protocols for low-latency control, data monitoring, and reporting at every level. Transitioning these network communications from wired Wide Area Networks (WANs) to wireless communication through 5G can provide additional benefits to their security and network configurability. However, introducing these wireless capabilities may also result in a degradation of network latency. In this paper, we propose utilizing 5G for smart grid communications, and we evaluate the latency impacts of encapsulating GOOSE, Modbus, and DNP3 for transmission over a 5G network. The OpenAirInterface open-source library is utilized to deploy an in-lab 5G Core Network and gNB for testing with off-the-shelf User Equipment (UE). This creates an effective 5G test platform for experimenting with different OT protocols such as GOOSE. The results are validated by measuring two different Intelligent Electronic Devices’ contact closure times for each network configuration. These tests are also conducted for varying packet sizes in order to isolate different sources of network latency. Our study outlines the latency impact of communication over 5G for time-critical and non-critical applications regarding their transition toward private 5G-based OT network implementations. The conducted experiments illustrate that in the case of GOOSE packets, simple encapsulation may exceed the protocol’s time-critical nature, and, therefore, additional measures must be taken to ensure a viable transition of GOOSE to 5G services. However, non-critical applications are shown to be viable for migration to 5G.

42 ENGINEERING↗
DOE OSTI2021

Analysis of Ion-Exchanged ZSM-5, BEA, and SSZ-13 Zeolite Trapping Materials under Realistic Exhaust Conditions

An industry-defined evaluation protocol was used to evaluate the hydrocarbon trapping (HCT) and passive NOx adsorption (PNA) potential for BEA, ZSM-5, and SSZ-13 zeolites with ion-exchanged Pd or Ag. All materials underwent 700 °C degreening prior to exposure to an industry-derived protocol gas stream, which included NOx, ethylene, toluene, and decane as measured trapping species as well as common exhaust gasses CO, H2O, O2, CO2, and H2. Evaluation showed that BEA and ZSM-5 zeolites were effective at trapping hydrocarbons (HCs), as saturation was not achieved after 30 min of exposure. SSZ-13 also stored HCs but was only able to adsorb 20–25% compared to BEA and ZSM-5. The presence of Ag or Pd did not impact the overall HC uptake, particularly in the first three minutes. Pd/zeolites had significantly lower THC release temperature, and it aided in the conversion of the released HCs; Ag only had a moderate effect in both areas. With respect to NOx adsorption, the level of uptake was much lower than HCs on all samples, and Ag or Pd was necessary with Pd being notably more effective. Additionally, only Pd/ZSM-5 and Pd/SSZ-13 continue to store a portion of the NOx above 200 °C, which is critical for downstream selective catalytic NOx reduction (SCR). Hydrothermal aging (800 °C for 50 h) of a subset of the samples were performed: BEA, Pd/BEA, ZSM-5, Pd/ZSM-5, and Pd/SSZ-13. There was a minimal effect on the HC storage, ~10% reduction in capacity with no effect on release temperature; however, only Pd/SSZ-13 showed significant NOx storage after aging.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2025

Modeling Protocol for Evaluation of Air Emissions for the Idaho National Laboratory Site-wide Permit to Construct and Facility Emission Cap (PTC/FEC)

This document outlines the methods to be used for calculating non-radiological air quality impacts from proposed emissions at the Idaho National Laboratory (INL) Site in support of applying for a renewal to a Permit to Construct with a Facility Emissions Cap (PTC/FEC). The INL Site is seeking to maintain a state enforceable limit on emissions to maintain synthetic minor status in accordance with Idaho Administrative Procedures Act (IDAPA) 58.01.01.175-181 and IDAPA 58.01.01.200-228.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2017

Best Practices and Testing Protocols for Benchmarking ORR Activities of Fuel Cell Electrocatalysts Using Rotating Disk Electrode

Abstract Thin-film-rotating disk electrodes (TF-RDEs) are the half-cell electrochemical system of choice for rapid screening of oxygen reduction reaction (ORR) activity of novel Pt supported on carbon black supports (Pt/C) electrocatalysts. It has been shown that the magnitude of the measured ORR activity and reproducibility are highly dependent on the system cleanliness, evaluation protocols, and operating conditions as well as ink formulation, composition, film drying, and the resultant film thickness and uniformity. Accurate benchmarks of baseline Pt/C catalysts evaluated using standardized protocols and best practices are necessary to expedite ultra-low-platinum group metal (PGM) catalyst development that is crucial for the imminent commercialization of fuel cell vehicles. We report results of evaluation in three independent laboratories of Pt/C electrocatalysts provided by commercial fuel cell catalyst manufacturers (Johnson Matthey, Umicore, Tanaka Kikinzoku Kogyo—TKK). The studies were conducted using identical evaluation protocols/ink formulation/film fabrication albeit employing unique electrochemical cell designs specific to each laboratory. The ORR activities reported in this work provide a baseline and criteria for selection and scale-up of novel high activity ORR electrocatalysts for implementation in proton exchange membrane fuel cells (PEMFCs).

Kocha, Shyam S.↗
DOE OSTI2015

Performance Test Protocol for Evaluating Inverters Used in Grid-Connected Photovoltaic Systems

Interconnection equipment that connects distributed resources (DR) to an electrical power system (EPS) are expected to do so efficiently. Standardized test procedures are necessary to establish methods for verifying inverter performance that leads to comparable results. These test procedures are provided as a repeatable, independent means of measuring inverter performance regarding maximum continuous power rating, conversion efficiency, and tare loss characteristics.

14 SOLAR ENERGY↗
NASA NTRS1992

Development of a test protocol for evaluating EVA glove performance

Testing gloved hand performance involves work from several disciplines. Evaluations performed in the course of reenabling a disabled hand, designing a robotic end effector or master controller, or hard-suit design have all yielded relevant information, and, in most cases, produced performance test methods. Most times, these test methods have been primarily oriented toward their parent discipline. For space operations, a comparative test which would provide a way to quantify pressure glove and end effector performance would be useful in dividing tasks between humans and robots. Such a test would have to rely heavily on sensored measurement, as opposed to questionnaires, to produce relevant data. However, at some point human preference would have to be taken into account. This paper presents a methodology for evaluating gloved hand performance which attempts to respond to these issues. Glove testing of a prototype glove design using this method is described.

Hinman, Elaine M.↗
DOE OSTI2022

Durability Evaluation of Advanced Fenestration Technologies

In the U.S., more than 40% of primary energy and 70% of electricity is consumed in residential and commercial buildings, resulting in annual energy costs of more than $430 billion. Approximately 35% of this consumption can be attributed to losses through the building envelope, of which windows are currently the weakest link. Multiple technologies are under development to improve this performance. This includes dynamic and photovoltaic glazing as well as emerging highly insulating technologies including vacuum insulating glass (VIG), aerogels and thin-glass based multi-pane configurations. While windows are specified based on expected performance as installed, the energy savings realized by high performance windows are delivered over time. This makes it critical to understand and maximize the durability of high-performance windows to ensure those projected energy savings are delivered. Present methods for evaluating durability are based on existing technologies. These methods may not apply or be adequate for newer emerging technologies which often present novel failure mechanisms that need to be understood and evaluated differently. In this presentation, we will discuss our efforts to define appropriate methods to improve the evaluation of many existing technologies as well as evaluate the durability of emerging window technologies. This work results from the combination of an extensive review of various international standards as well as existing scientific literature. In addition, input was gathered from multiple industry stakeholders regarding present practice as well as feedback on proposed improvements to existing methods. Here we will share these findings and present proposed improvements to developing and existing durability evaluation protocols.

building envelope↗
DOE OSTI2022

Model Assumptions and Data Characteristics: Impacts on Domain Adaptation in Building Segmentation

Studies on domain adaptation (DA) for remote sensing (RS) imagery analysis lack consistency in selection and description of evaluation scenarios. Without properly characterizing datasets, model assumptions, and evaluation scenarios, it is difficult to objectively compare DA methods and reach conclusions about their suitability across different applications. With this motivation, this work seeks to empirically assess to which extent the interaction between data characteristics and model assumptions influences the effectiveness of DA methods. Using the widely explored task of building footprint segmentation as a case study, we perform a large-scale study across over 200 DA scenarios that include variations across view angles, areas observed, and sensors used for data acquisition. Rather than adopting different model architectures or optimization criteria, we contrast the performances of two DA methods based on adversarial learning that differ only in their assumptions about source and target domains. Informed by metadata and data characteristics unveiled using traditional computer vision (CV) techniques as well as pretrained deep models, we provide a detailed meta-analysis of experiments highlighting the importance of accurately considering data assumptions for DA in RS segmentation tasks. As demonstrated by a “cherry-picking” exercise, different claims regarding which model is best could be made by selecting different subsets of evaluation scenarios. While well-calibrated assumptions can be beneficial, mismatching assumptions can lead to negative biases in DA applications. Furthermore, this study intends to motivate the community toward more consistent evaluation protocols while providing recommendations and insights toward creating novel benchmark datasets, documenting data characteristics, application-specific knowledge, and model assumptions.

42 ENGINEERING↗
DOE OSTI2026

Scaling open-weight large language models for hydropower regulatory information extraction: A systematic analysis

Information extraction from regulatory and technical documents using large language models (LLMs) involves practical trade-offs between extraction quality and computational cost. We evaluate eight open-weight LLMs spanning 0.6B–70B parameters on hydropower licensing documents and report deployment-oriented evidence under a unified extraction schema and evaluation protocol. Across the model set, we observe clear scale-dependent trends in both baseline extraction quality and the effectiveness of reflective reasoning (self-checking) under our fixed-prompt, no-augmentation setting. Mid-scale models often provide a favorable balance of accuracy and efficiency, whereas the smallest models show limited or inconsistent gains from the reasoning variants tested. Larger models achieve the highest overall F1 scores but incur substantially greater compute and infrastructure requirements. We further find that reliability failure modes can distort conventional metrics in this domain: in particular, high recall can coincide with systematic extraction errors when models fabricate values for fields that are absent from the source text, underscoring the importance of conservative null handling and evidence-grounded evaluation. Overall, our study provides a reproducible resource–performance comparison for open-weight LLM-based extraction in hydropower regulatory documentation and offers practical guidance for model selection under different deployment constraints.

Evaluation protocol↗
DOE OSTI2014

SU-E-J-35: Clinical Performance Evaluation of a Phase II Proton CT Scanner

Purpose: To develop the methodology to evaluate the clinical performance of a Phase II Proton CT scanner Methods: Range errors on the order of 3%-5% constitute a major uncertainty in current charged particle treatment planning based on Hounsfield Unit (HU)-relative stopping power (RSP) calibration curves. Within our proton CT collaboration, we previously developed and built a Phase I proton CT scanner that provided a sensitive area of 9 cm (axial) × 18 cm (in-plane). This scanner served to get initial experience with this new treatment planning tool and to incorporate lessons learned into the next generation design. A Phase II scanner was recently completed and is now undergoing initial performance testing. It will increase the proton acquisition rate and provide a larger detection area of 9 cm x 36 cm. We are now designing a comprehensive evaluation program to test the image quality, imaging dose, and range uncertainty associated with this scanner. The testing will be performed along the lines of AAPM TG 66. Results: In our discussion of the evaluation protocol we identified the following priorities. The image quality of proton CT images, in particular spatial resolution and low-density contrast discrimination, will be evaluated with the Catphan600 phantom. Initial testing showed that the Catphan uniformity phantom did not provide sufficient uniformity; it was thus replaced by a cylindrical water phantom. The imaging dose will be tested with a Catphan dose module, and compared to a typical cone beam CT dose for comparable image quality. Lastly, we developed a dedicated dosimetry range phantom based on the CIRS pediatric head phantom HN715. Conclusion: A formal evaluation of proton CT as a new tool for proton treatment planning is an important task. The availability of the new Phase II proton CT scanner will allow us to perform this task. This research is supported by the National Institute of Biomedical Imaging and Bioengineering of the NIH under award number R01EB013118. The content is solely the responsibility of the authors and does not necessarily represent the official views of the NIH.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗
DOE OSTI2025

Exponential Backoff and Its Security Implications for Safety-Critical OT Protocols over TCP/IP Networks

The convergence of Operational Technology (OT) and Information Technology (IT) networks has become increasingly prevalent with the growth of Industrial Internet of Things (IIoT) applications. This shift, while enabling enhanced automation, remote monitoring, and data sharing, also introduces new challenges related to communication latency and cybersecurity. Oftentimes, legacy OT protocols were adapted to the TCP/IP stack without an extensive review of the ramifications to their robustness, performance, or safety objectives. To further accommodate the IT/OT convergence, protocol gateways were introduced to facilitate the migration from serial protocols to TCP/IP protocol stacks within modern IT/OT infrastructure. However, they often introduce additional vulnerabilities by exposing traditionally isolated protocols to external threats. This study investigates the security and reliability implications of migrating serial protocols to TCP/IP stacks and the impact of protocol gateways, utilizing two widely used OT protocols: Modbus TCP and DNP3. Our protocol analysis finds a significant safety-critical vulnerability resulting from this migration, and our subsequent tests clearly demonstrate its presence and impact. A multi-tiered testbed, consisting of both physical and emulated components, is used to evaluate protocol performance and the effects of device-specific implementation flaws. Through this analysis of specifications and behaviors during communication interruptions, we identify critical differences in fault handling and the impact on time-sensitive data delivery. The findings highlight how reliance on lower-level IT protocols can undermine OT system resilience, and they inform the development of mitigation strategies to enhance the robustness of industrial communication networks.

DNP3↗
DOE OSTI2025

An Evaluation of Representation Learning Methods in Particle Physics Foundation Models

We present a systematic evaluation of representation learning objectives for particle physics within a unified framework. Our study employs a shared transformer-based particle-cloud encoder with standardized preprocessing, matched sampling, and a consistent evaluation protocol on a jet classification dataset. We compare contrastive (supervised and self-supervised), masked particle modeling, and generative reconstruction objectives under a common training regimen. In addition, we introduce targeted supervised architectural modifications that achieve state-of-the-art performance on benchmark evaluations. This controlled comparison isolates the contributions of the learning objective, highlights their respective strengths and limitations, and provides reproducible baselines. We position this work as a reference point for the future development of foundation models in particle physics, enabling more transparent and robust progress across the community.

Chen, Michael [Caltech]↗