Search NASA⌕ Search

SEARCH · Search NASA

Results for “Testing metrics”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 217 records · Page 12
All sourcesNASA NTRSDOE OSTI
DOE OSTI2017

Multi-mode evaluation of power-maximizing cross-flow turbine controllers

A general method for predicting and evaluating the performance of three candidate cross-flow turbine power-maximizing controllers is presented in this paper using low-order dynamic simulation, scaled laboratory experiments, and full-scale field testing. For each testing mode and candidate controller, performance metrics quantifying energy capture (ability of a controller to maximize power), variation in torque and rotation rate (related to drive train fatigue), and variation in thrust loads (related to structural fatigue) are quantified for two purposes. First, for metrics that could be evaluated across all testing modes, we considered the accuracy with which simulation or laboratory experiments could predict performance at full scale. Second, we explored the utility of these metrics to contrast candidate controller performance. For these turbines and set of candidate controllers, energy capture was found to only differentiate controller performance in simulation, while the other explored metrics were able to predict performance of the full-scale turbine in the field with various degrees of success. Finally, effects of scale between laboratory and full-scale testing are considered, along with recommendations for future improvements to dynamic simulations and controller evaluation.

13 HYDRO ENERGY↗
DOE OSTI2015

Clinical Validation of 4-Dimensional Computed Tomography Ventilation With Pulmonary Function Test Data

Purpose: A new form of functional imaging has been proposed in the form of 4-dimensional computed tomography (4DCT) ventilation. Because 4DCTs are acquired as part of routine care for lung cancer patients, calculating ventilation maps from 4DCTs provides spatial lung function information without added dosimetric or monetary cost to the patient. Before 4DCT-ventilation is implemented it needs to be clinically validated. Pulmonary function tests (PFTs) provide a clinically established way of evaluating lung function. The purpose of our work was to perform a clinical validation by comparing 4DCT-ventilation metrics with PFT data. Methods and Materials: Ninety-eight lung cancer patients with pretreatment 4DCT and PFT data were included in the study. Pulmonary function test metrics used to diagnose obstructive lung disease were recorded: forced expiratory volume in 1 second (FEV1) and FEV1/forced vital capacity. Four-dimensional CT data sets and spatial registration were used to compute 4DCT-ventilation images using a density change–based and a Jacobian-based model. The ventilation maps were reduced to single metrics intended to reflect the degree of ventilation obstruction. Specifically, we computed the coefficient of variation (SD/mean), ventilation V20 (volume of lung ≤20% ventilation), and correlated the ventilation metrics with PFT data. Regression analysis was used to determine whether 4DCT ventilation data could predict for normal versus abnormal lung function using PFT thresholds. Results: Correlation coefficients comparing 4DCT-ventilation with PFT data ranged from 0.63 to 0.72, with the best agreement between FEV1 and coefficient of variation. Four-dimensional CT ventilation metrics were able to significantly delineate between clinically normal versus abnormal PFT results. Conclusions: Validation of 4DCT ventilation with clinically relevant metrics is essential. We demonstrate good global agreement between PFTs and 4DCT-ventilation, indicating that 4DCT-ventilation provides a reliable assessment of lung function. Four-dimensional CT ventilation enables exciting opportunities to assess lung function and create functional avoidance radiation therapy plans. The present work provides supporting evidence for the integration of 4DCT-ventilation into clinical trials.

62 RADIOLOGY AND NUCLEAR MEDICINE↗
NASA NTRSDate not supplied

Rapid Aero Modeling for Urban Air Mobility Aircraft in Computational Experiments

Rapid Aero Modeling (RAM) applied to computational testing, RAM-C, is an approach to efficiently and automatically obtain aerodynamic models during computational investigations. RAM-C is designed to estimate models appropriate for flight dynamics studies and simulations. The approach responds to a demand for experimental efficiency and model fidelity that has increased with growing aircraft complexity and aerodynamic nonlinearities associated with hybrid and electric vertical takeoff and landing (eVTOL) aircraft. In an Urban Air Mobility (UAM) transportation system, it is expected that aircraft will embrace many features from both airplanes and rotorcraft. These vehicles present many more factors than conventional aircraft which can lead to increased computational costs and missed key factor interactions when applying traditional testing and modeling methods. RAM-C provides feedback loops around computational codes to rapidly guide testing toward aerodynamic models meeting user-defined fidelity goals. It combines and extends concepts from design of experiment theory and aircraft system identification theory that allow the user the freedom to choose, in advance of the test, a specific level of fidelity in terms of prediction error. RAM-C only collects enough data required to meet the user-specified prediction error requirements thus saving computational time and resources. The overall achievable fidelity of the final model also depends on the accuracy of the test facility, or in this case, the computational modeling approach. Previous studies to support development of the RAM-T process were conducted in wind tunnel tests to assess potential metrics, algorithms, and procedures. This paper presents results from the next steps taken and tests conducted for the development of RAM-C technology and highlights some of the unique features of RAM applied eVTOL configurations in a computational study.

Aerodynamics↗
DOE OSTI2017

Nano-indentation and laser-induced damage testing in optical multilayer-dielectric gratings [Nanomechanics and laser-induced damage in optical multilayer dielectric gratings]

Here, we demonstrate how a nanomechanical test can be used to generate metrics to complement laser-induced–damage testing (LIDT) measurements and show that differences in optical performance of the gratings (arising from changes in cleaning process and/or fabrication methods) can be related to their mechanical reliability. Data are presented on LIDT measurements in diffractive gratings of silica deposited on optical multilayers. The nano-indentation response of the diffraction gratings is measured in a new mode that allows for the extraction of a measurable metric characterizing the brittleness of the gratings, as well as their ductility. We show that lower LIDT’s are positively correlated with an increased grating brittleness, and therefore identify a nanomechanical approach to describe LIDT’s. We present extensive numerical simulations of nano-indentation tests and identify different deformation modes including stretching, shear concentration, and bending as precursors to mechanical failure in the nano-indentation test. The effects of geometrical inhomogeneities on enhanced stress generation in these gratings are specifically examined and addressed.

36 MATERIALS SCIENCE↗
NASA NTRS1985

Preliminary evaluation of a micro-based repeated measures testing system

A need exists for an automated performance test system to study the effects of various treatments which are of interest to the aerospace medical community, i.e., the effects of drugs and environmental stress. The ethics and pragmatics of such assessment demand that repeated measures in small groups of subjects be the customary research paradigm. Test stability, reliability-efficiency and factor structure take on extreme significance; in a program of study by the U.S. Navy, 80 percent of 150 tests failed to meet minimum metric requirements. The best is being programmed on a portable microprocessor and administered along with tests in their original formats in order to examine their metric properties in the computerized mode. Twenty subjects have been tested over four replications on a 6.0 minute computerized battery (six tests) and which compared with five paper and pencil marker tests. All tests achieved stability within the four test sessions, reliability-efficiencies were high (r greater than .707 for three minutes testing), and the computerized tests were largely comparable to the paper and pencil version from which they were derived. This computerized performance test system is portable, inexpensive and rugged.

Kennedy, Robert S.↗
DOE OSTI2020

Developing Flexible, Networked Lighting Control Systems That Reliably Save Energy in California Buildings

An important strategy to meet California's ambitious energy efficiency goals is to use innovative wireless communications, embedded sensors, data analytics and controls to significantly reduce lighting energy use in commercial buildings. This project developed a suite of networked lighting solutions to further this goal. The technologies include a platform for low-cost sensing, distributed intelligence and communications, the “PermaMote,” which is a self-powered sensor and controller for lighting applications. The project team also developed a task ambient daylighting system that integrates sensors with data-driven daylighting control using an open communication interface, called the “Readings-At-Desk” (RAD) system. To address the problem of building occupants being confused about how to operate traditional lighting control systems, the research team created content that could be the basis for a user interface standard for lighting controls. Finally, to address the difficulty of ensuring that advanced lighting control systems actually deliver their promised energy savings, the project team developed a new method for evaluating and specifying lighting systems’ performance. The research team validated these technologies in the laboratory, showing significant lighting energy savings, up to 73% for the PermaMote sensor system from occupancy control and daylight dimming features, compared to the same light source (LED replacement lamps) operated via simple on/off scheduling. The project team also developed a proposed standard lighting data model and user interface elements, which were contributed to the ANSI Lighting Systems Committee (C137) for standardization. Existing data models are incomplete and inconsistent, whereas the lighting-specific data model developed here is clear and comprehensive, to serve as a starting point for creating common, universally agreed upon semantic definitions of key lighting parameters, to promote interoperability. For the task on verifiable performance of lighting systems, the project team developed a more effective metric for capturing the actual energy impact of a lighting system over time — the energy usage intensity (kWh/ft2/year). Three commercial lighting systems were tested in FLEXLAB® using this new metric, and the tests show a wide range in the accuracy of the self-reported energy-use metric, from 0.5% to 28% error compared to direct measurement of lighting energy using dedicated submeters. Overall, the project team estimates that these advanced technologies can reduce California office lighting energy use by 20% (above and beyond normal advanced lighting controls mandated by Title 24), resulting in about 1,600 GWh/year in savings.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
DOE OSTI2019

Performance Metric for Cementitious Waste Form Inventory Release in the Integrated Disposal Facility

This report describes work supporting Phase 1 of the 2018 Department of Energy (DOE) Office of River Protection’s (ORP) Grand Challenge recipient proposal “Cementitious Immobilization of Treated LAW and Ancillary Benefits to Risk Reduction in Cross Site Transfer”. Phase 1 was to establish a performance metric for grout leachability against which laboratory leach data (as observed diffusivities) can be assed, followed by a comparison of historical test data against this metric. Historically, advancements in the cementitious (grout) waste form performance have been made over the last 40 years, with the majority of progress achieved within the past 8 years as part of the Washington River Protection Solutions, LLC testing program. This report presents a summary of that effort relevant to the Hanford Site.

12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗
NASA NTRS2017

Kepler Planet Detection Metrics: Per-Target Flux-Level Transit Injection Tests of TPS for Data Release 25

Quantifying the ability of a transiting planet survey to recover transit signals has commonly been accomplished through Monte-Carlo injection of transit signals into the observed data and subsequent running of the signal search algorithm (Gilliland et al., 2000; Weldrake et al., 2005; Burke et al., 2006). In order to characterize the performance of the Kepler pipeline (Twicken et al., 2016; Jenkins et al., 2017) on a sample of over 200,000 stars, two complementary injection and recovery tests are utilized:1. Injection of a single transit signal per target into the image or pixel-level data, hereafter referred to as pixel-level transit injection (PLTI), with subsequent processing through the Photometric Analysis (PA), Presearch Data Conditioning (PDC), Transiting Planet Search (TPS), and Data Validation (DV) modules of the Kepler pipeline. The PLTI quantification of the Kepler pipeline's completeness has been described previously by Christiansen et al. (2015, 2016); the completeness of the final SOC 9.3 Kepler pipeline acting on the Data Release 25 (DR25) light curves is described by Christiansen (2017).2. Injection of multiple transit signals per target into the normalized flux time series data with a subsequent transit search using a stream-lined version of the Transiting Planet Search (TPS) module. This test, hereafter referred to as flux-level transit injection (FLTI), is the subject of this document. By running a heavily modified version of TPS, FLTI is able to perform many injections on selected targets and determine in some detail which injected signals are recoverable. Significant numerical efficiency gains are enabled by precomputing the data conditioning steps at the onset of TPS and limiting the search parameter space (i.e., orbital period, transit duration, and ephemeris zero-point) to a small region around each injected transit signal.The PLTI test has the advantage that it follows transit signals through all processing steps of the Kepler pipeline, and the recovered signals can be further classified as planet candidates or false positives in the exact same manner as detections from the nominal (i.e., observed) pipeline run (Twicken et al., 2016, Thompson et al., in preparation). To date, the PLTI test has been the standard means of measuring pipeline completeness averaged over large samples of targets (Christiansen et al., 2015, 2016; Christiansen, 2017). However, since the PLTI test uses only one injection per target, it does not elucidate individual-target variations in pipeline completeness due to differences in stellar properties or astrophysical variability. Thus, we developed the FLTI test to provide a numerically efficient way to fully map individual targets and explore the performance of the pipeline in greater detail. The FLTI tests thereby allow a thorough validation of the pipeline completeness models (such as window function (Burke and Catanzarite, 2017a), detection efficiency (Burke Catanzarite, 2017b), etc.) across the spectrum of Kepler targets (i.e., various astrophysical phenomena and differences in instrumental noise). Tests during development of the FLTI capability revealed that there are significant target-to-target variations in the detection efficiency.

DR25↗
NASA NTRSDate not supplied

High-Altitude ADS-B/GPS LPV Flight Tests on a NASA ER-2 Research Airplane

The research presented in this paper describes the conceptual design of a system architecture that integrates Automatic Dependent Surveillance-Broadcast (ADS-B) and Global Positioning System (GPS) Localizer Performance Vertical (LPV) guidance technology onto a unique high-altitude research airplane: a United States Air Force (USAF) / Lockheed Martin (Bethesda, Maryland) Aeronautics U-2S airplane. The design features modern display capabilities to provide air-to-air surveillance and precision navigation, to adhere to Federal Aviation Administration (FAA) certification standards for operations in upper Class E airspace. The National Aeronautics and Space Administration (NASA) variant of the U-2S, now called the Earth Resources (ER-2) airplane, remains unrivaled in the art of sustained high-altitude flight for scientific expeditions. Capable of routinely cruising above flight level (FL) 650 that had been considered, at inception, the domain of only the most elite experimental research aircraft types. Nicknamed the Dragon Lady, this U-2S research testbed is still one of the most advanced aircraft in the world. The exceptional military design of the vehicle, security guidelines, and the performance envelope of the ER-2 posed unique challenges to the integration of modern civilian avionics. ADS-B epitomizes the next generation of surveillance technology, incorporating both air and ground aspects. ADS-B provides air traffic control (ATC) with a more accurate picture of the three-dimensional positioning of aircraft in various phases of flight, including en route, terminal, approach, and ground operations. The airborne surveillance system broadcasts its identification, position, altitude, velocity, and other information. GPS LPV represents a significant advancement in aviation technology, emphasizing the pivotal role that GPS and Performance-Based Navigation concepts will play in the foreseeable future. This technology represents a shift from sensor-based navigation to performance-based navigation, allowing for more flexible and efficient use of airspace. This research described herein is structured as follows: Section II, “Systems Background,” provides a systems background and description of an ADS-B and GPS LPV system equipped on the high-altitude ER-2 research airplane to satisfy the FAA airworthiness requirements for high-altitude flight operations. Section III, “Flight Test System,” describes the flight-test airplane systems. Section IV, “Analysis of GPS SBAS, Safety, and Ground Tests,” provides an overview of the GPS Satellite-Based Augmentation System (SBAS) and an analysis of the GPS LPV metrics, safety, and ground tests. Section V, “High-Altitude Flight Tests,” describes the high-altitude flight tests, including 3 flight-test results, analysis, human factors, and lessons learned. Section VI, the conclusion, draws insights from the lessons learned, discusses the design challenges associated with ADS-B and GPS LPV, and showcases the paramount significance of these pivotal technologies in aircraft surveillance and navigation.

Ricardo A. Arteaga↗
NASA NTRS1995

Rotorcraft flight control design using quantitative feedback theory and dynamic crossfeeds

A multi-input, multi-output controls design with robust crossfeeds is presented for a rotorcraft in near-hovering flight using quantitative feedback theory (QFT). Decoupling criteria are developed for dynamic crossfeed design and implementation. Frequency dependent performance metrics focusing on piloted flight are developed and tested on 23 flight configurations. The metrics show that the resulting design is superior to alternative control system designs using conventional fixed-gain crossfeeds and to feedback-only designs which rely on high gains to suppress undesired off-axis responses. The use of dynamic, robust crossfeeds prior to the QFT design reduces the magnitude of required feedback gain and results in performance that meets current handling qualities specifications relative to the decoupling of off-axis responses. The combined effect of the QFT feedback design following the implementation of low-order, dynamic crossfeed compensator successfully decouples ten of twelve off-axis channels. For the other two channels it was not possible to find a single, low-order crossfeed that was effective.

Cheng, Rendy P.↗
NASA NTRS2001

The Application of Time-Frequency Methods to HUMS

This paper reports the study of four time-frequency transforms applied to vibration signals and presents a new metric for comparing them for fault detection. The four methods to be described and compared are the Short Time Frequency Transform (STFT), the Choi-Williams Distribution (WV-CW), the Continuous Wavelet Transform (CWT) and the Discrete Wavelet Transform (DWT). Vibration data of bevel gear tooth fatigue cracks, under a variety of operating load levels, are analyzed using these methods. The new metric for automatic fault detection is developed and can be produced from any systematic numerical representation of the vibration signals. This new metric reveals indications of gear damage with all of the methods on this data set. Analysis with the CWT detects mechanical problems with the test rig not found with the other transforms. The WV-CW and CWT use considerably more resources than the STFT and the DWT. More testing of the new metric is needed to determine its value for automatic fault detection and to develop methods of setting the threshold for the metric.

Pryor, Anna H.↗
DOE OSTI2013

Non-Intrusive Load Monitoring Assessment: Literature Review and Laboratory Protocol

Unlike sub-metering, which requires that individual loads are equipped with their own meters, nonintrusive load monitors (NILM) use algorithms to discern the individual loads from a single metering point. Although a number of these technologies are now commercially available to utilities and homeowners, with several more to become available in the coming years, it is not yet understood how accurately these technologies can disaggregate a home’s electrical load. Pacific Northwest National Laboratory (PNNL) and its partners at the Northwest Energy Efficiency Alliance (NEEA) have identified and acquired six of these technologies and will be testing them in the PNNL Lab Homes as well as the owner-occupied Residential Test Bed operated by NEEA. As a result of this testing, PNNL and NEEA will evaluate the selected NILMs against a repeatable test protocol so that their accuracy can be compared across NILM technologies. In the field deployment in the NEEA Test Bed Homes, PNNL and NEEA will evaluate the real-world accuracy of the NILMs responding to dynamic loads. To evaluate the accuracy of NILM technologies regardless of their intended application, a standardized test protocol is required that applies representative loads and establishes consistent metrics against which the NILM can be evaluated and compared. To establish this standardized test protocol, a literature review was conducted to identify any test protocols or standardized testing approaches currently in use. The literature review indicated that no consistent conventions were currently in place for measuring the accuracy of these technologies. Consequently, PNNL developed a testing protocol and metrics to provide the basis for quantifying and analyzing the accuracy of commercially available NILM technologies. This report discusses the results of the literature review and the proposed test protocol and metrics in more detail.

NILM↗
DOE OSTI2017

Experimental Evaluation of Grid Support Enabled PV Inverter Response to Abnormal Grid Conditions

As revised interconnection standards for grid-tied photovoltaic (PV) inverters address new advanced grid support functions (GSFs), there is increasing interest in inverter performance in the case of abnormal grid conditions. The growth of GSF-enabled inverters has outpaced the industry standards that define their operation, although recently published updates to UL1741 Supplement SA define test conditions for GSFs such as volt-var control, frequency-watt control, and voltage/frequency ride-through, among others. This paper describes the results of a comparative experimental evaluation on four commercially available, three-phase PV inverters in the 24.0-39.8 kVA power range on their GSF capability and its effect on abnormal grid condition response. The evaluation examined the impact particular GSF implementations have on run-on times during islanding conditions, peak voltages in load rejection overvoltage scenarios, and peak currents during single-phase and three-phase fault events for individual inverters. Testing results indicated a wide variance in the performance of GSF enabled inverters to various test cases.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2015

WE-AB-201-00: Treatment Planning System Commissioning and QA

Treatment planning systems (TPS) are a cornerstone of modern radiation therapy. Errors in their commissioning or use can have a devastating impact on many patients. To support safe and high quality care, medical physicists must conduct efficient and proper commissioning, good clinical integration, and ongoing quality assurance (QA) of the TPS. AAPM Task Group 53 and related publications have served as seminal benchmarks for TPS commissioning and QA over the past two decades. Over the same time, continuing innovations have made the TPS even more complex and more central to the clinical process. Medical goals are now expressed in terms of the dose and margins around organs and tissues that are delineated from multiple imaging modalities (CT, MR and PET); and even temporally resolved (i.e., 4D) imaging. This information is passed on to optimization algorithms to establish accelerator movements that are programmed directly for IMRT, VMAT and stereotactic treatments. These advances have made commissioning and QA of the TPS much more challenging. This education session reviews up-to-date experience and guidance on this subject; including the recently published AAPM Medical Physics Practice Guideline (MPPG) #5 “Commissioning and QA of Treatment Planning Dose Calculations: Megavoltage Photon and Electron Beams”. Treatment Planning System Commissioning and QA: Challenges and Opportunities (Greg Salomons) This session will provide some key background and review publications describing prominent incidents relating to TPS commissioning and QA. Traditional approaches have been hardware and feature oriented. They aim to establish a functional configuration and establish specifications for regular testing of features (like dose calculation) to assure stable operation and detect failures. With the advent of more complex systems, more patient-specific testing has also been adopted. A number of actual TPS defects will be presented along with heuristics for identifying similar defects in the future. Finally, the Gamma test has become a popular metric for reporting TPS Commissioning and QA results. It simplifies complex testing into a numerical index, but noisy data and casual application can make it misleading. A brief review of the issues around the use of the Gamma test will be presented. TPS commissioning and QA: A process orientation and application of control charts (Michael Sharpe) A framework for commissioning a treatment planning system will be presented, focusing on preparations, practical aspects of configuration, priorities, specifications, and establishing performance. The complexity of the modern TPS make modular testing of features inadequate, and modern QA tools can provide “too much information” about the performance of techniques like IMRT and VMAT. We have adopted a process orientation and quality tools, like control charts, for ongoing TPS QA and assessment of patient-specific tests. The trending nature of these tools reveals the overall performance of the TPS system, and quantifies the variations that arise from individual plans, discrete calculations, and experimentation based on discrete measurements. Examples demonstrating application of these tools to TPS QA will be presented. TPS commissioning and QA: Incorporating the entire planning process (Sasa Mutic) The TPS and its features do not perform in isolation. Instead, the features and modules are key components in a complex process that begins with CT Simulation and extends to treatment delivery, along with image guidance and verification. Most importantly, the TPS is used by people working in a multi-disciplinary environment. It is very difficult to predict the outcomes of human interactions with software. Therefore, an interdisciplinary approach to training, commissioning and QA will be presented, along with an approach to the physics chart check and end-to-end testing as a tool for TPS QA. The role of standardization and automation in QA will also be discussed. The recommendations of MPPG #5 and practical implementation strategies (Jennifer Smilowitz) The recently published recommendations from Task Group No. 244, Medical Physics Practice Guideline on Commissioning and QA of Treatment Planning Dose Calculations: Megavoltage Photon and Electron Beams will be presented. The recommendations focus on the validation of commissioning data and dose calculations. Tolerance values for non-IMRT beam configurations are summarized based on established criteria and data collected by the IROC. More stringent evaluation criteria for IMRT dose calculations are suggested to test the limitations of the TPS dose algorithms for advanced delivery conditions. The MPPG encourages users to create a suite of validation tests for dose calculation for various conditions for static photon beams, heterogeneities, IMRT/VMAT and electron beams. This test suite is intended to be used for subsequent testing, including TPS software upgrades. In the past, the recommendations of some reports have not been widely implemented due to practical limitations. Implementation strategies, tools and processes developed by multiple centers for efficient and “do-able” MPPG # 5 testing will be presented, as well as a discussion on the overall validation experience. Learning Objectives: Identify some of the key documents relevant for TPS commissioning and QA Understand strategies for testing TPS software Gain a practical knowledge of the Gamma test criteria Increase familiarity with the process of commissioning a TPS Learn about the use of Control Charts for TPS QA Review the role of the TPS in the overall planning process Increase awareness of the link between TPS QA and chart checking Gain an increased appreciation for the importance of interdisciplinary communication Understand the new recommendations from MPPG #5 on TPS Dose Algorithm Commissioning and QC/QA Learn practical implementation processes and tools for MPPG #5 validation recommendations.

60 APPLIED LIFE SCIENCES↗
DOE OSTI2015

WE-AB-201-01: Treatment Planning System Commissioning and QA: Challenges and Opportunities

Treatment planning systems (TPS) are a cornerstone of modern radiation therapy. Errors in their commissioning or use can have a devastating impact on many patients. To support safe and high quality care, medical physicists must conduct efficient and proper commissioning, good clinical integration, and ongoing quality assurance (QA) of the TPS. AAPM Task Group 53 and related publications have served as seminal benchmarks for TPS commissioning and QA over the past two decades. Over the same time, continuing innovations have made the TPS even more complex and more central to the clinical process. Medical goals are now expressed in terms of the dose and margins around organs and tissues that are delineated from multiple imaging modalities (CT, MR and PET); and even temporally resolved (i.e., 4D) imaging. This information is passed on to optimization algorithms to establish accelerator movements that are programmed directly for IMRT, VMAT and stereotactic treatments. These advances have made commissioning and QA of the TPS much more challenging. This education session reviews up-to-date experience and guidance on this subject; including the recently published AAPM Medical Physics Practice Guideline (MPPG) #5 “Commissioning and QA of Treatment Planning Dose Calculations: Megavoltage Photon and Electron Beams”. Treatment Planning System Commissioning and QA: Challenges and Opportunities (Greg Salomons) This session will provide some key background and review publications describing prominent incidents relating to TPS commissioning and QA. Traditional approaches have been hardware and feature oriented. They aim to establish a functional configuration and establish specifications for regular testing of features (like dose calculation) to assure stable operation and detect failures. With the advent of more complex systems, more patient-specific testing has also been adopted. A number of actual TPS defects will be presented along with heuristics for identifying similar defects in the future. Finally, the Gamma test has become a popular metric for reporting TPS Commissioning and QA results. It simplifies complex testing into a numerical index, but noisy data and casual application can make it misleading. A brief review of the issues around the use of the Gamma test will be presented. TPS commissioning and QA: A process orientation and application of control charts (Michael Sharpe) A framework for commissioning a treatment planning system will be presented, focusing on preparations, practical aspects of configuration, priorities, specifications, and establishing performance. The complexity of the modern TPS make modular testing of features inadequate, and modern QA tools can provide “too much information” about the performance of techniques like IMRT and VMAT. We have adopted a process orientation and quality tools, like control charts, for ongoing TPS QA and assessment of patient-specific tests. The trending nature of these tools reveals the overall performance of the TPS system, and quantifies the variations that arise from individual plans, discrete calculations, and experimentation based on discrete measurements. Examples demonstrating application of these tools to TPS QA will be presented. TPS commissioning and QA: Incorporating the entire planning process (Sasa Mutic) The TPS and its features do not perform in isolation. Instead, the features and modules are key components in a complex process that begins with CT Simulation and extends to treatment delivery, along with image guidance and verification. Most importantly, the TPS is used by people working in a multi-disciplinary environment. It is very difficult to predict the outcomes of human interactions with software. Therefore, an interdisciplinary approach to training, commissioning and QA will be presented, along with an approach to the physics chart check and end-to-end testing as a tool for TPS QA. The role of standardization and automation in QA will also be discussed. The recommendations of MPPG #5 and practical implementation strategies (Jennifer Smilowitz) The recently published recommendations from Task Group No. 244, Medical Physics Practice Guideline on Commissioning and QA of Treatment Planning Dose Calculations: Megavoltage Photon and Electron Beams will be presented. The recommendations focus on the validation of commissioning data and dose calculations. Tolerance values for non-IMRT beam configurations are summarized based on established criteria and data collected by the IROC. More stringent evaluation criteria for IMRT dose calculations are suggested to test the limitations of the TPS dose algorithms for advanced delivery conditions. The MPPG encourages users to create a suite of validation tests for dose calculation for various conditions for static photon beams, heterogeneities, IMRT/VMAT and electron beams. This test suite is intended to be used for subsequent testing, including TPS software upgrades. In the past, the recommendations of some reports have not been widely implemented due to practical limitations. Implementation strategies, tools and processes developed by multiple centers for efficient and “do-able” MPPG #5 testing will be presented, as well as a discussion on the overall validation experience. Learning Objectives: Identify some of the key documents relevant for TPS commissioning and QA Understand strategies for testing TPS software Gain a practical knowledge of the Gamma test criteria Increase familiarity with the process of commissioning a TPS Learn about the use of Control Charts for TPS QA Review the role of the TPS in the overall planning process Increase awareness of the link between TPS QA and chart checking Gain an increased appreciation for the importance of interdisciplinary communication Understand the new recommendations from MPPG #5 on TPS Dose Algorithm Commissioning and QC/QA Learn practical implementation processes and tools for MPPG #5 validation recommendations.

60 APPLIED LIFE SCIENCES↗
DOE OSTI2015

WE-AB-201-02: TPS Commissioning and QA: A Process Orientation and Application of Control Charts

Treatment planning systems (TPS) are a cornerstone of modern radiation therapy. Errors in their commissioning or use can have a devastating impact on many patients. To support safe and high quality care, medical physicists must conduct efficient and proper commissioning, good clinical integration, and ongoing quality assurance (QA) of the TPS. AAPM Task Group 53 and related publications have served as seminal benchmarks for TPS commissioning and QA over the past two decades. Over the same time, continuing innovations have made the TPS even more complex and more central to the clinical process. Medical goals are now expressed in terms of the dose and margins around organs and tissues that are delineated from multiple imaging modalities (CT, MR and PET); and even temporally resolved (i.e., 4D) imaging. This information is passed on to optimization algorithms to establish accelerator movements that are programmed directly for IMRT, VMAT and stereotactic treatments. These advances have made commissioning and QA of the TPS much more challenging. This education session reviews up-to-date experience and guidance on this subject; including the recently published AAPM Medical Physics Practice Guideline (MPPG) #5 “Commissioning and QA of Treatment Planning Dose Calculations: Megavoltage Photon and Electron Beams”. Treatment Planning System Commissioning and QA: Challenges and Opportunities (Greg Salomons) This session will provide some key background and review publications describing prominent incidents relating to TPS commissioning and QA. Traditional approaches have been hardware and feature oriented. They aim to establish a functional configuration and establish specifications for regular testing of features (like dose calculation) to assure stable operation and detect failures. With the advent of more complex systems, more patient-specific testing has also been adopted. A number of actual TPS defects will be presented along with heuristics for identifying similar defects in the future. Finally, the Gamma test has become a popular metric for reporting TPS Commissioning and QA results. It simplifies complex testing into a numerical index, but noisy data and casual application can make it misleading. A brief review of the issues around the use of the Gamma test will be presented. TPS commissioning and QA: A process orientation and application of control charts (Michael Sharpe) A framework for commissioning a treatment planning system will be presented, focusing on preparations, practical aspects of configuration, priorities, specifications, and establishing performance. The complexity of the modern TPS make modular testing of features inadequate, and modern QA tools can provide “too much information” about the performance of techniques like IMRT and VMAT. We have adopted a process orientation and quality tools, like control charts, for ongoing TPS QA and assessment of patient-specific tests. The trending nature of these tools reveals the overall performance of the TPS system, and quantifies the variations that arise from individual plans, discrete calculations, and experimentation based on discrete measurements. Examples demonstrating application of these tools to TPS QA will be presented. TPS commissioning and QA: Incorporating the entire planning process (Sasa Mutic) The TPS and its features do not perform in isolation. Instead, the features and modules are key components in a complex process that begins with CT Simulation and extends to treatment delivery, along with image guidance and verification. Most importantly, the TPS is used by people working in a multi-disciplinary environment. It is very difficult to predict the outcomes of human interactions with software. Therefore, an interdisciplinary approach to training, commissioning and QA will be presented, along with an approach to the physics chart check and end-to-end testing as a tool for TPS QA. The role of standardization and automation in QA will also be discussed. The recommendations of MPPG #5 and practical implementation strategies (Jennifer Smilowitz) The recently published recommendations from Task Group No. 244, Medical Physics Practice Guideline on Commissioning and QA of Treatment Planning Dose Calculations: Megavoltage Photon and Electron Beams will be presented. The recommendations focus on the validation of commissioning data and dose calculations. Tolerance values for non-IMRT beam configurations are summarized based on established criteria and data collected by the IROC. More stringent evaluation criteria for IMRT dose calculations are suggested to test the limitations of the TPS dose algorithms for advanced delivery conditions. The MPPG encourages users to create a suite of validation tests for dose calculation for various conditions for static photon beams, heterogeneities, IMRT/VMAT and electron beams. This test suite is intended to be used for subsequent testing, including TPS software upgrades. In the past, the recommendations of some reports have not been widely implemented due to practical limitations. Implementation strategies, tools and processes developed by multiple centers for efficient and “do-able” MPPG #5 testing will be presented, as well as a discussion on the overall validation experience. Learning Objectives: Identify some of the key documents relevant for TPS commissioning and QA Understand strategies for testing TPS software Gain a practical knowledge of the Gamma test criteria Increase familiarity with the process of commissioning a TPS Learn about the use of Control Charts for TPS QA Review the role of the TPS in the overall planning process Increase awareness of the link between TPS QA and chart checking Gain an increased appreciation for the importance of interdisciplinary communication Understand the new recommendations from MPPG #5 on TPS Dose Algorithm Commissioning and QC/QA Learn practical implementation processes and tools for MPPG #5 validation recommendations.

60 APPLIED LIFE SCIENCES↗
DOE OSTI2015

WE-AB-201-03: TPS Commissioning and QA: Incorporating the Entire Planning Process

Treatment planning systems (TPS) are a cornerstone of modern radiation therapy. Errors in their commissioning or use can have a devastating impact on many patients. To support safe and high quality care, medical physicists must conduct efficient and proper commissioning, good clinical integration, and ongoing quality assurance (QA) of the TPS. AAPM Task Group 53 and related publications have served as seminal benchmarks for TPS commissioning and QA over the past two decades. Over the same time, continuing innovations have made the TPS even more complex and more central to the clinical process. Medical goals are now expressed in terms of the dose and margins around organs and tissues that are delineated from multiple imaging modalities (CT, MR and PET); and even temporally resolved (i.e., 4D) imaging. This information is passed on to optimization algorithms to establish accelerator movements that are programmed directly for IMRT, VMAT and stereotactic treatments. These advances have made commissioning and QA of the TPS much more challenging. This education session reviews up-to-date experience and guidance on this subject; including the recently published AAPM Medical Physics Practice Guideline (MPPG) #5 “Commissioning and QA of Treatment Planning Dose Calculations: Megavoltage Photon and Electron Beams”. Treatment Planning System Commissioning and QA: Challenges and Opportunities (Greg Salomons) This session will provide some key background and review publications describing prominent incidents relating to TPS commissioning and QA. Traditional approaches have been hardware and feature oriented. They aim to establish a functional configuration and establish specifications for regular testing of features (like dose calculation) to assure stable operation and detect failures. With the advent of more complex systems, more patient-specific testing has also been adopted. A number of actual TPS defects will be presented along with heuristics for identifying similar defects in the future. Finally, the Gamma test has become a popular metric for reporting TPS Commissioning and QA results. It simplifies complex testing into a numerical index, but noisy data and casual application can make it misleading. A brief review of the issues around the use of the Gamma test will be presented. TPS commissioning and QA: A process orientation and application of control charts (Michael Sharpe) A framework for commissioning a treatment planning system will be presented, focusing on preparations, practical aspects of configuration, priorities, specifications, and establishing performance. The complexity of the modern TPS make modular testing of features inadequate, and modern QA tools can provide “too much information” about the performance of techniques like IMRT and VMAT. We have adopted a process orientation and quality tools, like control charts, for ongoing TPS QA and assessment of patient-specific tests. The trending nature of these tools reveals the overall performance of the TPS system, and quantifies the variations that arise from individual plans, discrete calculations, and experimentation based on discrete measurements. Examples demonstrating application of these tools to TPS QA will be presented. TPS commissioning and QA: Incorporating the entire planning process (Sasa Mutic) The TPS and its features do not perform in isolation. Instead, the features and modules are key components in a complex process that begins with CT Simulation and extends to treatment delivery, along with image guidance and verification. Most importantly, the TPS is used by people working in a multi-disciplinary environment. It is very difficult to predict the outcomes of human interactions with software. Therefore, an interdisciplinary approach to training, commissioning and QA will be presented, along with an approach to the physics chart check and end-to-end testing as a tool for TPS QA. The role of standardization and automation in QA will also be discussed. The recommendations of MPPG #5 and practical implementation strategies (Jennifer Smilowitz) The recently published recommendations from Task Group No. 244, Medical Physics Practice Guideline on Commissioning and QA of Treatment Planning Dose Calculations: Megavoltage Photon and Electron Beams will be presented. The recommendations focus on the validation of commissioning data and dose calculations. Tolerance values for non-IMRT beam configurations are summarized based on established criteria and data collected by the IROC. More stringent evaluation criteria for IMRT dose calculations are suggested to test the limitations of the TPS dose algorithms for advanced delivery conditions. The MPPG encourages users to create a suite of validation tests for dose calculation for various conditions for static photon beams, heterogeneities, IMRT/VMAT and electron beams. This test suite is intended to be used for subsequent testing, including TPS software upgrades. In the past, the recommendations of some reports have not been widely implemented due to practical limitations. Implementation strategies, tools and processes developed by multiple centers for efficient and “do-able” MPPG #5 testing will be presented, as well as a discussion on the overall validation experience. Learning Objectives: Identify some of the key documents relevant for TPS commissioning and QA Understand strategies for testing TPS software Gain a practical knowledge of the Gamma test criteria Increase familiarity with the process of commissioning a TPS Learn about the use of Control Charts for TPS QA Review the role of the TPS in the overall planning process Increase awareness of the link between TPS QA and chart checking Gain an increased appreciation for the importance of interdisciplinary communication Understand the new recommendations from MPPG #5 on TPS Dose Algorithm Commissioning and QC/QA Learn practical implementation processes and tools for MPPG #5 validation recommendations.

60 APPLIED LIFE SCIENCES↗