Search NASA⌕ Search

SEARCH · Search NASA

Results for “EVALUATION”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 829 records · Page 46

A method to evaluate dose errors introduced by dose mapping processes for mass conserving deformations

Purpose: To present a method to evaluate the dose mapping error introduced by the dose mapping process. In addition, apply the method to evaluate the dose mapping error introduced by the 4D dose calculation process implemented in a research version of commercial treatment planning system for a patient case. Methods: The average dose accumulated in a finite volume should be unchanged when the dose delivered to one anatomic instance of that volume is mapped to a different anatomic instance--provided that the tissue deformation between the anatomic instances is mass conserving. The average dose to a finite volume on image S is defined as d{sub S}=e{sub s}/m{sub S}, where e{sub S} is the energy deposited in the mass m{sub S} contained in the volume. Since mass and energy should be conserved, when d{sub S} is mapped to an image R(d{sub S{yields}R}=d{sub R}), the mean dose mapping error is defined as {Delta}d{sub m}=|d{sub R}-d{sub S}|=|e{sub R}/m{sub R}-e{sub S}/m{sub S}|, where the e{sub R} and e{sub S} are integral doses (energy deposited), and m{sub R} and m{sub S} are the masses within the region of interest (ROI) on image R and the corresponding ROI on image S, where R and S are the two anatomic instances from the same patient. Alternatively, application of simple differential propagation yields the differential dose mapping error, {Delta}d{sub d}=|({partial_derivative}d/{partial_derivative}e)*{Delta}e+({partial_derivative}d/{partial_derivative}m)*{Delta}m|=|((e{sub S}-e{sub R})/m{sub R})-((m{sub S}-m{sub R})/m{sub R}{sup 2})*e{sub R}|={alpha}|d{sub R}-d{sub S}| with {alpha}=m{sub S}/m{sub R}. A 4D treatment plan on a ten-phase 4D-CT lung patient is used to demonstrate the dose mapping error evaluations for a patient case, in which the accumulated dose, D{sub R}={Sigma}{sub S=0}{sup 9}d{sub S{yields}R}, and associated error values ({Delta}D{sub m} and {Delta}D{sub d}) are calculated for a uniformly spaced set of ROIs. Results: For the single sample patient dose distribution, the average accumulated differential dose mapping error is 4.3%, the average absolute differential dose mapping error is 10.8%, and the average accumulated mean dose mapping error is 5.0%. Accumulated differential dose mapping errors within the gross tumor volume (GTV) and planning target volume (PTV) are lower, 0.73% and 2.33%, respectively. Conclusions: A method has been presented to evaluate the dose mapping error introduced by the dose mapping process. This method has been applied to evaluate the 4D dose calculation process implemented in a commercial treatment planning system. The method could potentially be developed as a fully-automatic QA method in image guided adaptive radiation therapy (IGART).

61 RADIATION PROTECTION AND DOSIMETRY↗

A quality assurance framework for the fully automated and objective evaluation of image quality in cone-beam computed tomography

Purpose: Thousands of cone-beam computed tomography (CBCT) scanners for vascular, maxillofacial, neurological, and body imaging are in clinical use today, but there is no consensus on uniform acceptance and constancy testing for image quality (IQ) and dose yet. The authors developed a quality assurance (QA) framework for fully automated and time-efficient performance evaluation of these systems. In addition, the dependence of objective Fourier-based IQ metrics on direction and position in 3D volumes was investigated for CBCT. Methods: The authors designed a dedicated QA phantom 10 cm in length consisting of five compartments, each with a diameter of 10 cm, and an optional extension ring 16 cm in diameter. A homogeneous section of water-equivalent material allows measuring CT value accuracy, image noise and uniformity, and multidimensional global and local noise power spectra (NPS). For the quantitative determination of 3D high-contrast spatial resolution, the modulation transfer function (MTF) of centrally and peripherally positioned aluminum spheres was computed from edge profiles. Additional in-plane and axial resolution patterns were used to assess resolution qualitatively. The characterization of low-contrast detectability as well as CT value linearity and artifact behavior was tested by utilizing sections with soft-tissue-equivalent and metallic inserts. For an automated QA procedure, a phantom detection algorithm was implemented. All tests used in the dedicated QA program were initially verified in simulation studies and experimentally confirmed on a clinical dental CBCT system. Results: The automated IQ evaluation of volume data sets of the dental CBCT system was achieved with the proposed phantom requiring only one scan for the determination of all desired parameters. Typically, less than 5 min were needed for phantom set-up, scanning, and data analysis. Quantitative evaluation of system performance over time by comparison to previous examinations was also verified. The maximum percentage interscan variation of repeated measurements was less than 4% and 1.7% on average for all investigated quality criteria. The NPS-based image noise differed by less than 5% from the conventional standard deviation approach and spatially selective 10% MTF values were well comparable to subjective results obtained with 3D resolution pattern. Determining only transverse spatial resolution and global noise behavior in the central field of measurement turned out to be insufficient. Conclusions: The proposed framework transfers QA routines employed in conventional CT in an advanced version to CBCT for fully automated and time-efficient evaluation of technical equipment. With the modular phantom design, a routine as well as an expert version for assessing IQ is provided. The QA program can be used for arbitrary CT units to evaluate 3D imaging characteristics automatically.

62 RADIOLOGY AND NUCLEAR MEDICINE↗

SU-E-J-12: A New Stereological Method for Tumor Volume Evaluation for Esophageal Cancer

Purpose: Stereological method used to obtain three dimensional quantitative information from two dimensional images is a widely used tool in the study of cells and pathology. But the feasibility of the method for quantitative evaluation of volumes with 3D image data sets for radiotherapy clinical application has not been explored. On the other hand, a quick, easy-to-use and reliable method is highly desired in image-guided-radiotherapy(IGRT) for tumor volume measurement for the assessment of response to treatment. To meet this need, a stereological method for evaluating tumor volumes for esophageal cancer is presented in this abstract. Methods: The stereology method was optimized by selecting the appropriate grid point distances and sample types. 7 patients with esophageal cancer were selected retrospectively for this study, each having pre and post treatment computed tomography (CT) scans. Stereological measurements were performed for evaluating the gross tumor volume (GTV) changes after radiotherapy and the results was compared with the ones by planimetric measurements. Two independent observers evaluated the reproducibility for volume measurement using the new stereological technique. Results: The intraobserver variation in the GTV volume estimation was 3.42±1.68cm3 (the Wilcoxon matched-pairs test Resultwas Z=−1.726,P=0.084>0.05); the interobserver variation in the GTV volume estimation was 22.40±7.23 cm3 (Z=−3.296,P=0.083>0.05), which showed the consistency in GTV volume calculation with the new method for the same and different users. The agreement level between the results from the two techniques was also evaluated. Difference between the measured GTVs was 20.10±5.35 cm3 (Z=−3.101,P=0.089>0.05). Variation of the measurement results using the two techniques was low and clinically acceptable. Conclusion: The good agreement between stereological and planimetric techniques proves the reliability of the stereological tumor volume estimations. The optimized stereological technique described in this abstract may provide a quick, unbiased and reproducible tool for tumor volume estimation for treatment response assessment. Supported by NSFC (#81041107, #81171342 and #31000784)

60 APPLIED LIFE SCIENCES↗

SU-F-T-552: A One-Year Evaluation of the QABeamChecker+ for Use with the CyberKnife System

Purpose: By attaching an adapter plate with fiducial markers to the QA BeamChecker+ (Standard Imaging, Inc., Middleton, WI), the output of the CyberKnife can be accurately, efficiently, and consistently evaluated. The adapter plate, known as the Cutting Board, allows for automated alignment of the QABC+ using the CK’s stereoscopic kV image-based treatment localization system (TLS). Described herein is an evaluation of the system following a year of clinical utilization. Methods: Based on a CT scan of the QABC+ and CB, a treatment plan is generated which delivers a beam to each of the 5 plane-parallel ionization chambers. Following absolute calibration of the CK, the QA plan is delivered, and baseline measurements are acquired (and automatically corrected for temperature and pressure). This test was performed at the beginning of each treatment day for a year. A calibration evaluation (using a water-equivalent slab and short thimble chamber) is performed every four weeks, or whenever the QABC+ detects a deviation of more than 1.0%. Results: During baseline evaluation, repeat measurements (n=10) were performed, with an average output of 0.25% with an SD of 0.11%. As a test of the reposition of the QABC+ and CB, ten additional measurements were performed where between each acquisition, the entire system was removed and re-positioned using the TLS. The average output deviation was 0.30% with a SD of 0.13%. During the course of the year, 187 QABC+ measurements and 13 slab-based measurements were performed. The output measurements of the QABC+ correlated well with slab-based measurements (R2=0.909). Conclusion: By using the QABC+ and CB, daily output was evaluated accurately, efficiently, and consistently. From setup to break-down (including analysis), this test required 5 minutes instead of approximately 15 using traditional techniques (collimator-mounted ionization chambers). Additionally, by automatically saving resultant output deviation to a database, trend analysis was simplified. Spectrum Medical Physics, LLC of Greenville, SC has a consulting contract with Standard Imaging of Middleton, WI.

60 APPLIED LIFE SCIENCES↗

WE-AB-207A-12: HLCC Based Quantitative Evaluation Method of Image Artifact in Dental CBCT

Purpose: Image artifacts are usually evaluated qualitatively via visual observation of the reconstructed images, which is susceptible to subjective factors due to the lack of an objective evaluation criterion. In this work, we propose a Helgason-Ludwig consistency condition (HLCC) based evaluation method to quantify the severity level of different image artifacts in dental CBCT. Methods: Our evaluation method consists of four step: 1) Acquire Cone beam CT(CBCT) projection; 2) Convert 3D CBCT projection to fan-beam projection by extracting its central plane projection; 3) Convert fan-beam projection to parallel-beam projection utilizing sinogram-based rebinning algorithm or detail-based rebinning algorithm; 4) Obtain HLCC profile by integrating parallel-beam projection per view and calculate wave percentage and variance of the HLCC profile, which can be used to describe the severity level of image artifacts. Results: Several sets of dental CBCT projections containing only one type of artifact (i.e. geometry, scatter, beam hardening, lag and noise artifact), were simulated using gDRR, a GPU tool developed for efficient, accurate, and realistic simulation of CBCT Projections. These simulated CBCT projections were used to test our proposed method. HLCC profile wave percentage and variance induced by geometry distortion are about 3∼21 times and 16∼393 times as large as that of the artifact-free projection, respectively. The increase factor of wave percentage and variance are 6 and133 times for beam hardening, 19 and 1184 times for scatter, and 4 and16 times for lag artifacts, respectively. In contrast, for noisy projection the wave percentage, variance and inconsistency level are almost the same with those of the noise-free one. Conclusion: We have proposed a quantitative evaluation method of image artifact based on HLCC theory. According to our simulation results, the severity of different artifact types is found to be in a following order: Scatter>Geometry>Beam hardening>Lag>Noise>Artifact-free in dental CBCT.

60 APPLIED LIFE SCIENCES↗

Evaluations of University of Wisconsin Silicon Carbide Temperature Monitors 300 LO and 400 LO B

Silicon carbide (SiC) temperature monitors 05R4-02-A KG1403 (300 LO) and 05R4-01-A KG1415 (400 LO B) were evaluated at the High Temperature Test Lab (HTTL) to determine their peak irradiation temperatures. HTTL measurements indicate that the peak irradiation temperature for the 300 LO monitor was 295 {+-} 20 °C and the peak irradiation temperature for the 400 LO B monitor was 294 {+-} 25 °C. Two silicon carbide (SiC) temperature monitors irradiated in the Advanced Test Reactor (ATR) were evaluated at the High Temperature Test Lab (HTTL) to determine their peak temperature during irradiation. These monitors were irradiated as part of the University of Wisconsin Pilot Project with a target dose of 3 dpa. Temperature monitors were fabricated from high density (3.203 g/cm 3 ) SiC manufactured by Rohm Haas with a nominal size of 12.5 mm x 1.0 mm x 0.75 mm (see Attachment A). Table 1 provides identification for each monitor with an expected peak irradiation temperature range based on preliminary thermal analysis (see Attachment B). Post irradiation calculations are planned to reduce uncertainties in these calculated temperatures. Since the early 1960s, SiC has been used as a post-irradiation temperature monitor. As noted in Reference 2, several researchers have observed that neutron irradiation induced lattice expansion of SiC annealed out when the post-irradiation annealing temperature exceeds the peak irradiation temperature. As noted in Reference 3, INL uses resistivity measurements to infer peak irradiation temperature from SiC monitors. Figure 1 depicts the equipment at the HTTL used to evaluate the SiC monitors. The SiC monitors are heated in the annealing furnace using isochronal temperature steps that, depending on customer needs, can range from 50 to 800 °C. This furnace is located under a ventilation hood within the stainless steel enclosure. The ventilation system is activated during heating so that any released vapors are vented through this system. Annealing temperatures are recorded using a National Institute of Standards and Technology (NIST) traceable thermocouple inserted into an alumina tube in the furnace. After each isochronal annealing, the specimens are placed in a specialized fixture located in the constant temperature chamber (maintained at 30 °C) for a minimum of 30 minutes. After the 30 minute wait time, each specimen's resistance is measured using the specialized fixture and a calibrated DC power analyzer. This report discusses the evaluation of the SiC monitors and presents the results. Testing was conducted in accordance with Reference 3. Sections 2 and 3 present the data collected for each monitor and provide interpretation of the data. Section 4 presents the evaluated temperature results.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗

Five-Year NRHP Re-Evaluation of Historic Buildings Assessment

The Lawrence Livermore National Laboratory (LLNL) 'Draft Programmatic Agreement among the Department of Energy and the California State Historic Preservation Officer Regarding Operation of Lawrence Livermore National Laboratory' requires a review and re-evaluation of the eligibility of laboratory properties for the National Register of Historic Places (NRHP) every five years. The original evaluation was published in 2005; this report serves as the first five-year re-evaluation. This re-evaluation includes consideration of changes within LLNL to management, to mission, and to the built environment. it also determines the status of those buildings, objects, and districts that were recommended as NRHP-eligible in the 2005 report. Buildings that were omitted from the earlier building list, those that have reached 50 years of age since the original assessment, and new buildings are also addressed in the re-evaluation.

99 GENERAL AND MISCELLANEOUS↗

Guideline for Performing Systematic Approach to Evaluate and Qualify Legacy Documents that Support Advanced Reactor Technology Activity

The establishment of a systematic process for the evaluation of historic technology information for use in advanced reactor licensing is described. Efforts are underway to recover and preserve Experimental Breeder Reactor II and Fast Flux Test Facility historical data. These efforts have generally emphasized preserving information from data-acquisition systems and hard-copy reports and entering it into modern electronic formats suitable for data retrieval and examination. The guidance contained in this document has been developed to facilitate consistent and systematic evaluation processes relating to quality attributes of historic technical information (with focus on sodium-cooled fast reactor (SFR) technology) that will be used to eventually support licensing of advanced reactor designs. The historical information may include, but is not limited to, design documents for SFRs, research-and-development (R&D) data and associated documents, test plans and associated protocols, operations and test data, international research data, technical reports, and information associated with past U.S. Nuclear Regulatory Commission (NRC) reviews of SFR designs. The evaluation process is prescribed in terms of SFR technology, but the process can be used to evaluate historical information for any type of advanced reactor technology. An appendix provides a discussion of typical issues that should be considered when evaluating and qualifying historical information for advanced reactor technology fuel and source terms, based on current light water reactor (LWR) requirements and recent experience gained from Next Generation Nuclear Plant (NGNP).

21 SPECIFIC NUCLEAR REACTORS AND ASSOCIATED PLANTS↗

Guidelines and Specifications for Enhanced Durability Evaluation of Insulating Glass and Vacuum Insulating Glass Units

The National Renewable Energy Laboratory (NREL) has conducted research into durability of advanced fenestration technologies for almost thirty years. NREL has recently begun to research potential methods to enhance durability evaluation of present market insulating glass units (IGU). This document outlines a proposed process flow for adoption of these methods by interested parties. The proposed evaluation process flow described below is still a topic of active research at NREL as well as with industry partners. NREL will periodically update this document based on research results, both internal and external to NREL that may inform development of improved durability evaluation. This document is also meant to serve as a guideline for interested parties who may wish to engage in enhanced durability evaluation of IGU's both internally or with external testing laboratories. NREL will work to support adoption of the suggestions in this guideline by interested partners in the hope that these practices are more broadly adopted by the industry. Further research into this area will continue to inform both adaptations of this guideline as well as development of improved durability evaluation standards.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

Evaluation of Building 23-620, Los Alamos Scientific Laboratory J-3 Office, Nevada National Security Site, Nye County, Nevada

The U.S. Department of Energy (DOE) National Nuclear Security Administration Nevada Field Office (NNSA/NFO) plans to demolish Building 23-620, the Los Alamos Scientific Laboratory (LASL) J-3 Office Building (Nevada State Historic Preservation Office [SHPO] Resource No. B15283) at the Nevada National Security Site (NNSS) in Nye County, Nevada. The NNSA/NFO is implementing a long-term project to modernize the town of Mercury for future mission needs. The project is considered an undertaking subject to review under Title 54 of United States Code (USC) § 306108, commonly known as Section 106 of the National Historic Preservation Act, Title 54 USC § 300101, et seq., and its implementing regulations, Title 36 of the Code of Federal Regulations (36 CFR) Part 800. In 2018, Desert Research Institute (DRI) completed an architectural survey of the town of Mercury. This effort resulted in the identification, recordation, and evaluation of the Mercury Historic District (MHD, SHPO Resource No. D230), including the identification of its contributing elements (Reno et al. 2018). The MHD was recommended eligible for listing in the National Register of Historic Places (NRHP, National Register) under the Secretary of the Interior’s (SOI) Significance Criteria A and C, as defined in 36 CFR Part 60.4, as a significant concentration of buildings and structures with a direct and important association with Cold War-era nuclear testing from 1951 through 1992. It has not been evaluated under Criteria B and D to date. As part of a larger modernization program for Mercury, the NNSA/NFO and the SHPO executed the 2018 Programmatic Agreement between the National Nuclear Security Administration Nevada Field Office and the Nevada State Historic Preservation Officer regarding Modernization and Operational Maintenance of the Nevada National Security Site, at Mercury in Nye County, Nevada (Mercury PA). The Mercury PA includes streamlined Section 106 procedures for undertakings in the MHD based on contributing element categories. Building 23-620 is identified in Appendix C of the Mercury PA as a Category I contributing element, indicating that it might be individually eligible for the NRHP. It is a historic property for the purposes of Section 106 compliance and subject to the stipulations of the Mercury PA. Per Stipulation VI of the Mercury PA, when the Area of Potential Effect (APE) for an undertaking includes Category I elements, the NNSA/NFO must evaluate the Category I elements for individual NRHP eligibility under all of the SOI Significance Criteria prior to initiating any activity that may affect the elements. The purpose of this report is to evaluate Building 23-620 as a potential individually eligible historic property in fulfillment of Stipulation VI of the Mercury PA. The evaluation detailed herein concludes that Building 23-620 is not individually eligible for listing in the NRHP. Although it retains aspects of integrity and continues to contribute to the MHD, it is not individually significant under any of the SOI Significance Criteria.

54 ENVIRONMENTAL SCIENCES↗

A Comparative Evaluation and Selection of High-Temperature Heat Exchangers for Application to Integrated Energy Systems

The following report aims to create a refined and well-structured method for comparatively evaluating heat exchanger technologies for integrated energy systems that caters customers’ specific needs while meeting engineering requirements. For the evaluation, this study elevates previous evaluation metrics, enhances the knowledge base via literature and market surveys, and identifies the figures of merit with robust rationales to enhance the quality of decisions made throughout the proposed heat exchanger evaluation process. The heat exchanger designs evaluated as part of the case study are shell and tube heat exchangers, printed circuit heat exchangers, plate heat exchangers, spiral heat exchangers, and heat pipe heat exchangers. The information presented in this report is meant for industries interested in making a preliminary screening process to identify the most suitable heat exchanger design for their application of interest.

42 ENGINEERING↗

Identification and Evaluation of Thermal Transport Components for Integrated Energy Systems

A proper evaluation and selection of component technologies is a critical aspect in the development of Integrated Energy System (IES). This study aims to investigate the key components and associated technologies required to build high-temperature heat transport systems for IES. Of particular interest is the component technologies needed to design and construct IES that combines advanced nuclear reactors with high-temperature industrial processes. This study particularly delved into knowledge base, evaluation metrics, and state-of-the-art commercial technologies, aiming to facilitate the evaluation and selection process of various thermal transport components. In addition, an evaluation process was proposed in order to assist in the optimal selection of the thermal transport components based on the knowledge base and evaluation metrics investigated through this study.

42 ENGINEERING↗

Evaluation of Heat Exchanger Technology for Integrated Energy System: Methodology and Preliminary Application

This report proposes a novel heat exchanger evaluation method, based on integrated QFD-AHP technique, to support strategic optimal selection of heat exchanger (HX) technology for integrated energy systems. The term “optimal selection” is defined as selecting a product (i.e., HX) that best meets various aspects of end user demand, including both technical and economic requirements. Compared to previous HX evaluation methods, the proposed approach allows decision makers to explicitly take into account the practical demand of customers (i.e., HX end users) which may vary depending on the HX application target. In addition, the proposed evaluation method can be used not only for high-level decisions to select HX type, but also for low-level decisions such as the selection of the key components including HX material or heat transfer fluid. The focus of this study is to describe the new HX evaluation method and to illustrate the overall HX evaluation and selection processes using the three nuclear heat application cases that represent the thermal integration between advanced nuclear reactors and industrial processes. As a preliminary study, three different types of HX, i.e., shell-and-tube heat exchanger, printed circuit heat exchanger, and heat pipe heat exchanger, were compared.

42 ENGINEERING↗

Hydropower Biological Evaluation Toolset Best Practice Guide

Field studies using live fish are necessary for the evaluation of turbine biological performance, but they cannot determine the specific hydraulic conditions or physical stresses experienced by the fish, the locations where deleterious conditions occur, or the specific causes of the biological response. Using the Sensor Fish (SF) sensing technology, this deficiency can be overcome because the SF can be released independently or concurrently with live fish directly into operating infrastructure, and it takes high-frequency measurements of hydraulic conditions such as pressure, acceleration, and rotation acting on a body in situ during downstream passage. The Hydropower Biological Evaluation Tools (HBET; Hou et al. 2018) software package, developed by Pacific Northwest National Laboratory (PNNL), is designed to assemble, organize, and process data collected by the PNNL-developed SF and by live fish. HBET was developed specifically to design SF field studies, process the raw data, and analyze the processed data efficiently and scientifically. Its objectives are to facilitate SF studies focused on characterizing hydraulic conditions and to apply SF data for evaluating the impacts on fish from passage through hydro-structures. HBET allows users to design new studies, analyze data, perform statistical analyses, and evaluate predicted biological responses. It can be used by researchers, turbine designers, hydropower operators, and regulators to evaluate hydro-structures to enhance environmental sustainability in a cost-effective manner.

13 HYDRO ENERGY↗

A Comparative Evaluation and Selection of High-Temperature Heat Exchangers for Application to Integrated Energy Systems

The following report aims to create a refined and well-structured method for comparatively evaluating heat exchanger technologies for integrated energy systems that caters customers’ specific needs while meeting engineering requirements. For the evaluation, this study elevates previous evaluation metrics, enhances the knowledge base via literature and market surveys, and identifies the figures of merit with robust rationales to enhance the quality of decisions made throughout the proposed heat exchanger evaluation process. The heat exchanger designs evaluated as part of the case study are shell and tube heat exchangers, printed circuit heat exchangers, plate heat exchangers, spiral heat exchangers, and heat pipe heat exchangers. The information presented in this report is meant for industries interested in making a preliminary screening process to identify the most suitable heat exchanger design for their application of interest.

42 - ENGINEERING↗

R&D for computational cognitive and social models : foundations for model evaluation through verification and validation (final LDRD report).

Sandia National Laboratories is investing in projects that aim to develop computational modeling and simulation applications that explore human cognitive and social phenomena. While some of these modeling and simulation projects are explicitly research oriented, others are intended to support or provide insight for people involved in high consequence decision-making. This raises the issue of how to evaluate computational modeling and simulation applications in both research and applied settings where human behavior is the focus of the model: when is a simulation 'good enough' for the goals its designers want to achieve? In this report, we discuss two years' worth of review and assessment of the ASC program's approach to computational model verification and validation, uncertainty quantification, and decision making. We present a framework that extends the principles of the ASC approach into the area of computational social and cognitive modeling and simulation. In doing so, we argue that the potential for evaluation is a function of how the modeling and simulation software will be used in a particular setting. In making this argument, we move from strict, engineering and physics oriented approaches to V&V to a broader project of model evaluation, which asserts that the systematic, rigorous, and transparent accumulation of evidence about a model's performance under conditions of uncertainty is a reasonable and necessary goal for model evaluation, regardless of discipline. How to achieve the accumulation of evidence in areas outside physics and engineering is a significant research challenge, but one that requires addressing as modeling and simulation tools move out of research laboratories and into the hands of decision makers. This report provides an assessment of our thinking on ASC Verification and Validation, and argues for further extending V&V research in the physical and engineering sciences toward a broader program of model evaluation in situations of high consequence decision-making.

97 MATHEMATICS AND COMPUTING↗

Collaborative Systemwide Monitoring and Evaluation Project (CSMEP) - Year 5 : Annual Report for FY 2008.

The Collaborative Systemwide Monitoring and Evaluation Project (CSMEP) is a coordinated effort to improve the quality, consistency, and focus of fish population and habitat data to answer key monitoring and evaluation questions relevant to major decisions in the Columbia River Basin. CSMEP was initiated by the Columbia Basin Fish and Wildlife Authority (CBFWA) in October 2003. The project is funded by the Bonneville Power Administration (BPA) through the Northwest Power and Conservation Council's Fish and Wildlife Program (NPCC). CSMEP is a major effort of the federal state and Tribal fish and wildlife managers to develop regionally integrated monitoring and evaluation (M&E) across the Columbia River Basin. CSMEP has focused its work on five monitoring domains: status and trends monitoring of populations and action effectiveness monitoring of habitat, harvest, hatcheries, and the hydrosystem. CSMEP's specific goals are to: (1) interact with federal, state and tribal programmatic and technical entities responsible for M&E of fish and wildlife, to ensure that work plans developed and executed under this project are well integrated with ongoing work by these entities; (2) document, integrate, and make available existing monitoring data on listed salmon, steelhead, bull trout and other fish species of concern; (3) critically assess strengths and weaknesses of these data for answering key monitoring questions; and (4) collaboratively design, implement and evaluate improved M&E methods with other programmatic entities in the Pacific Northwest. During FY2008 CSMEP biologists continued their reviews of the strengths and weaknesses (S&W) of existing subbasin inventory data for addressing monitoring questions about population status and trends at different spatial and temporal scales. Work was focused on Lower Columbia Chinook and steelhead, Snake River fall Chinook, Upper Columbia Spring Chinook and steelhead, and Middle Columbia River Chinook and steelhead. These FY2008 data assessments and others assembled over the years of the CSMEP project can be accessed on the CBFWA public website. The CSMEP web database (http://csmep.streamnet.org/) houses metadata inventories from S&W assessments of Columbia River Basin watersheds that were completed prior to FY2008. These older S&W assessments are maintained by StreamNet, but budget cutbacks prevented us from adding the new FY2008 assessments into the database. Progress was made in FY2008 on CSMEP's goals of collaborative design of improved M&E methods. CSMEP convened two monitoring design workshops in Portland (December 5 and 6, 2007 and February 11 and 12, 2008) to continue exploration of how best to integrate the most robust features of existing M&E programs with new approaches. CSMEP continued to build on this information to develop improved designs and analytical tools for monitoring the status and trends of fish populations and the effectiveness of hatchery and hydrosystem recovery actions within the Columbia River Basin. CSMEP did not do any new work on habitat or harvest effectiveness monitoring designs in FY2008 due to budget cutbacks. CSMEP presented the results of the Snake Basin Pilot Study to the Independent Scientific Review Panel (ISRP) in Portland on December 7, 2008. This study is the finalization of CSMEP's pilot exercise of developing design alternatives across different M&E domains within the Snake River Basin spring/summer Chinook ESU. This work has been summarized in two linked reports (CSMEP 2007a and CSMEP 2007b). CSMEP participants presented many of the analyses developed for the Snake Basin Pilot work at the Western Division American Fisheries Society (AFS) conference in Portland on May 4 to 7, 2008. For the AFS conference CSMEP organized a symposium on regional monitoring and evaluation approaches. A presentation on CSMEP's Cost Integration Database Tool and Salmon Viability Monitoring Simulation Model developed for the Snake Basin Pilot Study was also given to the Pacific Northwest Aquatic monitoring Partnership (PNAMP) steering committee in Portland on August 28, 2008. Further information on CSMEP strengths and weaknesses assessments and monitoring design products for FY2008 is presented in the main text and appendices of this Annual Report as well as being available on CBFWA's public website.

13 HYDRO ENERGY↗

Spring Chinook Salmon Oncorhynchus tshawytscha Supplementation in the Clearwater Subbasin ; Nez Perce Tribal Hatchery Monitoring and Evaluation Project, 2007 Annual Report.

The Nez Perce Tribal Hatchery (NPTH) program has the following goals (BPA, et al., 1997): (1) Protect, mitigate, and enhance Clearwater Subbasin anadromous fish resources; (2) Develop, reintroduce, and increase natural spawning populations of salmon within the Clearwater Subbasin; (3) Provide long-term harvest opportunities for Tribal and non-Tribal anglers within Nez Perce Treaty lands within four generations (20 years) following project initiation; (4) Sustain long-term fitness and genetic integrity of targeted fish populations; (5) Keep ecological and genetic impacts to non-target populations within acceptable limits; and (6) Promote Nez Perce Tribal management of Nez Perce Tribal Hatchery Facilities and production areas within Nez Perce Treaty lands. The NPTH program was designed to rear and release 1.4 million fall and 625,000 spring Chinook salmon. Construction of the central incubation and rearing facility NPTH and spring Chinook salmon acclimation facilities were completed in 2003 and the first full term NPTH releases occurred in 2004 (Brood Year 03). Monitoring and evaluation plans (Steward, 1996; Hesse and Cramer, 2000) were established to determine whether the Nez Perce Tribal Hatchery program is achieving its stated goals. The monitoring and evaluation action plan identifies the need for annual data collection and annual reporting. In addition, recurring 5-year program reviews will evaluate emerging trends and aid in the determination of the effectiveness of the NPTH program with recommendations to improve the program's implementation. This report covers the Migratory Year (MY) 2007 period of the NPTH Monitoring & Evaluation (M&E) program. There are three NPTH spring Chinook salmon treatment streams: Lolo Creek, Newsome Creek, and Meadow Creek. In 2007, Lolo Creek received 140,284 Brood Year (BY) 2006 acclimated pre-smolts at an average weight of 34.9 grams per fish, Newsome Creek received 77,317 BY 2006 acclimated pre-smolts at an average of 24.9 grams per fish, and Meadow Creek received 53,425 BY 2006 direct stream release parr at an average of 4.7 grams per fish. Natural and hatchery origin spring Chinook salmon pre-smolt emigrants were monitored from September - November 2006 and smolts from March-June 2007. Data on adult returns were collected from May-September. A suite of performance measures were calculated including total adult and spawner escapement, juvenile production, and survival probabilities. These measures were used to evaluate the effectiveness of supplementation and provide information on the capacity of the natural environment to assimilate and support supplemented salmon populations.

13 HYDRO ENERGY↗