Search NASA⌕ Search

SEARCH · Search NASA

Results for “evaluation methodology”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 991 records · Page 55

Space Transportation Operations: Assessment of Methodologies and Models

The systems design process for future space transportation involves understanding multiple variables and their effect on lifecycle metrics. Variables such as technology readiness or potential environmental impact are qualitative, while variables such as reliability, operations costs or flight rates are quantitative. In deciding what new design concepts to fund, NASA needs a methodology that would assess the sum total of all relevant qualitative and quantitative lifecycle metrics resulting from each proposed concept. The objective of this research was to review the state of operations assessment methodologies and models used to evaluate proposed space transportation systems and to develop recommendations for improving them. It was found that, compared to the models available from other sources, the operations assessment methodology recently developed at Kennedy Space Center has the potential to produce a decision support tool that will serve as the industry standard. Towards that goal, a number of areas of improvement in the Kennedy Space Center's methodology are identified.

Joglekar, Prafulla↗

Evaluation of an improved algorithm for producing realistic 3D breast software phantoms: Application for mammography

Purpose: This work presents an improved algorithm for the generation of 3D breast software phantoms and its evaluation for mammography. Methods: The improved methodology has evolved from a previously presented 3D noncompressed breast modeling method used for the creation of breast models of different size, shape, and composition. The breast phantom is composed of breast surface, duct system and terminal ductal lobular units, Cooper's ligaments, lymphatic and blood vessel systems, pectoral muscle, skin, 3D mammographic background texture, and breast abnormalities. The key improvement is the development of a new algorithm for 3D mammographic texture generation. Simulated images of the enhanced 3D breast model without lesions were produced by simulating mammographic image acquisition and were evaluated subjectively and quantitatively. For evaluation purposes, a database with regions of interest taken from simulated and real mammograms was created. Four experienced radiologists participated in a visual subjective evaluation trial, as they judged the quality of the simulated mammograms, using the new algorithm compared to mammograms, obtained with the old modeling approach. In addition, extensive quantitative evaluation included power spectral analysis and calculation of fractal dimension, skewness, and kurtosis of simulated and real mammograms from the database. Results: The results from the subjective evaluation strongly suggest that the new methodology for mammographic breast texture creates improved breast models compared to the old approach. Calculated parameters on simulated images such as {beta} exponent deducted from the power law spectral analysis and fractal dimension are similar to those calculated on real mammograms. The results for the kurtosis and skewness are also in good coincidence with those calculated from clinical images. Comparison with similar calculations published in the literature showed good agreement in the majority of cases. Conclusions: The improved methodology generated breast models with increased realism compared to the older model as shown in evaluations of simulated images by experienced radiologists. It is anticipated that the realism will be further improved using an advanced image simulator so that simulated images may be used in feasibility studies in mammography.

60 APPLIED LIFE SCIENCES↗

Evaluation of oxygen reduction activity by the thin-film rotating disk electrode methodology: The effects of potentiodynamic parameters

Here, an accurate and efficient assessment of activity is critical for the research and development of electrocatalysts for oxygen reduction reaction (ORR). Currently, the methodology combining the thin-film rotating disk electrode (TF-RDE) and potentiodynamic polarization is the most commonly used to pre-evaluate ORR activity, acquire kinetic data (i.e., kinetic current, Tafel slope, etc.), and gain understanding of the ORR mechanism. However, it is often neglected that appropriate potentiodynamic parameters have to be chosen to obtain reliable results. We first evaluate the potentiodynamic and potentiostatic polarization measurements with TF-RDE to examine the ORR activity of Pt nanoelectrocatalyst. Furthermore, our results demonstrate that besides depending on the nature of electrocatalyst, the apparent ORR kinetics also strongly depends on the associated potentiodynamic parameters, such as scan rate and scan region, which have a great effect on the coverage of adsorbed OH ad /O ad on Pt surface, thereby affecting the ORR activities of both nanosized and bulk Pt. However, the apparent Tafel slopes remained nearly the same, indicating that the ORR mechanism in all the measurements was not affected by different potentiodynamic parameters.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗

ANALYSIS OF EMERGING NDE TECHNIQUES: METHODS FOR EVALUATING AND IMPLEMENTING CONTINUOUS ONLINE MONITORING

One of the goals of the program for the proactive management of materials degradation (PMMD) is to manage proactively the in-service degradation of metallic components in aging NPPs. As some forms of degradation, such as stress corrosion cracking, are characterized by a long initiation time followed by a rapid growth phase, new inspection or monitoring technologies may be required. New nondestructive evaluation (NDE) techniques that may be needed include techniques to find stress corrosion cracking (SCC) precursors, on-line monitoring techniques to detect cracks as they initiate and grow, as well as advances in NDE technologies. This paper reports on the first part of the development of a methodology to determine the effectiveness of these emerging NDE techniques for managing metallic degradation. This methodology will draw from experience derived from evaluating techniques that have "emerged" in the past. The methodology will follow five stages: a definition of inspection parameters, a technical evaluation, laboratory testing, round robin testing, and the design of a performance demonstration program. This methodology will formalize the path taken for previous techniques and set a predictable course for future NDE techniques. This paper then applies the expert review section of the methodology to the acoustic emission technique to evaluate the use of acoustic emission in performing continuous online monitoring of reactor components.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗

Design Load Analysis for Wave Energy Converters: Preprint

This study demonstrates a systematic methodology for establishing the design loads of a wave energy converter (WEC). The proposed design load methodology incorporates existing design guidelines, where they exist, and follows a typical design progression; namely, advancing from many, quick, order-of-magnitude accurate, conceptual stage design computations to a few, computationally intensive, high-fidelity, design validation simulations. The goal of the study is to streamline and document this process based on quantitative evaluations of the design loads' accuracy at each design step and consideration for the computational efficiency of the entire design process. For the WEC, loads, and site conditions considered, this study demonstrates an efficient and accurate methodology of evaluating the design loads.

16 TIDAL AND WAVE POWER↗

Development of Real-Time Quantitative Safety Evaluation Method for Nuclear Power Plants

Human operators involved with large process control systems such as nuclear power plants (NPPs) perform significant roles in the operation of systems. In this reason, deficiencies in human factors such as poor HSI (Human System Interface) design, procedure, and training, are significant contributing factors to NPPs incidents and accidents. In this regard, many kinds of studies including human reliability analysis (HRA), operation support system, automation, have been conducted in order to reduce human errors. Among these fields of studies, automation has its own specialty since it mainly focuses on eliminating the human error-inducing tasks to reduce human error, while other topics are primarily focusing on reducing human error by aiding human operators during operation. It is commonly believed that replacing human operators to the automated system would guarantee greater efficiency, lower workloads, and fewer human errors. However, these beliefs are not always true in large and complex systems such as NPP, since excessive automation can generate new roles for human operators, change human operators' activities in unexpected ways, and inducing 'out-of-the-loop' (OOTL) problem. Especially, although human error probability drastically increases during abnormal situations in NPP due to overload of information, high workload, and short time available for diagnosis, automation is not actively adopted since conventional machine learning techniques are neither capable nor reliable to handle overwhelmingly complex abnormal situations in NPP. Recently, newly developed 'deep learning' techniques have been actively applied to many fields, and some of these studies proved that the artificial intelligences (AI) based on deep learning techniques show better performance than that of human being. In 2015, deep Q-network (DQN) which is one of the deep learning techniques was developed and applied to train AI that automatically plays various Atari 2800 games, and this AI surpassed the human-level playing in many kind of games. Also in 2016, 'AlphaGo', which was developed by 'Google Deepmind' based on deep learning technique to play the game of Go, was defeated Se-dol Lee who is the World Go champion with score of 4:1. In the point of view that the game of Go is considered as one of the challenging area in machine learning field, it is verified that these new machine learning techniques can perform well, even better than the human being in high-complexity tasks. Previously, most of AI applications in nuclear field were focusing on supporting human operators' tasks such as diagnosis and decision making by rapid classification and prediction, based on fixed training data sets. However, it is expected that AI can be applied to solve more complicated problems (i.e. automation during abnormal states in NPP) by adopting advanced machine learning methodologies. In order to apply the automation system in NPP, firstly it is needed to evaluate the status of the plant quantitatively in real-time, since the information about which state is better (i.e. safe) or worse (i.e. unsafe) is essential for training properly working automation AI. Currently, there are several methodologies for NPP safety evaluation such as probabilistic safety assessment (PSA), safety performance indicator (SPI). However, existing methodologies are not feasible for development of automated system since they are not capable for the cases which are not included in the underlying model, not able to consider delicate changes in the NPP, and hard to evaluate in real-time. Therefore, development of new NPP safety evaluation methodology which can cover previously mentioned requirements is needed to conduct the study on NPP automation further. In this study, the concept of early warning score (EWS) in medical field was adopted to develop the methodology which can evaluate the safety level of the NPP quantitatively and in real-time. (authors)

22 GENERAL STUDIES OF NUCLEAR REACTORS↗

WE-AB-BRA-11: Development of Physiological-Based Virtual Thorax Phantoms for Evaluation of Deformable Image Registration

Purpose: To develop a methodology of constructing physiological-based virtual thorax phantom based on hyperpolarized (HP) gas tagging MRI for evaluating deformable image registration (DIR). Methods: Three healthy subjects were imaged at both the end-of-inhalation (EOI) and the end-of-exhalation (EOE) phases using a high-resolution (2.5mm isovoxel) 3D proton MRI, as well as a hybrid MRI which combines HP gas tagging MRI and a low-resolution (4.5mm isovoxel) proton MRI. A sparse tagging displacement vector field (tDVF) was derived from the HP gas tagging MRI by tracking the displacement of tagging grids between EOI and EOE. Using the tDVF and the high-resolution MR images, we determined the motion model of the entire thorax in the following two steps: 1) the DVF inside of lungs was estimated based on the sparse tDVF using a novel multi-step natural neighbor interpolation method; 2) the DVF outside of lungs was estimated from the DIR between the EOI and EOE images (Velocity AI). The derived motion model was then applied to the high-resolution EOI image to create a deformed EOE image, forming the virtual phantom where the motion model provides the ground truth of deformation. Five DIR methods were evaluated using the developed virtual phantom. Errors in DVF magnitude (Em) and angle (Ea) were determined and compared for each DIR method. Results: Among the five DIR methods, free form deformation produced DVF results that are most closely resembling the ground truth (Em=1.04mm, Ea=6.63°). The two DIR methods based on B-spline produced comparable results (Em=2.04mm, Ea=13.66°; and Em =2.62mm, Ea=17.67°), and the two optical-flow methods produced least accurate results (Em=7.8mm; Ea=53.04°; Em=4.45mm, Ea=31.02°). Conclusion: A methodology for constructing physiological-based virtual thorax phantom based on HP gas tagging MRI has been developed. Initial evaluation demonstrated its potential as an effective tool for robust evaluation of DIR in the lung.

60 APPLIED LIFE SCIENCES↗

Review of Evaluation, Measurement and Verification Approaches Used to Estimate the Load Impacts and Effectiveness of Energy Efficiency Programs

Public and private funding for end-use energy efficiency actions is expected to increase significantly in the United States over the next decade. For example, Barbose et al (2009) estimate that spending on ratepayer-funded energy efficiency programs in the U.S. could increase from $3.1 billion in 2008 to $7.5 and 12.4 billion by 2020 under their medium and high scenarios. This increase in spending could yield annual electric energy savings ranging from 0.58% - 0.93% of total U.S. retail sales in 2020, up from 0.34% of retail sales in 2008. Interest in and support for energy efficiency has broadened among national and state policymakers. Prominent examples include {approx}$18 billion in new funding for energy efficiency programs (e.g., State Energy Program, Weatherization, and Energy Efficiency and Conservation Block Grants) in the 2009 American Recovery and Reinvestment Act (ARRA). Increased funding for energy efficiency should result in more benefits as well as more scrutiny of these results. As energy efficiency becomes a more prominent component of the U.S. national energy strategy and policies, assessing the effectiveness and energy saving impacts of energy efficiency programs is likely to become increasingly important for policymakers and private and public funders of efficiency actions. Thus, it is critical that evaluation, measurement, and verification (EM&V) is carried out effectively and efficiently, which implies that: (1) Effective program evaluation, measurement, and verification (EM&V) methodologies and tools are available to key stakeholders (e.g., regulatory agencies, program administrators, consumers, and evaluation consultants); and (2) Capacity (people and infrastructure resources) is available to conduct EM&V activities and report results in ways that support program improvement and provide data that reliably compares achieved results against goals and similar programs in other jurisdictions (benchmarking). The National Action Plan for Energy Efficiency (2007) presented commonly used definitions for EM&V in the context of energy efficiency programs: (1) Evaluation (E) - The performance of studies and activities aimed at determining the effects and effectiveness of EE programs; (2) Measurement and Verification (M&V) - Data collection, monitoring, and analysis associated with the calculation of gross energy and demand savings from individual measures, sites or projects. M&V can be a subset of program evaluation; and (3) Evaluation, Measurement, and Verification (EM&V) - This term is frequently seen in evaluation literature. EM&V is a catchall acronym for determining both the effectiveness of program designs and estimates of load impacts at the portfolio, program and project level. This report is a scoping study that assesses current practices and methods in the evaluation, measurement and verification (EM&V) of ratepayer-funded energy efficiency programs, with a focus on methods and practices currently used for determining whether projected (ex-ante) energy and demand savings have been achieved (ex-post). M&V practices for privately-funded energy efficiency projects (e.g., ESCO projects) or programs where the primary focus is greenhouse gas reductions were not part of the scope of this study. We identify and discuss key purposes and uses of current evaluations of end-use energy efficiency programs, methods used to evaluate these programs, processes used to determine those methods; and key issues that need to be addressed now and in the future, based on discussions with regulatory agencies, policymakers, program administrators, and evaluation practitioners in 14 states and national experts in the evaluation field. We also explore how EM&V may evolve in a future in which efficiency funding increases significantly, innovative mechanisms for rewarding program performance are adopted, the role of efficiency in greenhouse gas mitigation is more closely linked, and programs are increasingly funded from multiple sources often with multiple program administrators and intended to meet multiple purposes.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗

Development of Simplified Model Test Methods for Creep-Fatigue Evaluation

The Simplified Model Test (SMT) approach is an alternative creep-fatigue evaluation method that no longer requires the use of the damage interaction diagram, or D-diagram. The reason is that the combined effects of creep and fatigue are accounted for in the test data by means of a SMT specimen that is designed to replicate or bound the stress and strain redistribution that occurs in actual components when loaded in the creep regime. However, creep-fatigue experiments on SMT key feature articles are specialized and difficult to perform by the general research community. In this paper, two innovative SMT based creep-fatigue experimental methods are developed and implemented. These newly-developed SMT test methods have resolved all the critical challenges in the SMT key feature article testing and enable the potential of further development of the SMT based creep-fatigue evaluation method into a standard testing method. Scoping test results on Alloy 617 and SS 316H using the newly developed SMT methods are summarized and discussed. The concepts of the SMT methodology for creep-fatigue evaluation are explained.

Wang, Yanli↗

Systems engineering overview for regenerative life-support systems applicable to space habitats

The Life-Support Systems Group of the 1977 NASA Ames Summer Study conducted analyses initially to identify areas where partial or complete closure of the life-support system, for various types of space habitats, should be considered, and to develop a sound methodology for identifying, screening, and evaluating alternative closure schemes. Results of these analyses, including the systems engineering considerations that affect technology-development planning and management for regenerative life-support systems, are discussed. A recommended methodology is also presented as a basis for future technology-development activities in this area.

Spurlock, J.↗

Solar/Thermal Powerplant Simulation

Simulation program evaluates performances and energy costs of diverse solar/thermal powerplant configurations. Approach based on optimizing sizes of collector and storage subsystems to give minimum energy cost for specified plant rating and load factor. Methodology provides for consistent comparative evaluation of solar/thermal powerplants.

Bowyer, J. M.↗

Using computer-extracted image features for modeling of error-making patterns in detection of mammographic masses among radiology residents

Purpose: Mammography is the most widely accepted and utilized screening modality for early breast cancer detection. Providing high quality mammography education to radiology trainees is essential, since excellent interpretation skills are needed to ensure the highest benefit of screening mammography for patients. The authors have previously proposed a computer-aided education system based on trainee models. Those models relate human-assessed image characteristics to trainee error. In this study, the authors propose to build trainee models that utilize features automatically extracted from images using computer vision algorithms to predict likelihood of missing each mass by the trainee. This computer vision-based approach to trainee modeling will allow for automatically searching large databases of mammograms in order to identify challenging cases for each trainee. Methods: The authors’ algorithm for predicting the likelihood of missing a mass consists of three steps. First, a mammogram is segmented into air, pectoral muscle, fatty tissue, dense tissue, and mass using automated segmentation algorithms. Second, 43 features are extracted using computer vision algorithms for each abnormality identified by experts. Third, error-making models (classifiers) are applied to predict the likelihood of trainees missing the abnormality based on the extracted features. The models are developed individually for each trainee using his/her previous reading data. The authors evaluated the predictive performance of the proposed algorithm using data from a reader study in which 10 subjects (7 residents and 3 novices) and 3 experts read 100 mammographic cases. Receiver operating characteristic (ROC) methodology was applied for the evaluation. Results: The average area under the ROC curve (AUC) of the error-making models for the task of predicting which masses will be detected and which will be missed was 0.607 (95% CI,0.564-0.650). This value was statistically significantly different from 0.5 (p < 0.0001). For the 7 residents only, the AUC performance of the models was 0.590 (95% CI,0.537-0.642) and was also significantly higher than 0.5 (p = 0.0009). Therefore, generally the authors’ models were able to predict which masses were detected and which were missed better than chance. Conclusions: The authors proposed an algorithm that was able to predict which masses will be detected and which will be missed by each individual trainee. This confirms existence of error-making patterns in the detection of masses among radiology trainees. Furthermore, the proposed methodology will allow for the optimized selection of difficult cases for the trainees in an automatic and efficient manner.

60 APPLIED LIFE SCIENCES↗

Development of a Technical, Economic, and Risk Assessment Tool for the Evaluation of Work Reduction Opportunities

Efficient and cost-effective operation of a nuclear power plant (NPP) is essential to ensuring long-term economical and safe operation. Multiple cost saving opportunities exist, referred to here as work reduction opportunities (WRO). These WROs reduce plant operating costs by employing various cost-effective strategies (e.g., implementation of modern technologies). Identifying and objectively screening WROs is an essential task to help reduce overall costs. However, there is no comprehensive framework for assessing WROs in the nuclear industry and evaluating their impact on plant operations. This report presents a novel framework for systematically evaluating WROs from a technical, economic, and risk perspective. As NPPs continue to add new technology and implement modernization strategies into their current processes, potential WROs are commonly identified. Although most WROs have the potential to reduce costs, not all opportunities will result in significant cost savings due to unforeseen risks, large implementation costs, or benefits that fall short of expectations. Examples of this can be the result of a technology that is not fully developed, uncertainty in the amount of cost reduction, or difficulties introducing a new process into an organization. These uncertainties can manifest several ways and can result in a WRO with limited cost savings or even a loss of investment. The framework developed emphasizes the importance of effectively screening the WROs from a holistic perspective to objectively identify inefficiencies and ensure a positive impact to the organization. This report presents the Technical, Economic, and Risk Assessment (TERA) as a key methodology for the screening and evaluation of potential WROs. The TERA framework begins with a screening phase where the process is examined through a hybrid combination of Lean Six Sigma and Integrated Operations for Nuclear (ION) guiding principles. This framework examines the current processes using the Lean Six Sigma SIPOC (Suppliers, Inputs, Process, Outputs, Consumers) methodology but retains the ION key elements of People, Technology, Process, and Governance as important factors to the nuclear decision-making process. By combining the principles of Lean Six Sigma and ION, the developed screening process is specific to the nuclear industry in that it systematically evaluates WROs in order to implement new technology that is comprehensively evaluated. The TERA begins by mapping current processes as they relate to WROs and examining the inefficiencies. Furthermore, the created process map can be used to identify and evaluate potential solutions. Using key performance indicators (KPIs), the TERA evaluates each area—technology, economics, and risk—for uncertainties and to perform cost-benefit analysis. The results of the TERA are important KPIs that allow for an evaluation of different processes and technology implementations. This assessment enables decision-makers to compare various WROs based on metrics and then make informed decisions for which opportunity to implement first. This research includes not only the creation of the TERA framework, but also the evaluation of its performance. A case study for screening potential WROs at Southern Nuclear Company is presented that utilizes the TERA methodology. Through the use of TERA, various WROs were screened, and the solutions evaluated for cost-benefit expectations. The report concludes by summarizing the overall effort and implications for utility modernization. The performance of the screening and TERA are discussed as well as the impact on the nuclear industry. The TERA process enables utilities to evaluate and inform investment decisions for WROs and mitigate any potential risks. Through this research, we provide utilities with a valuable framework to optimize operations, reduce costs, and drive continuous process improvement.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗

Using State Estimation Residuals to Detect Abnormal SCADA Data

Detection of manipulated supervisory control and data acquisition (SCADA) data is critically important for the safe and secure operation of modern power systems. In this paper, a methodology of detecting manipulated SCADA data based on state estimation residuals is presented. A framework of the proposed methodology is described. Instead of using original SCADA measurements as the bad data sources, the residuals calculated based on the results of the state estimator are used as the input for the outlier detection process. The BACON algorithm is applied to detect outliers in the state estimation residuals. The IEEE 118-bus system is used as a test case to evaluate the effectiveness of the proposed methodology. The accuracy of the BACON method is compared with that of the 3-σ method for the simulated SCADA measurements and residuals.

SCADA, BACON algorithm, state estimation, outlier ↗

Risk assessment of turbine rotor failure using probabilistic ultrasonic non-destructive evaluations

The study presents a method and application of risk assessment methodology for turbine rotor fatigue failure using probabilistic ultrasonic nondestructive evaluations. A rigorous probabilistic modeling for ultrasonic flaw sizing is developed by incorporating the model-assisted probability of detection, and the probability density function (PDF) of the actual flaw size is derived. Two general scenarios, namely the ultrasonic inspection with an identified flaw indication and the ultrasonic inspection without flaw indication, are considered in the derivation. To perform estimations for fatigue reliability and remaining useful life, uncertainties from ultrasonic flaw sizing and fatigue model parameters are systematically included and quantified. The model parameter PDF is estimated using Bayesian parameter estimation and actual fatigue testing data. The overall method is demonstrated using a realistic application of steam turbine rotor, and the risk analysis under given safety criteria is provided to support maintenance planning.

42 ENGINEERING↗

Augmenting Topic Finding in the NASA Aviation Safety Reporting System using Topic Modeling

Context: The NASA Aviation and Safety Reporting System (ASRS) provides various publications to the aviation community (including individual anonymous reports, Callback, Database Search Requests, Directline, and Alerting Messages). Key to these publications are the timely processing of new reports, which is currently done mostly manually by ASRS staff, and which the volume increases yearly. Aim: We investigate whether existing topic modelling techniques are suitable to ease some of the manual effort, and to enhance it with additional visual cues regarding the process of grouping, sense making and labeling incoming (and previous) reports. Method: We evaluate the applicability of WarpLDA topic modelling results combined with three visualization tools, the first two of which have been extended by us in this work for ASRS: Termite, TopicFlow, and LDAVis. Based on the identified limitations in these tools, we propose a methodology for improving them, and evaluate their outputs using ASRS as our test dataset. Results: The user interfaces of Termite, Topicflow and LDAVis were found insufficient for sense-making of the narratives. Moreover, concerns regarding the stability of results due to the inherent randomness of topic modelling, and the lack of a measurable approach for evaluation against the existing ASRS manual workflow were also noted. Conclusion: While many tools to topic modeling and visualization have been proposed, more work is necessary before they can be applied in practical situations to improve existing manual workflows. The methodology presented and applied in this work contribute towards this effort.

ASRS↗

Resilience Design Patterns: A Structured Approach to Resilience at Extreme Scale

Reliability is a serious concern for future extreme-scale high-performance computing (HPC) systems. Projections based on the current generation of HPC systems and technology roadmaps suggest the prevalence of very high fault rates in future systems. While the HPC community has developed various resilience solutions, application-level techniques as well as system-based solutions, the solution space remains fragmented. There are no formal methods and metrics to integrate the various HPC resilience techniques into composite solutions, nor are there methods to holistically evaluate the adequacy and efficacy of such solutions in terms of their protection coverage, and their performance \& power efficiency characteristics. Additionally, few of the current approaches are portable to newer architectures and software environments that will be deployed on future systems. In this paper, we develop a structured approach to the design, evaluation and optimization of HPC resilience using the concept of design patterns. A design pattern is a general repeatable solution to a commonly occurring problem. We identify the problems caused by various types of faults, errors and failures in HPC systems and the techniques used to deal with these events. Each well-known solution that addresses a specific HPC resilience challenge is described in the form of a pattern. We develop a complete catalog of such resilience design patterns, which may be used by system architects, system software and tools developers, application programmers, as well as users and operators as essential building blocks when designing and deploying resilience solutions. We also develop a design framework that enhances a designer's understanding the opportunities for integrating multiple patterns across layers of the system stack and the important constraints during implementation of the individual patterns. It is also useful for defining mechanisms and interfaces to coordinate flexible fault management across hardware and software components. The resilience patterns and the design framework also enable exploration and evaluation of design alternatives and support optimization of the cost-benefit trade-offs among performance, protection coverage, and power consumption of resilience solutions. Here, the overall goal of this work is to establish a systematic methodology for the design and evaluation of resilience technologies in extreme-scale HPC systems that keep scientific applications running to a correct solution in a timely and cost-efficient manner despite frequent faults, errors, and failures of various types.

97 MATHEMATICS AND COMPUTING↗