Search NASA⌕ Search

SEARCH · Search NASA

Results for “addressing failure”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records

Addressing failures in exascale computing

We present here a report produced by a workshop on “Addressing Failures in Exascale Computing” held in Park City, Utah, August 4–11, 2012. The charter of this workshop was to establish a common taxonomy about resilience across all the levels in a computing system; discuss existing knowledge on resilience across the various hardware and software layers of an exascale system; and build on those results, examining potential solutions from both a hardware and software perspective and focusing on a combined approach. The workshop brought together participants with expertise in applications, system software, and hardware; they came from industry, government, and academia; and their interests ranged from theory to implementation. The combination allowed broad and comprehensive discussions and led to this document, which summarizes and builds on those discussions.

Snir, Marc↗

Addressing Failures in Molten Salt Thermal Energy Storage Tank for Central Receiver Concentrating Solar Power Plants

The thermal energy storage (TES) system is a critical component in concentrated solar power (CSP) plants that increases the plant's capacity factor and economic competitiveness by reducing the levelized cost of energy (LCOE) while simultaneously increasing the value of the delivered energy. Failures including molten-salt leaks and diverse localized cracking after several months to a few years of operation have been reported in hot tanks for CSP plants operating around the world. A model of a molten salt thermal energy storage tank was developed and validated to analyze the impact of different tank design features on the temperature and stress distributions as a function of typical plant operation conditions. Design features included the floor plate thicknesses, friction coefficients between the tank floor and the foundation, and the sparger ring location. Maximum stresses in the tank floor frequently surpassed the yield point of the material during operation, leading to a detriment of the tank's lifetime. Recommendations on design features to improve the reliability of new molten salt tanks for CSP plants are provided.

concentrating solar power↗

Using Rollback Avoidance to Mitigate Failures in Next-Generation Extreme-Scale Systems

High-performance computing (HPC) systems enable scientists to numerically model complex phenomena in many important physical systems. The next major milestone in the development of HPC systems is the construction of the rst supercomputer capable executing more than an exa op, 10 18 oating point operations per second. On systems of this scale, failures will occur much more frequently than on current systems. As a result, resilience is a key obstacle to building next-generation extremescale systems. Coordinated checkpointing is currently the most widely-used mechanism for handling failures on HPC systems. Although coordinated checkpointing remains e ective on current systems, increasing the scale of today's systems to build next-generation systems will increase the cost of fault tolerance as more and more time is taken away from the application to protect against or recover from failure. Rollback avoidance techniques seek to mitigate the cost of checkpoint/restart by allowing an application to continue its execution rather than rolling back to an earlier checkpoint when failures occur. These techniqes include failure prediction and preventive migration, replicated computation, fault-tolerant algorithms, and softwarebased memory fault correction. In this thesis, we examine how rollback avoidance techniques can be used to address failures on extreme-scale systems. Using a combination of analytic modeling and simulation, we evaluate the potential impact of rollback avoidance on these systems. We then present a novel rollback avoidance technique that exploits similarities in application memory. Finally, we examine the feasibility of using this technique to protect against memory faults in kernel memory.

97 MATHEMATICS AND COMPUTING↗

Modeling Freedom From Progression for Standard-Risk Medulloblastoma: A Mathematical Tumor Control Model With Multiple Modes of Failure

Purpose: As pediatric medulloblastoma (MB) is a relatively rare disease, it is important to extract the maximum information from trials and cohort studies. Here, a framework was developed for modeling tumor control with multiple modes of failure and time-to-progression for standard-risk MB, using published pattern of failure data. Methods and Materials: Outcome data for standard-risk MB published after 1990 with pattern of relapse information were used to fit a tumor control dose-response model addressing failures in both the high-dose boost volume and the elective craniospinal volume. Estimates of 5-year event-free survival from 2 large randomized MB trials were used to model the time-to-progression distribution. Uncertainty in freedom from progression (FFP) was estimated by Monte Carlo sampling over the statistical uncertainty in input data. Results: The estimated 5-year FFP (95% confidence intervals [CI]) for craniospinal doses of 15, 18, 24, and 36 Gy while maintaining 54 Gy to the posterior fossa was 77% (95% CI, 70%-81%), 78% (95% CI, 73%-81%), 79% (95% CI, 76%-82%), and 80% (95% CI, 77%-84%) respectively. The uncertainty in FFP was considerably larger for craniospinal doses below 18 Gy, reflecting the lack of data in the lower dose range. Conclusions: Estimates of tumor control and time-to-progression for standard-risk MB provides a data-driven setting for hypothesis generation or power calculations for prospective trials, taking the uncertainties into account. The presented methods can also be applied to incorporate further risk-stratification for example based on molecular biomarkers, when the necessary data become available.

62 RADIOLOGY AND NUCLEAR MEDICINE↗

PV Module and System Reliability Research

While photovoltaic (PV) technologies have experienced widespread success and adoption, continued growth of these technologies - especially new PV technologies - requires ongoing improvements to their reliability and the testing procedures, data, and standards that underpin them. To better understand and address failure mechanisms of PV modules and systems, NREL conducts testing, models failures, analyzes performance data, helps to develop standards, and convenes expert stakeholders. This work is focused both on improving the performance of industry-dominant PV technologies and rapidly de-risking new PV technologies so they can be proven reliable without decades of field testing.

14 SOLAR ENERGY↗

Evaluating Online Global Recovery with Fenix Using Application-Aware In-Memory Checkpointing Techniques

Exascale systems promise the potential for computation at unprecedented scales and resolutions, but achieving exascale by the end of this decade presents significant challenges. A key challenge is due to the very large number of cores and components and the resulting mean time between failures (MTBF) in the order of hours or minutes. Since the typical run times of target scientific applications are longer than this MTBF, fault tolerance techniques will be essential. An important class of failures that must be addressed is process or node failures. While checkpoint/restart (C/R) is currently the most widely accepted technique for addressing processor failures, coordinated, stable-storage-based global C/R might be unfeasible at exascale when the time to checkpoint exceeds the expected MTBF. This paper explores transparent recovery via implicitly coordinated, diskless, application-driven checkpointing as a way to tolerate process failures in MPI applications at exascale. The discussed approach leverages User Level Failure Mitigation (ULFM), which is being proposed as an MPI extension to allow applications to create policies for tolerating process failures. Specifically, this paper demonstrates how different implementations of application-driven in-memory checkpoint storage and recovery compare in terms of performance and scalability. We also experimentally evaluate the effectiveness and scalability of the Fenix online global recovery framework on a production system -the Titan Cray XK7 at ORNL- and demonstrate the ability of Fenix to tolerate dynamically injected failures using the execution of four benchmarks and mini-applications with different behaviors.

97 MATHEMATICS AND COMPUTING↗

Development of Leaching Processes for the Recovery of Rare Earth Elements from Acid Mine Drainage Precipitates

Acid mine drainage (AMD) has been a challenge for mine operators to address and has had significant environmental impacts when there is a failure to address it. Fortunately, there is a regulation in the US that requires the treatment of AMD before water can be discharged into the environment. AMD is generated when sulfide minerals are exposed to water and air from mining. The acidic water then leaches metals from the surrounding rock, creating the potential for environmental contamination of this acidic water containing dissolved metals. AMD can contain high concentrations of metals like iron, aluminum, and manganese and also have been shown to contain trace concentrations of critical minerals, including rare earth elements (REEs). This environmental waste stream is now being researched as a potential source for REEs. There is a patented process for processing and extracting REEs from AMD which produces rare earth oxide preconcentrate (REOP) from AMD treatment precipitates and a final stage mixed rare earth oxide (MREO) product. This body of research examines a selective leaching process for each of these products and determines the activation energy associated with the leaching processes. Acid leaching with HCl was examined to selectively leach REEs from REOP while contaminant metals remained in the solid residue. The maximum leaching recovery of the REEs from the REOP was approximately 90% and an activation energy of 6.4 kJ/mol at pH 3.0. Ammonium chloride leaching was examined to selectively remove contaminant metals from a MREO product. The ammonium chloride process successfully leached major contaminant metals in excess of 85% recovery and had a maximum activation energy of 40.0 kJ/mol.

01 COAL, LIGNITE, AND PEAT↗

Development of Criticality Safety Validation Guidance for NRC-Regulated Activities

Some guidance documents support validation of criticality safety evaluations for many system types, and some accommodate different regulators or regulatory requirements. While these reports address specific applications, no single document applies to all systems and incorporates both traditional and sensitivity/uncertainty-based validation techniques. Oak Ridge National Laboratory will develop a single reference report for the US Nuclear Regulatory Commission (NRC) that provides modern examples and suggested recommendations for a uniform approach for performing and documenting k eff calculation validations applicable to any system containing fissile material. The project will include a literature review and development of a new, comprehensive NUREG/CR document on nuclear criticality safety validation. The NUREG will address the following experiences from recent reviews: 1. Inappropriate critical experiment selection 2. Insufficient trending analysis 3. Incorrectly calculated 95/95 uncertainties 4. Failure to address how validation applies to safety analysis 5. Lack of goodness-of-fit testing in trending analyses 6. Incorrect use of commercial reactor critical data 7. Failure to meet bias determination methodology prerequisites 8. Inappropriate extrapolation of bias and uncertainty 9. Failure to identify validation analysis gaps and weaknesses 10. Inadequate validation analysis documentation the final report will provide guidance on use of trending analysis results, acceptable confidence levels for normality testing and trending analysis, management of validation analysis gaps and weaknesses, use of sensitivity and uncertainty analysis in validation analysis, and potential impact of correlations among critical experiments used in validation. It will also consolidate existing guidance documents using both traditional and sensitivity/uncertainty-based approaches. This document will support NRC-regulated activities, as well as validation used in criticality safety within other regulatory frameworks.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗

Development of criticality safety validation guidance for NRC-regulated activities

Some guidance documents support validation of criticality safety evaluations for many system types, and some accommodate different regulators or regulatory requirements. While these reports address specific applications, no single document applies to all systems and incorporates both traditional and sensitivity/uncertainty-based validation techniques. Oak Ridge National Laboratory will develop a single reference report for the US Nuclear Regulatory Commission (NRC) that provides modern examples and suggested recommendations for a uniform approach for performing and documenting k{sub eff} calculation validations applicable to any system containing fissile material. The project will include a literature review and development of a new, comprehensive NUREG/CR document on nuclear criticality safety validation. The NUREG will address the following experiences from recent reviews: 1) Inappropriate critical experiment selection; 2) Insufficient trending analysis; 3) Incorrectly calculated 95/95 uncertainties; 4) Failure to address how validation applies to safety analysis; 5) Lack of goodness-of-fit testing in trending analyses; 6) Incorrect use of commercial reactor critical data; 7) Failure to meet bias determination methodology prerequisites; 8) Inappropriate extrapolation of bias and uncertainty; 9) Failure to identify validation analysis gaps and weaknesses; and 10) Inadequate validation analysis documentation. The final report will provide guidance on use of trending analysis results, acceptable confidence levels for normality testing and trending analysis, management of validation analysis gaps and weaknesses, use of sensitivity and uncertainty analysis in validation analysis, and potential impact of correlations among critical experiments used in validation. It will also consolidate existing guidance documents using both traditional and sensitivity/uncertainty-based approaches. This document will support NRC-regulated activities, as well as validation used in criticality safety within other regulatory frameworks. (authors)

22 GENERAL STUDIES OF NUCLEAR REACTORS↗

Final Technical Report. FG02-06ER54881

For over 25 years the Applied Superconductivity Center has been integral to the successful development of superconductors for magnetic confinement fusion reactors, culminating in being a part of the “dream team” assembled by the ITER Organization to address the failure of prototype ITER cable-in-conduit conductors to pass testing in the SULTAN facility, an issue that if not addressed successfully would have limited the useful lifetime of the reactor. While being called on through the years to address the immediate needs of facility construction we also continued fundamental studies of fusion applicable superconductors so that magnet designers could make use of improved performance, reliability and cost. The annual Low Temperature Superconductors workshops, jointly organized with LBNL, became the most important meeting for dissemination of data between university laboratories, national laboratories and industry.

2212↗

Simulations of Failure via Three-Dimensional Cracking in Fuel Cladding for Advanced Nuclear Fuels

Enhancing performance of fuel cladding and duct alloys is a key means of increasing fuel burnup. This project will address the failure of fuel cladding via three-dimensional cracking models. Researchers will develop a simulation code for the failure of the fuel cladding and validate the code through experiments. The objective is to develop an algorithm to determine the failure of fuel cladding in the form of three-dimensional cracking due to prolonged exposure under varying conditions of pressure, temperature, chemical environment, and irradiation. This project encompasses the following tasks: 1. Simulate 3D crack initiation and growth under instantaneous and/or fatigue loads using a new variant of the material point method (MPM); 2. Simulate debonding of the materials in the crack path using cohesive elements, considering normal and shear traction separation laws; 3. Determine the crack propagation path, considering damage of the materials incorporated in the cohesive elements to allow the energy release rate to be minimized; 4. Simulate the three-dimensional fatigue crack growth as a function of loading histories; 5. Verify the simulation code by comparing results to theoretical and numerical studies available in the literature; 6. Conduct experiments to observe the crack path and surface profile in unused fuel cladding and validate against simulation results; and 7. Expand the adaptive mesh refinement infrastructure parallel processing environment to allow adaptive mesh refinement at the 3D crack fronts and adaptive mesh merging in the wake of cracks. Fuel cladding is made of materials such as stainless steels and ferritic steels with added alloying elements, which increase stability and durability under irradiation. As fuel cladding is subjected to water, chemicals, fission gas, pressure, high temperatures, and irradiation while in service, understanding performance is essential. In the fast fuel used in advanced burner reactors, simulations of the nuclear fuels are critical to understand the burnup, and thus the fuel efficiency.

11 NUCLEAR FUEL CYCLE AND FUEL MATERIALS↗

DMTN-260: Failure Modes and Error Handling for Prompt Processing

The Prompt Processing system will be responsible for processing roughly a thousand visits per night, and distributing the results in near real time, for at least ten years of Rubin Observatory operations. As such, it must be highly robust to algorithmic, network, and infrastructure failures, ranging from momentary glitches to extended downtimes. DMTN-219 introduced the initial design for the Prompt Processing framework; this document expands on the design to address expected failure modes and recovery strategies for each.

79 ASTRONOMY AND ASTROPHYSICS↗

Addressing technical barriers for reliable, safe removal of fluoride from drinking water using minimally processed bauxite ores

Throughout the developing world, over 200 million people drink groundwater containing fluoride concentrations surpassing the World Health Organization's maximum recommended contaminant level (WHO-MCL) of 1.5 mg F - /L, resulting in adverse health effects ranging from mottled tooth enamel to debilitating skeletal fluorosis. Existing technologies to remove fluoride from water, such as reverse osmosis and filtration with activated alumina, are expensive and are not accessible for low-income communities. Our group and others have demonstrated that minimally-processed bauxite ores can remove fluoride to safe levels at a fraction of the cost of activated alumina. We report results from testing for some technical challenges that may arise in field deployment of this technology at large scale, particularly in a sufficiently robust manner for application in development contexts. Anticipating possible modes of failure and addressing these challenges in advance in the laboratory is particularly important for technologies for vulnerable communities where the opportunity to re-launch pilot projects is limited and small failures can keep solutions from the people that need them most. This work addresses three potential technical barriers to reliable removal of fluoride from drinking water with bauxite ore from Visakhapatnam, Andhra Pradesh, India. We evaluate competition from co-occurring ions, adsorption reversibility, and potability of the product water with regards to leaching of undesirable ions during treatment with various adsorbent materials including raw and thermally activated bauxite, and synthetic gibbsite (a simple model system). Under the conditions tested, the presence of phosphate significantly impacts fluoride adsorption capacity on all adsorbents. Sulfate impacts fluoride adsorption on gibbsite, but not on either bauxite adsorbent. Nitrate and silicate (as silicic acid), tested only with gibbsite, do not affect fluoride adsorption capacity. Both thermally activated bauxite and gibbsite show non-reversible adsorption of fluoride at a pH of 6. Raw bauxite leached arsenic and manganese in a TCLP leaching test at levels indicating the need for ongoing monitoring of treated water, but not precluding safe deployment of bauxite as a fluoride remediation technology. Understanding these phenomena is crucial to ensure field deployment over large diverse geographical areas with aquifers varying in groundwater composition, and for ensuring that the appropriate engineering processes are designed for field implementation of this innovation.

42 ENGINEERING↗

Performance Efficient Multiresilience Using Checkpoint Recovery in Iterative Algorithms

In this paper, we address the design challenge of building multiresilient iterative high-performance computing (HPC) applications. Multiresilience in HPC applications is the ability to tolerate and maintain forward progress in the presence of both soft errors and process failures. We address the challenge by proposing performance models which are useful to design performance efficient and resilient iterative applications. The models consider the interaction between soft error and process failure resilience solutions. We experimented with a linear solver application with two distinct kinds of soft error detectors: one detector has high overhead and high accuracy, whereas the second has low overhead and low accuracy. We show how both can be leveraged for verifying the integrity of checkpointed state used to recover from both soft errors and process failures. Our results show the performance efficiency and resiliency benefit of employing the low overhead detector with high frequency within the checkpoint interval, so that timely soft error recovery can take place, resulting in less re-computed work.

Ashraf, Rizwan↗

Human Factors for Advanced Reactors

Existing light water reactors in the U.S. are primarily large baseload electricity generating facilities. The concept of operations for these plants remains largely unchanged since the advent of commercial nuclear power—the main control room serves as the hub of plant activities and is staffed with multiple licensed operators who work in tandem under the shift supervisor, and staff such as field workers support the control room remotely. While newer plants have brought the advent of digital human-machine interfaces to replace earlier analog and mechanical instrumentation and controls, much of the control process remains unchanged and manual. It is simply a newer version of legacy concepts. Advanced reactors potentially bring considerable changes to the size, fuel type, automation, and staffing of nuclear power plants, necessitating a fundamental shift not just from analog to digital, but further from human to automation, from onsite to remote, from control to monitoring, and from many to few operators. Despite this multitude of parallel evolutions in reactor designs, many of the vendors developing the next generation of reactors represent smaller research and development enterprises. It is therefore not feasible to address all aspects of plant design at the same time. In particular, the competing design aspects of new reactors present a significant challenge to the development of robust and human factored systems at the plant. As vendors develop new reactor designs, much of the early focus is naturally on the fuel and reactor system technology. Looming behind these early advances is the daunting prospect of first-of-a-kind control concepts that have not yet been developed or validated. A failure to address the human element of reactor design early will lead to missed opportunities. The quickest development process is the replication of existing concepts of operations at legacy plants, even when such systems were long ago surpassed by better human-machine technologies outside the nuclear industry. Conversely, attempting to undertake novel concepts of operations late in the design life cycle of a plant could result in protracted development efforts and delays in licensing and deployment. This does not have to happen, and it is imperative that human factors be considered now, early in the design of new reactors.

99 GENERAL AND MISCELLANEOUS↗

Assessment of failure modes of monolithic fuel plates

Monolithic fuel is a fuel form that is considered for the conversion of high performance research reactors. In order to qualify this new fuel system, the fuel plates should meet the safety standards and perform well in reactor. The fuel system must maintain its mechanical integrity, sustain a geometric stability and should have stable and predictable irradiation behavior. The requirement to maintain mechanical integrity under normal operating conditions is primarily demonstrated by a successful testing of fuel plates up to the limiting conditions defined by the fuel performance envelope, including an adequate margin. Although large number of plates have been tested with satisfactory thermo-mechanical performance, post-irradiation examination of plates from previous RERTR-12 experiments have revealed that pillowing occurred in several plates, rendering performance of these plates unacceptable. To address such failures, efforts are underway to define the mechanisms responsible for the in-reactor pillowing, and suggest improvements to the fuel plate design and operational envelope. For this purpose, selected plates from previous experiments were simulated to understand the thermo-mechanical response of the plates to the fission density and thermally induced stresses. Simulation results were then comparatively evaluated with post-irradiation examinations of selected plates. The simulation results and experimental observations established a possible correlation between failure by plate pillowing, high porosity and a presence of tensile stress state. The study has implied that porosity leading to degradation of material properties, accompanied by a sufficiently large tensile stress state can lead to a pillowing-type failure at reactor shutdown. This paper presents these findings, discusses such failure modes, and the influence of fuel burn-up and power on the magnitude of the shutdown-induced tensile stresses.

42 ENGINEERING↗