SEARCH · Search NASA

Results for “evaluation methodology”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 433 records · Page 24

All sources NASA NTRS DOE OSTI

NASA NTRS1993

Space network scheduling benchmark: A proof-of-concept process for technology transfer

This paper describes a detailed proof-of-concept activity to evaluate flexible scheduling technology as implemented in the Request Oriented Scheduling Engine (ROSE) and applied to Space Network (SN) scheduling. The criteria developed for an operational evaluation of a reusable scheduling system is addressed including a methodology to prove that the proposed system performs at least as well as the current system in function and performance. The improvement of the new technology must be demonstrated and evaluated against the cost of making changes. Finally, there is a need to show significant improvement in SN operational procedures. Successful completion of a proof-of-concept would eventually lead to an operational concept and implementation transition plan, which is outside the scope of this paper. However, a high-fidelity benchmark using actual SN scheduling requests has been designed to test the ROSE scheduling tool. The benchmark evaluation methodology, scheduling data, and preliminary results are described.

Moe, Karen↗

DOE OSTI2023

Evaluation of the Sum-of-Fractions Methodology for Water and Polyethylene Moderated Systems

Sum-of-Fractions is a method intended to assure a subcritical margin for aqueous solutions and slurries of fissionable isotopes. The method indicates that a system is subcritical if the sum of the ratios of the mass of each isotope in a mixture to its individual minimum subcritical mass limit is less than or equal to one. The basis of the Sum-of Fractions has historically been derived from allowances given in ANSI/ANS-8.15-1981. However, the allowance was removed in ANSI/ANS-8.15-2014 due to a lack of technical basis. A methodology was developed to assess the validity of using the Sum-of-Fractions for water or polyethylene moderated systems for the following nuclides: 232 U, 233 U, 234 U, 235 U, 237 Np, 236 Pu, 238 Pu, 239 Pu, 240 Pu, 241 Pu, 242 Pu, 241 Am, 242 mAm, 243 Am, 242 Cm, 243 Cm, 244 Cm, 245 Cm, 246 Cm, 247 Cm, 249 Cf, and 251 Cf. The methodology uses available benchmark data for mixtures of 233 U, 235 U, and 239 Pu to establish the calculational margin, and a mass limit reduction to establish the margin of subcriticality. Water or polyethylene moderated and reflected mixtures containing the nuclides are evaluated with SCALE 6.2.4. Including the calculational margin, subcritical mass limits for each nuclide were computed for optimally water or polyethylene moderated and fully reflected systems. These masses were used to create nuclide mixtures in which the sum of the mass to subcritical mass limit ratios is one. The various nuclide mixtures were modeled over a range of moderation and demonstrate the k eff does not exceed the calculational margin. For additional assurance of subcriticality, a significant mass reduction is applied to each computed minimum critical mass of the nuclides without adequate benchmark data consistent with the method in ANSI/ANS-8.15-2014.

07 ISOTOPE AND RADIATION SOURCES↗

NASA NTRS1991

IMPAC: An Integrated Methodology for Propulsion and Airframe Control

The National Aeronautics and Space Administration is actively involved in the development of enabling technologies that will lead towards aircraft with new/enhanced maneuver capabilities such as Short Take-Off Vertical Landing (STOVL) and high angle of attack performance. Because of the high degree of dynamic coupling between the airframe and propulsion systems of these types of aircraft, one key technology is the integration of the flight and propulsion control. The NASA Lewis Research Center approach to developing Integrated Flight Propulsion Control (IFPC) technologies is an in-house research program referred to as IMPAC (Integrated Methodology for Propulsion and Airframe Control). The goals of IMPAC are to develop a viable alternative to the existing integrated control design methodologies that will allow for improved system performance and simplicity of control law synthesis and implementation, and to demonstrate the applicability of the methodology to a supersonic STOVL fighter aircraft. Based on some preliminary control design studies that included evaluation of the existing methodologies, the IFPC design methodology that is emerging at the Lewis Research Center consists of considering the airframe and propulsion system as one integrated system for an initial centralized controller design and then partitioning the centralized controller into separate airframe and propulsion system subcontrollers to ease implementation and to set meaningful design requirements for detailed subsystem control design and evaluation. An overview of IMPAC is provided and detailed discussion of the various important design and evaluation steps in the methodology are included.

Garg, Sanjay↗

NASA NTRS2009

On Applying the Prognostic Performance Metrics

Prognostics performance evaluation has gained significant attention in the past few years. As prognostics technology matures and more sophisticated methods for prognostic uncertainty management are developed, a standardized methodology for performance evaluation becomes extremely important to guide improvement efforts in a constructive manner. This paper is in continuation of previous efforts where several new evaluation metrics tailored for prognostics were introduced and were shown to effectively evaluate various algorithms as compared to other conventional metrics. Specifically, this paper presents a detailed discussion on how these metrics should be interpreted and used. Several shortcomings identified, while applying these metrics to a variety of real applications, are also summarized along with discussions that attempt to alleviate these problems. Further, these metrics have been enhanced to include the capability of incorporating probability distribution information from prognostic algorithms as opposed to evaluation based on point estimates only. Several methods have been suggested and guidelines have been provided to help choose one method over another based on probability distribution characteristics. These approaches also offer a convenient and intuitive visualization of algorithm performance with respect to some of these new metrics like prognostic horizon and alpha-lambda performance, and also quantify the corresponding performance while incorporating the uncertainty information.

Saxena, Abhinav↗

NASA NTRS1992

Probabilistic evaluation of uncertainties and risks in aerospace components

This paper summarizes a methodology developed at NASA Lewis Research Center which computationally simulates the structural, material, and load uncertainties associated with Space Shuttle Main Engine (SSME) components. The methodology was applied to evaluate the scatter in static, buckling, dynamic, fatigue, and damage behavior of the SSME turbo pump blade. Also calculated are the probability densities of typical critical blade responses, such as effective stress, natural frequency, damage initiation, most probable damage path, etc. Risk assessments were performed for different failure modes, and the effect of material degradation on the fatigue and damage behaviors of a blade were calculated using a multi-factor interaction equation. Failure probabilities for different fatigue cycles were computed and the uncertainties associated with damage initiation and damage propagation due to different load cycle were quantified. Evaluations on the effects of mistuned blades on a rotor were made; uncertainties in the excitation frequency were found to significantly amplify the blade responses of a mistuned rotor. The effects of the number of blades on a rotor were studied. The autocorrelation function of displacements and the probability density function of the first passage time for deterministic and random barriers for structures subjected to random processes also were computed. A brief discussion was included on the future direction of probabilistic structural analysis.

Shah, A. R.↗

NASA NTRS2003

Adaptation of Mesoscale Weather Models to Local Forecasting

Methodologies have been developed for (1) configuring mesoscale numerical weather-prediction models for execution on high-performance computer workstations to make short-range weather forecasts for the vicinity of the Kennedy Space Center (KSC) and the Cape Canaveral Air Force Station (CCAFS) and (2) evaluating the performances of the models as configured. These methodologies have been implemented as part of a continuing effort to improve weather forecasting in support of operations of the U.S. space program. The models, methodologies, and results of the evaluations also have potential value for commercial users who could benefit from tailoring their operations and/or marketing strategies based on accurate predictions of local weather. More specifically, the purpose of developing the methodologies for configuring the models to run on computers at KSC and CCAFS is to provide accurate forecasts of winds, temperature, and such specific thunderstorm-related phenomena as lightning and precipitation. The purpose of developing the evaluation methodologies is to maximize the utility of the models by providing users with assessments of the capabilities and limitations of the models. The models used in this effort thus far include the Mesoscale Atmospheric Simulation System (MASS), the Regional Atmospheric Modeling System (RAMS), and the National Centers for Environmental Prediction Eta Model ( Eta for short). The configuration of the MASS and RAMS is designed to run the models at very high spatial resolution and incorporate local data to resolve fine-scale weather features. Model preprocessors were modified to incorporate surface, ship, buoy, and rawinsonde data as well as data from local wind towers, wind profilers, and conventional or Doppler radars. The overall evaluation of the MASS, Eta, and RAMS was designed to assess the utility of these mesoscale models for satisfying the weather-forecasting needs of the U.S. space program. The evaluation methodology includes objective and subjective verification methodologies. Objective (e.g., statistical) verification of point forecasts is a stringent measure of model performance, but when used alone, it is not usually sufficient for quantifying the value of the overall contribution of the model to the weather-forecasting process. This is especially true for mesoscale models with enhanced spatial and temporal resolution that may be capable of predicting meteorologically consistent, though not necessarily accurate, fine-scale weather phenomena. Therefore, subjective (phenomenological) evaluation, focusing on selected case studies and specific weather features, such as sea breezes and precipitation, has been performed to help quantify the added value that cannot be inferred solely from objective evaluation.

Manobianco, John T.↗

DOE OSTI2022

A risk-informed assessment tool to support review of license amendment requests for spent nuclear fuel dry cask storage systems

We report that while risk-informed safety evaluations are common methodologies used by the U.S. Nuclear Regulatory Commission (NRC), their application in the field of commercial spent nuclear fuel (SNF) dry storage has received limited attention. For instance, license amendment requests (LARs) for SNF dry cask storage systems (DCSSs) are evaluated according to U.S. NRC standard review plans (SRPs). However, risk-informed LAR review strategies could improve the understanding of critical system modifications, and lead to more predictable, efficient, and consistent LAR review processes. A methodology to develop a risk-informed LAR review tool is presented. Although focused on DCSSs, this methodology can be used to develop similar tools for a broad spectrum of safety-relevant systems. The end product (i.e., the tool) includes a tree diagram to help visualize the review risk significances assigned to a predefined set of system modifications and supporting rationale documentation. Instructions support correct tool application and provide the user with a path to incorporate currently unevaluated modifications in the tool structure. The demonstration of the methodology leads to the conclusion that risk-informed reviews of DCSS LARs are possible, although the tool precision could benefit from additional risk information. Further, the results indicate that many typical LARs involve low-risk or medium-risk modifications.

12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗

DOE OSTI2020

Sandia's Integrated Methodology for Energy and Infrastructure Resilience Analysis

Sandia National Laboratories' (Sandia) Resilient Energy Systems (RES) Strategic Initiative is establishing a strategic vision for U.S. energy systems' resilience through threat-informed research and development, enabling energy and interdependent infrastructure systems to successfully adapt in an environment of accelerating change. A key challenge in promoting energy systems resilience lies in developing rigorous resilience analysis methodologies to quantify system performance. Resilience analysis methodologies should enable evaluation of the consequences of various disruptions and the relative effectiveness of potential mitigations. To address this challenge, RES synthesized the common components of Sandia's resilience frameworks into an integrated methodology for energy and infrastructure resilience analysis. This report documents, demonstrates, and extends this methodology.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗

DOE OSTI2023

A Review of the Evaluation of Building Energy Code Compliance in the United States

Building energy codes are essential tools for achieving energy efficiency in buildings. However, the full energy savings potential of these codes can only be realized if buildings are constructed in compliance with them. Therefore, evaluating building energy code compliance is crucial in bridging the gap between the energy efficiency requirements set by energy codes and the actualized energy savings achieved. An energy code compliance evaluation serves as a mechanism to assess construction practices, evaluate the effectiveness of code enforcement, identify gaps in compliance, and guide strategies for improvement through training and education. Conducting code compliance evaluation activities involves field studies that require careful design and significant resources. Historically, more emphasis has been placed on developing and adopting building energy codes, while efforts to evaluate compliance have been relatively limited and lacking consistent approaches. The passage of the 2009 American Recovery and Reinvestment Act (ARRA), which mandated that states create plans for achieving 90% compliance within eight years, stimulated the need for an energy code compliance evaluation. As a result, federal, state, and local governments, and utilities have invested in the development of methodologies and tools for code compliance evaluation studies. This paper reviews the code compliance evaluation studies conducted in the United States over the past three decades. It describes and compares the methodologies and metrics used to assess building energy code compliance, summarizes the general elements and steps involved in the evaluation process, and discusses common issues in these studies. Over time, code compliance evaluation methodologies have evolved from isolated development within individual states, regions, and utilities, to widely accepted protocols applicable across different states and local jurisdictions. There has been a transition in compliance metrics, shifting from historical compliance rates to energy-consumption-oriented approaches.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

NASA NTRS1992

Measurement and analysis of operating system fault tolerance

This paper demonstrates a methodology to model and evaluate the fault tolerance characteristics of operational software. The methodology is illustrated through case studies on three different operating systems: the Tandem GUARDIAN fault-tolerant system, the VAX/VMS distributed system, and the IBM/MVS system. Measurements are made on these systems for substantial periods to collect software error and recovery data. In addition to investigating basic dependability characteristics such as major software problems and error distributions, we develop two levels of models to describe error and recovery processes inside an operating system and on multiple instances of an operating system running in a distributed environment. Based on the models, reward analysis is conducted to evaluate the loss of service due to software errors and the effect of the fault-tolerance techniques implemented in the systems. Software error correlation in multicomputer systems is also investigated.

Lee, I.↗

DOE OSTI2021

Development of an Assessment Methodology That Enables the Nuclear Industry to Evaluate Adoption of Advanced Automation

Nuclear power has a crucial role in providing safe, reliable, and economical carbon-free electricity for today and the future. For continued operation, many of the existing United States nuclear power plants will begin the subsequent license renewal process for extending their operating license periods. As plants extend their expected operating lifetimes, there is a significant opportunity to modernize. These plants have a much stronger business case with these extended mission periods to modernize and significantly enhance their economic viability in current and future energy markets by implementing digital technologies that support innovation, efficiency gains, and business-model transformation. Ensuring continued safety and reliability is crucial. Transformative digital technologies—including automation—that fundamentally change the concept of operation for the nuclear power plant operating model requires a critical focus on the human and technology integration element. Further, the nuclear industry has historically been reluctant to modernize due to having a risk adverse culture and lack of clarity for a transformative new state vision (Joe & Remer, 2019; Thomas et al., 2020). Common barriers include (1) the perceived value and return on investment (ROI) of digital technology, (2) the perceived risk associated with licensing, regulatory, and cybersecurity, and (3) insufficient guidance for performing digital modifications to power generation systems. This work presents a methodology to address these barriers and support the industry in adopting advanced automation and digital technology through developing a transformative vision and implementation strategy that will address the human and technology integration element. This research leverages previous LWRS Program and industry results. It draws specifically on previous LWRS Program research in the areas of advanced alarm systems, computer-based procedures, model informed decision support, and advanced human-system interface displays (e.g., overviews and task-based). The modernization methodology can be used to guide transformative thinking when integrating a set of vendor-specific capabilities to support a new concept of operations and a utility’s end-state vision. The results of this research are organized into six major sections: - Section 1 introduces the need for supporting large-scale digital modifications that will renew the technology base for extended operating life beyond 60 years - Section 2 describes the challenges that the nuclear industry is enduring with modernizing. - Section 3 summarizes the primary standards and guidance. - Section 4 presents earlier work from the LWRS Program regarding the development of a transformative conceptual design for an advanced control room of a hybrid plants. - Section 5 presents a methodology that is designed at addressing the challenges in the industry today in achieving a transformative new state vision and concept of operations. - Conclusions and next steps of this research are provided in Section 6.

99 GENERAL AND MISCELLANEOUS↗

NASA NTRS1979

Performance, physiological, and oculometer evaluation of VTOL landing displays

A methodological approach to measuring workload was investigated for evaluation of new concepts in VTOL aircraft displays. Physiological, visual response, and conventional flight performance measures were recorded for landing approaches performed in the NASA Visual Motion Simulator (VMS). Three displays (two computer graphic and a conventional flight director), three crosswind amplitudes, and two motion base conditions (fixed vs. moving base) were tested in a factorial design. Multivariate discriminant functions were formed from flight performance and/or visual response variables. The flight performance variable discriminant showed maximum differentation between crosswind conditions. The visual response measure discriminant maximized differences between fixed vs. motion base conditions and experimental displays. Physiological variables were used to attempt to predict the discriminant function values for each subject/condition trial. The weights of the physiological variables in these equations showed agreement with previous studies. High muscle tension, light but irregular breathing patterns, and higher heart rate with low amplitude all produced higher scores on this scale and thus represent higher workload levels.

North, R. A.↗

DOE OSTI2023

Automated generation of scientific workflow generators with WfChef

Scientific workflow applications have gained significant importance, and their automated and efficient execution on large-scale computing platforms has been the subject of extensive research and development. For these efforts to be successful, a solid experimental methodology is needed to evaluate workflow algorithms and systems. A foundation for this methodology is the availability of realistic workflow instances. Although public repositories provide workflow instances for a few scientific applications, these are limited in scope, and workflow instances are not available for all application scales of interest. To address this limitation, previous work has developed generators of synthetic workflow instances of arbitrary scales. Despite being popular, the implementation of these generators is a manual and labor-intensive process that requires expert application knowledge. As a result, these generators only target a handful of applications, even though there are hundreds of workflow applications in production. Here, we introduce WfChef , a fully automated framework for constructing a synthetic workflow generator for any scientific application. Based on an input set of workflow instances for a particular application, WfChef automatically produces a synthetic workflow generator. To measure the realism of the generated workflows, we define and evaluate several metrics. Using these metrics, we compare the realism of the workflows generated by WfChef generators to that of the workflows generated by the previously available, hand-crafted generators. We find that WfChef generators not only require zero development effort (because they are automatically produced), but also generate workflows that are more realistic than those generated by hand-crafted generators.

97 MATHEMATICS AND COMPUTING↗

NASA NTRS2010

Testing Requirements for Refractory Materials

Launch Pads 39A and 39B currently use refractory material (Fondu Fyre) in the flame trenches. This material was initially approved for the Saturn program. This material had a lifetime of 10 years according to the manufacturer, and it has been used for over 40 years. As a consequence, the Fondu Fyre at Launch Complex 39 requires repair subsequent to almost every launch. A review of the literature indicates that the gunned Fondu Fyre refractory product (WA-1G) was never tested prior to use. With the recent severe damage to the flame trenches, a new refractory material is sought to replace Fondu Fyre. In order to replace Fondu Fyre, a methodology to test and evaluate refractory products was developed. This paper outlines this methodology and discusses current testing requirements, as well as the laboratory testing that might be required. Furthermore, this report points out the necessity for subscale testing, the locations where this testing can be performed, and the parameters that will be necessary to qualify a product. The goal is to identify a more durable refractory material that has physical, chemical, and thermal properties suitable to withstand the harsh environment of the launch pads at KSC.

Calle, Luz Marina↗

NASA NTRS2011

Testing Requirements for Refractory Materials

Calle, Luz Marina↗

NASA NTRS2012

Testing Requirements for Refractory Materials

Launch Pads 39A and 39B currently use refractory material (Fondu Fyre) in the flame trenches. This material was initially approved for the Saturn program. This material had a lifetime of 10years according to the manufacturer, and it has been used for over 40 years. As a consequence, the Fondu Fyre at Launch Complex 39 requires repair subsequent to almost every launch. A review of the literature indicates that the gunned Fondu Fyre refractory product (WA-1 G) was never tested prior to use. With the recent severe damage to the flame trenches, a new refractory material is sought to replace Fondu Fyre. In order to replace Fondu Fyre, a methodology to test and evaluate refractory products was developed. This paper outlines this methodology and discusses current testing requirements, as well as the laboratory testing that might be required. Furthermore, this report points out the necessity for subscale testing, the locations where this testing can be performed, and the parameters that will be necessary to qualify a product. The goal is to identify a more durable refractory material that has physical, chemical, and thermal properties suitable to withstand the harsh environment of the launch pads at KSC.

Calle, Luz Marina↗

DOE OSTI2020

Economic Extraction and Recovery of REEs and Production of Clean Value-Added Products from Low-Rank Coal Fly Ash

The University of North Dakota (UND) Energy & Environmental Research Center (EERC) teamed with Pacific Northwest National Laboratory (PNNL) , the North Dakota Industrial Commission Lignite Research Program, and commercial partners Basin Electric Power Cooperative, Great River Energy, and Southern Company to execute a Cooperative Agreement funded by the U.S. Department of Energy’s (DOE’s) National Energy Technology Laboratory (NETL) focused on identifying unique pathways and pretreatments to extract rare-earth elements (REEs) from low-rank coal (LRC) ash in a more economical and environmentally benign manner than current methods. The EERC drew upon the unique and deep knowledge of the team regarding LRC geochemistry, ash transformation mechanisms during combustion, and fly ash chemistry in the approach to understand the portioning of REEs in LRC ash. A comprehensive characterization effort was undertaken to fully establish the form, associations, and partitioning of the REEs and other elements/minerals of interest in fly ash, as well as the ash chemistry, mineralogy, and morphology. Using this information, the team tested methods of REE extraction to take advantage of the unique and advantageous properties of LRC ash. Multiple complementary approaches were evaluated that combined both novel and proven methods to develop an extractive process for REEs and other high-value metals from LRC ash that provide technological and economic progress on the current state of the art. Methodologies were also evaluated for the potential to synergistically remove toxic metals while extracting other high-value metals from the ash, generating a clean, safe ash material as a value-added product. This would eliminate waste and offer significant additional revenue potential. Based on laboratory-scale testing, an initial high-level technical and economic analysis to estimate operating expenses and product revenues to guide future phases of technology development was implemented. By focusing on minimizing process steps and consumptive use of extraction solutions, a tunable, economically viable process for REE extraction from LRC ash was developed. The tunable process has the ability to be adjusted to adapt to the differing ash chemical and physical properties that are characteristic of LRC ash. This is key to deployment as a commercial process. In demonstrating the process, the EERC was able to illustrate mixed REE concentration greater than the goal of 2 wt%. High-level economic analysis of the process with LRC ash of moderate REE levels demonstrated the potential of the process, but recovery of just the REEs was insufficient to cover operational costs. With the additional extraction of other high-value metals found in LRC ash, the process economics approach would break even, and it is anticipated that with process optimization, the developed process could potentially be economically sound. The same economic analysis performed utilizing LRC ash with high levels of REEs showed the recovery of the REEs alone would make the process economically viable even without further optimization.

01 COAL, LIGNITE, AND PEAT↗

NASA NTRS2005

Developing a Methodology for Eliciting Subjective Probability Estimates During Expert Evaluations of Safety Interventions: Application for Bayesian Belief Networks

The NASA Aviation Safety Program (AvSP) has defined several products that will potentially modify airline and/or ATC operations, enhance aircraft systems, and improve the identification of potential hazardous situations within the National Airspace System (NAS). Consequently, there is a need to develop methods for evaluating the potential safety benefit of each of these intervention products so that resources can be effectively invested to produce the judgments to develop Bayesian Belief Networks (BBN's) that model the potential impact that specific interventions may have. Specifically, the present report summarizes methodologies for improving the elicitation of probability estimates during expert evaluations of AvSP products for use in BBN's. The work involved joint efforts between Professor James Luxhoj from Rutgers University and researchers at the University of Illinois. The Rutgers' project to develop BBN's received funding by NASA entitled "Probabilistic Decision Support for Evaluating Technology Insertion and Assessing Aviation Safety System Risk." The proposed project was funded separately but supported the existing Rutgers' program.

Wiegmann, Douglas A.a↗