Search NASA⌕ Search

SEARCH · Search NASA

Results for “evaluation methodology”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 649 records · Page 36

Sandia's Integrated Methodology for Energy and Infrastructure Resilience Analysis

Sandia National Laboratories' (Sandia) Resilient Energy Systems (RES) Strategic Initiative is establishing a strategic vision for U.S. energy systems' resilience through threat-informed research and development, enabling energy and interdependent infrastructure systems to successfully adapt in an environment of accelerating change. A key challenge in promoting energy systems resilience lies in developing rigorous resilience analysis methodologies to quantify system performance. Resilience analysis methodologies should enable evaluation of the consequences of various disruptions and the relative effectiveness of potential mitigations. To address this challenge, RES synthesized the common components of Sandia's resilience frameworks into an integrated methodology for energy and infrastructure resilience analysis. This report documents, demonstrates, and extends this methodology.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗

A Review of the Evaluation of Building Energy Code Compliance in the United States

Building energy codes are essential tools for achieving energy efficiency in buildings. However, the full energy savings potential of these codes can only be realized if buildings are constructed in compliance with them. Therefore, evaluating building energy code compliance is crucial in bridging the gap between the energy efficiency requirements set by energy codes and the actualized energy savings achieved. An energy code compliance evaluation serves as a mechanism to assess construction practices, evaluate the effectiveness of code enforcement, identify gaps in compliance, and guide strategies for improvement through training and education. Conducting code compliance evaluation activities involves field studies that require careful design and significant resources. Historically, more emphasis has been placed on developing and adopting building energy codes, while efforts to evaluate compliance have been relatively limited and lacking consistent approaches. The passage of the 2009 American Recovery and Reinvestment Act (ARRA), which mandated that states create plans for achieving 90% compliance within eight years, stimulated the need for an energy code compliance evaluation. As a result, federal, state, and local governments, and utilities have invested in the development of methodologies and tools for code compliance evaluation studies. This paper reviews the code compliance evaluation studies conducted in the United States over the past three decades. It describes and compares the methodologies and metrics used to assess building energy code compliance, summarizes the general elements and steps involved in the evaluation process, and discusses common issues in these studies. Over time, code compliance evaluation methodologies have evolved from isolated development within individual states, regions, and utilities, to widely accepted protocols applicable across different states and local jurisdictions. There has been a transition in compliance metrics, shifting from historical compliance rates to energy-consumption-oriented approaches.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

Measurement and analysis of operating system fault tolerance

This paper demonstrates a methodology to model and evaluate the fault tolerance characteristics of operational software. The methodology is illustrated through case studies on three different operating systems: the Tandem GUARDIAN fault-tolerant system, the VAX/VMS distributed system, and the IBM/MVS system. Measurements are made on these systems for substantial periods to collect software error and recovery data. In addition to investigating basic dependability characteristics such as major software problems and error distributions, we develop two levels of models to describe error and recovery processes inside an operating system and on multiple instances of an operating system running in a distributed environment. Based on the models, reward analysis is conducted to evaluate the loss of service due to software errors and the effect of the fault-tolerance techniques implemented in the systems. Software error correlation in multicomputer systems is also investigated.

Lee, I.↗

Development of an Assessment Methodology That Enables the Nuclear Industry to Evaluate Adoption of Advanced Automation

Nuclear power has a crucial role in providing safe, reliable, and economical carbon-free electricity for today and the future. For continued operation, many of the existing United States nuclear power plants will begin the subsequent license renewal process for extending their operating license periods. As plants extend their expected operating lifetimes, there is a significant opportunity to modernize. These plants have a much stronger business case with these extended mission periods to modernize and significantly enhance their economic viability in current and future energy markets by implementing digital technologies that support innovation, efficiency gains, and business-model transformation. Ensuring continued safety and reliability is crucial. Transformative digital technologies—including automation—that fundamentally change the concept of operation for the nuclear power plant operating model requires a critical focus on the human and technology integration element. Further, the nuclear industry has historically been reluctant to modernize due to having a risk adverse culture and lack of clarity for a transformative new state vision (Joe & Remer, 2019; Thomas et al., 2020). Common barriers include (1) the perceived value and return on investment (ROI) of digital technology, (2) the perceived risk associated with licensing, regulatory, and cybersecurity, and (3) insufficient guidance for performing digital modifications to power generation systems. This work presents a methodology to address these barriers and support the industry in adopting advanced automation and digital technology through developing a transformative vision and implementation strategy that will address the human and technology integration element. This research leverages previous LWRS Program and industry results. It draws specifically on previous LWRS Program research in the areas of advanced alarm systems, computer-based procedures, model informed decision support, and advanced human-system interface displays (e.g., overviews and task-based). The modernization methodology can be used to guide transformative thinking when integrating a set of vendor-specific capabilities to support a new concept of operations and a utility’s end-state vision. The results of this research are organized into six major sections: - Section 1 introduces the need for supporting large-scale digital modifications that will renew the technology base for extended operating life beyond 60 years - Section 2 describes the challenges that the nuclear industry is enduring with modernizing. - Section 3 summarizes the primary standards and guidance. - Section 4 presents earlier work from the LWRS Program regarding the development of a transformative conceptual design for an advanced control room of a hybrid plants. - Section 5 presents a methodology that is designed at addressing the challenges in the industry today in achieving a transformative new state vision and concept of operations. - Conclusions and next steps of this research are provided in Section 6.

99 GENERAL AND MISCELLANEOUS↗

Performance, physiological, and oculometer evaluation of VTOL landing displays

A methodological approach to measuring workload was investigated for evaluation of new concepts in VTOL aircraft displays. Physiological, visual response, and conventional flight performance measures were recorded for landing approaches performed in the NASA Visual Motion Simulator (VMS). Three displays (two computer graphic and a conventional flight director), three crosswind amplitudes, and two motion base conditions (fixed vs. moving base) were tested in a factorial design. Multivariate discriminant functions were formed from flight performance and/or visual response variables. The flight performance variable discriminant showed maximum differentation between crosswind conditions. The visual response measure discriminant maximized differences between fixed vs. motion base conditions and experimental displays. Physiological variables were used to attempt to predict the discriminant function values for each subject/condition trial. The weights of the physiological variables in these equations showed agreement with previous studies. High muscle tension, light but irregular breathing patterns, and higher heart rate with low amplitude all produced higher scores on this scale and thus represent higher workload levels.

North, R. A.↗

Automated generation of scientific workflow generators with WfChef

Scientific workflow applications have gained significant importance, and their automated and efficient execution on large-scale computing platforms has been the subject of extensive research and development. For these efforts to be successful, a solid experimental methodology is needed to evaluate workflow algorithms and systems. A foundation for this methodology is the availability of realistic workflow instances. Although public repositories provide workflow instances for a few scientific applications, these are limited in scope, and workflow instances are not available for all application scales of interest. To address this limitation, previous work has developed generators of synthetic workflow instances of arbitrary scales. Despite being popular, the implementation of these generators is a manual and labor-intensive process that requires expert application knowledge. As a result, these generators only target a handful of applications, even though there are hundreds of workflow applications in production. Here, we introduce WfChef , a fully automated framework for constructing a synthetic workflow generator for any scientific application. Based on an input set of workflow instances for a particular application, WfChef automatically produces a synthetic workflow generator. To measure the realism of the generated workflows, we define and evaluate several metrics. Using these metrics, we compare the realism of the workflows generated by WfChef generators to that of the workflows generated by the previously available, hand-crafted generators. We find that WfChef generators not only require zero development effort (because they are automatically produced), but also generate workflows that are more realistic than those generated by hand-crafted generators.

97 MATHEMATICS AND COMPUTING↗

A critical comparison of ten disposable cup LCAs

Disposable cups can be made from conventional petro-plastics, bioplastics, or paperboard (coated with petro-plastics or bioplastics). This study compared ten life cycle assessment (LCA) studies of disposable cups with the aim to evaluate the robustness of their results. The selected studies have only one impact category in common, namely climate change with global warming potential (GWP) as its category indicator. Quantitative GWP results of the studies were closer examined. GWPs within and across each study show none of the cup materials to be consistently better than the others. Comparison of the absolute GWPs (after correction for the cup volume) also shows no consistent better or worse cup material. An evaluation of the methodological choices and the data sets used in the studies revealed their influence on the GWP. The differences in GWP can be attributed to a multitude of factors, i.e., cup material and weight, production processes, waste processes, allocation options, and data used. These factors basically represent different types of uncertainty. Sensitivity and scenario analyses provided only the influence of one factor at once. A systematic and simultaneous use of sensitivity and scenario analyses could, in a next research, result in more robust outcomes. -- Highlights: • Conflicting results from life cycle assessment (LCA) on disposable cups • GWP results of LCAs did not point to a best or worst cup material. • Differences in GWP results are due to methodological choices and data sets used. • Standardized LCA: transparency of LCA studies, but still different in approaches.

54 ENVIRONMENTAL SCIENCES↗

Testing Requirements for Refractory Materials

Launch Pads 39A and 39B currently use refractory material (Fondu Fyre) in the flame trenches. This material was initially approved for the Saturn program. This material had a lifetime of 10 years according to the manufacturer, and it has been used for over 40 years. As a consequence, the Fondu Fyre at Launch Complex 39 requires repair subsequent to almost every launch. A review of the literature indicates that the gunned Fondu Fyre refractory product (WA-1G) was never tested prior to use. With the recent severe damage to the flame trenches, a new refractory material is sought to replace Fondu Fyre. In order to replace Fondu Fyre, a methodology to test and evaluate refractory products was developed. This paper outlines this methodology and discusses current testing requirements, as well as the laboratory testing that might be required. Furthermore, this report points out the necessity for subscale testing, the locations where this testing can be performed, and the parameters that will be necessary to qualify a product. The goal is to identify a more durable refractory material that has physical, chemical, and thermal properties suitable to withstand the harsh environment of the launch pads at KSC.

Calle, Luz Marina↗

Testing Requirements for Refractory Materials

Launch Pads 39A and 39B currently use refractory material (Fondu Fyre) in the flame trenches. This material was initially approved for the Saturn program. This material had a lifetime of 10 years according to the manufacturer, and it has been used for over 40 years. As a consequence, the Fondu Fyre at Launch Complex 39 requires repair subsequent to almost every launch. A review of the literature indicates that the gunned Fondu Fyre refractory product (WA-1G) was never tested prior to use. With the recent severe damage to the flame trenches, a new refractory material is sought to replace Fondu Fyre. In order to replace Fondu Fyre, a methodology to test and evaluate refractory products was developed. This paper outlines this methodology and discusses current testing requirements, as well as the laboratory testing that might be required. Furthermore, this report points out the necessity for subscale testing, the locations where this testing can be performed, and the parameters that will be necessary to qualify a product. The goal is to identify a more durable refractory material that has physical, chemical, and thermal properties suitable to withstand the harsh environment of the launch pads at KSC.

Calle, Luz Marina↗

Testing Requirements for Refractory Materials

Launch Pads 39A and 39B currently use refractory material (Fondu Fyre) in the flame trenches. This material was initially approved for the Saturn program. This material had a lifetime of 10years according to the manufacturer, and it has been used for over 40 years. As a consequence, the Fondu Fyre at Launch Complex 39 requires repair subsequent to almost every launch. A review of the literature indicates that the gunned Fondu Fyre refractory product (WA-1 G) was never tested prior to use. With the recent severe damage to the flame trenches, a new refractory material is sought to replace Fondu Fyre. In order to replace Fondu Fyre, a methodology to test and evaluate refractory products was developed. This paper outlines this methodology and discusses current testing requirements, as well as the laboratory testing that might be required. Furthermore, this report points out the necessity for subscale testing, the locations where this testing can be performed, and the parameters that will be necessary to qualify a product. The goal is to identify a more durable refractory material that has physical, chemical, and thermal properties suitable to withstand the harsh environment of the launch pads at KSC.

Calle, Luz Marina↗

Economic Extraction and Recovery of REEs and Production of Clean Value-Added Products from Low-Rank Coal Fly Ash

The University of North Dakota (UND) Energy & Environmental Research Center (EERC) teamed with Pacific Northwest National Laboratory (PNNL) , the North Dakota Industrial Commission Lignite Research Program, and commercial partners Basin Electric Power Cooperative, Great River Energy, and Southern Company to execute a Cooperative Agreement funded by the U.S. Department of Energy’s (DOE’s) National Energy Technology Laboratory (NETL) focused on identifying unique pathways and pretreatments to extract rare-earth elements (REEs) from low-rank coal (LRC) ash in a more economical and environmentally benign manner than current methods. The EERC drew upon the unique and deep knowledge of the team regarding LRC geochemistry, ash transformation mechanisms during combustion, and fly ash chemistry in the approach to understand the portioning of REEs in LRC ash. A comprehensive characterization effort was undertaken to fully establish the form, associations, and partitioning of the REEs and other elements/minerals of interest in fly ash, as well as the ash chemistry, mineralogy, and morphology. Using this information, the team tested methods of REE extraction to take advantage of the unique and advantageous properties of LRC ash. Multiple complementary approaches were evaluated that combined both novel and proven methods to develop an extractive process for REEs and other high-value metals from LRC ash that provide technological and economic progress on the current state of the art. Methodologies were also evaluated for the potential to synergistically remove toxic metals while extracting other high-value metals from the ash, generating a clean, safe ash material as a value-added product. This would eliminate waste and offer significant additional revenue potential. Based on laboratory-scale testing, an initial high-level technical and economic analysis to estimate operating expenses and product revenues to guide future phases of technology development was implemented. By focusing on minimizing process steps and consumptive use of extraction solutions, a tunable, economically viable process for REE extraction from LRC ash was developed. The tunable process has the ability to be adjusted to adapt to the differing ash chemical and physical properties that are characteristic of LRC ash. This is key to deployment as a commercial process. In demonstrating the process, the EERC was able to illustrate mixed REE concentration greater than the goal of 2 wt%. High-level economic analysis of the process with LRC ash of moderate REE levels demonstrated the potential of the process, but recovery of just the REEs was insufficient to cover operational costs. With the additional extraction of other high-value metals found in LRC ash, the process economics approach would break even, and it is anticipated that with process optimization, the developed process could potentially be economically sound. The same economic analysis performed utilizing LRC ash with high levels of REEs showed the recovery of the REEs alone would make the process economically viable even without further optimization.

01 COAL, LIGNITE, AND PEAT↗

Developing a Methodology for Eliciting Subjective Probability Estimates During Expert Evaluations of Safety Interventions: Application for Bayesian Belief Networks

The NASA Aviation Safety Program (AvSP) has defined several products that will potentially modify airline and/or ATC operations, enhance aircraft systems, and improve the identification of potential hazardous situations within the National Airspace System (NAS). Consequently, there is a need to develop methods for evaluating the potential safety benefit of each of these intervention products so that resources can be effectively invested to produce the judgments to develop Bayesian Belief Networks (BBN's) that model the potential impact that specific interventions may have. Specifically, the present report summarizes methodologies for improving the elicitation of probability estimates during expert evaluations of AvSP products for use in BBN's. The work involved joint efforts between Professor James Luxhoj from Rutgers University and researchers at the University of Illinois. The Rutgers' project to develop BBN's received funding by NASA entitled "Probabilistic Decision Support for Evaluating Technology Insertion and Assessing Aviation Safety System Risk." The proposed project was funded separately but supported the existing Rutgers' program.

Wiegmann, Douglas A.a↗

Ultraviolet brightness of celestial targets for Apollo 17

An evaluation of the ultraviolet flux from the stars expected in the various inertial-hold pointing directions and PTC scans during the Apollo 17 mission is presented. These directions and PTC scan poles for the nominal mission are listed. The methodology used in evaluating the flux, and the individual targets themselves is explained.

Fastie, W. G.↗

Semantic Segmentation of High-Resolution Satellite Imagery using Generative Adversarial Networks with Progressive Growing

With increase in urbanization and Earth Sciences research into urban areas, the need to quickly and accurately segment urban rooftop maps has never been greater. Cur-rent machine learning techniques struggle to produce high accuracy maps in dense urban zones where there is high image noise and foot print overlap. In this paper, we evaluate a training methodology for pixel-wise segmentation for high resolution satellite imagery using progressive growing of generative adversarial networks as a solution. We apply our model to segmenting building rooftops and compare these results to conventional methods for rooftop segmentation. We evaluate our approach using the SpaceNet version 2 and xView datasets. Our experiments show that for SpaceNet, progressive Generative Adversarial Network (GAN) training achieved a test accuracy of 93% compared to 89% for traditional GAN training and 87% for U-Net architecture, while for xView, we achieved 71% accuracy using progressive GAN training compared to 69% through traditional GAN training and 65% using U-Net.

Semantic↗

Evaluating software development by analysis of changes - Some data from the Software Engineering Laboratory

Basili and Weiss (1984) have discussed an approach for obtaining valid data which may be used to evaluate software development methodologies in a production environment. The methodology consists of five elements, including the identification of goals, the determination of questions of interest from the goals, the development of a data collection form, the development of data collection procedures, and the validation and analysis of the data. The current investigation is concerned with the presentation of the results from such an evaluation. The presented data were collected as part of studies reported by Basili et al. (1977). These studies had been conducted by NASA's Software Engineering Laboratory (SEL). Attention is given to an overview of the SEL, the application of the considered methodology, the results of a data analysis, and conclusions about the SEL environment.

Weiss, D. M.↗

A Cyber-Physical Anomaly Detection for Wide-Area Protection Using Machine Learning

Wide-area protection scheme (WAPS) provides system-wide protection by detecting and mitigating small and large-scale disturbances that are difficult to resolve using local protection schemes. As this protection scheme is evolving from a substation-based distributed remedial action scheme (DRAS) to the control center-based centralized RAS (CRAS), it presents severe challenges to their cybersecurity because of its heavy reliance on an insecure grid communication, and its compromise would lead to system failure. This article presents an architecture and methodology for developing a cyber-physical anomaly detection system (CPADS) that utilizes synchrophasor measurements and properties of network packets to detect data integrity and communication failure attacks on measurement and control signals in CRAS. The proposed machine leaning-based methodology applies a rules-based approach to select relevant input features, utilizes variational mode decomposition (VMD) and decision tree (DT) algorithms to develop multiple classification models, and performs final event identification using a rules-based decision logic. Here, we have evaluated the proposed methodology of CPADS using the IEEE 39 bus system for several performance measures (accuracy, recall, precision, and F-measure) in a cyber-physical testbed environment. Furthermore, our experimental results reveal that the proposed algorithm (VMD-DT) of CPADS outperforms the existing machine learning classifiers during noisy and noise-free measurements while incurring an acceptable processing overhead.

24 POWER TRANSMISSION AND DISTRIBUTION↗

An Evaluation of Oceanographic Optical Instruments and Deployment Methodologies

The primary objective of the Sea-viewing, Wide Field-of-view Sensor (SeaWiFS) Project is to produce water- leaving radiances with an uncertainty of 5% in clear-water regions and chlorophyll a concentrations within +/- 35% over the range of 0.05-50 mg/cu m. Any global mission, like SeaWiFS, requires validation data be submitted from a wide variety of investigators which places a significant challenge on quantifying the total uncertainty associated with the in situ measurements, because each investigator follows slightly different practices when it comes to implementing all of the steps associated with collecting field data, even those with a prescribed set of protocols. This study uses data from multiple cruises to quantify the uncertainties associated with implementing data collection procedures while utilizing differing in-water optical instruments and deployment methods. A comprehensive approach is undertaken and includes: (1) the use of a portable light source and in-water intercomparisons to monitor the stability of the field radiometers, (2) alternative methods for acquiring reference measurements, and (3) different techniques for making in-water profiles. The only system to meet the 5% radiometric objective of the SeaWiFS Project was a free-fall profiler using (relatively inexpensive) modular components, although a more sophisticated (and comparatively expensive) profiler using integral components was very close and only 1% higher. A relatively inexpensive system deployed with a winch and crane was also close, but the ship shadow contamination problem increased the total uncertainty to approximately 6.5%.

Hooker, Stanford B.↗