Search NASA⌕ Search

SEARCH · Search NASA

Results for “evaluation methodology”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 811 records · Page 45

A Methodology for Writing High Quality Requirement Specifications and for Evaluating Existing Ones

Requirements development and management have always been critical in the implementation of software systems-engineers are unable to build what analysts can not define. It is generally accepted that the earlier in the life cycle potential risks are identified the easier it is to eliminate or manage the conditions that introduce that risk. Problems that are not found until testing are approximately 14 times more costly to fix than if the problem was found in the requirement phase. The requirements specification, as the first tangible representation of the capability to be produced, establishes the basis for all of the project's engineering management and assurance functions. If the quality of the requirements specification is poor it can give rise to risks in all areas of the project. Recently, automated tools have become available to support requirements management. The use of these tools not only provides support in the definition and tracing of requirements, but it also opens the door to effective use of metrics in characterizing and assessing the quality of the requirement specifications.

Rosenberg, Linda↗

Evaluation of Contact Separation Force Testing as a Screening Methodology for Electrical Socket Contacts

During system level testing intermittent and permanent open circuit failures of mated, crimp removable, electrical contact pairs were experienced. The root cause of the failures was determined to be low (but not zero) contact forces applied by the socket contact tines against the engaging pin. The low contact force reduces the effectiveness of the wiping action of the socket tines against the pin. The observed failure mode may be produced when insufficient wiping during mate, demate and small relative movement in use allows for the accumulation of debris or insulating films that electrically separate the contact pair. The investigation identified at least three manufacturing process control problems associated with the socket contacts that enabled shipment of contacts susceptible to developing low contact forces: (1) Improper heat treatment of the socket tines resulting in plastic rather than elastic behavior; (2) Overly thinned socket tines at their base resulting in reduced pin retention forces; (3) Insufficient screening tests to identify parts susceptible to the aforementioned failure mechanisms. The results from an extensive screening program of socket contacts utilizing the industry standard contact separation force test procedures are described herein. The investigation shows this method to be capable of identifying initially weak sockets. However, sockets whose contact retention forces may degrade during use may not be screened out by pin retention testing alone. Further investigations are required to correlate low contact retention forces with increased electrical contact resistance in the presence of insulating films that may accumulate in the use environment.

Christopher Green↗

Overview of Liner Activities in Support of the International Forum for Aviation Research

This paper provides an overview of liner research being conducted by members of the International Forum for Aviation Research (IFAR). The IFAR consists of representatives from a number of national research labs and is established to enable information exchange on aviation research activities. NASA has provided three challenges that explore different aspects of acoustic liner research, and will compile the results from each of the partners into a common database when the challenges are completed. The goal of the first challenge is to compare experimental methods applied at different national labs for acoustic liner evaluation, and to compare impedance eduction methodologies for uniform liners. The second challenge is to evaluate the ability of 3D propagation codes to predict the effects of spanwise variable-impedance liners on the acoustic pressure field. The third challenge is to explore different approaches to simultaneously educe the impedances for each zone of a three-zone liner. Each challenge is intended to be conducted as a blind test. Thus, only representative results achieved by the NASA Liner Physics Team for each of the three challenges are presented herein. When the results from all participants are available, the compiled results are intended to be provided in future reports.

Jones, Michael G.↗

Information Analysis Methodology for Border Security Deployment Prioritization and Post Deployment Evaluation

Due to international commerce, cross-border conflicts, and corruption, a holistic, information driven, approach to border security is required to best understand how resources should be applied to affect sustainable improvements in border security. The ability to transport goods and people by land, sea, and air across international borders with relative ease for legitimate commercial purposes creates a challenging environment to detect illicit smuggling activities that destabilize national level border security. Smuggling activities operated for profit or smuggling operations driven by cross border conflicts where militant or terrorist organizations facilitate the transport of materials and or extremists to advance a cause add complexity to smuggling interdiction efforts. Border security efforts are further hampered when corruption thwarts interdiction efforts or reduces the effectiveness of technology deployed to enhance border security. These issues necessitate the implementation of a holistic approach to border security that leverages all available data. Large amounts of information found in hundreds of thousands of documents can be compiled to assess national or regional borders to identify variables that influence border security. Location data associated with border topics of interest may be extracted and plotted to better characterize the current border security environment for a given country or region. This baseline assessment enables further analysis, but also documents the initial state of border security that can be used to evaluate progress after border security improvements are made. Then, border security threats are prioritized via a systems analysis approach. Mitigation factors to address risks can be developed and evaluated against inhibiting factor such as corruption. This holistic approach to border security helps address the dynamic smuggling interdiction environment where illicit activities divert to a new location that provides less resistance to smuggling activities after training or technology is deployed at a given location. This paper will present an approach to holistic border security information analysis.

11 NUCLEAR FUEL CYCLE AND FUEL MATERIALS↗

Challenges in Implementing Methodologies for Nonproliferation Assessments

A handful of models for explaining and predicting States’ development of nuclear weapons programs have been proposed since the 1970s. Despite the array of techno-social variables and computational concepts employed in these models, no model has yet been established as an agreed-upon standard. Likewise, the International Atomic Energy Agency (IAEA)—one of the main institutions evaluating social, political, and technological information for assessments of States’ current nuclear capabilities—uses only a qualitative framework by which to evaluate such information to assess the correctness and completeness of a State’s declaration. In this paper, analysts familiar with both the development of techno-social modelling and the IAEA’s implementation of a safeguards system that is information driven discuss the challenges faced in the development, implementation, and evaluation of models and methodologies for nonproliferation assessments, based on experiences at Pacific Northwest National Laboratory (PNNL) and the IAEA.

11 NUCLEAR FUEL CYCLE AND FUEL MATERIALS↗

Evolution of the Next Exploration Toilet through Human-in-the-Loop (HITL) Testing

Human waste collection in space is a unique and necessary function that all crewmembers must perform. The variability in how each crewmember uses the toilet to urinate and defecate introduces complexities and challenges with regards to overall hardware design. Because of this variability, it is important to consider crew inputs in all aspects of a toilet design especially with regards to crew interfaces that could impact overall waste collection. Access to crew feedback is essential to the design process and should be considered early and often through the various design phases. In 2020, NASA started a project for the Human Landing System (HLS) program to develop a Government Furnished Equipment (GFE) toilet option. The project is known as the Lavatory On-Orbit (LOO). During the early development of the LOO, the project team conducted several crew evaluations to collect and summarize valuable crew feedback on system design, function, and overall usability to influence the next design iteration. Because every person could use the system differently in space, it was extremely important to collect and analyze the data in a very methodical manner to appropriately influence the design based on the evaluation results. Establishing a standard process ensures consistent data collection from one evaluation to another, helps to maintain privacy for each test subject’s inputs and removes any potential bias from test subject to test subject. To date, the team has completed four crew evaluations on prototypes for the different LOO hardware. This paper will summarize the methodology used to conduct the evaluations as well as how data was collected and analyzed. The paper will also provide details on each of the evaluations and how the design was updated based on the results.

toilet↗

Evolution of the Next Exploration Toilet Through Human-in-the-Loop (HITL) Testing

Human waste collection in space is a unique and necessary function that all crewmembers must perform. The variability in how each crewmember uses the toilet to urinate and defecate introduces complexities and challenges with regards to overall hardware design. Because of this variability, it is important to consider crew inputs in all aspects of a toilet design especially with regards to crew interfaces that could impact overall waste collection. Access to crew feedback is essential to the design process and should be considered early and often through the various design phases. In 2020, NASA started a project for the Human Landing System (HLS) program to develop a Government Furnished Equipment (GFE) toilet option. The project is known as the Lavatory On-Orbit (LOO). During the early development of the LOO, the project team conducted several crew evaluations to collect and summarize valuable crew feedback on system design, function, and overall usability to influence the next design iteration. Because every person could use the system differently in space, it was extremely important to collect and analyze the data in a very methodical manner to appropriately influence the design based on the evaluation results. Establishing a standard process ensures consistent data collection from one evaluation to another, helps to maintain privacy for each test subject’s inputs and removes any potential bias from test subject to test subject. To date, the team has completed four crew evaluations on prototypes for the different LOO hardware. This paper will summarize the methodology used to conduct the evaluations as well as how data was collected and analyzed. The paper will also provide details on each of the evaluations and how the design was updated based on the results.

toilet↗

Requirements and Recommendations for a Physical Attack Characterization Framework

This study seeks to identify existing frameworks or develop requirements and recommendations for a new framework that can consistently characterize physical attacks, analogous to MITRE ATT&CK®. MITRE ATT&CK is widely used across government, research organizations, and the cyber security community to characterize cyber attack tactics, techniques, and procedures (TTPs) in a consistent and commonly understood manner. While physical attack taxonomies, methodologies, and other tools for evaluating physical security do exist, many are sector and/or facility-type specific—and therefore not able to provide comparable scenarios across sectors—or are more focused on security assessment instead of the characterization of attacks themselves. A MITRE ATT&CK analog for physical attacks on critical infrastructure would provide a common language and structure for analysis of physical attacks. Existing attack characterization methodologies do not robustly address cyber-physical security risks. To fully understand a facility’s security needs, it is important to understand the entire vulnerability landscape from both a physical and a cyber perspective. To underscore this need, organizations such as the Cybersecurity and Infrastructure Security Agency (CISA) are calling for a coordinated approach to cyber and physical security, which they refer to as cyber and physical security convergence. A physical attack characterization framework that could be used jointly with MITRE ATT&CK would help support a more robust analysis in support of convergence, enabling the consistent characterization of attacks that utilize both cyber and physical tactics and techniques. This could provide analysts and stakeholders with a clearer understanding of how security mitigations deployed in the physical realm impact security risks in the cyber realm, and vice versa. In this study, the project team evaluates existing physical security taxonomies and methodologies to assess whether an existing method can be used to create a “physical half” of MITRE ATT&CK. This study then provides requirements and recommendations for a framework that can leverage aspects of existing methodologies. The goal of the final framework is for it to be widely adopted and referenced, regardless of critical infrastructure sector, facility type, or facility components. This study also identifies applicable use cases for when and how a framework could be applied across the various critical infrastructure sectors for a variety of attack types or motivations. Through a literature review of existing security-focused methodologies and taxonomies, engagement with relative stakeholders, evaluation of potential physical attack framework use cases, and subsequent identification of requirements, this study identified the following key findings and recommendations: There is a need for a new physical attack characterization framework; A physical attack framework should be interoperable with the MITRE ATT&CK framework; A physical attack framework should be broadly applicable, but with detailed tactics, techniques, and procedures that encompass the entire attack path; A physical attack framework should be based on observed or feasible events; A physical attack framework should adapt features from existing methodologies, frameworks, and taxonomies; A physical attack framework should be owned, overseen, and maintained by one organization.

45 MILITARY TECHNOLOGY, WEAPONRY, AND NATIONAL DEF↗

Poster - 44: Development and implementation of a comprehensive end-to-end testing methodology for linac-based frameless SRS QA using a modified commercial stereotactic anthropomorphic phantom

Purpose: An end-to-end testing methodology was designed to evaluate the overall SRS treatment fidelity, incorporating all steps in the linac-based frameless radiosurgery treatment delivery process. The study details our commissioning experience of the Steev (CIRS, Norfolk, VA) stereotactic anthropomorphic head phantom including modification, test design, and baseline measurements. Methods: Repeated MR and CT scans were performed with interchanging inserts. MR-CT fusion accuracy was evaluated and the insert spatial coincidence was verified on CT. Five non-coplanar arcs delivered a prescription dose to a 15 mm spherical CTV with 2 mm PTV margin. Following setup, CBCT-based shifts were applied as per protocol. Sequential measurements were performed by interchanging inserts without disturbing the setup. Spatial and dosimetric accuracy was assessed by a combination of CBCT hidden target, radiochromic film, and ion chamber measurements. To facilitate film registration, the film insert was modified in-house by etching marks. Results: MR fusion error and insert spatial coincidences were within 0.3 mm. Both CBCT and film measurements showed spatial displacements of 1.0 mm in similar directions. Both coronal and sagittal films reported 2.3 % higher target dose relative to the treatment plan. The corrected ion chamber measurement was similarly greater by 1.0 %. The 3 %/2 mm gamma pass rate was 99% for both films Conclusions: A comprehensive end-to-end testing methodology was implemented for our SRS QA program. The Steev phantom enabled realistic evaluation of the entire treatment process. Overall spatial and dosimetric accuracy of the delivery were 1 mm and 3 % respectively.

60 APPLIED LIFE SCIENCES↗

Quantitative Evaluation of Reliability Improvement: Case Study on a Self-healing Distribution System

In this work, we develop a methodology and tool to quantitatively evaluate the reliability of a self-healing system that considers practical distribution system features such as the distributed energy resources, microgrids, and service restoration strategies. Also, this paper addresses various practical issues when being applied to an actual Duke Energy distribution system, including the design of feasible and practical service restoration strategies that are used to identify the customer interruptions after a fault, and the incorporation of the utility’s historical reliability indices that are used to calibrate the failure rate and repair time of distribution system components such as overhead lines and underground cables. This case study demonstrates the effectiveness of the proposed method.

Dong, Jiaojiao↗

Quantitative Evaluation of Reliability Improvement: Case Study on a Self-healing Distribution System

In this work, we develop a methodology and tool to quantitatively evaluate the reliability of a self-healing system that considers practical distribution system features such as the distributed energy resources, microgrids, and service restoration strategies. Also, this paper addresses various practical issues when being applied to an actual Duke Energy distribution system, including the design of feasible and practical service restoration strategies that are used to identify the customer interruptions after a fault, and the incorporation of the utility's historical reliability indices that are used to calibrate the failure rate and repair time of distribution system components such as overhead lines and underground cables. This case study demonstrates the effectiveness of the proposed method.

Dong, Jiaojiao↗

Cost-benefit analysis of space technology

A discussion of the implications and problems associated with the use of cost-benefit techniques is presented. Knowledge of these problems is useful in the structure of a decision making process. A methodology of cost-benefit analysis is presented for the evaluation of space technology. The use of the methodology is demonstrated with an evaluation of ion thrusters for north-south stationkeeping aboard geosynchronous communication satellites. A critique of the concept of consumers surplus for measuring benefits is also presented.

Hein, G. F.↗

Topical Analyses Related to Co-located Industrial Facilities at Nuclear Power Plants

This report investigates topics of interest with regard to Nuclear Power Plants (NPPs) utilizing flexible plant operations and generation (FPOG). Previous reports have identified the risk associated with co-located hydrogen generation facilities. This report evaluates special topics with regard to co-location of both hydrogen and syngas production facilities. A literature review was conducted to evaluate overpressure mitigation techniques that may be available to the NPP to reduce the consequence of an overpressure event. Also, the overpressure consequence of a catastrophic hydrogen storage tank failure event was analyzed. A comparison of the similarities and differences between the methodology utilized in HyRAM+ and Regulatory Guide 1.91 (R.G. 1.91) was performed for overpressure analysis. Also, the trinitrotoluene (TNT) equivalency methodology was utilized to evaluate an overpressure event at a Syngas production facility.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗

A Formal Approach for Designing and Evaluating Procedures

Operator interaction with modern control systems is a topic of great concern in high-risk industries such as nuclear power and commercial aviation. The issues associated with such systems focus on the ability of the operators (e.g., pilots) to achieve mission goals safely while containing failures. Operators must be able to interact safely and reliably with highly automatic and complex systems across the full spectrum of possible operating conditions, including normal, abnormal, and emergency situations. In environments such as commercial aviation, operator interaction with the machine is specified through a set of standard operating procedures (SOP). A procedure represents a collective agreement on the 'best' way to perform a given task. The intent of this paper is to suggest a formal methodology, for designing and evaluating procedures, that is both reliable and systematic. Our approach involves two major elements: a model of the machine and a list of the operator's task specifications (goals). We use formal modeling paradigms for describing the system and super-imposing on it the operator's tasks. Such paradigms, based on recent frameworks such as Statecharts and Hierarchical Hybrid Machines appear to be adequate methods for analyzing operator interaction with modern control systems. To illustrate this methodology, we model and analyze the sequence of actions for an emergency procedure. The procedure, Irregular Engine Start, for a medium-range aircraft, specifies the sequence of immediate actions that must be performed by the crew to avoid an uncontrolled rise in engine temperature during start-up. A model of engine behavior during a hot start is constructed. It also describes the various actions that can be taken by the crew and the resulting outcomes. The model is then opened up as a tree of all possible action sequences. This action tree allows us to trace the correct sequences necessary to achieve the desired end-goal (secure and shut down of the engine). In conclusion, we argue that the current process of designing and evaluating procedures can be improved. We discuss the implications of this approach for designing and evaluating this and other types of procedures. We conclude with insights about the benefits and limitation of this methodology, and offer suggestions for future research.

Degani, Asaf↗

Manufacturing Process Design of a Micro-Scale Liquid-Liquid Extractor and Multi-phase Separator

Advances in process innovation within the chemical process industry are impeded by the cost of producing new innovative components. In this paper, various manufacturing process routes were evaluated for scaling up a micro-scale liquid-liquid extractor and multi-phase separator capable of separating immiscible phases that are used for the extraction of a chemical product into a lower boiling point solvent to make downstream distillation easier. A manufacturing process design methodology was used to evaluate both photochemical machining and binder jetting as a means for producing the micropost arrays within the separator. Cost curves show that for a 0.3 mm deep micropost array, photochemical machining was found to be much cheaper than binder jetting. However, because binder jetting processes enable microposts with over 13 times higher depth-to-width aspect ratios, binder jetting was found to decrease the overall costs of the extractor/separator by more than half for a 500 barrel per day plant, with breakeven as low as 15 barrels per day. These economics were confirmed by building extractor/separator plates and characterizing them dimensionally and functionally. The binder jet plates were found to perform better than photochemical plates leading to the complete separation of the organic phase from water. The utilization of slower sintering cycles combined with surface grinding provided acceptable dimensional tolerances for the binder jet plates. Using pressurized air during depowdering improved the economics for the binder jetting process.

42 ENGINEERING↗

Fuel Cell Airframe Integration Study for Short-Range Aircraft: Aircraft Propulsion and Subsystems Integration Evaluation - Volume 1

The objective of this study is to define the functionality and evaluate the propulsion and power system benefits derived from a Solid Oxide Fuel Cell (SOFC) based Auxiliary Power Unit (APU) for a future short range commercial aircraft, and to define the technology gaps to enable such a system. United Technologies Corporation (UTC) Integrated Total Aircraft Power System (ITAPS) methodologies were used to evaluate a baseline aircraft and several SOFC architectures. The technology benefits were captured as reductions of the mission fuel burn, life cycle cost, noise and emissions. As a result of the study, it was recognized that system integration is critical to maximize benefits from the SOFC APU for aircraft application. The mission fuel burn savings for the two SOFC architectures ranged from 4.7 percent for a system with high integration to 6.7 percent for a highly integrated system with certain technological risks. The SOFC APU itself produced zero emissions. The reduction in engine fuel burn achieved with the SOFC systems also resulted in reduced emissions from the engines for both ground operations and in flight. The noise level of the baseline APU with a silencer is 78 dBA, while the SOFC APU produced a lower noise level. It is concluded that a high specific power SOFC system is needed to achieve the benefits identified in this study. Additional areas requiring further development are the processing of the fuel to remove sulfur, either on board or on the ground, and extending the heat sink capability of the fuel to allow greater waste heat recovery, resolve the transient electrical system integration issues, and identification of the impact of the location of the SOFC and its size on the aircraft.

Gummalla, Mallika↗

Evaluation of Extreme Weather Impacts on Utility-scale Photovoltaic Plant Performance in the United States

The global energy system is undergoing significant changes, including a shift in energy generating technologies to more renewable energy sources. However, the dependence of renewable energy sources on local environmental conditions could also increase disruptions in service through exposures to compound, extreme weather events. By fusing three diverse datasets (operations and maintenance tickets, weather data, and production data), this analysis presents a novel methodology to identify and evaluate performance impacts arising from extreme weather events across diverse geographical regions. Text analysis of maintenance tickets identified snow, hurricanes, and storms as the leading extreme weather events affecting photovoltaic plants in the United States. Statistical techniques and machine learning were then implemented to identify the magnitude and variability of these extreme weather impacts on site performance. Impacts varied between event and non-event days, with snow events causing the greatest reductions in performance (54.5%), followed by hurricanes (12.6%) and storms (1.1%). Machine learning analysis identified key features in determining if a day is categorized as low performing, such as low irradiance, geographic location, weather features, and site size. The analysis improves our understanding of compound, extreme weather event impacts on photovoltaic systems, which can inform planning activities, especially as the industry continues to expand into new geographic and climatic regions around the world.

14 SOLAR ENERGY↗