SEARCH · Search NASA
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
Software to evaluate functions of one or two arguments is too often tested or certified, or the accuracy assessed, by ad hoc methods. Systematic methods to test, certify or assess the accuracy of single- or double-precision functions of one or two arguments, and supporting software, are described here.
Assurance cases are a method for providing assurance for a system by giving an argument to justify a claim about the system, based on evidence about its design, development, and tested behavior. In comparison with assurance based on guidelines or standards (which essentially specify only the evidence to be produced), the chief novelty in assurance cases is provision of an explicit argument. In principle, this can allow assurance cases to be more finely tuned to the specific circumstances of the system, and more agile than guidelines in adapting to new techniques and applications. The first part of this report (Sections 1-4) provides an introduction to assurance cases. Although this material should be accessible to all those with an interest in these topics, the examples focus on software for airborne systems, traditionally assured using the DO-178C guidelines and its predecessors. A brief survey of some existing assurance cases is provided in Section 5. The second part (Section 6) considers the criteria, methods, and tools that may be used to evaluate whether an assurance case provides sufficient confidence that a particular system or service is fit for its intended use. An assurance case cannot provide unequivocal "proof" for its claim, so much of the discussion focuses on the interpretation of such less-than-definitive arguments, and on methods to counteract confirmation bias and other fallibilities in human reasoning.
Techniques in solving integrodifferential equations for two-dimensional transverse electric (TE) polarization field computations are discussed. Analytical simplifications are presented to efficiently and accurately evaluate the impedance elements, especially the diagonal ones, without loss of accuracy in the moment-method solution to the integral equations using pulse expansion and point matching. With such a choice, one can minimize the efforts in the software development of a two-dimensional electromagnetics code. Meanwhile, analytical simplifications on the evaluation of the impedance elements are developed based on the principle that the error due to simplifications is of a higher-order small argument than the error caused by the discretization.
Arguments in safety cases are predominantly qualitative. This is partly attributed to the lack of sufficient design and operational data necessary to measure the achievement of high-dependability targets, particularly for safety-critical functions implemented in software. The subjective nature of many forms of evidence, such as expert judgment and process maturity, also contributes to the overwhelming dependence on qualitative arguments. However, where data for quantitative measurements is systematically collected, quantitative arguments provide far more benefits over qualitative arguments, in assessing confidence in the safety case. In this paper, we propose a basis for developing and evaluating integrated qualitative and quantitative safety arguments based on the Goal Structuring Notation (GSN) and Bayesian Networks (BN). The approach we propose identifies structures within GSN-based arguments where uncertainties can be quantified. BN are then used to provide a means to reason about confidence in a probabilistic way. We illustrate our approach using a fragment of a safety case for an unmanned aerial system and conclude with some preliminary observations
Visualization software is widely used in scientific and engineering research. But computed visualizations can be very misleading, and the errors are easy to miss. We feel that the software producing the visualizations must be thoroughly evaluated and the evaluation process as well as the results must be made available. Testing and evaluation of visualization software is not a trivial problem. Several methods used in testing other software are helpful, but these methods are (apparently) often not used. When they are used, the description and results are generally not available to the end user. Additional evaluation methods specific to visualization must also be developed. We present several useful approaches to evaluation, ranging from numerical analysis of mathematical portions of algorithms to measurement of human performance while using visualization systems. Along with this brief survey, we present arguments for the importance of evaluations and discussions of appropriate use of some methods.
Usability encompasses learnability, efficiency, memorability, effectiveness, and satisfaction. NASA’s standards for usability acceptance criteria focus on interfaces that help operators achieve their tasks efficiently, effectively, and with satisfaction. However, discussions on usability, especially regarding future highly automated and autonomous systems, rarely include trust. As NASA plans for long-duration exploration missions, it envisions astronauts operating more independently from Mission Control on Earth. This independence will drive the development of these highly automated and autonomous systems that astronauts will use daily. To prepare for this future, our team has developed a scheduling and execution software tool that facilitates self-scheduling, allowing astronauts to independently manage their own schedule without Mission Control’s involvement. Over many years, we have developed, matured, and evaluated our software tool in extreme environments, prioritizing user-centered design and high usability. These evaluations have included multiple campaigns in NASA analogs, including NEEMO, BASALT, and HERA, as well as technology demonstrations onboard the International Space Station. Our recent research on software interfaces for future astronaut autonomy revealed a strong correlation between usability and trust measures. In a controlled lab experiment, we asked novice users to perform a complex scheduling task, during which the software immediately validated the schedule’s constraints and checked for violations. We collected usability (User Experience Questionnaire, UEQ) and trust (Trust in Automated Systems scale, TAS) measures; significant, strong, and moderate correlations emerged between several of the UEQ metrics and TAS. These results support the argument for investing in usability early to enable and sustain trust in highly automated and autonomous systems.
This document defines and explains through examples the Friendly Argument Notation (FAN). FAN builds on previous work investigating text-based ways to express arguments [2, 3]. Its primary intended use is for creating and evaluating arguments about safety-critical systems, especially the types of arguments common within safety and assurance cases [4], but nothing in its design constrains its use to that domain. Compared to existing notations commonly used within this domain (for example [6]), FAN corresponds more closely to traditional argument concepts (for example [1]), allows greater flexibility in expression, provides for including counter-arguments, and requires less knowledge of computer-science-specific concepts. Only time and use will determine how beneficial these differences are in practice. This paper concentrates on showing how FAN looks to someone who is using it manually to develop or assess arguments. A later document will concentrate on providing the information necessary for software tools to be created for FAN.
This whitepaper provides strategic insights and recommendations into security cloud-based solutions for electric utilities, encompassing operational technology (OT), virtual power plants (VPP), distributed energy resources (DERs), applications, networks, and data storage as they transition to and leverage cloud infrastructure through managed service providers (MSPs) and cloud service providers (CSPs). Principles derived from established frameworks serve as a foundation for best practices across cybersecurity projects and remove the constraints of settling on a single framework. For organizations that prefer not to integrate a specific framework altogether, elements of the proposed approach could be adopted or tailored to best fit defined requirements and expected functionalities. The Cirrus assessment, a utility cloud feasibility tool, and the roadmap it provides serve as a precursor to this paper, which seeks to be a valuable resource for defining next steps following cloud technology integration feasibility appraisal. With its comprehensive approach to adoption, the Cirrus framework offers strategic guidance on responsibly preparing for or deploying a utility cloud solution. The previously published whitepaper, “Use Case-Informed Framework for Utility Cloud Migration,” details the guiding strategy, research, and deployment of cloud solutions within electric and interconnected grid systems. Before implementing the controls suggested in this document, it is recommended that stakeholders complete Cirrus's cloud integration assessment and pair the results with their unique cybersecurity controls to form a comprehensive cloud-based utility cybersecurity plan. The Cirrus outcome will consider a series of future architectures for the grid before and after the energy transition and evaluate the arguments for and against cloud applications for each electric and interconnected grid layer. This document is a companion to the original whitepaper, "Use Case-Informed Framework for Utility Cloud Migration" to further identify and recommend security controls based on Cirrus’s cloud integration assessment output. The following whitepaper outlines the cybersecurity controls that secure cloud-service models pertinent to the electric sector using the predefined categories identify, protect, detect, and respond and recover. The objective is to outline prescriptive security controls based on the type of architecture and data stored in the cloud. The focus includes dissecting the shared responsibility model and elucidating what on-premises Infrastructure as a Service (IaaS), Platform as a Service (PaaS), and Software as a Service (SaaS) entail. A pivotal consideration in this context is allocating responsibility for foundational cybersecurity aspects—having used Cirrus for the cloud integration assessment. The ensuing controls detailed herein also represent a checklist of controls necessary for a secure cloud transition, equipping utilities with the knowledge to navigate this digital transformation with confidence and strategic foresight in a safe and responsible manner.
Several attempts have been made recently to apply techniques such as model checking and theorem proving to the analysis of programs. This shall be seen as a current trend to analyze real software systems instead of just their designs. This includes our own effort to develop a model checker for Java, the Java PathFinder 1, one of the very first of its kind in 1998. However, model checking cannot handle very large programs without some kind of abstraction of the program. This paper describes a complementary scalable technique to handle such large programs. Our interest is turned on the observation part of the equation: How much information can be extracted about a program from observing a single execution trace? It is our intention to develop a technology that can be applied automatically and to large full-size applications, with minimal modification to the code. We present a tool, Java PathExplorer (JPaX), for exploring execution traces of Java programs. The tool prioritizes scalability for completeness, and is directed towards detecting errors in programs, not to prove correctness. One core element in JPaX is an instrumentation package that allows to instrument Java byte code files to log various events when executed. The instrumentation is driven by a user provided script that specifies what information to log. Examples of instructions that such a script can contain are: 'report name and arguments of all called methods defined in class C, together with a timestamp'; 'report all updates to all variables'; and 'report all acquisitions and releases of locks'. In more complex instructions one can specify that certain expressions should be evaluated and even that certain code should be executed under various conditions. The instrumentation package can hence be seen as implementing Aspect Oriented Programming for Java in the sense that one can add functionality to a Java program without explicitly changing the code of the original program, but one rather writes an aspect and compiles it into the original program using the instrumentation. Another core element of JPaX is an observation package that supports the analysis of the generated event stream. Two kinds of analysis are currently supported. In temporal analysis the execution trace is evaluated against formulae written in temporal logic. We have implemented a temporal logic evaluator on finite traces using the Maude rewriting system from SRI International, USA. Temporal logic is defined in Maude by giving its syntax as a signature and its semantics as rewrite equations. The resulting semantics is extremely efficient and can handle event streams of hundreds of millions events in few minutes. Furthermore, the implementation is very succinct. The second form of even stream analysis supported is error pattern analysis where an execution trace is analyzed using various error detection algorithms that can identify error-prone programming practices that may potentially lead to errors in some different executions. Two such algorithms focusing on concurrency errors have been implemented in JPaX, one for deadlocks and the other for data races. It is important to note, that a deadlock or data race potential does not need to occur in order for its potential to be detected with these algorithms. This is what makes them very scalable in practice. The data race algorithm implemented is the Eraser algorithm from Compaq, however adopted to Java. The tool is currently being applied to a code base for controlling a spacecraft by the developers of that software in order to evaluate its applicability.
The Overarching Properties (OPs) have been created by an inter-national working group and are being evaluated by the National Aeronautics and Space Administration (NASA), the Federal Aviation Administration (FAA), industry, and other certifying agencies in an effort to streamline certification processes. Their intent is to facilitate the use of alternative approaches and to al-low flexibility to combine the system, software, and complex hardware certification. The hope is that the FAA may eventually establish an Advisory Circular that offers the OPs as a Means of Compliance (MoC) for software approval (and eventually systems and hardware) by showing the product possesses the three OPs: Intent (specification of the intended behavior), Correctness (implementation of the intended behavior) and Innocuity (safety of unintended behavior). In the certification community, there is still a concern about the practicability of using such high level properties in certification. This paper aims to address that concern by showing possession of the OPs in an industrial case study using assurance arguments. The two main contributions of this paper are: a certification process based on OPs as Means of Compliance, and a certification argument for an on-board physical model of an UAV, as industrial example. We pro-pose a hybrid approach for the certification process that combines OPs with existing certification standards. Thus, OPs can be used for parts of a system that uses technologies that are not supported by current standards or for which existing standards require additional effort without commensurate additional safety assurance.
Ongoing design activities at Argonne National Laboratory are requiring a thorough verification of the Argonne Reactor Computation codes be performed. DIF3D is central to this system. The driver for this effort requires the 3D Cartesian, triangular-Z, and hexagonal-Z core geometry options of DIF3D be verified. Previous work identified the DIF3D features required to be verified to support current design activities, features of which are generally applicable to hexagonal-Z fast reactor designs. The scope of this verification effort includes verifying DIF3D’s ability to correctly translate the user’s model in to DIF3D’s preferred format, verifying that options planned for use have the desired effect, and verifying the correctness of the eigenvalue, fixed-source, forward, and adjoint solvers in DIF3D-FD and DIF3D-VARIANT. This manuscript provides the verification tasks and their results with respect to the features needed for current design activities. Since analytic solutions of the neutron diffusion and transport equations are either limited in scope or not possible, multiple tiers of problems unique to each solver and geometry type were implemented. Each of these tiers tests features independent and complementary arguments for why the separate testing of functionalities is acceptable. Finally, this separate testing was also supplemented with a high-level integral check of each the diffusion and transport capabilities and applicable geometries. To accommodate cases which an analytic solution is not feasible, MCNP6.2 was relied upon to provide a higher-order reference solution. This therefore required that the capabilities within MCNP6.2 which were relied upon for this work are also verified in this work. No MCNP discrepancies were noted in this effort. Note that the MCNP6.2 verification included in this work does not stand as a full verification of MCNP6.2, but merely verifies the features used in verifying DIF3D. The verification effort identified no issues that are debilitating or otherwise impactful to design usage of DIF3D, and thus DIF3D version 11.0, release 3012 is considered verified. The types of issues that were identified were predominantly in the areas of: unclear documentation, software bugs which were inconsequential to final results, editing options which were ignored in favor of printing more information than requested, bugs in the outputs of intermediate results, or secondary output binary file information which was not present. While not a bug, this verification report also identified that the algorithm used to evaluate the peak fast flux in a nodal transport solution can be quite unreliable due to the polynomial order used and the location of the peak within the mesh. The authors of the report therefore recommend the usage of the EvaluateFlux software (distributed with ARC) as a more robust alternative.
This work focused on capturing, using, and evolving a qualitative decision support structure across the life cycle of a project. The particular application of this study was towards business process reengineering and the representation of the business process in a set of Business Rules (BR). In this work, we defined a decision model which captured the qualitative decision deliberation process. It represented arguments both for and against proposed alternatives to a problem. It was felt that the subjective nature of many critical business policy decisions required a qualitative modeling approach similar to that of Lee and Mylopoulos. While previous work was limited almost exclusively to the decision capture phase, which occurs early in the project life cycle, we investigated the use of such a model during the later stages as well. One of our significant developments was the use of the decision model during the operational phase of a project. By operational phase, we mean the phase in which the system or set of policies which were earlier decided are deployed and put into practice. By making the decision model available to operational decision makers, they would have access to the arguments pro and con for a variety of actions and can thus make a more informed decision which balances the often conflicting criteria by which the value of action is measured. We also developed the concept of a 'monitored decision' in which metrics of performance were identified during the decision making process and used to evaluate the quality of that decision. It is important to monitor those decision which seem at highest risk of not meeting their stated objectives. Operational decisions are also potentially high risk decisions. Finally, we investigated the use of performance metrics for monitored decisions and audit logs of operational decisions in order to feed an evolutionary phase of the the life cycle. During evolution, decisions are revisisted, assumptions verified or refuted, and possible reassessments resulting in new policy are made. In this regard we implemented a machine learning algorithm which automatically defined business rules based on expert assessment of the quality of operational decisions as recorded during deployment.
Ongoing design activities at Argonne National Laboratory are requiring a thorough verification of the Argonne Reactor Computation codes be performed. DIF3D is central to this system. The driver for this effort requires the 3D Cartesian, triangular-Z, and hexagonal-Z core geometry options of DIF3D be verified. Previous work identified the DIF3D features required to be verified to support current design activities, features of which are generally applicable to hexagonal-Z fast reactor designs. The scope of this verification effort includes verifying DIF3D’s ability to correctly translate the user’s model in to DIF3D’s preferred format, verifying that options planned for use have the desired effect, and verifying the correctness of the eigenvalue, fixed-source, forward, and adjoint solvers in DIF3D-FD and DIF3D-VARIANT. This manuscript provides the verification tasks and their results with respect to the features needed for current design activities. Since analytic solutions of the neutron diffusion and transport equations are either limited in scope or not possible, multiple tiers of problems unique to each solver and geometry type were implemented. Each of these tiers tests features independent and complementary arguments for why the separate testing of functionalities is acceptable. Finally, this separate testing was also supplemented with a high-level integral check of each the diffusion and transport capabilities and applicable geometries. To accommodate cases which an analytic solution is not feasible, MCNP6.2 was relied upon to provide a higher-order reference solution. This therefore required that the capabilities within MCNP6.2 which were relied upon for this work are also verified in this work. No MCNP discrepancies were noted in this effort. Note that the MCNP6.2 verification included in this work does not stand as a full verification of MCNP6.2, but merely verifies the features used in verifying DIF3D. The verification effort identified no issues that are debilitating or otherwise impactful to design usage of DIF3D, and thus DIF3D version 11.0, release 3012 is considered verified. As some additional changes have been made to the ARC software since this point all versions between release 3012 and 3266 can be considered verified as version 3253 was used for all updates in this revision. The types of issues that were identified were predominantly in the areas of: unclear documentation, software bugs which were inconsequential to final results, editing options which were ignored in favor of printing more information than requested, bugs in the outputs of intermediate results, or secondary output binary file information which was not present. While not a bug, this verification report also identified that the algorithm used to evaluate the peak fast flux in a nodal transport solution can be quite unreliable due to the methodology used and the location of the peak within the mesh. The authors of the report therefore recommend the usage of the EvaluateFlux software (distributed with ARC) as a more robust alternative noting that DIF3D will properly notify the user when the peaking values it is providing are potentially incorrect.
Ongoing design activities at Argonne National Laboratory are requiring a thorough verification of the Argonne Reactor Computation codes be performed. DIF3D is central to this system. The driver for this effort requires the 3D Cartesian, triangular-Z, and hexagonal-Z core geometry options of DIF3D be verified. Previous work identified the DIF3D features required to be verified to support current design activities, features of which are generally applicable to hexagonal-Z fast reactor designs. The scope of this verification effort includes verifying DIF3D’s ability to correctly translate the user’s model in to DIF3D’s preferred format, verifying that options planned for use have the desired effect, and verifying the correctness of the eigenvalue, fixed-source, forward, and adjoint solvers in DIF3D-FD and DIF3D-VARIANT. This manuscript provides the verification tasks and their results with respect to the features needed for current design activities. Since analytic solutions of the neutron diffusion and transport equations are either limited in scope or not possible, multiple tiers of problems unique to each solver and geometry type were implemented. Each of these tiers tests features independent and complementary arguments for why the separate testing of functionalities is acceptable. Finally, this separate testing was also supplemented with a high-level integral check of each the diffusion and transport capabilities and applicable geometries. To accommodate cases which an analytic solution is not feasible, MCNP6.2 was relied upon to provide a higher-order reference solution. This therefore required that the capabilities within MCNP6.2 which were relied upon for this work are also verified in this work. No MCNP discrepancies were noted in this effort. Note that the MCNP6.2 verification included in this work does not stand as a full verification of MCNP6.2, but merely verifies the features used in verifying DIF3D. The verification effort identified no issues that are debilitating or otherwise impactful to design usage of DIF3D, and thus DIF3D version 11.0, release 3012 is considered verified. As some additional changes have been made to the ARC software since this point all versions between release 3012 and 3266 can be considered verified as version 3253 was used for all updates in this revision. The types of issues that were identified were predominantly in the areas of: unclear documentation, software bugs which were inconsequential to final results, editing options which were ignored in favor of printing more information than requested, bugs in the outputs of intermediate results, or secondary output binary file information which was not present. While not a bug, this verification report also identified that the algorithm used to evaluate the peak fast flux in a nodal transport solution can be quite unreliable due to the methodology used and the location of the peak within the mesh. The authors of the report therefore recommend the usage of the EvaluateFlux software (distributed with ARC) as a more robust alternative noting that DIF3D will properly notify the user when the peaking values it is providing are potentially incorrect.