Engineering topics
Meshkat, Leila
Publications and source records attributed to Meshkat, Leila.
Quantifying Cybersecurity Risk for NASA Missions
An end-to-end cyber risk assessment process is presented that is based on the combination of guidelines from the National Institute of Standards & Technology (NIST), the standard 5x5 risk matrix, and quantitative methods for generating loss exceedance curves. The NIST guidelines provide a framework for cyber risk assessment, and the standard 5x5 matrix is widely used across the industry for the representation of risk across multiple disciplines. Loss exceedance curves are a means of quantitatively assessing the loss that occurs due to a given risk profile. Combining these different techniques enables us to follow the guidelines, adhere to standard 5x5 risk management practices and develop quantitative metrics simultaneously. Our quantification process is based on the consideration of the NASA and JPL Cost Risk assessment modeling techniques as we define the cost associated with the cybersecurity risk profile of a mission as a function of the mission cost.
Reliability-Aware Requirements Development for Autonomy Software
Autonomy is the ability of a system to make decisions and take actions in the presence of uncertainty. Validating autonomy, therefore, is a matter of validating that the system makes intelligent decisions even when there might be discrepancies between the observed and the expected state of the world as well as when the possible outcome of each of the possible courses of action is uncertain. During the requirements engineering phase, the functions, including those that are required to be autonomous and their expected behavior are specified. However, in the case of autonomous software system, it is challenging to specify all possible scenarios that the software may encounter. Additionally, since autonomous behaviors cannot always be fully predetermined, it is difficult to reason about their completeness and correctness. While these challenges are also faced with non-autonomous system, they are more prevalent for autonomous system due to their complexity and emergent behavior. Our efforts have been three-fold. On one hand, we have developed a goal-based method for requirements decomposition. We have embedded this approach into a higher-level systems engineering framework and then developed methods for decision and reliability analysis within this framework. And finally, we use simulation analysis to validate the more theoretical methods and provide a feedback mechanism. The decision analysis approach (DA) helps in identifying and mitigating the run-time risks, by bringing to the forefront the uncertainties, decisions, interactions, and other factors that may cause autonomous software to make erroneous decisions. The DA framework is used to quantify, for each decision that the software can make, failure risk due to uncertainties. The information about the key decisions and circumstances that can cause autonomous system to make incorrect decisions are used to generate test scenarios to be run in a simulator to ensure that the system can handle error-prone circumstances. The results or outcomes from decisions, based on the simulated scenarios, can then be leveraged to further gain insight of the assured system. We combine a goal-based approach with simulation analysis to facilitate requirements development for autonomous systems and further provide a method for run-time systems level reliability considerations during the autonomous software development process. The DA and the reliability analysis methods provide a formalism for the consideration of uncertainty in the run time decision-making process for autonomous software. This formalism considers the uncertainty involved in the outcome of different courses of action, in terms of performance and cost, as well as a method to consider the system level reliability implications. System level reliability considerations for autonomous software decision making contribute to the satisfaction of the system level goals over the lifecycle of a given mission. We demonstrate our approach using a hypothetical rover path planning example. The requirements for our rover simulator are based on existing literature about Mars rovers and lessons learned from real world observations at JPL. The specifics of the autonomy design for the Mars rovers, that have not been previously cleared for external release, are not used for our demonstrations.
Risk considerations for autonomy software
This paper summarizes key findings related to methods for the risk considerations of autonomy software. Existing methods for Verification and Validation (V&V) of autonomy software are summarized and a method for the assurance of autonomy software is suggested and demonstrated on a command execution use case. Risk and reliability are defined in the context of autonomy and an approach for risk assessment of autonomy is presented using an example use case. Key insights regarding areas of uncertainty for autonomy are provided, along with a suggested architecture for the systematic consideration of reliability within the context of a given autonomous planner.
Behavior modeling for cybersecurity
In this paper, we elaborate on the key actors within the context of the cyber world at the Jet Propulsion Laboratory and use Bayesian Belief Networks to represent the causal and probabilistic relationships between the various elements that affect an actor and the likelihood of exploits and adverse consequences that can occur due to an exploit. We assess the effectiveness of each of the mitigative methods and the sensitivity of the system to each of the aggravating as well as mitigative factors. We use a combination of objective incident data and Subject Matter Expert knowledge as input to these models.
Data Analysis & Statistical Methods for Command File Errors
This paper explains current work on modeling for managing the risk of command file errors. It is focused on analyzing actual data from a JPL spaceflight mission to build models for evaluating and predicting error rates as a function of several key variables. We constructed a rich dataset by considering the number of errors, the number of files radiated, including the number commands and blocks in each file, as well as subjective estimates of workload and operational novelty. We have assessed these data using different curve fitting and distribution fitting techniques, such as multiple regression analysis, and maximum likelihood estimation to see how much of the variability in the error rates can be explained with these. We have also used goodness of fit testing strategies and principal component analysis to further assess our data. Finally, we constructed a model of expected error rates based on the what these statistics bore out as critical drivers to the error rate. This model allows project management to evaluate the error rate against a theoretically expected rate as well as anticipate future error rates.
Data Analysis & Statistical Methods for Command File Errors
No abstract available
Addressing the Hard Factors for Command File Errors by Probabilistic Reasoning
Command File Errors (CFE) are managed using standard risk management approaches at the Jet Propulsion Laboratory. Over the last few years, more emphasis has been made on the collection, organization, and analysis of these errors for the purpose of reducing the CFE rates. More recently, probabilistic modeling techniques have been used for more in depth analysis of the perceived error rates of the DAWN mission and for managing the soft factors in the upcoming phases of the mission. We broadly classify the factors that can lead to CFE's as soft factors, which relate to the cognition of the operators and hard factors which relate to the Mission System which is composed of the hardware, software and procedures used for the generation, verification & validation and execution of commands. The focus of this paper is to use probabilistic models that represent multiple missions at JPL to determine the root cause and sensitivities of the various components of the mission system and develop recommendations and techniques for addressing them. The customization of these multi-mission models to a sample interplanetary spacecraft is done for this purpose.
Soft Factors and Space Mission Failures: Quantifying the Effect of Management Decisions
No abstract available
Towards a Decision Support System for Space Flight Operations
The Mission Operations Directorate (MOD) at the Johnson Space Center (JSC) has put in place a Model Based Systems Engineering (MBSE) technological framework for the development and execution of the Flight Production Process (FPP). This framework has provided much added value and return on investment to date. This paper describes a vision for a model based Decision Support System (DSS) for the development and execution of the FPP and its design and development process. The envisioned system extends the existing MBSE methodology and technological framework which is currently in use. The MBSE technological framework currently in place enables the systematic collection and integration of data required for building an FPP model for a diverse set of missions. This framework includes the technology, people and processes required for rapid development of architectural artifacts. It is used to build a feasible FPP model for the first flight of spacecraft and for recurrent flights throughout the life of the program. This model greatly enhances our ability to effectively engage with a new customer. It provides a preliminary work breakdown structure, data flow information and a master schedule based on its existing knowledge base. These artifacts are then refined and iterated upon with the customer for the development of a robust end-to-end, high-level integrated master schedule and its associated dependencies. The vision is to enhance this framework to enable its application for uncertainty management, decision support and optimization of the design and execution of the FPP by the program. Furthermore, this enhanced framework will enable the agile response and redesign of the FPP based on observed system behavior. The discrepancy of the anticipated system behavior and the observed behavior may be due to the processing of tasks internally, or due to external factors such as changes in program requirements or conditions associated with other organizations that are outside of MOD. The paper provides a roadmap for the three increments of this vision. These increments include (1) hardware and software system components and interfaces with the NASA ground system, (2) uncertainty management and (3) re-planning and automated execution. Each of these increments provide value independently; but some may also enable building of a subsequent increment.
Modeling to Improve the Risk Reduction Process for Command File Errors
The Jet Propulsion Laboratory has learned that even innocuous errors in the spacecraft command process can have significantly detrimental effects on a space mission. Consequently, such Command File Errors (CFE), regardless of their effect on the spacecraft, are treated as significant events for which a root cause is identified and corrected. A CFE during space mission operations is often the symptom of imbalance or inadequacy within the system that encompasses the hardware and software used for command generation as well as the human experts and processes involved in this endeavor. As we move into an era of increased collaboration with other NASA centers and commercial partners, these systems become more and more complex. Consequently, the ability to thoroughly model and analyze CFEs formally in order to reduce the risk they pose is increasingly important. In this paper, we summarize the results of applying modeling techniques previously developed to the DAWN flight project. The original models were built with the input of subject matter experts from several flight projects. We have now customized these models to address specific questions for the DAWN flight project and formulating use cases to address their unique mission needs. The goal of this effort is to enhance the project's ability to meet commanding reliability requirements for operations and to assist them in managing their Command File Errors.
Towards a Decision Support System for Space Flight Operations
No abstract available
Towards a decision support system for space flight operations
The Mission Operations Directorate (MOD) at the Johnson Space Center (JSC) has put in place a Model Based Systems Engineering (MBSE) technological framework for the development and execution of the Flight Production Process (FPP). This framework has provided much added value and return on investment to date. This paper describes a vision for a model based Decision Support System (DSS) for the development and execution of the FPP and its design and development process. The envisioned system extends the existing MBSE methodology and technological framework which is currently in use. The MBSE technological framework currently in place enables the systematic collection and integration of data required for building an FPP model for a diverse set of missions. This framework includes the technology, people and processes required for rapid development of architectural artifacts. It is used to build a feasible FPP model for the first flight of spacecraft and for recurrent flights throughout the life of the program. This model greatly enhances our ability to effectively engage with a new customer. It provides a preliminary work breakdown structure, data flow information and a master schedule based on its existing knowledge base. These artifacts are then refined and iterated upon with the customer for the development of a robust end-to-end, high-level integrated master schedule and its associated dependencies. The vision is to enhance this framework to enable its application for uncertainty management, decision support and optimization of the design and execution of the FPP by the program. Furthermore, this enhanced framework will enable the agile response and redesign of the FPP based on observed system behavior. The differences between the anticipated system behavior and the observed behavior may be due to the processing of tasks internally, or due to external factors such as changes in program requirements or conditions associated with other organizations that are outside of MOD. The paper provides a roadmap for the four increments of this vision. These increments include (1) the existing capabilities (2) hardware and software system components and interfaces with the NASA ground system, (3) uncertainty management and (4) re-planning and automated execution. Each of these increments provides value independently; but some may also enable building of a subsequent increment.