Search NASA⌕ Search

SEARCH · Search NASA

Results for “Decision Making”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 73 records · Page 4

Integrating narratives into decision making for complex systems engineering design issues

Abstract Engineering decision making and design requires collaboration between groups from different disciplines, each with different tools, vocabulary, and concerns. Traditional engineering decision‐making tools are generally based on understanding the decision makers’ values, modeling uncertainty with probability, and selecting the alternative that maximizes utility. This rational approach to decision making may not be well understood or used by many stakeholders involved in the engineering design process. Constructing narratives, a basic means of human communication, may aid in engineering communication and comprehension and help with decision making. Narratives represent events by means of a story and usually include characters or agents who cause events and to whom events happen. This paper recommends three methods for how the use of narrative can be applied to the area of engineering decision making. These methods include connecting the decision maker to the analysis, creating narrative simulations for training decision makers, and fostering consensus in problems with multiple stakeholders. An illustrative example of designing a better cookstove for the developing world demonstrates the role that understanding narratives of various stakeholders can play for accomplishing complex systems engineering.

MacKenzie, Cameron A.↗

Stress and Aeronautical Team Decision Making: Strengthening the Weak Links

A model that characterizes pilots'decision making in flight will be presented. Elements of the model that appear most vulnerable to stress will be examined in light of accidents and incidents. The model includes two major components: Situation assessment and choice of a course of action. While based on Klein's Recognition-Primed Decision Making, it is tailored to the aviation environment which includes certain features that may be common to other domains: Primarily, aviation is highly proceduralized and options are generally well known. What appears to make decisions difficult are ambiguity, time pressure, and risk. In addition, decisions must often be made while carrying out the standard procedures of flight, including checklists, review of approach plates, standard briefings, and communication with air traffic controllers or cabin crew. The effects of stressors on decision making by pilots with varying levels of expertise will be explored, along with strategies for strengthening the weak links.

Orasanu, Judith↗

Charging Decision Making for Automated Electric Vehicles

Abstract: Overview/Significance With the emerging of autonomous driving and its application in electric vehicles, the charging decision making should be taken over by the vehicles. Autonomous electric vehicles will require the capability to make charging decisions according to the battery energy state, the travel demand and also the available charging infrastructure. The introduction of autonomous driving technology would remove the challenge of co-locating charging infrastructure with driver destinations and presents a driver-free method for EVs to reach nearby charging stations. This will significantly change the charging behavior of electric vehicles. EV driver will no longer need to be present at charging stations for charging actions. Automated EVs can drive to nearby charging stations to perform charging actions by themselves when necessary. Meanwhile, connected vehicles technology makes electric vehicles have the capability to sense and obtain pertinent information from nearby charging station networks and then calculate the corresponding costs and availability for charging. All these promising capabilities bring new opportunities to construct an automatic charging decision making system and improve energy efficiency of charging actions for automatic electric vehicles. Abstract: Methodology This research introduces optimization tools and case studies on charging decision making for automated electric vehicles under personal usage scenario. The designed optimization tools aim to provide charging strategies, i.e. the optimal choice of charging station outside home and the amount of charged energy, by considering constraints from personal daily itineraries and existing charging infrastructure. Abundant real world EV data is utilized to construct stochastic energy consumption prediction model under realistic driving conditions. Advanced dynamic programming technology is utilized to model and solve the potential multi-stage decision making process by considering realistic uncertainties of EV energy cost and dynamics of itinerary information. Large-scale simulation studies using Chicago itinerary data are demonstrated to illustrate energy benefits from automatic charging decision making for future autonomous EVs. Abstract: Results/Conclusions Optimal automatic charging decision making and its energy benefits have been studied for personal automated electric vehicles. A data-driven method based on large amount of real-world data and the corresponding real-time update algorithm have been proposed to construct multi-channel stochastic energy consumption prediction model. Based on the energy cost prediction, dynamic multi-stage charging decision-making models are established for optimal strategies during a daily itinerary. Derived charging strategies can minimize the monetary and energy cost of charging actions for autonomous vehicles. Simulations and case studies show the potential ability of autonomous electric vehicles to reduce the range anxiety and charging infrastructure dependency. Automatic charging decision making can mitigate the pain from electric vehicle charging necessity outside home. In conclusion, automated electric vehicles equipped with an appropriate automatic charging decision making system will help EV driver to relieve from heavy time cost of recharging, reduce the energy cost of performing charging actions and improve the energy efficiency in future electrified transportation system.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

Got risk? risk-centric perspective for spacecraft technology decision-making

A risk-based decision-making methodology conceived and developed at JPL and NASA has been used to aid in decision making for spacecraft technology assessment, adoption, development and operation. It takes a risk-centric perspective, through which risks are used as a reasoning step to interpose between mission objectives and risk mitigation measures.

risk↗

Stakeholder-Engaged Structured Decision Making for the Los Alamos Legacy Cleanup Mission - 20501

The Los Alamos National Laboratory (LANL) environmental legacy cleanup program requires decisions to be made for environmental remediation, decommissioning and disposal or management of radioactive waste. This legacy cleanup program was established to address groundwater contamination, material disposal areas (MDAs) that have been used to dispose of radioactive and other waste material, and 'aggregate areas' that might produce radioactive or other chemical waste as a result of remediation activities. The LANL site is regulated for environmental concerns under the Resource Conservation and Recovery Act (RCRA). However, some parts of LANL, such as material disposal area G (MDA G), have disposed of radioactive waste under DOE Order 435.1, and are subject to other regulations. For example, decommissioning the remote-handled TRU material stored in 33 shafts at MDA G falls under EPA's 40 CFR 191. Collectively, the regulations are all aimed in the same direction of finding the best solution, either through constructs such as 'as low as reasonably achievable' (ALARA), considering balancing factors as opposed to only cost and human health risk, and, under EPA regulations such as RCRA and NEPA, evaluating impact from all chemicals and both human health and ecological endpoints. Despite the basic goals and objectives of the regulations or their guidance, the main challenge is in their implementation. Arguably perhaps, but really in principle, all of these (and similar) regulations are asking for a decision analysis to be performed. Implementation challenges encountered have included lack of understanding of decision analysis in the industry, lack of effective stakeholder engagement in the decision analysis process, and lack of appreciation of the need to separate value judgments from science, the latter leading to developing conservative, or protective, science-based models. Conservative models lead to poor solutions, lack of effective stakeholder engagement leads to long drawn out protracted approaches to finding a solution (which also might never be found with this approach), and lack of understanding of decision analysis and Bayesian statistics causes poor models to be built, which creates unfortunate situations of 'garbage in, garbage out' becoming the basis for decision making. Stakeholder-engaged structured decision making (SDM) is an approach to solving problems that relies on the theory of decision science to involve stakeholders in the decision-making process. This approach incorporates stakeholder values using a scientifically rigorous methodology that separates value judgments from science in a way that helps avoid the pitfalls of biased, protective, or conservative modeling. This approach has its foundation in Keeney's 1992 treatise on value-focused thinking [1]. Keeney advocated a paradigm shift in decision making based on the idea that the standard way of thinking about decisions is backwards. The standard approach of focusing first on identifying alternatives rather than on articulating values results in a reactive approach with the emphasis on mechanics and fixed choices instead of the core values that have meaning to stakeholders. This paradigm shift effectively engages all stakeholders in the decision-making process while using a values focused thinking approach that can lead to the identification of decision opportunities and the creation of better alternatives. The intent is to be proactive and generate solutions that are related directly to values and objectives as identified by stakeholders. There are, perhaps, two overarching reasons why SDM can be used to benefit LANL's environmental legacy cleanup. Some of LANL's remaining waste management and environmental management problems are challenging and complex (for example, the Cr and RDX plumes, and MDAs) and while the traditional approach has, arguably, worked well for relatively simple risk-based problems, it cannot, or should not, be applied to more complex problems if the most effective and efficient solutions are desired. The second reason is cost. This has perhaps become more critical since publication of the Government Accountability Office (GAO) reports that DoE's environmental liability is considered a high-risk concern for the nation [2]. The focus of SDM is on structuring solutions to decision risk problems by first addressing stakeholder and decision maker values and subsequently developing decision objectives and ways to measure those objectives, preference weighting across objectives, identifying decision alternatives that best achieve those values, and characterizing uncertainty in predictions of the measures. Because a complete decision model is created using SDM, it can be evaluated to find the main elements of the model that drive, or predict, the best solution. This approach creates complete decision models that are transparent, traceable, reproducible and technically defensible. The science behind SDM, or decision analysis, is well founded, yet it is not unusual to see ad hoc approaches to decision making implemented under various environmental regulations that are pertinent to the LANL site, including NEPA, RCRA and DOE Order 435.1. Such ad hoc approaches are often not transparent or traceable, and lack reproducibility and technical defensibility. The LANL legacy cleanup program has embarked on using SDM to address the complex problems that remain. Stakeholder meetings have been held, and a prototype version of the stakeholder value system has been developed. Further meetings are expected in the future to address specific project needs. This is a long-term endeavor considering the complex environmental problems faced by DOE EM in Los Alamos (EM-LA), and careful planning, consideration of stakeholder value systems, and engagement with stakeholders throughout the SDM process is expected to lead to a successful endpoint. (authors)

12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗

A conceptual framework for negotiating public involvement in municipal waste management decision-making in the UK

Highlights: • A review of public involvement in waste management decision-making was undertaken. • Public involvement seen as necessary to create stronger foundations for decisions. • The mode and level of public involvement explored in different decision contexts. • Public involvement can be negotiated based on the technological and social context. • Early public involvement favoured where issues are contentious or uncertain. - Abstract: The technical expertise that politicians relied on in the past to produce cost-effective and environmentally sound solutions no longer provides sufficient justification to approve waste facilities. Local authorities need to find more effective ways to involve stakeholders and communities in decision-making since public acceptance of municipal waste facilities is integral to delivering effective waste strategies. This paper presents findings from a research project that explored attitudes towards greater levels of public involvement in UK waste management decision-making. The study addressed questions of perception, interests, the decision context, the means of engagement and the necessary resources and capacity for adopting a participatory decision process. Adopting a mixed methods approach, the research produced an empirical framework for negotiating the mode and level of public involvement in waste management decision-making. The framework captures and builds on theories of public involvement and the experiences of practitioners, and offers guidance for integrating analysis and deliberation with public groups in different waste management decision contexts. Principles in the framework operate on the premise that the decision about ‘more’ and ‘better’ forms of public involvement can be negotiated, based on the nature of the waste problem and wider social context of decision-making. The collection of opinions from the wide range of stakeholders involved in the study has produced new insights for the design of public engagement processes that are context-dependent and ‘fit-for-purpose’; these suggest a need for greater inclusivity in the case of contentious technologies and high levels of uncertainty regarding decision outcomes.

12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗

Role of PHM in Autonomous Decision-Making: Aerospace applications

There is an increased need for onboard decision-making capabilities in cyber-physical systems be it in energy, automotive, aviation, space, or other industries as they aim for increased efficiency, resiliency, and mission assurance capabilities. Emerging next-gen technologies such as multi-rover planetary missions, distributed satellites, unmanned ground and aerial vehicle operations and smart grid systems rely on in-time risk assessment and autonomous decision-making. One critical piece of the autonomy puzzle is reliable prediction of system behavior under time-varying and potentially uncertain environmental conditions. Further, if agent states change during operation such as initiation of faults or degradation, reliable diagnostic tools need to be investigated. In this tutorial, we will revise approaches that integrates existing physics-based and data-driven models of agents interacting with probability models of the environment and component operation state. Role of existing PHM methodologies as they feed into decision-making under uncertainty will be studied. Balancing critical trade-offs between high-fidelity prognostic models, prediction time-horizons and the computational requirements for in-time cost-effective decision-making will be discussed through the implementation of surrogate models. Finally, the audience will be introduced to a real-time application of in-time trajectory planning of an unmanned aerial system (UAS) based on its PHM assessments under uncertain and varying wind conditions.

decision-making↗

Decision making, procedural compliance, and outcomes definition in U.S. forest service planning processes

The National Environmental Policy Act (NEPA) dictates a process of analyzing and disclosing the likely impacts of proposed agency actions on the human environment. This study addresses two key questions related to NEPA implementation in the U.S. Forest Service: 1) how do Interdisciplinary (ID) team leaders and decision makers conceptualize the outcomes of NEPA processes? And 2), how does NEPA relate to agency decision making? We address these questions through two separate online surveys that posed questions about recently completed NEPA processes - the first with the ID team leaders tasked with carrying out the processes, and the second with the line officers responsible for making the processes' final decisions. Outcomes of NEPA processes include impacts on public relations, on employee morale and team functioning, on the achievement of agency goals, and on the achievement of NEPA's procedural requirements (disclosure) and substantive intent (minimizing negative environmental impacts). Although both tended to view public relations outcomes as important, decision makers' perceptions of favorable outcomes were more closely linked to the achievement of agency goals and process efficiency than was the case for ID team leaders. While ID team leaders' responses suggest that they see decision making closely integrated with the NEPA process, decision makers more commonly decoupled decision making from the NEPA process. These findings suggest a philosophical difference between ID team leaders and decision makers that may pose challenges for both the implementation and the evaluation of agency NEPA. We discuss the pros and cons of integrating NEPA with decision making or separating the two. We conclude that detaching NEPA from decision making poses greater risks than integrating them.

54 ENVIRONMENTAL SCIENCES↗

System risk quantification and decision making support using functional modeling and dynamic Bayesian network

Risk-informed decision-making requires a probabilistic assessment of the likelihood of success of control action, given the system status. This paper presents a systematic state transition modeling approach integrating dynamic probabilistic risk assessment with a decision-making process using a dynamic Bayesian network (DBN) coupled with functional modeling. A functional model designed with multilevel flow modeling (MFM) technique was used to build a system state structure inferred by energy, mass, and information flow so that one can verify the developed model with respect to system functionality. The MFM model represents the causal relationship among the nodes, which captures the structure of process parameters and control units. Each node may have multiple possible states, and the DBN structured by the MFM model represents the time-domain transitions among the defined states. Furthermore, the MFM-DBN integrated state transition modeling is a white-box approach that allows one to draw the system's risk profile by updating the system states and supports the decisions probabilistically with physical inference. An example of a simple heating system has been used to illustrate this process, including decision-making support based on quantitative risk profile. For demonstrating its applicability to a complex system operational decision making, a case study of station blackout accident scenario leading to the seal loss of coolant accident in a nuclear power plant is presented. The proposed approach effectively provided the risk profile along time for each option so that the operators can make the best decision, which minimizes the plant risk.

42 ENGINEERING↗

Probabilistic Risk Assessment for Decision Making During Spacecraft Operations

Decisions made during the operational phase of a space mission often have significant and immediate consequences. Without the explicit consideration of the risks involved and their representation in a solid model, it is very likely that these risks are not considered systematically in trade studies. Wrong decisions during the operational phase of a space mission can lead to immediate system failure whereas correct decisions can help recover the system even from faulty conditions. A problem of special interest is the determination of the system fault protection strategies upon the occurrence of faults within the system. Decisions regarding the fault protection strategy also heavily rely on a correct understanding of the state of the system and an integrated risk model that represents the various possible scenarios and their respective likelihoods. Probabilistic Risk Assessment (PRA) modeling is applicable to the full lifecycle of a space mission project, from concept development to preliminary design, detailed design, development and operations. The benefits and utilities of the model, however, depend on the phase of the mission for which it is used. This is because of the difference in the key strategic decisions that support each mission phase. The focus of this paper is on describing the particular methods used for PRA modeling during the operational phase of a spacecraft by gleaning insight from recently conducted case studies on two operational Mars orbiters. During operations, the key decisions relate to the commands sent to the spacecraft for any kind of diagnostics, anomaly resolution, trajectory changes, or planning. Often, faults and failures occur in the parts of the spacecraft but are contained or mitigated before they can cause serious damage. The failure behavior of the system during operations provides valuable data for updating and adjusting the related PRA models that are built primarily based on historical failure data. The PRA models, in turn, provide insight into the effect of various faults or failures on the risk and failure drivers of the system and the likelihood of possible end case scenarios, thereby facilitating the decision making process during operations. This paper describes the process of adjusting PRA models based on observed spacecraft data, on one hand, and utilizing the models for insight into the future system behavior on the other hand. While PRA models are typically used as a decision aid during the design phase of a space mission, we advocate adjusting them based on the observed behavior of the spacecraft and utilizing them for decision support during the operations phase.

dynamic fault trees↗

Acquisition and production of skilled behavior in dynamic decision-making tasks

Ongoing research investigating perceptual and contextual influences on skilled human performance in dynamic decision making environments is discussed. The research is motivated by two general classes of findings in recent decision making research. First, many studies suggest that the concrete context in which a task is presented has strong influences on the psychological processes used to perform the task and on subsequent performance. Second, studies of skilled behavior in a wide variety of task environments typically implicate the perceptual system as an important contributor to decision-making performance, either in its role as a mediator between the current decision context and stored knowledge, or as a mechanism capable of directly initiating activity through the development of a 'trained eye.' Both contextual and perceptual influences place limits on the ability of traditional utility-theoretic accounts of decision-making to guide display design, as variance in behavior due to contextual factors or the development of a perceptual skill is left unexplained. The author outlines a framework in which to view questions of perceptual and contextual influences on behavior and describe an experimental task and analysis technique which will be used to diagnose the possible role of perception in skilled decision making performance.

Kirlik, Alex↗

Modular evaluation method for subsurface activities (MEMSA). A novel approach for integrating social acceptance in a permit decision-making process for subsurface activities

We investigate how the decision support system ‘Modular Evaluation Method Subsurface Activities’ (MEMSA) can help facilitate an informed decision-making process for permit applications of subsurface activities. To this end, we analyze the extent the MEMSA approach allows for a dialogue between stakeholders in a transparent manner. We use the exploration permit for the underground gas storage facility at the Pieterburen salt dome (Netherlands) as a case study. The results suggest that the MEMSA approach is flexible enough to adjust to changing conditions. Furthermore, MEMSA provides a novel way for identifying structural problems and possible solutions in permit decision-making processes for subsurface activities, on the basis of the sensitivity analysis of intermediate rankings. We suggest that the planned size of an activity should already be specified in the exploration phase, because this would allow for a more efficient use of the subsurface as a whole. We conclude that the host community should be involved to a greater extent and in an early phase of the permit decision-making process, for example, already during the initial analysis of the project area of a subsurface activity. We suggest that strategic national policy goals are to be re-evaluated on a regular basis, in the form of a strategic vision for the subsurface, to account for timing discrepancies between the realization of activities and policy deadlines, because this discrepancy can have a large impact on the necessity and therefore acceptance of a subsurface activity.

12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗

Method and apparatus for constructing informative outcomes to guide multi-policy decision making

In Multi-Policy Decision-Making (MPDM), many computationally-expensive forward simulations are performed in order to predict the performance of a set of candidate policies. In risk-aware formulations of MPDM, only the worst outcomes affect the decision making process, and efficiently finding these influential outcomes becomes the core challenge. Recently, stochastic gradient optimization algorithms, using a heuristic function, were shown to be significantly superior to random sampling. In this disclosure, it was shown that accurate gradients can be computed-even through a complex forward simulation—using approaches similar to those in dep networks. The proposed approach finds influential outcomes more reliably, and is faster than earlier methods, allowing one to evaluate more policies while simultaneously eliminating the need to design an easily-differentiable heuristic function.

Olson, Edwin↗

Harnessing Condorcet Methods to Improve Decision-making Based on Ranked Data

Decision-makers must often choose between multiple alternatives based on their relative merits across a variety of criteria. These multiple-criteria decision-making (MCDM) problems must be approached in an objective, measurable, and transparent fashion to obtain well-supported outcomes. The challenge of using sets of ranked data to identify the optimal choice is not unlike the challenge of using ranked ballots to identify the winner of an election in a preference voting system. As such, the methods of rank aggregation used in elections can be applied to help resolve MCDM problems. Previous research into the design of elections and ballots has yielded many algorithms with well-understood properties. A subset of these, called Condorcet methods, reliably identify the “Condorcet winner,” if it exists, giving the result that would defeat any other in a pairwise comparison. That, in addition to several other desirable qualities, makes Condorcet methods like the Schulze method, Ranked Pairs, and Copeland’s method, effective and scalable means for tackling MCDM problems. These rank aggregation methods have been implemented in JavaScript functions for incorporation into a web-based MCDM decision tool. These algorithms can be applied in a wide variety of contexts (for example, determining the “best” environmental remediation method or selecting a subcontractor) to facilitate effective decision-making. The performance of the JavaScript Condorcet prototype tool was compared across the implemented algorithms and with both a non-Condorcet rank aggregation method as well as an implementation of the Simple Multi-Attribute Rating Technique (SMART) algorithm.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗

Can You Verifi This? Studying Uncertainty and Decision-Making About Misinformation using Visual Analytics

We describe a novel study of decision-making processes around misinformation on social media. Using a custom-built visual analytic system, we presented users with news content from social media accounts from a variety of news outlets, including outlets engaged in distributing misinformation. We conducted controlled experiments to study decision-making regarding the veracity of these news outlets and tested the role of confirmation bias (the tendency to ignore contradicting information) and uncertainty of information on human decision-making processes. Our findings reveal that the presence of conflicting information, presented to users in the form of cues, impacts the ability to judge the veracity of news in systematic ways. We also find that instructing participants even to explicitly disconfirm given hypotheses does not significantly impact their decision-making regarding misinformation when compared to a control condition. Our findings have the potential to inform the design of visual analytics systems so that they may be used to mitigate the effects of cognitive biases and stymie the spread of misinformation on social media.

misinformation, decision making, social media, vis↗

Analytical Framework for Homeowner-Focused Computational Cost-Benefit Mitigation Decision Making

As a result of increasing windstorm losses in the United States over the past 50 years, a variety of residential wind hazard mitigation strategies have been recommended. Wind hazard mitigation undoubtedly reduces windstorm losses; however, little customized information is available to support the economic decision-making process for individual stakeholders, especially at the consumer level. Therefore, there is a need for decision support tools to calculate, evaluate, and present the economic benefits of mitigation over variable decision-making time horizons for unique building characteristics in specific geographical locations. This paper presents a cost-benefit computational decision-making recommendation framework that evaluates and presents the cost effectiveness of mitigation, customized based on location, years of interest, and building characteristics. The decision-making framework consists of two cores. The user-independent core and user-dependent core which consists of eight steps, divided into four subprocesses: (1) data acquisition, (2) data evaluation, (3) cost-benefit calculation, and (4) results comparison and decision making. To demonstrate the framework, a wood-framed single-family home in light suburban terrain in Golden Meadow, Louisiana, is used as a case study, considering retrofit mitigation with a $15,000 budget and a decision-making time horizon of 30 years. The case study provides six mitigation options, with cost ranging between approximately $2,000 and $15,000 and net benefit/cost ratios ranging from 2.4 to 13. Once fully implemented in a decision making information tool, the developed mitigation recommendation framework will provide valuable information that can be used by home designers, builders, owners, and other stakeholders to better understand the economic facets of wind hazard mitigation, providing critical information as part of the overall decision-making process.

Orooji, Fatemeh↗

Prospective Architectures for Onboard vs Cloud-Based Decision Making for Unmanned Aerial Systems

This paper investigates propsective architectures for decision-making in unmanned aerial systems. When these unmanned vehicles operate in urban environments, there are several sources of uncertainty that affect their behavior, and decision-making algorithms need to be robust to account for these different sources of uncertainty. It is important to account for several risk-factors that affect the flight of these unmanned systems, and facilitate decision-making by taking into consideration these various risk-factors. In addition, there are several technical challenges related to autonomous flight of unmanned aerial systems; these challenges include sensing, obstacle detection, path planning and navigation, trajectory generation and selection, etc. Many of these activities require significant computational power and in many situations, all of these activities need to be performed in real-time. In order to efficiently integrate these activities, it is important to develop a systematic architecture that can facilitate real-time decision-making. Four prospective architectures are discussed in this paper; on one end of the spectrum, the first architecture considers all activities/computations being performed onboard the vehicle whereas on the other end of the spectrum, the fourth and final architecture considers all activities/computations being performed in the cloud, using a new service known as Prognostics as a Service that is being developed at NASA Ames Research Center. The four different architectures are compared, their advantages and disadvantages are explained and conclusions are presented.

Software engineering↗