Search NASA⌕ Search

SEARCH · Search NASA

Results for “Decision Making, Organizational”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 37 records · Page 2

Applying Costs, Risks and Values Evaluation (CRAVE) methodology to Engineering Support Request (ESR) prioritization

Given limited budget, the problem of prioritization among Engineering Support Requests (ESR's) with varied sizes, shapes, and colors is a difficult one. At the Kennedy Space Center (KSC), the recently developed 4-Matrix (4-M) method represents a step in the right direction as it attempts to combine the traditional criteria of technical merits only with the new concern for cost-effectiveness. However, the 4-M method was not adequately successful in the actual prioritization of ESRs for the fiscal year 1995 (FY95). This research identifies a number of design issues that should help us to develop better methods. It emphasizes that given the variety and diversity of ESR's one should not expect that a single method could help in the assessment of all ESR's. One conclusion is that a methodology such as Costs, Risks, and Values Evaluation (CRAVE) should be adopted. It also is clear that the development of methods such as 4-M requires input not only from engineers with technical expertise in ESR's but also from personnel with adequate background in the theory and practice of cost-effectiveness analysis. At KSC, ESR prioritization is one part of the Ground Support Working Teams (GSWT) Integration Process. It was discovered that the more important barriers to the incorporation of cost-effectiveness considerations in ESR prioritization lie in this process. The culture of integration, and the corresponding structure of review by a committee of peers, is not conducive to the analysis and confrontation necessary in the assessment and prioritization of ESR's. Without assistance from appropriately trained analysts charged with the responsibility to analyze and be confrontational about each ESR, the GSWT steering committee will continue to make its decisions based on incomplete understanding, inconsistent numbers, and at times, colored facts. The current organizational separation of the prioritization and the funding processes is also identified as an important barrier to the pursuit of cost-effectiveness. Perhaps the greatest barrier is that, at the working level, KSC's culture is so preoccupied with technical concerns that it seems almost oblivious to any cost concerns, let alone cost-effectiveness concerns. It is recommended that we must urgently begin to change that culture and seek a better balance between these two concerns.

Joglekar, Prafulla N.↗

Space Research Project Management Can Benefit from Engineering Technology Selection Methods

Many engineering methods have been developed to help management select technology for a system design or further research. The simplest way to compare technologies is to use a checklist containing all the more or less important selection criteria, so that nothing is overlooked. The criteria usually include cost, safety, reliability and maintainability, and potential problems such as noise generation and microgravity sensitivity. The next step typically is to weight and score all the criteria. The process of weighting and scoring is helpful in bringing out different priorities and reaching a shared point of view. Group technology selection methods are designed to highlight initial disagreements and produce a shared consensus. Often a frank discussion led by management rather than decision analysts can be more effective. The final selection depends on management and engineering judgment and may include programmatic and organizational factors that are beyond the engineering checklist. The objective of engineering technology selection methods is to provide engineering information to assist management in making sound decisions. Project management and technology selection are assumed to use rational engineering analytic methods, but they often do not. The reason is that human insight, intuition, and “gut feel,” rather than logic, more frequently determine our decisions. Project selection and management are strongly influenced by nonrational psychological influences, which can produce unjustified confidence and determination. Nevertheless, there is a strong need for space projects to do rational project analysis and selection. Demonstrating a rational spirit is necessary for a scientific and technical organization. Professional ethics at its best requires an open, honest, and fair process, without damaging politics. Rational analysis can help improve good projects and avoid selecting bad ones. A sanity check using rational analysis guided by a checklist can help avoid egregious and damaging errors.

Jones, Harry W.↗

Engineering Elegant Systems: Postulates, Principles, and Hypotheses of Systems Engineering

Systems engineering integrated both the system and the organizational engineering disciplines to produce an elegant system. The NASA Systems Engineering Research Consortium has developed systems engineering postulates, principles, and hypotheses defining the physical and social aspects of systems engineering. This paper presents an overview of the current revision of this basis for systems engineering. This basis addresses several key aspects of systems engineering including system specific approach, organizational influences, policy and law impacts, application across the system life cycle, the mathematical basis of systems engineering, decision making, clearly distinguishing verification from validation, and system optimization.

Watson, Michael D.↗

Engineering Elegant Systems: Postulates, Principles, and Hypotheses of Systems Engineering

Systems engineering integrated both the system and the organizational engineering disciplines to produce an elegant system. The NASA Systems Engineering Research Consortium has developed systems engineering postulates, principles, and hypotheses defining the physical and social aspects of systems engineering. This paper presents an overview of the current revision of this basis for systems engineering. This basis addresses several key aspects of systems engineering including system specific approach, organizational influences, policy and law impacts, application across the system life cycle, the mathematical basis of systems engineering, decision making, clearly distinguishing verification from validation, and system optimization.

Watson, Michael D.↗

Perceptions of Climate Risk and Use of Climate Risk Information by Natural Resource Conservation Stakeholders Participating in ADVANCE Projects in Asia and Latin America

Integrating climate risk information into resilience-building activities in the field is important to ensure that adaptation is based on the best available science. Despite this, many challenges exist when developing, communicating, and incorporating climate risk information. There are limited resources on how stakeholders perceive risks, use risk information, and what barriers exist to limit knowledge integration. This paper seeks to define the following: 1) What do conservation stakeholders consider to be the most significant climate risks they face now and possibly in the future? 2) What have been the most significant barriers to their using climate risk information? 3) What sources and types of knowledge would be most useful for these managers to overcome these barriers? A survey was conducted among stakeholders (n = 224) associated with World Wildlife Fund projects in tropical and subtropical countries. A very high proportion of stakeholders used climate risk information and yet faced integration-related challenges, which included too much uncertainty and the lack of a relevant scale for planning. The main factors preventing the use of climate risk information in decision-making were unavailability of climate risk information, no or limited financial or human resources available to respond, lack of organizational mandate or support, and no or limited institutional incentives. Comparing perceived current and future risks revealed a decline in concern for some future climate hazards. Survey respondents identified scientific reports, climate scientists, and online sources as the most useful information sources of climate risk information, while (i) maps and illustrations; (ii) scenarios format; and (iii) data tables, graphs, and charts were identified as user-friendly formats.

Climate risk↗

Collective Intelligence and Three Aspects of Planning in Organizations: A NASA Example

For many complex sociotechnical systems, planning seems to require explicit coordination; certainly, in executing a plan the activities of different actors must be tightly coupled. However, distributing the needed planning information can be very burdensome and error prone, because different groups need different collections of information, updated or kept current on different time cycles. Further, the information needed to form successful plans is often highly distributed, and while feedback about the success of prior plans may exist, it may not be available to those in a position of using this to improve plans or to detect and resolve other problems in the system (Weick, 1995). Tools to support various aspects of planning have been developed, and can provide a huge benefit to the individuals working on that aspect. To be tractable, most solutions address a quite bounded slice of work, isolating it from the larger context. Prospective planning takes place over multiple, nested cycles of decision making. This builds a plan that specifies activities of different granularity. "Subplans" may specify multiple parallel activities by different groups and individuals, as well as sequential, nested actions by a single actor. Planning produces valuable, sharable, external, representations: in addition to prospective use, plans support retrospective assessment and also action in the present. Viewing planning in a larger context - both temporal and organizational -- enables noticing what one does not know and generating more systemic and effective solutions. Viewing a problem as one of collective intelligence invites thinking about the larger organizational context. Many approaches to supporting collective intelligence do not support execution of highly contingent actions, distributed across many players, and hence provide incomplete support for planning. However, CI technology maybe helpful in managing the processes of gathering information for decision making in planning and of distributing plan information to various actors, needed on various time cycles.

Billman, Dorrit↗

Public science policy and administration

Science, the overwhelming concern of our time, is no longer a matter of private research and development but one of public policy and administration, in which government, industry, foundations, and educational institutions must all work together as never before. Few other single tasks are of such decisive importance to the collective and individual welfare of American citizens as the formulation of public science policy and the administration of scientific programs. Eleven national authorities of varied background in science, education, and government administration contribute their experience and their judgment in an effort to deal with the major aspects of the subject. Their focus is on the meeting of actual problems; they consider the decision making process in both public and public-private organizations. Topics are grouped in three general categories: personnel needs and resources, organizational problems and techniques, and the administrative role in policy leadership.

Rosenthal, A. H.↗

Improving the Transition of Earth Satellite Observations from Research to Operations

There are significant gaps between the observations, models, and decision support tools that make use of new data. These challenges include: 1) Decreasing the time to incorporate new satellite data into operational forecast assimilation systems, 2) Blending in-situ and satellite observing systems to produce the most accurate and comprehensive data products and assessments, 3) Accelerating the transition from research to applications through national test beds, field campaigns, and pilot demonstrations, and 4) Developing the partnerships and organizational structures to effectively transition new technology into operations. At the Short-term Prediction Research and Transition (SPORT) Center in Huntsville, Alabama, a NASA-NOAA-University collaboration has been developed to accelerate the infusion of NASA Earth science observations, data assimilation and modeling research into NWS forecast operations and decision-making. The SPoRT Center research focus is to improve forecasts through new observation capability and the regional prediction objectives of the US Weather Research Program dealing with 0-1 day forecast issues such as convective initiation and 24-hr quantitative precipitation forecasting. The near real-time availability of high-resolution experimental products of the atmosphere, land, and ocean from the Moderate Resolution Imaging Spectroradiometer (MODIS), the Advanced Infrared Spectroradiometer (AIRS), and lightning mapping systems provide an opportunity for science and algorithm risk reduction, and for application assessment prior to planned observations from the next generation of operational low Earth orbiting and geostationary Earth orbiting satellites. This paper describes the process for the transition of experimental products into forecast operations, current products undergoing assessment by forecasters, and plans for the future. The SPoRT Web page is at (http://www.ghcc.msfc.nasa.gov/sport).

Goodman, Steven J.↗

Definition of Earth Resources Policy and Management Problems in California

The activities of the Social Sciences Group in solving earth resources management problems as related to social factors, are reported. Major efforts of the Group revolved around identifying potential users of ERTS data, ascertain the user's needs, and assay the organizational impacts of new and technologically advanced sources of information. Attempts were also made to develop a linear programming model to be used in decision making with respect to resources being observed by ERTS and other remote sensing vehicles. The cost effectiveness of solving these management problems is discussed.

Churchman, C. W.↗

NASA Risk-Informed Decision Making Handbook

This handbook provides guidance for conducting risk-informed decision making in the context of NASA risk management (RM), with a focus on the types of direction-setting key decisions that are characteristic of the NASA program and project life cycles, and which produce derived requirements in accordance with existing systems engineering practices that flow down through the NASA organizational hierarchy. The guidance in this handbook is not meant to be prescriptive. Instead, it is meant to be general enough, and contain a sufficient diversity of examples, to enable the reader to adapt the methods as needed to the particular decision problems that he or she faces. The handbook highlights major issues to consider when making decisions in the presence of potentially significant uncertainty, so that the user is better able to recognize and avoid pitfalls that might otherwise be experienced.

Dezfuli, Homayoon↗

Event Report for The Ethical Artificial Intelligence Quantification Workshop

Artificial Intelligence (AI) is a powerful emerging technology area which requires special attention to using it ethically. AI ethics is still an emerging field, and the partners for this workshop and report seek to move AI ethics discussion ahead by experimenting with ways to measure AI ethics criteria. The following document describes the outcomes and learnings from The Ethical Artificial Intelligence Quantification Workshop held at the National Institute for Aerospace (NIA), Hampton, Virginia on May 12th, 2022. The purpose of the workshop was for participants to evaluate and experiment-with the methodology and process presented by AIEthics.World in cooperation with Intel Corporation. The meeting participants learned about the Ethical AI Certification and Maturity Model™ and applied the methodology to selected notional AI systems. The workshop facilitated the evaluation of the maturity of the AI system according to ethical considerations relevant to NASA, NIA and other participants. The workshop consisted of three main phases. The first phase focused on understanding and summarizing NASA’s ethical approaches, mission and values based on published documentation, discussions and individual insights & opinions of participants. This information was prioritized, weighted, ordered, and quantified in phase two, to formulate an alignment between human values (ethics) and their applicability to AI systems during all lifecycle phases. The first two phases were summarized as a form of ethical genealogy for artificial intelligence, specific to NASA’s ethical approaches. In the third and last phase of the workshop the participants evaluated notional examples of artificial intelligence to qualify and quantify its ability to adhere to the organizational ethics approaches, using the Ethical AI Certification and Maturity Model™. The workshop uses the concept of genealogy, in the traditional sense: the study and traceability of lines of ancestors in the process of evolutionary development from earlier forms. However, as it is applied to an Ethical AI definition, it is providing the insights to the necessary and mandatory traceability of content, data, metrics, telemetry, elements, and structures which are used in the AI’s lifecycle to foster and measure AI ethics in all steps of its lifecycle. The Ethical Artificial Intelligence Quantification Workshop provided NASA with the opportunity to apply the Ethical AI Certification and Maturity Model™, in combination with existing and well-known decision-making and quality control methods to identify the metrics and measurements for an Ethical AI and assess its ethical condition and quality aligned with NASA ethics approaches. The result of the workshop is the capacity for NASA to apply the maturity model assessment to its AI Systems as desired and if necessary, publish the ability of these AI Systems to adhere to the organizational ethical goals. AI ethics frameworks need to be customized for each application domain, for example, individual NASA Mission Directorates. General principles that work in one area such as AI/Machine Learning-based text analysis (the ethics of information-extraction) may need to be adapted for another such as sense-and-avoid decision-making in a flight environment. The workshop was conducted among approximately twenty NASA subject matter experts, so the elements noted above should be considered examples, not definitive NASA ethical AI principles, genealogy, etc. Generating a definitive AI ethics framework for an organization as diverse as NASA would require far more discussion, debate, review, etc. However, the workshop provided valuable insight into mechanisms and processes for quantifying AI ethical qualities.

Artificial Intelligence↗

Transferring Knowledge from Observations and Models to Decision Makers: An Overview and Challenges

Over the last 25 years, a tremendous progress has been made in the Earth science space-based remote sensing observations, technologies and algorithms. Such advancements have improved the predictability by providing lead-time and accuracy of forecast in weather, climate, natural hazards, and natural resources. It has further reduced or bounded the overall uncertainties by partially improving our understanding of planet Earth as an integrated system that is governed by non-linear and chaotic behavior. Many countries such US, European Community, Japan, China and others have invested billions of dollars in developing and launching space-based assets in the low earth (LEO) and geostationary (GEO) orbits. However, the wealth of this scientific knowledge that has potential of extracting monumental socio-economic benefits from such large investments have been slow in reaching to public and decision makers. For instance, there are a number of areas such as energy forecasting, aviation safety, agricultural competitiveness, disaster management, security, air quality and public health can directly take advantage. Nevertheless, we all live in a global economy that depends on access to the best available Earth Science information for all inhabitants of this planet. This paper surveys and examines a number such applications in terms of their architecture, maturity and economic applicability as they apply to the societal needs. A detailed analysis is also presented of various challenges and issues that pertain to a number of areas such as: (1) difficulties in making a speedy transition of data and information from observations and models to relevant Decision Support Systems (DSS) or tools, (2) data and models inter-operability issues, (3) limitations of spatial, spectral and temporal resolution, (4) communication limitations as dictated by the availability of image processing and data compression techniques. Additionally, the most critical element amongst all is the organizational and management boundaries that must be resolved at local, state, national and international levels to implement and realize free flow of such vital information. This paper also makes attempts to address this topic and discuss possible approaches to deal with this quandary.

Habib, Shahid↗

Organizing, staffing and initiating the program.

This paper describes and discusses those test laboratory management considerations prerequisite to test conduct and test reporting. These considerations are identified as (1) organizational requirements; they have to be determined, not only to implement a successful test program, but to match the communications needs of the other elements of the parent organization: (2) the role of supervision; decision making processes, types of decisions, and charters and role statements are presented: (3) planning; involves establishing objectives, definitions, procedures and the review and appraisal process: (4) delegation; some things cannot be delegated, others have to be.

Van Biene, F. K.↗

NASA Earth Systems Digital Twins (ESDT)

"Similarly to artificial intelligence, which is now revolutionizing many aspects of our daily lives, Earth system digital twin technologies have the potential to revolutionize the way Earth Science research will be conducted in the future, and how results and knowledge from this research will provide information to support decision making and yield impactful societal benefits. An Earth System Digital Twin or ESDT is a dynamic and interactive information system that first provides a digital replica of the past and current states of the Earth or Earth system as accurately and timely as possible; second, allows for computing forecasts of future states under nominal assumptions and based on the current replica; and third, offers the capability to investigate many hypothetical scenarios under varying impact assumptions. In other words, an ESDT provides the integrated What-Now, What-Next, and What-If pictures of the Earth or Earth system, by continuously ingesting newly observed data and by leveraging multiple interconnected models, machine learning as well advanced computing and visualization capabilities. Digital twins have been developed in engineering since 2002, but the interest in digital twins for the Earth domain is more recent and stems from the convergence of several developments: - The huge amount of diverse data that has now been collected continuously for more than 50 years, and that is becoming more and more difficult to access, understand, and utilize. - At the same time, because of climate change and its impacts the information produced by all of this data is becoming of interest to many new non-traditional users for analyzing and predicting various phenomena. - Because of advances in computational and visualization capabilities and the parallel unprecedented development of machine learning (ML), extracting relevant information from these large amounts of data and running complex models faster has become possible. As a result, it is becoming necessary and possible to build intuitive and interactive frameworks that will enable users with various skill levels and/or organizational hierarchy levels to easily access large amounts of targeted information along with the relevant tools and models (Earth system and human activity models), to support them in analyzing and visualizing this information, to help them understand interactions among models, to visualize the potential outcomes of various impacts, and to support decision or policy making. The full power of digital twins is that, through an integrated representation and standardized tools and software technologies, the same digital replica can address the needs of multiple users at various resolutions (spatial and temporal) and for various applications (science, economic, policy, etc.) – “from farmer to scientist”. With all these interests at stake, the challenges of building optimal digital twins are many and complex. The first challenge is to determine if a Digital Twin should be global or local, and multi-domain or thematic. For example, some domains such as Climate or Weather will require a global Digital Twin or Digital Twin capabilities while science areas such as Biodiversity might be more local. We can also envision that multiple thematic ESDTs, e.g., Air Quality, Wildfires, Hydrology could be federated or provide input to other ESDTs, either on a regional level or to a more global ESDT. Overall, we can imagine a future “web” of Digital Twins co-existing in a hierarchy or in a network, and capable of being connected or federated depending on the needs. This last point brings up the very important challenge of interoperability, including standards and protocols that will need to be built into these systems from the beginning. Each individual digital twin would have full flexibility in internal construction but would need standards-based interfaces (input and output) or hooks to make it compatible with others. Another challenge when building digital twins will be to decide how to organize each digital replica. Based on the applications targeted by the DT under implementation, various amounts and types of raw data, Analysis Ready Data (ARD) and information will need to be incorporated. Depending on the required latencies and needs of the users, various solutions can be considered, including Data Cubes, Data Lakes, pointers, or computing information on demand. We envision that each ESDT will choose a solution adapted to its specific objectives. Another important challenge is the type(s) of visualization that will be used, as well as the level of interactivity and refresh rate that will be required. Again, this will depend on the objectives of the ESDT, but also on the various users’ needs. In most cases, several types of visualizations and human interfaces will need to be offered depending on the projected users of that system. In parallel to the challenges highlighted above, there are also many tools and technologies that will need to be developed or improved for all types of digital twins. Among those are improved machine learning technologies, for example providing explainability, but also ML techniques for causality and providing a better integration of physics models. Additionally, reliable uncertainty quantification methods will be needed for all ESDT components, from validating data fusion and assimilation to assessing the accuracy of ML models and weighing the values of decisions supported by those systems. This presentation introduces the ESDT concept, presents several ESDT use cases, and a proposed ESDT architecture framework, as well as various technologies being developed by the Advanced Information Systems Technology (AIST) Program."

Earth Science Remote Sensing; Information Systems↗

Space Environments and Spacecraft Effects Concept: Transitioning Research to Operations and Applications

The National Aeronautics and Space Administration (NASA) is embarking on a course to expand human presence beyond Low Earth Orbit (LEO) while expanding its mission to explore the solar system. Destinations such as Near Earth Asteroids (NEA), Mars and its moons, and the outer planets are but a few of the mission targets. NASA has established numerous organizations specializing in specific space environments disciplines that will serve to enable these missions. To complement these existing discipline organizations, a concept is presented focusing on the development of a space environment and spacecraft effects organization. This includes space climate, space weather, natural and induced space environments, and effects on spacecraft materials and systems. This space environment and spacecraft effects organization would be comprised of Technical Working Groups (TWG) focusing on, for example: a) Charged Particles (CP), b) Space Environmental Effects (SEE), and c) Interplanetary and Extraterrestrial Environments (IEE). These technical working groups will generate products and provide knowledge supporting four functional areas: design environments, environment effects, operational support, and programmatic support. The four functional areas align with phases in the program mission lifecycle and are briefly described below. Design environments are used primarily in the mission concept and design phases of a program. Environment effects focuses on the material, component, sub-system and system-level selection and the testing to verify design and operational performance. Operational support provides products based on real time or near real time space weather observations to mission operators to aid in real time and near-term decision-making. The programmatic support function maintains an interface with the numerous programs within NASA and other federal agencies to ensure that communications are well established and the needs of the programs are being met. The programmatic support function also includes working in coordination with the program in anomaly resolution and generation of lesson learned documentation. The goal of this space environment and spacecraft effects organization is to develop decision-making tools and engineering products to support the mission phases of mission concept through operations by focusing on transitioning research to application. Products generated by this space environments and spacecraft effects organization are suitable for use in anomaly investigations. This paper will describe the organizational structure for this space environments and spacecraft effects organization, and outline the scope of conceptual TWG's and their relationship to the functional areas.

Edwards, D. L.↗

Anatomy of an organizational change effort at the Lewis Research Center

By 1979, after a long decline following the end of the Apollo program, the Lewis Research Center found its very existence endangered because it was not doing the kind of research that could attract funding at the time. New management under Andrew J. Stofan applied a program of strategic planning, participative management, and consensus decision making. A corporate-cultural change was effected which enabled Lewis to commit itself to four fundable research and development projects. Morale-building and training programs which were essential to this change are described.

Hawker, James R.↗

Space Environments and Effects Concept: Transitioning Research to Operations and Applications

The National Aeronautics and Space Administration (NASA) is embarking on a course to expand human presence beyond Low Earth Orbit (LEO) while expanding its mission to explore the solar system. Destinations such as Near Earth Asteroids (NEA), Mars and its moons, and the outer planets are but a few of the mission targets. NASA has established numerous offices specializing in specific space environments disciplines that will serve to enable these missions. To complement these existing discipline offices, a concept focusing on the development of space environment and effects application is presented. This includes space climate, space weather, and natural and induced space environments. This space environment and effects application is composed of 4 topic areas; characterization and modeling, engineering effects, prediction and operation, and mitigation and avoidance. These topic areas are briefly described below. Characterization and modeling of space environments will primarily focus on utilization during Program mission concept, planning, and design phases. Engineering effects includes materials testing and flight experiments producing data to be used in mission planning and design phases. Prediction and operation pulls data from existing sources into decision-making tools and empirical data sets to be used during the operational phase of a mission. Mitigation and avoidance will develop techniques and strategies used in the design and operations phases of the mission. The goal of this space environment and effects application is to develop decision-making tools and engineering products to support the mission phases of mission concept through operations by focusing on transitioning research to operations. Products generated by this space environments and effects application are suitable for use in anomaly investigations. This paper will outline the four topic areas, describe the need, and discuss an organizational structure for this space environments and effects application.

Edwards, David L.↗

Cracking the Code: An Integrated Approach to Meeting Client Needs at NASA’s Langley Research Center

Organizational development (OD) is an interdisciplinary field that encourages the integration of many different concepts and theories in meeting client requirements. However, OD practitioners often maintain a significant amount of knowledge and lack specific guidance on how to combine concepts into a unified approach to drive change, thus making the selection and integration of disparate approaches challenging. To assist OD practitioners in navigating these challenges, a theory synthesizing approach is used to demonstrate the incremental insights that can be gained through the innovative integration of concepts, rather than being restricted to one perspective. More specifically, a novel Integrated Approach Decision-Making Tool is presented that offers specific steps to take to reach an effective, innovative solution. To further illustrate how this tool can be used in practice, an example is provided that details how this tool was used to achieve project success at NASA’s Langley Research Center through integrating three common OD models. Taken together, this paper will inform change agents operating in a similar capacity at any organization by providing a roadmap for how to meet client requirements and achieve increased project success through creating integrated, novel OD interventions.

Interdisciplinary↗