Objective Structured Clinical Evaluation (OSCE) of an Artificial Intelligence (AI) Clinical Decision Support System (CDSS) Tool
Explore the source record for details and available documents.
Ariana Nelson↗
SEARCH · Search NASA
Results for “decision tools”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
Distributionally Robust Decision Making Leveraging Conditional Distributions
Distributionally robust optimization (DRO) is a powerful tool for decision making under uncertainty. It is particularly appealing because of its ability to leverage existing data. However, many practical problems call for decision-making with some auxiliary information, and DRO in the context of conditional distributions is not straightforward. We propose a conditional kernel distributionally robust optimization (CKDRO) method that enables robust decision making under conditional distributions through kernel DRO and the conditional mean operator in the reproducing kernel Hilbert space (RKHS). In particular, we consider problems where there is a correlation between the unknown variable y and an auxiliary observable variable x. Given past data of the two variables and a queried auxiliary variable, CKDRO represents the conditional distribution P(y|x) as the conditional mean operator in the RKHS space and quantifies the ambiguity set in the RKHS as well, which depends on the size of the dataset as well as the query point. To justify the use of RKHS, we demonstrate that the ambiguity set defined in RKHS can be viewed as a ball under a metric that is similar to the Wasserstein metric. The DRO is then dualized and solved via a finite dimensional convex program. The proposed CKDRO approach is applied to a generation scheduling problem and shows that the result of CKDRO is superior to common benchmarks in terms of quality and robustness.
Chen, Yuxiao↗
Advanced Technology Lifecycle Analysis System (ATLAS) Technology Tool Box (TTB)
The Advanced Technology Lifecycle Analysis System (ATLAS) is a decision support tool designed to aid program managers and strategic planners in determining how to invest technology research and development dollars. It is an Excel-based modeling package that allows a user to build complex space architectures and evaluate the impact of various technology choices. ATLAS contains system models, cost and operations models, a campaign timeline and a centralized technology database. Technology data for all system models is drawn from a common database, the ATLAS Technology Tool Box (TTB). The TTB provides a comprehensive, architecture-independent technology database that is keyed to current and future timeframes.
Doyle, Monica↗
Evaluation of Usability and Workload with Paper Strips as Compared to Virtual Flight Strips Used for Ramp Operations
This paper describes an experiment designed to compare the use of paper strips with the use of a new user interface, the Ramp Traffic Console (RTC), designed for use by ramp controllers to be used in place of paper strips. A Human-In-the-Loop (HITL) experiment was performed as the fifth study in a series of six HITL simulation experiments designed to evaluate a concept that provided advisories to the users. The RTC was designed to be used as a Decision Support Tool (DST) that provided advisories to ramp controllers regarding metering or pushback such that most of the delay was taken at the gate to save fuel and emissions. In addition to being a DST, an added benefit of the RTC is that it can provide real-time updates of flight data, airport and airspace status to the controller including Traffic Management Initiatives (TMI). The RTC was designed as new user interface that displays virtual strips on a terminal map drawn on a 27-inch touch screen monitor. The RTC was used in some conditions of the experiment by ramp controllers in place of paper strips and paper maps in the HITL environment. In other conditions the controllers were given paper strips and paper maps similar to what they currently use at Charlotte Douglas International Airport (CLT). The study described here, evaluated the use of the virtual strips displayed on the RTC as compared to the use of paper strips and paper map, using current ramp tower controllers at CLT as participants. The research question being asked was - How does management of ramp traffic affect user workload and usability ratings while using RTC to manage traffic in the ramp verses using paper strips? Workload for our purposes is defined by four components of the NASA-TLX (Task Load Index). Usability was assessed with two sets of usability questions - One set of usability questions addressed traffic management performance and the other set addressed issues of resources and efficiency. Both Post Run and Post Study questionnaire responses were gathered and the results were analyzed to assess controller workload and usability ratings under both conditions, virtual strips shown on RTC and Paper Strips. The results indicate that controllers perceived lower workload while using virtual strips displayed on RTC to manage ramp traffic. Usability ratings for Traffic management performance questions are lower in the virtual strip/RTC condition than in the paper strip condition showing a preference for RTC over Paper. Usability ratings for Resources and efficiency questions show mixed results. Additionally, the Post Study Questions show preference for RTC over paper strips. Results of this data analysis will be presented in this paper. This DST evaluation was an important step in researching and improving the tool, which was planned to be deployed in the field.
Aviation Decision Support Tools↗
NASA-Modified Precipitation Products to Improve EPA Nonpoint Source Water Quality Modeling for the Chesapeake Bay
The Environmental Protection Agency (EPA) has estimated that over 20,000 water bodies within the United States do not meet water quality standards. Ninety percent of the impairments are typically caused by nonpoint sources. One of the regulations in the Clean Water Act of 1972 requires States to monitor the Total Maximum Daily Load (TMDL), or the amount of pollution that can be carried by a water body before it is determined to be "polluted", for any watershed in the U.S.. In response to this mandate, the EPA developed Better Assessment Science Integrating Nonpoint Sources (BASINS) as a Decision Support Tool (DST) for assessing pollution and to guide the decision making process for improving water quality. One of the models in BASINS, the Hydrological Simulation Program -- Fortran (HSPF), computes daily stream flow rates and pollutant concentration at each basin outlet. By design, precipitation and other meteorological data from weather stations serve as standard model input. In practice, these stations may be unable to capture the spatial heterogeneity of precipitation events especially if they are few and far between. An attempt was made to resolve this issue by substituting station data with NASA modified/NOAA precipitation data. Using these data within HSPF, stream flow was calculated for seven watersheds in the Chesapeake Bay Basin during low flow periods, convective storm periods, and annual flows. In almost every case, the modeling performance of HSPF increased when using the NASA-modified precipitation data, resulting in better stream flow statistics and, ultimately, in improved water quality assessment.
Nigro, Joseph↗
High Value Opportunities to Advance Automation in Electric Grid Control Rooms
Grid operators and electric utilities are increasingly driven to develop automated processes and introduce decision support tools to reduce cognitive demand on System Operations staff and aid in the operational decision-making process. However, System Operation staff are traditionally hesitant to implement and accept new solutions, automation processes, and tools within the control room. This paper explores high-value opportunities to advance automation in electric grid control rooms for the purpose of improving grid safety, reliability, and resiliency during normal and emergency operating conditions.
24 POWER TRANSMISSION AND DISTRIBUTION↗
Development of the Surface Management System Integrated with CTAS Arrival Tools
The Surface Management System (SMS) developed by NASA Ames Research Center in coordination with the Federal Aviation Administration (FAA) is a decision support tool to help tower traffic coordinators and Ground/Local controllers in managing and controlling airport surface traffic in order to increase capacity, efficiency, and flexibility. SMS provides common situation awareness to personnel at various air traffic control facilities such as airport traffic control towers (ATCT s), airline ramp towers, Terminal Radar Approach Control (TRACON), and Air Route Traffic Control Center (ARTCC). SMS also provides a traffic management tool to assist ATCT traffic management coordinators (TMCs) in making decisions such as airport configuration and runway load balancing. The Build 1 of the SMS tool was installed and successfully tested at Memphis International Airport (MEM) and received high acceptance scores from ATCT controllers and coordinators, as well as airline ramp controllers. NASA Ames Research Center continues to develop SMS under NASA s Strategic Airspace Usage (SAU) project in order to improve its prediction accuracy and robustness under various modeling uncertainties. This paper reports the recent development effort performed by the NASA Ames Research Center: 1) integration of Center TRACON Automation System (CTAS) capability with SMS and 2) an alternative approach to obtain airline gate information through a publicly available website. The preliminary analysis results performed on the air/surface traffic data at the DFW airport have shown significant improvement in predicting airport arrival demand and IN time at the gate. This paper concludes with recommendations for future research and development.
Jung, Yoon C.↗
SARDA: An Integrated Concept for Airport Surface Operations Management
The Spot and Runway Departure Advisor (SARDA) is an integrated decision support tool for airlines and air traffic control tower enabling surface collaborative decision making (CDM) and departure metering in order to enhance efficiency of surface operations at congested airports. The presentation describes the concept and architecture of the SARDA as a CDM tool, and the results from a human-in-the-loop simulation of the tool conducted in 2012 at the FutureFlight Central, the tower simulation facility. Also, presented is the current activities and future plan for SARDA development. The presentation was given at the meeting with the FAA senior advisor of the Surface Operations Office.
surface collaborative decision making↗
A Typology of Decision-Making Tasks for Visualization
Despite decision-making being a vital goal of data visualization, little work has been done to differentiate decision-making tasks within the field. While visualization task taxonomies and typologies exist, they often focus on more granular analytical tasks that are too low-level to describe large complex decisions, which can make it difficult to reason about and design decision-support tools. In this paper, we contribute a typology of decision-making tasks that were iteratively refined from a list of design goals distilled from a literature review. Our typology is concise and consists of only three tasks: CHOOSE, ACTIVATE, and CREATE. Although decision types originating in other disciplines exist, we provide definitions for these tasks that are suitable for the visualization community. Our proposed typology offers two benefits. First, the ability to compose and hierarchically organize the tasks enables flexible and clear descriptions of decisions with varying levels of complexities. Second, the typology encourages productive discourse between visualization designers and domain experts by abstracting the intricacies of data, thereby promoting clarity and rigorous analysis of decision-making processes. We demonstrate the benefits of our typology through four case studies, and present an evaluation of the typology from semi-structured interviews with experienced members of the visualization community who have contributed to developing or publishing decision support systems for domain experts. Our interviewees used our typology to delineate the decision-making processes supported by their systems, demonstrating its descriptive capacity and effectiveness. Finally, we present preliminary findings on the usefulness of our typology for visualization design.
97 MATHEMATICS AND COMPUTING↗
An Integrated Tool Suite for En Route Radar Controllers in NextGen
This paper describes recent human-in-the-loop research in the Airspace Operations Laboratory at the NASA Ames Research Center focusing on en route air traffic management with advanced trajectory planning tools and increased levels of human-automation cooperation. The decision support tools were exercised in a simulation of seven contiguous high-altitude sectors. Preliminary data suggests the controllers were able to manage higher amounts of traffic as compared to today, while maintaining acceptable levels of workload.
Mercer, Joey↗
Analysis of ETMS Data Quality for Traffic Flow Management Decisions
The data needed for air traffic flow management decision support tools is provided by the Enhanced Traffic Management System (ETMS). This includes both the tools that are in current use and the ones being developed for future deployment. Since the quality of decision support provided by all these tools will be influenced by the quality of the input ETMS data, an assessment of ETMS data quality is needed. Motivated by this desire, ETMS data quality is examined in this paper in terms of the unavailability of flight plans, deviation from the filed flight plans, departure delays, altitude errors and track data drops. Although many of these data quality issues are not new, little is known about their extent. A goal of this paper is to document the magnitude of data quality issues supported by numerical analysis of ETMS data. Guided by this goal, ETMS data for a 24-hour period were processed to determine the number of aircraft with missing flight plan messages at any given instant of time. Results are presented for aircraft above 18,000 feet altitude and also at all altitudes. Since deviation from filed flight plan is also a major cause of trajectory-modeling errors, statistics of deviations are presented. Errors in proposed departure times and ETMS-generated vertical profiles are also shown. A method for conditioning the vertical profiles for improving demand prediction accuracy is described. Graphs of actual sector counts obtained using these vertical profiles are compared with those obtained using the Host data for sectors in the Fort Worth Center to demonstrate the benefit of preprocessing. Finally, results are presented to quantify the extent of data drops. A method for propagating track positions during ETMS data drops is also described.
Chatterji, Gano B.↗
Market Survey 2020: Commercial Clinical Decision Support Systems and Wellness Tools
For long-duration, deep space exploration missions, current methods for managing and supporting crew health and medical conditions will be unsuitable. Communication and data transmission lags will necessitate the use of a sophisticated clinical decision support system (CDSS) that will tailor diagnosis and treatment guidance that is context-sensitive for anticipated astronaut health, wellness, and medical conditions. A variety of clinical decision support (CDS) and wellness tools (WT) are currently available in the commercial market and a broad-brush survey of this market can provide an initial impression of the current state of the art which, in turn, can inform the roadmap of NASA deep space CDSS development and associated requirements. Such a survey was undertaken during the first six months of 2020 using directed convenience sampling to obtain information provided by vendors on their websites; both commercially available CDS and WT (such as those used to track and monitor nutrition, exercise, and sleep) were included. Areas assessed were item type (e.g., software/application, device); primary purpose of the item (e.g., diagnostic support, nutrition tracking); additional purposes (if any); reported features, capabilities, and functionality; setting of use (e.g., inpatient, outpatient); intended user (e.g., clinician, patient); location and sources of data/information used or produced by the item; integration with patient electronic health record (EHR); compliance with interoperability ontologies and standards (e.g., Health Level 7 [HL7], Systematized Nomenclature of Medicine – Clinical Terminology [SNOMED-CT]); and whether the item is knowledge-based (derived from research findings) or non-knowledge-based (derived through artificial intelligence, machine learning, advanced probability and statistics), among others. Ninety-seven (97) vendor websites describing 196 CDS and 73 WT (269 total) were reviewed and coded. The primary purpose of the majority of CDS reviewed is diagnosis or diagnosis/treatment/drug decision support—targeted for clinician use— and the primary purpose of the majority of WT reviewed is the monitoring of different health metrics, most often through the use of a biosensor device (e.g., blood pressure)—targeted for patient use. Very few CDS or WT appear to comply with major international interoperability standards or can be integrated with a patient’s EHR data. None consider contextual factors, such as conditions of the physical environment (e.g., CO2 levels). The majority of CDS and WT reviewed are non-knowledge, cloud- or web-based applications or software. Forty-three (43) major findings were identified and the implications those findings have for NASA will be discussed. Example major findings include: CDS-WT capabilities range from diagnosis to treatment applications, CDS-WT may be wearable or non-wearable and are technologically advanced and only a few CDS-WT tools referenced compliance to ensure interoperability, among other findings. Recommendations will also be offered that will help to address ExMC Gap, Medical-701: Enhance medical capabilities within an exploration medical system.
market survey↗
Real-Time Science Operations to Support a Lunar Polar Volatiles Rover Mission
Future human exploration of the Moon will likely rely on in situ resource utilization (ISRU) to enable long duration lunar missions. Prior to utilizing ISRU on the Moon, the natural resources (in this case lunar volatiles) must be identified and characterized, and ISRU demonstrated on the lunar surface. To enable future uses of ISRU, NASA and the CSA are developing a lunar rover payload that can (1) locate near subsurface volatiles, (2) excavate and analyze samples of the volatile-bearing regolith, and (3) demonstrate the form, extractability and usefulness of the materials. Such investigations are important both for ISRU purposes and for understanding the scientific nature of these intriguing lunar volatile deposits. Temperature models and orbital data suggest near surface volatile concentrations may exist at briefly lit lunar polar locations outside persistently shadowed regions. A lunar rover could be remotely operated at some of these locations for the approx. 2-14 days of expected sunlight at relatively low cost. Due to the limited operational time available, both science and rover operations decisions must be made in real time, requiring immediate situational awareness, data analysis, and decision support tools. Given these constraints, such a mission requires a new concept of operations. In this paper we outline the results and lessons learned from an analog field campaign in July 2012 which tested operations for a lunar polar rover concept. A rover was operated in the analog environment of Hawaii by an off-site Flight Control Center, a rover navigation center in Canada, a Science Backroom at NASA Ames Research Center in California, and support teams at NASA Johnson Space Center in Texas and NASA Kennedy Space Center in Florida. We find that this type of mission requires highly efficient, real time, remotely operated rover operations to enable low cost, scientifically relevant exploration of the distribution and nature of lunar polar volatiles. The field demonstration illustrated the need for science operations personnel in constant communications with the flight mission operators and the Science Backroom to provide immediate and continual science support and validation throughout the mission. Specific data analysis tools are also required to enable immediate data monitoring, visualization, and decision making. The field campaign demonstrated that this novel methodology of real-time science operations is possible and applicable to providing important new insights regarding lunar polar volatiles for both science and exploration.
Moon; Volatiles; Rover; Missions↗
Impact Real World System Validation
Introduction NASA has developed a new evidence-based data-driven probabilistic risk assessment and tradespace analysis tool as a successor to the Integrated Medical Model. This updated decision support tool is known as IMPACT (Informing Mission Planning via Analysis of Complex Tradespaces). IMPACT estimates the frequency and consequences of medical conditions that might arise during exploration missions. A validation analysis of IMPACT was performed with respect to a set of International Space Station (ISS) and Shuttle Transportation System (STS) real world system (RWS) referent data due to the limited referent data available from exploration missions. Methods Observed mission and crew characteristics from STS and ISS missions were used as model inputs within MEDPRAT (Medical Extensible Dynamic Probabilistic Risk Assessment Tool). For each mission, two hundred thousand simulations were generated. For each mission, model outputs included occurrence counts for each condition, total medical events (TME), and the probability of loss of crew life (LOCL). These simulated model outputs were compared to the RWS referent data. Results The predicted number of total medical events exceeded the total RWS medical events for ISS missions and combined ISS and STS missions and fell within the 90% confidence interval for STS missions. For the 32 ISS missions simulated by IMPACT, the number of total medical events was overpredicted for 19 missions and fell within the 90% confidence interval for 13 missions. For the 21 STS missions, the total number of medical events was overpredicted for 3 missions, fell within the 90% confidence interval for 16 missions, and was underpredicted for 2 missions. Combined, 29 missions were in range, 22 were overpredicted, and 2 were underpredicted. The predicted LOCL probability for the 32 ISS missions, the 21 STS missions, and the combined ISS and STS missions was consistent with the zero LOCL events observed in the RWS referent data. The validation analysis included a comparison of the number of medical events predicted by IMPACT and the number of medical events observed in the RWS data on a condition-by-condition basis. For ISS missions, 50 conditions were in range, 52 conditions were statistically underpowered (not enough observed sample to draw any conclusions on precision), 8 conditions were overpredicted, and 9 conditions were underpredicted. Overall, only 14% (17/119) of conditions were out of range for STS missions, 40 conditions were in range, 59 conditions were statistically underpowered, 10 conditions were overpredicted, and 10 conditions were underpredicted. Overall, only 17% (20/119) of conditions were out of range. For combined ISS and STS missions, 11 conditions were overpredicted, and 11 conditions were underpredicted. Overall, only 18% (22/119) of conditions were out of range. For combined ISS and STS missions, 49 conditions were in range, 46 conditions were statistically underpowered, 18 conditions were overpredicted, and 8 conditions were underpredicted. Overall, 21% (26/121) of conditions were out of range. Conclusion The results of this validation analysis should not be interpreted as a pass/fail test of the validity of IMPACT. Instead, this validation analysis should be used to assess some of the IMPACT outcomes in terms of consistencies and inconsistencies with the ISS and STS RWS referent data.
L. Boley↗
Flight Awareness Collaboration Tool Development
This is a PowerPoint presentation covering airline operations center (AOC) research. It reviews a dispatcher decision support tool called the Flight Awareness Collaboration Tool (FACT). FACT gathers information about winter weather onto one screen and includes predictive abilities. FACT should prove to be useful for airline dispatchers and airport personnel when they manage winter storms and their effect on air traffic. This material is very similar to other previously approved presentations.
airlines↗
Baseline Medical System Translation for the Impact Medical Database
NASA has developed a new evidence-based data-driven probabilistic risk assessment and tradespace analysis tool as a successor to the Integrated Medical Model (IMM). This updated decision support tool is known as IMPACT (Informing Mission Planning via Analysis of Complex Tradespaces). Whereas IMM focuses on the resources and risks associated with International Space Station (ISS) and Low Earth Orbit (LEO) missions, IMPACT estimates the frequency and consequences of medical conditions that might arise during exploration missions. One of the services offered by IMM is a series of generic or baseline medical systems associated with typical mission types or DRMs (Design Reference Missions), such that requestors may prioritize questions pertaining to the mission itself over the medical supplies indicated by the model outputs for that DRM. In a mission focused request, the appropriate baseline medical system is used in place of a prepared or optimized medical kit. In preparation, an effort was undertaken to create baseline systems of medical resources within IMPACT suitable for typical DRMs. Using an existing baseline medical system within IMM’s Integrated Medical Evidence Database (iMED) as a starting point, the resources found within the medical kit were compared to and substituted for equivalent resources available within the IMPACT MD (Medical Database). In consultation with clinicians with knowledge of IMPACT’s MD, each resource was matched as closely as possible to a similarly purposed resource in IMPACT, seeking to preserve the treatment capabilities and procedures offered by the IMM medical system while reconciling the differences in modeled resources and medical conditions between the models. To demonstrate the efficacy of this work, a prototype ISS medical system in IMPACT was translated from the baseline used in the IMM. The appropriate medical system was then run through its associated medical model to compare and validate the resultant risks and risk mitigation provided by each baseline ISS medical kit.
S Schwartz↗
User's guide to the Mission Analysis Evaluation and Space Trajectory Operations program (MAESTRO)
The MAESTRO system is a mission analysis tool designed to present to the user information necessary to make the various decisions required in the design and execution of a spaceflight mission. The system was designed so that it can be used in both the pre-launch mission planning phase of a mission and during the flight as an in-flight decision making tool. A description of each of the following modes is presented: (1) trajectory propagation mode; (2) retro-fire determination mode; (3) midcourse analysis determination mode; (4) Monte Carlo mode; (5) verification mode; (6) orbit stability mode; and (7) post injection trim mode. A description of the inputs necessary to run the program mode is given along with a sample case.
Lutzky, D.↗
Verification and Validation of the Stakeholder Tool for Assessing Radioactive Transportation (START) – 22323
The U.S. Department of Energy (DOE) Office of Integrated Waste Management is planning for the eventual transportation, storage, and disposal of spent nuclear fuel (SNF) and high-level radioactive waste (HLW) from nuclear power plant and DOE sites. The Stakeholder Tool for Assessing Radioactive Transportation (START) is a web-based, geospatial decision-support tool developed for evaluating routing options and other aspects of transporting SNF and HLW, covering rail, truck, barge, and intermodal infrastructure and operations in the continental United States. The verification and validation (V&V) process is intended to independently assess START to provide confidence in the ability of START to accurately provide intended results. The V&V process checks the START tool using a variety of methods, ranging from independent hand calculations to comparison of START performance and results to those of other codes. The V&V activity was conducted independently from the START development team with opportunities to provide feedback and collaborate throughout the process. The V&V analyzed attributes of transportation routes produced by START, including route distance and both population and population density captured within buffer zones around routes. Population in the buffer zone, population density in the buffer zone, and route distance were all identified as crucial outputs of the START code and were subject to V&V tasks. Some of the improvements identified through the V&V process were standardizing the underlying population data in START, changing the projection of the population raster data, and changes to the methodology used for population density to improve its applicability for expected users. This collaboration also led to suggested improvements to some of the underlying shape file segments within START. A formal series of test routes went through the V&V process from START version 3.2.1 following the implementation of the recommended improvements to START. The V&V of the buffer zone population and distance reported in START were achieved by utilizing a custom workflow developed in QGIS. This platform was chosen because it is fully independent of START. The buffer zone population and route distances in START showed excellent agreement to independent V&V test results. Over 200 route test cases were run; in all cases, the percent difference in the population within the buffer zone was less than +/- 5%, and the majority of cases were below +/- 1%. These cases were also duplicated in ArcMap, which is Environmental Systems Research Institute’s (ESRI) GIS desktop application. The ArcMap results also show good agreement; however, they cannot be considered independent of START because START also uses ESRI software. The V&V tasks are currently addressing population density and estimated radiation dose in START as well as implementing further automation of the V&V process to allow larger suites of test routes to be run more easily and quickly by less experienced GIS users. These improvements to the V&V process will make it a valuable tool for the development team and the stakeholders so that each new iteration of START can easily be checked for quality and consistency. This paper and presentation will describe the V&V process, findings and implications, and ongoing V&V activities.
START, DOE, Transporation, dose, GIS↗