Predicting Cognitive States Using Machine Learning Fusion Paradigms to Reduce Model Uncertainty
Explore the source record for details and available documents.
SEARCH · Search NASA
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
Explore the source record for details and available documents.
Abstract In this paper, we describe specific deployments of the Smart Connected Worker (SCW) Edge Platform for Smart Manufacturing through implementation of four instructive real‐world use cases that illustrate the role of people in a Smart Manufacturing paradigm through which affordable, scalable, accessible, and portable (ASAP) information technology (IT) acquires and contextualizes data into information for transmission to operation technologies (OT). For case one, the platform captures the relationships between energy consumption and human workflows for improved energy productivity while workers interact with machines during semiconductor manufacturing. The platform utilizes human cognition to identify anomalous machine behavior for root cause analysis of system faults via neural network (NN) that recognize alarm postures of workers with cameras. For case two, a smart assembly line is demonstrated for state monitoring and fault detection. Machine learning (ML) models are used to recognize system states and identify fault scenarios with human intervention. For case three, the platform monitors human–machine interactions to classify manufacturing machine states for proper operations and energy productivity. Internal energy states of individual or collections of manufacturing equipment are determined via NN based algorithms that disaggregate signals associated with smart metering typically deployed at manufacturing facilities. These methods predict the real time energy profile of each machine from the total energy profile of a manufacturing site. For case four, a software defined sensor system built with scientific workflow engines is demonstrated for contextualizing data from laser surface refraction for characterization, and diagnostics in the processing of additively manufactured titanium alloy.
The major focus of the present proposal was to examine psychophysiological variables related to hazardous states of awareness induced by monitoring automated systems. With the increased use of automation in today's work environment, people's roles in the work place are being redefined from that of active participant to one of passive monitor. Although the introduction of automated systems has a number of benefits, there are also a number of disadvantages regarding worker performance. Byrne and Parasuraman have argued for the use of psychophysiological measures in the development and the implementation of adaptive automation. While both performance based and model based adaptive automation have been studied, the use of psychophysiological measures, especially EEG, offers the advantage of real time evaluation of the state of the subject. The current study used the closed-loop system, developed at NASA-Langley Research Center, to control the state of awareness of subjects while they performed a cognitive vigilance task. Previous research in our laboratory, supported by NASA, has demonstrated that, in an adaptive automation, closed-loop environment, subjects perform a tracking task better under a negative than a positive, feedback condition. In addition, this condition produces less subjective workload and larger P300 event related potentials to auditory stimuli presented in a concurrent oddball task. We have also recently shown that the closed-loop system used to control the level of automation in a tracking task can also be used to control the event rate of stimuli in a vigilance monitoring task. By changing the event rate based on the subject's index of arousal, we have been able to produce improved monitoring, relative to various control groups. We have demonstrated in our initial closed-loop experiments with the the vigilance paradigm that using a negative feedback contingency (i.e. increasing event rates when the EEG index is low and decreasing event rates when the EEG index is high) results in a marked decrease of the vigilance decrement over a 40 minute session. This effect is in direct contrast to performance of a positive feedback group, as well as a number of other control groups which demonstrated the typical vigilance decrement. Interestingly, however, the negative feedback group performed at virtually the same level as a yoked control group. The yoked control group received the same order of changes in event rate that were generated by the negative feedback subjects using the closed-loop system. Thus it would appear to be possible to optimize vigilance performance by controlling the stimuli which subjects are asked to process.
The behavior of expert pilots in familiar situations is explored and the implications for better training programs and cockpit designs are stated. Experts in familiar operational situations performing highly practiced tasks are said to recognize and respond to complex situations using pattern recognition or intuition. For some tasks this class of behaviors is desirable; performance can be improved by reducing cognitive load and increasing speed and accuracy. Part-task training, training for monitoring and techniques for the transfer of knowledge can facilitate the development of these skills. Methods for promoting pattern recognition through pilot-aircraft interface design include the use of spatial presentations of information and providing triggering events. In some instances, the familiar, well-practiced behavior is not appropriate and it is desirable to prevent the response. When prevention is necessary, barriers can be constructed in the interface to remind the pilot of the inappropriateness of the response.
NASA and the commercial spacecraft community are working diligently to put the first woman on the moon in the 2024 timeframe. At the same time, NASA researchers are thinking about how to solve the even larger challenges that future deep space missions will bring. Space travel itself is difficult, but astronauts on deep space missions will face obstacles and unknowns never before experienced. In addition to the altered gravity and hostile/closed environment of a spacecraft, deep space crews will face increased radiation, isolation, and distance from Earth. During Extravehicular Activity (EVA, or “spacewalk”) operations, crew will experience increased physical and cognitive workload due to extended types, frequencies and durations of tasks performed on exploration missions in partial gravity environments. All of these stressors will impact crew physical and mental health and performance in difficult-to-anticipate ways. Crew autonomy may be one of the biggest challenges faced. Communication delays and blackouts will occur, and in those situations, the crew may not have access to the Mission Control Center (MCC). They may be forced to be solely dependent on each other and the available information onboard to stay alive, healthy, and achieve the mission. The only conceivable way to meet the challenges of Earth independence is to enable the crew to monitor their own health and performance -- preferably unobtrusively as they perform their duties. Technologies and techniques must be developed to aid the crew in these assessments. A deep space mission is expected to have relatively short periods of high cognitive demand, stress, and fatigue, alongside potentially long periods of cognitive underload during the transit, where boredom, loneliness, and depression can set in. Both ends of this spectrum are dangerous. Crew must be made aware when their task performance drops significantly, when their cognitive workload is too high, when they have lost situation awareness, or when they are too stressed or too fatigued to perform well. They must be able to identify these risks, and then mitigate them with countermeasures available onboard. A number of self-monitoring technologies are presently being explored by NASA to advance crew state determination capabilities. These range from real-time, physiological workload and situation awareness assessments, to crew health measurements determining physical and mental fitness for duty, to task performance metrics such as suit resource expenditures. For EVA tasks during surface exploration missions, biomedical information such as metabolic rate may be provided to crewmembers for situational awareness related to task performance efficiency. In addition, translation distances, hydration, nutrition, inspired CO2 exposure and other consumables usage rates may be useful input metrics for modeling individualized performance during tasks to inform crew or provide estimates of work efficiency. Oculomotor metrics such as gaze dwell time, pupilometry, and eye tracking collected in advanced helmet mounted displays could potentially be used to characterize crew situation awareness. This paper highlights some of these projects, and provides broader discussion about the need for advanced monitoring and smart technologies, as NASA takes the leap into the next generation of space exploration.
BACKGROUND Landing during exploration spaceflight may consist of both planned automated supervisory control and unplanned crew override. Supervisory control, particularly when performed under cognitive load with additional monitoring tasks, is essential for ensuring overall mission success during landing contingencies. Evaluating performance in a relevant Human Landing System (HLS) supervisory landing task after long-duration microgravity exposure can help identify potential risks from human error and sensorimotor alterations. Adaptive changes in the sensorimotor system can manifest during g-transitions as spatial disorientation. Although training and landing aids facilitate successful landings despite disorientation, these adaptive changes may heighten cognitive demand, which must be considered in the landing strategy. It is important to characterize these effects as soon as possible following the G-transition while the sensorimotor system remains in a state of adaptive flux to inform appropriate countermeasures. METHODS A Multi-attribute Lunar Table Battery (MALTB) task was developed for an iOS tablet device to provide flexible crew testing and training capabilities in-flight and on the ground. Elements of the tablet task were derived from the Multi-Attribute Task Battery (MATB, Cegarra et al. 2020). The task requires crew members to study a map of a planned landing site and memorizing the terrain and surface landmarks to inform potential divert maneuvers during landing. The user will oversee a series of approaches through touchdown simulations on the tablet with an external camera view of the Lunar surface. The primary responsibility of the crew member will be to execute a divert if the guidance recommended site is erroneous (e.g., the guidance projected landing target is not within 10m of the planned landing site center), or the projected landing site is no longer suitable due to surface obstacles. Considering vehicle maneuverability and fuel reserves, the divert capabilities will diminish as the task progresses. In cases where a divert is initiated, a new landing target will need to be designated by the user and will be evaluated for the proximity to the original pre-planned site. A secondary operational monitoring task will challenge the user's cognitive reserve by requiring the user to maintain several gauges within acceptable limits and respond to a visual indicator while completing the landing approach. Outcome measures include distance from the planned landing site to the user-initiate divert landing location, ground slope at the new landing site, time to divert, the ability to accomplish the secondary monitoring tasks, and perceived workload. The tablet task is being evaluated in a ground-based study to determine the learning effect of first-time users. The tablet task will be utilized in a flight study to test performance multiple times postflight. Future potential testing in-flight have been identified for capsules with iOS tablet devices. Multi-attribute Lunar Table Battery Task MALTB comprises video footage of thirty distinct landing conditions. The landing scenarios feature various landing sites (n = 3), hazardous object sizes (n = 6), sun azimuth degrees (n = 4), camera modes (i.e., fixed or gimbaled), and navigation bias (i.e., true or false). Each seventy second trial uses a sixty-degree constant glideslope trajectory. The application architecture and layout include user identification setup and data storing, a guided walkthrough of the task and interface components, practice mode for task familiarization, and a modified Bedford workload scale administered following task completion. The time-based dependent measures are saved locally to the iOS Files application and post-processing scripts have been developed to evaluate the remaining measures of performance. RELEVANCE This project will deliver an operational demonstration of crew monitoring capability following spaceflight and identify potential deficits that may require remediation. Comparison of individual vestibular and cognitive changes with crew performance will help better characterize the landing risks associated with sensorimotor alterations. ACKNOWLEDGEMENTS: This project is funded by NASA’s Human Research Program Human Health Countermeasures Element. REFERENCES Cegarra J, Valery B, Avril E, Calmettes C, Navarro J (2020) OpenMATB: A Multi-Attribute Task Battery promoting task customization, software extensibility and experiment replicability. Behav Res Methods 52:1980-1990 doi: 10.3758/s13428-020-01364-w
Environmental monitoring at geologic CO 2 storage sites is required by regulations for the purposes of environmental protection and emissions accounting in the case of leakage to surface. However, another very important goal of environmental monitoring is to assure stakeholders that the project is monitored for safety and effectiveness. With current efforts to optimize monitoring for cost-effectiveness, the question remains: will optimization of monitoring approaches degrade stakeholder assurance, or do heavily-instrumented sites communicate higher risk to a stakeholder? We report the results of a stakeholder survey in Gulf Coast states of the US where carbon capture and storage (CCS) is developing quickly. We rely on a 2 by 2 factorial experiment in which we manipulate message complexity (complex v. simple) and social norm (support from scientists v. support from community members). Subjects were randomly assigned to one of four conditions: 1) complex message with scientist support; 2) complex message with community member support; 3) simple message with scientist support; or 4) simple message with community member support. In addition to the experimental stimuli, subjects were also asked about their need for cognition, attitudes toward science and scientists, attitudes about climate change and support for carbon capture and storage (CCS). Our sample is drawn from residents in states bordering the western Gulf of Mexico (Texas, Louisiana, Florida) where CO 2 geologic storage is being planned both onshore and offshore. The results offer important implications for public outreach efforts to key stakeholders.
The goal of this project was to test the applicability of information theoretic learning (feasibility study) to develop new brain computer interfaces (BCI). The difficulty to BCI comes from several aspects: (1) the effective data collection of signals related to cognition; (2) the preprocessing of these signals to extract the relevant information; (3) the pattern recognition methodology to detect reliably the signals related to cognitive states. We only addressed the two last aspects in this research. We started by evaluating an information theoretic measure of distance (Bhattacharyya distance) for BCI performance with good predictive results. We also compared several features to detect the presence of event related desynchronization (ERD) and synchronization (ERS), and concluded that at least for now the bandpass filtering is the best compromise between simplicity and performance. Finally, we implemented several classifiers for temporal - pattern recognition. We found out that the performance of temporal classifiers is superior to static classifiers but not by much. We conclude by stating that the future of BCI should be found in alternate approaches to sense, collect and process the signals created by populations of neurons. Towards this goal, cross-disciplinary teams of neuroscientists and engineers should be funded to approach BCIs from a much more principled view point.
BACKGROUND Manual control during exploration spaceflight consists of both planned automated supervisory control and unplanned crew override. This crew override capability is critical to enable overall mission success during landing contingencies. However, the introduction of manual override capabilities must be implemented to enable crews to mitigate risks introduced by human error. Adaptive changes in the sensorimotor system can manifest during g-transitions as spatial disorientation. While training and landing aids enable successful landing through disorientation, these adaptive changes may increase cognitive demand that needs to be accounted for in the manual control strategy. It is important to characterize these effects as soon as possible following the G-transition to develop appropriate countermeasures. METHODS In this study, we will examine operational piloting tasks following International Space Station (ISS) missions in a simulated lunar landing using a six-degree-of-freedom (6DOF) motion base. The motion-based simulation will be implemented in our laboratory at the Johnson Space Center (JSC) and will be available within hours following the return from commercial crew landings. The primary goals of this study are (1) to understand the impact of spaceflight on crew ability to perform manual crew override tasks, (2) to examine how adaptive changes in vestibular and cognitive function relate to changes in manual crew override proficiency, and (3) compare performance during late “just-in-time” on-board training with early post-flight crew performance. The impact of spaceflight on piloting capability will be assessed from pre- versus post-flight changes in crewmembers assigned to either short duration (< 30 day) or long duration (~6- month) missions to the International Space Station (ISS). Individual differences in post-flight vestibular and cognitive changes include motion sickness reports, measures of tilt motion perception accuracy and precision, and dual-task tracking. During the 6DOF lunar simulation, the crew will manually takeover attitude and rate-of-descent to the nominal or re-designated landing aim point during the approach phase. The outcome measures for the lunar crew override tasks will be the percent time maintaining actual vehicle states, e.g., attitude and rate-of-descent, within recommended guidance during the landing approach, number and maximum deviation outside limits, and root mean square error (RMSE). Given that “just-in-time” (JIT) training is an operational expectation for the Human Landing System (HLS) program, all participants will perform late inflight JIT training for each manual crew override task in which they will participate. Crew proficiency will be captured inflight during JIT training that will be implemented on a laptop with hand controllers to allow the crewmember to practice the landing task procedures like the approach implemented for JIT training with Shuttle landing and ISS telerobotic tasks. COUNTERMEASURE UPDATES The lunar lander simulation has been implemented onto a two-laptop display system where subjects select from alternative landing points, based upon avoiding hazards that the on-board system identifies during the landing. The subject makes inputs using a rotational hand controller (i.e., joystick) and a translational hand controller, which are processed by simulated vehicle dynamics to update the vehicle attitude and rate of descent. Subjects will utilize a combination of flight, situation, and status displays to monitor the state of the simulated vehicle. The simulation exports tilt and translation movements to a 6DOF motion base synced to visual movements and provides representative vestibular cues to the subject while performing the task inside an enclosed cabin featuring a simulated lunar out-the-window view and a heads-down lander cockpit view. RELEVANCE This project will deliver an operational demonstration of crew override capability following spaceflight and identify potential deficits that may require remediation. Comparison of individual vestibular and cognitive changes with crew override performance will help better characterize the manual control risks associated with sensorimotor alterations. The inclusion of “just-in-time” on-board training will ensure we are characterizing changes in override proficiency with this expected countermeasure in place. ACKNOWLEDGEMENTS: The authors acknowledge contributions from Draper, HLS Joint Test Panel, Crew HLS Interfaces for Piloting Working Group, and Dynamic Skills Trainer (DST) Lab toward the development of the lunar landing simulation. This project is funded by the Human Health Countermeasures Element.
A method, system, apparatus and device for the monitoring, diagnosis and evaluation of the state of a dynamic pulmonary system is disclosed. This method and system provides the processing means for receiving sensed and/or simulated data, converting such data into a displayable object format and displaying such objects in a manner such that the interrelationships between the respective variables can be correlated and identified by a user. This invention provides for the rapid cognitive grasp of the overall state of a pulmonary critical function with respect to a dynamic system.
Automation-induced complacency has been documented as a cause or contributing factor in many airplane accidents throughout the last two decades. It is surmised that the condition results when a crew is working in highly reliable automated environments in which they serve as supervisory controllers monitoring system states for occasional automation failures. Although many reports have discussed the dangers of complacency, little empirical research has been produced to substantiate its harmful effects on performance as well as what factors produce complacency. There have been some suggestions, however, that individual characteristics could serve as possible predictors of performance in automated systems. The present study examined relationship between the individual differences of complacency potential, boredom proneness, and cognitive failure, automation-induced complacency. Workload and boredom scores were also collected and analyzed in relation to the three individual differences. The results of the study demonstrated that there are personality individual differences that are related to whether an individual will succumb to automation-induced complacency. Theoretical and practical implications are discussed.
Astronauts often use promethazine (PMZ) to counteract space motion sickness; however PMZ may cause drowsiness, which might impair cognitive function. In a NASA ground study, subjects received PMZ and their cognitive performance was then monitored over time. Subjects also reported sleepiness using the Karolinska Sleepiness Score (KSS), which ranges from 1 - 9. A problem arises when using KSS to establish an association between true sleepiness and performance because KSS scores tend to overly concentrate on the values 3 (fairly awake) and 7 (moderately tired). Therefore, we defined a latent sleepiness measure as a continuous random variable describing a subject s actual, but unobserved true state of sleepiness through time. The latent sleepiness and observed KSS are associated through a conditional probability model, which when coupled with demographic factors, predicts performance.
Crewed missions into deep space will require astronauts to respond autonomously to safety- and time-critical anomalies. These and other potential issues will require the continuous monitoring and recognition of potentially subtle, but complex, anomalous data patterns. NASA continues to investigate real-time metrics for Extravehicular Operations1 and for providing feedback to improve performance2. Since it is unknown how long-term exposure to deep space will affect crew health, continuous monitoring of clinical and subclinical status is warranted. NASA is characterizing crew medical decision-support needs and identifying the recommended requirements of a Clinical Decision Support System (CDSS) within the Exploration Medical Capability Element (ExMC) of the Human Research Program3. Early detection, and continuous monitoring, of performance may provide an effective prognostic capability for assessing crew health state, and the identified requirements could be provided for an in-vehicle CDSS that acts as an assistant for delivering optimal health and performance and medical care during exploration missions. There is evidence to suggest that automatic, unobtrusive collection of keystroke features, such as alphanumeric key latencies, backspaces or typing rhythm, could be a valuable performance screening tool. Keystroke features show significant differences between control subjects and patients with overt clinical illness, such as multiple sclerosis4 and Parkinson’s disease5, as well as behavioral health and performance issues, such as mild cognitive impairment6 and depression7. This work is exploring a capability for the assessment of early task performance decline in crew as they perform daily activities. Timestamped keyboard entries are continuously collected and stored. Nominal keystroke interactions are compared to those obtained after exposure to spaceflight stressors, e.g., fatigue, to determine the methods sensitivity. This work will pave the way toward an objective detection tool that may be deployed in a spaceflight setting.
Linea Research Corporation has developed a neurodiagnostic tool that detects behavioral stress markers for astronauts on long-duration space missions. Lightweight and compact, the device is unobtrusive and requires minimal time and effort for the crew to use. The system provides a real-time functional imaging of cortical activity during normal activities. In Phase I of the project, Linea Research successfully monitored cortical activity using multiparameter sensor modules. Using electroencephalography (EEG) and functional near-infrared spectroscopy signals, the company obtained photoplethysmography and electrooculography signals to compute the heart rate and frequency of eye movement. The company also demonstrated the functionality of an algorithm that automatically classifies the varying degrees of cognitive loading based on physiological parameters. In Phase II, Linea Research developed the flight-capable neurodiagnostic device. Worn unobtrusively on the head, the device detects and classifies neurophysiological markers associated with decrements in behavior state and cognition. An automated algorithm identifies key decrements and provides meaningful and actionable feedback to the crew and ground-based medical staff.
Hypobaric decompression sickness (DCS) research presents the medical monitor with the difficult task of assessing the onset and progression of DCS largely on the basis of subjective symptoms. Even with the introduction of precordial Doppler ultrasound techniques for the detection of venous gas emboli (VGE), correct prediction of DCS can be made only about 65 percent of the time according to data from the Armstrong Laboratory's (AL's) hypobaric DCS database. An AL research protocol concerned with exercise and its effects on denitrogenation efficiency includes implementation of a performance assessment test battery to evaluate cognitive functioning during a 4-h simulated 30,000 ft (9144 m) exposure. Information gained from such a test battery may assist the medical monitor in identifying early signs of DCS and subtle neurologic dysfunction related to cases of asymptomatic, but advanced, DCS. This presentation concerns the selection and integration of a test battery and the timely graphic display of subject test results for the principal investigator and medical monitor. A subset of the Automated Neuropsychological Assessment Metrics (ANAM) developed through the Office of Military Performance Assessment Technology (OMPAT) was selected. The ANAM software provides a library of simple tests designed for precise measurement of processing efficiency in a variety of cognitive domains. For our application and time constraints, two tests requiring high levels of cognitive processing and memory were chosen along with one test requiring fine psychomotor performance. Accuracy, speed, and processing throughout variables as well as RMS error were collected. An automated mood survey provided 'state' information on six scales including anger, happiness, fear, depression, activity, and fatigue. An integrated and interactive LOTUS 1-2-3 macro was developed to import and display past and present task performance and mood-change information.
NASA space systems, including mission operations on the ground and in space, are complex, dynamic, predominantly automated systems in which the human operator is a supervisory controller. The human operator monitors and fine-tunes computer-based control systems and is responsible for ensuring safe and efficient system operation. In such systems, the potential consequences of human mistakes and errors may be very large, and low probability of such events is likely. Thus, models of cognitive functions in complex systems are needed to describe human performance and form the theoretical basis of operator workstation design, including displays, controls, and decision support aids. The operator function model represents normative operator behavior-expected operator activities given current system state. The extension of the theoretical structure of the operator function model and its application to NASA Johnson mission operations and space station applications is discussed.
An experiment conducted using a head-up display (HUD) suggests that the demonstrated superiority of the HUD over a conventional instrument panel stems from its superior layout of information. A HUD display presents instrument-panel information to pilots in such a way that the symbols appear as a virtual image at optical infinity superimposed on the landscape. In the experiment conducted, the luminance of the display symbology and its angle subtended at the eye remained fixed, while optical distance and gaze angle were varied. Concomitant measures of eye movements, eye accommodative state, and decision-making time concerning airspeed, altitude and runway condition were obtained. It is found that, while looking straight ahead, at zero diopters, the HUD shortens decision time by 80 to 90 msec, not statistically significant at the 0.05 (slope of diopter/gaze interaction) level. The question of a cognitive overload induced by the luminous symbols of the HUD is subsequently addressed.
The 2014 Space Radiation Standing Review Panel (from here on referred to as the SRP) participated in a WebEx/teleconference with members of the Space Radiation Program Element, representatives from the Human Research Program (HRP), the National Space Biomedical Research Institute (NSBRI), and NASA Headquarters on November 21, 2014 (list of participants is in Section XI of this report). The SRP reviewed the updated Research Plan for the Risk of Cardiovascular Disease and Other Degenerative Tissue Effects from Radiation Exposure (Degen Risk). The SRP also received a status update on the Risk of Acute and Late Central Nervous System Effects from Radiation Exposure (CNS Risk), the Risk of Acute Radiation Syndromes Due to Solar Particle Events (ARS Risk), and the Risk of Radiation Carcinogenesis (Cancer Risk). The SRP thought the teleconference was very informative and that the Space Radiation Program Element did a great job of outlining where the Element is with respect to our state of knowledge on the risks of carcinogenesis, central nervous system effects, and the risk of cardiovascular disease and other degenerative tissue effects from exposure to space radiation. The SRP was impressed with the quality of research that is being conducted and the progress the Space Radiation Program Element has made in the past year. While much work has been done, the SRP had a few remaining questions regarding the broad applicability of these findings to a manned deep space mission (in terms of cognitive function, the paradigms were still hippocampal based and also using Alzheimer disease models). The SRP believes that NASA should consider developing an approach to follow astronauts long-term (beyond retirement) for potential side-effects/risks of space exposure that may be unknown. Radiation toxicities often occur decades after exposure, and potential consequences would be missed if intensified exams stop after retirement of the astronauts. In addition, while cancer is one consequence of radiation exposure that is monitored, potential other side effects (CNS, Alzheimer Disease, loss of cognitive function, etc.) are not included in long-term studies and would be missed. Inclusion of long-term data would be of benefit to the astronauts themselves who have given their service to the corps but also to future astronauts and the future of space exploration.