Search NASA⌕ Search

SEARCH · Search NASA

Results for “multimodal display”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records

The Role of Tactile Cueing in Multimodal Displays: Application in Complex Task Environments for Space Exploration

This paper reviews the potential role of tactile cueing in multimodal displays that will need to be developed to enable Extravehicular Activity (EVA) and telerobotic exploration for both Moon and Mars missions. Specific missions to be accomplished during EVA surface and space operations will include construction and assembly, surface and geologic exploration, and excavation for protective shelter from radiation and significant solar particle events. EVA crews have limited time, resources, mobility, visibility and dexterity, and activities take place in a perceptually impoverished environment. Limitations include a loss of spatial reference frame leading to spatial disorientation, a loss of information normally provided by the auditory and somato-sensory systems, and a restricted field of view. In addition, in non-surface operations crew must be able to navigate and operate in a fully six degree of freedom (DoF) environment. Fatigue resulting from suited EVA tasks will also critically affect performance and safety. In order to mitigate these factors, it is important that crews be well trained prior to executing actual missions. There is a critical need to design multimodal interfaces to optimize performance under these conditions for both training and actual operations. Because of the high level of complexity of the EVA environment on the one hand, and the “sensory restrictions” on the other, it makes sense to envisage enhancements to the astronauts’ visual environment. Such an enriched environment may include virtual auditory as well as tactile inputs.

multimodal displays↗

Advanced Multimodal Solutions for Information Presentation

High-workload, fast-paced, and degraded sensory environments are the likeliest candidates to benefit from multimodal information presentation. For example, during extra-vehicular activity (EVA) and telerobotic operations, the sensory restrictions associated with such a hostile environment provide a major challenge to maintaining the situation awareness (SA) required for safe operations. In particular, orientation, navigation, and collision avoidance are critical aspects of EVA tasks that need to be addressed to ensure the safety of the crew and the success of the mission. Multimodal displays hold promise to enhance situation awareness and task performance by utilizing different sensory modalities and maximizing their effectiveness based on appropriate interaction between modalities. Multimodal displays will also play an important role for long-duration information systems and will likely begin to be developed in the early phases of cislunar Gateway, and later lunar or Mars transit missions. Information systems are envisioned for LDEMs that require spacecraft with greater crew autonomy and increased dependence on computer-provided information needed to perform routine tasks, as well as time- and safety critical tasks. Such a system will require a single, common interface that is easy to learn and use and accesses key information from all relevant vehicle/habitat systems to enable task performance in both nominal and emergency conditions. Understanding of multimodal display technologies and their interactions will help to inform interface guidelines for LDEMs. The scope of the current report is an analysis of potential multimodal display technologies for long duration missions and, in particular, will focus on their potential role in EVA activities. The review will address multimodal (combined visual, auditory and/or tactile) displays investigated by the National Aeronautics and Space Administration (NASA), industry, and Department of Defense (DoD). It also considers the need for adaptive information systems to accommodate a variety of operational contexts such as crew status (e.g., fatigue, workload level) and task environment (e.g., EVA, habitat, rover, spacecraft). Current approaches to guidelines and best practices for combining modalities for the most effective information displays are also reviewed. Potential issues in developing interface guidelines for LDEMs are briefly considered.

multimodal displays↗

Advanced Multimodal Solutions for Information Presentation

High-workload, fast-paced, and degraded sensory environments are the likeliest candidates to benefit from multimodal information presentation. For example, during EVA (Extra-Vehicular Activity) and telerobotic operations, the sensory restrictions associated with a space environment provide a major challenge to maintaining the situation awareness (SA) required for safe operations. Multimodal displays hold promise to enhance situation awareness and task performance by utilizing different sensory modalities and maximizing their effectiveness based on appropriate interaction between modalities. During EVA, the visual and auditory channels are likely to be the most utilized with tasks such as monitoring the visual environment, attending visual and auditory displays, and maintaining multichannel auditory communications. Previous studies have shown that compared to unimodal displays (spatial auditory or 2D visual), bimodal presentation of information can improve operator performance during simulated extravehicular activity on planetary surfaces for tasks as diverse as orientation, localization or docking, particularly when the visual environment is degraded or workload is increased. Tactile displays offer a third sensory channel that may both offload information processing effort and provide a means to capture attention when urgently required. For example, recent studies suggest that including tactile cues may result in increased orientation and alerting accuracy, improved task response time and decreased workload, as well as provide self-orientation cues in microgravity on the ISS (International Space Station). An important overall issue is that context-dependent factors like task complexity, sensory degradation, peripersonal vs. extrapersonal space operations, workload, experience level, and operator fatigue tend to vary greatly in complex real-world environments and it will be difficult to design a multimodal interface that performs well under all conditions. As a possible solution, adaptive systems have been proposed in which the information presented to the user changes as a function of taskcontext-dependent factors. However, this presupposes that adequate methods for detecting andor predicting such factors are developed. Further, research in adaptive systems for aviation suggests that they can sometimes serve to increase workload and reduce situational awareness. It will be critical to develop multimodal display guidelines that include consideration of smart systems that can select the best display method for a particular contextsituation.The scope of the current work is an analysis of potential multimodal display technologies for long duration missions and, in particular, will focus on their potential role in EVA activities. The review will address multimodal (combined visual, auditory andor tactile) displays investigated by NASA, industry, and DoD (Dept. of Defense). It also considers the need for adaptive information systems to accommodate a variety of operational contexts such as crew status (e.g., fatigue, workload level) and task environment (e.g., EVA, habitat, rover, spacecraft). Current approaches to guidelines and best practices for combining modalities for the most effective information displays are also reviewed. Potential issues in developing interface guidelines for the Exploration Information System (EIS) are briefly considered.

multimodal display↗

Virtual workstation - A multimodal, stereoscopic display environment

A head-mounted, wide-angle, stereoscopic display system controlled by operator position, voice and gesture has been developed for use in a multipurpose interface environment. The system provides a multisensory, interactive display environment in which a user can virtually explore a 360-degree synthesized or remotely sensed environment and can viscerally interact with its components. Primary applications of the system are in telerobotics, management of large-scale integrated information systems, and human factors research. System configuration, application scenarios, and research directions are described.

Fisher, S. S.↗

Enhancing Situation Awareness of Automated Procedures Using Adaptive Multimodal Augmented Reality Displays

In future missions, crew must be more autonomous and perform tasks with limited help from MCC. Virtual Intelligent Task Assistant (VITA) will stand-in for MCC and help crew perform tasks. Prepare for and perform manual tasks to increase efficiency and accuracy. Take over partially completed task by summarizing progress made so far, and reviewing remaining tasks. Maintain situation awareness of ongoing automated procedures while crew is performing manual tasks.

Schreckenghost, D.↗

Multimodality Imaging Approach towards Primary Aortic Sarcomas Arising after Endovascular Abdominal Aortic Aneurysm Repair: Case Series Report

Primary aortic neoplasms are rare. Aortic sarcoma arising after endovascular aneurysm repair (EVAR) is a scarce subset of primary aortic malignancies, reports of which are infrequent in the published literature. The diagnosis of aortic sarcoma is challenging due to its non-specific clinical presentation, and the prognosis is poor due to delayed diagnosis, rapid proliferation, and propensity for metastasis. Post-EVAR, aortic sarcomas may mimic other more common aortic processes on surveillance imaging. Radiologists are rarely knowledgeable about this rare entity for which multimodality imaging and awareness are invaluable in early diagnosis. A series of three pathologically confirmed cases are presented to display the multimodality imaging features and clinical presentations of aortic sarcoma arising after EVAR.

62 RADIOLOGY AND NUCLEAR MEDICINE↗

Picosecond streak camera display of an intermodal coupling matrix at a multimode fiber splice

A technique is developed that permits direct visualization of the coupling matrix for all guided modes of a moderately multimode fiber optic at a splice or mode coupler. The matrix is formed by an array of spots at the output of a picosecond streak camera. The technique also permits unambiguous determination of the phase velocity and group velocity of the modes.

Louradour, F.↗

Multimodal information Management: Evaluation of Auditory and Haptic Cues for NextGen Communication Displays

Auditory communication displays within the NextGen data link system may use multiple synthetic speech messages replacing traditional ATC and company communications. The design of an interface for selecting amongst multiple incoming messages can impact both performance (time to select, audit and release a message) and preference. Two design factors were evaluated: physical pressure-sensitive switches versus flat panel "virtual switches", and the presence or absence of auditory feedback from switch contact. Performance with stimuli using physical switches was 1.2 s faster than virtual switches (2.0 s vs. 3.2 s); auditory feedback provided a 0.54 s performance advantage (2.33 s vs. 2.87 s). There was no interaction between these variables. Preference data were highly correlated with performance.

Begault, Durand R.↗

Piloted Evaluation of the H-Mode, a Variable Autonomy Control System, in Motion-Based Simulation

As aircraft become able to autonomously respond to a range of situations with performance surpassing human operators, we are compelled to look for new methods that help understand their use and guide the design of new, more effective forms of automation and interaction. The "H-mode" is one such method and is based on the metaphor of a well-trained horse. The concept allows the pilot to manage a broad range of control automation functionality, from augmented manual control to FMS-like coupling and automation initiated actions, using a common interface system and easily learned set of interaction skills. The interface leverages familiar manual control interfaces (e.g., the control stick) and flight displays through the addition of contextually dependent haptic-multimodal elements. The concept is relevant to manned and remotely piloted vehicles. This paper provides an overview of the H-mode concept followed by a presentation of the results from a recent evaluation conducted in a motion-based simulator. The evaluation focused on assessing the overall usability and flying qualities of the concept with an emphasis on the effects of turbulence and cockpit motion. Because the H-mode results in interactions between traditional flying qualities and management of higher-level flight path automation, these effects are of particular interest. The results indicate that the concept may provide a useful complement or replacement to conventional interfaces, and retains the usefulness in the presence of turbulence and motion.

Goodrich, Kenneth H.↗

Radiolabeled Peptide Scaffolds for PET/SPECT - Optical in Vivo Imaging of Carbohydrate-Lectin Interactions

The objective of this research is to develop phage display-selected peptides into radio- and fluoresecently- labeled scaffolds for the multimodal imaging of carbohydrate-lectin interactions. While numerous protein and receptor systems are being explored for the development of targeted imaging agents, the targeting and analysis of carbohydrate-lectin complexes in vivo remains relatively unexplored. Antibodies, nanoparticles, and peptides are being developed that target carbohydrate-lectin complexes in living systems. However, antibodies and nanoparticles often suffer from slow clearance and toxicity problems. Peptides are attractive alternative vehicles for the specific delivery of radionuclides or fluorophores to sites of interest in vivo, although, because of their size, uptake and retention may be less than antibodies. We have selected high affinity peptides that bind a specific carbohydrate-lectin complex involved in cell-cell adhesion and cross-linking using bacteriophage (phage) display technologies (1,2). These peptides have allowed us to probe the role of these antigens in cell adhesion. Fluorescent versions of the peptides have been developed for optical imaging and radiolabeled versions have been used in single photon emission computed tomography (SPECT) and positron emission tomography (PET) in vivo imaging (3-6). A benefit in employing the radiolabeled peptides in SPECT and PET is that these imaging modalities are widely used in living systems and offer deep tissue sensitivity. Radiolabeled peptides, however, often exhibit poor stability and high kidney uptake in vivo. Conversely, optical imaging is sensitive and offers good spatial resolution, but is not useful for deep tissue penetration and is semi-quantitative. Thus, multimodality imaging that relies on the strengths of both radio- and optical- imaging is a current focus for development of new in vivo imaging agents. We propose a novel means to improve the efficacy of radiolabeled and fluorescently labeled peptides, including our lectin/carbohydrate- targeting peptides, by displaying the targeting epitopes on small ~29 amino acid cyclic plant protein scaffolds known as cyclotides. Cyclotides are extremely stable molecules with long serum half-lives and low kidney uptake (7). More than one copy of the peptide can be engineered into the cyclotide loops, thus increasing the avidity of the peptide construct for its target.

59 BASIC BIOLOGICAL SCIENCES↗

Comparison of genetic algorithms with conjugate gradient methods

Genetic algorithms for mathematical function optimization are modeled on search strategies employed in natural adaptation. Comparisons of genetic algorithms with conjugate gradient methods, which were made on an IBM 1800 digital computer, show that genetic algorithms display superior performance over gradient methods for functions which are poorly behaved mathematically, for multimodal functions, and for functions obscured by additive random noise. Genetic methods offer performance comparable to gradient methods for many of the standard functions.

Bosworth, J. L.↗

System and Method for Training of State-Classifiers

Method and systems are disclosed for training state-classifiers for classification of cognitive state. A set of multimodal signals indicating physiological responses of an operator are sampled over a time period. A depiction of operation by the operator during the time period is displayed. In response to user input selecting a cognitive state for a portion of the time period, the one or more state-classifiers are trained. In training the state-classifiers, the set of multimodal signals sampled in the portion of the time period are used as input to the one or more state-classifiers and the selected one of the set of cognitive states is used as a target result to be indicated by the one or more state-classifiers.

Stephens, Chad L.↗

Folding at the Microscale: Enabling Multifunctional 3D Origami‐Architected Metamaterials

Abstract Mechanical metamaterials inspired by the Japanese art of paper folding have gained considerable attention because of their potential to yield deployable and highly tunable assemblies. The inherent foldability of origami structures enlarges the material design space with remarkable properties such as auxeticity and high deformation recoverability and deployability, the latter being key in applications where spatial constraints are pivotal. This work integrates the results of the design, 3D direct laser writing fabrication, and in situ scanning electron microscopic mechanical characterization of microscale origami metamaterials, based on the multimodal assembly of Miura‐Ori tubes. The origami‐architected metamaterials, achieved by means of microfabrication, display remarkable mechanical properties: stiffness and Poisson’s ratio tunable anisotropy, large degree of shape recoverability, multistability, and even reversible auxeticity whereby the metamaterial switches Poisson’s ratio sign during deformation. The findings here reported underscore the scalable and multifunctional nature of origami designs, and pave the way toward harnessing the power of origami engineering at small scales.

Lin, Zhaowen↗

WE-D-9A-03: CSDF: A Color Extension of the Grayscale Standard Display Function

Purpose: Use of color images in medical imaging has increased significantly the last few years. As of today there is no agreed standard on how color information needs to be visualized on medical color displays, resulting into large variability of color appearance and it making consistency and quality assurance a challenge. This paper presents a proposal for an extension of DICOM GSDF towards color. Methods: Visualization needs for several color modalities (multimodality imaging, nuclear medicine, digital pathology, quantitative imaging applications…) have been studied. On this basis a proposal was made for desired color behavior of color medical display systems and its behavior and effect on color medical images was analyzed. Results: Several medical color modalities could benefit from perceptually linear color visualization for similar reasons as why GSDF was put in place for greyscale medical images. An extension of the GSDF (Greyscale Standard Display Function) to color is proposed: CSDF (color standard display function). CSDF is based on deltaE2000 and offers a perceptually linear color behavior. CSDF uses GSDF as its neutral grey behavior. A comparison between sRGB/GSDF and CSDF confirms that CSDF significantly improves perceptual color linearity. Furthermore, results also indicate that because of the improved perceptual linearity, CSDF has the potential to increase perceived contrast of clinically relevant color features. Conclusion: There is a need for an extension of GSDF towards color visualization in order to guarantee consistency and quality. A first proposal (CSDF) for such extension has been made. Behavior of a CSDF calibrated display has been characterized and compared with sRGB/GSDF behavior. First results indicate that CSDF could have a positive influence on perceived contrast of clinically relevant color features and could offer benefits for quantitative imaging applications. Authors are employees of Barco Healthcare.

60 APPLIED LIFE SCIENCES↗

Effects of Type and Strength of Force Feedback on Movement Time in a Target Selection Task

Future cockpits will likely include new onboard technologies, such as cockpit displays of traffic information, to help support future flight deck roles and responsibilities. These new technologies may benefit from multimodal feedback to aid pilot information processing. The current study investigated the effects of multiple levels of force feedback on operator performance in an aviation task. Participants were presented with two different types of force feedback (gravitational and spring force feedback) for a discrete targeting task, with multiple levels of gain examined for each force feedback type. Approach time and time in target were recorded. Results suggested that the two highest levels of gravitational force significantly reduced approach times relative to the lowest level of gravitational force. Spring force level only affected time in target. Implications of these findings for the design of future cockpit displays will be discussed.

CDTI↗

Visually guided control of movement in the context of multimodal stimulation

Flight simulation has been almost exclusively concerned with simulating the motions of the aircraft. Physically distinct subsystems are often combined to simulate the varieties of aircraft motion. Visual display systems simulate the motion of the aircraft relative to remote objects and surfaces (e.g., other aircraft and the terrain). 'Motion platform' simulators recreate aircraft motion relative to the gravitoinertial vector (i.e., correlated rotation and tilt as opposed to the 'coordinated turn' in flight). 'Control loaders' attempt to simulate the resistance of the aerodynamic medium to aircraft motion. However, there are few operational systems that attempt to simulate the motion of the pilot relative to the aircraft and the gravitoinertial vector. The design and use of all simulators is limited by poor understanding of postural control in the aircraft and its effect on the perception and control of flight. Analysis of the perception and control of flight (real or simulated) must consider that: (1) the pilot is not rigidly attached to the aircraft; and (2) the pilot actively monitors and adjusts body orientation and configuration in the aircraft. It is argued that this more complete approach to flight simulation requires that multimodal perception be considered as the rule rather than the exception. Moreover, the necessity of multimodal perception is revealed by emphasizing the complementarity rather than the redundancy among perceptual systems. Finally, an outline is presented for an experiment to be conducted at NASA ARC. The experiment explicitly considers possible consequences of coordination between postural and vehicular control.

Riccio, Gary E.↗

Influence of friction stir processing on the room temperature fatigue cracking mechanisms of A356 aluminum alloy

Here, failure by fatigue is a common problem associated with cast aluminum alloys due to defects like shrinkage porosities, non-metallic inclusions, etc. Friction stir processing (FSP) has recently emerged as an effective technique for local modification of microstructure. This study investigates the fatigue crack initiation and growth mechanisms in cast and FSPed A356 aluminum alloy. Two sets of parameters were used to friction stir the cast alloy resulting in the complete modification the cast microstructure to a wrought microstructure. Both the FSPed microstructures exhibited severe abnormal grain growth (AGG) after heat treatment leading to a multimodal grain size distribution – the grain sizes ranging from a few microns to a few millimeters. One of the FSP conditions displayed an excellent improvement in fatigue life by an order of magnitude, while the other condition displayed an unexpectedly large scatter in fatigue lives. Detailed study of the fractured fatigue specimens by electron back scattered diffraction (EBSD) revealed that both, fatigue crack initiation and propagation, were intimately tied to the grain size as well as the grain misorientations in the microstructure.

36 MATERIALS SCIENCE↗

Robotic multimodality stereotactic brain tissue identification: work in progress

Real-time identification of tissue would improve procedures such as stereotactic brain biopsy (SBX), functional and implantation neurosurgery, and brain tumor excision. To standard SBX equipment has been added: (1) computer-controlled stepper motors to drive the biopsy needle/probe precisely; (2) multiple microprobes to track tissue density, detect blood vessels and changes in blood flow, and distinguish the various tissues being penetrated; (3) neural net learning programs to allow real-time comparisons of current data with a normative data bank; (4) three-dimensional graphic displays to follow the probe as it traverses brain tissue. The probe can differentiate substances such as pig brain, differing consistencies of the 'brain-like' foodstuff tofu, and gels made to simulate brain, as well as detect blood vessels imbedded in these substances. Multimodality probes should improve the safety, efficacy, and diagnostic accuracy of SBX and other neurosurgical procedures.

Brain/anatomy & histology/surgery↗