Search NASA⌕ Search

SEARCH · Search NASA

Results for “Augmented reality”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 91 records · Page 5

Developing Extended Reality Projects in Support of Design, Fabrication and Procedure

The goal of this internship was to improve and create virtual reality simulations and visualizations for use in parallel with the design, fabrication, and analysis of flight ready hardware for areas like the Environmental Control and Life Support Systems (ECLSS) and also Space Systems. Specifically, my work was done in the XRSpace lab at Marshall Space Flight Center (MSFC) with assistance directly and indirectly from workers at KSC, JSC and LaRC. Led by David Reynolds, the XRSpace lab develops products for various entities at NASA. The work done in the XRSpace lab focuses on Extended Reality (XR) solutions for both simulations and visualization capabilities. The goal of the lab is to support the larger systems of NASA and to help find ways that XR technologies can streamline and optimize the design process. Extended Reality is an umbrella term that encompasses Mixed Reality, Augmented Reality, and Virtual Reality. In this capacity, I was able to complete several elements in the design, building, testing, and deployment for a variety of immersive experiences, including a VR procedure simulation, visualization aids, and a 360-image capture tool.

Joseph Chambers↗

Human Systems Intergration & Engineering: HSI & E Design Influence

The design of space systems is a long and complicated process where changes to the design increase cost and schedule as the design progresses into implementation. Human Systems Integration uses tools such as mockups, Virtual Reality, Augmented Reality and human simulations early in the design process to evaluate designs for usability, operability, and maintainability. This early design influence provides cost and schedule savings during manufacturing and operations. Examples of how MSFC EV74 Human Systems Integration Team have used these tools to evaluate hardware designs will be discussed.

Tanya Andrews↗

An Evaluation of Extended Reality Technologies for Use in Verification Testing at NASA 2024 HRP IWS Abstract

BACKGROUND At NASA, verification testing is the formal process of ensuring that a product conforms to requirements set by a project or program. Some verification methods, such as Demonstrations and Test, require either the end product or a mockup of the product with sufficient fidelity to stand-in for the product during the test. Traditionally, these mockups have been physical (e.g., foam-core and wood) but there is growing interest in exploring new methods for testing with these mockups. These methods include virtual reality (VR), mixed reality, and augmented reality which are collectively referred to as eXtended Reality (XR) technologies. VR has already been adopted and used by many in the aerospace industry as a tool for use in early design phases (e.g., developmental testing) and may have the most potential for use in verification tests. Benefits of using VR mockups offer cost effectiveness, ease of iteration, simulation of hazardous conditions (e.g., an egress through a hatch with smoke obscuring vision), and the ability to simulate microgravity conditions, which are challenging to do with physical mockups. However, the validity of test results obtained from VR mockup demonstrations or testing, compared to the current gold standard of physical mockups, remains uncertain. It is unlikely that there is one clean answer as there are many different types of verification outcomes and each XR technology must be evaluated on its own merits. This is not an issue during developmental testing as the design is still in flux and the total success of the design is not dependent upon the results of a developmental test. Verification tests, however, only happen once, assuming no change to the design, and the results are used to certify the product. Therefore, establishing the validity of XR mockup-based verification outcomes is essential before considering them for any use in verification tests. OBJECTIVE AND METHOD To address this concern, the Human Research Program has funded a project to explore and qualify how XR technologies might be used in verification demonstration and testing at NASA. Currently, we are conducting a review of the literature on the utilization of XR mockups for design activities, prototyping, and user testing. We are employing the Strengths, Weaknesses, Opportunities, and Threats (SWOT) analysis method to identify the pros, cons, and barriers to adoption of XR technologies for verification testing at NASA. Additionally, we are developing a framework to guide the deployment of XR mockups for verification tests. Building upon available evidence from the literature and subject-matter expert feedback, our goal for the framework is to provide guidelines for which forms of XR mockups are suitable for a given verification test, when only physical mockups should be employed and to highlight areas for which more evidence is needed. To further refine our framework and to contribute to the body of evidence, we are planning a lab-based experiment comparing a VR mockup to a physical twin for a set of select verification outcomes. ANTICIPATED RESULTS In this presentation, we will present the work we conducted to evaluate XR technologies for use in verification tests at NASA. We will summarize and report our findings from the SWOT analysis and our lab-based study, and we will present the current state of the XR Technologies for Verification Testing framework. We will conclude by summarizing remaining work and future directions for the project. Technologies for Verification Testing framework. We will conclude by summarizing remaining work and future directions for the project.

Extended Reality↗

Seeing the Future of Spaceflight: Applications of Extended Reality (XR) Technologies

INTRODUCTION Advances in Earth Independent Medical Operations (EIMO) focused capabilities will be crucial for completing successful beyond low-earth-orbit (LEO) missions, where traditional Earth-based medical support and near real-time communication are limited, intermittent, or absent. In such restricted or limited communication environments, emerging extended reality (ER or XR) technologies may play a vital role in ensuring crew health and safety. Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR) are common terms used to describe technologies that "generate or modify" while “extended reality” (ER or XR) is often used as an umbrella term for these technologies. This review explores how such cutting-edge “extended reality” information delivery systems are currently applied across various disciplines within NASA. The review focuses on their capabilities, limitations, and prospective future applications as they pertain to Earth-Independent Medical Operations and its components of pre-mission planning, medical decision making, resource management, and task load management. METHODS/DESIGN The review began with a literature search to identify relevant disciplines (e.g., medical education, austere environments) where XR technologies have been studied to highlight areas where evidence is lacking. Previous and current NASA XR projects and applications, as well as commercial partnerships, such as those with Small Business Innovation Research (SBIR), were identified. Discussions with technical experts involved in the design, development, and application of these technologies allowed for classification based on their respective spaceflight applications. RESULTS AND CONCLUSION This review highlights current and emerging XR technologies that have been developed, tested, and implemented both terrestrially and in spaceflight by NASA, its international partners, the U.S. Armed Forces, and various relevant commercial entities. While these novel information delivery systems vary in their respective technology readiness levels (TRL), they have the potential to be implemented on Earth, in LEO, and during deep space exploration. These systems have the potential for impacting and optimizing many aspects of EIMO.

Yevgeniy Zhivotovskiy↗

Augmented Virtual Reality Laboratory

Real time motion tracking hardware has for the most part been cost prohibitive for research to regularly take place until recently. With the release of the Microsoft Kinect in November 2010, researchers now have access to a device that for a few hundred dollars is capable of providing redgreenblue (RGB), depth, and skeleton data. It is also capable of tracking multiple people in real time. For its original intended purposes, i.e. gaming, being used with the Xbox 360 and eventually Xbox One, it performs quite well. However, researchers soon found that although the sensor is versatile, it has limitations in real world applications. I was brought aboard this summer by William Little in the Augmented Virtual Reality (AVR) Lab at Kennedy Space Center to find solutions to these limitations.

Resolved Skeleton↗

RMIT AR Android Application Abstract

This summer, I've worked as an intern for IT-C1's Rapid Model Import Tool (RMIT) project under Bill Little. The goal of this project was to develop a mobile application for IT-C1's larger project, RMIT, so that it could be deployed in a convenient way. The current function of RMIT is to import file types that are not normally compatible with the Unity 3D software development engine and make them into files that can be placed into Unity for the specific purpose of viewing CAD models in virtual reality (VR). The mobile application that I worked on uses augmented reality (AR), using the camera on a mobile device, to show virtual models in the real world for the purpose of visualization and understanding. In addition to single-player capabilities, this application will include a multi-player mode so that collaborating engineers can see each other's work on their own devices.Working on this application required many steps. First, I hard to learn a lot about how mobile development is different from desktop development, and what the limitations of augmented reality are at its current state. From there, I was assigned new tasks and goals every two weeks, which included addingground plane and air plane functionality, designing a user interface, adding multi-player capability and allowing the user to manipulate models placed in a scene.

Dorian Thompson↗

Interacting with the COVID virus model in VR environment

The Disaster Response Complex at Idaho State University (ISU) hosts an annual Disaster Preparedness and Response Conference which will give a bird's eye view of disasters and pandemics in the U.S. We'll demonstrate virtual reality (VR) and augmented reality (AR) capabilities at the conference. This VR demo displays the structure and components of the COVID virus model, and let users to interact with it in an immersive environment. The 3D model used in this demo is open source and downloaded from this site: https://sketchfab.com/GLS.

3D↗

Intelligent Process Visualization through Nuclear Operation Process Modeling, Reasoning, and Object Detection from Field Videos (Final Report)

This report is a deliverable for the “Final Report” task of DOE NEET Project 19-16790, "Context-Aware Safety Information Display for Nuclear Field Workers." This project's overall goal is to test the hypothesis that integrating computer vision and process reasoning methods will enable proactive visualization of the safe operation and maintenance processes of Nuclear Power Plants (NPP) for field workers. Augmented Reality (AR) glasses adopting such proactive safety information visualization techniques can significantly increase personnel safety and reduce the NPP’s operating costs. The current practice of monitoring NPPs requires workers to switch between digital models, data, and physical workspaces in identifying relevant but potentially occluded objects and in assessing the risks of operation and maintenance processes. On the other hand, frequently changed field conditions require field workers to report to supervisors for real-time guidance. Such guidance is essential to ensure that changing conditions will not invalidate or endanger the work order and other ongoing processes that may jeopardize NPP operations. Additionally, incorrect recognition of equipment objects can result in communication errors and safety problems. AR techniques can assist engineers in viewing the physical workspaces with objects labeled with detailed operation procedures and safety reminders during field operations. The project team developed an “Intelligent Context-Aware Safety Information Display” (ICAD) for supporting Nuclear Power Plant (NPP) field workers in achieving safe and efficient execution of a series of operational tasks in uncertain and changing workspaces of an NPP. Before designing the ICAD-AR prototype, the project team synthesized NPP operational knowledge models through literature review studies, surveys, interviews with domain experts, and knowledge modeling. The project team conducted an extensive study of the operational procedures of various NPPs, and digital technologies that can support the safe and efficient execution of those procedures in different NPP operational contexts. This literature review helped the project team conduct surveys and interviews with nuclear engineers and field workers to identify three categories of information. The NPP knowledge modeling efforts reveal that the three categories of information identified have different levels of importance in a typical procedure of carrying out a series of tasks to achieve a specific NPP operation goal (e.g., shutdown, mode changes). These three categories of information include 1) Workspace dynamics – the changing spatial arrangements of workspaces, tools, protection equipment, and supporting materials, 2) Workflow prognostics – the dynamic dependencies between different parts of an NPP that functionally support and influence each other in terms of safety and efficiency, and 3) Hazards – objects and spaces that contain hazardous materials or physical conditions that can pose risks to workers or mechanical systems. The project team has profiled the importance levels of these categories of information into a knowledge model. This knowledge model specifies what types of information are more critical for a given task in a given workspace so that computers can automatically identify critical objects and sensors in a scene for delivering context-ware safety information to field workers through AR devices. Significant research development of this project results in technical research outcomes and a prototyping system that illustrates the technical feasibility of establishing an ICAD-AR system supporting the proactive safety information display for nuclear field workers. This final report summarizes the project team’s technological achievements in the past three years. Overall, the project team completed the development and integration of five techniques into a prototype ICAD Augmented Reality (ICAD-AR) system and demonstrated the developed system’s real-time execution in a mechanical room. The project team completed the analysis of using this prototype in other types of workspaces based on 3D image data and digital design models collected from two additional workspaces (a water treatment plant and a flow loop training facility). The integrated techniques include 1) Natural Language Processing (NLP) algorithms supporting the generation and updates of nuclear fieldwork process models based on text analysis of work packages and operation manuals; 2) sensor log analysis for predicting control actions in given sensor reading contexts; 3) computer vision algorithms for automatic localization and navigation of workers; 4) object detection algorithms for identifying task-related objects and correlated sensors for safety checking; 5) AR technique as a platform for supporting the integration. The testing results of these five techniques have shown that 1) the sensor log analysis model can predict the next control action with an accuracy of 0.883; 2) the trained natural language processing model can extract more than 80% of the critical information from paper-based procedures (PBPs); 3) the navigation algorithm with the integration of Visual Inertial Odometry (VIO) and Non-Recursive Bayesian Filter methods make operator’s trajectory estimation resilient to drift error; 4) the computer vision algorithm can detect task-specific and safety-critical objects with an average accuracy of 95.3%. The project team used work procedures collected from a flow loop training facility and two datasets collected from two mechanical rooms simulating the workspaces of NPPs to demonstrate the technical capabilities of the developed ICAD-AR prototype. The demonstration validated the technical feasibility of establishing the ICAD-AR system for nuclear field workers and identified the challenges in 1) automatic text analysis of work packages; 2) use of limited samples of sensor logs for predicting the proper timings of control actions; 3) reliably tracking workers and their task progress in mechanical rooms with many similar objects.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗

Internship in Augmented and Virtual Reality - Rapid Model Import Tool

The integration of virtual and augmented reality, sometimes called mixed reality, is an emerging technology which will likely skyrocket overnight much in the way smartphones did a decade ago. Kennedy Space Center's Augmented and Virtual Reality (AVR) Lab is developing a Rapid Model Import Tool (RMIT) to create a quick and efficient way to bring NASA's complex engineering 3D models into virtual and augmented environments. The long-term objective is to create a tool that will ultimately benefit KSC engineers. Its various uses within NASA can potentially span from astronaut training, to marketing, to public outreach, to name a few. Unity is a prolific cross-platform game engine that allows users to build high quality 2D and 3D games for desktop, mobile, web, and game console platforms. It is perhaps also the most widely used software for virtual reality game development. At the AVR lab, we are looking at alternative uses of Unity to build tools for NASA engineers to perform design, development, testing, and training on spacecraft, rocket delivery systems, ground support equipment, and facilities at KSC. As an intern for the RMIT project, I am charged with the task of performing research on Unity-compatible file types to develop an efficient, affordable, preservative process to bring models from CATIA 3D engineering software into the Unity environment. With a tool called the NASA Enterprise Visualization Application (NEVA), developed by the Boeing Design Visualization group at KSC, we are able to easily convert CATIA's design models to. DAE (also known as COLLADA) and .OBJ file formats. I first reduce the polygon count of the model within CATIA itself, make any necessary tweaks to reduce the model further, and then export using NEVA. The .OBJ or. DAE files that I am left with are then converted by another intern to a Unity-compatible file format using a custom Python script. I have generated extensive documentation of this process in a NEVA User Guide. By the end of this semester, we will have built a solid framework for RMIT based on a thorough understanding of virtual reality specifications and file requirements, allowing future software development teams to go forward with development on the custom tool.

Leap Motion↗

Performance Evaluation of Augmented Teleoperation of Contact Manipulation Tasks

At Argonne National Laboratory, an enhanced telerobotic concept has been suggested by providing augmented virtual fixtures for precise and dexterous manipulation. The proposed technology incorporated a human-robot interface that can assist personnel on the processes of Deactivation and Decommissioning in sites of high radiation. It has been implemented on a dual-arm robot system, known as Baxter, by integrating the Robotic Operating System (ROS) with 3D sensors and visual-haptic operator interface. However, what remained before its practical utilization is the capability for precisely placing the virtual fixtures. To this end, the objectives involved camera calibration, which required hardware fabrication and setup, as well as modification and implementation of ROS. The goal of this Research and Development is to develop an augmented-reality- based operator interface for human-inclusive, intelligent, and autonomous robotic system for the Deactivation and Decommissioning (D and D) of radioactive sites. The objectives are to develop: Collaborative Robot; Multi-modal Mixed-Reality Human-Robot Interface; Intelligent and Autonomous Behaviors. In order to achieve this, Baxter's Head-Kinect camera and Left-Hand camera need to be calibrated to overlap the real environment with the simulated environment. Camera calibration, which is crucial for precisely placing the virtual fixtures, was successful after running the simulations. The real and simulated environments overlapped, permitting the utilization of augmented teleoperation and the dexterous manipulation of Baxter for D and D applications in nuclear sites. Future steps: Performance evaluation of augmented teleoperation of contact manipulation tasks: Multiple-Camera 3D Reconstruction and Automation. Oculus Interface: Augmented reality and machine learning for enhanced telerobotic operation (Augmented Tele-autonomy): Machine-learning techniques for complex and flexible autonomous robotic behaviors. Interactive remote operation of robotic systems. Mesh cutting with simulation and real robot.

12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗

Human-Robot Interaction

Human-robot interaction (HRI) is a discipline investigating the factors affecting the interactions between humans and robots. It is important to evaluate how the design of interfaces affect the human's ability to perform tasks effectively and efficiently when working with a robot. By understanding the effects of interface design on human performance, workload, and situation awareness, interfaces can be developed to appropriately support the human in performing tasks with minimal errors and with appropriate interaction time and effort. Thus, the results of research on human-robot interfaces have direct implications for the design of robotic systems. For efficient and effective remote navigation of a rover, a human operator needs to be aware of the robot's environment. However, during teleoperation, operators may get information about the environment only through a robot's front-mounted camera causing a keyhole effect. The keyhole effect reduces situation awareness which may manifest in navigation issues such as higher number of collisions, missing critical aspects of the environment, or reduced speed. One way to compensate for the keyhole effect and the ambiguities operators experience when they teleoperate a robot is adding multiple cameras and including the robot chassis in the camera view. Augmented reality, such as overlays, can also enhance the way a person sees objects in the environment or in camera views by making them more visible. Scenes can be augmented with integrated telemetry, procedures, or map information. Furthermore, the addition of an exocentric (i.e., third-person) field of view from a camera placed in the robot's environment may provide operators with the additional information needed to gain spatial awareness of the robot. Two research studies investigated possible mitigation approaches to address the keyhole effect: 1) combining the inclusion of the robot chassis in the camera view with augmented reality overlays, and 2) modifying the camera frame of reference. The first study investigated the effects of inclusion and exclusion of the robot chassis along with superimposing a simple arrow overlay onto the video feed of operator task performance during teleoperation of a mobile robot in a driving task. In this study, the front half of the robot chassis was made visible through the use of three cameras, two side-facing and one forward-facing. The purpose of the second study was to compare operator performance when teleoperating a robot from an egocentric-only and combined (egocentric plus exocentric camera) view. Camera view parameters that are found to be beneficial in these laboratory experiments can be implemented on NASA rovers and tested in a real-world driving and navigation scenario on-site at the Johnson Space Center.

Sandor, Aniko↗

The Next Wave: Humans, Computers, and Redefining Reality

The Augmented/Virtual Reality (AVR) Lab at KSC is dedicated to " exploration into the growing computer fields of Extended Reality and the Natural User Interface (it is) a proving ground for new technologies that can be integrated into future NASA projects and programs." The topics of Human Computer Interface, Human Computer Interaction, Augmented Reality, Virtual Reality, and Mixed Reality are defined; examples of work being done in these fields in the AVR Lab are given. Current new and future work in Computer Vision, Speech Recognition, and Artificial Intelligence are also outlined.

Extended Reality↗

Visual Stability of Objects and Environments Viewed through Head-Mounted Displays

Virtual Environments (aka Virtual Reality) is again catching the public imagination and a number of startups (e.g. Oculus) and even not-so-startup companies (e.g. Microsoft) are trying to develop display systems to capitalize on this renewed interest. All acknowledge that this time they will get it right by providing the required dynamic fidelity, visual quality, and interesting content for the concept of VR to take off and change the world in ways it failed to do so in past incarnations. Some of the surprisingly long historical background of the technology that the form of direct simulation that underlies virtual environment and augmented reality displays will be briefly reviewed. An example of a mid 1990's augmented reality display system with good dynamic performance from our lab will be used to illustrate some of the underlying phenomena and technology concerning visual stability of virtual environments and objects during movement. In conclusion some idealized performance characteristics for a reference system will be proposed. Interestingly, many systems more or less on the market now may actually meet many of these proposed technical requirements. This observation leads to the conclusion that the current success of the IT firms trying to commercialize the technology will depend on the hidden costs of using the systems as well as the development of interesting and compelling content.

virtual environment↗

Usability Evaluation of a Flight-Deck Airflow Hazard Visualization System

Many aircraft accidents each year are caused by encounters with unseen airflow hazards near the ground, such as vortices, downdrafts, low level wind shear, microbursts, or turbulence from surrounding vegetation or structures near the landing site. These hazards can be dangerous even to airliners; there have been hundreds of fatalities in the United States in the last two decades attributable to airliner encounters with microbursts and low level wind shear alone. However, helicopters are especially vulnerable to airflow hazards because they often have to operate in confined spaces and under operationally stressful conditions (such as emergency search and rescue, military or shipboard operations). Providing helicopter pilots with an augmented-reality display visualizing local airflow hazards may be of significant benefit. However, the form such a visualization might take, and whether it does indeed provide a benefit, had not been studied before our experiment. We recruited experienced military and civilian helicopter pilots for a preliminary usability study to evaluate a prototype augmented-reality visualization system. The study had two goals: first, to assess the efficacy of presenting airflow data in flight; and second, to obtain expert feedback on sample presentations of hazard indicators to refine our design choices. The study addressed the optimal way to provide critical safety information to the pilot, what level of detail to provide, whether to display specific aerodynamic causes or potential effects only, and how to safely and effectively shift the locus of attention during a high-workload task. Three-dimensional visual cues, with varying shape, color, transparency, texture, depth cueing, and use of motion, depicting regions of hazardous airflow, were developed and presented to the pilots. The study results indicated that such a visualization system could be of significant value in improving safety during critical takeoff and landing operations, and also gave clear indications of the best design choices in producing the hazard visual cues.

Aragon, Cecilia R.↗

Visualizing an Exascale Data Center Digital Twin: Considerations, Challenges and Opportunities

Digital twins are an excellent tool to model, visualize, and simulate complex systems, to understand and optimize their operation. In this work, we present the technical challenges of real-time visualization of a digital twin of the Frontier supercomputer.We show the initial prototype and current state of the twin and highlight technical design challenges of visualizing such a large High Performance Computing (HPC) system. The goal is to understand the use of augmented reality as a primary way to extract information and collaborate on digital twins of complex systems. This leverages the spatio-temporal aspect of a 3D representation of a digital twin, with the ability to view historical and real-time telemetry, triggering simulations of a system state and viewing the results, which can be augmented via dashboards for details. Finally, we discuss considerations and opportunities for augmented reality of digital twins of large-scale, parallel computers.

Maiterth, Matthias↗

Time Counts! Some Comments on System Latency in Head-Referenced Displays

System response latency is a prominent characteristic of human-computer interaction. Laggy systems are; however, not simply annoying but substantially reduce user productivity. The impact of latency on head referenced display systems, particularly head-mounted systems, is especially disturbing since not only can it interfere with dynamic registration in augmented reality displays but it also can in some cases indirectly contribute to motion sickness. We will summarize several experiments using standard psychophysical discrimination techniques that suggest what system latencies will be required to achieve perceptual stability for spatially referenced computer-generated imagery. In conclusion I will speculate about other system performance characteristics that I would hope to have for a dream augmented reality system.

human-computer interaction↗

HydrogenAR: Interactive Data-Driven Presentation of Dispenser Reliability: Preprint

When delivering presentations to a co-located audience, we typically use slides with text and 2D graphics to complement the spoken narrative. Though presentations have largely been explored on 2D media, augmented reality (AR) allows presentation designers to add data and augmentations to existing physical infrastructure on display. This could provide a more engaging experience to the audience and support comprehension. With HydrogenAR, we present a novel application that leverages the benefits of data-driven storytelling with those of AR to explain the unique challenges of hydrogen dis- penser reliability. Utilizing physical props, situated data, and virtual augmentations and animations, HydrogenAR serves as a unique presentation tool, particularly critical for stakeholders, tour groups, and VIPs. HydrogenAR is a product of multiple collaborative de- sign iterations with a local Hydrogen Fuel research team and is evaluated through interviews with team members and a user study with end-users to evaluate the usability and quality of the interactive AR experience. Through this work, we provide design considera- tions for AR data-driven presentations and discuss how AR could be used for innovative content delivery beyond traditional slide-based presentations.

augmented reality↗