Search NASA⌕ Search

SEARCH · Search NASA

Results for “Extended Reality”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 37 records · Page 2

The Future of Visualization Technology Vis in Practice

Two decades ago, we witnessed an inflection point as graphics programming became commoditized. A comparable inflection point may be on the horizon, this time driven by the commoditization of immersive display technology with Augmented Reality (AR), Virtual Reality (VR), and Extended Reality (XR) technologies poised to redefine our interaction with data, environments, and collaboration. To drive the technology and its application, we should consider the lessons learned from successful large-scale immersive installations. Where these installations have thrived is by enhancing traditional workflows rather than entirely replacing them, supporting improved judgments of complex spaces, direct interaction in 3D, and embedding high-dimensional data.

AR/VR↗

Using XR for Improving Scientific Discovery With Numerical Weather Models

Our work explores the use of extended reality (XR) to im- prove scientific discovery with numerical weather/climate models that inform Earth science digital twins, specifically the NASA Goddard Earth Observing System (GEOS) global atmospheric model. The overall project is named the Vi- sualization And Lagrangian dynamics Immersive eXtended Reality Toolkit (VALIXR), which has two main areas of focus: (1) enhancing the understanding of and interaction with model output data through advanced visualizations in the XR environment, and (2) the integration of Lagrangian dynamics into the GEOS model, which allows a natural, feature-specific analysis of Earth science phenomena as op- posed to traditional, fixed-point Eulerian dynamics. Here, we report initial work on these focus areas.

Thomas Grubb↗

MenuFramework

The Menu Framework is a reusable Extended Reality multi-level menu system for the Unity Development Platform. The menu structure is architected for reusability to suit whatever need the developer may have for a multi-level, reusable menu framework. Menu Framework's architecture, GUI, and functionality are based on the paper "Extended Reality for Enhanced Training and Knowledge Capture" (SAND2020-2361C). This tool uses the Microsoft Mixed Reality ToolKit (MRTK) built in functionality to provide a suite of functionalities including bounding box, follow me, minimize, maximize, settings, and help commands. Sandia National Laboratories is a multimission laboratory managed and operated by National Technology & Engineering Solutions of Sandia, LLC, a wholly owned subsidiary of Honeywell International Inc., for the U.S. Department of Energy’s National Nuclear Security Administration under contract DE-NA0003525. SAND2021-1507 O

Klein, BrandonThorin↗

The Next Wave: Humans, Computers, and Redefining Reality

The Augmented/Virtual Reality (AVR) Lab at KSC is dedicated to " exploration into the growing computer fields of Extended Reality and the Natural User Interface (it is) a proving ground for new technologies that can be integrated into future NASA projects and programs." The topics of Human Computer Interface, Human Computer Interaction, Augmented Reality, Virtual Reality, and Mixed Reality are defined; examples of work being done in these fields in the AVR Lab are given. Current new and future work in Computer Vision, Speech Recognition, and Artificial Intelligence are also outlined.

Extended Reality↗

Immersive Digital Twin Laboratory for Engineering Education (CRADA Final Report)

This project aimed to create an immersive digital twin laboratory that incorporates advanced tracking and visualization capabilities. In collaboration with Fort Lewis College, NREL designed a state-of-the-art physical visualization laboratory, developed a software platform to enable interaction with tracked physical objects in the laboratory, and provided proof-of-concept curricula that included manipulating these tracked objects. The project was initiated to address the growing need for innovative educational tools in engineering education. As renewable energy systems, particularly solar installations, become more complex, there is a pressing need to bridge the gap between theoretical knowledge and practical application. Traditional methods of teaching solar engineering concepts often fall short of providing students with a comprehensive, hands-on understanding. This immersive digital twin laboratory was conceived to fill that gap by creating a safe, non-energized setting where students can interact with augmented solar installation objects, gaining valuable insights into system performance, design, and maintenance. The project utilized extended reality (XR) technologies, including head-mounted displays (HMDs) and a whole-room optical motion tracking system, to connect physical objects with their digital twins in real time. The laboratory was equipped with MagicLeap 2 HMDs, supported by a Vicon Vero 2.2 Optical Tracking System, which provided precise 6-degrees-of-freedom (6-DOF) tracking. We developed a software platform to manage the interaction between the tracked physical objects and their virtual counterparts, enabling real-time data synchronization, object recognition, and virtual overlays. We designed the system to be flexible and extendable, allowing for future integration of additional objects and curriculum. This research advances the field of engineering education by demonstrating the potential of immersive digital twin environments. The laboratory provides a dynamic learning space where students can experiment, collaborate, and learn without the risks associated with live experimentation. The ability to simulate and manipulate solar installation objects under various conditions has broad implications for workforce development, particularly in renewable energy. The project also highlights the economic feasibility of using XR technologies in educational settings, offering a cost-effective solution for institutions looking to enhance their curriculum. By fostering a deeper understanding of solar energy systems, this work contributes to the broader goal of supporting the global energy transition and preparing the next generation of engineers and technicians.

24 POWER TRANSMISSION AND DISTRIBUTION↗

Interactive Quantum Chemistry Enabled by Machine Learning, Graphical Processing Units, and Cloud Computing

Modern quantum chemistry algorithms are increasingly able to accurately predict molecular properties that are useful for chemists in research and education. Despite this progress, performing such calculations is currently unattainable to the wider chemistry community, as they often require domain expertise, computer programming skills, and powerful computer hardware. In this review, we outline methods to eliminate these barriers using cutting-edge technologies. We discuss the ingredients needed to create accessible platforms that can compute quantum chemistry properties in real time, including graphical processing units–accelerated quantum chemistry in the cloud, artificial intelligence–driven natural molecule input methods, and extended reality visualization. We end by highlighting a series of exciting applications that assemble these components to create uniquely interactive platforms for computing and visualizing spectra, 3D structures, molecular orbitals, and many other chemical properties.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗

Early Fault Detection in Nuclear Systems: A Digital Engineering Approach

Nuclear energy systems present unique challenges in terms of ensuring safety, reliability, and efficiency during their design and operation. Early fault detection is critical for mitigating risks and fostering system resilience. However, current methods often fall short at identifying faults during early stages, potentially leading to costly delays and safety risks. The present work proposes a comprehensive digital engineering approach that leverages digital twins, digital threads, model-based systems engineering, artificial intelligence, and immersive extended reality to support early fault detection in nuclear systems. Through a series of case studies, we highlight specific gaps in the fault detection mechanisms of traditional nuclear design and operation processes, then demonstrate a suite of solutions we are working to implement to address these shortcomings in similar projects. Our findings suggest that a digital engineering approach to design and operation can significantly improve fault detection, ultimately leading to reductions in risk.

42 - ENGINEERING↗

Immersive Technologies for Human-in-the-Loop Lunar Surface Simulations

NASA, the National Aeronautics and Space Administration, continually seeks innovative solutions to enhance its operations, particularly in the realms of testing, evaluation, and training for future missions. Immersive technologies, such as virtual, augmented, and mixed reality have proven to be powerful tools for realistic, interactive, and engaging environments. This paper explores how the Simulation and Graphics Branch at NASA’s Johnson Space Center (JSC) leverages immersive technology, modern commercial rendering engines, and physics-based systems simulations to develop human-in-the-loop systems for humanity’s return to the Moon through the Artemis program. When NASA returns to the Moon, astronauts will travel to the Moon’s South Pole where lighting conditions will cause a more complex operational environment. Human-in-the-loop testing plays a crucial role in NASA's mission planning, spacecraft and space systems development, and evaluation of operational scenarios. The development of immersive environments such as a lunar rover mockup at a video wall enables engineers and astronauts to simulate and experience mission scenarios, integrated spacecraft systems, and operational procedures in a relevant environment before deployment. By integrating realistic virtual environments, immersive technology allows for the visualization and interaction with virtual spacecraft models, mission landscapes, and complex operational tasks. This approach helps identify potential design flaws, operational challenges, and safety considerations. It also provides valuable insights for risk reduction and helps improve mission efficiency and effectiveness. With advanced motion tracking systems and custom virtual environments data can be gathered and evaluated to help NASA refine training protocols, develop specialized training procedures and optimize human-robotic interactions for future space missions. Furthermore, immersive technology offers opportunities for future training initiatives at NASA. The Virtual Reality Laboratory at JSC has pioneered training with Virtual Reality (VR) since the Hubble Space Telescope repair missions in the early 1990’s. Extended Reality (XR) simulations enable astronauts to rehearse complex spacewalks, spacecraft maneuvers, and extravehicular activities in a safe and controlled environment. By replicating the physical and cognitive challenges of space missions, immersive training experiences enhance astronauts' situational awareness, decision-making abilities, and adaptability to unexpected scenarios. Additionally, immersive technology facilitates collaborative training, allowing geographically dispersed crew and mission control personnel to engage in synchronized simulations, fostering teamwork and effective communication. The adoption of immersive technology in NASA's testing, evaluation, and future training programs has yielded significant benefits. By incorporating human-in-the-loop testing for studies involving Extra Vehicular Activities (EVA), surface mobility and landing systems, NASA can identify and mitigate risks, optimize operational procedures, and enhance mission success. Ultimately, immersive training experiences can empower astronauts to better navigate the complexities of space missions, ensuring their safety, productivity, and success in the dynamic and challenging environments they will experience at the Lunar South Pole.

Simulation Modeling Virtual Reality Immersive Tech↗

Immersive Technologies for Human-in-the-Loop Lunar Surface Simulations

NASA, the National Aeronautics and Space Administration, continually seeks innovative solutions to enhance its operations, particularly in the realms of testing, evaluation, and training for future missions. Immersive technologies, such as virtual, augmented, and mixed reality have proven to be powerful tools for immersing users in realistic, interactive, and engaging environments. This paper explores how the Simulation and Graphics Branch at NASA’s Johnson Space Center (JSC) leverages immersive technology, modern commercial rendering engines, and physics-based systems simulations to develop human-in-the-loop systems for humanity’s return to the Moon through the Artemis program. When NASA returns to the Moon, astronauts will travel to the Moon’s South Pole where lighting conditions will cause a more complex operational environment. Human-in-the-loop simulations play a crucial role in NASA’s mission planning, spacecraft and space systems development, and evaluation of operational scenarios. The development of immersive environments such as a lunar rover mockup at a video wall enables engineers and astronauts to simulate and experience mission scenarios, integrated spacecraft systems, and operational procedures in a relevant environment before deployment. By integrating realistic virtual environments, immersive technology allows for the visualization and interaction with virtual spacecraft models, mission landscapes, and complex operational tasks. This approach helps identify potential design flaws, operational challenges, and safety considerations. It also provides valuable insights for risk reduction and helps improve mission efficiency and effectiveness. With advanced motion tracking systems and custom virtual environments, data can be gathered and evaluated to help NASA refine training protocols, develop specialized training procedures, and optimize human-robotic interactions for future space missions. Furthermore, immersive technology offers opportunities for future training initiatives at NASA. The Virtual Reality Laboratory at JSC has pioneered training with Virtual Reality (VR) since the Hubble Space Telescope repair missions in the early 1990’s. Extended Reality (XR) simulations enable astronauts to rehearse complex spacewalks, spacecraft maneuvers, and extravehicular activities in a safe and controlled environment. By replicating the physical and cognitive challenges of space missions, immersive training experiences enhance astronauts’ situational awareness, decision-making abilities, and adaptability to unexpected scenarios. Additionally, immersive technology facilitates collaborative training, allowing geographically dispersed crew and mission control personnel to engage in synchronized simulations, fostering teamwork and effective communication. The adoption of immersive technology in NASA’s testing, evaluation, and future training programs has yielded significant benefits. By incorporating human-in-the-loop simulations for studies involving Extra Vehicular Activities (EVA), surface mobility and landing systems, NASA can identify and mitigate risks, optimize operational procedures, and enhance mission success. Ultimately, immersive simulation experiences can empower astronauts to better navigate the complexities of space missions, ensuring their safety, productivity, and success in the dynamic and challenging environments they will experience at the Lunar South Pole.

HITL↗

Exploration ToolKit (ExTK)

The Exploration Toolkit (ExTK) is a reusable Extended Reality (XR) system developed for incorporating and exploring 3-Dimensional (3-D) computer aided design (CAD) models in XR, with a primary focus on Augmented Reality (AR). The ExTK consists of a Developer Mode and a User Mode. In Developer Mode, ExTK provides developers with the ability to easily import 3-D CAD models and activate desired exploration functionality and layout. Multiple models can be added to a single instantiation of the ExTK using Unity's Scene capability. Exploration functions include scaling, rotating, explode/contract, animations, hiding parts, submodules, and measurement functions. In User Mode, ExTK provides a menu system that allows users to select models and initiate exploration functions. ExTK is architected for reusability and developers can customize the ExTK layout and functions according to application needs. ExTK is designed to be hardware agnostic, although initial development focused on the Microsoft HoloLens as the primary deployment platform. The ExTK is developed using the Unity Game Engine Development Platform. Sandia National Laboratories is a multimission laboratory managed and operated by National Technology & Engineering Solutions of Sandia, LLC, a wholly owned subsidiary of Honeywell International Inc., for the U.S. Department of Energy’s National Nuclear Security Administration under contract DE-NA0003525. SAND2021-1506 O

Klein, BrandonThorin↗

Sequential Training ToolKit - STTK

Sequential Training ToolKit, or STTK, is a software toolkit plugin for the Unity development platform that enables software developers to easily develop structured Sequential Training applications for gaming and extended reality (XR) environments. STTK is architected to easily import the toolkit as a plugin for the Unity platform and then interact with the graphical user interface (GUI) for drag-and-drop functionality. STTK also includes easy add-on plugins for Prefabs and auto-builds a State Machine. Sandia National Laboratories is a multimission laboratory managed and operated by National Technology & Engineering Solutions of Sandia, LLC, a wholly owned subsidiary of Honeywell International Inc., for the U.S. Department of Energy’s National Nuclear Security Administration under contract DE-NA0003525. SAND2021-3059 O

Klein, BrandonThorin↗

Toward a Smart Metaverse City: Immersive Realism and 3D Visualization of Digital Twin Cities

Metaverse and its related extended reality technologies can enable immersive, realistic, and participatory visualization of 3D data, and their use and the potential within a smart city can be effective for supporting urban research and urban operations management. This book chapter describes a vision for prototyping a “Smart Metaverse City” to combine the unique advantage of the Metaverse technology with the two-way connectivity of a digital twin city application. This synergy aims to create a virtual environment for immersive geovisualization to help researchers and the public understand the complex urban system through science-based and data-driven approaches. This book chapter selectively reviews past technological and paradigm advancements for collecting, analyzing, and visualizing 3D urban big data. Then we present a prototyping Geographic Information System (GIS) framework, together with some relevant data sources and open-source web technologies, to help researchers create a smart Metaverse city. We demonstrate our vision and discuss its application opportunities through a real-world example, a digital twin city developed at the Oak Ridge National Laboratory, to facilitate participatory, smart, and sustainable campus management.

Xu, Haowen↗

Complex Adaptive System of Systems (CASoS) Engineering Applications (V.1.0)

Complex Adaptive Systems of Systems, or CASoS, are vastly complex eco-socio-economic-technical systems which we must understand to design a secure future for the nation and the world. Perturbations/disruptions in CASoS have the potential for far-reaching effects due to highly-saturated interdependencies and allied vulnerabilities to cascades in associated systems. The Phoenix initiative approaches this high-impact problem space as engineers, devising interventions (problem solutions) that influence CASoS to achieve specific aspirations. CASoS embody the world's biggest problems and greatest opportunities: applications to real world problems are the driving force of our effort. We are developing engineering theory and practice together to create a discipline that is grounded in reality, extends our understanding of how CASoS behave, and allows us to better control those behaviors. Through application to real-world problems, Phoenix is evolving CASoS Engineering principles while growing a community of practice and the CASoS engineers to populate it.

42 ENGINEERING↗

Complex Adaptive Systems of Systems (CASOS) Engineering Environment

Complex Adaptive Systems of Systems, or CASoS, are vastly complex physical-socio-technical systems which we must understand to design a secure future for the nation. The Phoenix initiative implements CASoS Engineering principles combining the bottom up Complex Systems and Complex Adaptive Systems view with the top down Systems Engineering and System-of-Systems view. CASoS Engineering theory and practice must be conducted together to develop a discipline that is grounded in reality, extends our understanding of how CASoS behave and allows us to better control the outcomes. The pull of applications (real world problems) is critical to this effort, as is the articulation of a CASoS Engineering Framework that grounds an engineering approach in the theory of complex adaptive systems of systems. Successful application of the CASoS Engineering Framework requires modeling, simulation and analysis (MS and A) capabilities and the cultivation of a CASoS Engineering Community of Practice through knowledge sharing and facilitation. The CASoS Engineering Environment, itself a complex adaptive system of systems, constitutes the two platforms that provide these capabilities.

42 ENGINEERING↗

Game Based Learning For Earth Science Applications Training

Current NASA Earth capacity development programs employ mechanisms ranging from online resource sharing, and virtual and in-person trainings to share knowledge. While these programs are highly successful at engaging individuals around the world – in 2018, over 8000 individuals and over 2000 institutions from all 50 US states and over 140 countries were engaged through over 150 projects and trainings – user feedback has highlighted the desire for expanded hands-on, practical experiences in incorporating NASA EO insights with localized data and actions. We aim to address this gap by leveraging the benefits of game-based learning to build user skills in integrating NASA and local EO data to guide decisions for climate resiliency and hazard planning. This project is being executed as a two-phase crowdsourced challenge: 1) Phase 1 will require a well-researched product concept that reflects an understanding of NASA’s Earth data and tools and user needs, and proposes an innovative and interactive game or extended reality experience to train users in identifying relevant NASA data and applying insights to their climate resiliency decisions; 2) Winners of Phase 1 will be provided seed funding to develop a working prototype of the product. We aim to award 1-3 final winners to support the development of more than one game, thereby ensuring that NASA's diverse audiences around the world can access training games that best suit their needs and capabilities. This EO training game project fits in the NASA Earth Science Applied Sciences Program’s Capacity Development Program, contributing to the program mission of “helping people around the world better understand [NASA’s Earth] data and find ways to use them” (https://appliedsciences.nasa.gov/what-we-do/capacity-building). The final training game will complement existing programmatic activities of workforce development, trainings, and collaborative projects, while providing the unique value of providing interactive experiences to users and collecting real-time data and feedback to improve NASA’s Earth applications’ products and services related to climate resilience.

Human centered design↗

Earth Independent Medical Operations (EIMO) Datascope: Challenges and Potential Solutions

Data flows and storage/retrieval capacity are severely constrained during missions in space and challenges will become even greater during exploration class missions. There is a need for an artificial intelligence (AI)-based clinical decision support system (CDSS) to monitor and analyze data to provide real-time consultative support for crew medical officer (CMO) decision-making. EIMO is defined as the gradual transition of medical care and decision making from terrestrial to space-based assets, enabling support of astronaut health and performance and reducing overall mission risk. While a hallmark of this paradigm shift from low-earth orbit is that on-board care will increasingly become the responsibility of the astronauts for primary management and decision making, terrestrial assets will continue to be paramount in pre-mission screening and planning, as well as prevention, health maintenance and long-term care contingencies. New capabilities and systems that enable progressively more robust and resilient systems and crews will be necessary to reduce risk and increase probability of deep space exploration mission success. An aspiration for EIMO is to develop AI-enhanced solutions for analysis of crew health & performance data and to facilitate clinical decision support for autonomous medical operations. A “system of systems” approach is envisioned whereby EIMO will deploy AI-supported natural language processing and machine learning (ML) techniques to utilize embedded reference databases and real-time data streams [input vectors] from multiple data sources. Constituent input vectors may include environmental controls, countermeasures data, behavioral data, physiologic wearables, point-of-care laboratory tests, personalized medical records, inventory trade space risk assessments, COTS medical databases, and ground support inputs. An ideal AI capability would possess trained fusion algorithms to cross reference input vectors with medical ‘knowledge’ [cultivated database] to stratify relevant data streams for predictive and actionable capabilities. In addition, EIMO will feature mobility, in that it can be accessed and can push/pull data within and between multiple vehicles/habitats. Large amounts and variable sources of data can be leveraged to diagnose, inform treatment strategies, and potentially predict medical events and performance decrements. Inclusion of advanced training tools using extended reality will enable increasingly autonomous medical care to aid a CMO when ground support is unavailable or time-delayed beyond required action window, e.g., emergent medical situations. EIMO CDSS would require very large datasets to train pre-flight and significant amounts of data are needed to support ML via in-flight CDSS operations. An additional challenge will be to find sufficient data to train a model relevant to astronaut demographics. The rapid, accelerating evolution of this field creates a propitious solution space to leverage multi-modal AI through public-private partnership(s). The status of multi-modal AI systems today would preclude their use for long duration missions as they remain unreliable and are subject to “digital hallucinations” and other errors that could pose operational risk. A federated labs structure is being considered to test and optimize data flow from the multiple input vectors leading to field testing in suitable ground/flight analogs. Critical to the success of an EIMO CDSS will be integration and interoperability and success will be defined by a system that can serve as an in-flight medical consult for the CMO providing critical support during medical contingencies. Benefits to terrestrial medicine may be significant as an outflow of the EIMO medical system, particularly for remote areas and communities lacking significant infrastructure, personnel and resources.

J Lemery↗