Results for “Human-In-The-Loop” — Page 24

NASA NTRSDate not supplied

Effects of Communication Modality on Pilot-Controller Coordination during a Simulated m:N Operation

The last decade or so has seen growing interest in new control paradigms and concepts of operation for uncrewed aircraft systems (UAS) in which multiple aircraft are piloted remotely by a single or relatively small number of people. Referred to as “one-to-many” and “many-to- many” (alternatively, “multi-operator, multi-vehicle”)—and frequently expressed as the corresponding ratios, 1:N and m:N—such novel configurations of aircraft and the people who manage them are seen as critical to the path to future operations involving UAS. Examples of industry domains interested in these control paradigms are small package delivery services utilizing small UAS and passenger-carrying, short-range “Urban Air Mobility” (UAM) operations. Stakeholders in such operations have identified communication and coordination of flight activity with air traffic controllers (ATC) as a barrier to operations. In contrast to present-day flight operations, in which a pilot communicates with one ATC on one radio frequency for one aircraft, multi-vehicle operations potentially entail a significant increase in pilot task load for management of comms. New concepts, such as UAS Service Suppliers (USSs) and Providers of Services to UAM (PSUs), have been proposed to address the known bottleneck for Air Traffic Management (ATM) presented by multi-vehicle operations. While progress has been steadily made over years developing USSs and PSUs, it is generally expected that initial UAM operations will rely on traditional voice-over-radio communication with ATC for purposes of ATM. The current study was a human-in-the-loop simulation that had participants, each possessing a Private Pilot License, act as the ground-based pilot-in- command for multiple vehicles in a hypothetical UAM service in the San Francisco Bay Area. The experiment utilized a 2-by-3, within-subjects design in which the pilot’s Vehicle Load (4 vs. 12) and Comm System (Voice, Datalink, and a Hybrid) were manipulated. The task given to pilots was to use the Comm System to coordinate flight activity for all aircraft with appropriate controllers, having to obtain departure and arrival clearances at “vertiport” facilities and transition clearances for any intermediate airspaces along the route. Pilots were additionally responsible for compliance with vectoring instructions issued by ATC. Subjective workload questionnaires (NASA-TLX) were administered following each experimental trial. Screen recordings of the pilot’s Ground Control Station (GCS) and audio recordings of trials were subsequently coded to obtain performance metrics: response times and error rates. Presented in this paper are results related to pilot responses to vectoring instructions issued by ATC. Workload was found to be significantly higher in the 12-Vehicle condition compared to the 4-Vehicle condition, nearly maxing out the NASA-TLX overall workload scale. There was no significant difference made by the Comm System on workload ratings. Pilots’ response times to communications were fastest in the Voice condition, although overall “service time” for compliance was shorter in Datalink and Hybrid conditions in most cases. Errors by pilots were frequent in both Vehicle Load conditions, most perniciously when using the Voice system. The results of this study suggest tradeoffs in advantages and disadvantages of the three comm systems. Recommendations for communication system design are provided taking the tradeoffs into account.

urban air mobility↗

NASA NTRS2023

Using Virtual Reality to Envision Deployment of Spacesuit-Compatible Augmented Reality Displays for Lunar Surface Operations

The National Aeronautics and Space Administration (NASA) aims to land crew on the lunar surface to establish a sustainable presence and develop operational concepts for future long-duration missions. New technologies will be necessary to extend planning and execution capabilities for lunar surface activities. NASA’s Joint Augmented Reality Visual Informatics System (Joint AR) is one such technology. Joint AR is a suit-mounted augmented reality (AR) display and computes system which facilitates unprecedented information exchange and data visualization capabilities between mission support operators and suited crew. This paper describes challenges associated with developing AR technology for an envisioned work domain by applying a sociotechnical lens to the iterative testing and development of novel AR technology through virtual reality (VR). A VR testbed was established to simulate a representative lunar surface environment, enabling a series of three human-in-the-loop (HITL) tests evaluating AR navigation interfaces for exploration extravehicular activity (xEVA). Our findings identify several considerations for future Joint AR design and testing efforts, including challenges with data overload, attentional demands, and environment-related perceptual challenges. Trade-offs and potential approaches are discussed to mitigate these challenges and improve future Joint AR testing fidelity.

Jacob Keller↗

NASA NTRSDate not supplied

Helicopter Pilot Assessments of the Airborne Collision Avoidance System XR With Automated Maneuvering

The Airborne Collision Avoidance System X (ACAS X) is a next-generation collision avoidance tool developed to support different aircraft types and operations. The rotorcraft variant – referred to as ACAS XR – is designed to accommodate existing helicopter platforms as well as in-development, electric vertical takeoff and landing concepts, which are critical to the emerging concept of operations referred to as Advanced Air Mobility. The fundamental role of ACAS XR is to provide Detect and Avoid (DAA) and/or Collision Avoidance (CA) protection against airborne traffic. DAA alerting and guidance in the context of ACAS XR is caution-level and “suggestive,” and is to be used by the pilot if, and when, they decide to maneuver against an identified threat to DAA “well clear.” The CA alerting, by contrast, is warning-level and “directive,” with the pilot required to comply with the associated guidance to prevent a predicted Near Midair Collision (NMAC). The CA alerts generated by ACAS XR are referred to as Resolution Advisories (RAs) consistent with previous CA systems. Unlike earlier CA systems, ACAS XR issues RAs in the horizontal and vertical dimensions as well as multi-axis RAs (referred to as “Blended” RAs). According to the Minimal Operational Performance Standards of DAA systems for Unmanned Aircraft Systems, maneuvers to comply with RAs may be automated, whereas maneuvers based on DAA alerting assume a manual response. The current study was a human-in-the-loop simulation that presented rotorcraft pilots with ACAS XR alerts and guidance in a fixed-base eVTOL simulator with varying levels of automation. Objective results showed that pilots complied with all RAs within the expected 5-second time window and responded to DAA alerts quicker than in earlier studies with ACAS XU. Pilots often made larger horizontal deviations during Manual RAs, but often favored vertical and blended maneuvers. No NMACs occurred, and losses of well clear were mainly attributed to the obligation of the pilots and system to maneuver only after the CA phase of the encounter had begun. Other losses were due to pilots’ noncompliance or disregard for ACAS XR’s alerting and guidance. Noncompliance with RAs most frequently occurred when pilots determined they were too close to the terrain to continue to follow Descend RAs, performing vertical maneuvers instead of following Horizontal RAs, or rejecting Horizontal RA updates because they felt that enough maneuvering had been performed. Subjectively, pilots found the DAA and RA alerting and guidance intuitive and useful for VFR helicopter operations. Slightly more pilots preferred the Automated RA condition to the Manual RA condition. Lastly, they also felt that ACAS led to occasional unsafe Descend RAs. Caveats and future implications are discussed.

eVTOL↗

NASA NTRSDate not supplied

Helicopter Pilot Assessments of the Airborne Collision Avoidance System XR With Automated Maneuvering

The Airborne Collision Avoidance System X (ACAS X) is a next-generation collision avoidance tool developed to support different aircraft types and operations. The rotorcraft variant – referred to as ACAS XR – is designed to accommodate existing helicopter platforms as well as in-development, electric vertical takeoff and landing concepts, which are critical to the emerging concept of operations referred to as Advanced Air Mobility. The fundamental role of ACAS XR is to provide Detect and Avoid (DAA) and/or Collision Avoidance (CA) protection against airborne traffic. DAA alerting and guidance in the context of ACAS XR is caution-level and “suggestive,” and is to be used by the pilot if, and when, they decide to maneuver against an identified threat to DAA “well clear.” The CA alerting, by contrast, is warning-level and “directive,” with the pilot required to comply with the associated guidance to prevent a predicted Near Midair Collision (NMAC). The CA alerts generated by ACAS XR are referred to as Resolution Advisories (RAs) consistent with previous CA systems. Unlike earlier CA systems, ACAS XR issues RAs in the horizontal and vertical dimensions as well as multi-axis RAs (referred to as “Blended” RAs). According to the Minimal Operational Performance Standards of DAA systems for Unmanned Aircraft Systems, maneuvers to comply with RAs may be automated, whereas maneuvers based on DAA alerting assume a manual response. The current study was a human-in-the-loop simulation that presented rotorcraft pilots with ACAS XR alerts and guidance in a fixed-base eVTOL simulator with varying levels of automation. Objective results showed that pilots complied with all RAs within the expected 5-second time window and responded to DAA alerts quicker than in earlier studies with ACAS XU. Pilots often made larger horizontal deviations during Manual RAs, but often favored vertical and blended maneuvers. No NMACs occurred, and losses of well clear were mainly attributed to the obligation of the pilots and system to maneuver only after the CA phase of the encounter had begun. Other losses were due to pilots’ noncompliance or disregard for ACAS XR’s alerting and guidance. Noncompliance with RAs most frequently occurred when pilots determined they were too close to the terrain to continue to follow Descend RAs, performing vertical maneuvers instead of following Horizontal RAs, or rejecting Horizontal RA updates because they felt that enough maneuvering had been performed. Subjectively, pilots found the DAA and RA alerting and guidance intuitive and useful for VFR helicopter operations. Slightly more pilots preferred the Automated RA condition to the Manual RA condition. Lastly, they also felt that ACAS led to occasional unsafe Descend RAs. Caveats and future implications are discussed.

eVTOL↗

NASA NTRSDate not supplied

Desert Research and Technology Studies (D-RATS) 2022 Quicklook Report

This report summarizes the Desert Research and Technology Studies (D-RATS) 2022 analog tests. BACKGROUND - Artemis Challenges – NASA’s concept of operations (ConOps) for the Artemis mission architecture brings new challenges for human exploration of the lunar surface, including: (1) Low-angle, natural lighting at lunar poles; and (2) Exploration sites that challenge communication with Earth. - International Partner Involvement – NASA is working with the Japan Aerospace Exploration Agency (JAXA) to scope mission & functional requirements for an Artemis Pressurized Rover (PR), which JAXA may provide. - Charter – HQ Exploration Systems Development Mission Directorate (ESDMD) Moon to Mars Architecture Development Office (M2MADO) Strategy and Architectures (SA) chartered the Human-in-the-Loop (HITL) test team to investigate Artemis architectural questions related to pressurized rover ConOps. - Rationale – to inform the NASA/JAXA pressurized rover study-agreement. PLAN - Objectives – Analog tests conducted in October 2022 by the D-RATS team addressed three high-level objectives: 1. Investigate pressurized rover (PR) ConOps and capabilities for Artemis exploration 2. Integrate with JAXA engineers & astronauts and incorporate JAXA PR design elements into testing. 3. Re-establish analog field-testing skills & capabilities with rovers to investigate Artemis architecture ConOps. - Secondary Objectives – Work with other groups to leverage D-RATS field test for additional objectives. 4. Work with the Public Affairs Office (PAO) to perform D-RATS public outreach activities. 5. Coordinate with the Human Physiology Performance Protection & Operations (H-3PO) team to facilitate in-field evaluation of human health and performance (HHP) objectives. 6. Share D-RATS field-site and assets with Lunar LTE Studies (Lunar LiTES) team, to aid their study of the use of 4G/LTE communication protocols and devices for astronauts and robotic nodes on the lunar surface. - Team – Fully integrated test team comprised of members from 5 NASA centers, JAXA, and the United States Geological Survey (USGS) - Location – Black Point Lava Flow, ~40 miles north of Flagstaff, AZ HIGH-LEVEL OBJECTIVES ACCOMPLISHED - Investigated Pressurized Rover ConOps & Capabilities for Artemis Exploration (Objective 1) - Completed testing with 4 crew pairs, each spending 3 days and 2 nights in the rover conducting Artemis PR dayin-the-life activities (2 JAXA astronauts, 2 JAXA engineers, 1 NASA astronaut, 3 NASA engineers). - Collected detailed objective & subjective data supporting 10 strategic questions related to Artemis PR operations. - Field geologists present in field observed rover operations & EVAs. - Science team in Houston MCC communicated directly with crew. - Demonstrated crew-led and MCC-led PR teleoperation use cases during EVAs. - Integrated with JAXA Engineers & Astronauts and Incorporated JAXA PR Design Elements into Testing (Objective 2) - NASA & JAXA engineers, flight controllers, scientists, roboticists, and astronauts directly participated in and/or observed testing both in field and in MCC-Houston. - Incorporated JAXA PR design elements into both integrated and standalone testing at JSC and in the field. - Re-established Analog Field-Testing Skills & Capabilities with Rovers to Investigate Artemis Architecture ConOps (Objective 3) - Multiple teams successfully worked to establish and manage field-test base camp, monitor and maintain the rover, and plan and execute 2 weeks of consecutive field-testing with little to no breaks between crews. TEST OUTCOMES - Results will inform Artemis architecture ConOps & capabilities related to pressurized rover operations (see sections 2 for more details) - Summary and team detailed reports will be posted on the D-RATS 2022 wiki

Analog↗

NASA NTRSDate not supplied

A Virtual Reality Planning Environment for High-Risk, High-Latency Teleoperation

In-Space Servicing, Assembly, and Manufacturing has the potential to enable larger-scale and longer-lived infrastructure projects in space. Servicing in particular has the potential to vastly increase the usable lifetimes of satellites, both newly launched and existing. However, some of these servicing tasks require delicate manipulation and, to apply to existing satellites, need to be able to operate on structures that were not designed for servicing and, in some cases, where the exact shape of the structure is not known in advance. For this reason, human-in-the-loop ground-based teleoperation is the only option for some missions. Ground-based teleoperation presents its own difficulties. In addition to the challenge of performing delicate operations in constrained spaces, communication with the servicing platform is subject to a variable time delay on the order of 2-7 seconds, and the only visual feedback comes from camera views with limited situational awareness and viewing angles.

Teleoperation↗

NASA NTRSDate not supplied

Advanced Air Mobility Operations & Automation Part II Technical Lecture

The Airborne Collision Avoidance System X (ACAS X) is a next-generation collision avoidance tool developed to support different aircraft types and operations. The rotorcraft variant – referred to as ACAS XR – is designed to accommodate existing helicopter platforms as well as in-development, electric vertical takeoff and landing concepts, which are critical to the emerging concept of operations referred to as Advanced Air Mobility. The fundamental role of ACAS XR is to provide Detect and Avoid (DAA) and/or Collision Avoidance (CA) protection against airborne traffic. DAA alerting and guidance in the context of ACAS XR is caution-level and “suggestive,” and is to be used by the pilot if, and when, they decide to maneuver against an identified threat to DAA “well clear.” The CA alerting, by contrast, is warning-level and “directive,” with the pilot required to comply with the associated guidance to prevent a predicted Near Midair Collision (NMAC). The CA alerts generated by ACAS XR are referred to as Resolution Advisories (RAs) consistent with previous CA systems. Unlike earlier CA systems, ACAS XR issues RAs in the horizontal and vertical dimensions as well as multi-axis RAs (referred to as “Blended” RAs). According to the Minimal Operational Performance Standards of DAA systems for Unmanned Aircraft Systems, maneuvers to comply with RAs may be automated, whereas maneuvers based on DAA alerting assume a manual response. The current presentation discusses two human-in-the-loop simulations that presented rotorcraft pilots with ACAS XR alerts and guidance in eVTOL simulators with varying levels of automation. It also presents overviews of these studies as well as how they assisted live flight tests, which will occur throughout 2023.

air taxis↗

NASA NTRSDate not supplied

Using Virtual Reality to Envision Deployment of Spacesuit-Compatible Augmented Reality Displays for Lunar Surface Operations

The National Aeronautics and Space Administration (NASA) aims to land crew on the lunar surface to establish a sustainable presence and develop operational concepts for future long-duration missions. New technologies will be necessary to extend planning and execution capabilities for lunar surface activities. NASA’s Joint Augmented Reality Visual Informatics System (Joint AR) is one such technology. Joint AR is a suit-mounted augmented reality (AR) display and computes system which facilitates unprecedented information exchange and data visualization capabilities between mission support operators and suited crew. This paper describes challenges associated with developing AR technology for an envisioned work domain by applying a sociotechnical lens to the iterative testing and development of novel AR technology through virtual reality (VR). A VR testbed was established to simulate a representative lunar surface environment, enabling a series of three human-in-the-loop (HITL) tests evaluating AR navigation interfaces for exploration extravehicular activity (xEVA). Our findings identify several considerations for future Joint AR design and testing efforts, including challenges with data overload, attentional demands, and environment-related perceptual challenges. Trade-offs and potential approaches are discussed to mitigate these challenges and improve future Joint AR testing fidelity.

Matthew Miller↗

NASA NTRSDate not supplied

SPHINX: A Generalized Tool for Solar Energetic Particle Model Validation

The Integrated Solar Energetic Proton Alert/Warning (ISEP) project was established to aid in the transition of SEP models from the research realm to operational use. This project is a collaboration between the NASA Community Coordinated Model Center (CCMC), the NASA Moon to Mars Space Weather Analysis Office (M2M) and the NASA Space Radiation Analysis Group (SRAG). Through ISEP, the SEP Scoreboards (Intensity, Probability, and All Clear) were developed to visualize real time forecasts from SEP models for use in SRAG operations. To accomplish this, SRAG, CCMC, and M2M work closely with modelers on model development, CCMC works on Scoreboard implementation, and M2M provides human-in-the-loop space weather analysis and ensures that models run in real time.

space weather↗

NASA NTRSDate not supplied

Design Considerations for LTV HITL Testing of Pressurized Suited Crew on a Motion-Based Platform

The upcoming NASA Lunar Terrain Vehicle (LTV) will succeed the Apollo Lunar Roving Vehicle, performing both surface exploration and logistics transfer in NASA’s return to the Moon. Human-in-the-Loop (HITL) testing will play a key role in refining the design of the LTV, ensuring its usability and ability to accommodate the astronaut population. A motion-based platform and a lunar terrain and lighting model can simulate the conditions of the lunar south pole region. It can be used by NASA to conduct HITL testing in concert with HITL testing of the drivable Ground Reference Unit (GTU) concept vehicle. The dynamic motion of the platform combined with the mobility restrictions of pressurized suits could help improve NASA understanding of vehicle-suit-astronaut interfaces. NASA human factors practitioners have proposed multiple HITL test series to use the motion-based platform in conjunction with testing of the GTU in hopes that lessons learned can be applied to help select a commercial partner to develop the Artemis LTV.

LTV↗

NASA NTRSDate not supplied

Autonomous Control for Arbitrary Thruster Configurations and Mass Properties in Special Euclidean Group SE(3)

Most current methods for determining maneuvers and thrust firing sequences depend on explicit and predetermined commands generated by a combination of on-board systems and ground-based human-in-the-loop methods. For spacecraft and space structures with changing mass properties and thruster configurations, such as the Deep Space Gateway as it changes configurations throughout its lifetime, determining these commands can be time-consuming and computationally intensive. However, recent work within the Lie group SE(3) has offered ways of autonomously determining the location, power, precision, and capabilities of thrusters in any arbitrary position. Furthermore, a method for determining thruster firing sequences based on an arbitrary control input (both translational and rotational in a coupled, 6-element vector) and arbitrary thruster configurations has also recently been developed. When combining these methods, any spacecraft with any mass properties and thruster configurations can be understood in terms of controllability limits and thruster firing sequences can be generated quickly and with low computational load, thus extending the autonomous capabilities of deep space missions. In this work, this method is presented and explored in terms of convergence time to the desired pose. The capabilities of this method are also examined in the case of the Deep Space Gateway both in fully controllable configurations and uncontrollable configurations.

SE(3)↗

NASA NTRSDate not supplied

A Crew Seat for Human Exploration in Multiple Gravity Environments

This work attempts to develop a single crew seating solution that is applicable across a range of gravity environments encountered by spacecraft proposed in several conceptual spacecraft architectures. All of these spacecraft will need to provide some sort of stationary accommodation for the crew for performing various activities such as work in science laboratories, maintenance and repair facilities, medical care facilities, and spacecraft operations centers, as well as for basic habitation in crew quarters, entertainment / relaxation facilities, and crew dining facilities. Depending on the spacecraft or architecture, this stationary accommodation may be experienced continuously in microgravity, such as would be the case for the Deep Space Exploration Vehicle. Alternately, it could be in continuous lunar or Martian gravity, such as the Common Habitat base camps. It could experience fractional gravity, such as a Pressurized Rover for In-Space Missions at a Near Earth Asteroid or one of the Martian moons. It could alternate between artificial gravity and microgravity, such as the Nautilus-X. It could alternate between microgravity and lunar or Martian gravity, such as the SpaceX Starship Human Landing System or the Blue Origin Blue Moon Block 2 Human Landing System. Or it could be in continuous Earth gravity, such as ground trainer systems. Prior human spaceflight systems for stationary accommodation have been focused on microgravity applications. These systems have their own limitations and cannot be used in a gravity environment. A public crowdsourcing campaign generated dozens of ideas, which ultimately generated a Gecko Mobility Aids system for crew translation and the Multi-Gravity Crew Seat (MGCS) for stationary accommodation. The MGCS functions in gravity as a traditional terrestrial seat, performing functions of load for the overall body and forearms, as well as head and neck load relief while positioning the body within range of an intended task. In microgravity, the MGCS functions as a body restraint, securing the body against inadvertent drifting by applying a restraining pressure at the front and back of the thighs, shoulders, and back. The initial MGCS concept was developed in a NASA hackathon and was refined through a review of dozens of terrestrial seating styles. Additionally, a review of anthropometry and biomechanics data related to the neutral body posture was conducted to help inform the microgravity configuration of the MGCS. A series of CAD models were iteratively developed, with subject matter expert reviews leading to design improvements. A scale model was constructed and used with a humanoid model to demonstrate MGCS accommodation of a human-like body in both gravity and microgravity modes. Work to develop a full-scale protype of the MGCS is discussed, including design and fabrication of the headrest, arm rest, seat back, seat pan, seat base, and the conversion mechanisms. A 1-g human-in-the-loop evaluation of the prototype assessed the acceptability of performing seated activities in the MGCS, collecting data on the usability, comfort, and ease of ingress/egress. Based on the evaluation results, design modifications needed for reduced gravity testing are documented and initial work is indicated for a reduced gravity test plan.

Restraints and Mobility Aids↗

NASA NTRSDate not supplied

Highlights of NASA’s Orbital Debris Program Office In Situ and Laboratory Measurements

NASA’s Orbital Debris Program Office (ODPO) maintains various returned spacecraft materials, capabilities, and facilities used for in situ and laboratory measurements that directly support orbital debris environmental models. In situ measurements include the analysis of exposed and returned hardware surfaces. These surfaces serve as passive sensors for the small-sized micrometeoroid and orbital debris (MMOD) flux below the sensitivity of ground-based radar and optical sensors. Various instruments and techniques are used to determine the size and depth of selected impact features, and – if feasible – the composition of the projectile material. Analysis of the impactor residues enables the differentiation of MM and OD for debris below 1 mm to support modeling the OD environment. In addition, projectiles identified as OD can be further differentiated in low-, medium-, and high- density impactors based on chemical analyses. In addition to in situ measurements, the ODPO has also worked in collaboration with the U.S. Space Force Space Systems Command (formerly the U.S. Air Force Space and Missile Systems Center), the Aerospace Corporation, and the University of Florida on a laboratory-based hypervelocity impact test, DebriSat, conducted at the Air Force Arnold Engineering Development Complex in 2014. The resulting data from this impact test series are being analyzed to assess the fragments’ sizes/masses, materials/densities, shapes, and other parameters of interest. The DebriSat project provides the data needed to update NASA’s breakup models and size estimation models using the simulated orbital breakup of a modern, low Earth orbit spacecraft. Ultimately, over 200,000 fragments from this impact test will be stored at NASA Johnson Space Center (JSC) and further analyzed by the ODPO. This project will also use machine learning techniques to infer physical parameters of fragments embedded in the soft-catch foam used in the impact experiment. Applied to X-ray imagery of the foam panels, these techniques promise to minimize human-in-the-loop processes for fragment extraction and physical characterization. A brief overview of this project and data collected will be presented. Lastly, the ODPO provides various capabilities hosted at NASA JSC for optical inspections and measurements using a variety of techniques and scientific instrumentation to support both in situ and laboratory measurements. The ODPO’s Optical Measurement Center (OMC) is an advanced facility for photometric and spectroscopic laboratory measurements of targets, including fragments from the DebriSat project. The OMC simulates telescopic observations by using space-like illumination conditions and source-target-sensor orientation techniques. Additionally, the OMC is uniquely equipped to acquire pseudo-bidirectional reflectance distribution data for broadband photometric measurements, thus removing aspect angle dependencies that can affect target size estimates using the optical size estimation model. Narrow-band surface material characterization using spectroscopic instrumentation gives insight into how the albedo parameter – also important in the optical size estimation model – may vary depending on the state of the material. The OMC also performs simulations of photometric measurements using optical ray-tracing software to model the OMC optical throughput. In addition to the OMC, the ODPO houses a start-of-the-art Fragment Analysis Facility that uses multiple microscopic inspection instruments to support in situ measurements and material characterization. An overview of both facilities will be highlighted in this paper.

Orbital Debris↗

NASA NTRS2023

Using Machine Learning to Infer Material Properties of Debris Fragments from X-ray Images in the DebriSat Project

The DebriSat project is a collaboration effort with the NASA Orbital Debris Program Office, the U.S. Space Force Space Systems Command Center, The Aerospace Corporation, and the University of Florida. To date, over 200,000 fragments from this ground-based, hypervelocity impact experiment have been collected, and processing is underway to determine their physical characteristics, such as material, shape, color, characteristic length, and average cross-sectional area. The x-ray process is primarily used to identify the location of the fragments and estimated size for extraction, so that these physical characteristics can be assessed. This paper proposes a machine learning-based approach to characterize materials from x-ray images of debris fragments embedded in soft-catch foam used in the DebriSat project. The novel methodology discussed in this paper will highlight the use of x-ray imagery data to characterize these fragments without extraction or a human-in-the-loop. Both supervised and unsupervised machine learning techniques are utilized with this approach to infer the physical parameters of the fragments embedded in the soft-catch foam panels used in the impact experiment based on x-ray images of the foam panels. Additionally, 3D reconstructions of the extracted fragments are created with images taken from two different angles using the structure from motion (SfM) method. The characteristic lengths and shape from the 3D reconstruction, alongside the physical characteristics of the debris, are used in the inference of the material type. To develop and test the approach, a dataset of x-ray images of debris fragments of varying sizes and materials is collected. Supervised learning methods such as convolutional neural networks (CNNs), support vector machines (SVM), decision trees, and random forest classifiers are used due to the high-dimensional feature spaces of the debris and nonlinear decision boundaries for material categorization. Given the limited pre-labeled data of embedded debris materials smaller than 10 mm, unsupervised machine learning techniques such as clustering algorithms and autoencoders are used, in addition to supervised learning methods. The clustering algorithms group similar fragments together based on their physical properties, and autoencoders reduce the dimensionality of the x ray images and extract relevant features. The performance of the proposed approach's is analyzed using a range of statistical methods, including confusion matrices, receiver operating characteristic curves, and precision-recall curves. The results are compared with those obtained using a baseline approach that relies on manual identification and classification of debris fragments. To evaluate the effectiveness of different machine learning methods, statistical tests such as t-tests, ANOVA, and cross-validation are performed, comparing the performance of CNNs, SVMs, clustering algorithms, and autoencoders. Additional analysis needs to be conducted to identify any sources of bias or variability that may affect the results, such as variations in imaging conditions or fragmentation patterns. Other topics explored are limitations, refinements, and the potential use of semi-supervised learning techniques, such as self-training to label unlabeled datasets and co-training using x-ray images taken from two different angles as two different models.

Saik Anam Siam↗

NASA NTRS2023

Joint Augmented Reality Visual Informatics System: Concept of Operations

NASA proposed requirements for a digital display for an EVA spacesuit to provide relevant information to the crew member. The Joint Augmented Reality Visual Informatics System (Joint AR) project pursued four years of research and development towards a suit-display system in a near-eye, AR form factor. The project was responsible for developing software (custom graphics engine and core flight software), physical hardware prototyping (controls, projection display optics, suited display platform), virtual prototyping platform (a virtual reality testbed), and human-in-the-loop (HITL) operational testing informed by EVA flight controllers, crew members, and human factors engineers for con-ops definition. This document contains substantial updates to CTSD-ADV-1788 Rev. Basic. This revision was produced by the project to summarize the use-cases and and user experiences developed throughout the project, and refine the Basic revision originally drafted at the beginning of the project life cycle. The primary purpose of this document is to summarize and make available the scenario development efforts that have been pursued and explored within the Joint AR project. This includes descriptions of the scenarios themselves as well as corresponding potential of advanced informatics displays to support those specified scenarios. In doing so, this document provides a variety of approaches to deconstruct and hypothesize how future technological capabilities so that with future EVA work demands can be satisfied within future human planetary spaceflight missions.

Matthew Miller↗

NASA NTRS2024

Concept, Design, & Implementation of a Remote Vehicle Operations Center for Autonomous Missions

The National Aeronautics and Space Administration is supporting research to develop a prototype remote vehicle operations center at Langley Research Center to explore current and future advanced air mobility operations using small unmanned aerial systems vehicles as surrogates for future, larger-scale passenger carrying vehicles. The prototype facility known as the Remote Operations for Autonomous Missions (ROAM) Unmanned Aerial Systems (UAS) Operations Center is being used to explore different roles and responsibilities of remote operators managing multiple autonomous vehicles, with the goal of exploring human-autonomy teaming concepts that enable m:N operations (i.e., m operators managing N vehicles). ROAM has developed into a world-class research, development, and technology (RD&T) environment that can support both the collection of human factors data and the command and control of remote vehicles in beyond visual line of sight conditions. ROAM provides a key capability to enable full end-to-end hardware- and human-in-the-loop simulation testing, connecting with simulated small-UAS and creating a seamless Live-Virtual-Constructive (LVC) environment. This report describes the development of the ROAM UAS Operations Center from concept through design, culminating in the current implementation at NASA’s Langley Research Center.

CERTAIN↗

DOE OSTI2025

Interactive Rotated Object Detection for Novel Class Detection in Remotely Sensed Imagery

In this paper we propose IRTR-DETR an Interactive and Real-Time Rotated DEtection TRansformer that extends IRTDETR to predict rotated bounding boxes. IRTR-DETR maintains the Human-In-The-Loop (HIL) workflow of IRTDETR but introduces rotation-aware heads for improved detection of objects with arbitrary orientations. Similarly to IRTDETR IRTR-DETR can be trained with a small labeled sample set in an interactive setting but we show that it can also be pretrained on related but not identical data--such as a building damage dataset--before being applied to tasks like identifying buildings under construction. We demonstrate the efficacy of our approach on the publicly available Tiny-DOTA and xBD dataset as well as two study-cases on proprietary datasets of greenhouses and houses under construction ("waffle homes"). Detecting greenhouses is highly relevant in the context of damage assessment while "waffle homes" aid understanding typical floorplans and building codes in different areas both thereby supporting population modeling emergency response and policy planning. Our method outperforms the state of the art in interactive rotated object detection on the Tiny-DOTA dataset by 5.7 percent and improves upon the non interactive RTDETR by 7.85 to 19.39 percent (depending on the number of provided samples) while maintaining its real-time efficiency.

Burges, Marvin [ORNL] (ORCID:0000000312690769)↗

DOE OSTI2025

HEPTAPOD: Orchestrating High Energy Physics Workflows Towards Autonomous Agency

Many workflows in high-energy-physics (HEP) stand to benefit from recent advances in transformer-based large language models (LLMs). While early applications of LLMs focused on text generation and code completion, modern LLMs now support orchestrated agency: the coordinated execution of complex, multi-step tasks through tool use, structured context, and iterative reasoning. We introduce the HEP Toolkit for Agentic Planning, Orchestration, and Deployment (HEPTAPOD), an orchestration framework designed to bring this emerging paradigm to HEP pipelines. The framework enables LLMs to interface with domain-specific tools, construct and manage simulation workflows, and assist in common utility and data analysis tasks through schema-validated operations and run-card-driven configuration. To demonstrate these capabilities, we consider a representative Beyond the Standard Model (BSM) Monte Carlo validation pipeline that spans model generation, event simulation, and downstream analysis within a unified, reproducible workflow. HEPTAPOD provides a structured and auditable layer between human researchers, LLMs, and computational infrastructure, establishing a foundation for transparent, human-in-the-loop systems.

Menzo, Tony [Alabama U.; Fermilab] (ORCID:00000002↗