Human-in-the-Loop Robot Control for Human-Robot Collaboration: Human Intention Estimation and Safe Trajectory Tracking Control for Collaborative Tasks
Not provided.
Automation & Control Systems↗
SEARCH · Search NASA
Results for “Human in the Loop”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
Skin-integrated stretchable actuators toward skin-compatible haptic feedback and closed-loop human-machine interactions
Abstract Skin-integrated haptic interfaces that can relay a wealth of information from the machine to the human are of great interest. However, existing haptic devices are not yet able to produce haptic cues that are compatible with the skin. In this work, we present the stretchable soft actuators for haptic feedback, which can match the perception range, spatial resolution, and stretchability of the skin. Pressure-amplification structures are fabricated using a scalable self-assembly process to ensure an output pressure beyond the skin perception threshold. Due to the minimized device size, the actuator array can be fabricated with a sufficiently high spatial resolution, which makes the haptic device applicable for skin locations with the highest spatial acuity. A haptic feedback system is demonstrated by employing the developed soft actuators and highly sensitive pressure sensors. Two proof-of-concept applications are developed to illustrate the capability of transferring information related to surface textures and object shapes acquired at the robot side to the user side.
Chen, Si↗
Human-in-the-Loop Motion Control of a Two-DOF Hydraulic Backhoe Powered by the Hybrid Hydraulic Electric Architecture (HHEA)
Abstract The Hybrid Hydraulic-Electric Architecture (HHEA) combines the respective power density and control advantages of hydraulic and electric actuation to save energy for off-road vehicles. It uses a set of selectable common pressure rails to transmit the majority of power and electric actuation to modulate that power. As it is critical that off-road vehicles can perform tasks dexterously and exactly as commanded by the operator, the switchings between discrete pressure rails pose a potential challenge for smooth and precise motion. A control strategy consisting of a backstepping nominal control and least norm transition control has previously been developed to address this issue. It has been tested on 1 degree of freedom (DOF) testbeds where known trajectories were able to be tracked precisely. This paper presents the implementation of the HHEA motion control strategy on a 2-DOF backhoe operated by a human operator via a 2-DOF joystick. Unlike previous studies, the duty cycle is unknown beforehand and the decision to change pressure rails is taken in real-time. The efficacy of the motion control strategy has been validated experimentally. Several strategies to improve the user interface: control in workspace coordinates, pressure feedback, and velocity field-based task specification, have also been implemented and demonstrated to make operating the multiple DOF, HHEA actuated machine more intuitive to novice operators.
24 POWER TRANSMISSION AND DISTRIBUTION↗
Dynamic Human-in-the-Loop Simulated Nuclear Power Plant Thermal Dispatch System Demonstration and Evaluation Study
An Idaho National Laboratory research team performed a human-in-the-loop study with two formerly licensed retired operators to evaluate thermal power dispatch operations supported by the modified GSE Systems GPWR plant simulator. Virtual representations of the analog control panels were presented on touch-screen bays configured to mimic the control room layout in the newly renovated Human Systems Simulation Laboratory. The operators performed 15 scenarios covering normal evolutions to transition the plant from full turbine operation to joint turbine and thermal power dispatch operations in addition to transient response scenarios induced with simulated faults to evaluate the impact of thermal power dispatch system on operator and plant responses. A prototype human-system interface (HSI) was developed and displayed in tandem with the virtual analog panels to support the operators executing the procedurally drive evolutions and transient responses. An interdisciplinary team of operations experts, nuclear engineers, and human factors experts observed the operators performing the scenarios to evaluate the operations. Only a preliminary analysis of the results has been performed. The full analysis will be shared in a milestone report scheduled for release at the end of September; however, some high-level conclusions were readily available. Two high level findings for the system design were captured in the study. The manual control supported by the HSI to transition from standard operations to thermal power dispatch operation imposed a considerable amount of workload on the operators due to tedious manual valve manipulations and system monitoring required to verify their intended effect. An additional operator would be required in the control room to support the daily evolution. Automatic control for the transition was deemed a requirement for plant adoption without imposing additional staffing costs. The second finding was the necessity for an automatic thermal power dispatch system trip isolation function linked to a turbine and reactor trip signal. The operators completed scenarios with automatic isolation functionality and manually required actuation of the thermal power dispatch system. The operator response was sufficiently slower in the manual trip condition, such that operators were unable to manually actuate key post trip safety functions an indicate a degraded control capability that should be avoided. With an automatic trip signal little to no impact of the thermal power dispatch system was identified on the primary plant response and therefore the system could be readily and safely adopted. Together these two findings represent the need to support the adoption of thermal power dispatch capability into existing operations by leveraging automation to augment any additional operator tasking required to control and monitor an additional system beyond existing operations.
99 GENERAL AND MISCELLANEOUS↗
A Scalable Hardware-and-Human-in-the-Loop Grid-interactive Efficient Building Equipment Performance Dataset
This project developed a publicly available, high-fidelity dataset about the interactions among humans, homes, and heat pumps supporting grid interactive efficient buildings to balance demand on the grid with comfort for occupants. Laboratory measurements and simulations of the hardware capture the second-scale electric power dynamics of heat pumps providing grid services like load shifting and load shedding. Field measurements, behavior tracking, and qualitative surveys of people in their homes over multiple years—including experimentally adjusting the heating and cooling system to provide grid services—to capture the reciprocal effect of human behavior on grid services, and grid services on human comfort. Taken together, these data capture the complete Hardware and Human in the loop system for residential heat pumps, reducing large uncertainties in simulation for design, and models for control of heat pumps, and grid-interactive buildings.
24 POWER TRANSMISSION AND DISTRIBUTION↗
Agentic traffic intelligence: Augmented human-in-the-loop scenario generation for microscopic traffic simulation
Traditional microscopic traffic simulation generation often relies on static datasets and manual design, limiting its ability to simulate complex conditions easily. This paper presents a novel framework, Agentic Traffic Intelligence, which combines human approval large language models (LLMs), the Real-Twin tool, and multi-agent systems to perform realistic microscopic traffic simulation scenario generation. The proposed framework incorporates human-in-the-loop (HIL) control, retrieval-augmented generation (RAG), and multi-agent control mechanisms. HIL mechanisms are used to guide multiple LLMs focused on attributes for microscopic simulation generation and to improve the interpretability and transparency of LLM execution for users. RAG enhances context extraction by dynamically integrating external knowledge sources for traffic scenario generation foundations. A multi-agent architecture with supervisory control coordinates the interaction of simulation components, including traffic simulators, control logic, and calibration tools. This enables the synthesis of simulation-ready scenarios that reflect dynamic demand profiles and behavior controls. Furthermore, the framework fuses multisource traffic data with unstructured context and supports iterative refinement through interactive user feedback. Validated through microscopic simulation using Simulation of Urban Mobility, the generated scenarios demonstrate high-fidelity network generation with inflow and turn movement and behavioral calibration, offering a robust and efficient tool for stress-testing and optimizing urban mobility systems.
Hierarchical multi-agent control↗
Human-in-the-loop Sensing and Control for Commercial Building Energy Efficiency and Occupant Comfort
Most of the existing heating, ventilation and air conditioning (HVAC) systems in commercial buildings operate in a conservative manner by assuming maximum occupancy in each room during pre-specified periods of the week, leading to significant energy being wasted as rooms are over-conditioned compared to the actual requirements of the occupants. Though critical, our understanding of occupancy patterns and thermal comfort needs of the occupants in commercial buildings is lacking and it is well known that both of these quantities are stochastic and time-varying, thus requiring sensing solutions to estimate them. This project had the goal of designing, implementing and evaluating a hardware and software solution to ameliorate this challenge. In particular, a depth camera (one whose pixels reveal distance from the camera as opposed to color values) placed on doorways is used to detect entrance and exit events from thermal zones in the building, and thereby estimate their occupancy levels. This information is then fed to a novel control algorithm that can, through interactions with the HVAC system, learn how to provide control inputs that maximize comfort and minimize energy waste. The resulting system represents a significant improvement over existing controllers for commercial HVAC systems and allowed us to improve our understanding of the design of future human-in-the-loop control solutions. For this solution to be feasible, the project had target metrics for its performance and cost. In particular, entrance and exit events for occupants moving about the building would need to be detected with an accuracy higher than 97%; and the resulting control inputs derived from this information would need to lead to approximately 10% energy savings compared to a schedule-based controller. Furthermore, regarding the final hardware design, the project had a target bill of materials (BOM) cost for the sensing solution of less than US$200 per unit while using less than 25W of power on average. All of these target metrics were met or exceeded by our final proposed solution. We performed evaluations by deploying the system in over 20 rooms of different types across 6 commercial buildings in Pittsburgh, PA over the course of three years, and performing targeted controlled experiments to test its performance along the different metrics. The human-in-the-loop control solutions (both hardware and software) developed through this project are expected to lead to significant improvements in the comfort and energy efficiency of HVAC systems used in commercial buildings. The insights we developed through the project pave the way to HVAC systems that can condition interior spaces according to their real-time utilization and the thermal comfort needs of the occupants, thereby reducing energy use. They also open up a new learning-based way of configuring HVAC controllers without having to manually fine-tune them for each building. These innovations can significantly increase the adoption of novel control solutions by the industry and thereby save resources and reduce costs of operation.
32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
A human-in-the-loop explanation framework for morphologically transparent AI predictions from whole-slide images
Deep learning models enable the prediction of clinical endpoints from whole-slide images (WSIs), but many such models function as “black boxes”, lacking transparency about whether and which histomorphological patterns drive their predictions, hindering interpretability and clinical adoption. Here we propose a human-in-the-loop explanation framework, MorphoXAI, which provides both local and global interpretability for deep learning models by incorporating human-expert interpretations. At the global level, it reveals the histomorphological patterns on which the model consistently relies to distinguish between classes of WSIs, as well as the patterns associated with confusion between classes. At the local level, it indicates which of these patterns are used in the prediction of an individual WSI and which regions within the slide correspond to such patterns. We validated our method across multiple deep learning–based WSI analysis tasks spanning different tissue types. The results show that our framework generates explanations that accurately reflect the histomorphology underlying the model’s predictions at both global and local levels. For interpretability and clinical utility in diagnostic contexts, human evaluation results showed that our explanations were easy to interpret, rich in diagnostic features, and directly helpful for diagnostic decision-making, thereby enhancing pathologist-AI collaboration. Our work highlights that unifying global and local explanations and grounding them in expert-interpreted morphology enhances the interpretability and verifiability of deep learning models, thereby facilitating the transparent deployment of such models in clinical practice.
Lou, Peiliang↗
A General Framework for Human-in-the-Loop Cognitive Digital Twins
Modelling and analysis of systems that are equipped with sensors and connected to the Internet are becoming more automated and less human-dependent. However, bringing expert knowledge into the loop along with data obtained from Internet of Thing (IoT) devices minimizes the risk of making poor and unexplainable decisions and helps to assess the impact of different strategies before applying them in reality. While Digital Twins are more of a data-driven simulation of the physical system, Cognitive Digital Twins bring the human dimension into the modelling and simulation. In this paper, we aim to emphasize the crucial role of explainability and the underlying rationale behind automated or interactive decision-making processes. Furthermore, we propose an initial framework that delineates the specific points within the feedback loop of a cognitive digital twin where human involvement can be incorporated.
Niloofar, Parisa↗
Human-in-the-Loop: The Future of Machine Learning in Automated Electron Microscopy
Machine learning (ML) methods are progressively gaining acceptance in the electron microscopy community for de-noising, semantic segmentation, and dimensionality reduction of data post-acquisition. The introduction of the application programming interfaces (APIs) by major instrument manufacturers now allows the deployment of ML workflows in microscopes, not only for data analytics but also for real-time decision-making and feedback for microscope operation. However, the number of use cases for real-time ML remains remarkably small. Furthermore, we discuss some considerations in designing ML-based active experiments and pose that the likely strategy for the next several years will be human-in-the-loop automated experiments (hAE). In this paradigm, the ML learning agent directly controls beam position and image and spectroscopy acquisition functions, and a human operator monitors experiment progression in real and feature space of the system and tunes the policies of the ML agent to steer the experiment toward specific objectives.
47 OTHER INSTRUMENTATION↗
Machine-Learning-Driven Discovery of Water Splitting BaFe 2 O 4 and Human-in-the-Loop Improvement via Al-Substitution for Increased Thermal Stability
Thermochemical hydrogen (TCH) production offers a promising method for converting thermal energy into hydrogen fuel through heat-driven redox cycles of metal oxides. Here, in this work a defect graph neural network (dGNN) was used to predict oxygen vacancy formation energies ΔH V O combined with Materials Project predictions of oxygen chemical potential stability to screen candidate oxides via high-throughput database analysis. BaFe 2 O 4 was identified as a promising material for experimental validation based on its predicted ΔH V O , oxygen chemical potential stability range, and potential for tunable substitutions to improve thermal properties. Experimental validation using thermogravimetric analysis (TGA), stagnation flow reactor (SFR), X-ray diffraction (XRD), and electron microscopy confirmed positive water-splitting behavior but also revealed limitations in thermal stability under aggressive reduction conditions. To address this, a human-in-the-loop modification strategy was employed introducing Al substitution in BaFe 2–x Al x O 4 ; this modification improves thermal stability, alters the crystal structure and enhances overall performance. These results demonstrate a combined computational and experimental workflow in which machine learning accelerates identification of promising candidates, while targeted experimental design enables optimization of functional performance. This approach advances the development of robust, cost-effective TCH materials and highlights the importance of integrating data-driven discovery with human-guided materials design in paving the way for scalable hydrogen production technologies.
organic↗
Using Flory–Huggins-informed human-in-the-loop Bayesian optimization to map the phase diagram of polymer blends
Mapping the phase diagram of polymer blends is an essential step in controlling the structure–property relationship of polymer-based materials. However, traditional grid-based approaches are inefficient and rely on subjective judgements for terminating the experimental campaign. Artificial intelligence-guided experimentation offers a compelling alternative, especially when data-driven decision-making is interfaced with established polymer thermodynamics to improve efficiency and interpretability. Here, we introduce a physics-informed Bayesian optimization approach to guide the mapping of the phase diagram of a model blend containing poly(methyl methacrylate) and poly(styrene-ran-acrylonitrile). Physical information is derived from a Flory–Huggins representation of the spinodal curve, which is integrated into the Bayesian optimization process as a structured prior mean that acts as a soft constraint. Implemented as a human-in-the-loop workflow, the approach leverages optical imaging of film cloudiness with iterative Gaussian process surrogate modeling and a parameter selection decision policy to identify the composition-temperature conditions for sequential iterations. Convergence of kernel and Flory–Huggins-based hyperparameters provided a stopping criterion, ensuring an objective and interpretable termination of the experimental campaign. The framework recovered the known lower critical solution temperature (∼160 °C), while increasing material efficiency through targeted sampling. This work establishes a proof-of-concept for the application of Bayesian optimization workflows to study polymer blend miscibility.
36 MATERIALS SCIENCE↗
Building workflows for an interactive human-in-the-loop automated experiment (hAE) in STEM-EELS
Exploring the structural, chemical, and physical properties of matter on the nano- and atomic scales has become possible with the recent advances in aberration-corrected electron energy-loss spectroscopy (EELS) in scanning transmission electron microscopy (STEM). However, the current paradigm of STEM-EELS relies on the classical rectangular grid sampling, in which all surface regions are assumed to be of equal a priori interest. However, this is typically not the case for real-world scenarios, where phenomena of interest are concentrated in a small number of spatial locations, such as interfaces, structural and topological defects, and multi-phase inclusions. One of the foundational problems is the discovery of nanometer- or atomic-scale structures having specific signatures in EELS spectra. Herein, we systematically explore the hyperparameters controlling deep kernel learning (DKL) discovery workflows for STEM-EELS and identify the role of the local structural descriptors and acquisition functions in experiment progression. In agreement with the actual experiment, we observe that for certain parameter combinations the experiment path can be trapped in the local minima. We demonstrate the approaches for monitoring the automated experiment in the real and feature space of the system and knowledge acquisition of the DKL model. Based on these, we construct intervention strategies defining the human-in-the-loop automated experiment (hAE). This approach can be further extended to other techniques including 4D STEM and other forms of spectroscopic imaging. The hAE library is available on Github at https://github.com/utkarshp1161/hAE/tree/main/hAE.
Pratiush, Utkarsh [Univ. of Tennessee, Knoxville, ↗
A dynamic Bayesian optimized active recommender system for curiosity-driven partially Human-in-the-loop automated experiments
Optimization of experimental materials synthesis and characterization through active learning methods has been growing over the last decade, with examples ranging from measurements of diffraction on combinatorial alloys at synchrotrons, to searches through chemical space with automated synthesis robots for perovskites. In virtually all cases, the target property of interest for optimization is defined a priori with the ability to shift the trajectory of the optimization based on human-identified findings during the experiment is lacking. Thus, to highlight the best of both human operators and AI-driven experiments, here we present the development of a human–AI collaborated experimental workflow, via a Bayesian optimized active recommender system (BOARS), to shape targets on the fly with human real-time feedback. Here, the human guidance overpowers AI at early iteration when prior knowledge (uncertainty) is minimal (higher), while the AI overpowers the human during later iterations to accelerate the process with the human-assessed goal. We showcase examples of this framework applied to pre-acquired piezoresponse force spectroscopy of a ferroelectric thin film, and in real-time on an atomic force microscope, with human assessment to find symmetric hysteresis loops. It is found that such features appear more affected by subsurface defects than the local domain structure. This work shows the utility of human–AI approaches for curiosity driven exploration of systems across experimental domains.
36 MATERIALS SCIENCE↗
The future of self-driving laboratories: from human in the loop interactive AI to gamification
Recent developments in artificial intelligence (AI) and machine learning (ML), implemented through self-driving laboratories (SDLs), are rapidly creating unprecedented opportunities for the accelerated discovery and optimization of materials. This paper provides a joint analysis of SDLs from both academic and industry perspectives, highlighting the importance of integrating human intelligence in these systems. It discusses the necessity of careful planning in SDL design across physical, data, and workflow dimensions, including instrumental setup, experimental workflow, data management, and human–SDL interaction. The significance of integrating human input within SDLs, especially as the focus shifts from individual tools and tasks to the creation and management of complex workflows, is emphasized. The paper stresses on the crucial role of reward function design in developing forward-looking workflows and examines the interplay between hardware evolution, ML application across chemical processes, and the influence of reward systems in research. Ultimately, the article advocates for a future where SDLs blend human intuition in hypothesis formulation with AI's precision, speed, and data-handling capabilities.
97 MATHEMATICS AND COMPUTING↗
Toward model-free safety-critical control with humans in the loop
Not Available
Xiao, Wei (ORCID:0000000196226415)↗
Human-in-the-loop control of dynamics and robotics using augmented reality
Not Available
Wyckoff, Elijah (ORCID:0000000284672984)↗
A Tale of Two Simulators—A Comparative Human-in-the-Loop Nuclear Power Plant Operations Study on Thermal Power Dispatch for Hydrogen Production
A study was designed for a reconfigurable, full-scale, full-scope nuclear power plant control room simulator to compare two different thermal power dispatch systems, on separate simulator platforms, demonstrating a TPD concept of operation. A TPD system can provide a desirable alternative revenue source for utilities but requires addressing new and unique operational issues. The selection of representative scenarios and the scenario-based experimental design are presented as key elements to capture evidence for validating the developed TPD concept of operations overcome these operational issues.
Ulrich, Thomas A.↗