Machine Learning for Autonomous Drones Operations
Presentation to the Machine Learning & Artificial Intelligence Symposium on October 16, 2020
SEARCH · Search NASA
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
Presentation to the Machine Learning & Artificial Intelligence Symposium on October 16, 2020
Building energy modeling (BEM) is a multipurpose tool for building energy efficiency (EE). The U.S. Department of Energy Building Technologies Office (BTO) seeks to expand the use and effectiveness of BEM in the design and operation of commercial and residential buildings with the goal of achieving lasting reductions in total and peak energy use. This report identifies gaps and outlines recommended initiatives to achieve this goal, based on a combination of technical analysis and stakeholder input. In addition to BTO, this report can benefit BEM professionals (architects, mechanical engineers, energy consultants, building auditors, equipment manufacturers, and BEM software vendors) and BEM clients (building owners and operators, EE program administrators, EE service providers, policymakers, and policy and code jurisdictions such as states and cities). This report was developed in two phases. In the first, BTO worked with a team from Navigant Consulting (now Guidehouse) to characterize objectives, opportunities, and current activities; identify gaps and barriers; and define initiatives. To collect input, Navigant conducted telephone interviews and workshops with industry experts. The initial phase produced a draft report, which was released for public review in 2016 and yielded over 400 comments. Based on these comments, BTO compiled a second draft report that addressed many of those comments while acknowledging changes that had occurred both at BTO and in the industry. Unlike the first draft report, the second focused much more heavily on BTO’s own role, portfolio, and activities. BTO is a direct player in the BEM field - it funds the development of several significant software packages that are embedded in commercial products - and transparency about its goals and future plans is requisite. BTO recognizes that a great number of other public and private organizations contribute to the BEM enterprise. With the second draft report, BTO did not attempt to produce a blueprint for the industry as a whole, but rather a working document BTO can use to iteratively solicit stakeholder input and synthesize it into a program. BTO released the second draft report for public review in 2019. The second round of review generated 83 pages of feedback and comments - almost exactly the length of the draft report itself - a significant portion of which was collected and synthesized by IBPSA-USA Advocacy Committee. This final report incorporates this feedback. This report does not address the use of BEM in support of building-based grid services, a recent BTO initiative called Grid-interactive Efficient Buildings (GEB). In 2019, BTO published a report that specifically addresses the role of BEM - and other “integration” technologies - in GEB.
One overarching objective of this program of study was the accumulation of objective, scientifically valid information relating to auditory performance of bald and golden eagles that may be used to guide the development of acoustic alerting/deterrence technologies intended to discourage encroachment into wind energy air spaces. To that end, analyses aimed at the characterization of sensitivity to sound in bald and golden eagles, along with findings in the supra-threshold, dynamic frequency spaces related to response latencies and amplitudes, leads us to conclude that bald, and golden eagles navigate the same basic working auditory space, as in other known and thus far characterized members of the diurnal raptor family. Specifically, bald and golden eagles, along with other raptor species within the group, operate in an auditory space characterized by a frequency band at least four octaves wide and centered on 2 kHz, with an upper frequency limit between 6 and 10 kHz at 80 dB SPL and a lower frequency limit that almost certainly extends below 0.2 kHz. Consequently, we recommend that signal designers use these data as a guideline in efforts to design effective and efficient acoustic alerting/deterrent systems. It is important to note that signal energy broadcast outside of this frequency band at moderate levels will not contribute to the efficacy of a deterrent but will add an unnecessary fraction to the overall acoustic pollution budget. The importance of this consideration is heightened by contemporaneous concerns related to the transmission of noise broadcast by wind energy farms. In addition, based on analyses of data acquired from red-tailed hawks using the same experimental paradigm and data acquisition system, we conclude that auditory function in the red-tailed hawk is sufficiently like that observed in bald and golden eagles to permit its use as a surrogate species. Response waveforms, threshold-frequency curves, and input-output characteristics match those of eagles closely. It should be noted however, that differences in sensitivity and slightly extended high-frequency limits of hearing should be taken into account when extrapolating findings from one species to the others. Although the inclusion of behavioral tests of red-tailed hawks to acoustic stimuli was beyond the scope of this investigation, future efforts to assess response parameters like signal-type preference and habituation rate will further elucidate their suitability to serve as eagle surrogates in behavioral studies; nonetheless, the species in question are well matched with respect to basic auditory performance. A second essential objective of this program of study was the acoustic characterization of a subset of calls comprising the vocal repertoires of bald and golden eagles that may be used to supplement auditory performance findings in the effort to guide the development of acoustic alerting signals. With regard to that objective, the vocal repertoires of both bald and golden eagle species are rich and varied. While similar in spectrographic structure, distinctive differences are also clear. Generally, golden eagles produce some calls with shorter durations, and similar “sounding” calls exhibit distinctively different spectrographic patterns than those of bald eagles. Both species produce calls that contain a wide variety of nonlinear elements that operate to enhance the rich and varied nature of commonly observed vocal products. Comparison of the average power spectra of commonly observed bald and golden eagle calls with threshold-frequency curves leads to the conclusion that call energies fall within the frequency bounds of hearing. Further, the acoustic energy of calls considered in this report tend to fall into overlapping, but different frequency ranges of the acoustic sensitivity curve. This condition may encourage signal designers to vary the frequency content of acoustic deterrence signals in the field. Finally, preliminary observations relating to the tendencies and proclivities of bald eagles to attend to the acoustic landscape lead to the conclusion that eagles monitor their immediate sound environment assiduously. Individuals respond to a variety of natural and synthetic sound signals reliably and, perhaps most relevant in the context of the engineering of acoustic alerting/deterrence technologies, habituation to most sounds considered in this effort was minimal. These preliminary results, while calling for extended behavioral testing, are promising and set the stage for the exportation of behavioral studies into real world scenarios.
While standardized ontologies like the Brick schema address data heterogeneity in Building Automation Systems (BAS), accessing this semantic data remains a challenge as domain experts often lack the expertise to formulate complex SPARQL queries. To bridge this gap, we present BrickQA, a Large Language Model (LLM)-based framework that translates natural language into executable SPARQL queries through structured query decomposition, dynamic schema exploration, and inline validation. BrickQA utilizes an iterative reasoning agent to actively navigate graph topology through dynamic exploration actions without requiring exhaustive context injection or model fine-tuning. This approach effectively mitigates hallucinations, particularly in large-scale building knowledge graphs. Empirical evaluation on BuildingQA, a standardized benchmark, demonstrates that BrickQA significantly outperforms ReAct baselines, delivering a 0.291–0.355 absolute F1 improvement while achieving 3 × –12.7 × higher token cost-efficiency. Beyond these metrics, the framework maintains structural fidelity across heterogeneous buildings and remains resilient to ambiguous queries without requiring site-specific fine-tuning. Furthermore, a case study on operational analytics validates the framework’s capability to handle temporal and aggregation constraints, effectively transforming abstract semantic models into actionable facility management insights.1
The Navigation and Meteorological Data from Multiple Sensors on Airborne Platform (NAVMET-AIR) value-added product encompasses the aafnaviwg data set in the ARM Data Center (ADC), known as the IWG file, named for the Inter-Agency Working Group for Airborne Data and Telemetry Systems (IWGADTS). Its purpose is to produce a suite of tools to promote standardization of instrument interface, data format, and data processing. This data set’s contents are navigational and meteorological state variables at 1 Hz. It also contains higher-order data flags created by ARM Aerial Facility (AAF) scientists to aid analysis such as periods when the aircraft is flying level, operating in maneuvers, or flying through cloud. Due to the nature of research flights, the payload of the aircraft consists of many duplicate and overlapping instruments to ensure, in case of instrument failure, there is a backup to record the data. Measurements from multiple instruments are consolidated into a single file. Each variable of this data set is carefully chosen from the onboard instrumentation and quality checked by AAF scientists to create this wholistic and most accurate data set for airborne research. This document is intended to be a hub towards the individual “read me” files produced for each campaign. For reference about what instrument was used for calculations on specific days, the read me files are located in the ARM IOP archive at the addresses listed, or access the IOP (intensive operational period) database at https://www.archive.arm.gov/.
This white paper presents a comprehensive methodology for assessing utilities’ cloud postures and frameworks. It aims to produce a roadmap and strategy for a cloud-enabled grid future, providing guidance for integrators, asset owners, and operators. Instead of offering a yes or no answer for cloud implementation, this framework offers strategic guidance on responsibly preparing for and deploying cloud solutions. This paper delves into cloud-service models pertinent to the electric sector, dissecting the shared responsibility model and elucidating what on-premise infrastructure as a service (IaaS), platform as a service (PaaS), and software as a service (SaaS) entail. A pivotal consideration within the context of the shared responsibility model is the allocation of responsibility for foundational security aspects—a decision that will be informed by a comprehensive risk assessment. The ensuing discussion will present a checklist of certifications necessary for a secure cloud transition, equipping utilities with the knowledge to navigate this digital transformation with confidence and with a strategic roadmap. Furthermore, the paper outlines gaps in understanding the U.S. government’s role in shaping technology development and responsible use. Its purpose is to aid decision-making by offering support for risk-informed solutions that benefit those managing assets and operating in the cloud environment. The primary objective is to enhance the resilience and future readiness of a decarbonized electric grid, with cloud solutions as one viable option. The paper synthesizes information on current and future grid architectures and applications, considering both conservative and progressive energy transitions, along with scalable and distributed computing considerations.
The blue economy is a dynamic and rapidly growing movement that captures the interplay between economic, social, and ecological sustainability of the ocean and encompasses numerous maritime sectors and activities (e.g., commerce and trade; living resources; renewable energy; minerals, materials, and freshwater; and ocean health and data). The demand for ocean data to inform scientific, risk reduction, and national security needs is leading to a large increase in the number of deployed ocean observation and monitoring systems, most of which require increased power. Because ocean observation systems are often placed in remote locations, they primarily rely on energy storage (or in some cases in situ energy generation) to power instruments and equipment, which imposes limits on sampling rates, deployment times, and spatiotemporal resolution of data. The U.S. Department of Energy Water Power Technologies Office is exploring the potential for marine renewable energy (MRE) devices (largely wave and tidal energy converters) to provide power to support multiple blue economy opportunities. A portion of these opportunities focus on power at sea markets for providing power in off-grid and offshore locations to support a variety of ocean-based activities, including ocean observation and navigation, underwater vehicle charging, marine aquaculture, marine algae farming, and seawater mining. Initially, research has focused on better understanding how and where MRE can provide a consistent source of reliable power to extend ocean observing missions, including operation of autonomous underwater vehicles. Online surveys as well as phone and in-person interviews were conducted with experts in the field of ocean observing systems and observatories to gather end-user requirements, determine energy needs, identify opportunities for codevelopment, and pinpoint constraints for MRE to meet those needs. The surveys and interviews provided feedback on the potential for powering devices and vehicles using MRE, including identifying common themes and challenges that will inform foundational research and development steps needed to advance the integration of MRE with ocean observing systems. In most cases, additional power generation on the order of watts was identified as significantly beneficial to enhancing ocean observations capabilities.
The blue economy is a dynamic and rapidly growing movement that captures the interplay between economic, social, and ecological sustainability of the ocean and encompasses numerous maritime sectors and activities (e.g., commerce and trade; living resources; renewable energy; minerals, materials, and freshwater; and ocean health and data). The demand for ocean data to inform scientific, risk reduction, and national security needs is leading to a large increase in the number of deployed ocean observation and monitoring systems, most of which require increased power. Because ocean observation systems are often placed in remote locations, they primarily rely on energy storage (or in some cases in situ energy generation) to power instruments and equipment, which imposes limits on sampling rates, deployment times, and spatiotemporal resolution of data. The U.S. Department of Energy Water Power Technologies Office is exploring the potential for marine renewable energy (MRE) devices (largely wave and tidal energy converters) to provide power to support multiple blue economy opportunities. A portion of these opportunities focus on power at sea markets for providing power in off-grid and offshore locations to support a variety of ocean-based activities, including ocean observation and navigation, underwater vehicle charging, marine aquaculture, marine algae farming, and seawater mining. Initially, research has focused on better understanding how and where MRE can provide a consistent source of reliable power to extend ocean observing missions, including operation of autonomous underwater vehicles. Online surveys as well as phone and inperson interviews were conducted with experts in the field of ocean observing systems and observatories to gather end-user requirements, determine energy needs, identify opportunities for codevelopment, and pinpoint constraints for MRE to meet those needs. The surveys and interviews provided feedback on the potential for powering devices and vehicles using MRE, including identifying common themes and challenges that will inform foundational research and development steps needed to advance the integration of MRE with ocean observing systems. In most cases, additional power generation on the order of watts was identified as significantly beneficial to enhancing ocean observations capabilities.
This presentation explores the evolving definition of “critical load” in the electric grid, emphasizing the growing importance of digital infrastructure—particularly data centers—in grid resilience, restoration, and modernization. As utilities increasingly rely on AI-driven analytics and software-defined control systems, data centers have shifted from passive electricity consumers to essential computational hubs that enable National Critical Functions (NCFs) and support real-time grid operations. The deck examines the scale and impact of digital loads, the need for grid modernization to manage rapid load growth, and the diverse computing paradigms required for AI deployment. It introduces a tiered taxonomy for classifying critical loads, highlights operational dependencies between the grid and digital infrastructure, and discusses policy implications for integrating data centers into emergency planning and restoration protocols. Through case studies and practical frameworks, the presentation provides actionable insights for utilities, regulators, and planners navigating the digital transformation of the power sector.
Recent advancements in mobile robotics have displayed impressive capabilities in traversing and accessing areas that are inaccessible to humans either due to the characteristics of the environment or potential hazards. Furthermore, these advancements within the field of mobile robotics, more specifically uncrewed ground vehicles (UGVs), give the ability to potentially survey, observe, and map these inaccessible areas for humans. However, one of the most challenging areas to implement this technology is underground environments. The main challenge with implementing this technology in underground environments is the dependence on either GPS or RF communication for UGVs to navigate properly. Therefore, in order to properly demonstrate the mapping capabilities of the UGV this challenge must be resolved. The overall goal of this study is to demonstrate the mapping capabilities of a UGV while addressing this challenge and documenting the implementation and testing phase of the robot. The proposed solution to this challenge is to implement a SLAM algorithm onto the main computational device of the UGV utilizing the Robot Operating System (ROS). The algorithm is the open-source software package Slam Toolbox developed by Steve Macenski. Furthermore, the sllidar_ros2 package from Slamtec will be used to gather the lidar data from an A3M1 2D lidar. A separate program will be created to gather odometry information for our UGV robot. All of these software packages will run together in a Docker environment. Through working on this project I have developed a better understanding of the world of robotics/autonomous systems, especially with applications such as navigation and mapping. Furthermore, through this project, I have been given exposure to how research is conducted within a DOE lab setting. As robotics/autonomous systems become more advanced it's important to pursue more avenues of research such as this project as it will ensure the development of our capabilities.
In response to growing risks of power outages from extreme weather and aging infrastructure, communities are increasingly exploring the potential of microgrids to provide reliable energy access. Microgrids offer promising solutions to meet this challenge but face a complex landscape of non-technical barriers, particularly regulations concerning the provision and distribution of energy. Most existing legal and regulatory frameworks were designed for a centralized, one-way power system, and are often poorly suited to handle systems that independently balance distributed energy resources with local load. This three-part report series provides a strategic analysis of existing regulatory and legal factors affecting microgrid deployments to help non-technical community leaders and decision-makers better understand the feasibility of a microgrid in their community. -Report No. 1: Foundational Issues Facing Microgrids details the universal policy barriers all microgrids face, including utility interconnection processes, rate structures, and local permitting. -Report No. 2: Single Property Microgrids outlines how direct asset ownership and operating behind-the-meter can bypass some regulatory oversight, using the Blue Lake Rancheria microgrid as a case study. -Report No. 3: Multi-Property Microgrids tackles the complex challenges of crossing public rights-of-way and navigating utility franchise rights, highlighting the Coventry microgrid project. The series equips decision-makers with phased frameworks to navigate financial and regulatory complexities, engage effectively with local utilities and Authorities Having Jurisdiction (AHJs), and structure successful microgrid projects.
The high and rising cost of preserving and delivering housing in the U.S. requires changes to existing practices of finance, design, and construction. Innovative methods such as industrialized construction could offer the means to address housing undersupply while reducing delivery costs, operational costs and material waste in the building industry, but they face challenges to success and to scale. Simultaneously, construction and cleantech innovators themselves face skepticism from the traditional entrepreneurial ecosystem such as incubators and accelerators while attempting to navigate systems level challenges. To respond to this need, various public and private sector stakeholders have launched initiatives to support innovative companies. These include nonprofits such as Terner Labs and Ivory Innovations offering curated programming to architecture, engineering, and construction (AEC) startups; housing developers in Minnesota and California "bundling" multiple projects together to reach economies of scale with a consistent project team; public and private sector entities developing "catalogues" of pre-approved home designs in the U.S. and Canada. This exploratory paper documents several of these emerging ecosystem-development efforts to support innovative housing approaches, characterizing them by leading stakeholder and intervention strategy based on publicly available information. The paper finds that these initiatives share similar high level goals but vary in implementation, reflecting different stakeholder priorities, regional market and policy dynamics, and housing typologies. The early stage of these efforts offer limited data for comparing actual outcomes, but the paper highlights common qualitative themes and identifies opportunities for further research and potential coordination among these efforts.
SAND2022-14694 O CrystalVR is a CAD2VR plugin which enables the import of crystallographic information files (CIF) into a virtual reality (VR) environment for visualization and interaction. Crystallographic data is incredibly complex and difficult to navigate and visualize on a two-dimensional or 2D screen. CrystalVR empowers users to step inside their crystals, allowing them to intuitively grow, shrink, and scale the crystal structure. Sandia National Laboratories is a multimission laboratory managed and operated by National Technology & Engineering Solutions of Sandia, LLC, a wholly owned subsidiary of Honeywell International Inc., for the U.S. Department of Energy’s National Nuclear Security Administration under contract DE-NA0003525.
Boiler failure could cause loss of life and safety issues, cost hundreds of thousands of dollars in equipment repairs, property damage and production losses, and drive up the cost of electric power. Boiler maintenance is challenging and risky for inspectors working on scaffolding in confined hazardous spaces inside of a boiler and sometimes the space is hard to access. The operation is also time-consuming due to the large area of vertical structures for inspection and the tremendous effort needed for scaffolding. Recently, the use of robotics (e.g., drones and crawlers) in power plants for maintenance is growing rapidly. However, the existing robotics solutions show two notable technological gaps: no live repair capability, and no Artificial Intelligence (AI) for smart autonomy. The objective of this project is to develop an integrated autonomous robotic platform that is equipped with compact non-destructive evaluation (NDE) sensors to perform live inspection, operates onboard repair devices to perform live repair, and uses AI for intelligent data fusion and predictive analysis for automated and smart spatiotemporal inspection, analysis and repair of the furnace walls in coal-fired boilers. The approach to achieve the objective includes developing NDE sensors with signal processing techniques, designing and evaluating repair devices for robots based on fusion and solid-state technologies, and an autonomous robotic platform that can attach to and navigate on boiler furnace walls using magnetic drive tracks. The robot is also powered by AI to automate data gathering (e.g., 3D mapping and damage localization) and predictive analysis. This project has advanced the state-of-the-art by providing technological breakthroughs including compact NDE and repair tools for robots, AI capabilities for smart autonomy, and a robotic platform for automated boiler maintenance. This project has great potential to result in significant benefits including limiting or eliminating the need to send operators to assess difficult-to-access or hazardous areas, enabling automated live inspection and repair, avoiding time consuming scaffolding (especially for partial maintenance during unplanned outage), collecting comprehensive and well-organized data smartly, and avoiding or limiting the need for onsite or remote piloting technicians. The impacts can be tremendous in terms of the time and cost savings, reducing the risk for human operators, and increasing boiler reliability, usability, and efficiency. In addition, by developing the new technologies on the autonomous inspection and repair robot, by involving multiple undergraduate and graduate students working together with the faculty members on this project, and by generating knowledge and building up collaborations with industrial partners, this effort will significantly update the education capabilities, support long-term fundamental research, and maintain the leadership of Colorado School of Mines and Michigan State University in energy fields.
AMLRO (Active Machine Learning Reaction Optimizer) is an open-source framework designed to accelerate chemical reaction optimization using active learning with classical machine learning regression models. AMLRO integrates space-filling sampling strategies (e.g., Sobol and Latin Hypercube sampling) with iterative model training, prediction, and experiment selection to efficiently navigate complex reaction spaces. The platform supports multiple regression models, flexible multi-objective definitions, and user-defined parameter bounds, enabling data-efficient optimization from small initial datasets. AMLRO is designed for ease of use by experimentalists and can operate as a standalone decision-support tool or be integrated into closed-loop automated experimentation workflows.
Deep Eutectic Solvents (DESs) are a promising class of solvents for CO 2 capture. DESs are complex mixtures that can be designed to optimize CO solubility and overall capture process efficiency. However, the vast design landscape of DES mixtures makes experimental investigation prohibitive; as such, there is a need for computational models that can quickly and efficiently navigate the design space and inform data collection efforts. In this work, we propose Graph Neural Network (GNN) models for predicting CO 2 solubility for DESs; the GNN leverages a mixture graph representation that captures the molecular structure of the DES components as well as their intermolecular interactions. Here, we compare the GNN framework against alternative architectures (neural networks, graph convolution networks, and random forests) and data representations (molecular fingerprints, sigma profiles, and graphs). We show that the proposed approach offers superior predictive performance; specifically, we show that solubility can be predicted reliably directly from molecular structure (without the need of using sigma profiles as proposed in previous studies). This result is important, as obtaining sigma profiles requires expensive density functional theory computations. We also explored the ability of GNNs to predict solubility for new DES mixtures and operating conditions. We found that the model extrapolates across temperature reliably. However, we also found deficiencies in the ability of the model to predict solubility for DES mixtures, pressures, and molar ratio not included in the training sets; we show that this is due to an inherent lack of chemical diversity in datasets available in the literature. The proposed computational capabilities can thus help navigate the design space of DES and inform data collection efforts. Our models, data, and benchmarks are shared as Python code implemented in Jupyter notebooks.
Automated experiments in 4D scanning transmission electron microscopy (STEM) are implemented for rapid discovery of local structures, symmetry-breaking distortions, and internal electric and magnetic fields in complex materials. Deep kernel learning enables active learning of the relationship between local structure and 4D-STEM-based descriptors. With this, efficient and “intelligent” probing of dissimilar structural elements to discover desired physical functionality is made possible. This approach allows effective navigation of the sample in an automated fashion guided by either a predetermined physical phenomenon, such as strongest electric field magnitude, or in an exploratory fashion. Therefore, we verify the approach first on preacquired 4D-STEM data and further implement it experimentally on an operational STEM. The experimental discovery workflow is demonstrated using graphene and subsequently extended toward a lesser-known layered 2D van der Waals material, MnPS 3 . This approach establishes a pathway for physics-driven automated 4D-STEM experiments that enable probing the physics of strongly correlated systems and quantum materials and devices, as well as exploration of beam-sensitive materials.
This poster details the motor control improvements to Fermilab's Remote Viewing Robot (RVR). It is a robot tasked with remotely investigating issues within the accelerator tunnels at Fermilab. Initially controlled by a single Raspberry Pi that housed all the robot s operations, the RVR will now use a Raspberry Pi Pico W for its motor control. This was achieved using Pulse Width Modulation (PWM) to allow for precise speed and torque control for the robot. This enhances the robot s reliability. By addressing the challenge of navigating high radiation environments, the upgrade helps assist the RVR s goal of decreasing the need for human intervention in accelerator tunnels.