Results for “Computer Vision” — Page 35

NASA NTRS2024

Search Technology for Optimal Rescue Missions (STORM)

Natural disasters, such as earthquakes, hurricanes, and wildfires are responsible for the deaths of 60,000 to 90,000 people per year. Today, search and rescue (SAR) operations heavily rely on humans to find and deliver life-saving supplies to those affected by these disasters. However, these operations have limits in visibility, navigation, communication systems, and data availability in the area affected, as well as endangering the SAR personnel. Search Technology for Optimal Rescue Missions (STORM) discusses a new system for SAR teams using autonomous drones able to find and deliver supplies to people, without risking more lives in the process. The concept includes the use of two drone types, STORM Search and STORM Rescue, which will survey and locate survivors and be able to drop equipment to the survivors identified, respectively. These two types of drones were optimized in drone design and durability (such as the use of dihedral wings and a toroidal propeller), detection and navigation systems (sturdy thermal and Light Detection and Ranging [LiDAR] cameras), automation and system design (Machine Learning and Computer Vision), server-drone communication (Meshnets), weight, and cost. Once implemented, the STORM concept is expected to improve, ease, and speed up SAR operations, and most important of all, rescue lives that would have never been currently possible to find.

Astha Ingole↗

NASA NTRSDate not supplied

New Tools for Automating Arcjet Sample Recession Tracking and Analysis

Arcjet Computer Vision (arcjetCV) has been significantly upgraded to enhance accuracy and performance in tracking material recession and shock-material standoff in test videos. These improvements include integrating new machine learning models, developing a specialized edge detection class, and incorporating a more comprehensive training dataset. These upgrades have refined the software’s ability to automate time-resolved recession tracking, making it more precise and reliable for analyzing complex physical processes. In parallel, a new tool called STARscan (Spatial Targeting and Alignment Rig for Scanning) is being developed to capture detailed 3D surface data before and after testing. By comparing these pre- and post-test scans with arcjetCV’s automated video analysis results, users can achieve a more comprehensive assessment of material recession. This method enables cross-validation of results, improving confidence in the analysis of tested materials. The expanded capabilities of arcjetCV have been successfully demonstrated on videos from various facilities, including the NASA Ames arcjets, UIUC’s PlasmatronX, and the VKI Plasmatron. It has been adopted as a new standard for in-situ recession tracking by the Mars Sample Return Project and Orion. ArcjetCV’s improved efficiency and accuracy are critical for reducing testing uncertainties and validating heatshield material performance under extreme conditions. The software’s user-friendly graphical interface ensures ease of use, enabling seamless processing and precise analysis of arcjet videos, providing deeper insights into material behavior in hypersonic environments. ArcjetCV is now available on both PyPI and Conda, allowing easy installation via "pip install arcjetCV" or through the Conda package manager, ensuring broad accessibility and streamlined deployment for users across various platforms.

Ablation↗

NASA NTRSDate not supplied

3D Scanning System to Assess Gravity-Dependent Body Shape Changes

The human body shows unique morphological changes when exposed to different gravity conditions, including muscle atrophy, fluid shift, and spinal elongation. Such changes need to be incorporated for human-system integration in the vehicle habitat, garment, and spacesuit designs, as inaccurate body measurements can result in suboptimal crew protection that can potentially decrease injury tolerance. However, measurement tools have not been available for accurate assessments of body shape changes. This work aimed to develop a prototype 3-D body scanning system with the configuration and performance optimized for in-flight crewmember body scanning. A scan hardware system was developed using Intel RealSense commercial off-the-shelf 3D sensors. The sensor parameters were iteratively optimized and tested to obtain the performance level needed for body scanning. A scan booth structure was fabricated, with the overall size 4 x 4 x 8 feet. The specific number of sensors and mounting positions were determined by iterative simulations, which indicated that 16 cameras can capture 94% and 96% of body surface area from the 1st percentile female and 99th percentile male crew population subject. The mounted sensors were linked through a mix of USB-C and USB-3 cables and operated for data acquisition from a Linux laptop computer. A software prototype was developed using Python and Tkinter graphical user interface toolkit. A calibration procedure was also built using a panel of QR codes. A computer vision tool detected and decoded the unique ID and pattern locations of the QR codes. The calibration information determined the position and orientation of the 3D sensors with respect to each other. The scanner performance was assessed using a set of 3D printed custom manikins. The manikin size and shape were derived from the previous ISS study, which measured the crewmembers’ anthropometry changes across the different flight phases. The average anthropometric measurements at the pre-flight and flight day 15 were sampled and projected onto the 1st percentile female and 99th percentile male body shapes. Another pair of manikins were also 3D printed to simulate the neutral body posture, estimated from ISS microgravity environments. A preliminary analysis assessed the performance of the newly developed scanner against the reference scanner, which has been used at the NASA JSC for crew and test subject anthropometry. Although the new scanner data showed several artifacts and missing geometries in the occluded body areas such as armpits and crotch, overall shape matched with the reference scan. When the manikin surface coordinates were compared, a root mean square error of 1.3 cm was observed from the manikin torso segment. Linear measurements including the stature, knee height and circumference measurements at the chest and calf showed differences from the reference scan measurements, ranging between 0.3 and 0.9 cm. Overall, this work demonstrated a development framework for an in-flight scanner with design and operation optimized for crewmember body scanning. Further improvement can potentially provide previously unavailable anthropometric data from different gravitational environments, including 0-g, 1/6-g, and 1-g. Such data can improve suit fit, habitat design, exercise efficacy quantification and sizing of orthostatic intolerance garments.

K H Kim↗

NASA NTRSDate not supplied

Developing Methods for Exercise System Kinematics Tracking

BACKGROUND How to quantify the load and forces produced by exercise equipment and their Vibration Isolation and Stabilization (VIS) platforms in-flight is an active area of investigation. Kinematic tracking paired with system modeling can provide insights as well as verification and validation of simulations used for system design and development. Traditional motion capture methods can require significant cost in equipment procurement and crew-time, but newer lessons learned can be leveraged [1]. The VIS systems of current and future exercise hardware on the International Space Station (ISS) such as the Cycle Ergometer with Vibration Isolation System (CEVIS) and the European Enhanced Exploration Exercise Device (E4D) are not currently outfitted with IMUs or similar measurement devices. Video-based methods would enable use of multi-purpose, crew-familiar flight equipment. An initial exploration of video-based solutions was performed utilizing 2-camera video from crew cycling on Teal-CEVIS on the ISS. METHODS AND RESULTS Our group has scoped a variety of video-based object tracking methods. To date, we have primarily investigated computer vision toolkits such as openCV. Techniques explored include key-point detection, background subtraction, region-of interest tracking, color-based tracking, tag masking and tracking, and corner detection. Although object-tracking and 6D pose estimation is a rich field, space applications are a unique problem that are challenging for existing software and toolkits. The majority of the existing object-tracking applications involve vehicles/pedestrians and household objects with simple backgrounds. We have identified the following features which pose particular challenges for on-station exercise equipment tracking: Busy and visually cluttered background Low-textured tracking object with relatively small motions Occlusions and motion by human subject and loose, floating objects Limited number of video cameras with no fixed global references Limited ability to add tags, markers, or visual references to the tracking object Lack of training data for Machine Learning (ML) algorithms CONCLUSION We will summarize the efficacy of techniques tested for a ground mock-trial and the on-station exercise trial. It is likely that human-in-loop feedback or a conglomerate of methods is required. ML-based methods, like those implemented for human body tracking [2], may still be a viable option, but more training data and validation is needed.

L B Nilsson↗

NASA NTRS2024

3D Cloud Masking Across A Broad Swath Using Multi-Angle Polarimetry and Deep Learning

Understanding the 3-dimensional structure of clouds is of crucial importance to modeling our changing climate. Active sensors, such as radar and lidar, provide accurate vertical cloud profiles, but are mostly restricted to along-track sampling. Passive sensors can capture a wide swath, but struggle to see beneath cloud tops. In essence, both types of products are restricted to two dimensions: as a cross-section in the active case, and an image in the passive case. However, multi-angle sensor configurations contain implicit information about 3D structure, due to parallax and atmospheric path differences. Extracting that implicit information can be challenging, requiring computationally expensive radiative transfer techniques. Machine learning, as an alternative, may be able to capture some of the complexity of a full 3D radiative transfer solution with significantly less computational expense. In this work, we make three contributions towards understanding 3D cloud structure from multi-angle polarimetry. First, we introduce a large-scale, publicly available dataset that fuses existing cloud products into a format more amenable to machine learning. This dataset treats multi-angle polarimetry as an input, and radar-based vertical cloud profiles as an output. Second, we describe and evaluate strong baseline machine learning models based that predict these profiles from the passive imagery. Notably, these models are trained only on center-swath labels, but can predict cloud profiles over the entire passive imagery swath. Third, we leverage the information-theoretic nature of machine learning to draw conclusions about the relative utility of various sensor configurations, including spectral channels, viewing angles, and polarimetry. These findings have implications for Earth-observing missions such as NASA’s Plankton, Aerosol, Cloud-ocean Ecosystem (PACE) and Atmosphere Observing System (AOS) missions, as well as in informing future applications of computer vision to atmospheric remote sensing.

clouds↗

NASA NTRS2022

Bayesian Deep Learning for Segmentation for Autonomous Safe Planetary Landing

Hazard detection is critical for enabling autonomous landing on planetary surfaces. Current state-of-the-art methods leverage traditional computer vision approaches to automate the identification of safe terrain from input digital elevation models (DEMs). However, performance for these methods can degrade for input DEMs with increased sensor noise. In the last decade, deep learning techniques have been developed for various applications. Nevertheless, their applicability to safety-critical space missions has often been limited due to concerns regarding their outputs’ reliability. In response to these limitations, this paper proposes an application of the Bayesian deep learning segmentation method for hazard detection. The developed approach enables reliable, safe landing site detection by i) generating simultaneously a safety prediction map and its uncertainty map via Bayesian deep learning and semantic segmentation, and ii) using the uncertainty map to filter out the uncertain pixels in the prediction map so that the safe site identification is performed only based on the certain pixels (i.e., pixels for which the model is certain about its safety prediction). Experiments are presented with simulated data based on a Mars HiRISE digital terrain model by varying uncertainty threshold and noise levels to demonstrate the performance of the proposed approach.

Kento Tomita↗

NASA NTRSDate not supplied

ARCTRON: A Rapid Experimental Proving Ground for TPS Experiments and Arcjet Technology Development

Innovation in high-enthalpy facilities is fundamentally limited by the cost and risk of experimentation. New concepts for plasma control, diagnostics, facility components, and plasma-material interaction often require repeated iterations that are impractical to perform in production arcjets. As a result, promising ideas may remain unexplored or reach operational facilities only after significant development effort. ARCTRON is being developed as a rapid experimental proving ground where new ideas in plasma science, arcjet engineering, diagnostics, and material response can be conceived, tested, and quantitatively evaluated before transition to large-scale facilities. The platform combines radio-frequency (RF) and DC arc plasma generation, externally applied magnetic fields, configurable gas composition, reduced-pressure operation, laser heating, electrical biasing, and modular diagnostic access. These capabilities permit the plasma source, applied forcing, test article, and measurement configuration to be modified independently, allowing individual physical mechanisms to be isolated more readily than in a traditional test environment. One class of investigations addresses fundamental plasma-surface interaction physics. Conventional material tests often expose a specimen simultaneously to convective heating, reactive species, pressure, shear, radiation, and surface-current effects. The resulting material response may be measured accurately, while the contribution of each mechanism remains difficult to identify. ARCTRON is designed to vary these effects selectively. Plasma chemistry can be changed independently through configurable gas mixtures; magnetic fields and electrical biasing can modify charged-particle transport; laser heating can provide a non-plasma thermal input; and pressure, flow, and discharge mode can be varied over a broad operating space. This enables controlled tests of hypotheses involving surface catalycity, reactive-species transport, plasma-assisted oxidation, electromagnetic effects, shear, and the relative contributions of thermal and chemical loading. A second class of investigations enabled by this approach concerns the engineering of high-enthalpy facilities themselves. Arc-heated facilities are limited by electrode erosion, unstable arc attachment, localized heating, and damage to nozzles and other plasma-facing components. ARCTRON provides a lower-cost environment for testing concepts intended to mitigate these limitations. Candidate investigations include the use of applied magnetic fields to alter current paths and reduce plasma interaction with nozzle walls, ExB forcing to introduce controlled plasma rotation, magnetic or geometric approaches for distributing arc attachment, and alternative electrode or discharge configurations intended to reduce erosion and improve stability. Because the platform is reconfigurable, these concepts can be evaluated through repeated design--build--test cycles before they are considered for implementation in operational facilities. The platform also supports the development and validation of diagnostics that may be difficult to introduce initially into a large arcjet. Current and planned measurements include spatially resolved optical emission spectroscopy, electrostatic probes, fast imaging, pyrometry, calorimetry, laser-induced fluorescence, and absorption spectroscopy. These diagnostics are intended not merely to document a nominal operating condition, but to constrain the local plasma state and its relationship to component or material response. The modular facility geometry allows diagnostic concepts to be tested, calibrated, and compared under repeatable conditions before deployment in more demanding environments. ARCTRON is also supported by an integrated software suite. Automated control and data acquisition allow discharge parameters, gas composition, magnetic fields, diagnostic timing, and test configuration to be recorded as part of each experiment (STARDAC - Software for Testing, Analysis, Research Data, and Control). The Backend for Experiment Analysis, Storage, and Traceability (BEAST) is a database that provides the infrastructure needed to associate heterogeneous measurements with facility configuration, specimen identity, calibration state, geometry, and analysis provenance. This backend is particularly important for exploratory campaigns, in which many related configurations may be tested, and the value of an individual experiment depends on its connection to earlier and subsequent iterations. Complementary analysis capabilities, including computer-vision-based transient response measurements (arcjetCV), three-dimensional surface reconstruction (STARSCAN), and model-based Bayesian inference (SHIELD), and tomography data analysis (TOMATO, PuMA) can be incorporated when required by a specific hypothesis without becoming the focus of every campaign. The central objective of ARCTRON is therefore not to maximize heat flux or reproduce a complete flight environment. Its purpose is to reduce the cost and time required to ask consequential questions about plasma behavior, plasma-facing materials, diagnostics, and arcjet technology. By providing a controlled environment for rapid reconfiguration, mechanism isolation, quantitative measurement, and iterative engineering, ARCTRON can help mature concepts that would otherwise remain too speculative or too risky for evaluation in production facilities. The resulting knowledge can then guide the design of material models, focus test objectives in larger arcjets, reduce facility-development risk, and improve the physical basis of high-enthalpy ground testing. This work will present the ARCTRON architecture, operating modes, diagnostic suite, and digital experimental workflow. Initial experimental results from the first integrated operation of the facility will be presented, including flow characterization, power limitations, and deployment of the initial diagnostic suite. Ongoing development efforts aimed at catalycity characterization, magnetic plasma control, and advanced optical diagnostics will also be discussed, illustrating how the platform supports rapid iteration from concept to experiment.

experimental diagnostics↗

NASA NTRSDate not supplied

BEAST: Expanding Sustainable Data Infrastructure for High-Enthalpy Facilities

Reproducible, data-driven thermal protection system (TPS) research requires that experimental records from high-enthalpy testing be consistently structured, traceable, and accessible across campaigns and institutions. In practice, however, arcjet and plasma facilities data remain largely fragmented: raw diagnostics are stored in ad hoc formats, material sample histories are disconnected from test conditions, and metadata standards are absent, precluding systematic cross-campaign analysis and long-term reuse. BEAST (Backend for Experiment Analysis, Storage, and Traceability) is an open-source, web-based platform that addresses these limitations by providing a unified, queryable infrastructure for high-enthalpy ground-test data [1]. First presented at the 15th Ablation Workshop [2], BEAST has since undergone significant development. The platform ingests and structures multi-channel time-series diagnostics, facility configurations, and material property records within a common provenance model, ensuring end-to-end traceability from raw sensor acquisition to reduced experimental quantities. A versioned material library links specimen identity and processing history to the specific runs in which each sample was tested. An integrated modeling workbench enables training and evaluation of regression models directly on archived experimental data, supporting condition interpolation and the construction of empirical material response databases. Beyond its original deployment at NASA Ames Research Center, BEAST has been designed to be facility-agnostic, with ongoing efforts to extend its adoption to other facilities. Its modular architecture accommodates heterogeneous diagnostic setups and facility types, and its future open-source distribution allows institutions to build on a common data standard rather than maintaining isolated, bespoke solutions. BEAST is further integrated within a broader ecosystem of companion tools: arcjetCV [3] extracts recession rates and shock standoff distances from high-speed video using computer vision, and miniSTARscan [4] provides sub-minute, portable photogrammetric surface reconstruction of test articles before and after exposure. All tools share a common data schema, enabling seamless ingestion of surface geometry, imagery, and time-series data into a single, coherent experimental record.

Database↗

NASA NTRSDate not supplied

ARCTRON: A Rapid Experimental Proving Ground for TPS Experiments and Arcjet Technology Development

Innovation in high-enthalpy facilities is fundamentally limited by the cost and risk of experimentation. New concepts for plasma control, diagnostics, facility components, and plasma-material interaction often require repeated iterations that are impractical to perform in production arcjets. As a result, promising ideas may remain unexplored or reach operational facilities only after significant development effort. ARCTRON is being developed as a rapid experimental proving ground where new ideas in plasma science, arcjet engineering, diagnostics, and material response can be conceived, tested, and quantitatively evaluated before transition to large-scale facilities. The platform combines radio-frequency (RF) and DC arc plasma generation, externally applied magnetic fields, configurable gas composition, reduced-pressure operation, laser heating, electrical biasing, and modular diagnostic access. These capabilities permit the plasma source, applied forcing, test article, and measurement configuration to be modified independently, allowing individual physical mechanisms to be isolated more readily than in a traditional test environment. One class of investigations addresses fundamental plasma-surface interaction physics. Conventional material tests often expose a specimen simultaneously to convective heating, reactive species, pressure, shear, radiation, and surface-current effects. The resulting material response may be measured accurately, while the contribution of each mechanism remains difficult to identify. ARCTRON is designed to vary these effects selectively. Plasma chemistry can be changed independently through configurable gas mixtures; magnetic fields and electrical biasing can modify charged-particle transport; laser heating can provide a non-plasma thermal input; and pressure, flow, and discharge mode can be varied over a broad operating space. This enables controlled tests of hypotheses involving surface catalycity, reactive-species transport, plasma-assisted oxidation, electromagnetic effects, shear, and the relative contributions of thermal and chemical loading. A second class of investigations enabled by this approach concerns the engineering of high-enthalpy facilities themselves. Arc-heated facilities are limited by electrode erosion, unstable arc attachment, localized heating, and damage to nozzles and other plasma-facing components. ARCTRON provides a lower-cost environment for testing concepts intended to mitigate these limitations. Candidate investigations include the use of applied magnetic fields to alter current paths and reduce plasma interaction with nozzle walls, ExB forcing to introduce controlled plasma rotation, magnetic or geometric approaches for distributing arc attachment, and alternative electrode or discharge configurations intended to reduce erosion and improve stability. Because the platform is reconfigurable, these concepts can be evaluated through repeated design--build--test cycles before they are considered for implementation in operational facilities. The platform also supports the development and validation of diagnostics that may be difficult to introduce initially into a large arcjet. Current and planned measurements include spatially resolved optical emission spectroscopy, electrostatic probes, fast imaging, pyrometry, calorimetry, laser-induced fluorescence, and absorption spectroscopy. These diagnostics are intended not merely to document a nominal operating condition, but to constrain the local plasma state and its relationship to component or material response. The modular facility geometry allows diagnostic concepts to be tested, calibrated, and compared under repeatable conditions before deployment in more demanding environments. ARCTRON is also supported by an integrated software suite. Automated control and data acquisition allow discharge parameters, gas composition, magnetic fields, diagnostic timing, and test configuration to be recorded as part of each experiment (STARDAC - Software for Testing, Analysis, Research Data, and Control). The Backend for Experiment Analysis, Storage, and Traceability (BEAST) is a database that provides the infrastructure needed to associate heterogeneous measurements with facility configuration, specimen identity, calibration state, geometry, and analysis provenance. This backend is particularly important for exploratory campaigns, in which many related configurations may be tested, and the value of an individual experiment depends on its connection to earlier and subsequent iterations. Complementary analysis capabilities, including computer-vision-based transient response measurements (arcjetCV), three-dimensional surface reconstruction (STARSCAN), and model-based Bayesian inference (SHIELD), and tomography data analysis (TOMATO, PuMA) can be incorporated when required by a specific hypothesis without becoming the focus of every campaign. The central objective of ARCTRON is therefore not to maximize heat flux or reproduce a complete flight environment. Its purpose is to reduce the cost and time required to ask consequential questions about plasma behavior, plasma-facing materials, diagnostics, and arcjet technology. By providing a controlled environment for rapid reconfiguration, mechanism isolation, quantitative measurement, and iterative engineering, ARCTRON can help mature concepts that would otherwise remain too speculative or too risky for evaluation in production facilities. The resulting knowledge can then guide the design of material models, focus test objectives in larger arcjets, reduce facility-development risk, and improve the physical basis of high-enthalpy ground testing. This work will present the ARCTRON architecture, operating modes, diagnostic suite, and digital experimental workflow. Initial experimental results from the first integrated operation of the facility will be presented, including flow characterization, power limitations, and deployment of the initial diagnostic suite. Ongoing development efforts aimed at catalycity characterization, magnetic plasma control, and advanced optical diagnostics will also be discussed, illustrating how the platform supports rapid iteration from concept to experiment.

experimental diagnostics↗

NASA NTRSDate not supplied

BEAST: Expanding Sustainable Data Infrastructure for High-Enthalpy Facilities

Reproducible, data-driven thermal protection system (TPS) research requires that experimental records from high-enthalpy testing be consistently structured, traceable, and accessible across campaigns and institutions. In practice, however, arcjet and plasma facilities data remain largely fragmented: raw diagnostics are stored in ad hoc formats, material sample histories are disconnected from test conditions, and metadata standards are absent, precluding systematic cross-campaign analysis and long-term reuse. BEAST (Backend for Experiment Analysis, Storage, and Traceability) is an open-source, web-based platform that addresses these limitations by providing a unified, queryable infrastructure for high-enthalpy ground-test data [1]. First presented at the 15th Ablation Workshop [2], BEAST has since undergone significant development. The platform ingests and structures multi-channel time-series diagnostics, facility configurations, and material property records within a common provenance model, ensuring end-to-end traceability from raw sensor acquisition to reduced experimental quantities. A versioned material library links specimen identity and processing history to the specific runs in which each sample was tested. An integrated modeling workbench enables training and evaluation of regression models directly on archived experimental data, supporting condition interpolation and the construction of empirical material response databases. Beyond its original deployment at NASA Ames Research Center, BEAST has been designed to be facility-agnostic, with ongoing efforts to extend its adoption to other facilities. Its modular architecture accommodates heterogeneous diagnostic setups and facility types, and its future open-source distribution allows institutions to build on a common data standard rather than maintaining isolated, bespoke solutions. BEAST is further integrated within a broader ecosystem of companion tools: arcjetCV [3] extracts recession rates and shock standoff distances from high-speed video using computer vision, and miniSTARscan [4] provides sub-minute, portable photogrammetric surface reconstruction of test articles before and after exposure. All tools share a common data schema, enabling seamless ingestion of surface geometry, imagery, and time-series data into a single, coherent experimental record.

Database↗

DOE OSTI2012

Communication and Energy Efficiency in Visual Sensor Networks for People Localization

This paper addresses the communication and energy efficiency in collaborative visual sensor networks (VSNs) for people localization, a challenging computer vision problem of its own. We focus on the design of a light-weight and energy efficient solution where people are localized based on distributed camera nodes integrating the so-called certainty map generated at each node, that records the target non-existence information within the camera s field of view. We first present a dynamic itinerary for certainty map integration where not only each sensor node transmits a very limited amount of data but that a limited number of camera nodes is involved. Then, we perform a comprehensive analytical study to evaluate communication and energy efficiency between different integration schemes, i.e., centralized and distributed integration. Based on results obtained from analytical study and real experiments, the distributed method shows effectiveness in detection accuracy as well as energy and bandwidth efficiency.

Karakaya, Mahmut↗

DOE OSTI2015

An Evaluation of Image Based Techniques for Early Wildfire Detection and Fuel Mapping

Few events can cause the catastrophic impact to ecology, infrastructure, and human safety of a wildland fire along the wildland urban interface. The suppression of natural wildland fires over the past decade has caused a buildup of dry, dead surface fuels: a condition that, coupled with the right weather conditions, can cause large destructive wildfires that are capable of threatening both ancient tree stands and manmade infrastructure. Firefighters use fire danger models to determine staffing needs on high fire risk days; however models are only as effective as the spatial and temporal density of their observations. OKFIRE, an Oklahoma initiative created by a partnership between Oklahoma State University and the University of Oklahoma, has proven that fire danger assessments close to the fire – both geographically and temporally – can give firefighters a significant increase in their situational awareness while fighting a wildland fire. This paper investigates several possible solutions for a small Unmanned Aerial System (UAS) which could gather information useful for detecting ground fires and constructing fire danger maps. Multiple fire detection and fuel mapping programs utilize satellites, manned aircraft, and large UAS equipped with hyperspectral sensors to gather useful information. Their success provides convincing proof of the utility that could be gained from low-altitude UAS gathering information at the exact time and place firefighters and land managers are interested in. Close proximity, both geographically and operationally, to the end can reduce latency times below what could ever be possible with satellite observation. This paper expands on recent advances in computer vision, photogrammetry, and infrared and color imagery to develop a framework for a next-generation UAS which can assess fire danger and aid firefighters in real time as they observe, contain, or extinguish wildland fires. It also investigates the impact information gained by this system could have on pre-fire risk assessments through the development of very high resolution fuel maps.

54 ENVIRONMENTAL SCIENCES↗

DOE OSTI2015

Evidence of Long Range Dependence and Self-similarity in Urban Traffic Systems

Transportation simulation technologies should accurately model traffic demand, distribution, and assignment parame- ters for urban environment simulation. These three param- eters significantly impact transportation engineering bench- mark process, are also critical in realizing realistic traffic modeling situations. In this paper, we model and charac- terize traffic density distribution of thousands of locations around the world. The traffic densities are generated from millions of images collected over several years and processed using computer vision techniques. The resulting traffic den- sity distribution time series are then analyzed. It is found using the goodness-of-fit test that the traffic density dis- tributions follows heavy-tail models such as Log-gamma, Log-logistic, and Weibull in over 90% of analyzed locations. Moreover, a heavy-tail gives rise to long-range dependence and self-similarity, which we studied by estimating the Hurst exponent (H). Our analysis based on seven different Hurst estimators strongly indicate that the traffic distribution pat- terns are stochastically self-similar (0.5 H 1.0). We believe this is an important finding that will influence the design and development of the next generation traffic simu- lation techniques and also aid in accurately modeling traffic engineering of urban systems. In addition, it shall provide a much needed input for the development of smart cities.

long range dependence↗

DOE OSTI2016

Learning Weight Uncertainty with Stochastic Gradient MCMC for Shape Classification

Learning the representation of shape cues in 2D & 3D objects for recognition is a fundamental task in computer vision. Deep neural networks (DNNs) have shown promising performance on this task. Due to the large variability of shapes, accurate recognition relies on good estimates of model uncertainty, ignored in traditional training of DNNs, typically learned via stochastic optimization. This paper leverages recent advances in stochastic gradient Markov Chain Monte Carlo (SG-MCMC) to learn weight uncertainty in DNNs. It yields principled Bayesian interpretations for the commonly used Dropout/DropConnect techniques and incorporates them into the SG-MCMC framework. Extensive experiments on 2D & 3D shape datasets and various DNN models demonstrate the superiority of the proposed approach over stochastic optimization. Our approach yields higher recognition accuracy when used in conjunction with Dropout and Batch-Normalization.

convolutional neural network↗

DOE OSTI2017

AN INDUCTIVE MAPPING WITH CONVOLUTIONAL REPRESENTATIONS FOR HUMAN SETTLEMENT DETECTION: PRELIMINARY RESULTS

Undoubtedly, deep convolutional learning methods continue to improve performance in image-level classification for computer vision and remote sensing applications. However, the spatio-temporal nature of remote sensing imagery offers other interesting challenges in multiscale image understanding. Emerging opportunities include seeking fine-grained and neighborhood mapping with overhead imagery. Limitations due to lack of relevant scale ground-truth often mandates that these challenges are pursued as disjoint. We test this premise and explore representation spaces from a single deep convolutional network and their visualization to argue for a unified feature extraction framework. The objective is to utilize and re-purpose trained feature extractors without the need for network retraining on three remote sensing tasks i.e. superpixel mapping, pixel-level segmentation and semantic based image visualization. By leveraging the same convolutional feature extractors and viewing them as visual information extractors that encode different image representation spaces, we demonstrate surprising potential on multiscale experiments that incorporate edge-level details up to semantic-level informationWe test this premise and explore representation spaces from a single deep convolutional network and their visualization to argue for a novel unified feature extraction framework. The objective is to utilize and re-purpose trained feature extractors without the need for network retraining on three remote sensing tasks i.e. superpixel mapping, pixel-level segmentation and semantic based image visualization. By leveraging the same convolutional feature extractors and viewing them as visual information extractors that encode different image representation spaces, we demonstrate a preliminary inductive transfer learning potential on multiscale experiments that incorporate edge-level details up to semantic-level informationWe test this premise and explore representation spaces from a single deep convolutional network and their visualization to argue for a novel unified feature extraction framework. The objective is to utilize and re-purpose trained feature extractors without the need for network retraining on three remote sensing tasks i.e. superpixel mapping, pixel-level segmentation and semantic based image visualization. By leveraging the same convolutional feature extractors and viewing them as visual information extractors that encode different image representation spaces, we demonstrate a preliminary inductive transfer learning potential on multiscale experiments that incorporate edge-level details up to semantic-level information.

Lunga, Dalton↗

DOE OSTI2020

Graph Neural Networks for Particle Reconstruction in High Energy Physics detectors

Pattern recognition problems in high energy physics are notably different from traditional machine learning applications in computer vision. Reconstruction algorithms identify and measure the kinematic properties of particles produced in high energy collisions and recorded with complex detector systems. Two critical applications are the reconstruction of charged particle trajectories in tracking detectors and the reconstruction of particle showers in calorimeters. These two problems have unique challenges and characteristics, but both have high dimensionality, high degree of sparsity, and complex geometric layouts. Graph Neural Networks (GNNs) are a relatively new class of deep learning architectures which can deal with such data effectively, allowing scientists to incorporate domain knowledge in a graph structure and learn powerful representations leveraging that structure to identify patterns of interest. In this work we demonstrate the applicability of GNNs to these two diverse particle reconstruction problems.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗

DOE OSTI2020

Systematic Evaluation of Backdoor Data Poisoning Attacks on Image Classifiers

Backdoor data poisoning attacks have recently been demonstrated in computer vision research as a potential safety risk for machine learning (ML) systems. Traditional data poisoning attacks manipulate training data to induce unreliability of an ML model, whereas backdoor data poisoning attacks maintain system performance unless the MLmodel is presented with an input containing an embedded“trigger” that provides a predetermined response advantageous to the adversary. Our work builds upon prior back-door data-poisoning research for ML image classifiers and systematically assesses different experimental conditions including types of trigger patterns, persistence of trigger patterns during retraining, poisoning strategies, architectures (ResNet-50, NasNet, NasNet-Mobile), datasets (Flowers, CIFAR-10), and potential defensive regularization techniques (Contrastive Loss, Logit Squeezing, Manifold Mixup,Soft-Nearest-Neighbors Loss). Experiments yield four key findings. First, the success rate of backdoor poisoning at-tacks varies widely, depending on several factors, including model architecture, trigger pattern and regularization technique. Second, we find that poisoned models are hard to detect through performance inspection alone. Third, regularization typically reduces backdoor success rate, although it can have no effect or even slightly increase it, depending on the form of regularization. Finally, backdoors inserted through data poisoning can be rendered ineffective after just a few epochs of additional training on a small set of clean data without affecting the model’s performance.

Truong, Loc T.↗

DOE OSTI2020

Machine Learning for Automated Extraction of Building Geometry

As data science comes to buildings, the promise of using machine learning and novel sources of data has received much attention. Advances in machine learning and computer vision algorithms, combined with increased access to unstructured data (e.g., images and text), have created an opportunity for automated extraction of building characteristics – cost-effectively, and at scale. Acquisition of features such as footprint are time consuming and costly to acquire with today’s manual methods, but can be streamlined through intelligent software-based solutions applied to satellite images. When combined with aerial RGB and thermal images, full 3D geometries and thermal maps can be constructed to determine additional characteristics such as window to wall ratio, height, number of stories and envelope thermal characteristics. In this paper we present three contributions to accelerate these high potential opportunities: (1) a methodical analysis of how these features can be integrated into today’s simulation and data driven software tools to enhance efficiency measure identification and owner/operator decision making; (2) development and accuracy testing of open source deep neural network methods to extract building footprints from satellite imagery, including the curation and application of openly available GIS datasets for training and continued development by others; and (3) an open framework for drone-based image capture and creation of 3D building geometries. This work represents an important bridge between high-level studies that span diverse application areas and those that detail point solutions yet cannot be easily replicated or extended.

Touzani, Samir↗