Search NASA⌕ Search

SEARCH · Search NASA

Results for “Semantic segmentation”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 91 records · Page 5
All sourcesNASA NTRSDOE OSTI
DOE OSTI2021

Disentangling Rotational Dynamics and Ordering Transitions in a System of Self-Organizing Protein Nanorods via Rotationally Invariant Latent Representations

The dynamics of complex ordering systems with active rotational degrees of freedom exemplified by protein self-assembly is explored using a machine learning workflow that combines deep learning-based semantic segmentation and rotationally invariant variational autoencoder-based analysis of orientation and shape evolution. The latter allows for disentanglement of the particle orientation from other degrees of freedom and compensates for lateral shifts. The disentangled representations in the latent space encode the rich spectrum of local transitions that can now be visualized and explored via continuous variables. The time dependence of ensemble averages allows insight into the time dynamics of the system and, in particular, illustrates the presence of the potential ordering transition. Finally, analysis of the latent variables along the single-particle trajectory allows tracing these parameters on a single-particle level. The proposed approach is expected to be universally applicable for the description of the imaging data in optical, scanning probe, and electron microscopy seeking to understand the dynamics of complex systems where rotations are a significant part of the process.

representation learning↗
DOE OSTI2024

Deep learning approaches for instantaneous laser absorptance prediction in additive manufacturing

Abstract The quantification of absorbed light is essential for understanding laser-material interactions and melt pool dynamics in order to minimize defects in additively manufactured metal components. The geometry of a vapor depression formed during laser melting is closely related to laser energy absorption. This relationship has been observed by the state-of-the-art in situ high-speed synchrotron X-ray visualization and integrating sphere radiometry. These two techniques create a temporally resolved dataset consisting of vapor depression images and corresponding laser absorptance. In this work, we propose two different approaches to predict instantaneous laser absorptance. The end-to-end approach uses deep convolutional neural networks to learn implicit features of X-ray images automatically and predict the laser energy absorptance. The two-stage approach uses a semantic segmentation model to engineer geometric features and predict absorptance using classical regression models. While having distinct advantages, both approaches achieved a consistently low mean absolute error of less than 3.3%.

Chemistry↗
DOE OSTI2022

Towards automating structural discovery in scanning transmission electron microscopy *

Abstract Scanning transmission electron microscopy is now the primary tool for exploring functional materials on the atomic level. Often, features of interest are highly localized in specific regions in the material, such as ferroelectric domain walls, extended defects, or second phase inclusions. Selecting regions to image for structural and chemical discovery via atomically resolved imaging has traditionally proceeded via human operators making semi-informed judgements on sampling locations and parameters. Recent efforts at automation for structural and physical discovery have pointed towards the use of ‘active learning’ methods that utilize Bayesian optimization with surrogate models to quickly find relevant regions of interest. Yet despite the potential importance of this direction, there is a general lack of certainty in selecting relevant control algorithms and how to balance a priori knowledge of the material system with knowledge derived during experimentation. Here we address this gap by developing the automated experiment workflows with several combinations to both illustrate the effects of these choices and demonstrate the tradeoffs associated with each in terms of accuracy, robustness, and susceptibility to hyperparameters for structural discovery. We discuss possible methods to build descriptors using the raw image data and deep learning based semantic segmentation, as well as the implementation of variational autoencoder based representation. Furthermore, each workflow is applied to a range of feature sizes including NiO pillars within a La:SrMnO 3 matrix, ferroelectric domains in BiFeO 3 , and topological defects in graphene. The code developed in this manuscript is open sourced and will be released at github.com/nccreang/AE_Workflows .

47 OTHER INSTRUMENTATION↗
DOE OSTI2024

Unpaired image translation to mitigate domain shift in liquid argon time projection chamber detector responses

Deep learning algorithms often are developed and trained on a training dataset and deployed on test datasets. Any systematic difference between the training and a test dataset may severely degrade the final algorithm performance on the test dataset—what is known as the domain shift problem . This issue is prevalent in many scientific domains where algorithms are trained on simulated data but applied to real-world datasets. Typically, the domain shift problem is solved through various domain adaptation (DA) methods. However, these methods are often tailored for a specific downstream task, such as classification or semantic segmentation, and may not easily generalize to different tasks. This work explores the feasibility of using an alternative way to solve the domain shift problem that is not specific to any downstream algorithm. The proposed approach relies on modern Unpaired Image-to-Image (UI2I) translation techniques, designed to find translations between different image domains in a fully unsupervised fashion. In this study, the approach is applied to a domain shift problem commonly encountered in Liquid Argon Time Projection Chamber (LArTPC) detector research when seeking a way to translate samples between two differently distributed LArTPC detector datasets deterministically. This translation allows for mapping real-world data into the simulated data domain where the downstream algorithms can be run with much less domain-shift-related performance degradation. Conversely, using the translation from the simulated data to a real-world domain can increase the realism of the simulated dataset and reduce the magnitude of any systematic uncertainties. To evaluate the quality of the translations, we use both pixel-wise metrics and a downstream task to measure the effectiveness of UI2I methods for mitigating the domain shift problem. We adapted several popular UI2I translation algorithms to work on scientific data and demonstrated the viability of these techniques for solving the domain shift problem with LArTPC detector data. To facilitate further development of DA techniques for scientific datasets, the ‘Simple Liquid-Argon Track Samples’ dataset used in this study is also published.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2024

GWAS supported by computer vision identifies large numbers of candidate regulators of in planta regeneration in Populus trichocarpa

Plant regeneration is an important dimension of plant propagation and a key step in the production of transgenic plants. However, regeneration capacity varies widely among genotypes and species, the molecular basis of which is largely unknown. Association mapping methods such as genome-wide association studies (GWAS) have long demonstrated abilities to help uncover the genetic basis of trait variation in plants; however, the performance of these methods depends on the accuracy and scale of phenotyping. To enable a large-scale GWAS of in planta callus and shoot regeneration in the model tree Populus, we developed a phenomics workflow involving semantic segmentation to quantify regenerating plant tissues over time. We found that the resulting statistics were of highly non-normal distributions, and thus employed transformations or permutations to avoid violating assumptions of linear models used in GWAS. We report over 200 statistically supported quantitative trait loci (QTLs), with genes encompassing or near to top QTLs including regulators of cell adhesion, stress signaling, and hormone signaling pathways, as well as other diverse functions. Our results encourage models of hormonal signaling during plant regeneration to consider keystone roles of stress-related signaling (e.g. involving jasmonates and salicylic acid), in addition to the auxin and cytokinin pathways commonly considered. The putative regulatory genes and biological processes we identified provide new insights into the biological complexity of plant regeneration, and may serve as new reagents for improving regeneration and transformation of recalcitrant genotypes and species.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2025

Describing Point Defect Topology in 2D Energy Materials through Computer Vision

Point defects such as vacancies and impurity atoms strongly impact the performance of 2D materials. Traditional efforts often rely on manual detection, a process that is time-intensive, prone to human error, and challenging to scale. Here we leverage machine learning (ML) methods to identify and quantify vacancies within 2D transition metal carbides (Ti3C2, MXenes), aiming to expedite detection while improving accuracy. MXenes exhibit valuable defect-defined electrochemical properties, but we currently lack statistical understanding of defect topology needed to fully harness these materials. Here we employ a convolutional neural network for semantic segmentation of experimental MXene images, opening an opportunity to conduct a rigorous statistical study on defect hierarchy while investigating local relaxation in the lattice. We show how the integration of ML can yield fundamental insight into point defects, providing a powerful tool that will play an increasingly crucial role in the future of materials science.

2d materials↗
DOE OSTI2024

Human-in-the-Loop: The Future of Machine Learning in Automated Electron Microscopy

Machine learning (ML) methods are progressively gaining acceptance in the electron microscopy community for de-noising, semantic segmentation, and dimensionality reduction of data post-acquisition. The introduction of the application programming interfaces (APIs) by major instrument manufacturers now allows the deployment of ML workflows in microscopes, not only for data analytics but also for real-time decision-making and feedback for microscope operation. However, the number of use cases for real-time ML remains remarkably small. Furthermore, we discuss some considerations in designing ML-based active experiments and pose that the likely strategy for the next several years will be human-in-the-loop automated experiments (hAE). In this paradigm, the ML learning agent directly controls beam position and image and spectroscopy acquisition functions, and a human operator monitors experiment progression in real and feature space of the system and tunes the policies of the ML agent to steer the experiment toward specific objectives.

47 OTHER INSTRUMENTATION↗
DOE OSTI2021

Entropy and Boundary Based Adversarial Learning for Large Scale Unsupervised Domain Adaptation

Supervised semantic segmentation methods provide state-of-the-art performance, but their performance is limited by the amount of quality labeled data they need for training. Scarcity of labeled data and non-transferablity of models, due to cross-domain discrepancy makes it a bigger challenge for remote sensing imagery analysis. In this work, we approach this problem through adversarial learning, driven by entropy and boundary of region-of-interest for unsupervised domain adaptation. This concept helps with better boundary prediction and encourages target domain entropy maps (probability/uncertainty maps) to be similar to source domains. In particular, we showed that deriving informative entropy through the adversarial learning is essential to enable the adaptation. We used a large scale cross country building extraction dataset to validate the framework. The experimental results show the usefulness of considering boundary and entropy driven adversarial learning for adaptation.

Makkar, Nikhil↗
DOE OSTI2025

DeepAndes: A Self-Supervised Vision Foundation Model for Multispectral Remote Sensing Imagery of the Andes

By mapping sites at large scales usingremotely sensed data, archaeologists can generate unique insights into long-term demographic trends, interregional social networks, and human adaptations in the past. Remote sensing surveys complement field-based approaches, and their reach can be especially great when combined with deep learning and computer vision techniques. However, conventional supervised deep learning methods face challenges in annotating fine-grained archaeological features at scale. In addition, while recent vision foundation models have shown remarkable success in learning large-scale remote sensing data with minimal annotations, most off-the-shelf solutions are designed for RGB images rather than multispectral satellite imagery, such as the eight-band data used in our study. In this article, we introduce DeepAndes, a transformer-based vision foundation model trained on three million multispectral satellite images, specifically tailored for Andean archaeology. DeepAndes incorporates a customized DINOv2 self-supervised learning algorithm optimized for eight-band multispectral imagery, marking the first foundation model designed explicitly for the Andes region. We evaluate its image understanding performance through imbalanced image classification, image instance retrieval, and pixel-level semantic segmentation tasks. Our experiments show that DeepAndes achieves superior F1 scores, mean average precision, and Dice scores in few-shot learning scenarios, significantly outperforming models trained from scratch or pretrained on smaller datasets. This underscores the effectiveness of large-scale self-supervised pretraining in archaeological remote sensing.

Guo, Junlin [Vanderbilt Univ., Nashville, TN (Unit↗
DOE OSTI2023

Coarse-to-fine Task-driven Inpainting for Geoscience Images

The processing and recognition of geoscience images have wide applications. Most of existing researches focus on understanding the high-quality geoscience images by assuming that all the images are clear. However, in many real-world cases, the geoscience images might contain occlusions during the image acquisition. This problem actually implies the image inpainting problem in computer vision and multimedia. As far as we know, all the existing image inpainting algorithms learn to repair the occluded regions for a better visualization quality, they are excellent for natural images but not good enough for geoscience images, and they never consider the following geoscience task when developing inpainting methods. Here, this paper aims to repair the occluded regions for a better geoscience task performance and advanced visualization quality simultaneously, without changing the current deployed deep learning based geoscience models. Because of the complex context of geoscience images, we propose a coarse-to-fine encoder-decoder network with the help of designed coarse-to-fine adversarial context discriminators to reconstruct the occluded image regions. Due to the limited data of geoscience images, we propose a MaskMix based data augmentation method, which augments inpainting masks instead of augmenting original images, to exploit the limited geoscience image data. The experimental results on three public geoscience datasets for remote sensing scene recognition, cross-view geolocation and semantic segmentation tasks respectively show the effectiveness and accuracy of the proposed method. The code is available at: https://github.com/HMS97/Task-driven-Inpainting.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2024

OReole-FM: successes and challenges toward billion-parameter foundation models for high-resolution satellite imagery

While the pretraining of Foundation Models (FMs) for remote sensing (RS) imagery is on the rise, models remain restricted to a few hundred million parameters. Scaling models to billions of parameters has been shown to yield unprecedented benefits including emergent abilities, but requires data scaling and computing resources typically not available outside industry R&D labs. In this work, we pair high-performance computing resources including Frontier supercomputer, America's first exascale system, and high-resolution optical RS data to pretrain billion-scale FMs. Our study assesses performance of different pretrained variants of vision Transformers across image classification, semantic segmentation and object detection benchmarks, which highlight the importance of data scaling for effective model scaling. Moreover, we discuss construction of a novel TIU pretraining dataset, model initialization, with data and pretrained models intended for public release. By discussing technical challenges and details often lacking in the related literature, this work is intended to offer best practices to the geospatial community toward efficient training and benchmarking of larger FMs.

Ambrozio Dias, Philipe↗
DOE OSTI2021

Aerial 3D Building Reconstruction from Drone Imagery (A3DBR) v1

This toolkit is composed of several modules for extracting buildings geometrical and thermal characteristics from RGB and thermal imagery captured using a drone. - Building 3D reconstruction module: leverage a photogrammetry software to construct a 3D point cloud from RGB drone imagery, which is then used in conjunction with image processing and geometric methods to extract building footprint and building height (i.e., 3D model of the building). - Windows to wall ratio estimation module: leverage deep learning semantic segmentation modeling to detect windows on 2D drone RGB images. The detected windows are then projected onto the extracted building 3D model (using building 3D reconstruction module) and their area is computed to obtain window to wall ratio estimation. - Thermal anomalies detection module: leverage image processing and machine learning algorithm to detect on 2D drone thermal images potential thermal anomalies within building's facades and roofs.

Granderson, Jessica↗
DOE OSTI2025

Describing Point Defect Topology in 2D Energy Materials Through Computer Vision

Point defects such as vacancies and impurity atoms strongly impact the performance of 2D materials. Traditional efforts often rely on manual detection, a process that is time-intensive, prone to human error, and challenging to scale. Here we leverage machine learning (ML) methods to identify and quantify vacancies within 2D transition metal carbides (Ti3C2, MXenes), aiming to expedite detection while improving accuracy. MXenes exhibit valuable defect-defined electrochemical properties, but we currently lack statistical understanding of defect topology needed to fully harness these materials. We employ a convolutional neural network for semantic segmentation of experimental MXene images, opening an opportunity to conduct a rigorous statistical study on defect hierarchy while investigating local relaxation in the lattice. We show how the integration of ML can yield fundamental insight into point defects, providing a powerful tool that will play an increasingly crucial role in the future of materials science.

2D materials↗
DOE OSTI2026

Describing Point Defect Topology in 2D Energy Materials Through Computer Vision

Point defects such as vacancies and impurity atoms strongly impact the performance of 2D materials. Traditional efforts often rely on manual detection, a process that is time-intensive, prone to human error, and challenging to scale. Here we leverage machine learning (ML) methods to identify and quantify vacancies within 2D transition metal carbides (Ti3C2, MXenes), aiming to expedite detection while improving accuracy. MXenes exhibit valuable defect-defined electrochemical properties, but we currently lack statistical understanding of defect topology needed to fully harness these materials. Here we employ a convolutional neural network for semantic segmentation of experimental MXene images, opening an opportunity to conduct a rigorous statistical study on defect hierarchy while investigating local relaxation in the lattice. We show how the integration of ML can yield fundamental insight into point defects, providing a powerful tool that will play an increasingly crucial role in the future of materials science. ML is often not just a matter of straightforward application, and pretrained models proved ineffective in this case. Instead, we trained our own neural network (NN) and applied data augmentation techniques and fine-tuning to the training dataset. Since labeled microscopy data is often scarce, we developed training data from a previously published wide-frame MXene image, using customized Gaussian fitting to locate atomic positions. Our trained model was then applied to a large dataset of experimental images, enabling a statistical study of defect configurations across three samples prepared with different HF etchant concentrations (5%, 9.1%, and 12.5%), as shown in Fig. 1. This also allowed us to investigate local strain around vacancies, though we find that we are limited by the precision of measurements using high-angle annular dark field (HAADF) images, as shown in Fig. 2. This study demonstrates how ML enables large-scale, quantitative analysis of atomic defects - an otherwise infeasible task with traditional methods. While our NN was specialized for Ti3C2 MXenes, the pipeline we developed provides a foundation for future ML models tailored to other materials. Ultimately, we envision embedding the NN onto the microscope to give real-time feedback to the user. To make this a reality, continued work is necessary to fully understand the NN's capabilities and limitations. This study gets one step closer to our goals of automated experimentation moving away from traditional methods of manual labeling. As ML capabilities advance, we hope to continue adapting and applying these techniques in microscopy.

2D materials↗
DOE OSTI2023

Expanded genetic variation (SNP) and phenomics (image based) dataset for Populus trichocarpa

The image dataset consists of 11,791 images representing 1,219 genotypes of Populus trichocarpa undergoing in planta regeneration. Genotypes were imaged with a median of four weekly timepoints and a median of two replicates each. A representative and diverse subset of 249 images was annotated using the IDEAS annotation interface (ideas.eecs.oregonstate.edu) and these annotated images were used to train a deep semantic segmentation model (PSPNet), which was deployed for inference over the entire dataset. Annotated classes include specific stages of regeneration (callus and shoot) in addition to unregenerated plant material and background. Statistics of relative tissue area were extracted and used for downstream genetic association mapping in a genome-wide association study. The SNP dataset consists of over 40 million single-nucleotide polymorphisms across 1,323 wild accessions of Populus trichocarpa

09 BIOMASS FUELS↗
DOE OSTI2021

AI-Enabled Robotic NDE for Structural Damage Assessment and Repair

The aim of this paper is to develop the concept and a prototype of an intelligent mobile robotic platform that is integrated with nondestructive evaluation (NDE) capabilities for autonomous live inspection and repair. In many industrial environments, such as the application of power plant boiler inspection, human inspectors often have to perform hazardous and challenging tasks. There is a significant chance of injury, considering the confined spaces and limited visibility of the inspection environment and hazards such as pressurization and improper water levels. In order to provide a solution to eliminate these dangers, the concept of a new robotic system was developed and prototyped that is capable of autonomously sweeping the region to be inspected. The robot design contains systematic integration of components from robotics, NDE, and artificial intelligence (AI). A magnetic track system is used to navigate over the vertical steel structures required for examination. While moving across the inspection area, the robot uses an NDE sensor to acquire data for inspection and repair. This paper presents a design of a portable NDE scanning system based on eddy current array probes, which can be customized and installed on various mobile robot platforms. Machine learning methods are applied for semantic segmentation that will simultaneously localize and recognize defects without the need of human intervention. Experiments were conducted that show the NDE and repair capabilities of the system. Improvements in human safety and structural damage prevention, as well as lowering the overall costs of maintenance, are possible through the implementation of this robotic NDE system.

Materials Science↗
DOE OSTI2023

Interface-enhanced conductivities in surfactant-mediated, solution-grown ionic crystalline complexes

Renewable energy is increasingly relying on optimized electrolytes and interfaces. In this work, Tween 20 and sodium chloride are selected as a model system to reveal the effects of surfactants on salt crystallization in the context of ionic conductivity and interface optimization. At a varied crystallization speed and mix ratio, it is demonstrated that the resultant solution-grown ionic crystalline complexes can achieve a highly tunable ion transport with a controllable crystalline interface. X-ray diffraction results rule out the possibility of polymorphism in the NaCl/Tween 20 systems, which further supports the importance of an optimized crystalline network for optimizing permittivity or ionic conductivity. Raman mapping and machine learning techniques are used to perform semantic segmentation on highly heterogeneous NaCl/Tween 20 complexes. Furthermore, FTIR measurements demonstrate that inter- and intra-molecular interactions play critical roles in the formation of these crystals. This work lays a foundation toward future optimization of such complex ion systems for a specific salt or crystallization modifier in energy storage or ion transport applications.

36 MATERIALS SCIENCE↗
DOE OSTI2026

Deep Learning and Photogrammetric Reconstruction for Automated Crack Detection and Dimensional Measurement in Mining Operations

Surface crack detection and dimensional measurement at active mining sites present significant safety and operational challenges. Manual inspection methods are labor-intensive, spatially incomplete, and expose personnel to hazardous environments, while existing automated approaches have been developed primarily for concrete civil infrastructure and have not been validated on the complex, variable surfaces characteristic of mining environments. This dissertation presents an automated pipeline that integrates deep learning semantic segmentation with Structure-from-Motion photogrammetry to detect surface cracks and measure their aperture, length, and vertical displacement from standard RGB imagery acquired during routine Uncrewed Aerial Vehicle (UAV) survey operations, without requiring additional sensor hardware or manual measurement. The pipeline combines a U-Net architecture with an EfficientNet-B0 encoder, pretrained on the SDNET2018 concrete crack dataset and fine-tuned on a mining-specific dataset spanning laboratory concrete specimens, coal refuse impoundment embankments, and post-blast limestone quarry benches. Photogrammetric reconstruction is performed using COLMAP Structure-from-Motion and Multi-View Stereo, with crack segmentation masks projected into the reconstructed point cloud to enable three-dimensional vertical displacement measurement through local plane fitting and bimodal surface detection. The pipeline was validated across 36 controlled laboratory specimens at three imaging distances and four vertical displacement levels, achieving aperture measurement RMSE of 0.047 cm and R² of 0.954, and vertical displacement RMSE of 0.140 cm and R² of 0.966, against independent caliper measurements. Field application at a coal refuse impoundment in southwestern Pennsylvania detected 71 crack components across the embankment crest, with a dominant longitudinal crack exhibiting aperture values reaching 28 cm and a 95th percentile vertical displacement of 35.53 cm, consistent in magnitude and spatial distribution with simultaneously acquired LiDAR-derived estimates. Application across four post-blast limestone quarry bench datasets in California successfully characterized blast-induced fracture networks at ground sampling distances ranging from 0.59 to 1.23 cm/pixel, with detected crack geometries physically consistent with observable surface conditions at each site. The results demonstrate that deep learning-based crack detection and photogrammetric measurement can be integrated into routine UAV inspection workflows at mining sites, providing repeatable, scalable, and quantitative crack characterization across surface types, crack scales, and displacement magnitudes not previously addressed in the literature. The pipeline requires no dedicated surveying equipment beyond the UAV platforms already deployed at mine sites for survey and monitoring purposes, supporting practical adoption within existing operational workflows.

Crack detection, Dimensional Measurement↗