SEARCH · Search NASA

Results for “Semantic segmentation”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 91 records · Page 5

All sources NASA NTRS DOE OSTI

DOE OSTI2025

Describing Point Defect Topology in 2D Energy Materials through Computer Vision

2d materials↗

DOE OSTI2024

Human-in-the-Loop: The Future of Machine Learning in Automated Electron Microscopy

Machine learning (ML) methods are progressively gaining acceptance in the electron microscopy community for de-noising, semantic segmentation, and dimensionality reduction of data post-acquisition. The introduction of the application programming interfaces (APIs) by major instrument manufacturers now allows the deployment of ML workflows in microscopes, not only for data analytics but also for real-time decision-making and feedback for microscope operation. However, the number of use cases for real-time ML remains remarkably small. Furthermore, we discuss some considerations in designing ML-based active experiments and pose that the likely strategy for the next several years will be human-in-the-loop automated experiments (hAE). In this paradigm, the ML learning agent directly controls beam position and image and spectroscopy acquisition functions, and a human operator monitors experiment progression in real and feature space of the system and tunes the policies of the ML agent to steer the experiment toward specific objectives.

47 OTHER INSTRUMENTATION↗

DOE OSTI2021

Entropy and Boundary Based Adversarial Learning for Large Scale Unsupervised Domain Adaptation

Supervised semantic segmentation methods provide state-of-the-art performance, but their performance is limited by the amount of quality labeled data they need for training. Scarcity of labeled data and non-transferablity of models, due to cross-domain discrepancy makes it a bigger challenge for remote sensing imagery analysis. In this work, we approach this problem through adversarial learning, driven by entropy and boundary of region-of-interest for unsupervised domain adaptation. This concept helps with better boundary prediction and encourages target domain entropy maps (probability/uncertainty maps) to be similar to source domains. In particular, we showed that deriving informative entropy through the adversarial learning is essential to enable the adaptation. We used a large scale cross country building extraction dataset to validate the framework. The experimental results show the usefulness of considering boundary and entropy driven adversarial learning for adaptation.

Makkar, Nikhil↗

DOE OSTI2025

DeepAndes: A Self-Supervised Vision Foundation Model for Multispectral Remote Sensing Imagery of the Andes

By mapping sites at large scales usingremotely sensed data, archaeologists can generate unique insights into long-term demographic trends, interregional social networks, and human adaptations in the past. Remote sensing surveys complement field-based approaches, and their reach can be especially great when combined with deep learning and computer vision techniques. However, conventional supervised deep learning methods face challenges in annotating fine-grained archaeological features at scale. In addition, while recent vision foundation models have shown remarkable success in learning large-scale remote sensing data with minimal annotations, most off-the-shelf solutions are designed for RGB images rather than multispectral satellite imagery, such as the eight-band data used in our study. In this article, we introduce DeepAndes, a transformer-based vision foundation model trained on three million multispectral satellite images, specifically tailored for Andean archaeology. DeepAndes incorporates a customized DINOv2 self-supervised learning algorithm optimized for eight-band multispectral imagery, marking the first foundation model designed explicitly for the Andes region. We evaluate its image understanding performance through imbalanced image classification, image instance retrieval, and pixel-level semantic segmentation tasks. Our experiments show that DeepAndes achieves superior F1 scores, mean average precision, and Dice scores in few-shot learning scenarios, significantly outperforming models trained from scratch or pretrained on smaller datasets. This underscores the effectiveness of large-scale self-supervised pretraining in archaeological remote sensing.

Guo, Junlin [Vanderbilt Univ., Nashville, TN (Unit↗

DOE OSTI2023

Coarse-to-fine Task-driven Inpainting for Geoscience Images

The processing and recognition of geoscience images have wide applications. Most of existing researches focus on understanding the high-quality geoscience images by assuming that all the images are clear. However, in many real-world cases, the geoscience images might contain occlusions during the image acquisition. This problem actually implies the image inpainting problem in computer vision and multimedia. As far as we know, all the existing image inpainting algorithms learn to repair the occluded regions for a better visualization quality, they are excellent for natural images but not good enough for geoscience images, and they never consider the following geoscience task when developing inpainting methods. Here, this paper aims to repair the occluded regions for a better geoscience task performance and advanced visualization quality simultaneously, without changing the current deployed deep learning based geoscience models. Because of the complex context of geoscience images, we propose a coarse-to-fine encoder-decoder network with the help of designed coarse-to-fine adversarial context discriminators to reconstruct the occluded image regions. Due to the limited data of geoscience images, we propose a MaskMix based data augmentation method, which augments inpainting masks instead of augmenting original images, to exploit the limited geoscience image data. The experimental results on three public geoscience datasets for remote sensing scene recognition, cross-view geolocation and semantic segmentation tasks respectively show the effectiveness and accuracy of the proposed method. The code is available at: https://github.com/HMS97/Task-driven-Inpainting.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2024

OReole-FM: successes and challenges toward billion-parameter foundation models for high-resolution satellite imagery

While the pretraining of Foundation Models (FMs) for remote sensing (RS) imagery is on the rise, models remain restricted to a few hundred million parameters. Scaling models to billions of parameters has been shown to yield unprecedented benefits including emergent abilities, but requires data scaling and computing resources typically not available outside industry R&D labs. In this work, we pair high-performance computing resources including Frontier supercomputer, America's first exascale system, and high-resolution optical RS data to pretrain billion-scale FMs. Our study assesses performance of different pretrained variants of vision Transformers across image classification, semantic segmentation and object detection benchmarks, which highlight the importance of data scaling for effective model scaling. Moreover, we discuss construction of a novel TIU pretraining dataset, model initialization, with data and pretrained models intended for public release. By discussing technical challenges and details often lacking in the related literature, this work is intended to offer best practices to the geospatial community toward efficient training and benchmarking of larger FMs.

Ambrozio Dias, Philipe↗

DOE OSTI2021

Aerial 3D Building Reconstruction from Drone Imagery (A3DBR) v1

This toolkit is composed of several modules for extracting buildings geometrical and thermal characteristics from RGB and thermal imagery captured using a drone. - Building 3D reconstruction module: leverage a photogrammetry software to construct a 3D point cloud from RGB drone imagery, which is then used in conjunction with image processing and geometric methods to extract building footprint and building height (i.e., 3D model of the building). - Windows to wall ratio estimation module: leverage deep learning semantic segmentation modeling to detect windows on 2D drone RGB images. The detected windows are then projected onto the extracted building 3D model (using building 3D reconstruction module) and their area is computed to obtain window to wall ratio estimation. - Thermal anomalies detection module: leverage image processing and machine learning algorithm to detect on 2D drone thermal images potential thermal anomalies within building's facades and roofs.

Granderson, Jessica↗

DOE OSTI2025

Describing Point Defect Topology in 2D Energy Materials Through Computer Vision

2D materials↗

DOE OSTI2026

Describing Point Defect Topology in 2D Energy Materials Through Computer Vision

Point defects such as vacancies and impurity atoms strongly impact the performance of 2D materials. Traditional efforts often rely on manual detection, a process that is time-intensive, prone to human error, and challenging to scale. Here we leverage machine learning (ML) methods to identify and quantify vacancies within 2D transition metal carbides (Ti3C2, MXenes), aiming to expedite detection while improving accuracy. MXenes exhibit valuable defect-defined electrochemical properties, but we currently lack statistical understanding of defect topology needed to fully harness these materials. Here we employ a convolutional neural network for semantic segmentation of experimental MXene images, opening an opportunity to conduct a rigorous statistical study on defect hierarchy while investigating local relaxation in the lattice. We show how the integration of ML can yield fundamental insight into point defects, providing a powerful tool that will play an increasingly crucial role in the future of materials science. ML is often not just a matter of straightforward application, and pretrained models proved ineffective in this case. Instead, we trained our own neural network (NN) and applied data augmentation techniques and fine-tuning to the training dataset. Since labeled microscopy data is often scarce, we developed training data from a previously published wide-frame MXene image, using customized Gaussian fitting to locate atomic positions. Our trained model was then applied to a large dataset of experimental images, enabling a statistical study of defect configurations across three samples prepared with different HF etchant concentrations (5%, 9.1%, and 12.5%), as shown in Fig. 1. This also allowed us to investigate local strain around vacancies, though we find that we are limited by the precision of measurements using high-angle annular dark field (HAADF) images, as shown in Fig. 2. This study demonstrates how ML enables large-scale, quantitative analysis of atomic defects - an otherwise infeasible task with traditional methods. While our NN was specialized for Ti3C2 MXenes, the pipeline we developed provides a foundation for future ML models tailored to other materials. Ultimately, we envision embedding the NN onto the microscope to give real-time feedback to the user. To make this a reality, continued work is necessary to fully understand the NN's capabilities and limitations. This study gets one step closer to our goals of automated experimentation moving away from traditional methods of manual labeling. As ML capabilities advance, we hope to continue adapting and applying these techniques in microscopy.

2D materials↗

DOE OSTI2023

Expanded genetic variation (SNP) and phenomics (image based) dataset for Populus trichocarpa

The image dataset consists of 11,791 images representing 1,219 genotypes of Populus trichocarpa undergoing in planta regeneration. Genotypes were imaged with a median of four weekly timepoints and a median of two replicates each. A representative and diverse subset of 249 images was annotated using the IDEAS annotation interface (ideas.eecs.oregonstate.edu) and these annotated images were used to train a deep semantic segmentation model (PSPNet), which was deployed for inference over the entire dataset. Annotated classes include specific stages of regeneration (callus and shoot) in addition to unregenerated plant material and background. Statistics of relative tissue area were extracted and used for downstream genetic association mapping in a genome-wide association study. The SNP dataset consists of over 40 million single-nucleotide polymorphisms across 1,323 wild accessions of Populus trichocarpa

09 BIOMASS FUELS↗

DOE OSTI2021

AI-Enabled Robotic NDE for Structural Damage Assessment and Repair

The aim of this paper is to develop the concept and a prototype of an intelligent mobile robotic platform that is integrated with nondestructive evaluation (NDE) capabilities for autonomous live inspection and repair. In many industrial environments, such as the application of power plant boiler inspection, human inspectors often have to perform hazardous and challenging tasks. There is a significant chance of injury, considering the confined spaces and limited visibility of the inspection environment and hazards such as pressurization and improper water levels. In order to provide a solution to eliminate these dangers, the concept of a new robotic system was developed and prototyped that is capable of autonomously sweeping the region to be inspected. The robot design contains systematic integration of components from robotics, NDE, and artificial intelligence (AI). A magnetic track system is used to navigate over the vertical steel structures required for examination. While moving across the inspection area, the robot uses an NDE sensor to acquire data for inspection and repair. This paper presents a design of a portable NDE scanning system based on eddy current array probes, which can be customized and installed on various mobile robot platforms. Machine learning methods are applied for semantic segmentation that will simultaneously localize and recognize defects without the need of human intervention. Experiments were conducted that show the NDE and repair capabilities of the system. Improvements in human safety and structural damage prevention, as well as lowering the overall costs of maintenance, are possible through the implementation of this robotic NDE system.

Materials Science↗

DOE OSTI2023

Interface-enhanced conductivities in surfactant-mediated, solution-grown ionic crystalline complexes

Renewable energy is increasingly relying on optimized electrolytes and interfaces. In this work, Tween 20 and sodium chloride are selected as a model system to reveal the effects of surfactants on salt crystallization in the context of ionic conductivity and interface optimization. At a varied crystallization speed and mix ratio, it is demonstrated that the resultant solution-grown ionic crystalline complexes can achieve a highly tunable ion transport with a controllable crystalline interface. X-ray diffraction results rule out the possibility of polymorphism in the NaCl/Tween 20 systems, which further supports the importance of an optimized crystalline network for optimizing permittivity or ionic conductivity. Raman mapping and machine learning techniques are used to perform semantic segmentation on highly heterogeneous NaCl/Tween 20 complexes. Furthermore, FTIR measurements demonstrate that inter- and intra-molecular interactions play critical roles in the formation of these crystals. This work lays a foundation toward future optimization of such complex ion systems for a specific salt or crystallization modifier in energy storage or ion transport applications.

36 MATERIALS SCIENCE↗

DOE OSTI2026

Deep Learning and Photogrammetric Reconstruction for Automated Crack Detection and Dimensional Measurement in Mining Operations

Surface crack detection and dimensional measurement at active mining sites present significant safety and operational challenges. Manual inspection methods are labor-intensive, spatially incomplete, and expose personnel to hazardous environments, while existing automated approaches have been developed primarily for concrete civil infrastructure and have not been validated on the complex, variable surfaces characteristic of mining environments. This dissertation presents an automated pipeline that integrates deep learning semantic segmentation with Structure-from-Motion photogrammetry to detect surface cracks and measure their aperture, length, and vertical displacement from standard RGB imagery acquired during routine Uncrewed Aerial Vehicle (UAV) survey operations, without requiring additional sensor hardware or manual measurement. The pipeline combines a U-Net architecture with an EfficientNet-B0 encoder, pretrained on the SDNET2018 concrete crack dataset and fine-tuned on a mining-specific dataset spanning laboratory concrete specimens, coal refuse impoundment embankments, and post-blast limestone quarry benches. Photogrammetric reconstruction is performed using COLMAP Structure-from-Motion and Multi-View Stereo, with crack segmentation masks projected into the reconstructed point cloud to enable three-dimensional vertical displacement measurement through local plane fitting and bimodal surface detection. The pipeline was validated across 36 controlled laboratory specimens at three imaging distances and four vertical displacement levels, achieving aperture measurement RMSE of 0.047 cm and R² of 0.954, and vertical displacement RMSE of 0.140 cm and R² of 0.966, against independent caliper measurements. Field application at a coal refuse impoundment in southwestern Pennsylvania detected 71 crack components across the embankment crest, with a dominant longitudinal crack exhibiting aperture values reaching 28 cm and a 95th percentile vertical displacement of 35.53 cm, consistent in magnitude and spatial distribution with simultaneously acquired LiDAR-derived estimates. Application across four post-blast limestone quarry bench datasets in California successfully characterized blast-induced fracture networks at ground sampling distances ranging from 0.59 to 1.23 cm/pixel, with detected crack geometries physically consistent with observable surface conditions at each site. The results demonstrate that deep learning-based crack detection and photogrammetric measurement can be integrated into routine UAV inspection workflows at mining sites, providing repeatable, scalable, and quantitative crack characterization across surface types, crack scales, and displacement magnitudes not previously addressed in the literature. The pipeline requires no dedicated surveying equipment beyond the UAV platforms already deployed at mine sites for survey and monitoring purposes, supporting practical adoption within existing operational workflows.

Crack detection, Dimensional Measurement↗

DOE OSTI2025

A derecho climatology (2004–2021) in the United States based on machine learning identification of bow echoes

Due to their persistent widespread severe winds, derechos pose significant threats to human safety and property, with impacts comparable to many tornadoes and hurricanes. Yet, automated detection of derechos remains challenging due to the absence of spatiotemporally continuous observations and the complex criteria employed to define the phenomenon. This study presents an objective derecho detection approach capable of automatically identifying derechos through both observations and model results. The approach is grounded in a physically based definition of derechos and integrates three algorithms: (1) the Python Flexible Object Tracker (PyFLEXTRKR) algorithm to track mesoscale convective systems (MCSs), (2) a semantic segmentation convolutional neural network to identify bow echoes, and (3) a comprehensive classification algorithm to detect derechos within MCS life cycles and distinguish derecho-producing from non-derecho-producing MCSs. Using this approach, we developed a novel high-resolution (4 km and hourly) observational dataset of derechos and accompanying derecho-producing MCSs over the United States east of the Rocky Mountains from 2004 to 2021. The dataset consists of two subsets based on different gust speed data sources and is analyzed to document the climatology of derechos in the United States. On average, 12–15 derechos are identified per year, aligning with previous estimations (∼6–21 events annually). The spatial distribution and seasonal variation patterns are consistent with prior studies, showing peak occurrences in the Great Plains and the Midwest during the warm season. Additionally, during the study period, derechos account for approximately 3.1 % of measured damaging gusts (≥25.93 m s−1) over the eastern United States. The dataset is publicly available at https://doi.org/10.5281/zenodo.14835362 (Li et al., 2025).

54 ENVIRONMENTAL SCIENCES↗

DOE OSTI2023

Red Wine Fermentation Alters Grape Seed Morphology and Internal Porosity

During wine fermentation, grape berry components, including seeds, undergo extensive physical and chemical changes that result in the release of flavonoids, such as tannins, from seeds into wine. Understanding changes in seed morphology during fermentation is crucial for aiding the development of accurate prediction models for flavonoid extraction during winemaking, which enhances fermentation management and ensures consistency in wines from year to year. High-resolution x-ray microcomputed tomography (x-ray μCT) was used to investigate the effect of red wine fermentation on changes in grape seed morphology. Using a PyTorch-based implementation of a fully convolutional network with a Resnet-101 backbone for semantic segmentation of x-ray μCT images, we quantified extensive alteration to grape seed structure during fermentation. Image analyses revealed the development of a pore network breaking apart the seed endosperm by the end of fermentation, leading to an increase in surface area. Fermentation significantly altered grape seed morphology. Such alterations could enable transport of seed flavonoids from inside the endosperm and integument to outside the seed. Further research on the physical processes occurring in seeds during wine fermentation is necessary to build better physiochemical models.

59 BASIC BIOLOGICAL SCIENCES↗

DOE OSTI2021

Data supporting manuscript from L. Sheneman, G. Stephanopoulos, A.E. Vasdekis titled "Deep learning classification of lipid droplets in quantitative phase images" as currently under review at PLOS ONE. This includes: 1) raw and binary labeled Quantitative Phase Images (QPI) of Y. lipolytica cells used in the analyses described within the manuscript. 2) various derived data including classifier scores, etc.

Data supporting manuscript from L. Sheneman, G. Stephanopoulos, A.E. Vasdekis titled "Deep learning classification of lipid droplets in quantitative phase images" as currently under review at PLOS ONE. This includes: 1) raw and binary labeled Quantitative Phase Images (QPI) of Y. lipolytica cells used in the analyses described within the manuscript. 2) various derived data including classifier scores, etc.

ANN↗

DOE OSTI2021

Deep learning for lipid droplet recognition in quantitative phase images

This library of Python code is used for performing semantic segmentation of images using 6 different machine learning methods. Five of the methods are implemented entirely within the scikit-learn framework. The Convolutional Neural Network (CNN) method requires Keras with a TensorFlow backend and generally uses a different set of scripts in order to perform the complete training and evaluation.

Sheneman, Lucas↗

DOE OSTI2022

Scaling Building Energy Audits through Machine Learning Methods on Novel Drone Image Data

Building energy audits are time-consuming and labor-intensive. This paper describes a new method using machine learning (ML) techniques on novel data sources (drone images) to improve the identification of building characteristics and retrofit opportunities, and thereby reduce the effort for audits. The new ML method includes: (1) Building footprint extraction using line extraction, polygonization, and polygon-merging, (2) Building envelope extraction using PIX4d modeling software to reconstruct a building 3D model, (3) Visualization tool for viewing images from the 3D model, (4) Window-to-wall ratio (WWR) using state-of-art deep neural network semantic segmentation, (5) Envelope thermal anomaly detection using an unsupervised machine learning clustering algorithm, and (6) Rooftop energy equipment detection based on an object detection algorithm. The testing of this method involved a comparison of additional ML-generated information overlaid on current ‘state-of-practice’ audit and remote assessment baselines using evaluation metrics: labor time and associated cost, marginal benefits of using ML-generated information in workflows for audits and remote assessments, integration potential with existing processes and tools, and replicability/scalability of the method. In two test buildings in California that had comprehensive drawings and meter data available, the ML method effectively generated a building footprint, envelope, rooftop equipment, WWR, and locations of envelope thermal anomalies. Projected target segments of the ML method are sites with minimal drawings and energy data, and underserved sectors such as multistoried housing, disadvantaged communities, and schools for which the ML method can enable identification of building asset characteristics and prioritization of envelope retrofits and decentralized energy equipment retrofits.

Singh, Reshma↗