SEARCH · Search NASA
Results for “3D RGB”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
Evaluation of Techniques for Vegetation Removal from UAV-Based Photogrammetric DSM near WIPP Land Withdrawal Act Boundary, NM
The Waste Isolation Pilot Plant is located within New Mexico's karst landscape and is the United States' only deep geologic radioactive waste repository which isolates transuranic waste from defense activities underground in a bedded salt formation. Characteristic karst topography in this region is made up of material (such as limestone) that can be dissolved by water over a long period of time, and includes features such as springs, surface streams, sinking streams, caves, and sinkholes. Karst features can potentially impact the integrity of the waste repository in the future and compromise WIPP's performance, particularly when coupled with incompatible land-use activities within the Land Withdrawal Act (LWA) boundary as well as increased water withdrawals outside the LWA. In order to better capture the ground surface topography to support hydrology modeling efforts, the U.S. DOE has enlisted FIU to develop a high-resolution digital elevation model (DEM). In a previous field study, a high-resolution digital surface model (DSM) of an adjacent representative site, Basin 6, was developed by photogrammetry using aerial images captured with unmanned aerial vehicles. The DSM generated was based on point clouds representing the entire landscapes, including points of terrain, vegetation and infrastructure. These above-ground features need to be removed to create a DSM/DEM with good representation of the bare ground. In recent years, many techniques including both pre- and post-processing of DSMs, were developed for vegetation removal with varied success. This research focuses on applying and comparing various vegetation removal methods using point clouds generated of the Basin 6 pilot study area. The 3D RGB-based point cloud classification using Python scripts was a modification of the procedure by Themistocleous (2019). The method using Liblas classifies the point cloud three-dimensionally, whereas the method by Themistocleous (2019) is two-dimensional. The preliminary result of the method using Liblas, however, still had a large amount of vegetation remaining compared to the Pix4D Machine Learning method. The vegetation layer remains highly visible in the DEMs processed using the vegetation indices as opposed to the Pix4D Machine Learning method, which produces a DEM where mostly bare ground is seen. Further modifications to the input parameters are required to improve the vegetation vs. bare ground classification using this approach. The next step will be to improve the method using Liblas and to also test other vegetation removal methods found in the literature review to compare them against the two described here.
OPEN-Augmented Reality GUI for Bioenergy Crop Phenotyping and Precision Agriculture (Donald Danforth Plant Science Center Final Scientific Technical Report)
The project led by the Donald Danforth Plant Science Center, in collaboration with Arizona State University, George Washington University, and Saint Louis University, has made significant strides in advancing the phenotypic analysis of bioenergy crops through the development of an innovative AI processing pipeline. This initiative was primarily funded by ARPA-E, with additional cost-sharing provided by the participating institutions. The project successfully utilized a variety of sensors—3D scanners, thermal, RGB, and hyperspectral—to refine algorithms for data-driven trait signature identification and improve the classification and visualization of plant traits. The developed AI processing pipeline is capable of handling the complex, multidimensional data characteristic of dynamic agricultural environments. 1) Contributions to understanding: The research has advanced the field of plant phenomics by showcasing the synergistic use of various sensor data to enhance the precision of trait analysis in bioenergy crops. Through the integration of 3D scanners, thermal, RGB, and hyperspectral sensors, the project has developed robust data-driven trait signature algorithms and visualization techniques. These innovations have facilitated detailed monitoring and management of plant traits, providing vital insights into plant growth dynamics and stress responses. Further, the project has broadened our understanding of how machine learning can be effectively applied in multi-sensor environments to refine trait analysis. By leveraging diverse datasets, the research has not only improved the accuracy of phenotypic assessments but also established a versatile methodological framework that can be extended beyond agriculture to other fields requiring detailed phenotypic analysis. 2) Technical effectiveness and economic feasibility: The AI processing pipeline developed in this project demonstrated significant technical effectiveness, achieving high throughput analysis of extensive phenotypic data and meeting targeted accuracies. This system exemplified the capability of advanced machine learning technologies to efficiently manage and analyze large, complex datasets. Economically, the implementation of the project-developed pipelines may offer substantial cost savings across multiple sectors. It enhances data analysis processes and significantly reduces the need for manual data interpretation, thereby decreasing both the time and resources required. 3) Public benefit: The project has significantly broadened the scope of agricultural methodologies to enhance phenotypic analysis, with potential applications in various sectors beyond agriculture. Additionally, the initiative fostered an enriching educational and collaborative environment, significantly enhancing the technical skills of participants. It also made substantial contributions to the scientific community by providing open-access data sets and tools, encouraging ongoing research and development across various disciplines. Overall, the project not only met its scientific goals but also showcased the extensive utility of integrating advanced machine learning and sensor data analysis technologies. These advancements have proven instrumental in driving forward both theoretical research and practical applications, setting a strong foundation for future explorations and innovations in data-driven science.
Aerial 3D Building Reconstruction from Drone Imagery (A3DBR) v1
This toolkit is composed of several modules for extracting buildings geometrical and thermal characteristics from RGB and thermal imagery captured using a drone. - Building 3D reconstruction module: leverage a photogrammetry software to construct a 3D point cloud from RGB drone imagery, which is then used in conjunction with image processing and geometric methods to extract building footprint and building height (i.e., 3D model of the building). - Windows to wall ratio estimation module: leverage deep learning semantic segmentation modeling to detect windows on 2D drone RGB images. The detected windows are then projected onto the extracted building 3D model (using building 3D reconstruction module) and their area is computed to obtain window to wall ratio estimation. - Thermal anomalies detection module: leverage image processing and machine learning algorithm to detect on 2D drone thermal images potential thermal anomalies within building's facades and roofs.
Comprehension of Spatial Constraints by Neural Logic Learning from a Single RGB-D Scan
Autonomous industrial assembly relies on the precise measurement of spatial constraints as designed by computer-aided design (CAD) software such as SolidWorks. This paper proposes a framework for an intelligent industrial robot to understand the spatial constraints for model assembly. An extended generative adversary network (GAN) with a 3D long short-term memory (LSTM) network was designed to composite 3D point clouds from a single RGB-D scan. The spatial constraints of the segmented point clouds are identified by a neural-logic network that incorporates general knowledge of spatial constraints in terms of first-order logic. The model was designed to comprehend a complete set of spatial constraints that are consistent with industrial CAD software, including left, right, above, below, front, behind, parallel, perpendicular, concentric, and coincident relations. The accuracy of 3D model composition and spatial constraint identification was evaluated by the RGB-D scans and 3D models in the ABC dataset. The proposed model achieved 57.23% intersection over union (IoU) in 3D model composition, and over 99% in comprehending all spatial constraints.
Machine Learning for Automated Extraction of Building Geometry
As data science comes to buildings, the promise of using machine learning and novel sources of data has received much attention. Advances in machine learning and computer vision algorithms, combined with increased access to unstructured data (e.g., images and text), have created an opportunity for automated extraction of building characteristics – cost-effectively, and at scale. Acquisition of features such as footprint are time consuming and costly to acquire with today’s manual methods, but can be streamlined through intelligent software-based solutions applied to satellite images. When combined with aerial RGB and thermal images, full 3D geometries and thermal maps can be constructed to determine additional characteristics such as window to wall ratio, height, number of stories and envelope thermal characteristics. In this paper we present three contributions to accelerate these high potential opportunities: (1) a methodical analysis of how these features can be integrated into today’s simulation and data driven software tools to enhance efficiency measure identification and owner/operator decision making; (2) development and accuracy testing of open source deep neural network methods to extract building footprints from satellite imagery, including the curation and application of openly available GIS datasets for training and continued development by others; and (3) an open framework for drone-based image capture and creation of 3D building geometries. This work represents an important bridge between high-level studies that span diverse application areas and those that detail point solutions yet cannot be easily replicated or extended.
Voxel carving–based 3D reconstruction of sorghum identifies genetic determinants of light interception efficiency
Changes in canopy architecture traits have been shown to contribute to yield increases. Optimizing both light interception and light interception efficiency of agricultural crop canopies will be essential to meeting the growing food needs. Canopy architecture is inherently three-dimensional (3D), but many approaches to measuring canopy architecture component traits treat the canopy as a two-dimensional (2D) structure to make large scale measurement, selective breeding, and gene identification logistically feasible. We develop a high throughput voxel carving strategy to reconstruct 3D representations of sorghum from a small number of RGB photos. Our approach builds on the voxel carving algorithm to allow for fully automatic reconstruction of hundreds of plants. It was employed to generate 3D reconstructions of individual plants within a sorghum association population at the late vegetative stage of development. Light interception parameters estimated from these reconstructions enabled the identification of known and previously unreported loci controlling light interception efficiency in sorghum. The approach is generalizable and scalable, and it enables 3D reconstructions from existing plant high throughput phenotyping datasets. We also propose a set of best practices to increase 3D reconstructions’ accuracy.
PhytoOracle: Scalable, modular phenomics data processing pipelines
As phenomics data volume and dimensionality increase due to advancements in sensor technology, there is an urgent need to develop and implement scalable data processing pipelines. Current phenomics data processing pipelines lack modularity, extensibility, and processing distribution across sensor modalities and phenotyping platforms. To address these challenges, we developed PhytoOracle (PO), a suite of modular, scalable pipelines for processing large volumes of field phenomics RGB, thermal, PSII chlorophyll fluorescence 2D images, and 3D point clouds. PhytoOracle aims to ( i ) improve data processing efficiency; ( ii ) provide an extensible, reproducible computing framework; and ( iii ) enable data fusion of multi-modal phenomics data. PhytoOracle integrates open-source distributed computing frameworks for parallel processing on high-performance computing, cloud, and local computing environments. Each pipeline component is available as a standalone container, providing transferability, extensibility, and reproducibility. The PO pipeline extracts and associates individual plant traits across sensor modalities and collection time points, representing a unique multi-system approach to addressing the genotype-phenotype gap. To date, PO supports lettuce and sorghum phenotypic trait extraction, with a goal of widening the range of supported species in the future. At the maximum number of cores tested in this study (1,024 cores), PO processing times were: 235 minutes for 9,270 RGB images (140.7 GB), 235 minutes for 9,270 thermal images (5.4 GB), and 13 minutes for 39,678 PSII images (86.2 GB). These processing times represent end-to-end processing, from raw data to fully processed numerical phenotypic trait data. Repeatability values of 0.39-0.95 (bounding area), 0.81-0.95 (axis-aligned bounding volume), 0.79-0.94 (oriented bounding volume), 0.83-0.95 (plant height), and 0.81-0.95 (number of points) were observed in Field Scanalyzer data. We also show the ability of PO to process drone data with a repeatability of 0.55-0.95 (bounding area).
A Real2Sim Digital Twin Pipeline for Photorealistic Robot Simulation: Evaluating VLA Policy Deployment on a Bimanual Mobile Robot
Digital twins that are automatically constructed from robot sensor data offer a promising pathway for scalable Real2Sim and Sim2Real transfer. However, it remains an open question whether photorealistic reconstruction alone is sufficient to support reliable deployment of vision-language-action (VLA) policies. We present a generative-AI-assisted Real2Sim pipeline that generates simulation-ready digital twins from real-world RGB observations with minimal manual intervention. The pipeline uses prompted segmentation to isolate scene components and a generative 3D model to directly produce simulation assets, eliminating the need for traditional multi-view reconstruction or manual 3D modeling.\r\nTo evaluate simulation fidelity, we deploy and compare policies from two VLA models in both the real robot and the reconstructed\r\nsimulation under identical tasks and initial conditions. We compare joint-level action trajectories and analyze how divergence evolves over time in closed-loop execution. Although the reconstructed environments are visually accurate, we observe increasing trajectory divergence during closedloop operation. These results indicate that photorealistic reconstruction alone is insufficient to preserve closed-loop control behavior\r\nin VLA policies, particularly in contact-rich manipulation settings where small perceptual errors compound over time.
Autonomous elemental characterization enabled by a low cost robotic platform built upon a generalized software architecture
Despite the rapidly growing applications of robots in industry, the use of robots to automate tasks in scientific laboratories is less prolific due to the lack of generalized methodologies and the high cost of hardware. This paper focuses on the automation of characterization tasks necessary for reducing cost while maintaining generalization and proposes a software architecture for building robotic systems in scientific laboratory environments. A dual-layer (Socket.IO and ROS) action server design is the basic building block, which facilitates the implementation of a web-based front end for user-friendly operation and the use of ROS Behavior Trees for convenient task planning and execution. A robotic platform for automating mineral and material sample characterization is built upon the architecture, with an open-source, low-cost three-axis computer numerical control gantry system serving as the main robot. A handheld laser induced breakdown spectroscopy (LIBS) analyzer is integrated with a 3D printed adapter, enabling (1) automated 2D chemical mapping and (2) autonomous sample measurement (with the support of an RGB-Depth camera). We demonstrate the utility of automated chemical mapping by scanning the surface of a spodumene-bearing pegmatite core sample with a 1071-point dense hyperspectral map acquired at a rate of 1520 bits per second. Furthermore, we showcase the autonomy of the platform in terms of perception, dynamic decision-making, and execution, through a case study of LIBS measurement of multiple mineral samples. The platform enables controlled and autonomous chemical quantification in the laboratory that complements field-based measurements acquired with the same handheld device, linking resource exploration and processing steps in the supply chain for lithium-based battery materials.
Improving Building Footprint Extraction Using NAIP and 3DEP Lidar Derived Features with Deep Learning
Accurate building footprint extraction is critical for applications ranging from population estimation to disaster management. Although optical imagery provides detailed spectral information, it often struggles with shadows, occlusions, and background clutter in dense urban environments. Lidar data, by contrast, offer precise elevation and structural attributes but face challenges such as variable point density and noise. This study integrates multispectral imagery from the U.S. Department of Agriculture (USDA) National Agriculture Imagery Program (NAIP) with lidar-derived feature height and intensity from the U.S. Geological Survey (USGS) 3D Elevation Program (3DEP) to improve footprint extraction using a U-Net–based deep learning model. A six-band input stack (RGB, near-infrared, height, intensity) was developed, normalized, and tiled for training and evaluation against Microsoft Global Building Footprints (GBF). Results from the Houston, TX test site show that the six-band model achieved a precision of 0.86, recall of 0.88, F1 score of 0.87, and Intersection-over-Union (IoU) of 0.76, consistently outperforming four-band baselines by reducing false positives while maintaining sensitivity. Predictions on withheld Houston tiles confirmed strong within-region generalization, yielded a precision of 0.78, recall of 0.81, F1 score of 0.79, and IoU of 0.66. Qualitative analysis further revealed limitations stemming from both training label quality and vegetation–building confusion. These findings demonstrate the complementary value of integrating spectral and structural information for robust building footprint extraction and how domain adaptation strategies can be used to enhance cross-regional transferability.
Advanced Interactive 3D Visualization Tool for Customizable Analyses of Tomography Datasets in Material Science
Current methods for visualizing and analyzing 3D tomography datasets in materials science often lack the interactivity and depth required for detailed structural insights. This limitation restricts a researchers' ability to accurately interpret complex data, which is critical for advancing material innovations and understanding structural properties. To address this issue, we have developed a novel, web-based interactive 3D visualization and analysis tool from the Trame framework that offers customizable features to enhance data interpretability. The tool allows users to adjust parameters such as visible range, slice planes, data rotation, and layering, providing a more detailed and dynamic view of complex structures. Its user-friendly web interface increases the accessibility and ease of use for both novice and experienced researchers, to visualize large volumetric datasets. The tool supports a diverse range of data formats, making it versatile for various research applications. Unique capabilities include real-time data manipulation, automated feature detection, context-sensitive feedback, and real-time volume calculations and distributions per sliced region or layer, alongside the ability to quickly generate high-quality screenshots and videos for presentations and reports. These advancements offer a comprehensive solution for enhanced 3D data exploration, significantly improving the analysis process and communication of results in materials science.
Deep Point Cloud Building Envelope Segmentation (DeeP-CuBES) using Deep Learning
Building Information Modeling (BIM) plays an important role in building design and construction, particularly for achieving energy-efficient retrofits. Building envelope retrofits using panelized prefabricated system, such as those popularized by the Energiesprong program, need accurate as-built dimensions of facade features (windows, doors, etc.) to achieve the desired thermal and air tightness. Traditionally, building surveying is done manually, resulting in a time-consuming and labor-intensive process. Recently, 3D point clouds from terrestrial LiDAR have been used to automate the generation of as-built dimensions of existing buildings. However, automated BIM using LiDAR relies on solving the point cloud semantic segmentation (PCSS) problem. In this work, we propose a robust pipeline for solving the PCSS problem using deep neural networks, focusing on overcoming challenges posed by imbalanced datasets and complex architectural features. We introduce the first high-density, labeled, and validated building envelope point cloud dataset derived from multiple building scans, specifically curated to tackle challenges in facade-level segmentation. Results from the trained neural networks show that advanced attention-based architectures and incorporating radiometry (light intensity and RGB) features significantly boost segmentation accuracy for windows and doors.
Development of a Miniaturized Mobile Mapping System for In-Row, Under-Canopy Phenotyping
This paper focuses on the development of a miniaturized mobile mapping platform with advantages over current agricultural phenotyping systems in terms of acquiring data that facilitate under-canopy plant trait extraction. The system is based on an unmanned ground vehicle (UGV) for in-row, under-canopy data acquisition to deliver accurately georeferenced 2D and 3D products. The paper addresses three main aspects pertaining to the UGV development: (a) architecture of the UGV mobile mapping system (MMS), (b) quality assessment of acquired data in terms of georeferencing information as well as derived 3D point cloud, and (c) ability to derive phenotypic plant traits using data acquired by the UGV MMS. The experimental results from this study demonstrate the ability of the UGV MMS to acquire dense and accurate data over agricultural fields that would facilitate highly accurate plant phenotyping (better than above-canopy platforms such as unmanned aerial systems and high-clearance tractors). Plant centers and plant count with an accuracy in the 90% range have been achieved.
FY21 Progress Report: SRNL Analysis of ICCWR LCM and WAMS data for Corrosion and Cracking
The development of algorithms for machine learning and data analysis for the 3013 Surveillance Program is a collaborative effort by the Savannah River National Laboratory (SRNL) and the University of South Carolina (USC). For corrosion detection, Laser Confocal Microscope (LCM) or Wide Area 3D Measurement System (WAMS) data is extracted from large binary files, with software written to convert the data to physical attributes (e.g., height, color and grayscale values; all as functions of a location in a plane projection). A user-friendly Matlab Graphical User Interface (GUI) that reads data from either LCM or WAMS files was developed to integrate input data with software developed for processing and evaluation. The GUI can selectively download binary data, interrogate data attributes, label data, flag significant features, execute Machine Learning (ML) algorithms, output parameters for trained ML algorithms, report ML model accuracy with respect to labeled data, and generate graphical representations for various analyses. Features can be called out by user-specified thresholds, manual labeling or machine learning algorithms when they have been completed. The ability to rapidly label data is important because of the volume of data required for training machine learning algorithms. The GUI has the flexibility to allow addition of improved ML algorithms, methods for data visualization, and statistical computations. Statistical analyses via the GUI include areas of pits within a defined range of pit depths, correlations between Red-Green-Blue (RGB) or grayscale intensity and relative surface height, covariances between values associated with features, and feature histograms. The development of supervised machine learning algorithms, however, has been hindered by a lack of training data. The machine learning algorithms for crack identification are being refined but require improvements to the true positive rate for crack detection. This shortcoming is an artifact of the limited training data currently available, perhaps more so than the structure of the neural networks. At present, the best results are had from a consensus over an ensemble of randomly generated Deep Neural Network (DNN) or Convolutional Neural Network (CNN) algorithms. Although the consensus accuracy method has yielded optimum true positive and true negative rates in excess of 80%, additional validation testing is necessary. In addition to the suite of LCM data that was initially used, and which represents the majority of the work presented in this report, WAMS image data was also reviewed at a preliminary level. The review included a comparison between image resolution and dynamic range for each method. WAMS (ZON file) image data was found to have a pixel pitch of 3.69μm compared to 1 μm for the LCM (vk4 file) data, which implies a lower resolution for the WAMS images. Conversely, the ratio of dynamic range of the WAMS data to the LCM data was approximately 41:20 for height data, suggesting that information from WAMS should more accurately determine the depth of pits. At present, the significance of the greater dynamic range of the WAMS data relative to the LCM data has not yet been evaluated.