Results for “Computer vision”

DOE OSTI2023

RadioGalaxyNET: Dataset and novel computer vision algorithms for the detection of extended radio galaxies and infrared hosts

Abstract Creating radio galaxy catalogues from next-generation deep surveys requires automated identification of associated components of extended sources and their corresponding infrared hosts. In this paper, we introduce RadioGalaxyNET, a multimodal dataset, and a suite of novel computer vision algorithms designed to automate the detection and localization of multi-component extended radio galaxies and their corresponding infrared hosts. The dataset comprises 4 155 instances of galaxies in 2 800 images with both radio and infrared channels. Each instance provides information about the extended radio galaxy class, its corresponding bounding box encompassing all components, the pixel-level segmentation mask, and the keypoint position of its corresponding infrared host galaxy. RadioGalaxyNET is the first dataset to include images from the highly sensitive Australian Square Kilometre Array Pathfinder (ASKAP) radio telescope, corresponding infrared images, and instance-level annotations for galaxy detection. We benchmark several object detection algorithms on the dataset and propose a novel multimodal approach to simultaneously detect radio galaxies and the positions of infrared hosts.

Astronomy & Astrophysics↗

DOE OSTI2025

Revealing the evolution of order in materials microstructures using multi-modal computer vision

The development of high-performance materials for microelectronics, energy storage, and extreme environments depends on our ability to describe and direct property-defining microstructural order. Our present understanding is typically derived from laborious manual analysis of imaging and spectroscopy data, which is difficult to scale, challenging to reproduce, and lacks the ability to reveal latent associations needed for mechanistic models. Here, we demonstrate a multi-modal machine learning (ML) approach to describe order from electron microscopy analysis of the complex oxide La 1−x Sr x FeO 3 . We construct a hybrid pipeline based on fully and semi-supervised classification, allowing us to evaluate both the characteristics of each data modality and the value each modality adds to the ensemble. We observe distinct differences in the performance of uni- and multi-modal models, from which we draw general lessons in describing crystal order using computer vision.

36 MATERIALS SCIENCE↗

DOE OSTI2024

Microscopy modality transfer of steel microstructures: Inferring scanning electron micrographs from optical microscopy using generative AI

Scanning electron microscopy (SEM) is resource intensive, which limits its throughput in some applications. As an alternative, we propose applying computer vision and machine learning to generate high-quality synthetic SEM micrographs from micrographs obtained using light optical microscopy (LOM). Working with a correlated LOM/SEM dataset of dual-phase steel images, we test generative models of various architectures, including encoder-decoder networks, generative adversarial networks (GANs), and diffusion-based models. We find that the diffusion models significantly outperform other methods on both qualitative and quantitative assessments, while preserving key metallurgical meaning. This work establishes diffusion as the state-of-the-art for microscopy modality transfer and demonstrates the potential of AI-powered microscopy to enhance LOM with micron scale structural recreation.

Computer vision↗

DOE OSTI2025

Micrometer: Micromechanics transformer for predicting full field mechanical responses of heterogeneous materials

Predicting mechanical responses of heterogeneous materials across scales remains a significant challenge. Traditional computational methods often struggle with complex and multiscale nature of these materials, limiting their effectiveness in real-world applications. Here, in this paper, we introduce Micrometer, a vision transformer based deep learning model designed to predict full field mechanical responses of heterogeneous materials, bridging the gap between computer vision and solid mechanics problems. We show that Micrometer, trained on a large-scale high-resolution dataset of 2D fiber-reinforced composites, can achieve state-of-the-art performance in predicting microscale strain fields across a wide range of material properties and loading conditions. Our model demonstrates accuracy and computational efficiency in applications such as computational homogenization and multiscale modeling, reducing computational time by up to two orders of magnitude compared to conventional numerical solvers while maintaining less than 1 % errors in predicting macroscale stress fields. Furthermore, we showcase Micrometer’s adaptability through transfer learning experiments on new materials with limited data, highlighting its potential to tackle diverse scenarios in computational solid mechanics. These results represent a significant step towards AI-driven innovation in materials science, addressing the limitations of traditional numerical methods and paving the way for more efficient simulations of heterogeneous materials across various industrial applications.

Composite materials↗

NASA NTRS2026

Hydrology Copilot: A Cloud-Native Ai System for Hydrological Data Analysis

The emergence of AI-driven Earth observation systems promises to broaden access to petabyte-scale geospatial data beyond domain specialists. However, translating this vision into operational scientific infrastructure requires addressing fundamental challenges in data virtualization, code transparency, and domain-specific reasoning. We present Hydrology Copilot, a cloud-native AI framework for natural-language-driven analysis of Earth observation data. To demonstrate operational capabilities at scale, we implement the system using NASA's North American Land Data Assimilation System version 3 (NLDAS-3), which provides surface meteorological forcing and land-surface model output across North and Central America at 1-km resolution, from which drought diagnostics are derived. The system integrates five core contributions: (1) scalable data virtualization using Kerchunk-based cloud optimized access, achieving a 1.5 to 4.6 times improvement in I/O latency across benchmark queries spanning regional single-day extractions (4.6 times speedup) to continental monthly aggregations (1.5 times speedup); (2) transparent code generation through Microsoft Azure AI Foundry agents that expose executable Python workflows for scientific verification; (3) persistent conversational memory enabling multi-turn analytical discourse across sessions; (4) intelligent query validation that enforces dataset boundaries and resolves ambiguous requests before execution; and (5) a multi-agent architecture coordinating query parsing, code generation, and visualization. We evaluate the system through drought-monitoring workflows, demonstrating reliable code generation, accurate results validated against reference computations and the operational U.S. Drought Monitor, and efficient operation across increasingly complex tasks. By bridging natural-language interfaces with rigorous hydrological analysis, Hydrology Copilot advances beyond proof-of-concept demonstrations to provide a deployable framework for operational Earth science applications.

Data virtualization↗

NASA NTRS1992

Effect of Microgravity on Several Visual Functions During STS Shuttle Missions

Many astronauts and cosmonauts have commented on apparent changes in their vision while on-orbit. Comments have included statements of supposed improved distance acuity to decreased near vision capability. The purpose of this study was to assess not only changes in visual acuity, but expand the assessment to several other visual functions for a comprehensive battery of tests. Vision was assessed using an innovative device, the Visual Function Tester - Model 1 (VFT-1), which presents the tests at optical infinity and includes critical flicker fusion, stereopsis to 10 seconds-of-arc, visual acuity in small steps to 20/7.7, cyclophoria, lateral and vertical phoria, and retinal rivalry. Vision was assessed 2 times prelaunch at L-14 days and L-7 days, 3-4 times while on-orbit, at landing, and 2 times postlanding at L+3 days and L+7 days. There were 26 STS astronauts that participated, with data on 20 astronauts used for analysis. There was a typical wide variability between subjects in baseline visual performance for each parameter at the prelaunch sessions. There was a slight but statistically significant decrease in visual acuity while on-orbit that was not clinically significant. For stereopsis (i.e. depth perception), there was a small improvement on-orbit that was not statistically significant. There were no changes during space flight for any of the other visual parameters tested. A few individuals showed apparent changes in acuity and stereopsis. The possibility exists that microgravity affects the visual system of some individuals differently, as with space adaptation syndrome. Repeat data on 2 astronauts showed good repeatability between the 2 flights. These results pertain to only short term space flight on the STS shuttle, and longer flights are necessary to determine if there is any relationship between mission duration and these visual functions.

Melvin R O'Neal↗

NASA NTRSDate not supplied

NASA's Small Spacecraft and Distributed Systems: Development and Demonstration of Technologies Enabling Swarms and New Spacecraft Platforms with AI and Edge Computing

NASA’s Small Spacecraft & Distributed Systems (SSDS) within the Research and Technology Mission Directorate (RTMD) expands U.S. capability to execute unique missions through targeted investment, rapid development, and flight demonstration of small spacecraft technologies applicable to exploration, science and the commercial space sector. SSDS strategically invests in technology development and on-orbit demonstrations executed across NASA, other government agencies, industry, and academia. The program’s University SmallSat Technology Partnerships initiative awards academic researchers with the opportunity to collaborate with NASA to mature innovative technology. Capabilities aligned with RTMD’s technology shortfalls and interests - power, processing, propulsion, sensors, communications, autonomous navigation, architectures, and advanced applications like artificial intelligence (AI), machine learning, and edge computing - are prioritized in SSDS investments. These investments enable distributed, autonomous, and cooperative small spacecraft systems that support swarm missions extending beyond low Earth orbit into cislunar and deep space. This paper highlights representative SSDS flight demonstrations that mature these capabilities to enable a future operational infrastructure needed to support sustained exploration of the Moon and beyond. SSDS’s investment strategy emphasizes rapid development and on-orbit demonstration to validate spacecraft technologies required for swarms and distributed mission architectures. The Starling swarm technology demonstration mission exemplifies this approach by advancing distributed spacecraft autonomy, cooperative operations, and space situational awareness. Extended flight testing and ongoing studies of next generation swarm configurations and on-orbit space traffic monitoring and management continue to inform future swarm designs. DiskSat’s four-spacecraft demonstration mission represents SSDS’s strategic vision to expand the design space for future small spacecraft through its commitment to advance novel platform concepts that can impact how science is performed on orbit. Continuing to invest in future platforms, the notional PY12 concept is a 12-spacecraft swarm hosting neuromorphic processors and is envisioned as an on-orbit testbed for AI, edge computing, and positioning, navigation and timing technologies. SSDS also invests in single-spacecraft technology demonstrations that underpin the success of future swarm missions and accelerate the availability of validated technologies across the small spacecraft ecosystem. Examples of such demonstrations include Pathfinder Technology Demonstrator-3 (PTD-3), which performed high-rate optical communications; PTD-R, which demonstrated a camera capable of simultaneous ultraviolet and short-wave infrared optical sensing; and CAPSTONE, the Cislunar Autonomous Positioning System Technology and Operations Navigation Experiment, which validated autonomous navigation in cislunar space. Collectively, SSDS-funded demonstrations advance capabilities across swarms and illustrate a coordinated investment strategy to mature high-impact technologies required for autonomous, distributed, and cooperative small spacecraft systems for low Earth orbit, cislunar, and deep space applications. Technology demonstrations strengthen SSDS partnerships with industry, academia, and other government agencies, and promote small spacecraft community adoption of capabilities required to close technical gaps for swarm missions.

Jan Stupl↗

NASA NTRSDate not supplied

NASA's Small Spacecraft and Distributed Systems: Development and Demonstration of Technologies Enabling Swarms and New Spacecraft Platforms with AI and Edge Computing

NASA’s Small Spacecraft & Distributed Systems (SSDS) within the Research and Technology Mission Directorate (RTMD) expands U.S. capability to execute unique missions through targeted investment, rapid development, and flight demonstration of small spacecraft technologies applicable to exploration, science and the commercial space sector. SSDS strategically invests in technology development and on-orbit demonstrations executed across NASA, other government agencies, industry, and academia. The program’s University SmallSat Technology Partnerships initiative awards academic researchers with the opportunity to collaborate with NASA to mature innovative technology. Capabilities aligned with RTMD’s technology shortfalls and interests - power, processing, propulsion, sensors, communications, autonomous navigation, architectures, and advanced applications like artificial intelligence (AI), machine learning, and edge computing - are prioritized in SSDS investments. These investments enable distributed, autonomous, and cooperative small spacecraft systems that support swarm missions extending beyond low Earth orbit into cislunar and deep space. This paper highlights representative SSDS flight demonstrations that mature these capabilities to enable a future operational infrastructure needed to support sustained exploration of the Moon and beyond. SSDS’s investment strategy emphasizes rapid development and on-orbit demonstration to validate spacecraft technologies required for swarms and distributed mission architectures. The Starling swarm technology demonstration mission exemplifies this approach by advancing distributed spacecraft autonomy, cooperative operations, and space situational awareness. Extended flight testing and ongoing studies of next generation swarm configurations and on-orbit space traffic monitoring and management continue to inform future swarm designs. DiskSat’s four-spacecraft demonstration mission represents SSDS’s strategic vision to expand the design space for future small spacecraft through its commitment to advance novel platform concepts that can impact how science is performed on orbit. Continuing to invest in future platforms, the notional PY12 concept is a 12-spacecraft swarm hosting neuromorphic processors and is envisioned as an on-orbit testbed for AI, edge computing, and positioning, navigation and timing technologies. SSDS also invests in single-spacecraft technology demonstrations that underpin the success of future swarm missions and accelerate the availability of validated technologies across the small spacecraft ecosystem. Examples of such demonstrations include Pathfinder Technology Demonstrator-3 (PTD-3), which performed high-rate optical communications; PTD-R, which demonstrated a camera capable of simultaneous ultraviolet and short-wave infrared optical sensing; and CAPSTONE, the Cislunar Autonomous Positioning System Technology and Operations Navigation Experiment, which validated autonomous navigation in cislunar space. Collectively, SSDS-funded demonstrations advance capabilities across swarms and illustrate a coordinated investment strategy to mature high-impact technologies required for autonomous, distributed, and cooperative small spacecraft systems for low Earth orbit, cislunar, and deep space applications. Technology demonstrations strengthen SSDS partnerships with industry, academia, and other government agencies, and promote small spacecraft community adoption of capabilities required to close technical gaps for swarm missions.

Jan Stupl↗

NASA NTRSDate not supplied

BEAST: Expanding Sustainable Data Infrastructure for High-Enthalpy Facilities

Reproducible, data-driven thermal protection system (TPS) research requires that experimental records from high-enthalpy testing be consistently structured, traceable, and accessible across campaigns and institutions. In practice, however, arcjet and plasma facilities data remain largely fragmented: raw diagnostics are stored in ad hoc formats, material sample histories are disconnected from test conditions, and metadata standards are absent, precluding systematic cross-campaign analysis and long-term reuse. BEAST (Backend for Experiment Analysis, Storage, and Traceability) is an open-source, web-based platform that addresses these limitations by providing a unified, queryable infrastructure for high-enthalpy ground-test data [1]. First presented at the 15th Ablation Workshop [2], BEAST has since undergone significant development. The platform ingests and structures multi-channel time-series diagnostics, facility configurations, and material property records within a common provenance model, ensuring end-to-end traceability from raw sensor acquisition to reduced experimental quantities. A versioned material library links specimen identity and processing history to the specific runs in which each sample was tested. An integrated modeling workbench enables training and evaluation of regression models directly on archived experimental data, supporting condition interpolation and the construction of empirical material response databases. Beyond its original deployment at NASA Ames Research Center, BEAST has been designed to be facility-agnostic, with ongoing efforts to extend its adoption to other facilities. Its modular architecture accommodates heterogeneous diagnostic setups and facility types, and its future open-source distribution allows institutions to build on a common data standard rather than maintaining isolated, bespoke solutions. BEAST is further integrated within a broader ecosystem of companion tools: arcjetCV [3] extracts recession rates and shock standoff distances from high-speed video using computer vision, and miniSTARscan [4] provides sub-minute, portable photogrammetric surface reconstruction of test articles before and after exposure. All tools share a common data schema, enabling seamless ingestion of surface geometry, imagery, and time-series data into a single, coherent experimental record.

Database↗

NASA NTRSDate not supplied

BEAST: Expanding Sustainable Data Infrastructure for High-Enthalpy Facilities

Reproducible, data-driven thermal protection system (TPS) research requires that experimental records from high-enthalpy testing be consistently structured, traceable, and accessible across campaigns and institutions. In practice, however, arcjet and plasma facilities data remain largely fragmented: raw diagnostics are stored in ad hoc formats, material sample histories are disconnected from test conditions, and metadata standards are absent, precluding systematic cross-campaign analysis and long-term reuse. BEAST (Backend for Experiment Analysis, Storage, and Traceability) is an open-source, web-based platform that addresses these limitations by providing a unified, queryable infrastructure for high-enthalpy ground-test data [1]. First presented at the 15th Ablation Workshop [2], BEAST has since undergone significant development. The platform ingests and structures multi-channel time-series diagnostics, facility configurations, and material property records within a common provenance model, ensuring end-to-end traceability from raw sensor acquisition to reduced experimental quantities. A versioned material library links specimen identity and processing history to the specific runs in which each sample was tested. An integrated modeling workbench enables training and evaluation of regression models directly on archived experimental data, supporting condition interpolation and the construction of empirical material response databases. Beyond its original deployment at NASA Ames Research Center, BEAST has been designed to be facility-agnostic, with ongoing efforts to extend its adoption to other facilities. Its modular architecture accommodates heterogeneous diagnostic setups and facility types, and its future open-source distribution allows institutions to build on a common data standard rather than maintaining isolated, bespoke solutions. BEAST is further integrated within a broader ecosystem of companion tools: arcjetCV [3] extracts recession rates and shock standoff distances from high-speed video using computer vision, and miniSTARscan [4] provides sub-minute, portable photogrammetric surface reconstruction of test articles before and after exposure. All tools share a common data schema, enabling seamless ingestion of surface geometry, imagery, and time-series data into a single, coherent experimental record.

Database↗

NASA NTRS2005

The Effects of Lunar Dust on EVA Systems During the Apollo Missions

Mission documents from the six Apollo missions that landed on the lunar surface have been studied in order to catalog the effects of lunar dust on Extra-Vehicular Activity (EVA) systems, primarily the Apollo surface space suit. It was found that the effects could be sorted into nine categories: vision obscuration, false instrument readings, dust coating and contamination, loss of traction, clogging of mechanisms, abrasion, thermal control problems, seal failures, and inhalation and irritation. Although simple dust mitigation measures were sufficient to mitigate some of the problems (i.e., loss of traction) it was found that these measures were ineffective to mitigate many of the more serious problems (i.e., clogging, abrasion, diminished heat rejection). The severity of the dust problems were consistently underestimated by ground tests, indicating a need to develop better simulation facilities and procedures.

James R Gaier↗

NASA NTRS2007

The Effects of Lunar Dust on EVA Systems During the Apollo Missions

Mission documents from the six Apollo missions that landed on the lunar surface have been studied in order to catalog the effects of lunar dust on Extra-Vehicular Activity (EVA) systems, primarily the Apollo surface space suit. It was found that the effects could be sorted into nine categories: vision obscuration, false instrument readings, dust coating and contamination, loss of traction, clogging of mechanisms, abrasion, thermal control problems, seal failures, and inhalation and irritation. Although simple dust mitigation measures were sufficient to mitigate some of the problems (i.e., loss of traction) it was found that these measures were ineffective to mitigate many of the more serious problems (i.e., clogging, abrasion, diminished heat rejection). The severity of the dust problems were consistently underestimated by ground tests, indicating a need to develop better simulation facilities and procedures.

James R Gaier↗

DOE OSTI2024

RG-CAT: Detection pipeline and catalogue of radio galaxies in the EMU pilot survey

Abstract We present source detection and catalogue construction pipelines to build the first catalogue of radio galaxies from the 270$\rm deg^2$pilot survey of the Evolutionary Map of the Universe (EMU-PS) conducted with the Australian Square Kilometre Array Pathfinder (ASKAP) telescope. The detection pipeline uses Gal-DINO computer vision networks (Gupta et al. 2024, PASA, 41, e001) to predict the categories of radio morphology and bounding boxes for radio sources, as well as their potential infrared host positions. The Gal-DINO network is trained and evaluated on approximately 5 000 visually inspected radio galaxies and their infrared hosts, encompassing both compact and extended radio morphologies. We find that the Intersection over Union (IoU) for the predicted and ground-truth bounding boxes is larger than 0.5 for 99% of the radio sources, and 98% of predicted host positions are within$3^{\prime \prime}$of the ground-truth infrared host in the evaluation set. The catalogue construction pipeline uses the predictions of the trained network on the radio and infrared image cutouts based on the catalogue of radio components identified using theSelavysource finder algorithm. Confidence scores of the predictions are then used to prioritiseSelavycomponents with higher scores and incorporate them first into the catalogue. This results in identifications for a total of 211 625 radio sources, with 201 211 classified as compact and unresolved. The remaining 10 414 are categorised as extended radio morphologies, including 582 FR-I, 5 602 FR-II, 1 494 FR-x (uncertain whether FR-I or FR-II), 2 375 R (single-peak resolved) radio galaxies, and 361 with peculiar and other rare morphologies. Each source in the catalogue includes a confidence score. We cross-match the radio sources in the catalogue with the infrared and optical catalogues, finding infrared cross-matches for 73% and photometric redshifts for 36% of the radio galaxies. The EMU-PS catalogue and the detection pipelines presented here will be used towards constructing catalogues for the main EMU survey covering the full southern sky.

Astronomy & Astrophysics↗

NASA NTRS1965