Search NASA⌕ Search

SEARCH · Search NASA

Results for “workflow development”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 667 records · Page 37
All sourcesNASA NTRSDOE OSTI
DOE OSTI2026

Deep generative learning of magnetic frustration in artificial spin ice from magnetic force microscopy images

Increasingly large datasets of microscopic images with nanoscale resolution facilitate the development of machine learning methods to identify and analyze subtle physical phenomena embedded within the images. In this work, microscopic images of honeycomb lattice spin-ice samples serve as datasets from which we automate the calculation of net magnetic moments and directional orientations of spin-ice configurations. In the first stage of our workflow, machine learning models are trained to accurately predict magnetic moments and directions within spin-ice structures. Variational Autoencoders (VAEs), an emergent unsupervised deep learning technique, are employed to generate high-quality synthetic magnetic force microscopy (MFM) images and extract latent feature representations, thereby reducing experimental and segmentation errors. The second stage of proposed methodology enables precise identification and prediction of frustrated vertices and nanomagnetic segments, effectively correlating structural and functional aspects of microscopic images. This facilitates the design of optimized spin-ice configurations with controlled frustration patterns, enabling potential on-demand synthesis.

36 MATERIALS SCIENCE↗
DOE OSTI2025

Advancing $otsdaq$ for Optimized Data Acquisition

High-energy physics (HEP) experiments demand data acquisition (DAQ) systems capable of orchestrating complex detector operations, high data throughput, and responsive, real-time feedback. Traditional systems often have steep learning curves, making onboarding difficult for new users. The Off-The-Shelf Data Acquisition $otsdaq$ framework was developed to address these issues by providing a modular and flexible interface that is easier to operate while remaining customizable enough for experimental setups. As the upcoming Mu2e experiment prepares for deployment, improving stability, usability, and performance has become increasingly critical. Our work enhances $otsdaq$ with features that streamline visualization, correct data metrics, improve debugging workflows, and stabilize the user interface.

Mohammed, Ali (ORCID:0009000860386626)↗
NASA NTRS2019

Simplifying NASA Earth Science Data and Information Access Through Natural Language Processing Based Data Analysis and Visualization

NASA Earth science data collected from satellites, model assimilation, airborne missions, and field campaigns, are large, complex and evolving. Such characteristics pose great challenges for end users (e.g., Earth science and applied science users, students, citizen scientists), particularly for those who are unfamiliar with NASA's EOSDIS and thus unable to access and utilize datasets effectively. For example, a novice user may simply ask: what is the total rainfall for a flooding event in my county yesterday? For an experienced user (e.g., algorithm developer), a question can be: how did my rainfall product perform, compared to ground observations, during a flooding event? Nonetheless, with rapid information technology development such as natural language processing, it is possible to develop simplified Web interfaces and back-end processing components to handle such questions and deliver answers in terms of text, data, or graphic results directly to users.In this presentation, we describe the main challenges for end users with different levels of expertise in accessing and utilizing NASA Earth science data. Surveys reveal that most non-professional users normally do not want to download and handle raw data as well as conduct heavy-duty data processing tasks. Often they just want some simple graphics or data for various purposes. To them, simple and intuitive user interfaces are sufficient because complicated ones can be difficult and time-consuming to learn. Professionals also want such interfaces to answer many questions from datasets. One solution is to develop a natural language based search box like Google and the search results can be text, data, graphics and more. Now the challenge is, with natural language processing, can we design a system to process a scientific question typed in by a user? In this presentation, we describe our plan for such a prototype. The workflow is: 1) extract needed information (e.g., variables, spatial and temporal information, processing methods, etc.) from the input, 2) process the data in the backend, and 3) deliver the results (data or graphics) to the user.

Liu, Zhong↗
NASA NTRSDate not supplied

Machine-Learning for Safety Critical Airborne Applications Part II: Case Study

The exceptional progress in the field of Artificial Intelligence (AI) systems, enabled by Machine Learning (ML) technology in recent years provides historic opportunities for the aviation industry. Current certification standards for avionics were developed prior to the ML renaissance and have several fundamental incompatibilities with the ML technology. WG-114 is working hard to release a new standard as soon as possible but for now there is no recognized means of compliance for ML based systems even of low criticality. In this talk, we present the custom ML workflow that can be used comply with all objectives of the current certification standards for a low-criticality (DAL D and C) ML-based system. To illustrate the practical application of the custom ML workflow we present a case study of a system based on a Deep Neural Network (DNN) intended to detect and identify airport runway signs. We present the system design, data generation, training, and verification in detail and describe how the design assurance objectives can be met for a DAL D and DAL C systems.

Johann Schumann↗
NASA NTRSDate not supplied

NASA’s Moon Trek Portal: New Capabilities Supporting Mission Planning and Engagement

Introduction: NASA’s Moon Trek (https://trek.nasa.gov/moon/) is one of a growing number of interactive, browser-based, online portals for planetary data visualization and analysis produced by NASA’s Solar System Treks Project (SSTP). Moon Trek continues to be enhanced with new data and new capabilities enabling it to facilitate the planning and conducting of upcoming lunar missions by NASA, its commercial partners, and its international partners, as well as advancing its role as a valuable outreach tool. A Comprehensive Online Web Portal: Developed at NASA’s Jet Propulsion Laboratory (JPL) and managed as a project of NASA’s Solar System Exploration Research Virtual Institute (SSERVI) at NASA Ames Research Center, Moon Trek is a browser-based web portal. The portal provides easy-to-use tools for browsing, data layering, data product blending, and feature search among thousands of data products covering topography, mineralogy, elemental abundance, geology, and much more. Visualizations are provided in var-ious map projections, interactive 3D viewing, and in virtual reality. Using an in-house stereo workflow, SSTP is able to produce new NAC-based high-resolution mosaics and DEMs. Diverse Applications for Lunar Exploration: Baseline analytic tools available to all users include dis-tance measurement, elevation profiling, sun angle calculation, and 3D print file generation. More advanced account-level tools allow users to perform more computationally intensive analyses. These include ray-traced lighting analysis for user-specified areas over user-specified time/date ranges and time intervals, electro-static surface potential analysis, subsetting of large data products, slope analysis, and Lunar Laser Ranging geometry calculation. Artificial intelligence (AI) and ma-chine learning (ML) based tools have been implemented for crater detection and hazard analysis, boulder detection and hazard analysis, and rockfall detection. New Tools Facilitating Exploration: Additional, new tools have recently been added and others are in development, offering even greater functionality in con-ducting analyses of potential landing sites and areas of surface operations. The new Line-of-Sight tool facilitates communications planning between locations on the lunar surface, between any given site on the lunar surface and a specified ground station on the Earth, and between a site on the lunar surface and a relay asset in lunar orbit, all taking into account local lunar topography. The new Data Plotter tool provides both tabular and graphical representations of pixel values along a user specified path for a growing number of data products. The new NAC Finder tool will identify and pro-vide access to NAC images that intersect a user-defined path or bounded area. The SSTP development team is looking to leverage the capabilities of its existing AI and ML crater, boulder, and rockfall detection and analysis tools, and extend that technology to a generalized feature detector that can be trained on instances of specific types of landforms and then search the lunar surface for more examples of such features. New traverse planning tools are being developed with use cases in generalized concept studies and specific mission planning in mind. These will facilitate finding optimal traverse paths based on constraints such as slope, lighting, hazard avoidance, and communications. These will be complemented by new traverse visualization capabilities. Users will be able to interactively ride along with a rover, examining 3D views of the terrain while adjusting camera height and viewing angle along with selecting different data layer overlays to drape across the terrain. Engaging the Public: The capabilities being developed for mission planning are being leveraged to further enhance Moon Trek’s proven utility as a valuable public outreach resource. This includes providing multiple lev-els of engagement with different points of entry. At its simplest level, promoting understanding through visualization, media and the public will be able to easily visualize and conduct their own exploration of lunar sites targeted by NASA and its partners. For a more in-depth experience, we are working with our stakeholders to promote understanding through interaction by extend-ing our current landing site and traverse analysis capabilities, making simplified access to these tools available to those who want to explore more deeply key factors in planning a mission through interactive and possibly even gamified experiences. The highest degree of public outreach, focusing on engagement through scientific participation, could be achieved through our work with missions and the NASA Office of the Chief Scientist on Moon Trek’s extension as a tool with specialized capabilities for facilitating citizen science. In such scenarios, participants become members of extended mis-sion science teams, using dedicated and integrated interfaces to analyze mission data to help answer questions key to lunar science and exploration. We are work-ing with NASA’s Office of Communications, museums, planetariums, and the media to help them easily integrate accurate, detailed visualizations of NASA’s lunar destinations and exploration into their content/productions and to engage diverse audiences in diverse venues.

Moon Trek↗
DOE OSTI2026

Minimizing the Electromechanical Stresses in Poloidal Field Coils by Optimizing their Numbers and Locations using FREDA Framework

Poloidal field (PF) and central solenoid (CS) coils play a crucial role in sustaining the equilibrium and preserving the shape of highly confined tokamak plasmas. Ensuring that PF coil current and mechanical stress stay within superconducting and structural limitations is an important check in the design assessment. Minimizing the PF coil currents and mechanical stresses influences reliability, cost, and performance. A free-boundary MHD equilibrium code—FreeGS is employed within the fusion reactor design and assessment (FREDA) whole facility modeling (WFM) framework to construct the plasma equilibrium based on the configuration and currents in the PF coils. Here, we present the capability of the FreeGS code to minimize the currents, forces, and electromagnetic stresses on the PF coils by optimizing their number, sizes, structures, and locations while maintaining an MHD stable plasma configuration with a large confinement factor. The workflow is initialized with a configuration of plasma parameters and coils’ locations from the 0-D tokamak build systems code in the FREDA framework. Then, FreeGS is called to calculate the initial equilibrium at the minimum total current in PF coils. Thereafter, FreeGS’s internal optimizer minimizes the currents and hoop and central forces on the PF coils while maintaining the reference equilibrium. Finally, the input configuration is updated with the optimized parameters for equilibria over the ramp-up phase of a burning-plasma operation. FREDA’s whole facility optimization capability, which includes all magnetic field coil systems, blanket, vacuum vessel (VV), first wall, divertor, etc., is under development and out of the scope for this study.

Hassan, Ehab [ORNL] (ORCID:0000000181060301)↗
DOE OSTI2026

A Scoping Review of Mixed Initiative Visual Analytics in the Automation Renaissance

Artificial agents are increasingly integrated into data analysis workflows, carrying out tasks that were primarily done by humans. Our research explores how the introduction of automation recalibrates the dynamic between humans and automating technology. To explore this question, we conducted a scoping review encompassing twenty years of mixed-initiative visual analytic systems. To describe and contrast the relationship between humans and automation, we developed an integrated taxonomy to delineate the objectives of these mixed-initiative visual analytics tools, how much automation they support, and the assumed roles of humans. Here, we describe our qualitative approach of integrating existing theoretical frameworks with new codes we developed. Our analysis shows that the visualization research literature lacks consensus on the definition of mixed-initiative systems and explores a limited potential of the collaborative interaction landscape between people and automation. Our research provides a scaffold to advance the discussion of human-AI collaboration during visual data analysis. Our integrated taxonomy is available in the form of a web application on https://smonadjemi.github.io/miva.

Monadjemi, Shayan [ORNL] (ORCID:0000000293855969)↗
DOE OSTI2026

Data supporting the manuscript "Nanometer Scale Imaging to Develop Quantitative Descriptors of Bipolar Membrane Junction Structure"

This dataset contains atomic force microscopy images associated with the manuscript "Nanometer Scale Imaging to Develop Quantitative Descriptors of Bipolar Membrane Junction Structure" by Maria Kelly, Emily R. Dunn, Ellis A. Spickermann, Josephine N. Gruber, César A. Lasalde-Ramírez, P. N. Romero Zavala, Éowyn Lucas, Ankur Gupta, Harry A. Atwater, and Wilson A. Smith. The dataset contains both raw images as well as segmented images produced by the image processing workflow described in the manuscript. A readme file and meta data file are included to provide additional details regarding the sample identity, image acquisition parameters, and file naming scheme.

36 MATERIALS SCIENCE↗
NASA NTRS2014

Evolution of the Scope and Capabilities of Uplink Support Software for Mars Surface Operations

In January of 2004 both of the Mars Exploration Rover spacecraft landed safely, initiating daily surface operations at the Jet Propulsion Laboratory for what was anticipated to be approximately three months of mobile exploration. The longevity of this mission, still ongoing after ten years, has provided not only a tremendous return of scientific data but also the opportunity to refine and improve the methodology by which robotic Mars surface missions are commanded. Since the landing of the Mars Science Laboratory spacecraft in August of 2012, this methodology has been successfully applied to operate a Martian rover which is both similar to, and quite different from, its predecessors. For MER and MSL, daily uplink operations can be most broadly viewed as converting the combined interests of both the science and engineering teams into a spacecraft-safe set of transmittable command files. In order to accomplish these ends a discrete set of mission-critical software tools were developed which not only allowed for conformation to established JPL standards and practices but also enabled innovative technologies specific to each mission. Although these primary programs provided the requisite capabilities for meeting the high-level goals of each distinct phase of the uplink process, there was little in the way of secondary software to support the smooth flow of data from one phase to the next. In order to address this shortcoming a suite of small software tools was developed to aid in phase transitions, as well as to automate some of the more laborious and error-prone aspects of uplink operations. This paper describes the evolution of this software suite, from its initial attempts to merely shorten the duration of the operator's shift, to its current role as an indispensable tool enforcing workflow of the uplink operations process and agilely responding to the new and unexpected challenges of missions which can, and have, lasted many years longer than originally anticipated.

CoUGAR↗
NASA NTRS2017

Weather from 250 Miles Up: Visualizing Precipitation Satellite Data (and Other Weather Applications) Using CesiumJS

Geospatial weather visualization remains predominately a two-dimensional endeavor. Even popular advanced tools like the Nullschool Earth display 2-dimensional fields on a 3-dimensional globe. Yet much of the observational data and model output contains detailed three-dimensional fields. In 2014, NASA and JAXA (Japanese Space Agency) launched the Global Precipitation Measurement (GPM) satellite. Its two instruments, the Dual-frequency Precipitation Radar (DPR) and GPM Microwave Imager (GMI) observe much of the Earth's atmosphere between 65 degrees North Latitude and 65 degrees South Latitude. As part of the analysis and visualization tools developed by the Precipitation Processing System (PPS) Group at NASA Goddard, a series of CesiumJS [Using Cesium Markup Language (CZML), JavaScript (JS) and JavaScript Object Notation (JSON)] -based globe viewers have been developed to improve data acquisition decision making and to enhance scientific investigation of the satellite data. Other demos have also been built to illustrate the capabilities of CesiumJS in presenting atmospheric data, including model forecasts of hurricanes, observed surface radar data, and gridded analyses of global precipitation. This talk will present these websites and the various workflows used to convert binary satellite and model data into a form easily integrated with CesiumJS.

web applications↗
DOE OSTI2026

Integrated Neutronics Modeling for Inertial Fusion Energy Systems: Development and Application to LD-FIRST

Lawrence Livermore National Laboratory (LLNL) is proposing a new Laser Driven Fusion Integration Research and Science Test Facility (LD-FIRST) with the goal of providing an experimental testbed for future Inertial Fusion Energy (IFE) systems. However, IFE systems require detailed and accurate multiphysics modeling to quantify material damage, thermal loading, and tritium breeding within complex chamber environments. This article presents the first step in an integrated multiphysics framework that couples meshed CAD-based geometry within Monte Carlo neutronic simulations to enable high-fidelity analysis of IFE chamber concepts, with future coupling to external codes. The neutronics workflow utilizes OpenMC and its third-party capability to use CAD-based geometries through DAGMC and tally on unstructured meshes with Libmesh to evaluate neutron transport behavior, geometric fidelity, and material performance under reactor-relevant conditions. The use of tailored tallies on unstructured meshes in this framework allows direct transfer without interpolating to CFD simulation tools. Two IFE chambers were evaluated, both conceived by LLNL: HYLIFE-II and Laser IFE (LIFE). This work produced high-fidelity conformal surface and volumetric meshes of the HYLIFE-II and LIFE chambers with mapped spatial insight into material damage, thermal loading, and tritium breeding. The HYLIFE-II model was built utilizing available resources and used as a test case to verify that the neutronics framework can handle complex geometries. The LIFE chamber CAD was provided by LLNL and was the main focus of this work. This work analyzes multiple ternary alloy breeding materials for the LIFE chamber, across different 6 Li enrichments to produce data relevant to the LD-FIRST project. This work also investigates the level of model fidelity for the LIFE chamber, and results show that inclusion of detailed first wall and coolant structures increased the predicted tritium breeding ratio (TBR) by ~30%, highlighting the sensitivity of tritium breeding and the need for a high-fidelity simulation framework for IFE chambers. These developments provide a scalable toolset for the design and optimization of next-generation IFE chambers, forming a solid foundation for future coupled multiphysics analysis.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗
DOE OSTI2026

Navigating the Path to Autonomy: Real-World Lessons from an Air-Free Self-Driving Laboratory

While autonomous experimentation has promise to accelerate discovery in physcial sciences, the real-world integration of predictive models and experimentation is non-trivial. Here we describe the genesis of a self-driving laboratory (SDL) for air-sensitive chemistry at Argonne National Laboratory and demonstrate the experimental design considerations needed for high-throughput experiments before predictive models can lead to scientific discovery. Our SDL was designed to explore battery electrolyte stability. Our final SDL utilized plate readers in a glovebox with a nitrogen atmosphere to perform kinetic assays and screen hundreds of battery-relevant solvents. However, the roadmap to autonomy and airfree-friendly experimentation required the complex evaluation of several spectroscopic and chromatographic methods. The greatest experimental challenges were (a) developing long-term sampling methods that remained air-free; (b) accelerating kinetics to advance reactivity projections; and (c) ensuring labware compatibility with nonaqueous solvents used in battery chemistry. Our experiences highlight the practical gap between closed-loop aspirations and the realities of chemical discovery, offering lessons on the challenges of transferring every day laboratory workflows to autonomy. These results suggest a more realistic blueprint for autonomy in chemistry—one that balances thoughtful and realistic experimental formulation.

Robertson, Lily A.↗
DOE OSTI2024

Roadmap on methods and software for electronic structure based simulations in chemistry and materials

This Roadmap article provides a succinct, comprehensive overview of the state of electronic structure methods and software for molecular and materials simulations. Seventeen distinct sections collect insights by 51 leading scientists in the field. Each contribution addresses the status of a particular area, as well as current challenges and anticipated future advances, with a particular eye towards software related aspects and providing key references for further reading. Foundational sections cover density functional theory and its implementation in real-world simulation frameworks, Green's function based many-body perturbation theory, wave-function based and stochastic electronic structure approaches, relativistic effects and semiempirical electronic structure theory approaches. Subsequent sections cover nuclear quantum effects, real-time propagation of the electronic structure, challenges for computational spectroscopy simulations, and exploration of complex potential energy surfaces. The final sections summarize practical aspects, including computational workflows for complex simulation tasks, the impact of current and future high-performance computing architectures, software engineering practices, education and training to maintain and broaden the community, as well as the status of and needs for electronic structure based modeling from the vantage point of industry environments. Overall, the field of electronic structure software and method development continues to unlock immense opportunities for future scientific discovery, based on the growing ability of computations to reveal complex phenomena, processes and properties that are determined by the make-up of matter at the atomic scale, with high precision.

36 MATERIALS SCIENCE↗
NASA NTRS2011

Generic, Extensible, Configurable Push-Pull Framework for Large-Scale Science Missions

The push-pull framework was developed in hopes that an infrastructure would be created that could literally connect to any given remote site, and (given a set of restrictions) download files from that remote site based on those restrictions. The Cataloging and Archiving Service (CAS) has recently been re-architected and re-factored in its canonical services, including file management, workflow management, and resource management. Additionally, a generic CAS Crawling Framework was built based on motivation from Apache s open-source search engine project called Nutch. Nutch is an Apache effort to provide search engine services (akin to Google), including crawling, parsing, content analysis, and indexing. It has produced several stable software releases, and is currently used in production services at companies such as Yahoo, and at NASA's Planetary Data System. The CAS Crawling Framework supports many of the Nutch Crawler's generic services, including metadata extraction, crawling, and ingestion. However, one service that was not ported over from Nutch is a generic protocol layer service that allows the Nutch crawler to obtain content using protocol plug-ins that download content using implementations of remote protocols, such as HTTP, FTP, WinNT file system, HTTPS, etc. Such a generic protocol layer would greatly aid in the CAS Crawling Framework, as the layer would allow the framework to generically obtain content (i.e., data products) from remote sites using protocols such as FTP and others. Augmented with this capability, the Orbiting Carbon Observatory (OCO) and NPP (NPOESS Preparatory Project) Sounder PEATE (Product Evaluation and Analysis Tools Elements) would be provided with an infrastructure to support generic FTP-based pull access to remote data products, obviating the need for any specialized software outside of the context of their existing process control systems. This extensible configurable framework was created in Java, and allows the use of different underlying communication middleware (at present, both XMLRPC, and RMI). In addition, the framework is entirely suitable in a multi-mission environment and is supporting both NPP Sounder PEATE and the OCO Mission. Both systems involve tasks such as high-throughput job processing, terabyte-scale data management, and science computing facilities. NPP Sounder PEATE is already using the push-pull framework to accept hundreds of gigabytes of IASI (infrared atmospheric sounding interferometer) data, and is in preparation to accept CRIMS (Cross-track Infrared Microwave Sounding Suite) data. OCO will leverage the framework to download MODIS, CloudSat, and other ancillary data products for use in the high-performance Level 2 Science Algorithm. The National Cancer Institute is also evaluating the framework for use in sharing and disseminating cancer research data through its Early Detection Research Network (EDRN).

Foster, Brian M.↗
DOE OSTI2026

Reining in an Agentic Harness for High Energy Physics

Agentic systems now address tasks across theoretical, phenomenological, and experimental high energy physics (HEP), but their scientific capabilities remain difficult to reuse across different large language models, providers, and harnesses. We argue that stable parts of these workflows should be promoted into versioned scientific operations and exposed through common protocols. Existing general-purpose harnesses can then be specialized for HEP through task-specific sets of tools and skills, while community-maintained registries would make these capabilities discoverable and citable. We identify mismatches in conventions, assumptions, and domains of validity among independently developed operations as a potential obstacle to their composition, and discuss machine-readable scientific contracts as one possible solution. These design principles and evaluation guidelines provide a near-term path toward a portable and community-maintained agentic harness for HEP.

Menzo, Tony [Alabama U.; Fermilab] (ORCID:00000002↗
DOE OSTI2025

LLM Benchmarking with LLaMA2: Evaluating Code Development Performance Across Multiple Programming Languages

The rapid evolution of large language models (LLMs) has opened new possibilities for automating various tasks in software development. This paper evaluates the capabilities of the LLaMA 2-70B model in automating these tasks for scientific applications written in commonly used programming languages. Using representative test problems, we assess the model's capacity to generate code, documentation, and unit tests, as well as its ability to translate existing code between commonly used programming languages. Our comprehensive analysis evaluates the compilation, runtime behavior, and correctness of the generated and translated code. Additionally, we assess the quality of automatically generated code, documentation, and unit tests. Here, our results indicate that while LLaMA 2-70B frequently generates syntactically correct and functional code for simpler numerical tasks, it encounters substantial difficulties with more complex, parallelized, or distributed computations, requiring considerable manual corrections. We identify key limitations and suggest areas for future improvements to better leverage AI-driven automation in scientific computing workflows.

97 MATHEMATICS AND COMPUTING↗
NASA NTRSDate not supplied

Updates in Developing a Prototype Science Pipeline and Full-Volume, Global Hyperspectral Synthetic Data Sets for NASA’s Earth System Observatory’s Upcoming Surface, Biology and Geology Mission

The Surface Biology and Geology (SBG) mission recently passed mission confirmation review and has entered phase A – design and development. SBG will acquire high resolution solar-reflected spectroscopy and thermal infrared observations at a data rate of ~2.5 TB/day and generate products at ~40 TB/day. Given that the per-day volume is greater than NASA’s total extant airborne hyperspectral data collection, collecting, processing, disseminating, and exploiting the SBG data present new challenges. To meet these challenges, we have developed a prototype science pipeline and a full-volume global hyperspectral synthetic data set to help prepare for SBG’s flight (see poster GC42D-0730). Our science pipeline is based on the science processing technology developed for NASA’s Kepler and TESS planet-hunting missions. The pipeline infrastructure, Ziggy, provides a scalable architecture for robust, repeatable, and replicable science and application products that can be run on a range of systems from a laptop to the cloud or a supercomputer. Ziggy is compliant with NASA Procedural Requirement (NPR) 7150.2C, is at a technical readiness level (TRL) of 7 and has been released to github.com/nasa/ziggy. We integrated Ziggy with EO-1/Hyperion workflows to build a prototype pipeline and ingested the 17-year mission archive that provides globally sampled visible through shortwave infrared spectra that are representative of SBG data types and volumes. We fully implemented the first stage and processed the entire 55 TB Hyperion data set from the raw data (Level 0) to top-of-the-atmosphere radiance (Level 1R). We are currently evaluating the ISOFIT atmospheric correction module to convert the L1R data to surface reflectance (Level 2) before reprocessing the full data set to L2. Crosschecks are being performed with RadCalNet as well as with coincident observations by AVIRIS. We are also investigating modern methods for georectifying the Hyperion scenes. Finally, we describe an analysis of the cost to conduct forward processing and reprocessing campaigns for SBG on HECC with dedicated compute and storage resources using the resurrected Hyperion pipeline as a proxy for full-volume SBG data. The analysis demonstrates that SBG L0 data can be processed to L2 on HECC with full reprocessing campaigns every two years for ~$2.6M over a 7-year lifespan. Moreover, 69% of the system capacity would be available for other activities, possibly enabling future open-source science activities, including algorithm development, L3+ processing, .etc.

ESD↗
DOE OSTI2025

Streaming Data in HPC Workflows Using ADIOS

The “IO Wall” problem, in which the gap between computation rate and data access rate grows continuously, poses significant problems to scientific workflows which have traditionally relied upon using the filesystem for intermediate storage between workflow stages. One way to avoid this problem in scientific workflows is to stream data directly from producers to consumers and avoiding storage entirely. However, the manner in which this is accomplished is key to both performance and usability. This paper presents the Sustainable Staging Transport, an approach which allows direct streaming between traditional file writers and readers with few application changes. SST is an ADIOS “engine”, accessible via standard ADIOS APIs, and because ADIOS allows engines to be chosen at run-time, many existing file-oriented ADIOS workflows can utilize SST for direct application-to-application communication without any source code changes. This paper describes the design of SST and presents performance results from various applications that use SST, for feeding model training with simulation data with substantially higher bandwidth than the theoretical limits of Frontier’s file system, for strong coupling of separately developed applications for multiphysics multiscale simulation, or for in situ analysis and visualization of data to complete all data processing shortly after the simulation finishes.

Podhorszki, Norbert [ORNL] (ORCID:000000019647542X↗