SEARCH · Search NASA

Results for “adaptive workflow”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 145 records · Page 8

All sources NASA NTRS DOE OSTI

DOE OSTI2026

iDDS: intelligent distributed dispatch and scheduling for workflow orchestration

The intelligent distributed dispatch and scheduling (iDDS) service is a versatile workflow orchestration system designed for large-scale, distributed scientific computing. iDDS extends traditional workload and data management by integrating data-aware execution, conditional logic, and programmable workflows, enabling automation of complex and dynamic processing pipelines. Originally developed for the ATLAS experiment at the large hadron collider, iDDS has evolved into an experiment-agnostic platform that supports both template-driven workflows and a Function-as-a-Task model for Python-based orchestration. This paper presents the architecture and core components of iDDS, highlighting its scalability, modular message-driven design, and integration with systems such as PanDA and Rucio. We demonstrate its versatility through real-world use cases: fine-grained tape resource optimization for ATLAS, orchestration of large Directed Acyclic Graph (DAG) workflows for the Rubin Observatory, distributed hyperparameter optimization for machine learning applications, active learning for physics analyses, and AI-assisted detector design at the electron–ion collider. By unifying workload scheduling, data movement, and adaptive decision-making, iDDS reduces operational overhead and enables reproducible, high-throughput workflows across heterogeneous infrastructures. We conclude with current challenges and future directions, including interactive, cloud-native, and serverless workflow support.

97 MATHEMATICS AND COMPUTING↗

NASA NTRSDate not supplied

Science Workflows using Kamodo

Kamodo is a powerful python software package based on data functionalization. Once a given data set is functionalized, a large variety of capabilities are easily accessible in Kamodo, including unit conversions, custom analysis via function composition, interactive publication quality visualizations, and LaTeX encoding. The entirety of capabilities available in Kamodo are easily applied to both simulated and observed data across the multiple domains of Heliophysics and even in other disciplines. This work includes a variety of science workflows using Kamodo in combination with other resources, including with other python software packages, that expand the utility of Kamodo even further. These workflows include model-data comparisons, ensemble modeling examples, satellite mission planning examples, and other applications, all of which are freely available on CCMC’s Kamodo Github page for the community to adapt to their own uses (https://github.com/nasa/Kamodo). We invite the community to use these workflows and to contribute their own to share.

software↗

NASA NTRSDate not supplied

Science Workflows using Kamodo

python↗

DOE OSTI2024

Integrating adaptive learning with post hoc model explanation and symbolic regression to build interpretable surrogate models

Abstract We develop a materials informatics workflow to build an interpretable surrogate model for micromagnetic simulations. Our goal is to predict the energy barrier of a moving isolated skyrmion in rare-earth-free $$\hbox {Mn}_4$$ Mn 4 N. Our approach integrates adaptive learning with post hoc model explanation and symbolic regression methods. We discuss an unexplored acquisition function (information condensing active learning) within the adaptive learning loop and compare it with the known standard deviation function for efficient navigation of the search space. Model-agnostic post hoc explanation techniques then uncover trends learned by the trained model, which we then leverage to constrain the expressions used for symbolic regression. Graphical abstract

Biswas, Ankita↗

DOE OSTI2024

Integrating HPC, AI, and Workflows for Scientific Data Analysis: Report from Dagstuhl Seminar 23352

The Dagstuhl Seminar 23352, titled “Integrating HPC, AI, and Workflows for Scientific Data Analysis,” held from August 27 to September 1, 2023, was a significant event focusing on the synergy between High-Performance Computing (HPC), Artificial Intelligence (AI), and scientific workflow technologies. The seminar recognized that modern Big Data analysis in science rests on three pillars: workflow technologies for reproducibility and steering, AI and Machine Learning (ML) for versatile analysis, and HPC for handling large data sets. These elements, while crucial, have traditionally been researched separately, leading to gaps in their integration. The seminar aimed to bridge these gaps, acknowledging the challenges and opportunities at the intersection of these technologies. The event highlighted the complex interplay between HPC, workflows, and ML, noting how ML has increasingly been integrated into scientific workflows, thereby enhancing resource demands and bringing new requirements to HPC architectures, like support for GPUs and iterative computations. The seminar also addressed the challenges in adapting HPC for large-scale ML tasks, including in areas like deep learning, and the need for workflow systems to evolve to leverage ML in data analysis fully. Moreover, the seminar explored how ML could optimize scientific workflow systems and HPC operations, such as through improved scheduling and fault tolerance. A key focus was on identifying prestigious use cases of ML in HPC and understanding their unique, unmet requirements. The stochastic nature of ML and its impact on the reproducibility of data analysis on HPC systems was also a topic of discussion.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2018

Machine Detection of Enhanced Electromechanical Energy Conversion in PbZr 0.2 Ti 0.8 O 3 Thin Films

Many energy conversion, sensing, and microelectronic applications based on ferroic materials are determined by the domain structure evolution under applied stimuli. New hyperspectral, multidimensional spectroscopic techniques now probe dynamic responses at relevant length and time scales to provide an understanding of how these nanoscale domain structures impact macroscopic properties. Such approaches, however, remain limited in use because of the difficulties that exist in extracting and visualizing scientific insights from these complex datasets. Using multidimensional band-excitation scanning probe spectroscopy and adapting tools from both computer vision and machine learning, an automated workflow is developed to featurize, detect, and classify signatures of ferroelectric/ferroelastic switching processes in complex ferroelectric domain structures. This approach enables the identification and nanoscale visualization of varied modes of response and a pathway to statistically meaningful quantification of the differences between those modes. Lastly, among other things, the importance of domain geometry is spatially visualized for enhancing nanoscale electromechanical energy conversion.

36 MATERIALS SCIENCE↗

DOE OSTI2022

NATURF

The Neighborhood Adaptive Tissues for Urban Resilience Futures tool (NATURF) is a Python workflow that generates les readable by the Weather Research and Forecasting (WRF) model. The NATURF Python modules use shapefiles containing building footprint and height data as input to calculate 132 building parameters at any resolution and converts the parameters into a binary file format.

Allen-Dumas, MelissaRee [Oak Ridge National Lab. (↗

DOE OSTI2019

Harnessing the power of supercomputers using the PanDA Pilot 2 in the ATLAS Experiment

The unprecedented computing resource needs of the ATLAS experiment at LHC have motivated the Collaboration to become a leader in exploiting High Performance Computers (HPCs). To meet the requirements of HPCs, the PanDA system has been equipped with two new components; Pilot 2 and Harvester, that were designed with HPCs in mind. While Harvester is a resource-facing service which provides resource provisioning and workload shaping, Pilot 2 is responsible for payload execution on the resource. The presentation focuses on Pilot 2, which is a complete rewrite of the original PanDA Pilot used by ATLAS and other experiments for well over a decade. Pilot 2 has a flexible and adaptive design that allows for plugins to be defined with streamlined workflows. In particular, it has plugins for specific hardware infrastructures (HPC/GPU clusters) as well as for dedicated workflows defined by the needs of an experiment. Examples of dedicated HPC workflows are discussed in which the Pilot either uses an MPI application for processing fine-grained event level service under the control of the Harvester service or acts like an MPI application itself and runs a set of job in an assemble. In addition to describing the technical details of these workflows, results are shown from its deployment on Titan (OLCF) and other HPCs in ATLAS.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

DOE OSTI2016

Towards a Scalable and Adaptive Application Support Platform for Large-Scale Distributed E-Sciences in High-Performance Network Environments

The advent of large-scale collaborative scientific applications has demonstrated the potential for broad scientific communities to pool globally distributed resources to produce unprecedented data acquisition, movement, and analysis. System resources including supercomputers, data repositories, computing facilities, network infrastructures, storage systems, and display devices have been increasingly deployed at national laboratories and academic institutes. These resources are typically shared by large communities of users over Internet or dedicated networks and hence exhibit an inherent dynamic nature in their availability, accessibility, capacity, and stability. Scientific applications using either experimental facilities or computation-based simulations with various physical, chemical, climatic, and biological models feature diverse scientific workflows as simple as linear pipelines or as complex as a directed acyclic graphs, which must be executed and supported over wide-area networks with massively distributed resources. Application users oftentimes need to manually configure their computing tasks over networks in an ad hoc manner, hence significantly limiting the productivity of scientists and constraining the utilization of resources. The success of these large-scale distributed applications requires a highly adaptive and massively scalable workflow platform that provides automated and optimized computing and networking services. This project is to design and develop a generic Scientific Workflow Automation and Management Platform (SWAMP), which contains a web-based user interface specially tailored for a target application, a set of user libraries, and several easy-to-use computing and networking toolkits for application scientists to conveniently assemble, execute, monitor, and control complex computing workflows in heterogeneous high-performance network environments. SWAMP will enable the automation and management of the entire process of scientific workflows with the convenience of a few mouse clicks while hiding the implementation and technical details from end users. Particularly, we will consider two types of applications with distinct performance requirements: data-centric and service-centric applications. For data-centric applications, the main workflow task involves large-volume data generation, catalog, storage, and movement typically from supercomputers or experimental facilities to a team of geographically distributed users; while for service-centric applications, the main focus of workflow is on data archiving, preprocessing, filtering, synthesis, visualization, and other application-specific analysis. We will conduct a comprehensive comparison of existing workflow systems and choose the best suited one with open-source code, a flexible system structure, and a large user base as the starting point for our development. Based on the chosen system, we will develop and integrate new components including a black box design of computing modules, performance monitoring and prediction, and workflow optimization and reconfiguration, which are missing from existing workflow systems. A modular design for separating specification, execution, and monitoring aspects will be adopted to establish a common generic infrastructure suited for a wide spectrum of science applications. We will further design and develop efficient workflow mapping and scheduling algorithms to optimize the workflow performance in terms of minimum end-to-end delay, maximum frame rate, and highest reliability. We will develop and demonstrate the SWAMP system in a local environment, the grid network, and the 100Gpbs Advanced Network Initiative (ANI) testbed. The demonstration will target scientific applications in climate modeling and high energy physics and the functions to be demonstrated include workflow deployment, execution, steering, and reconfiguration. Throughout the project period, we will work closely with the science communities in the fields of climate modeling and high energy physics including Spallation Neutron Source (SNS) and Large Hadron Collider (LHC) projects to mature the system for production use.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2025

El Agente: An autonomous agent for quantum chemistry

Computational chemistry tools are widely used to study the behavior of chemical phenomena. Yet, the complexity of these tools can make them inaccessible to non-specialists and challenging even for experts. In this work, we introduce El Agente Q, an LLM-based multi-agent system that dynamically generates and executes quantum chemistry workflows from natural language user prompts. The system is built on a novel cognitive architecture featuring a hierarchical memory framework that enables flexible task decomposition, adaptive tool selection, post-analysis, and autonomous file handling and submission. El Agente Q is benchmarked on six university-level course exercises and two case studies, demonstrating robust problem-solving performance (averaging >87% task success) and adaptive error handling through in situ debugging. It also supports longer-term, multi-step task execution for more complex workflows, while maintaining transparency through detailed action trace logs. Together, these capabilities lay the foundation for increasingly autonomous and accessible quantum chemistry.

agentic systems↗

DOE OSTI2019

Decommissioning of a Historic Waste Storage Facility in Austria - 19059

Nuclear Engineering Seibersdorf Ltd. (NES) performed decommissioning of an old waste storage facility, the so called 'bunker', which was commissioned in 1968 and used until 1978 for the storage of radioactive waste from medical, industrial and research activities, as well as from the operation of the Austrian research reactor ASTRA. Due to the layout of the bunker the initial decommissioning plan was to decontaminate the inner walls of the caverns and afterwards perform clearance measurements on the standing structure. However, throughout the first phase of decommissioning it turned out, that contaminations penetrated deeper into the concrete structure as initially anticipated, which made an adoption of the decommissioning strategy necessary. By changing the workflow and to remove non-contaminated parts of the structure first, also the strategy for the clearance measurements had to be adapted. The selective decommissioning strategy and the demand that measurements shall be performed by an accredited body, required to set up an adequate communication structure to keep the workflow going. Moreover, a fast communication between NES, the competent authority and the authority's (external) experts was necessary to receive clearance licenses on time, and to prevent backlog of dismantled material. The paper reports on the initial and the adapted decommissioning strategy and the challenges within the changed approach. Applied decontamination and decommissioning methods and the implemented radiation protection measures will be discussed. Furthermore, information on the methodologies for clearance measurements, the collaboration of the accredited body with the decommissioning team, as well as on the communication between NES and the competent authority will be given. Finally, a balance of radioactive waste vs. conventionally disposed waste will be shown. (authors)

12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗

DOE OSTI2018

Evaluation of dose recalculation vs dose deformation in a commercial platform for deformable image registration with a computational phantom

Deformable image registration (DIR) is an important tool for mapping both dose and contours of a new set of planning images when recurrent, that is, adaptive radiotherapy, or further treatment, that is, re-treatment, is required. The aim of this study was to evaluate the need for plan recalculation in deformed anatomies and to develop a reliable workflow for validating the DIR algorithm to be applied to dose warping such as dose accumulation (DA) for adaptive radiation therapy, and dose summation (DS) in the case of re-treatment. A set of 3 computational phantoms was developed to validate the application of B-Spline-based registrations for dose mapping among the various computed tomography image data sets. Two different versions were defined for each phantom to simulate clinical needs: adaptive radiotherapy and re-treatment; a DIR was performed to obtain a displacement vector field (DVF) for dose applications. Comparison of calculated and deformed doses was carried out by means of known markers inside the virtual phantoms. The differences were evaluated using a 3% dose index as acceptance criteria. A paired Wilcoxon signed-rank test was carried out to test the statistical significance of differences within the markers. Significant differences were only observed for the deformed dose in the DA test; no significant differences were observed for recalculated dose values in the DA and DS tests. The dose index is in accordance with these results. The warping dose process obtained by registering various sets of images was validated; a recalculation approach seems to be more accurate for DA purposes and image-guided adaptive radiation therapy (IGART) applications.

62 RADIOLOGY AND NUCLEAR MEDICINE↗

DOE OSTI2018

A View from ORNL: Scientific Data Research Opportunities in the Big Data Age

One of the core issues across computer and computational science today is adapting to, managing, and learning from the influx of "Big Data". In the commercial space, this problem has led to a huge investment in new technologies and capabilities that are well adapted to dealing with the sorts of human-generated logs, videos, texts, and other large-data artifacts that are processed and resulted in an explosion of useful platforms and languages (Hadoop, Spark, Pandas, etc.). However, translating this work from the enterprise space to the computational science and HPC community has proven somewhat difficult, in part because of some of the fundamental differences in type and scale of data and timescales surrounding its generation and use. We describe a forward-looking research and development plan which centers around the concept of making Input/Output (I/O) intelligent for users in the scientific community, whether they are accessing scalable storage or performing in situ workflow tasks. Much of our work is based on our experience with the Adaptable I/O System (ADIOS 1.X), and our next generation version of the software ADIOS 2.X [1].

Klasky, Scott↗

NASA NTRSDate not supplied

Comparing Unstructured Adaptive Mesh Solutions for the High Lift Common Research Model Airfoil

Discretization error is a common source of uncertainty in Computational Fluid Dynamics (CFD) analyses. Traditional means of controlling discretization error through fixed-mesh refinement studies has proven to be difficult particularly when modeling complex geometries and flow fields. One reason for this is that mesh generation in today’s production CFD workflow is often a labor intensive process that is heavily dependent on user judgment. Unstructured mesh adaptation is known to be an efficient way to control discretization errors in CFD. Adaptive methods replace user based decision making with automated processes that optimize a mesh to reduce discretization error. This paper compares the application of multiple solution adaptive techniques in combination with multiple flow solvers to solve for the flow field about a 2D airfoil section of the NASA High-Lift Common Research Model (HL-CRM). By driving the adaptive mesh processes to a similar level of mesh convergence, the ability to achieve consistent results between multiple adaptive techniques and flow solvers is demonstrated. Mesh convergence for the various adaptive mesh approaches is compared identifying potential areas for improvement and providing mesh generation guidance for future workshops.

mesh adaptation high-lift 2D airfoil↗

NASA NTRSDate not supplied

Comparing Unstructured Adaptive Mesh Solutions for the High Lift Common Research Model Airfoil

mesh adaptation↗

DOE OSTI2017

Topology-Aware Performance Optimization and Modeling of Adaptive Mesh Refinement Codes for Exascale

Here, we introduce a topology-aware performance optimization and modeling workflow for AMR simulation that includes two new modeling tools, ProgrAMR and Mota Mapper, which interface with the BoxLib AMR framework and the SSTmacro network simulator. ProgrAMR allows us to generate and model the execution of task dependency graphs from high-level specifications of AMR-based applications, which we demonstrate by analyzing two example AMR-based multigrid solvers with varying degrees of asynchrony. Mota Mapper generates multiobjective, network topology-aware box mappings, which we apply to optimize the data layout for the example multigrid solvers. While the sensitivity of these solvers to layout and execution strategy appears to be modest for balanced scenarios, the impact of better mapping algorithms can be significant when performance is highly constrained by network hop latency. Furthermore, we show that network latency in the multigrid bottom solve is the main contributing factor preventing good scaling on exascale-class machines.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2026

Self-supervised and multi-fidelity learning for extended predictive soil spectroscopy

Infrared spectroscopy is a cost-effective, non-destructive, and environmentally benign technology that is increasingly recognized as an important solution for meeting the global demand for soil data. While both near-infrared (NIR) and mid-infrared (MIR) diffuse reflectance spectroscopy enable rapid estimation of soil properties, they present a significant trade-off: NIR offers superior scalability and lower operational costs, whereas MIR provides higher analytical fidelity by capturing fundamental molecular vibrations. In this study, we propose a self-supervised, multi-fidelity learning framework designed to bridge this gap. Our approach leverages large-scale MIR spectral libraries to learn a compact, transferable latent representation, into which NIR spectra are subsequently aligned for downstream prediction. The workflow consists of pretraining a latent model on a large MIR library, adapting the representation using a smaller paired NIR–MIR dataset, and evaluating generalization on an independent external test set. Across a range of chemical and physical soil properties, we found that MIR-derived embeddings improved prediction accuracy relative to baseline models that used raw MIR inputs. Predictions derived from the spectrum conversion (NIR to MIR) task did not match the performance of the original MIR spectra but were similar or superior to predictive performance of NIR-only models, suggesting the unified spectral latent space can effectively leverage the larger and more diverse MIR dataset for prediction of soil properties not well represented in current NIR libraries.

54 ENVIRONMENTAL SCIENCES↗

DOE OSTI2024

Towards a High Fidelity Training Environment for Autonomous Cyber Defense Agents

Cyber defenders are overwhelmed by the frequency and scale of attacks against their networks. This problem will only be exacerbated as attackers leverage AI to automate their workflows. Autonomous cyber defense capabilities could aid defenders by automating operations and adapting dynamically to novel threats. However, existing training environments fall short in areas such as generalization, explainability, scalability, and transferability, making it intractable to train agents that will be effective in real networks. In this paper we take an important step towards creating autonomous cyber defense agents — we present a high fidelity training environment called Cyberwheel that includes both simulation and emulation capabilities. Cyberwheel simplifies customization of the training network and easily allows redefining the agent’s reward function, observation space, and action space to support rapid experimentation of novel approaches to agent design. It also provides visibility into agent behaviors necessary for agent evaluation and sufficient documentation / examples to lower the barrier to entry. As an example use case of Cyberwheel, we present initial results training an autonomous agent to deploy cyber deception strategies in simulation.

Oesch, T↗