Search NASA⌕ Search

SEARCH · Search NASA

Results for “Distributed Data Management”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 37 records · Page 2
All sourcesNASA NTRSDOE OSTI
DOE OSTI2025

A Data-Driven Voltage Control Strategy for Distribution Grids With Distributed Energy Resources

Traditionally, distribution system control approaches have been model-based. The deployment of advanced metering infrastructure has provided electric utilities with the capability of data-driven control with real-time measurements. The shift from model-based to data-driven control represents a significant advancement in the management of distribution systems, offering a more adaptive approach to system control because of the ability to dynamically adapt to changing conditions without the need for system modeling. Here, in this paper, a behavioral data-driven control method is developed to provide voltage regulation to an actual distribution system by controlling the legacy devices and distributed energy resource (DER) assets. The studied distribution system has a load tap changer and three capacitor banks as the legacy devices and photovoltaic systems as the DERs. The performance of the proposed control algorithm is validated using a laboratory test bed setup considering multiple scenarios. The results show that the proposed control achieved 99% voltage regulation.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2024

TrustDER: Trusted, Private and Scalable Coordination of Distributed Energy Resources

In this project, the Stanford and SLAC Teams have developed a Trusted, Private and Scalable platform for coordinating Coordination of Distributed Energy Resources (TrustDER). This is a layered system that ensures private, trusted and scalable coordination and monitoring of DERs. It accommodates a variety of resources, such as solar generation, gensets and loads, with a particular focus on battery systems-based resources, as they are a transformational technology experiencing fast growth in adoption by large critical facilities. The platform can be used as standalone or added to existing aggregation systems to enable trust, privacy and resilience. TrustDER consists of layers that address each of the shortcomings of the existing state of the art. Each layer in the platform can operate independently but provides information to the layers above it to enable a novel form of overall coordination architecture. The project consists of several tasks, with each task dedicated to the design of each layer. Task 2 Resource Virtualization defined a software abstraction layer for distributed energy resources (DERs). The goal of this abstraction was to simplify the implementation of algorithms utilizing cooperation of DERs resources in a variety of use cases. Task 3 is on Secure ID for Asset Authentication. Identity Management Systems (IDMS) are a foundational infrastructure for interactions between entities (organizations, users, devices, and services). Secure ID is blockchain-based a distributed identity management system allowing (1) identity provisioning, (2) authentication, (3) authorization, and (4) identity data sharing for IoT-enabled assets on the electricity grid. In this project, the SLAC team focused on designing and testing Keymaker, a protocol for authenticating device identity managed by Secure ID. Task 5 Private and Safe Integration is focused on the design and evaluation of a DER cooperation scheme which allows for the aggregation of DERs without impacting network reliability. The approach is designed based on realistic assumptions regarding data availability, communication infrastructure limitations, and privacy. Task 6 Scalable Distributed Privacy for Information explored how virtualized batteries could be managed privately. Specifically, it examined the case in which a principal provides a partitioned battery to multiple clients. Task 7 Use Cases was to ensure that this technology was applied in relevant situations and scenarios. Primarily, this means that virtualization needed to be employed in a manner that either improved flexibility, bolstered security or privacy, or decreased costs.

25 ENERGY STORAGE↗
DOE OSTI2026

Powered By CADET

The Capacity Expansion Decision Support for Distribution Networks (CADET) is a Python-based library and framework for creating electrical distribution system capacity planning tools for cost-effective, reliable power delivery. It enables the creation of modular, scalable, and extensible distribution capacity planning tools by providing a high-level optimization interface, parameter and options data managers, optimization constraint and objective libraries, generalized nomenclature, a system for tracking and modifying distribution network changes, optimization solution validation, and other capabilities. This webinar will describe 1) the motivation for creating CADET, 2) key designs, and 3) several use cases.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2026

iDDS: intelligent distributed dispatch and scheduling for workflow orchestration

The intelligent distributed dispatch and scheduling (iDDS) service is a versatile workflow orchestration system designed for large-scale, distributed scientific computing. iDDS extends traditional workload and data management by integrating data-aware execution, conditional logic, and programmable workflows, enabling automation of complex and dynamic processing pipelines. Originally developed for the ATLAS experiment at the large hadron collider, iDDS has evolved into an experiment-agnostic platform that supports both template-driven workflows and a Function-as-a-Task model for Python-based orchestration. This paper presents the architecture and core components of iDDS, highlighting its scalability, modular message-driven design, and integration with systems such as PanDA and Rucio. We demonstrate its versatility through real-world use cases: fine-grained tape resource optimization for ATLAS, orchestration of large Directed Acyclic Graph (DAG) workflows for the Rubin Observatory, distributed hyperparameter optimization for machine learning applications, active learning for physics analyses, and AI-assisted detector design at the electron–ion collider. By unifying workload scheduling, data movement, and adaptive decision-making, iDDS reduces operational overhead and enables reproducible, high-throughput workflows across heterogeneous infrastructures. We conclude with current challenges and future directions, including interactive, cloud-native, and serverless workflow support.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2024

Towards Cross-Facility Workflows Orchestration through Distributed Automation

Modern science relies on end-to-end workflows that incorporate experimental instruments and utilize edge, cloud, or high-performance computing and storage resources. These components are geographically dispersed across various user facilities and interconnected through high-speed networks. In this paper, we present Zambeze, an automated distributed framework designed to facilitate this new class of cross-facility workflows. Utilizing swarm intelligence principles, Zambeze orchestrates science campaigns by managing distributed autonomous agents. These agents can offer a suite of services, including computing, storage, and data management. We demonstrate the feasibility of Zambeze through a real-world application involving electron microscopy, enhanced with Artificial Intelligence capabilities.

Skluzacek, Tyler↗
DOE OSTI2025

The environmental impact, carbon emissions and sustainability of computing in the ATLAS experiment

ATLAS, a general-purpose experiment at the Large Hadron Collider (LHC), makes use of a large internationally-distributed computing infrastructure, including over 10 6 TB of managed data on disk and tape and almost one million simultaneously running CPU cores. Upgrades for the High-Luminosity LHC (HL-LHC) will increase the required computing resources by a factor of 3–4 by the beginning of the 2030s, and by an order of magnitude before the conclusion of data taking at the beginning of the 2040s. These resources are spread over around 100 computing sites worldwide. Efforts are underway within the experiment to evaluate and mitigate various aspects of the environmental impact of the sites, with the additional long-term goal of making recommendations to the sites that will significantly reduce the total expected environmental impact in the HL-LHC era. These efforts take several forms: building awareness in the experiment community, adjusting aspects of the computing policy, and modifications of data center configurations, either in ways that take advantage of particular features of ATLAS workloads or in generic ways that reduce the environmental impact of the computing resources. This paper describes the ongoing investigations and approaches that have already provided useful and actionable outcomes.

Aad, G. [CNRS/IN2P3] (ORCID:0000000266654934)↗
DOE OSTI2025

Networked Microgrid Energy Management Considering Ownership and Control Structures: A Comparison

With the growing deployment of microgrids, networked microgrids have emerged for their additional advantages of economy, reliability and resilience by coordinating the operation of multiple microgrids. As microgrids are operated with different ownership, objectives and functionalities, the formed networked microgrids show characteristics of mixed ownership, inconsistent objectives and various functionalities. To enable the coordinated operation of networked microgrids, three control structures, i.e., centralized, distributed and decentralized, have been constructed in the literature. However, the data sharing enabling these different paradigms and the resulted value propositions are not well defined, leading to poor resource management and resilience, etc. To solve this issue, a complete comparison of networked microgrid energy management under centralized, distributed and decentralized structures are performed. As a novel contribution, the required minimum data exchange of networked microgrid energy management under three control structures are identified, respectively. The value propositions are calculated and compared against each other by the results of case studies.

control structures↗
DOE OSTI2025

Architectural Approaches for Integrating ADMS and DERMS: Challenges, Comparisons, and Real-World Use Cases

The electrical distribution landscape is rapidly transforming due to the proliferation of distributed energy resources (DERs) such as solar panels, wind turbines, battery storage systems, combined heat and power units, and electric vehicles, introducing variability and uncontrollability that traditional grid operators are ill-equipped to manage. This transformation is further accelerated by advancements in Information and Communication Technology infrastructure that connects control centers with end devices, demanding automation and a deeper understanding of new technologies by utility personnel. Advanced grid control techniques using system-level optimization, Artificial Intelligence, and Machine Learning at the enterprise level and distributed level are evolving to address these issues. There is also an opportunity to utilize the enormous data created by these new DER technologies in the grid. Advanced Distribution Management Systems (ADMS) and Distributed Energy Resource Management Systems (DERMS) are critical in addressing these challenges by automating grid operations and enhancing reliability. Given the relatively recent development of ADMS and DERMS, and the still relatively low level of ADMS and DERMS deployment in the industry, there is a notable deficiency in the comprehensive understanding of the challenges and benefits associated with these new technologies, especially with their complementary natures and integration architectures. This paper aims to bridge the knowledge gap in ADMS and DERMS integration, presenting three distinct integration architectures currently available, and discussing the challenges and benefits of each architecture to guide utilities, industry professionals, and researchers in optimizing grid management and decision-making processes for a resilient and efficient energy future.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2024

Enhancing Discoverability and Management of Atmospheric Data at Scale: Solutions from the ARM Data Center

The Atmospheric Radiation Measurement (ARM) is a multi-laboratory and multi-institutional U.S. Department of Energy (DOE) Office of Science National User Facility. The ARM Data Center (ADC), located at Oak Ridge National Laboratory, collects, archives, and shares vast atmospheric data crucial for climate research. The ADC manages over 7 PB of data from 460 instruments worldwide, processing it into more than 11,000 diverse data products using the Network Common Data Form (NetCDF) for machine-independent accessibility. The primary challenge addressed in this paper is the efficient management and distribution of vast and diverse datasets essential for the climate research community, enhancing accessibility through advanced tools like Data Discovery. The ADC has developed advanced infrastructure and software architecture to handle the continuous influx of heterogeneous data to enhance data discoverability, resulting in increased scientific collaboration. In 2023, users from over 34 countries downloaded and utilized ARM data, resulting in 1,455 publications. The ADC’s efforts have significantly improved the discoverability and usability of atmospheric data, fostering extensive scientific research and collaboration. This paper details the solutions implemented by the ADC team for efficient data discovery and distribution, and it demonstrates ARM’s capability of staging processed data for scientific analysis.

Shah, Chirag [ORNL] (ORCID:0000000203145737)↗
DOE OSTI2025

Scientific Data Management Beyond Traditional Computing Boundaries

Scientific data management is undergoing a fundamental transformation driven by the convergence of artificial intelligence (AI)/machine learning workflows, distributed computing and storage environments, and exponential data growth. Here, we analyze how these developments address current limitations while enabling new capabilities for cross-facility collaboration and AI-driven research.

Widener, Patrick [Oak Ridge National Laboratory (O↗
DOE OSTI2025

Resilient Entanglement Distribution in a Multihop Quantum Network

The evolution of quantum networking requires architectures capable of dynamically reconfigurable entanglement distribution to meet diverse user needs and ensure tolerance against transmission disruptions. We introduce multihop quantum networks to improve network reach and resilience by enabling quantum communications across intermediate nodes, thus broadening network connectivity and increasing scalability. We present multihop two-qubit polarization-entanglement distribution within a quantum network at the Oak Ridge National Laboratory campus. Our system uses wavelength-selective switches for adaptive bandwidth management on a software-defined quantum network that integrates a quantum data plane with classical data and control planes, creating a flexible, reconfigurable mesh. Our network distributes entanglement across six nodes within three subnetworks, each located in a separate building, optimizing quantum state fidelity and transmission rate through adaptive resource management. Additionally, we demonstrate the network's resilience by implementing a link recovery approach that monitors and reroutes quantum resources to maintain service continuity despite link failures—paving the way for scalable and reliable quantum networking infrastructures.

Alshowkan, Muneer [Oak Ridge National Laboratory (↗
DOE OSTI2026

Integrated Energy-Water Data for Cross-Sector Resilience

This white paper focuses on the “energy-for-water” domain, addressing the urgent need for integrated, empirical data to support regional management, benchmarking, and research on improving efficiency and developing technologies for water and wastewater management systems. The costs and energy required for the supply, treatment, and distribution of water and wastewater lack a standard data collection mechanism and centralized database or storage infrastructure, limiting data-driven decision-making across interdependent infrastructure systems.

42 ENGINEERING↗
DOE OSTI2024

Efficient distributed continual learning for steering experiments in real-time

Deep learning has emerged as a powerful method for extracting valuable information from large volumes of data. However, when new training data arrives continuously (i.e., is not fully available from the beginning), incremental training suffers from catastrophic forgetting (i.e., new patterns are reinforced at the expense of previously acquired knowledge). Training from scratch each time new training data becomes available would result in extremely long training times and massive data accumulation. Rehearsal-based continual learning has shown promise for addressing the catastrophic forgetting challenge, but research to date has not addressed performance and scalability. To fill this gap, we propose an approach based on a distributed rehearsal buffer that efficiently complements data-parallel training on multiple GPUs to achieve high accuracy, short runtime, and scalability. It leverages a set of buffers (local to each GPU) and uses several asynchronous techniques for updating these local buffers in an embarrassingly parallel fashion, all while handling the communication overheads necessary to augment input minibatches using unbiased, global sampling. We further propose a generalization of rehearsal buffers to support both classification and generative learning tasks, as well as more advanced rehearsal strategies (notably Dark Experience Replay, leveraging knowledge distillation). We illustrate this approach with a real-life HPC streaming application from the domain of ptychographic image reconstruction. Furthermore, we run extensive experiments on up to 128 GPUs of the ThetaGPU supercomputer to compare our approach with baselines representative of training-from-scratch (the upper bound in terms of accuracy) and incremental training (the lower bound). Results show that rehearsal-based continual learning achieves a top-5 validation accuracy close to the upper bound, while simultaneously exhibiting a runtime close to the lower bound.

Asynchronous data management↗
DOE OSTI2025

A Data Science and Machine Learning Platform Supporting Large Particle Accelerator Control and Diagnostics Applications Final Report: SBIR Initial Phase II DE-SC0022583

The Machine Learning Data Platform (MLDP) is a product providing full-stack support for data science, Machine Learning, and Artificial Intelligence (ML/AI) applications at particle accelerator and large experimental physics facilities. It supports ML/AI applications from front-end, high-speed acquisition of heterogeneous, time-series data, through data archiving and management, to back-end analysis. The MLDP embodies a “data-science ready” platform for data analysis and ML/AI applications in diagnosis, modelling, control, and optimization of these facilities. It provides data scientists and applications a consistent, datacentric interface to archive data standardizing implementation and deployment of ML/AI algorithms to different operations configurations within the same facility, or between facilities. Being an open-source, public-domain project, the MLDP is intended for broadest possible impact by increasing accessibility and minimizing the required expertise for installation and operation. The MLDP can also be deployed at user facilities for experimental data collection, archiving, and analysis. It is capable of acquisition and archiving of heterogeneous data from experimental equipment (e.g., images, arrays, structures, etc.) along with system hardware configurations (e.g., scalars, tables), control system process variables, and any metadata required for provenance. Thus, the MLDP can manage experimental data through its entire lifecycle, from acquisition and archiving, through analysis and investigation, to release and final publication.

43 PARTICLE ACCELERATORS↗
DOE OSTI2024

A Framework to Demonstrate a DNP3 Interface With a CIM-Based Data Integration Platform: Preprint

The contemporary electrical grid is characterized by its complexity and abundance of data. A control-rich environment supported by information and communication technologies within an Advanced Distribution Management System (ADMS) presents a viable and cost-effective option for utility companies aiming to implement advanced real-time analytical schemes for monitoring and remotely controlling distribution feeders. Modular platform-based approaches to distribution operations require a structured framework for acquiring field device measurements, performing analytics, converting the setpoint to the correct protocol, and sending it on the appropriate communications network to the field devices. We present the development and deployment of an application service to integrate an open-source standardsbased platform with an ADMS test bed with field devices using the Distributed Network Protocol (DNP3) for data exchange. The step-by-step procedure for establishing the DNP3-Master service on an open-source distribution platform is outlined, comprehensively explaining the Master setup process. Moreover, sample use case results highlight the capabilities of the DNP3- Master service setup. Results demonstrate the scalability and configurability of the DNP3-Master service, making it adaptable for integration with other relevant applications, thus providing potential opportunities for real-world field trials and real-time assessments.

ADMS↗
DOE OSTI2026

An Autonomous MCP Bridge to Rucio: Enhancing Data Management Accessibility for High Energy Physics

The Rucio Data Management System [1] is an important tool used by High Energy Physics experiments, including those at Fermi National Accelerator Laboratory, to store and manage exabyte-scale scientific datasets. Despite its central role in coordinating data across globally distributed storage sites, Rucio's command line interface (CLI) presents a steep learning curve, and makes it difficult for scientists to navigate through. To solve this issue, a containerized Model Context Protocol (MCP) [2] server was built that connects Large Language Models directly to Rucio, allowing AI agents to handle data tasks by using simple, natural language rather than memorized terminal commands. The core engineering focus of this project was moving the server away from slow terminal commands that require text parsing and replacing them with a native Python Client API toolset and a planned REST API framework. Moving to the Python API handles data operations directly in memory, which helps clear up formatting errors, provides the AI with clean, structured JSON data and speeds up tool execution. To prove that the system actually works, a benchmarking pipeline was also built with various questions to test the AI across four different model configurations. The questions included finding data scopes, tracking down specific datasets, and checking replication rules. Through benchmarking, early runs showed that with raw terminal text, the model would get confused and stuck, whereas switching to the Python API to feed the AI clean, structured data yielded massive improvement. By creating an intelligent and autonomous bridge to a storage network, this project shows how AI can be implemented in scientific data management, which ultimately helps scientists at Fermilab spend less time sorting through data and more time focusing on their experiments and analysis.

Akella, Kashyap [William Rainey Harper Coll.]↗
DOE OSTI2025

Integrated Distribution Planning

The contemporary distribution planning landscape is comprised of an increasing number of factors that require integration into the engineering of the modern electric grid. Expectations for electric utilities to accommodate heightened awareness of stakeholders' interest in things like decarbonization, resilience and equity are growing. As these interests are formed into objectives, many jurisdictions will experience increasing levels of load modifying technologies like DER, building and industrial electrification and electric vehicles which prove not only to challenge the capabilities of the grid; but the processes by which planning for it is traditionally done. Other related factors that strain the conventional distribution planning mold are the swelling amount and sources of data associated with these technologies and the need it creates for improved capabilities in the processes and tools that manage it. As the complexity of the distribution system expands, so will the distribution system's effects on the transmission and generation systems that it is a part of. Forecasting distribution system load and DER are examples of areas where this complexity will manifest, and harmonizing distribution forecasting with transmission and generation forecasting requires higher amounts of intentionality as these typically separate processes become a solitary one. Of course, core activities do not cease as a utility begins to integrate these other factors, and in this webinar we explore specifics of how distribution planning can be expected to evolve as progress towards Integrated Distribution System Planning is made.

24 POWER TRANSMISSION AND DISTRIBUTION↗