Search NASA⌕ Search

SEARCH · Search NASA

Results for “distributed services”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 73 records · Page 4
All sourcesNASA NTRSDOE OSTI
DOE OSTI2020

Customizable Scale-Out Key-Value Stores

Enterprise KV stores are often not well suited for HPC applications, and thus cumbersome end-to-end KV design customization is required to meet the needs of modern HPC applications. To this end, in this article we present bespoKV, an adaptive, extensible, and scale-out KV store framework. bespoKV decouples the KV store design into the control plane for distributed management and the data plane for local data store. For the control plane, bespoKVprovides pre-built modules, called controlets, supporting common distributed functionalities (e.g., replication, consistency, and topology) and their various combinations. This decoupling allows bespoKV to take a user-provided single-server KV store, called a datalet, and transparently enables a scalable and fault-tolerant distributed KV store service. The resulting distributed stores are also adaptive to consistency or topology requirement changes and can be easily extended for new types of services. Such specializations enable innovative uses of KV stores in HPC applications, especially for emerging applications that utilize KV-friendly workloads. We evaluate bespoKV in a local testbed as well as in a public cloud settings. Experiments show that bespoKV-enabled distributed KV stores scale horizontally to a large number of nodes, and performs comparably and sometimes 1.2× to 2.6× better than the state-of-the-art systems.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2024

ATLAS Data Analysis using a Parallel Workflow on Distributed Cloud-based Services with GPUs

A new type of parallel workflow is developed for the ATLAS experiment at the Large Hadron Collider, that makes use of distributed computing combined with a cloud-based infrastructure. This has been developed for a specific type of analysis using ATLAS data, one popularly referred to as Simulation-Based Inference (SBI). The JAX library is used for the parts of the workflow to compute gradients as well as accelerate program execution using just-in-time compilation, which becomes essential in a full SBI analysis and can also offer significant speed-ups in more traditional types of analysis.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗
DOE OSTI2024

Experimental Analysis of Distribution Network Voltage Regulation Using Smart Inverters

Smart inverters (SIs) have demonstrated their potential to provide grid services for both transmission and distribution systems. One of these grid services, distribution network voltage regulation by SIs, has the potential to improve network voltage regulation through controlling the reactive and active power output of the SIs. Voltage regulation by SIs will be distributed and might be better suited to controlling local conditions to complement traditional voltage-regulating assets, e.g., tap-changing transformers, capacitor banks, and line voltage regulators. There is a gap in the literature on comparing the SI response characteristics when the SIs are controlled by a local controller or external control signals. This paper presents an experimental study to characterize SI reactive power regulation responses to two different control methods: autonomous control and remote dispatch. We found that SI reactive power regulation responses exhibit important differences between these methods in terms of delays and ramp rate. Finally, power-hardware-in-the-loop (PHIL) tests were conducted to evaluate the performance of these two methods. The PHIL test results show that the SI response characteristics for autonomous control and remote dispatch need to be considered when planning for distribution network voltage regulation using SIs.

autonomous control↗
DOE OSTI2020

Large-Scale Adoption of Self-Synchronized Universal Droop Controller-Based Inverters to Enable Ancillary Services for Different Modes of Distribution System Operations

This work proposes the large-scale adoption of self-synchronized universal droop controller (SUDC)-based inverters to enable ancillary services for different modes of distribution system operations. The IEEE 123 bus system was modeled on a real-time simulator to study the performance of large-scale adoption of SUDC inverters in a distribution system. The resulting data collected shows that the voltage and the frequency were regulated within ranges, such as less than 5% for voltage and less than 0.5% for frequency, under different load variations and grid operations. Also, the black start was achieved within 0.4 s without any voltage overshoot. Through the simulation and validation on a small microgrid and the IEEE 123 bus distribution system, it can be concluded that the SUDC was successfully adopted to regulate the voltage and the frequency within the given ranges, and black start achieved within 1 s without voltage overshoot for different modes of distribution system operations.

14 SOLAR ENERGY↗
DOE OSTI2022

Automatic DDoS Attack Detection on SDNs: Preprint

Denial of Service (DoS) and Distributed Denial of Service (DDoS) attacks pose a serious threat to computing networks - especially to critical systems within the U.S. electrical grid. As attack mechanisms have increased in complexity and variety, more sophisticated detection mechanisms have become necessary to ensure network security. This paper explores the use of artificial intelligence to automate the process of detection and mitigation of DoS and DDoS attacks within the framework of Software-Defined Networking (SDN), to a high degree. Machine learning algorithms are trained to recognize DoS and DDoS attacks and are deployed in real-time to mitigate malicious network traffic. The results show a well-tuned gradient-boosted decision tree detecting DoS and DDoS attacks, as well as initial successful mitigation of attacks within an SDN framework.

cyber detection↗
DOE OSTI2020

Mochi: Composing Data Services for High-Performance Computing Environments

Technology enhancements and the growing breadth of application workflows running on high-performance computing (HPC) platforms drive the development of new data services that provide high performance on these new platforms, provide capable and productive interfaces and abstractions for a variety of applications, and are readily adapted when new technologies are deployed. The Mochi framework enables composition of specialized distributed data services from a collection of connectable modules and subservices. Rather than forcing all applications to use a one-size-fits-all data staging and I/O software configuration, Mochi allows each application to use a data service specialized to its needs and access patterns. This paper introduces the Mochi framework and methodology. The Mochi core components and microservices are described. Examples of the application of the Mochi methodology to the development of four specialized services are detailed. Finally, a performance evaluation of a Mochi core component, a Mochi microservice, and a composed service providing an object model is performed. The paper concludes by positioning Mochi relative to related work in the HPC space and indicating directions for future work.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2024

A Managed Tokens Service for Securely Keeping and Distributing Grid Tokens

Fermilab is transitioning authentication and authorization for grid operations to using bearer tokens based on the WLCG Common JWT (JSON Web Token) Profile. One of the functionalities that Fermilab experimenters rely on is the ability to automate batch job submission, which in turn depends on the ability to securely refresh and distribute the necessary credentials to experiment job submit points. Thus, with the transition to using tokens for grid operations, we needed to create a service that would obtain, refresh, and distribute tokens for experimenters’ use. This service would avoid the need for experimenters to be experts in obtaining their own tokens and would better protect the most sensitive long-lived credentials. Further, the service needed to be widely scalable, as we are currently keeping credentials active for approximately 15 experiments, each with 1-3 different credentials, and distributing those credentials to 2-20 submit points per experiment, with those numbers steadily increasing. To address these issues, we created and deployed a Managed Tokens service. The service is written in Go, taking advantage of that language’s native concurrency primitives to easily be able to scale operations as we onboard experiments. The service uses as its first credentials a set of kerberos keytabs, stored on the same secure machine that the Managed Tokens service runs on. These kerberos credentials allow the service to use htgettoken via condor_vault_storer to store vault tokens in the HTCondor credential managers (credds) that run on the batch system scheduler machines (HTCondor schedds); as well as downloading a local, shorter-lived copy of the vault token. The kerberos credentials are then also used to distribute copies of the locally-stored vault tokens to experiment submit points. When experimenters schedule jobs to be submitted, these distributed vault tokens are used to access a Hashicorp Vault instance (run separately from the Managed Tokens service), and previously-stored refresh tokens there are used to obtain the bearer token that is submitted with the job. We will discuss here the design of the Managed Tokens service, including elaborating on certain choices we made with regards to concurrent operations, configuration, monitoring, and deployment.

Bhat, Shreyas↗
DOE OSTI2021

A Load Switching Group based Feeder-level Microgrid Energy Management Algorithm for Service Restoration in Power Distribution System

This paper presents a load switching group based energy management system (LSG-EMS) for operating microgrids on a distribution feeder powered by one or multiple grid-forming distributed energy resources. Loads on a distribution feeder are divided into load switching groups that can be remotely switched on and off. The LSG-EMS algorithm, formulated as a mixed-integer linear programming (MILP) problem, has an objective function of maximizing the served loads while minimizing the total number of switching actions. A new set of topology constraints are developed for allowing multiple microgrids to be formed on the feeder and selecting the optimal supply path. Customer comfort is accounted for by maximizing the supply duration in the customer preferred service period and enforcing a minimum service duration. The proposed method is demonstrated on a modified IEEE 33-bus system using actual customer data. Simulation results show that the LSG-EMS successfully coordinates multiple grid-forming sources by selecting an optimal supply topology that maximizes the supply period of both the critical and noncritical loads while minimizing customer service interruptions in the service restoration process.

Hu, Rongxing↗
DOE OSTI2025

A Managed Tokens Service for Securely Keeping and Distributing Grid Tokens

Fermilab is transitioning authentication and authorization for grid operations to using bearer tokens based on the WLCG Common JWT (JSON Web Token) Profile. One of the functionalities that Fermilab experimenters rely on is the ability to automate batch job submission, which in turn depends on the ability to securely refresh and distribute the necessary credentials to experiment job submit points. Thus, with the transition to using tokens for grid operations, we needed to create a service that would obtain, refresh, and distribute tokens for experimenters' use. This service would avoid the need for experimenters to be experts in obtaining their own tokens and would better protect the most sensitive long-lived credentials. Further, the service needed to be widely scalable, as Fermilab hosts many experiments, each of which would need their own credentials. To address these issues, we created and deployed a Managed Tokens Service. The service is written in Go, taking advantage of that language's native concurrency primitives to easily be able to scale operations as we onboard experiments. The service uses as its first credentials a set of kerberos keytabs, stored on the same secure machine that the Managed Tokens service runs on. These kerberos credentials allow the service to use htgettoken via condor_vault_storer to store vault tokens in the HTCondor credential managers (credds) that run on the batch system scheduler machines (HTCondor schedds); as well as downloading a local, shorter-lived copy of the vault token. The kerberos credentials are then also used to distribute copies of the locally-stored vault tokens to experiment submit points.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗
DOE OSTI2021

Artificial Intelligence for Energy Systems Cybersecurity

Artificial intelligence and machine learning systems have the potential to influence the future design and implementation of cybersecurity systems for the power grid. These systems may enhance the overall operation of the power system by leveraging and making sense of massive amounts of data. However, we must also understand how AI/ML will need to be protected from cyber threat actors. We discuss the existing insights the NREL team has developed using AI/ML systems and then present resources including ESIF and the Cyber Energy Emulation Platform that can be used to generate training data and insights. We end by offering suggestions on priority research paths for AI in cybersecurity.

artificial intelligence↗
DOE OSTI2023

HEPnOS: a Specialized Data Service for High Energy Physics Analysis

In this paper, we present HEPnOS, a distributed data service for managing data produced by high-energy physics (HEP) experiments. Using HEPnOS, HEP applications can use HPC resources more effciently than traditional fle-based applications. The fle-based model leads to a rigid, chunk-based allocation of computational resources and limits the number of cores that can be used concurrently by an HEP application. The fundamental problem is that organizing domain-specifc data into fles inadvertently introduces a single, artifcial, confated tuning parameter that puts key optimization goals into confict: larger fle sizes reduce metadata overhead and thus improve I/O effciency, but smaller fle sizes provide more opportunity for workfow parallelism and load balancing. In this work, we introduce a domain-specifc data service that decouples that constraint so that data can be accessed and processed in its natural granularity while still maintaining I/O effciency. By removing the constraints introduced by fle handling we are able to obtain better scaling and make effcient use of more cores for processing a fxed-sized data sample. We demonstrate the improved scalability by using an application developed in the fle-based paradigm and comparing it to a version modifed to use HEPnOS.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗
DOE OSTI2026

Operational Evolution of FTS3: A DevOps Driven Approach to Elastic Operations

The File Transfer Service (FTS3) is a distributed data movement service developed at CERN and widely used to transfer data across the Worldwide LHC Computing Grid (WLCG). At Fermilab, FTS3 supports data transfers for multiple experiments, including Intensity Frontier experiments such as DUNE, enabling reliable data movement between WebDAV endpoints in Europe and the Americas.​ At CHEP 2021, we reported on the initial containerized deployment of FTS3 on OKD, the community Kubernetes distribution of Red Hat OpenShift. In this work, we present the subsequent evolution of this deployment, focusing on new operational capabilities introduced to improve scalability, robustness, and long-term maintainability.​ We describe the adoption of more secure and reproducible container build workflows, the integration of DevOps-driven operational practices, and enhancements in monitoring and automation. A key new result is the introduction of horizontal scaling and elastic resource management, allowing FTS3 components to dynamically adapt to workload variations while maintaining service reliability. We also discuss improvements in fault tolerance and operational procedures derived from production experience.​ Finally, we summarize lessons learned from operating FTS3 as a Kubernetes-native service and outline how these developments have improved the resilience and efficiency of data movement operations at Fermilab.

Munoz Flores, Victor Leopoldo [Fermilab]↗
DOE OSTI2024

Diaspora: Resilience-Enabling Services for Real-Time Distributed Workflows

The need for real-time processing to enable automated decision making and experimental steering has driven a shift from high-performance computing workflows on a centralized system to a distributed approach that integrates remote data sources, edge devices, and diverse compute facilities. Under this paradigm, data can be processed close to the source where it is generated, thus reducing latency and bandwidth usage. System resilience is thus a key challenge, requiring distributed workflows to survive component failures and to meet stringent quality-of-service requirements, which results in the need to mitigate anomalies such as congestion and low availability of resources. To address these challenges, we propose Diaspora, a unified resilience framework that is inspired by event-driven communication patterns used in public clouds. Specifically, we propose an event fabric that extends across sites, facilities, and computations to provide timely, reliable, and accurate information about data, application, and resource status. On top of the event fabric, we build resilience-enabling services that combine QoS-aware data streaming, resilient data views, resilient compute and data resources, and anomaly detection and prediction, all of which collectively enhance workflow resilience for these scientific cases.

Rao, Nageswara↗
DOE OSTI2024

Utilizing Distributed Heterogeneous Computing with PanDA in ATLAS

In recent years, advanced and complex analysis workflows have gained increasing importance in the ATLAS experiment at CERN, one of the large scientific experiments at LHC. Support for such workflows has allowed users to exploit remote computing resources and service providers distributed worldwide, overcoming limitations on local resources and services. The spectrum of computing options keeps increasing across the Worldwide LHC Computing Grid (WLCG), volunteer computing, high-performance computing, commercial clouds, and emerging service levels like Platform-as-a-Service (PaaS), Container-as-a-Service (CaaS) and Function-as-a-Service (FaaS), each one providing new advantages and constraints. Users can significantly benefit from these providers, but at the same time, it is cumbersome to deal with multiple providers, even in a single analysis workflow with fine-grained requirements coming from their applications’ nature and characteristics. In this paper, we will first highlight issues in geographically-distributed heterogeneous computing, such as the insulation of users from the complexities of dealing with remote providers, smart workload routing, complex resource provisioning, seamless execution of advanced workflows, workflow description, pseudointeractive analysis, and integration of PaaS, CaaS, and FaaS providers. We will also outline solutions developed in ATLAS with the Production and Distributed Analysis (PanDA) system and future challenges for LHC Run4.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2023

Valuation of Distributed Wind Turbines Providing Multiple Market Services

The role of wind turbines has traditionally been limited to providing energy capacity to the grid, but the availability of smart inverters and recent regulatory changes provide the technical and policy capability for wind turbines to also provide ancillary services. However, in contrast to the technical and policy aspects, the valuation of distributed wind turbines providing such services has not been thoroughly studied. This paper presents an optimal market-participation method for distributed wind turbines and valuates different strategies in California Independent System Operator’s balancing area. The services include energy capacity, regulation up and down, and reserves. An optimization problem is formulated to determine optimal power output for each service and demonstrated using historical data for one complete year. The revenues from multiple services are quantified, and a sensitivity analysis is performed to relate market prices with revenues. It is found that the optimal strategy generates 6% more revenue compared to the revenue from participating in the energy market only. Also, the reduced energy prices in future scenarios increase the relative importance of market participation in ancillary services.

Bhatti, Bilal Ahmad↗
DOE OSTI2021

Large-Scale Adoption of Self Synchronized Universal Droop Controller-Based Inverters to Enable Ancillary Services for Different Modes of Distribution System Operations

This poster presents the results of using the self-synchronized universal droop controller to enable solar power to provide ancillary services so that the use of solar power can be increased. A 300-node system using the IEEE 123 bus test feeder are modeled on a real time power system simulator with the results presented.

Arce, Laura↗