Search NASA⌕ Search

SEARCH · Search NASA

Results for “Data Distribution Service”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 127 records · Page 7

Development of Distributed Microgrid Controllers Using HELICS [SWR-21-94]

Development of Distributed Microgrid Controllers Using HELICS uses HELICS framework develop distributed microgrid controllers (DMCs) for coordinated grid service. Development of Distributed Microgrid Controllers Using HELICS includes thirteen agents, including OpenDSS agent, data manager agent, and eleven DMC agents. Development of Distributed Microgrid Controllers Using HELICS can be used as a standard framework to develop fully distributed control solutions that dispatches simulated DERs in simulation platform.

Wang, Jing↗

A Network-Aware Distributed Energy Resource Aggregation Framework for Flexible, Cost-Optimal, and Resilient Operation

To efficiently use the ubiquitous behind-the-meter distributed energy resources (DERs) in distribution systems for providing grid services, this paper presents a hierarchical control framework for DER optimal aggregation and control. We first develop a convex optimization model to evaluate the DER flexibility, and then use a convex model-predictive-control based approach to dispatch those DERs. The hierarchical control framework consists of a utility controller, community aggregators and multiple home energy management systems. The flexibility of the DERs is evaluated by each controller in the hierarchy such that the resultant flexibility is feasible given its operational domain. Based on the determined flexibility, the hierarchical controllers then compute optimal setpoints for the DERs to help the distribution system regulate node voltages and provide other distribution grid services. Numerical simulations performed on a model of a real distribution feeder in Colorado, using actual DER data in a residential community demonstrate that the proposed approach can effectively alleviate voltage issues and support resilient operation.

24 POWER TRANSMISSION AND DISTRIBUTION↗

Coordinating an operational data distribution network for CMIP6 data

The distribution of data contributed to the Coupled Model Intercomparison Project Phase 6 (CMIP6) is via the Earth System Grid Federation (ESGF). The ESGF is a network of internationally distributed sites that together work as a federated data archive. Data records from climate modelling institutes are published to the ESGF and then shared around the world. It is anticipated that CMIP6 will produce approximately 20 PB of data to be published and distributed via the ESGF. In addition to this large volume of data a number of value-added CMIP6 services are required to interact with the ESGF; for example the citation and errata services both interact with the ESGF but are not a core part of its infrastructure. With a number of interacting services and a large volume of data anticipated for CMIP6, the CMIP Data Node Operations Team (CDNOT) was formed. The CDNOT coordinated and implemented a series of CMIP6 preparation data challenges to test all the interacting components in the ESGF CMIP6 software ecosystem. This ensured that when CMIP6 data were released they could be reliably distributed.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗

Diaspora: Resilience-Enabling Services for Real-Time Distributed Workflows

The need for real-time processing to enable automated decision making and experimental steering has driven a shift from high-performance computing workflows on a centralized system to a distributed approach that integrates remote data sources, edge devices, and diverse compute facilities. Under this paradigm, data can be processed close to the source where it is generated, thus reducing latency and bandwidth usage. System resilience is thus a key challenge, requiring distributed workflows to survive component failures and to meet stringent quality-of-service requirements, which results in the need to mitigate anomalies such as congestion and low availability of resources. To address these challenges, we propose Diaspora, a unified resilience framework that is inspired by event-driven communication patterns used in public clouds. Specifically, we propose an event fabric that extends across sites, facilities, and computations to provide timely, reliable, and accurate information about data, application, and resource status. On top of the event fabric, we build resilience-enabling services that combine QoS-aware data streaming, resilient data views, resilient compute and data resources, and anomaly detection and prediction, all of which collectively enhance workflow resilience for these scientific cases.

Rao, Nageswara↗

Daymet: Daily Surface Weather Data on a 1-km Grid for North America, Version 2.

More information: http://daymet.ornl.gov Presenter: Ranjeet Devarakonda Environmental Sciences Division Oak Ridge National Laboratory (ORNL) Daymet: Daily Surface Weather Data and Climatological Summaries provides gridded estimates of daily weather parameters for North America, including daily continuous surfaces of minimum and maximum temperature, precipitation occurrence and amount, humidity, shortwave radiation, snow water equivalent, and day length. The current data product (Version 2) covers the period January 1, 1980 to December 31, 2013 [1]. The prior product (Version 1) only covered from 1980-2008. Data are available on a daily time step at a 1-km x 1-km spatial resolution in Lambert Conformal Conic projection with a spatial extent that covers the conterminous United States, Mexico, and Southern Canada as meteorological station density allows. Daymet data can be downloaded from 1) the ORNL Distributed Active Archive Center (DAAC) search and order tools (http://daac.ornl.gov/cgi-bin/cart/add2cart.pl?add=1219) or directly from the DAAC FTP site (http://daac.ornl.gov/cgi-bin/dsviewer.pl?ds_id=1219) and 2) the Single Pixel Tool [2] and THREDDS (Thematic Real-time Environmental Data Services) Data Server [3]. The Single Pixel Data Extraction Tool allows users to enter a single geographic point by latitude and longitude in decimal degrees. A routine is executed that translates the (lon, lat) coordinates into projected Daymet (x,y) coordinates. These coordinates are used to access the Daymet database of daily-interpolated surface weather variables. Daily data from the nearest 1 km x 1 km Daymet grid cell are extracted from the database and formatted as a table with one column for each Daymet variable and one row for each day. All daily data for selected years are returned as a single (long) table, formatted for display in the browser window. At the top of this table is a link to the same data in a simple comma-separated text format, suitable for import into a spreadsheet or other data analysis software. The Single Pixel Data Extraction Tool also provides the option to download multiple coordinates programmatically. A multiple extractor script is freely available to download at http://daymet.ornl.gov/files/daymet.zip. The ORNL DAAC s THREDDS data server (TDS) provides customized visualization and access to Daymet time series of North American mosaics. Users can subset and download Daymet data via a variety of community standards, including OPeNDAP, NetCDF Subset service, and Open Geospatial Consortium (OGC) Web Map/Coverage Service. The ORNL DAAC TDS also exposes Daymet metadata through its ncISO service to facilitate harvesting Daymet metadata records into 3rd party catalogs. References: [1] Thornton, P.E., M.M. Thornton, B.W. Mayer, N. Wilhelmi, Y. Wei, R. Devarakonda, and R.B. Cook. 2014. Daymet: Daily Surface Weather Data on a 1-km Grid for North America, Version 2. Data set. Available on-line [http://daac.ornl.gov] from Oak Ridge National Laboratory Distributed Active Archive Center, Oak Ridge, Tennessee, USA. [2] Devarakonda R., et al. 2012. Daymet: Single Pixel Data Extraction Tool. Available on-line [http://daymet.ornl.go/singlepixel.html]. [3] Wei Y., et al. 2014. Daymet: Thematic Real-time Environmental Data Services. Available on-line [http://daymet.ornl.gov/thredds_tiles.html].

Thornton, Peter E↗

Sequential Perturbation-based FDIA Detection using DERs for Unbalanced Distribution System

The power distribution system with its reliance on automated monitoring and control infrastructure makes it a complicated cyber-physical system and vulnerable to various cyber-attacks. This paper proposes a moving target defense inspired false data injection attack (FDIA) detection mechanism for an unbalanced distribution system. In the proposed grid diagnosis service framework, the distribution system operator judiciously manipulates power outputs of a subset of inverter-based distribution energy resources (DERs) to create secret low magnitude voltage perturbations which are inconsequential to the normal operation of the grid. A mixed-integer-linear-programming algorithm is developed to select the optimal set of DERs that can create a voltage perturbation signal of the required magnitude at each sensor location at a minimum cost. Then, a sequential detector is applied that detects for the FDIA as measurements are received from individual sensors. The performance of the proposed perturbation-based FDIA detection framework is demonstrated via simulation of the IEEE 123 bus test system.

Attack Detection↗

The microeconomics of residential photovoltaics: Tariffs, network operation and maintenance, and ancillary services in distribution-level electricity markets

Here, we develop a microeconomic model of a distribution-level electricity market that takes explicit account of residential photovoltaics (PV) adoption. The model allows us to study the consequences of most tariffs on PV adoption and the consequences of increased residential PV adoption under the assumption of economic sustainability for electric utilities. We also validated the model using U.S. data and extend it to consider different pricing schemes for operation and maintenance costs of the distribution network and for ancillary services. Results show that net metering promotes more environmental benefits and social welfare than other tariffs. But, if costs to operate the distribution network increase, net metering will amplify the unequal distribution of surplus among households. In conclusion, maintaining the economic sustainability of electric utilities under net metering may become extremely difficult unless the uneven distribution of surplus is legitimated by environmental benefits.

14 SOLAR ENERGY↗

Integration of end-user Cloud storage for CMS analysis

End-user Cloud storage is increasing rapidly in popularity in research communities thanks to the collaboration capabilities it offers, namely synchronisation and sharing. CERN IT has implemented a model of such storage named, CERNBox, integrated with the CERN AuthN and AuthZ services. To exploit the use of the end-user Cloud storage for the distributed data analysis activity, the CMS experiment has started the integration of CERNBox as a Grid resource. This will allow CMS users to make use of their own storage in the Cloud for their analysis activities as well as to benefit from synchronisation and sharing capabilities to achieve results faster and more effectively. It will provide an integration model of Cloud storages in the Grid, which is implemented and commissioned over the world’s largest computing Grid infrastructure, Worldwide LHC Computing Grid (WLCG). In this paper, we present the integration strategy and infrastructure changes needed in order to transparently integrate end-user Cloud storage with the CMS distributed computing model. We describe the new challenges faced in data management between Grid and Cloud and how they were addressed, along with details of the support for Cloud storage recently introduced into the WLCG data movement middleware, FTS3. Finally, the commissioning experience of CERNBox for the distributed data analysis activity is also presented.

97 MATHEMATICS AND COMPUTING↗

Towards Cross-Facility Workflows Orchestration through Distributed Automation

Modern science relies on end-to-end workflows that incorporate experimental instruments and utilize edge, cloud, or high-performance computing and storage resources. These components are geographically dispersed across various user facilities and interconnected through high-speed networks. In this paper, we present Zambeze, an automated distributed framework designed to facilitate this new class of cross-facility workflows. Utilizing swarm intelligence principles, Zambeze orchestrates science campaigns by managing distributed autonomous agents. These agents can offer a suite of services, including computing, storage, and data management. We demonstrate the feasibility of Zambeze through a real-world application involving electron microscopy, enhanced with Artificial Intelligence capabilities.

Skluzacek, Tyler↗

Nuclear Physics Network Requirements Review Report

The Energy Sciences Network (ESnet) is the Office of Science’s high-performance network user facility, delivering highly reliable data transport capabilities optimized for the requirements of data-intensive science. In essence, ESnet is the circulatory system that enables the U.S. Department of Energy (DOE) science mission by connecting each and every DOE lab and its user facilities. ESnet is funded and stewarded by the Advanced Scientific Computing Research (ASCR) Program and managed and operated by the Scientific Networking Division at Lawrence Berkeley National Laboratory (LBNL). ESnet is widely regarded as a global leader in the research and education networking community. ESnet connects DOE national laboratories, user facilities, and major experiments so scientists can use remote instruments and computing resources as well as share data with collaborators, transfer large data sets, and access distributed data repositories. While ESnet provides network connectivity, it cannot be characterized as an internet service provider as it is specifically built to provide a range of network services that are tailored to meet the unique requirements of DOE’s data-intensive science.

97 MATHEMATICS AND COMPUTING↗

Nuclear Physics Network Requirements Review Report

The Energy Sciences Network (ESnet) is the Office of Science’s high-performance network user facility, delivering highly reliable data transport capabilities optimized for the requirements of data-intensive science. In essence, ESnet is the circulatory system that enables the U.S. Department of Energy (DOE) science mission by connecting each and every DOE lab and its user facilities. ESnet is funded and stewarded by the Advanced Scientific Computing Research (ASCR) Program, and managed and operated by the Scientific Networking Division at Lawrence Berkeley National Laboratory (LBNL). ESnet is widely regarded as a global leader in the research and education networking community. ESnet connects DOE national laboratories, user facilities, and major experiments so scientists can use remote instruments and computing resources as well as share data with collaborators, transfer large data sets, and access distributed data repositories. While ESnet provides network connectivity, it cannot be characterized as an internet service provider as it is specifically built to provide a range of network services that are tailored to meet the unique requirements of DOE’s data-intensive science. In short, ESnet’s mission is to enable and accelerate scientific discovery by delivering unparalleled network infrastructure, capabilities, and tools. ESnet’s vision is summarized by these three points. 1. Scientific progress will be completely unconstrained by the physical location of instruments, people, computational resources, or data. 2. Collaborations at every scale, in every domain, will have the information and tools they need to achieve maximum benefit from scientific facilities, global networks, and emerging network capabilities. 3. ESnet will foster the partnerships and pioneer the technologies necessary to ensure that these transformations occur. In May 2019, ESnet and the Office of Nuclear Physics (NP) of the DOE Office of Science organized an ESnet requirements review of NP-supported activities. Preparation for this event included identification of key stakeholders to the process: program and facility management, research groups, technology providers, and a number of external observers. These individuals were asked to prepare formal case study documents about their relationship to the NP program to build a complete understanding of the current, near-term, and long-term status, expectations, and processes that will support the science going forward. A series of pre-planning meetings better prepared case study authors for this task and facilitated a smooth transition to the in-person review.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗

ESGF Future Architecture Report (V.1.1)

This document reports the findings of the Earth System Grid Federation Future Architecture Face-to-Face Meeting held in the UK 5-7 November 2019, attended by technical representatives from the ESGF partner organisations. ESGF is an International federated archive that provides access to Earth science data. It is fundamentally a distributed file management system with its main function being data dissemination through the provision of services for efficient replication and search. It has been in operation for over ten years and has undergone continuous improvements and enhancements over this time span. The Future Architecture is a major review of the software system including a fundamental reassessment of its requirements with a view to develop a new overall system architecture. This has been initiated in the light of challenges around the maintenance of the existing software legacy and a changing technology landscape and practices in the communities it serves. The work is being carried out under the oversight of the ESGF Executive Committee1,2 and is coordinated activity between ESGF and the European IS-ENES3 project.

54 ENVIRONMENTAL SCIENCES↗

Fault-Tolerant and Elastic Streaming MapReduce with Decentralized Coordination

The MapReduce programming model, due to its simplicity and scalability, has become an essential tool for processing large data volumes in distributed environments. Recent Stream Processing Systems (SPS) extend this model to provide low-latency analysis of high-velocity continuous data streams. However, integrating MapReduce with streaming poses challenges: first, the runtime variations in data characteristics such as data-rates and key-distribution cause resource overload, that inturn leads to fluctuations in the Quality of the Service (QoS); and second, the stateful reducers, whose state depends on the complete tuple history, necessitates efficient fault-recovery mechanisms to maintain the desired QoS in the presence of resource failures. We propose an integrated streaming MapReduce architecture leveraging the concept of consistent hashing to support runtime elasticity along with locality-aware data and state replication to provide efficient load-balancing with low-overhead fault-tolerance and parallel fault-recovery from multiple simultaneous failures. Our evaluation on a private cloud shows up to 2:8 improvement in peak throughput compared to Apache Storm SPS, and a low recovery latency of 700 -1500 ms from multiple failures.

97 MATHEMATICS AND COMPUTING↗

Extending the Remote Control Capabilities in the CMS Detector Control System with Remote Procedure Call Services

The CMS Detector Control System (DCS) is implemented as a large distributed and redundant system, with applications interacting and sharing data in multiple ways. The CMS XML-RPC is a software toolkit implementing the standard Remote Procedure Call (RPC) protocol, using the Extensible Mark-up Language (XML) and a custom lightweight variant using the JavaScript Object Notation (JSON) to model, encode and expose resources through the Hypertext Transfer Protocol (HTTP). The CMS XML-RPC toolkit complies with the standard specification of the XML-RPC protocol that allows system developers to build collaborative software architectures with self-contained and reusable logic, and with encapsulation of well-defined processes. The implementation of this protocol introduces not only a powerful communication method to operate and exchange data with web-based applications, but also a new programming paradigm to design service-oriented software architectures within the CMS DCS domain. This paper presents details of the CMS XML-RPC implementation in WinCC Open Architecture (OA) Control Language using an object-oriented approach.

43 PARTICLE ACCELERATORS↗

Proposed Application for an Entity Component System in an Energy Services Interface

An Entity Component System is a data-oriented architecture originally developed to streamline video game performance. Despite being quite new, Entity Component Systems are relatively well established within the video game industry due to the cutting edge nature of research into performance, especially around graphics. However, Entity Component Systems have not been widely examined or adopted outside of that industry. We propose adopting an Entity Component Systems framework to serve the needs of an Energy Service Interfaces. We examine the needs of an Energy Service Interface, give an overview of open-source Entity Component Systems (ECSs) libraries, examine some preliminary performance results for ECSs, and explore the traditional approach to fulfilling the needs of an Energy Service Interface (ESI) with database architectures.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

Enabling NVM for Data-Intensive Scientific Services

Specialized, transient data services are playing an increasingly prominent role in data-intensive scientific computing. These services offer flexible, on-demand pairing of applications with storage hardware using semantics that are optimized for the problem domain. Concurrent with this trend, upcoming scientific computing and big data systems will be deployed with emerging NVM technology to achieve the highest possible price/productivity ratio. Clearly, therefore, we must develop techniques to facilitate the confluence of specialized data services and NVM technology. In this work we explore how to enable the composition of NVM resources within transient distributed services while still retaining their essential performance characteristics. Our approach involves eschewing the conventional distributed file system model and instead projecting NVM devices as remote microservices that leverage user-level threads, RPC services, RMA-enabled network transports, and persistent memory libraries in order to maximize performance. We describe a prototype system that incorporates these concepts, evaluate its performance for key workloads on an exemplar system, and discuss how the system can be leveraged as a component of future data-intensive architectures.

Carns, Philip↗

Production Operations Management System (POMS) for Fermilab Experiments

The Production Operations Management System (POMS) software allows production teams and analysis groups across multiple Fermilab experiments to launch, modify and monitor large scale campaigns of related Monte Carlo or data processing jobs, and currently manages the majority of production computing of experiments at Fermilab.POMS provides a web service interface that enables automated job submission on distributed resources according to customers’ requests, as well as subsequent monitoring of those submissions as well as recovery of failed submissions, debugging and record keeping.POMS interfaces with existing HEP data access, processing, movement and monitoring tools at Fermilab, including Jobsub (for job submission), SAM (for data management), and Landscape (for monitoring), and in combination with them handles the creation of intermediate datasets of files for multistage campaigns.POMS visualizes these campaigns with an interactive GUI interface. An important feature of POMS is a one-to-one connection between the GUI campaign visualizer and a text based representation of the complete campaign configuration. An extensible library of template campaign configurations is available. The templates are user modifiable and map cleanly to and from the GUI description of the campaign.This paper will discuss the current usage, history, and some technical details of POMS, as well as planned future extensions, particularly interfacing with Rucio as well as SAM for file movement.

Mengel, Marc↗

Blockchain for Fault-Tolerant Grid Operations

Distribution system fault/failure has a direct impact on customers. Distribution systems are not designed with the contingency of one or more elements failing as the transmission system, however, distribution systems are more likely to have faults in comparison to transmission and generation systems. Most of the distribution systems have a radial design with the protection system assumption of unidirectional power flow. The radial design results in the disconnection of customers for any given component failure. The rapid deployment of distributed energy resources conflicts with the assumption of unidirectional power flow. To maintain/improve the distribution system fault tolerance for a grid with high penetration of distributed energy resources requires improvement. Blockchain can add value to improve fault-tolerant grid operations. That can be achieved using blockchain’s core features of distributed consensus-based decision-making process and immutability. In the process of preparing for an event or system restoration, there is a need for reliable data and system situational awareness. The system situational awareness enables accurate planning for possible scenarios and system restoration after an event, an area where blockchain’s distributed architecture can help. The value of blockchain for a fault-tolerant grid operation is elaborated with three use cases: 1) fault location, isolation, and service restoration; 2) topology identification; and 3) data configuration. These use cases demonstrate how blockchain-based architecture can facilitate bridging the vulnerabilities of the existing implementation.

24 POWER TRANSMISSION AND DISTRIBUTION↗