Search NASA⌕ Search

SEARCH · Search NASA

Results for “data grid middleware”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records

PNNL Data-Intensive Computing for a Smarter Energy Grid

The Middleware for Data-Intensive Computing (MeDICi) Integration Framework, an integrated platform to solve data analysis and processing needs, supports PNNL research on the U.S. electric power grid. MeDICi is enabling development of visualizations of grid operations and vulnerabilities, with goal of near real-time analysis to aid operators in preventing and mitigating grid failures.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗

Scalable Real Time Data Management for Smart Grid

This paper presents GridMW, a scalable and reliable data middleware for smart grids. Smart grids promise to improve the efficiency of power grid systems and reduce green house emissions through incorporating power generation from renewable sources and shaping demand to match the supply. As a result, power grid systems will become much more dynamic and require constant adjustments, which requires analysis and decision making applications to improve the efficiency and reliability of smart grid systems.

54 ENVIRONMENTAL SCIENCES↗

Big PanDa Workflow Management on Titan for High Energy and Nuclear Physics and for Future Extreme Scale Scientific Application

Over a three year period, from 2016-2019, this project demonstrated the scientific benefits of integrating the Titan supercomputer at Oak Ridge Leadership Computing Facility into traditional high throughput grid based distributed computing systems managed by PanDA, the workflow management system used for the execution of all distributed computing applications by the ATLAS experiment at the Large Hadron Collider. PanDA manages millions of batch jobs daily at hundreds of clusters worldwide on request by thousands of physicist users, and processes more than an exabyte of data annually using grid middleware. High levels of operational use of Titan was sustained by PanDA in order to meet the physics goals of ATLAS. The success of this project led to the use of other supercomputers worldwide by ATLAS, and to the adoption of PanDA by other experiments and other scientists. Multiple innovative operational and computer science research goals were achieved supporting the use of supercomputers for scientific domains with large scale distributed data and distributed processing needs.

97 MATHEMATICS AND COMPUTING↗

Distributed Monte Carlo production for D0

The D0 collaboration uses a variety of resources on four continents to pursue a strategy of flexibility and automation in the generation of simulation data. This strategy provides a resilient and opportunistic system which ensures an adequate and timely supply of simulation data to support D0's physics analyses. A mixture of facilities, dedicated and opportunistic, specialized and generic, large and small, grid job enabled and not, are used to provide a production system that has adapted to newly developing technologies. This strategy has increased the event production rate by a factor of seven and the data production rate by a factor of ten in the last three years despite diminishing manpower. Common to all production facilities is the SAM (Sequential Access to Metadata) data-grid. Job submission to the grid uses SAMGrid middleware which may forward jobs to the OSG, the WLCG, or native SAMGrid sites. The distributed computing and data handling system used by D0 will be described and the results of MC production since the deployment of grid technologies will be presented.

99 GENERAL AND MISCELLANEOUS↗

A Distributed Middleware Architecture for Attack-Resilient Communications in Smart Grids

Distributed Energy Resources (DERs) are being increasingly accepted as an excellent complement to traditional energy sources in smart grids. As most of these generators are geographically dispersed, dedicated communications investments for every generator are capital cost prohibitive. Real-time distributed communications middleware, which supervises, organizes and schedules tremendous amounts of data traffic in smart grids with high penetrations of DERs, allows for the use of existing network infrastructure. In this paper, we propose a distributed attack-resilient middleware architecture that detects and mitigates the congestion attacks by exploiting the Quality of Experience (QoE) measures to complement the conventional Quality of Service (QoS) information to detect and mitigate the congestion attacks effectively. The simulation results illustrate the efficiency of our proposed communications middleware architecture.

24 POWER TRANSMISSION AND DISTRIBUTION↗

Optimization and Control Theory for Smart (Power) Grids [Slides]

The national interest in a broad class of problems loosely characterized as the smart grid is rapidly growing. At its core, the smart grid is the modernization of our existing electric power systems through improvements in efficiency, increased robustness and resiliency, integration of time-variable renewable generation, and reduction of environmental impacts. To date, the focus of most commercial entities has been on development of hardware such as smart meters and green technologies. However, with the exception of a handful of academic researchers, there has been limited research on optimization, control and more broadly, the information science required to effectively utilize this hardware. Our proposal targets development of the basic information science that will effectively bridge this gap. Using our expertise in information theory, network analysis, infrastructure analysis and power engineering, we will develop new algorithms for optimal design of the nation's next generation power transmission and distribution system that address the realities and challenges of the future grid. We focus on three core information science challenges in Grid Design, Grid Control and Grid Stability by considering a set of challenging problems. First, we study efficient and robust integration of geographically widespread renewable generation by solving large, cutting-edge optimization problems that incorporate difficult constraints imposed by the transmission of electrical power. Second, we consider two load balancing problems in the low-voltage distribution grid: development of robust and efficient distributed control techniques and algorithms based on queuing theory. These tools will enable high penetration of small-scale distributed generation and plug-in hybrid vehicles with limited or no communications. We will also develop efficient control algorithms for maintaining an operational distribution grid via switching of redundant lines. Third, we will develop a preventive toolbox capable of detecting instabilities and failures prior to their occurrence. Damaging failures and cascading outages, which can result in billions of dollars worth of damage, have substantial economic consequences. We will focus on estimating the probability of these outages, identifying their signatures and precursors, and developing detection algorithms for alerting the transmission grid operator. The new challenges in IS&T for smart grids require coherent advances in analysis and control, stability and reliability metrics, state estimation, data aggregation and assimilation, and communication middleware for the grid as well as understanding consumer response.

24 POWER TRANSMISSION AND DISTRIBUTION↗

An integrated security framework for GOSS power grid analytics platform

In power grid operations, security is an essential component for any middleware platform. Security protects data against unwanted access as well as cyber attacks. GridOpticsTM Software System (GOSS) is an open source power grid analytics platform that facilitates ease of access between applications and data sources and promotes development of advanced analytical applications. GOSS contains an API that abstracts many of the difficulties in connecting to various heterogeneous data sources. A number of applications and data sources have already been implemented to demonstrate functionality and ease of use. A security framework has been implemented which leverages widely accepted, robust JavaTM security tools in a way such that they can be interchanged as needed. This framework supports the complex fine-grained, access control rules identified for the diverse data sources already in GOSS. Performance and reliability are also important considerations in any power grid architecture. An evaluation is done to determine the overhead cost caused by security within GOSS and ensure minimal impact to performance.

Gibson, Tara D.↗

GridOPTICS Software System

GridOPTICS Software System (GOSS) is a middleware that facilitates creation of new, modular and flexible operational and planning platforms that can meet the challenges of the next generation power grid. GOSS enables Department of Energy, power system utilities, and vendors to build better tools faster. GOSS makes it possible to integrate Future Power Grid Initiative software products/prototypes into existing power grid software systems, including the PNNL PowerNet and EIOC environments. GOSS is designed to allow power grid applications developed for different underlying software platforms installed in different utilities to communicate with ease. This can be done in compliance with existing security and data sharing policies between the utilities. GOSS not only supports one-to-one data transfer between applications, but also publisher/subscriber scheme. To support interoperability requirements of future EMS, GOSS is designed for CIM compliance. In addition to this, it supports authentication and authorization capabilities to protect the system from cyber threats. In summary, the contributions of the GOSS middleware are as follows: A platform to support future EMS development. A middleware that promotes interoperability between power grid applications. A distributed architecture that separates data sources from power grid applications. Support for data exchange with either one-to-one or publisher/subscriber interfaces. An authentication and authorization scheme for limiting the access to data between utilities.

Akyol, BoraA↗

GridOPTICS Software System

GridOPTICS Software System (GOSS) is a middleware that facilitates creation of new, modular and flexible operational and planning platforms that can meet the challenges of the next generation power grid. GOSS enables Department of Energy, power system utilities, and vendors to build better tools faster. GOSS makes it possible to integrate Future Power Grid Initiative software products/prototypes into existing power grid software systems, including the PNNL PowerNet and EIOC environments. GOSS is designed to allow power grid applications developed for different underlying software platforms installed in different utilities to communicate with ease. This can be done in compliance with existing security and data sharing policies between the utilities. GOSS not only supports one-to-one data transfer between applications, but also publisher/subscriber scheme. To support interoperability requirements of future EMS, GOSS is designed for CIM compliance. In addition to this, it supports authentication and authorization capabilities to protect the system from cyber threats. In summary, the contributions of the GOSS middleware are as follows: A platform to support future EMS development. A middleware that promotes interoperability between power grid applications. A distributed architecture that separates data sources from power grid applications. Support for data exchange with either one-to-one or publisher/subscriber interfaces. An authentication and authorization scheme for limiting the access to data between utilities.

Akyol, Bora A↗

PNNL Future Power Grid Initiative-developed GridOPTICS Software System (GOSS)

The power grid is changing and evolving. One aspect of this change is the growing use of smart meters and other devices, which are producing large volumes of useful data. However, in many cases, the data can’t be translated quickly into actionable guidance to improve grid performance. There's a need for innovative tools. The GridOPTICS(TM) Software System, or GOSS, developed through PNNL's Future Power Grid Initiative, is open source and became publicly available in spring 2014. The value of this middleware is that it easily integrates grid applications with sources of data and facilitates communication between them. Such a capability provides a foundation for developing a range of applications to improve grid management.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

Dynamic Distribution of High-Rate Data Processing from CERN to Remote HPC Data Centers

The prompt reconstruction of the data recorded from the Large Hadron Collider (LHC) detectors has always been addressed by dedicated resources at the CERN Tier-0. Such workloads come in spikes due to the nature of the operation of the accelerator and in special high load occasions experiments have commissioned methods to distribute (spill-over) a fraction of the load to sites outside CERN. The present work demonstrates a new way of supporting the Tier-0 environment by provisioning resources elastically for such spilled-over workflows onto the Piz Daint Supercomputer at CSCS. Furthermore, this is implemented using containers, tuning the existing batch scheduler and reinforcing the scratch file system, while still using standard Grid middleware. ATLAS, CMS and CSCS have jointly run selected prompt data reconstruction on up to several thousand cores on Piz Daint into a shared environment, thereby probing the viability of the CSCS high performance computer site as on demand extension of the CERN Tier-0, which could play a role in addressing the future LHC computing challenges for the high luminosity LHC.

97 MATHEMATICS AND COMPUTING↗

CDF GlideinWMS usage in grid computing of high energy physics

Many members of large science collaborations already have specialized grids available to advance their research in the need of getting more computing resources for data analysis. This has forced the Collider Detector at Fermilab (CDF) collaboration to move beyond the usage of dedicated resources and start exploiting Grid resources. Nowadays, CDF experiment is increasingly relying on glidein-based computing pools for data reconstruction. Especially, Monte Carlo production and user data analysis, serving over 400 users by central analysis farm middleware (CAF) on the top of Condor batch system and CDF Grid infrastructure. Condor is designed as distributed architecture and its glidein mechanism of pilot jobs is ideal for abstracting the Grid computing by making a virtual private computing pool. We would like to present the first production use of the generic pilot-based Workload Management System (glideinWMS), which is an implementation of the pilot mechanism based on the Condor distributed infrastructure. CDF Grid computing uses glideinWMS for its data reconstruction on the FNAL campus Grid, user analysis and Monte Carlo production across Open Science Grid (OSG). We review this computing model and setup used including CDF specific configuration within the glideinWMS system which provides powerful scalability and makes Grid computing working like in a local batch environment with ability to handle more than 10000 running jobs at a time.

99 GENERAL AND MISCELLANEOUS↗

Building the International Lattice Data Grid

We present the International Lattice Data Grid (ILDG), a loosely federated grid-of-grids for sharing data from Lattice Quantum Chromodynamics (LQCD) simulations. The ILDG comprises of metadata, file-format and web-service standards, which can be used to wrap regional data-grid interfaces, allowing seamless access to catalogues and data in a diverse set of collaborating regional grids. We discuss the technological underpinnings of the ILDG, primarily the metadata and the middleware, and offer a critique of its various aspects with the hindsight of the design work and the two years of production.

99 GENERAL AND MISCELLANEOUS↗

SLoG: Large-Scale Logging Middleware for HPC and Big Data Convergence

Cloud developers traditionally rely on purpose-specific services to provide the storage model they need for an application. In contrast, HPC developers have a much more limited choice, typically restricted to a centralized parallel file system for persistent storage. Unfortunately, these systems often offer low performance when subject to highly concurrent, conflicting I/O patterns. This makes difficult the implementation of inherently concurrent data structures such as distributed shared logs. Yet, this data structure is key to applications such as computational steering, data collection from physical sensor grids, or discrete event generators. In this paper we tackle this issue. We present SLoG, shared log middleware providing a shared log abstraction over a parallel file system, designed to circumvent the aforementioned limitations. We evaluate SLoG’s design on up to 100,000 cores of the Theta supercomputer: the results show high append velocity at scale while also providing substantial benefits for other persistent backend storage systems.

big data↗

An Attack-Resilient Middleware Architecture for Grid Integration of Distributed Energy Resources

In recent years, the increasing penetration of Distributed Energy Resources (DERs) has made an impact on the operation of the electric power systems. In the grid integration of DERs, data acquisition systems and communications infrastructure are crucial technologies to maintain system economic efficiency and reliability. Since most of these generators are relatively small, dedicated communications investments for every generator are capital cost prohibitive. Combining real-time attack-resilient communications middleware with Internet of Things (IoTs) technologies allows for the use of existing infrastructure. In our paper, we propose an intelligent communication middleware that utilizes the Quality of Experience (QoE) metrics to complement the conventional Quality of Service (QoS) evaluation. Furthermore, our middleware employs deep learning techniques to detect and defend against congestion attacks. The simulation results illustrate the efficiency of our proposed communications middleware architecture.

24 POWER TRANSMISSION AND DISTRIBUTION↗

Extending the distributed computing infrastructure of the CMS experiment with HPC resources

Particle accelerators are an important tool to study the fundamental properties of elementary particles. Currently the highest energy accelerator is the LHC at CERN, in Geneva, Switzerland. Each of its four major detectors, such as the CMS detector, produces dozens of Petabytes of data per year to be analyzed by a large international collaboration. The processing is carried out on the Worldwide LHC Computing Grid, that spans over more than 170 compute centers around the world and is used by a number of particle physics experiments. Recently the LHC experiments were encouraged to make increasing use of HPC resources. While Grid resources are homogeneous with respect to the used Grid middleware, HPC installations can be very different in their setup. In order to integrate HPC resources into the highly automatized processing setups of the CMS experiment a number of challenges need to be addressed. For processing, access to primary data and metadata as well as access to the software is required. At Grid sites all this is achieved via a number of services that are provided by each center. However at HPC sites many of these capabilities cannot be easily provided and have to be enabled in the user space or enabled by other means. At HPC centers there are often restrictions regarding network access to remote services, which is again a severe limitation. The paper discusses a number of solutions and recent experiences by the CMS experiment to include HPC resources in processing campaigns.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

MeDICi Software Superglue for Data Analysis Pipelines

The Middleware for Data-Intensive Computing (MeDICi) Integration Framework is an integrated middleware platform developed to solve data analysis and processing needs of scientists across many domains. MeDICi is scalable, easily modified, and robust to multiple languages, protocols, and hardware platforms, and in use today by PNNL scientists for bioinformatics, power grid failure analysis, and text analysis.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗

Automated Cache Performance Analysis And Optimization

While there is no lack of performance counter tools for coarse-grained measurement of cache activity, there is a critical lack of tools for relating data layout to cache behavior to application performance. Generally, any nontrivial optimizations are either not done at all, or are done ”by hand” requiring significant time and expertise. To the best of our knowledge no tool available to users measures the latency of memory reference instructions for particular addresses and makes this information available to users in an easy-to-use and intuitive way. In this project, we worked to enable the Open|SpeedShop performance analysis tool to gather memory reference latency information for specific instructions and memory addresses, and to gather and display this information in an easy-to-use and intuitive way to aid performance analysts in identifying problematic data structures in their codes. This tool was primarily designed for use in the supercomputer domain as well as grid, cluster, cloud-based parallel e-commerce, and engineering systems and middleware. Ultimately, we envision a tool to automate optimization of application cache layout and utilization in the Open|SpeedShop performance analysis tool. To commercialize this software, we worked to develop core capabilities for gathering enhanced memory usage performance data from applications and create and apply novel methods for automatic data structure layout optimizations, tailoring the overall approach to support existing supercomputer and cluster programming models and constraints. In this Phase I project, we focused on infrastructure necessary to gather performance data and present it in an intuitive way to users. With the advent of enhanced Precise Event-Based Sampling (PEBS) counters on recent Intel processor architectures and equivalent technology on AMD processors, we are now in a position to access memory reference information for particular addresses. Prior to the introduction of PEBS counters, cache behavior could only be measured reliably in the aggregate across tens or hundreds of thousands of instructions. With the newest iteration of PEBS technology, cache events can be tied to a tuple of instruction pointer, target address (for both loads and stores), memory hierarchy, and observed latency. With this information we can now begin asking questions regarding the efficiency of not only regions of code, but how these regions interact with particular data structures and how these interactions evolve over time. In the short term, this information will be vital for performance analysts understanding and optimizing the behavior of their codes for the memory hierarchy. In the future, we can begin to ask how data layouts might be changed to improve performance and, for a particular application, what the theoretical optimal performance might be. The overall benefit to be produced by this effort was a commercial quality easy-to- use and scalable performance tool that will allow both beginner and experienced parallel programmers to automatically tune their applications for optimal cache usage. Effective use of such a tool can literally save weeks of performance tuning effort. Easy to use. With the proposed innovations, finding and fixing memory performance issues would be more automated and hide most to all of the performance engineer expertise ”under the hood” of the Open|SpeedShop performance tool. One of the biggest public benefits from the proposed innovations is that it makes performance analysis more usable to a larger group of application developers. Intuitive reporting of results. The Open|SpeedShop performance analysis tool has a rich set of intuitive, yet detailed reports for presenting performance results to application developers. Our goal was to leverage this existing technology to present the results from our memory performance addition to Open|SpeedShop. Suitable for experts as well as novices. Application performance is getting more difficult to measure as the hardware platforms they run on become more complicated. This makes life difficult for the application developer, in that they need to know more about the hardware platform, including the memory system hierarchy, in order to understand the performance of their application. Some application developers are comfortable in that sceario, while others want to do their scientific research and not have to understand all the nuances in the hardware platform they are running their application on. Our proposed innovations were aimed to support both experts and novice performance analysts. Useful in many markets. The enhancement to Open|SpeedShop would appeal to a broader market space, as it will be useful in scientific, commercial, and cloud computing environments. Our goal was to use technology developed initially at the and Lawrence Livermore National Laboratory combined with the development and commercial software experience of the Argo Navis Technologies, LLC (ANT) to form a powerful combination to delivery these objectives.

97 MATHEMATICS AND COMPUTING↗