Search NASA⌕ Search

SEARCH · Search NASA

Results for “middlewares”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records

A Distributed Middleware Architecture for Attack-Resilient Communications in Smart Grids

Distributed Energy Resources (DERs) are being increasingly accepted as an excellent complement to traditional energy sources in smart grids. As most of these generators are geographically dispersed, dedicated communications investments for every generator are capital cost prohibitive. Real-time distributed communications middleware, which supervises, organizes and schedules tremendous amounts of data traffic in smart grids with high penetrations of DERs, allows for the use of existing network infrastructure. In this paper, we propose a distributed attack-resilient middleware architecture that detects and mitigates the congestion attacks by exploiting the Quality of Experience (QoE) measures to complement the conventional Quality of Service (QoS) information to detect and mitigate the congestion attacks effectively. The simulation results illustrate the efficiency of our proposed communications middleware architecture.

24 POWER TRANSMISSION AND DISTRIBUTION↗

Definition and implementation of a SAML-XACML profile for authorization interoperability across Grid middleware implementations in OSG and EGEE.

In order to ensure interoperability between middleware and authorization infrastructures used in the Open Science Grid (OSG) and the Enabling Grids for E-science (EGEE) projects, an Authorization Interoperability activity was initiated in 2006. The interoperability goal was met in two phases: firstly, agreeing on a common authorization query interface and protocol with an associated profile that ensures standardized use of attributes and obligations; and secondly implementing, testing, and deploying on OSG and EGEE, middleware that supports the interoperability protocol and profile. The activity has involved people from OSG, EGEE, the Globus Toolkit project, and the Condor project. This paper presents a summary of the agreed-upon protocol, profile and the software components involved.

97 MATHEMATICS AND COMPUTING↗

Middleware for a Heterogeneous CAV Fleet

This paper introduces CAN to ROS, a model-based code generation tool used in development, testing, and deployment of a heterogeneous fleet of vehicles with robotic sensing in ROS. Code generation supports two main features: (1) self-configuration for deployment in a heterogeneous vehicle fleet, and (2) quick iteration for testing and development of reading vehicle sensors and robotic control. This tool features the ability to detect the vehicle it is in and regenerate and rebuild itself at runtime to provide the proper two-way bridge between ROS and the sensed on-board vehicle sensor network. Code generation relies on a per-model defined JSON to map a CAN database (DBC) to the desired ROS topic names and message types. The live ROS publishing of CAN messages allows for instant feedback, and the code regeneration allows for adjustments in DBC or vehicle JSON to iteratively hone in on new vehicle signals. Generated ROS nodes are written in C++ for runtime use in lightweight embedded computers. This has been tested in vehicles from three different Original Equipment Manufacturers (OEMs), and can be extended to support a wide array of vehicles. By using a unifying ROS specification, a heterogeneous set of vehicles can be unified into a fleet with abstracted model-specific details; this opens the door for developing cross-model software applications for vehicle control, connected vehicle applications, or fleet monitoring systems.

42 ENGINEERING↗

...And Eat it Too: High Read Performance in Write-Optimized HPC I/O Middleware File Formats

As HPC applications run on increasingly high process counts on larger and larger machines, both the frequency of checkpoints needed for fault tolerance and the resolution and size of Data Analysis Dumps are expected to increase proportionally. In order to maintain an acceptable ratio of time spent performing useful computation work to time spent performing I/O, write bandwidth to the underlying storage system must increase proportionally to this increase in the checkpoint and computation size. Unfortunately, popular scientific self-describing file formats such as netCDF and HDF5 are designed with a focus on portability and flexibility. Extra care and careful crafting of the output structure and API calls is required to optimize for write performance using these APIs. To provide sufficient write bandwidth to continue to support the demands of scientific applications, the HPC community has developed a number of I/O middleware layers, that structure output into write-optimized file formats. However, the obvious concern with any write optimized file format would be a corresponding penalty on reads. In the log-structured filesystem, for example, a file generated by random writes could be written efficiently, but reading the file back sequentially later would result in very poor performance. Simulation results require efficient read-back for visualization and analytics, and though most checkpoint files are never used, the efficiency of a restart is very important in the face of inevitable failures. The utility of write speed improving middleware would be greatly diminished if it sacrificed acceptable read performance. In this paper we examine the read performance of two write-optimized middleware layers on large parallel machines and compare it to reading data natively in popular file formats. The two I/O middleware layer examined in this paper are the Adaptable IO System (ADIOS), a library-based approach developed at Oak Ridge National Laboratory to provide a high-level IO API that can be used in place of netCDF or HDF5 to do much more aggressive write-behind and efficient reordering of data locations within the file; and the Parallel Log-structured Filesystem (PLFS), a stackable FUSE filesystem approach developed at Los Alamos National Laboratory that decouples concurrent writes to improve the speed of checkpoints. Since ADIOS is an I/O componentization that affords selection of different I/O methods at or during runtime, through a single API, users can have access to MPI-IO, Posix-IO, HDF5, netCDF, and staging methods. The ADIOS BP file format is a new log-file format that has a superset of the features of both HDF5 and netCDF, but is designed to be portable and flexible while being optimized for writing. PLFS, on the other hand, is mounted as a stackable filesystem on top of an existing parallel filesystem. Reads or writes to the PLFS filesystem are transparently translated into operations on per-process log files stored in the underlying parallel filesystem. Since PLFS performs this translation without application modification, users can write in HDF5, netCDF, or app-specific file formats and PLFS will store the writes in a set of efficiently written log-formatted files, while presenting the user with a logical 'flat' file on reads. Despite their different approaches, the commonality behind both of these middleware systems is that they both write to a log file format. As shown in previous publications, writes are fully optimized in both systems, sometimes resulting in 100x improvements over writing data in popular file formats. But as mentioned above, writing is only one part of the story. In this paper we examine the read performance of our middleware layers on large parallel machines and compare these to reading data either natively or from other popular file formats. We compare the reading performance in two different scenarios: (1) Reading back restarts from the same number of processors as wrote the data and (2) Reading back restart data from a different number of processors. We observe that not only can write-optimized I/O middleware be built to not penalize read speeds, but for important workloads, techniques that improve write performance can, perhaps counterintuitively, improve read speeds over reading from a contiguously organized file format. In the remainder of this paper, we investigate this further through case studies of PLFS and ADIOS on simulation checkpoint restarts.

99 GENERAL AND MISCELLANEOUS↗

Middleware Architecture Evaluation for Dependable Self-managing Systems

Middleware provides infrastructure support for creating dependable software systems. A specific middleware implementation plays a critical role in determining the quality attributes that satisfy a system’s dependability requirements. Evaluating a middleware architecture at an early development stage can help to pinpoint critical architectural challenges and optimize design decisions. In this paper, we present a method and its application to evaluate middleware architectures, driven by emerging architecture patterns for developing self-managing systems. Our approach focuses on two key attributes of dependability, reliability and maintainability by means of fault tolerance and fault prevention. We identify the architectural design patterns necessary to build an adaptive self-managing architecture that is capable of preventing or recovering from failures. These architectural patterns and their impacts on quality attributes create the context for middleware evaluation. Our approach is demonstrated by an example application -- failover control of a financial application on an enterprise service bus.

97 MATHEMATICS AND COMPUTING↗

An Attack-Resilient Middleware Architecture for Grid Integration of Distributed Energy Resources

In recent years, the increasing penetration of Distributed Energy Resources (DERs) has made an impact on the operation of the electric power systems. In the grid integration of DERs, data acquisition systems and communications infrastructure are crucial technologies to maintain system economic efficiency and reliability. Since most of these generators are relatively small, dedicated communications investments for every generator are capital cost prohibitive. Combining real-time attack-resilient communications middleware with Internet of Things (IoTs) technologies allows for the use of existing infrastructure. In our paper, we propose an intelligent communication middleware that utilizes the Quality of Experience (QoE) metrics to complement the conventional Quality of Service (QoS) evaluation. Furthermore, our middleware employs deep learning techniques to detect and defend against congestion attacks. The simulation results illustrate the efficiency of our proposed communications middleware architecture.

24 POWER TRANSMISSION AND DISTRIBUTION↗

A Conceptual Architecture for a Quantum-HPC Middleware

Abstract-Quantum computing promises potential for science and industry by solving certain computationally complex problems faster than classical computers. Quantum computing systems evolved from coupled to classical computing nodes (HPC). With the increasing monolithic systems towards modular architectures comprising multiple quantum processing units (QPUs) scale, middleware systems that facilitate the efficient coupling of quantum-classical computing are becoming critical. Through an in-depth analysis of quantum applications, integration patterns and systems, we identified a gap in understanding Quantum HPC middleware systems. We present a conceptual middleware to facilitate reasoning about quantum-classical integration and serve as the basis for a future middleware system. An essential contribution of this paper lies in leveraging well-established high performance computing abstractions for managing workloads, tasks, and resources to integrate quantum computing into HPC systems seamlessly

97 MATHEMATICS AND COMPUTING↗

Cloud object store for archive storage of high performance computing data using decoupling middleware

Cloud object storage is enabled for archived data, such as checkpoints and results, of high performance computing applications using a middleware process. A plurality of archived files, such as checkpoint files and results, generated by a plurality of processes in a parallel computing system are stored by obtaining the plurality of archived files from the parallel computing system; converting the plurality of archived files to objects using a log structured file system middleware process; and providing the objects for storage in a cloud object storage system. The plurality of processes may run, for example, on a plurality of compute nodes. The log structured file system middleware process may be embodied, for example, as a Parallel Log-Structured File System (PLFS). The log structured file system middleware process optionally executes on a burst buffer node.

97 MATHEMATICS AND COMPUTING↗

Cloud object store for checkpoints of high performance computing applications using decoupling middleware

Cloud object storage is enabled for checkpoints of high performance computing applications using a middleware process. A plurality of files, such as checkpoint files, generated by a plurality of processes in a parallel computing system are stored by obtaining said plurality of files from said parallel computing system; converting said plurality of files to objects using a log structured file system middleware process; and providing said objects for storage in a cloud object storage system. The plurality of processes may run, for example, on a plurality of compute nodes. The log structured file system middleware process may be embodied, for example, as a Parallel Log-Structured File System (PLFS). The log structured file system middleware process optionally executes on a burst buffer node.

97 MATHEMATICS AND COMPUTING↗

A collaborative network middleware project by Lambda Station, TeraPaths, and Phoebus

The TeraPaths, Lambda Station, and Phoebus projects, funded by the US Department of Energy, have successfully developed network middleware services that establish on-demand and manage true end-to-end, Quality-of-Service (QoS) aware, virtual network paths across multiple administrative network domains, select network paths and gracefully reroute traffic over these dynamic paths, and streamline traffic between packet and circuit networks using transparent gateways. These services improve network QoS and performance for applications, playing a critical role in the effective use of emerging dynamic circuit network services. They provide interfaces to applications, such as dCache SRM, translate network service requests into network device configurations, and coordinate with each other to setup up end-to-end network paths. The End Site Control Plane Subsystem (ESCPS) builds upon the success of the three projects by combining their individual capabilities into the next generation of network middleware. ESCPS addresses challenges such as cross-domain control plane signalling and interoperability, authentication and authorization in a Grid environment, topology discovery, and dynamic status tracking. The new network middleware will take full advantage of the perfSONAR monitoring infrastructure and the Inter-Domain Control plane efforts and will be deployed and fully vetted in the Large Hadron Collider data movement environment.

99 GENERAL AND MISCELLANEOUS↗

Identifying gaps in grid middleware on fast networks with the advanced networking initiative

As of 2012, a number of US Department of Energy (DOE) National Laboratories have access to a 100 Gb/s wide-area network backbone. The ESnet Advanced Networking Initiative (ANI) project is intended to develop a prototype network, based on emerging 100 Gb/s Ethernet technology. The ANI network will support DOE's science research programs. A 100 Gb/s network test bed is a key component of the ANI project. The test bed offers the opportunity for early evaluation of 100Gb/s network infrastructure for supporting the high impact data movement typical of science collaborations and experiments. In order to make effective use of this advanced infrastructure, the applications and middleware currently used by the distributed computing systems of large-scale science need to be adapted and tested within the new environment, with gaps in functionality identified and corrected. As a user of the ANI test bed, Fermilab aims to study the issues related to end-to-end integration and use of 100 Gb/s networks for the event simulation and analysis applications of physics experiments. In this paper we discuss our findings from evaluating existing HEP Physics middleware and application components, including GridFTP, Globus Online, etc. in the high-speed environment. These will include possible recommendations to the system administrators, application and middleware developers on changes that would make production use of the 100 Gb/s networks, including data storage, caching and wide area access.

Dykstra, Dave↗

ReSCo: A Middleware Component for Reliable Service Composition in Pervasive Systems

Service composition schemes create high-level application services by combining several basic services. Service composition schemes for dynamic, open systems, such as those found in pervasive environments, must be cognizant of the possibility of failures and attacks. In open systems, it is seldom feasible to guarantee the reliability of each node prior to access; however, there may be several possible ways to compose the same high-level service, each having a different (though possibly overlapping) set of nodes that can satisfy the composition. We approach this problem with a Reliable Service Composition middleware component, ReSCo, to determine trustworthy compositions and nodes for service composition in dynamic, open systems. ReSCo is a modular, adaptive middleware component that selects from possible composition paths and nodes to enhance reliability of service compositions. ReSCo can work with a broad range of both service composition algorithms and trust establishment mechanisms.

97 MATHEMATICS AND COMPUTING↗

The architecture of an event correlation service for adaptive middleware-based applications

Loosely coupled component communication driven by events is a key mechanism for building middleware- based applications that must achieve reliable qualities of service in an adaptive manner. In such a system, events that encapsulate state snapshots of a running system are generated by monitoring components. Hence, an event correlation service is necessary for correlating monitored events from multiple sources. The requirements for the event correlation raise two challenges: to seamlessly integrate event correlation services with other services and applications; and to provide reliable event management with minimal delay. This paper describes our experience in the design and implementation of an event correlation service. The design encompasses an event correlator and an event proxy that are integrated with an architecture for adaptive middleware components. The implementation utilizes the common-based event (CBE) specification and stateful Web service technologies to support the deployment of the event correlation service in a distributed architecture. We evaluate the performance of the overall solution in a test bed and present the results in terms of the trade-off between the flexibility and the performance overhead of the architecture

99 GENERAL AND MISCELLANEOUS↗

STREAM2016: Streaming Requirements, Experience, Applications and Middleware Workshop

The Department of Energy (DOE) Office of Science (SC) facilities including accelerators, light sources and neutron sources and sensors that study, the environment, and the atmosphere, are producing streaming data that needs to be analyzed for next-generation scientific discoveries. There has been an explosion of new research and technologies for stream analytics arising from the academic and private sectors. However, there has been no corresponding effort in either documenting the critical research opportunities or building a community that can create and foster productive collaborations. The two-part workshop series, STREAM: Streaming Requirements, Experience, Applications and Middleware Workshop (STREAM2015 and STREAM2016), were conducted to bring the community together and identify gaps and future efforts needed by both NSF and DOE. This report describes the discussions, outcomes and conclusions from STREAM2016: Streaming Requirements, Experience, Applications and Middleware Workshop, the second of these workshops held on March 22-23, 2016 in Tysons, VA. STREAM2016 focused on the Department of Energy (DOE) applications, computational and experimental facilities, as well software systems. Thus, the role of “streaming and steering” as a critical mode of connecting the experimental and computing facilities was pervasive through the workshop. Given the overlap in interests and challenges with industry, the workshop had significant presence from several innovative companies and major contributors. The requirements that drive the proposed research directions, identified in this report, show an important opportunity for building competitive research and development program around streaming data. These findings and recommendations are consistent with vision outlined in NRC Frontiers of Data and National Strategic Computing Initiative (NCSI) [1, 2]. The discussions from the workshop are captured as topic areas covered in this report's sections. The report discusses four research directions driven by current and future application requirements reflecting the areas identified as important by STREAM2016. These include (i) Algorithms, (ii) Programming Models, Languages and Runtime Systems (iii) Human-in-the-loop and Steering in Scientific Workflow and (iv) Facilities.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗

Definition and Implementation of a SAML-XAMCL Profile for Authorization Interoperability Across Grid Middleware in OSG and EGEE

In order to ensure interoperability between middleware and authorization infrastructures used in the Open Science Grid (OSG) and the Enabling Grids for E-science (EGEE) projects, an Authorization Interoperability activity was initiated in 2006. The interoperability goal was met in two phases: firstly, agreeing on a common authorization query interface and protocol with an associated profile that ensures standardized use of attributes and obligations; and secondly implementing, testing, and deploying on OSG and EGEE, middleware that supports the interoperability protocol and profile. The activity has involved people from OSG, EGEE, the Globus Toolkit project, and the Condor project. This paper presents a summary of the agreed-upon protocol, profile and the software components involved.

97 MATHEMATICS AND COMPUTING↗

Value-of-Information based Middleware for Autonomic Querying of Distributed Sensor Databases

With particular focus on distributed situational awareness in defense and security applications, we propose a value-of-information based middleware framework as a paradigm shift from crawl-index based centralized search. The proposed approach seeks to improve efficiency in search and retrieval by transforming sensors into programmable information points and enabling ubiquitous data and software flows through the infrastructure of the distributed network. We explain the different components of such a middleware framework to organize, tag and notify emerging spatial, temporal and causal patterns from the sensor measurements. We conclude the paper with a brief discussion on the top-down programming model that can realize the framework as a reconfigurable sensor query system.

Sukumar, Sreenivas R.↗

SLoG: Large-Scale Logging Middleware for HPC and Big Data Convergence

Cloud developers traditionally rely on purpose-specific services to provide the storage model they need for an application. In contrast, HPC developers have a much more limited choice, typically restricted to a centralized parallel file system for persistent storage. Unfortunately, these systems often offer low performance when subject to highly concurrent, conflicting I/O patterns. This makes difficult the implementation of inherently concurrent data structures such as distributed shared logs. Yet, this data structure is key to applications such as computational steering, data collection from physical sensor grids, or discrete event generators. In this paper we tackle this issue. We present SLoG, shared log middleware providing a shared log abstraction over a parallel file system, designed to circumvent the aforementioned limitations. We evaluate SLoG’s design on up to 100,000 cores of the Theta supercomputer: the results show high append velocity at scale while also providing substantial benefits for other persistent backend storage systems.

big data↗

An Autonomic Middleware Solution for Coordinating Multiple QoS Controls

Adaptive self-managing applications can adapt their behavior through components that monitor the application behavior and provide feedback controls. This paper outlines an autonomic approach to coordinate multiple controls for managing service quality using executable control models. In this approach, controls are modeled as process models. Moreover, controls with cross-cutting concerns are provisioned by a dedicated process model. The flexibility of this approach allows composing new controls from existing control components. The coordination of their dependencies is performed within a unified architecture framework for modeling, deploying and executing these models. We integrate the process modeling and execution techniques into a middleware architecture to deliver such features. To demonstrate the practical utilization of this approach, we employ it to manage fail-over and over-loading controls for a service oriented loan brokering application. The empirical results further validate that this solution is not only sensitive to resolving cross-cutting interests of multiple controls, but also lightweight as it incurs low computational overhead.

99 GENERAL AND MISCELLANEOUS↗