Search NASA⌕ Search

SEARCH · Search NASA

Results for “distributed systems, data management”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 127 records · Page 7

Data Distribution System (DDS) and Solar Dynamic Observatory Ground Station (SDOGS) Integration Manager

The DDS SDOGS Integration Manager (DSIM) provides translation between native control and status formats for systems within DDS and SDOGS, and the ASIST (Advanced Spacecraft Integration and System Test) control environment in the SDO MOC (Solar Dynamics Observatory Mission Operations Center). This system was created in response for a need to centralize remote monitor and control of SDO Ground Station equipments using ASIST control environment in SDO MOC, and to have configurable table definition for equipment. It provides translation of status and monitoring information from the native systems into ASIST-readable format to display on pages in the MOC. The manager is lightweight, user friendly, and efficient. It allows data trending, correlation, and storing. It allows using ASIST as common interface for remote monitor and control of heterogeneous equipments. It also provides failover capability to back up machines.

Pham, Kim↗

Bi-Level Volt-VAR Optimization to Coordinate Smart Inverters with Voltage Control Devices

Conservation voltage reduction (CVR) uses voltage control to reduce customer power demand and help improve the operational efficiency of the electric power distribution system. With the advent of advanced distribution management systems (ADMS), the underlying data and control-rich platform can be utilized to enable a centralized Volt-VAR optimization (VVO) scheme that includes smart inverter controls along with legacy voltage control devices such as voltage regulators and capacitor banks to ensure a maximum CVR benefit. The objective of this paper to present a VVO approach that controls system’s legacy voltage control devices and coordinates their operation with smart inverter control. An optimal power flow (OPF) formulation is proposed by developing linear and nonlinear power flow approximations for a three-phase unbalanced electric power distribution system. A bi-level VVO approach is proposed where, Level-1 optimizes the control of legacy devices and smart inverters using a linear approximate three-phase power flow. In Level-2, the control parameters for smart inverters are adjusted to obtain an optimal and feasible solution by solving the approximate nonlinear OPF model. Level-1 is modeled as a Mixed Integer Linear Program (MILP) while Level-2 as a Quadratically Constrained Quadratic Program (QCQP). The proposed approach is validated using IEEE test cases and the results demonstrate the applicability of the framework in achieving the CVR objective.

Jha, Rahul Ranjan↗

Autonomous semantic data discovery for distributed networked systems

Systems, methods, techniques and apparatuses for managing distributed applications of networked intelligent agents are disclosed. The agents are operably to autonomously discover semantic profiles and associated data of other agents in a networked system participating in a given application. The agents need not be in direct communication with or known to all the other agents in the networked system.

Brissette, Alexander↗

Efficient Bulk Data Replication for the Earth System Grid

The Earth System Grid (ESG) community faces the difficult challenge of managing the distribution of massive data sets to thousands of scientists around the world. To move data replicas efficiently, the ESG has developed a data transfer management tool called the Bulk Data Mover (BDM). We describe the performance results of the current system and plans towards extending the techniques developed so far for the up- coming project, in which the ESG will employ advanced networks to move multi-TB datasets with the ulti- mate goal of helping researchers understand climate change and its potential impacts on world ecology and society.

97 MATHEMATICS AND COMPUTING↗

Data Management in the Continuum: Cross-facility Object-based Data Transfers

Scientific workflows are evolving from relying on a monolithic storage subsystem at a single High-Performance Computing (HPC) facility to using geographically distributed file systems, repositories, and cloud storage. As a result, storing, accessing, transferring, and managing scientific data have become highly complex and prone to performance inefficiencies. This paper delves into these challenges by exploring an optimized end-to-end interface designed to seamlessly connect various local and remote storage systems, enabling efficient data movement of objects across HPC–Cloud and HPC–HPC environments. We showcase this capability through an object-focused data management runtime system, discuss the effects of relaxed consistency semantics in distributed object scenarios, and illustrate its application in an earthquake simulation workflow. Besides reducing the amount of data by selectively transferring regions of interest, our facility-local results achieved a speedup of 45 × over an optimized HDF5 usage and 15 × over the HDF5 with caching by using the new interface in PDC-XF.

Bez, Jean Luca↗

A comparison of different database technologies for the CMS AsyncStageOut transfer database

AsyncStageOut (ASO) is the component of the CMS distributed data analysis system (CRAB) that manages users transfers in a centrally controlled way using the File Transfer System (FTS3) at CERN. It addresses a major weakness of the previous, decentralized model, namely that the transfer of the user’s output data to a single remote site was part of the job execution, resulting in inefficient use of job slots and an unacceptable failure rate. Currently ASO manages up to 600k files of various sizes per day from more than 500 users per month, spread over more than 100 sites. ASO uses a NoSQL database (CouchDB) as internal bookkeeping and as way to communicate with other CRAB components. Since ASO/CRAB were put in production in 2014, the number of transfers constantly increased up to a point where the pressure to the central CouchDB instance became critical, creating new challenges for the system scalability, performance, and monitoring. This forced a re-engineering of the ASO application to increase its scalability and lowering its operational effort. In this contribution we present a comparison of the performance of the current NoSQL implementation and a new SQL implementation, and how their different strengths and features influenced the design choices and operational experience. We also discuss other architectural changes introduced in the system to handle the increasing load and latency in delivering output to the user.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗

Online Model-Free DER Dispatch Via Adaptive Voltage Sensitivity Estimation and Chance Constrained Programming

This paper proposes an online data-driven distributed energy resource management system (DERMS) for distribution system optimal DER dispatch as well as voltage regulation. Here, the key innovation is to leverage the Local Sensitivity Factor (LSF) for transforming the DER control into a computationally efficient linear programming (LP) problem. By taking real-time measurements, the estimation of LSF eliminates the need for an accurate distribution system model as well as full nodal load information, which is difficult to achieve in practice. A robust recursive least squares method is also developed to ensure the robust estimation of LSF, which is initialized using reasonable values from model-derived LSFs. This allows the system to adapt to changing operational conditions effectively. A scenario-based, chance-constrained framework is further employed to ensure voltage remains within acceptable limits in the presence of measurement and estimation uncertainties. Test results on a real-world, 759-node distribution network located in western Colorado, U.S., validate the effectiveness and robustness of the proposed control approach and demonstrate its superior performance as compared to alternative methods.

24 POWER TRANSMISSION AND DISTRIBUTION↗

Kelpie: FY2014 Project Update

The ASC CSSE project Kelpie is a research and development project focused on developing a distributed, in-memory data management system that can be leveraged in a number of high-performance computing (HPC) applications. After FY13s demonstration that a key/value data store could be implemented on top of the Nessie RDMA/RPC library, we began refactoring Kelpie in FY14 in order to make it more usable by other research teams that need it for upcoming milestones. This report provides a summary of the different efforts in FY14 that took place to make Kelpie a more usable system.

97 MATHEMATICS AND COMPUTING↗

Information logistics: A production-line approach to information services

Logistics can be defined as the process of strategically managing the acquisition, movement, and storage of materials, parts, and finished inventory (and the related information flow) through the organization and its marketing channels in a cost effective manner. It is concerned with delivering the right product to the right customer in the right place at the right time. The logistics function is composed of inventory management, facilities management, communications unitization, transportation, materials management, and production scheduling. The relationship between logistics and information systems is clear. Systems such as Electronic Data Interchange (EDI), Point of Sale (POS) systems, and Just in Time (JIT) inventory management systems are important elements in the management of product development and delivery. With improved access to market demand figures, logisticians can decrease inventory sizes and better service customer demand. However, without accurate, timely information, little, if any, of this would be feasible in today's global markets. Information systems specialists can learn from logisticians. In a manner similar to logistics management, information logistics is concerned with the delivery of the right data, to the ring customer, at the right time. As such, information systems are integral components of the information logistics system charged with providing customers with accurate, timely, cost-effective, and useful information. Information logistics is a management style and is composed of elements similar to those associated with the traditional logistics activity: inventory management (data resource management), facilities management (distributed, centralized and decentralized information systems), communications (participative design and joint application development methodologies), unitization (input/output system design, i.e., packaging or formatting of the information), transportations (voice, data, image, and video communication systems), materials management (data acquisition, e.g., EDI, POS, external data bases, data entry) and production scheduling (job, staff, and project scheduling).

Adams, Dennis↗

A guide to NASA's Pilot Land Data System (PLDS)

NASA's Pilot Land Data System (PLDS) is a distributed information management system designed to support NASA's land science community. The PLDS provides a wide range of services including management of information about scientific data, access to a library of scientific data, a data ordering capability, communications, connection to data analysis facilities, and electronic mail. The PLDS provides these services by offering the scientist the capability to search for and order data, and to communicate electronically with other scientists and computers. Three functions enable scientists to find what data are available and where they reside. The first two, Find data summaries and Read detailed descriptions give summary and detailed descriptions about data sets or groups of related data sets, science, projects, and institutions which archive land data. The third, gives information about specific pieces of data. This last function has two components, Search systemwide inventory and Search local inventory. The first component enables the user to find data elements (images, geological samples, transects, maps, etc.) that exist anywhere in the PLDS while the second has only information about data at the local site. The first enables the user to find pieces of data from several different data sets with the same temporal and spatial coverage and other elements common to most data sets, while the second allows the user to select a data set based on these descriptors and on those that are unique to a data set. The PLDS provides capabilities that enable electronic file transfers, intercomputer connection, and electronic mail. Both TCP/IP and DECnet protocols are supported via the NASA Science Internet (NIS). Access is also available through Telenet.

Source record↗

Advanced software tools space station focused technology

Information is given in outline form on advanced software tools for the Space Station data management system. The Space Station data management system is identified as a highly distributed system with payload users controlling experiments and processing payload data from home facilities.

Nelson, Robert W.↗

AsyncStageOut: Distributed User Data Management for CMS Analysis

AsyncStageOut (ASO) is a new component of the distributed data analysis system of CMS, CRAB, designed for managing users' data. It addresses a major weakness of the previous model, namely that mass storage of output data was part of the job execution resulting in inefficient use of job slots and an unacceptable failure rate at the end of the jobs. ASO foresees the management of up to 400k files per day of various sizes, spread worldwide across more than 60 sites. It must handle up to 1000 individual users per month, and work with minimal delay. This creates challenging requirements for system scalability, performance and monitoring. ASO uses FTS to schedule and execute the transfers between the storage elements of the source and destination sites. It has evolved from a limited prototype to a highly adaptable service, which manages and monitors the user file placement and bookkeeping. To ensure system scalability and data monitoring, it employs new technologies such as a NoSQL database and re-uses existing components of PhEDEx and the FTS Dashboard. We present the asynchronous stage-out strategy and the architecture of the solution we implemented to deal with those issues and challenges. The deployment model for the high availability and scalability of the service is discussed. The performance of the system during the commissioning and the first phase of production are also shown, along with results from simulations designed to explore the limits of scalability.

Riahi, H.↗

Leveraging AMI data for distribution system model calibration and situational awareness

The many new distributed energy resources being installed at the distribution system level require increased visibility into system operations that will be enabled by distribution system state estimation (DSSE) and situational awareness applications. Reliable and accurate DSSE requires both robust methods for managing the big data provided by smart meters and quality distribution system models. This paper presents intelligent methods for detecting and dealing with missing or inaccurate smart meter data, as well as the ways to process the data for different applications. It also presents an efficient and flexible parameter estimation method based on the voltage drop equation and regression analysis to enhance distribution system model accuracy. Finally, it presents a 3-D graphical user interface for advanced visualization of the system state and events. Moreover, we demonstrate this paper for a university distribution network with the state-of-the-art real-time and historical smart meter data infrastructure.

24 POWER TRANSMISSION AND DISTRIBUTION↗

SWARM: Reimagining scientific workflow management systems in a distributed world

Modern scientific workflows process massive amounts of data from diverse instruments and sensors, leveraging geographically distributed, heterogeneous compute and storage resources—from leadership-class systems to edge devices—connected by high-performance networks. The diversity of resources introduces challenges in harnessing their full potential, with resilience issues arising across applications, system software, networks, storage, and hardware. Today, workflow management systems (WMS) coordinate the execution of computation and data management tasks across target resources. However, WMS’s centralized nature makes them vulnerable to faults and scalability issues that may result in failures of entire computational campaigns. In conclusion, this paper introduces a novel agentic framework for workflow management, fully distributing and decentralizing the WMS functions and modeling them as swarm intelligence agents infused with advanced artificial intelligence solutions and traditional distributed computing algorithms that can make coordinated decisions in the presence of failures of the underlying cyberinfrastructure.

Swarm intelligence↗

Performance issues in management of the Space Station Information System

The onboard segment of the Space Station Information System (SSIS), called the Data Management System (DMS), will consist of a Fiber Distributed Data Interface (FDDI) token-ring network. The performance of the DMS in scenarios involving two kinds of network management is analyzed. In the first scenario, how the transmission of routine management messages impacts performance of the DMS is examined. In the second scenario, techniques for ensuring low latency of real-time control messages in an emergency are examined.

Johnson, Marjory J.↗

BigPanDA: PanDA Workload Management System and its Applications beyond ATLAS

Modern experiments collect peta-scale volumes of data and utilize vast, geographically distributed computing infrastructure that serves thousands of scientists around the world. Requirements for rapid, near real-time data processing, fast analysis cycles and need to run massive detector simulations to support data analysis pose special premium on efficient use of available computational resources. A sophisticated Workload Management System (WMS) is needed to coordinate the distribution and processing of data and jobs in such environment. The ATLAS experiment at CERN uses PanDA (Production and Data Analysis) Workload Management System for managing the workflow for all data processing on over 150 data centers. While PanDAcurrently uses more than 250,000 cores with a peak performance of 0.3 petaFLOPS, it runs around 2 million jobs per day on hundreds of Grid sites and serving thousands of ATLAS users. In 2017 about 1.5 exabytes of data were processed with PanDA.In 2012 BigPanDA project project was started with aim to introduce new types of computing resources into ATLAS computing infrastructure, but also to offering PanDA features to different data-intensive applications for projects and experiments outside of ATLAS and High-Energy and Nuclear Physics. In this article we will present accomplishments and discuss possible directions for future work.

Svirin, Pavlo↗

NASA Exhibits

A series of NASA presentations for the Supercomputing 2001 conference are summarized. The topics include: (1) Mars Surveyor Landing Sites "Collaboratory"; (2) Parallel and Distributed CFD for Unsteady Flows with Moving Overset Grids; (3) IP Multicast for Seamless Support of Remote Science; (4) Consolidated Supercomputing Management Office; (5) Growler: A Component-Based Framework for Distributed/Collaborative Scientific Visualization and Computational Steering; (6) Data Mining on the Information Power Grid (IPG); (7) Debugging on the IPG; (8) Debakey Heart Assist Device: (9) Unsteady Turbopump for Reusable Launch Vehicle; (10) Exploratory Computing Environments Component Framework; (11) OVERSET Computational Fluid Dynamics Tools; (12) Control and Observation in Distributed Environments; (13) Multi-Level Parallelism Scaling on NASA's Origin 1024 CPU System; (14) Computing, Information, & Communications Technology; (15) NAS Grid Benchmarks; (16) IPG: A Large-Scale Distributed Computing and Data Management System; and (17) ILab: Parameter Study Creation and Submission on the IPG.

Deardorff, Glenn↗

Space station data system analysis/architecture study. Task 3: Trade studies, DR-5, volume 2

Results of a Space Station Data System Analysis/Architecture Study for the Goddard Space Flight Center are presented. This study, which emphasized a system engineering design for a complete, end-to-end data system, was divided into six tasks: (1); Functional requirements definition; (2) Options development; (3) Trade studies; (4) System definitions; (5) Program plan; and (6) Study maintenance. The Task inter-relationship and documentation flow are described. Information in volume 2 is devoted to Task 3: trade Studies. Trade Studies have been carried out in the following areas: (1) software development test and integration capability; (2) fault tolerant computing; (3) space qualified computers; (4) distributed data base management system; (5) system integration test and verification; (6) crew workstations; (7) mass storage; (8) command and resource management; and (9) space communications. Results are presented for each task.

Source record↗