Search NASA⌕ Search

SEARCH · Search NASA

Results for “Distributed dynamic resource management”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records
All sourcesNASA NTRSDOE OSTI
DOE OSTI2022

Electric Access System Enhancement (EASE): Assessment of a Distributed Energy Resource Management System for Enabling Dynamic Hosting Capacity

The Electric Access System Enhancement (EASE) project demonstrated a scalable, interoperable, and cost-effective means of integrating high penetration of distributed energy resources (DERs). The control architecture developed leveraged a Distributed Energy Resource Management System, Distribution System Operator for transacting energy, and 3rd Party DER Aggregator platform. The project identified ways to enhance the customer interconnection process to the grid and improve access to information from DERs and optimize the usage of DERs to provide energy services in a simulated day-ahead shadow market while maintain grid reliability. By integrating these capabilities into a scalable system of systems, the DSO can effectively balance DER generation and customer demand on the distribution network. This capability allows the grid to host more DER than traditionally possible on wires alone. Hosting more DER has the added benefit of supplying increased demand growth, sometimes beyond the capacity limits of the distribution network itself. This is known as a dynamic hosting capacity, which could help utilities manage the forecasted growth in electricity demand as California switch to electric vehicles and appliances. This could help to establish energy storage and PV generation as a “pseudo firm” generation resource mix if managed appropriately and provide sufficient resource adequacy for distribution capacity upgrade deferrals.

14 SOLAR ENERGY↗
DOE OSTI2022

Design and implementation of dynamic I/O control scheme for large scale distributed file systems

In this paper, we have analyzed the input/output (I/O) activities of Cori, which is a high-performance computing system at the National Energy Research Scientific Computing Center at Lawrence Berkeley National Laboratory. Our analysis results indicate that most users do not adjust storage configurations but rather use the default settings. In addition, owing to the interference from many applications running simultaneously, the performance varies based on the system status. To configure file systems autonomously in complex environments, we developed DCA-IO, a dynamic distributed file system configuration adjustment algorithm that utilizes the system log information to adjust storage configurations automatically. Our scheme aims to improve the application performance and avoid interference from other applications without user intervention. Moreover, DCA-IO uses the existing system logs and does not require code modifications, an additional library, or user intervention. To demonstrate the effectiveness of DCA-IO, we performed experiments using I/O kernels of real applications in both an isolated small-sized Lustre environment and Cori. Our experimental results shows that our scheme can improve the performance of HPC applications by up to 263% with the default Lustre configuration.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2020

Distributed Resources for the Earth System Grid Federation (ESGF) Advanced Management (DREAM). Final Report

Distributed Resources for the Earth System Grid Federation (ESGF) Advanced Management (DREAM) is a proposed system that will enable data from an infinite number of diverse sources to be organized and accessed from anywhere using any handheld or other computer device. The approach offers a powerful roadmap for the creation and integration of a unified knowledge base of an entire ecosystem, including its many geophysical, geographical, social, political, agricultural, energy, transportation, and cyber aspects. The resulting aggregation of data has the potential to generate an informational universe of unprecedented size that has never before been possible due to the prohibitive costs, managerial complexity, and technical barriers associated with ever-changing exponential-growth data flows. We envision that DREAM will accelerate discovery by enabling climate researchers, among other types of researchers, to manage, analyze, and visualize data from earth-scale measurements and simulations. DREAM’s success will be built on proven components that leverage existing services and resources. A key building block for DREAM will be the ESGF, chaired by Dean N. Williams. Expanding on the existing ESGF, the project will ensure that the access, storage, movement, and analysis of the large quantities of data that are processed and produced by diverse science projects can be dynamically distributed with proper resource management. Much of the Office of Science data is currently generated by multiple stand-alone facilities. DREAM can collect data accumulated from these facilities and incorporate it into a fully integrated network accessible from anywhere in the world. The result is a completely new paradigm shift for data management, analysis, and visualization enabling researchers to: Manage their calculations, data, tools, and research results; Ensure that all data are sharable, reproducible and (re)usable—accompanied by appropriate metadata describing its provenance, syntax, and semantics at creation; Advance application performance by selectively adapting APIs and services in response to scientific requirements and architectural complexities; and Provide scalable interactive resource management—navigate data and metadata at multiple levels, provide architecture-aware data integration, analysis and visualization tools. We will engage closely with DOE, NASA, and NOAA science groups working at the leading edge of computing. These engagements—in domains such as biology, climate, and hydrology—will allow us to advance disciplinary science goals and inform our development of technologies that can accelerate discovery across DOE more broadly. We will advertise and promote our technologies via dedicated workshops, tutorials, and sessions at conferences, stand-alone events with broad inter-disciplinary invitation, and engagements with leadership facilities.

54 ENVIRONMENTAL SCIENCES↗
NASA NTRS2003

Toward a Dynamically Reconfigurable Computing and Communication System for Small Spacecraft

Future science missions will require the use of multiple spacecraft with multiple sensor nodes autonomously responding and adapting to a dynamically changing space environment. The acquisition of random scientific events will require rapidly changing network topologies, distributed processing power, and a dynamic resource management strategy. Optimum utilization and configuration of spacecraft communications and navigation resources will be critical in meeting the demand of these stringent mission requirements. There are two important trends to follow with respect to NASA's (National Aeronautics and Space Administration) future scientific missions: the use of multiple satellite systems and the development of an integrated space communications network. Reconfigurable computing and communication systems may enable versatile adaptation of a spacecraft system's resources by dynamic allocation of the processor hardware to perform new operations or to maintain functionality due to malfunctions or hardware faults. Advancements in FPGA (Field Programmable Gate Array) technology make it possible to incorporate major communication and network functionalities in FPGA chips and provide the basis for a dynamically reconfigurable communication system. Advantages of higher computation speeds and accuracy are envisioned with tremendous hardware flexibility to ensure maximum survivability of future science mission spacecraft. This paper discusses the requirements, enabling technologies, and challenges associated with dynamically reconfigurable space communications systems.

Kifle, Muli↗
DOE OSTI2020

Methods and Experiences for Developing Abstractions for Data-intensive, Scientific Applications

Developing software for scientific applications that require the integration of diverse types of computing, instruments, and data present challenges that are distinct from commercial software. These applications require scale, and the need to integrate various programming and computational models with evolving and heterogeneous infrastructure. Pervasive and effective abstractions for distributed infrastructures are thus critical; however, the process of developing abstractions for scientific applications and infrastructures is not well understood. While theory-based approaches for system development are suited for well-defined, closed environments, they have severe limitations for designing abstractions for scientific systems and applications. The design science research (DSR) method provides the basis for designing practical systems that can handle real-world complexities at all levels. In contrast to theory-centric approaches, DSR emphasizes both practical relevance and knowledge creation by building and rigorously evaluating all artifacts. In this work, we show how DSR provides a well-defined framework for developing abstractions and middleware systems for distributed systems. Specifically, we address the critical problem of distributed resource management on heterogeneous infrastructure over a dynamic range of scales, a challenge that currently limits many scientific applications. We use the pilot-abstraction, a widely used resource management abstraction for high-performance, high throughput, big data, and streaming applications, as a case study for evaluating the DSR activities. For this purpose, we analyze the research process and artifacts produced during the design and evaluation of the pilot-abstraction. We find DSR provides a concise framework for iteratively designing and evaluating systems. Finally, we capture our experiences and formulate different lessons learned.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2024

Scaling Ensembles of Data-Intensive Quantum Chemical Calculations for Millions of Molecules

Deep learning models are efficient computational tools that can accelerate the inverse design of molecules with desired functional properties by generating predictions at a fraction of the time required by traditional quantum chemical approaches. To ensure that a model maintains accuracy and transferability across broad regions of the chemical space explored during the inverse design, it must be trained on massively large volumes of simulation data. This requires running large-scale ensemble quantum chemical calculations on high-performance computing (HPC) systems for data collection. However, the efficient execution of such large ensemble calculations and the management of large volumes of output data require tools that can judiciously utilize computational resources and manage metadata overhead on the file system. Therefore, we present a high-performance, scalable, ensemble management framework for performing data-intensive quantum chemical electronic structure calculations for organic molecules. This framework provides abstractions to plug different ab initio, first principles, and first principles-based semi-empirical methods and executes them efficiently at large scale on HPC systems. It dynamically distributes tasks to resources and uses tiered storage for managing large collections of files. We employed this framework to process over ten million organic molecules and generate open-source datasets that provide UV-vis absorption spectra by running time-dependent density-functional tight-binding calculations. It is the largest database containing molecular optical spectra that were simulated with quantum chemical methods in a consistent manner.

Mehta, Kshitij↗
NASA NTRS2000

Information Power Grid: Distributed High-Performance Computing and Large-Scale Data Management for Science and Engineering

We use the term "Grid" to refer to distributed, high performance computing and data handling infrastructure that incorporates geographically and organizationally dispersed, heterogeneous resources that are persistent and supported. This infrastructure includes: (1) Tools for constructing collaborative, application oriented Problem Solving Environments / Frameworks (the primary user interfaces for Grids); (2) Programming environments, tools, and services providing various approaches for building applications that use aggregated computing and storage resources, and federated data sources; (3) Comprehensive and consistent set of location independent tools and services for accessing and managing dynamic collections of widely distributed resources: heterogeneous computing systems, storage systems, real-time data sources and instruments, human collaborators, and communications systems; (4) Operational infrastructure including management tools for distributed systems and distributed resources, user services, accounting and auditing, strong and location independent user authentication and authorization, and overall system security services The vision for NASA's Information Power Grid - a computing and data Grid - is that it will provide significant new capabilities to scientists and engineers by facilitating routine construction of information based problem solving environments / frameworks. Such Grids will knit together widely distributed computing, data, instrument, and human resources into just-in-time systems that can address complex and large-scale computing and data analysis problems. Examples of these problems include: (1) Coupled, multidisciplinary simulations too large for single systems (e.g., multi-component NPSS turbomachine simulation); (2) Use of widely distributed, federated data archives (e.g., simultaneous access to metrological, topological, aircraft performance, and flight path scheduling databases supporting a National Air Space Simulation systems}; (3) Coupling large-scale computing and data systems to scientific and engineering instruments (e.g., realtime interaction with experiments through real-time data analysis and interpretation presented to the experimentalist in ways that allow direct interaction with the experiment (instead of just with instrument control); (5) Highly interactive, augmented reality and virtual reality remote collaborations (e.g., Ames / Boeing Remote Help Desk providing field maintenance use of coupled video and NDI to a remote, on-line airframe structures expert who uses this data to index into detailed design databases, and returns 3D internal aircraft geometry to the field); (5) Single computational problems too large for any single system (e.g. the rotocraft reference calculation). Grids also have the potential to provide pools of resources that could be called on in extraordinary / rapid response situations (such as disaster response) because they can provide common interfaces and access mechanisms, standardized management, and uniform user authentication and authorization, for large collections of distributed resources (whether or not they normally function in concert). IPG development and deployment is addressing requirements obtained by analyzing a number of different application areas, in particular from the NASA Aero-Space Technology Enterprise. This analysis has focussed primarily on two types of users: the scientist / design engineer whose primary interest is problem solving (e.g. determining wing aerodynamic characteristics in many different operating environments), and whose primary interface to IPG will be through various sorts of problem solving frameworks. The second type of user is the tool designer: the computational scientists who convert physics and mathematics into code that can simulate the physical world. These are the two primary users of IPG, and they have rather different requirements. The results of the analysis of the needs of these two types of users provides a broad set of requirements that gives rise to a general set of required capabilities. The IPG project is intended to address all of these requirements. In some cases the required computing technology exists, and in some cases it must be researched and developed. The project is using available technology to provide a prototype set of capabilities in a persistent distributed computing testbed. Beyond this, there are required capabilities that are not immediately available, and whose development spans the range from near-term engineering development (one to two years) to much longer term R&D (three to six years). Additional information is contained in the original.

Johnston, William E.↗
NASA NTRS2000

Adaptive Management of Computing and Network Resources for Spacecraft Systems

It is likely that NASA's future spacecraft systems will consist of distributed processes which will handle dynamically varying workloads in response to perceived scientific events, the spacecraft environment, spacecraft anomalies and user commands. Since all situations and possible uses of sensors cannot be anticipated during pre-deployment phases, an approach for dynamically adapting the allocation of distributed computational and communication resources is needed. To address this, we are evolving the DeSiDeRaTa adaptive resource management approach to enable reconfigurable ground and space information systems. The DeSiDeRaTa approach embodies a set of middleware mechanisms for adapting resource allocations, and a framework for reasoning about the real-time performance of distributed application systems. The framework and middleware will be extended to accommodate (1) the dynamic aspects of intra-constellation network topologies, and (2) the complete real-time path from the instrument to the user. We are developing a ground-based testbed that will enable NASA to perform early evaluation of adaptive resource management techniques without the expense of first deploying them in space. The benefits of the proposed effort are numerous, including the ability to use sensors in new ways not anticipated at design time; the production of information technology that ties the sensor web together; the accommodation of greater numbers of missions with fewer resources; and the opportunity to leverage the DeSiDeRaTa project's expertise, infrastructure and models for adaptive resource management for distributed real-time systems.

Pfarr, Barbara↗
DOE OSTI2020

How fast can one resize a distributed file system?

Efficient resource utilization becomes a major concern as large-scale distributed computing infrastructures keep growing in size. Malleability, the possibility for resource managers to dynamically increase or decrease the amount of resources allocated to a job, is a promising way to save energy and costs. However, state-of-the-art parallel and distributed storage systems have not been designed with malleability in mind. The reason is mainly the supposedly high cost of data transfers required by resizing operations. Nevertheless, as network and storage technologies evolve, old assumptions about potential bottlenecks can be revisited. In this study, we evaluate the viability of malleability as a design principle for a distributed storage system. We specifically model the minimal duration of the commission and decommission operations. To show how our models can be used in practice, we evaluate the performance of these operations in HDFS, a relevant state-of-the-art distributed file system. We show that the existing decommission mechanism of HDFS is good when the network is the bottleneck, but can be accelerated by up to a factor 3 when storage is the limiting factor. We also show that the commission in HDFS can be substantially accelerated. With the highlights provided by our model, we suggest improvements to speed both operations in HDFS. We discuss how the proposed models can be generalized for distributed file systems with different assumptions and what perspectives are open for the design of efficient malleable distributed file systems.

97 MATHEMATICS AND COMPUTING↗
NASA NTRS2003

Cardea: Providing Support for Dynamic Resource Access in a Distributed Computing Environment

The environment framing the modem authorization process span domains of administration, relies on many different authentication sources, and manages complex attributes as part of the authorization process. Cardea facilitates dynamic access control within this environment as a central function of an inter-operable authorization framework. The system departs from the traditional authorization model by separating the authentication and authorization processes, distributing the responsibility for authorization data and allowing collaborating domains to retain control over their implementation mechanisms. Critical features of the system architecture and its handling of the authorization process differentiate the system from existing authorization components by addressing common needs not adequately addressed by existing systems. Continuing system research seeks to enhance the implementation of the current authorization model employed in Cardea, increase the robustness of current features, further the framework for establishing trust and promote interoperability with existing security mechanisms.

Lepro, Rebekah↗
DOE OSTI2020

Power Grid Simulation Testbed for Transactive Energy Management Systems

To effectively engage demand-side and distributed energy resources (DERs) for dynamically maintaining the electric power balance, the challenges of controlling and coordinating building equipment and DERs on a large scale must be overcome. Although several control techniques have been proposed in the literature, a significant obstacle to applying these techniques in practice is having access to an effective testing platform. Performing tests at scale using real equipment is impractical, so simulation offers the only viable route to developmental testing at scales of practical interest. Existing power-grid testbeds are unable to model individual residential end-use devices for developing detailed control formulations for responsive loads and DERs. Furthermore, they cannot simulate the control and communications at subminute timescales. To address these issues, this paper presents a novel power-grid simulation testbed for transactive energy management systems. Detailed models of primary home appliances (e.g., heating and cooling systems, water heaters, photovoltaic panels, energy storage systems) are provided to simulate realistic load behaviors in response to environmental parameters and control commands. The proposed testbed incorporates software as it will be deployed, and enables deployable software to interact with various building equipment models for end-to-end performance evaluation at scale.

24 POWER TRANSMISSION AND DISTRIBUTION↗
NASA NTRS2003

Resource Management for Real-Time Adaptive Agents

Increased autonomy and automation in onboard flight systems offer numerous potential benefits, including cost reduction and greater flexibility. The existence of generic mechanisms for automation is critical for handling unanticipated science events and anomalies where limitations in traditional control software with fixed, predetermined algorithms can mean loss of science data and missed opportunities for observing important terrestrial events. We have developed such a mechanism by adding a Hierarchical Agent-based ReaLTime technology (HART) extension to our Dynamic Resource Management (DRM) middleware. Traditional DRM provides mechanisms to monitor the realtime performance of distributed applications and to move applications among processors to improve real-time performance. In the HART project we have designed and implemented a performance adaptation mechanism to improve reaktime performance. To use this mechanism, applications are developed that can run at various levels of quality. The DRM can choose a setting for the quality level of an application dynamically at run-time in order to manage satellite resource usage more effectively. A groundbased prototype of a satellite system that captures and processes images has also been developed as part of this project to be used as a benchmark for evaluating the resource management framework A significant enhancement of this generic mission-independent framework allows scientists to specify the utility, or "scientific benefit," of science observations under various conditions like cloud cover and compression method. The resource manager then uses these benefit tables to determine in redtime how to set the quality levels for applications to maximize overall system utility as defined by the scientists running the mission. We also show how maintenance functions llke health and safety data can be integrated into the utility framework. Once thls framework has been certified for missions and successfully flight tested it can be reused with little development overhead for other missions. In contrast, current space missions llke Swift manage similar types of resource trade -off completely with the scientific application code itself, and such code must be re-certified and tested for each mission even if a large portion of the code base is shared. This final report discusses some of the major issues motivating this research effort, provides a literature review of the related work, discusses the resource management framework and ground-based satellite system prototype that has been developed, indicates what work is yet to be performed, and provides a list of publications resulting from this work.

Welch, Lonnie↗
DOE OSTI2020

Post-Disturbance Dynamic Distribution System Restoration with DGs and Mobile Resources

Distributed generations (DGs) can act as emergency power supplies when distribution systems suffer from outages. However, the generation capabilities of DGs are generally limited by a number of factors including weather conditions, fuel limitations, etc. In this context, mobile resources that are able to reallocate resources to desired locations are regarded as important complements to conventional fixed DGs in assisting distribution system restoration. In this paper, a distribution system restoration model with DGs and mobile resources is proposed. Firstly, the dispatch and allocation of mobile resources are modeled with respect to the characteristics of the traffic network. Then the developed mobile resource models are integrated into the distribution system restoration model to co-optimize the scheduling of DGs and mobile resources. Uncertainty factors are managed by a model predictive control approach so that system operators can dynamically adjust the restoration strategy with the up-to-date information. The effectiveness of the proposed method is validated through an IEEE 13-bus test system.

distributed generations (DGs)↗
DOE OSTI2024

From Resilient and Ready to Used and Useful: Managing Temporal and Locational Uncertainty in Electrification, DER Adoption, and Climate Adaptation

Grid planning decisions involve weighing risks against benefits. The best decisions will facilitate development of electrical infrastructure that minimizes risk and maximizes benefits. With the rapidly evolving energy landscape, today's planner must discern new loads and demand cycles; embrace the operational complexity of climate risk; revise settled standards; and anticipate and manage the system impacts of distributed energy resource (DER) adoption. Only by managing the combined uncertainty of these dynamic processes can a planner hope to make effective decisions. Our analysis focuses on the management of temporal and locational uncertainty, and particularly on the risks presented to customers by the mismanagement of the factors that are the sources of these uncertainties.

climate↗
NASA NTRSDate not supplied

Investigation of Intelligent Resource Management for Aviation Communications

The emergence of new aerial vehicles into the airspace as part of new initiatives, such as Advanced Air Mobility (AAM), will place growing demand for spectrum resources to support airspace operations. The traditional approach of using fixed channel allocations within standard service volumes will not allow for dynamic and efficient distribution of resources based on airspace demand; consequently, a new approach to aviation spectrum management will be required to meet the anticipated needs of airspace users. The National Aeronautics and Space Administration (NASA) is investigating the application of advanced concepts to implement a novel spectrum management approach that allows for the intelligent utilization of aviation spectrum throughout the airspace while maintaining the quality of service prescribed by aeronautical standards. This technical investigation evaluates the dynamic assignment of resources for both air-ground and air-air communication links applicable to both the emerging AAM initiative as well as the existing air traffic management system. The performance of the proposed spectrum management concepts will be evaluated using a custom modeling and simulation capability that is currently under development. The implementation of these approaches is anticipated to facilitate increased spectrum utilization efficiency and enhanced airspace capacity, which will better serve the needs of future applications.

Aeronautics↗
NASA NTRSDate not supplied

Investigation of Intelligent Resource Management for Aviation Communications

The emergence of new aerial vehicles into the airspace as part of new initiatives, such as Advanced Air Mobility (AAM), will place growing demand for spectrum resources to support airspace operations. The traditional approach of using fixed channel allocations within standard service volumes will not allow for dynamic and efficient distribution of resources based on airspace demand; consequently, a new approach to aviation spectrum management will be required to meet the anticipated needs of airspace users. The National Aeronautics and Space Administration (NASA) is investigating the application of advanced concepts to implement a novel spectrum management approach that allows for the intelligent utilization of aviation spectrum throughout the airspace while maintaining the quality of service prescribed by aeronautical standards. This technical investigation evaluates the dynamic assignment of resources for both air-ground and air-air communication links applicable to both the emerging AAM initiative as well as the existing air traffic management system. The performance of the proposed spectrum management concepts will be evaluated using a custom modeling and simulation capability that is currently under development. The implementation of these approaches is anticipated to facilitate increased spectrum utilization efficiency and enhanced airspace capacity, which will better serve the needs of future applications.

Eric Knoblock↗
NASA NTRS2002

An Experimental Framework for Executing Applications in Dynamic Grid Environments

The Grid opens up opportunities for resource-starved scientists and engineers to harness highly distributed computing resources. A number of Grid middleware projects are currently available to support the simultaneous exploitation of heterogeneous resources distributed in different administrative domains. However, efficient job submission and management continue being far from accessible to ordinary scientists and engineers due to the dynamic and complex nature of the Grid. This report describes a new Globus framework that allows an easier and more efficient execution of jobs in a 'submit and forget' fashion. Adaptation to dynamic Grid conditions is achieved by supporting automatic application migration following performance degradation, 'better' resource discovery, requirement change, owner decision or remote resource failure. The report also includes experimental results of the behavior of our framework on the TRGP testbed.

Huedo, Eduardo↗