Search NASA⌕ Search

SEARCH · Search NASA

Results for “Data Distribution Service”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 523 records · Page 29

Priority-enabled Load Balancing for Dispersed Computing

Opportunistic managed access to local in-network compute resources can improve the performance of distributed applications and reduce the dependence on shared network resources. Instead of backhauling application data to a centralized cloud data center for processing, networked services may be adaptively and continuously dispersed into shared compute resources that are closer to the source of need. While this approach has several benefits, support for mission-aware access to computation is often an afterthought, and is implemented as a brittle extension over traditional load-balancer solutions.In this work, we investigate the design of two priority-aware resource allocation strategies and two load-balancing dispatching strategies as first class citizens in an open-source dispersed computing middleware. We present a control theoretic analysis of these load-balancing primitives to identify weaknesses and strengths in our design, and recommend future directions. In parallel, we prototype two priority-aware allocation algorithms to validate our priority predictions. In initial experiments our prototype shows substantial gains in processing prioritized load. Finally, we make our source-code and experimental configurations open source.

Paulos, Aaron↗

An Execution Service for Grid Computing

This paper describes the design and implementation of the IPG Execution Service that reliably executes complex jobs on a computational grid. Our Execution Service is part of the IPG service architecture whose goal is to support location-independent computing. In such an environment, once n user ports an npplicntion to one or more hardware/software platfrms, the user can describe this environment to the grid the grid can locate instances of this platfrm, configure the platfrm as required for the application, and then execute the application. Our Execution Service runs jobs that set up such environments for applications and executes them. These jobs consist of a set of tasks for executing applications and managing data. The tasks have user-defined starting conditions that allow users to specih complex dependencies including task to execute when tasks fail, afiequent occurrence in a large distributed system, or are cancelled. The execution task provided by our service also configures the application environment exactly as specified by the user and captures the exit code of the application, features that many grid execution services do not support due to dflculties interfacing to local scheduling systems.

Smith, Warren↗

CPS Testbed Architectures for WAMPAC using Industrial Substation and Control Center Platforms and Attack-Defense Evaluation

Advanced persistent threats and cyberattacks can impact wide-area monitoring, protection, and control (WAMPAC) system operation. Many cyber-physical system (CPS) testbeds have been developed for attack-defense experimentation and attack-resiliency tools evaluation for WAMPAC, but they are limited to a simulation-and-emulation based environment. This paper presents a quasi-realistic CPS attack-defense testbed-based framework for WAMPAC applications using the industrial substation and control center platforms such as eTerra integrated with the hardware-in-the-loop CPS smart grid testbed available at Iowa State University. The proposed framework includes various combinations of industry-grade substation and control center platforms, communication topologies, real-time digital simulators, and a novel cyber-physical distributed intrusion-and-anomaly detection system (D-IADS) for WAMPAC applications. The D-IADS includes a master at the control center and geographically distributed sensor devices at each substation. Each D-IADS sensor deployed at a substation or control center network monitors ingress and egress traffic, detect intrusions, and dispatch alerts to the D-IADS master. The D-IADS master centrally monitors and analyze the alerts and controls D-IADS sensors. We considered an EMP60 synthetic CPS grid as a case study to demonstrate the framework and proposed D-IADS for WAMPAC applications against cyberattack vectors such as Man-in-the-Middle DNP3 attack, denial-of-service, and data-integrity attacks.

24 POWER TRANSMISSION AND DISTRIBUTION↗

Recent evolutions in the LTDP CDF project

In the latest years, CNAF (the national center of the Italian Institute for Nuclear Physics INFN dedicated to Research and Development on Information and Communication Technologies) has been working on the Long Term Data Preservation (LTDP) project for the CDF experiment, active at Fermilab from 1990 to 2011. The main aims of the project are to protect the most relevant part of the CDF RUN-2 data collected between 2001 and 2011 and already stored on tape at CNAF (4 PB), as well as to ensure the availability and the access to the analysis facility to those data over time. Lately, the CDF database, hosting information about CDF datasets such as their structure, file locations and metadata, has been imported from Fermilab to CNAF. Also, the Sequential Access via Metadata (SAM) station data handling tool for CDF data management, that allows to manage data transfers and to retrieve information from the CDF database, has been properly installed and configured at CNAF. This was a fundamental step in the perspective of a complete decommissioning of CDF services on Fermilab side. An access system has been designed and tested to submit CDF analysis jobs, using CDF software distributed via CERN Virtual Machine File System (CVMFS) and requesting delivery of CDF files stored on CNAF tapes, as well as data present only on Fermilab storage archive. Moreover, the availability and the correctness of all CDF data stored on CNAF tapes has been verified. This paper describes all these recent evolutions in detail, presenting the future plans for the LTDP project at CNAF.

Fornari, Federico↗

iez: Resource Contention Aware Load Balancing for Large-Scale Parallel File Systems

Parallel I/O performance is crucial to sustaining scientific applications on large-scale High-Performance Computing (HPC) systems. However, I/O load imbalance in the underlying distributed and shared storage systems can significantly reduce overall application performance. There are two conflicting challenges to mitigate this load imbalance: (i) optimizing systemwide data placement to maximize the bandwidth advantages of distributed storage servers, i.e., allocating I/O resources efficiently across applications and job runs; and (ii) optimizing client-centric data movement to minimize I/O load request latency between clients and servers, i.e., allocating I/O resources efficiently in service to a single application and job run. Moreover, existing approaches that require application changes limit wide-spread adoption in commercial or proprietary deployments. We propose iez, an “end-to-end control plane” where clients transparently and adaptively write to a set of selected I/O servers to achieve balanced data placement. Our control plane leverages realtime load information for distributed storage server global data placement while our design model leverages trace-based optimization techniques to minimize I/O load request latency between clients and servers. We evaluate our proposed system on an experimental cluster for two common use cases: synthetic I/O benchmark IOR for large sequential writes and a scientific application I/O kernel, HACC-I/O. Results show read and write performance improvements of up to 34% and 32%, respectively, compared to the state of the art.

Wadhwa, Bharti↗

Design prediction for long term stress rupture service of composite pressure vessels

Extensive stress rupture studies on glass composites and Kevlar composites were conducted by the Lawrence Radiation Laboratory beginning in the late 1960's and extending to about 8 years in some cases. Some of the data from these studies published over the years were incomplete or were tainted by spurious failures, such as grip slippage. Updated data sets were defined for both fiberglass and Kevlar composite stand test specimens. These updated data are analyzed in this report by a convenient form of the bivariate Weibull distribution, to establish a consistent set of design prediction charts that may be used as a conservative basis for predicting the stress rupture life of composite pressure vessels. The updated glass composite data exhibit an invariant Weibull modulus with lifetime. The data are analyzed in terms of homologous service load (referenced to the observed median strength). The equations relating life, homologous load, and probability are given, and corresponding design prediction charts are presented. A similar approach is taken for Kevlar composites, where the updated stand data do show a turndown tendency at long life accompanied by a corresponding change (increase) of the Weibull modulus. The turndown characteristic is not present in stress rupture test data of Kevlar pressure vessels. A modification of the stress rupture equations is presented to incorporate a latent, but limited, strength drop, and design prediction charts are presented that incorporate such behavior. The methods presented utilize Cartesian plots of the probability distributions (which are a more natural display for the design engineer), based on median normalized data that are independent of statistical parameters and are readily defined for any set of test data.

Robinson, Ernest Y.↗

Evolution of Information Management at the GSFC Earth Sciences (GES) Data and Information Services Center (DISC): 2006-2007

Increasingly sophisticated National Aeronautics and Space Administration (NASA) Earth science missions have driven their associated data and data management systems from providing simple point-to-point archiving and retrieval to performing user-responsive distributed multisensor information extraction. To fully maximize the use of remote-sensor-generated Earth science data, NASA recognized the need for data systems that provide data access and manipulation capabilities responsive to research brought forth by advancing scientific analysis and the need to maximize the use and usability of the data. The decision by NASA to purposely evolve the Earth Observing System Data and Information System (EOSDIS) at the Goddard Space Flight Center (GSFC) Earth Sciences (GES) Data and Information Services Center (DISC) and other information management facilities was timely and appropriate. The GES DISC evolution was focused on replacing the EOSDIS Core System (ECS) by reusing the In-house developed disk-based Simple, Scalable, Script-based Science Product Archive (S4PA) data management system and migrating data to the disk archives. Transition was completed in December 2007

Kempler, Steven↗

Meteorological Support Interface Control Working Group (MSICWG) Instrumentation, Data Format, and Networks Document

The purpose of this document is to provide an overview of instrumentation discussed at the Meteorological Interface Control Working Group (MSICWG), a reference for data formats currently used by members of the group, a summary of proposed formats for future use by the group, an overview of the data networks of the group's members. This document will be updated as new systems are introduced, old systems are retired, and when the MSICWG community necessitates a change to the formats. The MSICWG consists of personnel from the National Aeronautics and Space Administration (NASA) Kennedy Space Center (KSC), NASA Marshall Space Flight Center (MSFC), NASA Johnson Space Center (JSC), National Oceanic and Atmospheric Administration National Weather Service Spaceflight Meteorology Group (SMG), and the United States Air Force (USAF) 45th Space Wing and Weather Squadron. The purpose of the group is to coordinate the distribution of weather related data to support NASA space launch related activities.

Brenton, James↗

Grid accounting service: state and future development

During the last decade, large-scale federated distributed infrastructures have been continually developed and expanded. One of the crucial components of a cyber-infrastructure is an accounting service that collects data related to resource utilization and identity of users using resources. The accounting service is important for verifying pledged resource allocation per particular groups and users, providing reports for funding agencies and resource providers, and understanding hardware provisioning requirements. It can also be used for end-to-end troubleshooting as well as billing purposes. In this work we describe Gratia, a federated accounting service jointly developed at Fermilab and Holland Computing Center at University of Nebraska-Lincoln. The Open Science Grid, Fermilab, HCC, and several other institutions have used Gratia in production for several years. The current development activities include expanding Virtual Machines provisioning information, XSEDE allocation usage accounting, and Campus Grids resource utilization. We also identify the direction of future work: improvement and expansion of Cloud accounting, persistent and elastic storage space allocation, and the incorporation of WAN and LAN network metrics.

Levshina, T.↗

Moving Beyond Direct Load Control: A Maturity Model for Realizing the Promise of Demand Flexibility

This report discusses a new maturity model that regulators and utilities can use to guide and expand demand flexibility programs and enable the resources to provide more grid services. The model has six demand flexibility categories: planning and design; customer engagement; program operations; evaluation, measurement and verification; distributed energy resource orchestration; and data infrastructure. Within each category, capabilities are identified and described on a maturity scale that ranges from performing below expectations to improving on best practices.

24 POWER TRANSMISSION AND DISTRIBUTION↗

Satellite Embedding-Based Population Imputation for Areas with Missing Building Footprint Data: A Computer Vision-Based Approach

High-resolution population modeling is important for supporting effective decision-making across diverse sectors. LandScan Mosaic generates population estimates at the level of individual buildings and aggregates them to 3 arc-second grids, and this approach performs well in regions where building footprint data are comprehensive and reliable. However, large portions of the globe still suffer from incomplete, sparse, or entirely missing building stock datasets, creating a structural limitation for strictly building-based population models. To address this research gap, this study proposes a computer vision-based framework that employs Google Earth Engine satellite embeddings and UNet, which allows us to directly impute grid-level population estimates in building-data-deficient areas. Applied to Taiwan as a case study, the framework achieved strong predictive performance with R$^{2}$ of 0.89, RMSE of 18.70, and MAE of 8.41, outperforming traditional machine learning approaches. Notably, the proposed framework effectively addressed building false-positive errors inherent in Global Human Settlement Layer (GHSL) data, correctly identifying uninhabited areas that were erroneously classified as populated. The framework also offers significant advantages for global population mapping, particularly in terms of scalability and temporal consistency, thereby extending the coverage and accuracy of high-resolution population products in data-scarce regions worldwide. Urban planners, decision makers, and related stakeholders can obtain granular population distributions to support more accurate and targeted infrastructure investment, service delivery, resource allocation, and risk assessment decisions.

97 MATHEMATICS AND COMPUTING↗

Grand Valley Ecological Forecasting : Assessing Trends in Pinyon-Juniper Habitat Relative to Drought, Beetle Infestation, Wildland Fires, and Treatment to Plan Future Management Strategies

Drought, beetle infestation, and more frequent wildfires are changing the composition and distribution of the pinyon-juniper woodland and sagebrush ecosystems of the Grand Valley in western Colorado. Land managers must consider short- and long-term goals for restoration as well as budgetary and personnel limitations after such disturbances. Satellite remote sensing can provide long-term and continuous vegetation monitoring to assess where restoration is needed most and where treatment practices are most likely to succeed. Harnessing Earth observation data, our team set out to observe trends in disturbances and the distribution of pinyon-juniper woodlands and sagebrush communities of National Park Service (NPS) and Bureau of Land Management (BLM) lands within the Grand Valley. We used imagery from the Moderate Resolution Imaging Spectroradiometer (MODIS) aboard the Aqua and Terra satellites, and Landsat sensors to map landcover change within these ecosystems from 1984-2021. Additionally, we analyzed disturbed areas and treatment sites to understand their effect on long-term vegetation health and recovery. Results showed that pinyon-juniper woodland has expanded in extent more than other landcover types, indicating woody encroachment into sagebrush ecosystems. We also found that wildfire disturbances had lasting impacts up to 20 years post-disturbance. Pre-fire treatment practices showed mixed results regarding their effectiveness at stopping fires and promoting post-fire recovery. These results will provide context to public land managers in the Grand Valley when developing management plans, ecological monitoring locations, and implementing treatment practices for future disturbances.

Garrett Powers↗

DATA-DRIVEN ALGORITHMS FOR DISTRIBUTION SYSTEM OPERATION AND CONTROL

Transactive energy systems are an alternative solution for the problems that distribution system operators face due to an increase in the use of distributed energy resources(DERs) and resulting much required scalability in managing active distribution system. To meet the reliability, sustainability and security requirement, ecient management and control of DERs including distributed generators, energy storage systems, and demand response is necessary. DERs can provide services such as reactive power compensation, voltage control, energy balancing, peak load reduction. These edge resources can also be used for system restoration after the extreme events within regulatory bound and enabling resiliency of the electric grid. While there are numbers of related work focused on solving these problems, availability of additional data resources has not been utilized to its potential to enable the reliable, sustainable and resilient electric grid. In order to address these challenges, data-driven algorithms have been proposed in this dissertation, including a) Volt/Var control, b) activity aware demand response and c) estimation of behind the meter distributed energy resources. Results indicate satisfactory performance of the developed algorithms.

ZHANG, YUE↗

Intelligent Systems Technologies and Utilization of Earth Observation Data

The addition of raw data and derived geophysical parameters from several Earth observing satellites over the last decade to the data held by NASA data centers has created a data rich environment for the Earth science research and applications communities. The data products are being distributed to a large and diverse community of users. Due to advances in computational hardware, networks and communications, information management and software technologies, significant progress has been made in the last decade in archiving and providing data to users. However, to realize the full potential of the growing data archives, further progress is necessary in the transformation of data into information, and information into knowledge that can be used in particular applications. Sponsored by NASA s Intelligent Systems Project within the Computing, Information and Communication Technology (CICT) Program, a conceptual architecture study has been conducted to examine ideas to improve data utilization through the addition of intelligence into the archives in the context of an overall knowledge building system (KBS). Potential Intelligent Archive concepts include: 1) Mining archived data holdings to improve metadata to facilitate data access and usability; 2) Building intelligence about transformations on data, information, knowledge, and accompanying services; 3) Recognizing the value of results, indexing and formatting them for easy access; 4) Interacting as a cooperative node in a web of distributed systems to perform knowledge building; and 5) Being aware of other nodes in the KBS, participating in open systems interfaces and protocols for virtualization, and achieving collaborative interoperability.

Ramapriyan, H. K.↗

Spacecraft-Initiated Scheduling of Commercial Communications Services

We propose a software framework enabling a spacecraft in Earth orbit to schedule its own access to space communications services via ground stations and relay satellites. All operations are automated, without a human in the loop, allowing for highly responsive service to meet time-varying needs. We describe a modular software architecture allowing the framework to interface with the scheduling systems of several providers - both government and commercial. Specifically, we present the results of an experiment with Amazon Web Services (AWS) Ground Station, an operational commercial service provider. Proposed software executed scheduling of a ground station antenna with minimal lead time (less than 20 minutes before the start of a pass). During the contact AWS cloud infrastructure was used to demodulate, process, and distribute data products from an orbiting satellite. We believe this work demonstrates the framework's ability to interface with additional providers and potential to integrate more complex decision-making in the future to schedule optimal contacts.

Scheduling↗

Event recognition using signal spectrograms in long pulse experiments

As discharge duration increases, real-time complex analysis of the signal becomes more important. In this context, data acquisition and processing systems must provide models for designing experiments which use event oriented plasma control. One example of advanced data analysis is signal classification. The off-line statistical analysis of a large number of discharges provides information to develop algorithms for the determination of the plasma parameters from measurements of magnetohydrodinamic waves, for example, to detect density fluctuations induced by the Alfven cascades using morphological patterns. The need to apply different algorithms to the signals and to address different processing algorithms using the previous results necessitates the use of an event-based experiment. The Intelligent Test and Measurement System platform is an example of architecture designed to implement distributed data acquisition and real-time processing systems. The processing algorithm sequence is modeled using an event-based paradigm. The adaptive capacity of this model is based on the logic defined by the use of state machines in SCXML. The Intelligent Test and Measurement System platform mixes a local multiprocessing model with a distributed deployment of services based on Jini.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗

The NPOESS Preparatory Project (NPP) Science Data Segment (SDS) Data Depository and Distribution Element (SD3E) System Architecture

The National Polar-orbiting Operational Environmental Satellite System (NPOESS), the U.S. Government's future low-Earth orbiting satellite system, will monitor global weather and environmental conditions. Serving as a risk reduction for NPOESS, the NPOESS Preparatory Project (NPP) will provide remotely sensed atmospheric, land, ocean, ozone, and sounder data that will serve the meteorological and global climate change scientific communities. The National Aeronautics and Space Administration (NASA) NPP Science Data Segment's (SDS) primary role is to independently assess the quality of the NPP science and environmental data records for their ability to support climate research. The SDS is composed of nine elements; an input element that receives data from the operational agencies and acts as a buffer, a calibration analysis element, five elements devoted to measurement based quality assessment, an element used to test algorithmic improvements, and an element that provides overall science direction. Each element requires a set of sensor specific science data products for their evaluation. There are four NPP sensors that will be flown on the NPP observatory. They are the Visible Infrared Imagining Radiometer Suite (VIIRS), the Advanced Technology Microwave Sounder (ATMS), the Cross-Track Infrared Sounder (CrIS), and the Ozone Mapper/Profiler Suite (OMPS). It is estimated that these four sensors combined will make daily data requests for approximately six terabytes of NPP science products from the operational data providers. As a result, issues associated with duplicate data requests, data transfers of large volumes of diverse products, and data transfer failures raised concerns with respect to the network traffic and bandwidth consumption. Therefore, a central data broker system for receiving and buffering data requests and data products for the SDS was developed. The data element for this system is called the SDS Data Depository and Distribution Element (SD3E). It supports science mission data assessment by assuring the timely and validated acquisition and subsequent transfer of the NPP Science Mission data to the SDS Elements and NPP Science Team. The six science elements that interface with the SD3E span across the NASA Goddard Space Flight Center (GSFC), the NASA Jet Propulsion Laboratory (JPL), and the University of Wisconsin. As the primary communication vehicle for the science elements and science team, the SD3E has an interface to the operational data providers: National Environment Satellite, Data, and Information Service (NESDIS) Interface Data Processing System (IDPS) and the National Oceanic Atmospheric Administration's (NOAA) Comprehensive Large Array-data Stewardship system (CLASS) Archive Data System (ADS), that are responsible for product generation and archive and distribution respectively. The SD3E is designed to be a semi-customizable and semi-automated system. This system is designed to provide flexibility and ease of use for the science users in accessing the latest data products by creating a rolling data cache that temporarily stores the products locally before transferring the data to the SDS Measurement based elements for the land, ocean, atmosphere, sounder, and ozone. This paper describes the design and architecture of one of the nine SDS elements, the SD3E, and how this system has provided a mechanism for efficient data exchange, how it has helped in alleviating some of the network traffic and usage, and how it has contributed to reducing operational costs.

Ho, Evelyn L.↗

Integrated System and Application Continuous Performance Monitoring and Analysis Capability

Scientific applications run on high-performance computing (HPC) systems are critical for many national security missions within Sandia and the NNSA complex. However, these applications often face performance degradation and even failures that are challenging to diagnose. To provide unprecedented insight into these issues, the HPC Development, HPC Systems, Computational Science, and Plasma Theory & Simulation departments at Sandia crafted and completed their FY21 ASC Level 2 milestone entitled "Integrated System and Application Continuous Performance Monitoring and Analysis Capability." The milestone created a novel integrated HPC system and application monitoring and analysis capability by extending Sandia's Kokkos application portability framework, Lightweight Distributed Metric Service (LDMS) monitoring tool, and scalable storage, analysis, and visualization pipeline. The extensions to Kokkos and LDMS enable collection and storage of application data during run time, as it is generated, with negligible overhead. This data is combined with HPC system data within the extended analysis pipeline to present relevant visualizations of derived system and application metrics that can be viewed at run time or post run. This new capability was evaluated using several week-long, 290-node runs of Sandia's ElectroMagnetic Plasma In Realistic Environments ( EMPIRE ) modeling and design tool and resulted in 1TB of application data and 50TB of system data. EMPIRE developers remarked this capability was incredibly helpful for quickly assessing application health and performance alongside system state. In short, this milestone work built the foundation for expansive HPC system and application data collection, storage, analysis, visualization, and feedback framework that will increase total scientific output of Sandia's HPC users.

97 MATHEMATICS AND COMPUTING↗