Search NASA⌕ Search

SEARCH · Search NASA

Results for “Data Distribution Service”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 73 records · Page 4

Joint Genome Institute Analysis Workflow Service (JAWS) v2.7

The U.S. Department of Energy Joint Genome Institute (JGI) has developed the JGI Analysis Workflow Service (JAWS) as a distributed framework to run computational workflows across diverse high-performance computing (HPC) and cloud environments. JAWS enhances the reusability, scalability, and robustness of scientific workflows by orchestrating data movement, code execution, and results retrieval across multiple DOE facilities. At its core, JAWS integrates the Cromwell workflow engine to run workflows expressed in the Workflow Description Language (WDL), ensuring portability and interoperability. To provide consistent runtime environments, JAWS employs container technologies such as Shifter, Apptainer, and Docker. Workflow tasks are managed via HTCondor on HPC backends, while Globus ensures secure and efficient data transfer between sites. JAWS is deployed as a multi-site workflow manager across national laboratory computing facilities, with dedicated instances supporting community projects such as the National Microbiome Data Collaborative (NMDC) and KBase. This distributed, service-oriented architecture enables users to "write once, run anywhere," providing scalable, production-quality workflow execution.

Kirton, Edward↗

Hardening DOE R&D Software Tools for Web-based Visualization SBIR Phase I Final Report

Ubiquitous web-based visualization is essential to delivering large-scale data visualization to various stakeholders, from the scientist to the board member. These stakeholders will not tolerate a stalled application or a pop-up window asking them to wait for the processing to complete. They require a responsive and interactive visualization environment with high-quality imagery suitable for detailed analysis and boardroom presentations. At Kitware, Inc., we have accomplished web visualization to this point, leveraging state-of-the-art tools like HTML5, CSS3, SVG, Canvas, and WebGL. Solutions that leverage a combination of these technologies are necessary to handle workloads that vary significantly in data size efficiently. However, it is not always practical to move large data to the web client for visualization. Kitware's ParaView as a Service combines client-side visualization using both distributed processing and remote rendering on big data impractical to move. Existing distributed processing and remote rendering solution's interactivity is below the expectations of web-based applications. Our project examined proposed solutions to the areas outlined above in ParaView as a Service. We have investigated concurrent pipelines, streaming images, progressive rendering, and optimization of algorithms and data movement to address these concerns. For the Phase I project, we completed the proposed work plan. As a result, the project produced three prototypes of essential importance for web visualization and the ParaView as a Service community. We created a simple desktop application for an interactive streamline placement prototype, a web-based interactive streamline placement prototype, and a web-based progressive rendering utilizing raytracing prototype. These prototypes relied on the hardening of emerging software toolkits funded by the Department of Energy (DOE) Advanced Scientific Computing Research (ASCR) program (such as ParaView, VTK-m, and Mochi). We blended these components into web-based visualization prototypes that meet the industry's expectations for interactivity and responsiveness. The Phase I project had four essential focus areas: 1. Develop prototype ParaView as a Service backend server using asynchronous, non-blocking design principles. 2. Develop a prototype web application that uses the ParaView as a Service backend server for remote data visualization. 3. Implement image streaming with encoding/compression and progressive rendering capabilities in the proposed platform. 4. Evaluate the prototype developed and summarize observations, including the challenges and pitfalls of our approach. After our successful completion of Phase I, we are strongly positioned to propose a successful Phase II project.

Geveci, Berk↗

FTS3: Data Movement Service in containers deployed in OKD [Slides]

The File Transfer Service (FTS3) is a data movement service developed at CERN which is used to distribute the majority of the Large Hadron Collider's data across the Worldwide LHC Computing Grid (WLCG) infrastructure. At Fermilab, we have deployed FTS3 instances for Intensity Frontier experiments (e.g. DUNE) to transfer data in America and Europe, using a container-based strategy. In this article we summarize our experience building docker images based on work from the SLATE project (slateci.io) and deployed in OKD, the community distribution of Red Hat OpenShift. Additionally, we discuss our method of certificate management and maintenance utilizing Kubernetes CronJobs. Finally, we also report on the two different configurations currently running at Fermilab, comparing and contrasting a Docker-based OKD deployment against a traditional RPM-based deployment.

97 MATHEMATICS AND COMPUTING↗

FTS3: Data Movement Service in containers deployed in OKD

The File Transfer Service (FTS3) is a data movement service developed at CERN which is used to distribute the majority of the Large Hadron Collider's data across the Worldwide LHC Computing Grid (WLCG) infrastructure. At Fermilab, we have deployed FTS3 instances for Intensity Frontier experiments (e.g. DUNE) to transfer data in America and Europe, using a container-based strategy. In this article we summarize our experience building docker images based on work from the SLATE project (slateci.io) and deployed in OKD, the community distribution of Red Hat OpenShift. Additionally, we discuss our method of certificate management and maintenance utilizing Kubernetes CronJobs. Finally, we also report on the two different configurations currently running at Fermilab, comparing and contrasting a Docker-based OKD deployment against a traditional RPM-based deployment.

Lobato Pardavila, Lorena↗

Building a Wide-Area File Transfer Performance Predictor: An Empirical Study

Wide-area data transfer is central to geographically distributed scientific workflows. Faster delivery of data is important for these workflows. Predictability is equally (or even more) important. With the goal of providing a reasonably accurate estimate of data transfer time to improve resource allocation & scheduling for workflows and enable end-to-end data transfer optimization, we apply machine learning methods to develop predictive models for data transfer times over a variety of wide area networks. To build and evaluate these models, we use 201,388 transfers, involving 759 million files totaling 9 PB transferred, over 115 heavily used source-destination pairs (“edges”) between 135 unique endpoints. We evaluate the models for different retraining frequencies and different window size of history data. In the best case, the resulting models have a median prediction error of ≤21% for 50% of the edges, and ≤32% for 75% of the edges. We present a detailed analysis of these results that provides insights into the cause of some of the high errors. We envision that the performance predictor will be informative for scheduling geo-distributed workflows. The insights also suggest obvious directions for both further analysis and transfer service optimization.

Liu, Zhengchun↗

Cosimulating Integrated Energy Systems With Heterogeneous Digital Twins: Matching a Connected World

Energy system integration promises in-creased resiliency and the unlocking of synergies, while also contributing to our goal of decarbonization. It is enabled by both old and new technologies, glued together with data and digital services. Hydrolyzers, heat pumps, distributed renewable generation, smart buildings, and the digital grid edge are all currently the subject of integration with the power system and the energy sector at large. To plan and operate such a multidisciplinary and multisectoral system properly, insight, tools, and expertise are all needed. Further, this is exactly where the state of the art fails to deliver: tools for integrated energy systems (IESs) are still in their infancy, and many times, even academia treats these sectors separately, producing experts in each of them but not across.

24 POWER TRANSMISSION AND DISTRIBUTION↗

Final Report for DOE Award ER25756

The SciDAC-funded Center for Enabling Distributed Petascale Science (CEDPS) was established to address technical challenges that arise due to the frequent geographic distribution of data producers (in particular, supercomputers and scientific instruments) and data consumers (people and computers) within the DOE laboratory system. Its goal is to produce technical innovations that meet DOE end-user needs for (a) rapid and dependable placement of large quantities of data within a distributed high-performance environment, and (b) the convenient construction of scalable science services that provide for the reliable and high-performance processing of computation and data analysis requests from many remote clients. The Center is also addressing (c) the important problem of troubleshooting these and other related ultra-high-performance distributed activities from the perspective of both performance and functionality

97 MATHEMATICS AND COMPUTING↗

CWS4DB: A Customizable Web Service for Efficient Access to Distributed Nuclear Physics Relational Databases

Tech-X proposes to develop a system providing an efficient access for the High-Energy and Nuclear Physics data stored in distributed heterogeneous relational databases. The system will consist of a generic Web Service, customized client APIs, and tools that facilitate generation of custom client APIs. The access to the data will be mediated by mechanisms that provide data caching and adaptive scheduling of the query plans.

99 GENERAL AND MISCELLANEOUS↗

DataFed

There is a need to easily and uniformly capture, access, enrich, organize, search, and collaborate on working scientific data sets across geographically distributed experiment, computation, and data analytics facilities. DataFed provides a set of services and applications enabling simple and uniform cross-facility scientific data management with federated (distributed) data repositories. This solution greatly reduces effort required by scientist to organize and/or move data between facilities/institutes, Reduces likelihood of misidentification and subsequent misuse of data, and improves scientific reproducibility through deep, domain-specific context and provenance capture.

Stansberry, DaleV.↗

High Penetration PV Deployment in the Arizona Public Service System

In an effort to better understand the impacts of high penetrations of photovoltaic (PV) generators on distribution systems, Arizona Public Service (APS) and its partners have begun work on a multi-year project to develop the tools and knowledgebase needed to safely and reliably integrate high penetrations of utility and residential scale PV. Building upon the APS Community Power Project - Flagstaff Pilot, this project will analyze the impact of PV on a representative feeder in northeast Flagstaff. To quantify and catalog the effects of the estimated 1.5 MW of PV that will be installed on the feeder (both smaller units at homes as well as large, centrally located systems), high-speed weather and electrical data acquisition systems and digital 'smart' meters are being designed and installed to facilitate monitoring and to build and validate comprehensive, high-resolution models of the distribution system. These models will be used to analyze the impacts of the PV on distribution circuit protection systems (including anti-islanding), predict voltage regulation and phase balance issues, and develop volt/var control schemes. The goal of this paper is to provide insight and lessons learned on the early stages of high penetration PV deployment. Primarily focusing on modeling and data acquisition, this paper describes the overall project, early results, and plans for future phases of the project.

14 SOLAR ENERGY↗

Model of Operation-and-Maintenance Costs for Photovoltaic Systems

This article presents a method for calculating costs associated with operation and maintenance (O&M) of photovoltaic (PV) systems. It compiles details regarding the cost and frequency of multiple O&M services to estimate annual O&M costs ($\$$/year) for each year of an analysis period, the net present value ($\$$) of life cycle costs accumulated over the analysis period, and the reserve account amount ($\$$). Here we show that this method is an improvement over the previous averaged or levelized per-unit ($\$$/kW/year) valuations for estimating PV O&M costs, because it allows a detailed selection of services to perform based on system size, market served (e.g., residential, commercial, or utility), type and configuration of system components (e.g., micro-, string or central inverter) and site and environmental conditions (e.g., snow, pollen, bird populations). This model also distinguishes costs that vary from year to year and increase at different rates over time because of heuristic failure distributions (e.g. Weibull or Lognormal distribution) based on actuarial data for many of the services. This cost model was created by the PV O&M Working Group of researchers and industry, sponsored by DOE Solar Energy Technologies Office, and has been published in an on-line version hosted by SunSpec Alliance at apsuite.sunspec.org. A spreadsheet version is included with this paper as supplementary material.

14 SOLAR ENERGY↗

Toward Human-Centric Transportation and Energy Metrics: Influence of Mode, Vehicle Occupancy, Trip Distance, and Fuel Economy

Traditional metrics measuring transportation and energy outcomes can be augmented to better represent impacts on people's lives and systems-level performance. In this context, this study introduces two novel metrics: road capacity (as number of people traveling and accessing services) and energy intensity (as energy use for people traveling and accessing services). Current national-level distributions of available data in the United States for factors contributing to the two new integrated metrics are used as context to evaluate potential outcomes. These factors include vehicle occupancy, mode share, fuel economy, and trip distance. Variations in input values provide insights on how these factors shape efficiencies in road capacity and energy intensity. Parametric sensitivity analysis indicates that the impact of each input depends upon the metric being evaluated. For the human-centered road capacity mobility metric, increasing vehicle occupancy has the largest effect – twice that of increasing mode share for bike, walk, and transit. For the energy intensity mobility metric, the effect of improving fuel economy is the largest. However, when the focus is on accessibility (instead of mobility), for both metrics the effect of lowering average trip distance is the largest. Additionally, a novel interactive tool to visualize the results for various parameter combinations makes the metrics practitioner ready. The findings suggest that the diffusion of new human-centric metrics that benchmark outcomes associated with road capacity and energy may be significant in motivating new sustainable transportation investments and efficient utilization of infrastructure, mobility assets, and services.

ADVANCED PROPULSION SYSTEMS↗

Belle II Conditions Database

The Belle II experiment at KEK is preparing for taking first collision data in early 2018. For the success of the experiment it is essential to have information about varying conditions available to systems worldwide in a fast and efficient manner that is straightforward for both the user and maintainer. The Belle II Conditions Database was designed to make maintenance as easy as possible. To this end, a HTTP REST service was developed with industry-standard tools such as Swagger for the API interface development, Payara for the Java EE application server, and the Hazelcast in-memory data grid for support of scalable caching as well as transparent distribution of the service across multiple sites. On the client side, the online and offline software has to be able to obtain conditions data from the Belle II Conditions Database in a robust and reliable way under very different situations. As such the client side interface to the Belle II Conditions Database has been designed with a variety of access mechanisms which allow the software to be used with and without an internet connection. Different methods to access the payload information are implemented to allow for a high level of customization per site and to simplify testing of new payloads locally. Changes to the conditions data are usually handled transparently but users can actively check whether an object has changed or register callback functions to be called whenever a conditions data object is updated. In addition a command line user interface has been developed to simplify inspection and modification of the database contents.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗

Mesh-free data transfer algorithms for partitioned multiphysics problems: Conservation, accuracy, and parallelism

In this study we analyze and extend mesh-free algorithms for three-dimensional data transfer problems in partitioned multiphysics simulations. We first provide a direct comparison between a mesh-based weighted residual method using the common-refinement scheme and two mesh-free algorithms leveraging compactly supported radial basis functions: one using a spline interpolation and one using a moving least square reconstruction. Through the comparison we assess both the conservation and accuracy of the data transfer obtained from each of the methods. We do so for a varying set of geometries with and without curvature and sharp features and for functions with and without smoothness and with varying gradients. Our results show that the mesh-based and mesh-free algorithms are complementary with cases where each was demonstrated to perform better than the other. We then focus on the mesh-free methods by developing a set of algorithms to parallelize them based on sparse linear algebra techniques. This includes a discussion of fast parallel radius searching in point clouds and restructuring the interpolation algorithms to leverage data structures and linear algebra services designed for large distributed computing environments. The scalability of our new algorithms is demonstrated on a leadership class computing facility using a set of basic scaling studies. Finally, these scaling studies show that for problems with reasonable load balance, our new algorithms for both spline interpolation and moving least square reconstruction demonstrate both strong and weak scalability using more than 100,000 MPI processes with billions of degrees of freedom in the data transfer operation.

97 MATHEMATICS AND COMPUTING↗

PV O&M Cost Model [SWR-21-14]

This tool presents a method for calculating costs associated with the operation and maintenance (O&M) of photovoltaic (PV) systems. The tool compiles details regarding the cost and frequency of multiple O&M services to estimate annual O&M costs ($/year) for each year of an analysis period, the net present value ($) of life cycle costs accumulated over the analysis period, and the reserve account amount ($) that might be required to fund unexpected repairs. The reserve account includes parts inventory and is important for providing a source of funds to make repairs quickly and avoid lost production. This method allows a detailed selection of services to perform based on system size, market served (e.g., residential, commercial, or utility), type and configuration of system components (e.g., micro-, string, or central inverter), and site and environmental conditions (e.g., pollen, bird populations) which is an improvement over simple per unit valuations of O&M costs ($/kW/year). This model also distinguishes costs that vary from year to year and increase at different rates over time as modeled by heuristic failure distributions (e.g., Weibull or Lognormal distribution) based on actuarial data for many of the services.

Walker, Andy↗

Obtaining and Managing Answer Quality for Online Data-Intensive Services

Online data-intensive (OLDI) services use anytime algorithms to compute over large amounts of data and respond quickly. Interactive response times are a priority, so OLDI services parallelize query execution across distributed software components and return best effort answers based on the data so far processed. Omitted data from slow components could lead to better answers, but tracing online how much better the answers could be is difficult. Here, we introduce Ubora, a design method to measure the effect of slow-running components on the quality of answers. Ubora randomly samples online queries and executes them a second time. The first online execution omits data from slow components and provides interactive answers. The second execution uses mature results from intermediate components completed after the online execution finishes. Ubora uses memoization to speed up mature executions by replaying network messages exchanged between components. Our systems-level implementation works for a wide range of services, including Hadoop/Yarn, Apache Lucene, the EasyRec Recommendation Engine, and the OpenEphyra question-answering system. Ubora computes answer quality with more mature executions per second than competing approaches that do not use memoization. With Ubora, we illustrate that answer quality is effective at guiding online admission control. While achieving the same answer quality on high-priority queries, our adaptive controller had 55% higher peak throughput on low-priority queries than a competing controller guided by the rate of timeouts.

answer quality↗

Comparison between the Strength Levels of Baseline Nuclear-Grade Graphite and Graphite Irradiated in AGC-2

This report details the initial comparison of mechanical strength properties between the cylindrical nuclear-grade graphite specimens irradiated in the second Advanced Graphite Creep (AGC-2) experiment with the established baseline, or unirradiated, mechanical properties compiled in the Baseline Graphite Characterization program. The overall comparative analysis will describe the development of an appropriate test protocol for irradiated specimens, the execution of the mechanical tests on the AGC-2 sample population, and will further discuss the data in terms of developing an accurate irradiated property distribution in the limited amount of irradiated data by leveraging the considerably larger property datasets being captured in the Baseline Graphite Characterization program. Integrating information on the inherent variability in nuclear-grade graphite with more complete datasets is one of the goals of the VHTR Graphite Materials program. Between “sister” specimens, or specimens with the same geometry machined from the same sub-block of graphite from which the irradiated AGC specimens were extracted, and the Baseline datasets, a comprehensive body of data will exist that can provide both a direct and indirect indication of the full irradiated property distributions that can be expected of irradiated nuclear-grade graphite while in service in a VHTR system. While the most critical data will remain the actual irradiated property measurements, expansion of this data into accurate distributions based on the inherent variability in graphite properties will be a crucial step in qualifying graphite for nuclear use as a structural material in a VHTR environment.

21 SPECIFIC NUCLEAR REACTORS AND ASSOCIATED PLANTS↗

Time-Based Data Streams: Fundamental Concepts for a Data Resource for Streams

Real time data, which we call data streams, are readings from instruments, environmental, bodily or building sensors that are generated at regular intervals and often, due to their volume, need to be processed in real time. Often a single pass is all that can be made on the data, and a decision to discard or keep the instance is made on the spot. Too, the stream is for all practical purposes indefinite, so decisions must be made on incomplete knowledge. This notion of data streams has a different set of issues from a file, for instance, that is byte streamed to a reader. The file is finite, so the byte stream is becomes a processing convenience more than a fundamentally different kind of data. Through the duration of the project we examined three aspects of streaming data: the first, techniques to handle streaming data in a distributed system organized as a collection of web services, the second, the notion of the dashboard and real time controllable analysis constructs in the context of the Fermi Tevatron Beam Position Monitor, and third and finally, we examined provenance collection of stream processing such as might occur as raw observational data flows from the source and undergoes correction, cleaning, and quality control. The impact of this work is severalfold. We were one of the first to advocate that streams had little value unless aggregated, and that notion is now gaining general acceptance. We were one of the first groups to grapple with the notion of provenance of stream data also.

97 MATHEMATICS AND COMPUTING↗