SEARCH · Search NASA
Results for “Data Movement”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
Transport-Support Workflow Composition and Optimization for Big Data Movement in High-Performance Networks
Not provided.
Moving from low level eye movement data to meaningful content in dynamic environments.
Abstract not provided.
Commutative Data Reordering: A New Technique to Reduce Data Movement Energy on Sparse Inference Workloads.
Abstract not provided.
Empirical Study of Molecular Dynamics Workflow Data Movement: DYAD vs. Standard I/O Systems
Explore the source record for details and available documents.
Analytics4X: General-Purpose Framework for Analysis and Optimization of HPC Data Movement
Explore the source record for details and available documents.
GROWTH on S190426c: Real-time Search for a Counterpart to the Probable Neutron Star–Black Hole Merger using an Automated Difference Imaging Pipeline for DECam
The discovery of a transient kilonova following the gravitational-wave (GW) event GW170817 highlighted the critical need for coordinated rapid and wide-field observations, inference, and follow-up across the electromagnetic spectrum. In the southern hemisphere, the Dark Energy Camera (DECam) on the Blanco 4 m telescope is well suited to this task, as it is able to cover wide fields quickly while still achieving the depths required to find kilonovae like the one accompanying GW170817 to ~500 Mpc, the binary neutron star (NS) horizon distance for current generation of LIGO/Virgo collaboration (LVC) interferometers. Here, as part of the multi-facility follow-up by the Global Relay of Observatories Watching Transients Happen collaboration, we describe the observations and automated data movement, data reduction, candidate discovery, and vetting pipeline of our target-of-opportunity DECam observations of S190426c, the first possible NS-black hole merger detected in GWs. Starting 7.5 hr after S190426c, over 11.28 hr of observations, we imaged an area of 525 deg 2 ( r band) and 437 deg 2 ( z band); this was 16.3% of the total original localization probability, and nearly all of the probability visible from the southern hemisphere. The machine-learning-based pipeline was optimized for fast turnaround, delivering transients for human vetting within 17 minutes, on average, of shutter closure. We reported nine promising counterpart candidates 2.5 hr before the end of our observations. One hour after our data-taking ended (roughly 20 hr after the announcement of S190426c), LVC released a refined skymap that reduced the probability coverage of our observations to 8.0%, demonstrating a critical need for localization updates on shorter (~hour) timescales. Our observations yielded no detection of a bona fide counterpart to m z = 21.7 and m r = 22.2 at the 5σ level of significance, consistent with the refined LVC positioning. Here, we view these observations and rapid inferencing as an important real-world test for this novel end-to-end wide-field pipeline.
Automatic Data Distribution for CFD Applications on Structured Grids
Development of HPF versions of NPB and ARC3D showed that HPF has potential to be a high level language for parallelization of CFD applications. The use of HPF requires an intimate knowledge of the applications and a detailed analysis of data affinity, data movement and data granularity. Since HPF hides data movement from the user even with this knowledge it is easy to overlook pieces of the code causing low performance of the application. In order to simplify and accelerate the task of developing HPF versions of existing CFD applications we have designed and partially implemented ADAPT (Automatic Data Distribution and Placement Tool). The ADAPT analyzes a CFD application working on a single structured grid and generates HPF TEMPLATE, (RE)DISTRIBUTION, ALIGNMENT and INDEPENDENT directives. The directives can be generated on the nest level, subroutine level, application level or inter application level. ADAPT is designed to annotate existing CFD FORTRAN application performing computations on single or multiple grids. On each grid the application can considered as a sequence of operators each applied to a set of variables defined in a particular grid domain. The operators can be classified as implicit, having data dependences, and explicit, without data dependences. In order to parallelize an explicit operator it is sufficient to create a template for the domain of the operator, align arrays used in the operator with the template, distribute the template, and declare the loops over the distributed dimensions as INDEPENDENT. In order to parallelize an implicit operator, the distribution of the operator's domain should be consistent with the operator's dependences. Any dependence between sections distributed on different processors would preclude parallelization if compiler does not have an ability to pipeline computations. If a data distribution is "orthogonal" to the dependences of an implicit operator then the loop which implements the operator can be declared as INDEPENDENT.
Complex Network Analysis and Intelligent Monitoring Platform
Data Products has particular interest in developing an efficient network analysis and monitoring system that utilizes advanced machine learning methods to mine complex network data of the type generated within large data centers, with an objective to provide tangible business-usable intelligence. The Laboratory is currently the lead institution for a U.S. Department of Energy Office of Science (DOE/SC) network research project called BigData Express (BDE). The project seeks to provide an orchestration service for high-performance data movement of data sets for large-scale science. This collaboration is believed to potentially be a highly symbiotic one, with Data Products platform development benefiting from the rich sets of network data available in the BDE development environment, and BDE benefiting from the Data Products machine learning analysis of BDE-generated data movement.
Compendium of Tour-Based Freight Modeling Literature
Transportation officials, metropolitan planning organizations (MPOs), and companies of varying size can all benefit from freight movement data. Obtaining freight movement data and route choice information is crucial to planning, improving current and future transportation infrastructure, locating businesses, improving efforts to streamline goods movement, especially urban goods movement, and reducing energy usage within freight transportation. Obtaining data on freight movements can also help in better planning transportation infrastructure to improve safety. Also, with increasing demands and changing consumption habits, mobility being offered as a service, and with new technologies such as 3-D printing, the landscape of freight planning is changing, as businesses compete to meet the needs of a growing customer base, while transportation planners aim at meeting these needs by maintaining and updating the transportation infrastructure.For decades, transportation research has focused on passenger travel behavior. Much of this research has examined route decision choice and, more recently, trip chaining of personal vehicles, both individually and collectively. Discrete choice modeling has commonly been used to better understand passenger movement and predict route decisions based on geographic and demographic data from Census data and personal surveys, and micro-simulation has been used to model estimated movements through the transportation network. However, there is a growing interest in examining freight movements, and although previous research has been done to look at freight route choice regarding individual trips, there is increasing interest in freight tours. This paper is an overview of the current tour-based freight modeling methodology found in the literature. Background information on tour-based freight modeling is provided, as well as common modeling methods, data sources, and findings. Consideration is given to new technologies and safety considerations excluded from the current literature. Suggestions are provided for future tour-based freight modeling studies.
Network performance analysis for HPC datacenters (net_perf) v1.0
The software has two main features: (1) identify data movement trends in HPC data centers that use network flow monitoring (2) analyze the performance of individual data flows under the existing data movement management strategy and identify performance bottlenecks that impede timely data availability for science workflows. Its main advantage is that it is tailored for HPC network traffic by considering HPC data movement management intricacies.
RISE: Reducing I/O Contention in Staging-based Extreme-Scale In-situ Workflows
While in-situ workflow formulations have addressed some of the data-related challenges associated with extreme-scale scientific workflows, these workflows involve complex interactions and different modes of data exchange. In the context of increasing system complexity, such workflows present significant resource management challenges, requiring complex cost-performance tradeoffs. This paper presents RISE, an intelligent staging-based data management middleware, which builds on the DataSpaces framework and performs intelligent scheduling of data management operations to reduce I/O contention. In RISE, data are always written immediately to local buffers to reduce the effect of the transfer impact upon application performance. RISE identifies applications’ data access patterns and moves data towards data consumers only when the network is expected to be idle, reducing the impact of asynchronous background data movement upon critical data read/write requests. Here, we experimentally demonstrate that RISE can take advantage of staging nodes to offload data during writes without degrading application data movement performance.
An Analysis Workflow-Aware Storage System for Multi-Core Active Flash Arrays
Here, the need for novel data analysis is urgent in the face of a data deluge from modern applications. Traditional approaches to data analysis incur significant data movement costs, moving data back and forth between the storage system and the processor. Emerging Active Flash devices enable processing on the flash, where the data already resides. An array of such Active Flash devices allows us to revisit how analysis workflows interact with storage systems. By seamlessly blending together the flash storage and data analysis, we create an analysis workflow-aware storage system, AnalyzeThis. Our guiding principle is that analysis-awareness be deeply ingrained in each and every layer of the storage, elevating data analyses as first-class citizens, and transforming AnalyzeThis into a potent analytics-aware appliance. To evaluate the AnalyzeThis system, we have adopted both emulation and simulation approaches. In particular, we have evaluated AnalyzeThis by implementing the AnalyzeThis storage system on top of the Active Flash Array's emulation platform. We have also implemented an event-driven AnalyzeThis simulator, called AnalyzeThisSim, which allows us to address the limitations of the emulation platform, e.g., performance impact of using multi-core SSDs. The results from our emulation and simulation platforms indicate that AnalyzeThis is a viable approach for expediting workflow execution and minimizing data movement.
Characterization of bandwidth requirements of algorithms for extreme scale science
Technology trends have resulted in widely different rates of improvement in computational throughput as compared to data movement rates in computer systems. With future systems, the cost of data movement through the memory hierarchy is expected to become even more dominant relative to the cost of performing arithmetic operations, both in terms of time and energy. Therefore, optimizing data access costs will become ever more critical in the coming years. However, unlike the well understood model of computational complexity in terms of the number of elementary operations to execute an algorithm, the inherent data-movement complexity of computations is unknown for most algorithms. Given the crucial importance of optimizing data access costs in scalable parallel systems, it is of great interest to develop techniques for understanding the inherent data movement complexity of an algorithm. In this project, we focus on this fundamental and significant problem.
Data Jockey: Automatic Data Management for HPC Multi-Tiered Storage Systems
We present the design and implementation of Data Jockey, a data management system for HPC multi-tiered storage systems. As a centralized data management control plane, Data Jockey automates bulk data movement and placement for scientific workflows and integrates into existing HPC storage infrastructures. Data Jockey simplifies data management by eliminating human effort in programming complex data movements, laying datasets across multiple storage tiers when supporting complex workflows, which in turn increases the usability of multitiered storage systems emerging in modern HPC data centers.Specifically, Data Jockey presents a new data management scheme called “goal driven data management” that can automatically infer low-level bulk data movement plans from declarative high-level goal statements that come from the lifetime of iterative runs of scientific workflows. While doing so, Data Jockey aims to minimize data wait times by taking responsibility for datasets that are unused or to be used, and aggressively utilizing the capacity of the upper, higher performant storage tiers.We evaluated a prototype implementation of Data Jockey under a synthetic workload based on a year’s worth of Oak Ridge Leadership Computing Facility’s (OLCF) operational logs. Our evaluations suggest that Data Jockey leads to higher utilization of the upper storage tiers while minimizing the programming effort of data movement compared to human involved, per-domain adhoc data management scripts.
When to use rsync
We have endeavored to show, using a series of data transfer results obtained from two testbeds, when to use the popular data copying tool rsync and related tools. Tests have been conducted in local area network (LAN) and wide area network (WAN) environments. We conclude that for files in a certain size range and network latency ≦ 10 ms round trip time (RTT), rsync is still useful for data moving tasks in the category 4 of the U.S. DOE Technical Report “Data Movement Categories”. For more demanding data movement requirements, tools of different classes are suggested. Sample histograms from two DOE user facilities are provided to further support our conclusions.
From Data to Models and Analytics: An Integrated Scientific Computing Platform to Accelerate Environmental Research SBIR Phase I (Final Technical Report)
Phase I research was conducted to investigate the technical feasibility of an integrated open source platform GeoLib to provide turnkey solutions for model-data integration for environmental systems science (ESS), using modern web technologies, advances in HPC and cloud computing and high-performance visualization. A key feature of our platform is that it minimizes and automates data movement between data storage and computing systems. Another critical feature of the design is that it enables users to perform high-fidelity post-processing by performing high-performance visualization server-side and in-situ on the compute cluster close to the data. The investigation, which was performed over a 12-month period ending April 2019, centered on developing prototype software to demonstrate the key enabling technologies required for this system. This included: (i) rapid access to data from multiple remote sources, via a unified data interface; (ii) software for scalable geospatial visualization of raster, vector, and point cloud data on the web; (iii) development of an easy-to-use API for creating simulation and modeling workflows; (iv) remote data visualization and processing on middle-tier servers; and (v) remote data processing on high-performance computing systems. A critical part of this work is that all of these capabilities be accessible to end users through standard desktop tools such as Project Jupyter, which is widely adopted for ESS simulation and modeling.