SEARCH · Search NASA

Results for “Data Movement”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 109 records · Page 6

All sources NASA NTRS DOE OSTI

DOE OSTI2019

Analytical Modeling of Cache Behavior for Affine Programs

Modern optimizing compiler design integrates program transformation strategies aimed at reducing data movement to/from main memory by exploiting the data cache hierarchy. But in reality, the effect of these transformations on the actual cache miss count is ignored in compilers due to the lack of precise compile-time models of misses for hierarchical caches. The current state of practice for cache miss analysis is based on accurate simulation. However, simulation requires time proportional to the dataset/problem size as well as the number of distinct cache configurations of interest to be evaluated. This paper takes a fundamentally different approach, made possible by focusing on polyhedral programs with static control flow: instead of relying on costly simulation, this paper develops the first closed-form solution for exact modeling of misses in a set-associative cache hierarchy. It enabling simulation-less selection of program transformations at compile-time to optimize cache misses. A push-button tool implementing the approach is developed and used for extensive validation of the complete framework.

affine programs, cache modeling, set associativity↗

DOE OSTI2018

ARCHIE: Data Analysis Acceleration with Array Caching in Hierarchical Storage

© 2018 IEEE. Scientific data analysis typically involves reading massive amounts of data that was generated by simulations, experiments, and observations. Performance of reading such large volumes of data from disk-based file systems is often poor because of the slow and mechanical components in the disks. Recent supercomputing systems are adding non-volatile storage layers in a hierarchy to handle the performance gap between fast main memory and slow disk-based storage. Software libraries for managing this hierarchy not only need efficient reading of data but also reduce user-involvement for cross-layer data movement. Furthermore, these libraries need to support array data access patterns into hierarchical storage management as scientific data is often organized in array-based data structures. Existing software typically manage individual storage layers requiring significant manual process in moving data among them. In this paper, we introduce a new array caching in hierarchical storage (ARCHIE) to accelerate array data analysis in a seamless fashion. ARCHIE evaluates array access patterns and prefetches data with array semantics between storage layers. Our evaluation shows that ARCHIE outperforms state-of-the-art file systems, i.e., Lustre and DataWarp, on a production supercomputing system by up to 5.8× in accessing data by scientific analysis applications.

Dong, Bin↗

DOE OSTI2019

Extreme-Scale Distribution-Based Data Analysis

Scientists overview and identify regions of interest by transforming data into compact information descriptors that characterize simulation results and allow detailed analysis on demand. Among many existing feature descriptors, statistical information derived from data samples is a promising approach to taming the big data avalanche because data distributionscomputed from a population can compactly describe the presence and characteristics of salient data features with minimal data movement. The ability to computationally summarize and process data using distributions also provides an efficient and representative capture of information that can adjust to size and resource constraints, with the added benefit that uncertainty associated with results can be quantified and communicated. In this project, we have developed a distribution-based data analysis and visualization framework for optimizing in situ processing and analysis of extreme-scale scientific data.

97 MATHEMATICS AND COMPUTING↗

NASA NTRS1996

[The Strategic Organization of Skill]

Eye-movement software was developed in addition to several studies that focused on expert-novice differences in the acquisition and organization of skill. These studies focused on how increasingly complex strategies utilize and incorporate visual look-ahead to calibrate action. Software for collecting, calibrating, and scoring eye-movements was refined and updated. Some new algorithms were developed for analyzing corneal-reflection eye movement data that detect the location of saccadic eye movements in space and time. Two full-scale studies were carried out which examined how experts use foveal and peripheral vision to acquire information about upcoming environmental circumstances in order to plan future action(s) accordingly.

Roberts, Ralph↗

NASA NTRS2009

Enhanced Video-Oculography System

A previously developed video-oculography system has been enhanced for use in measuring vestibulo-ocular reflexes of a human subject in a centrifuge, motor vehicle, or other setting. The system as previously developed included a lightweight digital video camera mounted on goggles. The left eye was illuminated by an infrared light-emitting diode via a dichroic mirror, and the camera captured images of the left eye in infrared light. To extract eye-movement data, the digitized video images were processed by software running in a laptop computer. Eye movements were calibrated by having the subject view a target pattern, fixed with respect to the subject s head, generated by a goggle-mounted laser with a diffraction grating. The system as enhanced includes a second camera for imaging the scene from the subject s perspective, and two inertial measurement units (IMUs) for measuring linear accelerations and rates of rotation for computing head movements. One IMU is mounted on the goggles, the other on the centrifuge or vehicle frame. All eye-movement and head-motion data are time-stamped. In addition, the subject s point of regard is superimposed on each scene image to enable analysis of patterns of gaze in real time.

Moore, Steven T.↗

NASA NTRSDate not supplied

Can Tracked Great Frigatebirds (Fregata minor) be Used to Measure the Dynamics of the Planetary Boundary Layer Height?

Observing the dynamics of the planetary boundary layer (PBL) will require creative combinations of space-based and in-situ measurements. Animals fitted with biologgers have been used to gather data and parameterize physical models in various settings and may be a useful addition to the suite of space-based measurements that NASA is considering. For example, great frigatebirds (Fregata minor) generally fly below 700m but have been shown to routinely climb up to 2,000 meters during the day, occasionally reaching heights over 4,000 meters. This suggests that they may be tracking boundary layer height dynamics as they soar and glide on air currents with little energetic cost. We show here that the high-altitude climbing heights of great frigatebirds carrying GPS/accelerometer biologgers (e-obs GmbH Bird Solar 15g) at Palmrya Atoll, Pacific Ocean, are consistent with the long-term climatology (2006-2019) of the local planetary boundary layer height, and that these soaring heights can be reliably extracted from the tag data. Additional study is needed to determine how well movement data from tracked birds can constrain PBL height, but this work demonstrates the potential for animal-borne sensors to complement space-based measurements. Biologging data were contributed by the USGS and the Nature Conservancy's Palmyra Bluewater Research (PBR) study and analyzed as part of NASA’s Internet of Animals project.

Planetary Boundary Layer Height↗

DOE OSTI2019

FTLADS: Object-Logging Based Fault-Tolerant Big Data Transfer System Using Layout Aware Data Scheduling

The layout-aware data scheduling (LADS) data movement framework optimizes congestion for end-to-end data transfers. During data transfer, LADS can avoid congested storage elements by exploiting the underlying storage layout at each endpoint. This improves the I/O bandwidth and hence the data transfer rate across high-speed networks. However, the absence of fault tolerance (FT) in LADS results in data retransmission overhead and may lead to possible data integrity issues upon faults. In this paper, we propose object-logging FT mechanisms to avoid transmitting the objects that are successfully written into the parallel file system (PFS) at the sink end. Depending on the number of log files created for the whole dataset, we have classified our FT mechanisms into three different categories: file logger, transaction logger, and universal logger. Also, to address the space overhead, we have proposed different methods of populating the log files with the information of the successfully transferred objects. We have evaluated the data transfer performance and recovery time overhead of the proposed object-logging-based FT mechanisms on the LADS data transfer framework. Our experimental results reflect that FT mechanisms exhibit negligible overhead (<; 1%) with respect to the data transfer time. Yet, the fault recovery time is 10% higher than the total data transfer time at any fault point.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2025

Accelerating Advanced Light Source Science Through Multi-Facility HPC Workflows

Synchrotron light sources support a wide array of techniques to investigate materials, often producing complex, high-volume data that challenge traditional workflows. At the Advanced Light Source (ALS), we developed infrastructure to move microtomography data over ESnet to ALCF and NERSC, where CPU- and GPU-based algorithms generate 3D reconstructed volumes of experimental samples. We employ two data movement and reconstruction models: real-time processing as data streams directly to NERSC compute nodes, and automated file transfer to NERSC and ALCF file systems. The streaming pipeline provides users with feedback in under ten seconds, while the file-based workflow produces high-quality reconstructions suitable for deeper analysis in 20-30 minutes. This infrastructure enables users to utilize HPC resources without direct access to backend systems. We plan to extend this architecture to more endstations, supporting our beamline scientists and users.

Abramov, David↗

NASA NTRS1985

Polyamines as Possible Modulators of Gravity-induced Calcium Transport in Plants

Data from various laboratories indicate a probable relationship between calcium movement and some aspects of graviperception and tropistic bending responses. The movement of calcium in response to gravistimulation appears to be rapid, polar and opposite in direction to polar auxin transport. What might be the cause of such rapid Ca(2+) movement? Data from studies on polyamine (PA) metabolism may furnish a clue. A transient increase in the activity of ornithine decarboxylase (ODC) and titers of various PAs occurs within 60 seconds after hormonal stimulation of animal cells, followed by Ca(2+) transport out of the cells. Through the use of specific inhibitors, it was shown that the enhanced PA synthesis from ODC was essential not only for Ca(2+) transport, but also for Ca(2+) transport-dependent endocytosis and the movement of hexoses and amino acids across the plasmalemma. In plants, rapid changes in arginine decarboxylase (ADC) activity occur in response to various plant stresses. Physical stresses associated with gravisensor displacement and reorientation of a plant in the gravitational field could similarly activate ADC and that resultant increases in PA levels might initiate transient perturbations in Ca(2+) homeostasis.

Galston, A. W.↗

DOE OSTI2024

Analog Computing for Science

Conventional digital computing faces fundamental physical limits: large scale computing systems already con sume tens of Megawatts of power, Dennard scaling has ended, and data movement costs dominate application performance. Next generation experimental facilities generate data at rates that overwhelm conventional pro cessing and demand real-time analysis at the source. Analog computing, which exploits the continuous dynamics of physical systems to perform computation, promises a transformative path toward orders-of-magnitude gains in energy efficiency and time-to-solution for scientific workloads.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2017

Challenges and Opportunities of User-Level File Systemsfor HPC

The performance gap between magnetic disks and data processing on HPC systems has become that huge that an efficient data processing can only be achieved by introducing non-volatile memory (NVRAM) as a new storage tier. Although the benefits of hierarchical storage have been adequately demonstrated to the point that the newest leadership class HPC systems will employ burst buffers, critical questions remain for supporting hierarchical storage systems, including: How should we present hierarchical storage systems to user applications, such that they are easy to use and that application code is portable across systems? How should we manage data movement through a storage hierarchy for best performance and resilience of data? How do the particular I/O use cases mandate the way we manage data? There have been many efforts to explore this space in the form of file systems, with increasingly more implemented at the user level. This is because it is relatively easy to swap in new, specialized user-level file systems for use by applications on a case-by-case basis, as opposed to the current mainstream approach of using general-purpose, system-level file systems which may not be optimized for HPC workloads and must be installed by administrators. In contrast, file systems at the user level can be tailored for specific HPC workloads for high performance and can be used by applications without administrator intervention. Many user-level file system developers have found themselves “having to reinvent the wheel” to implement various optimizations in their file systems. Thus, a main goal of this meeting was to bring together experts in I/O performance, file systems, and storage, and collectively explore the space of current and future problems and solutions for I/O on hierarchical storage systems in order to begin a community effort in enabling user-level file system support for HPC systems. We had a lively week of learning about each other’s approaches as well as unique I/O use cases that can influence the design of a community-driven file and storage system standards. The agenda for this meeting contained talks from participants on the following high level topics: HPC storage and I/O support today; what do HPC users need for I/O; existing userlevel file system efforts; object stores and other alternative storage systems; and components for building user-level file systems. The talks were short and intended to be conversation starters for more in-depth discussions with the whole group. The participants engaged in lengthy discussions on various questions that arose from the talks including: Are we ready to program to a memory hierarchy versus block devices? Are the needs of HPC users reflected in our existing file systems and storage systems? Should we drop or keep POSIX moving forward? What do we mean when we say “user-level file system”? Do we all mean the same thing? How should the IO 500 benchmark be defined so it is fair and useful? and How are stage-in and stage-out actually going to work? The report for this seminar contains a record of the talks from the participants as well as the resulting discussions. Our hope is that the effort initiated during this seminar will result in long-term collaborations that will benefit the HPC community as a whole.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2020

Compiler and Runtime Approaches to Enable Large-Scale Irregular Programs. Final report, July 2013 - July 2019

While regular algorithms, characterized by operations on dense matrices and arrays, have long been the mainstay of scientific, high-performance computing, irregular algorithms, which feature unpredictable accesses to pointer-based data structures, are becoming increasingly common in high performance computing, arising in graph analysis, data mining and visualization, among other domains. Unfortunately, the defining characteristics of irregular applications, their dynamic, unpredictable, data-dependent access patterns and data layouts, make achieving high performance on large scale systems difficult. Scaling applications to peta- and exa-scale requires carefully controlling communication and data movement and placement, an inherently difficult task when access patterns and data layouts are unpredictable! Most irregular applications that attain high performance must be painstakingly hand-written and hand-tuned, with few common principles or paradigms uniting various implementations and easing future development. Despite the increasing importance of irregular applications, there is little programmer knowledge, and even less compiler ability, devoted to optimizing them. This project aims to solve these problems. By allowing programmers to write irregular applications in high level forms, with at most a few annotations highlighting key structural properties, programmers can focus on developing their algorithms and methods. The compiler and run-time system can take on the tedious task of optimizing the application for execution at large scales, and can automatically provide efficient implementations. This will provide portability and ease maintenance for existing irregular applications, but, more importantly, open up whole new domains of computational science to large-scale, high-performance simulation codes.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2022

Vector-Matrix Multiplication Engine for Neuromorphic Computation with a CBRAM Crossbar Array [Slides]

The core function of many neural network algorithms is the dot product, or vector matrix multiply (VMM) operation. Crossbar arrays utilizing resistive memory elements can reduce computational energy in neural algorithms by up to five orders of magnitude compared to conventional CPUs. Moving data between a processor, SRAM, and DRAM dominates energy consumption. By utilizing analog operations to reduce data movement, resistive memory crossbars can enable processing of large amounts of data at lower energy than conventional memory architectures.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2017

Integration of end-user Cloud storage for CMS analysis

End-user Cloud storage is increasing rapidly in popularity in research communities thanks to the collaboration capabilities it offers, namely synchronisation and sharing. CERN IT has implemented a model of such storage named, CERNBox, integrated with the CERN AuthN and AuthZ services. To exploit the use of the end-user Cloud storage for the distributed data analysis activity, the CMS experiment has started the integration of CERNBox as a Grid resource. This will allow CMS users to make use of their own storage in the Cloud for their analysis activities as well as to benefit from synchronisation and sharing capabilities to achieve results faster and more effectively. It will provide an integration model of Cloud storages in the Grid, which is implemented and commissioned over the world’s largest computing Grid infrastructure, Worldwide LHC Computing Grid (WLCG). In this paper, we present the integration strategy and infrastructure changes needed in order to transparently integrate end-user Cloud storage with the CMS distributed computing model. We describe the new challenges faced in data management between Grid and Cloud and how they were addressed, along with details of the support for Cloud storage recently introduced into the WLCG data movement middleware, FTS3. Finally, the commissioning experience of CERNBox for the distributed data analysis activity is also presented.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2018

Assessing the Implementation of an SDM at Sandia and LLNL

This report assesses the proposed implementation of a Scientific Data Management system that could be co-developed by LLNL and SNL and installed at each lab. In the current view, the systems would not be connected to each other. This report examines the different collaboration possibilities and status of planning and implementation of the teams at LLNL and SNL. The report uses identified requirements from previous work that has been done and documented in [1]. In that work, a set of simulation data management requirements was developed during a series of meetings, starting from requirements developed by Sandia Laboratory. These requirements are expected to evolve, and it is recommended that these requirements be revisited and updated for the following reasons: 1- Advancement in the computing hardware and software which leads to more processing power and better data movement speeds. 2- Advancement in computer algorithms which creates opportunities for better data management and analysis. 3- The need to expand and add to the capabilities of the SDM tools due to advancement in the applications software and due to new requirements by the scientific community. This report builds upon the experience from the work that has been done by some of the team members and the current SDM development efforts at LLNL and Sandia.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗

NASA NTRS1978

A single user efficiency measure for evaluation of parallel or pipeline computer architectures

A precise statement of the relationship between sequential computation at one rate, parallel or pipeline computation at a much higher rate, the data movement rate between levels of memory, the fraction of inherently sequential operations or data that must be processed sequentially, the fraction of data to be moved that cannot be overlapped with computation, and the relative computational complexity of the algorithms for the two processes, scalar and vector, was developed. The relationship should be applied to the multirate processes that obtain in the employment of various new or proposed computer architectures for computational aerodynamics. The relationship, an efficiency measure that the single user of the computer system perceives, argues strongly in favor of separating scalar and vector processes, sometimes referred to as loosely coupled processes, to achieve optimum use of hardware.

Jones, W. P.↗

DOE OSTI2020

An Integrated Indexing and Search Service for Distributed File Systems

Data services such as search, discovery, and management in scalable distributed environments have traditionally been decoupled from the underlying file systems, and are often deployed using external databases and indexing services. However, modern data production rates, looming data movement costs, and the lack of metadata, entail revisiting the decoupled file system-data services design philosophy. In this article, we present TagIt, a scalable data management service framework aimed at scientific datasets, which can be integrated into prevalent distributed file system architectures. A key feature of TagIt is a scalable, distributed metadata indexing framework, which facilitates a flexible tagging capability to support data discovery. Furthermore, the tags can also be associated with an active operator, for pre-processing, filtering, or automatic metadata extraction, which we seamlessly offload to file servers in a load-aware fashion. We have integrated TagIt into two popular distributed file systems, i.e., GlusterFS and CephFS. Our evaluation demonstrates that TagIt can expedite data search operation by up to 10× over the extant decoupled approach.

97 MATHEMATICS AND COMPUTING↗

NASA NTRS1994

Performance Measurement, Visualization and Modeling of Parallel and Distributed Programs

This paper presents a methodology for debugging the performance of message-passing programs on both tightly coupled and loosely coupled distributed-memory machines. The AIMS (Automated Instrumentation and Monitoring System) toolkit, a suite of software tools for measurement and analysis of performance, is introduced and its application illustrated using several benchmark programs drawn from the field of computational fluid dynamics. AIMS includes (i) Xinstrument, a powerful source-code instrumentor, which supports both Fortran77 and C as well as a number of different message-passing libraries including Intel's NX Thinking Machines' CMMD, and PVM; (ii) Monitor, a library of timestamping and trace -collection routines that run on supercomputers (such as Intel's iPSC/860, Delta, and Paragon and Thinking Machines' CM5) as well as on networks of workstations (including Convex Cluster and SparcStations connected by a LAN); (iii) Visualization Kernel, a trace-animation facility that supports source-code clickback, simultaneous visualization of computation and communication patterns, as well as analysis of data movements; (iv) Statistics Kernel, an advanced profiling facility, that associates a variety of performance data with various syntactic components of a parallel program; (v) Index Kernel, a diagnostic tool that helps pinpoint performance bottlenecks through the use of abstract indices; (vi) Modeling Kernel, a facility for automated modeling of message-passing programs that supports both simulation -based and analytical approaches to performance prediction and scalability analysis; (vii) Intrusion Compensator, a utility for recovering true performance from observed performance by removing the overheads of monitoring and their effects on the communication pattern of the program; and (viii) Compatibility Tools, that convert AIMS-generated traces into formats used by other performance-visualization tools, such as ParaGraph, Pablo, and certain AVS/Explorer modules.

Yan, Jerry C.↗