SEARCH · Search NASA

Results for “distributed caching”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

24 records · Page 2

All sources NASA NTRS DOE OSTI

DOE OSTI2025

Data Management in the Continuum: Cross-facility Object-based Data Transfers

Scientific workflows are evolving from relying on a monolithic storage subsystem at a single High-Performance Computing (HPC) facility to using geographically distributed file systems, repositories, and cloud storage. As a result, storing, accessing, transferring, and managing scientific data have become highly complex and prone to performance inefficiencies. This paper delves into these challenges by exploring an optimized end-to-end interface designed to seamlessly connect various local and remote storage systems, enabling efficient data movement of objects across HPC–Cloud and HPC–HPC environments. We showcase this capability through an object-focused data management runtime system, discuss the effects of relaxed consistency semantics in distributed object scenarios, and illustrate its application in an earthquake simulation workflow. Besides reducing the amount of data by selectively transferring regions of interest, our facility-local results achieved a speedup of 45 × over an optimized HDF5 usage and 15 × over the HDF5 with caching by using the new interface in PDC-XF.

Bez, Jean Luca↗

DOE OSTI2022

Memory access optimization for particle operations in computational fluid dynamics-discrete element method simulations

Computational Fluid Dynamics - Discrete Element Method is used to model gas-solid systems in several applications in energy, pharmaceutical and petrochemical industries. Computational performance bottlenecks often limit the problem sizes that can be simulated at industrial scale. The data structures used to store several millions of particles in such large-scale simulations have a large memory footprint that does not fit into the processor cache hierarchies on current high-performance-computing platforms, leading to reduced computational performance. This paper specifically addresses this aspect of memory access bottlenecks in industrial scale simulations. The use of space-filling curves to improve memory access patterns is described and their impact on computational performance is quantified in both shared and distributed memory parallelization paradigms. The Morton space filling curve applied to uniform grids and k-dimensional tree partitions are used to reorder the particle data-structure thus improving spatial and temporal locality in memory. The performance impact of these techniques when applied to two benchmark problems, namely the homogeneous-cooling-system and a fluidized-bed, are presented. We report these optimization techniques lead to approximately two-fold performance improvement in particle focused operations such as neighbor-list creation and data-exchange, with ~ 1.5 times overall improvement in a fluidization simulation with 1.27 million particles.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2020

Striped Data Analysis Framework

A columnar data representation is known to be an efficient way for data storage, specifically in cases when the analysis is often done based only on a small fragment of the available data structures. A data representation like Apache Parquet is a step forward from a columnar representation, which splits data horizontally to allow for easy parallelization of data analysis. Based on the general idea of columnar data storage, working on the [LDRD Project], we have developed a striped data representation, which, we believe, is better suited to the needs of High Energy Physics data analysis. A traditional columnar approach allows for efficient data analysis of complex structures. While keeping all the benefits of columnar data representations, the striped mechanism goes further by enabling easy parallelization of computations without requiring special hardware. We will present an implementation and some performance characteristics of such a data representation mechanism using a distributed no-SQL database or a local file system, unified under the same API and data representation model. The representation is efficient and at the same time simple so that it allows for a common data model and APIs for wide range of underlying storage mechanisms such as distributed no-SQL databases and local file systems. Striped storage adopts Numpy arrays as its basic data representation format, which makes it easy and efficient to use in Python applications. The Striped Data Server is a web service, which allows to hide the server implementation details from the end user, easily exposes data to WAN users, and allows to utilize well known and developed data caching solutions to further increase data access efficiency. We are considering the Striped Data Server as the core of an enterprise scale data analysis platform for High Energy Physics and similar areas of data processing. We have been testing this architecture with a 2TB dataset from a CMS dark matter search and plan to expand it to multiple 100 TB or even PB scale. We will present the striped format, Striped Data Server architecture and performance test results.

Gutsche, Oliver↗

DOE OSTI2025

An Nginx-based Content Distribution Network for HEP

With the move to HTTP/WebDAV and JSON Web Tokens as a standard protocol for transfers within the WLCG distributed storage network, a large amount of off-the-shelf technologies become viable for meeting the requirements of a Storage Element (SE). In this work, we explore the capabilities and performance of the OpenResty framework, which extends the nginx server with the LuaJIT scripting language, to recreate the feature set of a SE. We demonstrate token-authenticated HTTP read, write, and WebDAV third-party copy features, as well as a storage federation with HTTP redirect, proxy, and caching capabilities. We further explore the performance scaling in terms of throughput and requests per second.

Aarora, Aashay [UC, San Diego]↗

DOE OSTI2021

Perseverance’s Scanning Habitable Environments with Raman and Luminescence for Organics and Chemicals (SHERLOC) Investigation

The Scanning Habitable Environments with Raman and Luminescence for Organics and Chemicals (SHERLOC) is a robotic arm-mounted instrument on NASA’s Perseverance rover. SHERLOC has two primary boresights. The Spectroscopy boresight generates spatially resolved chemical maps using fluorescence and Raman spectroscopy coupled to microscopic images (10.1 μm/pixel). The second boresight is a Wide Angle Topographic Sensor for Operations and eNgineering (WATSON); a copy of the Mars Science Laboratory (MSL) Mars Hand Lens Imager (MAHLI) that obtains color images from microscopic scales (~13 μm/pixel) to infinity. SHERLOC Spectroscopy focuses a 40 μs pulsed deep UV neon-copper laser (248.6 nm), to a ~100 μm spot on a target at a working distance of ~48 mm. Fluorescence emissions from organics, and Raman scattered photons from organics and minerals, are spectrally resolved with a single diffractive grating spectrograph with a spectral range of 250 to ~370 nm. Because the fluorescence and Raman regions are naturally separated with deep UV excitation (<250 nm), the Raman region ~ 800 – 4000 cm -1 (250 to 273 nm) and the fluorescence region (274 to ~370 nm) are acquired simultaneously without time gating or additional mechanisms. SHERLOC science begins by using an Autofocus Context Imager (ACI) to obtain target focus and acquire 10.1 μm/pixel greyscale images. Chemical maps of organic and mineral signatures are acquired by the orchestration of an internal scanning mirror that moves the focused laser spot across discrete points on the target surface where spectra are captured on the spectrometer detector. ACI images and chemical maps (< 100 μm/mapping pixel) will enable the first Mars in situ view of the spatial distribution and interaction between organics, minerals, and chemicals important to the assessment of potential biogenicity (containing CHNOPS). Single robotic arm placement chemical maps can cover areas up to 7x7 mm in area and, with the < 10 min acquisition time per map, larger mosaics are possible with arm movements. This microscopic view of the organic geochemistry of a target at the Perseverance field site, when combined with the other instruments, such as Mastcam-Z, PIXL, and SuperCam, will enable unprecedented analysis of geological materials for both scientific research and determination of which samples to collect and cache for Mars sample return.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗

DOE OSTI2024

I/O performance studies of analysis workloads on production and dedicated resources at CERN

The recent evolutions of the analysis frameworks and physics data formats of the LHC experiments provide the opportunity of using central analysis facilities with a strong focus on interactivity and short turnaround times, to complement the more common distributed analysis on the Grid. In order to plan for such facilities, it is essential to know in detail the performance of the combination of a given analysis framework, of a specific analysis and of the installed computing and storage resources. This contribution describes performance studies performed at CERN, using the EOS disk-based storage, either directly or through an XCache instance, from both batch resources and highperformance compute nodes which could be used to build an analysis facility. A variety of benchmarks, both synthetic and based on real-world physics analyses and their corresponding input datasets, are utilized. In particular, the RNTuple format from the ROOT project is put to the test and compared to the latest version of the TTree format, and the impact of caches is assessed. In addition, we assessed the difference in performance between the use of storage system specific protocols, like XRootd, and FUSE. The results of this study are intended to be a valuable input in the design of analysis facilities, at CERN and elsewhere.

Sciabà, Andrea↗