Search NASA⌕ Search

SEARCH · Search NASA

Results for “HPSS”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

12 records
All sourcesNASA NTRSDOE OSTI
DOE OSTI2015

Data Transfer Study HPSS Archiving

The movement of the large amounts of data produced by codes run in a High Performance Computing (HPC) environment can be a bottleneck for project workflows. To balance filesystem capacity and performance requirements, HPC centers enforce data management policies to purge old files to make room for new computation and analysis results. Users at Oak Ridge Leadership Computing Facility (OLCF) and many other HPC user facilities must archive data to avoid data loss during purges, therefore the time associated with data movement for archiving is something that all users must consider. This study observed the difference in transfer speed from the originating location on the Lustre filesystem to the more permanent High Performance Storage System (HPSS). The tests were done with a number of different transfer methods for files that spanned a variety of sizes and compositions that reflect OLCF user data. This data will be used to help users of Titan and other Cray supercomputers plan their workflow and data transfers so that they are most efficient for their project. We will also discuss best practice for maintaining data at shared user facilities.

Big Data↗
DOE OSTI2016

Milestone Report - Level-2 Milestone 5589: Modernization and Expansion of LLNL Archive Disk Cache

This report documents Livermore Computing (LC) activities in support of ASC L2 milestone 5589: Modernization and Expansion of LLNL Archive Disk Cache, due March 31, 2016. The full text of the milestone is included in Attachment 1. The description of the milestone is: Description: Configuration of archival disk cache systems will be modernized to reduce fragmentation, and new, higher capacity disk subsystems will be deployed. This will enhance archival disk cache capability for ASC archive users, enabling files written to the archives to remain resident on disk for many (6–12) months, regardless of file size. The milestone was completed in three phases. On August 26, 2015 subsystems with 6PB of disk cache were deployed for production use in LLNL’s unclassified HPSS environment. Following that, on September 23, 2015 subsystems with 9 PB of disk cache were deployed for production use in LLNL’s classified HPSS environment. On January 31, 2016, the milestone was fully satisfied when the legacy Data Direct Networks (DDN) archive disk cache subsystems were fully retired from production use in both LLNL’s unclassified and classified HPSS environments, and only the newly deployed systems were in use.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2019

Profiling the Usage of an Extreme-Scale Archival Storage System

Profiling the archival storage system in scientific computing environments has received much less attention compared to the parallel file system, but is equally important since it stores the final data products safely, for a long duration. In this paper, we analyze eight years worth of data transfer logs for accessing the archival file system (HPSS) in the Oak Ridge Leadership Computing Facility (OLCF), which has been hosting the world's largest supercomputers and file systems. Our analysis encompasses about 135 million data transfer activities to the 80 PB High Performance Storage System (HPSS), between 2010 and 2017. We analyze the logs from several dimensions, including studying the workload characteristics (e.g., access patterns, frequency of accesses and temporal behavior), file system characteristics (e.g., directory depth, file system scaling trends, file types), and scientific user behavior (e.g., domain-specific usage and organization). Based on the analysis, we derive insights into the future evolution of the archive in terms of provisioning, desired features and functionality from the archive software, role and right sizing of the archive tiers, quota management, and the importance of smart and efficient metadata and storage management. We believe our study will prove useful for both operating current archival storage and better provisioning future systems.

Sim, Hyogi↗
DOE OSTI2018

Experience and Possibilities of Using Energy Resources of Water Reservoirs for a Complex Purpose Located in an Arid Zone. Part II

This paper shows the dependence of the energy indicators of advanced HPSs at reservoirs built for a complex purpose that are located in an arid zone on the genesis of feeding rivers, hydrological regimes, and water use. It reveals the possibility of increasing the reliability of the electric power supply of the reservoirs and prospective hydroelectric power stations (HPSs) in them by wind power plants (WPPs) using the wind energy in the zone of the reservoirs.

13 HYDRO ENERGY↗
DOE OSTI2023

Exploring Future Storage Options for ATLAS at the BNL/SDCC facility

The ATLAS experiment is expected to deliver an unprecedented amount of scientific data in the High Luminosity(HL-LHC) era. As the demand for disk storage capacity in ATLAS continues to rise steadily, the BNL Scientific Data and Computing Center (SDCC) faces challenges in terms of cost implications for maintaining multiple disk copies and adapting to the coming ATLAS storage requirements. To address these challenges, the SDCC Storage team has undertaken a thorough analysis of the ATLAS experiment's requirements, matching them to suitable storage options and strategies, and has explored alternatives to enhance or replace the current storage solution. This paper aims to present the main challenges encountered while supporting big data experiments such as ATLAS. We describe the experiment's specific requirements and priorities, particularly focusing on the critical storage system characteristics of the high-luminosity run and how the key storage components provided by the Storage team work together: the dCache disk storage system; its archival back-end, HPSS; and its OS-level backend Storage. Specifically, we investigate a novel approach to integrate Lustre and XRootD. In this setup, Lustre serves as backend storage and XRootD acts as an access layer frontend, supporting various grid access protocols. Additionally, we also describe the validation and commissioning tests, including the performance comparison between dCache and XRootd. Furthermore, we provide a performance and cost analysis comparing OpenZFS and LINUX MD RAID, evaluate different storage software stacks, and showcase stress tests conducted to validate Third Party Copy (TPC) functionality.

99 GENERAL AND MISCELLANEOUS↗
DOE OSTI2019

Understanding data motion in the modern HPC data center

The utilization and performance of storage, compute, and network resources within HPC data centers have been studied extensively, but much less work has gone toward characterizing how these resources are used in conjunction to solve larger scientific challenges. To address this gap, we present our work in characterizing workloads and workflows at a data-center-wide level by examining all data transfers that occurred between storage, compute, and the external network at the National Energy Research Scientific Computing Center over a three-month period in 2019. Using a simple abstract representation of data transfers, we analyze over 100 million transfer logs from Darshan, HPSS user interfaces, and Globus to quantify the load on data paths between compute, storage, and the wide-area network based on transfer direction, user, transfer tool, source, destination, and time. We show that parallel I/O from user jobs, while undeniably important, is only one of several major I/O workloads that occurs throughout the execution of scientific workflows. We also show that this approach can be used to connect anomalous data traffic to specific users and file access patterns, and we construct time-resolved user transfer traces to demonstrate that one can systematically identify coupled data motion for individual workflows.

Lockwood, GK↗
DOE OSTI2021

ESnet Secure Copy (EScp) v0.6

EScp is a high speed transfer tool with a similar command line syntax to scp. Unlike SCP it is designed to transfer files at high speed, thus far we have been able to show 100gbit/s transfers, although I expect that the throughput should scale in proportion to the network interface, i.e. I expect 400gbit/s performance on our 400gbit/s test bed. EScp achieves good performance through an innovative design (multithreaded, zero copy transfers), along with pluggable filters and I/O engines. As an example, you can switch from POSIX i/O to UIO by checking a different engine. It also natively supports encryption, and cheksums for file verification and transport security. AAA is through standard SSH (same as SCP). By taking advantage of filters, EScp supports transferring unstructured data and/or I/O to non-posix data sources. Examples include streaming data (i.e. from equipment), transferring data to the cloud, and/or supporting non-posix file systems (like HPSS).

Shiflett, Charles↗
DOE OSTI2025

olcf/hsi_xfer

A wrapper for the `hsi` tool that provides a simple user interface to allow the most efficient and least stressful (to the system) way to migrate data off of HPSS filesystems

Wynne, JamesRiley [Oak Ridge National Laboratory (↗
DOE OSTI2022

US Department of Energy, Office of Science High Performance Computing Facility Operational Assessment 2021: Oak Ridge Leadership Computing Facility

Oak Ridge National Laboratory’s (ORNL’s) Leadership Computing Facility (OLCF) continues to surpass its operational target goals of supporting users; delivering fast, reliable computational ecosystems; creating innovative solutions for high-performance computing (HPC) needs; contributing to the community to build the next generation HPC workforce, and managing risks, safety, and security associated with operating some of the most powerful computers in the world. The results can be seen in the cutting-edge science conducted by users and the praise from the research community. Calendar year (CY) 2021 saw continued excellence in research supported by the OLCF’s leadership-class computing resources, including Summit (the nation’s most powerful supercomputer), the global scratch file system Alpine, the Scalable Protected Infrastructure (SPI), the Exploratory Visualization Environment for Research in Science and Technology (EVEREST), and the archival mass-storage resource High-Performance Storage System (HPSS). While maintaining access and exceptional user support for Summit, the OLCF continued to make progress on the installation and deployment of Frontier, which will be the nation’s first exascale system when it comes online at the start of CY 2023. Users have already begun running and optimizing scientific codes on Crusher, the OLCF test and development system equipped with Frontier’s architecture. Throughout the year, the OLCF maintained a strong culture of operational excellence, including risk management, workplace safety, and cybersecurity. The OLCF’s rigorous risk management strategy anticipated and mitigated risks, and at this time there are no high-priority operational risks. Similarly, ORNL and the OLCF were committed to operating under the US Department of Energy’s (DOE’s) safety regulations that ensure a safe workplace. Technical staff tracked and monitored existing threats and vulnerabilities within the OLCF while continually developing tools and practices to enhance operations without increasing the facility’s risk. CY 2021 was filled with outstanding results and accomplishments, including a very high rating from users on overall satisfaction for the eighth consecutive year; a tremendous number of node hours delivered to 1,671 researchers on Summit; and the successful delivery of the allocation split of roughly 60%, 20%, and 20% of core-hours offered for the Innovative and Novel Computational Impact on Theory and Experiment (INCITE), Advanced Scientific Computing Research Leadership Computing Challenge (ALCC), and Director’s Discretionary (DD) programs, respectively (Section 2). COVID-19 research remained a focus in 2021, and the ALCC and DD programs allocated over 1 million Summit hours to the COVID-19 High Performance Computing Consortium. These accomplishments, coupled with the high utilization rates (i.e., overall and capability usage), represent the fulfillment of the promise of leadership class machines: efficient facilitation of leadership-class computational applications.

97 MATHEMATICS AND COMPUTING↗