SEARCH · Search NASA

Results for “File System”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 199 records · Page 11

All sources NASA NTRS DOE OSTI

DOE OSTI2021

Demystifying asynchronous I/O Interference in HPC applications

With increasing complexity of HPC workflows, data management services need to perform expensive I/O operations asynchronously in the background, aiming to overlap the I/O with the application runtime. However, this may cause interference due to competition for resources: CPU, memory/network bandwidth. The advent of multi-core architectures has exacerbated this problem, as many I/O operations are issued concurrently, thereby competing not only with the application but also among themselves. Furthermore, the interference patterns can dynamically change as a response to variations in application behavior and I/O subsystems (e.g. multiple users sharing a parallel file system). Without a thorough understanding, I/O operations may perform suboptimally, potentially even worse than in the blocking case. To fill this gap, here we investigate the causes and consequences of interference due to asynchronous I/O on HPC systems. Specifically, we focus on multi-core CPUs and memory bandwidth, isolating the interference due to each resource. Then, we perform an in-depth study to explain the interplay and contention in a variety of resource sharing scenarios such as varying priority and number of background I/O threads and different I/O strategies: sendfile, read/write, mmap/write underlining trade-offs. The insights from this study are important both to enable guided optimizations of existing background I/O, as well as to open new opportunities to design advanced asynchronous I/O strategies.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2022

AI4IO: A suite of AI-based tools for IO-aware scheduling

Traditional workload managers do not have the capacity to consider how IO contention can increase job runtime and even cause entire resource allocations to be wasted. Whether from bursts of IO demand or parallel file systems (PFS) performance degradation, IO contention must be identified and addressed to ensure maximum performance. In this paper, we present AI4IO (AI for IO), a suite of tools using AI methods to prevent and mitigate performance losses due to IO contention. AI4IO enables existing workload managers to become IO-aware. Currently, AI4IO consists of two tools: PRIONN and CanarIO. PRIONN predicts IO contention and empowers schedulers to prevent it. CanarIO mitigates the impact of IO contention when it does occur. We measure the effectiveness of AI4IO when integrated into Flux, a next-generation scheduler, for both small- and large-scale IO-intensive job workloads. Our results show that integrating AI4IO into Flux improves the workload makespan up to 6.4%, which can account for more than 18,000 node-h of saved resources per week on a production cluster in our large-scale workload.

Wyatt, II, Michael R.↗

DOE OSTI2018

Analysis Report for Exascale Storage Requirements for Scientific Data.

Over the next 10 years, the Department of Energy will be transitioning from Petascale to Exascale Computing resulting in data storage, networking, and infrastructure requirements to increase by three orders of magnitude. The technologies and best practices used today are the result of a relatively slow evolution of ancestral technologies developed in the 1950s and 1960s. These include magnetic tape, magnetic disk, networking, databases, file systems, and operating systems. These technologies will continue to evolve over the next 10 to 15 years on a reasonably predictable path. Experience with the challenges involved in transitioning these fundamental technologies from Terascale to Petascale computing systems has raised questions about how these will scale another 3 or 4 orders of magnitude to meet the requirements imposed by Exascale computing systems. This report is focused on the most concerning scaling issues with data storage systems as they relate to High Performance Computing- and presents options for a path forward. Given the ability to store exponentially increasing amounts of data, far more advanced concepts and use of metadata will be critical to managing data in Exascale computing systems.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2016

Recorded Information Management Records Move Project

Throughout the history of Sandia National Laboratories, both federal and corporate records have been stored in 12 structures in remote areas of the lab, none of which are compliant storage facilities under federal regulations. Upon inspection by the National Archives and Records Administration (NARA), findings were levied against Sandia along with recommendations on correcting the issues. The RM Department anticipated over 38,000 boxes of legacy records in Sandia’s current holdings. The current storage areas present an abundance of unknown issues and risks associated with the move project. Along with the large amount of records, and media types, this project faces issues with over 50 years of ever changing filing systems and human error pertaining to data entry, boxes being processed and destroyed. This document presents the final data and results pertaining to the Recorded Information Management Department corrective actions in a well laid out project plan that accommodated safety, security, environmental concerns and of course the records.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗

DOE OSTI2020

CephFS experiments on stria.sandia.gov

This report is an institutional record of experiments conducted to explore performance of a vendor installation of CephFS on the SNL stria cluster. Comparisons between CephFS, the Lustre parallel file system, and NFS were done using the IOR and MDTEST benchmarking tools, a test program which uses the SEACAS/Trilinos IOSS library, and the checkpointing activity performed by the LAMMPS molecular dynamics simulation.

74 ATOMIC AND MOLECULAR PHYSICS↗

DOE OSTI2020

CephFS experiments on stria.sandia.gov

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2022

Position Papers for the ASCR Workshop on the Management and Storage of Scientific Data

The purpose of this workshop is to identify priority research directions in the area of data management for high-performance and scientific computing above and beyond HPC’s traditional "the parallel file system is the data-management system" model. Supporting the breadth of the DOE mission, including the explosion of AI uses and the growing needs of experimental and observational science, motivates revisiting our assumptions about data management. There are many facets of this topic to explore including: (1) Interfaces for accessing data that resides on traditional persistent storage as well as memory devices; (2) Storage-system architecture design that supports scientific workflows on varied hierarchical storage and networking devices; (3) Devising metadata management infrastructure to support FAIR principles (Findability, Accessibility, Interoperability, and Reusability); (4) Capturing provenance information about scientific data; (5) Utilizing AI to learn I/O patterns of emerging workloads for efficient data management; (6) Providing data management support for AI and complex workflows; and (7) Understanding the overlap between traditional storage systems and I/O (SSIO) efforts and data management. While the program committee has identified these topics as important areas for discussion, we welcome position papers from the community that propose additional topics of interest for discussion at the workshop. The workshop agenda will include breakout sessions for discussing these and selected topic areas to inform priority research directions for data management for high-performance and scientific computing.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2022

Report for the ASCR Workshop on the Management and Storage of Scientific Data

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2022

US Department of Energy, Office of Science High Performance Computing Facility Operational Assessment 2021: Oak Ridge Leadership Computing Facility

Oak Ridge National Laboratory’s (ORNL’s) Leadership Computing Facility (OLCF) continues to surpass its operational target goals of supporting users; delivering fast, reliable computational ecosystems; creating innovative solutions for high-performance computing (HPC) needs; contributing to the community to build the next generation HPC workforce, and managing risks, safety, and security associated with operating some of the most powerful computers in the world. The results can be seen in the cutting-edge science conducted by users and the praise from the research community. Calendar year (CY) 2021 saw continued excellence in research supported by the OLCF’s leadership-class computing resources, including Summit (the nation’s most powerful supercomputer), the global scratch file system Alpine, the Scalable Protected Infrastructure (SPI), the Exploratory Visualization Environment for Research in Science and Technology (EVEREST), and the archival mass-storage resource High-Performance Storage System (HPSS). While maintaining access and exceptional user support for Summit, the OLCF continued to make progress on the installation and deployment of Frontier, which will be the nation’s first exascale system when it comes online at the start of CY 2023. Users have already begun running and optimizing scientific codes on Crusher, the OLCF test and development system equipped with Frontier’s architecture. Throughout the year, the OLCF maintained a strong culture of operational excellence, including risk management, workplace safety, and cybersecurity. The OLCF’s rigorous risk management strategy anticipated and mitigated risks, and at this time there are no high-priority operational risks. Similarly, ORNL and the OLCF were committed to operating under the US Department of Energy’s (DOE’s) safety regulations that ensure a safe workplace. Technical staff tracked and monitored existing threats and vulnerabilities within the OLCF while continually developing tools and practices to enhance operations without increasing the facility’s risk. CY 2021 was filled with outstanding results and accomplishments, including a very high rating from users on overall satisfaction for the eighth consecutive year; a tremendous number of node hours delivered to 1,671 researchers on Summit; and the successful delivery of the allocation split of roughly 60%, 20%, and 20% of core-hours offered for the Innovative and Novel Computational Impact on Theory and Experiment (INCITE), Advanced Scientific Computing Research Leadership Computing Challenge (ALCC), and Director’s Discretionary (DD) programs, respectively (Section 2). COVID-19 research remained a focus in 2021, and the ALCC and DD programs allocated over 1 million Summit hours to the COVID-19 High Performance Computing Consortium. These accomplishments, coupled with the high utilization rates (i.e., overall and capability usage), represent the fulfillment of the promise of leadership class machines: efficient facilitation of leadership-class computational applications.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2018

A Methodology for the Rapid Development of Scalable HPC Data Services

There is growing evidence in the scientifc computing community that parallel file systems are not sufficient for all HPC storage workloads. This realization has motivated extensive research in new storage system designs. The question of which alternative we should turn towards implies that there could be a single answer satisfying a wide range of very diverse applications. We argue that such a generic solution does not exist. Instead, custom data services should be designed and tailored to the needs of specific applications on specific hardware. Furthermore, custom data services should be designed in close collaboration with users. In this paper, we present a methodology for the rapid development of such data services. This methodology promotes the design of reusable building blocks that can be composed together efficiently through a runtime based on high-performance threading, tasking, and remote procedure calls. We illustrate the success of our methodology by showcasing three examples of data services designed from the same building blocks, yet targeting entirely different applications.

Dorier, Mattieu↗

DOE OSTI2020

System and method of managing large data files

Disclosed are systems and software that provide a high-performance, extensible file format and web API for remote data access and a visual interface for data viewing, query, and analysis. The described system can support storage of raw spectroscopic data such as neural recording data, MSI data, metadata, and derived analyses in a single, self-describing format that may be compatible by a large range of analysis software.

Bowen, Benjamin P.↗

DOE OSTI2018

Advanced TidGen Power System - OpenFOAM Version 5 CFD Case Files

The TidGen Power System generates emission-free electricity from tidal currents and connects directly into existing grids using smart grid technology. The power system consists of three major subsystems: shore-side power electronics, mooring system, and turbine generator unit (TGU) device. This submission contains supporting CFD files, case files and geometry for the Advanced TidGen. TSR = Tip speed ratio Cp = Power coefficient Cl = Lift coefficient Cd = Drag coefficient

16 TIDAL AND WAVE POWER↗

DOE OSTI2017

Challenges and Opportunities of User-Level File Systemsfor HPC

The performance gap between magnetic disks and data processing on HPC systems has become that huge that an efficient data processing can only be achieved by introducing non-volatile memory (NVRAM) as a new storage tier. Although the benefits of hierarchical storage have been adequately demonstrated to the point that the newest leadership class HPC systems will employ burst buffers, critical questions remain for supporting hierarchical storage systems, including: How should we present hierarchical storage systems to user applications, such that they are easy to use and that application code is portable across systems? How should we manage data movement through a storage hierarchy for best performance and resilience of data? How do the particular I/O use cases mandate the way we manage data? There have been many efforts to explore this space in the form of file systems, with increasingly more implemented at the user level. This is because it is relatively easy to swap in new, specialized user-level file systems for use by applications on a case-by-case basis, as opposed to the current mainstream approach of using general-purpose, system-level file systems which may not be optimized for HPC workloads and must be installed by administrators. In contrast, file systems at the user level can be tailored for specific HPC workloads for high performance and can be used by applications without administrator intervention. Many user-level file system developers have found themselves “having to reinvent the wheel” to implement various optimizations in their file systems. Thus, a main goal of this meeting was to bring together experts in I/O performance, file systems, and storage, and collectively explore the space of current and future problems and solutions for I/O on hierarchical storage systems in order to begin a community effort in enabling user-level file system support for HPC systems. We had a lively week of learning about each other’s approaches as well as unique I/O use cases that can influence the design of a community-driven file and storage system standards. The agenda for this meeting contained talks from participants on the following high level topics: HPC storage and I/O support today; what do HPC users need for I/O; existing userlevel file system efforts; object stores and other alternative storage systems; and components for building user-level file systems. The talks were short and intended to be conversation starters for more in-depth discussions with the whole group. The participants engaged in lengthy discussions on various questions that arose from the talks including: Are we ready to program to a memory hierarchy versus block devices? Are the needs of HPC users reflected in our existing file systems and storage systems? Should we drop or keep POSIX moving forward? What do we mean when we say “user-level file system”? Do we all mean the same thing? How should the IO 500 benchmark be defined so it is fair and useful? and How are stage-in and stage-out actually going to work? The report for this seminar contains a record of the talks from the participants as well as the resulting discussions. Our hope is that the effort initiated during this seminar will result in long-term collaborations that will benefit the HPC community as a whole.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2026

Representation and Impact of Water Head on Power System Planning and Operation

Representing water head information in power system model files, can provide a more realistic model of the system and thereby inform operation and planning personnel in the decision-making process. This article describes a procedure for modifying the power system model files (steady-state and dynamic) to represent water head information. Additionally, the impact of representing the water head on power system reliability studies including contingency analysis, cascading failure analysis and dynamic frequency response analysis has been investigated, using the modified power system models. This paper considers the detailed Western Electricity Coordination Council model during summer and winter conditions as the test system for the impact analysis. Results show that under reduced water head: 1) the number of critical voltage and branch flow violations increases; 2) chances of cascading failure and island formation increases; and 3) frequency nadir decreases as compared to those of the base cases where the water head information is not represented.

13 - HYDRO ENERGY↗

DOE OSTI2025

Data from: Short-Term Inhibition and Long-Term Enhancement of IrreversibleTrace Metal Binding to Goethite in Multi-Metal Systems

These files include the data used in all figures from the following paper: Short-Term Inhibition and Long-Term Enhancement of Irreversible Trace Metal Binding to Goethite in Multi-Metal Systems Below is a brief description of each of the figures and associated data. Fig 1. Ni adsorption isotherms to goethite in the presence and absence of Zn and Cd Fig 2. % of Ni, Zn, and Cd exchanged as a function of time in isotope exchange experiments conducted after 2 days, 30 days, and 60 days aging. Fig. 3. This shows the % of Ni, Cd, and Zn in single metal and mixed metal systems that is non-exchangeable, both in units of % non-exchangeable and in micromoles of metal per gram of goethite. Fig. 4. Particle size distributions of Ni-Zn and Ni-Cd systems after aging. Fig 5. Ni and Zn XANES fits for Ni-Cd and Ni-Zn systems after 30 and 60 days aging. Linear combination fits were performed between freshly adsorbed and incorporated phases (synthesized as Ni- or Zn-substituted goethite) to determine the proportions of each. Figure S1. Dissolved Metal Concentrations at the Start of Isotope Exchange Figure S2. Reactor pH with Aging Figure S3. Dissolved Cd and Ni Concentrations during Isotope Exchange Figure S4. Dissolved Zn and Ni Concentrations during Isotope Exchange Figure S5. Tracer Mole Fractions in Ni-Zn Isotope Exchange Experiments Figure S6. Tracer Mole Fractions in Ni-Cd Isotope Exchange Experiments

58 GEOSCIENCES↗

DOE OSTI2022

Study of reference burnup steps optimization in fuel segment data file generation for NEXUS/ANC9 code system

For any two-step core design code system, the cross-section files are the primary factor to determine the accuracy of the system prediction. For a once-through cross-section system to cover all potential applicable conditions, the system may need to perform tens of thousands state points of lattice calculations. These calculations take a significant amount of CPU times and computing power, especially when a fine or ultra-fine energy group library is used. As computer power increases, so does the calculational complexity. Therefore, it is important to make the lattice calculations effective and efficient. Based on the Westinghouse core design code system NEXUS/ANC9, this work focuses on the reference burnup steps used by the lattice code when performing the calculations. Following the fundamental cross-section methodology and analyzing the contribution of each individual terms, this study provides an applicable solution of the optimized reference burnup steps. The test results show that, with the existing cross-section methodology, it is possible to significantly reduce the lattice calculation cases and cross-section file generation time without sacrificing the accuracy of system prediction. (authors)

22 GENERAL STUDIES OF NUCLEAR REACTORS↗

DOE OSTI2023

Data from: Long-term changes in soil carbon and nitrogen fractions in switchgrass, native grasses, and no-till corn bioenergy production systems

These files contain data from soil and root samples use in this publication. The R script uses this data to perform the statistical analysis used in the publication.

Bioenergy feedstock↗

DOE OSTI2024

State Requirements for Electric Distribution System Planning

Utilities have conducted distribution planning since they first began building and operating electricity systems. But filing these plans for regulatory and stakeholder review is a relatively recent phenomenon. This report summarizes legislative and regulatory requirements for regulated electric utilities to file some type of distribution system plan in 20 U.S. jurisdictions. Some plans focus on expedited cost recovery for certain types of distribution system improvements; other plans focus on investments for grid modernization or distributed energy resources. Increasingly, states are adopting requirements for Integrated Distribution Plans. Such plans provide holistic grid investment strategies that address state and local policies and increasing complexity at the grid edge. The report covers the following topics for distribution system plans, highlighting advanced practices: -State goals and objectives -Procedural requirements -Forecasting loads and distributed energy resources -Hosting capacity analysis -Baseline information requirements -Grid modernization strategy -Grid needs assessment -Non-wires solutions -Reliability and resilience analyses -Stakeholder engagement -Equity -Pilots -Coordination with other planning processes The report includes links to legislation; regulatory requirements, proceedings, and orders; and filed utility plans. The U.S. Department of Energy’s Office of Electricity provided funding support.

24 POWER TRANSMISSION AND DISTRIBUTION↗