Computing erasure metadata and data layout prior to storage using a processing platform

Bent, John M.; Faibish, Sorin; Ting, Dennis P.J.; Teymouri, Sasan; Pedone, Jr., James M.; Grider, Gary

Home/Archive/DOE OSTI

DOE OSTI · 1494131

Computing erasure metadata and data layout prior to storage using a processing platform

Bent, John M.·Faibish, Sorin·Ting, Dennis P.J.·Teymouri, Sasan·Pedone, Jr., James M.·Grider, Gary

View original source ↗Read source document ↗Export metadata ↓

Abstract

Techniques are provided for computing data and metadata layout prior to storage in a storage system using a processing platform. An exemplary processing platform comprises one or more of a compute node and a burst buffer appliance. The processing platform communicates with a plurality of the compute nodes over a network, wherein a plurality of applications executing on the plurality of compute nodes generate a plurality of data objects; computes erasure metadata for one or more of the data objects on at least one of the compute nodes; and provides the erasure metadata with the corresponding one or more data objects to a storage system. The processing platform optionally determines a full set of the data objects to be stored and queries the storage system to determine an anticipated layout of the full set of the data objects to be stored. The anticipated layout allows special handling, for example, for small files and large files that are identified based on predefined criteria.

Explore related subjects

96 KNOWLEDGE MANAGEMENT AND PRESERVATION

Keep this discovery

Explore connections, maps & timelines

BibTeX RIS

Bent, John M., Faibish, Sorin, Ting, Dennis P.J., Teymouri, Sasan, Pedone, Jr., James M., Grider, Gary. 2018-11-20. Computing erasure metadata and data layout prior to storage using a processing platform. https://www.osti.gov/biblio/1494131

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related reports

DOE OSTI2026

Ontologies for Intelligent Data Science

As anyone even vaguely aware of current technology can tell you, machine learning (ML) and artificial intelligence (AI) have made exceptional breakthroughs in recent years. Generative artificial intelligence (GAI) emerged circa 2022 dominated by Large Language Models (LLMs) and generative tools for images emerged at about the same time.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗

DOE OSTI2026

Data Cards for Standardized Metadata Across DOE-Aligned Data Initiatives: Toward Transparent, Interoperable, and Governed Dataset Documentation

As data-intensive research, advanced computing, and artificial intelligence become increasingly central to scientific and operational workflows, the need for consistent, transparent, and machine-actionable documentation has grown correspondingly. Multiple DOE-aligned communities—including Office of Science, Genesis Mission, American Science Cloud (AmSC), National Nuclear Security Administration (NNSA) stewardship and governance, and related cross-laboratory collaborations—have independently developed metadata practices to support discovery, access, reuse, repository deposit, and compliance.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗

DOE OSTI2026

Verification and Validation of Performance with Dissemination of Best Practices in District Energy and CHP for Enhanced Resiliency, Energy Efficiency, and Cybersecurity

This report contains the results of the International District Energy Association’s work to analyze, validate, and verify performance data of existing district energy systems and identify industy best practices for the purpose of improving system reliability, resiliency, and efficiency, and to accellerate decarbonization. In addition to a technical evaluation of the surveyed systems and identification of a series of technical performance metrics, the report illustrates the accompanying operations and financial best practices employed by surveyed systems to fully serve their customer base. Additionally, the third chapter of the report describes the current landscape of cybersecurity threats and counteracting measures, and recommends a series of steps for effectively guarding highly networked district energy systems against cybersecurity attacks.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗