Search NASA⌕ Search

SEARCH · Search NASA

Results for “Data Reuse”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 199 records · Page 11
All sourcesNASA NTRSDOE OSTI
NASA NTRS2004

SAMZ: Satellite-Derived Management Zones

The term "satellite-derived management zones" (SAMZ) denotes agricultural management zones that are subdivisions of large fields and that are derived from images of the fields acquired by instruments aboard Earth orbiting satellites during approximately the past 15 years. "SAMZ" also denotes the methodology and the software that implements the methodology for creating such zones. The SAMZ approach is one of several products of continuing efforts to realize a concept of precision agriculture, which involves optimal variations in seeding, in application of chemicals, and in irrigation, plus decisions to farm or not to farm certain portions of fields, all in an effort to maximize profitability in view of spatial and temporal variations in the growth and health of crops and in the chemical and physical conditions of soils. As used here, "management zone" signifies, more precisely, a subdivision of a field within which the crop production behavior is regarded as homogeneous. From the perspective of precision agriculture, management zones are the smallest subdivisions between which the seeding, application of chemicals, and other management parameters are to be varied. In the SAMZ approach, the main sources of data are the archives of satellite imagery that have been collected over the years for diverse purposes. One of the main advantages afforded by the SAMZ approach is that the data in these archives can be reused for purposes of precision agriculture at low cost. De facto, these archives contain information on all sources of variability within a field, including weather, crop types, crop management, soil types, and water drainage patterns. The SAMZ methodology involves the establishment of a Web-based interface based on an algorithm that generates management zones automatically and quickly from archival satellite image data in response to requests from farmers. A farmer can make a request by either uploading data describing a field boundary to the Web site or else drawing the boundary on a reference image. Hence, a farmer can start to engage in precision farming shortly after gaining access to the Web site, without need for incurring the high costs of conventional precision-agriculture data-collection practices that include collecting soil samples, mapping electrical conductivity of soil, and compiling multi-year crop-yield data. Given the boundary of a field, a SAMZ server computes the zones within the field in a three-stage process. In the first stage, a vector-valued image of the field is constructed by assembling, from the archives, the equivalent of a stack of the available images of the field (see figure). In the second stage, the vector-valued image is analyzed by use of a wavelet transform that detects spatial variations considered significant for precision farming while suppressing small-scale heterogeneities that are regarded as insignificant. In the third stage, a segmentation algorithm assembles the zones from smaller regions that have been identified in the wavelet analysis.

Source record↗
NASA NTRS2005

Satellite-Derived Management Zones

The term "satellite-derived management zones" (SAMZ) denotes agricultural management zones that are subdivisions of large fields and that are derived from images of the fields acquired by instruments aboard Earth-orbiting satellites during approximately the past 15 years. "SAMZ" also denotes the methodology and the software that implements the methodology for creating such zones. The SAMZ approach is one of several products of continuing efforts to realize a concept of precision agriculture, which involves optimal variations in seeding, in application of chemicals, and in irrigation, plus decisions to farm or not to farm certain portions of fields, all in an effort to maximize profitability in view of spatial and temporal variations in the growth and health of crops, and in the chemical and physical conditions of soils. As used here, "management zone" signifies, more precisely, a subdivision of a field within which the crop-production behavior is regarded as homogeneous. From the perspective of precision agriculture, management zones are the smallest subdivisions between which the seeding, application of chemicals, and other management parameters are to be varied. In the SAMZ approach, the main sources of data are the archives of satellite imagery that have been collected over the years for diverse purposes. One of the main advantages afforded by the SAMZ approach is that the data in these archives can be reused for purposes of precision agriculture at low cost. De facto, these archives contain information on all sources of variability within a field, including weather, crop types, crop management, soil types, and water drainage patterns. The SAMZ methodology involves the establishment of a Web-based interface based on an algorithm that generates management zones automatically and quickly from archival satellite image data in response to requests from farmers. A farmer can make a request by either uploading data describing a field boundary to the Web site or else drawing the boundary on a reference image. Hence, a farmer can start to engage in precision farming shortly after gaining access to the Web site, without the need for incurring the high costs of conventional precision-agriculture data-collection practices that include collecting soil samples, mapping electrical conductivity of soil, and compiling multiyear crop-yield data. Given the boundary of a field, a SAMZ server computes the zones within the field in a three-stage process. In the first stage, a vector-valued image of the field is constructed by assembling, from the archives, the equivalent of a stack of the available images of the field (see figure). In the second stage, the vector-valued image is analyzed by use of a wavelet transform that detects spatial variations considered significant for precision farming while suppressing small-scale heterogeneities that are regarded as insignificant. In the third stage, a segmentation algorithm assembles the zones from smaller regions that have been identified in the wavelet analysis.

Lepoutre, Damien↗
DOE OSTI2023

PNNL DataHub NIAID Program Project: Modeling Host Responses to Understand Severe Human Virus Infections, Multi-Omic Viral Dataset Catalog Collection

The National Institute of Allergy and Infectious Diseases (NIAID) "Modeling Host Responses to Understand Severe Human Virus Infections" program project was a highly integrated and comprehensive systems biology research core, funded by the National Institute of Health (U19AI106772) from 2013-06-01 to 2018-05-31, investigating the complex host response to category A, B, and C priority pathogen infections. Resulting project deliverables include an extensive comprehensive collections of linked primary and secondary transformation viral experimental infection data. Here we provide a never before released comprehensive infectious disease collection of primary and secondary transformation multi-Omics data profiling a series of priority pathogen primary experimental studies for enhanced open access to viral Omics lifecycle datasets and project metadata. Using a highly integrated and multidisciplinary approach, linked primary data and metadata supporting secondary normalization datasets, provide critical information necessary for research reproducibility and long-term preservation. Enabling on-demand data access for research community consumption and developer reuse, serves to support new mechanistic insights and discoveries into host-pathogen interactions for aiding future biohazard data preparedness efforts in emergency response to global health crises involving viral infection.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2023

Effectiveness and predictability of in-network storage cache for Scientific Workflows

Large scientific collaborations often have multiple scientists accessing the same set of files while doing different analyses, which create repeated accesses to the large amounts of shared data located far away. These data accesses have long latency due to distance and occupy the limited bandwidth available over the wide-area network. To reduce the wide-area network traffic and the data access latency, regional data storage caches have been installed as a new networking service. To study the effectiveness of such a cache system in scientific applications, we examine the Southern California Petabyte Scale Cache for a high-energy physics experiment. By examining about 3TB of operational logs, we show that this cache removed 67.6% of file requests from the wide-area network and reduced the traffic volume on wide-area network by 12. 3TB (or 35.4%) an average day. The reduction in the traffic volume (35.4%) is less than the reduction in file counts (67.6%) because the larger files are less likely to be reused. Due to this difference in data access patterns, the cache system has implemented a policy to avoid evicting smaller files when processing larger files. We also build a machine learning model to study the predictability of the cache behavior. Tests show that this model is able to accurately predict the cache accesses, cache misses, and network throughput, making the model useful for future studies on resource provisioning and planning.

Sim, Caitlin↗
DOE OSTI2024

A global database of dissolved organic matter (DOM) concentration measurements in coastal waters (CoastDOM v1)

Abstract. Measurements of dissolved organic carbon (DOC), nitrogen (DON), and phosphorus (DOP) concentrations are used to characterize the dissolved organic matter (DOM) pool and are important components of biogeochemical cycling in the coastal ocean. Here, we present the first edition of a global database (CoastDOM v1; available at https://doi.org/10.1594/PANGAEA.964012, Lønborg et al., 2023) compiling previously published and unpublished measurements of DOC, DON, and DOP in coastal waters. These data are complemented by hydrographic data such as temperature and salinity and, to the extent possible, other biogeochemical variables (e.g. chlorophyll a, inorganic nutrients) and the inorganic carbon system (e.g. dissolved inorganic carbon and total alkalinity). Overall, CoastDOM v1 includes observations of concentrations from all continents. However, most data were collected in the Northern Hemisphere, with a clear gap in DOM measurements from the Southern Hemisphere. The data included were collected from 1978 to 2022 and consist of 62 338 data points for DOC, 20 356 for DON, and 13 533 for DOP. The number of measurements decreases progressively in the sequence DOC > DON > DOP, reflecting both differences in the maturity of the analytical methods and the greater focus on carbon cycling by the aquatic science community. The global database shows that the average DOC concentration in coastal waters (average ± standard deviation (SD): 182±314 µmol C L−1; median: 103 µmol C L−1) is 13-fold higher than the average coastal DON concentration (13.6±30.4 µmol N L−1; median: 8.0 µmol N L−1), which is itself 39-fold higher than the average coastal DOP concentration (0.34±1.11 µmol P L−1; median: 0.18 µmol P L−1). This dataset will be useful for identifying global spatial and temporal patterns in DOM and will help facilitate the reuse of DOC, DON, and DOP data in studies aimed at better characterizing local biogeochemical processes; closing nutrient budgets; estimating carbon, nitrogen, and phosphorous pools; and establishing a baseline for modelling future changes in coastal waters.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2026

Transplatformer: translating toxicogenomic profiles between generations of platforms

Background Transcriptomic profiling technologies have advanced the analysis of biological and toxicological responses. However, substantial differences in probe design, dynamic range, gene coverage, and preprocessing pipelines across platforms introduce artifacts that limit cross-study integration and hinder the reuse of historical datasets. We aim to develop computational methods for accurate cross-platform translation to maximize the value of legacy resources. Results We present TransPlatformer a deep learning framework for translating gene expression profiles across heterogeneous toxicogenomics platforms. TransPlatformer employs a novel attention-based architecture to map high-dimensional fold-change vectors from legacy microarray technologies to current platforms. Models are trained and evaluated using DrugMatrix, spanning three technological generations. We investigate mixed-tissue, single-tissue, and cross-tissue training paradigms and benchmark performance against multilayer perceptron and matrix-completion baselines. In mixed-tissue training, TransPlatformer achieves a greater than 50% reduction in mean absolute error (0.043 vs. 0.09) and nearly doubles Pearson correlation ( ≈ 0.71 vs. 0.37) relative to baseline methods. Importantly, TransPlatformer preserves rare but biologically meaningful over- and under-expressed signals, with mean absolute error below 0.22. Single-tissue models yield further improvements for well-represented organs, such as a 10% reduction in liver mean absolute error, while underscoring the need for data augmentation strategies in low-sample tissues.ra Conclusions TransPlatformer provides an effective and scalable computational solution for cross-platform transcriptomic translation. By enabling biologically faithful harmonization of gene expression data, the proposed approach facilitates the reuse of legacy toxicogenomics datasets, enhances downstream biomarker discovery, and supports more reproducible predictive modeling in toxicology.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2022

GPU cache management based on locality type detection

Wavefront loading in a processor is managed and includes monitoring a selected wavefront of a set of wavefronts. Reuse of memory access requests for the selected wavefront is counted. A cache hit rate in one or more caches of the processor is determined based on the counted reuse. Based on the cache hit rate, subsequent memory requests of other wavefronts of the set of wavefronts are modified by including a type of reuse of cache lines in requests to the caches. In the caches, storage of data in the caches is based on the type of reuse indicated by the subsequent memory access requests. Reused cache lines are protected by preventing cache line contents from being replaced by another cache line for a duration of processing the set of wavefronts. Caches are bypassed when streaming access requests are made.

Zhang, Xianwei↗
DOE OSTI2026

RC-SFA Data Management Templates and Guidance for Standardized, Reusable AI-Ready Data Packages

This data package provides templates and supporting documentation developed by the River Corridor Science Focus Area (RC-SFA; https://www.pnnl.gov/projects/river-corridor) to communicate its approach to managing and publishing AI-ready data. The package is intended to help data users and data producers understand the structures, metadata practices, and quality-control approaches that support consistent, reusable, and machine-actionable data products across RC-SFA studies. Rather than focusing on a single experimental dataset, this package documents the data management framework used to make RC-SFA data easier to find, ingest, navigate, and interpret. The materials in this package reflect RC-SFA practices for standardized data package organization, including the use of a human- and machine-readable README, file-level metadata, data dictionaries, descriptive file naming, method identifiers, and automated and review-based quality assurance procedures. Together, these components illustrate how RC-SFA extends FAIR data principles toward AI-readiness by prioritizing deep metadata, consistency across data packages, and support for informed downstream reuse by both humans and computational tools. This dataset is comprised of (1) readme; (2) presentation slides with an overview of RC-SFA approach and guidance; (3) document of RC-SFA best practices; (4) data dictionary (dd); (5) file level metadata (flmd); and a subfolder containing templates for dd and flmd. All files are .csv and .pdf. For details on how to navigate data packages generated by this project, see https://data.ess-dive.lbl.gov/portals/PNNLRiverCorridorSFA/About.

AI-readiness↗
DOE OSTI2023

Lawrence Livermore National Laboratory Experimental Test Site, Site 300: Soil Study Institutional Review Board #1081 Soil Sampling and Analysis Plan

This Soil Sampling and Analysis Plan (SAP) was prepared by the Environmental Functional Area/Technical Services Department (TSD) of the Environment, Safety & Health (ES&H) Directorate for the Project Management Office (PMO) to collect soil quality data at an area located at Lawrence Livermore National Laboratory’s (LLNL’s) Experimental Test Site, Site 300 (Site 300) (project) (Figures 1 and 2). The purpose of the project is to determine the chemical and radiological quality of soil in the project area to support feasibility and design of future improvements. This SAP identifies chemicals of concern and describes the procedures for collection and analysis of environmental samples, and evaluation of analytical data (chemical and radiological) to determine the reuse potential of shallow soil at the project area. This SAP follows the criteria established in LLNL’s Soils Screening and Management Plan (SSMP) (LLNL 2022), which is consistent with U.S. Environmental Protection Agency (EPA) guidance for developing Data Quality Objectives for environmental data (EPA 2006) and the Multi-Agency Radiation Survey and Site Investigation Manual (MARSSIM) guidance (U.S. NRC, U.S. EPA, U.S. DOE, U.S. DOD 2000). The scope of this SAP is based on preliminary design information provided by PMO.

54 ENVIRONMENTAL SCIENCES↗
NASA NTRSDate not supplied

An Overview of NASA’s Catalog of Archived Suborbital Earth Science Investigations (CASEI): Supporting FAIR and Open Access to Airborne and Field Data

Since 2019, NASA’s Airborne Data Management Group (ADMG) within the Interagency Implementation and Advanced Concepts Team (IMPACT) has worked to promote and ensure the discoverability and accessibility of the agency’s non-satellite Earth science observations. A primary component of this effort is the development of NASA’s Catalog of Archived Suborbital Earth Science Investigations (CASEI) and the vetting of key contextual details required to sustain this unique inventory of airborne and field metadata. CASEI provides information on the science objectives motivating data collection, key events/time periods in the observational record aligned with the science objectives, complementary simultaneous observations, programmatic details, and much more. The diverse set of data formats and disciplines served by CASEI have required the implementation of a common data model to organize suborbital observation metadata and efficiently connect appropriate campaigns, platforms, and instruments. The CASEI inventory provides a single entry point for users to search and browse NASA’s airborne and field data archives, regardless of which repository is responsible for their stewardship. This presentation will provide a summary of the motivations for and the development of the CASEI system. Particular attention will be granted to how CASEI facilitates discovery and reuse of these lesser-known NASA data, supporting the Open Science vision and enhancing the return on investments made to collect these unique and varied observations. An up-to-date summary of CASEI inventory content and initial metrics will be provided. Current and future avenues ADMG is pursuing to enhance both CASEI and specific components of suborbital data stewardship at various stages of the data life cycle will also be discussed.

Stephanie M. Wingo↗
NASA NTRS2014

Anticipated Changes in Conducting Scientific Data-Analysis Research in the Big-Data Era

A Big-Data environment is one that is capable of orchestrating quick-turnaround analyses involving large volumes of data for numerous simultaneous users. Based on our experiences with a prototype Big-Data analysis environment, we anticipate some important changes in research behaviors and processes while conducting scientific data-analysis research in the near future as such Big-Data environments become the mainstream. The first anticipated change will be the reduced effort and difficulty in most parts of the data management process. A Big-Data analysis environment is likely to house most of the data required for a particular research discipline along with appropriate analysis capabilities. This will reduce the need for researchers to download local copies of data. In turn, this also reduces the need for compute and storage procurement by individual researchers or groups, as well as associated maintenance and management afterwards. It is almost certain that Big-Data environments will require a different "programming language" to fully exploit the latent potential. In addition, the process of extending the environment to provide new analysis capabilities will likely be more involved than, say, compiling a piece of new or revised code.We thus anticipate that researchers will require support from dedicated organizations associated with the environment that are composed of professional software engineers and data scientists. A major benefit will likely be that such extensions are of higherquality and broader applicability than ad hoc changes by physical scientists. Another anticipated significant change is improved collaboration among the researchers using the same environment. Since the environment is homogeneous within itself, many barriers to collaboration are minimized or eliminated. For example, data and analysis algorithms can be seamlessly shared, reused and re-purposed. In conclusion, we will be able to achieve a new level of scientific productivity in the Big-Data analysis environments.

Kuo, Kwo-Sen↗
DOE OSTI2025

The Surface-Topography Challenge: A Multi-Laboratory Benchmark Study to Advance the Characterization of Topography

Surface performance is critically influenced by topography in virtually all real-world applications. The current standard practice is to describe topography using one of a few industry-standard parameters. The most commonly reported number is Ra, the average absolute deviation of the height from the mean line (at some, not necessarily known or specified, lateral length scale). However, other parameters, particularly those that are scale-dependent, influence surface and interfacial properties; for example the local surface slope is critical for visual appearance, friction, and wear. The present Surface-Topography Challenge was launched to raise awareness for the need of a multi-scale description, but also to assess the reliability of different metrology techniques. In the resulting international collaborative effort, 153 scientists and engineers from 64 research groups and companies across 20 countries characterized statistically equivalent samples from two different surfaces: a “rough” and a “smooth” surface. The results of the 2088 measurements constitute the most comprehensive surface description ever compiled. We find wide disagreement across measurements and techniques when the lateral scale of the measurement is ignored. Consensus is established through scale-dependent parameters while removing data that violates an established resolution criterion and deviates from the majority measurements at each length scale. Our findings suggest best practices for characterizing and specifying topography. The public release of the accumulated data and presented analyses enables global reuse for further scientific investigation and benchmarking.

42 ENGINEERING↗
DOE OSTI2026

VirJenDB: a FAIR (meta)data and bioinformatics platform for all viruses

High-throughput sequencing has generated an unprecedented volume of data. However, researcher-submitted data in repositories requires extensive curation and quality control for reuse. These tasks are hindered by the multiplicity of repositories, the sheer volume of the data, and the complexity of virus (meta)data curation. To address these challenges, VirJenDB offers a user-friendly platform to facilitate versioned, community-driven curation, and ontology development. Virus sequences were ingested from 16 sources, including ~200 fields of metadata or standards, covering taxonomy, sample, and host information. Up to 85 metadata fields have undergone at least one round of curation, and are linked to 15.4 million virus sequences, with 88 % from those infecting eukaryotes and the remaining infecting prokaryotes. Subsets were created, including a novel collection of 0.91 million viral operational taxonomic unit (vOTU) sequences across all viruses, while keeping the original sequences from each vOTU to facilitate downstream analyses, e.g. sequence variation. The VirJenDB web portal (https://www.virjendb.org) provides HTTPS and Application Programming Interface (API) access to the sequence datasets and metadata, offering a search engine, filtering, download, visualizations, and documentation. VirJenDB aims to connect the phage and eukaryotic virus research communities by supporting webtool integration, meta-analyses, and metadata schema extensions.

Saghaei, Shahram↗
DOE OSTI2022

Williston Basin Carbon Ore, Rare Earth, and Critical Minerals (CORE-CM) Program

The U.S. Department of Energy’s Office of Fossil Energy awarded 13 CORE-CM programs as part of the CORE-CM Initiative, designed to develop the technology and upstream and midstream supply chains to extract rare-earth elements (REEs) and critical minerals (CMs) from the nation’s coal supplies. The intent is to catalyze regional economic growth and job creation while strengthening the use of domestic resources. The Williston Basin CORE-CM Program aims to drive the expansion and transformation of coal and coal-based resource usage within the Williston Basin to produce REEs, CMs, and nonfuel carbon-based products (CBPs). The work constitutes Phase 1 of a long-term program. The objectives of Phase 1 are to identify the existing knowledge base and gaps and to develop a series of assessments/plans, including an initial basinal assessment; a characterization and data acquisition plan; a waste stream reuse plan; a basinal strategies assessment for infrastructure, industries, and business; a technology assessment, development, and field testing plan; a technology innovation center plan(s); and a stakeholder outreach and education plan. The goal of this project is to initiate the development of a new industry in the resource-rich Williston Basin that will catalyze economic growth and job creation through a coalition team of private industry, university, and state, local, and federal government entities. A coalition team of nearly 30 private industry, university, and state, local, and federal government partners was formed for this program. Partners include the University of North Dakota Energy & Environmental Research Center, Institute for Energy Studies, and Nistler College; North Dakota State University; Montana Tech University; Pacific Northwest National Laboratory; Critical Materials Institute; North Dakota Geological Survey; South Dakota Geological Survey; U.S. Geological Survey; North American Coal Corporation; BNI Energy; Basin Electric Power Cooperative; Minnkota Power Cooperative; and many more.

Kay, John P.↗
DOE OSTI2023

Williston Basin Carbon Ore, Rare Earth, and Critical Minerals (CORE-CM) Program

The U.S. Department of Energy’s (DOE’s) Office of Fossil Energy awarded 13 Carbon Ore, Rare-Earth and Critical Minerals (CORE-CM) programs as part of the CORE-CM Initiative, designed to develop the technology and upstream and midstream supply chains to extract rare-earth elements (REEs) and CMs from the nation’s coal supplies. The intent is to catalyze regional economic growth and job creation while strengthening the use of domestic resources. The Williston Basin CORE-CM Program aims to drive the expansion and transformation of coal and coal-based resource usage within the Williston Basin to produce REEs, CMs, and nonfuel carbon-based products (CBPs). The work constitutes Phase 1 of a long-term program. The objectives of Phase 1 are to identify the existing knowledge base and gaps and to develop a series of assessments/plans, including an initial basinal assessment; a characterization and data acquisition plan; a waste stream reuse plan; a basinal strategies assessment for infrastructure, industries, and business; a technology assessment, development, and field-testing plan; a technology innovation center plan(s); and a stakeholder outreach and education plan. Originally, the CORE-CM programs were to do little if any sample collection as part of the Phase 1 program. As part of the ongoing interactions with DOE regarding the progress of the 13 CORE-CM programs, DOE has decided to extend Phase 1 with additional funding to allow all of the Phase 1 CORE-CM programs to collect and characterize more samples that have potential to be sources of REEs and CMs. The goal of this project is to initiate the development of a new industry in the resource-rich Williston Basin that will generate economic growth and job creation through a coalition team of private industry; university; and state, local, and federal government entities. A coalition team of nearly 30 private industry; university; and state, local, and federal government partners was formed for this program. Partners include the University of North Dakota Energy & Environmental Research Center, Institute for Energy Studies, and Nistler College; North Dakota State University; Montana Tech University; Pacific Northwest National Laboratory; Critical Materials Institute; North Dakota Geological Survey; South Dakota Geological Survey; U.S. Geological Survey; North American Coal Corporation; BNI Energy; Basin Electric Power Cooperative; Minnkota Power Cooperative; and many more.

Kay, John P.↗
NASA NTRS1991

Space Shuttle Orbiter carbon-carbon oxidation performance

Reusable, oxidation protected reinforced carbon carbon (RCC) has been successfully flown on forty Shuttle Orbiter flights. Thermal testing of the silicon carbide coated, reinforced carbon-carbon to determine its oxidation characteristics has been performed in both radiant and convective (plasma arc jet) heating test facilities. Subsurface oxidation of the RCC substrate as a result of oxygen penetrating micro cracks (fizzures) in the coating was characterized as a function of temperature and pressure for both convective and radiant environments. High temperature testing was performed to establish coating recession for over-temperature flight conditions experienced on abort trajectories. Suggested methods for using these test data to establish multi-mission reuse (i.e., mission life) and single mission limits are presented.

Curry, D. M.↗
DOE OSTI2021

Efficient loading of reduced data ensembles produced at ORNL SNS/HFIR neutron time-of-flight facilities

We present algorithmic improvements to the loading operations of certain reduced data ensembles produced from neutron scattering experiments at Oak Ridge National Laboratory (ORNL) facilities. Ensembles from multiple measurements are required to cover a wide range of the phase space of a sample material of interest. They are stored using the standard NeXus schema on individual HDF5 files. This makes it a scalability challenge, as the number of experiments stored increases in a single ensemble file. The present work follows up on our previous efforts on data management algorithms, to address identified input output (I/O) bottlenecks in Mantid, an open-source data analysis framework used across several neutron science facilities around the world. We reuse an in-memory binary-tree metadata index that resembles data access patterns, to provide a scalable search and extraction mechanism. In addition, several memory operations are refactored and optimized for the current common use cases, ranging most frequently from 10 to 180, and up to 360 separate measurement configurations. Results from this work show consistent speed ups in wall-clock time on the Mantid LoadMD routine, ranging from 19% to 23% on average, on ORNL production computing systems. The latter depends on the complexity of the targeted instrument-specific data and the system I/O and compute variability for the shared computational resources available to users of ORNL’s Spallation Neutron Source (SNS) and the High Flux Isotope Reactor (HFIR) instruments. Nevertheless, we continue to highlight the need for more research to address reduction challenges as experimental data volumes, user time and processing costs increase.

Godoy, William↗