Precision Cascade: A novel algorithm for multi-precision extreme compression
Explore the source record for details and available documents.
73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
SEARCH · Search NASA
Results for “data tiering”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
Flexible and Effective Object Tiering for Heterogeneous Memory Systems
Computing platforms that package multiple types of memory, each with their own performance characteristics, are quickly becoming mainstream. To operate efficiently, heterogeneous memory architectures require new data management solutions that are able to match the needs of each application with an appropriate type of memory. As the primary generators of memory usage, applications create a great deal of information that can be useful for guiding memory management, but the community still lacks tools to collect, organize, and leverage this information effectively. To address this gap, this work introduces a novel software framework that collects and analyzes object-level information to guide memory tiering. The framework includes tools to monitor the capacity and usage of individual data objects, routines that aggregate and convert this information into tier recommendations for the host platform, and mechanisms to enforce these recommendations according to user-selected policies. Moreover, the developed tools and techniques are fully automatic, work on standard Linux systems, and do not require modification or recompilation of existing software. Using this framework, this study evaluates and compares the impact of a variety of design choices for memory tiering, including different policies for prioritizing objects for the fast memory tier as well as the frequency and timing of migration events. In conclusion, the results, collected on a modern Intel platform with conventional DDR4 SDRAM as well as Intel Optane NVRAM, show that guiding data tiering with object-level information can enable significant performance and efficiency benefits compared with standard hardware- and software-directed data-tiering strategies for a diverse set of memory-intensive workloads.
97 MATHEMATICS AND COMPUTING↗
Current and future directions in network biology
Network biology is an interdisciplinary field bridging computational and biological sciences that has proved pivotal in advancing the understanding of cellular functions and diseases across biological systems and scales. Although the field has been around for two decades, it remains nascent. It has witnessed rapid evolution, accompanied by emerging challenges. These stem from various factors, notably the growing complexity and volume of data together with the increased diversity of data types describing different tiers of biological organization. We discuss prevailing research directions in network biology, focusing on molecular/cellular networks but also on other biological network types such as biomedical knowledge graphs, patient similarity networks, brain networks, and social/contact networks relevant to disease spread. In more detail, we highlight areas of inference and comparison of biological networks, multimodal data integration and heterogeneous networks, higher-order network analysis, machine learning on networks, and network-based personalized medicine. Following the overview of recent breakthroughs across these five areas, we offer a perspective on future directions of network biology. Additionally, we discuss scientific communities, educational initiatives, and the importance of fostering diversity within the field. This article establishes a roadmap for an immediate and long-term vision for network biology.
59 BASIC BIOLOGICAL SCIENCES↗
FitCache: A Transparent Drop-In Framework for Multi-Tier Caching to Accelerate Distributed Deep Learning Workloads
Training in Deep learning (DL) remains highly compute- and data-intensive, with I/O becoming a critical bottleneck as models and datasets scale. Recent studies report that data loading can dominate training time, especially on large-scale HPC systems with shared parallel file systems (PFS). Existing caching approaches either rely on single-tier designs or require intrusive modifications to training pipelines, limiting their portability and effectiveness. In this work, we present FitCache, a transparent drop-in framework for multi-tier caching to accelerate distributed DL training by coordinating fast local memory (e.g., DRAM, Persistent Memory (PMem)) and NVMe as hierarchical caches atop PFS. Our design adapts to hardware diversity, i.e., if NVMe is missing, memory transparently acts as a caching tier, ensuring stable performance. FitCache transparently intercepts I/O requests and issues concurrent fetches across all tiers, returning data from the fastest responder without centralized metadata or static redirection paths. FitCache adapts to dynamic workloads and heterogeneous clusters while maintaining POSIX compatibility. Experiments on Frontier (2048 GPUs) and smaller research clusters show that FitCache reduces training time by up to 40% and per-batch I/O latency by up to 71.6% compared to Lustre Orion PFS, offering a drop-in solution for scalable DL training.
Hu, Guangxing [ORNL] (ORCID:0009000283203614)↗
Models, data, and scripts associated with “Prediction of Distributed River Sediment Respiration Rates using Community-Generated Data and Machine Learning”
This data package is associated with the publication “Prediction of Distributed River Sediment Respiration Rates using Community-Generated Data and Machine Learning’’ submitted to the Journal of Geophysical Research: Machine Learning and Computation (Scheibe et al. 2024). River sediment respiration observations are expensive and labor intensive to obtain and there is no physical model for predicting this quantity. The Worldwide Hydrobiogeochemisty Observation Network for Dynamic River Systems (WHONDRS) observational data set (Goldman et al.; 2020) is used to train machine learning (ML) models to predict respiration rates at unsampled sites. This repository archives training data, ML models, predictions, and model evaluation results for the purposes of reproducibility of the results in the associated manuscript and community reuse of the ML models trained in this project. One of the key challenges in this work was to find an optimum configuration for machine learning models to work with this feature-rich (i.e. 100+ possible input variables) data set. Here, we used a two-tiered approach to managing the analysis of this complex data set: 1) a stacked ensemble of ML models that can automatically optimize hyperparameters to accelerate the process of model selection and tuning and 2) feature permutation importance to iteratively select the most important features (i.e. inputs) to the ML models. The major elements of this ML workflow are modular, portable, open, and cloud-based, thus making this implementation a potential template for other applications. This data package is associated with the GitHub repository found at Please see the file level metadata (flmd; “sl-archive-whondrs_flmd.csv”) for a list of all files contained in this data package and descriptions for each. Please see the data dictionary (dd; “sl-archive-whondrs_dd.csv”) for a list of all column headers contained within comma separated value (csv) files in this data package and descriptions for each. The GitHub repository is organized into five top-level directories: (1) “input_data” holds the training data for the ML models; (2) “ml_models” holds machine learning models trained on the data in “input_data”; (3) “scripts” contains data preprocessing and postprocessing scripts and intermediate results specific to this data set that bookend the ML workflow; (4) “examples” contains the visualization of the results in this repository including plotting scripts for the manuscript (e.g., model evaluation, FPI results) and scripts for running predictions with the ML models (i.e., reusing the trained ML models); (5) “output_data” holds the overall results of the ML model on that branch. Each trained ML model resides on its own branch in the repository; this means that inputs and outputs can be different branch-to-branch. Furthermore, depending on the number of features used to train the ML models, the preprocessing and postprocessing scripts, and their intermediate results, can also be different branch-to-branch. The “main-*” branches are meant to be starting points (i.e. trunks) for each model branch (i.e. sprouts). Please see the Branch Navigation section in the top-level README.md in the GitHub repository for more details. There is also one hidden directory “.github/workflows”. This hidden directory contains information for how to run the ML workflow as an end-to-end automated GitHub Action but it is not needed for reusing the ML models archived here. Please the top-level README.md in the GitHub repository for more details on the automation.
13C↗
Memory instruction for memory tiers
Various embodiments provide for one or more processor instructions and memory instructions that enable a memory sub-system to copy, move, or swap data across (e.g., between) different memory tiers of the memory sub-system, where each of the memory tiers is associated with different memory locations (e.g., different physical memory locations) on one or more memory devices of the memory sub-system.
Roberts, David Andrew↗
Prediction of Distributed River Sediment Respiration Rates Using Community-Generated Data and Machine Learning
River sediment microbial respiration is a key indicator of ecosystem functioning and the biogeochemical fluxes across this critical zone link surface and subsurface waters. As such, there is tremendous interest in measuring and mapping these respiration rates. Respiration observations are expensive and labor intensive; there is limited data available to the community. An open science, collaborative initiative is collecting samples for respiration rate analysis and multi-scale metadata; this evolving data set is being used for making machine learning (ML) predictions at unsampled sites to help inform continued community engagement. However, it is a challenge to find an optimum configuration for ML models to work with this feature-rich (i.e., 100+ possible input variables) data set. Here, we present results from a two-tiered approach to managing the analysis of this complex data set: (a) a stacked ensemble of models that automatically optimizes hyperparameters and manages the training of many models and (b) feature permutation importance to detect the most important features in the models. The major elements of this workflow are modular, portable, open, and cloud-based thus making this implementation a potential template for other applications. The models developed here predict that sediment organic matter chemistry is one of the most important features for predicting sediment respiration rate. Other larger-scale, important features fall into the categories of climatic, ecological, geological, and fluvial settings. Leveraging these larger-scale features to generate data-driven estimates of river sediment respiration rates reveals spatially consistent but heterogeneous patterns across the river network of the Columbia River Basin.
54 ENVIRONMENTAL SCIENCES↗
Higher than expected N 2 O emissions from soybean crops in the Pampas Region of Argentina: Estimates from DayCent simulations and field measurements
In developing countries, agriculture generally represents a large fraction of GHG emissions reported in National Inventories, and emissions are typically estimated using Tier 1 IPCC guidelines. However, field data and locally adapted simulation models can improve the accuracy of IPCC estimations. In this report we aimed to quantify anthropogenic N2O emissions from croplands of Argentina through field measurements, model simulations and IPCC guidelines. Here we measured N 2 O emissions and their controlling factors in 62 plots of the Pampas Region with corn, soybean and wheat/soybean crops and in unmanaged grasslands. We accounted for gross emissions from crops and background emissions from unmanaged grasslands to calculate net anthropogenic emissions from crops as the difference between them. We calibrated and evaluated the DayCent model and then simulated different weather and management scenarios. Finally, we applied IPCC guidelines to estimate anthropogenic N 2 O emissions at the same plots. The DayCent model accurately simulated annual N 2 O emission for all crops as compared to measured data (RMSE = 1.4 g N ha -1 day -1 ). Measured and simulated emissions in soybean crops were higher than in corn and wheat/soybean crops. Gross N 2 O emissions ranged from 1.4 to 5.1 kg N ha -1 yr -1 for current environmental (soil and weather) and management (crops and fertilizer doses) conditions. Background emissions ranged between 1.1 and 1.3 kg N ha -1 yr -1 , and therefore net anthropogenic emissions ranged from 0.3 to 4.0 kg N ha-1 yr -1 . IPCC Tier 1 emission factors underestimated N2O releases from soybean, that were on average 4.87 times greater when estimated with DayCent and observations (0.53 vs 2.47 and 2.69 kg N ha -1 yr -1 , respectively). On the contrary, IPCC estimates for corn and wheat/soybean crops were similar to modeled and measured values. Our results suggest that N 2 O emissions from the vast 15 million ha of soybean croplands in the Pampas Region may be substantially underestimated.
59 BASIC BIOLOGICAL SCIENCES↗
RuralAI in Tomato Farming: Integrated Sensor System, Distributed Computing, and Hierarchical Federated Learning for Crop Health Monitoring
Precision horticulture is evolving due to scalable sensor deployment and machine learning (ML) integration. These advancements boost the operational efficiency of individual farms, balancing the benefits of analytics with autonomy requirements. However, given concerns that affect wide geographic regions (e.g., climate change), there is a need to apply models that span farms. Federated learning (FL) has emerged as a potential solution. FL enables decentralized ML across different farms without sharing private data. Traditional FL assumes simple two-tier network topologies and, thus, falls short of operating on more complex networks found in real-world agricultural scenarios. Networks vary across crops and farms and encompass various sensor data modes, extending across jurisdictions. New hierarchical FL (HFL) approaches are needed for more efficient and context-sensitive model sharing, accommodating regulations across multiple jurisdictions. Here, we present the RuralAI architecture deployment for tomato crop monitoring, featuring sensor field units for soil, crop, and weather data collection. HFL with personalization is used to offer localized and adaptive insights. Model management, aggregation, and transfers are facilitated via a flexible approach, enabling seamless communication between local devices, edge nodes, and the cloud.
60 APPLIED LIFE SCIENCES↗
Data Analysis Approach for Large Data Volumes in a Connected Community
Recent advancements within smart neighborhoods where utilities are enabling automatic control of appliances such as heating, ventilation, and air conditioning (HVAC) and water heater (WH) systems are providing new opportunities to minimize energy costs through reduced peak load. This requires systematic collection, storage, management, and in-memory processing of large volumes of streaming data for fast performance. In this paper, we propose a multi-tier layered IoT software framework that enables effective descriptive and predictive data analysis for understanding live operation of the neighborhood, fault identification, and future opportunities for further optimization of load curves. We then demonstrate how we achieve live situational awareness of the connected neighborhood through a suite of visualization components. Finally, we discuss a few analytic dashboards that address questions such as peak load reductions obtained due to optimization, customer preference for automatic control of appliances (do they override the automatic control of HVAC?, etc.). 1 1 This manuscript has been authored by UT-Battelle, LLC under Contract No. DE-AC05-00OR22725 with the U.S. Department of Energy. The United States Government retains and the publisher, by accepting the article for publication, acknowledges that the United States Government retains a nonexclusive, paid-up, irrevocable, world-wide license to publish or reproduce the published form of this manuscript, or allow others to do so, for United States Government purposes. The Department of Energy will provide public access to these results of federally sponsored research in accordance with the DOE Public Access Plan (http://energy.gov/downloads/doe-public-access-plan).
Chinthavali, Supriya↗
An Efficient Storage-Driven Machine Learning Model for Performance in the Era of Multimodal Scientific Data
Scientific workflows are increasingly relying on machine learning (ML), simulation, and hybrid techniques to predict, understand, and optimize the behavior of complex experiments. High-performance computing has greatly improved researchers’ ability to acquire diverse data modalities in these workflows. Recent studies suggest that the performance of machine learning models can be improved by integrating data from various sources. Unfortunately, these workloads pose unprecedent pressure on the network storage to meet the demands associated with accessing these multimodal data. To mitigate the impact of intensive IO, we propose a solution that utilizes a multi-tier High-Performance Computing (HPC) distributed storage and data processing framework, placing computation where the data resides for better performance. By adopting this project, the scientific community will gain new opportunities to explore multimodal storage-driven possibilities, integrating multiple scientific data sources with advanced streaming frameworks. Additionally, our framework effectively utilizes computing resources and bridges the gaps identified by HPC experts. Our proposed approach tackles scalability and persistence challenges by leveraging native persistency, which has posed difficulties in traditional approaches. Furthermore, we seek to enhance fault-tolerance and load-balance of computations by leveraging real-time streaming in diverse scientific computing environments, thereby propelling advanced scientific computing research into the next generation.
97 MATHEMATICS AND COMPUTING↗
MIGHTEE: the nature of the radio-loud AGN population
ABSTRACT We study the nature of the faint radio source population detected in the MeerKAT International GHz Tiered Extragalactic Exploration (MIGHTEE) Early Science data in the COSMOS field, focusing on the properties of the radio-loud active galactic nuclei (AGNs). Using the extensive multiwavelength data available in the field, we are able to classify 88 per cent of the 5223 radio sources in the field with host galaxy identifications as AGNs (35 per cent) or star-forming galaxies (54 per cent). We select a sample of radio-loud AGNs with redshifts out to z ∼ 6 and radio luminosities 1020 < L1.4 GHz/W Hz−1 < 1027 and classify them as high-excitation and low-excitation radio galaxies (HERGs and LERGs). The classification catalogue is released with this work. We find no significant difference in the host galaxy properties of the HERGs and LERGs in our sample. In contrast to previous work, we find that the HERGs and LERGs have very similar Eddington-scaled accretion rates; in particular we identify a population of very slowly accreting AGNs that are formally classified as HERGs at these low radio luminosities, where separating into HERGs and LERGs possibly becomes redundant. We investigate how black hole mass affects jet power, and find that a black hole mass ≳ 107.8 M⊙ is required to power a jet with mechanical power greater than the radiative luminosity of the AGN (Lmech/Lbol > 1). We discuss that both a high black hole mass and black hole spin may be necessary to launch and sustain a dominant radio jet.
Whittam, I. H. (ORCID:0000000322655983)↗
A spatially resolved spectral analysis of giant radio galaxies with MeerKAT
ABSTRACT In this study we report the spatially resolved, wideband spectral properties of three giant radio galaxies (GRGs) in the COSMOS field: MGTC J095959.63+024608.6, MGTC J100016.84+015133.0, and MGTC J100022.85+031520.4. One of these galaxies, MGTC J100022.85+031520.4, is reported here for the first time, with a projected linear size of 1.29 Mpc at a redshift of 0.1034. Unlike the other two, it is associated with a brightest cluster galaxy (BCG), making it one of the few GRGs known to inhabit cluster environments. We examine the spectral age distributions of the three GRGs using new MeerKAT UHF-band (544–1088 MHz) observations, and L-band (900–1670 MHz) data from the MeerKAT International GHz Tiered Extragalactic Exploration (MIGHTEE) survey. We test two models of spectral ageing, the Jaffe–Perola and Tribble models, using the Broadband Radio Astronomy Tools (brats) software, and find that they agree well with each other. We estimate the Tribble spectral age for MGTC J095959.63+024608.6 as 68 Myr, for MGTC J100016.84+015133.0 as 47 Myr, and for MGTC J100022.85+031520.4 as 67 Myr. We find significant disagreements between these spectral age estimates and the estimates of the dynamical ages of these GRGs, modelled in cluster and group environments. Our results highlight the need for additional processes that are not accounted for in either the dynamic age or the spectral age estimations.
Charlton, K. K. L. (ORCID:0000000229252047)↗
Host Onboarding Tool (HObT) v1.0.0
The Host OnBoarding Tool (Hobt) is a publicly accessible, web-based software designed to organize and share information about microbial hosts under development at the Agile BioFoundry (ABF). It streamlines the assessment, tracking, and sharing of information related to microbial host development and provides a centralized platform where users can rapidly evaluate hosts' readiness for various bio processes. HObT leverages the Tier System, a standardized host development framework that organizes and assesses microbial hosts based on their readiness for biomanufacturing. Each tier outlines key targets—including genetic tools, growth conditions, omics data, and predictive models—needed to transform new or emerging microbes into established production platforms. By applying clear criteria for advancement, the Tier System helps users quickly evaluate each organism's current development status, identify gaps in available knowledge or tools, and prioritize future strain improvement efforts. Through its user-friendly interface, HObT encourages contributions of new data and insights from researchers, fostering collaboration and accelerating host development. By providing structured guidance for microbial strain advancement, HObT and the Tier System support more systematic, rapid, and cost-effective development of non-traditional microbial hosts, ultimately enhancing the efficiency and impact of biomanufacturing research and applications.
Plahar, Hector [Lawrence Berkeley National Laborat↗
DETAIL Component Scaling and Methodology Comparison
The purpose of this study was to develop a process to convert input signals from one facility into another by reflecting geometric and environmental settings. The Dynamic Energy Transport and Integration Laboratory (DETAIL) is a research facility in development. Its aim is to emulate the daily interactions among power production industry systems and receive real-time data from those systems as inputs. To convert signals and ensure that the temporal sequences and magnitudes reflect laboratory settings, the ability to scale and project data is essential. To demonstrate this ability, Dynamical System Scaling (DSS) and Hierarchical Two-Tiered Scaling (H2TS) (methodologies that enable systems to scale and project or extrapolate data sets to desired environments while conserving the observed behavior based on first principles) were applied to DETAIL’s thermocline thermal storage system in the Thermal Energy Distribution System (TEDS) facility and solid-oxide electrolysis cell in the High Temperature Hydrogen Electrolysis (HTHE) facility. Both thermocline and electrolysis cell systems were successfully scaled, and test cases were conducted to generate a doubly accelerated energy charge and discharge in reference to past experimental data from the facilities. The research results represented a case for the thermocline system that required signals to be accelerated without altering the stored energy. To enhance the quality of the accelerated data, error propagation analyses were conducted on DSS post-processing terms to determine the consequences of raw-data-associated errors.
08 HYDROGEN↗
Grid Operator Analytics and Assessment Tools for Inverter- Based Resources Dominated Grid (GOAAT-IBR) Project Update
This presentation provides an update on the OPTIMA GOAAT project, with emphasis on the cloud-native data platform developed in-house to ingest, manage, and operationalize high-resolution power system data. Since our last NASPI presentation, accessible via OSTI ID #2671437, the project team advanced the design and deployment of a scalable architecture capable of handling both synchronized and non-synchronized streams, including PMU, point-on-wave (POW), COMTRADE, and SCADA data. These materials review the project status, recent progress, and key lessons learned. The core of the presentation examines the architecture and engineering of our cloud-native ingestion and data management platform. We then explain how pipelines were designed to collect, normalize, time-align, store, and serve heterogeneous data at scale. We will discuss design choices such as data models, streaming versus batch ingestion, storage tiers, and interoperability with analytics applications. Practical experiences with cloud-native technologies were shared during the event, including benefits, limitations, and integration challenges in a utility environment, along with methods used to improve performance, reduce latency, and optimize resource usage. The presentation also showcases user interface designs and visualization tools that convert raw measurements and analytics results into intuitive, actionable insights for operators and engineers. During the presentation examples were provided demonstrating how visualization, event views, and summarized analytics enhance situational awareness and support operational decision-making. These use cases illustrate how a well-designed data infrastructure can bridge the gap between high-volume measurements and practical grid operations.
Aminifar, Farrokh↗
SESAME: ASCII2 File Format
A new ASCII format for SESAME data is explicitly defined and dubbed ASCII2. This format fixes some of the onerous limitations of the legacy ASCII-styled format in addition to relaxing the FORTRAN style fixed-format layout of data tables. With the exception of the multi-tiered indexes of the legacy binary format, the new ASCII2 is more compatible with the binary in that it does not restrict the extent of various integer data (e.g., SESAME material and table numbers) to six digits nor does it restrict the extent or precision of the various floating point data. There will be very limited support of the legacy ASCII-styled formats in the future.
36 MATERIALS SCIENCE↗
Repurposing of the Run 2 CMS High Level Trigger Infrastructure as a Cloud Resource for Offline Computing
The former CMS Run 2 High Level Trigger (HLT) farm is one of the largest contributors to CMS compute resources, providing about 25k job slots for offline computing. This CPU farm was initially employed as an opportunistic resource, exploited during inter-fill periods, in the LHC Run 2. Since then, it has become a nearly transparent extension of the CMS capacity at CERN, being located on-site at the LHC interaction point 5 (P5), where the CMS detector is installed. This resource has been configured to support the execution of critical CMS tasks, such as prompt detector data reconstruction. It can therefore be used in combination with the dedicated Tier 0 capacity at CERN, in order to process and absorb peaks in the stream of data coming from the CMS detector. The initial configuration for this resource, based on statically configured VMs, provided the required level of functionality. However, regular operations of this cluster revealed certain limitations compared to the resource provisioning and use model employed in the case of WLCG sites. A new configuration, based on a vacuum-like model, has been implemented for this resource in order to solve the detected shortcomings. This paper reports about this redeployment work on the permanent cloud for an enhanced support to CMS offline computing, comparing the former and new models’ respective functionalities, along with the commissioning effort for the new setup.
46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗