Search NASA⌕ Search

SEARCH · Search NASA

Results for “collective input/output”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

14 records
All sourcesNASA NTRSDOE OSTI
DOE OSTI2014

Collective input/output under memory constraints

Compared with current high-performance computing (HPC) systems, exascale systems are expected to have much less memory per node, which can significantly reduce necessary collective input/output (I/O) performance. In this study, we introduce a memory-conscious collective I/O strategy that takes into account memory capacity and bandwidth constraints. The new strategy restricts aggregation data traffic within disjointed subgroups, coordinates I/O accesses in intranode and internode layers, and determines I/O aggregators at run time considering memory consumption among processes. We have prototyped the design and evaluated it with commonly used benchmarks to verify its potential. The evaluation results demonstrate that this strategy holds promise in mitigating the memory pressure, alleviating the contention for memory bandwidth, and improving the I/O performance for projected extreme-scale systems. Given the importance of supporting increasingly data-intensive workloads and projected memory constraints on increasingly larger scale HPC systems, this new memory-conscious collective I/O can have a significant positive impact on scientific discovery productivity.

collective input/output↗
DOE OSTI2019

Adaptive Learning for Concept Drift in Application Performance Modeling

Supervised learning is a promising approach for modeling the performance of applications running on large HPC systems. A key assumption in supervised learning is that the training and testing data are obtained under the same conditions. However, in production HPC systems these conditions might not hold because the conditions of the platform can change over time as a result of hardware degradation, hardware replacement, software upgrade, and configuration updates. These changes could alter the data distribution in a way that affects the accuracy of the predictive performance models and render them less useful; this phenomenon is referred to as concept drift. Ignoring concept drift can lead to suboptimal resource usage and decreased efficiency when those performance models are deployed for tuning and job scheduling in production systems. To address this issue, we propose a concept-drift-aware predictive modeling approach that comprises two components: (1) an online Bayesian changepoint detection method that can automatically identify the location of events that lead to concept drift in near-real time and (2) a moment-matching transformation inspired by transfer learning that converts the training data collected before the drift to be useful for retraining. We use application input/output performance data collected on Cori, a production supercomputing system at the National Energy Research Scientific Computing Center, to demonstrate the effectiveness of our approach. The results show that concept-drift-aware models obtain significant improvement in accuracy; the median absolute error of the best-performing Gaussian process regression improved by 58.8% when the proposed approaches were used.

adaptive learning↗
DOE OSTI2024

Codiscovering graphical structure and functional relationships within data: A Gaussian Process framework for connecting the dots

Most problems within and beyond the scientific domain can be framed into one of the following three levels of complexity of function approximation. Type 1: Approximate an unknown function given input/output data. Type 2: Consider a collection of variables and functions, some of which are unknown, indexed by the nodes and hyperedges of a hypergraph (a generalized graph where edges can connect more than two vertices). Given partial observations of the variables of the hypergraph (satisfying the functional dependencies imposed by its structure), approximate all the unobserved variables and unknown functions. Type 3: Expanding on Type 2, if the hypergraph structure itself is unknown, use partial observations of the variables of the hypergraph to discover its structure and approximate its unknown functions. These hypergraphs offer a natural platform for organizing, communicating, and processing computational knowledge. While most scientific problems can be framed as the data-driven discovery of unknown functions in a computational hypergraph whose structure is known (Type 2), many require the data-driven discovery of the structure (connectivity) of the hypergraph itself (Type 3). We introduce an interpretable Gaussian Process (GP) framework for such (Type 3) problems that does not require randomization of the data, access to or control over its sampling, or sparsity of the unknown functions in a known or learned basis. Its polynomial complexity, which contrasts sharply with the super-exponential complexity of causal inference methods, is enabled by the nonlinear ANOVA capabilities of GPs used as a sensing mechanism.

Science & Technology - Other Topics↗
DOE OSTI2024

Julienne v1.0.0

Julienne is a compiler-portable unit-testing framework for Fortran software projects, including those that use the parallel/accelerator-programming features of Fortran 2023. Julienne achieves portability across compilers through minimalism and isolation. The minimal design ensures that Julienne uses only features supported by the majority of Fortran compilers. The isolation through zero dependencies ensures that no other projects block Julienne from building with a particular compiler. Julienne also contains with additional services that support its unit-testing code. These include functions for manipulating strings, command lines and input/output format strings; and a user-defined collective subroutine for verifying that all processes pass a test in parallel testing. Julienne's name derives from the term for vegetables sliced into thin strings: julienne vegetables. Julienne captures the authors' most frequently used thin slice of the Veggies and Sourcery software repositories while avoiding certain compiler limitations of the those two packages.

Rouson, Damian↗
DOE OSTI2017

Virtualized hardware environments for supporting digital I and C verification - 324

The technology of virtualized hardware testing platforms within the nuclear context in order to provide a framework for evaluating virtualized hardware/software approaches is discussed. To understand the applicability of virtualized hardware approaches, we have developed a virtual hardware model of an actual smart sensor using the Imperas product OVPsim development environment, as is seen in . An evaluation of the modeling effort required to create the digital I and C smart sensor is provided, followed by an analysis and findings on the various tools in the OVPsim environment for supporting testing, evidence collecting, and analysis. The analyzed tools include traditional input/output testing, model-based testing, mutation testing, and fault injection techniques. Finally, the goal of this research effort is to analyze virtual hardware platforms and present findings to the nuclear community on the efficacy and viability of this technology as a complementary means to actual hardware-in-the-loop testing Factory Acceptance Testing for Nuclear Power Plant (NPP) digital I and C systems. (authors)

22 GENERAL STUDIES OF NUCLEAR REACTORS↗
DOE OSTI2022

Modeling pre-Exascale AMR Parallel I/O Workloads via Proxy Applications

The present work investigates the modeling of preexascale input/output (I/O) workloads of Adaptive Mesh Refinement (AMR) simulations through a simple proxy application. We collect data from the AMReX Castro framework running on the Summit supercomputer for a wide range of scales and mesh partitions for the hydrodynamic Sedov case as a baseline to provide sufficient coverage to the formulated proxy model. The non-linear analysis data production rates are quantified as a function of a set of input parameters such as output frequency, grid size, number of levels, and the Courant-Friedrichs-Lewy (CFL) condition number for each rank, mesh level and simulation time step. Linear regression is then applied to formulate a simple analytical model which allows to translate AMReX inputs into MACSio proxy I/O application parameters, resulting in a simple “kernel” approximation for data production at each time step. Results show that MACSio can simulate actual AMReX nonlinear “static” I/O workloads to a certain degree of confidence on the Summit supercomputer using the present methodology. The goal is to provide an initial level of understanding of AMR I/O workloads via lightweight proxy applications models to facilitate autotune data management strategies in anticipation of exascale systems.

Godoy, William↗
DOE OSTI2021

Scalable FBP decomposition for cone-beam CT reconstruction

Filtered Back-Projection (FBP) is a fundamental compute intense algorithm used in tomographic image reconstruction. Cone-Beam Computed Tomography (CBCT) devices use a cone-shaped X-ray beam, in comparison to the parallel beam used in older CT generations. Distributed image reconstruction of cone-beam datasets typically relies on dividing batches of images into different nodes. This simple input decomposition, however, introduces limits on input/output sizes and scalability.We propose a novel decomposition scheme and reconstruction algorithm for distributed FPB. This scheme enables arbitrarily large input/output sizes, eliminates the redundancy arising in the end-to-end pipeline and improves the scalability by replacing two communication collectives with only one segmented reduction. Finally, we implement the proposed decomposition scheme in a framework that is useful for all current-generation CT devices (7th gen). In our experiments using up to 1024 GPUs, our framework can construct 40963 volumes, for real-world datasets, in under 16 seconds (including I/O).

Chen, Peng↗
DOE OSTI2026

A network of soil moisture, soil temperature, air temperature, net radiation, ground heat flux and ground water for Chicago, Illinois

This dataset contains environmental monitoring data collected using solar-powered Multi-Function Research (MFR) Long Range Wide Area (LoRaWAN)-enabled nodes at 11 sites in Chicago, Illinois, as part of the DOE Urban Integrated Field Lab CROCUS project. The MFR node system consists of an Input/Output Digital Input Module (IB8) interface box (ICT International) providing wired connections for environmental sensors and an MFR-Node-L data logger that manages power, data processing, and LoRaWAN communication. The wireless data are ingested via Sage network (https://sagecontinuum.org/) nodes that contain LoRaWAN antennae. Measurements were collected from 11 MFR nodes deployed across Chicago State University (CSU), Northeastern Illinois University (NEIU), Northwestern University (NU), University of Illinois Chicago (UIC), West Woodlawn "Blacks in Green" (BIG), and Indian Boundary Prairies (IBP). Each MFR node supports a consistent suite of sensors measuring atmospheric, soil, and hydrological variables. Atmospheric measurements include 2m air temperature (°C), 2m vapor pressure deficit (kPa), and 2m shortwave/longwave radiation (incoming and outgoing, W/m²) measured using ATH-VPD and Apogee SN500 sensors. Soil measurements include volumetric water content (VWC, %) and temperature (°C) at four depths (15, 30, 45, and 60 cm below surface) using Meter Teros54 sensors, and heat flux (W/m²) at 10 cm depth using Huske HFP01-05 sensors. At selected locations, Meter Hydros21 sensors measure groundwater depth (mm), specific conductivity (dS/m), and temperature (°C). The dataset includes timestamps, site identifiers with location names, device IDs, Global Positioning System (GPS) coordinates, variable names with units, measurement depths, values, sensor names, and Sage node identifiers. All timestamps are in local Chicago time (CDT/CST). Quality control flags are provided using a 6-bit binary system indicating physical range violations, step spikes, 24-hour flat-line conditions, 6-hour jitter, 7-day ultra-low variance, and persistent high offset. Data is provided in CSV and CF-compliant NetCDF formats. This dataset is part of a larger collection of CROCUS environmental monitoring data, including linked datasets from Air Quality Transmitter (AQT) sensors, Weather Transmitter (WXT) sensors, and Sap Flow Meter (SFM1x) sensors.

Chicago↗
DOE OSTI2023

Distributed out-of-memory NMF on CPU/GPU architectures

We propose an efficient distributed out-of-memory implementation of the non-negative matrix factorization (NMF) algorithm for heterogeneous high-performance-computing systems. The proposed implementation is based on prior work on NMFk, which can perform automatic model selection and extract latent variables and patterns from data. In this work, we extend NMFk by adding support for dense and sparse matrix operation on multi-node, multi-GPU systems. The resulting algorithm is optimized for out-of-memory problems where the memory required to factorize a given matrix is greater than the available GPU memory. Memory complexity is reduced by batching/tiling strategies, and sparse and dense matrix operations are significantly accelerated with GPU cores (or tensor cores when available). Input/output latency associated with batch copies between host and device is hidden using CUDA streams to overlap data transfers and compute asynchronously, and latency associated with collective communications (both intra-node and inter-node) is reduced using optimized NVIDIA Collective Communication Library (NCCL) based communicators. Benchmark results show significant improvement, from 32X to 76x speedup, with the new implementation using GPUs over the CPU-based NMFk. Good weak scaling was demonstrated on up to 4096 multi-GPU cluster nodes with approximately 25,000 GPUs when decomposing a dense 340 Terabyte-size matrix and an 11 Exabyte-size sparse matrix of density 10 -6 .

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2023

JuliaOceanWaves

SAND2023-05107O JuliaOceanWaves is a collection of packages and functions for modeling wave energy converters. The open-source software is designed to be used by the wave energy community to model new concepts, perform design optimization, and more. Capabilities include: • A representation of ocean wave spectra and input/output (e.g. read from National Oceanic and Atmospheric Administration buoys) and post-processing functionalities; • A solver for the linear potential flow radiation and diffraction equations for wave forcing on floating structures. • Solving the floating body linear dynamics in the frequency domain. Future additions to the software will include solving non-linear floating body dynamics in the time-domain through time-marching Ordinary Differential Equation solvers and the pseudo-spectral optimization method. Sandia National Laboratories is a multimission laboratory managed and operated by National Technology & Engineering Solutions of Sandia, LLC, a wholly owned subsidiary of Honeywell International Inc., for the U.S. Department of Energy’s National Nuclear Security Administration under contract DE-NA0003525.

Michelen Strofer, Carlos↗
DOE OSTI2015

Rethinking key–value store for parallel I/O optimization

Key-value stores are being widely used as the storage system for large-scale internet services and cloud storage systems. However, they are rarely used in HPC systems, where parallel file systems are the dominant storage solution. In this study, we examine the architecture differences and performance characteristics of parallel file systems and key-value stores. We propose using key-value stores to optimize overall Input/Output (I/O) performance, especially for workloads that parallel file systems cannot handle well, such as the cases with intense data synchronization or heavy metadata operations. We conducted experiments with several synthetic benchmarks, an I/O benchmark, and a real application. We modeled the performance of these two systems using collected data from our experiments, and we provide a predictive method to identify which system offers better I/O performance given a specific workload. The results show that we can optimize the I/O performance in HPC systems by utilizing key-value stores.

Hyperdex↗
DOE OSTI2021

Open Radiation Monitoring: Conceptual System Design

The Open Radiation Monitoring (ORM) Project seeks to develop and demonstrate a modular radiation detection architecture designed specifically for use in arms control treaty verification (ACTV) applications that will facilitate rapid development of trusted systems to meet the needs of potential future treaties. Development of trusted systems to support potential future treaties is a complex and costly endeavor that typically results in a purpose-built system designed to perform one specific task. The majority of prior trusted system development efforts have relied on the use of commercial embedded computers or microprocessors to control the system and process the acquired data. These processors are complex, making authentication and certification of measurement systems and collected data challenging and time consuming. We believe that a modular architecture can be used to reduce more complex systems to a series of single-purpose building blocks that could be used to implement a variety of detection modalities with shared functionalities. With proper design, the functionality of individual modules can be confirmed through simple input/output testing, thereby facilitating equipment inspection and in turn building trust in the equipment by all treaty parties. Furthermore, a modular architecture can be used to control data flow within the measurement system, reducing the risk of "hidden switches" and constraining the amount of sensitive information that could potentially be inadvertently leaked. This report documents a conceptual modular system architecture that is designed to facilitate inspection in an effort to reduce overall authentication and certification burden. As of publication, this architecture remains in a conceptual phase and additional funding is required to prove out the utility of a modular architecture and test the assumptions used to rationalize the design.

61 RADIATION PROTECTION AND DOSIMETRY↗
DOE OSTI2015

Fast equilibration of coarse-grained polymeric liquids

The study of macromolecular systems may require large computer simulations that are too time consuming and resource intensive to execute in full atomic detail. The integral equation coarse-graining approach by Guenza and co-workers enables the exploration of longer time and spatial scales without sacrificing thermodynamic consistency, by approximating collections of atoms using analytically-derived soft-sphere potentials. Because coarse-grained (CG) characterizations evolve polymer systems far more efficiently than the corresponding united atom (UA) descriptions, we can feasibly equilibrate a CG system to a reasonable geometry, then transform back to the UA description for a more complete equilibration. Automating the transformation between the two different representations simultaneously exploits CG efficiency and UA accuracy. By iteratively mapping back and forth between CG and UA, we can quickly guide the simulation towards a configuration that would have taken many more time steps within the UA representation alone. Accomplishing this feat requires a diligent workflow for managing input/output coordinate data between the different steps, deriving the potential at runtime, and inspecting convergence. Here in this article, we present a lightweight workflow environment that accomplishes such fast equilibration without user intervention. The workflow supports automated mapping between the CG and UA descriptions in an iterative, scalable, and customizable manner. We describe this technique, examine its feasibility, and analyze its correctness.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2019

FELIX-Based Readout of the Single-Phase ProtoDUNE Detector

Large liquid argon (LAr) time projection chambers (TPCs) have been adopted for the Deep Underground Neutrino Experiment (DUNE) experiment's far detector, which will be composed of four 17-kton detectors situated 1.5 km underground at the Sanford Underground Research Facility. This represents a large increase in scale compared to existing experiments. Both single- and dual-phase technologies will be validated at CERN, in cryostats capable of accommodating full-size detector modules, and exposed to low-energy charged particle beams. This program, called ProtoDUNE, also allows for extensive tests of data acquisition strategies. The Front-End LInk eXchange (FELIX) readout system was initially developed within the ATLAS collaboration and is based on custom field-programmable gate array (FPGA)-based Peripheral Component Interconnect Express input/output cards, connected through point-to-point links to the detector front end and hosted in commodity servers. FELIX will be used in the single-phase ProtoDUNE setup to read the data coming from 2560 anode wires organized in a single anode plane assembly (APS) structure. With a continuous readout at a sampling rate of 2 MHz, the system must deal with an input rate of 96 Gb/s. An external trigger will preselect time windows of 5 ms with interesting activity expected inside the detector. Event building will occur for triggered events, at a target rate of 25 Hz; the readout system will form fragments from the data samples matching the time window, carry out lossless compression, and forward the data to event building nodes over 10-Gb/s Ethernet. This paper discusses the design and implementation of this readout system as well as the first operational experience.

47 OTHER INSTRUMENTATION↗