Search NASA⌕ Search

SEARCH · Search NASA

Results for “Visual data”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 1,405 records · Page 78
All sourcesNASA NTRSDOE OSTI
DOE OSTI2023

MaNGA DynPop – I. Quality-assessed stellar dynamical modelling from integral-field spectroscopy of 10K nearby galaxies: a catalogue of masses, mass-to-light ratios, density profiles, and dark matter

ABSTRACT This is the first paper in our series on the combined analysis of the Dynamics and stellar Population (DynPop) for the MaNGA survey in the final SDSS Data Release 17 (DR17). Here, we present a catalogue of dynamically determined quantities for over 10 000 nearby galaxies based on integral-field stellar kinematics from the MaNGA survey. The dynamical properties are extracted using the axisymmetric Jeans Anisotropic Modelling (JAM) method, which was previously shown to be the most accurate for this kind of study. We assess systematic uncertainties using eight dynamical models with different assumptions. We use two orientations of the velocity ellipsoid: either cylindrically aligned JAMcyl or spherically aligned JAMsph. We also make four assumptions for the models’ dark versus luminous matter distributions: (1) mass-follows-light, (2) free NFW dark halo, (3) cosmologically constrained NFW halo, (4) generalized NFW dark halo, i.e. with free inner slope. In this catalogue, we provide the quantities related to the mass distributions (e.g. the density slopes and enclosed mass within a sphere of a given radius for total mass, stellar mass, and dark matter mass components). We also provide the complete models which can be used to compute the full luminous and mass distribution of each galaxy. Additionally, we visually assess the qualities of the models to help with model selections. We estimate the observed scatter in the measured quantities which decreases as expected with improvements in quality. For the best data quality, we find a remarkable consistency of measured quantities between different models, highlighting the robustness of the results.

Astronomy & Astrophysics↗
DOE OSTI2025

Enhancing synchrotron radiation micro-CT images using deep learning: an application of Noise2Inverse on bone imaging

In bone-imaging research, in situ synchrotron radiation micro-computed tomography (SRµCT) mechanical tests are used to investigate the mechanical properties of bone in relation to its microstructure. Low-dose computed tomography (CT) is used to preserve bone's mechanical properties from radiation damage, though it increases noise. To reduce this noise, the self-supervised deep learning method Noise2Inverse was used on low-dose SRµCT images where segmentation using traditional thresholding techniques was not possible. Simulated-dose datasets were created by sampling projection data at full, one-half, one-third, one-fourth and one-sixth frequencies of an in situ SRµCT mechanical test. After convolutional neural networks were trained, Noise2Inverse performance on all dose simulations was assessed visually and by analyzing bone microstructural features. Visually, high image quality was recovered for each simulated dose. Lacunae volume, lacunae aspect ratio and mineralization distributions shifted slightly in full, one-half and one-third dose network results, but were distorted in one-fourth and one-sixth dose network results. Following this, new models were trained using a larger dataset to determine differences between full dose and one-third dose simulations. Significant changes were found for all parameters of bone microstructure, indicating that a separate validation scan may be necessary to apply this technique for microstructure quantification. Noise present during data acquisition from the testing setup was determined to be the primary source of concern for Noise2Inverse viability. While these limitations exist, incorporating dose calculations and optimal imaging parameters enables self-supervised deep learning methods such as Noise2Inverse to be integrated into existing experiments to decrease radiation dose.

Obata, Yoshihiro (ORCID:0000000303659129)↗
DOE OSTI2024

Data-Informed Evaluation Framework for Integrated Energy Systems: Insights from Power, Process Heat, and Hydrogen Production Applications

The multi-criteria decision analysis (MCDA) framework provides a systematic evaluation of the diverse preferences and performance metrics associated with alternative solutions. This approach is advantageous over a single-criterion methodology, which are only valid under conditions that assume ceteris paribus or an "apples-to-apples" comparison. However, selecting suitable technologies for integrated energy systems (IES) can be likened to an "apples-to-oranges" comparison, given the heterogeneous factors at stake. These factors include economics and performance parameters, geological compatibility, and environmental impacts. Consequently, past research has often employed a mixture of qualitative and quantitative criteria tailored to the specific interests of each study. While the method proves effective in handling the intricate interplay of criteria, the resulting rankings and scores can vary from study to study. This inconsistency is introduced from the use of subjectively defined thresholds and weights. As a result, decision-makers frequently find it challenging to establish clear connections between specific criteria and the resulting scores, as the transformation of criteria into ordinal scores results in a substantial loss of information. To address this challenge, we introduce a data-informed IES evaluation framework that offers comprehensive, interpretable, and traceable evaluations backed by quantifiable rationale. First, we identified key IES evaluation criteria from a decade of literature, focusing on relevant IES applications in power, process heat, and hydrogen production. We leveraged state-of-the-art cost estimates from the Idaho National Laboratory (INL) and technical data from 78 reactor designs from the International Atomic Energy Agency (IAEA) and the Organization for Economic Co-operation and Development - Nuclear Energy Agency (OECD-NEA). Lastly, we established thresholds by analyzing the mean, variance, root mean square, and slope of values across alternatives, categorizing the preferences of decision-makers into distinct utility functions, such as linear, saturating, exponential, and stepwise. Our approach yielded two main outcomes: (1) it provided consistent assessments across different stakeholder groups and (2) it visualized uncertainties in the decision-making context via comprehensive sensitivity analysis. To demonstrate the impact of our framework, we conducted case studies on 6 reactor designs (AP1000, NuScale, BWRX-300, Xe-100, eVinci, iMSR) for the three applications. Our data-driven framework proved to be highly effective in addressing heterogenous uncertainties faced by varied decision-makers? preferences and IES applications, as well as cost and technical estimates of advanced reactors.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗
DOE OSTI2024

Data-Informed Evaluation Framework for Integrated Energy Systems: Insights from Power, Process Heat, and Hydrogen Production Applications

The multi-criteria decision analysis (MCDA) framework provides a systematic evaluation of the diverse preferences and performance metrics associated with alternative solutions. This approach is advantageous over a single-criterion methodology, which are only valid under conditions that assume ceteris paribus or an "apples-to-apples" comparison. However, selecting suitable technologies for integrated energy systems (IES) can be likened to an "apples-to-oranges" comparison, given the heterogeneous factors at stake. These factors include economics and performance parameters, geological compatibility, and environmental impacts. Consequently, past research has often employed a mixture of qualitative and quantitative criteria tailored to the specific interests of each study. While the method proves effective in handling the intricate interplay of criteria, the resulting rankings and scores can vary from study to study. This inconsistency is introduced from the use of subjectively defined thresholds and weights. As a result, decision-makers frequently find it challenging to establish clear connections between specific criteria and the resulting scores, as the transformation of criteria into ordinal scores results in a substantial loss of information. To address this challenge, we introduce a data-informed IES evaluation framework that offers comprehensive, interpretable, and traceable evaluations backed by quantifiable rationale. First, we identified key IES evaluation criteria from a decade of literature, focusing on relevant IES applications in power, process heat, and hydrogen production. We leveraged state-of-the-art cost estimates from the Idaho National Laboratory (INL) and technical data from 78 reactor designs from the International Atomic Energy Agency (IAEA) and the Organization for Economic Co-operation and Development - Nuclear Energy Agency (OECD-NEA). Lastly, we established thresholds by analyzing the mean, variance, root mean square, and slope of values across alternatives, categorizing the preferences of decision-makers into distinct utility functions, such as linear, saturating, exponential, and stepwise. Our approach yielded two main outcomes: (1) it provided consistent assessments across different stakeholder groups and (2) it visualized uncertainties in the decision-making context via comprehensive sensitivity analysis. To demonstrate the impact of our framework, we conducted case studies on 6 reactor designs (AP1000, NuScale, BWRX-300, Xe-100, eVinci, iMSR) for the three applications. Our data-driven framework proved to be highly effective in addressing heterogenous uncertainties faced by varied decision-makers? preferences and IES applications, as well as cost and technical estimates of advanced reactors.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗
DOE OSTI2021

Co-expression networks in Chlamydomonas reveal significant rhythmicity in batch cultures and empower gene function discovery

The unicellular green alga Chlamydomonas reinhardtii is a choice reference system for the study of photosynthesis and chloroplast metabolism, cilium assembly and function, lipid and starch metabolism, and metal homeostasis. Despite decades of research, the functions of thousands of genes remain largely unknown, and new approaches are needed to categorically assign genes to cellular pathways. Growing collections of transcriptome and proteome data now allow a systematic approach based on integrative co-expression analysis. We used a dataset comprising 518 deep transcriptome samples derived from 58 independent experiments to identify potential co-expression relationships between genes. We visualized co-expression potential with the R package corrplot, to easily assess co-expression and anti-correlation between genes. We extracted several hundred high-confidence genes at the intersection of multiple curated lists involved in cilia, cell division, and photosynthesis, illustrating the power of our method. Surprisingly, Chlamydomonas experiments retained a significant rhythmic component across the transcriptome, suggesting an underappreciated variable during sample collection, even in samples collected in constant light. Our results therefore document substantial residual synchronization in batch cultures, contrary to assumptions of asynchrony. We provide step-by-step protocols for the analysis of co-expression across transcriptome data sets from Chlamydomonas and other species to help foster gene function discovery.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2025

Heat transfer and visual observation of pool boiling of next-generation low-pressure refrigerant: R1336mzz(Z)

R1336mzz(Z) is a non-flammable and non-toxic hydrofluoroolefin (HFO), which is a potential alternative working fluid for multiple applications such as centrifugal chillers, industrial high temperature heat pumps, and Organic Rankine Cycles. Pool boiling is one of the primary heat transfer modes in shell-and-tube evaporators, and there is limited data for R1336mzz(Z) due to its recent introduction. This study investigated the pool boiling characteristics of R1336mzz(Z) using a closed-loop apparatus. Heat transfer coefficients on a smooth aluminum surface were measured at various heat fluxes with a heating block, and the results were compared to predictions from various correlations. Additionally, a high-speed camera was employed to visualize and analyze the pool boiling behavior of this next-generation low-pressure refrigerant.

Yang, Cheng-Min [ORNL]↗
DOE OSTI2020

Pyridoxal 5'-phosphate dependent reactions: Analyzing the mechanism of aspartate aminotransferase

Enzyme catalysis is the primary activity in energy and information metabolism and enzyme cofactors are key to the catalytic ability of most enzymes. Pyridoxal 5'-phosphate (PLP) cofactor, derived from Vitamin B6, is widely distributed in nature and has significant latitude in catalytic diversity. X-ray crystallography has revealed the structures of diverse PLP dependent enzymes from multiple families. But these structures are incomplete, lacking the positions of protons essential for understanding enzymatic mechanisms. Here, we review the diversity of PLP and discuss the use of neutron crystallography and joint X-ray/neutron refinement of Fold Type I aspartate aminotransferase to visualize the positions of protons in both the internal and external aldimine forms. Strategies used to prepare extremely large crystals required for neutron diffraction and the approach to data refinement including the PLP cofactor are discussed. The observed positions of protons, including one located in a previously unknown low-barrier hydrogen bond, have been used to create more accurate models for computational analysis. The results revealed a new mechanism for the transaminase reaction where hyperconjugation is key to reducing the energy barrier which finally provides a clear explanation of the Dunathan alignment.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2022

Developing Novel Performance Measures for Traffic Congestion Management and Operational Planning Based on Connected Vehicle Data

In this study, the authors present their efforts in exploring a new type of traffic data, referred to as internet-connected vehicle (ICV) data, for traffic congestion management and operational planning. Most currently manufactured vehicles contain onboard GPS and cellular modules, and they constantly connect to automobile manufacturers' clouds via cellular networks and upload their status. Some automobile manufacturers have recently redistributed the nonpersonal part of such data, such as geolocation, to third-party organizations for innovative applications. Compared with the traditional vehicle GPS data, the ICV data contain high-resolution GPS waypoints accompanied with the vehicles' abnormal moving events (e.g., hard braking). The ICV data also have huge potential in congestion management and operational planning. They explore to identify and analyze traffic congestion on both freeways and arterials using the ICV data. The ICV data adopted for this research are redistributed by Wejo Data Service, representing 10%-15% of all moving vehicles in the Dallas-Fort Worth (DFW) area in Texas. Through one case study for a freeway segment and one for an arterial segment, new traffic performance metrics based on the characteristics of ICV data have been presented. The highlights of these efforts are as follows: (I) queue length and propagation at freeway bottlenecks can be directly measured based on where and when most internet-connected vehicles slow down and join the queue; (II) an internet-connected vehicle's actual delay time on arterials can be directly measured according to its slow movement percentage, without assuming the nondelay travel speed; and (III) the ICV data set are also combined with the high-resolution traffic signal events to generate a ground-truth time-space diagram (TSD) on arterials - a common visualization of arterial signal performance for transportation planning and operations.

33 ADVANCED PROPULSION SYSTEMS↗
DOE OSTI2025

Planar shock-induced bubble collapse and jetting in water captured via X-ray phase contrast imaging

Shock wave–bubble interactions in water manifest rich dynamics driven by a combination of strong pressure and density mismatches. They have a wide variety of applications, including the injection of pharmaceuticals, and through scaling, enable the exploration of various aspects of high-energy-density systems such as inertial confinement fusion. In this work, the interaction between a micrometric nitrogen bubble and a planar shock wave, characterized by a Mach number of M = 1.24 and a peak pressure of p max = 0.57 , is experimentally recorded using ultra-high-speed x-ray phase contrast imaging. Highly resolved radiographs provide access to all phase discontinuities along the beam path, offering quantities such as the time-varying bubble size, the speed of a jet produced during the bubble collapse, and the time evolution of the shock wave front, which are critical benchmark data for numerical scheme validation. This study addresses the lack of well-characterized, repeatable, and high spatiotemporal resolution experiments at negative Atwood numbers by providing shock–bubble visualization and corresponding numerical simulation.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗
DOE OSTI2021

Detection of the Metabolic Response to Drought Stress Using Hyperspectral Reflectance

Drought is the most important limitation on crop yield. Understanding and detecting drought stress in crops is vital for improving water use efficiency through effective breeding and management. Leaf reflectance spectroscopy offers a rapid, non-destructive alternative to traditional techniques for measuring plant traits involved in a drought response. We measured drought stress in six glasshouse-grown agronomic species using physiological, biochemical and spectral data. In contrast to physiological traits, leaf metabolite concentrations revealed drought stress before it was visible to the naked eye. We used full-spectrum leaf reflectance data to predict metabolite concentrations using partial least-squares regression, with validation R2 values of 0.49 to 0.87. We show for the first time that spectroscopy may be used for the quantitative estimation of proline and abscisic acid, demonstrating the first use of hyperspectral data to detect a phytohormone. We used linear discriminant analysis and partial least squares discriminant analysis to differentiate between droughted and watered plants based on measured traits (accuracy: 71%) and raw spectral data (66%). Finally, we validated our glasshouse-developed models in an independent field trial. We demonstrate that spectroscopy can detect drought stress via underlying biochemical changes, before visual differences occur, representing a powerful advance for measuring limitations on yield.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2024

Investigating Resilience of Loops in HPC Programs: A Semantic Approach with LLMs

Soft errors have become one of the major concerns for the error resilience of the HPC applications as those errors may cause HPC applications to generate serious outcomes such as silent data corruptions (SDCs). Protecting the applications from soft errors is an essential while challenging task. Among different approaches, obtaining a profound understanding of the resilience proneness of an application is very important to devise efficient error detection and recovery strategies. Given the scale of the HPC applications both in the code size and execution time, there are often cases that the error propagation analysis on such applications would produce a massive volume of unstructured data, which requires a significant amount of efforts, to process and to obtain indicating actions towards error protection. In this paper, we present a control-flow based visual analysis framework to help the users conduct error propagation analysis and identify the critical sections of a program that may have a higher likelihood of leading to erroneous outcomes when affected by the control flow related errors. We also design and implement the scalable visualization framework - ResilienceVis that efficiently and effectively visualizes the affected program states under errors and the propagation traces for an application in a user-friendly manner, and eventually, we combine the analysis and visualization to exhibit the error-proneness of the different sections of applications.

Jiang, Hailong↗
DOE OSTI2021

Adaptive Spatially Aware I/O for Multiresolution Particle Data Layouts

Large-scale simulations on nonuniform particle distributions that evolve over time are widely used in cosmology, molecular dynamics, and engineering. Such data are often saved in an unstructured format that neither preserves spatial locality nor provides metadata for accelerating spatial or attribute subset queries, leading to poor performance of visualization tasks. Furthermore, the parallel I/O strategy used typically writes a file per process or a single shared file, neither of which is portable or scalable across different HPC systems. We present a portable technique for scalable, spatially aware adaptive aggregation that preserves spatial locality in the output. We evaluate our approach on two supercomputers, Stampede2 and Summit, and demonstrate that it outperforms prior approaches at scale, achieving up to 2.5× faster writes and reads for nonuniform distributions. Furthermore, the layout written by our method is directly suitable for visual analytics, supporting low-latency reads and attribute-based filtering with little overhead.

Usher, Will↗
DOE OSTI2024

Prediction of microstructure formation in laser powder bed fusion process.

The datasets are results analyzing the predicted microstructures in a single track during the laser powder bed fusion additive manufacturing process. They are the outputs by running the opensource code, muMatScale (The code can be cited at: Yuan, Lang, Fattebert, Jean-Luc, and Sabau, Adrian. (2023, August 03). muMatScale. [Computer software]. https://github.com/lang-yuan/muMatScale. https://doi.org/10.11578/dc.20240112.2.) For each set of data, it contains the time-dependent information of Temperature, Fraction Solid, Grain ID, Grain Angle ( crystallographic orientations by Euler angles), and solute Composition. The dataset can be visualized by Paraview (https://www.paraview.org/ ). The 6 datasets are: 1. Baseline_base1_n4e14_dt20: baseline case with bulk nucleation density of 4E+14/m^3, undercooling of 20K, substrate nucleation density of 1.5E+15/m^3 2. Nuc_n4e15_dt20: case with bulk nucleation density of 4E+15/m^3, undercooling of 20K, substrate nucleation density of 1.5E+15/m^3 3. Nuc_n4e15_dt50: case with bulk nucleation density of 4E+15/m^3, undercooling of 50K, substrate nucleation density of 1.5E+15/m^3 4. Nuc_n6e15_dt20: case with bulk nucleation density of 4E+16/m^3, undercooling of 20K, substrate nucleation density of 1.5E+15/m^3 5. Base4_n4e14_dt20: case with bulk nucleation density of 4E+14/m^3, undercooling of 20K, substrate nucleation density of 6.0E+15/m^3 6. Base16_n4e14_dt20: case with bulk nucleation density of 4E+14/m^3, undercooling of 20K, substrate nucleation density of 2.4E+16/m^3

36 MATERIALS SCIENCE↗
DOE OSTI2025

An experimental and computational investigation of the structure and spectroscopic signatures of α -UO 3

α-UO 3 is a common intermediate compound found in the nuclear fuel cycle, yet the exact crystal structure of this material has long been debated. Inconsistent computational and experimental data in previous works has led to varying conclusions between authors. Furthermore, to ensure the validity of our results in this work, the structural and spectroscopic signatures of pure phase α-UO 3 are investigated using powder X-ray diffraction and optical vibrational spectroscopy (infrared and Raman). Rietveld refinement of powder X-ray diffraction data on pure phase α-UO 3 collected in this work allows us to propose an alteration to the currently accepted C2mm structure (a = 3.9705 Å, b = 6.8553 Å, c = 4.15955 Å, α = β = γ = 90°) for α-UO 3 with no uranyl [UO 2 2+ ] bonds. Raman spectra collected using two excitation wavelengths (two instruments using 532 nm and one 785 nm) are presented, and differences with recently published results are discussed. Infrared spectra from two instruments used here agree well with recently published results, but the spectral range encompassed in our data extends past what has been reported with modern techniques. Additionally, we provide tentative vibrational mode assignments based on density functional perturbation theory calculations and resulting phonon eigenvector visualizations. Unexpected features in the optical vibrational spectra of α-UO 3 are explained by unique features in the structure we present.

38 RADIATION CHEMISTRY, RADIOCHEMISTRY, AND NUCLEA↗
DOE OSTI2023

Computed Tomography Scanning and Geophysical Measurements of the CarbonSAFE Seal Integrity Wells in the Illinois Basin

The U.S. Department of Energy’s (DOE) National Energy Technology Laboratory (NETL) researchers in Morgantown, West Virginia, utilized computed tomography (CT) facilities and the Multi-Sensor Core Logger (MSCL) to evaluate the integrity of three cores (Geophysical Monitor #2 (GM2), Verification Well #1 (VW #1), and Wabash #1 for the Illinois Basin Carbon Storage Assurance Facility Enterprise (CarbonSAFE) efforts. This work was carried out in collaboration with the Illinois State Geological Survey (ISGS) as part of their efforts to analyze cores from two field locations, including the Illinois Basin-Decatur Project (IBDP) and second stage Illinois Industrial Carbon Capture and Storage (IL-ICCS) sites at Decatur, Illinois, and the Wabash CarbonSAFE project at the Wabash Valley Resources Integrated Gasification Combined Cycle plant in Terre Haut, Indiana. The results of this study are presented in several formats and are available online on the Energy Data eXchange (EDX). The rock characterization was conducted using non-destructive techniques, allowing for future analysis of the cores. While the equipment used did not provide direct visualization of shale pore space, it allowed for detection of fractures and discontinuities. Low resolution CT imagery with the NETL medical CT scanner was performed on the entire core. Qualitative analysis of the medical CT images, coupled with x-ray fluorescence (XRF), P-wave, and magnetic susceptibility measurements from the MSCL were utilized to identify areas of interest for further study as well as fractured zones. The ability to quickly identify key areas for more detailed study with higher resolution will save time and resources in future studies. The combination of methods used provided a multi-scale analysis of this core and provided both a macro and micro description of the core that is relevant for many subsurface energy-related examinations traditionally performed at NETL.

58 GEOSCIENCES↗
DOE OSTI2022

LensingETC: A Tool to Optimize Multifilter Imaging Campaigns of Galaxy-scale Strong Lensing Systems

Imaging data is the principal observable required to use galaxy-scale strong lensing in a multitude of applications in extragalactic astrophysics and cosmology. In this paper, we develop Lensing Exposure Time Calculator (LensingETC; to optimize the efficiency of telescope-time usage when planning multifilter imaging campaigns for galaxy-scale strong lenses. This tool simulates realistic data tailored to specified instrument characteristics and then automatically models them to assess the power of the data in constraining lens model parameters. We demonstrate a use case of this tool by optimizing a two-filter observing strategy (in the IR and ultraviolet-visual (UVIS)) within the limited exposure time per system allowed by a Hubble Space Telescope (HST) Snapshot program. We find that higher resolution is more advantageous to gain constraining power on the lensing observables, when there is a trade-off between signal-to-noise ratio and resolution; for example, between the UVIS and IR filters of the HST. We also find that, whereas a point-spread function (PSF) with sub-Nyquist sampling allows the sample mean for a model parameter to be robustly recovered for both galaxy–galaxy and point-source lensing systems, a sub-Nyquist-sampled PSF introduces a larger scatter than a Nyquist-sampled one in the deviation from the ground truth for point-source lens systems.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗
DOE OSTI2021

Accelerating Multigrid-based Hierarchical Scientific Data Refactoring on GPUs

Rapid growth in scientific data and a widening gap between computational speed and I/O bandwidth make it increasingly infeasible to store and share all data produced by scientific simulations. Instead, we need methods for reducing data volumes: ideally, methods that can scale data volumes adaptively so as to enable negotiation of performance and fidelity tradeoffs in different situations. Multigrid-based hierarchical data representations hold promise as a solution to this problem, allowing for flexible conversion between different fidelities so that, for example, data can be created at high fidelity and then transferred or stored at lower fidelity via logically simple and mathematically sound operations. However, the effective use of such representations has been hindered until now by the relatively high costs of creating, accessing, reducing, and otherwise operating on such representations. We describe here highly optimized data refactoring kernels for GPU accelerators that enable efficient creation and manipulation of data in multigrid-based hierarchical forms. We demonstrate that our optimized design can achieve up to 250 TB/s aggregated data refactoring throughput—83% of theoretical peak—on 1024 nodes of the Summit supercomputer. We showcase our optimized design by applying it to a large-scale scientific visualization workflow and the MGARD lossy compression software.

Chen, Jieyang↗
DOE OSTI2024

DaYu: Optimizing Distributed Scientific Workflows by Decoding Dataflow Semantics and Dynamics

The combination of ever-growing scientific datasets and distributed workflow complexity creates I/O performance bottlenecks due to data volume, velocity, and variety. Although the increasing use of descriptive data formats (e.g., HDF5, netCDF) helps organize these datasets, it also creates obscure bottlenecks due to the need to translate high level operations into file addresses and then into low-level I/O operations. To address this challenge, we introduce DaYu, a method and toolset for analyzing (a) semantic relationships between logical datasets and file addresses, (b) how dataset operations translate into I/O, and (c) the combination across entire workflows. DaYu's analysis and visualization enables identification of critical bottlenecks and reasoning about remediation. We describe our methodology and propose optimization guidelines. Evaluation on scientific workflows demonstrates up to 3.7x performance improvements in I/O time for obscure bottlenecks. The time and storage overhead for DaYu's time-ordered data is typically under 0.2% of runtime and 0.25% of data volume, respectively.

Tang, Meng↗