A Data Toolkit for Ultrahigh-resolution E3SM Simulation on Massively Parallel Processing Systems
Explore the source record for details and available documents.
Wang, Dali↗
SEARCH · Search NASA
Results for “Massive Data”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
VISILIENCE: An Interactive Visualization Framework for Resilience Analysis using Control-Flow Graph
Soft errors have become one of the major concerns for the error resilience of HPC applications, as those errors can cause HPC applications to generate serious outcomes such as Silent Data Corruptions (SDCs). A large body of approaches has been proposed to analyze the resilience of HPC applications. However, existing studies rarely address the challenges of the analysis result perception. Specifically, resilience analysis techniques often produce a massive volume of unstructured data, making it difficult for programmers to conduct the resilience analysis due to non-intuitive raw data. Furthermore, different analysis models produce diverse results with multiple levels of details, which may create hurdles to compare and explore the resilience of HPC program execution. To this end, we present VISILIENCE, an interactive VISual resILIENCE analysis framework to allow programmers to facilitate the resilience analysis of HPC applications. In particular, VISILIENCE leverages an effective visualization approach Control Flow Graph (CFG) to present a function execution. In addition, three widely-used models for resilience analysis (i.e., Y-Branch, IPAS, and TRIDENT) are seamlessly embedded into the framework for resilience analysis and result comparison. Multiple case studies have been conducted to demonstrate the effectiveness of our proposed framework VISILIENCE.
Jiang, Hailong↗
The Portals 4.3 Network Programming Interface
This report presents a specification for the Portals 4 network programming interface. Portals 4 is intended to allow scalable, high-performance network communication between nodes of a parallel computing system. Portals 4 is well suited to massively parallel processing and embedded systems. Portals 4 represents an adaption of the data movement layer developed for massively parallel processing platforms, such as the 4500-node Intel TeraFLOPS machine. Sandia's Cplant cluster project motivated the development of Version 3.0, which was later extended to Version 3.3 as part of the Cray Red Storm machine and XT line. Version 4 is targeted to the next generation of machines employing advanced network interface architectures that support enhanced offload capabilities.
97 MATHEMATICS AND COMPUTING↗
CoreCruncher : Fast and Robust Construction of Core Genomes in Large Prokaryotic Data Sets
The core genome represents the set of genes shared by all, or nearly all, strains of a given population or species of prokaryotes. Inferring the core genome is integral to many genomic analyses, however, most methods rely on the comparison of all the pairs of genomes; a step that is becoming increasingly difficult given the massive accumulation of genomic data. Here, we present CoreCruncher; a program that robustly and rapidly constructs core genomes across hundreds or thousands of genomes. CoreCruncher does not compute all pairwise genome comparisons and uses a heuristic based on the distributions of identity scores to classify sequences as orthologs or paralogs/xenologs. Although it is much faster than current methods, our results indicate that our approach is more conservative than other tools and less sensitive to the presence of paralogs and xenologs. CoreCruncher is freely available from: https://github.com/lbobay/CoreCruncher. CoreCruncher is written in Python 3.7 and can also run on Python 2.7 without modification. It requires the python library Numpy and either Usearch or Blast. Certain options require the programs muscle or mafft.
59 BASIC BIOLOGICAL SCIENCES↗
Cosmological Constraints from Gas Mass Fractions of Massive, Relaxed Galaxy Clusters
We present updated cosmological constraints from measurements of the gas mass fractions (f(gas)) of massive, dynamically relaxed galaxy clusters. Our new data set has greater leverage on models of dark energy, thanks to the addition of the Perseus cluster at low redshifts, two new clusters at redshifts z greater than or similar to 1, and significantly longer observations of four clusters at 0.6 < z < 0.9. Our low-redshift (z < 0.16) f(gas) data, combined with the cosmic baryon fraction measured from the cosmic microwave background (CMB), imply a Hubble constant of h = 0.722 +/- 0.067. Combining the full f(gas) data set with priors on the cosmic baryon density and the Hubble constant, we constrain the dark energy density to be Omega(Lambda) = 0.865 +/- 0.119 in non-flat Lambda cold dark matter (cosmological constant) models, and its equation of state to be in flat, constant-w models, respectively 41percent and 29percent tighter than our previous work, and comparable to the best constraints available from other probes. Combining f(gas), CMB, supernova, and baryon acoustic oscillation data, we also constrain models with global curvature and evolving dark energy. For the massive, relaxed clusters employed here, we find the scaling of f(gas) with mass to be consistent with a constant, with an intrinsic scatter that corresponds to just similar to 3percent in distance.
clusters: general – X-rays:↗
DEMONSTRATION OF A DATA-DRIVEN PHYSICS-BASED APPROACH FOR COMPUTATIONALLY EFFICIENT CFD PREDICTION OF TWO-PHASE BUBBLY FLOW
To realize efficient computational fluid dynamics (CFD) prediction of two-phase flow, a multi-scale physics-guided data-driven approach, Feature Similarity Measurement (FSM) technique was developed for error estimation in two-phase flow simulation using coarse-mesh CFD, to achieve a comparable accuracy as fine-mesh simulations with fast-running feature. By defining physics-guided parameters and variable gradients as physical features, FSM has the capability to capture the underlying local patterns in the coarse-mesh CFD simulation. Massive low-fidelity data and respective high-fidelity data are used to explore the underlying information relevant to the main simulation errors and the effects of phenomenological scaling. By learning from previous simulation data, a surrogate model using deep feedforward neural network (DFNN) can be developed and trained to estimate the simulation error of coarse-mesh CFD. In a demonstration case of two-phase bubbly flow, the DFNN model well captured and corrected the unphysical “peaks” in the velocity and void fraction profiles near the wall in the coarse-mesh configuration, even for extrapolative predictions. The research documented supports the feasibility of the physics-guided deep learning methods for coarse mesh CFD simulations which has a potential for the efficient industrial design.
42 ENGINEERING↗
Investigating Resilience of Loops in HPC Programs: A Semantic Approach with LLMs
Soft errors have become one of the major concerns for the error resilience of the HPC applications as those errors may cause HPC applications to generate serious outcomes such as silent data corruptions (SDCs). Protecting the applications from soft errors is an essential while challenging task. Among different approaches, obtaining a profound understanding of the resilience proneness of an application is very important to devise efficient error detection and recovery strategies. Given the scale of the HPC applications both in the code size and execution time, there are often cases that the error propagation analysis on such applications would produce a massive volume of unstructured data, which requires a significant amount of efforts, to process and to obtain indicating actions towards error protection. In this paper, we present a control-flow based visual analysis framework to help the users conduct error propagation analysis and identify the critical sections of a program that may have a higher likelihood of leading to erroneous outcomes when affected by the control flow related errors. We also design and implement the scalable visualization framework - ResilienceVis that efficiently and effectively visualizes the affected program states under errors and the propagation traces for an application in a user-friendly manner, and eventually, we combine the analysis and visualization to exhibit the error-proneness of the different sections of applications.
Jiang, Hailong↗
Computationally efficient CFD prediction of bubbly flow using physics-guided deep learning
To realize efficient computational fluid dynamics (CFD) prediction of two-phase flow, a multi-scale framework was proposed in this paper by applying a physics-guided data-driven approach. Instrumental to this framework, Feature Similarity Measurement (FSM) technique was developed for error estimation in two-phase flow simulation using coarse-mesh CFD, to achieve a comparable accuracy as fine-mesh simulations with fast-running feature. In this work, by defining physics-guided parameters and variable gradients as physical features, FSM has the capability to capture the underlying local patterns in the coarse-mesh CFD simulation. Massive low-fidelity data and respective high-fidelity data are used to explore the underlying information relevant to the main simulation errors and the effects of phenomenological scaling. By learning from previous simulation data, a surrogate model using deep feedforward neural network (DFNN) can be developed and trained to estimate the simulation error of coarse-mesh CFD. In a demonstration case of two-phase bubbly flow, the DFNN model well captured and corrected the unphysical “peaks” in the velocity and void fraction profiles near the wall in the coarse-mesh configuration, even for extrapolative predictions. The research documented supports the feasibility of the physics-guided deep learning methods for coarse mesh CFD simulations which has a potential for the efficient industrial design.
22 GENERAL STUDIES OF NUCLEAR REACTORS↗
Detecting the stochastic gravitational wave background from massive gravity with pulsar timing arrays
We explore the potential of pulsar timing arrays (PTAs) such as NANOGrav, EPTA, and PPTA to detect the stochastic gravitational wave background in theories of massive gravity. In general relativity, the function describing the dependence of the correlation between the arrival times of signals from two pulsars on the angle between them is known as the Hellings-Downs curve. We compute the analogous overlap reduction function for massive gravity, including the additional polarization states and the correction due to the mass of the graviton, and compare the result with the Hellings-Downs curve. The primary result is a complete analytical form for the analog Hellings-Downs curve, providing a starting point for future numerical studies aimed at a detailed comparison between PTA data and the predictions of massive gravity. We study both the massless limit and the stationary limit as checks on our calculation, and discuss how our formalism also allows us to study the impact of massive spin-2 dark matter candidates on data from PTAs.
71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗
Multiscale Modeling Meets Machine Learning: What Can We Learn?
Machine learning is increasingly recognized as a promising technology in the biological, biomedical, and behavioral sciences. There can be no argument that this technique is incredibly successful in image recognition with immediate applications in diagnostics including electrophysiology, radiology, or pathology, where we have access to massive amounts of annotated data. However, machine learning often performs poorly in prognosis, especially when dealing with sparse data. This is a field where classical physics-based simulation seems to remain irreplaceable. In this review, we identify areas in the biomedical sciences where machine learning and multiscale modeling can mutually benefit from one another: Machine learning can integrate physics-based knowledge in the form of governing equations, boundary conditions, or constraints to manage ill-posted problems and robustly handle sparse and noisy data; multiscale modeling can integrate machine learn- ing to create surrogate models, identify system dynamics and parameters, analyze sensitivities, and quantify uncertainty to bridge the scales and understand the emergence of function. With a view towards applications in the life sciences, we discuss the state of the art of combining machine learning and multiscale modeling, identify applications and opportunities, raise open questions, and address potential challenges and limitations. We anticipate that it will stimulate discussion within the community of computational mechanics and reach out to other disciplines including mathematics, statistics, computer science, artificial intelligence, biomedicine, systems biology, and precision medicine to join forces towards creating robust and efficient models for biological systems.
machine learning, multiscale modeling, physics-bas↗
Session Introduction: Graph Representations and Algorithms in Biomedicine
Connectivity is a fundamental property of biological systems: on the cellular level, proteins interact with each other to form protein-protein interaction networks (PPIs); on the organism level, neurons are arranged in a network; and on a community-level, species can have complex relationships with one another that drive the development and balance of an ecosystem. Graphs, representations of systems consisting of entities as vertices and their connections as edges, are a useful structure to characterize many such systems. Such models can be used to understand biological systems that naturally have a network structure, including PPIs, biological neurons, and ecosystems. In today’s information age, graph representations and algorithms (often in combination with machine learning techniques) are used to organize massive amounts of related data, much of which may be heterogeneous or unstructured, and identify patterns that represent novel biological insights. PSB’s 2023 session “Graph Representations and algorithms in Biomedicine,” encompasses modern developments in graph theory and its applications to various fields of biomedicine. This session includes a wide range of research - knowledge graphs built from text-mined health data, heterogeneous networks using multi-omic databases, and graphs refined to represent uncertainty or improve memory usage.
Chrisman, Brianna S.↗
Stealthy Cyber Anomaly Detection On Large Noisy Multi-material 3D Printer Datasets Using Probabilistic Models
As Additive Layer Manufacturing (ALM) becomes pervasive in industry, its applications in safety critical component manufacturing are being explored and adopted. However, ALM's reliance on embedded computing renders it vulnerable to tampering through cyber-attacks. Sensor instrumentation of ALM devices allows for rigorous process and security monitoring, but also results in a massive volume of noisy data for each run. As such, in-situ, near-real-time anomaly detection is very challenging. The ideal algorithm for this context is simple, computationally efficient, minimizes false positives, and is accurate enough to resolve small deviations. In this paper, we present a probabilistic-model-based approach to address this challenge. To test our approach, we analyze current measurements from a polymer composite 3D printer during emulated tampering attacks. Our results show that our approach can consistently and efficiently locate small changes in the presence of substantial operational noise.
Yoginath, Srikanth↗
TURBO: Terabits/s Using Reconfigurable Bandwidth Optics (Final Report)
Large scale, big science applications are generating petabytes of data that need to be shared among multiple locations. Moving a petabyte of data at 100 Gb/s takes 22 hours. Reducing this to single digit hours or below is clearly attractive and with future applications anticipating exabyte scale transfers, multi-terabit per second capacities are needed. This scale of capacity is available using optical systems. Core optical systems have aggregate per fiber capacities on the order of 10 Tb/s and more than 100 Tb/s system experiments have been realized in the lab. But large optical capacities are not just reserved for the core as similar system capacities are available in metro networks right up to the enterprise, campus, or data center. In general, the same technology is used today in the metro area as in the long-haul networks—reconfigurable optical add drop multiplexing (ROADM) node-based wavelength division multiplexing (WDM) systems. Data centers already support massive internal capacities on the scale of 100’s of Tb/s. As new photonic integrated technologies mature, for example through initiatives such as the Integrated Photonics Innovative Manufacturing Institute (IP-IMI), the cost of optical interfaces is expected to decrease, and higher capacity and higher performance interfaces will become more affordable. This set of circumstances creates the potential that Tb/s capacities will be available at the enterprise and campus level to support large scale science network applications.
42 ENGINEERING↗
Workshop Report for Next-Gen AI for Proliferation Detection: Accelerating the Development and Use of Explainability Methods to Design AI Systems Suitable for Nonproliferation Mission Applications
Artificial intelligence (AI) promises powerful new capabilities in an expansive array of applications. One area is proliferation detection, where AI can provide transformative tools to achieve objectives currently inaccessible using conventional methods. In particular, AI affords the opportunity to use new indicators and process massive amounts of heterogeneous data that can increase sensitivity to proliferant activities and push proliferation detection to the earliest possible stages. However, existing AI tools are not ready to achieve such potential, falling short in regard to some important considerations involved in proliferation detection. One crucial aspect that requires improvement is the explainability of the most powerful AI algorithms (i.e., understanding how the algorithms actually arrive at their conclusions)-the lack of which prevents widespread adoption in national security missions. The Data Science and AI portfolio within the National Nuclear Security Administration's Office of Defense Nuclear Nonproliferation Research and Development is driving the development of next-generation AI for proliferation detection through the expertise and work of the national laboratories and partners in academia. The "Next-Gen AI for Proliferation Detection: Accelerating the Development and Use of Explainability Methods to Design AI Systems Suitable for Nonproliferation Mission Applications" workshop aimed to further this objective. The meeting was held virtually on Sept. 15-16, 2020 and included more than 170 participants, primarily from the national laboratories with additional contributions from university researchers and mission partners. Each day included a keynote presentation, four technical presentations about cutting-edge research in explainability, and a panel to explore considerations in applying explainability and developing AI systems that can accelerate the transition of emerging AI technologies among partners and end users to solve critical mission questions. This report summarizes the Next-Gen AI for Proliferation Detection workshop's content and findings.
98 NUCLEAR DISARMAMENT, SAFEGUARDS, AND PHYSICAL P↗
PNNL DataHub Project: Omics Lethal Human Viruses Project Profiling of the Host Response to Ebola Virus Infection, Processed Experimental Dataset Catalog
Ebola virus (EBOV) is high risk biological agent, classified as a Category A priority pathogen (Flaviviridae) by the National Institute of Allergy and Infectious Diseases (NIAID), known to cause hemorrhagic fever with high mortality rates in humans. Lethal host-pathogen invasion mechanisms and the cellular intricacies behind these fatal infections still remain unclear. The NIAID Modeling Host Responses to Understand Severe Human Virus Infections Research Program project (2013-2018) aimed to develop an improved comprehensive understanding of the host response to a suite of viruses causing lethal infections leveraging a systems biology approach. Herein, PNNL sub-projects provide a never before released comprehensive infectious disease collection of primary and secondary transformation multi-Omics data profiling a series of priority pathogen primary experimental studies for enhanced open-access to viral Omics datasets and project lifecycle metadata. Secondary host-pathogen viral dataset downloads contain one or more statistically processed (normalization data transformation) quantitative dataset collections resulting in qualitative expression analyses of primary host-pathogen experimental study designs. Leveraging unique high-resolution Omics capabilities for proteomics (P), metabolomics (M), lipidomics (L), and transcriptomics (T) dataset downloads each have a direct relationship to a primary sample submission corresponding to a specific Ebola virus [NCBITAXON:186536] (Zaire/Makona or Zaire/Mayinga) experimental infection study. Human host samples types include peripheral blood mononuclear cells isolated from blood plasma ["PBMC", BTO:0001025], human hepatoma carcinoma cells ["HUH", BTO:0001950], human umbilical vein endothelial cells ["HUVEC", BTO:0001949], immortalized human hepatocyte cells ["IHH", BTO:0006147], and human histiocytic lymphoma cells ["U937", BTO:0001412].
59 BASIC BIOLOGICAL SCIENCES↗
Omics-Lethal Human Viruses, Ebola Experiment EH001
The purpose of this experiment was to evaluate the human patient peripheral blood mononuclear cells (PBMC) response to Zaire Ebola Makona virus infection during the 2013-2016 epidemic in West Africa. Samples were obtained from whole blood collections from human patients in 2015 who were naturally infected with Ebola virus during the West African Ebola virus epidemic and healthy individuals (6 month collections) where resulting blood serum was processed for proteome, metabolome, and lipidome expression analysis. Secondary host-associated viral dataset downloads contain one or more statistically processed (normalization data transformation) quantitative dataset collections resulting in qualitative expression analyses of primary host-pathogen experimental study designs. Leveraging unique high-resolution Omics capabilities for proteomics, metabolomics, and lipidomics dataset downloads each have a direct relationship to a primary sample submission corresponding to a specific Ebola virus infection.
59 BASIC BIOLOGICAL SCIENCES↗
Omics-Lethal Human Viruses, Ebola Experiment EHUH002
The purpose of this experiment was to evaluate the human host response to Zaire Ebola wild-type virus and mutant virus infection. Samples were obtained from human hepatoma carcinoma cells (HUH-7) infected with Zaire Ebola ΔVP30-WT background for proteome, metabolome, and lipidome expression analysis. Secondary host-associated viral dataset downloads contain one or more statistically processed (normalization data transformation) quantitative dataset collections resulting in qualitative expression analyses of primary host-pathogen experimental study designs. Leveraging unique high-resolution Omics capabilities for proteomics, metabolomics, and lipidomics dataset downloads each have a direct relationship to a primary sample submission corresponding to a specific Ebola virus infection.
59 BASIC BIOLOGICAL SCIENCES↗
PNNL DataHub Project: Omics Lethal Human Viruses Project Profiling of the Host Response to Influenza A Virus Infection, Processed Experimental Dataset Catalog
Influenza A virus (IAV) is a high risk biological agent, classified as a Category C priority pathogen (Orthomyxoviridae) by the National Institute of Allergy and Infectious Diseases (NIAID), and is known to cause severe respiratory disease with high mortality rates in humans. Lethal host-pathogen invasion mechanisms and the cellular intricacies behind these fatal infections still remain unclear. The NIAID Modeling Host Responses to Understand Severe Human Virus Infections Research Program project (2013-2018) aimed to develop an improved comprehensive understanding of the host response to a suite of viruses causing lethal infections leveraging a systems biology approach. Herein, PNNL sub-projects provide a never before released comprehensive infectious disease collection of primary and secondary transformation multi-Omics data profiling a series of priority pathogen primary experimental studies for enhanced open-access to viral Omics datasets and project lifecycle metadata. Secondary host-pathogen viral dataset downloads contain one or more statistically processed (normalization data transformation) quantitative dataset collections resulting in qualitative expression analyses of primary host-pathogen experimental study designs. Leveraging unique high-resolution Omics capabilities for proteomics (P), metabolomics (M), lipidomics (L), and transcriptomics (T) dataset downloads each have a direct relationship to a primary sample submission corresponding to a specific Influenza A virus [NCBITAXON:11320] experimental infection study. Host sample types include human lung adenocarcinoma cells ["Calu-3", BTO:0002750] and whole mouse lung [BTO:0000763] tissue collections.
59 BASIC BIOLOGICAL SCIENCES↗