Search NASA⌕ Search

SEARCH · Search NASA

Results for “high-throughput computing”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 613 records · Page 34

The Evolution of the Pegasus Workflow Management Software

Since 2001 the Pegasus Workflow Management System has evolved into a robust and scalable system that automates the execution of a number of complex applications running on a variety of heterogeneous, distributed high-throughput, and high-performance computing environments. Pegasus was built on the principle of separation between the workflow description and workflow execution, providing the ability to port and adapt the workflow based on the target execution environment. Through its user-driven research and development it has adapted to the needs of a number of scientific communities, utilizing and developing novel algorithms and software engineering solutions. This paper describes the evolution of Pegasus over time and provides motivations behind the design decisions. The paper concludes with selected lessons learned.

97 MATHEMATICS AND COMPUTING↗

High-Throughput Data Processing at FRIB Using ESnet

Real-time or nearly real-time (nearline) data processing methods are critical tools as detector technologies and data acquisition (DAQ) systems allow for higher data rates and volumes. The introduction of the energy sciences network (ESnet), a U.S. Department of Energy (DOE) supported high-speed network for scientific research, creates opportunities to leverage the computing power of DOE facilities like the National Energy Research Scientific Computing Center (NERSC). As a first step toward realizing a DOE Office of Science Integrated Research Infrastructure (IRI) pattern, an automated workflow was developed to remotely process data obtained from a nuclear physics experiment at the Facility for Rare Isotope Beams (FRIB) at NERSC with data transferred between FRIB and NERSC over ESnet. The workflow demonstrated the ability to process one week’s worth of experimental data in approximately 90 min and was used successfully for nearline analysis during a recently completed FRIB experiment. Here, a summary of the workflow development and results of recent demonstrations will be presented.

Data processing↗

Fast 2D Bicephalous Convolutional Autoencoder for Compressing 3D Time Projection Chamber Data

High-energy large-scale particle colliders produce data at high speed in the order of 1 terabytes per second in nuclear physics and petabytes per second in high energy physics. Developing real-time data compression algorithms to reduce such data at high throughput to fit permanent storage has drawn increasing attention. Specifically, at the newly constructed sPHENIX experiment at the Relativistic Heavy Ion Collider (RHIC), a time projection chamber is used as the main tracking detector, which records particle trajectories in a volume of three-dimensional (3D) cylinder. The resulting data are usually very sparse with occupancy around 10.8%. Such sparsity presents a challenge to conventional learning-free lossy compression algorithms, such as SZ, ZFP, and MGARD. The 3D convolutional neural network (CNN)-based approach, Bicephalous Convolutional Autoencoder (BCAE), outperforms traditional methods both in compression rate and reconstruction accuracy. BCAE can also utilize the computation power of graphical processing units suitable for deployment in a modern heterogeneous highperformance computing environment. This work introduces two BCAE variants: BCAE++ and BCAE-2D. BCAE++ achieves a 15% better compression ratio and a 77% better reconstruction accuracy measured in mean absolute error compared with BCAE. BCAE-2D treats the radial direction as the channel dimension of an image, resulting in a 3× speedup in compression throughput. In addition, we demonstrate an unbalanced autoencoder with a larger decoder can improve reconstruction accuracy without significantly sacrificing throughput. Lastly, we observe both the BCAE++ and BCAE-2D can benefit more from using half-precision mode in throughput (76 - 79% increase) without loss in reconstruction accuracy. The source code and links to data and pretrained models can be found at https://github.com/BNL-DAQ-LDRD/NeuralCompression_v2

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗

Integrating and Characterizing HPC Task Runtime Systems for hybrid AI-HPC workloads

Scientific workflows increasingly involve both HPC and machine-learning tasks, combining MPI-based simulations, training, and inference in a single execution. Launchers such as Slurm’s srun constrain concurrency and throughput, making them unsuitable for dynamic and heterogeneous workloads. We present a performance study of RADICAL-Pilot (RP) integrated with Flux and Dragon, two complementary runtime systems that enable hierarchical resource management and high-throughput function execution. Using synthetic and production-scale workloads on Frontier, we characterize the task execution properties of RP across runtime configurations. RP+Flux sustains up to 930 tasks/s, and RP+Flux+Dragon exceeds 1,500 tasks/s with over 99.6% utilization. In contrast, srun peaks at 152 tasks/s and degrades with scale, with utilization below 50%. For IMPECCABLE.v2 drug discovery campaign, RP+Flux reduces makespan by 30–60% relative to srun/Slurm and increases throughput more than four times on up to 1,024. These results demonstrate hybrid runtime integration in RP as a scalable approach for hybrid AI-HPC workloads.

HPC-AI↗

wastewater_virus

This repo contains software used to clean and assemble high-throughput sequencing data containing viruses. The input is raw illumina sequencing reads and the output is a database of high-quality viral genomes. The specific application is to wastewater viral concentrates but it is not restricted to that sample type. The software is composed of Nextflow workflows and a set of custom Python and bash scripts that call publicly available bioinformatics tools to accomplish obvious tasks in data analysis in a high performance computing environment. For detailed information, please see the repo's README file.

Kantor, Rose [Lawrence Livermore National Laborato↗

ConPADE: Genome Assembly Ploidy Estimation from Next-Generation Sequencing Data

As a result of improvements in genome assembly algorithms and the ever decreasing costs of high-throughput sequencing technologies, new high quality draft genome sequences are published at a striking pace. With well-established methodologies, larger and more complex genomes are being tackled, including polyploid plant genomes. Given the similarity between multiple copies of a basic genome in polyploid individuals, assembly of such data usually results in collapsed contigs that represent a variable number of homoeologous genomic regions. Unfortunately, such collapse is often not ideal, as keeping contigs separate can lead both to improved assembly and also insights about how haplotypes influence phenotype. Here, we describe a first step in avoiding inappropriate collapse during assembly. In particular, we describe ConPADE (Contig Ploidy and Allele Dosage Estimation), a probabilistic method that estimates the ploidy of any given contig/scaffold based on its allele proportions. In the process, we report findings regarding errors in sequencing. The method can be used for whole genome shotgun (WGS) sequencing data. We also show applicability of the method for variant calling and allele dosage estimation. Results for simulated and real datasets are discussed and provide evidence that ConPADE performs well as long as enough sequencing coverage is available, or the true contig ploidy is low. We show that ConPADE may also be used for related applications, such as the identification of duplicated genes in fragmented assemblies, although refinements are needed.

59 BASIC BIOLOGICAL SCIENCES↗

The genome-scale metabolic model for the purple non-sulfur bacterium Rhodopseudomonas palustris Bis A53 accurately predicts phenotypes under chemoheterotrophic, chemoautotrophic, photoheterotrophic, and photoautotrophic growth conditions

The purple non-sulfur bacterium Rhodopseudomonas palustris is recognized as a critical microorganism in the nitrogen and carbon cycle and one of the most common members in wastewater treatment communities. This bacterium is metabolically extremely versatile. It is capable of heterotrophic growth under aerobic and anaerobic conditions, but also able to grow photoautotrophically as well as mixotrophically. Therefore R . palustris can adapt to multiple environments and establish commensal relationships with other organisms, expressing various enzymes supporting degradation of amino acids, carbohydrates, nucleotides, and complex polymers. Moreover, R . palustris can degrade a wide range of pollutants under anaerobic conditions, e.g., aromatic compounds such as benzoate and caffeate, enabling it to thrive in chemically contaminated environments. However, many metabolic mechanisms employed by R . palustris to breakdown and assimilate different carbon and nitrogen sources under chemoheterotrophic or photoheterotrophic conditions remain unknown. Systems biology approaches, such as metabolic modeling, have been employed extensively to unravel complex mechanisms of metabolism. Previously, metabolic models have been reconstructed to study selected capabilities of R . palustris under limited experimental conditions. Here, we developed a comprehensive metabolic model (M-model) for R . palustris Bis A53 ( i DT1294) consisting of 2,721 reactions, 2,123 metabolites, and comprising 1,294 genes. We validated the model using high-throughput phenotypic, physiological, and kinetic data, testing over 350 growth conditions. i DT1294 achieved a prediction accuracy of 90% for growth with various carbon and nitrogen sources and close to 80% for assimilation of aromatic compounds. Moreover, the M-model accurately predicts dynamic changes of growth and substrate consumption rates over time under nine chemoheterotrophic conditions and demonstrated high precision in predicting metabolic changes between photoheterotrophic and photoautotrophic conditions. This comprehensive M-model will help to elucidate metabolic processes associated with the assimilation of multiple carbon and nitrogen sources, anoxygenic photosynthesis, aromatic compound degradation, as well as production of molecular hydrogen and polyhydroxybutyrate.

59 BASIC BIOLOGICAL SCIENCES↗

Disentangling multidimensional spatio-temporal data into their common and aberrant responses

With the advent of high-throughput measurement techniques, scientists and engineers are starting to grapple with massive data sets and encountering challenges with how to organize, process and extract information into meaningful structures. Multidimensional spatio-temporal biological data sets such as time series gene expression with various perturbations over different cell lines, or neural spike trains across many experimental trials, have the potential to acquire insight about the dynamic behavior of the system. For this potential to be realized, we need a suitable representation to understand the data. A general question is how to organize the observed data into meaningful structures and how to find an appropriate similarity measure. A natural way of viewing these complex high dimensional data sets is to examine and analyze the large-scale features and then to focus on the interesting details. Since the wide range of experiments and unknown complexity of the underlying system contribute to the heterogeneity of biological data, we develop a new method by proposing an extension of Robust Principal Component Analysis (RPCA), which models common variations across multiple experiments as the lowrank component and anomalies across these experiments as the sparse component. We show that the proposed method is able to find distinct subtypes and classify data sets in a robust way without any prior knowledge by separating these common responses and abnormal responses. Thus, the proposed method provides us a new representation of these data sets which has the potential to help users acquire new insight from data.

59 BASIC BIOLOGICAL SCIENCES↗

Phase Mapper: Accelerating Materials Discovery with AI

From the stone age, to the bronze, iron age, and modern silicon age, the discovery and characterization of new materials has always been instrumental to humanity's progress and development. With the current pressing need to address sustainability challenges and find alternatives to fossil fuels, we look for solutions in the development of new materials that will allow for renewable energy. To discover materials with the required properties, materials scientists can perform high-throughput materials discovery, which includes rapid synthesis and characterization via X-ray diffraction (XRD) of thousands of materials. A central problem in materials discovery, the phase map identification problem, involves the determination of the crystal structure of materials from materials composition and structural characterization data. This analysis is traditionally performed mainly by hand, which can take days for a single material system. In this work we present Phase-Mapper, a solution platform that tightly integrates XRD experimentation, AI problem solving, and human intelligence for interpreting XRD patterns and inferring the crystal structures of the underlying materials. Phase-Mapper is compatible with any spectral demixing algorithm, including our novel solver, AgileFD, which is based on convolutive non-negative matrix factorization. AgileFD allows materials scientists to rapidly interpret XRD patterns, and incorporates constraints to capture prior knowledge about the physics of the materials as well as human feedback. With our system, materials scientists have been able to interpret previously unsolvable systems of XRD data at the Department of Energy’s Joint Center for Artificial Photosynthesis, including the Nb-Mn-V oxide system, which led to the discovery of new solar light absorbers and is provided as an illustrative example of AI-enabled high throughput materials discovery

Computer Science↗

jaxhps: An elliptic PDE solver built with machine learning in mind

Elliptic partial differential equations (PDEs) can model many physical phenomena, such as electrostatics, acoustics, wave propagation, and diffusion. In scientific machine learning settings, a high-throughput PDE solver may be required to generate a training dataset, run in the inner loop of an iterative algorithm, or interface directly with a deep neural network. To provide value to machine learning users, such a PDE solver must be compatible with standard automatic differentiation frameworks, scale efficiently when run on graphics processing units (GPUs), and maintain high accuracy for a large range of input parameters. We have designed the jaxhps package with these use-cases in mind by implementing a highly efficient and accurate solver for elliptic problems with native hardware acceleration and automatic differentiation support.

97 MATHEMATICS AND COMPUTING↗

Plant Single-Cell Solutions for Energy and the Environment (Second Workshop Report)

Plants are important sources of energy and materials, and they collectively represent a critical component of Earth’s ecosystem. With increasing environmental stresses due to climate change and intensive agricultural practices, the need for resilient plants is greater than ever before. To secure plant resources for bioenergy, biomaterials, food, and ecosystem adaptation, a deeper understanding of the fundamental biology of plants at a cellular level is urgently needed. Plants contain a multitude of specialized cell types that compose tissues and organs. Pathogens often target specific cell types within plants, and the response of one cell to a particular stimulus is likely to be distinct from its neighbor because of underlying molecular and contextual differences. Understanding how these responses are distributed among cells, the main goal of single-cell approaches, will substantially enhance our ability to use targeted engineering for improving plant productivity and resilience. Furthermore, single-cell approaches are necessary to understand the interactions between plants and other ecosystem members such as fungi, bacteria, and archaea. Unlocking these gene-response mechanisms at a cellular level can improve our ability to adapt plants to environmental stresses, increasing their utility as feedstocks for biomaterials and bioenergy. Recent advances in high-throughput sequencing, mass spectrometry, microfluidics and miniaturization, artificial intelligence and machine learning, and bioinformatics have greatly improved our ability to detect and understand processes at a cellular level. In mammalian systems, single-cell transcriptomics has already led to many advances, such as newly identified cell types and cell-targeted treatment of diseases, and mass spectrometry-based single-cell proteomics has recently been demonstrated as a promising emerging technology. However, plant single-cell omics has lagged behind mammalian approaches due to the high cost of the technologies relative to available resources and to the innate biological features of plants, including the complexity of the cell wall and polyploidy. To better understand how single-cell methods could enable plant science, Lawrence Berkeley National Laboratory (Berkeley Lab) hosted a workshop on April 29, 2021, that brought together a diverse group of leaders in plant and/or single-cell biology. Attendees represented federal research programs and domestic and international academic institutions. During the workshop, three presenters described the current state of research in both experimental and computational approaches. While the focus of the workshop was on factors preventing plant biology researchers from fully adopting single-cell methodologies, workshop participants agreed that most barriers could be overcome with focused, strategic investment and coordinated efforts among institutions leading to significant scientific discoveries that would be difficult to obtain using more conventional technologies.

59 BASIC BIOLOGICAL SCIENCES↗

A Platform Technology for High-throughput Atomically Precise Manufacturing: Mechatronics at the Atomic Scale

The main objective of this project is to invent the necessary enabling technologies for high throughput atomically precise manufacturing (APM). APM is an emerging technology that refers to any manufacturing capability that enables fabrication of atomically precise structures, components, and devices under programmable control. APM will require positional assembly at the atomic and/or molecular scale, as well as at the nano and microscales using hierarchical assembly to create products ranging from nanoscale and quantum devices to macroscale systems and materials. This project builds on the expectation that commercial viability of APM will depend on a high level of parallelism to achieve the required throughput, a capability that currently does not exist. This project is the first attempt to address this key technological bottleneck.

77 NANOSCIENCE AND NANOTECHNOLOGY↗

Building a FAIR data ecosystem for incorporating single-cell transcriptomics data into agricultural genome to phenome research

Introduction The agriculture genomics community has numerous data submission standards available, but the standards for describing and storing single-cell (SC, e.g., scRNA- seq) data are comparatively underdeveloped. Methods To bridge this gap, we leveraged recent advancements in human genomics infrastructure, such as the integration of the Human Cell Atlas Data Portal with Terra, a secure, scalable, open-source platform for biomedical researchers to access data, run analysis tools, and collaborate. In parallel, the Single Cell Expression Atlas at EMBL-EBI offers a comprehensive data ingestion portal for high-throughput sequencing datasets, including plants, protists, and animals (including humans). Developing data tools connecting these resources would offer significant advantages to the agricultural genomics community. The FAANG data portal at EMBL-EBI emphasizes delivering rich metadata and highly accurate and reliable annotation of farmed animals but is not computationally linked to either of these resources. Results Herein, we describe a pilot-scale project that determines whether the current FAANG metadata standards for livestock can be used to ingest scRNA-seq datasets into Terra in a manner consistent with HCA Data Portal standards. Importantly, rich scRNA-seq metadata can now be brokered through the FAANG data portal using a semi-automated process, thereby avoiding the need for substantial expert curation. We have further extended the functionality of this tool so that validated and ingested SC files within the HCA Data Portal are transferred to Terra for further analysis. In addition, we verified data ingestion into Terra, hosted on Azure, and demonstrated the use of a workflow to analyze the first ingested porcine scRNA-seq dataset. Additionally, we have also developed prototype tools to visualize the output of scRNA-seq analyses on genome browsers to compare gene expression patterns across tissues and cell populations. This JBrowse tool now features distinct tracks, showcasing PBMC scRNA-seq alongside two bulk RNA-seq experiments. Discussion We intend to further build upon these existing tools to construct a scientist-friendly data resource and analytical ecosystem based on Findable, Accessible, Interoperable, and Reusable (FAIR) SC principles to facilitate SC-level genomic analysis through data ingestion, storage, retrieval, re-use, visualization, and comparative annotation across agricultural species.

Genetics & Heredity↗

Data-driven integration of genome-scale regulatory and metabolic network models

Microbes are diverse and extremely versatile organisms that play vital roles in all ecological niches. Understanding and harnessing microbial systems will be key to the sustainability of our planet. One approach to improving our knowledge of microbial processes is through data-driven and mechanism-informed computational modeling. Individual models of biological networks (such as metabolism, transcription, and signaling) have played pivotal roles in driving microbial research through the years. These networks, however, are highly interconnected and function in concert a fact that has led to the development of a variety of approaches aimed at simulating the integrated functions of two or more network types. Though the task of integrating these different models is fraught with new challenges, the large amounts of high-throughput data sets being generated, and algorithms being developed, means that the time is at hand for concerted efforts to build integrated regulatory-metabolic networks in a data-driven fashion. Lastly, in this perspective, we review current approaches for constructing integrated regulatory-metabolic models and outline new strategies for future development of these network models for any microbial system.

59 BASIC BIOLOGICAL SCIENCES↗

A novel statistical methodology for quantifying the spatial arrangements of axons in peripheral nerves

A thorough understanding of the neuroanatomy of peripheral nerves is required for a better insight into their function and the development of neuromodulation tools and strategies. In biophysical modeling, it is commonly assumed that the complex spatial arrangement of myelinated and unmyelinated axons in peripheral nerves is random, however, in reality the axonal organization is inhomogeneous and anisotropic. Present quantitative neuroanatomy methods analyze peripheral nerves in terms of the number of axons and the morphometric characteristics of the axons, such as area and diameter. In this study, we employed spatial statistics and point process models to describe the spatial arrangement of axons and Sinkhorn distances to compute the similarities between these arrangements (in terms of first- and second-order statistics) in various vagus and pelvic nerve cross-sections. We utilized high-resolution transmission electron microscopy (TEM) images that have been segmented using a custom-built high-throughput deep learning system based on a highly modified U-Net architecture. Our findings show a novel and innovative approach to quantifying similarities between spatial point patterns using metrics derived from the solution to the optimal transport problem. We also present a generalizable pipeline for quantitative analysis of peripheral nerve architecture. Our data demonstrate differences between male- and female-originating samples and similarities between the pelvic and abdominal vagus nerves.

59 BASIC BIOLOGICAL SCIENCES↗

New Orthophoto Generation Strategies from UAV and Ground Remote Sensing Platforms for High-Throughput Phenotyping

Remote sensing platforms have become an effective data acquisition tool for digital agriculture. Imaging sensors onboard unmanned aerial vehicles (UAVs) and tractors are providing unprecedented high-geometric-resolution data for several crop phenotyping activities (e.g., canopy cover estimation, plant localization, and flowering date identification). Among potential products, orthophotos play an important role in agricultural management. Traditional orthophoto generation strategies suffer from several artifacts (e.g., double mapping, excessive pixilation, and seamline distortions). The above problems are more pronounced when dealing with mid- to late-season imagery, which is often used for establishing flowering date (e.g., tassel and panicle detection for maize and sorghum crops, respectively). In response to these challenges, this paper introduces new strategies for generating orthophotos that are conducive to the straightforward detection of tassels and panicles. The orthophoto generation strategies are valid for both frame and push-broom imaging systems. The target function of these strategies is striking a balance between the improved visual appearance of tassels/panicles and their geolocation accuracy. The new strategies are based on generating a smooth digital surface model (DSM) that maintains the geolocation quality along the plant rows while reducing double mapping and pixilation artifacts. Moreover, seamline control strategies are applied to avoid having seamline distortions at locations where the tassels and panicles are expected. The quality of generated orthophotos is evaluated through visual inspection as well as quantitative assessment of the degree of similarity between the generated orthophotos and original images. Several experimental results from both UAV and ground platforms show that the proposed strategies do improve the visual quality of derived orthophotos while maintaining the geolocation accuracy at tassel/panicle locations.

54 ENVIRONMENTAL SCIENCES↗

Understanding Growth Dynamics and Yield Prediction of Sorghum Using High Temporal Resolution UAV Imagery Time Series and Machine Learning

Unmanned aerial vehicles (UAV) carrying multispectral cameras are increasingly being used for high-throughput phenotyping (HTP) of above-ground traits of crops to study genetic diversity, resource use efficiency and responses to abiotic or biotic stresses. There is significant unexplored potential for repeated data collection through a field season to reveal information on the rates of growth and provide predictions of the final yield. Generating such information early in the season would create opportunities for more efficient in-depth phenotyping and germplasm selection. This study tested the use of high-resolution time-series imagery (5 or 10 sampling dates) to understand the relationships between growth dynamics, temporal resolution and end-of-season above-ground biomass (AGB) in 869 diverse accessions of highly productive (mean AGB = 23.4 Mg/Ha), photoperiod sensitive sorghum. Canopy surface height (CSM), ground cover (GC), and five common spectral indices were considered as features of the crop phenotype. Spline curve fitting was used to integrate data from single flights into continuous time courses. Random Forest was used to predict end-of-season AGB from aerial imagery, and to identify the most informative variables driving predictions. Improved prediction of end-of-season AGB (RMSE reduction of 0.24 Mg/Ha) was achieved earlier in the growing season (10 to 20 days) by leveraging early- and mid-season measurement of the rate of change of geometric and spectral features. Early in the season, dynamic traits describing the rates of change of CSM and GC predicted end-of-season AGB best. Late in the season, CSM on a given date was the most influential predictor of end-of-season AGB. The power to predict end-of-season AGB was greatest at 50 days after planting, accounting for 63% of variance across this very diverse germplasm collection with modest error (RMSE 1.8 Mg/ha). End-of-season AGB could be predicted equally well when spline fitting was performed on data collected from five flights versus 10 flights over the growing season. This demonstrates a more valuable and efficient approach to using UAVs for HTP, while also proposing strategies to add further value.

54 ENVIRONMENTAL SCIENCES↗

SQuaD: Smart Quantum Detection for Photon Recognition and Dark Count Elimination

Quantum detectors of single photons are an essential component for quantum information processing across computing, communication and networking. Today's quantum detection system, which consists of single photon detectors, timing electronics, control and data processing software, is primarily used for counting the number of single photon detection events. However, it is largely incapable of extracting other rich physical characteristics of the detected photons, such as their wavelengths, polarization states, photon numbers, or temporal waveforms. This work, for the first time, demonstrates a smart quantum detection system, SQuaD, which integrates a field programmable gate array (FPGA) with a neural network model, and is designed to recognize the features of photons and to eliminate detector dark-count. The SQuaD is a fully integrated quantum system with high timing-resolution data acquisition, onboard multi-scale data analysis, intelligent feature recognition and extraction, and feedback-driven system control. Our \name experimentally demonstrates 1) reliable photon counting on par with the state-of-the art commercial systems; 2) high-throughput data processing for each individual detection events; 3) efficient dark count recognition and elimination; 4) up to 100% accurate feature recognition of photon wavelength and polarization. Additionally, we deploy the SQuaD to an atomic (erbium ion) photon emitter source to realize noise-free control and readout of a spin qubit in the telecom band, enabling critical advances in quantum networks and distributed quantum information processing.

Linne, Karl C. [U. Chicago (main)] (ORCID:00090009↗