Results for “data pipelines” — Page 48

DOE OSTI2020

Genomic Diversity Evaluation of Populus trichocarpa Germplasm for Rare Variant Genetic Association Studies

Genome-wide association studies are powerful tools to elucidate the genome-to-phenome relationship. In order to explain most of the observed heritability of a phenotypic trait, a sufficient number of individuals and a large set of genetic variants must be examined. The development of high-throughput technologies and cost-efficient resequencing of complete genomes have enabled the genome-wide identification of genetic variation at large scale. As such, almost all existing genetic variation becomes available, and it is now possible to identify rare genetic variants in a population sample. Rare genetic variants that were usually filtered out in most genetic association studies are the most numerous genetic variations across genomes and hold great potential to explain a significant part of the missing heritability observed in association studies. Rare genetic variants must be identified with high confidence, as they can easily be confounded with sequencing errors. In this study, we used a pre-filtered data set of 1,014 pure Populus trichocarpa entire genomes to identify rare and common small genetic variants across individual genomes. We compared variant calls between Platypus and HaplotypeCaller pipelines, and we further applied strict quality filters for improved genetic variant identification. Finally, we only retained genetic variants that were identified by both variant callers increasing calling confidence. Based on these shared variants and after stringent quality filtering, we found high genomic diversity in P. trichocarpa germplasm, with 7.4 million small genetic variants. Importantly, 377k non-synonymous variants (5% of the total) were uncovered. We highlight the importance of genomic diversity and the potential of rare defective genetic variants in explaining a significant portion of P. trichocarpa's phenotypic variability in association genetics. The ultimate goal is to associate both rare and common alleles with poplar's wood quality traits to support selective breeding for an improved bioenergy feedstock.

Genetics & Heredity↗

DOE OSTI2025

SITCOMTN-161: PSF assessment in the field of Abell 360 and shapeHSM shear profile using LSSTComCam data

The Rubin LSSTComCam on-sky campaign performed at the end of 2024 provided observations of the Abell 360 galaxy cluster; these data allow a preliminary study of cluster weak lensing analysis using Rubin Data Preview 1 (DP1) data. Among all the steps required for such analyses, accurate modeling of the PSF is essential. This work uses several diagnostics, mostly based on the residuals between the second moments of stars and the PSF model, to characterize the accuracy of the PSF modeling in the A360 field. We find the level of the residuals to be sufficiently low not to hinder the measurement of the tangential shear profile around A360. With a simple source selection process, we demonstrate that outputs of the LSST Science Pipelines can be used to detect the tangential shear profile in Abell 360 at the 3.6σ level, and our analysis indicates that contamination from PSF modeling systematics is negligible.

Dell'Antonio, Ian [Brown University]↗

DOE OSTI2020

Benchmarking materials property prediction methods: the Matbench test set and Automatminer reference algorithm

Abstract We present a benchmark test suite and an automated machine learning procedure for evaluating supervised machine learning (ML) models for predicting properties of inorganic bulk materials. The test suite, Matbench, is a set of 13 ML tasks that range in size from 312 to 132k samples and contain data from 10 density functional theory-derived and experimental sources. Tasks include predicting optical, thermal, electronic, thermodynamic, tensile, and elastic properties given a material’s composition and/or crystal structure. The reference algorithm, Automatminer, is a highly-extensible, fully automated ML pipeline for predicting materials properties from materials primitives (such as composition and crystal structure) without user intervention or hyperparameter tuning. We test Automatminer on the Matbench test suite and compare its predictive power with state-of-the-art crystal graph neural networks and a traditional descriptor-based Random Forest model. We find Automatminer achieves the best performance on 8 of 13 tasks in the benchmark. We also show our test suite is capable of exposing predictive advantages of each algorithm—namely, that crystal graph methods appear to outperform traditional machine learning methods given ~10 4 or greater data points. We encourage evaluating materials ML algorithms on the Matbench benchmark and comparing them against the latest version of Automatminer.

36 MATERIALS SCIENCE↗

DOE OSTI2020

Carbon Cycling, Environmental & Rural Economic Impacts of Collecting & Processing Specific Woody Feedstocks in Biofuels

Woody biomass will be an essential feedstock for a large-scale cellulosic biofuel industry. The life cycle carbon accounting for the production of biofuels from woody feedstocks is complex and has engendered significant controversy. Issues such as below ground carbon, carbon debt, varied regional forest practices, multiple parallel forest product lines, and development of realistic counterfactual scenarios all contribute to the complexity. DOE funded CORRIM to develop comprehensive and definitive lifecycle inventories and assessments on the production of fuels from woody feedstocks. CORRIM brought together expertise in forest practices, short rotation woody crops, bioconversion of woody biomass, process modeling, and life cycle assessment to successfully accomplish this goal. The CORRIM team has developed data on forest productivity, fuel usage, and fuel production for six regionally specific forest systems. These six forest systems include three current commercial systems; southern pine plantations, Douglas-fir plantations, naturally regenerated Northeastern (NE) spruce/fir, and three ‘short rotation woody crops’, poplar, eucalyptus and willow, which are at different stages of demonstration in the US. The fuel production systems include cellulosic ethanol and bio-oil based hydrocarbons. The project was successfully reviewed at the BETO Program review in March 2015, 2017, and 2019 and has resulted currently in 19 publications and reports and 35 presentations at both national and international conferences, with more in the pipeline. See publications and presentations for links to each document. Finally, and most importantly, CORRIM has gone beyond the scope of the original DOE proposal to work closely with GREET at Argonne National Laboratory (ANL) to incorporate all the life cycle data and scenario models into their modeling system. GREET is the most widely used and definitive information source for evaluating lifecycle carbon emissions for fuels. The incorporation of CORRIM data from this project guarantees the results of the research will be extensively used and widely disseminated. Technical process improvements and policy relevant accomplishments are detailed in the relevant programmatic sections in the full report including citations therein. Highlights are summarized here for easy reference.

09 BIOMASS FUELS↗

DOE OSTI2022

Application of the metabolic modeling pipeline in KBase to categorize reactions, predict essential genes, and predict pathways in an isolate genome

The DOE Systems Biology Knowledgebase (KBase) platform offers a range of powerful tools for the reconstruction, refinement, and analysis of genome-scale metabolic models built from microbial isolate genomes. In this chapter, we describe and demonstrate these tools in action with an analysis of isoprene production in the Bacillus subtilis DSM genome. Two different methods are applied to build initial metabolic models for the DSM genome, then the models are gapfilled in three different growth conditions. Next, flux balance analysis (FBA) and flux variability analysis (FVA) techniques are applied to both study the growth of these models in minimal media and classify reactions within each model based on essentiality and functionality. The models are applied with the FBA method to predict essential genes, which are then compared to an updated list of essential genes obtained for B. subtilis 168, a very similar strain to the DSM isolate. The models are also applied to simulate Biolog growth conditions, and these results are compared with Biolog data collected for B. subtilis 168. Finally, the DSM metabolic models are applied to explore the pathways and genes responsible for producing isoprene in this strain. These studies demonstrate the accuracy and utility of models generated from the KBase pipelines, as well as exploring the tools available for analyzing these models.

DOE knowledgebase↗

DOE OSTI2021

Application of the Metabolic Modeling Pipeline in KBase to Categorize Reactions, Predict Essential Genes, and Predict Pathways in an Isolate Genome

The DOE Systems Biology Knowledgebase (KBase) platform offers a range of powerful tools for the reconstruction, refinement, and analysis of genome-scale metabolic models built from microbial isolate genomes. In this chapter, we describe and demonstrate these tools in action with an analysis of isoprene production in the Bacillus subtilis DSM genome. Two different methods are applied to build initial metabolic models for the DSM genome, then the models are gapfilled in three different growth conditions. Next, flux balance analysis (FBA) and flux variability analysis (FVA) techniques are applied to both study the growth of these models in minimal media and classify reactions within each model based on essentiality and functionality. The models are applied with the FBA method to predict essential genes, which are then compared to an updated list of essential genes obtained for B. subtilis 168, a very similar strain to the DSM isolate. The models are also applied to simulate Biolog growth conditions, and these results are compared with Biolog data collected for B. subtilis 168. Finally, the DSM metabolic models are applied to explore the pathways and genes responsible for producing isoprene in this strain. These studies demonstrate the accuracy and utility of models generated from the KBase pipelines, as well as exploring the tools available for analyzing these models.

Allen, Benjamin↗

DOE OSTI2026

Temporal Convolutional Network Using Empirical Mode Decomposition to Detect Faults in Grid Connected Systems

Grid-connected power electronic systems require timely and reliable fault detection to prevent equipment damage and reduce downtime. This paper presents a forecasting-based anomaly detection pipeline that decomposes voltage and current measurements into intrinsic mode functions (IMFs) using empirical mode decomposition (EMD), then trains a causal temporal convolutional network (TCN) on normal-operation IMF data to predict short-horizon future dynamics. Deviations between forecasts and observations are summarized as reliability-weighted residual scores and thresholded per sensor using robust statistics with temporal persistence constraints to suppress false positives. To reduce runtime, EMD is performed on downsampled signals for detection, while raw-rate EMD is applied only within a short region of interest for high-frequency interpretability near detected events. Results on a simulated grid-connected converter system demonstrate that IMF-domain forecasting improves anomaly separability relative to raw-signal forecasting and provides interpretable evidence of faults across decomposition channels.

Sutton, Elizabeth [ORNL] (ORCID:0009000078885935)↗

DOE OSTI2026

YOLO11 to SAM2 pipeline for feature extraction from nuclear test films

The response to the effects of nuclear detonations is supported by models that describe the evolution of the nuclear fireball and cloud and the associated transport of active debris. Validation of those descriptions relies on data from the nuclear test operations. Video records of those events offer a rich source of information that was exploited to a limited extent in historic analyses. Computer vision and machine learning techniques are powerful tools that can be used to increase the number of measurements that can be obtained from those films. In this work, we apply computer vision techniques to automatically track the temporal evolution of the nuclear fireball. In particular, we apply You Only Look Once 11 (YOLO11) and Segment Anything Model 2 (SAM2) in combination with minimal human intervention to digitized versions of the original nuclear test films. As part of the proposed workflow, the YOLO11 model is applied to films to determine bounding boxes for the fireball within each frame. These are then used as inputs to SAM2, which uses image segmentation to determine the fireball boundaries and their temporal evolution. We assess the accuracy of our approach by using it to determine the energy released during the Trinity nuclear test and comparing the results with previous analyses based on manual measurements.

Van Exel, Kimberly [ORNL] (ORCID:0009000877463894)↗

DOE OSTI2026

Wasserstein Normalized Autoencoder for Anomaly Detection in ProtoDUNE Vertical-Drift Detector

ProtoDUNE Vertical Drift needs a selective triggering algorithm. The detector sits on Earth's surface, so cosmic activity dominates its data. Our goal in this paper is to trigger on neutrino events more robustly than the current deployed Analog-to-Digital Converter Simple Window (ADCSW) model and, eventually, search for signals of Beyond Standard Model (BSM) physics at DUNE as our ultimate North Star objective. As a step towards this goal, we evaluate a Wasserstein Normalized Autoencoder (WNAE) on simulated collection-plane only windows of shape $1\times10\times10$ where Neutrinos act as our BSM-proxy and Cosmic-ray Muons serve as our learned background. The network parameters are fitted using only cosmic-ray muon events as background in order to maintain an unsupervised pipeline. Training uses finite-step Langevin $x^-$ samples, positive-sample reconstruction energy, and an empirical sliced $2$-Wasserstein objective to learn a normalized Boltzmann energy model. We then calibrate on a nominal $5\,\mathrm{Hz}$ operating threshold calculated from cosmic validation data. Both WNAE and ADCSW accept 311 of 194,083 held-out cosmic background events at this $5\,\mathrm{Hz}$ threshold. We found that WNAE accepts 9,677 of 34,634 neutrino-proxy events $(27.9\pm0.24)\%$, compared with 10,076 $(29.1\pm0.24)\%$ for ADCSW, an observed WNAE-minus-ADCSW difference of $-1.15\%$. At another nominal $2\,\mathrm{Hz}$ target threshold, the corresponding efficiencies are $(20.5\pm0.22)\%$ and $(22.6\pm0.22)\%$, respectively. Of the WNAE-selected neutrino proxies at $5\,\mathrm{Hz}$, $(20.8\pm0.4)\%$ of the classified neutrino-proxy events are unique to WNAE, where the uncertainty is an absolute binomial standard error of $0.4\%$.

Zheng, Jake [U. Chicago (main)] (ORCID:00090002189↗

DOE OSTI2026

Wasserstein Normalized Autoencoder for Anomaly Detection in ProtoDUNE Vertical-Drift Detector

ProtoDUNE Vertical Drift needs a selective triggering algorithm. The detector sits on Earth's surface, so cosmic activity dominates its data. Our goal in this paper is to trigger on neutrino events more robustly than the current deployed Analog-to-Digital Converter Simple Window (ADCSW) model and, eventually, search for signals of Beyond Standard Model (BSM) physics at DUNE as our ultimate North Star objective. As a step towards this goal, we evaluate a Wasserstein Normalized Autoencoder (WNAE) on simulated collection-plane only windows of shape $1\times10\times10$ where Neutrinos act as our BSM-proxy and Cosmic-ray Muons serve as our learned background. The network parameters are fitted using only cosmic-ray muon events as background in order to maintain an unsupervised pipeline. Training uses finite-step Langevin $x^-$ samples, positive-sample reconstruction energy, and an empirical sliced $2$-Wasserstein objective to learn a normalized Boltzmann energy model. We then calibrate on a nominal $5\,\mathrm{Hz}$ operating threshold calculated from cosmic validation data. Both WNAE and ADCSW accept 311 of 194,083 held-out cosmic background events at this $5\,\mathrm{Hz}$ threshold. We found that WNAE accepts 9,677 of 34,634 neutrino-proxy events $(27.9\pm0.24)\%$, compared with 10,076 $(29.1\pm0.24)\%$ for ADCSW, an observed WNAE-minus-ADCSW difference of $-1.15\%$. At another nominal $2\,\mathrm{Hz}$ target threshold, the corresponding efficiencies are $(20.5\pm0.22)\%$ and $(22.6\pm0.22)\%$, respectively. Of the WNAE-selected neutrino proxies at $5\,\mathrm{Hz}$, $(20.8\pm0.4)\%$ of the classified neutrino-proxy events are unique to WNAE, where the uncertainty is an absolute binomial standard error of $0.4\%$.

Zheng, Jake [Chicago U.] (ORCID:0009000218901379)↗

DOE OSTI2021

Planck constraints on the tensor-to-scalar ratio

In this work, we present constraints on the tensor-to-scalar ratio r using Planck data. We use the latest release of Planck maps (PR4), processed with the NPIPE code, which produces calibrated frequency maps in temperature and polarization for all Planck channels from 30 GHz to 857 GHz using the same pipeline. We computed constraints on r using the BB angular power spectrum, and we also discuss constraints coming from the TT spectrum. Given Planck's noise level, the TT spectrum gives constraints on r that are cosmic-variance limited (with $\sigma$ r = 0.093), but we show that the marginalized posterior peaks towards negative values of r at about the 1.2$\sigma$ level. We derived Planck constraints using the BB power spectrum at both large angular scales (the 'reionization bump') and intermediate angular scales (the 'recombination bump') from $\ell$=2 to 150, and find a stronger constraint than that from TT, with $\sigma$ r = 0.069. The Planck BB spectrum shows no systematic bias, and is compatible with zero, given both the statistical noise and the systematic uncertainties. The likelihood analysis using B modes yields the constraint r < 0.158 at 95% confidence using more than 50% of the sky. This upper limit tightens to r < 0.069 when Planck EE, BB, and EB power spectra are combined consistently, and it tightens further to r < 0.056 when the Planck TT power spectrum is included in the combination. Finally, combining Planck with BICEP2/Keck 2015 data yields an upper limit of r < 0.044.

79 ASTRONOMY AND ASTROPHYSICS↗

DOE OSTI2023

Scalable Volume Visualization for Big Scientific Data Modeled by Functional Approximation

Considering the challenges posed by the space and time complexities in handling extensive scientific volumetric data, various data representations have been developed for the analysis of large-scale scientific data. Multivariate functional approximation (MFA) is an innovative data model designed to tackle substantial challenges in scientific data analysis. It computes values and derivatives with high-order accuracy throughout the spatial domain, mitigating artifacts associated with zero- or first-order interpolation. However, the slow query time through MFA makes it less suitable for interactively visualizing a large MFA model. In this work, we develop the first scalable interactive volume visualization pipeline, MFA-DVV, for the MFA model encoded from large-scale datasets. Our method achieves low input latency through distributed architecture, and its performance can be further enhanced by utilizing a compressed MFA model while still maintaining a high-quality rendering result for scientific datasets. We conduct comprehensive experiments to show that MFA-DVV can decrease the input latency and achieve superior visualization results for big scientific data compared with existing approaches.

big scientific dataset↗

DOE OSTI2025

One-shot gas detection with transformer paired neural networks in Mako collected longwave infrared hyperspectral imagery

To date, careful data treatment workflows and statistical detectors are used to perform hyperspectral image (HSI) detection of any gas contained in a spectral library, which is often expanded with physics models to incorporate different spectral characteristics. In general, surrounding evidence or known gas-release parameters are used to provide confidence in or confirm detection capability, respectively. This makes quantifying detection performance difficult as it is nearly impossible to develop an absolute ground truth for gas target pixel presence in collected HSI. Consequently, developing and comparing new detection methods, especially machine learning (ML)-based methods, is susceptible to subjectivity in derived detection map quality. Here, in this work, we demonstrate the first use of transformer-based paired neural networks (PNNs) for one-shot gas target detection for multiple gases while providing quantitative classification and detection metrics for their use on labeled data. Terabytes of training data are generated from a database of long-wave infrared HSI obtained from historical Mako sensor campaigns over Los Angeles. By incorporating labels, singular signature representations, and a model development pipeline, we can tune and select PNNs to detect multiple gas targets that are not seen in training on a quantitative basis. We additionally assess our test set detections using interpretability techniques widely employed with ML-based predictors, but less common with detection methods relying on learned latent spaces.

Hyperspectral imaging↗

DOE OSTI2022

Expansion of the ORNL VALID Library [Slides]

VALID continues as a library of high-quality models used for testing SCALE and nuclear data. Recent expansion of VALID was the largest impactful change in a decade. Adding 233 U experiments circa 2017 added more cases. There are a huge number of cases in the pipeline, but a lack of qualified staff and funding to complete reviews. Future plans should simplify the process and increase the availability of models and results for external users.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗

DOE OSTI2025

Control-Agnostic Beam Instrumentation with Redis at the Core

Redis isn’t a database — it’s our protocol. Fermilab’s RedisAdapter provides a high-performance, control-system-agnostic bridge between digitized beam data and downstream consumers such as ACNET and EPICS. It forms the foundation of three new software components deployed across MicroTCA-based digitizers: GMMDM, a runtime for memory-mapped data movement from Zynq-based platforms; GRAFE, a front end for Redis-to-ACNET presentation; and GREFE, an EPICS IOC front end. Together, these tools enable modular, standardized instrumentation pipelines. Precision timing is handled via White Rabbit PPS distribution, allowing nanosecond-scale synchronization across crates. This architecture, originally prototyped in Booster BPM systems, is now deployed on modern hardware and designed to meet the performance, modularity, and scalability requirements of the PIP-II era.

Steinkamp, Derek [Fermilab] (ORCID:000900027228626↗

DOE OSTI2026

BICEP/Keck. XX. Component-separated Maps of the Polarized Cosmic Microwave Background and Thermal Dust Emission Using Planck and BICEP/Keck Observations through the 2018 Observing Season

We present component-separated polarization maps of the cosmic microwave background (CMB) and Galactic thermal dust emission, derived using data from the BICEP/Keck experiments through the 2018 observing season and Planck. By employing a maximum-likelihood method that utilizes observing matrices, we produce unbiased maps of the CMB and dust signals. We outline the computational challenges and demonstrate an efficient implementation of the component map estimator. We show methods to compute and characterize power spectra of these maps, opening up an alternative way to infer the tensor-to-scalar ratio from our data. We compare the results of this map-based separation method with the baseline BICEP/Keck analysis. Our analysis demonstrates consistency between the two methods, finding an 84% correlation between the pipelines.

cosmic inflation↗

DOE OSTI2022

Map-based cosmology inference with lognormal cosmic shear maps

ABSTRACT Most cosmic shear analyses to date have relied on summary statistics (e.g. ξ+ and ξ−). These types of analyses are necessarily suboptimal, as the use of summary statistics is lossy. In this paper, we forward-model the convergence field of the Universe as a lognormal random field conditioned on the observed shear data. This new map-based inference framework enables us to recover the joint posterior of the cosmological parameters and the convergence field of the Universe. Our analysis properly accounts for the covariance in the mass maps across tomographic bins, which significantly improves the fidelity of the maps relative to single-bin reconstructions. We verify that applying our inference pipeline to Gaussian random fields recovers posteriors that are in excellent agreement with their analytical counterparts. At the resolution of our maps – and to the extent that the convergence field can be described by the lognormal model – our map posteriors allow us to reconstruct all summary statistics (including non-Gaussian statistics). We forecast that a map-based inference analysis of LSST-Y10 data can improve cosmological constraints in the σ8–Ωm plane by $\approx\!{30}{{\ \rm per\ cent}}$ relative to the currently standard cosmic shear analysis. This improvement happens almost entirely along the $S_8=\sigma _8\Omega _{\rm m}^{1/2}$ directions, meaning map-based inference fails to significantly improve constraints on S8.

79 ASTRONOMY AND ASTROPHYSICS↗

DOE OSTI2025

DESI 2024 II: sample definitions, characteristics, and two-point clustering statistics

We present the samples of galaxies and quasars used for DESI 2024 cosmological analyses, drawn from the DESI Data Release 1 (DR1). We describe the construction of largescale structure (LSS) catalogs from these samples, which include matched sets of synthetic reference ‘randoms’ and weights that account for variations in the observed density of the samples due to experimental design and varying instrument performance. We detail how we correct for variations in observational completeness, the input ‘target’ densities due to imaging systematics, and the ability to confidently measure redshifts from DESI spectra. We then summarize how remaining uncertainties in the corrections can be translated to systematic uncertainties for particular analyses. We describe the weights added to maximize the signalto-noise of DESI DR1 2-point clustering measurements. We detail measurement pipelines applied to the LSS catalogs that obtain 2-point clustering measurements in configuration and Fourier space. The resulting 2-point measurements depend on window functions and normalization constraints particular to each sample, and we present the corrections required to match models to the data. We compare the configuration- and Fourier-space 2-point clustering of the data samples to that recovered from simulations of DESI DR1 and find they are, generally, in statistical agreement to within 2% in the inferred real-space over-density field. The LSS catalogs, 2-point measurements, and their covariance matrices will be released publicly with DESI DR1.

79 ASTRONOMY AND ASTROPHYSICS↗