Search NASA⌕ Search

SEARCH · Search NASA

Results for “data pipelines”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 973 records · Page 54
All sourcesNASA NTRSDOE OSTI
DOE OSTI2024

SPT clusters with DES and HST weak lensing. I. Cluster lensing and Bayesian population modeling of multiwavelength cluster datasets

We present a Bayesian population modeling method to analyze the abundance of galaxy clusters identified by the South Pole Telescope (SPT) with a simultaneous mass calibration using weak gravitational lensing data from the Dark Energy Survey (DES) and the Hubble Space Telescope (HST). We discuss and validate the modeling choices with a particular focus on a robust, weak-lensing-based mass calibration using DES data. For the DES Year 3 data, we report a systematic uncertainty in weak-lensing mass calibration that increases from 1% at z = 0.25 to 10% at z = 0.95 , to which we add 2% in quadrature to account for uncertainties in the impact of baryonic effects. We implement an analysis pipeline that joins the cluster abundance likelihood with a multiobservable likelihood for the Sunyaev-Zel’dovich effect, optical richness, and weak-lensing measurements for each individual cluster. We validate that our analysis pipeline can recover unbiased cosmological constraints by analyzing mocks that closely resemble the cluster sample extracted from the SPT-SZ, SPTpol ECS, and SPTpol 500d surveys and the DES Year 3 and HST-39 weak-lensing datasets. This work represents a crucial prerequisite for the subsequent cosmological analysis of the real dataset.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2021

Models and Processes to Extract Drug-like Molecules From Natural Language Text

Researchers worldwide are seeking to repurpose existing drugs or discover new drugs to counter the disease caused by severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2). A promising source of candidates for such studies is molecules that have been reported in the scientific literature to be drug-like in the context of viral research. However, this literature is too large for human review and features unusual vocabularies for which existing named entity recognition (NER) models are ineffective. We report here on a project that leverages both human and artificial intelligence to detect references to such molecules in free text. We present 1) a iterative model-in-the-loop method that makes judicious use of scarce human expertise in generating training data for a NER model, and 2) the application and evaluation of this method to the problem of identifying drug-like molecules in the COVID-19 Open Research Dataset Challenge (CORD-19) corpus of 198,875 papers. We show that by repeatedly presenting human labelers only with samples for which an evolving NER model is uncertain, our human-machine hybrid pipeline requires only modest amounts of non-expert human labeling time (tens of hours to label 1778 samples) to generate an NER model with an F-1 score of 80.5%—on par with that of non-expert humans—and when applied to CORD’19, identifies 10,912 putative drug-like molecules. This enriched the computational screening team’s targets by 3,591 molecules, of which 18 ranked in the top 0.1% of all 6.6 million molecules screened for docking against the 3CLPro protein.

60 APPLIED LIFE SCIENCES↗
DOE OSTI2022

PATOKA: Simulating Electromagnetic Observables of Black Hole Accretion

Abstract The Event Horizon Telescope (EHT) has released analyses of reconstructed images of horizon-scale millimeter emission near the supermassive black hole at the center of the M87 galaxy. Parts of the analyses made use of a large library of synthetic black hole images and spectra, which were produced using numerical general relativistic magnetohydrodynamics fluid simulations and polarized ray tracing. In this article, we describe the PATOKA pipeline, which was used to generate the Illinois contribution to the EHT simulation library. We begin by describing the relevant accretion systems and radiative processes. We then describe the details of the three numerical codes we use, iharm , ipole , and igrmonty , paying particular attention to differences between the current generation of the codes and the originally published versions. Finally, we provide a brief overview of simulated data as produced by PATOKA and conclude with a discussion of limitations and future directions.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2024

Leveraging Existing Assets for Long Duration Energy Storage

Increased renewables penetration to electrical grid is necessary to reduce overall emissions from the electrical power generation sector. Nonetheless, its integration creates challenges to grid operators who must match the power being generated by intermittent renewables and other traditional energy sources with the demand from consumers, while ensuring the reliability and power quality for the entire system. Energy storage has been proposed as an alternative to natural gas peaking plants and a form to deliver excess renewable energy generation at times of peak demand. For energy storage to provide benefits to end customers (energy consumers), it must be reliable, efficient, and cost effective. The Illinois Sustainable Technology Center (ISTC), one of the surveys that integrate the Prairie Research Institute (PRI), aims to develop a Center for Energy Storage at Existing Assets (CESEA) at UIUC with the participation of Waste Pressure Corp and Ecotek Engineering USA LLC. CESEA will focus on LDES systems that can integrate to existing infrastructure in a manner that reduces the initial capital expenditure and demonstrates the ability to repurpose fossil assets that would otherwise become stranded, to serve the energy transition. CESEA aims to leverage UIUC’s unique facilities to validate LDES systems performance at a relevant operating environment. UIUC’s facilities include a 85-MW combined heat and power (CHP) power plant, two (2) solar PV plants totaling over 18 MWdc of installed capacity, an electrical grid along with a substation at transmission and distribution voltages, a 22-mile gas pipeline network operating at two pressure levels, along with steam and chilled water distribution networks. The new LDES systems will connect to the existing UIUC grid through a new test electrical station, which will have the capacity to accommodate additional connections to test new devices and technologies as part of future CESEA R&D activities. The test electrical station will contain meters, instrumentation, and controls to accurately capture data and allow optimization of control algorithms. CESEA will initially focus on technologies that: i) utilize existing equipment or facilities to perform at least one of the process steps in LDES (charging, storage, or discharging), ii) leverage mature or commercially available components or controls, iii) show potential for cost-leadership in 10+ hour storage at a commercial scale. Initial technologies that were identified to meet these criteria include Compressed Gas Energy Storage (CGES), and TES. CGES stores electricity by raising the pressure of a compressible gas inside a control volume and converting the stored energy to electricity via expansion-generation. CGES is a generalization of CAES that covers any working gas (not just air). A successful CGES demo will help to circumvent many challenges faced by CAES (long development times due to site prospecting, high cost of compression and storage, heat recovery management, etc.) by: 1) utilizing existing infrastructure (compressors, pipelines, underground storage or pressure vessels) used in the transportation and storage of industrial gases for LDES charging and storage; 2) deploying over sites already-developed for industrial applications with minor additional work; 3) leveraging the price structure of commercial industrial gas to cover the costs of electricity used during charging. A previous DOE-sponsored conceptual study (DE-FE-0032018) estimated the levelized cost of energy of a 1.1 MW / 17 MWh CGES system at $0.08/kWh, with a commercial 10x scale system cost estimated at <$0.04/kWh (Giardinella, 2022). The pilot-sized system was estimated to avoid up to 2693 tons of CO2/year.

25 ENERGY STORAGE↗
DOE OSTI2022

Automatic recognition system for document digitization in nuclear power plants

With the increasing number of data-driven models in nuclear applications, large volumes of numerical data are required to accurately model and predict the health status of a plant component. However, many historical operation logs that contain useful information are not fully utilized due to the lack of a systematic approach of digitization. To overcome this issue, this study proposes an automatic pipeline for extracting information from handwritten tabular documents collected from nuclear power plants. In our pipeline, we first denoise scanned documents with morphological operations, and then extract relevant parts from individual pages using both traditional computer vision and neural network methods. Handwriting recognition is applied to obtain text and numbers. As the most challenging step is how to crop only relevant information, the main focus of our paper is to detect tables and cells from scanned handwritten documents. Here we evaluate the efficiency and accuracy of our proposed method on handwritten operational reports obtained from a real-world case study. The results demonstrate the high accuracy and practicality of our proposed method.

21 SPECIFIC NUCLEAR REACTORS AND ASSOCIATED PLANTS↗
DOE OSTI2025

The Simons Observatory: validation of reconstructed power spectra from simulated filtered maps for the small aperture telescope survey

We present a transfer function-based method to estimate angular power spectra from filtered maps for cosmic microwave background (CMB) surveys. This is especially relevant for experiments targeting the faint primordial gravitational wave signatures in CMB polarisation at large scales, such as the Simons Observatory (SO) small aperture telescopes. While timestreams can be filtered to mitigate the contamination from low-frequency noise, usual methods that calculate the mode coupling at individual multipoles can be challenging for experiments covering large sky areas or reaching few-arcminute resolution. The method we present here, although approximate, is more practical and faster for larger data volumes. We validate it through the use of simulated observations approximating the first year of SO data, going from half-wave plate-modulated timestreams to maps, and using simulations to estimate the mixing of polarisation modes induced by an example of time-domain filtering. We show its performance through an example null test and with an end-to-end pipeline that performs inference on cosmological parameters, including the tensor-to-scalar ratio r. The performance demonstration uses simulated observations at multiple frequency bands. We find that the method can recover unbiased parameters for our simulated noise levels.

CMBR experiments↗
DOE OSTI2022

Star-spots and magnetism: testing the activity paradigm in the Pleiades and M67

We measure star-spot filling fractions for 240 stars in the Pleiades and M67 open star clusters using APOGEE high-resolution H-band spectra. For this work, we developed a modified spectroscopic pipeline which solves for star-spot filling fraction and star-spot temperature contrast. We exclude binary stars, finding that the large majority of binaries in these clusters (80 per cent) can be identified from Gaia DR3 and APOGEE criteria – important for field star applications. Our data agree well with independent activity proxies, indicating that this technique recovers real star-spot signals. In the Pleiades, filling fractions saturate at a mean level of 0.248 ± 0.005 for active stars with a decline at slower rotation; we present fitting functions as a function of Rossby number. In M67, we recover low mean filling fractions of 0.030 ± 0.008 and 0.003 ± 0.002 for main sequence GK stars and evolved red giants, respectively, confirming that the technique does not produce spurious spot signals in inactive stars. Star-spots also modify the derived spectroscopic effective temperatures and convective overturn time-scales. Effective temperatures for active stars are offset from inactive ones by −109 ± 11 K, in agreement with the Pecaut & Mamajek empirical scale. Star-spot filling fractions at the level measured in active stars changes their inferred overturn time-scale, which biases the derived threshold for saturation. Finally, we identify a population of stars statistically discrepant from mean activity–Rossby relations and present evidence that these are genuine departures from a Rossby scaling. Our technique is applicable to the full APOGEE catalogue, with broad applications to stellar, galactic, and exoplanetary astrophysics.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2020

An advanced workflow for single-particle imaging with the limited data at an X-ray free-electron laser

An improved analysis for single-particle imaging (SPI) experiments, using the limited data, is presented here. Results are based on a study of bacteriophage PR772 performed at the Atomic, Molecular and Optical Science instrument at the Linac Coherent Light Source as part of the SPI initiative. Existing methods were modified to cope with the shortcomings of the experimental data: inaccessibility of information from half of the detector and a small fraction of single hits. The general SPI analysis workflow was upgraded with the expectation-maximization based classification of diffraction patterns and mode decomposition on the final virus-structure determination step. The presented processing pipeline allowed us to determine the 3D structure of bacteriophage PR772 without symmetry constraints with a spatial resolution of 6.9 nm. The obtained resolution was limited by the scattering intensity during the experiment and the relatively small number of single hits.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗
DOE OSTI2022

Root system architecture and environmental flux analysis in mature crops using 3D root mesocosms

Current methods of root sampling typically only obtain small or incomplete sections of root systems and do not capture their true complexity. To facilitate the visualization and analysis of full-sized plant root systems in 3-dimensions, we developed customized mesocosm growth containers. While highly scalable, the design presented here uses an internal volume of 45 ft 3 (1.27 m 3 ), suitable for large crop and bioenergy grass root systems to grow largely unconstrained. Furthermore, they allow for the excavation and preservation of 3-dimensional root system architecture (RSA), and facilitate the collection of time-resolved subterranean environmental data. Sensor arrays monitoring matric potential, temperature and CO 2 levels are buried in a grid formation at various depths to assess environmental fluxes at regular intervals. Methods of 3D data visualization of fluxes were developed to allow for comparison with root system architectural traits. Following harvest, the recovered root system can be digitally reconstructed in 3D through photogrammetry, which is an inexpensive method requiring only an appropriate studio space and a digital camera. We developed a pipeline to extract features from the 3D point clouds, or from derived skeletons that include point cloud voxel number as a proxy for biomass, total root system length, volume, depth, convex hull volume and solidity as a function of depth. Ground-truthing these features with biomass measurements from manually dissected root systems showed a high correlation. We evaluated switchgrass, maize, and sorghum root systems to highlight the capability for species wide comparisons. We focused on two switchgrass ecotypes, upland (VS16) and lowland (WBC3), in identical environments to demonstrate widely different root system architectures that may be indicative of core differences in their rhizoeconomic foraging strategies. Finally, we imposed a strong physiological water stress and manipulated the growth medium to demonstrate whole root system plasticity in response to environmental stimuli. Hence, these new “3D Root Mesocosms” and accompanying computational analysis provides a new paradigm for study of mature crop systems and the environmental fluxes that shape them.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2024

Statistical Association between the Candidate Repeating FRB 20200320A and a Galaxy Group

Abstract We present results from angular cross correlations between select samples of CHIME/FRB repeaters and galaxies in three photometric galaxy surveys, which have shown correlations with the first CHIME/FRB catalog containing repeating and nonrepeating sources: Wide-field Infrared Survey Explorer (WISE) × SCOS, DESI-BGS, and DESI-LRG. We find a statistically significant correlation (p-value <0.001, after accounting for look-elsewhere factors) between a sample of repeaters with an extragalactic dispersion measure (DM) > 395 pc cm −3 and WISE × SCOS galaxies with redshiftz> 0.275. We demonstrate that the correlation arises surprisingly because of a statistical association between FRB 20200320A (extragalactic DM ≈ 550 pc cm −3 ) and a galaxy group in the same dark matter halo at redshiftz≈ 0.32. We estimate that the host halo, along with an intervening halo at redshiftz≈ 0.12, accounts for at least ∼30% of the extragalactic DM. Our results strongly motivate incorporating galaxy group and cluster catalogs into direct host association pipelines for FRBs with ≲ 1 ′ localization precision, effectively utilizing the two-point information to constrain FRB properties such as their redshift and DM distributions. In addition, we find marginal evidence for a negative correlation at 99.4% confidence limit between a sample of repeating FRBs with baseband data (median extragalactic DM = 354 pc cm −3 ) and DESI-LRG galaxies with redshift 0.3 ≤z< 0.45, suggesting that the repeaters might be more prone than apparent nonrepeaters to propagation effects in FRB–galaxy correlations due to intervening free electrons over angular scales ∼0.°5.

Astronomy & Astrophysics↗
DOE OSTI2022

Towards Low-Overhead Resilience for Data Parallel Deep Learning

Data parallel techniques have been widely adopted both in academia and industry as a tool to enable scalable training of deep learning models. At scale, DL training jobs can fail due to software or hardware bugs, may need to be preempted or terminated due to unexpected events, or may perform suboptimally because they were misconfigured. Under such circumstances, there is a need to recover and/or reconfigure data-parallel DL training jobs on-the-fly, while minimizing the impact on the accuracy of the DNN model and the runtime overhead. In this regard, state-of-art techniques adopted by the HPC community mostly rely on checkpoint-restart, which inevitably leads to loss of progress, thus increasing the runtime overhead. In this paper we explore alternative techniques that exploit the properties of modern deep learning frameworks (overlapping of gradient averaging and weight updates with local gradient computations through pipeline parallelism) to reduce the overhead of resilience/elasticity. To this end we introduce a failure simulation framework and two resilience strategies (immediate mini-batch rollback and lossy forward recovery), which we study compared with checkpoint-restart approaches in a variety of settings in order to understand the trade-offs between the accuracy loss of the DNN model and the runtime overhead.

data-parallel training↗
DOE OSTI2026

Operating advanced scientific instruments with AI agents that learn on the job

Advanced scientific user facilities, such as next generation X-ray light sources and self-driving laboratories, are revolutionizing scientific discovery by automating routine tasks and enabling rapid experimentation and characterizations. However, these facilities must continuously evolve to support new experimental workflows, adapt to diverse user projects, and meet growing demands for more intricate instruments and experiments. This continuous development introduces significant operational complexity, necessitating a focus on usability, reproducibility, and intuitive human-instrument interaction. In this work, we explore the integration of agentic AI, powered by Large Language Models (LLMs), as a transformative tool to achieve this goal. We present our approach to developing a human-in-the-loop pipeline for operating advanced instruments including an X-ray nanoprobe beamline and an autonomous robotic station dedicated to the design and characterization of materials. Specifically, we evaluate the potential of various LLMs as trainable scientific assistants for orchestrating complex, multi-task workflows, which also include multimodal data, optimizing their performance through optional human input and iterative learning. We demonstrate the ability of AI agents to bridge the gap between advanced automation and user-friendly operation, paving the way for more adaptable and intelligent scientific facilities.

Large Language Models↗
DOE OSTI2022

The DESI N -body Simulation Project – II. Suppressing sample variance with fast simulations

ABSTRACT Dark Energy Spectroscopic Instrument (DESI) will construct a large and precise three-dimensional map of our Universe. The survey effective volume reaches $\sim 20\, h^{-3}\, \mathrm{Gpc}^{3}$. It is a great challenge to prepare high-resolution simulations with a much larger volume for validating the DESI analysis pipelines. AbacusSummit is a suite of high-resolution dark-matter-only simulations designed for this purpose, with $200\, h^{-3}\, \mathrm{Gpc}^{3}$ (10 times DESI volume) for the base cosmology. However, further efforts need to be done to provide a more precise analysis of the data and to cover also other cosmologies. Recently, the CARPool method was proposed to use paired accurate and approximate simulations to achieve high statistical precision with a limited number of high-resolution simulations. Relying on this technique, we propose to use fast quasi-N-body solvers combined with accurate simulations to produce accurate summary statistics. This enables us to obtain 100 times smaller variance than the expected DESI statistical variance at the scales we are interested in, e.g. $k \lt 0.3\, h\, \mathrm{Mpc}^{-1}$ for the halo power spectrum. In addition, it can significantly suppress the sample variance of the halo bispectrum. We further generalize the method for other cosmologies with only one realization in AbacusSummit suite to extend the effective volume ∼20 times. In summary, our proposed strategy of combining high-fidelity simulations with fast approximate gravity solvers and a series of variance suppression techniques sets the path for a robust cosmological analysis of galaxy survey data.

cosmology: theory↗
DOE OSTI2025

PyCMG-based Simulation of Volumetric Concrete Microstructure

Concrete is a complex, heterogeneous material with a microstructure composed of aggregates, cement paste, and pores spanning multiple length scales. Understanding this microstructure is critical for advancing the performance, durability, and modeling of concrete-based systems. While experimental imaging such as X-ray computed tomography (XCT) provides valuable insights, generating large datasets with detailed ground truth annotations is both costly and labor-intensive due to challenges in segmenting similar phases, such as aggregates and cement paste, that often share similar attenuation properties. To address this, we developed a pipeline to simulate realistic 3D concrete microstructures using the open-source Python package PyCMG. This simulation effort focuses on generating high-fidelity, annotated microstructures that can serve as training or benchmarking datasets for image analysis, segmentation algorithms, and machine learning models, particularly in scenarios where experimental data is scarce.

Ziabari, Amir [Oak Ridge National Laboratory; ORNL↗
DOE OSTI2022

The SDSS-Gaia View of the Color–Magnitude Relation for Blue Horizontal-branch Stars

We present an updated sample of blue horizontal-branch (BHB) stars selected from the photometric and spectroscopic data from Sloan Digital Sky Survey and its associated project Sloan Extension for Galactic Understanding and Exploration (SEGUE). With these data, we selected candidates for A-type stars in the color–color space and then a mixture modeling technique was implemented in order to distinguish between BHB and main-sequence/blue-straggler stars based on their surface gravity values (log g) estimated by the SEGUE Stellar Parameter Pipeline. Our robust approach allows us to attribute individual probabilities of each star truly being in the BHB stage. Hence, our method is advantageous in comparison to previous SEGUE BHB selections that adopted simple log g cuts. We also revisit the color–magnitude relation for these stars and propose two calibrations, based on updated distances for Galactic globular clusters, to estimate absolute magnitudes with (g – r) 0 and (u – r) 0 colors.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2024

The Dark Energy Survey Supernova Program: Cosmological Biases from Host Galaxy Mismatch of Type Ia Supernovae

Abstract Redshift measurements, primarily obtained from host galaxies, are essential for inferring cosmological parameters from type Ia supernovae (SNe Ia). Matching SNe to host galaxies using images is nontrivial, resulting in a subset of SNe with mismatched hosts and thus incorrect redshifts. We evaluate the host galaxy mismatch rate and resulting biases on cosmological parameters from simulations modeled after the Dark Energy Survey 5 Yr (DES-SN5YR) photometric sample. For both DES-SN5YR data and simulations, we employ the directional light radius method for host galaxy matching. In our SN Ia simulations, we find that 1.7% of SNe are matched to the wrong host galaxy, with redshift differences between the true and matched hosts of up to 0.6. Using our analysis pipeline, we determine the shift in the dark energy equation of state parameter (Δ w ) due to including SNe with incorrect host galaxy matches. For SN Ia–only simulations, we find Δ w = 0.0013 ± 0.0026 with constraints from the cosmic microwave background. Including core-collapse SNe and peculiar SNe Ia in the simulation, we find that Δ w ranges from 0.0009 to 0.0032, depending on the photometric classifier used. This bias is an order of magnitude smaller than the expected total uncertainty on w from the DES-SN5YR sample of ∼0.03. We conclude that the bias on w from host galaxy mismatch is much smaller than the uncertainties expected from the DES-SN5YR sample, but we encourage further studies to reduce this bias through better host-matching algorithms or selection cuts.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2024

Learning From User Behavior: A Survey-Assist Algorithm for Longitudinal Mobility Data Collection

GPS-based travel surveys are widely used in mobility studies to gather crucial qualitative data, like purpose, transportation mode and replaced mode. However, survey response still poses a burden to users, especially in long-term mobility studies, leading to response fatigue. We explore a survey-assist strategy to ease this burden by a novel, user-level modeling approach that leverages past responses from each user to predict responses for new trips, without relying on external data sources like GIS data. We investigate three main algorithms for predicting responses: (i) clustering trips and extrapolating responses for similar trips, (ii) using random forest classification, and (iii) clustering that uses a hybrid algorithm to determine spatial structure, which is then fed as input to a classic random forest classifier. The clustering approach can flexibly predict responses for even complex qualitative survey questions; it achieved F-scores of 65%. The random forest pipeline uses architecture that restricts it to predicting three predetermined survey questions: trip purpose, mode, and replaced mode. However, it achieved F-scores of 78%. While the survey-assist approach has been implemented by several proprietary systems, to our knowledge, this is the first exploration in the academic literature. It follows that this is also the first rigorous evaluation of multiple algorithms that can implement the approach. The evaluation uses a large scale, publicly available, longitudinal dataset consisting of ~ 92k trips from 235 users over a period of roughly one and a half years. With this approach, travel surveys can be pre-filled with the predicted responses for each trip, thus streamlining the survey process for users. Combined with an active learning system that requests user input on low-confidence predictions, models can be updated and improved over time to better support the long-term collection of longitudinal qualitative data.

clustering↗