Search NASA⌕ Search

SEARCH · Search NASA

Results for “Science Pipeline”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 253 records · Page 14
All sourcesNASA NTRSDOE OSTI
DOE OSTI2023

Wildfire Smoke PM2.5 per Zipcode

This dataset contains daily aggregated measurements of daily PM2.5 from ambient wildfire smoke in the contiguous United States, spanning from 2006 to 2016. The data is sourced from a study by Childs et al. (2022), titled "Daily Local-Level Estimates of Ambient Wildfire Smoke PM2.5 for the Contiguous US" published in Environmental Science & Technology. To compute the standardized weight across different zip codes, we computed the weight calculation on a 10km grids. To compute the weight, we obtain area-weights that add up to 1 in each polygon of zipcodes. This enabled us to calculate the smoke values per zipcode for the aforementioned period. Those interested in replicating our data processing pipeline can access it at https://github.com/NSAPH-Data-Processing/smoke_aggregation.

PM2.5↗
DOE OSTI2025

Accelerating Control Systems with GitOps: A Path to Automation and Reliability

GitOps is a foundational approach for modernizing infrastructure by leveraging Git as the single source of truth for declarative configurations. The poster explores how GitOps transforms traditional control system infrastructure, services and applications by enabling fully automated, auditable, and version-controlled infrastructure management. Cloud-native and containerized environments are shifting the ecosystem not only in the IT industry but also within the computational science field, as is the case of CERN and Diamond Light Source among other Accelerator/Science facilities which are slowly shifting towards modern software and infrastructure paradigms. The ACORN project, which aims to modernize Fermilab’s control system infrastructure and software is implementing proven best-practices and cutting-edge technology standards including GitOps, containerization, infrastructure as code and modern data pipelines for control system data acquisition and the inclusion of AI/ML in our accelerator complex.

Gonzalez, M. [Fermilab]↗
DOE OSTI2023

Eight Up-Coming Biotech Tools to Combat Climate Crisis

Biotechnology has a high potential to substantially contribute to a low-carbon society. Several green processes are already well established, utilizing the unique capacity of living cells or their instruments. Beyond that, the authors believe that there are new biotechnological procedures in the pipeline which have the momentum to add to this ongoing change in our economy. Eight promising biotechnology tools were selected by the authors as potentially impactful game changers: (i) the Wood–Ljungdahl pathway, (ii) carbonic anhydrase, (iii) cutinase, (iv) methanogens, (v) electro-microbiology, (vi) hydrogenase, (vii) cellulosome and, (viii) nitrogenase. Some of them are fairly new and are explored predominantly in science labs. Others have been around for decades, however, with new scientific groundwork that may rigorously expand their roles. In the current paper, the authors summarize the latest state of research on these eight selected tools and the status of their practical implementation. We bring forward our arguments on why we consider these processes real game changers.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2025

Sequence length scaling in vision transformers for scientific images on frontier

Vision Transformers (ViTs) are pivotal for foundational models in scientific imagery, including Earth science applications, due to their capability to process large sequence lengths. While transformers for text have inspired scaling sequence lengths in ViTs, adapting these for ViTs introduces unique challenges. We develop distributed sequence parallelism for ViTs, enabling them to handle up to 1M tokens. Our approach, leveraging DeepSpeed-Ulysses and Long-Sequence-Segmentation with model sharding, is the first to apply sequence parallelism in ViT training, achieving a 94% batch scaling efficiency on 2,048 AMD-MI250X GPUs. Evaluating sequence parallelism in ViTs, particularly in models up to 10B parameters, highlighted substantial bottlenecks. We countered these with hybrid sequence, pipeline, and flash attention strategies, to scale beyond single GPU memory limits. Our method significantly enhances climate modeling accuracy by 20% in temperature predictions, marking the first training of a vision transformer model to convergence with a sequence length of 188K tokens, using full self-attention.

Tsaris, Aristeidis (aris) [ORNL] (ORCID:0000000277↗
DOE OSTI2021

Integrated System and Application Continuous Performance Monitoring and Analysis Capability

Scientific applications run on high-performance computing (HPC) systems are critical for many national security missions within Sandia and the NNSA complex. However, these applications often face performance degradation and even failures that are challenging to diagnose. To provide unprecedented insight into these issues, the HPC Development, HPC Systems, Computational Science, and Plasma Theory & Simulation departments at Sandia crafted and completed their FY21 ASC Level 2 milestone entitled "Integrated System and Application Continuous Performance Monitoring and Analysis Capability." The milestone created a novel integrated HPC system and application monitoring and analysis capability by extending Sandia's Kokkos application portability framework, Lightweight Distributed Metric Service (LDMS) monitoring tool, and scalable storage, analysis, and visualization pipeline. The extensions to Kokkos and LDMS enable collection and storage of application data during run time, as it is generated, with negligible overhead. This data is combined with HPC system data within the extended analysis pipeline to present relevant visualizations of derived system and application metrics that can be viewed at run time or post run. This new capability was evaluated using several week-long, 290-node runs of Sandia's ElectroMagnetic Plasma In Realistic Environments ( EMPIRE ) modeling and design tool and resulted in 1TB of application data and 50TB of system data. EMPIRE developers remarked this capability was incredibly helpful for quickly assessing application health and performance alongside system state. In short, this milestone work built the foundation for expansive HPC system and application data collection, storage, analysis, visualization, and feedback framework that will increase total scientific output of Sandia's HPC users.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2021

Integrated System and Application Continuous Performance Monitoring and Analysis Capability (Final)

Scientific applications run on high-performance computing (HPC) systems are critical for many national security missions within Sandia and the NNSA complex. However, these applications often face performance degradation and even failures that are challenging to diagnose. To provide unprecedented insight into these issues, the HPC Development, HPC Systems, Computational Science, and Plasma Theory & Simulation departments at Sandia crafted and completed their FY21 ASC Level 2 milestone entitled "Integrated System and Application Continuous Performance Monitoring and Analysis Capability." The milestone created a novel integrated HPC system and application monitoring and analysis capability by extending Sandia’s Kokkos application portability framework, Lightweight Distributed Metric Service (LDMS) monitoring tool, and scalable storage, analysis, and visualization pipeline. The extensions to Kokkos and LDMS enable collection and storage of application data during run time, as it is generated, with negligible overhead. This data is combined with HPC system data within the extended analysis pipeline to present relevant visualizations of derived system and application metrics that can be viewed at run time or post run. This new capability was evaluated using several week-long, 290-node runs of Sandia’s ElectroMagnetic Plasma In Realistic Environments (EMPIRE) modeling and design tool and resulted in 1TB of application data and 50TB of system data. EMPIRE developers remarked this capability was incredibly helpful for quickly assessing application health and performance alongside system state. In short, this milestone work built the foundation for expansive HPC system and application data collection, storage, analysis, visualization, and feedback framework that will increase total scientific output of Sandia’s HPC users.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗
DOE OSTI2022

ThunderSecure: deploying real-time intrusion detection for 100G research networks by leveraging stream-based features and one-class classification network

Nowadays, data generated by large-scale scientific experiments are on the scale of petabytes per month. These data are transferred through dedicated high-bandwidth networks (40/100G) across distributed sites for processing, storage, and analysis. Like general purpose networks, research networks experience intrusions. However, monitoring anomalies in such high-speed network traffics is challenging given current cyber-infrastructure. Moreover, traditional network intrusion detection systems (NIDS) are signature based. However, anomaly patterns are difficult to define and that rulesets are often not updated frequently enough to reflect the changes of attack behaviors. We present ThunderSecure, a high-throughput, unsupervised learning-based intrusions detection system for 100G research networks. ThunderSecure implements an efficient packet processing and detection pipeline using multi-cores and GPUs. It extracts statistical and temporal features from real-time network data streams and feeds them to a one-class anomaly detection network. A baseline of normal distribution will be created based on the training observation. Testing traffic deviated from the learned profile will be marked as anomalies. We trained ThunderSecure on hundreds of billions of science data packets mirrored from two 100G network connections at Fermi National Accelerator Laboratory. The detection performance was evaluated on traffic captured from the same research network days and weeks after the training with different types of attack flows injected. Results show that ThunderSecure can recognize science data traffic captured long after the training and made nearly certain detection on the segment of the streams where anomalous flows were injected.

100G research network↗
DOE OSTI2023

A streamlined approach to characterize microalgae strains for biomass productivity under dynamic climate simulation conditions

Screening microalgae strains under static light and temperature flask conditions cannot directly quantify the biomass productivities of algae growing in the dynamically fluctuating light and temperature conditions of outdoor ponds. In this effort, we describe a testing pipeline that screens for productivity under climate simulated conditions. A validated, miniaturized Laboratory Environmental Algae Pond Simulator (mini-LEAPS) photobioreactor was used to determine optimal medium salinities and biomass productivities of cold-tolerant microalgae collected from Arctic, subarctic, Antarctic, and subalpine habitats. Strains characterized in this effort include: Chlorella antarctica UTEX1959, Chlorella sp. UTEXSNO69, Chloromonas rosae UTEXSNO11, Phaeodactylum tricornutum UTEX646, and Stichococcus minutus CCALA727. Observed productivities of the characterized strains ranged from 1.74 ± 0.25 to 8.18 ± 0.81 g m –2 day –1 (as ash-free dry weight). For each strain, a temperature tolerance profile was generated to identify the most appropriate season (s) for cultivation. The two most promising strains, Chlorella sp. UTEX SNO69 and P. tricornutum UTEX646, were tested alongside the DISCOVR winter benchmark strain Monoraphidium minutum 26B-AM in outdoor open ponds at the PNNL Algae Testbed (PAT) in Arizona. Further, under cold-season outdoor pond conditions, Chlorella sp. UTEXSNO69 achieved average areal biomass productivities of 8.21 ± 0.50 g m –2 day –1 , not significantly different from that of the benchmark strain (8.52 ± 0.43 g m –2 day –1 , p > 0.05). Under spring outdoor pond conditions, P. tricornutum UTEX646 achieved average areal biomass productivities 10.34 ± 0.27 g m –2 day –1 , not significantly different from that of the benchmark (10.07 ± 0.50 g m –2 day –1 ) under the tested conditions (p > 0.05). The streamlined pipeline was thus demonstrated to successfully characterize productive microalgae strains for outdoor deployment by weeding out algae strains that were not productive under the tested dynamic light and temperature conditions.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2021

Rational design and testing of abiotic stress‐inducible synthetic promoters from poplar cis ‐regulatory elements

Summary Abiotic stress resistance traits may be especially crucial for sustainable production of bioenergy tree crops. Here, we show the performance of a set of rationally designed osmotic‐related and salt stress‐inducible synthetic promoters for use in hybrid poplar. De novo motif‐detecting algorithms yielded 30 water‐deficit (SD) and 34 salt stress (SS) candidate DNA motifs from relevant poplar transcriptomes. We selected three conserved water‐deficit stress motifs (SD18, SD13 and SD9) found in 16 co‐expressed gene promoters, and we discovered a well‐conserved motif for salt response (SS16). We characterized several native poplar stress‐inducible promoters to enable comparisons with our synthetic promoters. Fifteen synthetic promoters were designed using various SD and SS subdomains, in which heptameric repeats of five‐to‐eight subdomain bases were fused to a common core promoter downstream, which, in turn, drove a green fluorescent protein (GFP) gene for reporter assays. These 15 synthetic promoters were screened by transient expression assays in poplar leaf mesophyll protoplasts and agroinfiltrated Nicotiana benthamiana leaves under osmotic stress conditions. Twelve synthetic promoters were induced in transient expression assays with a GFP readout. Of these, five promoters (SD18‐1, SD9‐2, SS16‐1, SS16‐2 and SS16‐3) endowed higher inducibility under osmotic stress conditions than native promoters. These five synthetic promoters were stably transformed into Arabidopsis thaliana to study inducibility in whole plants. Herein, SD18‐1 and SD9‐2 were induced by water‐deficit stress, whereas SS16‐1, SS16‐2 and SS16‐3 were induced by salt stress. The synthetic biology design pipeline resulted in five synthetic promoters that outperformed endogenous promoters in transgenic plants.

09 BIOMASS FUELS↗
DOE OSTI2025

The Effect of Flow on CO2 Corrosion of Self-Healing Metallic Coatings

Internal corrosion is an issue that affects natural gas pipelines, a significant part of the United States energy infrastructure. Over time, this corrosion has worn away longstanding pipelines due to the original construction materials used for the lines and impurities in the gas and liquid streams flowing through them. The main impurities in natural gas are H2O, CO2, H2S, and O2. One of the leading causes of corrosion is the CO2 dissolved in water, giving rise to carbonic acid formation, which can further dissolve the steel pipe. The National Energy Technology Laboratory (NETL) has been studying different solutions to this problem. One potential answer is a self-healing sacrificial metallic coating applied by a novel cold spray technique. Previous studies at NETL have shown that the zinc-rich self-healing coating can withstand corrosion through passivation and create a barrier to the diffusion of corrosive species. This study explored the impact of flow on corrosion with this coating and carbon steel when exposed to a saturated CO2 environment by simulating the pipeline flow profile in a small-scale lab setting. The samples were evaluated using multiple electrochemical techniques that found corrosion rates and were backed up with surface analysis to conclude the behavior of the corrosion mechanisms.

CO2 corrosion↗
DOE OSTI2020

Towards Interactive, Reproducible Analytics at Scale on HPC Systems

The growth in scientific data volumes has resulted in a need to scale up processing and analysis pipelines using High Performance Computing (HPC) systems. These workflows need interactive, reproducible analytics at scale. The Jupyter platform provides core capabilities for interactivity but was not designed for HPC systems. In this paper, we outline our efforts that bring together core technologies based on the Jupyter Platform to create interactive, reproducible analytics at scale on HPC systems. Our work is grounded in a real world science use case-applying geophysical simulations and inversions for imaging the subsurface. Our core platform addresses three key areas of the scientific analysis workflow-reproducibility, scalability, and interactivity. We describe our implemention of a system, using Binder, Science Capsule, and Dask software. We demonstrate the use of this software to run our use case and interactively visualize real-Time streams of HDF5 data.

containers↗
DOE OSTI2026

Opportunities in Hydropower and Pumped Storage Hydropower

Hydropower and pumped storage hydropower (PSH) are established technologies with a long-standing presence in the United States, but the industry remains dynamic with an active relicensing pipeline and growing interest in PSH to provide large-scale energy storage and grid services. NLR has developed a range of tools, data, and analysis that supports the evaluation of hydropower and PSH opportunities, complementing other DOE-supported efforts in this space. This presentation initiates a conversation around the future of hydropower and PSH and how it can inform the National Academy of Sciences investigation of possible DOE-supported regional energy-water technology pilots.

13 HYDRO ENERGY↗
DOE OSTI2022

Transitioning from File-Based HPC Workflows to Streaming Data Pipelines with openPMD and ADIOS2

This paper aims to create a transition path from file-based IO to streaming-based workflows for scientific applications in an HPC environment. By using the openPMP-api, traditional workflows limited by filesystem bottlenecks can be overcome and flexibly extended for in situ analysis. The openPMD-api is a library for the description of scientific data according to the Open Standard for Particle-Mesh Data (openPMD). Its approach towards recent challenges posed by hardware heterogeneity lies in the decoupling of data description in domain sciences, such as plasma physics simulations, from concrete implementations in hardware and IO. The streaming backend is provided by the ADIOS2 framework, developed at Oak Ridge National Laboratory. This paper surveys two openPMD-based loosely-coupled setups to demonstrate flexible applicability and to evaluate performance. In loose coupling, as opposed to tight coupling, two (or more) applications are executed separately, e.g. in individual MPI contexts, yet cooperate by exchanging data. This way, a streaming-based workflow allows for standalone codes instead of tightly-coupled plugins, using a unified streaming-aware API and leveraging high-speed communication infrastructure available in modern compute clusters for massive data exchange. We determine new challenges in resource allocation and in the need of strategies for a flexible data distribution, demonstrating their influence on efficiency and scaling on the Summit compute system. The presented setups show the potential for a more flexible use of compute resources brought by streaming IO as well as the ability to increase throughput by avoiding filesystem bottlenecks.

Poeschel, Franz↗
DOE OSTI2022

Protocol for single-cell isolation and genome amplification of environmental microbial eukaryotes for genomic analysis

We describe environmental microbial eukaryotes (EMEs) sample collection, single-cell isolation, lysis, and genome amplification, followed by the rDNA amplification and OTU screening for recovery of high-quality species-specific genomes for de novo assembly. These protocols are part of our pipeline that also includes bioinformatic methods. The pipeline and its application on a wide range of phyla of different sample complexity are described in our complementary paper. In addition, this protocol describes optimized lysis, genome amplification, and OTU screening steps of the pipeline. For complete details on the use and execution of this protocol, please refer to Ciobanu et al. (2021).

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2024

Accelerating the identification of novel secondary metabolites in bioenergy plant root exudates using MicroED

Small molecule metabolites drive inter- and intraspecies communication and dependencies in diverse biological systems, yet a large proportion of these important chemical compounds remain uncharacterized in plants and microbes. Approximately 90% of the metabolites in root exudate profiles are unknown compounds, despite the importance of root exudate composition in plant-microbe interactions. We need advanced analytical capabilities that will support rapid discovery and structural elucidation of metabolites from biological samples that may be limited in quantity and high in complexity. To fill this gap, this project aimed to develop an integrated workflow involving metabolite extraction, separation, and crystallization from plant root exudates followed by characterization using nuclear magnetic resonance (NMR) spectroscopy, mass spectrometry, and microcrystal electron diffraction (MicroED). Using crude root exudates from sorghum, this project successfully developed higher throughput exudate fractionation strategies to obtain pure compounds for crystallization and identified crystals in multiple fractions that diffracted. Additional efforts to increase the throughput of high-quality crystal generation for MicroED, such as crystallization screening and crystallization chaperone exploration, will be needed to further advance root exudate metabolite identification. The overall optimized sample preparation process can then be integrated with the existing data collection and data analysis pipelines for MicroED at PNNL to facilitate more rapid natural product discovery.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2025

Accelerating Advanced Light Source Science Through Multi-Facility HPC Workflows

Synchrotron light sources support a wide array of techniques to investigate materials, often producing complex, high-volume data that challenge traditional workflows. At the Advanced Light Source (ALS), we developed infrastructure to move microtomography data over ESnet to ALCF and NERSC, where CPU- and GPU-based algorithms generate 3D reconstructed volumes of experimental samples. We employ two data movement and reconstruction models: real-time processing as data streams directly to NERSC compute nodes, and automated file transfer to NERSC and ALCF file systems. The streaming pipeline provides users with feedback in under ten seconds, while the file-based workflow produces high-quality reconstructions suitable for deeper analysis in 20-30 minutes. This infrastructure enables users to utilize HPC resources without direct access to backend systems. We plan to extend this architecture to more endstations, supporting our beamline scientists and users.

Abramov, David↗
DOE OSTI2024

Machine Learning (ML) Classifier to Assist Metadata Creation

The Atmospheric Radiation Measurement (ARM) Data Center is responsible for the timely collection, archival, and curation of science data products. These products are freely available through an online data repository. Metadata creation is paramount for scientific users to find and access over seven petabytes of atmospheric science data. The hierarchical metadata structure allows users to search for information at both broad and narrow levels. This project aims to leverage 30 years’ worth of manually created metadata to enable machine predictions of broad-term classifications from narrow-term descriptions. These classification predictions would assist metadata coordinators with their term selections. This paper discusses the cleaning and preprocessing of the training data, the pipeline developed to determine the best model for this task, and the creation of an API metadata classifier for ARM measurement metadata. Our results show that the Linear Support Vector Classification (LinearSVC) algorithm, along with the Term Frequency – Inverse Document Frequency (TF-IDF) vectorizer, is well-suited for our multi-class classification task. Lengthier input training data led to better results, and artificial balancing was unnecessary for this particular use case. This predictive classifier enhances efficiency in metadata creation, as well as supports greater consistency and accuracy in metadata tagging.

Collier, Hannah [ORNL] (ORCID:0000000341284292)↗
DOE OSTI2021

Remote methane sensor for emissions from pipelines and compressor stations using chirped-laser dispersion spectroscopy

Leak rates of methane (CH 4 ) from the natural gas supply chain result in lost profit from unsold product, public safety and property concerns due to potential explosion hazards, and a potentially large source of economic damages from legal liabilities. Yet large measurement challenges exist in identifying and quantifying CH 4 leak rates along the vast number and type of components in the natural gas supply chain. This is particularly true of the “midstream” components involved in the gathering, processing, compression, transmission, and storage of natural gas. This project developed and deployed new advances in chirped laser dispersion spectroscopy (CLaDS) to detect methane leaks from pipelines, compressor stations, and other midstream infrastructure from a remote position (standoff detection). The system was deployed from a van to measure fugitive methane leaks from a local compressor station as a proof-of-concept. The technique was validated through mobile laboratory measurements with in-situ sensors as well as controlled releases of methane. The system also mapped a plume from a controlled release of methane by tracking a small unmanned aerial system (sUAS) that carried a corner cube retroreflector which reflected the beam back to the instrument. The drone-based system quantified a leak rate to within 30% of the actual rate and localized the emission location within 5 m of the actual release location at standoff distances of 25-45 m. This sUAS-reflector tracking approach has benefits for mapping leak locations remotely using small drones flying around a facility. Benefits of a commercial sensor with these capabilities include reductions of leaks for pipeline operators (more profit), earlier detection of leaks to avoid catastrophic explosion hazards for public health and to mitigate property damage, and reduced methane emissions to the atmosphere (improving air quality).

03 NATURAL GAS↗