SEARCH · Search NASA

Results for “process data”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 487 records · Page 27

All sources NASA NTRS DOE OSTI

DOE OSTI2022

DeepAdversaries: examining the robustness of deep learning models for galaxy morphology classification

With increased adoption of supervised deep learning methods for work with cosmological survey data, the assessment of data perturbation effects (that can naturally occur in the data processing and analysis pipelines) and the development of methods that increase model robustness are increasingly important. In the context of morphological classification of galaxies, we study the effects of perturbations in imaging data. In particular, we examine the consequences of using neural networks when training on baseline data and testing on perturbed data. We consider perturbations associated with two primary sources: (a) increased observational noise as represented by higher levels of Poisson noise and (b) data processing noise incurred by steps such as image compression or telescope errors as represented by one-pixel adversarial attacks. We also test the efficacy of domain adaptation techniques in mitigating the perturbation-driven errors. We use classification accuracy, latent space visualizations, and latent space distance to assess model robustness in the face of these perturbations. For deep learning models without domain adaptation, we find that processing pixel-level errors easily flip the classification into an incorrect class and that higher observational noise makes the model trained on low-noise data unable to classify galaxy morphologies. On the other hand, we show that training with domain adaptation improves model robustness and mitigates the effects of these perturbations, improving the classification accuracy up to 23% on data with higher observational noise. Domain adaptation also increases up to a factor of ${\approx}2.3$ the latent space distance between the baseline and the incorrectly classified one-pixel perturbed image, making the model more robust to inadvertent perturbations. Successful development and implementation of methods that increase model robustness in astronomical survey pipelines will help pave the way for many more uses of deep learning for astronomy.

79 ASTRONOMY AND ASTROPHYSICS↗

DOE OSTI2020

What can we learn from N 2 O isotope data? – Analytics, processes and modelling

The isotopic composition of nitrous oxide (N 2 O) provides useful information for evaluating N 2 O sources and budgets. Due to the co–occurrence of multiple N 2 O transformation pathways, it is, however, challenging to use isotopic information to quantify the contribution of distinct processes across variable spatiotemporal scales. Here, we present an overview of recent progress in N 2 O isotopic studies and provide suggestions for future research, mainly focusing on: analytical techniques; production and consumption processes; and interpretation and modelling approaches. Comparing isotope–ratio mass spectrometry (IRMS) with laser absorption spectroscopy (LAS), we conclude that IRMS is a precise technique for laboratory analysis of N 2 O isotopes, while LAS is more suitable for in situ/inline studies and offers advantages for site–specific analyses. When reviewing the link between the N 2 O isotopic composition and underlying mechanisms/processes, we find that, at the molecular scale, the specific enzymes and mechanisms involved determine isotopic fractionation effects. In contrast, at plot–to–global scales, mixing of N 2 O derived from different processes and their isotopic variability must be considered. We also find that dual isotope plots are effective for semi–quantitative attribution of co–occurring N 2 O production and reduction processes. More recently, process–based N 2 O isotopic models have been developed for natural abundance and 15 N–tracing studies, and have been shown to be effective, particularly for data with adequate temporal resolution. Here, despite the significant progress made over the last decade, there is still great need and potential for future work, including development of analytical techniques, reference materials and inter–laboratory comparisons, further exploration of N 2 O formation and destruction mechanisms, more observations across scales, and design and validation of interpretation and modelling approaches. Synthesizing all these efforts, we are confident that the N 2 O isotope community will continue to advance our understanding of N 2 O transformation processes in all spheres of the Earth, and in turn to gain improved constraints on regional and global budgets.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗

DOE OSTI2023

Development of ML FPGA Filter for Particle Identification and Tracking in Real Time

Real-time data processing is a frontier field in experimental particle physics. Machine Learning methods are widely used and have proven to be very powerful in particle physics. The growing computational power of modern FPGA boards allows us to add more sophisticated algorithms for real time data processing. Many tasks could be solved using modern Machine Learning (ML) algorithms which are naturally suited for FPGA architectures. The FPGA-based machine learning algorithm provides an extremely low, sub-microsecond, latency decision and makes information-rich data sets for event selection. We report work has started to evaluate an FPGA based Machine Learning (ML) algorithm for a real-time particle identification and tracking with Transition Radiation Detector (TRD) and e/m calorimeter. The first target is the GlueX experiment, with a plan to build a TRD based on GEM technology. GlueX trigger latency is 3.3 μs.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

DOE OSTI2019

Provenance–aware workflow for data quality management and improvement for large continuous scientific data streams

Data quality assessment, management and improvement is integral part of any big data intensive scientific research to ensure accurate, reliable and reproducible science discoveries. The task of maintaining the quality of data, however, is non-trivial and pose a challenge for a program like Department of Energy’s Atmospheric Radiation Measurement (ARM) that collects data from hundreds of instruments across the world,and distributes thousands of streaming data products that are continuously growing in near-real-time in an archive 1.7 Petabyte in size and growing. We present a computational data processing workflow to collect the data quality issue via an easy and intuitive web-based portal that allows reporting of any quality issues for any site, facility or instruments at a granularity down to individual variable in the data files. Portal allows instrument specialists and scientists to provide corrective actions in form of symbolic equation. A parallel processing framework applies the data improvement to large volume of data in efficient parallel environment, while optimizing data transfer and file I/O operations. Corrected files are systematically versioned and archived. A provenance tracking module tracks and records any change made to the data during its entire life cycle which are communicated transparently to the scientific users. Developed in Python using open source technologies, software architecture enables efficient and fast management and improvement of data in an operational data center environment.

Kumar, Jitendra↗

DOE OSTI2024

Darshan for HEP applications

Modern HEP workflows must manage increasingly large and complex data collections. HPC facilities may be employed to help meet these workflows’ growing data processing needs. However, a better understanding of the I/O patterns and underlying bottlenecks of these workflows is necessary to meet the performance expectations of HPC systems.Darshan is a lightweight I/O characterization tool that captures concise views of HPC application I/O behavior. It intercepts application I/O calls at runtime, records file access statistics for each process, and generates log files detailing application I/O access patterns.Typical HEP workflows include event generation, detector simulation, event reconstruction, and subsequent analysis stages. A study of the I/O behavior of the ATLAS simulation and filtering stage, and the CMS simulation workflow using Darshan is presented, including insights into the I/O operations and data access size.

Wang, Rui↗

DOE OSTI2026

Data for Yield from Iowa’s first commercial miscanthus fields: implications of spatial variability for productivity and sustainability beyond research plots

This dataset contains biomass yield measurements and associated vegetation index data collected from commercial Miscanthus × giganteus fields in eastern Iowa during the 2022–2023 growing seasons. The data support the analyses presented in the article: “Yield From Iowa's First Commercial Miscanthus Fields: Implications of Spatial Variability for Productivity and Sustainability Beyond Research Plots.” We collected 105 ground-truth biomass samples from four mature commercial fields (>4 years old) covering 92.81 ha. Samples were taken from 3 m² quadrats that were hand-harvested in alignment with commercial harvest timing. Stem biomass (excluding leaves) was weighed, moisture-corrected, and converted to dry-matter yield expressed in Mg DM ha⁻¹. Sampling locations were selected to capture spatial variability visible in aerial imagery and were recorded using RTK GPS. Each biomass observation was paired with vegetation indices derived from high-resolution PlanetScope satellite imagery (3 m resolution). Images were acquired throughout the growing season, and indices were calculated to evaluate their ability to predict end-of-season biomass yield. Statistical and machine learning approaches were used to identify key predictors, and a linear regression model based on end-of-July Green Normalized Difference Vegetation Index (GNDVI) was developed and evaluated. This repository includes the data used in that modeling workflow. Management practices, economic data, full imagery time series, and additional methodological details are described in the associated publication and are not included here. The dataset consists of three comma-separated value (CSV) files: 1. Combine_Groundtruth_Yield_VI_22_23.csv This file contains ground-truth biomass yield measurements and associated key vegetation index values collected during the 2022 and 2023 growing seasons. Rows: 105 observations Columns: Year — Year of observation (2022 or 2023) Field — Field location identifier Sample_number — Unique sample identifier GNDVI_End_Jul — Green Normalized Difference Vegetation Index calculated at end of July GNDVI_End_Aug — Green Normalized Difference Vegetation Index calculated at end of August NDRE_End_Aug — Normalized Difference Red Edge index calculated at end of August Biomass_Stem_Yield_MgDM/ha — Measured stem biomass yield (megagrams dry matter per hectare) 2. trainData_GNDVI.csv This file contains the subset of observations used to train the predictive relationship between July GNDVI and biomass yield. Rows: 76 observations Columns: Unnamed: 0 — Row index retained from the original data processing workflow GNDVI_End_Jul — GNDVI at end of July Stem_Yield_MgDM/ha — Observed stem biomass yield (Mg DM ha⁻¹) 3. testData_GNDVI.csv This file contains the test dataset used to evaluate model performance. Rows: 29 observations Columns: Unnamed: 0 — Row index retained from the original data processing workflow GNDVI_End_Jul — GNDVI at end of July Predicted_Yield_MgDM/ha — Model-predicted stem biomass yield (Mg DM ha⁻¹) Observed_Yield_MgDM/ha — Measured stem biomass yield (Mg DM ha⁻¹)

Potential yield, yield gap, in-field management, y↗

DOE OSTI2025

Characterization of contaminants in the Lyman-alpha forest auto-correlation with DESI

Baryon Acoustic Oscillations can be measured with sub-percent precision above redshift two with the Lyman-α (Lyα) forest auto-correlation and its cross-correlation with quasar positions. This is one of the key goals of the Dark Energy Spectroscopic Instrument (DESI) which started its main survey in May 2021. We present in this paper a study of the contaminants to the Lyα forest which are mainly caused by correlated signals introduced by the spectroscopic data processing pipeline as well as astrophysical contaminants due to foreground absorption in the intergalactic medium. Notably, an excess signal caused by the sky background subtraction noise is present in the Lyα auto-correlation in the first line-of-sight separation bin. We use synthetic data to isolate this contribution, we also characterize the effect of spectro-photometric calibration noise, and propose a simple model to account for both effects in the analysis of the Lyα forest. We then measure the auto-correlation of the quasar flux transmission fraction of low redshift quasars, where there is no Lyα forest absorption but only its contaminants. We demonstrate that we can interpret the data with a two-component model: data processing noise and triply ionized Silicon and Carbon auto-correlations. This result can be used to improve the modeling of the Lyα auto-correlation function measured with DESI.

79 ASTRONOMY AND ASTROPHYSICS↗

DOE OSTI2022

Enabling energy‐efficient manufacturing of pharmaceutical solid oral dosage forms via integrated techno‐economic analysis and advanced process modeling

Abstract The global pharmaceutical industry is a trillion‐dollar market. However, the pharmaceutical sector often lags in manufacturing innovation and automation which limits its potential to maximize energy efficiency. The integration of techno‐economic analysis (TEA) with advanced process models as part of an overarching smart manufacturing platform, can help industries create business models, which can be adapted for manufacturing to reduce energy consumption and operating costs while ensuring product quality which can further enable a more sustainable process operation. In this study, a rational design of experiment on three unit‐operations (wet granulation, drying, and milling) was performed on a batch (case 1) and continuous (case 2) pharmaceutical process to obtain experimental data. Process models for predicting product quality and energy efficiency of each of the three‐unit operations were developed. The experimental data were used to validate the models and good agreement was observed. The energy consumption of each unit operation was calculated using statistical models relating the power consumption and the process parameters. The developed process models and energy models were further integrated into a TEA framework, which quantified the energy and monetary cost of manufacturing for both batch and continuous manufacturing cases. With this integrated framework, energy costs savings of ~33% was obtained in the continuous manufacturing process (case 2) over the batch process (case 1).

Sampat, Chaitanya↗

DOE OSTI2023

Time-warping invariant quantum recurrent neural networks via quantum-classical adaptive gating

Adaptive gating plays a key role in temporal data processing via classical recurrent neural networks (RNNs), as it facilitates retention of past information necessary to predict the future, providing a mechanism that preserves invariance to time warping transformations. This paper builds on quantum RNNs (QRNNs), a dynamic model with quantum memory, to introduce a novel class of temporal data processing quantum models that preserve invariance to time-warping transformations of the (classical) input-output sequences. The model, referred to as time warping-invariant QRNN (TWI-QRNN), augments a QRNN with a quantum–classical adaptive gating mechanism that chooses whether to apply a parameterized unitary transformation at each time step as a function of the past samples of the input sequence via a classical recurrent model. The TWI-QRNN model class is derived from first principles, and its capacity to successfully implement time-warping transformations is experimentally demonstrated on examples with classical or quantum dynamics.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2023

In-depth analysis on parallel processing patterns for high-performance Dataframes

The Data Science domain has expanded monumentally in both research and industry communities during the past decade, predominantly owing to the Big Data revolution. Artificial Intelligence (AI) and Machine Learning (ML) are bringing more complexities to data engineering applications, which are now integrated into data processing pipelines to process terabytes of data. Typically, a significant amount of time is spent on data preprocessing in these pipelines, and hence improving its efficiency directly impacts the overall pipeline performance. The community has recently embraced the concept of Dataframes as the de-facto data structure for data representation and manipulation. However, the most widely used serial Dataframes today (R, pandas) experience performance limitations while working on even moderately large data sets. We believe that there is plenty of room for improvement by taking a look at this problem from a high-performance computing point of view. In a prior publication, we presented a set of parallel processing patterns for distributed dataframe operators and the reference runtime implementation, Cylon. In this paper, we are expanding on the initial concept by introducing a cost model for evaluating the said patterns. Furthermore, we evaluate the performance of Cylon on the ORNL Summit supercomputer.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2024

Long-Term Statistical Process Monitoring of an Ultrafiltration Water Treatment Process

As water treatment technology has improved, the amount of available process data has substantially increased, making real-time, data-driven fault detection a reality. One shortcoming of the fault detection literature is that methods are usually evaluated by comparing their performance on hand-picked, short-term case studies, which yields no insight into long-term performance. In this work, we first evaluate multiple statistical and machine learning approaches for detrending process data. Then, we evaluate the performance of a PCA-based fault detection approach, applied to the detrended data, to monitor influent water quality, filtrate quality, and membrane fouling of an ultrafiltration membrane system for indirect potable reuse. Based on two short case studies, the adaptive lasso detrending method is selected, and the performance of the multivariate approach is evaluated over more than a year. The method is tested for different sets of three critical tuning parameters, and we find that for long-term, autonomous monitoring to be successful, these parameters should be carefully evaluated. However, in comparison with industry standards of simpler, univariate monitoring or daily pressure decay tests, multivariate monitoring produces substantial benefits in long-term testing.

ammonia↗

DOE OSTI2025

Integration of the FastIC front-end electronics into the Picosec MicroMegas detector

The Picosec MicroMegas collaboration aims to develop gaseous fast-timing detectors; experimentally, intrinsic time resolutions from around 50 ps to better than 20 ps are obtained, depending on the exact detector configuration. Parts of developments focus on exploring various options of fast-timing multi-channel front-end electronics, to meet the data processing demands of experimental applications. One option is the FastIC, which was developed for reading out positive and negative input polarity sensors with intrinsic amplification. In this paper, the first results from reading out the gaseous Picosec MicroMegas detector with the FastIC are presented. In laboratory studies, a basic description of the data processing chain was performed using a function generator. The results from test beam measurements are used to characterise the timing performance and the charge processing of the combination of FastIC and Picosec MicroMegas, as well as to demonstrate the possibility of a multi-channel detector readout. Although the timing-at-threshold level of the FastIC introduces a time walk of around 1.5 ns, time resolutions of around 50 ps have been achieved.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗

DOE OSTI2021

Verification and validation of developed short-term forecasting models

Recent advancements in machine learning (ML) and artificial intelligence (AI) technologies provide an opportunity for leveraging data-driven algorithms to predict future nuclear power plant (NPP) operating conditions by using recorded plant process data. Successfully implementing these models can lead to cost-reducing, conditioned-based predictive maintenance through optimized maintenance schedules and a reduction of unnecessary maintenance activities. This report discusses the verification and validation of short-term forecasting processes (i.e., data cleaning, feature selection, model optimization, and forecasting) developed in previous reports. The verification and validation (V&V) process demonstrates the expected precision and accuracy when the ML model encounters new datasets from different systems. Shapley additive explanations were used as the primary means of feature selection across these different data set. Individual models were trained for each data set, then validated through a cross-validation procedure. In this report, two different ML models were tasked to predict variables from three different plant process data sets with varying prediction horizons. The results indicate that support vector regression (SVR) outperformed long short-term memory (LSTM) neural networks in regard to each data set and each prediction horizon in this study, but further tuning and optimization could improve long short-term memory results. However, each forecasting model showed reduced performance as the prediction horizon was extended from 1 hour to 1 day ahead. Research is ongoing to evaluate the optimal input variable space, which is based on a given set of process parameters, to further improve forecasting accuracy.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗

DOE OSTI2020

Mixed core processor unit

A method and apparatus for processing data. The data is sent to a processor unit comprising a group of neural cores, a group of digital processing cores, and a routing network connecting the group of digital processing cores. The data is processed in the processor unit to generate a result.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2025

Using the ATLAS experiment software on heterogeneous resources

With the large dataset expected from 2030 onwards by the HL-LHC at CERN, the ATLAS experiment is reaching the limits of the current data processing model in terms of traditional CPU resources based on x86_64 architectures and an extensive program for software upgrades towards the HL-LHC has been set up. The ARM CPU architecture is becoming a competitive and energy efficient alternative. Accelerators like GPUs are available in any recent HPC. In the past years ATLAS has successfully ported its full data processing and simulation software framework Athena to ARM and has invested significant effort in porting parts of the reconstruction and simulation algorithms to GPUs. We report on the successful usage of the ATLAS experiment offline and online software framework Athena on ARM and GPUs through the PanDA workflow management system at various WLCG sites. Furthermore we report on performance optimizations of the builds for ARM CPUs and the GPU integration efforts. We will discuss performance comparisons of different ARM and x86_64 architectures on WLCG resources and Cloud compute providers like GCP and AWS using ATLAS productions workflows as used in the Hep-Score23 benchmark suite.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗

DOE OSTI2023

Wave Detection and Tracking Within a Rotating Detonation Engine Through Object Detection

As the operational time window of experimental rotating detonation engines (RDEs) is expanded and the technology matures toward integration within gas turbines, monitoring techniques must evolve to offer computationally efficient and highly time-resolved diagnostics. In this study, computer vision object detection methodology that seeks to reduce data processing time and calculate wave velocity within drastically reduced time intervals as compared to traditional high-frame-rate RDE images analysis techniques is proposed. The adapted you-only-look-once object detection network is trained to detect individual detonation waves within single down-axis RDE images. The wave location and rotational direction detected within a frame are tracked through a series of high-speed images to calculate the frame-to-frame wave velocity with the time-step resolution of $\mathrm{20 μs}$ across a series of frames. The analysis of the annotation box size and image linearization effects is presented, demonstrating the lowest frame-to-frame velocity total uncertainty of $\mathrm{±3.8\%}$ and the highest classification speed of 9.5 frames per second using linearized images. Linearized images “unwrap” the RDE annulus pixel region to a reduced image size. Here, this new method offers great reductions in data processing times and unsteady detonation behavior insight at intervals more comparable to the timescales of detonation wave interactions via the application of machine learning to experimental RDE data.

33 ADVANCED PROPULSION SYSTEMS↗

DOE OSTI2021

KS4A-Omics1.0_FspDS682

Soil fungi facilitate the translocation of inorganic nutrients from soil minerals to other microorganisms and plants. This ability is particularly advantageous in impoverished soils, because fungal mycelial networks can bridge otherwise spatially disconnected and inaccessible nutrient hotspots. However, the molecular mechanisms underlying fungal mineral weathering and transport through soil remains poorly understood. Here, we addressed this knowledge gap by directly visualizing nutrient acquisition and transport through fungal hyphae in a mineral doped soil micromodel using a multimodal imaging approach. Here, we observed how a representative of common saprotrophic soil fungi, Fusarium sp. DS 682, exhibited a mechanosensory response (thigmotropism) around obstacles and through pore spaces (~12 μm) in the presence of minerals.This study establishes the significance of fungal biology and nutrient translocation mechanisms in maintaining fungal growth under water and nutrient limitations in a soil-like microenvironment, using a high-throughput multi-omic analysis approach. Data package KS4A-Omics.1.0_FspDS682 (Publication: Fungal Mineral Weathering Mechanisms Revealed Through Direct Molecular Visualization) contents reported here are the first version (1.0) and contain pre- and post-processed data using high throughput data capture technologies for multi-omic analysis and integration, this data package contains raw and post-processed experimental data for X-Ray Absorption Near Edge Structure Spectroscopy (XANES/XRF), Optical Microscopy, Proteomics, Scanning Electron Microscope (SEM), Time-of-Flight Secondary Ion Mass Spectroscopy (ToF-SIMS), X-Ray Diffraction Spectroscopy (XRD) files, and X- Ray Photoelectron Spectroscopy (XPS) using EMSL capabilities. This data package DOI contains a comprehensive collection of high-throughput multi-omics data and process metadata catalog. Support files include additional data download contents “Read Me” with dataset descriptor information and data source method application ontologies (see data dictionary section). Reported data download content is structured for compliance with reported guidelines provided by community standard initiatives and publisher stakeholder policies supporting FAIR data principles.

47 OTHER INSTRUMENTATION↗

DOE OSTI2025

Adaptive anomaly detection for identifying attacks in cyber-physical systems: A systematic literature review

Modern cyberattacks in cyber-physical systems (CPS) rapidly evolve and cannot be deterred effectively with most current methods, which focus on characterizing past threats. Adaptive anomaly detection (AAD) is among the most promising techniques to detect evolving cyberattacks, with an emphasis on fast data processing and model adaptation. AAD has been researched extensively; however, to the best of our knowledge, our work is the first systematic literature review (SLR) on current research in this field. We present a comprehensive SLR, gathering 397 relevant papers and systematically analyzing 65 of them (47 research and 18 survey papers) on AAD in CPS from 2013 to November 2023. We introduce a novel taxonomy considering attack types, CPS application, learning paradigm, data management, and algorithms. Our findings show that most studies addressed either model adaptation or data processing, but rarely both simultaneously. This indicates a research gap in fully adaptive solutions. We also categorize algorithms, datasets, and attack characteristics, and summarize strengths and weaknesses across the literature. Our review provides a structured and accessible reference for researchers and practitioners, offering insights into key trends and highlighting limitations in current approaches. Finally, we outline several future research directions, including the need for integrated real-time processing and adaptive learning, explainability, and uncertainty quantification in AAD for CPS.

Adaptation↗