Search NASA⌕ Search

SEARCH · Search NASA

Results for “Streaming Data”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 631 records · Page 35
All sourcesNASA NTRSDOE OSTI
DOE OSTI2021

Misuse Detection for a Generalized SFR Test Reactor

Sodium-cooled Fast Reactors (SFRs) present unique challenges for international safeguards. SFRs possess neutron physics characteristics that if configured appropriately could produce more fissile material than consumed. An adversary state may choose to build an SFR, justified by a lack of domestic natural uranium and limited access or interest to procuring uranium from international markets. Once constructed, the state may choose to misuse the SFR for the purpose of diverting fissile plutonium from declared operation. This work shows that a demonstration SFR does not need to be configured as a plutonium breeder to create one Significant Quantity (SQ) of plutonium in a short amount of time (e.g., one to few years). However, such an extreme case of misuse would change the core reactivity in such a way as to be easily indicated by deviations of control rod position compared to declared operation. In this work a contrived SFR demonstration reactor was modeled for the purpose of exploring proliferation scenarios and how such misuse could be detected using the SFR's Reactor Data Acquisition System (RDAS). Typically, the International Atomic Energy Agency (IAEA) does not have access to the control rod position, power, thermal, pressure sensing and indicating systems of nuclear power plants. However, this work shows that such data streams can be compared against a parallel detailed simulation model (a Digital Twin) to detect possible misuse.

98 NUCLEAR DISARMAMENT, SAFEGUARDS, AND PHYSICAL P↗
NASA NTRS1994

Multifractal characterizations of nonstationarity and intermittency in geophysical fields: Observed, retrieved, or simulated

Geophysical data rarely show any smoothness at any scale, and this often makes comparison with theoretical model output difficult. However, highly fluctuating signals and fractal structures are typical of open dissipative systems with nonlinear dynamics, the focus of most geophysical research. High levels of variability are excited over a large range of scales by the combined actions of external forcing and internal instability. At very small scales we expect geophysical fields to be smooth, but these are rarely resolved with available instrumentation or simulation tools; nondifferentiable and even discontinuous models are therefore in order. We need methods of statistically analyzing geophysical data, whether measured in situ, remotely sensed or even generated by a computer model, that are adapted to these characteristics. An important preliminary task is to define statistically stationary features in generally nonstationary signals. We first discuss a simple criterion for stationarity in finite data streams that exhibit power law energy spectra and then, guided by developments in turbulence studies, we advocate the use of two ways of analyzing the scale dependence of statistical information: singular measures and qth order structure functions. In nonstationary situations, the approach based on singular measures seeks power law behavior in integrals over all possible scales of a nonnegative stationary field derived from the data, leading to a characterization of the intermittency in this (gradient-related) field. In contrast, the approach based on structure functions uses the signal itself, seeking power laws for the statistical moments of absolute increments over arbitrarily large scales, leading to a characterization of the prevailing nonstationarity in both quantitative and qualitative terms. We explain graphically, step by step, both multifractal statistics which are largely complementary to each other. The geometrical manifestations of nonstationarity and intermittency, 'roughness' and 'sparseness', respectively, are illustrated and the associated analytical (differentiability and continuity) properties are discussed. As an example, the two techniques are applied to a series of recent measurements of liquid water distributions inside marine stratocumulus decks; these are found to be multifractal over scales ranging from approximately 60 m to approximately 60 km. Finally, we define the 'mean multifractal plane' and show it to be a simple yet comprehensive tool with many applications including data intercomparison, (dynamical or stochastic) model and retrieval validations.

Davis, Anthony↗
DOE OSTI2024

Time series methods for the analysis of soundscapes and other cyclical ecological data

Biodiversity monitoring has entered an era of ‘big data’, exemplified by a near-continuous collection of sounds, images, chemical and other signals from organisms in diverse ecosystems. Such data streams have the potential to help identify new threats, assess the effectiveness of conservation interventions, as well as generate new ecological insights. However, appropriate analytical methods are often still missing, particularly with respect to characterizing cyclical temporal patterns. Here, we present a framework for characterizing and analysing ecological responses that represent nonstationary, complex temporal patterns and demonstrate the value of using Fourier transforms to decorrelate continuous data points. In our example, we use a framework based on three approaches (spectral analysis, magnitude squared coherence, and principal component analysis) to characterize differences in tropical forest soundscapes within and across sites and seasons in Gabon. By reconstructing the underlying, cyclic behaviour of the soundscape for each site, we show how one can identify circadian patterns in acoustic activity. Soundscapes in the dry season had a complex diel cycle, requiring multiple harmonics to represent daily variation, while in the wet season there was less variance attributable to the daily cyclic patterns. Our framework can be applied to most continuous, or near-continuous ecological data collected at a fine temporal resolution, allowing ecologists to explore patterns of temporal autocorrelation at multiple levels for biologically meaningful trends. Such methods will become indispensable as biological big data are used to understand the impact of anthropogenic pressures on biodiversity and to inform efforts to mitigate them.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2024

On learning what to learn: Heterogeneous observations of dynamics and establishing possibly causal relations among them

Abstract Before we attempt to (approximately) learn a function between two sets of observables of a physical process, we must first decide what the inputs and outputs of the desired function are going to be. Here we demonstrate two distinct, data-driven ways of first deciding “the right quantities” to relate through such a function, and then proceeding to learn it. This is accomplished by first processing simultaneous heterogeneous data streams (ensembles of time series) from observations of a physical system: records of multiple observation processes of the system. We determine (i) what subsets of observables are common between the observation processes (and therefore observable from each other, relatable through a function); and (ii) what information is unrelated to these common observables, therefore particular to each observation process, and not contributing to the desired function. Any data-driven technique can subsequently be used to learn the input–output relation—from k-nearest neighbors and Geometric Harmonics to Gaussian Processes and Neural Networks. Two particular “twists” of the approach are discussed. The first has to do with the identifiability of particular quantities of interest from the measurements. We now construct mappings from a single set of observations from one process to entire level sets of measurements of the second process, consistent with this single set. The second attempts to relate our framework to a form of causality: if one of the observation processes measures “now,” while the second observation process measures “in the future,” the function to be learned among what is common across observation processes constitutes a dynamical model for the system evolution.

Sroczynski, David W.↗
DOE OSTI2022

Robust Event Classification Using Imperfect Real-world PMU Data

Here, this paper studies robust event classification using imperfect real-world phasor measurement unit (PMU) data. By analyzing the real-world PMU data, we find it is challenging to directly use this dataset for event classifiers due to the low data quality observed in PMU measurements and event logs. To address these challenges, we develop a novel machine learning framework for training robust event classifiers, which consists of three main steps: data preprocessing, fine-grained event data extraction, and feature engineering. Specifically, the data preprocessing step addresses the data quality issues of PMU measurements (e.g., bad data and missing data); in the fine-grained event data extraction step, a model-free event detection method is developed to accurately localize the events from the inaccurate event timestamps in the event logs; and the feature engineering step constructs the event features based on the patterns of different event types, in order to improve the performance and the interpretability of the event classifiers. Based on the proposed framework, we develop a workflow for event classification using the real-world PMU data streaming into the system in real time. Using the proposed framework, robust event classifiers can be efficiently trained based on many off-the-shelf lightweight machine learning models. Numerical experiments using the real-world dataset from the Western Interconnection of the U.S power transmission grid show that the event classifiers trained under the proposed framework can achieve high classification accuracy while being robust against low-quality data.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗
DOE OSTI2022

Classification of Nuclear Reactor Operations Using Spatial Importance and Multisensor Networks

Distributed multisensor networks record multiple data streams that can be used as inputs to machine learning models designed to classify operations relevant to proliferation at nuclear reactors. The goal of this work is to demonstrate methods to assess the importance of each node (a single multisensor) and region (a group of proximate multisensors) to machine learning model performance in a reactor monitoring scenario. This, in turn, provides insight into model behavior, a critical requirement of data-driven applications in nuclear security. Using data collected at the High Flux Isotope Reactor at Oak Ridge National Laboratory via a network of Merlyn multisensors, two different models were trained to classify the reactor’s operational state: a hidden Markov model (HMM), which is simpler and more transparent, and a feed-forward neural network, which is less inherently interpretable. Traditional wrapper methods for feature importance were extended to identify nodes and regions in the multisensor network with strong positive and negative impacts on the classification problem. These spatial-importance algorithms were evaluated on the two different classifiers. The classification accuracy was then improved relative to baseline models via feature selection from 0.583 to 0.839 and from 0.811 ± 0.005 to 0.884 ± 0.004 for the HMM and feed-forward neural network, respectively. While some differences in node and region importance were observed when using different classifiers and wrapper methods, the nodes near the facility’s cooling tower were consistently identified as important—a conclusion further supported by studies on feature importance in decision trees. Node and region importance methods are model-agnostic, inform feature selection for improved model performance, and can provide insight into opaque classification models in the nuclear security domain.

Tibbetts, Jake↗
DOE OSTI2018

Linking big models to big data: efficient ecosystem model calibration through Bayesian model emulation

Data-model integration plays a critical role in assessing and improving our capacity to predict ecosystem dynamics. Similarly, the ability to attach quantitative statements of uncertainty around model forecasts is crucial for model assessment and interpretation and for setting field research priorities. Bayesian methods provide a rigorous data assimilation framework for these applications, especially for problems with multiple data constraints. However, the Markov chain Monte Carlo (MCMC) techniques underlying most Bayesian calibration can be prohibitive for computationally demanding models and large datasets. We employ an alternative method, Bayesian model emulation of sufficient statistics, that can approximate the full joint posterior density, is more amenable to parallelization, and provides an estimate of parameter sensitivity. Analysis involved informative priors constructed from a meta-analysis of the primary literature and specification of both model and data uncertainties, and it introduced novel approaches to autocorrelation corrections on multiple data streams and emulating the sufficient statistics surface. We report the integration of this method within an ecological workflow management software, Predictive Ecosystem Analyzer (PEcAn), and its application and validation with two process-based terrestrial ecosystem models: SIPNET and ED2. In a test against a synthetic dataset, the emulator was able to retrieve the true parameter values. A comparison of the emulator approach to standard brute-force MCMC involving multiple data constraints showed that the emulator method was able to constrain the faster and simpler SIPNET model's parameters with comparable performance to the brute-force approach but reduced computation time by more than 2 orders of magnitude. The emulator was then applied to calibration of the ED2 model, whose complexity precludes standard (brute-force) Bayesian data assimilation techniques. Both models are constrained after assimilation of the observational data with the emulator method, reducing the uncertainty around their predictions. Performance metrics showed increased agreement between model predictions and data. Our study furthers efforts toward reducing model uncertainties, showing that the emulator method makes it possible to efficiently calibrate complex models.

58 GEOSCIENCES↗
NASA NTRS2018

Land Surface Reflectances from Geostationary Sensors

GEONEX is a processing pipeline that produces a suite of satellite land surface products using data streams from the latest geostationary (GEO) sensors including the GOES016/ABI and the Himawari-8/AHI. The suite, created collaboratively by scientists from NASA and NOAA, includes top-of-atmosphere (TOA) reflectances, land surface reflectances (LSRs), vegetation indices, LAI/fPAR, and other downstream products. As a key component of the GEONEX product processing, we have adapted the Multi-Angle Implementation of Atmospheric Correction (MAIAC) algorithm to produce LSRs from the TOA data. Because the algorithm depends on building "stacks" of images, we first run internal geo-registration checks to ensure geo-spatial accuracy and consistency of the input (L1B) data before transferring them from the geostationary projection into a tile system in geographic grids. Scan-time is inferred from metadata and applied to calculate the sun-sensor angles for each grid cell. The MAIAC algorithm is run to detect clouds/shadows, estimate aerosol optical thickness (AOT), perform atmospheric corrections, and generate LSRs. We have processed 18-months (from 2016/04 onward) of AHI data over East Asia and Oceania at a 10-minute time step and 10-months (from 2018/01 onward) of ABI data over North and South Americas at a 15-minute time step. As a verification measure, we compare the GEONEX (AHI/ABI) surface reflectances with the standard MODIS products (MOD09GA) and the MODIS MAIAC products over pixels that have similar sun-view geometries. The results indicate general linear relationships between GEONEX and corresponding MODIS LSRs. In particular, the RMSEs between GEONEX and MOD09 data are comparable to those between MOD09 and MODIS MAIAC products, suggesting that the uncertainties of GEONEX LSRs fall into an acceptable range. However, direct comparisons of LSRs over pixels with different sun-view angles are not as straightforward and require more modeling efforts to correct the directional effects. Evaluation of such angular influences on the downstream products (e.g., vegetation indices) is also under investigation.

Geostationary satellite; Remote Sensing; Atmospher↗
NASA NTRS2023

Heterogeneity and Chemical Reactivity of the Remote Troposphere Defined By Aircraft Measurements - Corrected

The NASA Atmospheric Tomography (ATom) mission built a photochemical climatology of air parcels based on in situ measurements with the NASA DC-8 aircraft along objectively planned profiling transects through the middle of the Pacific and Atlantic oceans. In this paper we present and analyze a data set of 10 s (2 km) merged and gap-filled observations of the key reactive species driving the chemical budgets of O3 and CH4 (O3, CH4, CO, H2O, HCHO, H2O2, CH3OOH, C2H6, higher alkanes, alkenes, aromatics, NOx, HNO3, HNO4, peroxyacetyl nitrate, and other organic nitrates), consisting of 146 494 distinct air parcels from ATom deployments 1 through 4. Six models calculated the O3 and CH4 photochemical tendencies from this modeling data stream for ATom 1. We find that 80 %–90 % of the total reactivity lies in the top 50 % of the parcels and 25 %–35 % in the top 10 %, supporting previous model-only studies that tropospheric chemistry is driven by a fraction of all the air. Surprisingly, the probability densities of species and reactivities averaged on a model scale (100 km) differ only slightly from the 2 km ATom 10 s data, indicating that much of the heterogeneity in tropospheric chemistry can be captured with current global chemistry models. Comparing the ATom reactivities over the tropical oceans with climatological statistics from six global chemistry models, we find generally good agreement with the reactivity rates for O3 and CH4. Models distinctly underestimate O3 production below 2 km relative to the mid-troposphere, and this can be traced to lower NOx levels than observed. Attaching photochemical reactivities to measurements of chemical species allows for a richer, yet more constrained-to-what-matters, set of metrics for model evaluation.

Tropospheric Chemistry↗
DOE OSTI2022

Inference-Optimized AI and High Performance Computing for Gravitational Wave Detection at Scale

We introduce an ensemble of artificial intelligence models for gravitational wave detection that we trained in the Summit supercomputer using 32 nodes, equivalent to 192 NVIDIA V100 GPUs, within 2 h. Once fully trained, we optimized these models for accelerated inference using NVIDIA TensorRT. We deployed our inference-optimized AI ensemble in the ThetaGPU supercomputer at Argonne Leadership Computer Facility to conduct distributed inference. Using the entire ThetaGPU supercomputer, consisting of 20 nodes each of which has 8 NVIDIA A100 Tensor Core GPUs and 2 AMD Rome CPUs, our NVIDIA TensorRT-optimized AI ensemble processed an entire month of advanced LIGO data (including Hanford and Livingston data streams) within 50 s. Our inference-optimized AI ensemble retains the same sensitivity of traditional AI models, namely, it identifies all known binary black hole mergers previously identified in this advanced LIGO dataset and reports no misclassifications, while also providing a 3X inference speedup compared to traditional artificial intelligence models. We used time slides to quantify the performance of our AI ensemble to process up to 5 years worth of advanced LIGO data. In this synthetically enhanced dataset, our AI ensemble reports an average of one misclassification for every month of searched advanced LIGO data. We also present the receiver operating characteristic curve of our AI ensemble using this 5 year long advanced LIGO dataset. This approach provides the required tools to conduct accelerated, AI-driven gravitational wave detection at scale.

97 MATHEMATICS AND COMPUTING↗
NASA NTRS1973

Transmission of radiometer data from the Synchronous Meteorological Satellite

The Synchronous Meteorological Satellite uses a spin scanner radiometer which generates eight visual signals and two infrared signals. These signals are multiplexed and converted into a 28-Mbps data stream. This signal is transmitted to ground by quadriphase modulation at 1686.1 MHz. On the ground, the digital signal is reconstructed to an analog signal. To conserve bandwidth, an analog-to-digital converter with a nonlinear transfer function was used for the visual signals. The size of the quantization step was made proportional to the noise output of the scanner photomultiplier tube which increases as the square root of incident light. The radiometer data transmission link was simulated on a digital computer to determine the transfer function. Some results of the simulation are shown.

Davies, R. S.↗
NASA NTRS1999

Retrieval Assimilation and Modeling of Atmospheric Water Vapor from Ground- and Space-Based GPS Networks: Investigation of the Global and Regional Hydrological Cycles

Uncertainty over the response of the atmospheric hydrological cycle (particularly the distribution of water vapor and cloudiness) to anthropogenic forcing is a primary source of doubt in current estimates of global climate sensitivity, which raises severe difficulties in evaluating its likely societal impact. Fortunately, a variety of advanced techniques and sensors are beginning to shed new light on the atmospheric hydrological cycle. One of the most promising makes use of the sensitivity of the Global Positioning System (GPS) to the thermodynamic state, and in particular the water vapor content, of the atmosphere through which the radio signals propagate. Our strategy to derive the maximum benefit for hydrological studies from the rapidly increasing GPS data stream will proceed in three stages: (1) systematically analyze and archive quality-controlled retrievals using state-of-the-art techniques; (2) employ both currently available and innovative assimilation procedures to incorporate these determinations into advanced regional and global atmospheric models and assess their effects; and (3) apply the results to investigate selected scientific issues of relevance to regional and global hydrological studies. An archive of GPS-based estimation of total zenith delay (TZD) data and water vapor where applicable has been established with expanded automated quality control. The accuracy of the GPS estimates is being monitored; the investigation of systematic errors is ongoing using comparisons with water vapor radiometers. Meteorological packages have been implemented. The accuracy and utilization of the TZD estimates has been improved by implementing a troposphere gradient model. GPS-based gradients have been validated as real atmospheric moisture gradients, establishing a link between the estimated gradients and the passage of weather fronts. We have developed a generalized ray tracing inversion scheme that can be used to analyze occultation data acquired from space- or land-based receivers. The National Center for Atmospheric Research mesoscale model (version MM5) has been adapted for Southern California, and assimilation studies are underway. Additional information is contained in the original.

Dickey, Jean O.↗
DOE OSTI2019

Strategies to minimize the influence of instrumental bias in neutron scattering

The influence of instrumental bias on neutron scattering measurements that involve a change between two experimental states is studied analytically and computationally. Different sources of bias were applied to an unperturbed (perfect) data stream. The influence of bias on the measurement was quantified and means to mitigate it is demonstrated. We find frequent changes of state are better than fewer, and evenly alternating between changes of state is often unwise.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗
DOE OSTI2020

Noise-Based Selection of Robust Inherited Model for Accurate Continual Learning

There is a growing demand for an intelligent system to continually learn knowledge from a data stream. Continual learning requires both the preservation of previous knowledge (i.e., avoiding catastrophic forgetting) and the acquisition of new knowledge. Different from previous works that focus only on model adaptation (e.g., regularization, network expansion, memory rehearsal, etc.), we propose a novel training scheme named acquisitive learning (AL), which emphasizes both the knowledge inheritance and knowledge acquisition. AL starts from an elaborately selected model with pre-trained knowledge (the inherited model) and then adapts it to new data using segmented training. The selection is achieved by injecting random noise to various inherited models for better model robustness, which promises higher accuracy in further knowledge acquisition. The approach is validated by the visualization of the loss landscape and quantitative roughness measurement. The combination of the selective inherited model and knowledge acquisition reduces catastrophic forgetting by 10X on the CIFAR-100 dataset.

Du, Xiaocong↗
DOE OSTI2023

Model Agnostic Bayesian Framework for Online Anomaly/Event Detection in PMU Data

Phasor measurement units (PMU) are integral to the modernization and automation plan of the electric power industry. A PMU data signature contains system-level events (e.g., faults, generation/load change, etc.) and any measurement/device-related errors. Therefore, the reliable and resilient operation of power systems is equivalent to the quality of the PMU data and the situation awareness provided by its data signature. Despite recent progress, current state-of-the-art methods are not fool-proof and have certain limitations tracing an error/abnormality to sensor sub-components and grid systems. This is because of technical challenges imposed by the scarcity of the labeled information, loss of data quality, and non-stationarity of data. In this paper, we consider the online PMU data stream as an output of a stochastic process and pose the anomaly/event detection as a changepoint detection problem dealing with detecting parameter changes in the underlying stochastic processes. The proposed model-agnostic framework relies on: (a) feature extraction utilizing the minimum volume enclosing ellipsoids (MVEE) method from raw PMU observations and (b) a Bayesian framework of changepoint detection. The validity of the proposed methodology is discussed through numerical experiments on real-world utility-scale PMU data.

Hossain, Ramij Raja↗
DOE OSTI2026

Multi-modal dynamic radiography using short-pulse laser-generated probe beams

Radiography is an important tool for the interrogation of dynamic experiments in the fields of dynamic properties of materials, and in condensed matter, high explosive, and high-energy-density physics. Multi-modal radiography advances the hypothesis that combining the information delivered by multiple radiographic modalities can lead to more constrained (improved) “reconstruction” of the scene than can be obtained from a single probe. We identify four modalities: multi-probe, time sequence, multi-view, and multi-messenger. Multi-probe radiography is a promising candidate for a next-generation dynamic radiographic facility. High-energy X-rays are the most frequently used probe for dynamic radiography, although recent developments show the utility of proton (pRad), electron (eRad), and neutron probe beams. Because each probing species interacts with material in the radiographic scene through quantitatively different mechanisms, each returns independent information about the scene, which can add extra constraints to the reconstruction process. How to conduct detailed, quantitative “co-analysis” of multiple data streams remains an area of active research. Multi-beam, short-pulse, laser-generated probes offer sufficient dose, an appropriate spectrum, and appropriate spatio-temporal resolution to produce high-quality dynamic radiographs. This paper reports on technology development to advance the state of the art of multi-modal/multi-probe radiography and the pursuit of both deterministic and inferential (AI/ML assisted) co-analysis methodologies to produce more constrained reconstructions from multi-modal data.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗
DOE OSTI2022

1.2.4.404 - Data-Driven Approach for Hydropower Plant Controller Prototyping Using Remote Hardware in the Loop (DR-HIL)

Real-time prototyping of hydropower plant controls is important for reducing the cost and the risk of field deployment. This project will 1) collect design and operational data from actual hydro plants and 2) use a physics-informed machine learning approach for real-time emulation of hydropower plants, including hydro turbine and hydrodynamics. The data-driven models will be interfaced with digital real-time simulation at NREL’s Flatirons campus for hardware-in-the-loop (HIL) testing of the governor hardware device or controller-HIL (CHIL). The proposed approach will also establish the connectivity based remote CHIL testing capability using real-time data streams from an actual hydro plant. This integrated hydro-plant emulation with CHIL will be used to prototype hydro-governor controls and eventually provide an opportunity to test hydropower integrated with various technologies (e.g. conventional and renewable generation, energy conversion, etc.) as HIL.

controls prototyping↗
DOE OSTI2023

Signal Processing Based Method for Real-Time Anomaly Detection in High-Performance Computing

Performance anomalies can manifest as irregular execution times or abnormal execution events for many reasons, including network congestion and resource contention. Detecting such anomalies in real-time by analyzing the details of performance traces at scale is impractical due to the sheer volume of data High-Performance Computing (HPC) applications produce. In this paper, we propose formulating HPC performance anomaly detection as a signal-processing problem where anomalies can be treated as noise. We evaluate our proposed method in comparison with two other commonly used anomaly detection techniques of varying complexity based on their detection accuracy and scalability. Since real-time in-situ anomaly detection at a large scale requires lightweight methods that can handle a large volume of streaming data, we find that our proposed method provides the best trade-off. We then implement the proposed method in Chimbuko, the first online, distributed, and scalable workflow-level performance trace analysis framework. We compare our proposed signal-based anomaly detection algorithm with two other methods using a function of their accuracy, F1 score, and detection overhead. Our experiments demonstrate that our proposed approach achieves a 99% improvement for the benchmark datasets and a 93% improvement with Chimbuko traces.

99 GENERAL AND MISCELLANEOUS↗