Search NASA⌕ Search

SEARCH · Search NASA

Results for “streaming data”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 1,333 records · Page 74

In-Situ Inference: Bringing Advanced Data Science Into Exascale Simulations

As simulations generate ever-increasing amounts of data, there are correspondingly richer opportunities for analysis and scientific discovery - discoveries that will be missed if most of the data must be discarded before it is analyzed. Because future exascale architectures will be increasingly storage-limited, it will not be possible to save the vast majority of simulation data for later analysis, requiring analysis to occur “in-situ” within the simulation. However, existing in-situ data analysis frameworks provide little or no support for one of the most sophisticated forms of data science: probabilistic statistical modeling or uncertainty quantification (UQ), and the accompanying challenge of inference - fitting those statistical models to massive simulation output. Our goal is to develop the fundamental statistical algorithms and computer science needed to perform statistical inference in-situ (in HPC simulations) to the full stream of data those simulations generate.

97 MATHEMATICS AND COMPUTING↗

Artificial Intelligence for Data Center Operations (AIOps): Cooperative Research and Development (Final Report)

High performance computing data centers will increasingly need to rely on automation to keep pace with exascale growth in compute capability and to manage and optimize the data center environment and facility resources. Artificial intelligence and machine learning approaches provide the means to improve HPC data center operational efficiency, by learning historical trends and training models to operate on real-time data collected from both IT and facilities sources. NREL has developed methods of real-time collection, aggregation and streaming of these data in the ESIF HPC Data Center and has collected a significant dataset of relevant metrics across computer systems, racks, environmental, building and utility sources for research into various predictive analytics problems. HPE's Advanced Technology Group (ATG) is doing comprehensive research into exascale monitoring and management for High Performance Computing (HPC) systems (hereinafter HPE's Data Monitoring/ Management Technology). NREL and HPE will collaborate to add Artificial Intelligence (AI) to NREL's real-time data collection/ aggregation/ streaming system and HPE's Data Monitoring/ Management System, with the goal of improving the operational efficiency of NREL's Energy Systems Integration Facility (ESIF) HPC Data Center through data analytics on both historical and real-time data from IT systems and facilities operations. This collaboration will consist of efforts in Data Management, Data Analytics, and AI/ML Optimization for both manual and autonomous intervention in data center operations. This will be a multi-year, multi-staged effort with a goal towards building capabilities for an Advanced Smart Facility, and demonstration of these techniques in the NREL ESIF HPC Data Center.

97 MATHEMATICS AND COMPUTING↗

Bayesian Analysis of the Cosmic Microwave Background

There is a wealth of cosmological information encoded in the spatial power spectrum of temperature anisotropies of the cosmic microwave background! Experiments designed to map the microwave sky are returning a flood of data (time streams of instrument response as a beam is swept over the sky) at several different frequencies (from 30 to 900 GHz), all with different resolutions and noise properties. The resulting analysis challenge is to estimate, and quantify our uncertainty in, the spatial power spectrum of the cosmic microwave background given the complexities of "missing data", foreground emission, and complicated instrumental noise. Bayesian formulation of this problem allows consistent treatment of many complexities including complicated instrumental noise and foregrounds, and can be numerically implemented with Gibbs sampling. Gibbs sampling has now been validated as an efficient, statistically exact, and practically useful method for low-resolution (as demonstrated on WMAP 1 and 3 year temperature and polarization data). Continuing development for Planck - the goal is to exploit the unique capabilities of Gibbs sampling to directly propagate uncertainties in both foreground and instrument models to total uncertainty in cosmological parameters.

methods - statistical↗

A Preliminary Development of The Intelligent Change Detection System (ICDS): Using Machine Learning to Combat Change Blindness in Remote Operation Environments

The emergence of Advance Air Mobility (AAM) will increase the number and types of aerial vehicles operating in shared airspaces, which will subsequently affect the amount of actionable data that ground control station operators (GCSOs) will be expected to manage. In this environment, GCSOs are particularly susceptible to the visual perceptual phenomenon known as change blindness, in which a stimulus undergoes a change without the change being noticed by its observer. A machine agent designed to detect change blindness and mitigate the effects could improve human agent performance in a data-saturated remote operations environment. The objective of this paper is to describe a proof-of-concept system architecture that integrates real-time eye tracking and vehicle telemetry data to prevent human-agent errors resulting from change blindness while operating aircraft from a remote ground control station (GCS).

Real-Time Streaming Data Architecture↗

A Preliminary Development of The Intelligent Change Detection System (ICDS): Using Machine Learning to Combat Change Blindness in Remote Operation Environments

The emergence of Advance Air Mobility (AAM) will increase the number and types of aerial vehicles operating in shared airspaces, which will subsequently affect the amount of actionable data that ground control station operators (GCSOs) will be expected to manage. In this environment, GCSOs are particularly susceptible to the visual perceptual phenomenon known as change blindness, in which a stimulus undergoes a change without the change being noticed by its observer. A machine agent designed to detect change blindness and mitigate the effects could improve human agent performance in a data-saturated remote operations environment. The objective of this paper is to describe a proof-of-concept system architecture that integrates real-time eye tracking and vehicle telemetry data to prevent human-agent errors resulting from change blindness while operating aircraft from a remote ground control station (GCS).

Real-Time Streaming Data Architecture↗

A search for planetary Nebulae with the Sloan digital sky survey: the outer regions of M31

We have developed a method to identify planetary nebula (PN) candidates in imaging data of the Sloan Digital Sky Survey (SDSS). This method exploits the SDSS's five-band sampling of emission lines in PN spectra, which results in a color signature distinct from that of other sources. Selection criteria based on this signature can be applied to nearby galaxies in which PNe appear as point sources. We applied these criteria to the whole area of M31 as scanned by the SDSS, selecting 167 PN candidates that are located in the outer regions of M31. The spectra of 80 selected candidates were then observed with the 2.2 m telescope at Calar Alto Observatory. These observations and cross-checks with literature data show that our method has a selection rate efficiency of about 90%, but the efficiency is different for the different groups of PN candidates. In the outer regions of M31, PNe trace different well-known morphological features like the Northern Spur, the NGC 205 Loop, the G1 Clump, etc. In general, the distribution of PNe in the outer region 8 < R < 20 kpc along the minor axis shows the {sup e}xtended disk{sup —}a rotationally supported low surface brightness structure with an exponential scale length of 3.21 ± 0.14 kpc and a total mass of ∼10{sup 10} M {sub ☉}, which is equivalent to the mass of M33. We report the discovery of three PN candidates with projected locations in the center of Andromeda NE, a very low surface brightness giant stellar structure in the outer halo of M31. Two of the PNe were spectroscopically confirmed as genuine PNe. These two PNe are located at projected distances along the major axis of ∼48 Kpc and ∼41 Kpc from the center of M31 and are the most distant PNe in M31 found up to now. With the new PN data at hand we see the obvious kinematic connection between the continuation of the Giant Stream and the Northern Spur. We suggest that 20%-30% of the stars in the Northern Spur area may belong to the Giant Stream. In our data we also see a possible kinematic connection between the Giant Stream and PNe in Andromeda NE, suggesting that Andromeda NE could be the core or remnant of the Giant Stream. Using PN data we estimate the total mass of the Giant Stream progenitor to be ≈10{sup 9} M {sub ☉}. About 90% of its stars appear to have been lost during the interaction with M31.

79 ASTRONOMY AND ASTROPHYSICS↗

Computer Program Recognizes Patterns in Time-Series Data

A computer program recognizes selected patterns in time-series data like digitized samples of seismic or electrophysiological signals. The program implements an artificial neural network (ANN) and a set of N clocks for the purpose of determining whether N or more instances of a certain waveform, W, occur within a given time interval, T. The ANN must be trained to recognize W in the incoming stream of data. The first time the ANN recognizes W, it sets clock 1 to count down from T to zero; the second time it recognizes W, it sets clock 2 to count down from T to zero, and so forth through the Nth instance. On the N + 1st instance, the cycle is repeated, starting with clock 1. If any clock has not reached zero when it is reset, then N instances of W have been detected within time T, and the program so indicates. The program can readily be encoded in a field-programmable gate array or an application-specific integrated circuit that could be used, for example, to detect electroencephalographic or electrocardiographic waveforms indicative of epileptic seizures or heart attacks, respectively.

Hand, Charles↗

Big Data Synchrophasor Monitoring and Analytics for Resiliency Tracking (BDSMART)

This report contains key findings from a project titled Big Data Synchrophasor Monitoring and Analytics for Resiliency Tracking (BDSMART), which was carried out through a collaborative effort of a team of researchers from Texas A&M Engineering Experiment Station, Temple University, and Quanta Technology, LLC. The in-kind support came from OSIsoft (acquired by AVEVA), which provided their PI Historian software to demonstrate the use case of streaming PMU data. The first section of the report describes the project goals and objectives related to the development of Machine Learning (ML) models capable of detecting and classifying events by processing phasor measurements captured in the field by Phasor Measurement Units (PMUs). The data for this study was contributed by the utilities/ISOs from the Western and Eastern interconnects and ERCOT, further referred to as Interconnect B (IC B), Interconnect A (IC A), and Interconnect C (IC C), respectively. The approach that the BDSMART Research Team proposed and the key research tasks defined by the team are outlined in this section. The next section describes the technical approach. We first discuss the data constraints related to the PMU measurements and data interpretation constraints imposed by the data contributors. They provided neither the topological information of the grid nor PMU placement locations and captured recorded data at very few locations in the system with the reporting rate of either 30 or 60 fps. The recordings are mostly positive sequence voltage, frequency, and ROCOF, and in some limited cases, three-phase voltages and currents. We then reflect on the bad data issues that stem from poor recording practices and vague definitions of the PMU status bits to supposedly be used for bad data identification. Finally, the data discovery points to imprecise time stamps with incomplete event start/end time, as well as inconsistent and incomplete event labeling, which combined make the implementation of the data models using supervising learning quite challenging. Following the data discovery study, we hypothesize that because the IC B data has the most complete labels, we should focus our model development on that data and then test it on data from other interconnects. We also define the common metrics used to evaluate the results from the ML algorithm tests. We concluded this section by summarizing the common ML models we used and explaining how we implemented and tested them. The issues from this section are expanded in the Training Dataset Report from this project. The final section of this report deals with the accomplishments and conclusions. As the accomplishments, we formulate the problem we are solving and what is achieved by solving the problem. We then reflect on each of the analytics tools we developed and point out the performance of each tool when applied to solving the mentioned problems. We reference this work for further details to the papers we published on each tool. In the conclusions, we give recommendations on how to improve future PMU recording practices to facilitate the ML algorithm implementation and guidance for the future standardization work aimed at clarifying the ambiguities associated with the PMU status bits. We finally list future tasks that can bring about further improvements in the proposed algorithms. The issues from this section are expanded in the Training, and Test Dataset Report filed at the project completion date.

97 MATHEMATICS AND COMPUTING↗

Contaminant Source Identification from Finite Sensor Data: Perron–Frobenius Operator and Bayesian Inference

Sensors in the built environment ensure safety and comfort by tracking contaminants in the occupied space. In the event of contaminant release, it is important to use the limited sensor data to rapidly and accurately identify the release location of the contaminant. Identification of the release location will enable subsequent remediation as well as evacuation decision-making. In previous work, we used an operator theoretic approach—based on the Perron–Frobenius (PF) operator—to estimate the contaminant concentration distribution in the domain given a finite amount of streaming sensor data. In the current work, the approach is extended to identify the most probable contaminant release location. The release location identification is framed as a Bayesian inference problem. The Bayesian inference approach requires considering multiple release location scenarios, which is done efficiently using the discrete PF operator. The discrete PF operator provides a fast, effective and accurate model for contaminant transport modeling. The utility of our PF-based Bayesian inference methodology is illustrated using single-point release scenarios in both two and three-dimensional cases. The method provides a fast, accurate, and efficient framework for real-time identification of contaminant source location.

42 ENGINEERING↗

Temporal Characterization and Filtering of Sensor Data to Support Anomaly Detection

Here, we present an approach for characterizing complex temporal behavior in the sensor measurements of a system in order to support detection of anomalies in that system. We first characterize typical behavior by extending a hidden Markov model-based approach to time series alignment. We then use a trace of that learned behavior to develop a particle filter that enables efficient estimation of the filtering distribution on the state space. This produces filtered residuals that can then be used in an anomaly detection framework. Our motivating example is the daily behavior of a building’s heating, ventilation, and air conditioning (HVAC) system, using sensor measurements that arrive every minute and induce a state space with 15,120 states. We provide an end-to-end demonstration of our approach showing improved performance of anomaly detection after application of alignment and filtering compared to the unaligned data. The proposed model is implemented as a computationally efficient R package alignts (align time series) built with R and Fortran 95 with OpenMP support.

47 OTHER INSTRUMENTATION↗

Waste-to-Energy Evaluation: U.S. Virgin Islands

This NREL technical report evaluates the environmental impact and fundamental economics of waste-to-energy (WTE) technology based on available data from commercially operating WTE facilities in the United States. In particular, it considers life-cycle impacts of WTE as compared to landfill disposal and various forms of electrical generation, as well as WTE impacts on source reduction or recyclingprograms. In addition, it evaluates the economics and potential environmental impact of WTE in the U.S. Virgin Islands (USVI) based on existing USVI waste stream characterization data, recycling challenges unique to the USVI, and the results of cost and environmental modeling of four municipal solid waste (MSW) management options, including landfill, refuse-derived fuel (RDF) production,recycling, and gassification plus RDF.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

Ocean observations with EOS/MODIS: Algorithm development and post launch studies

Several significant accomplishments were made during the present reporting period. We have completed our basic study of using the 1.38 micron MODIS band for removal of the effects of thin cirrus clouds and stratospheric aerosol. The results suggest that it should be possible to correct imagery for thin cirrus clouds with optical thicknesses as large as 0.5 to 1.0. We have also acquired reflectance data for oceanic whitecaps during a cruise on the RV Malcolm Baldrige in the Gulf of Mexico. The reflectance spectrum of whitecaps was found to be similar to that for breaking waves in the surf zone measured by Frouin, Schwindling and Deschamps. We installed a CIMEL sun photometer at Fort Jefferson on the Dry Tortugas off Key West in the Gulf of Mexico. The instrument has yielded a continuous stream of data since February. It shows that the aerosol optical thickness at 669 nm is often less than 0.1 in winter. This suggests that the Southern Gulf of Mexico will be an excellent winter site for vicarious calibration. In addition, we completed a study of the effect of vicarious calibration, i.e., the accuracy with which the radiance at the top of the atmosphere (TOA) can be predicted from measurement of the sky radiance at the bottom of the atmosphere (BOA). The results suggest that the neglect of polarization in the aerosol optical property inversion algorithm and in the prediction code for the TOA radiances is the largest error associated with the radiative transfer process. Overall, the study showed that the accuracy of the TOA radiance prediction is now limited by the racliometric calibration error in the sky radiometer. Finally, considerable coccolith light scattering data were obtained in the Gulf of Maine with a flow-through instrument, along with data relating to calcite concentration and the rate of calcite production.

Gordan, Howard R.↗

Low-Cutoff, High-Pass Digital Filtering of Neural Signals

The figure depicts the major functional blocks of a system, now undergoing development, for conditioning neural signals acquired by electrodes implanted in a brain. The overall functions to be performed by this system can be summarized as preamplification, multiplexing, digitization, and high-pass filtering. Other systems under development for recording neural signals typically contain resistor-capacitor analog low-pass filters characterized by cutoff frequencies in the vicinity of 100 Hz. In the application for which this system is being developed, there is a requirement for a cutoff frequency of 5 Hz. Because the resistors needed to obtain such a low cutoff frequency would be impractically large, it was decided to perform low-pass filtering by use of digital rather than analog circuitry. In addition, it was decided to timemultiplex the digitized signals from the multiple input channels into a single stream of data in a single output channel. The signal in each input channel is first processed by a preamplifier having a voltage gain of approximately 50. Embedded in each preamplifier is a low-pass anti-aliasing filter having a cutoff frequency of approximately 10 kHz. The anti-aliasing filters make it possible to couple the outputs of the preamplifiers to the input ports of a multiplexer. The output of the multiplexer is a single stream of time-multiplexed samples of analog signals. This stream is processed by a main differential amplifier, the output of which is sent to an analog-to-digital converter (ADC). The output of the ADC is sent to a digital signal processor (DSP).

Mojarradi,Mohammad↗

Model Data Archive Associated with Manuscript "Fire-altered Carbon Pools Create Disturbance Memory in Stream Dissolved Organic Carbon"

This data package supports the publication “Fire-altered Carbon Pools Create Disturbance Memory in Stream Dissolved Organic Carbon” by Li et al. (2026). The package contains processed model inputs, configuration files, restart files, simulation outputs, scripts, and visualization products used to evaluate post-fire dissolved organic carbon (DOC) dynamics in the Naches River Watershed, Washington, USA, following the 2021 Schneider Springs Fire. The modeling workflow couples ELM-BGC, the biogeochemistry-enabled Energy Exascale Earth System Model Land Model; ATS, the Advanced Terrestrial Simulator for integrated surface-subsurface hydrology; and PFLOTRAN, a reactive transport model for multicomponent aqueous geochemistry. Together, these models simulate how wildfire-induced changes in vegetation, litter, coarse woody debris, and soil organic matter influence DOC production, transport, and reaction from burned hillslopes to stream networks. The archive includes preprocessed meteorological, geospatial, hydrologic, and biogeochemical forcing data; ELM-BGC-derived DOC source terms; ATS mesh files; PFLOTRAN reactive-transport inputs; model configuration files; spin-up and transient restart files; watershed-scale diagnostic outputs; stream concentration time series; and figures or visualization files used to inspect and reproduce key results. File types include Hierarchical Data Format 5 (HDF5) files for gridded forcing and model-coupling data, model input and configuration files for ELM-BGC, ATS, and PFLOTRAN, restart and simulation-output files generated by the modeling workflow, tabular or time-series diagnostic outputs, scripts for post-processing and figure generation, and image or visualization products associated with the manuscript. Use of the package depends on the intended task. Re-running the simulations requires the relevant modeling software, including ELM-BGC, ATS, and PFLOTRAN as ATS's geochemical engine. Inspecting outputs and reproducing figures requires Python with scientific plotting libraries such as Matplotlib, and three-dimensional model outputs may be viewed with ParaView. Geographic information system files or maps may be inspected with ArcGIS Pro or comparable GIS software. The data package is intended to enable traceability, reuse, and partial reproduction of the coupled land-to-watershed hydro-biogeochemical modeling workflow used to test how wildfire disturbance affects terrestrial carbon pools and downstream DOC dynamics.

ATS↗

Fault-Tolerant and Elastic Streaming MapReduce with Decentralized Coordination

The MapReduce programming model, due to its simplicity and scalability, has become an essential tool for processing large data volumes in distributed environments. Recent Stream Processing Systems (SPS) extend this model to provide low-latency analysis of high-velocity continuous data streams. However, integrating MapReduce with streaming poses challenges: first, the runtime variations in data characteristics such as data-rates and key-distribution cause resource overload, that inturn leads to fluctuations in the Quality of the Service (QoS); and second, the stateful reducers, whose state depends on the complete tuple history, necessitates efficient fault-recovery mechanisms to maintain the desired QoS in the presence of resource failures. We propose an integrated streaming MapReduce architecture leveraging the concept of consistent hashing to support runtime elasticity along with locality-aware data and state replication to provide efficient load-balancing with low-overhead fault-tolerance and parallel fault-recovery from multiple simultaneous failures. Our evaluation on a private cloud shows up to 2:8 improvement in peak throughput compared to Apache Storm SPS, and a low recovery latency of 700 -1500 ms from multiple failures.

97 MATHEMATICS AND COMPUTING↗

Streaming Readout and Remote Compute

Streaming Readout Data Acquisition systems coupled with distributed resources spread over vast geographic distances present new challenges to the next generation of experiments. High bandwidth modern network connectivity opens the possibility to utilize large, general-use, HTC systems that are not necessarily located close to the experiment. Near real-time response rates and workflow colocation can provide high reliability solutions to ensure efficient use of beam time. This talk will focus on a few technologies currently being developed at Jefferson Lab and in collaboration with ESnet to support fully streaming DAQ systems.

Lawrence, David↗