Search NASA⌕ Search

SEARCH · Search NASA

Results for “data-driven approach”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 361 records · Page 20

Assessing shellfish water exposure to fecal bacteria pollution in Salish Sea: three-dimensional modeling and implications for monitoring

Fecal bacteria (FB) contamination poses significant risks to shellfish safety and management in coastal and estuarine waters. Despite extensive pollution identification and correction efforts, FB contamination in shellfish-growing areas persists in the Salish Sea, highlighting the need to identify overlooked sources and better understand FB transport from riverine and shoreline inputs to shellfish beds. To address this, a high-resolution three-dimensional hydrodynamic model coupled with FB kinetics was developed and applied to a case study site in Salish Sea—Portage Bay—to simulate freshwater plume circulation, flushing dynamics, and bacterial transport. Daily FB loading from the major freshwater inflow—Nooksack River was generated by both linear interpolation and integrating a machine learning approach (XGBoost), trained on historical hydrological and meteorological data. The model successfully reproduced both the magnitude and seasonal variation of FB concentrations in Portage Bay for the year of 2021, demonstrating that simplified FB kinetics with first-order decay due to mortality was effective in this dynamic coastal environment with short flushing time. Model results identified the Nooksack River as the dominant far-field FB source, while scenario simulations showed that near-field coastal stormwater outfalls elevated local FB levels following rainfall, particularly under low-flow conditions. The XGBoost prediction provided comparable or superior accuracy to linear interpolation, particularly during periods of missing observational data, by capturing short-term variability and event-driven loading more effectively. Integrating data-driven riverine FB inputs with mechanistic coastal numerical modeling provides a robust framework for operational forecasting of shellfish bed exposure risk and supports adaptive monitoring and management of shellfish growing areas in the Salish Sea and similar coastal systems.

Salish Sea↗

Dynamic Control of Sodium Cold Trap Purification Temperature Using LSTM System Identification

This study investigates the dynamic regulation of the sodium cold trap purification temperature at Argonne National Laboratory’s liquid sodium test facility, employing long short-term memory (LSTM) system identification techniques. The investigation introduces an innovative hybrid approach by integrating model predictive control (MPC) based on first principles dynamic models with a multi-step time–frequency LSTM model in predicting the temperature profiles of a sodium cold trap purification system. The long short-term memory–model predictive controller (LSTM-MPC) model employs a sliding window scheme to gather training samples for multi-step prediction, leveraging historical data to construct predictive models that capture the non-linearities of the complex system dynamics without explicitly modeling the underlying physical processes. The performance of the LSTM-MPC and MPC were evaluated through simulation experiments, where both models were assessed on their capacity to maintain the cold trap temperature within predefined set-points while minimizing deviations and overshoots. Results obtained show how the data-driven LSTM-MPC model demonstrates stability and adaptability. In contrast, the traditional MPC model exhibits irregularities, particularly evident as overshoots around set-point limits, which can potentially compromise its effectiveness over long prediction time intervals. The findings obtained offer valuable insights into integrating data-driven techniques for enhancing real-time monitoring systems.

LSTM-MPC↗

Probabilistic Deliverability Assessment of Distributed Energy Resources via Scenario-Based AC Optimal Power Flow

As electric grids decarbonize and distributed energy resources (DERs) become increasingly prevalent, interconnection assessments must evolve to reflect operational variability and control flexibility. This paper highlights key modeling limitations observed in practice and reviews approaches for modeling uncertainty. It then introduces a Probabilistic Deliverability Assessment (PDA) framework designed to complement and extend existing procedures. The framework integrates scenario-based AC optimal power flow (AC OPF), corrective dispatch, and optional multi-temporal constraints. Together, these form a structured methodology for quantifying DER utilization, deliverability, and reliability under uncertainty in load, generation, and topology. Outputs include interpretable metrics with confidence intervals that inform siting decisions and evaluate compliance with reliability thresholds across sampled operating conditions. A case study on Puerto Rico’s publicly available bulk power system model demonstrates the framework’s application using minimal input data, consistent with current interconnection practice. Across staged fossil generation retirements, the PDA identifies high-value DER sites and regions requiring additional reactive power support. Results are presented through mean dispatch signals, reliability metrics, and geospatial visualizations, demonstrating how the framework provides transparent, data-driven siting recommendations. The framework’s modular design supports incremental adoption within existing workflows, encouraging broader use of AC OPF in interconnection and planning contexts.

14 SOLAR ENERGY↗

GenAI-Based Digital Twins Aided Data Augmentation Increases Accuracy in Real-Time Cokurtosis-Based Anomaly Detection of Wearable Data

Early detection of potential infectious disease outbreaks is crucial for developing effective interventions. In this study, we introduce advanced anomaly detection methods tailored for health datasets collected from wearables, offering insights at both individual and population levels. Leveraging real-world physiological data from wearables, including heart rate and activity, we developed a framework for the early detection of infection in individuals. Despite the availability of data from recent pandemics, substantial gaps remain in data collection, hindering method development. To bridge this gap, we utilized Wasserstein Generative Adversarial Networks (WGANs) to generate realistic synthetic wearable data, augmenting our dataset for training. Subsequently, we use these augmented datasets to implement a cokurtosis-based technique for anomaly detection in multivariate time-series data. Our approach includes a comprehensive assessment of uncertainties in synthetic data compared to the actual data upon which it was modeled, as well as the uncertainty associated with fine-tuning anomaly detection thresholds in physiological measurements. Through our work, we present an enhanced method for early anomaly detection in multivariate datasets, with promising applications in healthcare and beyond. This framework could revolutionize early detection strategies and significantly impact public health response efforts in future pandemics.

Data-Driven Digital Twins↗

Flux Cube Reconstruction from Slitless Spectroscopy

Slitless spectroscopy enables efficient, large-area surveys without target preselection, yet it faces challenges from source blending, higher noise, and lost spatial–spectral information. We present an advanced, nonparametric, data-driven algorithm that leverages multiple dispersion angles to reconstruct three-dimensional flux distributions, providing low-resolution integral field unit capabilities from slitless data. By treating each pixel as an independent element, our method naturally handles source confusion without requiring prior assumptions regarding redshifts, templates, or model libraries. We validate the algorithm using simulated Roman Space Telescope wide-field slitless spectroscopy images that are equivalent to what is expected from the High-Latitude Time-Domain Survey. First, we demonstrate that a host-galaxy model reconstructed from multiple dispersion angles can be used to accurately subtract host light from a transient, recovering a Type Ia supernova spectrum with minimal bias. Second, we showcase a high-fidelity flux-cube reconstruction of a complex galaxy, successfully measuring the redshift and recovering continuum, emission, and absorption features. This approach highlights the potential of multi-dispersion-angle slitless data to provide spatially resolved spectral information in a nonparametric way, which is traditionally accessible only with integral field spectroscopy, opening a new window into large, unbiased, and spatially resolved studies of galaxy evolution.

Griggio, M. [Space Telescope Science Institute, Ba↗

Equation-Free Coarse Control of Distributed Parameter Systems via Local Neural Operators

The control of high-dimensional distributed parameter systems (DPS) remains a challenge when explicit coarse-grained equations are unavailable. Classical equation-free (EF) approaches rely on fine-scale simulators treated as black-box timesteppers. However, repeated simulations for steady-state computation, linearization, and control design are often computationally prohibitive, or the microscopic timestepper may not even be available, leaving us with data as the only resource. We propose a data-driven alternative that uses local neural operators, trained on spatiotemporal microscopic/mesoscopic data, to obtain efficient short-time solution operators. These surrogates are employed within Krylov subspace methods to compute coarse steady and unsteady-states, while also providing Jacobian information in a matrix-free manner. Krylov-Arnoldi iterations then approximate the dominant eigenspectrum, yielding reduced models that capture the open-loop slow dynamics without explicit Jacobian assembly. Both discrete-time Linear Quadratic Regulator (dLQR) and pole-placement (PP) controllers are based on this reduced system and lifted back to the full nonlinear dynamics, thereby closing the feedback loop.

93B52, 93C20, 47N70, 65J15, 65M32, 68T07, 68T20, 6↗

Leveraging machine learning to enhance aerosol classification using Single-Particle Mass Spectrometry

Advancing automated classification of atmospheric aerosols from Single-Particle Mass Spectrometry (SPMS) data remains challenging due to overlapping ion signatures, compositional diversity, and limited labeled data. This study evaluates supervised and semi-supervised learning frameworks to enhance aerosol identification by jointly leveraging labeled and unlabeled spectra. Four models were compared: a supervised Support Vector Machine (SVM), a self-training SVM, a stacked autoencoder classifier, and a stacked autoencoder trained using a temporal-ensembling Mean Teacher approach. All models achieved high and stable accuracies (90.0 %–91.1 %), surpassing previous results on the same dataset (87 %) and matching the performance of state-of-the-art deep learning methods. Despite small global metric differences (≤ 1 %), semi-supervised variants yielded up to 5 %–10 % improvements for compositionally rare particle types – such as soot (0.77 % of spectra, F1-score: 0.93–0.97) and hazelnut pollen (0.98 % of spectra, F1-score: 0.97–1.00) – equating to roughly ∼ 187 additional correctly classified spectra. These gains are scientifically significant, as such rare particles exert disproportionate influence on radiative absorption and ice nucleation processes; their improved detection reduces modeled uncertainties in aerosol absorption optical depth and mixed-phase cloud ice nucleation rates. The models' residual misclassifications (≈ 9 %) largely arise from true spectral overlap among chemically adjacent species (e.g., Na- vs. K-feldspar, coated vs. uncoated feldspars), reflecting physical compositional continuity rather than algorithmic error. Collectively, these findings demonstrate that leveraging unlabeled data to learn robust spectral representations and refine classification enhances both fidelity and interpretability, bridging data-driven analysis with aerosol–climate process understanding.

54 ENVIRONMENTAL SCIENCES↗

Global Methane Budget 2000–2020

Abstract. Understanding and quantifying the global methane (CH4) budget is important for assessing realistic pathways to mitigate climate change. CH4 is the second most important human-influenced greenhouse gas in terms of climate forcing after carbon dioxide (CO2), and both emissions and atmospheric concentrations of CH4 have continued to increase since 2007 after a temporary pause. The relative importance of CH4 emissions compared to those of CO2 for temperature change is related to its shorter atmospheric lifetime, stronger radiative effect, and acceleration in atmospheric growth rate over the past decade, the causes of which are still debated. Two major challenges in quantifying the factors responsible for the observed atmospheric growth rate arise from diverse, geographically overlapping CH4 sources and from the uncertain magnitude and temporal change in the destruction of CH4 by short-lived and highly variable hydroxyl radicals (OH). To address these challenges, we have established a consortium of multidisciplinary scientists under the umbrella of the Global Carbon Project to improve, synthesise, and update the global CH4 budget regularly and to stimulate new research on the methane cycle. Following Saunois et al. (2016, 2020), we present here the third version of the living review paper dedicated to the decadal CH4 budget, integrating results of top-down CH4 emission estimates (based on in situ and Greenhouse Gases Observing SATellite (GOSAT) atmospheric observations and an ensemble of atmospheric inverse-model results) and bottom-up estimates (based on process-based models for estimating land surface emissions and atmospheric chemistry, inventories of anthropogenic emissions, and data-driven extrapolations). We present a budget for the most recent 2010–2019 calendar decade (the latest period for which full data sets are available), for the previous decade of 2000–2009 and for the year 2020. The revision of the bottom-up budget in this 2025 edition benefits from important progress in estimating inland freshwater emissions, with better counting of emissions from lakes and ponds, reservoirs, and streams and rivers. This budget also reduces double counting across freshwater and wetland emissions and, for the first time, includes an estimate of the potential double counting that may exist (average of 23 Tg CH4 yr−1). Bottom-up approaches show that the combined wetland and inland freshwater emissions average 248 [159–369] Tg CH4 yr−1 for the 2010–2019 decade. Natural fluxes are perturbed by human activities through climate, eutrophication, and land use. In this budget, we also estimate, for the first time, this anthropogenic component contributing to wetland and inland freshwater emissions. Newly available gridded products also allowed us to derive an almost complete latitudinal and regional budget based on bottom-up approaches. For the 2010–2019 decade, global CH4 emissions are estimated by atmospheric inversions (top-down) to be 575 Tg CH4 yr−1 (range 553–586, corresponding to the minimum and maximum estimates of the model ensemble). Of this amount, 369 Tg CH4 yr−1 or ∼ 65 % is attributed to direct anthropogenic sources in the fossil, agriculture, and waste and anthropogenic biomass burning (range 350–391 Tg CH4 yr−1 or 63 %–68 %). For the 2000–2009 period, the atmospheric inversions give a slightly lower total emission than for 2010–2019, by 32 Tg CH4 yr−1 (range 9–40). The 2020 emission rate is the highest of the period and reaches 608 Tg CH4 yr−1 (range 581–627), which is 12 % higher than the average emissions in the 2000s. Since 2012, global direct anthropogenic CH4 emission trends have been tracking scenarios that assume no or minimal climate mitigation policies proposed by the Intergovernmental Panel on Climate Change (shared socio-economic pathways SSP5 and SSP3). Bottom-up methods suggest 16 % (94 Tg CH4 yr−1) larger global emissions (669 Tg CH4 yr−1, range 512–849) than top-down inversion methods for the 2010–2019 period. The discrepancy between the bottom-up and the top-down budgets has been greatly reduced compared to the previous differences (167 and 156 Tg CH4 yr−1 in Saunois et al. (2016, 2020) respectively), and for the first time uncertainties in bottom-up and top-down budgets overlap. Although differences have been reduced between inversions and bottom-up, the most important source of uncertainty in the global CH4 budget is still attributable to natural emissions, especially those from wetlands and inland freshwaters. The tropospheric loss of methane, as the main contributor to methane lifetime, has been estimated at 563 [510–663] Tg CH4 yr−1 based on chemistry–climate models. These values are slightly larger than for 2000–2009 due to the impact of the rise in atmospheric methane and remaining large uncertainty (∼ 25 %). The total sink of CH4 is estimated at 633 [507–796] Tg CH4 yr−1 by the bottom-up approaches and at 554 [550–567] Tg CH4 yr−1 by top-down approaches. However, most of the top-down models use the same OH distribution, which introduces less uncertainty to the global budget than is likely justified. For 2010–2019, agriculture and waste contributed an estimated 228 [213–242] Tg CH4 yr−1 in the top-down budget and 211 [195–231] Tg CH4 yr−1 in the bottom-up budget. Fossil fuel emissions contributed 115 [100–124] Tg CH4 yr−1 in the top-down budget and 120 [117–125] Tg CH4 yr−1 in the bottom-up budget. Biomass and biofuel burning contributed 27 [26–27] Tg CH4 yr−1 in the top-down budget and 28 [21–39] Tg CH4 yr−1 in the bottom-up budget. We identify five major priorities for improving the CH4 budget: (i) producing a global, high-resolution map of water-saturated soils and inundated areas emitting CH4 based on a robust classification of different types of emitting ecosystems; (ii) further development of process-based models for inland-water emissions; (iii) intensification of CH4 observations at local (e.g. FLUXNET-CH4 measurements, urban-scale monitoring, satellite imagery with pointing capabilities) to regional scales (surface networks and global remote sensing measurements from satellites) to constrain both bottom-up models and atmospheric inversions; (iv) improvements of transport models and the representation of photochemical sinks in top-down inversions; and (v) integration of 3D variational inversion systems using isotopic and/or co-emitted species such as ethane as well as information in the bottom-up inventories on anthropogenic super-emitters detected by remote sensing (mainly oil and gas sector but also coal, agriculture, and landfills) to improve source partitioning. The data presented here can be downloaded from https://doi.org/10.18160/GKQ9-2RHT (Martinez et al., 2024).

54 ENVIRONMENTAL SCIENCES↗

Extraction and Analysis of Time Series Data from Building Automation Systems Using Large Language Models

Semantic schemas like Haystack 4, Brick and ASHRAE standard 223 enable the structured, standardized, and machine-readable representation of building data, facilitating interoperability, data integration, and advanced analytics. However, extracting information from these models requires specialized expertise in SPARQL and other programming languages, skills that are not commonly found among building professionals. Recent advancements in Large Language Models (LLMs), such as ChatGPT, enable the construction of queries using natural language, making it easier for individuals to interact with these systems in a manner that resembles everyday speech. However, these methods have not yet been tested on building semantic ontologies. This paper introduces a novel workflow and tool for enabling users to ask questions about a specific building's data, using natural language and receive answers automatically generated by GPT-4o. Our approach integrates semantic ontologies with advanced LLM capabilities to automate three critical steps: (1) generating SPARQL queries to retrieve time series references from ontological models, (2) extracting the corresponding time series data from the Building Automation System, and (3) performing computations and visualizations tailored to the user's query. The proposed method simplifies access to BAS data, allowing both domain experts and non-specialists to conduct sophisticated analyses without needing extensive technical knowledge of semantic web technologies. By demonstrating this pipeline, we facilitate more accessible and scalable data-driven decision-making in building operations and management.

Mulayim, Ozan Baris↗

A Probabilistic Approach to Load Modeling for Central HVAC Systems in Large Commercial Buildings for Retrofit Decisions Under Uncertainty

Retrofitting central HVAC systems in large commercial buildings with advanced technologies like heat recovery chillers (HRCs) offers a significant opportunity to enhance energy efficiency. However, analyzing these retrofits is challenging with traditional whole-building simulation tools, which require intensive calibration and struggle to model innovative system configurations and controls. To overcome these limitations, this study proposes a load profilebased retrofit analysis framework that provides better decisions under uncertainty. The main focus of this paper is the development of a probabilistic load profile model that can be used in the framework by using exploratory data analysis (EDA) of measured building data to properly quantify its inherent variability. A non-parametric Gaussian Process (GP) model was employed to capture the time- and weather-dependent characteristics of the heating load while explicitly modeling its uncertainty. The model's effectiveness is demonstrated through strong predictive performance on unseen data and physically interpretable insights into load behavior. This data-driven, probabilistic load profile serves as a robust and flexible input for subsequent system simulations, enabling a more confident and statistically sound analysis of retrofit potential.

Ham, S W↗

SIRTF Science Operations System Design

SIRTF Science Operations System Design William B. Green Manager, SIRTF Science Center California Institute of Technology M/S 310-6 1200 E. California Blvd., Pasadena CA 91125 (626) 395 8572 Fax (626) 568 0673 bgreen@ipac.caltech.edu. The Space Infrared Telescope Facility (SIRTF) will be launched in December 2001, and perform an extended series of science observations at wavelengths ranging from 20 to 160 microns for five years or more. The California Institute of Technology has been selected as the home for the SIRTF Science Center (SSC). The SSC will be responsible for evaluating and selecting observation proposals, providing technical support to the science community, performing mission planning and science observation scheduling activities, instrument calibration during operations and instrument health monitoring, production of archival quality data products, and management of science research grants. The science payload consists of three instruments delivered by instrument Principal Investigators located at University of Arizona, Cornell, and Harvard Smithsonian Astrophysical Observatory. The SSC is responsible for design, development, and operation of the Science Operations System (SOS) which will support the functions assigned to the SSC by NASA. The SIRTF spacecraft, mission profile, and science instrument design have undergone almost ten years of refinement. SIRTF development and operations activities are highly cost constrained. The cost constraints have impacted the design of the SOS in several ways. The Science Operations System has been designed to incorporate a set of efficient, easy to use tools which will make it possible for scientists to propose observation sequences in a rapid and automated manner. The use of highly automated tools for requesting observations will simplify the long range observatory scheduling process, and the short term scheduling of science observations. Pipeline data processing will be highly automated and data-driven, utilizing a variety of tools developed at JPL, the instrument development teams, and Space Telescope Science Institute to automate processing. An incremental ground data system development approach has been adopted, featuring periodic deliveries that are validated with the flight hardware throughout the various phases of system level development and testing. This approach minimizes development time and decreases operations risk. This paper will describe the top level architecture of the SOS and the basic design concepts. A summary of the incremental development approach will be presented. Examples of the unique science user tools now under final development prior to the first proposal call scheduled for mid-2000 will be shown.

Green, William↗

The Orion GN and C Data-Driven Flight Software Architecture for Automated Sequencing and Fault Recovery

The Orion Crew Exploration Vehicle (CET) is being designed to include significantly more automation capability than either the Space Shuttle or the International Space Station (ISS). In particular, the vehicle flight software has requirements to accommodate increasingly automated missions throughout all phases of flight. A data-driven flight software architecture will provide an evolvable automation capability to sequence through Guidance, Navigation & Control (GN&C) flight software modes and configurations while maintaining the required flexibility and human control over the automation. This flexibility is a key aspect needed to address the maturation of operational concepts, to permit ground and crew operators to gain trust in the system and mitigate unpredictability in human spaceflight. To allow for mission flexibility and reconfrgurability, a data driven approach is being taken to load the mission event plan as well cis the flight software artifacts associated with the GN&C subsystem. A database of GN&C level sequencing data is presented which manages and tracks the mission specific and algorithm parameters to provide a capability to schedule GN&C events within mission segments. The flight software data schema for performing automated mission sequencing is presented with a concept of operations for interactions with ground and onboard crew members. A prototype architecture for fault identification, isolation and recovery interactions with the automation software is presented and discussed as a forward work item.

King, Ellis↗

Algorithms for Spectral Decomposition with Applications to Optical Plume Anomaly Detection

The analysis of spectral signals for features that represent physical phenomenon is ubiquitous in the science and engineering communities. There are two main approaches that can be taken to extract relevant features from these high-dimensional data streams. The first set of approaches relies on extracting features using a physics-based paradigm where the underlying physical mechanism that generates the spectra is used to infer the most important features in the data stream. We focus on a complementary methodology that uses a data-driven technique that is informed by the underlying physics but also has the ability to adapt to unmodeled system attributes and dynamics. We discuss the following four algorithms: Spectral Decomposition Algorithm (SDA), Non-Negative Matrix Factorization (NMF), Independent Component Analysis (ICA) and Principal Components Analysis (PCA) and compare their performance on a spectral emulator which we use to generate artificial data with known statistical properties. This spectral emulator mimics the real-world phenomena arising from the plume of the space shuttle main engine and can be used to validate the results that arise from various spectral decomposition algorithms and is very useful for situations where real-world systems have very low probabilities of fault or failure. Our results indicate that methods like SDA and NMF provide a straightforward way of incorporating prior physical knowledge while NMF with a tuning mechanism can give superior performance on some tests. We demonstrate these algorithms to detect potential system-health issues on data from a spectral emulator with tunable health parameters.

Srivastava, Askok N.↗

Global CO2 Distributions over Land from the Greenhouse Gases Observing Satellite (GOSAT)

January 2009 saw the successful launch of the first space-based mission specifically designed for measuring greenhouse gases, the Japanese Greenhouse gases Observing SATellite (GOSAT). We present global land maps (Level 3 data) of column-averaged CO2 concentrations (X(sub CO2)) derived using observations from the GOSAT ACOS retrieval algorithm, for July through December 2009. The applied geostatistical mapping approach makes it possible to generate maps at high spatial and temporal resolutions that include uncertainty measures and that are derived directly from the Level 2 observations, without invoking an atmospheric transport model or estimates of CO2 uptake and emissions. As such, they are particularly well suited for comparison studies. Results show that the Level 3 maps for July to December 2009 on a lO x 1.250 grid, at six-day resolution capture much of the synoptic scale and regional variability of X(sub CO2), in addition to its overall seasonality. The uncertainty estimates, which reflect local data coverage, X(sub CO2) variability, and retrieval errors, indicate that the Southern latitudes are relatively well-constrained, while the Sahara Desert and the high Northern latitudes are weakly-constrained. A probabilistic comparison to the PCTM/GEOS-5/CASA-GFED model reveals that the most statistically significant discrepancies occur in South America in July and August, and central Asia in September to December. While still preliminary, these results illustrate the usefulness of a high spatiotemporal resolution, data-driven Level 3 data product for direct interpretation and comparison of satellite observations of highly dynamic parameters such as atmospheric CO2.

Hammerling, Dorit M.↗

Semi-Supervised Eigenbasis Novelty Detection

Recent discoveries in high-time-resolution radio astronomy data have focused attention on a new class of events. Fast transients are rare pulses of radio frequency energy lasting from microseconds to seconds that might be produced by a variety of exotic astrophysical phenomena. For example, X-ray bursts, neutron stars, and active galactic nuclei are all possible sources of short-duration, transient radio signals. It is difficult to anticipate where such signals might appear, and they are most commonly discovered through analysis of high-time- resolution data that had been collected for other purposes. Transients are often faint and difficult to detect, so improved detection algorithms can directly benefit the science yield of all such commensal monitoring. A new detection algorithm learns a low-dimensional linear manifold for describing the normal data. High reconstruction error indicates a novel signal that does not match the patterns of normal data. One unsupervised portion of the manifold model adapts its representation in response to recent data. A second supervised portion of the model is made of a basis trained in advance using labeled examples of RFI; this prevents false positives due to these events. For a linear model, an orthonormalization operation is used to combine these bases prior to the anomaly detection decision. Another novel aspect of the approach lies in combining basis vectors learned in an unsupervised, online fashion from the data stream with supervised basis vectors learned in advance from known examples of false alarms. Adaptive, data-driven detection is achieved that is also informed by existing domain knowledge about signals that may be statistically anomalous, but are not interesting and should therefore be ignored. The method was evaluated using data from the Parkes Multibeam Survey. This data set was originally collected to search for pulsars, which are astronomical sources that emit radio pulses at regular periods. However, several non-pulsar anomalies have recently been discovered in this dataset, making it a compelling test case. By explicitly filtering known false alarm patterns, the approach yields significantly better performance than current transient detection methods.

Wagstaff, Kiri L.↗

Active Learning with Rationales for Identifying Operationally Significant Anomalies in Aviation

A major focus of the commercial aviation community is discovery of unknown safety events in flight operations data. Data-driven unsupervised anomaly detection methods are better at capturing unknown safety events compared to rule-based methods which only look for known violations. However, not all statistical anomalies that are discovered by these unsupervised anomaly detection methods are operationally significant (e.g., represent a safety concern). Subject Matter Experts (SMEs) have to spend significant time reviewing these statistical anomalies individually to identify a few operationally significant ones. In this paper we propose an active learning algorithm that incorporates SME feedback in the form of rationales to build a classifier that can distinguish between uninteresting and operationally significant anomalies. Experimental evaluation on real aviation data shows that our approach improves detection of operationally significant events by as much as 75% compared to the state-of-the-art. The learnt classifier also generalizes well to additional validation data sets.

anomaly detection↗

Data-Driven State of Health Estimation for Second-Life Batteries Using Interpolated Synthetic Data and Feature Selection

Accurate estimation of the State of Health (SOH) for second-life batteries (SLBs) is crucial given their increasing use in energy storage applications. Precise SOH prediction is essential for safe operation and robust battery management systems. A major challenge is the limited availability of datasets for building reliable degradation models. To address this, synthetic data generation through linear interpolation is performed to extend the available data, making it more representative of real-world battery operating conditions. By analyzing feature correlation with SOH, the most relevant features are selected for the model. The proposed approach employs a convolutional neural network (CNN) model trained on this interpolated, feature-selected dataset, using time series data of voltage, temperature, and current over a cycle. By focusing on highly correlated features, the model achieves over 95% accuracy, with mean absolute error and root mean squared error up to 2.27% and 2.64%, respectively, in SOH estimation for two battery datasets tested. These results highlight the potential of combining synthetic data generation and feature selection to enhance SOH predictions, showcasing the superior performance of the proposed CNN model for both new batteries and SLBs.

feature selection↗

Data about data – when, why and how metadata can support the digital plant

A structured approach for recording data quality and contextual information about how and why a signal exists – i.e. metadata – is central to interpret and use sensor data correctly. This is becoming increasingly important with the global trend with data-driven applications such as digital twins and AI-models. But a structured metadata collection and organization of sensor data is not routine in most plants, which can result in lost information and missed opportunities to make use of the investments made in the data collection. Therefore, the IWA task group on Metadata Collection and Organization in wastewater resource recovery systems (MetaCO) was initiated in 2020 and recently delivered the IWA scientific and technical report number 31. The report gives and in-depth description about metadata in water resources recovery facilities (WRRFs) and is available as open access at IWA publishing. The report is the outcome of the collaboration between more than 80 water professionals with the intention to serve WRRF data users with a guide on how to structure and make use of metadata throughout the data pipeline in order to maximize the value of sensor data.

Alferes, Janelcy [VITO, Belgium]↗