Search NASA⌕ Search

SEARCH · Search NASA

Results for “correlation mining”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 37 records · Page 2
All sourcesNASA NTRSDOE OSTI
DOE OSTI2018

A Big Data Analytics Framework for HPC Log Data: Three Case Studies Using the Titan Supercomputer Log

Reliability, availability and serviceability (RAS) logs of high performance computing (HPC) resources, when closely investigated in spatial and temporal dimensions, can provide invaluable information regarding system status, performance, and resource utilization. These data are often generated from multiple logging systems and sensors that cover many components of the system. The analysis of these data for finding persistent temporal and spatial insights faces two main difficulties: the volume of RAS logs makes manual inspection difficult and the unstructured nature and unique properties of log data produced by each subsystem adds another dimension of difficulty in identifying implicit correlation among recorded events. To address these issues, we recently developed a multi-user Big Data analytics framework for HPC log data at Oak Ridge National Laboratory (ORNL). This paper introduces three in-progress data analytics projects that leverage this framework to assess system status, mine event patterns, and study correlations between user applications and system events. We describe the motivation of each project and detail their workflows using three years of log data collected from ORNL's Titan supercomputer.

Park, Byung↗
DOE OSTI2018

Variation in microbial community structure correlates with heavy-metal contamination in soils decades after mining ceased

Microorganisms play vital roles in Earth's biogeochemical cycles. Identifying disturbances in microbial communities due to anthropogenic contamination can provide insights into the health of ecosystems. Picher, Oklahoma, was the site of large-scale mining operations for Pb, Zn, and other heavy metals until the mid-1950s, operating within the Tri-State Mining District (TSMD) of Missouri, Kansas and Oklahoma. Although mining ceased decades ago, high concentrations of heavy metals (>1000 ppm) remain in area soil and water systems. Previously, we mapped metal concentrations on samples collected from mine tailings in Picher and along cardinal-direction transects within an 8.05-km radius of the town. To elucidate changes in microbial community structure due to regional metal contamination, 16S rRNA gene sequences and qPCR calculations of total Bacteria and Archaea were analyzed against these metal concentrations. Bacteria were negatively and significantly correlated with Pb, Cd, Zn, and Mg; however, Archaea was only significantly and positively correlated with pH. Illumina sequencing of 16S rRNA genes showed significant differences in microbial communities of chat and west transect samples. Comparison of soil chemistry with community structure indicated that Al, Pb, Cd, and Zn significantly impacted community composition and distribution of individual OTUs. Mapping the distribution of heavy-metal contamination and microbial communities in these soils represents the first step in understanding effects of long-term, heavy-metal contamination at a basic trophic level.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2023

An Investigation on the Pollen-Induced Soiling Losses in Utility-Scale PV Plants

Soiling, the accumulation of dust and other contaminants on the surface of photovoltaic (PV) modules, is a common factor that can negatively impact the performance of PV systems. In this study, the authors aim to analyze the impact of pollen on soiling losses in PV systems located in North Carolina, USA, particularly during the spring season. The performance data of two utility-scale PV plants was collected and analyzed using the two soiling extraction methods. Environmental data, including croplands and vegetation was also collected and analyzed to identify correlations with soiling losses. The results of the study may help improve understanding of necessary operation and maintenance activities for PV plants and provide new insights into the phenomenon of pollen deposition on PV systems.

correlation↗
DOE OSTI2018

Analysis of seismicity at observed at station BCE using subspace detectors

A dynamic correlation processor (DCP) was used to process 90 days of 3-component seismic data recorded by station BCE. Two methods of detector creation were compared. The first attempted to span the signal space with a small number of subspace detectors having minimal projections on one another, and the second used a much larger number of detectors with substantial inter-template projections. After two passes over the data the results were compared, and the second approach was found to have produced more than 3 times the number of detections. For events in common, the average correlation statistic for the second approach was 0.77 compared to a statistic of 0.65 for the first approach. Also, analysis of the detection statistics as a function of SNR showed a strong functional relationship between SNR and correlation statistic for the second approach not found in the first approach. I interpret this to mean that the intra-group variability of signals produced by the second method is due primarily to noise rather than variations in source or path. These results show that (in an environment with large numbers of repeaters) if precise classification is a goal, it is better to operate a DCP with many correlators with substantial inter-template projections and that fully span the signal space. The performance of the DCP was also measured using a small set of “ground truth” picks for BCE from the Chip Brogan catalog. The catalog spanned less than three days out of the 90 days of the experiment and is also incomplete, but the comparison identified 12 signals missed by the DCP because of low SNR. Combining detections by the DCP and picks from the Brogan catalog, identifies 76 unique signals. Of these, the DCP detected 64 (84%) and the Catalog contained between 48/76 (63%) and 48/62 (77%). Analysis of time-of-day and day-of-week statistics was not useful in identifying the signal sources. However, magnitude recurrence statistics were consistent with mining induced seismicity (a preponderance of small sources). Failure of the temporal discriminants may indicate that the events being detected are not coupled in the same way to the mining activity, e.g. secondary failures. Correlation data from explosions at the Nevada test site (NTS) were used to bound the separation of events within groups. Nearly all the NTS correlations with the same passband and time bandwidth product as the detections and having a correlation statistic >= 0.8 are from sources separated by less than 1 km. Even at correlation values of 0.6 nearly all the source separations are less than about 3 km. Because the DCP detectors were formed to span the signal space at a correlation value of 0.9 and because the intra-group variability is most likely due to noise and not source variability, it is likely that the sources associated with each detector are separated by one km or less, and almost certain that the majority are separated by no more than 3 km.

47 OTHER INSTRUMENTATION↗
DOE OSTI2025

STILGAR End-of-Project Report

The Subsurface Tunnel Imaging LeveraGed by Analysis of Rayleigh wave ellipticity (STILGAR) project demonstrated an integrated geophysical approach for detecting, locating, and characterizing underground structural changes using dense seismic arrays and advanced inversion techniques. Field campaigns were conducted at two operational mines—the Redmond salt mine (Utah) and Graymont Pleasant Gap limestone mine (Pennsylvania)—providing real-world testbeds for monitoring anthropogenic subsurface activity. At the Redmond salt mine, seismic interferometry combined with back-projection inversion successfully identified continuous, low-amplitude signals from mining operations. The approach differentiated stationary from migrating anthropogenic sources, captured daily operational cycles, and validated the potential of passive seismic monitoring for remote detection of underground activity. At the Graymont Pleasant Gap mine, two dense seismic deployments in the spring and fall of 2023 generated over 4 TB of high-resolution data. Key outcomes included the relocation of 199 underground and 8 surface explosions with accuracies within tens of meters and the development of a 3D P-wave velocity model using the triple-difference tomography algorithm (tomoTD) that resolved major structural features such as the mine entrance, low-velocity tunnels, and roof-collapse areas. Ambient noise cross-correlation and back-projection analyses revealed persistent sources linked to ongoing mining activity, whereas horizontal-to-vertical spectral ratio (HVSR) and ellipticity studies confirmed stable site responses across seasons and identified soil thickness trends consistent with regional erosional and depositional processes. Checkerboard and sensitivity tests further validated the robustness of the tomographic results. Overall, the findings emphasize that although significant progress has been made in subsurface imaging, further work is needed to enhance the detection and localization of underground structures. Accurate imaging requires higher frequencies, yet anthropogenic sources tend to dominate the seismic record at those frequencies, and high-frequency surface waves are affected by higher modes that complicate interpretation. The improved detection and localization of human-induced signals enabled detailed temporal and spatial mapping of daily mine operations, demonstrating the feasibility of continuous anthropogenic source monitoring. Sensitivity to signals from nontraditional sources, such as fan operations, highlights the broader applicability of this approach to other industrial environments in which continuous and impulsive signals are present. The field campaigns produced a substantial volume of high-quality seismic data, supporting the development and testing of new methods for seismic source characterization and subsurface imaging. Future deployments should include sensors capable of recording lower frequencies to probe deeper structures, increase bandwidth to enhance resolution and sensitivity to both shallow and deep targets, and collect additional large-scale datasets to refine imaging and source characterization techniques. Moreover, conducting 3D modeling studies of seismic wavefields at higher frequencies will provide a better understanding of wave scattering and cavity–wavefield interactions in complex underground environments. In conclusion, the STILGAR project demonstrated that integrated seismic monitoring can effectively characterize underground operations, capturing both natural and anthropogenic signals. The approaches developed provide a foundation for improved detection, localization, and imaging of subsurface structures and are directly transferable to broader industrial monitoring applications.

58 GEOSCIENCES↗
DOE OSTI2018

In Situ Monitoring of Groundwater Contamination Using the Kalman Filter

This study presents a Kalman filter-based framework to establish a real-time in situ monitoring system for groundwater contamination based on in situ measurable water quality variables, such as specific conductance (SC) and pH. First, this framework uses principal component analysis (PCA) to identify correlations between the contaminant concentrations of interest and in situ measurable variables. It then applies the Kalman filter to estimate contaminant concentrations continuously and in real-time by coupling data-driven concentration-decay models with the previously identified data correlations. We demonstrate our approach with historical groundwater data from the Savannah River Site F-Area: We use SC and pH data to estimate tritium and uranium concentrations over time. Results show that the developed method can estimate these contaminant concentrations based on in situ measurable variables. The estimates remain reliable with less frequent or no direct measurements of the contaminant concentrations, while capturing the dynamics of short- and long-term contaminant concentration changes. In addition, we show that data mining, such as PCA, is useful to understand correlations in groundwater data and to design long-term monitoring systems. As a result, the developed in situ monitoring methodology is expected to improve long-term groundwater monitoring by continuously confirming the contaminant plume's stability and by providing an early warning system for unexpected changes in the plume's migration.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2023

Vitamin interdependencies predicted by metagenomics-informed network analyses and validated in microbial community microcosms

Abstract Metagenomic or metabarcoding data are often used to predict microbial interactions in complex communities, but these predictions are rarely explored experimentally. Here, we use an organism abundance correlation network to investigate factors that control community organization in mine tailings-derived laboratory microbial consortia grown under dozens of conditions. The network is overlaid with metagenomic information about functional capacities to generate testable hypotheses. We develop a metric to predict the importance of each node within its local network environments relative to correlated vitamin auxotrophs, and predict that a Variovorax species is a hub as an important source of thiamine. Quantification of thiamine during the growth of Variovorax in minimal media show high levels of thiamine production, up to 100 mg/L. A few of the correlated thiamine auxotrophs are predicted to produce pantothenate, which we show is required for growth of Variovorax , supporting that a subset of vitamin-dependent interactions are mutualistic. A Cryptococcus yeast produces the B-vitamin pantothenate, and co-culturing with Variovorax leads to a 90-130-fold fitness increase for both organisms. Our study demonstrates the predictive power of metagenome-informed, microbial consortia-based network analyses for identifying microbial interactions that underpin the structure and functioning of microbial communities.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2025

Machine Learning for Well Log Analysis in Uranium Mining

This project explores the use of Artificial Intelligence (AI) and Machine Learning (ML) techniques to automate well log analysis for uranium mining. Geophysical log data—spontaneous potential, resistivity, and gamma ray—were used to classify lithology, correlate well logs and identify roll front zonation patterns, which are critical for locating uranium ore bodies. Supervised ML algorithms such as eXtreme Gradient Boosting (XGBoost), Categorical Boosting (CatBoost), and Random Forest were trained to classify lithology with high accuracy. Gradient Boosting Machines (GBM), XGBoost, Random Forest, and Neural Networks were also used for role front zone identification. Moreover, a Fast Dynamic Time Warping (FastDTW) algorithm was employed for well log correlation. Additionally, sample lag was addressed using dynamic programming. Results demonstrate the potential of AI and ML to streamline well log analysis and enhance uranium exploration workflows.

11 - NUCLEAR FUEL CYCLE AND FUEL MATERIALS↗
DOE OSTI2018

Incremental nonnegative matrix factorization based on correlation and graph regularization for matrix completion

Matrix factorization is widely used in recommendation systems, text mining, face recognition and computer vision. As one of the most popular methods, nonnegative matrix factorization and its incremental variants have attracted much attention. The existing incremental algorithms are established based on the assumption of samples are independent and only update the new latent variable of weighting coefficient matrix when the new sample comes, which may lead to inferior solutions. To address this issue, we investigate a novel incremental nonnegative matrix factorization algorithm based on correlation and graph regularizer (ICGNMF). The correlation is mainly used for finding out those correlated rows to be updated, that is, we assume that samples are dependent on each other. We derive the updating rules for ICGNMF by considering the correlation. We also present tests on widely used image datasets, and show ICGNMF reduces the error by comparing other methods.

Zhang, Xiaoxia↗
DOE OSTI2017

XRootD popularity on hadoop clusters

Performance data and metadata of the computing operations at the CMS experiment are collected through a distributed monitoring infrastructure, currently relying on a traditional Oracle database system. This paper shows how to harness Big Data architectures in order to improve the throughput and the efficiency of such monitoring. A large set of operational data - user activities, job submissions, resources, file transfers, site efficiencies, software releases, network traffic, machine logs - is being injected into a readily available Hadoop cluster, via several data streamers. The collected metadata is further organized running fast arbitrary queries; this offers the ability to test several Map&Reduce-based frameworks and measure the system speed-up when compared to the original database infrastructure. By leveraging a quality Hadoop data store and enabling an analytics framework on top, it is possible to design a mining platform to predict dataset popularity and discover patterns and correlations.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2016

Fault Tolerant Frequent Pattern Mining

FP-Growth algorithm is a Frequent Pattern Mining (FPM) algorithm that has been extensively used to study correlations and patterns in large scale datasets. While several researchers have designed distributed memory FP-Growth algorithms, it is pivotal to consider fault tolerant FP-Growth, which can address the increasing fault rates in large scale systems. In this work, we propose a novel parallel, algorithm-level fault-tolerant FP-Growth algorithm. We leverage algorithmic properties and MPI advanced features to guarantee an O(1) space complexity, achieved by using the dataset memory space itself for checkpointing. We also propose a recovery algorithm that can use in-memory and disk-based checkpointing, though in many cases the recovery can be completed without any disk access, and incurring no memory overhead for checkpointing. We evaluate our FT algorithm on a large scale InfiniBand cluster with several large datasets using up to 2K cores. Our evaluation demonstrates excellent efficiency for checkpointing and recovery in comparison to the disk-based approach. We have also observed 20x average speed-up in comparison to Spark, establishing that a well designed algorithm can easily outperform a solution based on a general fault-tolerant programming model.

Shohdy, Sameh↗
DOE OSTI2024

Critical Assessment of Electronic Structure Descriptors for Predicting Perovskite Catalytic Properties

The discovery and design of materials which can efficiently catalyze the oxygen reduction and evolution reactions at reduced temperatures is important for facilitating the widespread adoption of fuel cell and electrolyzer technologies. Numerous studies have produced correlations between catalytic properties, such as oxygen surface exchange or electrode area specific resistance (ASR), and properties of the catalyst material. However, correlations have historically been limited in scope (e.g., using only a few materials or at a single temperature) and it has been difficult to provide detailed assessments of their robustness. Here, in this study, we assess the ability of the O p-band center electronic structure descriptor, obtained from density functional theory (DFT) calculations, to correlate with oxygen surface exchange rates, diffusivities, and area specific resistances for a large database of perovskite oxide catalytic properties. By data mining the literature, we obtain 747 catalytic property value data points spanning 299 unique perovskite compositions from 313 studies. We assess linear correlations of each property with the O p-band center and find generally modest correlations that are qualitatively useful (prediction mean absolute errors of about 0.5 log units are typical), where the correlations are improved at higher temperatures (e.g., 800 °C vs. 500 °C) and significantly improve when considering fits to the subset of materials which have multiple independent measurements. These findings suggest that the spread of property data is significantly influenced by experimental uncertainty, and subsequent measurements of additional materials will likely improve the O p-band center correlations.

30 DIRECT ENERGY CONVERSION↗
DOE OSTI2017

Coupling Big Data Analytics and Reactive Transport Modeling for Cost-effective Groundwater Monitoring - 17163

This study presents an innovative approach for sustainable and cost-effective groundwater monitoring. This approach takes advantage of recent advances in various technologies: (1) in situ autonomous sensors, (2) big data analytics, and (3) parallel high-performance computing for flow and reactive transport modeling. In situ sensors are used to periodically measure the key variables (such as pH, redox potential, electrical conductivity, and groundwater level), which control contaminant mobility and the plume spatial and temporal distribution. Based on a limited number of groundwater sampling, the data analytics methods-data mining and machine learning-allow us to identify and quantify the correlations between the in situ-measured variables and contaminant concentrations, and also to detect significant changes associated with the plume mobility. In addition, a state-of-the-art parallel numerical flow and reactive transport simulator Amanzi, and uncertainty quantification software Agni are used to provide an improved physical and mechanistic understanding of the contaminated groundwater system behavior, and to predict the long-term plume distribution for optimizing and adapting the monitoring strategy. Amanzi and Agni were developed as part of the Advanced Simulation Capability for Environmental Management (ASCEM) program of the DOE Office of Environmental Management. Such modeling is critical, particularly, for assessing the impact of climate change and associated hydrological shifts. The developed approach is expected to significantly reduce the groundwater sampling frequency and associated cost. In addition, the real-time information on plume mobility serves as an early warning system, improving the resiliency of contaminated or potentially contaminated sites. We demonstrate this approach using as an example the Savannah River Site (SRS) F-Area, where groundwater is contaminated by various radionuclides, including uranium, tritium and technetium. (authors)

12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗
DOE OSTI2016

Insights into twinning in $\mathrm{Mg AZ31: A}$ combined $\mathrm{EBSD}$ and machine learning study

To explore the driving forces behind deformation twinning in Mg AZ31, a machine learning framework is utilized to mine data obtained from electron backscatter diffraction (EBSD) scans in order to extract correlations in physical characteristics that cause twinning. The results are intended to inform physics-based models of twin nucleation and growth. A decision tree learning environment is selected to capture the relationships between microstructure and twin formation; this type of model effectively highlights the more influential characteristics of the local microstructure. Trees are assembled to analyze both twin nucleation in a given grain, and twin propagation across grain boundaries. Each model reveals a unique combination of crystallographic attributes that affect twinning in the Mg. Twin nucleation is found to be mostly controlled by a combination of grain size, basal Schmid factor, and bulk dislocation density while twin propagation is affected most by grain boundary length, basal Schmid factor, angle from grain boundary plane to the RD plane, and grain boundary misorientation. The machine learning framework can be readily adapted to investigate other relationships between microstructure and material response.

36 MATERIALS SCIENCE↗
DOE OSTI2017

Metagenomic discovery of polybrominated diphenyl ether biosynthesis by marine sponges

Naturally produced polybrominated diphenyl ethers (PBDEs) pervade the marine environment and structurally resemble toxic man-made brominated flame retardants. PBDEs bioaccumulate in marine animals and are likely transferred to the human food chain. However, the biogenic basis for PBDE production in one of their most prolific sources, marine sponges of the order Dysideidae, remains unidentified. Here, we report the discovery of PBDE biosynthetic gene clusters within sponge microbiome-associated cyanobacterial endosymbionts by employing an unbiased metagenome mining approach. By expression of PBDE biosynthetic genes in heterologous cyanobacterial hosts, we correlate the structural diversity of naturally produced PBDEs to modifications within PBDE biosynthetic gene clusters in multiple sponge holobionts. Our results establish the genetic and molecular foundation for the production of PBDEs in one of the most abundant natural sources of these molecules, further setting the stage for a metagenomic-based inventory of other PBDE sources in the marine environment.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2020

The geometry of signal space: a case study of direct mapping between seismic signals and event distribution

SUMMARY Under favourable circumstances, seismic waveforms corresponding to an ensemble of events related by a common, spatially distributed process collectively exhibit a regular, signal-space geometry. When events in the ensemble have a common, or nearly common, source mechanism, this geometry is a distorted image of the distribution of events in the source region. The signal-space image can be visualized using a relatively simple waveform alignment and projection operation. Ensemble waveform correlation measurements can be inverted to estimate the distribution of the events in the source region, up to an arbitrary rotation, reflection and scaling, with residual distortion. We demonstrate these concepts with synthetic waveforms and with observations of long-wall mining induced seismicity for which substantial ground truth information is available. Our experience with these data has implications for location, correlation detection and machine learning and possible application to studies of repeating events in induced, volcanic and glacial seismicity. Our results place limits on the widely held assumption that waveform correlation is a useful measure of event separation. We suggest that the constraints on event separation need to be evaluated in the context of a population of related events, whose waveforms sample the signal space image of the source region. A better indicator of event separation is the length of the shortest path in signal space along the image.

58 GEOSCIENCES↗
DOE OSTI2021

LL18-Signal Analysis-NDD2Ad (Final Report)

Under favorable circumstances, seismic waveforms corresponding to an ensemble of events related by a common, spatially-distributed process collectively exhibit a regular, signal-space geometry. When events in the ensemble have a common, or nearly common, source mechanism, this geometry is a distorted image of the distribution of events in the source region. The signal space image can be visualized using a relatively simple waveform alignment and projection operation. Ensemble waveform correlation measurements can be inverted to estimate the distribution of the events in the source region, up to an arbitrary rotation, reflection, and scaling with residual distortion. We demonstrate these concepts with synthetic waveforms and with observations of long-wall mining induced seismicity for which substantial ground truth information is available. Our experience with these data has implications for location, correlation detection and machine learning, and possible application to studies of repeating events in induced, volcanic, and glacial seismicity. Our results place limits on the widely-held assumption that waveform correlation is a useful measure of event separation. We suggest that the constraints on event separation need to be evaluated in the context of a population of related events, whose waveforms sample the signal space image of the source region. A better indicator of event separation is the length of the shortest path in signal space along the image.

58 GEOSCIENCES↗
DOE OSTI2016

Isotopic evidence for reductive immobilization of uranium across a roll-front mineral deposit

We use uranium (U) isotope ratios to detect and quantify the extent of natural U reduction in groundwater across a roll front redox gradient. Our study was conducted at the Smith Ranch-Highland in situ recovery (ISR) U mine in eastern Wyoming, USA, where economic U deposits occur in the Paleocene Fort Union formation. To evaluate the fate of aqueous U in and adjacent to the ore body, we investigated the chemical composition and isotope ratios of groundwater samples from the roll-front type ore body and surrounding monitoring wells of a previously mined area. The 238 U/ 235 U of groundwater varies by approximately 3‰ and is correlated with U concentrations. Fluid samples down-gradient of the ore zone are the most depleted in 238 U and have the lowest U concentrations. Activity ratios of 234 U/ 238 U are ~5.5 up-gradient of the ore zone, ~1.0 in the ore zone, and between 2.3 and 3.7 in the down-gradient monitoring wells. High-precision measurements of 234 U/ 238 U and 238 U/ 235 U allow for development of a conceptual model that evaluates both the migration of U from the ore body and the extent of natural attenuation due to reduction. We find that the premining migration of U down-gradient of the delineated ore body is minimal along eight transects due to reduction in or adjacent to the ore body, whereas two other transects show little or no sign of reduction in the down-gradient region. Lastly, these results suggest that characterization of U isotopic ratios at the mine planning stage, in conjunction with routine geochemical analyses, can be used to identify where more or less postmining remediation will be necessary.

54 ENVIRONMENTAL SCIENCES↗