Search NASA⌕ Search

SEARCH · Search NASA

Results for “Data Mining”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 397 records · Page 22

Defining a Metric Space of Host Logs and Operational Use Cases

Host logs, in particular, Windows Event Logs, are a valuable source of information often collected by security operation centers (SOCs). The semi-structured nature of host logs inhibits automated analytics, and while manual analysis is common, the sheer volume makes manual inspection of all logs impossible. Although many powerful algorithms for analyzing time-series and sequential data exist, utilization of such algorithms for most cyber security applications is either infeasible or requires tailored, research-intensive preparations. In particular, basic mathematic and algorithmic developments for providing a generalized, meaningful similarity metric on system logs is needed to bridge the gap between many existing sequential data mining methods and this currently available but under-utilized data source. In this paper, we provide a rigorous definition of a metric product space on Windows Event Logs, providing an embedding that allows for the application of established machine learning and time-series analysis methods. We then demonstrate the utility and flexibility of this embedding with multiple use-cases on real data: (1) comparing known infected to new host log streams for attack detection and forensics, (2) collapsing similar streams of logs into semantically-meaningful groups (by user, by role), thereby reducing the quantity of data but not the content, (3) clustering logs as well as short sequences of logs to identify and visualize user behaviors and background processes over time. Overall, we provide a metric space framework for general host logs and log sequences that respects semantic similarity and facilitates a wide variety of data science analytics to these logs without data-specific preparations for each.

Verma, Miki↗

Mr. Scan: extreme scale density-based clustering using a tree-based network of GPGPU nodes, In: SC '13 Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis

Density-based clustering algorithms are a widely-used class of data mining techniques that can find irregularly shaped clusters and cluster data without prior knowledge of the number of clusters it contains. DBSCAN is the most well-known density-based clustering algorithm. We introduce our version of DBSCAN, called Mr. Scan, which uses a hybrid parallel implementation that combines the MRNet tree-based distribution network with GPGPU-equipped nodes. Mr. Scan avoids the problems of existing implementations by effectively partitioning the point space and by optimizing DBSCAN's computation over dense data regions. We tested Mr. Scan on both a geolocated Twitter dataset and image data obtained from the Sloan Digital Sky Survey. At its largest scale, Mr. Scan clustered 6.5 billion points from the Twitter dataset on 8,192 GPU nodes on Cray Titan in 17.3 minutes. All other parallel DBSCAN implementations have only demonstrated the ability to cluster up to 100 million points.

Computer Science↗

Pansophy enhancements for SRF through collecting and analyzing inputs/outputs to further project efficiency in reporting and performance

SRF cavity and Cryomodule testing and production requires a consistent means of collecting and analyzing data against quality and production parameters. JLab's engineering data management system, Pansophy, is utilized to assist project leaders and subject matter experts (SMEs) with such tasks, by providing a means to data mine key parameter indicators (KPI) and production planning and status data. Recent enhancements to reporting and trending have been utilized for the LCLS-II and CEBAF 12GeV upgrade projects. Further enhancements are being planned for future projects, like SNS-PPU, such as KPI trending, KPI quality, vendor quality, production timelines and user defined queries. Being able to understand past trends will assist with enhancements to future projects.

Bookwalter, Valerie D.↗

Automated Wildfire Detection Through Artificial Neural Networks

We have tested and deployed Artificial Neural Network (ANN) data mining techniques to analyze remotely sensed multi-channel imaging data from MODIS, GOES, and AVHRR. The goal is to train the ANN to learn the signatures of wildfires in remotely sensed data in order to automate the detection process. We train the ANN using the set of human-detected wildfires in the U.S., which are provided by the Hazard Mapping System (HMS) wildfire detection group at NOAA/NESDIS. The ANN is trained to mimic the behavior of fire detection algorithms and the subjective decision- making by N O M HMS Fire Analysts. We use a local extremum search in order to isolate fire pixels, and then we extract a 7x7 pixel array around that location in 3 spectral channels. The corresponding 147 pixel values are used to populate a 147-dimensional input vector that is fed into the ANN. The ANN accuracy is tested and overfitting is avoided by using a subset of the training data that is set aside as a test data set. We have achieved an automated fire detection accuracy of 80-92%, depending on a variety of ANN parameters and for different instrument channels among the 3 satellites. We believe that this system can be deployed worldwide or for any region to detect wildfires automatically in satellite imagery of those regions. These detections can ultimately be used to provide thermal inputs to climate models.

Miller, Jerry↗

Goddard Atmospheric Composition Data Center: Aura Data and Services in One Place

The Goddard Atmospheric Composition Data and Information Services Center (AC-DISC) is a portal to the Atmospheric Composition specific, user driven, multi-sensor, on-line, easy access archive and distribution system employing data analysis and visualization, data mining, and other user requested techniques for the better science data usage. It provides convenient access to Atmospheric Composition data and information from various remote-sensing missions, from TOMS, UARS, MODIS, and AIRS, to the most recent data from Aura OMI, MLS, HIRDLS (once these datasets are released to the public), as well as Atmospheric Composition datasets residing at other remote archive site.

Leptoukh, G.↗

Plots, Calculations and Graphics Tools (PCG2). Software Transfer Request Presentation

This slide presentation reviews the development of the Plots, Calculations and Graphics Tools (PCG2) system. PCG2 is an easy to use tool that provides a single user interface to view data in a pictorial, tabular or graphical format. It allows the user to view the same display and data in the Control Room, engineering office area, or remote sites. PCG2 supports extensive and regular engineering needs that are both planned and unplanned and it supports the ability to compare, contrast and perform ad hoc data mining over the entire domain of a program's test data.

Richardson, Marilou R.↗

Experiences with string matching on the Fermi Architecture

String matching is at the core of many real-world applications, such as security, bioinformatic, data mining. All these applications requires the ability to match always growing data sets against large dictionaries effectively, fastly and possibly in real time. Unfortunately, string matching is a computationally intensive procedure which poses significant challenges on current software and hardware implementations. Graphic Processing Units (GPU) have become an interesting target for such high-throughput applications, but the algorithms and the data structures need to be redesigned to be parallelized and adapted to the underlining hardware, coping with the limitations imposed by these architectures. In this paper we present an efficient implementation of the Aho-Corasick string matching algorithm on GPU, showing how we progressively redesigned the algorithm and the data structures to fit on the architecture. We then evaluate the implementation on single and multiple Tesla C2050 (T20 ``Fermi'' based) boards, comparing them to the previous Tesla C1060 (T10 based) solutions and equivalent multicore implementations on x86 CPUs. We discuss the various tradeoffs of the different architectures.

99 GENERAL AND MISCELLANEOUS↗

In Situ Monitoring of Groundwater Contamination Using the Kalman Filter

This study presents a Kalman filter-based framework to establish a real-time in situ monitoring system for groundwater contamination based on in situ measurable water quality variables, such as specific conductance (SC) and pH. First, this framework uses principal component analysis (PCA) to identify correlations between the contaminant concentrations of interest and in situ measurable variables. It then applies the Kalman filter to estimate contaminant concentrations continuously and in real-time by coupling data-driven concentration-decay models with the previously identified data correlations. We demonstrate our approach with historical groundwater data from the Savannah River Site F-Area: We use SC and pH data to estimate tritium and uranium concentrations over time. Results show that the developed method can estimate these contaminant concentrations based on in situ measurable variables. The estimates remain reliable with less frequent or no direct measurements of the contaminant concentrations, while capturing the dynamics of short- and long-term contaminant concentration changes. In addition, we show that data mining, such as PCA, is useful to understand correlations in groundwater data and to design long-term monitoring systems. As a result, the developed in situ monitoring methodology is expected to improve long-term groundwater monitoring by continuously confirming the contaminant plume's stability and by providing an early warning system for unexpected changes in the plume's migration.

54 ENVIRONMENTAL SCIENCES↗

Unsupervised learning of representative local atomic arrangements in molecular dynamics data

Molecular dynamics (MD) simulations present a data-mining challenge, given that they can generate a considerable amount of data but often rely on limited or biased human interpretation to examine their information content. By not asking the right questions of MD data we may miss critical information hidden within it. Here we combine dimensionality reduction (UMAP) and unsupervised hierarchical clustering (HDBSCAN) to quantitatively characterize prevalent coordination environments of chemical species within MD data. By focusing on local coordination, we significantly reduce the amount of data to be analyzed by extracting all distinct molecular formulas within a given coordination sphere. We then efficiently combine UMAP and HDBSCAN with alignment or shape-matching algorithms to partition these formulas into structural isomer families indicating their relative populations. The method was employed to reveal details of cation coordination in electrolytes based on molecular liquids.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗

A Scalable Monitoring for the CMS Filter Farm Based on Elasticsearch

A flexible monitoring system has been designed for the CMS File-based Filter Farm making use of modern data mining and analytics components. All the metadata and monitoring information concerning data flow and execution of the HLT are generated locally in the form of small documents using the JSON encoding. These documents are indexed into a hierarchy of elasticsearch (es) clusters along with process and system log information. Elasticsearch is a search server based on Apache Lucene. It provides a distributed, multitenant-capable search and aggregation engine. Since es is schema-free, any new information can be added seamlessly and the unstructured information can be queried in non-predetermined ways. The leaf es clusters consist of the very same nodes that form the Filter Farm thus providing natural horizontal scaling. A separate central” es cluster is used to collect and index aggregated information. The fine-grained information, all the way to individual processes, remains available in the leaf clusters. The central es cluster provides quasi-real-time high-level monitoring information to any kind of client. Historical data can be retrieved to analyse past problems or correlate them with external information. We discuss the design and performance of this system in the context of the CMS DAQ commissioning for LHC Run 2.

Andre, J. M.↗

Problem Reporting Taxonomy and Data Preparation Tool Evaluation

A member of the NASA Engineering and Safety Center (NESC) Systems Engineering Office (SEO) Technical Discipline Team (TDT) requested a SEO-managed activity to perform a gap analysis on the proposed NASA Standard 0006, "Common NASA Taxonomy for Problem Reporting, Analysis, and Resolution", and to create an input filter and set of instructions for using the data-mining/data-cleansing tool TechOasis1 with Space Shuttle Program (SSP) problem reporting data. The work that achieved these objectives and deployment of TechOasis are discussed in this report.

Beil, Robert J.↗

Determining the Number of Clusters in a Data Set Without Graphical Interpretation

Cluster analysis is a data mining technique that is meant ot simplify the process of classifying data points. The basic clustering process requires an input of data points and the number of clusters wanted. The clustering algorithm will then pick starting C points for the clusters, which can be either random spatial points or random data points. It then assigns each data point to the nearest C point where "nearest usually means Euclidean distance, but some algorithms use another criterion. The next step is determining whether the clustering arrangement this found is within a certain tolerance. If it falls within this tolerance, the process ends. Otherwise the C points are adjusted based on how many data points are in each cluster, and the steps repeat until the algorithm converges,

Aguirre, Nathan S.↗

An Adaptive Memory Interface Controller for Improving Bandwidth Utilization of Hybrid and Reconfigurable Systems

Emerging applications such as data mining, bioinformatics, knowledge discovery, social network analysis are irregular. They use data structures based on pointers or linked lists, such as graphs, unbalanced trees or unstructures grids, which generates unpredictable memory accesses. These data structures usually are large, but difficult to partition. These applications mostly are memory bandwidth bounded and have high synchronization intensity. However, they also have large amounts of inherent dynamic parallelism, because they potentially perform a task for each one of the element they are exploring. Several efforts are looking at accelerating these applications on hybrid architectures, which integrate general purpose processors with reconfigurable devices. Some solutions, which demonstrated significant speedups, include custom-hand tuned accelerators or even full processor architectures on the reconfigurable logic. In this paper we present an approach for the automatic synthesis of accelerators from C, targeted at irregular applications. In contrast to typical High Level Synthesis paradigms, which construct a centralized Finite State Machine, our approach generates dynamically scheduled hardware components. While parallelism exploitation in typical HLS-generated accelerators is usually bound within a single execution flow, our solution allows concurrently running multiple execution flow, thus also exploiting the coarser grain task parallelism of irregular applications. Our approach supports multiple, multi-ported and distributed memories, and atomic memory operations. Its main objective is parallelizing as many memory operations as possible, independently from their execution time, to maximize the memory bandwidth utilization. This significantly differs from current HLS flows, which usually consider a single memory port and require precise scheduling of memory operations. A key innovation of our approach is the generation of a memory interface controller, which dynamically maps concurrent memory accesses to multiple ports. We present a case study on a typical irregular kernel, Graph Breadth First search (BFS), exploring different tradeoffs in terms of parallelism and number of memories.

BFS↗

Tools and Data Services from the GSFC Earth Sciences DAAC for Aura Science Data Users

In these times of rapidly increasing amounts of archived data, tools and data services that manipulate data and uncover nuggets of information that potentially lead to scientific discovery are becoming more and more essential. The Goddard Space Flight Center (GSFC) Earth Sciences (GES) Distributed Active Archive Center (DAAC) has made great strides in facilitating science and applications research by, in consultation with its users, developing innovative tools and data services. That is, as data users become more sophisticated in their research and more savvy with information extraction methodologies, the GES DAAC has been responsive to this evolution. This presentation addresses the tools and data services available and under study at the GES DAAC, applied to the Earth sciences atmospheric data. Now, with the data from NASA's latest Atmospheric Chemistry mission, Aura, being readied for public release, GES DAAC tools, proven successful for past atmospheric science missions such as MODIS, AIRS, TRMM, TOMS, and UARS, provide an excellent basis for similar tools updated for the data from the Aura instruments. GES DAAC resident Aura data sets are from the Microwave Limb Sounder (MLS), Ozone Monitoring Instrument (OMI), and High Resolution Dynamics Limb Sounder (HIRDLS). Data obtained by these instruments afford researchers the opportunity to acquire accurate and continuous visualization and analysis, customized for Aura data, will facilitate the use and increase the usefulness of the new data. The Aura data, together with other heritage data at the GES DAAC, can potentially provide a long time series of data. GES DAAC tools will be discussed, as well as the GES DAAC Near Archive Data Mining (NADM) environment, the GIOVANNI on-line analysis tool, and rich data search and order services. Information can be found at: http://daac.gsfc.nasa.gov/upperatm/aura/. Additional information is contained in the original extended abstract.

Kempler, S.↗

Using Time Series Segmentation for Deriving Vegetation Phenology Indices from MODIS NDVI Data

Characterizing vegetation phenology is a highly significant problem, due to its importance in regulating ecosystem carbon cycling, interacting with climate changes, and decision-making of croplands managements. While ground based sensors, such as the AmeriFlux sensors, can provide measurements at high temporal resolution (every hour) and can be used to accurately calculate vegetation phenology indices, they are limited to only a few sites. Remote sensing data, such as the Normalized Difference Vegetation Index (NDVI), collected using the MODerate Resolution Imaging Spectroradiometer (MODIS), can provide global coverage, though at a much coarser temporal resolution (16 days). In this study we use data mining based time series segmentation methods to derive phenology indices from NDVI data, and compare it with the phenology indices derived from the AmeriFlux data using a widely used model fitting approach. Results show a significant correlation (as high as 0.60) between the indices derived from these two different data sources. This study demonstrates that data driven methods can be effectively employed to provide realistic estimates of vegetation phenology indices using periodic time series data and has the potential to be used at large spatial scales and for long-term remote sensing data.

Chandola, Varun↗

Using Dynamic Quantum Clustering to Analyze Hierarchically Heterogeneous Samples on the Nanoscale

Dynamic Quantum Clustering (DQC) is an unsupervised, high visual data mining technique. DQC was tested as an analysis method for X-ray Absorption Near Edge Structure (XANES) data from the Transmission X-ray Microscopy (TXM) group. The TXM group images hierarchically heterogeneous materials with nanoscale resolution and large field of view. XANES data consists of energy spectra for each pixel of an image. It was determined that DQC successfully identifies structure in data of this type without prior knowledge of the components in the sample. Clusters and sub-clusters clearly reflected features of the spectra that identified chemical component, chemical environment, and density in the image. DQC can also be used in conjunction with the established data analysis technique, which does require knowledge of components present.

99 GENERAL AND MISCELLANEOUS↗

Redefining the Practice of Peer Review Through Intelligent Automation—Part 3: Automated Report Analysis and Data Reconciliation

One method for addressing existing peer review limitations is the assignment of peer review cases on a completely blinded basis, in which the peer reviewer would create an independent report which can then be cross-referenced with the primary reader report of record. By leveraging existing computerized data mining techniques, one could in theory automate and objectify the process of report data extraction, classification, and analysis, while reducing time and resource requirements intrinsic to manual peer review report analysis. Once inter-report analysis has been performed, resulting inter-report discrepancies can be presented to the radiologist of record for review, along with the option to directly communicate with the peer reviewer through an electronic data reconciliation tool aimed at collaboratively resolving inter-report discrepancies and improving report accuracy. All associated report and reconciled data could in turn be recorded in a referenceable peer review database, which provides opportunity for context and user-specific education and decision support.

42 ENGINEERING↗