A probabilistic approach to mining massive Earth science data sets
Explore the source record for details and available documents.
remote sensing↗
SEARCH · Search NASA
Results for “Data Mining”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
Pre-Launch GOES-R Risk Reduction Activities for the Geostationary Lightning Mapper
The GOES-R Geostationary Lightning Mapper (GLM) is a new instrument planned for GOES-R that will greatly improve storm hazard nowcasting and increase warning lead time day and night. Daytime detection of lightning is a particularly significant technological advance given the fact that the solar illuminated cloud-top signal can exceed the intensity of the lightning signal by a factor of one hundred. Our approach is detailed across three broad themes which include: Data Processing Algorithm Readiness, Forecast Applications, and Radiance Data Mining. These themes address how the data will be processed and distributed, and the algorithms and models for developing, producing, and using the data products. These pre-launch risk reduction activities will accelerate the operational and research use of the GLM data once GOES-R begins on-orbit operations. The GLM will provide unprecedented capabilities for tracking thunderstorms and earlier warning of impending severe and hazardous weather threats. By providing direct information on lightning initiation, propagation, extent, and rate, the GLM will also capture the updraft dynamics and life cycle of convective storms, as well as internal ice precipitation processes. The GLM provides information directly from the heart of the thunderstorm as opposed to cloud-top only. Nowcasting applications enabled by the GLM data will expedite the warning and response time of emergency management systems, improve the dispatch of electric power utility repair crews, and improve airline routing around thunderstorms thereby improving safety and efficiency, saving fuel and reducing delays. The use of GLM data will assist the Bureau of Land Management (BLM) and the Forest Service in quickly detecting lightning ground strikes that have a high probability of causing fires. Finally, GLM data will help assess the role of thunderstorms and deep convection in global climate, and will improve regional air quality and global chemistry/climate modeling. The GLM has a robust design that benefits and improves upon its strong heritage of NASA-developed LEO predecessors, the Optical Transient Detector (OTD) and the Lightning Imaging Sensor (LIS). GLM will have a substantially larger number of pixels within the focal plane, two lens systems, and multiple Real-Time Event Processors REPS for on-board event detection and data compression to provide continuous observations of the Americas and adjacent oceans.
Goodman, S. J.↗
Requirements for Predictive Analytics
It is important to have a clear understanding of how traditional Business Intelligence (BI) and analytics are different and how they fit together in optimizing organizational decision making. With tradition BI, activities are focused primarily on providing context to enhance a known set of information through aggregation, data cleansing and delivery mechanisms. As these organizations mature their BI ecosystems, they achieve a clearer picture of the key performance indicators signaling the relative health of their operations. Organizations that embark on activities surrounding predictive analytics and data mining go beyond simply presenting the data in a manner that will allow decisions makers to have a complete context around the information. These organizations generate models based on known information and then apply other organizational data against these models to reveal unknown information.
97 MATHEMATICS AND COMPUTING↗
A novel procedure on next generation sequencing data analysis using text mining algorithm
Background: Next-generation sequencing (NGS) technologies have provided researchers with vast possibilities in various biological and biomedical research areas. Efficient data mining strategies are in high demand for large scale comparative and evolutional studies to be performed on the large amounts of data derived from NGS projects. Topic modeling is an active research field in machine learning and has been mainly used as an analytical tool to structure large textual corpora for data mining. Methods: We report a novel procedure to analyse NGS data using topic modeling. It consists of four major procedures: NGS data retrieval, preprocessing, topic modeling, and data mining using Latent Dirichlet Allocation (LDA) topic outputs. The NGS data set of the Salmonella enterica strains were used as a case study to show the workflow of this procedure. The perplexity measurement of the topic numbers and the convergence efficiencies of Gibbs sampling were calculated and discussed for achieving the best result from the proposed procedure. Results: The output topics by LDA algorithms could be treated as features of Salmonella strains to accurately describe the genetic diversity of fliC gene in various serotypes. The results of a two-way hierarchical clustering and data matrix analysis on LDA-derived matrices successfully classified Salmonella serotypes based on the NGS data. The implementation of topic modeling in NGS data analysis procedure provides a new way to elucidate genetic information from NGS data, and identify the gene-phenotype relationships and biomarkers, especially in the era of biological and medical big data. Conclusion: The implementation of topic modeling in NGS data analysis provides a new way to elucidate genetic information from NGS data, and identify the gene-phenotype relationships and biomarkers, especially in the era of biological and medical big data.
59 BASIC BIOLOGICAL SCIENCES↗
Using ADOPT Algorithm and Operational Data to Discover Precursors to Aviation Adverse Events
The US National Airspace System (NAS) is making its transition to the NextGen system and assuring safety is one of the top priorities in NextGen. At present, safety is managed reactively (correct after occurrence of an unsafe event). While this strategy works for current operations, it may soon become ineffective for future airspace designs and high density operations. There is a need for proactive management of safety risks by identifying hidden and "unknown" risks and evaluating the impacts on future operations. To this end, NASA Ames has developed data mining algorithms that finds anomalies and precursors (high-risk states) to safety issues in the NAS. In this paper, we describe a recently developed algorithm called ADOPT that analyzes large volumes of data and automatically identifies precursors from real world data. Precursors help in detecting safety risks early so that the operator can mitigate the risk in time. In addition, precursors also help identify causal factors and help predict the safety incident. The ADOPT algorithm scales well to large data sets and to multidimensional time series, reduce analyst time significantly, quantify multiple safety risks giving a holistic view of safety among other benefits. This paper details the algorithm and includes several case studies to demonstrate its application to discover the "known" and "unknown" safety precursors in aviation operation.
aviation safet↗
Addressing Data Center Cooling Needs through the Use of Subsurface Thermal Energy Storage Systems
This study aims to evaluate the feasibility of addressing the cooling needs for information technology (IT) equipment in data centers by using reservoir thermal energy storage (RTES) to provide reliable and sustainable low-temperature fluid. This project focuses on the technical viability of operating such a system in Houston, Texas, which is a representative location for crypto mining data centers. An analysis has been performed to investigate the technical feasibility with climate data for Houston. Results show that data centers on the scale of 30 MW, and operating at a temperature of 27 degrees C, can be reliably cooled by a combined RTES and dry coolers setup. A techno-economic analysis will be performed and energy/water saving benefits will be quantified in the future. In addition, the study will be extended to other representative data center locations with different climates and geographical locations.
cooling↗
Evolving landscape of low-energy nuclear physics publications
Evolution of low-energy nuclear physics publications over the last 120 years has been analyzed using nuclear physics databases. An extensive study of Nuclear Science References, Experimental Nuclear Reaction Data (EXFOR), and Evaluated Nuclear Structure Data File (ENSDF) contents provides a unique picture of refereed and non-refereed nuclear physics references. Significant fractional contributions of non-refereed reports, private communications and conference proceedings in EXFOR and ENSDF databases in the 1970’s reflect extensive experimental campaigns and an insufficient number of research journals. This trend has been reversed in recent years because the number of measurements is much lower, while number of journals is higher. In addition, nuclear physics results are mainly published in a limited number of journals, such as Physical Review C and Nuclear Physics A. In the present work, historic publication trends and averages have been extracted and analyzed using nuclear data mining techniques. Lastly, the results of this study and implications are discussed and conclusions presented.
73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
Alpha Seeding for Support Vector Machines
A key practical obstacle in applying support vector machines to many large-scale data mining tasks is that SVM's generally scale quadratically (or worse) in the number of examples or support vectors.
Support Vector Machines Alpha Seeding Data Mining↗
Applications of Anomaly Detection and Precursor Identification in Airspace Operations
As we continue to advance the U.S. National Airspace into the next generation of air traffic, we face challenges in both increase in complexity, as well as, a significant growth in traffic volume. Addressing these challenges, while maintaining the same level of safety is an important application of data mining. Because of these significant shifts in airspace design and usage there is a need to identify current and emergent safety risks along with their potential precursors. In recent years NASA has made advancements in developing scalable methods to address this effort in the Big Data paradigm. Multiple kernel anomaly detection approaches have been employed on both surveillance radar data and flight operational quality assurance data to identify operationally significant safety risks. Additionally, events have been explored with a recently developed precursor identification tool to discover states that reveal an increased probability of a safety event. These tools can be used to discover emerging safety risks that may not be currently monitored, which allows for mitigation tactics to be employed and ultimately make the overall airspace safer. This talk will discuss an overview of these methods and a discussion of the findings.
anomaly detection↗
Using dissimilarity metrics to identify interesting designs
We report on application of a combination of data mining steps guided by user-directives, and appropriate visualizations to present the results of these steps to the users.
design heuristic search optimization data mining↗
HPRT promotes proliferation and metastasis in head and neck squamous cell carcinoma through direct interaction with STAT3
Highlights: • The higher expression of HPRT in HNSCC tissue was related to poor prognosis of patients. • Overexpression of HPRT increased the gene expression of epithelial mesenchymal transition markers. • HPRT directly interacted with STAT3. • Knocking down HPRT significantly decreased tumour growth and enhanced the anticancer effect against HNSCC xenografts. Increasing effort has been put into finding novel molecular pathways to improve the efficiency of EGFR inhibitors against head and neck squamous cell cancer (HNSCC). In this study, we performed data mining and bioinformatically analysed RNA-Seq data downloaded from TCGA and confirmed that higher expression of HPRT in HNSCC tissue was related to poor prognosis of patients. Then, we conducted in vitro and in vivo loss- and gain-of-function experiments to demonstrate the role of HPRT in HNSCC cell lines. Overexpression of HPRT increased the gene expression of epithelial mesenchymal transition markers via direct interaction with STAT3. Knocking down HPRT significantly decreased tumour growth and enhanced the anticancer effect of EGFR inhibitors against HNSCC xenografts. In conclusion, HPRT is a binding partner of STAT3 that promotes EMT and proliferation. Our findings support HPRT as a promising prognostic indicator and potential therapeutic target for HNSCC.
60 APPLIED LIFE SCIENCES↗
Parallel Multivariate Spatio-Temporal Clustering of Large Ecological Datasets on Hybrid Supercomputers
A proliferation of data from vast networks of remote sensing platforms (satellites, unmanned aircraft systems (UAS), airborne etc.), observational facilities (meteorological, eddy covariance etc.), state-of-the-art sensors, and simulation models offer unprecedented opportunities for scientific discovery. Unsupervised classification is a widely applied data mining approach to derive insights from such data. However, classification of very large data sets is a complex computational problem that requires efficient numerical algorithms and implementations on high performance computing (HPC) platforms. Additionally, increasing power, space, cooling and efficiency requirements has led to the deployment of hybrid supercomputing platforms with complex architectures and memory hierarchies like the Titan system at Oak Ridge National Laboratory. The advent of such accelerated computing architectures offers new challenges and opportunities for big data analytics in general and specifically, large scale cluster analysis in our case. Although there is an existing body of work on parallel cluster analysis, those approaches do not fully meet the needs imposed by the nature and size of our large data sets. Moreover, they had scaling limitations and were mostly limited to traditional distributed memory computing platforms. We present a parallel Multivariate Spatio-Temporal Clustering (MSTC) technique based on k-means cluster analysis that can target hybrid supercomputers like Titan. We developed a hybrid MPI, CUDA and OpenACC implementation that can utilize both CPU and GPU resources on computational nodes. We describe performance results on Titan that demonstrate the scalability and efficacy of our approach in processing large ecological data sets.
Sreepathi, Sarat↗
SU-E-J-04: A Data-Driven, Response-Based, Multi-Criteria Decision Support System for Personalized Lung Radiation Treatment Planning
Purpose: To develop a decision support tool to predict a patient's potential overall survival (OS) and radiation induced toxicity (RIT) based on clinical factors and responses during the course of radiotherapy, and suggest appropriate radiation dose adjustments to improve therapeutic effect. Methods: Important relationships between a patient's basic information and their clinical features before and during the radiation treatment are identified from historical clinical data by using statistical learning and data mining approaches. During each treatment period, a data analysis (DA) module predicts radiotherapy features such as time to local progression (TTLP), time to distant metastases (TTDM), radiation toxicity to different organs, etc., under possible future treatment plans based on patient specifics or responses. An information fusion (IF) module estimates intervals for a patient's OS and the probabilities of RIT from a treatment plan by integrating the outcomes of module DA. A decision making (DM) module calculates “satisfaction” with the predicted radiation outcome based on trade-offs between OS and RIT, and finds the best treatment plan for the next time period via multi-criteria optimization. Results: Using physical and biological data from 130 lung cancer patients as our test bed, we were able to train and implement the 3 modules of our decision support tool. Examples demonstrate how it can help predict a new patient's potential OS and RIT with different radiation dose plans along with how these combinations change with dose, thus presenting a range of satisfaction/utility for use in individualized decision support. Conclusion: Although the decision support tool is currently developed from a small patient sample size, it shows the potential for the improvement of each patient's satisfaction in personalized radiation therapy. The radiation treatment outcome prediction and decision making model needs to be evaluated with more patients and demonstrated for use in radiation treatments for other cancers. P01-CA59827;R01CA142840.
60 APPLIED LIFE SCIENCES↗
LIB Design Module for Grid Energy System Application
We will employ a machine learning approach with intelligent data mining and database construction to analyze enormous data repositories for identifying and extracting geographic-dependent cell design specifications from publicly accessible grid-scale energy storage usage databases in an automated way at scale.
Liu, Dianying [Pacific Northwest National Laborato↗
Dynamic Non-Hierarchical File Systems for Exascale Storage
This constitutes the final report for “Dynamic Non-Hierarchical File Systems for Exascale Storage”. The ultimate goal of this project was to improve data management in scientific computing and high-end computing (HEC) applications, and to achieve this goal we proposed: to develop the first, HEC-targeted, file system featuring rich metadata and provenance collection, extreme scalability, and future storage hardware integration as core design goals, and to evaluate and develop a flexible non-hierarchical file system interface suitable for providing more powerful and intuitive data management interfaces to HEC and scientific computing users. Data management is swiftly becoming a serious problem in the scientific community – while copious amounts of data are good for obtaining results, finding the right data is often daunting and sometimes impossible. Scientists participating in a Department of Energy workshop noted that most of their time was spent “...finding, processing, organizing, and moving data and it’s going to get much worse”. Scientists should not be forced to become data mining experts in order to retrieve the data they want, nor should they be expected to remember the naming convention they used several years ago for a set of experiments they now wish to revisit. Ideally, locating the data you need would be as easy as browsing the web. Unfortunately, existing data management approaches are usually based on hierarchical naming, a 40 year-old technology designed to manage thousands of files, not exabytes of data. Today’s systems do not take advantage of the rich array of metadata that current high-end computing (HEC) file systems can gather, including content-based metadata and provenance1 information. As a result, current metadata search approaches are typically ad hoc and often work by providing a parallel management system to the “main” file system, as is done in Linux (the locate utility), personal computers, and enterprise search appliances. These search applications are often optimized for a single file system, making it difficult to move files and their metadata between file systems. Users have tried to solve this problem in several ways, including the use of separate databases to index file properties, the encoding of file properties into file names, and separately gathering and managing provenance data, but none of these approaches has worked well, either due to limited usefulness or scalability, or both. Our research addressed several key issues: High-performance, real-time metadata harvesting: extracting important attributes from files dynamically and immediately updating indexes used to improve search; Transparent, automatic, and secure provenance capture: recording the data inputs and processing steps used in the production of each file in the system; Scalable indexing: indexes that are optimized for integration with the file system; Dynamic file system structure: our approach provides dynamic directories similar to those in semantic file systems, but these are the native organization rather than a feature grafted onto a conventional system. In addition to these goals, our research effort will include evaluating the impact of new storage technologies on the file system design and performance. In particular, the indexing and metadata harvesting functions can potentially benefit from the performance improvements promised by new storage class memories.
42 ENGINEERING↗
Coastal Typologies: Methods for Improving Representation of Arctic Coastal Environments, Starting with Alaska's Northern Slope [Slides]
The objectives of the project included: Broaden QGIS (mapping) and Python (data analysis) skill set; Contribute to overall development of coastal typologies by conducting data mining and analysis; Expand scientific literacy; Narrow data sources to those most useful for project; Convert chosen data sets into image formats for analysis; and, Utilize python to analyze images.
58 GEOSCIENCES↗
Spatio-Temporal Surrogates for Interaction of a Jet with High Explosives: Part II - Clustering Extremely High-Dimensional Grid-Based Data
Building an accurate surrogate model for the spatio-temporal outputs of a computer simulation is a challenging task. A simple approach to improve the accuracy of the surrogate is to cluster the outputs based on similarity and build a separate surrogate model for each cluster. This clustering is relatively straightforward when the output at each time step is of moderate size. However, when the spatial domain is represented by a large number of grid points, numbering in the millions, the clustering of the data becomes more challenging. In this report, we consider output data from simulations of a jet interacting with high explosives. These data are available on spatial domains of different sizes, at grid points that vary in their spatial coordinates, and in a format that distributes the output across multiple files at each time step of the simulation. We first describe how we bring these data into a consistent format prior to clustering. Borrowing the idea of random projections from data mining, we reduce the dimension of our data by a factor of thousand, making it possible to use the iterative k-means method for clustering. We show how we can use the randomness of both the random projections, and the choice of initial centroids in k-means clustering, to determine the number of clusters in our data set. Our approach makes clustering of extremely high dimensional data tractable, generating meaningful cluster assignments for our problem, despite the approximation introduced in the random projections.
97 MATHEMATICS AND COMPUTING↗
Interactive Computer Graphics
Aerospace data analysis tools that significantly reduce the time and effort needed to analyze large-scale computational fluid dynamics simulations have emerged this year. The current approach for most postprocessing and visualization work is to explore the 3D flow simulations with one of a dozen or so interactive tools. While effective for analyzing small data sets, this approach becomes extremely time consuming when working with data sets larger than one gigabyte. An active area of research this year has been the development of data mining tools that automatically search through gigabyte data sets and extract the salient features with little or no human intervention. With these so-called feature extraction tools, engineers are spared the tedious task of manually exploring huge amounts of data to find the important flow phenomena. The software tools identify features such as vortex cores, shocks, separation and attachment lines, recirculation bubbles, and boundary layers. Some of these features can be extracted in a few seconds; others take minutes to hours on extremely large data sets. The analysis can be performed off-line in a batch process, either during or following the supercomputer simulations. These computations have to be performed only once, because the feature extraction programs search the entire data set and find every occurrence of the phenomena being sought. Because the important questions about the data are being answered automatically, interactivity is less critical than it is with traditional approaches.
Kenwright, David↗