Search NASA⌕ Search

Engineering topics

Chandola, Varun

Publications and source records attributed to Chandola, Varun.

At least 19 records

Large Deviations Anomaly Detection (LAD) for collection of multivariate time series data: Applications to COVID-19 data

Time series anomaly detection is frequently used to identify extreme behaviors within a single time series. Identifying extreme trends in relation to a collection of other time series, on the other hand, is frequently of significant interest, such as in public health policy, social justice, and pandemic propagation. Using concepts from large deviations theory , we propose an algorithm that can scale to large collections of time series data. This paper expands on the LAD algorithm presented in Guggilam et al. (2022). The proposed algorithm is an online anomaly detection method for identifying anomalies in a collection of multivariate time series that takes advantage of the algorithm’s ability to scale to high-dimensional data. We show how the proposed Large Deviations Anomaly Detection (LAD) algorithm can be used to identify regions with anomalous trends in COVID-19 cases, deaths, biweekly growth rates, vaccinations, and fatality rates. Several of the observed anomalous trends are associated with regions that have demonstrated poor response to the COVID pandemic.

97 MATHEMATICS AND COMPUTING↗

Physics informed machine learning for chemistry tabulation

Modeling of turbulent combustion system requires modeling the underlying chemistry and the turbulent transport. Solving both systems simultaneously is computationally prohibitive. Instead, given the difference in scales at which the two sub-systems evolve, the two sub-systems are typically (re)solved separately. Popular approaches such as the Flamelet Generated Manifolds (FGM) use a two-step strategy where the governing reaction kinetics are pre-computed and mapped to a low-dimensional manifold, characterized by a few reaction progress variables (model reduction) and the manifold is then “looked-up” during the run-time to estimate the high-dimensional system state by the turbulent transport system. While existing works have focused on these two steps independently, in this work we show that joint learning of the progress variables and the look-up model, can yield more accurate results. Here, we build on the base formulation and implementation to include the dynamically generated Thermochemical State Variables (Lower Dimensional Dynamic Source Terms). We discuss the challenges in the implementation of this deep neural network architecture and experimentally demonstrate its superior performance.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗

WebGIobe - A cloud-based geospatial analysis framework for interacting with climate data

While climate models have evolved over time to produce high fidelity and high resolution climate forecasts, visualization and analysis of the output of the model simulations has been limited, typically constrained to single dimensional charts for visualization and basic aggregate statistics for analytics. Same is true for the large troves of observational data available from meteorological stations all over the world. For richer understanding of climate and the impact of climate change, one needs computational tools that allow researchers, policymakers, and general public, to interact with the climate data. In this paper, we describe, webGlobe, a browser based GIS framework for interacting with climate data, and other datasets available in similar format. webGlobe is a unique resource that allows unprecedented access to climate data through a browser. The framework also allows for deploying machine learning based analytical applications on the climate data without putting computational burden on the client. Instead, webGlobe uses a client-server framework, where the server, deployed on a cloud infrastructure, allows for dynamic allocation of resources for running compute-intensive applications. The capabilities of the framework will be discussed in context of a use case: identifying extreme events from real and simulated climate data using a Gaussian process based change detection algorithm.

Sharma, Arun↗

Machine learning for energy-water nexus: challenges and opportunities

Modeling the interactions of water and energy systems is important to the enforcement of infrastructure security and system sustainability. To this end, recent technological advancement has allowed the production of large volumes of data associated with functioning of these sectors. We are beginning to see that statistical and machine learning techniques can help elucidate characteristic patterns across these systems from water availability, transport, and use to energy generation, fuel supply, and customer demand, and in the interdependencies among these systems that can leave these systems vulnerable to cascading impacts from single disruptions. In this paper, we discuss ways in which data and machine learning can be applied to the challenges facing the energy-water nexus along with the potential issues associated with the machine learning techniques themselves. We then survey machine learning techniques that have found application to date in energy-water nexus problems. We conclude by outlining future research directions and opportunities for collaboration among the energy-water nexus and machine learning communities that can lead to mutual synergistic advantage.

24 POWER TRANSMISSION AND DISTRIBUTION↗

Fraud Detection in Healthcare

We discuss the problem of fraud detection in healthcare in this chapter. Given the recent scrutiny of the ineciencies in the US healthcare system, identifying fraud has been on the forefront of the eorts towards reducing the healthcare costs. In this chapter we will focus on understanding the issue of healthcare fraud in detail, and review methods that have been proposed in the literature to combat this issue using data driven approach.

data analytics↗

Knowledge Discovery from Massive Healthcare Claims Data

The role of big data in addressing the needs of the present healthcare system in US and rest of the world has been echoed by government, private, and academic sectors. There has been a growing emphasis to explore the promise of big data analytics in tapping the potential of the massive healthcare data emanating from private and government health insurance providers. While the domain implications of such collaboration are well known, this type of data has been explored to a limited extent in the data mining community. The objective of this paper is two fold: first, we introduce the emerging domain of big"healthcare claims data to the KDD community, and second, we describe the success and challenges that we encountered in analyzing this data using state of art analytics for massive data. Specically, we translate the problem of analyzing healthcare data into some of the most well-known analysis problems in the data mining community, social network analysis, text mining, and temporal analysis and higher order feature construction, and describe how advances within each of these areas can be leveraged to understand the domain of healthcare. Each case study illustrates a unique intersection of data mining and healthcare with a common objective of improving the cost-care ratio by mining for opportunities to improve healthcare operations and reducing hat seems to fall under fraud, waste,and abuse.

healthcare fraud↗

Image Based Characterization of Formal and Informal Neighborhoods in an Urban Landscape

The high rate of global urbanization has resulted in a rapid increase in informal settlements, which can be de ned as unplanned, unauthorized, and/or unstructured housing. Techniques for ef ciently mapping these settlement boundaries can bene t various decision making bodies. From a remote sensing perspective, informal settlements share unique spatial characteristics that distinguish them from other types of structures (e.g., industrial, commercial, and formal residential). These spatial characteristics are often captured in high spatial resolution satellite imagery. We analyzed the role of spatial, structural, and contextual features (e.g., GLCM, Histogram of Oriented Gradients, Line Support Regions, Lacunarity) for urban neighborhood mapping, and computed several low-level image features at multiple scales to characterize local neighborhoods. The decision parameters to classify formal-, informal-, and non-settlement classes were learned under Decision Trees and a supervised classi cation framework. Experiments were conducted on high-resolution satellite imagery from the CitySphere collection, and four different cities (i.e., Caracas, Kabul, Kandahar, and La Paz) with varying spatial characteristics were represented. Overall accuracy ranged from 85% in La Paz, Bolivia, to 92% in Kandahar, Afghanistan. While the disparities between formal and informal neighborhoods varied greatly, many of the image statistics tested proved robust.

58 GEOSCIENCES↗

Anomaly Detection for Discrete Sequences: A Survey

This survey attempts to provide a comprehensive and structured overview of the existing research for the problem of detecting anomalies in discrete/symbolic sequences. The objective is to provide a global understanding of the sequence anomaly detection problem and how existing techniques relate to each other. The key contribution of this survey is the classification of the existing research into three distinct categories, based on the problem formulation that they are trying to solve. These problem formulations are: 1) identifying anomalous sequences with respect to a database of normal sequences; 2) identifying an anomalous subsequence within a long sequence; and 3) identifying a pattern in a sequence whose frequency of occurrence is anomalous. We show how each of these problem formulations is characteristically distinct from each other and discuss their relevance in various application domains. We review techniques from many disparate and disconnected application domains that address each of these formulations. Within each problem formulation, we group techniques into categories based on the nature of the underlying algorithm. For each category, we provide a basic anomaly detection technique, and show how the existing techniques are variants of the basic technique. This approach shows how different techniques within a category are related or different from each other. Our categorization reveals new variants and combinations that have not been investigated before for anomaly detection. We also provide a discussion of relative strengths and weaknesses of different techniques. We show how techniques developed for one problem formulation can be adapted to solve a different formulation, thereby providing several novel adaptations to solve the different problem formulations. We also highlight the applicability of the techniques that handle discrete sequences to other related areas such as online anomaly detection and time series anomaly detection.

99 GENERAL AND MISCELLANEOUS↗

iGlobe: An Interactive Visualization and Analysis Framework for Geospatial Data

The iGlobe system is a desktop-based visualization and analysis environment which allows seamless integration of multiple geospatial data sets from varied sources and provides an interface to interactively analyze the different data sets and apply sophisticated data analysis and mining algorithms in a near real time fashion. The framework is highly desirable in domains such as earth and climate sciences where great emphasis is placed on simultaneous analysis of different data sets such as remote sensing images, climate model simulation outputs, and other environmental and demographic databases, to understand weather and climate systems and the impact of climate change in nature and people.

Chandola, Varun↗

A Scalable Gaussian Process Analysis Algorithm for Biomass Monitoring

Biomass monitoring is vital for studying the carbon cycle of earth's ecosystem and has several significant implications, especially in the context of understanding climate change and its impacts. Recently, several change detection methods have been proposed to identify land cover changes in temporal profiles (time series) of vegetation collected using remote sensing instruments, but do not satisfy one or both of the two requirements of the biomass monitoring problem, i.e., {\em operating in online mode} and {\em handling periodic time series}. In this paper, we adapt Gaussian process regression to detect changes in such time series in an online fashion. While Gaussian process (GP) have been widely used as a kernel based learning method for regression and classification, their applicability to massive spatio-temporal data sets, such as remote sensing data, has been limited owing to the high computational costs involved. We focus on addressing the scalability issues associated with the proposed GP based change detection algorithm. This paper makes several significant contributions. First, we propose a GP based online time series change detection algorithm and demonstrate its effectiveness in detecting different types of changes in {\em Normalized Difference Vegetation Index} (NDVI) data obtained from a study area in Iowa, USA. Second, we propose an efficient Toeplitz matrix based solution which significantly improves the computational complexity and memory requirements of the proposed GP based method. Specifically, the proposed solution can analyze a time series of length $t$ in $O(t^2)$ time while maintaining a $O(t)$ memory footprint, compared to the $O(t^3)$ time and $O(t^2)$ memory requirement of standard matrix manipulation based methods. Third, we describe a parallel version of the proposed solution which can be used to simultaneously analyze a large number of time series. We study three different parallel implementations: using threads, MPI, and a hybrid implementation using threads and MPI. Experimental results show that the hybrid implementation scales better than the multi-threaded and MPI based implementations. The application of the proposed scalable algorithm is demonstrated in analyzing massive remote sensing observation data. The hybrid implementation, using 1536 computing cores, can analyze an NDVI data set for the Iowa study area in nearly 5 seconds, while a serial algorithm, using standard Cholesky decomposition based routines, takes several days to process the same data set.

09 BIOMASS FUELS↗

GX-Means: A model-based divide and merge algorithm for geospatial image clustering

One of the practical issues in clustering is the specification of the appropriate number of clusters, which is not obvious when analyzing geospatial datasets, partly because they are huge (both in size and spatial extent) and high dimensional. In this paper we present a computationally efficient model-based split and merge clustering algorithm that incrementally finds model parameters and the number of clusters. Additionally, we attempt to provide insights into this problem and other data mining challenges that are encountered when clustering geospatial data. The basic algorithm we present is similar to the G-means and X-means algorithms; however, our proposed approach avoids certain limitations of these well-known clustering algorithms that are pertinent when dealing with geospatial data. We compare the performance of our approach with the G-means and X-means algorithms. Experimental evaluation on simulated data and on multispectral and hyperspectral remotely sensed image data demonstrates the effectiveness of our algorithm.

99 GENERAL AND MISCELLANEOUS↗

A Gaussian Process Based Online Change Detection Algorithm for Monitoring Periodic Time Series

Online time series change detection is a critical component of many monitoring systems, such as space and air-borne remote sensing instruments, cardiac monitors, and network traffic profilers, which continuously analyze observations recorded by sensors. Data collected by such sensors typically has a periodic (seasonal) component. Most existing time series change detection methods are not directly applicable to handle such data, either because they are not designed to handle periodic time series or because they cannot operate in an online mode. We propose an online change detection algorithm which can handle periodic time series. The algorithm uses a Gaussian process based non-parametric time series prediction model and monitors the difference between the predictions and actual observations within a statistically principled control chart framework to identify changes. A key challenge in using Gaussian process in an online mode is the need to solve a large system of equations involving the associated covariance matrix which grows with every time step. The proposed algorithm exploits the special structure of the covariance matrix and can analyze a time series of length T in O(T^2) time while maintaining a O(T) memory footprint, compared to O(T^4) time and O(T^2) memory requirement of standard matrix manipulation methods. We experimentally demonstrate the superiority of the proposed algorithm over several existing time series change detection algorithms on a set of synthetic and real time series. Finally, we illustrate the effectiveness of the proposed algorithm for identifying land use land cover changes using Normalized Difference Vegetation Index (NDVI) data collected for an agricultural region in Iowa state, USA. Our algorithm is able to detect different types of changes in a NDVI validation data set (with ~80% accuracy) which occur due to crop type changes as well as disruptive changes (e.g., natural disasters).

54 ENVIRONMENTAL SCIENCES↗

iGlobe: an interactive visualization and analysis framework for geospatial data

We demonstrate an interactive visualization and analysis system for integrating climate data with other geospatial data sets, such as environmental and demographic data. The \eviz system is a desktop-based visualization and analysis environment which allows seamless integration of multiple geospatial data sets from varied sources and provides an interface to interactively analyze the different data sets and apply sophisticated data analysis and mining algorithms in a near real time fashion. The framework is highly desirable in domains such as earth and climate sciences where great emphasis is placed on simultaneous analysis of different data sets such as remote sensing images, climate model simulation outputs, and other environmental and demographic databases, to understand weather and climate systems and the impact of climate change on nature and people.

54 ENVIRONMENTAL SCIENCES↗

Machine Learning Approaches for High-resolution Urban Land Cover Classification: A Comparative Study

The proliferation of several machine learning approaches makes it difficult to identify a suitable classification technique for analyzing high-resolution remote sensing images. In this study, ten classification techniques were compared from five broad machine learning categories. Surprisingly, the performance of simple statistical classification schemes like maximum likelihood and Logistic regression over complex and recent techniques is very close. Given that these two classifiers require little input from the user, they should still be considered for most classification tasks. Multiple classifier systems is a good choice if the resources permit.

97 MATHEMATICS AND COMPUTING↗

Rapid Damage Assessment Using High-resolution Remote Sensing Imagery: Tools and Techniques

Accurate damage assessment caused by major natural and anthropogenic disasters is becoming critical due to increases in human and economic loss. This increase in loss of life and severe damages can be attributed to growing population, as well as human migration to disaster prone regions of the world. Rapid damage assessment and dissemination of accurate information is critical for creating an effective emergency response. Remote sensing and geographic information systems (GIS) based techniques and tools are important in disaster damage assessment and reporting activities. In this review, we will look into the state of the art techniques in damage assessment using remote sensing and GIS.

99 GENERAL AND MISCELLANEOUS↗

Implementing a Gaussian Process Learning Algorithm in Mixed Parallel Environment

In this paper, we present a scalability analysis of a parallel Gaussian process training algorithm to simultaneously analyze a massive number of time series. We study three different parallel implementations: using threads, MPI, and a hybrid implementation using threads and MPI. We compare the scalability for the multi-threaded implementation on three different hardware platforms: a Mac desktop with two quad-core Intel Xeon processors (16 virtual cores), a Linux cluster node with four quad-core 2.3 GHz AMD Opteron processors, and SGI Altix ICE 8200 cluster node with two quad-core Intel Xeon processors (16 virtual cores). We also study the scalability of the MPI based and the hybrid MPI and thread based implementations on the SGI cluster with 128 nodes (2048 cores). Experimental results show that the hybrid implementation scales better than the multi-threaded and MPI based implementations. The hybrid implementation, using 1536 cores, can analyze a remote sensing data set with over 4 million time series in nearly 5 seconds while the serial algorithm takes nearly 12 hours to process the same data set.

99 GENERAL AND MISCELLANEOUS↗