Search NASA⌕ Search

SEARCH · Search NASA

Results for “algorithms scalability”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 559 records · Page 31
All sourcesNASA NTRSDOE OSTI
DOE OSTI2025

Efficient analysis of small-angle scattering curves for large biomolecular assemblies using Monte Carlo methods

Structure elucidation from small-angle scattering curves of large biomolecular assemblies is notoriously challenging. This is because the simulation of high-resolution features in the structure of large macromolecular assemblies, such as de novo protein assemblies, is computationally demanding when it needs to cover a broad range of length scales. Conventional methods, such as the numerical approximation to the Debye equation or the use of spherical harmonics, do not scale well as the size of the assembly increases, which limits their application to small structures (e.g. individual proteins). This work explores the effectiveness of a Monte Carlo method to simulate and fit scattering curves for large biomolecular assemblies spanning over ranges covering atomic and molecular detail (e.g. spacing and orientation of proteins in an assembly) as well as large-scale (hundreds of nanometres) features. Owing to its speed and scalability, it can be combined with a fitting algorithm to extract structural features from experimental small-angle scattering curves in biomolecular assemblies that are otherwise intractable for interpretation. This work first demonstrates the effectiveness of the tool using experimental small-angle X-ray scattering (SAXS) data from tile-like proteins that assemble into 1D tube-like macromolecular structures. Here, the diameter distribution of tubes is extracted from SAXS fits, and this is quantitatively compared with distributions from electron microscopy. SAXS data are also obtained from 2D sheet-like protein assemblies, and the proposed method is used to quantify structural features such as the separation distance between protein building blocks and the flexing of the sheet. An open-source implementation of the methodology is provided for use in a broad range of biological systems involving multi-scale scattering analysis.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2019

Data-Driven Load Modeling and Forecasting of Residential Appliances

The expansion of residential demand side management programs and increased deployment of controllable loads require accurate appliance-level load modeling and forecasting. This paper proposes a conditional hidden semi-Markov model to describe the probabilistic nature of residential appliance demand. Model parameters are estimated directly from power consumption data using scalable statistical learning methods. We also propose an algorithm for short-term load forecasting as a key application for appliance-level load models. Case studies performed using granular sub-metered power measurements from various types of appliances demonstrate the effectiveness of the proposed load model for short-term prediction.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
DOE OSTI2026

Expanding Access to Science Participation: A FAIR Framework for Petascale Data Visualization and Analytics

The massive data generated by scientists daily serve as both a major catalyst for new discoveries and innovations, as well as a significant roadblock that restricts access to the data. Here, our paper introduces a new approach to removing Big Data barriers and democratizing access to petascale data for the broader scientific community. Our novel data fabric abstraction layer allows user-friendly querying of scientific information while hiding the complexities of dealing with file systems or cloud services. We enable FAIR (Findable, Accessible, Interoperable, and Reusable) access to datasets such as NASA’s petascale climate datasets. Our paper presents an approach to managing, visualizing, and analyzing petabytes of data within a browser on equipment ranging from the top NASA supercomputer to commodity hardware like a laptop. Our novel data fabric abstraction utilizes state-of-the art progressive compression algorithms and machine-learning insights to power scalable visualization dashboards for petascale data. The result provides users with the ability to identify extreme events or trends dynamically, expanding access to scientific data and further enabling discoveries. We validate our approach by improving the ability of climate scientists to visually explore their data via three fully interactive dashboards. We further validate our approach by deploying the dashboards and simplified training materials in the classroom at a minority-serving institution. These dashboards, released in simplified form to the general public, contribute significantly to a broader push to democratize the access and use of climate data.

Computer science↗
DOE OSTI2014

Parallel Infrastructure Modeling and Inversion Module for E4D

Electrical resistivity tomography ERT is a method of imaging the electrical conductivity of the subsurface. Electrical conductivity is a useful metric for understanding the subsurface because it is governed by geomechanical and geochemical properties that drive subsurface systems. ERT works by injecting current into the subsurface across a pair of electrodes, and measuring the corresponding electrical potential response across another pair of electrodes. Many such measurements are strategically taken across an array of electrodes to produce an ERT data set. These data are then processed through a computationally demanding process known as inversion to produce an image of the subsurface conductivity structure that gave rise to the measurements. Data can be inverted to provide 2D images, 3D images, or in the case of time-lapse 3D imaging, 4D images. ERT is generally not well suited for environments with buried electrically conductive infrastructure such as pipes, tanks, or well casings, because these features tend to dominate and degrade ERT images. This reduces or eliminates the utility of ERT imaging where it would otherwise be highly useful for, for example, imaging fluid migration from leaking pipes, imaging soil contamination beneath leaking subsurface tanks, and monitoring contaminant migration in locations with dense network of metal cased monitoring wells. The location and dimension of buried metallic infrastructure is often known. If so, then the effects of the infrastructure can be explicitly modeled within the ERT imaging algorithm, and thereby removed from the corresponding ERT image. However, there are a number of obstacles limiting this application. 1) Metallic infrastructure cannot be accurately modeled with standard codes because of the large contrast in conductivity between the metal and host material. 2) Modeling infrastructure in true dimension requires the computational mesh to be highly refined near the metal inclusions, which increases computational demands. 3) The ERT imaging algorithm requires specialized modifications to accommodate high conductivity inclusions within the computational mesh. The solution to each of these challenges was implemented within E4D (formerly FERM3D), which is a parallel ERT imaging code developed at PNNL (IPID #30249). The infrastructure modeling module implement in E4D uses a method of decoupling the model at the metallic interface(s) boundaries, into several well posed sub-problems (one for each distinct metallic inclusion) that are subsequently solved and recombined to form the global solution. The approach is based on the immersed interface method, with has been applied for similar problems in other fields (e.g. semiconductor industry). Comparisons to analytic solutions have shown the results to be very accurate, addressing item 1 above. The solution is implemented about an unstructured mesh, which enables arbitrary shapes to be efficiently modelled, thereby addressing item 2 above. In addition, the algorithm is written in parallel and shows excellent scalability, which also addresses equation 2 above. Finally, because only the boundaries of metallic inclusions are modeled, there are no high conductivity cells within the modeling mesh, and the problem described by item 3 above is no longer applicable.

Johnson, TimC.↗
DOE OSTI2017

Extreme scale unstructured adaptive CFD for aerodynamic flow control

Understanding the flow of fluid, either liquid or gas, through and around solid bodies has challenged man since the dawn of scientific inquiry. Many of the great minds of science and mathematics have progressively built up a hierarchy of fluid models since fluid flow impacts our lives in so many fundamental ways - from the early days of flow of water in viaducts to cities, to today's flight of planes. This chapter is concerned with the computational modeling of turbulent flow around aerodynamic bodies such as planes and wind turbines. In this case, viscous effects near the solid bodies create very thin boundary layers that yield highly anisotropic (gradients normal to the surface may be 10 6 larger than gradients along the surface) solutions to the governing non-linear partial differential equations (PDE); the Navier-Stokes equations. Furthermore, turbulent flows develop extremely broad ranges of length and time scales motivating the use of discretization methods capable of employing adaptivity and implicit time integration. The combination of these features - non-linear, anisotropy, adaptivity, and implicit - dramatically raise the complexity of the discretization, posing large challenges to efficient scalable parallel implementation. However, through careful design, the more complex algorithms can provide great reductions in computational cost relative to simpler methods (e.g., Cartesian grids with explicit time integration) that are easier to mate efficiently to hardware. In this chapter, we not only describe our approach but we also address the fact that while complex algorithms may never be as efficient flop-for-flop as simple methods, in the important measure of science-per-core-hour, they can still win big by making complex features like adaptivity and implicit methods as efficient and scalable as possible.

42 ENGINEERING↗
DOE OSTI2016

The Livermore Brain: Massive Deep Learning Networks Enabled by High Performance Computing

The proliferation of inexpensive sensor technologies like the ubiquitous digital image sensors has resulted in the collection and sharing of vast amounts of unsorted and unexploited raw data. Companies and governments who are able to collect and make sense of large datasets to help them make better decisions more rapidly will have a competitive advantage in the information era. Machine Learning technologies play a critical role for automating the data understanding process; however, to be maximally effective, useful intermediate representations of the data are required. These representations or “features” are transformations of the raw data into a form where patterns are more easily recognized. Recent breakthroughs in Deep Learning have made it possible to learn these features from large amounts of labeled data. The focus of this project is to develop and extend Deep Learning algorithms for learning features from vast amounts of unlabeled data and to develop the HPC neural network training platform to support the training of massive network models. This LDRD project succeeded in developing new unsupervised feature learning algorithms for images and video and created a scalable neural network training toolkit for HPC. Additionally, this LDRD helped create the world’s largest freely-available image and video dataset supporting open multimedia research and used this dataset for training our deep neural networks. This research helped LLNL capture several work-for-others (WFO) projects, attract new talent, and establish collaborations with leading academic and commercial partners. Finally, this project demonstrated the successful training of the largest unsupervised image neural network using HPC resources and helped establish LLNL leadership at the intersection of Machine Learning and HPC research.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2017

Extreme Scale Unstructured Adaptive CFD: From Multiphase Flow to Aerodynamic Flow Control

Understanding the flow of fluid, either liquid or gas, through and around solid bodies has challenged man since the dawn of scientific inquiry. Many of the great minds of science and math have progressively built up a hierarchy of fluid models. This report is concerned with the computational modeling of turbulent flow around aerodynamic bodies such as planes and wind turbines. In this case, viscous effects near the solid bodies create very thin boundary layers that yield highly anisotropic (gradients normal to the surface may be 10^6 times larger than gradients along the surface) solutions to the governing non-linear partial differential equations (PDE); the Navier-Stokes equations. Furthermore, turbulent flows develop extremely broad ranges of length and time scales.This disparity motivates the use of discretization methods capable of employing adaptivity and implicit time integration. The combination of these features (non-linear, anisotropy, adaptivity, andimplicit) dramatically raise the complexity of the discretization, posing large challenges to efficient scalable parallel implementation. However, through careful design, the more complex algorithms can provide great reductions in computational cost relative to simpler methods (e.g., Cartesian grids with explicit time integration) that are easier to mate efficiently to hardware. In this report, we not only describe our approach but we also address the fact that while complex algorithms may never be as efficient flop-for-flop as simple methods, in the important measure of science-per-core-hour, they can still win big by making complex features like adaptivity and implicit methods as efficient and scalable as possible.

42 ENGINEERING↗
DOE OSTI2024

Generating An Advanced Cross-section Library For HTGR Pebble Bed Depletion Calculations Using Reduced-Order Model Generation Techniques

For code development, Advanced Reactor Technologies - Gas Cooled Reactors Program (ART-GCR) rely on a collaboration with the Nuclear Energy Advanced Modeling and Simulation (NEAMS) program, but the cross sections generation and the methodology definition is part of this program area goals. Based on previous studies in FY23, the size of microscopic cross section libraries increases rapidly with the number of tabulations, requiring significant amount of memory and drastically slowing down the Griffin calculations when evaluating cross sections via the multivariate linear interpolation approach. Rising to these challenges, this work investigates constructing Reduced-order Models (ROMs) for the multi-group microscopic cross sections to accelerate the cross section evaluation in Griffin. A database of multigroup cross sections is first collected considering all possible parameters that a designer could change for optimization. Down-selection of the ROM techniques afterward shows Deep Neural Network (DNN) as the best candidate when jointly consider memory efficiency, predictive accuracy, computational cost, scalability, flexibility and ease of implementation of the algorithms in comparison to the multidimensional interpolation. This work develops a specific interface that enables the cross section predictions using pre-trained DNN models into Griffin leveraging the existing ROM capabilities. DNNs have been trained for all isotopes for use in Griffin. Preliminary Griffin testing shows that DNNs exhibit exceptional predictive accuracy and the use of DNNs provides orders of magnitude improvement in memory efficiency compared to conventional interpolation techniques. With such ROM techniques, it holds great promise to further increase the fidelity of the Pebble Bed Reactor (PBR) simulation by increasing the number of tabulations/state variables during cross section evaluation, while maintaining the computational cost affordable in Griffin.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗
NASA NTRSDate not supplied

Trend Analysis of AI/ML Tools and Services in NASA

Usage of Machine Learning (ML) algorithms within NASA’s Science Mission Directorates have been increasing over the years. This can be quantitatively observed in the upward trends of ML usage found by analyzing the publications and presentations (in affiliation with NASA) available through NASA Technical Reports Server (NTRS) and PubMed Central(PMC). Identifying the problem types and class of ML algorithms used to tackle them across the divisions can present opportunities for collaborations, interdisciplinary projects and knowledge transfer for sustainable partnerships. In this presentation, we will present the trend analysis of ML algorithms used in different SMD divisions based on the publications and presentations publicly available. We identify these trends by leveraging ML algorithms which are able to search through the publication texts semantically; which are also highly scalable. We will also present an analysis on the available opensource tools and services in NASA leveraging AI/ML algorithms. This work will provide ample avenues for collaborative efforts across different disciplines based on the surfaced trends.

Slesa Adhikari↗
DOE OSTI2010

Parallel octree-based hexahedral mesh generation for eulerian to lagrangian conversion.

Computational simulation must often be performed on domains where materials are represented as scalar quantities or volume fractions at cell centers of an octree-based grid. Common examples include bio-medical, geotechnical or shock physics calculations where interface boundaries are represented only as discrete statistical approximations. In this work, we introduce new methods for generating Lagrangian computational meshes from Eulerian-based data. We focus specifically on shock physics problems that are relevant to ASC codes such as CTH and Alegra. New procedures for generating all-hexahedral finite element meshes from volume fraction data are introduced. A new primal-contouring approach is introduced for defining a geometric domain. New methods for refinement, node smoothing, resolving non-manifold conditions and defining geometry are also introduced as well as an extension of the algorithm to handle tetrahedral meshes. We also describe new scalable MPI-based implementations of these procedures. We describe a new software module, Sculptor, which has been developed for use as an embedded component of CTH. We also describe its interface and its use within the mesh generation code, CUBIT. Several examples are shown to illustrate the capabilities of Sculptor.

99 GENERAL AND MISCELLANEOUS↗
DOE OSTI2023

hIPPYlib-MUQ: A Bayesian Inference Software Framework for Integration of Data with Complex Predictive Models under Uncertainty

Bayesian inference provides a systematic framework for integration of data with mathematical models to quantify the uncertainty in the solution of the inverse problem. However, the solution of Bayesian inverse problems governed by complex forward models described by partial differential equations (PDEs) remains prohibitive with black-box Markov chain Monte Carlo (MCMC) methods. We present hIPPYlib-MUQ, an extensible and scalable software framework that contains implementations of state-of-the art algorithms aimed to overcome the challenges of high-dimensional, PDE-constrained Bayesian inverse problems. These algorithms accelerate MCMC sampling by exploiting the geometry and intrinsic low-dimensionality of parameter space via derivative information and low rank approximation. The software integrates two complementary open-source software packages, hIPPYlib and MUQ. hIPPYlib solves PDE-constrained inverse problems using automatically-generated adjoint-based derivatives, but it lacks full Bayesian capabilities. MUQ provides a spectrum of powerful Bayesian inversion models and algorithms, but expects forward models to come equipped with gradients and Hessians to permit large-scale solution. By combining these two complementary libraries, we created a robust, scalable, and efficient software framework that realizes the benefits of each and allows us to tackle complex large-scale Bayesian inverse problems across a broad spectrum of scientific and engineering disciplines. To illustrate the capabilities of hIPPYlib-MUQ, we present a comparison of a number of MCMC methods available in the integrated software on several high-dimensional Bayesian inverse problems. These include problems characterized by both linear and nonlinear PDEs, various noise models, and different parameter dimensions. The results demonstrate that large (~ 50×) speedups over conventional black box and gradient-based MCMC algorithms can be obtained by exploiting Hessian information (from the log-posterior), underscoring the power of the integrated hIPPYlib-MUQ framework.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2022

PowerNet: Multi-agent Deep Reinforcement Learning for Scalable Powergrid Control

This paper develops an efficient multi-agent deep reinforcement learning algorithm for cooperative controls in powergrids. Specifically, we consider the decentralized inverter-based secondary voltage control problem in distributed generators (DGs), which is first formulated as a cooperative multi-agent reinforcement learning (MARL) problem. We then propose a novel on-policy MARL algorithm, PowerNet, in which each agent (DG) learns a control policy based on (sub-)global reward but local states and encoded communication messages from its neighbors. Motivated by the fact that a local control from one agent has limited impact on agents distant from it, we exploit a novel spatial discount factor to reduce the effect from remote agents, to expedite the training process and improve scalability. Furthermore, a differentiable, learning-based communication protocol is employed to foster the collaborations among neighboring agents. In addition, to mitigate the effects of system uncertainty and random noise introduced during on-policy learning, we utilize an action smoothing factor to stabilize the policy execution. To facilitate training and evaluation, we develop PGSim, an efficient, high-fidelity powergrid simulation platform. Here, experimental results in two microgrid setups show that the developed PowerNet outperforms the conventional model-based control method, as well as several state-of-the-art MARL algorithms. The decentralized learning scheme and high sample efficiency also make it viable to large-scale power grids.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2013

Developing highly scalable fluid solvers for enabling multiphysics simulation.

We performed an investigation into explicit algorithms for the simulation of incompressible flows using methods with a finite, but small amount of compressibility added. Such methods include the artificial compressibility method and the lattice-Boltzmann method. The impetus for investigating such techniques stems from the increasing use of parallel computation at all levels (processors, clusters, and graphics processing units). Explicit algorithms have the potential to leverage these resources. In our investigation, a new form of artificial compressibility was derived. This method, referred to as the Entropically Damped Artificial Compressibility (EDAC) method, demonstrated superior results to traditional artificial compressibility methods by damping the numerical acoustic waves associated with these methods. Performance nearing that of the lattice- Boltzmann technique was observed, without the requirement of recasting the problem in terms of particle distribution functions; continuum variables may be used. Several example problems were investigated using a finite-di erence and finite-element discretizations of the EDAC equations. Example problems included lid-driven cavity flow, a convecting Taylor-Green vortex, a doubly periodic shear layer, freely decaying turbulence, and flow over a square cylinder. Additionally, a scalability study was performed using in excess of one million processing cores. Explicit methods were found to have desirable scaling properties; however, some robustness and general applicability issues remained.

Clausen, Jonathan R↗
DOE OSTI2016

Parallel FETI‐DP algorithm for efficient simulation of large‐scale EM problems

Summary An efficient parallelization of the dual‐primal finite‐element tearing and interconnecting (FETI‐DP) algorithm is presented for large‐scale electromagnetic simulations. As a nonoverlapping domain decomposition method, the FETI‐DP algorithm formulates a global interface problem, whose iterative solution is accelerated with a solution of a global corner problem. To achieve a good load balance for parallel computation, the original computational domain is decomposed into subdomains with similar sizes and shapes. The subdomains are then distributed to processors based on their close proximity to minimize inter‐processor communication. The parallel generalized minimal residual method, enhanced with the iterative classical Gram‐Schmidt orthogonalization scheme to reduce global communication, is adopted to solve the global interface problem with a fast convergence rate. The global corner‐related coarse problem is solved iteratively with a parallel communication‐avoiding biconjugate gradient stabilized method to minimize global communication, and its convergence is accelerated by a diagonal preconditioner constructed from the coarse system matrix. To alleviate neighboring communication overhead, the non‐blocking communication approach is employed in both generalized minimal residual and communication‐avoiding biconjugate gradient stabilized iterative solutions. Three numerical examples are presented to demonstrate the accuracy, scalability, and capability of the proposed parallel FETI‐DP algorithm for electromagnetic modeling of general objects and antenna arrays. Copyright © 2016 John Wiley & Sons, Ltd.

Zhang, Kedi↗
NASA NTRS2002

Field-Programmable Gate Array Computer in Structural Analysis: An Initial Exploration

This paper reports on an initial assessment of using a Field-Programmable Gate Array (FPGA) computational device as a new tool for solving structural mechanics problems. A FPGA is an assemblage of binary gates arranged in logical blocks that are interconnected via software in a manner dependent on the algorithm being implemented and can be reprogrammed thousands of times per second. In effect, this creates a computer specialized for the problem that automatically exploits all the potential for parallel computing intrinsic in an algorithm. This inherent parallelism is the most important feature of the FPGA computational environment. It is therefore important that if a problem offers a choice of different solution algorithms, an algorithm of a higher degree of inherent parallelism should be selected. It is found that in structural analysis, an 'analog computer' style of programming, which solves problems by direct simulation of the terms in the governing differential equations, yields a more favorable solution algorithm than current solution methods. This style of programming is facilitated by a 'drag-and-drop' graphic programming language that is supplied with the particular type of FPGA computer reported in this paper. Simple examples in structural dynamics and statics illustrate the solution approach used. The FPGA system also allows linear scalability in computing capability. As the problem grows, the number of FPGA chips can be increased with no loss of computing efficiency due to data flow or algorithmic latency that occurs when a single problem is distributed among many conventional processors that operate in parallel. This initial assessment finds the FPGA hardware and software to be in their infancy in regard to the user conveniences; however, they have enormous potential for shrinking the elapsed time of structural analysis solutions if programmed with algorithms that exhibit inherent parallelism and linear scalability. This potential warrants further development of FPGA-tailored algorithms for structural analysis.

Singleterry, Robert C., Jr.↗
NASA NTRS2023

Easy, Scalable Subsetting of GEDI Point Clouds

The GEDI Subsetter, a Python tool developed for NASA’s Multi-mission Algorithm and Analysis Platform (MAAP), optimizes the accessibility and visualization of GEDI point clouds by enabling users to efficiently subset data in a convenient, scalable manner. Complex science data often requires users to learn new software skills and handle many large files. Handling and cleaning large data sets is tedious and error-prone. These challenges significantly impede analysis. One of the goals of NASA's MAAP is to provide a platform that lowers the barrier to conducting research and analysis at scale. When a group of MAAP users wanted to conduct above-ground biomass estimation using GEDI data, we found that their existing workflow for leveraging GEDI data suffered from the barriers mentioned above. Furthermore, their workflow did not scale easily beyond a small number of granules. We found that existing tools related to GEDI data retrieval and subsetting were too limiting, so the GEDI Subsetter was written to support MAAP users’ needs. Being able to run many subsetting jobs simultaneously in the MAAP, and parallelizing the code itself, has led to significant speed improvements in obtaining relevant data, reducing subsetting time from hours to minutes. MAAP users can now more quickly and easily obtain only the data relevant to their research, by choosing which GEDI collection they want to work with (L1A, L2A, L2B, or L4A), and how they want to subset it, by specifying an area of interest, a temporal range, and relevant attributes. This has significantly reduced the feedback loop for users, allowing them to much more quickly subset GEDI data and begin their analysis. Although the GEDI Subsetter originally targeted users of the MAAP, it is generalized such that it can also be used outside of the MAAP and includes a command-line interface for convenience. Furthermore, with minor modifications, it should be possible to use it with non-GEDI data as the general pattern should be applicable to other sparse/track-based sensors.

Charles Daniels↗
DOE OSTI2021

Scalable Pattern Matching in Metadata Graphs via Constraint Checking

Pattern matching is a fundamental tool for answering complex graph queries. Unfortunately, existing solutions have limited capabilities: They do not scale to process large graphs and/or support only a restricted set of search templates or usage scenarios. Moreover, the algorithms at the core of the existing techniques are not suitable for today’s graph processing infrastructures relying on horizontal scalability and shared-nothing clusters, as most of these algorithms are inherently sequential and difficult to parallelize. In this article we present an algorithmic pipeline that bases pattern matching on constraint checking. The key intuition is that each vertex and edge participating in a match has to meet a set of constraints implicitly specified by the search template. These constraints can be verified independently and typically are less expensive to compute than searching the full template. The pipeline we propose generates these constraints and iterates over them to eliminate all the vertices and edges that do not participate in any match, thus reducing the background graph to a subgraph that is the union of all template matches—the complete set of all vertices and edges that participate in at least one match. Additional analysis can be performed on this annotated, reduced graph, such as full match enumeration, match counting, or computing vertex/edge centrality. Furthermore, a vertex-centric formulation for constraint checking algorithms exists, and this makes it possible to harness existing high-performance, vertex-centric graph processing frameworks. This technique (i) enables highly scalable pattern matching in metadata (labeled) graphs; (ii) supports arbitrary patterns with 100% precision; (iii) enables tradeoffs between precision and time-to-solution, while always selects all vertices and edges that participate in matches, thus offering 100% recall; and (iv) supports a set of popular data analytics scenarios. We implement our approach on top of HavoqGT, an open-source asynchronous graph processing framework, and demonstrate its advantages through strong and weak scaling experiments on massive scale real-world (up to 257 billion edges) and synthetic (up to 4.4 trillion edges) labeled graphs, respectively, and at scales (1,024 nodes / 36,864 cores), orders of magnitude larger than used in the past for similar problems. This article serves two purposes: First, it synthesises the knowledge accumulated during a long-term project. Second, it presents new system features, usage scenarios, optimizations, and comparisons with related work that strengthen the confidence that pattern matching based on iterative pruning via constraint checking is an effective and scalable approach in practice. The new contributions include the following: (i) We demonstrate the ability of the constraint checking approach to efficiently support two additional search scenarios that often emerge in practice, interactive incremental search and exploratory search. (ii) We empirically compare our solution with two additional state-of-the-art systems, Arabsque and TriAD. (iii) We show the ability of our solution to accommodate a more diverse range of datasets with varying properties, e.g., scale, skewness, label distribution, and match frequency. (iv) We introduce or extend a number of system features (e.g., work aggregation, load balancing, and the ability to cap the generated traffic) and design optimizations and demonstrate their advantages with respect to improving performance and scalability. (v) We present bottleneck analysis and insights into artifacts that influence performance. (vi) We present a theoretical complexity argument that motivates the performance gains we observe.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2016

CLUSTERING-BASED FEATURE LEARNING ON VARIABLE STARS

The success of automatic classification of variable stars depends strongly on the lightcurve representation. Usually, lightcurves are represented as a vector of many descriptors designed by astronomers called features. These descriptors are expensive in terms of computing, require substantial research effort to develop, and do not guarantee a good classification. Today, lightcurve representation is not entirely automatic; algorithms must be designed and manually tuned up for every survey. The amounts of data that will be generated in the future mean astronomers must develop scalable and automated analysis pipelines. In this work we present a feature learning algorithm designed for variable objects. Our method works by extracting a large number of lightcurve subsequences from a given set, which are then clustered to find common local patterns in the time series. Representatives of these common patterns are then used to transform lightcurves of a labeled set into a new representation that can be used to train a classifier. The proposed algorithm learns the features from both labeled and unlabeled lightcurves, overcoming the bias using only labeled data. We test our method on data sets from the Massive Compact Halo Object survey and the Optical Gravitational Lensing Experiment; the results show that our classification performance is as good as and in some cases better than the performance achieved using traditional statistical features, while the computational cost is significantly lower. With these promising results, we believe that our method constitutes a significant step toward the automation of the lightcurve classification pipeline.

79 ASTRONOMY AND ASTROPHYSICS↗