Search NASA⌕ Search

SEARCH · Search NASA

Results for “query optimization”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records
All sourcesNASA NTRSDOE OSTI
DOE OSTI2018

Apache Calcite: A Foundational Framework for Optimized Query Processing Over Heterogeneous Data Sources

Apache Calcite is a foundational software framework that provides query processing, optimization, and query language support to many popular open-source data processing systems such as Apache Hive, Apache Storm, Apache Flink, Druid, and MapD. The goal of this paper is to formally introduce Calcite to the broader research community, brie y present its history, and describe its architecture, features, functionality, and patterns for adoption. Calcite's architecture consists of a modular and extensible query optimizer with hundreds of built-in optimization rules, a query processor capable of processing a variety of query languages, an adapter architecture designed for extensibility, and support for heterogeneous data models and stores (relational, semi-structured, streaming, and geospatial). This exible, embeddable, and extensible architecture is what makes Calcite an attractive choice for adoption in big-data frameworks. It is an active project that continues to introduce support for the new types of data sources, query languages, and approaches to query processing and optimization.

Begoli, Edmon↗
DOE OSTI2015

Query Optimization for Graph Analytics on Linked Data Using SPARQL

Triplestores that support query languages such as SPARQL are emerging as the preferred and scalable solution to represent data and meta-data as massive heterogeneous graphs using Semantic Web standards. With increasing adoption, the desire to conduct graph-theoretic mining and exploratory analysis has also increased. Addressing that desire, this paper presents a solution that is the marriage of Graph Theory and the Semantic Web. We present software that can analyze Linked Data using graph operations such as counting triangles, finding eccentricity, testing connectedness, and computing PageRank directly on triple stores via the SPARQL interface. We describe the process of optimizing performance of the SPARQL-based implementation of such popular graph algorithms by reducing the space-overhead, simplifying iterative complexity and removing redundant computations by understanding query plans. Our optimized approach shows significant performance gains on triplestores hosted on stand-alone workstations as well as hardware-optimized scalable supercomputers such as the Cray XMT.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗
DOE OSTI2017

Optimizing the query performance of block index through data analysis and I/O modeling

Indexing technique has become an efficient tool to enable scientists to directly access the most relevant data records. But, the time and space requirements of building and storing indexes are expensive in the traditional approaches, such as R-tree and bitmaps. Recently, we started to address this issue by using the idea of "block index", and our previous work has shown promising results from comparing it against other well-known solutions, including ADIOS, SciDB, and FastBit. In this work, we further improve the technique from both theoretical and implementation perspectives. Driven by an extensive effort in characterizing scientific datasets and modeling I/O systems, we presented a theoretical model to analyze its query performance with respect to a given block size configuration. We also introduced three optimization techniques to achieve a 2.3x query time reduction comparing to the original implementation.

Wu, Tzuhsien↗
DOE OSTI2023

Leveraging Application Data Constraints to Optimize Database-Backed Web Applications

Exploiting the relationships among data is a classical query optimization technique. As persistent data is increasingly being created and maintained programmatically, prior work that infers data relationships from data statistics misses an important opportunity. We present Coco, the first tool that identifies data relationships by analyzing database-backed applications. Once identified, Coco leverages the constraints to optimize the application's physical design and query execution. Instead of developing a fixed set of predefined rewriting rules, Coco employs an enumerate-test-verify technique to automatically exploit the discovered data constraints to improve query execution. Each resulting rewrite is provably equivalent to the original query. Using 14 real-world web applications, our experiments show that Coco can discover numerous data constraints from code analysis and improve real-world application performance significantly.

Computer Science↗
DOE OSTI2021

ParChain: a framework for parallel hierarchical agglomerative clustering using nearest-neighbor chain

This paper studies the hierarchical clustering problem, where the goal is to produce a dendrogram that represents clusters at varying scales of a data set. We propose the ParChain framework for designing parallel hierarchical agglomerative clustering (HAC) algorithms, and using the framework we obtain novel parallel algorithms for the complete linkage, average linkage, and Ward's linkage criteria. Compared to most previous parallel HAC algorithms, which require quadratic memory, our new algorithms require only linear memory, and are scalable to large data sets. ParChain is based on our parallelization of the nearest-neighbor chain algorithm, and enables multiple clusters to be merged on every round. We introduce two key optimizations that are critical for efficiency: a range query optimization that reduces the number of distance computations required when finding nearest neighbors of clusters, and a caching optimization that stores a subset of previously computed distances, which are likely to be reused. Experimentally, we show that our highly-optimized implementations using 48 cores with two-way hyper-threading achieve 5.8--110.1x speedup over state-of-the-art parallel HAC algorithms and achieve 13.75--54.23x self-relative speedup. Compared to state-of-the-art algorithms, our algorithms require up to 237.3x less space. Our algorithms are able to scale to data set sizes with tens of millions of points, which existing algorithms are not able to handle.

Computer Science↗
DOE OSTI2024

QED: A Powerful Query Equivalence Decider for SQL

Checking query equivalence is of great significance in database systems. Prior work in automated query equivalence checking sets the first steps in formally modeling and reasoning about query optimization rules, but only supports a limited number of query features. In this paper, we present Qed, a new framework for query equivalence checking based on bag semantics. Qed uses a new formalism called Q-expressions that models queries using different normal forms for efficient equivalence checking, and models features such as integrity constraints and NULLs in a principled way unlike prior work. Our formalism also allows us to define a new query fragment that encompasses many real-world queries with a complete equivalence checking algorithm, assuming a complete first-order theory solver. Empirically, Qed can verify 299 out of 444 query pairs extracted from the Calcite framework and 979 out of 1287 query pairs extracted from CockroachDB, which is more than 2× the number of cases proven by prior state-of-the-art solver.

Computer Science↗
DOE OSTI2023

CEDS Differential Privacy (CEDSDP) v0.1

A Python package that provides differentially private queries optimized for energy systems' data. It may be used to publish queries such as clustering, averaging, metadata inference, etc. that are useful for a variety of grid-related analytics, including cyberattack detection.

Peisert, Sean↗
DOE OSTI2022

Scalable and Energy-Efficient Methods for Interactive Exploration of Scientific Data

The main scientific contributions of this project are the following novel concepts for multidimensional arrays: shape-based similarity join (SIGMOD 2016), incremental view maintenance (SIGMOD 2017), user-defined stencil functions (HPDC 2017), and distributed caching for in-situ processing (SSDBM 2018). Building on our collaboration with the astrophysics group at LBNL, we applied these techniques to the data generated in the Palomar Transient Factory (PTF) astronomical survey. They played a pivotal role in the first-ever observation of a neutron star merger, which produces gravitational waves and turns out to be the origin of heavy elements, including gold. This has lead to a Science magazine article that has received extensive media coverage on ACM TechNews, Slashdot, FiveThirtyEight, and Quanta Magazine, among others. Additionally, two other articles detailing related aspects of the same discovery have been published in the Astrophysical Journal Letters journal. These publications have more than 3,000 citations according to Google Scholar (as of February 2022). This cross-disciplinary collaboration provided very good opportunities to apply database techniques to real-life scientific problems. The fact that they facilitated major discoveries in astrophysics proves the importance of our research. In addition to the work on multidimensional array databases, this project has also developed stochastic gradient descent (SGD) optimization algorithms for training large scale machine learning models, methods for querying in-situ data, and a database query optimizer based on sketch synopses.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2015

Knowledge management information tool analytics with distributed database engine - 15737

The D and D KM-IT is a web-based knowledge management information tool custom built for the deactivation and decommissioning (D and D) user community. The system allows interested users to post information which goes through a custom work flow. To better improve the performance and user experience, the analysis of server logs is being conducted using MongoDB. MongoDB is a document database that provides high performance, high availability, and easy scalability. Objectives: - Feasibility study of Distributed Database Engine, MongoDB for web analytics; - Analyze server logs to evaluate D and D KM-IT module usage; - Generate useful information from the analysis. Method: - The server logs were loaded and stored in MongoDB for analysis; - Analysis and visualization was performed using NodeJS and Java Script library. Results: - It was possible to store log files in MongoDB; - Log files with different time stamps and groups from different servers can be aggregated; - Queries can be written to extract meaningful information from aggregated log files (e.g. users from different countries/organizations). Conclusions: - MongoDB is useful for server log analytics as the analyst can immediately access, analyze, and visualize MongoDB data based on the requirements; - The information can be extracted to develop custom reports; - Optimized queries can be run for fast data access. Path forward: - Custom queries need to be developed to extract information leading to improved user information.

12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗
DOE OSTI2024

Learning Quantum States and Unitaries of Bounded Gate Complexity

While quantum state tomography is notoriously hard, most states hold little interest to practically minded tomographers. Given that states and unitaries appearing in nature are of bounded gate complexity, it is natural to ask if efficient learning becomes possible. In this work, we prove that to learn a state generated by a quantum circuit with G two-qubit gates to a small trace distance, a sample complexity scaling linearly in G is necessary and sufficient. We also prove that the optimal query complexity to learn a unitary generated by G gates to a small average-case error scales linearly in G . While sample-efficient learning can be achieved, we show that under reasonable cryptographic conjectures, the computational complexity for learning states and unitaries of gate complexity G must scale exponentially in G . We illustrate how these results establish fundamental limitations on the expressivity of quantum machine-learning models and provide new perspectives on no-free-lunch theorems in unitary learning. Together, our results answer how the complexity of learning quantum states and unitaries relate to the complexity of creating these states and unitaries. Published by the American Physical Society 2024

Zhao, Haimeng (ORCID:0000000166751489)↗
DOE OSTI2015

Semiautomated head-and-neck IMRT planning using dose warping and scaling to robustly adapt plans in a knowledge database containing potentially suboptimal plans

Purpose: Prior work by the authors and other groups has studied the creation of automated intensity modulated radiotherapy (IMRT) plans of equivalent quality to those in a patient database of manually created clinical plans; those database plans provided guidance on the achievable sparing to organs-at-risk (OARs). However, in certain sites, such as head-and-neck, the clinical plans may not be sufficiently optimized because of anatomical complexity and clinical time constraints. This could lead to automated plans that suboptimally exploit OAR sparing. This work investigates a novel dose warping and scaling scheme that attempts to reduce effects of suboptimal sparing in clinical database plans, thus improving the quality of semiautomated head-and-neck cancer (HNC) plans. Methods: Knowledge-based radiotherapy (KBRT) plans for each of ten “query” patients were semiautomatically generated by identifying the most similar “match” patient in a database of 103 clinical manually created patient plans. The match patient’s plans were adapted to the query case by: (1) deforming the match beam fluences to suit the query target volume and (2) warping the match primary/boost dose distribution to suit the query geometry and using the warped distribution to generate query primary/boost optimization dose-volume constraints. Item (2) included a distance scaling factor to improve query OAR dose sparing with respect to the possibly suboptimal clinical match plan. To further compensate for a component plan of the match case (primary/boost) not optimally sparing OARs, the query dose volume constraints were reduced using a dose scaling factor to be the minimum from either (a) the warped component plan (primary or boost) dose distribution or (b) the warped total plan dose distribution (primary + boost) scaled in proportion to the ratio of component prescription dose to total prescription dose. The dose-volume constraints were used to plan the query case with no human intervention to adjust constraints during plan optimization. Results: KBRT and original clinical plans were dosimetrically equivalent for parotid glands (mean/median doses), spinal cord, and brainstem (maximum doses). KBRT plans significantly reduced larynx median doses (21.5 ± 6.6 Gy to 17.9 ± 3.9 Gy), and oral cavity mean (32.3 ± 6.2 Gy to 28.9 ± 5.4 Gy) and median (28.7 ± 5.7 Gy to 23.2 ± 5.3 Gy) doses. Doses to ipsilateral parotid gland, larynx, oral cavity, and brainstem were lower or equivalent in the KBRT plans for the majority of cases. By contrast, KBRT plans generated without the dose warping and dose scaling steps were not significantly different from the clinical plans. Conclusions: Fast, semiautomatically generated HNC IMRT plans adapted from existing plans in a clinical database can be of equivalent or better quality than manually created plans. The reductions in OAR doses in the semiautomated plans, compared to the clinical plans, indicate that the proposed dose warping and scaling method shows promise in mitigating the impact of suboptimal clinical plans.

60 APPLIED LIFE SCIENCES↗
DOE OSTI2017

HodDB: Design and Analysis of a Query Processor for Brick.

Brick is a recently proposed metadata schema and ontology for describing building components and the relationships between them. It represents buildings as directed labeled graphs using the RDF data model. Using the SPARQL query language, building-agnostic applications query a Brick graph to discover the set of resources and relationships they require to operate. Latency-sensitive applications, such as user interfaces, demand response and modelpredictive control, require fast queries — conventionally less than 100ms. We benchmark a set of popular open-source and commercial SPARQL databases against three real Brick models using seven application queries and find that none of them meet this performance target. This lack of performance can be attributed to design decisions that optimize for queries over large graphs consisting of billions of triples, but give poor spatial locality and join performance on the small dense graphs typical of Brick. We present the design and evaluation of HodDB, a RDF/SPARQL database for Brick built over a node-based index structure. HodDB performs Brick queries 3-700x faster than leading SPARQL databases and consistently meets the 100ms threshold, enabling the portability of important latency-sensitive building applications.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
DOE OSTI2023

Optimal adjustment sets for causal query estimation in partially observed biomolecular networks

Abstract Causal query estimation in biomolecular networks commonly selects a ‘valid adjustment set’, i.e. a subset of network variables that eliminates the bias of the estimator. A same query may have multiple valid adjustment sets, each with a different variance. When networks are partially observed, current methods use graph-based criteria to find an adjustment set that minimizes asymptotic variance. Unfortunately, many models that share the same graph topology, and therefore same functional dependencies, may differ in the processes that generate the observational data. In these cases, the topology-based criteria fail to distinguish the variances of the adjustment sets. This deficiency can lead to sub-optimal adjustment sets, and to miss-characterization of the effect of the intervention. We propose an approach for deriving ‘optimal adjustment sets’ that takes into account the nature of the data, bias and finite-sample variance of the estimator, and cost. It empirically learns the data generating processes from historical experimental data, and characterizes the properties of the estimators by simulation. We demonstrate the utility of the proposed approach in four biomolecular Case studies with different topologies and different data generation processes. The implementation and reproducible Case studies are at https://github.com/srtaheri/OptimalAdjustmentSet.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2015

A Selectivity based approach to Continuous Pattern Detection in Streaming Graphs

Cyber security is one of the most significant technical challenges in current times. Detecting adversarial activities, prevention of theft of intellectual properties and customer data is a high priority for corporations and government agencies around the world. Cyber defenders need to analyze massive-scale, high-resolution network flows to identify, categorize, and mitigate attacks involving networks spanning institutional and national boundaries. Many of the cyber attacks can be described as subgraph patterns, with prominent examples being insider infiltrations (path queries), denial of service (parallel paths) and malicious spreads (tree queries). This motivates us to explore subgraph matching on streaming graphs in a continuous setting. The novelty of our work lies in using the subgraph distributional statistics collected from the streaming graph to determine the query processing strategy. We introduce a ``Lazy Search" algorithm where the search strategy is decided on a vertex-to-vertex basis depending on the likelihood of a match in the vertex neighborhood. We also propose a metric named ``Relative Selectivity" that is used to select between different query processing strategies. Our experiments performed on real online news, network traffic stream and a synthetic social network benchmark demonstrate 10-100x speedups over non-incremental, selectivity agnostic approaches.

dynamic graphs↗
DOE OSTI2023

Characterizing Families of Spectral Similarity Scores and Their Use Cases for Gas Chromatography–Mass Spectrometry Small Molecule Identification

Metabolomics provides a unique snapshot into the world of small molecules and the complex biological processes that govern the human, animal, plant, and environmental ecosystems encapsulated by the One Health modeling framework. However, this “molecular snapshot” is only as informative as the number of metabolites confidently identified within it. The spectral similarity (SS) score is traditionally used to identify compound(s) in mass spectrometry approaches to metabolomics, where spectra are matched to reference libraries of candidate spectra. Unfortunately, there is little consensus on which of the dozens of available SS metrics should be used. This lack of standard SS score creates analytic uncertainty and potentially leads to issues in reproducibility, especially as these data are integrated across other domains. In this work, we use metabolomic spectral similarity as a case study to showcase the challenges in consistency within just one piece of the One Health framework that must be addressed to enable data science approaches for One Health problems. Here, using a large cohort of datasets comprising both standard and complex datasets with expert-verified truth annotations, we evaluated the effectiveness of 66 similarity metrics to delineate between correct matches (true positives) and incorrect matches (true negatives). We additionally characterize the families of these metrics to make informed recommendations for their use. Our results indicate that specific families of metrics (the Inner Product, Correlative, and Intersection families of scores) tend to perform better than others, with no single similarity metric performing optimally for all queried spectra. This work and its findings provide an empirically-based resource for researchers to use in their selection of similarity metrics for GC-MS identification, increasing scientific reproducibility through taking steps towards standardizing identification workflows.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2024

Two-Stage Estimation and Variance Modeling for Latency-Constrained Variational Quantum Algorithms

The quantum approximate optimization algorithm (QAOA) has enjoyed increasing attention in noisy, intermediate-scale quantum computing with its application to combinatorial optimization problems. QAOA has the potential to demonstrate a quantum advantage for NP-hard combinatorial optimization problems. As a hybrid quantum-classical algorithm, the classical component of QAOA resembles a simulation optimization problem in which the simulation outcomes are attainable only through a quantum computer. The simulation that derives from QAOA exhibits two unique features that can have a substantial impact on the optimization process: (i) the variance of the stochastic objective values typically decreases in proportion to the optimality gap, and (ii) querying samples from a quantum computer introduces an additional latency overhead. In this paper, we introduce a novel stochastic trust-region method derived from a derivative-free, adaptive sampling trust-region optimization method intended to efficiently solve the classical optimization problem in QAOA by explicitly taking into account the two mentioned characteristics. The key idea behind the proposed algorithm involves constructing two separate local models in each iteration: a model of the objective function and a model of the variance of the objective function. Exploiting the variance model allows us to restrict the number of communications with the quantum computer and also helps navigate the nonconvex objective landscapes typical in QAOA optimization problems. In conclusion, we numerically demonstrate the superiority of our proposed algorithm using the SimOpt library and Qiskit when we consider a metric of computational burden that explicitly accounts for communication costs.

Derivative-free Optimization↗
DOE OSTI2022

Stress-constrained topology optimization of lattice-like structures using component-wise reduced order models

We report lattice-like structures can provide a combination of high stiffness with light weight that is useful in many applications, but a resolved finite element mesh of such structures results in a computationally expensive discretization. This computational expense may be particularly burdensome in many-query applications, such as optimization. We develop a stress-constrained topology optimization method for lattice-like structures that uses component-wise reduced order models as a cheap surrogate, providing accurate computation of stress fields while greatly reducing run time relative to a full order model. We demonstrate the ability of our method to produce large reductions in mass while respecting a constraint on the maximum stress in a pair of test problems. The ROM methodology provides a speedup of about 150x in forward solves compared to full order static condensation and provides a relative error of less than 5% in the relaxed stress.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2025

Quantum Algorithm for Linear Non-unitary Dynamics with Near-Optimal Dependence on All Parameters

We introduce a family of identities that express general linear non-unitary evolution operators as a linear combination of unitary evolution operators, each solving a Hamiltonian simulation problem. This formulation can exponentially enhance the accuracy of the recently introduced linear combination of Hamiltonian simulation (LCHS) method [An, Liu, and Lin, Physical Review Letters, 2023]. For the first time, this approach enables quantum algorithms to solve linear differential equations with both optimal state preparation cost and near-optimal scaling in matrix queries on all parameters.

Applied Dynamical Systems↗