Search NASA⌕ Search

SEARCH · Search NASA

Results for “Graph databases”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 37 records · Page 2
All sourcesNASA NTRSDOE OSTI
DOE OSTI2019

A Semantic Search Capability for a Grid Model Repository

The operation of electric grids is becoming more challenging due to continued increases in wind and solar generation, the proliferation of decentralization generation resources, and cyber security threats. To support the needed research, new large scale realistic public grid models have been developed and placed in a repository for access by researchers. To assist researchers in easily finding grid models that are suitable for their specific research needs, an approach to searching numerous large grid models was evaluated, prototyped and developed. Here, the research identified multiple possible approaches to using graph database techniques and a natural language processing for the search algorithm. Results of the prototype, the final approach for production repository and the search algorithm formulation are described.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2019

SONET: a semantic ontological network graph for managing points of interest data heterogeneity

Scalability, standardization, and management are important issues when working with very large Volunteered Geographic Information (VGI). VGI is a rich and valuable source of Points of Interest (POI) information, but its inherent heterogeneity in content, structure, and scale across sources present major challenges for interlinking data sources for analysis. To be useful at scale, the raw information needs to be transformed into a standardized schema that can be easily and reliably used by data analysts. In this work, we tackle the problem of unifying POI categories (e.g. restaurants, temple, and hotel) across multiple data sources to aid in improving land use maps and population distribution estimation as well as support data analysts wishing to fuse multiple data sources with the OpenStreetMap (OSM) mapping platform or working with projects that are already configured in the OSM schema and wish to add additional sources of information. Graph theory and its implementation through the SONET graph database, provides a programmatic way to organize, store, and retrieve standardized POI categories at multiple levels of abstraction. Additionally, it addresses category heterogeneity across data sources by standardizing and managing categories in a way that makes cross-domain analysis possible.

Palumbo, Rachel↗
DOE OSTI2021

DISCOverflow

Procedure for reverse engineering binary files and storing the result in a graph database structure for later analysis. This is achieved through the use of Python, angr, and OrientDB Community Edition.

Beckman, BryanR↗
DOE OSTI2023

idaholab/cape2stix

This software allows for the conversion, extraction, and transformation of malware behavior data from "Malware Configuration And Payload Extraction" (CAPEv2) sandbox reports, to Structured Threat Information eXpression (STIX). This allows for further analysis to be performed, sharing of threat data, and transit to a graph database.

Cutshaw, Michael [Idaho National Laboratory (INL),↗
DOE OSTI2020

Annotated Translated Disassembled Code

Procedure for generating function/library embedding based on disassembled binary data from a graph database and processing those embedding through supervised machine learning techniques.

Beckman, BryanR.↗
DOE OSTI2022

Schema Grapher

PNNL contributions to project site managed by federal agency. Schema Grapher is an application that takes in CSV datasets and converts them to RDF for loading into a graph database. Schema Grapher is responsible for the extract and transform part of an ETL (i.e., extract, transform, load) pipeline.

Avila, Andrew↗
DOE OSTI2017

Data Structures for Extreme Scale Computing

As computing problems of national importance grow, the government meets the increased demand by funding the development of ever larger systems. The overarching goal of the work supported in part by this grant is to increase efficiency of programming and performing computations on these large computing systems. In past work, we have demonstrated that some of these computations once thought to require expensive hardware designs and/or complex, special-purpose programming may be executed efficiently on low-cost commodity cluster computing systems using a general-purpose “latency-tolerant” programming framework. One important developed application of the ideas underlying this framework is graph database technology supporting social network pattern matching used by US intelligence agencies to more quickly identify potential terrorist threats. This database application has been spun out by the Pacific Northwest National Laboratory, a Department of Energy Laboratory, into a commercial start-up, Trovares Inc. We explore an alternative application of the same underlying ideas to a well-studied challenge arising in engineering: solving unstructured sparse linear equations. Solving these equations is key to predicting the behavior of large electronic circuits before they are fabricated. Predicting that behavior ahead of fabrication means that designs can optimized and errors corrected ahead of the expense of manufacture.

42 ENGINEERING↗
DOE OSTI2022

Illuminating the pathways to carbon liberation: a systems approach to characterizing the consequential unknowns of carbon transformation and loss from thawing permafrost peatlands (Final Report)

The IsoGenie3 Project delivered new systems-level insights into carbon cycling in thawing permafrost landscapes, with an emphasis on methane and carbon dioxide emissions. From >200 samples from the site collected over a decade, co-analyzed for geochemistry and microbiology, the team recovered ~1,500 assembled microbial genomes and ~1,900 viral population genomes, revealing appreciable genetic novelty - from a new highly abundant bacterial phylum, to novel methane consumers and their activities, to rampant viral novelty. IsoGenie3 linked these organisms to carbon compound transformations (which define the cycling of organic matter in soils, and the loss of the greenhouse gases carbon dioxide and methane), and saw that the microbes at each stage of permafrost thaw had different genetic potential to degrade categories of compounds, expressed that genetic potential differently, and actually transformed carbon compounds into greenhouse gases in different ways. IsoGenie 3 identified that some of the thaw-stage differences were due to plant-microbiome relationships; the plant species across the thaw gradient contributed different carbon compounds into the soil, and hosted distinct microbiota (differing among parts of plants as well as species). Lastly, microbes in the saturated post-thaw conditions appeared likely to contribute to the mobilization and toxification of mercury released during thaw. In parallel with ongoing field sampling and analysis, hypotheses arising from field observations were tested via lab incubation experiments. When communities are taken out of their native habitats, they behave differently, and the team first rigorously quantified the magnitude of this effect on microbiome composition and functional capacity, organic matter composition, and gas production; overall the main system processes were maintained in the lab incubations under the conditions tested. Further, the microbial data could inform geochemical reaction network models of those processes. Then, the team ran experiments with additions of compounds, varying temperature, and “live” vs. “dead” peat (the latter having been gamma irradiated, with a few additional variants to control for methodological artifacts). From these, we (a) determined the importance of plant-derived soluble phenolic compounds in bogs’ extraordinary recalcitrance of organic matter, and carbon gas emissions skewed to carbon dioxide; (b) proposed an abiotic ‘tanning’ mechanism, which could contribute to Sphagnum’s inhibitory effect on anaerobic decomposition through alteration of N availability. IsoGenie3 illuminated longer-term and landscape-scale interactions of permafrost thaw and carbon cycling, advancing knowledge of the drivers of methane dynamics not only across in the permafrost-associated peatland (where hydrology and plant communities dictate microbiomes) but also their interconnected lakes (where sediment carbon quality and resident microbiota are determined by position within lake, and lake features). By leveraging observations of site methane dynamics extending well before this project, the team was able to construct a 44-year portrait of the interplay of permafrost thaw, hydrology, vegetation dynamics, and carbon gas emissions, and the doubling of the fully-thawed fens over this time. From the detailed study of this focal site, IsoGenie3 also aimed to improve model representation of these kinds of sites and processes. To improve predictions of methane transformations, we incorporated acetate and isotope dynamics into the ‘DNDC’ biogeochemistry model. In addition, recovered genomes were grouped into ‘functional groups’, i.e. the genomes that perform a specific function of interest, then used to parameterize maximum growth rate and optimum growth temperature (via signatures in their sequence composition) for the BioCrunch model. The BioCrunch model was then in turn used to test the impact of increasing functional resolution of the microbes, on the carbon gas emissions. Lastly for modeling, the ecosys model was parameterized from the microbial and other data, and used to evaluate drivers of e.g. change in methane emissions. Finally, this project also led to the development of a range of new methods and tools, a new metric of organic matter decomposability, as well as a graph-database solution to multidisciplinary data storage and querying. This project’s ongoing analyses at our focal site also contributed to broader advancements in understanding elements of genetic plasticity and methane metabolism, climate change microbiology and community assembly, global peatland geochemistry and Arctic lakes’ roles in climate feedbacks.

54 ENVIRONMENTAL SCIENCES↗
NASA NTRSDate not supplied

Enhancing NASA Earth Science Data Discovery from Scientific Publications

Earth observations from space borne instruments have evolved explosively in the past decades. Following closely are reanalysis systems assimilating model and observational data, yielding even longer records and larger number of variables. Thanks to advances in internet technology, it is now easier than ever to visualize and analyze these data using web interfaces. On the other hand, it also becomes an increasingly daunting task to build upon the existing knowledge published in various peer reviewed sources, and navigate toward the most relevant data, analysis, and visualization. We present an analysis of a subset of publications that utilized a popular visualization web interface at the NASA Goddard Earth Science Data and Information Services Center. Known as "Giovanni", it allows researchers from wide backgrounds to work with hundreds of variables from space observations and assimilation systems. Since coming online more than a decade ago, Giovanni has been credited in more than 100 papers per year, and the total count now is estimated to be nearly 1,500. Many of these papers contain valuable information about when, where and how Giovanni has been used, and hence forge an opportunity to learn and share the knowledge of which variables were used for what research projects. The purpose of our work is to retrieve the information from the papers and organize it as a knowledge repository which links together datasets, variables, places, dates and phenomena all of which reflect the essence of the published research. Since the publications are unstructured texts, we use natural language processing along with machine learning methods in the retrieval process. One of the challenges is deciphering the dataset names, because in many cases researchers refer to variables, rather than the datasets containing them. To constrain the number of terms, we deploy Earth Science ontologies as dictionaries for the term extraction. We demonstrate that storing these terms and underlying ontologies, along with datasets, variables and papers in the knowledge graph database, enables various linkages between all these entities facilitating the data discovery. Thus, we are setting a qualitatively new stage in improvements of web data interfaces, where machine learning techniques are used to establish and optimize usage-based discovery of data.

Irina V Gerasimov↗
NASA NTRSDate not supplied

Modeling Atmospheric Science Knowledge from Research Publications

NASA Earth Science Data Centers contain enormous amounts of remote sensing digital data. It is often a significant challenge for users to find data suitable for their research topic in these vast archives. One of the approaches is the usage-driven dataset discovery, where users seek publications on projects similar to their intended study. For this approach to be effective, users need a clear connection between the underlying data in the publications and the study objectives; this is not often apparent to non-expert users. Tools and methodologies that can help facilitate and organize these connections are therefore valuable for creating improved knowledge mappings, which can be further used by search engines to suggest data or publications best tailored to a user’s specific research goal. As an illustration of these challenges, in this work we focus on the atmospheric chemistry processes related to Earth environmental impacts such as ozone depletion, aerosols, smog formation, acid rain, and radiative forcing. We further limit our study to publications that use data from the Microwave Limb Sounder (MLS) instrument flown on the Aura Earth Observing System. To create knowledge representations of science carried out in these publications, we use existing ontologies such as the Global Change Master Directory (GCMD) and Semantic Web for Earth and Environmental Terminology (SWEET). These ontologies together encompass term dictionaries that include measured variables, names of molecules or radicals, mission and instrument names, locations, action words, among many others. Based on these terms acknowledge graph database was populated with the terms retrieved from scientific publications that study atmospheric chemistry. These databases can be used to further enhance the automation of knowledge discovery and facilitate machine learning and artificial intelligence algorithms or applications. These tools and methods can also be extended to apply to content from other related Earth science domains.

Irina Gerasimov↗
NASA NTRSDate not supplied

Using Machine Learning to Classify Earth Science Articles

Developing a machine learning pipeline to automatic the classification of research article topics, where topics are geophysical descriptors. The resulting classified documents will be added to the Usage-Based Discovery graph database, along with corresponding NASA Earth Science datasets, with the intent of linking the NASA data product with uses of it that can be searched by topic. The project will aid in the following goals (1) creating a wider representation of how and how much NASA Earth Science data products are used, (2) identifying when NASA data products have been used even when they are not clearly cited, and (3) providing the UBD search tool end-user with set of recommended datasets based on a given topic query.

Machine Learning↗
DOE OSTI2023

Digital Twin for Optimizing Real-time Economy of the Integrated Energy Systems

Economic and safe operation of integrated energy systems (IES) requires real-time optimization (RTO) of the control and actions conducted on each system component. In this regard, digital twins (DTs), which consist of a physical system, a virtual system, and the data communication that occurs between the two, are essential for effective RTO. Through the data warehouse, the virtual system is constantly updated with real-time data from the physical system, and functions as the model in the optimization framework. The reduced-order model of the dynamic process model in the virtual system is used in the optimization framework. The optimization results are then returned, via the data warehouse, as control actions to the physical system. This work demonstrates the software capabilities of DT assets for an IES in the context of preparing a DT for an experimental system comprised of Idaho National Laboratory (INL)’s Thermal Energy Delivery System and battery system. For the virtual demonstration, the DTs encompass (1) a physical system, including the Modelica models of the Thermal Energy Delivery System and the battery system; (2) virtual optimization via the Optimization of Real-Time Capacity Allocation (ORCA) platform; and (3) the open-source data warehouse software DeepLynx. This work assesses the performance of ORCA, which utilizes a reduced-order model built using the Risk Analysis Virtual Environment (RAVEN) and trained on the Modelica models and real-time data pipeline through the graph database hosted in DeepLynx. The proposed optimization workflow will be an RTO model based on DTs and the data they generate.

25 ENERGY STORAGE↗
DOE OSTI2020

The IsoGenie database: an interdisciplinary data management solution for ecosystems biology and environmental research

Modern microbial and ecosystem sciences require diverse interdisciplinary teams that are often challenged in “speaking” to one another due to different languages and data product types. Here we introduce the IsoGenie Database, a de novo developed data management and exploration platform, as a solution to this challenge of accurately representing and integrating heterogenous environmental and microbial data across ecosystem scales. The IsoGenieDB is a public and private data infrastructure designed to store and query data generated by the IsoGenie Project, a ~10 year DOE-funded project focused on discovering ecosystem climate feedbacks in a thawing permafrost landscape. The IsoGenieDB provides (i) a platform for IsoGenie Project members to explore the project’s interdisciplinary datasets across scales through the inherent relationships among data entities, (ii) a framework to consolidate and harmonize the datasets needed by the team’s modelers, and (iii) a public venue that leverages the same spatially explicit, disciplinarily integrated data structure to share published datasets. The IsoGenieDB is also being expanded to cover the NASA-funded Archaea to Atmosphere (A2A) project, which scales the findings of IsoGenie to a broader suite of Arctic peatlands, via the umbrella A2A Database (A2A-DB). The IsoGenieDB’s expandability and flexible architecture allow it to serve as an example ecosystems database.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2023

Session Introduction: Graph Representations and Algorithms in Biomedicine

Connectivity is a fundamental property of biological systems: on the cellular level, proteins interact with each other to form protein-protein interaction networks (PPIs); on the organism level, neurons are arranged in a network; and on a community-level, species can have complex relationships with one another that drive the development and balance of an ecosystem. Graphs, representations of systems consisting of entities as vertices and their connections as edges, are a useful structure to characterize many such systems. Such models can be used to understand biological systems that naturally have a network structure, including PPIs, biological neurons, and ecosystems. In today’s information age, graph representations and algorithms (often in combination with machine learning techniques) are used to organize massive amounts of related data, much of which may be heterogeneous or unstructured, and identify patterns that represent novel biological insights. PSB’s 2023 session “Graph Representations and algorithms in Biomedicine,” encompasses modern developments in graph theory and its applications to various fields of biomedicine. This session includes a wide range of research - knowledge graphs built from text-mined health data, heterogeneous networks using multi-omic databases, and graphs refined to represent uncertainty or improve memory usage.

Chrisman, Brianna S.↗
DOE OSTI2024

Acceleration of Graph Neural Network-Based Prediction Models in Chemistry via Co-Design Optimization on Intelligence Processing Units

Atomic structure prediction and associated property calculations are the bedrock of chemical physics. Since high-fidelity ab initio modeling techniques for computing the structure and properties can be prohibitively expensive, this motivates the development of machine-learning (ML) models that make these predictions more efficiently. Training graph neural networks over large atomistic databases introduces unique computational challenges such as the need to process millions of small graphs with variable size and support communication patterns that are distinct from learning over large graphs such as social networks. We demonstrate a novel hardware-software co-design approach to scale up the training of atomistic graph neural networks (GNN) for structure and property prediction. First, to eliminate redundant computation and memory associated with alternative padding techniques and to improve throughput via minimizing communication, we formulate the effective coalescing of the batches of variable-size atomistic graphs as the bin packing problem and introduce a hardware-agnostic algorithm to pack these batches. In addition, we propose hardware-specific optimizations including a planner and vectorization for the gather-scatter operations targeted for Graphcore’s Intelligence Processing Unit (IPU), as well as model-specific optimizations such as merged communication collectives and optimized softplus. Putting these all together, we demonstrate the effectiveness of the proposed co-design approach by providing an implementation of a well-established atomistic GNN on the Graphcore IPUs. We evaluate the training performance on multiple atomistic graph databases with varying degrees of graph counts, sizes and sparsity. Here, we demonstrate that such a co-design approach can reduce the training time of atomistic GNNs and can improve the performance by up to 1.5× compared to the baseline implementation of the model on the IPUs. Additionally, we compare our IPU implementation with a Nvidia GPU-based implementation and show that our atomistic GNN implementation on the IPUs can run 1.8× faster on average compared to the execution time on the GPUs.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2016

Efficient Synthesis of Graph Methods: a Dynamically Scheduled Architecture

RDF databases naturally map to a graph representation and employ languages, such as SPARQL, that implements queries as graph pattern matching routines. Graph methods exhibit an irregular behavior: they present unpredictable, fine-grained data accesses, and are synchronization inten- sive. Graph data structures expose large amounts of dy- namic parallelism, but are difficult to partition without gen- erating load unbalance. In this paper, we present a novel ar- chitecture to improve the synthesis of graph methods. Our design addresses the issues of these algorithms with two com- ponents: a Dynamic Task Scheduler (DTS), which reduces load unbalance and maximize resource utilization, and a Hi- erarchical Memory Interface controller (HMI), which pro- vides support for concurrent memory operations on multi- ported/multi-banked shared memories. We evaluate our ap- proach by generating the accelerators for a set of SPARQL queries from the Lehigh University Benchmark (LUBM). We first analyze the load unbalance of these queries, showing that execution time among tasks can differ even of order of magnitudes. We then synthesize the queries and com- pare the performance of the resulting accelerators against the current state of the art. Experimental results show that our solution provides a speedup over the serial implementa- tion close to the theoretical maximum and a speedup up to 3.45 over a baseline parallel implementation. We conclude our study by exploring the design space to achieve maximum memory channels utilization. The best design used at least three of the four memory channels for more than 90% of the execution time.

architecture↗
DOE OSTI2021

Enabling Scalable and Extensible Memory-mapped Datastores in Userspace

Exascale workloads are expected to incorporate data-intensive processing in close coordination with traditional physics simulations. These emerging scientific, data-analytics and machine learning applications need to access a wide variety of datastores in flat files and structured databases. Programmer productivity is greatly enhanced by mapping datastores into the application process's virtual memory space to provide a unified “in-memory” interface. Currently, memory mapping is provided by system software primarily designed for generality and reliability. However, scalability at high concurrency is a formidable challenge on exascale systems. Also, there is a need for extensibility to support new datastores potentially requiring HPC data transfer services. In this article, we present UMap , a scalable and extensible userspace service for memory-mapping datastores. Furthermore, through decoupled queue management, concurrency aware adaptation, and dynamic load balancing, UMap enables application performance to scale even at high concurrency. We evaluate UMap in data-intensive applications, including sorting, graph traversal, database operations, and metagenomic analytics. Our results show that UMap as a userspace service outperforms an optimized kernel-based service across a wide range of intra-node concurrency by 1.22-1.9 × . We performed two case studies to demonstrate UMap 's extensibility. First, a new datastore residing in remote memory is incorporated into UMap as an application-specific plugin. Second, we present a persistent memory allocator Metall built atop UMap for unified storage/memory.

97 MATHEMATICS AND COMPUTING↗