Search NASA⌕ Search

SEARCH · Search NASA

Results for “Unstructured data”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 109 records · Page 6

FY08 LDRD Final Report LOCAL: Locality-Optimizing Caching Algorithms and Layouts

This project investigated layout and compression techniques for large, unstructured simulation data to reduce bandwidth requirements and latency in simulation I/O and subsequent post-processing, e.g. data analysis and visualization. The main goal was to eliminate the data-transfer bottleneck - for example, from disk to memory and from central processing unit to graphics processing unit - through coherent data access and by trading underutilized compute power for effective bandwidth and storage. This was accomplished by (1) designing algorithms that both enforce and exploit compactness and locality in unstructured data, and (2) adapting offline computations to a novel stream processing framework that supports pipelining and low-latency sequential access to compressed data. This report summarizes the techniques developed and results achieved, and includes references to publications that elaborate on the technical details of these methods.

99 GENERAL AND MISCELLANEOUS↗

A Pipeline for Large Data Processing Using Regular Sampling for Unstructured Grids

Large simulation data requires a lot of time and computational resources to compute, store, analyze, visualize, and run user studies. Today, the largest cost of a supercomputer is not hardware but maintenance, in particular energy consumption. Our goal is to balance energy consumption and cognitive value of visualizations of resulting data. This requires us to go through the entire processing pipeline, from simulation to user studies. To reduce the amount of resources, data can be sampled or compressed. While this adds more computation time, the computational overhead is negligible compared to the simulation time. We built a processing pipeline at the example of regular sampling. The reasons for this choice are two-fold: using a simple example reduces unnecessary complexity as we know what to expect from the results. Furthermore, it provides a good baseline for future, more elaborate sampling methods. We measured time and energy for each test we did, and we conducted user studies in Amazon Mechanical Turk (AMT) for a range of different results we produced through sampling.

97 MATHEMATICS AND COMPUTING↗

Surface reconstruction from scattered data through pruning of unstructured grids

This paper describes an algorithm for reconstructing a surface from a randomly digitized object. Scan data (treated as a cloud of points) is first tesselated out to its convex hull using Delaunay triangulation. The line-of-sight between each surface point and the scanning device is traversed, and any tetrahedra which are pierced by it are removed. The remaining tetrahedra form an approximate solid model of the scanned object. Due to the inherently limited resolution of any scan, this algorithm requires two additional procedures to produce a smooth, polyhedral surface: one process removes long, narrow tetrahedra which span indentations in the surface between digitized points; the other smooths sharp edges. The results for a moderately resolved sample body and a highly resolved aircraft are displayed.

Maksymiuk, C. M.↗

Dynamic in-context learning with conversational models for data extraction and materials property prediction

The advent of natural language processing and large language models (LLMs) has revolutionized the extraction of data from unstructured scholarly papers. However, ensuring data trustworthiness remains a significant challenge. In this paper, we introduce PropertyExtractor, an open-source tool that leverages advanced conversational LLMs such as Google gemini-pro and OpenAI gpt-4, blends zero-shot with few-shot in-context learning, and employs engineered prompts for the dynamic refinement of structured information hierarchies—enabling autonomous, efficient, scalable, and accurate identification, extraction, and verification of material property data. Our tests on material data demonstrate precision and recall that exceed 95% with an error rate of ∼9%, highlighting the effectiveness and versatility of the toolkit. Finally, databases for 2D material thicknesses, a critical parameter for device integration, and energy bandgap values are developed using PropertyExtractor. In particular, for the thickness database, the rapid evolution of the field has outpaced both experimental measurements and computational methods, creating a significant data gap. Our work addresses this gap and showcases the potential of PropertyExtractor as a reliable and efficient tool for the autonomous generation of various material property databases, advancing the field.

Ekuma, Chinedu E. (ORCID:0000000258527556)↗

Computing Axisymmetric Jet Screech Tones using Unstructured Grids

The purpose of this paper is to show that computations with an aeroacoustic feedback loop, the jet screech noise, can be obtained using truly unstructured grid technology. Numerical results are presented for a nozzle with two different lip thicknesses which will be referred to in this paper as a thin and a thick lip nozzle respectively. The space-time conservation element and solution element (CE/SE) method is used to solve the conservation laws of the compressible axisymmetric Navier-Stokes equations. The equations are time marched to predict the unsteady flow and the near-field screech tone noise issuing from an underexpanded circular jet. The CE/SE method uses an unstructured grid based data structure. The unstructured grids for these calculations are generated based on the method of Delaunay triangulation. Comparisons of numerical results with available experimental data are shown for flows corresponding to several different jet Mach numbers. Generally good agreement is obtained in terms of flow physics, screech tone frequency, and sound pressure level.

Jorgenson, Philip C. E.↗

Computing Axisymmetric Jet Screech Tones Using Unstructured Grids

The space-time conservation element and solution element (CE/SE) method is used to solve the conservation law form of the compressible axisymmetric Navier-Stokes equations. The equations are time marched to predict the unsteady flow and the near-field screech tone noise issuing from an underexpanded circular jet. The CE/SE method uses an unstructured grid based data structure. The unstructured grids for these calculations are generated based on the method of Delaunay triangulation. The purpose of this paper is to show that an acoustics solution with a feedback loop can be obtained using truly unstructured grid technology. Numerical results are presented for two different nozzle geometries. The first is considered to have a thin nozzle lip and the second has a thick nozzle lip. Comparisons with available experimental data are shown for flows corresponding to several different jet Mach numbers. Generally good agreement is obtained in terms of flow physics, screech tone frequency, and sound pressure level.

Jorgenson, Philip C. E.↗

Learning Global Proliferation Expertise Evolution Using AI-Driven Analytics and Public Information

Detecting and anticipating global proliferation expertise and capability evolution from unstructured, noisy, and incomplete public data streams is a highly desired, but extremely challenging task. Here, in this article, we present our pioneering data-driven approach to support the non-proliferation mission to detect and explain the evolution of proliferation expertise and capability development globally from terabytes of publicly available information (PAI), focusing on our knowledge extraction pipeline and descriptive analytics. We first discuss how we fuse nine open-source data streams, including multilingual data, to convert 4 TB of unstructured data to structured knowledge and encode dynamically evolving proliferation expertise representations—content and context graphs. For this, we rely on natural language processing (NLP) and deep learning (DL) models to perform information extraction, topic modeling, and distributed text representation (aka embedding) learning. We then present interactive, usable, and explainable descriptive analytics to refine domain knowledge and present it in a human-understandable form. Finally, we introduce future work avenues that will leverage our dynamic knowledge representations and descriptive analytics to enable predictive and prescriptive inferences to achieve real-time domain understanding and contextual reasoning about global proliferation expertise and capability evolution.

98 NUCLEAR DISARMAMENT, SAFEGUARDS, AND PHYSICAL P↗

Big Data Framework with Machine Learning for D and D Applications - 19108

The nuclear industry is experiencing a steady increase in maintenance costs even though plants are maintained under high levels of safety, capability and reliability. Nuclear power plants are expected to run every unit at maximum capacity at all times, efficiently utilizing assets with minimal downtime. Surveillance and maintenance of nuclear decommissioning infrastructure provides lot challenges with respect to maintenance or decommissioning of the buildings. There is a need for a framework to analyze the huge amount of data generated by the sensors on the nuclear reactor components as well as structures, to monitor the conditions of these building over a period of time. Emerging technologies such as big data analytics have become a requirement in the nuclear industry to improve structural health monitoring and diagnostics. FIU will make use of existing mature technologies in the areas of imaging, robotics, big data, and machine learning/deep learning to assess the structural integrity of aging facilities at DOE sites. As these facilities await decommissioning, there is a need to understand the structural health of these structures. Many of these facilities were built over 50 years ago and in some cases these facilities have gone beyond the operational life expectancy. In other cases, the facilities have been placed in a state of 'cold and dark' and they are sitting unused, awaiting decommissioning. Finally, some older facilities are one-of-a-kind operational/production facilities supporting critical DOE missions and cannot be shut down. In any of these scenarios, the structural integrity of these facilities may be compromised, so it is imperative that adequate inspections and data collection/analysis be performed on a continuous and ongoing basis. The primary goals of the research include collecting various formats of data such as structured, unstructured and streaming data from the various sensors deployed in buildings, collect videos/pictures from various imaging sources, ingest them using a Hadoop distributed file system and process the data using Spark to perform batch processing and real time analytics. Research and development on various machine learning/deep learning algorithms will also be performed to analyze the heterogeneous data collected from nuclear decommission infrastructure. FIU will design the big data framework to ingest, store and process huge amounts of heterogeneous data collected from many sources and optimize the algorithms to provide insights into the data and predict anomalies observed when compared against baseline conditions. Various modules of this framework will include heterogeneous data sources, message broker, Hadoop distributed file system, Spark for stream and batch processing, machine learning/ deep learning, Cassandra - persistent data store and visualization. (authors)

12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗

Gradient Calculation Methods on Arbitrary Polyhedral Unstructured Meshes for Cell-Centered CFD Solvers

A survey of gradient reconstruction methods for cell-centered data on unstructured meshes is conducted within the scope of accuracy assessment. Formal order of accuracy, as well as error magnitudes for each of the studied methods, are evaluated on a complex mesh of various cell types through consecutive local scaling of an analytical test function. The tests highlighted several gradient operator choices that can consistently achieve 1st order accuracy regardless of cell type and shape. The tests further offered error comparisons for given cell types, leading to the observation that the "ideal" gradient operator choice is not universal. Practical implications of the results are explored via CFD solutions of a 2D inviscid standing vortex, portraying the discretization error properties. A relatively naive, yet largely unexplored, approach of local curvilinear stencil transformation exhibited surprisingly favorable properties

Meshes↗

A Case for Database Filesystems

Data intensive science is offering new challenges and opportunities for Information Technology and traditional relational databases in particular. Database filesystems offer the potential to store Level Zero data and analyze Level 1 and Level 3 data within the same database system [2]. Scientific data is typically composed of both unstructured files and scalar data. Oracle SecureFiles is a new database filesystem feature in Oracle Database 11g that is specifically engineered to deliver high performance and scalability for storing unstructured or file data inside the Oracle database. SecureFiles presents the best of both the filesystem and the database worlds for unstructured content. Data stored inside SecureFiles can be queried or written at performance levels comparable to that of traditional filesystems while retaining the advantages of the Oracle database.

99 GENERAL AND MISCELLANEOUS↗

krowkee

krowkee is a toolkit for scalably and efficiently summarizing many data streams in distributed memory. krowkee is intended for applications where one needs to summarize huge loosely structured data, such as matrices or graphs, where individual components such rows/columns or vertex adjacency information are impractical to store and directly inspect. krowkee ingests these objects as data streams - unstructured, arbitrarily ordered lists of updates - and accumulates summaries thereof in the form of data sketches.

Dunton, AlecM.↗

Big Data Meets HPC Log Analytics: Scalable Approach to Understanding Systems at Extreme Scale

Today's high-performance computing (HPC) systems are heavily instrumented, generating logs containing information about abnormal events, such as critical conditions, faults, errors and failures, system resource utilization, and about the resource usage of user applications. These logs, once fully analyzed and correlated, can produce detailed information about the system health, root causes of failures, and analyze an application's interactions with the system, providing valuable insights to domain scientists and system administrators. However, processing HPC logs requires a deep understanding of hardware and software components at multiple layers of the system stack. Moreover, most log data is unstructured and voluminous, making it more difficult for system users and administrators to manually inspect the data. With rapid increases in the scale and complexity of HPC systems, log data processing is becoming a big data challenge. This paper introduces a HPC log data analytics framework that is based on a distributed NoSQL database technology, which provides scalability and high availability, and the Apache Spark framework for rapid in-memory processing of the log data. The analytics framework enables the extraction of a range of information about the system so that system administrators and end users alike can obtain necessary insights for their specific needs. We describe our experience with using this framework to glean insights from the log data about system behavior from the Titan supercomputer at the Oak Ridge National Laboratory.

Park, Byung↗

Searching for 'Unknown Unknowns'

The NASA Engineering and Safety Center (NESC) was established to improve safety through engineering excellence within NASA programs and projects. As part of this goal, methods are being investigated to enable the NESC to become proactive in identifying areas that may be precursors to future problems. The goal is to find unknown indicators of future problems, not to duplicate the program-specific trending efforts. The data that is critical for detecting these indicators exist in a plethora of dissimilar non-conformance and other databases (without a common format or taxonomy). In fact, much of the data is unstructured text. However, one common database is not required if the right standards and electronic tools are employed. Electronic data mining is a particularly promising tool for this effort into unsupervised learning of common factors. This work in progress began with a systematic evaluation of available data mining software packages, based on documented decision techniques using weighted criteria. The four packages, which were perceived to have the most promise for NASA applications, are being benchmarked and evaluated by independent contractors. Preliminary recommendations for "best practices" in data mining and trending are provided. Final results and recommendations should be available in the Fall 2005. This critical first step in identifying "unknown unknowns" before they become problems is applicable to any set of engineering or programmatic data.

Parsons, Vickie S.↗

Recent Updates to the CFD General Notation System (CGNS)

The CFD General Notation System (CGNS) - a general, portable, and extensible standard for the storage and retrieval of computational fluid dynamics (CFD) analysis data has been in existence for more than a decade (Version 1.0 was released in May 1998). Both structured and unstructured CFD data are covered by the standard, and CGNS can be easily extended to cover any sort of data imaginable, while retaining backward compatibility with existing CGNS data files and software. Although originally designed for CFD, it is readily extendable to any field of computational analysis. In early 2011, CGNS Version 3.1 was released, which added significant capabilities. This paper describes these recent enhancements and highlights the continued usefulness of the CGNS methodology.

Rumsey, Christopher L.↗

Survey of Current State of the Art Entity-Relation Extraction Tools

In the area of information extraction from text data, there exists a number of tools with the capability of extracting entities, topics, and their relationships with one another from both structured and unstructured text sources. Such information has endless uses in a number of domains, however, the solutions to getting this information are still in early stages and has room for improvement. The topic has been explored from a research perspective by academic institutions, as well as formal tool creation from corporations but has not made much advancement since the early 2000's. Overall, entity extraction, and the related topic of entity linking, is common among these tools, though with varying degrees of accuracy, while relationship extraction is more difficult to find and seems limited to same sentence analysis. In this report, we take a look at the top state of the art tools currently available and identify their capabilities, strengths, and weaknesses. We explore the common algorithms in the successful approaches to entity extraction and their ability to efficiently handle both structured and unstructured text data. Finally, we highlight some of the common issues among these tools and summarize the current ability to extract relationship information.

97 MATHEMATICS AND COMPUTING↗

All Source Sensor Analysis

ASSA is a software application that processes binary data into summarized index tables that can be used to organize features contained within the data. ASSA's index tables can also be used to search for user specified features. ASSA is designed to organize and search for patterns in unstructured binary data streams or archives, such as video, images, audio, and network traffic. ASSA is basically a very general search engine used to search for any pattern in any binary data stream. It has uses in video analytics, image analysis, audio analysis, searching hard-drives, monitoring network traffic, etc.

Trease, Harold↗

MarmotViz User Guide

MarmotViz is an illustrative visualization plug-in for ParaView. It is intended for generating enhanced visualizations of time-varying datasets on unstructured connected meshes. A detailed description of the implemented algorithms and program structure can be found in the related document: Generalized framework and algorithms for illustrative visualization of time-varying data on unstructured meshes . This document provides details for compiling/building MarmotViz, using the MarmotViz plug-in in ParaView, and extending the MarmotViz plug-in.

42 ENGINEERING↗