Search NASA⌕ Search

SEARCH · Search NASA

Results for “Unstructured data”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records

Combining structured and unstructured data in a configurable web-based logbook

A typical electronic logbook is designed as a general-purpose system for recording time-ordered events and actions and, therefore, allows for a great flexibility in recording information, but the data is unstructured. To better position it in a specific context (e.g., a, test facility, a group activity log) it needs to support both structured data (keyword, authors, etc) and unstructured data (text, title, attachments) in that context. To do this, a logbook system can define a set of attributes, possibly built as a hierarchy. These application-specific attributes will be associated with each entry. To be flexible, such a system has to be configurable to allow for tailoring it to each specific environment. The paper describes a design, functionality, and experiences with WebLog, a database-configurable electronic logbook developed with the J2EE Web technology. Various functional and technical properties of the system are discussed, including views, searches, threads of entries, an automated alerting system as well as integration with other applications.

99 GENERAL AND MISCELLANEOUS↗

GMLS-Nets: A Framework for Learning from Unstructured Data

Data fields sampled on irregularly spaced points arise in many applications in the sciences and engineering. For regular grids, Convolutional Neural Networks (CNNs) have been successfully used to gaining benefits from weight sharing and invariances. We generalize CNNs by introducing methods for data on unstructured point clouds based on Generalized Moving Least Squares (GMLS). GMLS is a nonparametric technique for estimating linear bounded functionals from scattered data, and has recently been used in the literature for solving partial differential equations. By parameterizing the GMLS estimator, we obtain learning methods for operators with unstructured stencils. In GMLS-Nets the necessary calculations are local, readily parallelizable, and the estimator is supported by a rigorous approximation theory. We show how the framework may be used for unstructured physical data sets to perform functional regression to identify associated differential operators and to regress quantities of interest. The results suggest the architectures to be an attractive foundation for data-driven model development in scientific machine learning applications.

42 ENGINEERING↗

GMLS-Nets: Scientific Machine Learning Methods for Unstructured Data

Data fields sampled on irregularly spaced points arise in many applications in the sciences and engineering. For regular grids, Convolutional Neural Networks (CNNs) have been successfully used to gaining benefits from weight sharing and invariances. We generalize CNNs by introducing methods for data on unstructured point clouds based on Generalized Moving Least Squares (GMLS). GMLS is a nonparametric technique for estimating linear bounded functionals from scattered data, and has recently been used in the literature for solving partial differential equations. By parameterizing the GMLS estimator, we obtain learning methods for operators with unstructured stencils. In GMLS-Nets the necessary calculations are local, readily parallelizable, and the estimator is supported by a rigorous approximation theory. We show how the framework may be used for unstructured physical data sets to perform functional regression to identify associated differential operators and to regress quantities of interest. The results suggest the architectures to be an attractive foundation for data-driven model development in scientific machine learning applications

97 MATHEMATICS AND COMPUTING↗

GMLS-Nets: A Framework for Learning from Unstructured Data

Data fields sampled on irregularly spaced points arise in many applications in the sciences and engineering. For regular grids, Convolutional Neural Networks (CNNs) have been successfully used to gaining benefits from weight sharing and invariances. We generalize CNNs by introducing methods for data on unstructured point clouds based on Generalized Moving Least Squares (GMLS). GMLS is a non-parametric technique for estimating linear bounded functionals from scattered data, and has recently been used in the literature for solving partial differential equations. By parameterizing the GMLS estimator, we obtain learning methods for operators with unstructured stencils. In GMLS-Nets the necessary calculations are local, readily parallelizable, and the estimator is supported by a rigorous approximation theory. We show how the framework may be used for unstructured physical data sets to perform functional regression to identify associated differential operators and to regress quantities of interest. The results suggest the architectures to be an attractive foundation for data-driven model development in scientific machine learning applications

97 MATHEMATICS AND COMPUTING↗

Scraping Unstructured Data to Explore the Relationship between Rainfall Anomalies and Vector-Borne Disease Outbreaks

According to the World Health Organization (WHO), vector-borne diseases such as malaria and dengue account for 17% of all infectious disease cases and lead to more than 700,000 deaths per year. Tracking and predicting the spread of vector-borne diseases is a vital task that could save hundreds of thousands of lives annually. Oftentimes, the first reports of vector-borne disease outbreaks occur through emails and online reporting systems long before they are officially documented. Tracking and predicting the emergence and spread of vector-borne disease outbreaks requires extracting data from these unstructured sources in combination with historical weather and climate data to understand the underlying background triggers and disease dynamics. In this work, we develop a data extraction pipeline for the online outbreak reporting website ProMED-mail that utilizes a web scraper, transformer neural network summarizer, and named entity recognizer to obtain a dataset of malaria, dengue, zika, and chikungunya outbreaks over the last 30 years. This scraped dataset was further analyzed in association with global rainfall anomalies derived from NASA’s Integrated Multi-satellitE Retrievals for GPM [Global Precipitation Mission] (IMERG) dataset. This preliminary analysis was to understand the effect of global rainfall patterns on the spread of vector-borne diseases. Analysis of the ProMED-mail and GPM data shows that vector-borne disease outbreaks are clustered towards the tropics and outbreaks are often amplified during the rainy seasons. Our scraped dataset can be a valuable tool in creating comprehensive georeferenced disease records for modeling and predicting future outbreaks.

Web scraping↗

A General Framework for Error-controlled Unstructured Scientific Data Compression

Data compression plays a key role in reducing storage and I/O costs. Traditional lossy methods primarily target data on rectilinear grids and cannot leverage the spatial coherence in unstructured mesh data, leading to suboptimal compression ratios. We present a multi-component, error-bounded compression framework designed to enhance the compression of floating-point unstructured mesh data, which is common in scientific applications. Our approach involves interpolating mesh data onto a rectilinear grid and then separately compressing the grid interpolation and the interpolation residuals. This method is general, independent of mesh types and typologies, and can be seamlessly integrated with existing lossy compressors for improved performance. We evaluated our framework across twelve variables from two synthetic datasets and two real-world simulation datasets. The results indicate that the multi-component framework consistently outperforms state-of-the-art lossy compressors on unstructured data, achieving, on average, a 2.3 − 3.5× improvement in compression ratios, with error bounds ranging from 1 × 10 the −6 to 1×10−2. We further investigate impact of hyperparameters, such as grid spacing and error allocation, to deliver optimal compression ratios in diverse datasets.

Gong, Qian↗

Emulation automation and model checking

A method of automating emulations is provided. The method comprising collecting publicly available network data over a predefined time interval, wherein the collected network data might comprise structured and unstructured data. Any unstructured data is converted into structured data. The original and converted structured data is stored in a database and compared to known network vulnerabilities. An emulated network is created according to the collected network data and the comparison of the structured data with known vulnerabilities. Virtual machines are created to run on the emulated network. Director programs and guest actor programs are run on the virtual machines, wherein the actor programs imitate real user behavior on the emulated network. The director programs deliver task commands to the guest actor programs to imitate real user behavior. The imitated behavior is presented to a user via an interface.

Urias, Vincent↗

Compressing unstructured mesh data from simulations using machine learning

The amount of data output from a computer simulation has grown to terabytes and petabytes as increasingly complex simulations are being run on massively parallel systems. As we approach exaflop computing in the next decade, it is expected that the I/O subsystem will not be able to write out these large volumes of data. In this paper, we explore the use of machine learning to compress the data before it is written out. Despite the computational constraints that limit us to using very simple learning algorithms, our results show that machine learning is a viable option for compressing unstructured data. Furthermore, we demonstrate that by simply using a better sampling algorithm to generate the training set, we can obtain more accurate results compared to random sampling, but at no extra cost. Further, by carefully selecting and incorporating points with high prediction error, we can improve reconstruction accuracy without sacrificing the compression rate.

97 MATHEMATICS AND COMPUTING↗

Multilevel Techniques for Compression and Reduction of Scientific Data---The Unstructured Case

Previous work on multilevel techniques for compression and reduction of scientific data is extended to the case of data given on unstructured meshes in two and three dimensions. The centerpiece of the work is a decomposition algorithm which is shown to be optimal, in terms of both storage and operational complexity, applicable to unstructured grids in both two and three dimensions, and which implicitly gives a Riesz basis that can be exploited to reduce the data while maintaining rigorous bounds on the loss incurred. The flexibility of the approach is illustrated by applications to potential flow around an airfoil and the effect of compression on quantities of interest relevant to airfoil design; compression of computational simulation of a nonlinear reaction-diffusion system with special attention given to the problem of time series reduction; and, data from a simulation of magnetically confined plasma in a fusion reactor reduced so as to preserve the electric field computed from the data.

97 MATHEMATICS AND COMPUTING↗

A Framework for Compressing Unstructured Scientific Data via Serialization

We present a general framework for compressing unstructured scientific data with known local connectivity. A common application is simulation data defined on arbitrary finite element meshes. The framework employs a greedy topology preserving reordering of original nodes which allows for seamless integration into existing data processing pipelines. This reordering process depends solely on mesh connectivity and can be performed offline for optimal efficiency. However, the algorithm’s greedy nature also supports on-the-fly implementation. The proposed method is compatible with any compression algorithm that leverages spatial correlations within the data. The effectiveness of this approach is demonstrated on a large-scale real dataset using several compression methods, including MGARD, SZ, and ZFP.

Reshniak, Viktor [ORNL] (ORCID:0000000315454462)↗