Search NASA⌕ Search

SEARCH · Search NASA

Results for “Unstructured data”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 91 records · Page 5

Towards Automatically Matching Security Advisories to CPEs: String Similarity-based Vendor Matching

When a vulnerability is reported by the National Vulnerability Database (NVD), affected products are listed in the structured Common Platform Enumeration (CPE) format. Unfortunately, if the vulnerability is in a software library (e.g., Log4j), it will not include CPEs for each product containing that library. In these cases, security operators need to manually read the vendor's or third-party security advisories to see if their product is affected. However, these advisories do not report affected products in a structured format, which prevents automated processing, This paper makes the first effort towards automatically constructing structured CPEs for the vulnerable products in a non-NVD security advisory from the unstructured data in the advisory. Since this is a very challenging problem, this paper specifically focuses on the initial but key step of matching the un-structured vendor names in security advisories to the structured vendor representations in the standard CPE format. We explore the feasibility of using string similarity to solve the problem. The basic idea is to compare a vendor name from the non-NVD advisory with each vendor in the official CPE dictionary. The CPE vendor with the highest similarity score to the advisory's vendor will be considered as the match. We first conduct an experimental, comparative study of multiple mainstream string similarity metrics for this matching problem. To improve the performance, we then design a new string similarity metric that is adapted from an existing metric by weighing different tokens in the advisory's vendor name differently.

McClanahan, Kylie↗

An Overview of Internet biosurveillance

Internet biosurveillance utilizes unstructured data from diverse Web-based sources to provide early warning and situational awareness of public health threats. The scope of source coverage ranges from local based media in the vernacular to international media in widely read languages. Internet biosurveillance is a timely modality available to government and public health officials, health care workers, and the public and private sector, serving as a real-time complementary approach to traditional indicator-based public health disease surveillance methods. Internet biosurveillance also supports the broader activity of epidemic intelligence. This review covers the current state of the field of Internet biosurveillance and provides a perspective on the future of the field.

digital disease detection↗

Developing an Ontology for Cyber Security Knowledge Graphs

In this paper we describe an ontology developed for a cyber security knowledge graph database. This is intended to provide an organized schema that incorporates information from a large variety of structured and unstructured data sources, and includes all relevant concepts within the domain. We compare the resulting ontology with previous efforts, discuss its strengths and limitations, and describe areas for future work.

Iannacone, Michael↗

ESnet Secure Copy (EScp) v0.6

EScp is a high speed transfer tool with a similar command line syntax to scp. Unlike SCP it is designed to transfer files at high speed, thus far we have been able to show 100gbit/s transfers, although I expect that the throughput should scale in proportion to the network interface, i.e. I expect 400gbit/s performance on our 400gbit/s test bed. EScp achieves good performance through an innovative design (multithreaded, zero copy transfers), along with pluggable filters and I/O engines. As an example, you can switch from POSIX i/O to UIO by checking a different engine. It also natively supports encryption, and cheksums for file verification and transport security. AAA is through standard SSH (same as SCP). By taking advantage of filters, EScp supports transferring unstructured data and/or I/O to non-posix data sources. Examples include streaming data (i.e. from equipment), transferring data to the cloud, and/or supporting non-posix file systems (like HPSS).

Shiflett, Charles↗

Topological Analysis of The SPOKE Graph

The SPOKE graph [2, 6] is a sparse decorated semantic graph representing a collection of knowledge collected in many scientific databases from the fields of healthcare, biochemistry, chemistry, biology, et cetera. This knowledge graph is stored as a relational dataset decorated with metadata on each constituent vertex and edge. Formally, the graph is G(V, E, D), where V is a set of n vertices V := {1, ..., n} and edges of the form (i, j) ϵ E for i, j ϵ V, and table D that for any item in V υ E stores unstructured data such as vertex/edge type, nature of a relationship, et cetera. D(i) = {data involving vertex i ϵ V}, and D(i, j) = {data involving edge (i, j) ϵ E}. Here, we treat the graph as undirected in the sense that a direct relationship for (i, j) causes a (possibly opposite) reverse direct relationship for (j, i). The SPOKE graph G(V, E, D) is formed by processing a collection of relational datasets from medicine, chemistry, and biology, connecting many entities. Here, we analyze an instance from 2019, Spoke-20190707, where a graph file contains 6.16M edges and associated metadata and a vertex file contains 2.15M vertices and the associated metadata. There are 12 different types of vertex entities; all edge types used are implicit (see §2). There is other metadata in D on edges and vertices, but we just use the topology and the vertex labels in this report. SPOKE is growing as more knowledge is gained and more datasets are added. SPOKE is likely to grow 10x during the next phase of this project, and we therefore would like to consider topoligical analysis techniques that are scalable to several orders of magnitude larger than the current dataset (say >1B edges).

59 BASIC BIOLOGICAL SCIENCES↗

Mapped Moments to a Cartesian Grid (MMCG) Value-Added Product Report

Objective analysis (OA) is a method of mapping unstructured data to a structured grid. In the context of scanning radar data, OA is used to interpolate data in antenna coordinates (range, azimuth, and elevation) onto a regularly spaced Cartesian grid (Trapp and Doswell 2000). The U.S. Department of Energy Atmospheric Radiation Measurement (ARM) user facility’s Mapped Moments to a Cartesian Grid (MMCG) Value-Added Product (VAP) uses the Python ARM Radar Toolkit (Py-ART), a data model-driven interactive architecture for working with weather radar data, to map the data to a Cartesian grid (Helmus and Collis 2016). MMCG, with Py-ART built in, has the ability to take radar data in antenna coordinates and map the gates to a Cartesian grid using inverse distance weight functions such as Cressman (square) and Barnes (exponential), but also can filter the data during the interpolation. MMCG also allows arbitrary formulations for the radius of influence, which are matched to particular radar scanning strategies. This creates a complex parameter space for optimizing the retention of storm structure detail while minimizing artifacts. MMCG takes data processed with ARM’s Corrected Precipitation Radar Moments in Antenna Coordinates (CMAC) VAP and maps it to a Cartesian grid as the output product. A variety of fields that have been mapped to the Cartesian grid are then saved to plots to complement each grid file.

54 ENVIRONMENTAL SCIENCES↗

AI for Nuclear Safeguards Verification

The International Atomic Energy Agency (IAEA) utilizes AI/ML to analyze open-source information, including satellite imagery and scientific publications, to verify the completeness of State declarations regarding nuclear activities. AI/ML already assist the IAEA with automating processes and analysis of large datasets, including satellite imagery and unstructured data, improving efficiency and effectiveness of safeguards implementation. AI/ML in nuclear safeguards come with its own challenges that include the need for large, unbiased datasets, the risk of AI-generated fake information, including the potential for manipulation of satellite imagery.

97 MATHEMATICS AND COMPUTING↗

Adaptive grid embedding for the two-dimensional flux-split Euler equations

A numerical algorithm is presented for solving the 2-D flux-split Euler equations using a multigrid method with adaptive grid embedding. The method uses an unstructured data set along with a system of pointers for communication on the irregularly shaped grid topologies. An explicit two-stage time advancement scheme is implemented. A multigrid algorithm is used to provide grid level communication and to accelerate the convergence of the solution to steady state. Results are presented for a subcritical airfoil and a transonic airfoil with 3 levels of adaptation. Comparisons are made with a structured upwind Euler code which uses the same flux integration techniques of the present algorithm. Good agreement is obtained with converged surface pressure coefficients. The lift coefficients of the adaptive code are within 2 1/2 percent of the structured code for the sub-critical case and within 4 1/2 percent of the structured code for the transonic case using approximately one-third the number of grid points.

Warren, Gary Patrick↗

Adaptive grid embedding for the two-dimensional Euler equations

A numerical algorithm is presented for solving the two-dimensional flux-split Euler equations using a multigrid method with adaptive grid embedding. The method uses an unstructured data set along with a system of pointers for communication on the irregularly shaped grid topologies. An explicit two-stage time advancement scheme is implemented. A multigrid algorithm is used to provide grid level communication and to accelerate the convergence of the solution to steady state. Results are presented for an NACA 0012 airfoil in a freestream with Mach numbers of 0.95 and 1.054. Excellent resolution of the shock structures is obtained with the adaptive grid embedding method with significantly fewer grid points than the comparable structured grid.

Warren, Gary P.↗

Cartesian based grid generation/adaptive mesh refinement

Grid adaptation has recently received attention in the computational fluid dynamics (CFD) community as a means to capture the salient features of a flowfield by either moving grid points of a structured or by adding cells in an unstructured manner. An approach based on a background cartesian mesh is investigated from which the geometry is 'cut' out of the mesh. Once the mesh is obtained, a solution on this coarse grid is found, that indicates which cells need to be refined. This process of refining/solving continues until the flow is grid refined in terms of a user specified global parameter (such as drag coefficient etc.). The advantages of this approach are twofold: the generation of the base grid is independent of the topology of the bodies or surfaces around/through which the flow is to be computed, and the resulting grid (in uncut regions) is highly isotropic, so that the truncation error is low. The flow solver (which, along with the grid generation is still under development) uses a completely unstructured data base, and is a finite volume, upwinding scheme. Current and future work will address generating Navier-Stokes suitable grids by using locally aligned and normal face/cell refining. The attached plot shows a simple grid about two turbine blades.

Coirier, William J.↗

Application of multigrid and adaptive grid embedding to the two-dimensional flux-split Euler equations

A numerical algorithm is presented for solving the two-dimensional flux-split Euler equations using a multigrid method with adaptive grid embedding. The method uses an unstructured data set along with a system of pointers for communication on the irregularly shaped grid topologies. An explicit two-stage time-advancement scheme is implemented. A multigrid algorithm is used to provide grid level communication and to accelerate the convergence of the solution to steady state. Results are presented for a NACA 0012 aerofoil in a free stream with a Mach number of 0.85 and an angle of attack of 1.0 degree. Excellent resolution of the shock structures is obtained with the adaptive grid embedding method with significantly fewer grid points than the comparable structured grid.

Warren, Gary P.↗

Language Model For Earth Science: Exploring Potential Downstream Applications As Well As Current Challenges

The use of deep learning techniques to build transformer language models such as SciBERT and GPT3 have transformed the natural language technology (NLT) landscape. These new NLTs are being used in speech to text and vice versa, auto-mated text classification, sentiment analysis, topic modeling, text summarization, and cognitive assistants. While Earth science has no shortage of unstructured data such as journal and conference papers, little efforts have focused on harnessing NLTs for knowledge extraction and supporting the scientific process. This paper surveys the use of language models in different science. BERT-E, a new Earth science-specific language model, is presented. BERT-E is generated using a transfer learning solution. A language model that has already been trained for general Science (SciBERT) is fine-tuned using abstracts and full text extracted from various Earth science-related articles. A downstream keywords classification application is used for evaluation, and the use of BERT-E shows improved performance. The need to develop a robust set of benchmarks in evaluating the language model such as BERT-E is discussed. Finally, example applications are presented to inspire additional ideas for applications using domain-specific language models.

R Ramachandran↗

Contextualizing Air Traffic Management Conversations using Natural Language Understanding

Efficient management of air traffic and mitigation of delays depend on extracting actionable information from unstructured data, such as dialogues from the Federal Aviation Administration’s (FAA’s) Air Traffic Control System Command Center (ATCSCC) telecons. This study presents a pipeline utilizing Natural Language Processing (NLP) methods for Intent Classification (IC) and Slot Filling (SF) to identify and extract Traffic Management Initiatives (TMIs) from aviation-specific dialogues. We leveraged DeBERTa, a pre-trained transformer model, and fine-tuned it to the nuances of the aviation domain. Despite challenges posed by annotation complexities, the IC model achieved promising results with a weighted average F1-score of 0.81. Our results are close to those of human annotators, which demonstrates the model’s strong alignment with human-level performance. The SF model also showed strong performance, achieving a weighted F1-score of 0.97, which demonstrates its effectiveness in accurately predicting key slots. Our analysis revealed limitations in handling less frequent intents and slot labels due to data sparsity, motivating future efforts to adopt joint IC-SF modeling and data augmentation strategies. This research highlights the potential of domain-specific NLP to streamline decision-making in the aviation industry and improve the management of TMIs.

Air Traffic Control Management↗

Pattern Search in Multi-structure Data: A Framework for the Next-Generation Evidence-based Medicine

With the advent of personalized and evidence-based medicine, the need for a framework to analyze/interpret quantitative measurements (blood work, toxicology, etc.) with qualitative descriptions (specialist reports after reading images, bio-medical knowledge-bases) to predict diagnostic risks is fast emerging. Addressing this need, we pose and address the following questions (i) How can we jointly analyze both qualitative and quantitative data ? (ii) Is the fusion of multi-structure data expected to provide better insights than either of them individually ? We present experiments on two bio-medical data sets - mammography and traumatic brain studies to demonstrate architectures and tools for evidence-pattern search.

data fusion↗

Using GTO-Velo to Facilitate Communication and Sharing of Simulation Results in Support of the Geothermal Technologies Office Code Comparison Study

The Geothermal Technologies Office Code Comparison Study (GTO-CCS) aims to support the DOE Geothermal Technologies Office in organizing and executing a model comparison activity. This project is directed at testing, diagnosing differences, and demonstrating modeling capabilities of a worldwide collection of numerical simulators for evaluating geothermal technologies. Teams of researchers are collaborating in this code comparison effort, and it is important to be able to share results in a forum where technical discussions can easily take place without requiring teams to travel to a common location. Pacific Northwest National Laboratory has developed an open-source, flexible framework called Velo that provides a knowledge management infrastructure and tools to support modeling and simulation for a variety of types of projects in a number of scientific domains. GTO-Velo is a customized version of the Velo Framework that is being used as the collaborative tool in support of the GTO-CCS project. Velo is designed around a novel integration of a collaborative Web-based environment and a scalable enterprise Content Management System (CMS). The underlying framework provides a flexible and unstructured data storage system that allows for easy upload of files that can be in any format. Data files are organized in hierarchical folders and each folder and each file has a corresponding wiki page for metadata. The user interacts with Velo through a web browser based wiki technology, providing the benefit of familiarity and ease of use. High-level folders have been defined in GTO-Velo for the benchmark problem descriptions, descriptions of simulator/code capabilities, a project notebook, and folders for participating teams. Each team has a subfolder with write access limited only to the team members, where they can upload their simulation results. The GTO-CCS participants are charged with defining the benchmark problems for the study, and as each GTO-CCS Benchmark problem is defined, the problem creator can provide a description using a template on the metadata page corresponding to the benchmark problem folder. Project documents, references and videos of the weekly online meetings are shared via GTO-Velo. A results comparison tool allows users to plot their uploaded simulation results on the fly, along with those of other teams, to facilitate weekly discussions of the benchmark problem results being generated by the teams. GTO-Velo is an invaluable tool providing the project coordinators and team members with a framework for collaboration among geographically dispersed organizations.

Collaborative Tools↗

JSONize: A Scalable Machine Learning Pipeline to Model Medical Notes as Semi-structured Documents

The Department of Veteran's Affairs (VA) archives the largest corpora of clinical notes in their corporate data warehouse (CDW) as unstructured text data. Unstructured text easily supports keyword searches and regular expressions. Often these simple searches do not adequately support the complex searches that need to be performed on notes. For example, a researcher may want all notes with a Duke Treadmill Score less than 5 or people that smoke more than 1 pack per day. Range queries like this and more can be supported by modelling text as semi-structured documents. In this paper, we implement a scalable machine learning pipeline that models plain medical text as useful semi-structured documents. We improve on existing models and achieve a F1-score of 0.912 and scale our methods to the entire VA corpus.

Rush III, Everett↗

Reverse-mode differentiation in arbitrary tensor network format: with application to supervised learning.

This paper describes an efficient reverse-mode differentiation algorithm for contraction operations of tensor networks that may have arbitrary and unconventional network topologies. The approach leverages the tensor contraction tree of Evenbly and Pfeifer (2014), which provides an instruction set for the contraction sequence of a network. We show that this tree can be efficiently leveraged for differentiation of a full tensor network contraction using a recursive scheme that exploits (1) the bilinear property of contraction and (2) the property that trees have single path from root to leaves. While differentiation of tensor-tensor contraction is already possible in most automatic differentiation packages, we show that exploiting these two additional properties in the specific context of contraction sequences can improve efficiency. Following a description of the algorithm and computational complexity analysis, we investigate its utility for gradient-based supervised learning for low-rank function recovery and for fitting real-world unstructured datasets. We demonstrate improved performance over alternating least-squares optimization approaches and the capability to handle heterogeneous and arbitrary tensor network formats. When compared to alternating minimization algorithms, we find that the gradient-based approach requires a smaller oversampling ratio (number of samples compared to number model parameters) for recovery. This increased efficiency extends to fitting unstructured data of varying dimensionality and when employing a variety of tensor network formats. Here, we show improved learning using the hierarchical Tucker method over the tensor-train in high-dimensional settings on a number of benchmark problems.

97 MATHEMATICS AND COMPUTING↗

Compression-based integral curve data reuse framework for flow visualization

Currently, by default, integral curves are repeatedly re-computed in different flow visualization applications, such as FTLE field computation, source-destination queries, etc., leading to unnecessary resource cost. We present a compression-based data reuse framework for integral curves, to greatly reduce their retrieval cost, especially in a resource-limited environment. In our design, a hierarchical and hybrid compression scheme is proposed to balance three objectives, including high compression ratio, controllable error, and low decompression cost. Specifically, we use and combine digitized curve sparse representation, floating-point data compression, and octree space partitioning to adaptively achieve the objectives. Results have shown that our data reuse framework could acquire tens of times acceleration in the resource-limited environment compared to on-the-fly particle tracing, and keep controllable information loss. Moreover, our method could provide fast integral curve retrieval for more complex data, such as unstructured mesh data.

97 MATHEMATICS AND COMPUTING↗