Search NASA⌕ Search

SEARCH · Search NASA

Results for “NetFlow”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

18 records
All sourcesNASA NTRSDOE OSTI
DOE OSTI2017

NetFlow Dynamics

NetFlow Dynamics is a web-accessible analysis environment for simulating dynamic flows of materials on model networks. Performing a simulation requires both the NetFlow Dynamics application and a network model which is a description of the structure of the nodes and edges of a network including the flow capacity of each edge and the storage capacity of each node, and the sources and sinks of the material flowing on the network. NetFlow Dynamics consists of databases for storing network models, algorithms to calculate flows on networks, and a GIS-based graphical interface for performing simulations and viewing simulation results. Simulated flows are dynamic in the sense that flows on each edge of the network and inventories at each node change with time and can be out of equilibrium with boundary conditions. Any number of network models could be simulated using Net Flow Dynamics. To date, the models simulated have been models of petroleum infrastructure. The main model has been the National Transportation Fuels Model (NTFM), a network of U.S. oil fields, transmission pipelines, rail lines, refineries, tank farms, and distribution terminals. NetFlow Dynamics supports two different flow algorithms, the Gradient Flow algorithm and the Inventory Control algorithm, that were developed specifically for the NetFlow Dynamics application. The intent is to add additional algorithms in the future as needed. The ability to select from multiple algorithms is desirable because a single algorithm never covers all analysis needs. The current algorithms use a demand-driven capacity-constrained formulation which means that the algorithms strive to use all available capacity and stored inventory to meet desired flows to sinks, subject to the capacity constraints of each network component. The current flow algorithms are best suited for problems in which a material flows on a capacity-constrained network representing a supply chain in which the material supplied can be stored at each node of the network. In the petroleum models, the flowing materials are crude oil and refined products that can be stored at tank farms, refineries, or terminals (i.e. the nodes of the network). Examples of other network models that could be simulated are currency flowing in a financial network, agricultural products moving to market, or natural gas flowing on a pipeline network.

Corbet Jr., Thomas F↗
DOE OSTI2016

Inferring Netflow Data

Netflow data is a specific format for looking at network traffic. With this format, many security tests are easy to run. Cloud Service Providers produce logs that do not contain all of the content that Netflow data contains. There is a possibility that the fields that are in Netflow data but absent in Cloud Service Provider logs can be inferred from the fields that are present in the Cloud Service Provider logs. By inferring the missing data fields, the same test that can be run with Netflow data would be possible to run on the logs produced by Cloud Service Providers. Cloud Service Providers include Amazon Web Services and Box.com. There are multiple different types of Cloud Services, and each provider handles them differently and produces different logs. There are IaaS (meaning infrastructure as a service), Saas (meaning software as a service), and Paas (meaning platform as a service). Each provides a different use for the user.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2016

Netflow Theory Manual

The purpose of this report is to document the theoretical models utilized by the computer code NETFLOW. This report will focus on the theoretical models used to analyze high Mach number fully compressible transonic flows in piping networks.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2014

Discrete Mathematical Approaches to Graph-Based Traffic Analysis

Modern cyber defense and anlaytics requires general, formal models of cyber systems. Multi-scale network models are prime candidates for such formalisms, using discrete mathematical methods based in hierarchically-structured directed multigraphs which also include rich sets of labels. An exemplar of an application of such an approach is traffic analysis, that is, observing and analyzing connections between clients, servers, hosts, and actors within IP networks, over time, to identify characteristic or suspicious patterns. Towards that end, NetFlow (or more generically, IPFLOW) data are available from routers and servers which summarize coherent groups of IP packets flowing through the network. In this paper, we consider traffic analysis of Netflow using both basic graph statistics and two new mathematical measures involving labeled degree distributions and time interval overlap measures. We do all of this over the VAST test data set of 96M synthetic Netflow graph edges, against which we can identify characteristic patterns of simulated ground-truth network attacks.

Joslyn, Cliff A.↗
DOE OSTI2014

A Network Management System for Handling Scientific Data Flows

Large scientific data transfers often occur at high rates causing increased burstiness in Internet traffic. To limit the adverse effects of these high-rate large-sized flows, which are referred to as $\alpha $ flows, on delay-sensitive audio/video flows, a network management system called Alpha Flow Traffic Engineering System (AFTES) is proposed for intra-domain traffic engineering. An offline approach is used in which AFTES analyzes NetFlow records collected by routers, extracts source–destination address prefixes of $\alpha $ flows, and uses these prefixes to configure firewall filters at ingress routers of a provider’s network to redirect future $\alpha $ flows to traffic-engineered paths and isolated queues. The effectiveness of this scheme was evaluated through an analysis of 7 months of NetFlow data obtained from an ESnet router. For this data set, 91 % of bytes generated by $\alpha $ flows during high-rate intervals would have been directed had AFTES been deployed. The negative aspect of using address prefixes in firewall filters, i.e., the redirection of $\beta $ flows to $\alpha $-flow paths/queues, was also quantified.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2017

Automating Network Node Behavior Characterization by Mining Communication Patterns

Enterprise networks of scale are complex, dynamic computing environments that respond to evolv- ing business objectives and requirements. Characteriz- ing system behaviors in these environments is essential for network management and cyber security operations. Characterization of system’s communication is typical and is supported using network flow information (NetFlow). Related work has characterized behavior using theoretical graph metrics; results are often difficult to interpret by enterprise staff. We propose a different approach, where flow information is mapped to sets of tags that contextualize the data in terms of network principals and enterprise concepts. Frequent patterns are then extracted and are expressed as behaviors. Behaviors can be com- pared, identifying systems expressing similar behaviors. We evaluate the approach using flow information collected by a third party.

cyber system modeling↗
DOE OSTI2013

Massive Scale Cyber Traffic Analysis: A Driver for Graph Database Research

We describe the significance and prominence of network traffic analysis (TA) as a graph- and network-theoretical domain for advancing research in graph database systems. TA involves observing and analyzing the connections between clients, servers, hosts, and actors within IP networks, both at particular times and as extended over times. Towards that end, NetFlow (or more generically, IPFLOW) data are available from routers and servers which summarize coherent groups of IP packets flowing through the network. IPFLOW databases are routinely interrogated statistically and visualized for suspicious patterns. But the ability to cast IPFLOW data as a massive graph and query it interactively, in order to e.g.\ identify connectivity patterns, is less well advanced, due to a number of factors including scaling, and their hybrid nature combining graph connectivity and quantitative attributes. In this paper, we outline requirements and opportunities for graph-structured IPFLOW analytics based on our experience with real IPFLOW databases. Specifically, we describe real use cases from the security domain, cast them as graph patterns, show how to express them in two graph-oriented query languages SPARQL and Datalog, and use these examples to motivate a new class of "hybrid" graph-relational systems.

Joslyn, Cliff A.↗
DOE OSTI2019

Diventi

Diventi provides a write optimized index of network logs such as netflow or zeek. The program will ingest various network traffic logs (zeek connection logs or NetflowV5, v9 or IPFix logs) and index each IP address and time of its activity. This database can then be queried for timely tracking of network activity. It currently uses the ft-index library from Percona but is adaptable to any key-value store that supports the Berkeley DB api.SAND2019-15037 M Sandia National Laboratories is a multimission laboratory managed and operated by National Technology & Engineering Solutions of Sandia, LLC, a wholly owned subsidiary of Honeywell International Inc., for the U.S. Department of Energy’s National Nuclear Security Administration under contract DE-NA0003525.

Kroeger, Thomas↗
DOE OSTI2020

Hylia: Time-Series Library for Network Operations (Hylia) v1

The ability to analyze time-series data in network operations is critical to ensure optimal operations and management in wide area networks, sensor networks, and cloud networks. Hylia is a collection of multiple time-series prediction algorithms designed specifically to work with network monitoring data sets such as SNMP, NETFLOW and other monitoring data. It is designed to recognize these features and perform the extrapolation of the data.

Kiran, Mariam↗
DOE OSTI2016

Analysis of a Full Scale Blowdown Due to a Mechanical Failure of a Pressure Relief Device in a Natural Gas Vehicle Maintenance Facility

A computational fluid dynamics (CFD) analysis of a natural gas vehicle experiencing a mechanical failure of a pressure relief device on a full CNG cylinder was completed to determine the resulting amount and location of flammable gas. The resulting overpressure if it were to ignite was also calculated. This study completes what is discussed in Ekoto et al. which covers other related leak scenarios. We are not determining whether or not this is a credible release, rather just showing the result of a possible worst case scenario. The Sandia National Laboratories computational tool Netflow was used to calculate the leak velocity and temperature. The in - house CFD code Fuego was used to determine the flow of the leak into the maintenance garage. A maximum flammable mass of 35 kg collected along the roof of the garage. This would result in an overpressure that could do considerable damage if it were to ignite at the time of this maximum volume. It is up to the code committees to decide whet her this would be a credible leak, but if it were, there should be preventions to keep the flammable mass from igniting.

03 NATURAL GAS↗
DOE OSTI2021

NFD Methodology Report

The NetFlow Dynamics (NFD) model was developed for estimating the availability of a commodity supplied by a national- or regional-scale infrastructure following unexpected disruption of one or more of its components. The large scope of the disruptions of interest produce changes in availability lasting days to weeks. Consequently, the model does not resolve daily variations in system state and does not include the long-term processes that cause infrastructures to evolve as assets are added and removed according to owners ’planning decisions. NFD simulates fluid flow, including petroleum and other incompressible fluids, as well as natural gas and other compressible fluids, through pipeline networks characterized by limits on transmission capacity and storage. It was designed to enable efficient exploration of possible transmission system responses to large-scale disruptions lasting for days or longer. The model formulation reflects constraints on transmission and storage capacity imposed by the physical system assets. Those capacity limits are input parameters and are not derived from more basic system properties such as pipeline diameters and compressor power. A system’s response to a large disruption is controlled by operational decisions as well as damage to physical assets. The NFD model formulation allows users to efficiently consider alternative scenarios about the way remaining capacity might be used so that the analysis result appropriately reflects uncertainties about operator response.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2016

Utilizing Genetic Programming to Identify Failures in ICS Networks

Previously, researchers have attempted to apply machine learning techniques to network anomaly detection problems. Due to the staggering amount of variety that can occur in normal networks, the results have often been underwhelming. These challenges are far less pronounced when considering industrial control system (ICS) networks. The recurrent nature of these networks results in less noise and more consistent patterns for a machine learning algorithm to recognize. Here, we propose a method of evolving decision trees through genetic programming (GP) in order to detect network anomalies, such as device outages. Our approach extracts over a dozen features from network packet captures and netflows, normalizes them, and relates them in decision trees using fuzzy logic operators. Furthermore, the trees were used to detect three specific network events from three different points on the network across a statistically significant number of runs and achieved 100% accuracy on five of the nine experiments. When the trees attempted to detect more challenging events at points of presence further from the occurrence, the accuracy averaged to above 98%. Finally, using our method, all of the evolutionary cycles of the GP algorithm are computed a-priori, allowing the best resultant trees to be deployed as semi-real-time sensors with little overhead.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2023

polishCLR: A Nextflow Workflow for Polishing PacBio CLR Genome Assemblies

Long-read sequencing has revolutionized genome assembly, yielding highly contiguous, chromosome-level contigs. However, assemblies from some third generation long read technologies, such as Pacific Biosciences (PacBio) continuous long reads (CLR), have a high error rate. Such errors can be corrected with short reads through a process called polishing. Although best practices for polishing non-model de novo genome assemblies were recently described by the Vertebrate Genome Project (VGP) Assembly community, there is a need for a publicly available, reproducible workflow that can be easily implemented and run on a conventional high performance computing environment. Here, we describe polishCLR (https://github.com/isugifNF/polishCLR), a reproducible Nextflow workflow that implements best practices for polishing assemblies made from CLR data. PolishCLR can be initiated from several input options that extend best practices to suboptimal cases. It also provides re-entry points throughout several key processes, including identifying duplicate haplotypes in purge_dups, allowing a break for scaffolding if data are available, and throughout multiple rounds of polishing and evaluation with Arrow and FreeBayes. PolishCLR is containerized and publicly available for the greater assembly community as a tool to complete assemblies from existing, error-prone long-read data.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2015

A Visual Analytics Paradigm Enabling Trillion-Edge Graph Exploration

We present a visual analytics paradigm and a system prototype for exploring web-scale graphs. A web-scale graph is described as a graph with ~one trillion edges and ~50 billion vertices. While there is an aggressive R&D effort in processing and exploring web-scale graphs among internet vendors such as Facebook and Google, visualizing a graph of that scale still remains an underexplored R&D area. The paper describes a nontraditional peek-and-filter strategy that facilitates the exploration of a graph database of unprecedented size for visualization and analytics. We demonstrate that our system prototype can 1) preprocess a graph with ~25 billion edges in less than two hours and 2) support database query and visualization on the processed graph database afterward. Based on our computational performance results, we argue that we most likely will achieve the one trillion edge mark (a computational performance improvement of 40 times) for graph visual analytics in the near future.

graph visualization↗
DOE OSTI2021

Anonymization of Network Traces Data through Condensation-based Differential Privacy

Network traces are considered a primary source of information to researchers, who use them to investigate research problems such as identifying user behavior, analyzing network hierarchy, maintaining network security, classifying packet flows, and much more. However, most organizations are reluctant to share their data with a third party or the public due to privacy concerns. Therefore, data anonymization prior to sharing becomes a convenient solution to both organizations and researchers. Although several anonymization algorithms are available, few of them allow sufficient privacy (organization need), acceptable data utility (researcher need), and efficient data analysis at the same time. This article introduces a condensation-based differential privacy anonymization approach that achieves an improved tradeoff between privacy and utility compared to existing techniques and produces anonymized network trace data that can be shared publicly without lowering its utility value. Our solution also does not incur extra computation overhead for the data analyzer. A prototype system has been implemented, and experiments have shown that the proposed approach preserves privacy and allows data analysis without revealing the original data even when injection attacks are launched against it. When anonymized datasets are given as input to graph-based intrusion detection techniques, they yield almost identical intrusion detection rates as the original datasets with only a negligible impact.

97 MATHEMATICS AND COMPUTING↗