Search NASA⌕ Search

SEARCH · Search NASA

Results for “Query log”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 37 records · Page 2

Expandable Log Analyzing Framework

Prior to my internship, I was informed that a previous intern had built a tool to analyse MongoDB logs and look for invalid access attempts, which served as a great reference point for my project. I was initially tasked with expanding on her prototype and filling in the gaps such as integrating it with the main monitoring tool the lab uses. Eventually, the scope grew, expanding to support other databases and a growing collection of tools. I organized the framework around an observer pattern, meaning one point in the program sending updates to the rest of the framework. Every time a log was read and parsed, it was sent to be processed by the tools, using the type of event as a means to determine which tools should get a chance to act on the log. This decouples the tools from the log reader, making future updates and additions much easier. The framework processes MongoDB logs at ~135,000 entries per second and PostgreSQL logs at ~170,500 entries per second, accurately detecting anomalies such as slow queries and connections from unknown addresses. This framework serves to fill gaps in database monitoring tools currently implemented at the lab, such as tracking failed authentication for PostgreSQL and MongoDB which had very minimal or none before this framework. National labs such as Fermilab hold sensitive data and valuable computing resources, making them attractive targets. Monitoring intrusion attempts on databases is made much easier by this comprehensive monitoring suite.

Clark, Dylan [Unlisted, IL]↗

Expandable Log Analyzing Framework

Prior to my internship, I was informed that a previous intern had built a tool to analyse MongoDB logs and look for invalid access attempts, which served as a great reference point for my project. I was initially tasked with expanding on her prototype and filling in the gaps such as integrating it with the main monitoring tool the lab uses. Eventually, the scope grew, expanding to support other databases and a growing collection of tools. I organized the framework around an observer pattern, meaning one point in the program sending updates to the rest of the framework. Every time a log was read and parsed, it was sent to be processed by the tools, using the type of event as a means to determine which tools should get a chance to act on the log. This decouples the tools from the log reader, making future updates and additions much easier. The framework processes MongoDB logs at ~135,000 entries per second and PostgreSQL logs at ~170,500 entries per second, accurately detecting anomalies such as slow queries and connections from unknown addresses. This framework serves to fill gaps in database monitoring tools currently implemented at the lab, such as tracking failed authentication for PostgreSQL and MongoDB which had very minimal or none before this framework. National labs such as Fermilab hold sensitive data and valuable computing resources, making them attractive targets. Monitoring intrusion attempts on databases is made much easier by this comprehensive monitoring suite.

Clark, Dylan [Unlisted, IL]↗

Database-Agnostic Log Analysis and Monitoring Framework

Prior to my internship, I was informed that a previous intern had built a tool to analyse MongoDB logs and look for invalid access attempts, which served as a great reference point for my project. I was initially tasked with expanding on her prototype and filling in the gaps such as integrating it with the main monitoring tool the lab uses. Eventually, the scope grew, expanding to support other databases and a growing collection of tools. I organized the framework around an observer pattern, meaning one point in the program sending updates to the rest of the framework. Every time a log was read and parsed, it was sent to be processed by the tools, using the type of event as a means to determine which tools should get a chance to act on the log. This decouples the tools from the log reader, making future updates and additions much easier. The framework processes MongoDB logs at ~135,000 entries per second and PostgreSQL logs at ~170,500 entries per second, accurately detecting anomalies such as slow queries and connections from unknown addresses. This framework serves to fill gaps in database monitoring tools currently implemented at the lab, such as tracking failed authentication for PostgreSQL and MongoDB which had very minimal or none before this framework. National labs such as Fermilab hold sensitive data and valuable computing resources, making them attractive targets. Monitoring intrusion attempts on databases is made much easier by this comprehensive monitoring suite.

Clark, Dylan [Unlisted, US, IL; Fermilab]↗

Diventi

Diventi provides a write optimized index of network logs such as netflow or zeek. The program will ingest various network traffic logs (zeek connection logs or NetflowV5, v9 or IPFix logs) and index each IP address and time of its activity. This database can then be queried for timely tracking of network activity. It currently uses the ft-index library from Percona but is adaptable to any key-value store that supports the Berkeley DB api.SAND2019-15037 M Sandia National Laboratories is a multimission laboratory managed and operated by National Technology & Engineering Solutions of Sandia, LLC, a wholly owned subsidiary of Honeywell International Inc., for the U.S. Department of Energy’s National Nuclear Security Administration under contract DE-NA0003525.

Kroeger, Thomas↗

Combining Tensor Decompositions and Graph Analytics to Provide Cyber Situational Awareness at HPC Scale

This paper describes MADHAT (Multidimensional Anomaly Detection fusing HPC, Analytics, and Tensors), an integrated workflow that demonstrates the applicability of HPC resources to the problem of maintaining cyber situational awareness. MADHAT combines two high-performance packages: ENSIGN for large-scale sparse tensor decompositions and HAGGLE for graph analytics. Tensor decompositions isolate coherent patterns of network behavior in ways that common clustering methods based on distance metrics cannot. Parallelized graph analysis then uses directed queries on a representation that combines the elements of identified patterns with other available information (such as additional log fields, domain knowledge, network topology, whitelists and blacklists, prior feedback, and published alerts) to confirm or reject a threat hypothesis, collect context, and raise alerts. MADHAT was developed using the collaborative HPC Architecture for Cyber Situational Awareness (HACSAW) research environment and evaluated on structured network sensor logs collected from Defense Research and Engineering Network (DREN) sites using HPC resources at the U.S. Army Engineer Research and Development Center DoD Supercomputing Resource Center (ERDC DSRC). To date, MADHAT has analyzed logs with over 650 million entries.

Cyber security, Tensor decomposition, graph analyt↗

Bridging the gaps : joining information sources with Splunk.

Supercomputers are composed of many diverse components, operated at a variety of scales, and function as a coherent whole. The resulting logs are thus diverse in format, interrelated at multiple scales, and provide evidence of faults across subsystems. When combined with system configuration information, insights on both the downstream effects and upstream causes of events can be determined. However, difficulties in joining the data and expressing complex queries slow the speed at which actionable insights can be obtained. Effectively connecting data experts and data miners faces similar hurdles. This paper describes our experience with applying the Splunk log analysis tool as a vehicle to combine both data, and people. Splunk's search language, lookups, macros, and subsearches reduce hours of tedium to seconds of simplicity, and its tags, saved searches, and dashboards offer both operational insights and collaborative vehicles.

99 GENERAL AND MISCELLANEOUS↗

Bridging the gaps : joining information sources with Splunk.

Supercomputers are composed of many diverse components, operated at a variety of scales, and function as a coherent whole. The resulting logs are thus diverse in format, interrelated at multiple scales, and provide evidence of faults across subsystems. When combined with system configuration information, insights on both the downstream effects and upstream causes of events can be determined. However, difficulties in joining the data and expressing complex queries slow the speed at which actionable insights can be obtained. Effectively connecting data experts and data miners faces similar hurdles. This paper describes our experience with applying the Splunk log analysis tool as a vehicle to combine both data, and people. Splunk's search language, lookups, macros, and subsearches reduce hours of tedium to seconds of simplicity, and its tags, saved searches, and dashboards offer both operational insights and collaborative vehicles.

99 GENERAL AND MISCELLANEOUS↗

Fallon FORGE: Well Log Database

Master well database for all logs and other well data pertaining to the Fallon FORGE site. Tables are formatted in NGDS Teir 3 schema where appropriate; redundancies between tables are omitted, and should be queried off of the main WellHeader table. All relationships are set on the "HeaderURI" primary key. Tables: WellHeader - well locations and information ChemFluid - aqueous chemistry analyses ChemRock - rock chemistry analyses DensityCore - rock densities from core FracturesInduced - induced borehole fractures FracturesNatural - natural borehole factures LithIntercepts - lithology picks from cuttings and core MagSuscCore - magnetic susceptibility for core samples MagSuscPmag - magnetic susceptibility for pmag samples PTSLogs - pressure-temperature-spin logs WirelineLogs - digital wireline logs (21-31 only)

15 GEOTHERMAL ENERGY↗

XRootD popularity on hadoop clusters

Performance data and metadata of the computing operations at the CMS experiment are collected through a distributed monitoring infrastructure, currently relying on a traditional Oracle database system. This paper shows how to harness Big Data architectures in order to improve the throughput and the efficiency of such monitoring. A large set of operational data - user activities, job submissions, resources, file transfers, site efficiencies, software releases, network traffic, machine logs - is being injected into a readily available Hadoop cluster, via several data streamers. The collected metadata is further organized running fast arbitrary queries; this offers the ability to test several Map&Reduce-based frameworks and measure the system speed-up when compared to the original database infrastructure. By leveraging a quality Hadoop data store and enabling an analytics framework on top, it is possible to design a mining platform to predict dataset popularity and discover patterns and correlations.

97 MATHEMATICS AND COMPUTING↗

Survival Effect of Neoadjuvant Radiotherapy Before Esophagectomy for Patients With Esophageal Cancer: A Surveillance, Epidemiology, and End-Results Study

Purpose: The role of neoadjuvant radiotherapy (NeoRT) before definitive surgery for esophageal cancer remains controversial. This study used a large population-based database to assess the effect of NeoRT on survival for patients treated with definitive surgery. Methods and Materials: The overall survival (OS) and cause-specific survival for patients with Stage T2-T4, any N, M0 (cT2-T4M0) esophageal cancer who had undergone definitive surgery between 1998 and 2004 were analyzed by querying the Surveillance, Epidemiology, and End-Results database. Kaplan-Meier survival curves were generated and univariate comparisons were made using the log-rank test. Cox proportional hazards survival regression multivariate analysis was performed with NeoRT, T stage (T2 vs. T3-T4), pathologic nodal status (pN0 vs. pN1), number of nodes dissected (>10 vs. {<=}10), histologic type (adenocarcinoma vs. squamous cell carcinoma), age (<65 vs. {>=}65 years), and gender as covariates. Results: A total of 1,033 patients were identified. Of these, 441 patients received NeoRT and 592 underwent esophagectomy alone; 77% were men, 67% had adenocarcinoma, and 72% had Stage T3-T4 disease. The median OS and cause-specific survival were both significantly greater for patients who received NeoRT compared with esophagectomy alone (27 vs. 18 months and 35 vs. 21 months, respectively, p <0.0001). The 3-year OS rate was also significantly greater in the NeoRT group (43% vs. 30%). On multivariate analysis, NeoRT, age <65 years, adenocarcinoma histologic type, female gender, pN0 status, >10 nodes dissected, and Stage T2 disease were all independently correlated with increased OS. Conclusion: These results support the use of NeoRT for patients with esophageal cancer. Prospective studies are needed to confirm these results.

62 RADIOLOGY AND NUCLEAR MEDICINE↗

When Workflow Management Systems and Logging Systems Meet: Analyzing Large-Scale Execution Traces

This poster shows the benefits of integrating a workflow management system with logging and log mining capabilities. By combing two existing, mature technologies: Pegasus-WMS and Netlogger, we are able to efficiently process execution logs of earthquake science workflows consisting of hundreds of thousands to one million tasks. In particular we show results of processing logs of CyberShake, a workflow application running on the TeraGrid. Client-side tools allow scientists to quickly gather statistics about a workflow run and find out which tasks executed, where they were executed, what was their runtime, etc. These statistics can be used to understand the performance characteristics of a workflow and help tune the execution parameters of the workflow management system. This poster shows the scalability of the system presenting results of uploading task execution records into the system and by showing results of querying the system for overall workflow performance information.

97 MATHEMATICS AND COMPUTING↗

The Fabric for Frontier Experiments Project at Fermilab

The FabrIc for Frontier Experiments (FIFE) project is a new, far-reaching initiative within the Fermilab Scientific Computing Division to drive the future of computing services for experiments at FNAL and elsewhere. It is a collaborative effort between computing professionals and experiment scientists to produce an end-to-end, fully integrated set of services for computing on the grid and clouds, managing data, accessing databases, and collaborating within experiments. FIFE includes 1) easy to use job submission services for processing physics tasks on the Open Science Grid and elsewhere, 2) an extensive data management system for managing local and remote caches, cataloging, querying, moving, and tracking the use of data, 3) custom and generic database applications for calibrations, beam information, and other purposes, 4) collaboration tools including an electronic log book, speakers bureau database, and experiment membership database. All of these aspects will be discussed in detail. FIFE sets the direction of computing at Fermilab experiments now and in the future, and therefore is a major driver in the design of computing services worldwide.

Kirby, Michael↗

Crawling the Control System

Information about accelerator operations and the control system resides in various formats in a variety of places on the lab network. There are operating procedures, technical notes, engineering drawings, and other formal controlled documents. There are programmer references and API documentation generated by tools such as doxygen and javadoc. There are the thousands of electronic records generated by and stored in databases and applications such as electronic logbooks, training materials, wikis, and bulletin boards and the contents of text-based configuration files and log files that can also be valuable sources of information. The obvious way to aggregate all these sources is to index them with a search engine that users can then query from a web browser. Toward this end, the Google "mini" search appliance was selected and implemented because of its low cost and its simple web-based configuration and management. In addition to crawling and indexing electronic documents, the appliance provides an API that has been used to supplement search results with live control system data such as current values of EPICS process variables and graphs of recent data from the archiver.

43 PARTICLE ACCELERATORS↗

Collection And Analysis Of Telemetry For The Cyote Heuristic

CATCH CLI focuses on gathering telemetry data, storing it in the Neo4j database, querying for Mitre ATT&CK patterns, and creating STIX 2.1 reports. Key Components: Analysis Modules: Analyze data to detect attack patterns. GoSTOTS Collection Engines: Collect telemetry data. These tools can be used together or individually. Analysis modules rely on data from specific engines to identify attack patterns. Source Code Organization: Engines: CATCH/catch/cmd/collection Modules: CATCH/catch/cmd/analysis CGUI Overview CATCH Graphical User Interface (CGUI) offers a graphical shell to execute CATCH CLI, allowing easy editing of: Analysis Modules Database configurations Profiles (collection and device settings) Neo4j Overview Neo4j is a graph database using the Cypher query language, storing data in JSON. It seamlessly integrates with STIX 2.1 data for: Data Submission: CATCH Collection Engines Data Querying: Analysis Modules CATCH modifies STIX 2.1 data for Neo4j submission and reverts it back during querying. STIG Overview Structured Threat Intelligence Graph (STIG) is a tool for creating, editing, querying, analyzing, and visualizing threat intelligence using STIX 2.1 and storing data in Neo4j. Usage Tools can be run: Manually (CLI): Refer to CATCH documentation User Interface: Run ./cgui/CGUI or go run ./cgui/ Additional Information Logging System: Detailed in the config documentation Further Documentation: Available for CATCH and CGUI

Madsen, MichaelJ. [Idaho National Laboratory (INL)↗

Social media based NPL system to find and retrieve ARM data: Concept paper

Information connectivity and retrieval has a role in our daily lives. The most pervasive source of online information is databases. The amount of data is growing at rapid rate and database technology is improving and having a profound effect. Almost all online applications are storing and retrieving information from databases. One challenge in supplying the public with wider access to informational databases is the need for knowledge of database languages like Structured Query Language (SQL). Although the SQL language has been published in many forms, not everybody is able to write SQL queries. Another challenge is that it may not be practical to make the public aware of the structure of the database. There is a need for novice users to query relational databases using their natural language. To solve this problem, many natural language interfaces to structured databases have been developed. The goal is to provide more intuitive method for generating database queries and delivering responses. Social media makes it possible to interact with a wide section of the population. Through this medium, and with the help of Natural Language Processing (NLP) we can make the data of the Atmospheric Radiation Measurement Data Center (ADC) more accessible to the public. We propose an architecture for using Apache Lucene/Solr [1], OpenML [2,3], and Kafka [4] to generate an automated query/response system with inputs from Twitter5, our Cassandra DB, and our log database. Using the Twitter API and NLP we can give the public the ability to ask questions of our database and get automated responses.

Devarakonda, Ranjeet↗

Global disease monitoring and forecasting with Wikipedia

Infectious disease is a leading threat to public health, economic stability, and other key social structures. Efforts to mitigate these impacts depend on accurate and timely monitoring to measure the risk and progress of disease. Traditional, biologically-focused monitoring techniques are accurate but costly and slow; in response, new techniques based on social internet data, such as social media and search queries, are emerging. These efforts are promising, but important challenges in the areas of scientific peer review, breadth of diseases and countries, and forecasting hamper their operational usefulness. We examine a freely available, open data source for this use: access logs from the online encyclopedia Wikipedia. Using linear models, language as a proxy for location, and a systematic yet simple article selection procedure, we tested 14 location-disease combinations and demonstrate that these data feasibly support an approach that overcomes these challenges. Specifically, our proof-of-concept yields models with up to 0.92, forecasting value up to the 28 days tested, and several pairs of models similar enough to suggest that transferring models from one location to another without re-training is feasible. Based on these preliminary results, we close with a research agenda designed to overcome these challenges and produce a disease monitoring and forecasting system that is significantly more effective, robust, and globally comprehensive than the current state of the art.

59 BASIC BIOLOGICAL SCIENCES↗

A Scalable Monitoring for the CMS Filter Farm Based on Elasticsearch

A flexible monitoring system has been designed for the CMS File-based Filter Farm making use of modern data mining and analytics components. All the metadata and monitoring information concerning data flow and execution of the HLT are generated locally in the form of small documents using the JSON encoding. These documents are indexed into a hierarchy of elasticsearch (es) clusters along with process and system log information. Elasticsearch is a search server based on Apache Lucene. It provides a distributed, multitenant-capable search and aggregation engine. Since es is schema-free, any new information can be added seamlessly and the unstructured information can be queried in non-predetermined ways. The leaf es clusters consist of the very same nodes that form the Filter Farm thus providing natural horizontal scaling. A separate central” es cluster is used to collect and index aggregated information. The fine-grained information, all the way to individual processes, remains available in the leaf clusters. The central es cluster provides quasi-real-time high-level monitoring information to any kind of client. Historical data can be retrieved to analyse past problems or correlate them with external information. We discuss the design and performance of this system in the context of the CMS DAQ commissioning for LHC Run 2.

Andre, J. M.↗

Retrieving Historical Electrorefining Data

Pyrochemical Operations began at Los Alamos National Laboratory (LANL) during 1962 (1). Electrorefining (ER) has been implemented as a routine process since the 1980’s. The process data that went through the ER operation was recorded but had never been logged in an online database. Without a database new staff members are hindered in their work by the lack of information. To combat the issue a database in Access was created to collect the historical data. The years from 2000 onward were entered and queries were created to analyze trends. These trends will aid engineering and operations staff to reach optimal performance for the startup of the new lines.

38 RADIATION CHEMISTRY, RADIOCHEMISTRY, AND NUCLEA↗