Results for “Logging” — Page 3

DOE OSTI2026

Expandable Log Analyzing Framework

Prior to my internship, I was informed that a previous intern had built a tool to analyse MongoDB logs and look for invalid access attempts, which served as a great reference point for my project. I was initially tasked with expanding on her prototype and filling in the gaps such as integrating it with the main monitoring tool the lab uses. Eventually, the scope grew, expanding to support other databases and a growing collection of tools. I organized the framework around an observer pattern, meaning one point in the program sending updates to the rest of the framework. Every time a log was read and parsed, it was sent to be processed by the tools, using the type of event as a means to determine which tools should get a chance to act on the log. This decouples the tools from the log reader, making future updates and additions much easier. The framework processes MongoDB logs at ~135,000 entries per second and PostgreSQL logs at ~170,500 entries per second, accurately detecting anomalies such as slow queries and connections from unknown addresses. This framework serves to fill gaps in database monitoring tools currently implemented at the lab, such as tracking failed authentication for PostgreSQL and MongoDB which had very minimal or none before this framework. National labs such as Fermilab hold sensitive data and valuable computing resources, making them attractive targets. Monitoring intrusion attempts on databases is made much easier by this comprehensive monitoring suite.

Clark, Dylan [Unlisted, IL]↗

DOE OSTI2026

Expandable Log Analyzing Framework

Prior to my internship, I was informed that a previous intern had built a tool to analyse MongoDB logs and look for invalid access attempts, which served as a great reference point for my project. I was initially tasked with expanding on her prototype and filling in the gaps such as integrating it with the main monitoring tool the lab uses. Eventually, the scope grew, expanding to support other databases and a growing collection of tools. I organized the framework around an observer pattern, meaning one point in the program sending updates to the rest of the framework. Every time a log was read and parsed, it was sent to be processed by the tools, using the type of event as a means to determine which tools should get a chance to act on the log. This decouples the tools from the log reader, making future updates and additions much easier. The framework processes MongoDB logs at ~135,000 entries per second and PostgreSQL logs at ~170,500 entries per second, accurately detecting anomalies such as slow queries and connections from unknown addresses. This framework serves to fill gaps in database monitoring tools currently implemented at the lab, such as tracking failed authentication for PostgreSQL and MongoDB which had very minimal or none before this framework. National labs such as Fermilab hold sensitive data and valuable computing resources, making them attractive targets. Monitoring intrusion attempts on databases is made much easier by this comprehensive monitoring suite.

Clark, Dylan [Unlisted, IL]↗

NASA NTRS2020

What Can We Learn from One Billion Ground System Log Messages?

Shortage of log-based data in a ground system they have traditionally been the under achievers in a satellite ground system. This is due to several factors: Once log messages scroll out of view on the TTC event console window they are soon forgotten. Application and system log files are scattered across directories within a system, across a multitude of servers, and across one or more databases making access cumbersome. Typical tools to perform log file content searching are generally crude and typically only employed as part of trouble-shooting exercises.As we move towards satellite constellations and fleets and add even more status information, the number of messages keeps growing. One mission now estimates that they could generate 3,000,000 messages per day 1 billion per year - for the life of their mission. What to do with those 1 billion messages? That is the challenge. With the recent technological advances in the management of large data sets, text-based processing, and data analytics, there are now capabilities that we can provide to the ground system engineers and satellite operators to address what we postulate are missed opportunities. Advanced real-time log analysis can allow us to be less reactionary in favor of being more proactive. Analytics goals include the ability to: Identify root cause of unexpected events, failures or error conditions enabled by correlating disparate data. Detect security breaches attempts before they are successful. Help admins ensure IT resources continue running optimally. Identify trends and patterns that may indicate impending failures or error conditions for valuable assets before they happen. Compare satellites in a fleet or constellation in terms of number of alarms, number of command sent to them, etc.. Answer questions like "Are the operations support needs increasing over the past year?" or "Have we seen this combination of alarm conditions before?" But really, once the tools are readily available the users will start realizing what can be done with their new powers. In this presentation we will show the results of analyzing millions of actual mission operations log messages, how the results can be displayed to the user, and how new products now available as open source can be applied to the challenges of large scale time-tagged text-based mission operations messages. Flight operations team members believe that this is a powerful new option for how they assess overall system and space asset health. Technical descriptions of the design, tools, and storage will be provided. One billion messages? Bring'em on!

Orsborne, Sharon↗

DOE OSTI2026

Database-Agnostic Log Analysis and Monitoring Framework

Prior to my internship, I was informed that a previous intern had built a tool to analyse MongoDB logs and look for invalid access attempts, which served as a great reference point for my project. I was initially tasked with expanding on her prototype and filling in the gaps such as integrating it with the main monitoring tool the lab uses. Eventually, the scope grew, expanding to support other databases and a growing collection of tools. I organized the framework around an observer pattern, meaning one point in the program sending updates to the rest of the framework. Every time a log was read and parsed, it was sent to be processed by the tools, using the type of event as a means to determine which tools should get a chance to act on the log. This decouples the tools from the log reader, making future updates and additions much easier. The framework processes MongoDB logs at ~135,000 entries per second and PostgreSQL logs at ~170,500 entries per second, accurately detecting anomalies such as slow queries and connections from unknown addresses. This framework serves to fill gaps in database monitoring tools currently implemented at the lab, such as tracking failed authentication for PostgreSQL and MongoDB which had very minimal or none before this framework. National labs such as Fermilab hold sensitive data and valuable computing resources, making them attractive targets. Monitoring intrusion attempts on databases is made much easier by this comprehensive monitoring suite.

Clark, Dylan [Unlisted, US, IL; Fermilab]↗

NASA NTRS2012

Requirements-Driven Log Analysis Extended Abstract

Imagine that you are tasked to help a project improve their testing effort. In a realistic scenario it will quickly become clear, that having an impact is diffcult. First of all, it will likely be a challenge to suggest an alternative approach which is significantly more automated and/or more effective than current practice. The reality is that an average software system has a complex input/output behavior. An automated testing approach will have to auto-generate test cases, each being a pair (i; o) consisting of a test input i and an oracle o. The test input i has to be somewhat meaningful, and the oracle o can be very complicated to compute. Second, even in case where some testing technology has been developed that might improve current practice, it is then likely difficult to completely change the current behavior of the testing team unless the technique is obviously superior and does everything already done by existing technology. So is there an easier way to incorporate formal methods-based approaches than the full edged test revolution? Fortunately the answer is affirmative. A relatively simple approach is to benefit from possibly already existing logging infrastructure, which after all is part of most systems put in production. A log is a sequence of events, generated by special log recording statements, most often manually inserted in the code by the programmers. An event can be considered as a data record: a mapping from field names to values. We can analyze such a log using formal methods, for example checking it against a formal specification. This separates running the system for analyzing its behavior. It is not meant as an alternative to testing since it does not address the important in- put generation problem. However, it offers a solution which testing teams might accept since it has low impact on the existing process. A single person might be assigned to perform such log analysis, compared to the entire testing team changing behavior.

log analysis↗

DOE OSTI2025

Machine Learning for Well Log Analysis in Uranium Mining

This project explores the use of Artificial Intelligence (AI) and Machine Learning (ML) techniques to automate well log analysis for uranium mining. Geophysical log data—spontaneous potential, resistivity, and gamma ray—were used to classify lithology, correlate well logs and identify roll front zonation patterns, which are critical for locating uranium ore bodies. Supervised ML algorithms such as eXtreme Gradient Boosting (XGBoost), Categorical Boosting (CatBoost), and Random Forest were trained to classify lithology with high accuracy. Gradient Boosting Machines (GBM), XGBoost, Random Forest, and Neural Networks were also used for role front zone identification. Moreover, a Fast Dynamic Time Warping (FastDTW) algorithm was employed for well log correlation. Additionally, sample lag was addressed using dynamic programming. Results demonstrate the potential of AI and ML to streamline well log analysis and enhance uranium exploration workflows.

11 - NUCLEAR FUEL CYCLE AND FUEL MATERIALS↗

DOE OSTI2024

Advancing subsurface analysis: Integrating computer vision and deep learning for the near real-time interpretation of borehole image logs in the Illinois Basin-Decatur Project

The accurate quantification and mapping of subsurface natural fracture systems using borehole imaging logs are critical for the success of CO 2 sequestration in geologic formations, optimization of engineered geothermal systems, and hydrocarbon production enhancement. However, traditional interpretation processes suffer from time-consuming procedures and human bias. To address these challenges and expedite fracture analysis, we investigated the application of integrated computer vision and DL workflows to automate image log analysis. Specifically, the design of our workflow was crafted to swiftly detect fractures and baffles by using actual electrical resistivity of borehole wall from microresistivity imaging device alongside their binary representation. This novel approach significantly reduces computational time while providing invaluable insights. By incorporating conventional logging and microseismic data, we present a regional subsurface natural fracture mapping technique. Through the minimization of human bias in image log analysis, our automated workflow achieves reduced fracture interpretation time and costs while ensuring robust and reproducible results. We demonstrated the efficacy of our approach by applying the workflow to the Illinois Basin-Decatur Project site. The automated workflow successfully identified major fractured zones, multiple baffles, and an interbedded layer with a high resolution of 0.01 ft or 0.12 in. (0.3 cm) and can be upscaled to any desired resolution. Validation through microseismic and image log interpretations allows for accurate and near-real-time mapping of fractures and baffles, significantly enhancing CO 2 pressure forecasting and postinjection site care. Our approach stands out due to its robustness, consistency, and reduced computational cost compared with alternative feature extraction technologies. It presents exciting possibilities for advancing CO 2 sequestration and engineered geothermal efforts by offering comprehensive and efficient fracture mapping solutions. This technology can contribute significantly to the optimization of CO 2 sequestration projects, facilitating sustainable environmental practices, and combating climate change.

Geochemistry & Geophysics↗

DOE OSTI2021

Virtual Log-Structured Storage for High-Performance Streaming

Over the past decade, given the higher number of data sources (e.g., Cloud applications, Internet of things) and critical business demands, Big Data transitioned from batch-oriented to real-time analytics. Stream storage systems, such as Apache Kafka, are well known for their increasing role in real-time Big Data analytics. For scalable stream data ingestion and processing, they logically split a data stream topic into multiple partitions. Stream storage systems keep multiple data stream copies to protect against data loss while implementing a stream partition as a replicated log. This architectural choice enables simplified development while trading cluster size with performance and the number of streams optimally managed. This paper introduces a shared virtual log-structured storage approach for improving the cluster throughput when multiple producers and consumers write and consume in parallel data streams. Stream partitions are associated with shared replicated virtual logs transparently to the user, effectively separating the implementation of stream partitioning (and data ordering) from data replication (and durability). We implement the virtual log technique in the KerA stream storage system. When comparing with Apache Kafka, KerA improves the cluster ingestion throughput by up to 4x when multiple producers write over hundreds of data streams.

consistent stream ordering↗

DOE OSTI2023

Logging Crew Attributes by Region in the Southeast USA

Urbanization, shrinking markets, and reduced forestry investment may affect harvesting efficiency in regions of the US South. To monitor these conditions, logging businesses have been tracked by surveys conducted by universities and trade associations. This project used a sampling approach coordinated with FIA utilization studies to sample logging crews based on a harvesting location. The approach was used to develop relationships among firm attributes and site attributes in six southeastern states (AL, GA, FL, NC, SC, and VA) from 2011 to 2018. The data included harvest attributes (location, harvest size and stand type) and logging firm attributes (production, crew labor, crew number, the number of machines by type, and machine age). For crew capital value, an equation was developed for this study using machine number and average machine age. The data from logging crews on 419 harvests were analyzed by region, harvest size, and stand type. Mean values for crew labor ranged from 3.1 to 7.1 workers. The average capital value per crew ranged from $\$220,000$ to $\$524,000$ per crew in the Coastal Plain with a narrower range in the Piedmont. In the Coastal Plain, higher productivity was detected for larger harvests and pine versus hardwood and mixed stands; however, in the Piedmont those trends were less obvious. Ratio of feller-bunchers, skidders and loaders were mostly 1:1:1 or 1:2:1 with 41% and 24% of samples, respectively. There were notable trends among Coastal Plain loggers regarding capital value and productivity with evidence supported by a production function. The differences in Piedmont (e.g., ownership size, market access, terrain, population density, etc.) may combine to limit daily production and labor productivity.

54 ENVIRONMENTAL SCIENCES↗

NASA NTRS1963

A THEORETICAL AND EXPERIMENTAL INVESTIGATION OF FLUSH MOUNTING CONFIGURATIONS FOR LOG PERIODIC STRUCTURES

Many types of log periodic structures have been described in the literature; but, as far as is known, flush-mounted log periodic antennas have not been studied heretofore. The purpose of this study is to investigate, primarily by experimental methods, the input impedance and radiation characteristics of a particular type of log periodic antenna when flush-mounted over a suitable cavity or recess in a conducting surface. The particular type of log periodic antenna chosen for use in the investigation is referred to as a monopole-tooth antenna and is described in detail in Part II of the report. A suitable cavity over which the monopole-tooth antenna may be mounted is described in Part III, and experimentally determined characteristics of the antenna-cavity assembly are presented in Part IV. The first objective of the study was to obtain an optimum or nearoptimum design of the antenna-cavity assembly. This objective has been reached within the limits of the experimental approach, and the results obtained are summarized in Part V of the report. A second objective, which will be reported upon at a future date, is to establish optimum design criteria and performance characteristics of the antenna-cavity assembly when the cavity is filled with low-loss dielectric material.

RADIATION FUNCTION↗

DOE OSTI2021

Combined Structural Analysis of Core and Image Log of TGH 76-31 South East of Mt Baker, Washington State

Despite active volcanism, few geothermal energy resources have been developed in the Cascades Range. Temperature Gradient Hole 76-31 was drilled to ~440 m measured depth to probe for zones where fractures provide fluid conduits that transport deep volcanic heat to shallow depths that could support baseload, carbon neutral electrical generation. These zones were predicted by a Play Fairway assessment (PFA) of resource potential along a zone 11 km west-southwest of the summit of Mount Baker Volcano. Rock core, temperature logs, and an acoustic image log were obtained. By comparison to outcrops, the core has been initially interpreted as the Chilliwack group comprised of partially metamorphosed basaltic to andesitic volcanics. Core mapping reveals complex, steeply dipping networks of fractures and brecciation along slickensided strike slip faults; clay alteration is common in many of these structures. The majority of fractures are thoroughly healed by layers of chlorite and calcite; preservation of pyrite indicates anoxic conditions. The majority of fracture porosity resides in very dense fractures a few centimeters or less in length. The image log provides good insights into attitude of fractures that fully transect the core, but generally underestimates fracture density. The combination of complex, non-planar fracture zones containing many short fractures and healing promote misinterpretation of natural fracture attitude and density in the image log. The healing and anoxic minerals are consistent with the conductive temperature gradient measured in the well below a shallow isothermal zone, although, several fractures are open or only partially healed and resulted in fluid entries into the well. Here, the equilibrated measured temperature gradient of 64ºC/km and calculated heat flow of 145 mW/m2 is more than twice the regional average, indicating local influence of the Mt Baker magmatic system at the Little Park Creek TGH site.

Cascades↗

DOE OSTI2025

Informing Plant Asset Reliability and Availability Through AI-Driven Analysis of Operator Logs

The availability and reliability of nuclear power plant (NPP) structures, systems, and components (SSCs) are critical parameters for NPP safety. Tracking these parameters is necessary but costly and labor-intensive, requiring the collection and evaluation of SSC event data such as shutdowns, startups, and failures. To show how these events are needed for the parameters an example is given: one measure of reliability is based on the number of equipment failure events and the number of run hours (i.e., the time from a startup event to a shutdown event). Here, this work investigates using artificial intelligence (AI) to mine NPP operator log entry texts for SSC event data. Four AI approaches were explored for identifying these events, including natural language processing (NLP) methods, generative AI, generative AI combined with NLP, and topic modeling. A key challenge addressed with all four approaches is the brevity of operator log entries. Among these four a neural network–based NLP method was shown to be the most promising for this application, achieving F1 scores of 86.0% for shutdowns, 92.2% for startups, and 80.4% for failures on a subject-matter-expert-curated dataset from NPP operator logs, compared to a baseline of 66.6% for a random classifier. This shows that NLP methods can perform better than generative AI. Additionally, the NLP methods combined with generative AI were shown to perform better than generative AI alone. Generative AI was most successful at providing the background information for the NLP methods to use. This work demonstrates the potential to use AI to automate parameter collection from NPP operator log entries and other records.

97 - MATHEMATICS AND COMPUTING↗

DOE OSTI2024

PyTorch Implementation of Log-Additive Convolutional Neural Networks

This code is a collection of python code that defines, trains, and tests Log-Additive Convolutional Neural Networks. The model components and training routine are based on the PyTorch python library. The code implements the Log-Additive Convolutional Neural Networks as described in Pagendam et al. 2023. In addition to the Log-Additive Convolutional Neural Networks, this library also defines the Log-Normal Density loss function as described in Pagendam et al. 2023. Code from this paper is not publicly available, so the Pytorch implementation of this type of model is unique to this library.

Callis, Skylar↗

DOE OSTI2022

Analyzing the logs of even the smallest Information Technology (IT) system can be a challenge considering they can generate millions of lines of log data in a very short time. Splunk® Enterprise is an industry leading tool that allows analysis of log data, which can enhance troubleshooting capabilities, improve system performance, and improve the security posture of an IT system. Ivanti Connect Secure© (ICS) is a market-leading platform powered by the Ivanti Secure Socket Layer Virtual Private Network (SSL VPN) appliance, providing an architecture for secure access to and protection of network resources. This paper describes an approach for using Splunk Enterprise search capabilities to perform advanced data analysis of ICS logs.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2024

Well-Log Derived Geomechanical Analysis of Microseismicity in the Mt. Simon Saline Aquifers (Illinois Basin - Decatur Project)

The Illinois Basin Decatur Project (IBDP) successfully demonstrated the safe geologic storage of carbon dioxide at a commercial scale. Within the IBDP project three deep wells (injection (CCS1), monitoring (VW1), geophysical (GM1)) were competed and geophysical logs were recorded. During injection and post-injection periods microseismic monitoring was conducted to create a miscoseismic catalog. The correlations between microseimic attributes and geomechanical well logs define major geomechanical drivers of microseismic expression to understand a reservoir response to CO2 injection in geological context. Utilizing standard sonic and density well logs, the dynamic elastic moduli were calculated and employed to correlate with microseismic pseudo-logs. A multi-dimensional Mu-rho and Lambda-rho (MRLR) hyperdimensional plots display of meaningful data and uncovered subtle relationships between elastic properties of sandstones and the seismological attributes of recorded microseismicity.

Myshakin, Evgeniy↗

NASA NTRS1981

A complete X-ray sample of the high latitude sky from HEAO-1 A-2: log N lo S and luminosity functions

An experiment was performed in which a complete X-ray survey of the 8.2 steradians of the sky at galactic latitudes where the absolute value of b is 20 deg down to a limiting sensitivity of 3.1 x ten to the minus 11th power ergs/sq cm sec in the 2-10 keV band. Of the 85 detected sources 17 were identified with galactic objects, 61 were identified with extragalactic objects, and 7 remain unidentified. The log N - log S relation for the non-galactic objects is well fit by the Euclidean relationship. The X-ray spectra of these objects were used to construct log N - log S in physical units. The complete sample of identified sources was used to construct X-ray luminosity functions, using the absolute maximum likelihood method, for clusters galaxies and active galactic nuclei.

Piccinotti, G.↗

NASA NTRS1991

Solar-A reformatted data files and observing log

An overview is presented of the Solar-A telemetry data files which are to be created and the format and organization which the files are to use. The organization chosen is to be efficient in space, to facilitate access to the data, and to allow the data to be transportable to different machines. An observing log file is to be created automatically, using the reformatted data files as the input. It will be possible to perform searches with the observing log to list cases where instruments are in certain modes and/or seeing certain signal levels. A user will be able to search the observing log and obtain a list of all cases where a given set of conditions are satisfied. An event log will be created listing the times when the instrument or spacecraft modes change.

Morrison, M. D.↗

NASA NTRS1993

Optimal message log reclamation for independent checkpointing

Independent (uncoordinated) check pointing for parallel and distributed systems allows maximum process autonomy but suffers from possible domino effects and the associated storage space overhead for maintaining multiple checkpoints and message logs. In most research on check pointing and recovery, it was assumed that only the checkpoints and message logs older than the global recovery line can be discarded. It is shown how recovery line transformation and decomposition can be applied to the problem of efficiently identifying all discardable message logs, thereby achieving optimal garbage collection. Communication trace-driven simulation for several parallel programs is used to show the benefits of the proposed algorithm for message log reclamation.

Wang, Yi-Min↗