SEARCH · Search NASA

Results for “Log data”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 37 records · Page 2

All sources NASA NTRS DOE OSTI

DOE OSTI2017

Big Data Meets HPC Log Analytics: Scalable Approach to Understanding Systems at Extreme Scale

Today's high-performance computing (HPC) systems are heavily instrumented, generating logs containing information about abnormal events, such as critical conditions, faults, errors and failures, system resource utilization, and about the resource usage of user applications. These logs, once fully analyzed and correlated, can produce detailed information about the system health, root causes of failures, and analyze an application's interactions with the system, providing valuable insights to domain scientists and system administrators. However, processing HPC logs requires a deep understanding of hardware and software components at multiple layers of the system stack. Moreover, most log data is unstructured and voluminous, making it more difficult for system users and administrators to manually inspect the data. With rapid increases in the scale and complexity of HPC systems, log data processing is becoming a big data challenge. This paper introduces a HPC log data analytics framework that is based on a distributed NoSQL database technology, which provides scalability and high availability, and the Apache Spark framework for rapid in-memory processing of the log data. The analytics framework enables the extraction of a range of information about the system so that system administrators and end users alike can obtain necessary insights for their specific needs. We describe our experience with using this framework to glean insights from the log data about system behavior from the Titan supercomputer at the Oak Ridge National Laboratory.

Park, Byung↗

DOE OSTI2025

Event Log / Raw Data

The WFIP3 event log is a curated record spanning 578 days of meteorological phenomena and field observations that complements the campaign’s high-frequency measurements. The log combines manually documented daily weather discussions with automatically derived indicators of key atmospheric processes, providing standardized, publicly available context to support model evaluation, forecast verification, and case-study selection for offshore boundary-layer research.

17 WIND ENERGY↗

DOE OSTI2021

Utah FORGE: Well 56-32 Drilling Data and Logs

This dataset consists of drilling data (Pason data spreadsheets, daily reports, days v. depth, mud logs), Schlumberger logs (FMI, shear anisotropy analysis, memory, sonic, array induction/spectral density/dual spaced neutron/gamma ray/caliper, spectral GR/temperature, and Gardner density correlation), and an end of well report (EOWR) for Utah FORGE well 56-32. This is a vertical well that will be used for seismic monitoring. It was drilled between February 7th and February 21st 2021 to a depth of 9,145 feet. More information about this well can be found at: https://utahforge.com/2021/02/09/drilling-progress-of-well-56-32/ (linked below)

15 GEOTHERMAL ENERGY↗

DOE OSTI2018

Cross-geography scientific data transferring trends and behavior

Wide area data transfers play an important role in many science applications but rely on expensive infrastructure that often delivers disappointing performance in practice. In response, we present a systematic examination of a large set of data transfer log data to characterize transfer characteristics, including the nature of the datasets transferred, achieved throughput, user behavior, and resource usage. This analysis yields new insights that can help design better data transfer tools, optimize networking and edge resources used for transfers, and improve the performance and experience for end users. Our analysis shows that (i) most of the datasets as well as individual files transferred are very small; (ii) data corruption is not negligible for large data transfers; and (iii) the data transfer nodes utilization is low. Insights gained from our analysis suggest directions for further analysis.

Liu, Zhengchun↗

DOE OSTI2026

On Road Testing Data

This dataset provides the following on road testing data: - Videos - In-vehicle dash camera videos during different testing scenarios. - Signal controller data - NTCIP log data and processed signal timing data from the corresponding signal controllers - Vehicle data - Vehicle data recorded during the testing, including GNSS, communication, CAN signals.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

DOE OSTI2025

Site H - Event Log / Derived Data

This dataset contains the event log table that provides 10-minute wind statistics from the scanning lidar at AWAKEN's site H. This is a good dataset to start from for people unfamiliar with the AWAKEN project.

17 WIND ENERGY↗

DOE OSTI2025

Site A1 - Event Log / Derived Data

This dataset contains the event log table with 10-minute wind statistics from the scanning lidar at AWAKEN's site A1. This is a good dataset to start from for people unfamiliar with the AWAKEN project.

17 WIND ENERGY↗

DOE OSTI2025

Site A2 - Event Log / Derived Data

This dataset contains the event log table with 10-minute wind statistics from the scanning lidar at AWAKEN's site A2. This is a good dataset to start from for people unfamiliar with the AWAKEN project.

17 WIND ENERGY↗

DOE OSTI2025

Sites A1, A2, & H - Event Log / Derived Data

This dataset contains the event log table that provides 10-minute wind statistics from the scanning lidars at AWAKEN's sites A1, A2, and H. This is a good dataset to start from for people unfamiliar with the AWAKEN project.

17 WIND ENERGY↗

DOE OSTI2023

Malicious Cyber Activity Detection using Zigzag Persistence

In this study we synthesize zigzag persistence from topological data analysis with autoencoder-based approaches to detect malicious cyber activity, and derive analytic insights. Cybersecurity aims to safeguard computers, networks, and servers from various forms of malicious attacks, including network damage, data theft, and activity monitoring. We focus on the cybersecurity domain and investigate the detection of malicious activity using log data. We consider the dynamics of the log data and explore the changing topology of a hypergraph representation of this data to gain insights into the underlying activity. These hypergraphs capture complex interactions between processes, together with their temporal information. To study the changing topology we use zigzag persistence, which captures how topological features persist at multiple dimensions over time. We observe that this detects malicious activity in a cyber data set. To automate this detection we implement an autoencoder trained on a vectorization of the resulting zigzag persistence barcodes. Our experimental results demonstrate the effectiveness of the autoencoder in detecting malicious activity. Overall, this study highlights the potential of zigzag persistence and its combination with temporal hypergraphs for analyzing cybersecurity log data and detecting malicious behavior.

hypergraphs, temporal hypergraph, topological data↗

DOE OSTI2018

CO 2 -SCREEN tool: Application to the oriskany sandstone to estimate prospective CO 2 storage resource

Here, the ability to accurately predict the CO 2 storage resource in saline formations is important to make high-level, energy-related government policy and business decisions. CO 2 -SCREEN (Storage prospeCtive Resource Estimation Excel aNalysis) is a tool developed by the United States Department of Energy – National Energy Technology Laboratory (US-DOE-NETL) to screen saline formations for prospective CO 2 storage resources. CO 2 -SCREEN uses DOE methods and equations to serve as a consistent mechanism for calculating prospective CO 2 storage resources. CO 2 -SCREEN is comprised of two files: an Excel file used for inputs and outputs and a GoldSim Player file used to run Monte Carlo simulations. CO 2 -SCREEN requires input of physical geologic parameters (i.e. thickness, porosity) as well as efficiency factor ranges (i.e. net-to-gross thickness) to calculate a mass storage estimate. An application of CO 2 -SCREEN is demonstrated here using well log data from the Oriskany Sandstone portion in Pennsylvania. The Oriskany Sandstone is divided into 20 km x 20 km grid cells in which 151 cells contain well log data. CO 2 -SCREEN calculates prospective CO 2 storage resource for each grid cell based on the well log data and uses lithology and depositional environment information for efficiency factor ranges. The Oriskany Sandstone CO 2 storage resource estimate for Pennsylvania, calculated by CO 2 -SCREEN, ranges from 0.07 to 1.28 gigatons (Gt) with a P 50 value of 0.32 Gt. This resource assessment analysis is done to demonstrate the use of CO 2 -SCREEN and results are comparable to previous studies which encourages the application of CO 2 -SCREEN to other saline formations and warrants exploring the expansion of this tool to assess the CO 2 storage resource in other formations such as shale and depleted oil and gas reservoirs.

03 NATURAL GAS↗

DOE OSTI2019

Clustered BSRs: Evidence for gas hydrate-bearing turbidite complexes in folded regions, example from the Perdido Fold Belt, northern Gulf of Mexico

We detail previously undocumented but extensive gas hydrate accumulations in the mouth of Perdido Canyon in the northern Gulf of Mexico. The accumulations are located within central parts of structural domes (four-way closures) and are characterized by stacked, high-amplitude bottom simulating reflections (BSRs) that we call clustered BSRs. Seismic data from Perdido Canyon show two clustered BSRs associated with turbidite sequences within two dome folds formed from tectonic folding and salt diapir rise. The northwestern (NW) and southeastern (SE) clustered BSRs have aerial extents of ~25 km2 and 50 km2, respectively. Well log data confirm gas hydrate occurs above the NW clustered BSR, within a 225 m-thick consistently high-resistivity interval that we interpret as gas hydrate in near-vertical fractures and turbidite sands. The SE dome is only drilled at the edge of the BSR; nevertheless, the well log data indicate that a 30 m-thick gas hydrate accumulation is present. Gas chromatographic logs in both domes suggest a gradual transition from predominantly microbial gas below the BSR (500–1000 m below seafloor (mbsf)) to thermogenic gas at 1000–2000 mbsf. Based on the well log data and seismic stratigraphic analysis, we find gas hydrate is concentrated in fractures in marine mud, as well as in the pores of submarine fan turbidities, where saturations reach as high as 75%. An estimate of the total gas hydrate-bound gas volume at standard temperature and pressure is between 0.04 and 0.17 trillion cubic meters (TCM) assuming average hydrate saturation of 5-20% in a ~45 m thick turbidite sand unit above the Perdido Canyon BSR area. Measured BSR extent and gas volume estimates indicate that the NW and SE reservoirs are among the largest gas hydrate occurrences known in the Gulf of Mexico.

58 GEOSCIENCES↗

DOE OSTI2022

Analyzing the logs of even the smallest Information Technology (IT) system can be a challenge considering they can generate millions of lines of log data in a very short time. Splunk® Enterprise is an industry leading tool that allows analysis of log data, which can enhance troubleshooting capabilities, improve system performance, and improve the security posture of an IT system. Ivanti Connect Secure© (ICS) is a market-leading platform powered by the Ivanti Secure Socket Layer Virtual Private Network (SSL VPN) appliance, providing an architecture for secure access to and protection of network resources. This paper describes an approach for using Splunk Enterprise search capabilities to perform advanced data analysis of ICS logs.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2018

Spatio-Temporal Analysis of HPC I/O and Connection Data

The HPC system consists of a set of layers of software and hardware for I/O and networking. System logs are helpful resources to understand what is going on in the system. A challenge is that it is non-trivial to analyze the logs maintained in various levels of the stack. Independent analysis might lead to an incomplete conclusion due to the limited coverage of each log. This work takes a comprehensive approach to analysis that incorporates the logs in the multiple layers and components, in order to facilitate the detection of anomalous activities. This research aims to identify and predict potential performance bottlenecks in the HPC system, by capturing the temporal variation patterns from heterogeneous, high-dimensional, and non-linear log data. In this paper, we share our preliminary efforts for spatio-temporal analysis of HPC I/O and connection data, with our initial observations from the analysis of one-week HPC log data sets collected from one of NERSC systems.

Choi, Jinhwan↗

DOE OSTI2019

Enhancing trap and fault seal analyses by integrating observations from HR3D seismic data with well logs and conventional 3D seismic data, Texas inner shelf

An understanding of trap and fault seal quality is critical for assessing hydrocarbon prospectivity. To achieve this, modern analytical techniques leverage well data and conventional industry-standard 3D seismic data to evaluate the trap, and any faults displacing the reservoir and top seal intervals. Above all, geological interpretation provides the framework of trap and fault seal analyses, but can be hindered by the data resolution, quality and acquisition style of the conventional seismic data. Furthermore, limiting the analysis to only the petroleum system at depth may lead to erroneous perceptions because interpreting overburden features, such as shallow faults or gas chimneys, can provide valuable observations with respect to container performance, and can to help validate trap and fault seal predictions. Here, a supplement to conventional 3D data are high-resolution 3D seismic (HR3D) data, which provide detailed images of the overburden geology. This study utilizes an HR3D seismic volume in the San Luis Pass area of the Texas inner shelf, where shallow fault tips and a sizeable gas chimney are interpreted over an unsuccessful hydrocarbon prospect. Static post-drill fault seal and trap analyses suggest that the primary fault displacing the structural closure could have withheld columns of gas c. 100 m high, but disagree with our HR3D seismic interpretations and dry-well analyses. From our results, we hypothesize that tertiary gas migration through fault conduits reduced the hydrocarbon column in the prospective Early Miocene reservoir, and may have resulted from continued movement along the intersecting faults. Overall, this study reinforces the importance of understanding the overburden geology and geohistory of faulted prospects, and demonstrates the utility of pre-drill HR3D acquisition when conducting trap and fault seal analyses.

3D seismic↗

DOE OSTI2025

Electricity Baseline 2021 Background Data and Log File

The ElectricityLCI v2 Python package (https://github.com/USEPA/ElectricityLCI/tree/v2.0) was used to generate the 2021 electricity baseline: a regionalized life cycle inventory model of U.S. electricity generation, consumption, and distribution using standardized facility and generation data. ElectricityLCI implements a local data store for downloading and accessing public data on an individual's computer. The data store follows the folder definition provided by USEPA's esupy Python package (https://github.com/USEPA/esupy), which utilizes the appdirs Python dependency (https://pypi.org/project/appdirs/). An overview of the ElectricityLCI data stores may be found on the README (https://github.com/USEPA/ElectricityLCI/blob/v2.0/README.md#data-store). This submission includes the background data used to generate the 2021 electricity baseline inventory. Each zip archive stores the source files as found in their data stores. Sub-folders in each of the data stores are archived separately. For example, stewi.zip contains the JSON files, while stewi.facility.zip is the 'facility' sub-folder of stewi data store that stores the parquet files. To reproduce the data store, extract each zip file and drag-and-drop sub-folders in to their appropriate root folders to recreate the data stores, then copy the root folders to your data store folder (as returned by running the following on the command line: python -c "import appdirs; print(appdirs.user_data_dir())"). The main five data stores include: 'electricitylci', 'facilitymatcher', 'fedelemflowlist', 'stewi', and 'stewicombo'. The log file generated by the 2021 model run is also included, which contains the statements at the DEBUG level and above.

Electricity; LCA; LCI; Life Cycle; data inventory↗

DOE OSTI2025

Electricity Baseline 2020 Background Data and Log File

The ElectricityLCI v2 Python package (https://github.com/USEPA/ElectricityLCI/tree/v2.0) was used to generate the 2020 electricity baseline: a regionalized life cycle inventory model of U.S. electricity generation, consumption, and distribution using standardized facility and generation data. ElectricityLCI implements a local data store for downloading and accessing public data on an individual's computer. The data store follows the folder definition provided by USEPA's esupy Python package (https://github.com/USEPA/esupy), which utilizes the appdirs Python dependency (https://pypi.org/project/appdirs/). An overview of the ElectricityLCI data stores may be found on the README (https://github.com/USEPA/ElectricityLCI/blob/v2.0/README.md#data-store). This submission includes the background data used to generate the 2020 electricity baseline inventory. Each zip archive stores the source files as found in their data stores. Sub-folders in each of the data stores are archived separately. For example, stewi.zip contains the JSON files, while stewi.facility.zip is the 'facility' sub-folder of stewi data store that stores the parquet files. To reproduce the data store, extract each zip file and drag-and-drop sub-folders in to their appropriate root folders to recreate the data stores, then copy the root folders to your data store folder (as returned by running the following on the command line: python -c "import appdirs; print(appdirs.user_data_dir())"). The main five data stores include: 'electricitylci', 'facilitymatcher', 'fedelemflowlist', 'stewi', and 'stewicombo'. The log file generated by the 2020 model run is also included, which contains the statements at the DEBUG level and above.

Electricity; LCA; LCI; Life Cycle; data inventory↗

DOE OSTI2025

Electricity Baseline 2022 Background Data and Log File

The ElectricityLCI v2 Python package (https://github.com/USEPA/ElectricityLCI/tree/v2.0) was used to generate the 2022 electricity baseline: a regionalized life cycle inventory model of U.S. electricity generation, consumption, and distribution using standardized facility and generation data. ElectricityLCI implements a local data store for downloading and accessing public data on an individual's computer. The data store follows the folder definition provided by USEPA's esupy Python package (https://github.com/USEPA/esupy), which utilized the appdirs Python dependency (https://pypi.org/project/appdirs/). This submission includes the background data used to generate the 2022 electricity baseline inventory. Each zip archive stores the source files as found in their data stores. Sub-folders in each of the data stores are archived separately. For example, stewi.zip contains the JSON files, while stewi.facility.zip is the 'facility' sub-folder of stewi data store that stores the parquet files. To reproduce the data store, extract each zip file and drag-and-drop sub-folders in to their appropriate root folders to recreate the data stores, then copy the root folders to your data store folder (as returned by running the following on the command line: `python -c "import appdirs; print(appdirs.user_data_dir())"`). The main five data stores include: 'electricitylci', 'facilitymatcher', 'fedelemflowlist', 'stewi', and 'stewicombo'. The log file generated by the 2022 model run is also included, which contains the statements at the DEBUG level and above.

Electricity; LCA; data inventory↗